- Add replayed_from_friendly_id column to ClickHouse task_runs_v2 table
- Create migration 012_add_task_runs_v2_replayed_from.sql
- Update replication service to include the new field
- Add getRunReplays query to ClickHouse client (optimized with org/project/env filter)
- Update SpanPresenter to query replays from ClickHouse
- Add "Replayed as" section in run Details tab showing linked replay runs with status
The ClickHouse query filters by organization_id, project_id, and environment_id
in the correct order to match the primary key for optimal query performance.
This enables users to see which runs have been replayed from the original run,
addressing the feedback about tracking replay status of failed runs.
Slack thread: https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1767609682537389
Closes #<issue>
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
_[Describe the steps you took to test this change]_
---
## Changelog
_[Short description of what has changed]_
---
## Screenshots
_[Screenshots]_
💯
This fixes a regression introduced in #2778 - stable sort is required
for deterministic builds, but we can safely preserve order for the user
package.json during package updates
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/build@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## trigger.dev@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
- `@trigger.dev/build@4.3.1`
- `@trigger.dev/schema-to-json@4.3.1`
## @trigger.dev/core@4.3.1
### Patch Changes
- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))
## @trigger.dev/python@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.3.1`
- `@trigger.dev/core@4.3.1`
- `@trigger.dev/build@4.3.1`
## @trigger.dev/react-hooks@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/redis-worker@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/rsc@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/schema-to-json@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/sdk@4.3.1
### Patch Changes
- feat: Support for new batch trigger system
([#2779](https://github.com/triggerdotdev/trigger.dev/pull/2779))
- feat(sdk): Support debouncing runs when triggering with new debounce
options
([#2794](https://github.com/triggerdotdev/trigger.dev/pull/2794))
- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Add support for resetting idempotency keys both from ui and sdk
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Created a new run with a idempotency idempotencyKey.
- Started a new run with the same task and got redirected to the first
run.
- Deleted the key from the UI on the run details
- Started a new run with the same task and it created a new one
- Did the above steps using the SDK
---
## Changelog
- Add new action route for resetting idempotency keys via UI
- Add reset button in Idempotency section of run detail view
- Added API and SDK for resetting imdepotency
- Updated docs page for this feature
---
## Screenshots
_[Screenshots]_
<img width="438" height="363" alt="Screenshot 2025-12-11 at 11 56 37"
src="https://github.com/user-attachments/assets/30b8ef5e-8aac-4d04-b57a-9bf30d085dcb"
/>
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.
## Usage
```typescript
await myTask.trigger({ userId: "123" }, {
debounce: {
key: "user-123-update",
delay: "5s",
mode: "leading", // default
}
});
```
- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.
### "trailing" mode overrides
When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:
- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`** - Machine preset (cpu/memory)
## Behavior
- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)
Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.
This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.
For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.
## New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:
- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200
This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.
## Batch queue concurrency limits
The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.
There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.
## Batch trigger settings
- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue
### Configuration Recommendations by Use Case
**High-throughput priority (fairness acceptable at 0.98+):**
```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```
**Strict fairness priority (throughput can be lower):**
```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
## Changelog
- Add disableAdjacentRows prop to TaskRunsTable component to control
table state encoding
- Pass rootOnlyDefault prop from loader to TaskRunsTable for proper
state management
- Disable adjacent run navigation in schedule, waitpoint, and other
inspector views
- Preserve adjacent run navigation on main runs list page with rootOnly
filter support
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## trigger.dev@4.3.0
### Minor Changes
- feat(cli): deterministic image builds for deployments
([#2778](https://github.com/triggerdotdev/trigger.dev/pull/2778))
- feat(cli): enable zstd compression for deployment images
([#2773](https://github.com/triggerdotdev/trigger.dev/pull/2773))
### Patch Changes
- The new `triggeredVia` field is now populated in deployments via the
CLI. ([#2767](https://github.com/triggerdotdev/trigger.dev/pull/2767))
- fix(dev): stop max listeners exceeded warning messages when running
more than 10 runs concurrently
([#2771](https://github.com/triggerdotdev/trigger.dev/pull/2771))
- Upgrade @modelcontextprotocol/sdk to 1.24.3
([#2768](https://github.com/triggerdotdev/trigger.dev/pull/2768))
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
- `@trigger.dev/build@4.3.0`
- `@trigger.dev/schema-to-json@4.3.0`
## @trigger.dev/core@4.3.0
### Minor Changes
- feat(cli): deterministic image builds for deployments
([#2778](https://github.com/triggerdotdev/trigger.dev/pull/2778))
### Patch Changes
- The new `triggeredVia` field is now populated in deployments via the
CLI. ([#2767](https://github.com/triggerdotdev/trigger.dev/pull/2767))
## @trigger.dev/build@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/python@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
- `@trigger.dev/build@4.3.0`
- `@trigger.dev/sdk@4.3.0`
## @trigger.dev/react-hooks@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/redis-worker@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/rsc@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/schema-to-json@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/sdk@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This PR makes our image builds deterministic and reproducible by
ensuring that identical source code always produces the same image
layers and image digest. This means that deployments where nothing has
changed will no longer invalidate the image cache in our worker cluster
nodes, thus avoid making the cold starts for runs worse.
**Context**
New deployments currently increase the cold start times for runs, as
they generate a new image which needs to be pulled in the worker cluster
where runs are executed. It happens also when the source code for the
deployment has not changed due to non-deterministic steps in our build
system. This addresses the latter issue by making builds reproducible.
**Main changes**
- Avoided baking `TRIGGER_DEPLOYMENT_ID` and
`TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the
supervisor instead.
- Used `json-stable-stringify` for consistent key ordering in the files
we generate for the build, e.g., `package.json`, `build.json`,
`index.json`.
- Removed `metafile.json` from the image contents as it is not actually
used in the container. This is only relevant for the `analyze` command.
- Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker
builds to normalize file timestamps.
- Removed some `timings` and `outputHashes` from build outputs and
manifests.
The builds are now reproducible for both native build server and Depot
paths. This should also lead to better image layer cache reuse in
general.
- Add replace prop to LinkButton to use history replacement for adjacent
run navigation
- Preserve span and tab params when navigating between adjacent runs
- Disable animations for completed spans in timeline to improve
performance
- Include spanId in runs list navigation for better context preservation
- Direct link to task test page when filtering by single task with no
runs
- Fix minor styling issue with run friendlyId display padding
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Set up the local github application, and tested its connection with
trigger.dev
Checked:
- Change is backwards compatible
- Actions/ux are uniform across the pages (empty Tasks, Deployments,
project settings)
- Connecting GH, Connecting Repo, disconnecting Repo, modifying settings
---
## Changelog
- Create new resource route for GitHub settings management with loader
and actions
- Add GitHubSettingsPresenter to fetch connected repos and installations
- Implement GitHubSettingsPanel component for reusable GitHub
configuration UI
- Refactor project settings page to use shared GitHubSettingsPanel
component
- Integrate GitHub connection flow into empty state onboarding for Tasks
and Deployments
- Add support for GitHub repo connection, disconnection, and branch
tracking settings
- Include redirect URL support for seamless navigation after GitHub
actions
- Remove duplicate GitHub connection code from project settings route
---
## Screenshots
https://github.com/user-attachments/assets/8fc24699-640b-4f9e-afd8-b26edc945218🐐
---------
Co-authored-by: James Ritchie <james@trigger.dev>
This will speed up ice cold starts (*) for two reasons:
- better compression ratio
- faster decompression
This is a minor release because zstd compression will now be enabled by
default for all deployments.
(*) ice cold starts happen when deploy images are not cached on the
worker node yet. These cold start durations are highly dependent on
image size and as it turns out, also the type of compression used.
Fixed the issue where Cmd+Left Arrow was being intercepted by the
TreeView component on task runs screen.
Solution:
Added a check in the getTreeProps keyboard handler to detect when
metaKey (Cmd on macOS) is pressed with Left Arrow. When detected, the
handler returns early without preventing the default browser behavior,
allowing Chrome's native back navigation to work.
Closes #<issue>
## ✅ Checklist
- [ x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ x] The PR title follows the convention.
- [ x] I ran and tested the code works
---
## Testing
Went to the task details page and confirmed that Cmd + Left Arrow will
navigate back
---
## Changelog
Modified TreeView.tsx to check for e.metaKey before handling Left Arrow
key events
When Cmd+Left is pressed, the event is no longer prevented, allowing
browser default behavior
---
Co-authored-by: Mihai Popescu <mihaipopescu@Mihais-MacBook-Pro.local>
Deployments are affected by general API rate limits, this is just a
quick fix by whitelisting the deployment related endpoints. In a follow
up PR we'll add a separate rate limiter for this group of endpoints.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Manual testing of the task run pages
---
## Changelog
- Add previous/next run navigation buttons to run detail page header
- Support [ and ] keyboard shortcuts to jump between adjacent runs
- Preserve runs table state (filters, pagination) when navigating
- Preload adjacent page runs at boundaries for seamless navigation
- Add actions prop to PageTitle component
- Document shortcut in keyboard shortcuts panel
- Store current filter state from runs table as `tableState` search
param when navigating to individual run pages
- Restore filters when navigating back from run detail view to runs list
- Update `v3RunPath` and `v3RunSpanPath` helpers to accept optional
searchParams
- Use `useOptimisticLocation` to capture current search params in
TaskRunsTable
- Parse `tableState` param in run detail route and pass filters to back
button
- This improves UX by remembering filter selections (task, status, date
range, etc.) when users click into a run and then navigate back to the
runs list
- Add new text-below variant that shows "Click to copy" tooltip on hover
and "Copied" on click. Also add controlled open/onOpenChange props to
SimpleTooltip for managing tooltip visibility.
---
## Screenshots
https://github.com/user-attachments/assets/5067bbe0-1bcd-4e75-80a7-f56dabd5ed69
Changes in this PR:
- Arbitrary refs are now allowed when triggering the release workflow
manually (ref must be on the main branch).
- Release summary is now displayed in the GH job output; makes for a
nicer experience when approving the release workflow.
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Updates `BatchTriggerV3Service` to send `batch.id` (not `friendlyId`)
as `batchId` to `TriggerTaskService.call`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
68bf8df4a29bdcac5bfb2806bc417541f69ffd6e. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Centralizes SIGTERM handling in `DevSupervisor` and removes per-run
SIGTERM listeners in `DevRunController` to avoid
MaxListenersExceededWarning under high concurrency.
>
> - **Dev runtime**:
> - **SIGTERM handling**: Add centralized handler in
`packages/cli-v3/src/dev/devSupervisor.ts` to gracefully stop all run
controllers; unregisters on `shutdown()`.
> - **Cleanup**: Remove per-controller `SIGTERM` listener and handler
from `packages/cli-v3/src/entryPoints/dev-run-controller.ts` to reduce
event listeners and warnings.
> - **Changeset**: Add patch note in
`.changeset/fuzzy-ghosts-admire.md`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5ad2f5341829cebf6fd37a3c616a2db5e4ad936a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
This PR applies a small change to the deployments table to keep track
of:
- where the deployment was triggered from
- build server metadata, if the build server was involved
We recently upgraded the ECR sdk version. Our ECR repo exists check
relies on the type of the error thrown and the new ECR sdk version seems
to have broken that behavior. This PR adds a workaround to the issue.
The manual trigger is currently enabled only for the prerelease job.
This PR adds it for the normal release flow too, as it is useful to
retrigger failed release workflows.
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## trigger.dev@4.2.0
### Minor Changes
- feat(cli): upgrade bun deployments to v1.3.3
([#2756](https://github.com/triggerdotdev/trigger.dev/pull/2756))
### Patch Changes
- fix(otel): exported logs and spans will now have matching trace IDs
([#2724](https://github.com/triggerdotdev/trigger.dev/pull/2724))
- The `--force-local-build` flag is now renamed to just `--local-build`
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
- fix(cli): header will always print the correct profile
([#2728](https://github.com/triggerdotdev/trigger.dev/pull/2728))
- feat: add ability to set custom resource properties through
trigger.config.ts or via the OTEL_RESOURCE_ATTRIBUTES env var
([#2704](https://github.com/triggerdotdev/trigger.dev/pull/2704))
- feat(cli): implements content-addressable store for the dev CLI build
outputs, reducing disk usage
([#2725](https://github.com/triggerdotdev/trigger.dev/pull/2725))
- Added support for native build server builds in the deploy command
(`--native-build-server`)
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
- Updated dependencies:
- `@trigger.dev/build@4.2.0`
- `@trigger.dev/core@4.2.0`
- `@trigger.dev/schema-to-json@4.2.0`
## @trigger.dev/build@4.2.0
### Patch Changes
- syncVercelEnvVars to skip API and read env vars directly from
env.process for Vercel build environments. New syncNeonEnvVars build
extension for syncing environment variablesfrom Neon database projects
to Trigger.dev. The extension automatically detects branches and builds
appropriate PostgreSQL connection strings for non-production, non-dev
environments (staging, preview).
([#2729](https://github.com/triggerdotdev/trigger.dev/pull/2729))
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/core@4.2.0
### Patch Changes
- fix: prevent ERR_IPC_CHANNEL_CLOSED errors from causing an unhandled
exception on TaskRunProcess
([#2743](https://github.com/triggerdotdev/trigger.dev/pull/2743))
- Added support for native build server builds in the deploy command
(`--native-build-server`)
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
## @trigger.dev/python@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.2.0`
- `@trigger.dev/sdk@4.2.0`
- `@trigger.dev/core@4.2.0`
## @trigger.dev/react-hooks@4.2.0
### Patch Changes
- fix: prevent infinite useEffect when passing an array of tags to
useRealtimeRunsWithTag
([#2705](https://github.com/triggerdotdev/trigger.dev/pull/2705))
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/redis-worker@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/rsc@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/schema-to-json@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/sdk@4.2.0
### Patch Changes
- fix(sdk): Re-export schemaTask types to prevent the TypeScript error
TS2742: The inferred type of 'task' cannot be named without a reference
to '@trigger.dev/core/v3'. This is likely not portable.
([#2735](https://github.com/triggerdotdev/trigger.dev/pull/2735))
- feat: add ability to set custom resource properties through
trigger.config.ts or via the OTEL_RESOURCE_ATTRIBUTES env var
([#2704](https://github.com/triggerdotdev/trigger.dev/pull/2704))
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
**Background**
Currently the changeset PR creation and the publishing is handled by the
same workflow. This is not ideal:
- The build steps are executed on every run of the pipeline, even though
they're only needed for the publish case.
- The PR creation workflow does not need permissions to publish to npm,
only the release path needs them.
- Adding an approval step is painful as we'd need to also approve each
changeset PR creation workflow run.
**Changes in this PR**
- Separated the changeset PR creation into its own workflow and minimum
permission set.
- Added a GH environment with an approval step for the package
publishing workflow (also for prereleases).
- New publish workflow runs will not cancel in-progress runs; helps
avoid partial failures in publishing.
These changes also enable hardening the npm OIDC setup by tying it to a
GH environment that requires approval.
Vercel's NeonDB integration renders database connection environment
variables at runtime, which means Trigger.dev cannot directly sync these
values during the build process. This change adds support for fetching
branch-specific NeonDB connection strings via the Neon API.
feat(build): Add syncNeonEnvVars extension and improve Vercel env var
syncing
Add a new `syncNeonEnvVars` build extension for syncing environment
variables
from Neon database projects to Trigger.dev. The extension automatically
detects
branches and builds appropriate PostgreSQL connection strings for
non-production
environments (staging, dev, preview).
Features of `syncNeonEnvVars`:
- Fetches branch-specific database credentials from Neon API
- Generates all standard Postgres connection strings (DATABASE_URL,
POSTGRES_URL,
POSTGRES_PRISMA_URL, etc.) with both pooled and unpooled variants
- Supports custom database name, role name, and env var prefix options
- Skips automatically in Vercel environments (Neon's Vercel integration
handles this)
- Skips for production environments (designed for preview/staging/dev
branches)
Improvements to `syncVercelEnvVars`:
- When running in a Vercel build environment (detected via VERCEL env
var),
values are now read from process.env instead of the Vercel API response
- This ensures the build uses the actual runtime values Vercel provides
- Removed embedded Neon-specific logic (now handled by separate
extension)
- Simplified and cleaned up the extension code
Documentation updates for both extensions with usage examples and
configuration
options.
Closes#2714
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Set up Vercel + Trigger.dev envs, used Vercel's chat-bot-ai template.
When viewing runs filtered to a single task, the "Create a test run" and
"Run a test" buttons now navigate directly to the task-specific test
page instead of the generic test page.
This improves UX by pre-populating the test form with the filtered task,
saving users from having to manually select it again.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Screenshots
<img width="1706" height="1392" alt="image"
src="https://github.com/user-attachments/assets/d8b5d445-73b5-426c-83a4-90ac2a95b955"
/>
Add validation for logical replication publication configuration. Helps
diagnose an issue where runs are no longer replicated to clickhouse
because of a configuration issue with the replication publication.
## Problem
The `LogicalReplicationClient` only checked if a publication existed,
not if it was correctly configured. This caused a silent failure where:
- Replication would start successfully
- Transaction boundaries (begin/commit) were received
- **But no actual data changes were replicated**
This happened when a publication existed but:
1. Had no tables associated with it
2. Was missing required actions (e.g., `delete`)
## Solution
Added `#validatePublicationConfiguration()` method that validates:
- ✅ Publication includes the expected table
- ✅ Publication has all required actions configured
When validation fails, error messages include the exact SQL command to
fix the issue:
**Missing table:**
```
Publication 'task_runs_to_clickhouse_v1_publication' exists but has NO TABLES configured.
Expected table: "public.TaskRun".
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication ADD TABLE "TaskRun";
```
**Missing actions:**
```
Publication 'task_runs_to_clickhouse_v1_publication' is missing required actions.
Expected: [insert, update, delete], Current: [insert, update], Missing: [delete].
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication SET (publish = 'insert, update, delete');
```
This prevents silent data loss and makes debugging configuration issues
much easier.
Adds a manual trigger to the `release.yml` workflow for publishing
prerelease versions. Needs to be in the same workflow file due to a NPM
limitation on how OIDC claims are checked.
Currently there is a validation step on the ref for the prerelease: it
must be merged to the main branch. We can revisit this in the future in
case we find it too limiting.
To avoid doing a migration for the ECR repo cache settings, we lazily do
it on the next deployment for that project. Failures to update the repo
settings are just logged and will not cause the deployment to fail.
Fixes this type of error when exporting a `schemaTask` in a monorepo:
```
error TS2742: The inferred type of 'helloWorldSchema' cannot be named without a reference to '@trigger.dev/core/v3'. This is likely not portable.
```
This PR adds support for CLI deployments using the native build server.
**Background**
The deployment command currently does the following:
- bundles the code
- submits the build context to our external build provider and waits for
the build
- triggers deployment state transitions using the platform API
Upstream build provider outages cause issue with deployments,
potentially blocking deployments entirely. We recently introduced the
`--force-local-build` flag as a fallback to enable deployment without a
dependency on the upstream build provider, though it requires users to
have docker in their systems. This PR continues that work by providing a
remote build path which uses our own build server and does not rely on
the external provider.
**Changes in this PR**
Introduced the new `--native-build-server` flag, which does the
following:
- scans all files relevant for the Trigger deployment and evaluates
ignore rules
- packages it up in an archive and uploads it as a deployment artifact
- queues the deployment and triggers the build
- streams logs from the build server
This no longer relies on external build services. Also deployment state
transitions happen on the server-side, giving us more flexibility to
evolve the flow and schemas of related deployment API endpoints. In
general it gives us better control of the whole build and deployment
process. This path will eventually become the default.
The `--detach` flag is also new, allowing to trigger deployments without
waiting for the result.
The deployment artifacts are uploaded via pre-signed URLs to avoid
unnecessary load on the platform. The new `/artifacts` endpoint
generates the pre-signed URLs; size limits are enforced on s3. This
endpoint is deliberately generic, we could extend it in the future to
upload other artifacts client-side in a similar way, e.g., large payload
packets.
This is a step which we currently need to do manually and it's rather
painful. The lockfile update is necessary due to cross references in our
packages.
Added it as a separate job instead of a step to start from fresh
workspace, as the state that the `changeset` step leaves the workdir is
not explicitly clear to the reader.
Add a new `syncNeonEnvVars` build extension for syncing environment variables
from Neon database projects to Trigger.dev. The extension automatically detects
branches and builds appropriate PostgreSQL connection strings for non-production
environments (staging, dev, preview).
Features of `syncNeonEnvVars`:
- Fetches branch-specific database credentials from Neon API
- Generates all standard Postgres connection strings (DATABASE_URL, POSTGRES_URL,
POSTGRES_PRISMA_URL, etc.) with both pooled and unpooled variants
- Supports custom database name, role name, and env var prefix options
- Skips automatically in Vercel environments (Neon's Vercel integration handles this)
- Skips for production environments (designed for preview/staging/dev branches)
Improvements to `syncVercelEnvVars`:
- When running in a Vercel build environment (detected via VERCEL env var),
values are now read from process.env instead of the Vercel API response
- This ensures the build uses the actual runtime values Vercel provides
- Removed embedded Neon-specific logic (now handled by separate extension)
- Simplified and cleaned up the extension code
Documentation updates for both extensions with usage examples and configuration
options.
* Add the release version to changeset PRs
* Add missing id-token permission, needed for oidc
* Remove a couple of unnecesary steps
* Reference the `changeset-release/main` branch explicitly
Vercel's NeonDB integration renders database connection environment
variables at runtime, which means Trigger.dev cannot directly sync
these values during the build process. This change adds support for
fetching branch-specific NeonDB connection strings via the Neon API.
Changes:
- Discover NEON_PROJECT_ID from incoming Vercel environment variables
- Call NeonDB API to search for branches matching the git branch name
- Filter branches to find exact matches with Vercel environment prefix
(e.g., "preview/branch-name") to avoid false positives from partial
string matches
- Retrieve branch endpoints and select the write endpoint (or first
available)
- Build connection strings (DATABASE_URL, POSTGRES_URL, etc.) using
the branch endpoint host while preserving user/password credentials
Safety measures for non-production environments:
- Filter out all Neon-related env vars (DATABASE_URL, PGHOST, etc.)
before calling the Neon API to prevent accidental use of production
database credentials
- Only add branch-specific database env vars if a matching Neon branch
is found and the API call succeeds
- If neonDbAccessToken is not provided or the API fails, non-production
environments will not receive any database connection env vars
Usage:
Users must provide a NEON_ACCESS_TOKEN (via options or env var) to
enable automatic branch resolution for preview deployments. Production
environments continue to use Vercel's standard env var sync without
modification.
schedule spans can sometimes show as generic spans when using the
task_events_v2 table because of the inserted_at filter. Increasing the
buffer for the start time does the trick and doesn’t cause any perf
Issues (and is in general just more robust)
When using custom OTLP exporters via `telemetry.exporters` and
This occurred when tasks were triggered **without** a parent trace
context (e.g., via API or dashboard). In this scenario: - Spans were
correctly rewritten to use the generated `externalTraceId` - Logs kept
their original internal trace ID due to a bug in the early return logic
### Root Cause
In `ExternalLogRecordExporterWrapper.transformLogRecord()`, the early
return condition incorrectly included `!this.externalTraceContext`:
```typescript
if (!logRecord.spanContext || !this.externalTraceId ||
!this.externalTraceContext) { return logRecord; // Bug: Returns early
when externalTraceContext is undefined }
// This fallback logic was never reached:
const externalTraceId = this.externalTraceContext
? this.externalTraceContext.traceId
: this.externalTraceId;
```
### Fix
1. **Reordered logic in `transformLogRecord()`**: Move the
1. `externalTraceId` calculation before the early return, and check the
1. culated value instead of `this.externalTraceContext`:
```typescript
const externalTraceId = this.externalTraceContext
? this.externalTraceContext.traceId
: this.externalTraceId;
if (!logRecord.spanContext || !externalTraceId) {
return logRecord;
}
```
2. **Clarified `_isExternallySampled` logic**: Updated both
2. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper`
2. explicitly handle the case where there's no external trace context
2. a generated `externalTraceId` exists:
```typescript
this._isExternallySampled = externalTraceContext
? isTraceFlagSampled(externalTraceContext.traceFlags)
: !!externalTraceId;
```
### Impact
Logs and spans from the same task run will now have matching trace IDs
when exported to external observability tools, enabling proper trace correlation regardless of whether the task was triggered with or without a parent trace context.
`telemetry.logExporters` in `trigger.config.ts`, logs and spans were
exported with **different trace IDs**, breaking trace correlation in
external observability tools like Datadog.
* stop deleting the first dev version files on the first change, prevents system failures
* prevent dev runs getting stuck in dequeued status by deleting workers
* add changeset
This PR fixes the `x-trigger-branch` header support for targeting specific preview branches when managing environment variables. The header was documented but not actually being extracted or used in the environment variable API routes. Additionally, the query logic in `authenticatedEnvironmentForAuthentication` was fundamentally broken—it searched for environments with both `slug: "preview"` (parent environment property) AND a specific `branchName` (child environment property), which no environment could satisfy simultaneously. The fix extracts the branch name using `branchNameFromRequest()` and correctly queries for child branch environments using `type: "PREVIEW"` and the specific `branchName`. This ensures that environment variable operations (create, update, get, list) properly target individual preview branches instead of affecting all preview environments.
* Don't use the organization max concurrency anymore
* Early draft of the concurrency page
* WIP adding a new stepper input component
* Move stepper to be alphabetical
* When max value is reached, disabled the + button
* Show placeholder if you delete all numbers
* Make all the html input values available to the component
* Adds size variants
* Move stepper into its own component
* Work on showing the extra concurrency
* The purchase form styling and functionality (minus actually purchasing)
* New style for outline input fields
* Concurrency purchasing working
* Purchasing concurrency and quota emails working
* Improvements to the modal
* Show cost breakdown in the modal
* Fix for allocated concurrency including DEV
* Improved types
* Allocating concurrency is working
* Live updates total env concurrency
* Implemented reset
* Fix for concurrency allocation editing across multiple projects
* Tabular numbers
* Added an error from allocating concurrency
* Fixes for allocating concurrency where it didn't calculate correctly
* "Increase limit" link to concurrency page
* Indent environments
* Added Preview limit when updating concurrency for an org
* Show error when changing plan fails
* Added maximumProjectCount column to Org
* Limit project count and display a rich error toast (with title and button now)
* Added title and button to toasts. Use it for new project error
* @trigger.dev/platform 1.0.20
* Allow submitting zero concurrency so you can downgrade back to nothing
* Use the server as the truth for omitted environments
* Updated the pricing panels
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* prisma extension fixes WIP
* More prisma stuff
* more prisma stuff
* remove changelog
* upgrade github workflows to use node 20.19 because installing prisma@7 breaks with lower versions
* Don't use generate for the prisma reference projects
* make sure it works if no mode is passed in
* Update aws sdk ecr client to the latest version
* Exlude the cache tag from the immutability enforcement
* Attach a policy to ECR repos to expire untagged images
* Fix filterType
* Fix for the MCP tool that gets logs for debugging runs
This was broken when we changed the data on the backend that returns
log/span data from runs. We changed the data structured and the internal
API that the MCP client uses was failing to parse with the Zod schema
* add changeset
* Revert "add changeset"
This reverts commit 86eca836d5907fa0d0f8ac595d4d5ebade140514.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Use read-only project-scoped s2 tokens for streaming deployment logs
* Add http2 to remix polyfills
Needed for using s2 client-side.
* Stream build-server logs in the deployment details page
* Disable 12-hour format in the DateTime component
* Enable collapsing the logs panel
* Auto-collapse logs for succesful/timedout/queued deployments
* Make S2 env vars optional
* Show the logs section only for gh-triggered deployments
* Cache s2 access tokens in redis
* Reset streaming state
* Expose 12h format as a param for the Datetime components
* docs: deployments with local builds
Adds a section to the deployment docs page about the option to deploy using a locally built image.
* Update installation link
* Add an API endpoint to query remote build provider status
* Show local build hint for failed deployments when Depot is down
* Show the local build flag in the help output
* Add changeset
* Fix import
* Fix docs link
* chore(runner): move max duration logic into parent process
* chore(rsc): remove type-marker package.json
* add changeset
* chore(core): remove irrelevant test after our changes
* chore(core): clarify we don't care about the timeout promise
* fix(webapp): display correct concurrency override base value
* fix(webapp): persist concurrency overrides on deploy
* fix(webapp): use correct override base value type
* fix(webapp): override input is bounded by env concurrency
* Enable skipping image push during deployment finalization step
* Add endpoint to generate registry credentials for a deployment
* Add a --force-local-build flag to the deployment command to skip remote build
* Do not show the new flag in the help output
* Add changeset
* Remove registry login logs from onLog, not useful
* Rename var
* Update platform package to the latest version
* Limit local dev concurrency using the dev environment concurrency limit
Previously it was limited to max of 25, no matter the environment limit
* Have global dev limit
The Postgres query to get the latest run ids for the test page was very
slow when there were a lot of runs and/or versions.
This now uses the standard runs list we use everywhere else.
* feat(queues): add ability to override concurrency limit via API and dashboard
* Updates the modal layout and tweaks copy
* Improves the dropdown menu item
* Popover supports both Button and LinkButton
* Right align the columns and fix the dropdown menu item styles
* Organize imports,
* Fix spinner icon in dropdown menu
* Remove unused props
* Adds a tooltip to the Concurrency override badge
* Fixes console error with popover menu
* typo
* Fixes incorrect className
* Minimal buttons to view runs
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* WIP using ClickHouse for the tags filter list
* WIP on tags listing
* Webapp: exclude test files when typechecking
* Tags filtering working with CH
* Remove unused import
* The AI filter should only look at the last past 30d of tags
* Do the text query in ClickHouse
* Deal with encoded characters better
* More encoding fixes
* Fix for wrong items being checked
* Put applied tags back
* Add the env.id to the dependencies array
* chore(billing): improve logs to distinguish between failure modes
* fix(engine): default to paid placement on billing errors
* chore(engine): set plan type according to paying field when missing
- Move S3 credentials from secrets.objectStore to s3.auth
- Update external PostgreSQL config to use databaseUrl/directUrl approach
- Add existingSecret support for PostgreSQL with secretKeys
- Add TLS configuration for external Redis
- Add existingSecret support for Redis, ClickHouse and S3
- Add complete external S3 configuration example
- Improve secure credential management documentation
These changes align the production example file with the current values.yaml
structure, making it easier to configure external services with better
support for secret management.
* Allow shortcuts hook to work if undefined
* Conditionally show shortcut button if only 1 result
* LinkButton can accept conditionally shown shortcuts
* docs: deploying using the github integration
* Add hint in the gh actions docs page
* Remove extra space
* Add a couple of hints to the build config fields
* feat(build-server): add option to specify pre-build command
Adds an option to specify a pre-build command in the build settings. Can
be useful for projects that need a step before the build, e.g., to
generate a prisma client.
Also, remove the install directory in favor of simplicity. Both
pre-build and install commands are run from the root of the repo. Users
that need to run the commands in a different dir can just prepend to the
command, e.g., `cd apps/web && pnpm run primsa:migrate`
* Fix spelling
* Show hint if preview branches are disabled in the project
* Enable preview deployments only if the preview environemtn is enabled
* Fix prisma reference
* fix(otel): prevent unpaired unicode surrogate pairs from causing insert errors
* only check parts of the string that are not going to get truncated
remove unnecessary taks
* fix(charts): use bitnamilegacy repo for MinIO Object Browser
The Bitnami `minio-object-browser` image has been removed from Docker
Hub. This patch updates the Trigger Helm chart to reference the
`bitnamilegacy/minio-object-browser` repository under the
`s3.console.image` section, ensuring that the legacy MinIO browser pod
can still be deployed without image pull errors.
All other MinIO components continue using the `bitnamilegacy/minio`
image for consistency across Bitnami Legacy dependencies.
* chore(helm): bump Helm chart version to 4.0.4
The Depot build init with `depot.build.v1.BuildService.createBuild` fails surprisingly often due to transient errors, causing the whole deployment to fail. This PR adds a simple retry mechanism with backoff using p-retry. This should improve the failure rate.
* chore(docker): use bitnami legacy repo
* chore(helm): use bitnami legacy repo
* Make Helm webapp chart images configurable
Adds configurability for init and token syncer container images through
new values in the Helm chart configuration
* chore(helm): refactor utility image config
* chore(helm): bump chart version to 4.0.3
---------
Co-authored-by: LeoKaynan <leokaynan@hotmail.com>
* feat(webapp): rate limit magic-link login attempts
Adds a simple rate limiter to the login with magic link flow. Similar implementation to the MFA rate limits.
* Fix error message
* Add an env var feature flags for login rate limiting
* Use BoolEnv instead of `0`/`1`
* Parse xff properly
* feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing
feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status
* Handle FINISHED snapshot in the repair
* fix: use higher entropy invite tokens
We currently use CUIDs for invite tokens, which are generated using
a pattern and are not cryptographically secure. This PR switches to
a higher entropy string generated with `nanoid`.
* Dedupe the invite emails in the application
* fix: org scoping in the select plan flow
Adds proper org scoping in the loader and action in the plans page.
* Fix billing alerts scope
* Fix org usage page scope
* Fix token revoking flow scope check
* Throw error for failed PAT revokes instead of silent failure
* fix(webapp): toast message issue after gh app installation
Fixes an issue with displaying toasts messages in the project settings
page. The github callback cookie was interfering with the flash cookie used
for toast messages.
* Do not set a tracking branch in the staging env by default
* Add canceledAt to the deployment db schema
* Expose an api endpoint to cancel deployments
* Show the canceled status description in the dashboard
* Enable canceling deployments from the dashboard
* Show cancelation reason in the deployment details
* Make verifyProjectMembership a function for consistency
* Apply some good 🐰 suggestions
* Add installing status to the deployment db schema
* Replace the deployments /start endpoint with /progress
* Show the installing status in the dashboard
* Add installing status to the api schema and cli
* Add changeset
Depot builds have short-lived tokens and their TTL is not exposed in the SDK. As queued deployments can stay in the queue for an arbitrary amount of time, deferring the remote build creation helps avoid expired Depot token issues.
* Cancel run events which then propogate cancellation status to span ancestors
* WIP
* convert closing cached run spans to new system
* converted expired complete span event to new method
* move v3 over to new methods
* Convert getDetailedTraceSummary to use the new ancestor override stuff
* remove debug logs
* Don't return UNSPECIFIED task events in getRunEvents
* fix the call site for cancelling run event in v3
* Add changeset
* remove methods
This PR adapts the deployment initialization endpoint to handle build server deployments with older CLI versions gracefully.
When we introduced automatic deployments via the build server, we slightly changed the deployment flow
mainly in the initialization and starting step: now deployments are first initialized in the `PENDING` status
and updated to `BUILDING` once the build server dequeues the build job.
Newer versions of the `deploy` command in the CLI will automatically attach to the existing deployment
and continue with the build process. For older versions, we can't change the command's client-side behavior,
so we need to handle this case here in the initialization endpoint. As we control the env variables which
the git meta is extracted from in the build server, we can use those to pass the existing deployment ID
to this endpoint. This doesn't affect the git meta on the deployment as it is set prior to this step using the
/start endpoint. It's a rather hacky solution, but it will do for now as it enables us to avoid degrading the
build server experience for users with older CLI versions. We'll eventually be able to remove this workaround
once we stop supporting 3.x CLI versions.
* Fix `current` badge inconsistency in the deployment details page
* Add custom hook for auto revalidation based on an interval and/or focus change
* Use the autoRevalidate hook for live reloading of the deployments page
* Extract autoReloadPollIntervalMs to an env var
* Replace the sse-based autoreload in bulk actions and queues page with the simpler autoRevalidate hook
* Enable setting the initial status on deployment creation
* Expose endpoint to start deployments
* Extend build timeout on deployment start
* Use separate timeout value for queued deployments
* Add startedAt to the deployment schema
* Show the new startedAt instead of createdAt in the dashboard
* Show github user tag also in the deployment details page
* Show `pending` deployment status as `queued` in the dashboard
* Apply some good 🐰 suggestions
* Add missing return
* Improve styling of onboarding pricing plans
* type only import
* Improve the onboarding plan page so it scrolls on smaller screens
* Adds additional pricing for bolt-ons for the Pro plan
* fix text wrapping issue
* add bg color back in
* Change the color to indigo
* Pro tier pricing information now matches the marketing site
* Update the button styles to secondary
* WIP adding separate links to Parent and Root runs
* TextLink now supports optional shortcuts
* Adds shortcut keys to the root and parent links + the shortcut help panel
* Adds new icons for root and parent
* root friendlyId works
* Updates icons for jump to root and parent
* Copy tweak
* Improve how the Free tier shows no preview branches
* Improve the wording in the tooltip
* Align the x icon better
* Show price for additional preview branches
* Change the shortcut key
* Fixes button alignment
* Adds nested dependencies task hello-world
* Fixes typo “Cancelled”
* Removes taskIdentifier, not needed
* Removes unused taskIdentifier
This PR enables setting project build settings in the settings page:
root directory, install command and trigger config file path.
For most cases there should be no need to set these explicitly.
* Mo-Stashed changes
* fix(core): prettyPrintingPacket will now do a structuredClone on non-circular references instead of outputting [Circular]
This also fixes an issue with replaying of runs that include
non-circular references
* Add external build data and image platform to the get deployment endpoint
* If provided, attach to an existing deployment in the deploy command
* Check status for existing deployments
* Add changeset
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility
* A few improvements via the 🐇 review
* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs
* Add changeset
* Add a CLI command to list and view env vars
* Add changeset
* Restrict pemissions on env files created with `env pull`
* Escape env vars when exporting to file
* Switch changeset to patch
* Fix settigns page delete project width issue
* Apply a couple of touch-ups to the project settings page
* Add UI flow to connect gh repos
* Enabling adding another gh account in the ui
* Enable connecting a repo to a project
* Enable updating git settings
* Enable disconnecting gh repos from a project
* Remove prisma migration drifts
* Hide git settings when github app is disabled
* Fix migration order
* Avoid using `location` to avoid SSR issues
* Make branch tracking optional
* Disable save buttons when there are no field changes
* Disable delete project button unless the input matches the project slug
* Show connected repo connectedAt date
* Check that tracking branch exists when updating git settings
* Show tracking branch hint in the deployments page
* Fix positioning issue of the pagination pane in the deployments page
* Use mono font for branch names
* Add link to git settings
* Show tracking branch hint for the preview env too
* Add a confirmation prompt on repo disconnect
* Add link to configure repo access in gh
* Add rel prop to github links
* Automatically open repo connection modal after app installation
* Apply some fixes suggested by mr rabbit
* Fix flash cookie issue
* Extract project settings actions into a service
* Extract project settings loader into a presenter service
* Introduce neverthrow for error handling
* Try out neverthrow for error handling in the project setting flows
* Move env gh branch resolution to the presenter service
* feat(webapp): add support for running web services (api, engine, webapp) in cluster mode for better perf
* cleaned up signal handling and resolved some valid 🐇 issues
* Add schemas for gh app installations
* Implement gh app installation flow
* Make the gh app configs optional
* Add additional org check on gh app installation callback
* Save account handle and repo default branch on install
* Do repo hard deletes in favor of simplicity
* Disable github app by default
* Fix gh env schema union issue
* Use octokit's iterator for paginating repos
* Parse gh app install callback with a discriminated union
* Remove duplicate env vars
* Use bigint for github integer IDs
* Sanitize redirect paths in the gh installation and auth flow
* Regenerate migration after rebase on main to fix ordering
* Handle gh install updates separately from new installs
* feat(supervisor): dynamic queue consumer pool
* add changeset
* fix: correctly handle zero median and even samples
* feat(supervisor): consumer pool metrics
* fix tests
* more tests and fixes
* decrease default scaling cooldowns
* don't treat initial pool size as scale up
* handle scale down when queue length drops to zero
* remove changeset, supervisor changes only
* add damping factor env var
* feat(webapp): upgrade otel packages and add more metrics
* add env var to disable additional detectors
* expose more prisma metrics
* chore(webapp): drop node 16 support
* Added MCP to the intro
* Added human-in-the-loop and new build extensions
* Waitpoint notes
* Moved openai guardrails example to Python
* Added a connection limit note
* Added Supabase + Prisma note to supabase auth
* Updates based on what the rabbit said
* Moved the supavisor section to the prismaExtension docs
* docs: update triggering.mdx fixing wrong argument numeration for options
Update the docs triggering.mdx: text indicating options argument position inside `tasks.trigger` function.
Options object is a third argument instead of stated second argument.
* update triggering.mdx to change options argument position in batchTrigger
* feat(mcp): add wait_for_run_to_complete tool so agents don't spam the get_run_details call after triggering
This also fixes the search docs MCP tool
* Install mcp using the latest tag, not the specific version
* more eager dequeuing, queue cooloff periods, return workerQueueLength when dequeueing
* Cache worker group authentication and remove old self-hosted worker code (only managed is currently supported)
* add additional spans during dequeue
* Add env vars and additional spans
* Initial work on upgrading to 6.14.0
Set the output to node_modules still to make it easier
* Use ./generated Prisma folder, update types to fix issues
* Docker compose restart Clickhouse
* Prisma instrumentation update
* Docker
* Removed database dockerignore file, add generated prisma client to the top-level one
* Delete v3-catalog package.json
* Resolved pnpm lock file
* Log errors for very slow queries
* Create schema and migration for organization access tokens
* Add helpers for creating and authenticating OATs
* Adapt the auth service to also accept OATs
* Accept OATs in the whoami v2 endpoint
* Enable deployments with the CLI using OATs
* Avoid reading env variables directly in the token utils
* Remove duplicate cli token utils
* Validate ENCRYPTION_KEY length when parsing env vars
* Make token utils a server-only module
* Disallow revoking already revoked OATs
* Simplify generics in authenticateRequest
* Use 32 bytes mock encryption key in the test setup
* Update dummy encryption key values in tests and templates
* Add a column in the OATs table to differentiate between user and system generated
* Simplify args for v3ProjectPath
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Add index on org id and createdAt
* Avoid storing the encrypted oat token and its obfuscated version in the DB at all
It is a safer approach. Also we do not need to ever read the decrypted token value after creation.
* Fix prisma update condition
* Add token type to the OAT table index
* Accept OATs in the mcp auth flow
* Simplify env auth flow around the /projects endpoints
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Add MCP Server documentation to documentation site
This change introduces comprehensive documentation for the Trigger.dev MCP Server, covering its installation, usage, and available tools. The addition is necessary to provide users with detailed guidance on leveraging the MCP Server for managing Trigger.dev projects, tasks, and deployment processes.
- Updated `docs.json` to include a new section for the MCP Server documentation, improving navigation and accessibility.
- Added `mcp-introduction.mdx` as an introductory guide to the MCP Server, detailing its purpose and installation steps.
- Included `mcp-tools.mdx`, a document outlining each tool available in the MCP Server, complete with descriptions and example usages.
- The new content is aimed at improving user understanding and efficiency in using Trigger.dev's MCP Server capabilities.
* Refactor MCP installation documentation
The changes were necessary to streamline the MCP installation guidance and align our documentation with the latest client support capabilities. Removing the manual installation section and redundant information helps reduce confusion for users.
- Updated installation instructions to lead with auto-configuration for supported clients.
- Cleaned up the options and examples sections to enhance clarity.
- Reformatted client configuration details for better readability.
- Removed outdated links in the "Next Steps" section, focusing on more direct resources such as available MCP tools.
* mcp docs edits
* Add debugMode to get_run_details
* add cancel_run
* Add list deployments tools
* Add list preview branches tool
* Docs: adds agent rules page
* Adds redirect for the CLI path to work
* Adds subagent install image
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* fix(v3): prevent saturated queues from dominating dequeue attempts by adding a cooloff period of successive failed dequeues
* Add env vars and add a marqs shutdown on SIGTERM/INT
* fix: span events are optional, as logs don't have them
* remove CI: true env var to prevent massive tool responses
* fix github resolver relative path
* prevent the dev command from asking for user input when not interactive
* Remove lodash.get because it's deprecated
* span output can be object other than an object, like a string
* prevent large traces from causing get_run_details failures
* Decreased max log lines, made it a tool input property
* Some install mcp tweaks
* Remove variables from dequeue log message
* Continue snapshot throw json
* Waiting for deploy error removed
* Realtime ECONNRESET is expected
* Redis worker logErrors changes, removed ids
* Preview branch without a branch shouldn't log an error, it's a user provided issue
* "Task run is not in a cancellable state" isn't an error, it's expected
* "CreateCheckpointService: Child run already resumed" is expected
* "CreateCheckpointService: Batch already resumed" is expected
* "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures
* Ignore "PrismaClient error"
* Don't log Redis worker DLQ errors if we're ignoring
* "Failed to parse machine config" is fine, sometimes a config is null or undefined
* "Failed to parse machine config" for v3
* MetadataTooLargeError shouldn't log an error
* Don't log an error when the snapshot shouldn't be created, it's normal for this to happen
* Slack alerts, skip `account_inactive` errors
* v3 finalize run with no locked isn't an error
* Another false error
* Finalize run CRASHED runs were logging errors
* All slack alert errors are warnings except invalid blocks
* Adds new features table to top of v4 upgrade guide
* Adds wait idempotency to wait-until, wait-for, and wait-for-token pages
* Adds new priority docs page and updates the v4 upgrade guide
* Adds new task lifecycle hooks
* Removes the message about requiring tasks to be exported
* Adds new global lifecycle hooks section
* Moves sections from upgrade guide into the table
* Adds hidden task page
* Improves the global lifecycle hooks section
* Updates middleware and locals section
* Adds new useWaitToken page to the react hooks section
* Adds a new ai.tool section
* Moves Docker (legacy) page into self-hosting section
* Removes known issues from v4 upgrade guide
* Replace “toolTask” with “ai.tool” in the Streams page example
* Renames guide to “Migrating from v3” and adds redirect
* Remove references to v4
* Removes changelog from migration guide
* The installation guide now references `@latest update`
* Changes all references from `/sdk/v3` to `/sdk`
* Updates @v4-beta to @latest
* Fixed broken link
* Fixes broken link
* Adds an upgrade to v4 using AI section
* Fixes 2 broken links
* Adds an entry for targetting preview branches
* Updates the run statuses
* Adds boolean helpers section to the runs and realtime pages
* Updates the concurrency page
* Updates the test page to include the new options
* Adds SDK and curl options for the preview branch targeting
* Updates new bulk actions page
* Remove the releasing concurrency section
* Got rid of some more @v4-beta mentions
* Improved rate limit docs
* Improved migrating docs
* Removed commented sections of the docs
* useWaitToken hook
* Fixed the description
* Fix for missing test image
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Dan <8297864+D-K-P@users.noreply.github.com>
* fix: external traces now respect parent sampling, and prevent broken traces when there is no external trace context
* Add changeset
* improve trace flag handling and better internal host checking
* the traceFlags are now being properly passed through as a number
* add tier scheduling support to supervisor
* add billing info to dequeued message w/o cache
* add cache with best effort invalidation
* fix invalidate circular dep
* add changeset
* use new plan type on runs as fallback during dequeue
* tidy up
* be more explicit with plan type fallback
* remove additional billing check from hot path
* switch to placement tags
* update changeset
* update platform package
* start using new entitlement response
* ensure skipChecks optimization validates at batch level
* add optional items to add to queue manager limits
* make the bool env helper only accept boolean defaults
* remove redundant private field
* update placement tag helper to prevent unsupported tags
Small change that shows the github user in the deployments page in cases
where it was triggered by the github app integration.
These deployments are not triggered using a PAT, so we cannot associate
them with a Trigger user ID.
This PR adapts the deployment command to also evaluate build server env
variables when creating the git meta. The build server will initially
only support deployments triggered by the github app, but we might add
other git providers in the future.
Sticking to the GH actions naming convention for the env variables set
by the build server in favor of consistency.
* fix: sentry memory leak by disabling includeLocalVariables
* Enhance heap snapshot consistency and labeling
To facilitate more accurate and consistent heap memory snapshots, a new function forceConsistentGC was added before taking a snapshot. This ensures the garbage collector (GC) runs multiple times, stabilizing the heap state for more reliable analysis. This is particularly helpful when debugging memory-related issues.
Updates to the memory-leak-detector script now allow labeling of snapshot runs using the --label flag. This helps in distinguishing different runs for easier tracking and comparison of memory usage across test sessions. Additionally, the --expose-gc flag ensures that the GC can be manually triggered during test runs, leading to more consistent memory states and potentially uncovering hidden memory leaks.
* Refactor forceConsistentGC for improved readability
The function forceConsistentGC was refactored to enhance code readability and consistency. The main improvements include:
- Updated syntax for consistent string quotation and spacing.
- Simplified garbage collection by removing specific major/minor GC calls, as the distinction isn't necessary.
- Implemented minor changes to arrow function formatting for consistency.
These changes neither impact the program logic nor the function behavior but help maintain code quality standards and readability.
* Fix memory leak by removing request.clone() usage
Identified that the memory leak in the project was linked to the usage of request.clone() within the `@sentry/remix` package's callRouteAction handler. Although initially suspected as a Sentry issue, the problem appears to arise from the handling of request.clone() in Remix version 2.1.0. By removing the call to request.clone(), the memory leak has been resolved.
- Introduced garbage collection execution before snapshot to manage memory allocation effectively.
- Improved error handling and timeout mechanisms in the memory leak detector to enhance its resilience during runtime.
- Expanded testing for both GET and POST requests to monitor and validate potential memory leaks better. The POST requests involve sending large payloads to stress-test the system.
- The modification particularly focuses on enhancing robust memory tracking and providing detailed progress reporting during request phases.
* patch @sentry/remix to prevent memory leaks
* Fix pnpm lock
* undo some unrelated changes
* The logger now supports metadata
* Added metadata to ServiceValidationError in some critical places
* Don't ack the heartbeat if there's a mismatch, it might prevent a brand new one
* Don't ack the heartbeat inside stalled. By returning it will be acked IF the deduplication key matches
* Only start an attempt if not finished. Send message to worker if pending executing
* Fix the exit process reason tet
* Fixed cancelling test since bug fix
The old behaviour was wrong for pending executing in the test
* set correct run status on snapshot after dequeue
* set run status back to PENDING when we requeue
* remove retrying after failure status from v4 and fix tests
* fix one last test
* Adds a background image dashboard wrapper
* Dashboard background image
* Adds a background image to the welcome onboarding page
* Background is constructed of 3 images
* Adds the background to the create org page
* Adds a background to the choose plan page
* Change the default button spinner color to white
* Adds the background to the create new project page
* Updates the invite team member page
* Adds background image to received invite page
* Reordered react hooks + frontend sections
* Updated the overview and nav
* Separated out SWR hooks
* Restructured metadata sections
* Improved backend docs
* Fixed broken link
* Fixed broken links
* Updated the structure
* Restructured overview
* Updated examples cards
* Improved overview
* Updated how it works
* Updated auth
* Added type safety to the run object page
* made the subscribe description clearer
* Fixed links in triggering
* Fixed link
* Removed examples footers
* Copy tweak
* Consolidated metadata and subscribe pages
* moved metadata task examples to metadata
* Fixed links
* Fixing links like zelda
* Removed dead import
* Clearer titles
* Cap R for Realtime
* Fixes broken link
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Map new allowedMasterQueues → allowedWorkerQueues
* ClickHouse worker_queue on task runs
* Added the Region to the run inspector
* Pass a region in when triggering
* Added a changeset
* Added triggering regions docs
* Added region to the ctx
* Fix for backfiller masterQueue/workerQueue
* Add manual monorepo setup guide for Trigger.dev
This commit introduces a new "Manual setup" guide for setting up Trigger.dev in projects, specifically focused on monorepo configurations. The guide outlines two primary approaches for monorepos: creating a dedicated tasks package and integrating tasks directly within apps. This detailed documentation aims to help developers manually configure their projects, bypassing automated steps, and understanding the setup better.
- Provides step-by-step instructions for both 'Tasks as a package' and 'Tasks in apps' approaches.
- Includes example configurations for various package managers, environment setups, and runtime options.
- Enhances user understanding of Trigger.dev's configuration requirements in complex monorepo environments.
The purpose of adding this guide is to enable developers to seamlessly integrate Trigger.dev into their monorepos, whether they choose to abstract tasks into packages or embed them within individual applications. This flexibility supports diverse project structures while maintaining consistency with Trigger.dev's operational prerequisites.
* Correct tasks usage in documentation
The 'tasks as package' section in the documentation had an incorrect example under 'Use tasks in your apps' which needed correction to align with the actual package usage.
- Fixed incorrect import of tasks by updating to the correct import from '@repo/tasks/trigger'.
- Updated the syntax to use 'tasks.trigger' with type parameters, following the new pattern established for triggering tasks with TypeScript.
- Added error handling to catch and log errors during task execution, returning a meaningful error message instead of just failing silently.
This update ensures developers have an accurate reference when implementing tasks in their applications, especially given the breaking changes in TypeScript compatibility due to recent package updates.
* Update package configuration for Zod 3-4 compatibility
The recent Zod package updates from version 3 to 4 introduce breaking changes in TypeScript compatibility that require adjustments in our project configuration files. The primary updates involve adding type annotations and modifying import statements to support the shift without breaking existing functionality.
- Updated `package.json` and initialization files to align with new compatibility requirements.
- Modified server task examples to explicitly use type imports for improved error handling and consistency.
- Adjusted workspace and project settings to maintain compatibility and improve configuration clarity.
- Provided references to current examples for better implementation guidance.
* More manual setup guide steps
* Fix bun docs link
* Initial Regions page
* Fix for bad attribute name
* Switching regions is working. Some style improvements
* Use a dialog for confirmation, not great yet
* Added allowedMasterQueues
* Improves flag icons
* Improves the region switch modal with more info
* Adds “suggest a region” table row
* New icons for the buttons
* Improved the tooltip information
* New “small” badge style
* Make the default badge live in its column
* Better DO icon size
* Remove unused export of regions options
* Show upgrade message for free users to get static IPs
* Admins can view all regions and switch at will
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Update zod package to version 3.25.76 across all modules
Update the zod library from version 3.23.8 to 3.25.76 in multiple package files to ensure compatibility and take advantage of new features or bug fixes introduced in recent releases. Keeping all modules synchronized with the latest version of zod helps maintain consistency across the project and reduces potential compatibility issues.
- Modified zod version in apps/supervisor, webapp, and various internal packages.
- Updated zod references in pnpm-lock.yaml to reflect the new version.
- Ensure dependencies that rely on zod are using the updated version to avoid mismatches.
* Add changeset
* Add payload schema handling for task indexing
This change introduces support for handling payload schemas during task indexing. By incorporating the `payloadSchema` attribute into various components, we ensure that each task's payload structure is clearly defined and can be validated before processing.
- Updated the TaskManifest and task metadata structures to include an optional `payloadSchema` attribute. This addition allows for more robust validation and handling of task payloads.
- Enhanced several core modules to export and utilize the new `getSchemaToJsonSchema` function, providing easier conversion of schema types to JSON schemas.
- Modified the database schema to store the `payloadSchema` attribute, ensuring that the payload schema information is persisted.
- The change helps in maintaining consistency in data handling and improves the integrity of task data across the application.
* Refactor: Remove getSchemaToJsonSchema in favor of schemaToJsonSchema
The `getSchemaToJsonSchema` function was removed and replaced with `schemaToJsonSchema` across the codebase. This update introduces a new `@trigger.dev/schema-to-json` package to handle conversions of schema validation libraries to JSON Schema format, centralizing the functionality and improving maintainability.
- Removed `getSchemaToJsonSchema` exports and references.
- Added new schema conversion utility `@trigger.dev/schema-to-json`.
- Updated `trigger-sdk` package to utilize `schemaToJsonSchema` for payloads.
- Extensive testing coverage included to ensure conversion accuracy across various schema libraries including Zod, Yup, ArkType, Effect, and TypeBox.
- The update ensures consistent and reliable schema conversions, facilitating future enhancements and supporting additional schema libraries.
* Add support for Zod 4 in schema-to-json
This change enhances the schema-to-json package by adding support for Zod version 4, which introduces the native `toJsonSchema` method. This method facilitates a direct conversion of Zod schemas to JSON Schema format, improving performance and reducing reliance on the `zod-to-json-schema` library.
- Updated README to reflect Zod 4 support with native method and retained support for Zod 3 via existing library.
- Modified package.json to allow installation of both Zod 3 and 4 versions.
- Implemented handling for Zod 4 schemas in `src/index.ts` using their native method.
- Added a test case to verify the proper conversion of Zod 4 schemas to JSON Schema.
- Included a script for updating the package version based on the root package.json.
- Introduced a specific TypeScript config for source files.
* Revise schema-to-json for bundle safety and tests
The package @trigger.dev/schema-to-json has been revised to ensure bundle safety by removing direct dependencies on schema libraries such as Zod, Yup, and Effect. This change minimizes bundle size and enhances tree-shaking by allowing external conversion libraries to be utilized only at runtime if necessary. As a result, the README was updated to reflect this usage pattern.
- Introduced `initializeSchemaConverters` function to load necessary conversion libraries at runtime, keeping the base package slim.
- Adjusted test suite to initialize converters before tests, ensuring accurate testing of schema conversion capabilities.
- Updated `schemaToJsonSchema` function to dynamically check for availability of conversion libraries, improving flexibility without increasing the package size.
- Added configuration files for Vitest to support the new testing framework, reflecting the transition from previous test setups.
These enhancements ensure that only the schema libraries actively used in an application are bundled, optimizing performance and resource usage.
* Refine JSON Schema typing across packages
The changes introduce stricter typing for JSON Schema-related definitions, specifically replacing vague types with more precise ones, such as using `z.record(z.unknown())` instead of `z.any()` and `Record<string, unknown>` in place of `any`. This is part of an effort to better align with common practices and improve type safety in the packages.
- Updated the `payloadSchema` in several files to use `z.record(z.unknown())`, enhancing the type strictness and consistency with JSON Schema Draft 7 recommendations.
- Added `@types/json-schema` as a dependency, utilizing its definitions for improved type clarity and adherence to best practices in TypeScript.
- Modified various comments to explicitly mention JSON Schema Draft 7, ensuring developers are aware of the JSON Schema version being implemented.
- These adjustments are informed by research into how popular libraries and tools handle JSON Schema typing, aiming to integrate best practices for improved maintainability and interoperability.
* Add JSON Schema examples using various libraries
The change introduces extensive examples of using JSON Schemas in the 'references/hello-world' project within the 'trigger.dev' repository. These examples utilize libraries like Zod, Yup, and TypeBox for JSON Schema conversion and validation. The new examples demonstrate different use cases, including automatic conversion with schemaTask, manual schema provision, and schema conversion at build time. We also updated the dependencies in 'package.json' to include the necessary libraries for schema conversion and validation.
- Included examples of processing tasks with JSON Schema using libraries such as Zod, Yup, TypeBox, and ArkType.
- Showcased schema conversion techniques and type-safe JSON Schema creation.
- Updated 'package.json' to ensure all necessary dependencies for schema operations are available.
- Created illustrative scripts that cover task management from user processing to complex schema implementations.
* Refactor SDK to encapsulate schema-to-json package
The previous implementation required users to directly import and initialize functions from the `@trigger.dev/schema-to-json` package, which was not the intended user experience. This change refactors the SDK so that all necessary functions and types from `@trigger.dev/schema-to-json` are encapsulated within the `@trigger.dev/*` packages.
- The examples in `usage.ts` have been updated to clearly mark `@trigger.dev/schema-to-json` as an internal-only package.
- Re-export JSON Schema types and conversions in the SDK to improve developer experience (DX).
- Removed unnecessary direct dependencies on `@trigger.dev/schema-to-json` from user-facing code, ensuring initialization and conversion logic is handled internally.
- Replaced instances where users were required to manually perform schema conversions with automatic handling within the SDK for simplification and better maintainability.
* Add JSONSchema type for payloadSchema in tasks
The change was necessary to improve type safety by using a proper JSONSchema type definition instead of a generic Record<string, unknown>. This enhances the developer experience and ensures that task payloads conform to the JSON Schema Draft 7 specification. The JSONSchema type is now re-exported from the SDK for user convenience, hiding internal complexity and maintaining a seamless developer experience.
- Added JSONSchema type based on Draft 7 specification
- Updated task metadata and options to use JSONSchema type
- Hid internal schema conversion logic from users by re-exporting types from SDK
- Improved bundle safety and dependency management
* Add JSON schema testing and revert package dependencies
This commit introduces a comprehensive set of JSON schema testing within the monorepo, specifically adding a new test project in `references/json-schema-test`. This includes a variety of schema definitions and tasks utilizing multiple validation libraries to ensure robust type-checking and runtime validation.
Additionally, the dependency versions for `@effect/schema` have been adjusted from `^0.76.5` to `^0.75.5` to maintain compatibility across the project components. This ensures consistent behavior and compatibility with existing code bases without introducing breaking changes or unexpected behavior due to version discrepancies.
Key updates include:
- Added new test project with extensive schema validation tests.
- Ensured type safety across various task implementations.
- Reverted dependency versions to ensure compatibility.
- Created multiple schema tasks using libraries like Zod, Yup, and others for thorough testing.
* Refactor JSON Schema test files for clarity
Whitespace and formatting changes were applied across the `json-schema-test` reference project to enhance code readability and cohesion. This included removing unnecessary trailing spaces and ensuring consistent indentation patterns, which improves maintainability and readability by following the project's code style guidelines.
- Renamed JSONSchema type annotations to adhere to TypeScript conventions, ensuring that all schema definitions properly satisfy the JSONSchema interface.
- Restructured some object declarations for improved clarity, especially within complex schema definitions.
- These adjustments are crucial for better future maintainability, reducing potential developer errors when interacting with these test schemas.
* Fixed some stuff
* WIP
* we now convert schema to jsonSchema on the CLI side via the indexing
* Remove the json-schema-test reference project
* Improve schema-to-json peer deps and fix effect schema
* Explain the casting and match the version numbers
* Fixed a bunch more schema stuff
* Don't clean files that might be written to
* Don't use a custom version of vitest in the new package
* fix attw in schema-to-json
* prevent infinite retry loop for unicode errors
* structured logs for prisma events
* preserve all prisma event fields
* either use structured logs or stdout, never both
* split runs repo tests
* decrease test shards to 8
* First draft billing alerts page
* Budget alert form working
* Don't let free plan users change the billing alert amount
* Fix missing key in map in the form
* Disable queues/org from admin API endpoint
* Don't allow resuming if runsEnabled is false
* Refer to "Billing alerts" not "Plans"
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Form missing dependencies fix
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Deal with thrown errors, fix for duplicating email fields
* Added a RuntimeEnvironment organizationId index
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Cleanup context and execution creation, cache stuff, add parent and root task run ids
* more efficient by using friendly IDs instead of doing joins
* metadata.root/parent now reference current run when run has no root/parent
* Adding changeset
* try to make test less flaky
* Clean imports
* Another attempt to fix the flaky test
* Fix usage by still passing durationMs and costInCents to the execution, just not the run.ctx
* Improve dynamic flush scheduler for otel data
The changes introduce a more flexible and adaptive dynamic flush scheduler to address production issues where the system wasn't flushing data fast enough, causing memory growth and crashes. This issue arises from the existing scheduler handling only a single flush at a time, limiting concurrency and failing to cope with the influx of logs.
- Added configuration options for setting minimum and maximum concurrency levels, maximum batch size, and memory pressure threshold. These parameters ensure that flush operations adjust dynamically based on workload and pressure.
- Implemented `pLimit` to facilitate concurrent flush operations, with adjustments made according to batch queue length and memory pressure.
- Metrics reporting improvements were added to monitor the dynamic behavior of the flush scheduler, aiding in identifying performance issues and optimizing the operation accordingly.
* Implement load shedding for TaskEvent records
This change introduces load shedding mechanisms to manage TaskEvent
records, particularly those of kind LOG, when the system experiences
high volumes and is unable to flush to the database in a timely
manner. The addition aims to prevent overwhelming the system and
ensure critical tasks are prioritized.
- Added configuration options for `loadSheddingThreshold` and
`loadSheddingEnabled` in multiple modules to activate load shedding.
- Introduced `isDroppableEvent` function to allow specific events to
be dropped when load shedding is enabled.
- Ensured metrics are updated to reflect dropped events and load
shedding status, providing visibility into system performance
during high load conditions.
- Updated loggers to inform about load shedding state changes,
ensuring timely awareness of load management activities.
* Fix undefined 'queuePressure' variable in DynamicFlushScheduler
The 'queuePressure' variable was being used without being defined
in the DynamicFlushScheduler class, causing potential runtime
errors. This commit adds the missing definition and ensures that
the variable is correctly calculated based on the 'totalQueuedItems'
and 'memoryPressureThreshold'.
- Addressed code inconsistencies and improved formatting.
- Defined 'queuePressure' in the 'adjustConcurrency' method
to prevent potential undefined errors.
- Enhanced readability by maintaining consistent spacing and
format across the file, contributing to the stability and
maintainability of the code.
- Adjusted batch size logic based on the newly defined 'queuePressure'
variable.
* Refactor concurrency adjustment logic in scheduler
The concurrency adjustment logic in the dynamic flush scheduler has been refactored to improve clarity and maintainability. This change moves the calculation of pressure metrics outside of the conditional blocks to ensure they are always determined prior to decision-making.
- The queue pressure and time since last flush calculations were moved up in the code to be independent of the 'backOff' condition.
- This refactor sets up the groundwork for more reliable concurrency scaling and better performance monitoring capabilities. The overall logic of adjusting concurrency based on system pressure metrics remains unchanged.
This adjustment addresses ongoing issues with the scheduler that were not resolved by previous changes.
* Some tweaks
* Sentry WIP
* Configure sentry for uploading and releasing during the publish webapp step
* Delete source maps after uploading
* Forward logger.error calls to sentry through Logger.onError
* Couple tweaks to the dockerfile
* Premerge run batch before sending it to clickhouse
* Pre-order batch items before sending to clickhouse in favor of performance
* existing
* Emit event on batch flushes
* When merging batches, keep the last occurrence items with the same version
* Add a couple of tests
* Initial burst changes to queue page
* Added tooltip and changed wording around
* View runs from Queues page
* Fix for ugly Version filter "Current" badge
The normal database needs to be mocked because an import somewhere is
using it which causes issues with the test container dbs and hanging at
the end.
Strategy copied from the trigger test file.
* Set the default replication concurrency to 2 for self-hosters
100 was a bit crazy
* Made the run repository an interface, deferring just to CH for now
* Added run repository feature flag, allowing passing a default when getting a flag
* Switch run repository using a feature flag
* Added spans
* Pass the default repository in, so we can try Postgres in the tests
* Fallback to Postgres if ClickHouse errors
* Update feature flags API endpoint
* Queue in run table and filtering
* Debounce the filter changes
* Remove console log
* Added machine filtering
* Added version filtering
* Filter by version in the db
* Removed duplicate classes
* Version filtering hasFilters consistency
* Added queues and machines to the bulk action summary
* runs.list filtering for queue and machine
* Fix for machine errors
* Input field now has accessory instead of shortcut
* First experiments with the UI
* Got the fake filtering working
* AI filtering is working pretty well ✨
* Started working on tool calling
* Tool calling is working
* Styling progress
* Working on the error
* Errors work, improved the styling
* Nice glow effect
* Tweak the darkness of the text field
* Re-ordered the UI, set AI settings to use system prompt and telemetry
* Refactored to make it testable
* Added basic evals
* Better time inputs and evals
* Removed some code comments
* Remove unused useSearchParam change
* Tidy imports
* If no OpenAI API key send json back
* Fix for merge conflict with duplicate query filters
* Another conflict resolved
* Another merge conflict resolved
* Pass the model in, allow changing it
* Queue in run table and filtering
* Debounce the filter changes
* Remove console log
* Added machine filtering
* Added version filtering
* Filter by version in the db
* Removed duplicate classes
* Version filtering hasFilters consistency
* Added queues and machines to the bulk action summary
* runs.list filtering for queue and machine
* Fix for machine errors
* Added experimental_devProcessCwdInBuildDir config option
Added experimental_devProcessCwdInBuildDir config option to opt-in to new process.cwd behavior when executing tasks in the dev CLI. Currently process.cwd maps to the "root" of your trigger.dev project (the directory that contains your trigger.config.ts file). Setting experimental_devProcessCwdInBuildDir to true changes process.cwd to instead be the temporary build directory inside of the .trigger directory.
This makes it so the files added via the additionalFiles extension can be read using the same path in dev and deployed tasks.
* Remove claude code reference because it breaks windows tests 😡
* WIP
* Make release concurrency system extremely simple, everything just releases all the time
* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column
* WIP new release concurrency system
* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint
Also removed deadlock detection, and added environment burst concurrency
* Added new DEQUEUED status
Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests
* Introduce the new "current dequeued concurrency set"
* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing
* Remove waitpoint test for QUEUED_EXECUTING
* Add isWaiting
* Add changeset
* Use createdAt for ordering realtime runs instead of number
* Clarify the envCurrentDequeuedKey usage
* mock the db.server file to fix the tests
* Updated changset "EXECUTED" -> "EXECUTING"
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Access machinePreset from the run list presenter
* New icons for machine presets
* New icon + name combo label for the machine preset
* Adds new “Machine” column to the runs list
* Make a separate component for the machine tooltip info
* add machinePreset to the span presenter
* Show the Machine in the Details tab in the Run inspector
* Show an admin only separator
* Fix docs icon in the button
* Small padding tweak
* Move the Machine nearer the costs
* Don't cast the machine preset
* Remove typecast, better to have a bad label if we add a new machine and don't update thos
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Use sexy scrollbars for templates and recent runs popover content
@Nick this is for you!!1
* Fix TabButton component disabled state
* Disable payload tab when replaying runs with large payloads
* Fix for filtering by run id
Simplified to have a single filter for runId which accepts an array. This unifies the filtering by multiple runs (used by the waitpoint page) and the individual run filtering. It fixes a bug with bulk actions
* Truncate long bulk action titles
* Fix regression that broke superjson behavior for the replay flow
* Ignore undefined type metadata when overriding superjson payloads
Addresses https://github.com/triggerdotdev/trigger.dev/issues/1968
* Add a few unit tests for replaceSuperJsonPayload
* Add ID of the replayedFrom run to the TaskRun schema
* Propagate the replayedFrom run ID in the replay flow
* Link the replayed run in the run details pane
* useSearchParams has
* useSearchParams has
* useSearchParams has
* Consistent way to get the run filters
* Consistent way to get the run filters
* Consistent way to get the run filters
* Initial work on the new bulk actions
* Initial work on the new bulk actions
* Initial work on the new bulk actions
* WIP actions and filtering
* WIP actions and filtering
* WIP actions and filtering
* Empty filter arrays are set to undefined
* Empty filter arrays are set to undefined
* Empty filter arrays are set to undefined
* WIP prisma schema
Removed extra runtimeEnvironmentId
* WIP prisma schema
Removed extra runtimeEnvironmentId
* WIP prisma schema
Removed extra runtimeEnvironmentId
* Migrations
* Migrations
* Migrations
* BulkActionGroup changed some columns around
* BulkActionGroup changed some columns around
* BulkActionGroup changed some columns around
* New badge variant, removed unused ones
* New badge variant, removed unused ones
* New badge variant, removed unused ones
* Bulk action button
* Bulk action button
* Bulk action button
* Make the next runs page the default now
* Make the next runs page the default now
* Make the next runs page the default now
* Improved the RadioButton style
* Improved the RadioButton style
* Improved the RadioButton style
* Remove the old bulk action bar
* Remove the old bulk action bar
* Remove the old bulk action bar
* More UI progress
* More UI progress
* More UI progress
* Lots of UI changes to the Runs page
* Lots of UI changes to the Runs page
* Lots of UI changes to the Runs page
* Fixed period filter resetting everything
* Fixed period filter resetting everything
* Fixed period filter resetting everything
* Improved the Switch secondary style
* Improved the Switch secondary style
* Improved the Switch secondary style
* Buggy filter fixes
* Buggy filter fixes
* Buggy filter fixes
* Improved the filter display and fixed a bug with search param from object
* Improved the filter display and fixed a bug with search param from object
* Improved the filter display and fixed a bug with search param from object
* Clear button is minimal
* Clear button is minimal
* Clear button is minimal
* Using a presenter now
* Using a presenter now
* Using a presenter now
* Bulk actions are created, but not actually processed (yet)
* Bulk actions are created, but not actually processed (yet)
* Bulk actions are created, but not actually processed (yet)
* Bulk replay/cancel is working
* Bulk replay/cancel is working
* Bulk replay/cancel is working
* Multiple fixes, added bulk column to PG
* Multiple fixes, added bulk column to PG
* Multiple fixes, added bulk column to PG
* Bulk action run filtering working using CH
* Bulk action run filtering working using CH
* Bulk action run filtering working using CH
* Replay setting the bulk id on the runs
* Replay setting the bulk id on the runs
* Replay setting the bulk id on the runs
* Properly cap the time when doing a bulk action
* Properly cap the time when doing a bulk action
* Properly cap the time when doing a bulk action
* If the bulk action isn't recent, add it to the dropdown anyway
* If the bulk action isn't recent, add it to the dropdown anyway
* If the bulk action isn't recent, add it to the dropdown anyway
* Blank version of the bulk actions page
* Blank version of the bulk actions page
* Blank version of the bulk actions page
* Individually selected runs working
* Individually selected runs working
* Individually selected runs working
* Use selected mode if runs are checked
* Use selected mode if runs are checked
* Use selected mode if runs are checked
* Added the modal
* Added the modal
* Added the modal
* Marked the old bulk actions stuff as deprecated
* Marked the old bulk actions stuff as deprecated
* Marked the old bulk actions stuff as deprecated
* Renamed bulk action file
* Renamed bulk action file
* Renamed bulk action file
* Bulk run filter with the name and a default
* Bulk run filter with the name and a default
* Bulk run filter with the name and a default
* WIP on bulk actions page
* WIP on bulk actions page
* WIP on bulk actions page
* Updated panel, added new truncated id component
* Updated panel, added new truncated id component
* Updated panel, added new truncated id component
* Style improvements to the radio buttons
* Style improvements to the radio buttons
* Style improvements to the radio buttons
* Added an option action completion email
* Added an option action completion email
* Added an option action completion email
* Adds a blank state for the bulk actions page
* Adds a blank state for the bulk actions page
* Adds a blank state for the bulk actions page
* Nicer completed email
* Nicer completed email
* Nicer completed email
* Don't open the bulk action panel if there are no runs
* Don't open the bulk action panel if there are no runs
* Don't open the bulk action panel if there are no runs
* Runs blank state and bulk action accordion
* Runs blank state and bulk action accordion
* Runs blank state and bulk action accordion
* Updates secondary/small switch style
* Updates secondary/small switch style
* Updates secondary/small switch style
* Pagination buttons no longer split in twain (WIP)
* Pagination buttons no longer split in twain (WIP)
* Pagination buttons no longer split in twain (WIP)
* Aborting working
* Aborting working
* Aborting working
* Bulk action live reloading
* Bulk action live reloading
* Bulk action live reloading
* ListPagination works correctly in all states
* ListPagination works correctly in all states
* ListPagination works correctly in all states
* Run page, show friendlyId instead of number
* Run page, show friendlyId instead of number
* Run page, show friendlyId instead of number
* Bulk action help open by default if you have none
* Bulk action help open by default if you have none
* Bulk action help open by default if you have none
* Extra status filtering step because of replication delay
* Extra status filtering step because of replication delay
* Extra status filtering step because of replication delay
* Wider bulk action onboarding
* Wider bulk action onboarding
* Wider bulk action onboarding
* More sensible widths on the bulk action side panel
* More sensible widths on the bulk action side panel
* More sensible widths on the bulk action side panel
* Border color tweak to the RadioButton
* Border color tweak to the RadioButton
* Border color tweak to the RadioButton
* Improved the accordion component hover states
* Improved the accordion component hover states
* Improved the accordion component hover states
* Updates the bulk action blank state images to the latest UI
* Updates the bulk action blank state images to the latest UI
* Updates the bulk action blank state images to the latest UI
* Added R and C shortcuts back in
* Added R and C shortcuts back in
* Added R and C shortcuts back in
* Fix for selecting a single run
* Fix for selecting a single run
* Fix for selecting a single run
* Improved exit icon, added shortcut to modal
* Improved exit icon, added shortcut to modal
* Improved exit icon, added shortcut to modal
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Fix for grid layout when 1 page of bulk actions visible
* Fix for grid layout when 1 page of bulk actions visible
* Fix for grid layout when 1 page of bulk actions visible
* Removed the ... on the abort button
* Removed the ... on the abort button
* Removed the ... on the abort button
* Removed the ... on the abort button
* Animate the progress bar
* Set TZ="UTC" in the env example
* Filter summary in the bulk inspector
* Improves the pagination styling
* Improves the pagination styling
* Delete old bulk action routes
* Removed old Postgres RunListPresenter
* Retry any replication error where the message contains "timeout"
* Increase wait to make test less flaky
* The test was using run id instead of friendly id
* Safer array access
* Remove error log if there's a bad status
* Nicer frontend type safety with the bulk action and mode
* Switched a log to a debug log
* Retry replication unless the error is a known non-retry error
Flip the strategy to retry by default
* Make ClickHouse required
* Backfill run replication admin API endpoint
* Set a CLICKHOUSE_URL for unit tests
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Rearrange the layout of the replay modal
* Add run options to the replay modal
* Handle payload and metadata correctly in the shared json editor
* Apply run options in replays
* Fix clear button issue in json editor
Memoization of the clear function caused some problems. Removing it should not cause performance issues.
* Update replay modal hint
* Move machine and version fields to the top for visibility
* Use the same field ordering in the test page
* Reload queues and versions on env override
* Adapt json editor to fill full height
* Clean up a few excessive ternaries
* Avoid ui jump on env selection
* Switch to sexy scrollbars for scheduled tasks in the test page
* Improve TaskRunProcess health detection so we don't try and reuse an unhealthy process
This was happening after the process was killed internally, like by an OOM error
* Add changeset
* don't try interactive login in ci, link to docs
* add ci note to cli deploy docs
* add ci note to github actions
* add changeset
* make cron test less strict
* update node-22 image
* update bun image
* disable io_uring
* fix fallback bun path
* prevent duplicate warnings
* add runtime and version to deployments
* runtime icons
* fallback to nodejs
* prevent empty table cell menu
* log if local build on deploy
* pass io_uring env var to child
* denormalize runtime and version, display on run details
* add changesets
* disable pr checks for changeset commits..
* add runtime data to deployed bg workers
* Introduce request idempotency to prevent duplicate triggers
* Implement request idempotency on trigger
* Use x-trigger-request-idempotency-key header instead
Plus cleanup shared logic in the routes
* Add changeset
* Oops, lets not hardcode a 408
* A couple of improvements
* Add new prisma model for task run templates
* Create run templates in a new service
* Add modal to create run templates in the test page
* Show templates list and apply values when selected
* Hide template creation time in the dropdown list, only show date
* Enable deleting run templates
* Show success toast on template creation
* Validate template label length
* Improve the template creation success indicator
* Use formAction consistently to differentiate submissions
* Type formAction for better editor support
* Prettify run template payload and metadata
* Add triggerSource, concurrencyKey and ttl to run templates
* Switch to sexy scrollbars in the options pane
* Add missing field descriptions
* Add a run options link to the docs
* Make the copy for the field descriptions more concise
* Fix tooltip issue in the Select component
* Implement a new primitive UI component for picking durations
* Implement a new component to input run tags
* Expose all run options in the test run page
* Add subtle animations when adding/removing run tags in the test page
* Add a new resource endpoint for fetching queues
* Fetch usable queues for the selected task
* Fix width display issue in the select component
* Enable locking a run to a version from the test page
* Disable entering max attemps <0
* Validate tags
* Add recent runs popover
* Only show latest version for development environments
* Update run options when selecting a recent run
* Rearrange the test page layout
* Add subtle animation to the duration picker segments on focus
* Improve queue selection dropdown styling
* Fix disabled state issue for the SelectTrigger component
* Disable version selection field for dev envs
* Add usage hints next to the run option fields
* Add machine preset to the run options list
* Allow arbitrary queue inputs for v1 engine runs
* Show truncated run ID instead of run numbers for recent runs
Run numbers will soon get deprecated due to contention issues
* Fix duplicate queue issue
* Extract common elements across the standard and scheduled test task forms
* Apply values from recent runs to scheduled tasks too
* Add additional run options for scheduled tasks
* Use a slightly smaller font size for run option labels
* Disallow commas in the run tag input field
* Switch to a custom icon for recent runs button
* Flatten the load function test task result object
* Avoid redefining machine presets, use zod schema instead
* Fix ClockRotateLeftIcon jsx issues
* Remove recent runs button tooltip as it causes nesting errors
* Adjust the page layout to make it clear which task is currently selected
* Inline the tab group with the copy/clear buttons
* Adds a new route for logging in with mfa
* New path for security page
* Adds “Security” link to account side menu
* Update the Switch component to allow label positions left and right
* Optionally hide the Close button in the Dialog title bar
* Installs `qrcode` react package for generating QR codes.
* CopyButton component now takes children
* New Security route for setting up MFA
* Adds new OTP package for the chadcn InputOTP component
* Adds new InputOTP chadcn component
* Adds InputOTP chadcn component to the MFA login screen
* InputOTP component supports variant styles
* Improvements to form handling
* Show a confirmation modal before you can disable MFA
* Revert redirect back to the dashboard for now
* Implement MFA enabling and disabling
* Refactor and cleanup mfa management code
* More cleanup
* Handle errors in the management action
* Implement mfa login flow
* recovery code input should be password
* Implement rate limiting on the mfa validation endpoint
* Better error ux
* Implement mfa emails and apply James' updates
* Use latest @better-auth/utils
* Improvements via CodeRabbit review
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* create repo if doesn't exist
* fresh auth token for each deploy
* optional assume role
* log when machine overrides enabled
* make test repo namespace configurable
* assume role fix and env var changes
* improve ecr check
* improve tag parsing
* tag parsing tests
* track if repo created and fix test
* missing tryCatch for sts call
* processKeepAlive disabled performance fix
When processKeepAlive was disabled, the eagerly created TaskRunProcess being discarded and a whole new TaskRunProcess created at execution time.
* Don't use a TaskRunProcess if it's being killed, fixes retryImmediately
* Add retry logic for insert operations
Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.
* Replication settings are configurable
* Log out the runIds for failed batches
* Detecting bad JSON in run replication and ignoring it
* Reproduced split unicode error
* Move output file
* Massively improved the performance
* Minor performance improvements
* Unskip tests
* Remove unused test in CH package
* Fix for the ClickHouse UI explorer
* RunReplication keepAlive defaults to false
* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs
* ClickHouse package doesn't need to be built anymore for the webapp
* Set the concurrency_key from the run replication service
* v4: current concurrency sweeper
* Fix webapp tests
* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs
* Improved the mark phase
* Ensure cron jobs get rescheduled even if the handler throws an error
* Better property names
* fix both internal and external registry host
* add next steps to self-hosting overview
* separate into webapp and registry ingress
* fix external registry config check
* fix localhost registry warning
* bump chart version
* Fix incorrect logs on new schedule engine triggered taskss
Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine
* Fixed schedule recovery failing test
* shallow clone
* fix image tag locking example
* electric should use db url env var
* add htpasswd note
* use local driver with log rotation by default
* make worker urls more easily configurable
* configure dev otel endpoint via .env
* increase min recommended worker specs
* move worker url section
* Runs filter by org id and add created at to ordering
* CopyableText can accept an alternative value for copying
* The runs table now shows the ID instead of run number
* Paginating back/forwards fix
* The task stats need org id and project id too
* Fix realtime re-subscribing stale data issue
Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended
* removed logs
* Install the kapa sdk
* WIP using the SDK for the Kapa Ask AI widget
* Removes old kapa from root
* Now rendering everything inside the dialog component
* Fixes min-height of dialog content
* Remove kapa from root
* prevents kapa using reCaptcha
* Adds more functionailty with temporary UI placement for now
* Reset conversation button
* Adds a new sparkle list icon
* Adds some example questions as a blank state
* Animate in the example questions
* use “marked” package to render markdown
* Improve some animations
* Submit a question from the URL param
* adds custom scroll bar styling
* fixes modal to correct height after re-opening it
* Add button to stop generating answer mid-stream
* Adds buttons states to show submitting, generating, submittable
* Adds a helpfull sentence in the blank state
* Show a message if the chat returns an error
* Adds reset chat and feedback buttons to the bottom of an answer
* Makes sure you can give feedback in the different states of chat
* Adds a suble background to the dialog
* Fix a button inside button error
* Improve the shortcut esc key on dialog and sheet component
* Fix classname error
* organize imports
* Use our custom focus-visible
* Move the Tooltip for the button into the AskAI component
* Improved error message
* Organize imports
* Animated the modal gradient
* Small layout improvements
* Adds most asked questions from Kapa
* border glow tweak
* AskAI component is now a hook that can take a question
* remove kapa script
* Add a delay before the modal opens when usign the URL params
* Remove old component
* Update to the latest Kapa version
* Rephrased error message
* Use correct types for conversation
* Fixed types for addFeedback
* Adds DOMPurify package
* removed unused const
* Removed unnecessary platform specification
* Reset the timeout when the ai panel pops up
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fix for coderabbit bad commit
* Clean up imports
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
### PR: Optimize **TaskRun** indexes for hot-path queries
**What changed**
| Object | Type | Purpose |
| ------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `taskrun_runtime_id_desc_idx` | **BTREE** `(runtimeEnvironmentId, id DESC) INCLUDE (createdAt)` | Eliminates explicit sort for the “latest task runs” query (`ORDER BY id DESC`) while remaining index-only. |
| `taskrun_runtime_createdat_idx` | **BTREE** `(runtimeEnvironmentId, createdAt DESC) INCLUDE (id)` | Accelerates the filter-only path that scans by `createdAt >= …` without any ordering requirement. |
| `taskrun_createdat_brin` | **BRIN** on `createdAt` (`pages_per_range = 128`) | Lets the planner skip whole blocks older than the time window for both queries at < 100 MB cost. |
| *(cleanup)* | **DROP** `TaskRun_runtimeEnvironmentId_createdAt_id_idx` | Retires the 3-column index once the new ones are built. |
**Key details**
* All indexes created **CONCURRENTLY** to avoid write blocking.
* `fillfactor = 90` on b-trees for balanced space vs. future growth.
* Net disk usage drops **≈ 15–20 GB** while each query now gets a purpose-built access path.
**Why**
* Remove planner Sort nodes for the top-N “latest runs” view.
* Speed up environment-filtered range scans.
* Shrink index bloat and improve cache efficiency.
* Add createdAt filter to realtime subscribing with tags
* Filter realtime colums and expose ability to skip some columns
* Add sharding support for electric
* Use unkey cache for the created at filter caching
* Remove 2 unused indexes on TaskRun
* Run list now filters by a single runtime environment
* Remove project ID indexes
* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)
* WIP clickhouse powered runs list
stuff
* Improve the query to get the latest tasks for the task list presenter
* Update the usage task list to use clickhouse
* Implement next runs list powered by clickhouse
* Add new index for TaskRun for the runs list, by environment ID
* Add runTags gin index
* Handle possibly malicious inputs
* Ignore claude settings
* Better handling not finding an environment on the schedule page
* Use ms since epoch in test, not seconds
* Remove unused function
* Fix test
* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)
* Fixed the query builder to correct the group by / order by order
* Make sure runs.list still works
* Create small-birds-arrive.md
* remove unused env vars
* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)
* increase fallback branch limit to 100M
* more unused env vars and example
* hide usage page when self-hosted
* fix for init flow
* set default or concurrency to 300 to match 100 on env
* fail fast when registry env vars are empty strings
* WIP
* Run queue now works with the worker queue / master queue split
* Acking should also cause the master queue to be processed
* Convert run engine tests and run engine to use runQueue changes
* Include the util files in the test tsconfig
* coordinator target should be es2020 as well
* providers target 2020
* Fix the triggerTask tests in the webapp
* v4 now working with the new worker queues, and added the legacy master queue migration stuff
* report worker queue lengths via opentelemetry metrics
* Adding lock metrics
* Release concurrency bucket metrics
* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount)
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class
* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker
* Fix CodeRabbit suggestions
* return undefined from dequeueFromWorkerQueue, not null
* Remove message from worker queue in certain circumstances when acking
* Update log
* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var
* Change how the run queue master queue consumers are disabled internally
* Fixed tests
* process the queue on nack
* Fix more tests
* Fix priority tests
* Fixed dequeueing test
* remove registry proxy
* remove --self-hosted flag
* automatically set network build flag
* update syncEnvVars debug log
* improve switch command
* always display deploy errors if they exist
* fix stuck deploy command after finalize error
* webapp-driven deploys, multi-platform support, lots of fixes
* add worker deployment migration
* rename image platform env var
* only try to sync parent env vars for preview deployments
* add KEEP_TMP_DIRS
* supervisor: docker api version lock, auth, multi-platform
* set image ref on create, validate digest
* use metadata for digest, fix local multi-platform builds
* print git meta branch before commit
* improve push and load flag handling
* make runs after local builds compatible with load and push
* small improvement for platform overrides
* add image platform to dequeued message
* remove deprecated init request body fields
* fix fail deployment id param
* remove build debug logs
* pass report merge with no tests
* structured run debug logs
* add required env var for tests
* should not be an error log
* add changeset
* Draft preview branches docs
* More wip on docs
* Explain what the GH action does
* Added manual deploy/archive instructions
* Added dashboard section to the docs
* More wip on the preview branch docs
* Added preview/branch to the CLI deploy docs
* Add preview branch note to triggering docs
* Added preview branch notes to the API keys docs
* Added preview stuff to the Context docs
* CLI preview archive command
* Add a better explanation and details about env vars
* Fix for weird reference to the dashboard
* Prefer branch variables, but load preview envs too
* WIP with vercel preview env var syncing
* Return success if either child or parent env vars are synced
* Removed log of env vars
* Remove another log
* disable run debug logs by default
* lightweight webapp health check
* disable debug logs for dev runs
* disable run debug logs for supervisor client
* add changeset
* cursor should ignore .env files
* fix for duplicate builds when starting dev
* output metafile in dev
* attach metafile to background worker
* attach import timings to worker manifest
* warn during dev if imports take more than 1s
* add analyze command
* update disable warnings flag message
* add changeset
* Initial preview migrations
* Modified the staging endpoint to create preview environments
* Added isBranchableEnvironment to RuntimeEnvironment
* Staging = yellow Preview = orange
* Changed the env sort order
* Set isBranchableEnvironment correctly. Create preview for new projects
* Very basic branch menu
* Creating branches from the dashboard
* Fix for string icons on project delete page
* Don’t show branch API keys
* WIP on the manage branches page
* RuntimeEnvironment added projectId index
* Only create the parentEnvironmentId column if it doesn’t exist already
* Improved the limit wording
* Add search to the branch list
* contains in both places
* Many style improvements
* Branch dropdown and v4 badge
* Arching/unarchive branches working in the dashboard
* Tidied imports
* Change preview slug from `prev` to `preview`
* Use correct color for side menu preview branch icon
* Upsert the branch and use the shortcode as a unique constraint
* Upserting working with nice messages in the dashboard
* Better errors when upserting branches
* Button shortcut, don’t allow event to propagate
* Better duplicate error message
* Filter out archived branches from the env selector
* Archiving/creating tweaked some more
* Add an archived banner to the app, fixes for archived branches and upsells
* Fixed pagination
* Disable editing schedules, pausing queues, testing tasks
* Don’t allow replaying if the env is archived
* When deploying detect the correct environment
* Get the projectClient when there’s a branch
* createGitMeta function, most code from the vercel CLI repo
* Deploy, getting the correct environment client
* Added git column to WorkerDeployment
* Add GitMeta to core schemas
* Create branch when deploying
* WIP on branch support in the API
* Delete old createTaskRunAttempt fn
* apiAuth remove export from internal functions
* Rename env var to “TRIGGER_PREVIEW_BRANCH”
* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs
* First preview deploy and run working
* Set the preview branch in the main SDK
* Added git links to the preview branches table
* Better errors when replaying/testing archived branches
* Don’t dequeue archived environments
* Env var resolution with parent environment
* Hello world default machine small-2x to save my memory
* Fix for more env var functions
* Only return non-archived envs
* Switch to controlled state for the checkboxes
* Uncheck everything when PREVIEW is checked
* WIP on branch UI
* Show the preview branch label on the env vars list
* Fix for overriding env vars
* Adding preview branch env vars working
* Progress on new env vars
* Only allow selecting a single branch
* Layout fix when there are errors
* Set the defaultValue so there are some fields
* Conform fix for team invite page
* Archived environments don’t run scheduled tasks
* Added Git data to deployments
* Added git data to the deployment inspector
* Don’t allow upserting schedules when archived
* Deduplicate and blacklist some env vars
* Fix for wrong conform function being used
* Show a better error if all vars were blacklisted
* Added environment variable search (by key and value)
* Improved preview branch icon
* Replay now supports branches
* Schedule page render branches properly
* Show the env icon in bottom-left of the test page
* When editing older schedules (that have multi-env) show preview branches correctly
* Fix for incorrect disallowed branch name character
* Extract and improve the directory verification code
* WIP for CLI preview archive command
* Improved the preview branch action buttons
* Redirect to the project if we don’t find a matching env
* Archiving branch via the CLI working
* Fix for archiving branches
* Public access token test task
* JWTs working are with preview branches
* Add branch and git data to the Run ctx
* Updated GitMeta functions to work in CI
* Added pullRequestState
* Archive when deploying if the PR is closed/merged
* Fix for the changesets guide
* Fix for CLI dev bug introduced
* CLI promote now supports preview branches
* Add PR title. Reordered them and added tooltips
* syncEnvVars working with branches
* Added preview branch support to syncVercelEnvVars()
* Detect the branch from Vercel env var (set during build)
* Allow passing a branch in
* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI
* Temp delete
* Improved regenerate api key modal
* Added Accordion component (with styles)
* Redesigned the API keys page
* Revert "Temp delete"
This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.
* Changeset
* Fixed docs link
* The new branch panel closes when a branch is created
* Update apps/webapp/app/services/upsertBranch.server.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Removed findUniques from WorkerGroupTokenService
* Made the parentEnvironmentId migrations safe
* Latest lockfile
* Update packages/cli-v3/src/commands/workers/build.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Move isValidGitBranchName to a separate file
* Move the sanitize fn too
* removeBlacklistedVariables moved to a separate file
* Moved deduplicateVariableArray to a separate file…
* Fix broken sanitizeBranchName import
* Another import fix…
* Improved blacklisted error message
* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* v4: Can now set project ref using the TRIGGER_PROJECT_REF env var
* Add changeset
* Improve the name of the resolvEnvVars function and move setting NODE_ENV into the dev supervisor
* Removed moved function
* Simulate "next steps" when calculating the next timestamp to prevent millions of iterations on dev schedues
* Add additional tests
* Fix typescript issue
* shard unit tests
* temp enable for all pushes
* fix test workflow
* update to latest vitest and only add to root package.json
* additionally use default reporter
* gather reports before uploading
* split up slow replication tests
* split up unit tests workflow
* move workflows to parent dir
* use new paths in parent workflow
* prevent artifact clashes
* we always need to create the reports dir
* speed up merge reports
* gather reports even when tests fail
* fix artifact patterns
* increase shards
* disable push trigger again
* improve dequeue snapshot test reliability
* Refill release concurrency tokens when a run is cancelled
* Improved release concurrency accounting system + a sweeper to auto-refill tokens for snapshots that are no longer the latest snapshot on a run (e.g. the run has moved to a new snapshot state)
* Fix order of arguments to the releasings sweeper
* Add a heartbeat for SUSPENDED snapshots, where when stalled will attempt to continue the run if unblocked
* v4: fix race condition when continuing run when blocked at the same time
* Attempt to fix flaky e2e test
* Make waitpoint race condition test less flaky
* Upgrade posthog-node to clear axios vulns
* Upgrade @slack/web-api to use a secure version of axios
* Update parse-duration to fix security vulns
* Mitigate against the ws DoS vuln
* Upgrade body-parser to 1.20.3 in the webapp
* Upgrade react-use to 17.5.1 to remove the fast-loops transitive dep
* Remove unused babel dev deps and config file that's no longer used
* Upgrade prismjs to 1.30.0 and bundle parse-duration now
* upgrade express to 4.20.0 to fix issue with XSS when redirecting
* Upgrade @conform/zod to 0.9.2
* runs replication leader lock expiration fix
* Allow configuring the container image --max-old-space-size using NODE_MAX_OLD_SPACE_SIZE
* Ability to configure the clickhouse keep alive settings
* Add some logging because we might not be able to do telemetry
* Add external log exporters and fix missing external trace exporters in deployed tasks
* Generate the external traceID correctly and exporter 3rd party logs with the external traceID as well
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* WIP adding a side menu panel to display incident statuses
* Fixes re-rendering bug and copy tweak
* cache the betterstack response using unkey
* Style the button to fit the panel colors
* improve route to add worker group, handles existing groups gracefully
* add option to remove default worker group from project
* separate project id field
* update supervisor readme with route changes
* Initial commit with a plan for what we’re going to do
* Some initial types and improved plan
* Add Waitpoint resolver
* Add resolver + status index
* Remove type + status index
* Only drop if exists
* Remove type index
* Update waitpoint list presenter to use resolver
* Added resolver to the engine
* Made the existing waitpoint list presenter more flexible
* Initial implentation ofr wait.forHttpCallback()
* Added the callback endpoint (no API rate limit)
* schema version
* Added jsdocs, removed schema version because of errors
* Show callback URL if it’s set
* Dashboard pages and panels
* Remove todos
* Added temporary icon
* Added a blank state
* Some tweaks and added a Replicate example
* Implement unwrap() for httpCallback
* Added unwrap to wait.forToken() as well
* Improved jsdocs
* Added docs
* Added unwrap to the token docs
* Show a dash if there are no tags
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make the timeout error safer
* Fixed migrations… should use id desc not createdAt desc
* Fixed page title
* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first
* Respect the max content length by getting the length of the body
* Added more docs details about the callback format
* Remove code comment
* Improved the error
* Added a hash to the HTTP callback URLs
* Add the apiKey to the API input type to fix TS error
* Return the error responses. They were being caught and not preserved
* The content-length header is required. Deal with an empty body
* Removed unused types
* Added some new span icons
* Reworked http callback to be a create call then just use wait.forToken()
* Added a changeset
* Updated the docs
* Updated the wait overview docs
* Simplify to just a call
* WIP stripping right back to waitpoints just having a URL associated with them…
* More deletions
* Remove missing icon
* Updated the changeset
* Add URL to the token return types
* Remove wait for http callback page
* Updated docs
* More tidying
* Type and import fix
* Remove unused import
* Some type fixes for the retrieve
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* remove dead code
* rename managed to shared runtime manager
* rename to resolve waitpoint for clarity
* add resolver id helper
* store and correctly resolve waipoints that come in early
* fix ipc message type change
* branded type for resolver ids
* add fixme comments
* remove more unused ipc schemas
* fix entitlement validation when client doesn't exist
* restore hello world reference workspace imports
* runtime manager debug logs
* prefix engine run logs
* managed run logger accepts nested props
* runtime suspendable state and improved logs
* require suspendable state for checkpoints, fix snapshot processing queue
* add terminal link as cli module so we can more easily patch it
* apply cursor patch
* add license info
* remove terminal-link package and add deprecation notice
* remove old patch
* remove terminal-link from sdk
* rename snapshot module
* add cli test tsconfig
* add run logger base type
* add snapshot manager tests
* fix cli builds
* improve QUEUED_EXECUTING test
* changeset
* make testcontainers wait until container has stopped
* require unit tests for publishing again
* avoid mutation during iteration when resolving pending waitpoints
* improve debug logs and make them less noisy
* always update poller snapshot id for accurate logs
* detach task run process handlers
* check for env overrides in a few more places and add verbose logs
* log when poller is still executing when we stop it
* add supervisor to publish workflow
* always print full deploy logs in CI
* Revert "avoid mutation during iteration when resolving pending waitpoints"
This reverts commit 87b0ce1e5b.
* disable pre
* print prerelease script errors
* Revert "disable pre"
This reverts commit 9403409637.
* misc fixes
* better debug logs
* add snapshots since methods and route
* prep for snapshots since
* improve deprecated execution detection
* update supervisor and schema
* properly log http server errors
* detect restore after failed snapshot fetch
* run and snapshot id can be overridden
* fix restore detection
* fix deprecation checks, move into snapshot manager
* less logs
* rename snapshot manager stop
* restore detection was moved into snapshot manager
* fix notifier logs
* make runtime manager status a debug log
* no need to attach runtime status twice
* findUnique -> findFirst
* sort snapshots by created at everywhere
* update testcontainers packages
* increase cleanup timeout and add better logs
* small tweaks
* decrease docker network size so we can have more of them
* add a test flow to check this all works
* await all engine.quit calls
* add docker diagnostics
* reduce number of test runs
* improve network attachment output
* add setup logs
* log redis setup errors
* add cleanup log type
* stop redis container if setup fails
* disable ipv6
* tidy things up a bit
* fix unit tests workflow
* decrease container cleanup timeout
* fix types
* fix webapp typecheck
* make testcontainers wait until container has stopped
* require unit tests for publishing again
* add failing test case
* make it pass
* add retry threshold ms env var and use it
* Combines the Running and Concurrency limit cols into 1
* Display a badge when a queue is at the concurrency limit
* Colors the Running/Limit column text amber if the concurrency limit is hit
* Turns the “Running” big number amber and shows “At concurrency limit” text
* BigNumber now handles big values using formatNumber and formatNumberCompact
Also includes some responsive improvements to make sure things wrap when it gets tight
* Adds a new col for showing how the queue is limited
* Reinstates a threshold for making very big numbers compact
* Added border to make the search bar not float
* Adds a tooltip to display the UTC, timezone and local time for DateTime
* Made it safer for SSR
* Makes the copy function into a hook and separate button + adds copy dateTime button to the date tooltip
* Adds an extra-small size copy button
* Show the offset (UTC +1) next to your local time
* Optionally don’t show the date tooltip. Defaults to true (always show)
* Tidy imports
* Refactor the tooltip content to remove duplicated markup
* Coderabbit suggestion to include support for minutes for some timezones like Nepal
* add terminal link as cli module so we can more easily patch it
* apply cursor patch
* add license info
* remove terminal-link package and add deprecation notice
* remove old patch
* remove terminal-link from sdk
* changeset
* Made a better AI icon and improved how it looks on the create new schedules inspector
* WIP adding kapa ai to the app
* WIP adding a new Ask AI button to the side menu
* WIP using the react example from the docs
* Align the AI button in the bottom bar
* Kapa widget now works
* Trigger the Kapa modal from the custom button
* Fix imports
* Adds Ask AI shortcut to Shortcuts panel
* Adds a new enter shortcut key
* Adds a prop so you can optionally hide the shortcut key
* Latest
* Moved the Kapa/Help stuff into a component, out of root
* WIP using onModalClose
* Creates a wrapper provider to block shortcuts while kapa modal is open (has bugs)
* Fixes button alignment
* Hide the shortcut key at the button layer
* Fix for enable/disable shortcut keys globally
* Kapa is working
* You can bring up the shortcut keys without opening the help panel
* TODO remove listeners
* remove imports and fix invalid tailwind class
* style kapa widget as best i can
* Remove Kapa event listeners
* Allow passing in a query
* Open the AI widget if there’s a URL param
* Much cleaner implementation for Kapa
* Trying to auto-open the Kapa widget when the page loads
* Delay opening the widget because it was causing issues
* Improved Kapa widget colors
* Added an AI help link to the CLI
* We don’t need this anymore
* Exit with 1, indicating an error. This is important for CI
* Removed old auto-opening code
* Added some code comments to explain some of the annoying stuff
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key
* Safely release the reader lock
* Fix for Schedules list page slow loading
Getting BackgroundWorkerTask was very slow (Prisma was getting every single one…)
* Same fix for the upserting of schedules in the dashboard
* Fix priority tests by dequeuing in sequence
* Expect this test to fail until we support parallel dequeuing of an env with streaming
* Fix one and expect fail from the other
* Fix for batch trigger test dequeuing
* Fixed pending version test
* Locked task runs will now require queues and tasks to be in the locked version
* Client errors caught in a run function now will skip retrying
* Extracted out the trigger queues logic
* extract validation, idempotency keys, payloads to concerns
* Extracted out a bunch of more stuff and getting trigger tests to work
* Add queue and locked version tests
* Deadlock detection WIP
* more deadlock detection
* Only detect deadlocks when the parent run is waiting on the child run
* Improve the error experience around deadlocks
* A couple tweaks to make CodeRabbit happy and fixing the tests in CI
* Fixed failing test
* Changeset
* wip
* Make sure to scope queries to the runtime env
* Added error throwing tests to runlock
* Added some more run lock tests
* Added customConditions so the test can use the Logger
* Fix for JSON error
* Added a span to the run lock with a name
* Increase v4 visibility timeouts
* Added some logging for #continueRunIfUnblocked: run has no checkpoint
* If a run is finished or pending cancel, don’t try and continue it
* Remove setting the invisibility timeout because it’s already done in the dequeue Lua script
* Remove orphaned queue items when dequeuing
* Added an ack to the visibility timeout test
* refactor: docker compose migration
* fix compose download link
* set static name for electric container
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Add a custom react hook to enable fuzzy searching in item lists
* Use fuzzy filtering in the tasks view list
* Use fuzzy filtering in the test tasks list
* Remove the old tasks filtering react hook
* Allow empty payloads in test runs
* Expose a basic linter configs in the code editor component
* Touch up the linting error elements in the code editor component
* Added some Redis worker debounce tests (one failing that reproduces a bug)
* Added some tests for acking
* Added a deduplicationKey to prevent acking when items are queued
* The worker passes the deduplicationKey back in for acking
* Improved logs and removed events from test
* Fixes runLimiter check on #dequeueRuns
This was causing issues in the dev environment with logs being spammed with `[DevSupervisor] dequeueRuns. Run limit reached, trying again later` and making the worker unresponsive and lose connection.
* Create polite-impalas-care.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* multiple queue consumer in the same supervisor instance
* disable pre
* Revert "disable pre"
This reverts commit 4b154398982758d09667531e3a64761da83664eb.
* Completed batch waitpoints when we completed the BatchTaskRun
* Try complete the batch faster now it’s being used operationally
* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
* WIP on secret env vars
* Editing individual env var values is working
* Sort the env vars by the key
* Deleting values
* Allowing setting secret env vars
* Added medium switch style
* Many style changes to the env var form
* “Copy text” -> “Copy”
* Draw a divider between hidden buttons
* Env var tweaks
* Don’t show Dev:you anymore
* Grouping the same env var keys together
* Styles improved
* Improved styling of edit panel
* Fix bun detection, dev flushing, and init command (#1914)
* update nypm to support text-based bun lockfiles
* add nypm changeset
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add changesets
* chore: remove pre.json after exiting pre mode
* init command to install v4-beta packages
* Revert "chore: remove pre.json after exiting pre mode"
This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.
* make init default to cli version for all packages
* Release 4.0.0-v4-beta.1 (#1916)
* chore: Update version for release (v4-beta)
* Release 4.0.0-v4-beta.1
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Both run engines will only lock to versions they can handle (#1922)
* run engine v1 will only lock to v1 deployments
* run engine v2 will only lock to managed v2 deployments
* test: create background worker and deployment with correct engine version
* Add links to and from deployments (#1921)
* link from deployments tasks to filtered runs view
* jump to deployment
* don't add version links for dev (yet)
* Fix current worker deployment getter (#1924)
* only return last v1 deployment in the shared queue consumer
* be explicit about only returning managed deployments
* Add a docs page for the human-in-the-loop example project (#1919)
* Add a docs page for the human-in-the-loop example project
* Order guides, example projects and example tasks alphabetically in the docs list
* Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles
* fix retry spans
* only download debug logs if admin
* add nypm changeset
* pull out env override logic
* use runner env gather helper
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add logger
* add execution heartbeat service
* add snapshot poller service
* fix poller
* add changesets
* create socket in constructor
* enable strictPropertyInitialization
* deprecate dequeue from version
* start is not async
* dependency injection in prep for tests
* add warm start count to all controller logs
* add restore count
* pull out run execution logic
* temp disable pre
* add a controller log when starting an execution
* refactor execution and squash some bugs
* cleanup completed docker containers by default
* execution fixes and logging improvements
* don't throw afet abort cleanup
* poller should use private interval
* rename heartbeat service file
* rename HeartbeatService to IntervalService
* restore old heartbeat service but deprecate it
* use the new interval service everywhere
* Revert "temp disable pre"
This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.
* add changeset
* replace all run engine find uniques with find first
* Release 4.0.0-v4-beta.2 (#1928)
* chore: Update version for release (v4-beta)
* Release 4.0.0-v4-beta.2
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Remove batch ID carryover for non-batch waits (#1930)
* add failing test case
* do not carry over previous batch id when blocking with waitpoint
* delete irrelevant test
* Delete project (#1913)
* Delete project
- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in
* Mark the project as deleted last
* Fix for overriding local variable
* Added a todo for deleting env queues
* Remove todo
* Improve usage flushing (#1931)
* add flush to global usage api
* enable controller debug logs
* initialize usage manager after env overrides
* add previous run id to more debug logs
* add changeset
* For secret env vars, don’t return the value
* Added a new env var repository function for getting secrets with redactions
* Test task for env vars
* Delete heartbeat file, merge mess up
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Saadi Myftija <saadi.myftija@gmail.com>
* drive run heartbeats from child process
* track last heartbeat
* add changeset
* use v4-specific heartbeat interval env var and decrease to 20s
* delete old heartbeat service
* remove heartbeat handler during execution cleanup
* add flush to global usage api
* enable controller debug logs
* initialize usage manager after env overrides
* add previous run id to more debug logs
* add changeset
* Delete project
- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in
* Mark the project as deleted last
* Fix for overriding local variable
* Added a todo for deleting env queues
* Remove todo
* update nypm to support text-based bun lockfiles
* fix retry spans
* only download debug logs if admin
* add nypm changeset
* pull out env override logic
* use runner env gather helper
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add logger
* add execution heartbeat service
* add snapshot poller service
* fix poller
* add changesets
* create socket in constructor
* enable strictPropertyInitialization
* deprecate dequeue from version
* start is not async
* dependency injection in prep for tests
* add warm start count to all controller logs
* add restore count
* pull out run execution logic
* temp disable pre
* add a controller log when starting an execution
* refactor execution and squash some bugs
* cleanup completed docker containers by default
* execution fixes and logging improvements
* don't throw afet abort cleanup
* poller should use private interval
* rename heartbeat service file
* rename HeartbeatService to IntervalService
* restore old heartbeat service but deprecate it
* use the new interval service everywhere
* Revert "temp disable pre"
This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.
* add changeset
* replace all run engine find uniques with find first
* run engine v1 will only lock to v1 deployments
* run engine v2 will only lock to managed v2 deployments
* test: create background worker and deployment with correct engine version
* update nypm to support text-based bun lockfiles
* add nypm changeset
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add changesets
* chore: remove pre.json after exiting pre mode
* init command to install v4-beta packages
* Revert "chore: remove pre.json after exiting pre mode"
This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.
* make init default to cli version for all packages
* WIP on hiding the queued time
* Fix for position/sizes of spans
* Fix for duration squashing
* Tweaked some styles on the timeline
* Fix for retry spans appearing in the wrong place
* Added “Q” shortcut key
* Fix for in progress span durations
* WIP upgrade to v4 docs
* WIP
* Upgrade to v4 docs
* Add wait for token docs and improve upgrade guide
* Added known issues to upgrade guide, and moved the migration section higher up
* Update to say "There are no known issues at the moment.”
* Docs upgrade guide package version. Need to use exact packages for the CLI to work
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* attach all run controller logs to the run
* make controller-level pre-suspend wait duration configurable
* snapshot status should remain EXECUTING for short retry delays
* add suspended process error
* ensure clean slate before waiting for next run
* treat immediate retries as warm starts
* fix for finished runs waiting forever
* Use completedAt instead of updatedAt
* Disable verbose SSE logs
* Engine rate limiter: whitelist engine worker-actions
* Show the compute column in the runs table
* The compute column should be invocation + compute cost
* don't hardcode max old space size in deployment images
* flags: treat underscores as hyphens
* append attempt number to runner name if >1
* improve retry spans for oom errors
* Make sure releaseConcurrencyOnWaitpoint is set on taskQueue
* Allow task queue to have releaseConcurrencyOnWaitpoint (required to fix custom queues)
* Test for releaseConcurrencyOnWaitpoint
* Added a warning if you use the same queue twice with different settings
* Improved the error some more
* set correct worker name on create
* dev: kill run process on failed cancel attempt
* add missing prod timeline events
* add more debug logs after api call failures
* import events are always visible
* set dequeued at on new runners
* skip fork event test
* fix timeline test for import event
* Better trace icon
* Better Waitpoint token icon
* Fix for bad jsx
* Warm and cold start icons
* Tooltips now use a <Portal> so they appear on top
* Warm start components
* Added warm start markers to the Run page and inspector
* Fix for getting the correct value from the metadata
* Better trace icon, with fallback to a passed in one
* Removed unused isWarmStart function
* make worker nodetype configurable
* make max dequeue count configurable from supervisor
* deterministic runner ids
* update runner id on restore
* disable resource monitor
* explicit run controller type imports
* enhance debug logs
* run engine: add ability to selectively disable release concurrency queue consumers via env var
Also added some additional logging
* actually pass the disableConsumers option through to the release concurrency system
* Fixed CLI e2e failing tests
* v4: eagerly fork child process before warm start
Also fixes an issue where the attempt span events weren't coming through in the partial spans
* Making flushing a safer operation
* Add changeset
* Improve attempt metrics when in a warm start, and pass the STYLE_VARIANT for a warm or cold start on attempt spans
* Improved the react email docs and added a video
* Copy
* Added a link to the cursor rules
* Added satori example
* Copy updates
* Added react pdf link
* remove dummy metrics from heartbeat
* fix heartbeat timeouts..
* reset clock when resolving waitpoints
* optionally set metadata url when scheduling run
* don't use global vitest as it's outdated
* don't run test files in parallel
* handle special graceful shutdown code
* exit codes for success and failure can now be set and overridden
* ensure immediate cleanup in tests
* update lockfile after conflict
* AI SDK sql generator and approval example WIP
* Create some nice spans and extract the tools out so we can get the type of the streams
* Enable AI SDK telemetry
* Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons
* Created a separate d3-chat example, split out from the openai-agents example
* Fixed the tool options being passed to metadata in `ai.tool`
* Add a link to the run
* Slightly improved design
* Add a crawler task using crawl4ai
* Use a tool to get the userId
* Couple of tweaks
* Adding markdown rendering to assistant messages and added an e2b based chart rendering task
* Backup to anthropic
* Add changeset
* Delete old wait-for pages & create new wait-for-token page
* Link to new wait-for-token page and update description
* Adds an upgrade to v4 coming soon docs page
* Removes unncessary property from Note component
* Adds DevPresenceBanner to the Run page
* Adds DevPresenceBanner to the Run list page
* Makes the DevConnection component reusable and moves components to the DevPresence.tsx file
* SideMenu: only show dev presence when project engine === V2
* WIP on disconnected banner on v3 and v4
* v3 dev connection working. Made it slightly red
* Only show the disconnected banner on v3 if the run is not completed
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* logs for optional services
* print env vars on startup in debug mode
* routes need to explicitly ask to keep connection alive
* log indicators for now
* make workload api listen host configurable
* expose supervisor metrics and make more configurable
* configurable pull secrets, no defaults
* remove restore route
* run controller to handle queued executing
* fix v3 deploys in v4 project
* update admin worker route
* only start pod cleaner et al in k8s mode
* set new worker group as default if none yet
* make image ref optional
* checkpoint image ref is optional for output as well
* export feature flag const
* one last image ref type fix
* make runner intervals configurable
* ability to set arbitrary env vars on new runners
* set default runtime back to node 21
* move all runner env vars to the same section
* cli: add dev lock file to prevent 2 dev processes running at the same time in the same dir
* Make sure the .trigger dir exists before creating the dev.lock file
* logs for optional services
* print env vars on startup in debug mode
* routes need to explicitly ask to keep connection alive
* log indicators for now
* make workload api listen host configurable
* expose supervisor metrics and make more configurable
* configurable pull secrets, no defaults
* remove restore route
* run controller to handle queued executing
* fix v3 deploys in v4 project
* update admin worker route
* only start pod cleaner et al in k8s mode
* set new worker group as default if none yet
* Ensure the task list uses the dev environment of the logged in user
* Ensure the test task action uses the correct dev environment
* Fixed issue with missing case statement scope, wrapping in block
* Updates the large login buttons to the new purple style
* Updates the dev checking connection icon to include 3 dots on the screen
* Fixes svg property names
* Removed old condition that handled old v2 tables
* Adds support for a single button to be shown on hover in a table cell
* Makes the “hiddenButtons” support a single button instead of adding a new property
* Format the docs link for Schedules like the other for consistency
* D3 demo WIP
* Agent working and completing token successfully
* Sending OpenAI Agent SDK spans through to the platform now works
* A couple of perf tweaks for enriching events
* New time period filter (permanently displayed)
* Batches, and fix for blank state
* Waitpoint token filtering
* Tags query: remove things we’re not using
* Run tag and waitpoint tags use startsWith for faster search
* Fix for run page on wrong env. Added schedule last triggered column
* Removed the Redis pubsub, just use the presence key
* Improve the dev presence responsiveness
* The CLI presence connection recovers when the webapp is restarted
* Dev schedules are now working for v4
* Refactored to make the dev presence stuff
* Got rid of stupid extra /dev, added a connecting state with icon
* Remove unused Redis client
* Move to indigo primary button to better fit in with the app UI style
* Updates Pause queue button to secondary
* Updates some blank state CTA buttons to primary
* WIP
* Completing waitpoints using public access tokens
* fixed example
* adding agent-loops reference, adding useWaitToken react hook, adding a way to throw an error to output from the run, adding an icon option to logger.trace
* Fixed metadata system
* Check if the waitpoint is COMPLETED already in the complete endpoint and return true
* New format for the functions instead of “onWait() task”
* Make the icon sizes match the hero icons
* Updating icons for the new task hooks
* Adds new icons to the RunIcon.tsx component
* Improves the size of the menu icons
* Fixes the missaligned environment dropdown and dev connected button
* Changes the button colours so they are all unique
* More icon updates
* Adds new variables for main page icons
* Improves icons sizes
* Using new color variables in the side menu and blank states
* Adds preview environment color variable
* Align the env icon in the menu
* Updates the env icons to be the correct size and padding to match hero icons
* Label uses new colors, removes unused cases and new env icons
* Lower case env label
* Removes unused code
* Use full env title
* Organize imports
* More variants of the runs icon to work at smaller sizes
* small padding adjustment
* reformat init function span
* Fix last init formatting
* New colour for settings
* Organize switch statement for icons nicely
* Use new color variable
* Update icon svgs
* Renamed lifecycle hook icon to Function icon
* Renamed function icon file name
* Added waitpoints/tokens to the sidebar
* Added indexes to the Waitpoint time for filtering
* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now
* MVP of waitpoint token page
* Added status
* Expiry of timeout/ttl
* Improvements to the waitpoint table
* Improved columns and icon
* Changes from the RunTag copy on hover branch
* Fix for nested button error
* Added waitpoint tags to the DB/table
* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)
* Added tags to waitpoints
* Removed todos that have been done
* Added token support for releaseConcurrency. Also added a ton of JSDocs
* Added releaseConcurrency to the API token endpoint…
* WIP on waitpoint page filters
* Fix for tags filtering
* Waitpoint filters working
* Fix for badly named function
* WaitpointPresenter used from SpanPresenter
* Waitpoint detail panel WIP
* Fix for server client hydration issue with CodeBlock
* Selected waitpoint panel
* Added a blank state
* Added waitpoint docs link
* Fix for animated number going past the target
* Fix for the queue list pagination and upgrade status
* Engine version error for waitpoint token list
* RunTag component doesn’t get squished and hover behaviour is nicer
* Associating runs with waitpoints
* Added triggered icon
* Link directly to the waitpoint
* Fix for TS error on waitpoint retrieve
* Added CopyableText component, used for waitpoint id in the table
* Removed the confetti 🎊
* Deleted some old images
* Moved some schemas/types to core. Use `id` instead of `friendlyId`
* Added wait.listTokens() function. Made some changes to the types to make it nicer
* WIP wait.retrieveToken()
* wait.retrieveToken working
* Added data to retrieve token
* Separate ApiWaitpointPresenter completely
* Added completed time to the waitpoint detail panel
* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect
* Fix for applied idempotency key filter dropdown showing the id field
* Use parentheses to make sure the token list query respects idempotency key correctly
* Use the proper logger, and have a decent message with info to track the bug down
* Pass the org title into the Avatar
* Better error when failing to creating a manual waitpoint after X attempts
* Improve the new run engine Trigger/Batch trigger service class names
* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation
* Fixing the create tags in the new run engine trigger task service
* patch k8s client to allow field selector on informers
* add pod cleaner and tests
* add failed pod handler and tests
* remove supervisor test script for now
* Adds “plan” to the plan name if it’s a paid plan so they all format correctly
* Fixes menu item having incorrect height
* Adds gap between switch org menu items
* Moves main action button to right side in blank state
* Simpler way to add “plan” to the plan name
* Styled the plan badge a bit nicer
* Project page now matches the other settings page layouts
* Fixes colour of queue icon
* Fixes table blank state not spanning enough cols
* Organize imports
* Unified the blank state panels and improved some copy in them
* Improved the copy in the no deployed tasks blank state
* Make the upgrade icon solid and indigo
* Transition the hover state on the side menu section
* Replaces many props with a single ‘accessory’ React node on the Info Panel component
* Organize imports
* Organize imports
* Removes animation on the dropdown menu icon
* Improves Settings and Usage button states
* Side menu sections animate with easeInOut
* Incorporates docs button into the text and adds title to the clipboard field
* Icons look better as the small variant
* Improved the typography layout of the org name and small links
* Made the divide line show when the side menu scrolls
* Made the impersonating icon amber so it stands out more
* Adds an optional amber border to the org dropdown
* Delete the proxy app (was v2)
* Delete RunPresenterElectric
* Select the best proj/org/env
* Storing current proj/env in DB. Initial selection logic working with tasks page
* 2sm needed to be in the Tailwind merge list
* Move the task stream route (although we don’t actually use the env for now)
* Alerts moved from /v3
* API keys page moved from /v3
* Concurrency page moved from /v3
* WIP on side menu sections
* Improved the accordion animation
* Moved schedules from /v3
* More pages moved
* Move pages working
* Run page working
* Schedules working
* Moved deployments
* Alert pages moved
* Delete electric hooks, not used
* Started setting up blank states
* Test page working
* Removed “Select task” from the test page
* Some work on deployment page
* Style tweaks
* Redirect from project root to approriate env
* Improved env selector styling
* Fix for jsx errors
* Better min width on env selector
* Improved the env switching logic
* Added deployments to env routing
* Redirect deployments to the correct env
* Redirect run from proj to env
* JSX icon fix
* Only allow single env schedules from now on
* Remove env var count from the API keys page
* Move improvements and redirects
* Project settings moved
* Fix for scroll area on test page
* Tweaked the test design
* Made recent payloads column narrower
* Improved the test layout some more
* Added org icon, new project selector menu
* WIP on org switching menu
* Org switching is working
* New menu working well, removed old side menu items
* Buttons can now have a component name or an actual component for their icons
* Removed the Projects page, instead redirect appropriately
* Fix for broken blank states
* Minor run table improvements
* Removed unused switcher log and logic
* Concurrency page fix for invalid html, improved layout
* Minor improvements
* Moved the side menu to the project level
* Improved account styling
* Moved org settings pages (with redirects)
* Add current plan to billing side menu link
* Upgrade to get staging from env dropdown
* New env badge on concurrency limits page
* Show Run Engine version in span presenter
* New promote icon
* Concurrency limits page is the sum of engine v1 + v2 queues
* Fix for missing batch import
* Added currentConcurrencyOfEnvQueue function
* Basic avatar setting working
* Avatar setting is working
* You can change the color of your icon
* Avatar improvements
* Bugfix for mising prop
* Removed some old env badges
* Fixed replaying
* Removed EnvironmentLabel
* Old env badge deleted, changed everywhere to the new one
* Fix for Slack integration paths
* Fix for waitpoint completion form moving
* Bulk replay/cancel env fix
* Fix for alert webhook path
* Redirect projects/v3/* to project/*
* Fixes for CLI redirect routes
* Remove welcome email (unused)
* Change how we count schedules towards your limits
* Use new schedules limits when checking a schedule
* Added projectId back in to task queries (indexes)
* WIP dev presence
* CLI modal
* Moved things around and use Context
* Fix for p inside p
* Dev connected status on run page
* Correct dev env (not a teammates)
* Show disconnected message at the end
* Minor tweak on project dropdown icon padding
* Fix for inconsistent date format for presence
* Added a message when pushing to the billing page
* Center the team page
* Project settings page centered
* Improvements to the dev presence
* handle warm start service failure on supervisor side
* export zodfetch wrapper
* add warm start client
* rename to keepalive
* add missing warm start header
* make heartbeat and snapshot poll interval configurable
* create warm start client in constructor
* add warm start run debug log
* re-enable checkpoints and improve error messages
* reduce run pod container name cardinality
* move runner id generator into core
* fix runner id import
* log when no checkpoint client and we try to restore
* move run controller constructor to the top
* fix import
* support env overrides after restore
* fix and restructure dockerignore
* switch to using pnpm deploy
* pass webapp node image as build arg
* ensure pnpm is downloaded at build, not runtime
This just adds the main components for the MCP server. It hooks on the
existing `dev` command and can be enabled by passing the `--mcp` flag.
Currently only the `trigger-task` tool is exposed, which enables users
trigger tasks via MCP and see the resulting run.
Still WIP :)
* Delete v2 Stripe routes
* Delete v2 billing/usage pages
* Delete v2 integration pages
* Delete v2 project pages
* Deleted a load of components and services
* Deleted a load more components, presenters and services
* Deleted another 100 files or so…
* Removed old v2 paths
* Removed named icons from form titles
* Removed more string icons
* Delete NamedIcon
* Fixed some type errors
* Delete endpointApi
* Removed v2 from core/sdk
* Post merge fixes
* added explicit return types
* using the new sdk export without v3
* Delete old v2 file
* Added explicit return types because TS was complaining…
* Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue
* Fix for removal of NamedIcon in new route
* Removed strange eslintrc rule
* Use the new redis client
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* re2: fix @trigger.dev/core exports
* re2: WIP env based queue selection algo
* more wip
* WIP
* Get run engine tests to pass
* Adding tests for the fair dequeueing strat in the run engine
* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing
* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic
* Fixed webapp typechecks
* dev now depends on build, fixed supervisor typecheck
* Fixed run engine tests
* Fixed e2e tests
* dockerignore node_modules in subdirectories
* image tag action should handle re2 tags
* add supervisor containerfile
* add publish worker re2 workflow
* fix copypasta
* require branch check
* add more granular service control to supervisor session
* fix supervisor api domain for split setups
* remove default workload api domain
* option to disable workload api
* fix bool env var coercion
* Added describe to tests that were missing it
* Added a function to get the maxOldSpaceSize
* Make it easy to take `NODE_OPTIONS` and set the old space flag
* Added a zed task to rebuild the packages
* Moved isOOMRunError and added SIBABRT condition
* Deduplication flags function with tests
* Export flags file
* On TaskRunProcess, set max old space and deduplicate the flags with priority order
* Move retrying logic to a separate function, it was getting very messy
* Created new test file for attempt failures
* Allow setting retry settings for tests
* Some retrying tests, including OOM
* More failure condition tests
* Fix for OOM retrying
* Complete the attempt span if it was an OOM error
* Remove old broken import
* Fixed order of exports
* Fix for docker-provider checkpoints import
* Adds “Explore by build extension” to the intro page
* Adds instructions for clearing the build cache
* Add a tip for running trigger dev + next dev concurrently
* left align table headers
* Adds trouble shooting for pino
* Adds corepack bug and workaround
* Removes reference to undici as it messes with spans
* Left align table headers
* Adds example for exporting logs to Axiom
* In dev, the worker group is optional when triggering tasks (the master queue is defined by the environment). Also deprecated the TaskEvent.isDebug column and using TaskEventKind.LOG instead for debug events
* Fixed a couple of type issues
* More type fixes
* bump worker version
* Suggested glossary for the RunEngine, TBC
* Removed BatchTaskRun changes from this branch, they were done in main
* Set the BatchTaskRun status to completed when all runs are completed
* When dequeuing respect passed in maxResources
* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds
* Didn’t hit save… the new props when triggering tasks passed through
* Idempotency expiration + waitpoint edge case
* WIP on creating checkpoint, parking for now
* fix worker routes
* upgrade webapp node types to support generic event emitter
* separate event bus handler singleton and run failure alerts
* duration waits
* fix execution snapshot debug spans
* task waits
* fix event bus types
* temporary fix for react hook run handle type
* disable run notifications for now
* convert any typecasts to expect errors to more easily fix later
* fix webapp types after node types upgrade
* updateEnvConcurrencyLimits across marqs and the runqueue
* Pass proper values into the run engine
* RunQueue settings and removed unused rebalancing workers
* Remove rebalancing prop
* Tidied more things up
* Update/remove queue limits for MARQS and RunQueue
* taskQueue/concurrencyLimit changes ported back into the RunEngine
* Reworked completing waitpoints to improve performance and reduce race conditions
* Improved test robustness
* Down to a single run lock only when a run is totally unblocked and ready to continue
* warm starts, worker notifications, wait fixes
* Fix for Run Engine poll interval env var
* Expect the waitpoint to be completed quickly
* If a run is locked then it’s too late to expire it
* Added VALKEY_ env vars and plugged them into the run engine
* Extracted and updated the guard queue function so it can be used when batching
* Added logging and universal concurrency changes to trigger task v1
* Added notes back in
* Bump @trigger.dev/worker to 3.3.7
* reportInvocationUsage for the runAttemptStarted event
* improve execution snapshot span debug span start times
* Unfriendly IDs
* update lockfile
* Created a shared determineEngineVersion function
* disable unfinished commands
* save new cli config to different location, misc fixes
* add basic engine version check via current deploy
* new run engine will default to node 22 runtime
* block some actions for projects on previous run engine
* fix worker group tests
* fix triggerAndWait test
* one typescript version to rule them all
* redlock type patch
* fix type issues caused by ts-reset
* improve cleanup scripts
* add missing socket.io dep
* fix run notification handler type
* fix worker group test again
* generate prisma client for e2e tests
* remove worker group tests for now
* prevent image pull rate limits during unit tests
* increase timeout for queue concurrency limit test
* generate prisma client for preview release
* same node types everywhere
* Updated engine readme, removed legacy system notes
* use default machine preset from platform package
* worker instances plural in schema
* disable pnpm update notifications
* return worker group details from connect call
* add workers admin route
* fix heartbeat route return type
* move deployment labels to core apps
* refactor run controller env schema
* Add firstAttemptStartedAt to TaskRun
* RunEngine 2.0 batch trigger support (#1581)
* Make it clear when BatchTriggerV2Service is used
* Copy of BatchTriggerV2Service
* WIP batch triggering
* Allow blocking a run with multiple waitpoints at once. Made it atomic
* Removed unused param
* New batch service
* Pass through the parentRunId and resumeParentOnCompletion
* Use the new batch service, and correct trigger task version
* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point
* Removed the $transaction and early exit if nothing changed
* Adedd a simple batch task to the hello world reference catalog
* Fix for batch waits not working
* Added parentRunId in a couple more places
* Removed waitForBatch log
* Added another parentRunId
* Expanded the example to include all the different triggers
* More changes to blocking to support continuing after idempotent completed runs
* Fix for the wrong type when blocking a run
* remove @map
* optimise worker auth query
* add engine version header to core api client requests
* remove unique constraint for default group id
* consolidate migrations
* the first managed worker becomes the global default
* Debug events off by default, added an admin toggle to show them
* worker group name can't be an empty string
* add exec helper to core
* move machine resources to core
* add pre-dequeue callback to determine max resources
* optionally skip dequeue
* bump worker package
* move worker to core
* fix ReadableStream type error
* fix another type issue
* update a few more tsconfigs
* add metadata changes introduced in #1563
* Run Engine 2.0 trigger idempotency (#1613)
* Return isCached from the trigger API endpoint
* Fix for the wrong type when blocking a run
* Render the idempotent run in the inspector
* Event repository for idempotency
* Debug events off by default, added an admin toggle to show them
* triggerAndWait idempotency span
* Some improvements to the reference idempotency task
* Removed the cached tracing from the SDK
* Server-side creating cached span
* Improved idempotency test task
* Create cached task spans in a better way
* Idempotency span support inc batch trigger
* Simplified how the spans are done, using more of the existing code
* Improved the idempotency test task
* Added Waitpoint Batch type, add to TaskRunWaitpoint with order
* Pass batch ids through to the run engine when triggering
* Added batchIndex
* Better batch support in the run engine
* Added settings to batch trigger service, before major overhaul
* Allow the longer run/batch ids in the filters
* Changed how batching works, includes breaking changes in CLI
* Removed batch idempotency because it gets put on the runs instead
* Added `runs` to the batch.retrieve call/API
* Set firstAttemptStartedAt when creating the first attempt
* Do nothing when receiving a BATCH waitpoint
* Some fixes in the new batch trigger service… mostly just passing missing optional params through
* Tweaked the idempotency test task for more situations
* Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️
* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch
* Support for the same run multiple times in the same batch
* Small tweaks
* Make sure to complete batches, even if they’re not andWait ones
* Export RunDuplicateIdempotencyKeyError from the run engine
* Latest lockfile
* Trigger with a machine (old run engine)
* RE2, allow setting machine when triggering
* Fix for new glob patterns
* add max run count to dequeue from version route
* add worker instance name env var and header
* queue consumer pre skip callback
* poll for more runs after final execution errors
* fix dequeue search param schema
* add shortcut to debug switch
* expose run engine timeouts as env vars
* make warm start durations configurable
* add optional status to json reply helper
* fix preSkip hook, add debug logs
* BLOCKED_BY_WAITPOINTS -> SUSPENDED
* exit controller when run suspended
* check if already replied before http reply
* run controller will wait for next run after the current one is suspended
* cancel run button shortcut
* minimal event repository environment type
* fix update metadata call
* run suspension and misc fixes wip
* change debug shortcut to shift + D
* Started work on the Dev supervisor
* Formatting
* Fix for bad imports
* Before rebuilding SSE
* Presence updating from the CLI working via SSE
* add worker notification debug logs
* send run:stop when exiting run phase
* skip current snapshot poll on worker notification
* add more logs and route to submit run debug logs
* add worker and runner ids to snapshots
* improve run notification debug logs
* add workload debug log route
* misc run controller fixes and refactor
* prevent parallel execution of critical functions
* update bun to 1.2.1
* WIP with dev dequeuing
* Method to convert friendlyIds to non-friendly, do nothing with actual ids
* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2
* Runs with ttls were getting immediately expired… oops.
* Pass the Waiting for deploy reason through, so we have it on the execution snapshots
* Fixed the logic for getting the right background worker for a run
* Use the correct ID when dequeuing…
* determineEngineVersion is now fully functional
* Rate limiter ignores the dev endpoints
* Retrieving a batch gives you the runIds
* Set a unique version for the RE2 BatchTaskRun
* add provisional changeset
* The start of dev run execution is working
* First dev run working
* Moved the dev run controller closer to what Nick did with the managed one
* export exec output type
* Heartbeat fix: don’t heartbeat if _isHeartbeating == false
* Dev runs get notifications, some dev bug fixes
* Improved logging or dequeuing
* We need to dequeue runs from the latest version too, for triggerAndWait
* Ported Eric’s validateWorkerManifest with nicer errors
* When flattening an idempotency key if part is undefined, return undefined
* Dev logging fixes
* Remove sigterm listener
* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment
* Deleted some old files. Renamed “managed” to “deploy”
* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)
* Dev master queues should work differently
* Deleting old workers
* Added debounce function to core
* Improvement to canceling
* WIP on debounce canceling on socket disconnection
* Added environment data to execution snapshots
* Dev runs that have stalled get “Canceled” with a reason explaining why
* Show CLI messaged when a connection to the platform is lost/restored
* Fix TriggerTask after merge
* Add trigger task v2 max attempts, replace some findUniques
* Port the new queue logic to the run engine
* More fixes post-merge
* We weren’t setting a `retryConfig` up for the tests… it’s now required
* Start the Redis worker inside the Run Engine… 🤦♂️
* Trying to make the testcontainers more reliable
* Added keyPrefix: "engine:”
* Badly placed bracket in trigger task
* Better Redis namespacing
* Fix for expired run not getting removed from the queue
* Don’t create a redis client in the testcontainers, return the redisOptions instead
* Cleanup redis client in the run lock tests
* Fix for the RunQueue not supporting keyPrefix
* Updated more of the RunQueue scripts rebalancing
* Trying to make Redis more robust in the tests…
* Improved test resiliciency more
* Fix for delays (checkpoint check)
* Increase the timeout slightly to fix ttl test
* Added priority support when triggering
* More wip trying to make test containers more reliable
* batchTriggerAndWait test is still failing… some wip to try fix it
* Fixed redis tests now we’re not providing a client
* Separate Redis clients for the run engine worker/queue/runlock
* Made the wait for duration test more resilient
* Added idempotencyKeyExpiresAt to Waitpoints
* Waitpoint timeouts and idempotency expiry
* Use finishWaitpoint, removed extra worker job
* Added waitpoint idempotency tests
* Creating resume tokens is working
* Some improvements to the resume tokens
* Moved resumeTokens to just be wait functions 🥳
* Delete old RuntimeManagers
* Wait for token is working
* Better test for the wait tokens
* Improved the test task some more
* Hide the accessories in the span inspector
* WIP on waitpoint inspector
* WIP on complete waitpoint form
* Span overview panel can be changed based on the entity type
* Improved the waitpoint display
* WIP on completing waitpoint form
* Use the existing CodeBlock for the tip
* Style improvements
* Complete waitpoint
* All waitpoint sidebar variants
* Waits now use a pause icon
* Durations waits use the API to create/block with a waitpoint, not the runtime
* Fix for engine.blockRunWithWaitpoint required org id
* Removed old wait code from the run controllers/task run process
* Form action for skipping a datetime waitpoint
* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)
* Fix for glitchy hourglass animation
* Completed waitpoints display better
* Increase Redis maxRetriesPerRequest to 20 (default)
* Completing and skipping waitpoints is working
* Remove the database prisma dev command, since we need to use create only now. Updated docs
* Added skip timeout, reworked the UI
* Tweaked spacing
* Added payload limit to waitpoint token completion from dashboard
* Test idempotency works on wait.for and wait.until
* Moved the worker-actions to /engine/ from /api/
* Moved dev engine endpoints to /engine/ from /api/
* Separate /engine/ rate limiter
* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet
* WIP post-merge conflicts
* Set taskEventStore column in the new engine
* Remove duplicate keys
* Post-merge fixes
* Fix for span merge layout
* Use executedAt instead of firstAttemptStartedAt
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Removed unnecessary classes from Dialog
* Display large modal with code and copy button
* Fixes button position if chrome is shown
* Removes the duplication of the Highlight component
* Record cold start and execution metrics on attempt executions. Add cold start metrics as span events on attempt spans and display them in the run dashboard
* Add deployed tasks run timeline metrics
* Add Dequeued event to run timeline and cleanup the run timeline code
* Adds variants to storybook
* WIP adding new span styles
* Added offset progress bar animation
* More storybook states
* Adds support for the full vertical span to show the same state
* Adds error state to timelineLine
* Added additional state
* Added more line styling
* Added progress state to dequeued
* Added another state to storybook
* Fixed classname error
* Updated styles for the span timeline points
* Fixes alignment of timeline follow cursor indicator
* Adds help text tooltip to timeline span type titles
* Fixes type error
* Tweaked wording of tooltips
* Fixed type error (check this)
* Moved isAdmin to a higher level
* removed unused admin props
* Removed unused Admin filter
* Fixed border styling
* made the opacity of the timeline states 30% less
* Undo type cast
* Added a diminished style that’s used for spans (grey progress bar)
* Adds new storybook state
* Fixed timeline state
* Removed state if span isn’t the first
* Changed the timestamp span icon
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Improve the MarQS priority system by moving future messages into the LRE worker and using a priority timestamp offset to define priority in messages
* Add a test to make sure priority offsets don't unfairly favor environments
* requeuing should clear concurrency sets
* Heartbeats should only reschedule existing heartbeat jobs
* Fix type error
* Moves API reference to a new dropdown section
* Move toubleshooting higher up
* Separates the reference Overview page into new pages
* Removed Projects api and redirects old page to overview
* typo
* Format the machine override example code
* Adds note to add `fluent-ffmpeg` to external
* Updates onFailure with more clarification
* Adds debugging in vs code to Troubleshooting
* Removed link
* New MarQS method requeueMessage, an atomic version of replace message
* Remove redundant call to remove the message from the env queue in requeueMessage
* run engine v1: orgs are no longer considered for concurrency
* Add reserve concurrency concept to allow waiting to resume parent tasks to release concurrency at the env level for child tasks to use (or else there is a deadlock). WIP recursive tasks
* child tasks inherit the queue timestamp from their parent tasks to prioritize completing child tasks based on when their parent started
* handle reserve concurrency with recursive deadlocks
* Finish docs update for concurrency
* Some fixes from badge conflict resolution
* WIP priority queues
* Implement MarQS priority queues
* Fix the migrations
* Adds more Prisma documentation
* Adds more info and a diagram to improve idempotencyKeyTTLs
* Adds more clarity to the run with TTL docs section
* Limits page copy improvements
* Adds schema properties to the alert webhooks
* typo
* Adds feature list to the sveltekit plugin page
* Removed Migrations section in side menu
* Tidy up the table links
* Creates a community packages section
* Small tweak
* Adds info and links to community packages
* Updated mintlify broken links version in workflow file
* fixed broken link
* add timeout support to sendWithAck
* coordinator will retry completion submission
* actually retry
* increase default retries
* something went wrong there, add this back in
* add changeset
* don't crash run on lazy attempt errors and count on retry
* remove abort controller after checkpoint in all cases
* duration checkpoints
* manual checkpoints
* add busybox to coordinator image
* add async toggle to process in background
* remove early returns
* add changeset for manual checkpoint schema
* Uses image cards for the frameworks
* Removes old snippets
* New AI agents side menu section
* WIP adding new ai agent pages
* Better overview page
* More copy added to the agent example pages
* Copy improvements
* Removes “Creating a project” page and side menu section
* Fixes broken links
* Updates to the latest Mintlify version, fixes issues, changes theme
* Adds descriptions to the main dropdown menu items
* Reformatted Introduction docs ‘landing page’
* Retry heartbeat timeouts by putting back in the queue (#1689)
* If there’s a heartbeat error and no attempts we put it back in the queue to try again
* When nacking, return whether it was put back in the queue or not
* Try and nack, if it fails then fail the run
* Consolidated switch statement
* Fail executing/retrying runs
* OOM retrying on larger machines (#1691)
* OOM retrying on larger machines
* Create forty-windows-shop.md
* Update forty-windows-shop.md
* Only retry again if the machine is different from the original
* Kubernetes OOMs appear as non-zero sigkills, adding support for treating these as OOMs
* Complete the original attempt span if retrying due to an OOM
* Revert "Complete the original attempt span if retrying due to an OOM"
This reverts commit 5f652c6212.
* chore: Update version for release (#1666)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.3.14
* Set machine when triggering docs
* Batch queue runs that are waiting for deploy (#1693)
* Detect ffmpeg OOM errors, added manual OutOfMemoryError (#1694)
* Detect ffmpeg OOM errors, added manual OutOfMemoryError
* Create eighty-spies-knock.md
* Improved the machines docs, including the new OutOfMemoryError
* chore: Update version for release (#1695)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.3.15
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created (#1696)
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created
* Add env var for partition window in seconds
* Make startCreatedAt required in task event store
* Don't create an attempt if the run is final, batchTriggerAndWait bad continue fix (#1698)
* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)
* Don’t create an attempt if the run is already in a final status
* Don’t get all the columns for the query. Improved the logging.
* Added a log to the batch example
* Filter out the undefined values
* Fix missing logs on child runs by using the root task run createdAt if it exists (#1697)
* Provider changes to support image cache (#1700)
* add env var for additional pull secrets
* make static images configurable
* optional image prefixes
* optional labels with sample rates
* add missing core paths
* remove excessive logs
* Fix run container exits after OOM retries (#1701)
* remove unused imports
* tell run to exit before force requeue
* handle exit for case where we already retried after oom
* improve retry span and add machine props
* don't try to exit run in dev
* Upgrade local dev to use electric beta.15 (#1699)
* Text fixes
* Removed pnpm files
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* WIP with webhook SDK function and types
* JSDocs added to the schema
* Webhooks are working
* Expanded the alert docs
* Remove duplicate export of waitUntil.js
* Use uncrypto
* Don’t rate limit webhooks
* Create slow-olives-fix.md
* remove unused imports
* tell run to exit before force requeue
* handle exit for case where we already retried after oom
* improve retry span and add machine props
* don't try to exit run in dev
* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)
* Don’t create an attempt if the run is already in a final status
* Don’t get all the columns for the query. Improved the logging.
* Added a log to the batch example
* Filter out the undefined values
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created
* Add env var for partition window in seconds
* Make startCreatedAt required in task event store
* OOM retrying on larger machines
* Create forty-windows-shop.md
* Update forty-windows-shop.md
* Only retry again if the machine is different from the original
* If there’s a heartbeat error and no attempts we put it back in the queue to try again
* When nacking, return whether it was put back in the queue or not
* Try and nack, if it fails then fail the run
* Consolidated switch statement
* Fail executing/retrying runs
* missing construction option in `AudioWaveformExtension`
* Create lemon-fireants-repair.md
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Added isPrismaRetriableError()
* Retry completeBatchTaskRunItem if they fail because of a retriable Prisma error
* Retry using Redis worker
* Handle more retriable errors. Add special condition in for race condition error
* Added Postgres connection_timeout with default 20s
* Added a simple batchTriggerAndWait example
* Move the task run heartbeats to RedisWorker
* Move alerts to redis worker, improving redis worker
* Fix typecheck errors
* Use single threaded tests for redis worker
* Enable/disable the redis workers independently
* Remove preview release from PR checks
- remove calls to findUnique, especially when including/selecting relations
- add some missing indexes
- add spans to $transaction calls to help track down long running txs
* Automatically retry TriggerTaskService when hitting a unique constraint error on idempotency key
* improve batch completion system for run engine v1
* Rename batch stuff to v3 so it's not confusing
* Handle unique constraint error on BatchTaskRunItem creation and allow different limits for batchTrigger and batchTriggerAndWait
* Documentation edits made through Mintlify web editor
* Moves files to correct path and improves copy
* Corrects video props to camel case and removes deprecated prop
* Link to the new bulk actions page from the replaying page
---------
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
* Remove unnecessary disabled org check
* Add support for separate redis servers for caching, rate limiter, marqs, and the pub/sub stuff (they all default to the previously used env vars for redis)
* Remove the socket count from shared queue consumer cause it can take up to 5s to complete
* Add support for re-using queue snapshots across dequeues
* Only consider the top N orgs when dequeuing, to help mitigate large spikes in queues (like around the hour and half hour marks)
* Various fixes for run engine v1
- Make sure there are connected providers before sending a scheduled attempt message, nack and retry if there are not
- Fail runs that fail task heartbeats when pending and locked
- More and better logging around shared queue consumer
- Fix bug when failing a task run with no attempt
* Prevent findUnique from bringing down our database
* engine v1 fairness improvements (env based queue selection)
* Add ability to bias env selection by concurrency limit and capacity
* Add some randomization to the list of queues within each evn
* Configure biases using env vars
* Typo
* revert v3 catalog project ref change
* Fix v2 marqs
* reduces the thickness of the focus-visible
* More subtle Input style
* Updates focus-visible styles for the SimpleSelect
* Tables tab through rows and action cells only
* Refined focus visible style
* Adds the custom focus visible to the TextLink
* Moves the onClick handling to the table row rather than the cell
* Makes table row heights consistent
* Adds gap between task search bar and button
* Prevents long tag values from wrapping
* Removes unnecessary rows from table header
* Added gap between search and filters
* Fixes the schedules table pagination staying fixed to the bottom of the page
* Removed more unnecessary header table rows to prevent them being selectable
* Removed isSelected styles (not working)
* Added <tr> back to the main Table compontent
* Table row handles modifier keys
* Adds to={path} to the TableRow only
* Revert "Adds to={path} to the TableRow only"
This reverts commit 8a814d434ff85b8ab9c2305d336074f659d86094.
* Revert "Table row handles modifier keys"
This reverts commit b2394741fa2ed6b908e00673b1a395ae89ca7146.
* Table reverted to use linked cells rather than rows
* Set the tab index of a cell and style the table row when tabbed
* Tabbed row style applied to the sticky cells
* Adds isTabbableCell to each table
* Improves the spcificity of the row highlighting
* Reduces the height of the task rows to match the other tables
* Adds tab styles to fill in row dividers top and bottom
* Removed old row onClick and to props
* Removed duplicate table header row
* Creates table style variants so tables look good in the inspector panels
* Full width tables on the schedules inspector panel
* Reduce padding on Alerts page table rows
* Updates the Deploy page inspector table to the new style
* Removes the duplicate Table Row from the table headers in v2
* Adds TableRow to the EventsTable
* Adds TableRow to more tables
* Adds TableRow to more tables
* Adds TableRow to more tables
* Adds TableRow to more tables
* Fix for showing correct cursor for linked table rows
* Removed variants from some of the table components
* Efficiency improvements to SharedQueueConsumer and MarQS for run engine v1
* Reliabily resolve task queue for a run and ack runs where we can't find the queue
* engine v1: stop nacking after reaching the max nack count, ack to remove the message
* Handle division by 0 possibility in distributeQueues
* Add --experimental-global-webcrypto node option fix "crypto is not defined error" on Node.js 18 in dev
* Only add flag when on node 18
* Missed the function
* Show a good error if TRIGGER_ACCESS_TOKEN isn’t a PAT
* Require maxDuration in trigger.config from now on
* Changeset
* Update the trigger.config templates copy, and set the default to 1hr
* Set the maxDuration in the trigger.config files inside the repo
* Fixed the e2e tests by setting the package to 0.0.0-prerelease-20250116195421
* WIP preventing concurrent waits, throw an error
* Added ConcurrentWaitError (not retryable)
* Move preventMultipleWaits out of the RuntimeAPI
* Added preventMultipleWaits to the devRuntimeManager
* Added throwable InternalError. Plus new TASK_DID_CONCURRENT_WAIT code
* Docs link for troubleshooting concurrent waits
* Docs for troubleshooting concurrent waits
* preventMultipleWaits function
* Added TASK_DID_CONCURRENT_WAIT code
* Deal with InternalErrors that skipRetrying
* Added preventMultipleWaits to prod
* workerDeployment change findUnique to findFirst
* Changed the worker deployment queries to only select what they need, some of the columns are huge
* Only get the bare minimum data for tasks too
* WIP langsmith & AI SDK integration
* Add changeset
* Add exporter support to deployed tasks
* Better support for external exporters and group exporters and instrumentations under the telemetry config property
* Missing changes
* Added shortcuts (with tooltip) to the pagination
* WIP fixing the disabled hover state
* Styled the tooltip
* Added a shortcuts panel to the help menu
* Adds new shortcut to list
* Changes “meta” for “mod”
* Adds more shortcuts to the list
* Adding shortcut to open the shortcuts panel
* tweak gap between shortcut letters
* button component now has icon spacing adjustment (for lucide icons)
* Fixed some ilegal markup
* Pagination uses disabled prop rather than a disabled wrapper
* Improved the Switch styles
* Makes the shortcut modifier optional
* Added new icon based shortcut keys for mac and win
* Updated PC modifier shortcuts
* Adds a new windows key icon
* Allows variants and react nodes to be used as the modifier key
* Adds more shortcuts to the storybook
* Adds missing focus-visible styles to the pagination
* Removed test modifier keys
* number style is tabular
* Update apps/webapp/app/components/primitives/ShortcutKey.tsx
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Tooltip now just 1 prop on the button component
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Increase the number of active streams from 2 to 5
* WIP parent metadata updates
* Fix noop metadata manager
* Implement run metadata updates from ancestor tasks
* Add changeset
* Add ability to stream into parent and root task runs
* Don't duplicate exporting run metadata types
* Add ability to fetch streams through metadata
* Couple of fixes from CodeRabbit
* Fix metadata tests
* Make sure streams are subscribed in the "background"
* Move the stream subscription stuff to the API client, expose it through `runs.fetchStream`
* Fixed run stream tests
* Fix realtime safari bug because of missing ReadableStream async iterable support
Limit to only safari
* Fix missing TIMED_OUT run status
* When coercing realtime date strings, make sure they are set to UTC
* Added new side menu section and updated intro
* Updated the fal ai realtime project
* Copy update
* Added example project
* Swapped walkthrough and repo
* Restore original fal task page
* Added links back to the intro
* Updated login page logos
* Adds an isSelected state to the Table
* Toast style now matches the design
* Adds a space between the upgrade panel and the list of users
* Fix streaming splits in realtime streams v2
* Add changeset
* Skip all flaky tests 😡
* Improve the way we stream from tasks to the server
* Improve the v1 realtime streams (Redis)
* Turn on the relay realtime stream service
* Improved the relay realtime cleanup
* Fixed consuming realtime runs w/streams after the run is already finished
* Remove some logs
* Update changeset
* Fixed runStream tests
* Upgrading from Hobby to Pro displays a dialog to confirm
* Improved the logic for upgrading to pro
* Added the spinner component to storybook
* Improved the dark style spinner
* Realtime streams now powered by electric, and fix the streaming update duplicate issues by converting the electric Shape materialized view into a ReadableStream of changes
* Ensure realtime subscription stops when runs are finished, and add an onComplete handle to use realtime hooks
* Fix tests
* Mutliple streams can be now consumed simultaneously
* Update prerelease script
* Add changeset
* Make it core
* Handle API error responses when streaming
* JWT scopes for tags and batches can now access runs that have the tag or are in the batch
- useTaskTrigger can now submit options
- auto-generated batch trigger public access tokens no longer need each individual run ID scope
* Add changeset
* Added task scopes to work like tags and batches
Also removed scopes for tags when auto-generating a public access token as that could be dangerous.
* WIP adding a side help panel to the tasks page
* Optionally display the shortcut before the trailing icon in the button
* Updated the close icon
* WIP adding a new side help panel
* WIP adding content to the side help panel
* WIP new side help panel content
* Removed images as not needed any more
* Added content to the side help menu
* Help panel open/closed state stored as cookie
* Removed the icons from the docs and examples links
* Update Infisical instructions to new SDK
Current documentation is using old Infisical SDK implementation.
New SDK has a different name, and authenticates on a method, not in the class constructor.
* Fix import and client secret env
* WIP batch trigger v2
* Fix for the DateField being one month out… getUTCMonth() is zero indexed 🤦♂️
* Added a custom date range filter
* Deal with closing the custom date range
* Child runs filter
* Fix for the clear button untoggling the child runs
* WIP batchTriggerV2
* Finished removing rate limit from the webapp
* Added an index TaskRun to make useRealtimeBatch performant
* Renamed the period filter labels to be “Last X mins”
* Denormalize background worker columns into TaskRun
* Use the runTags column on TaskRun
* Add TaskRun ("projectId", "id" DESC) index
* Improved the v2 batch trigger endpoint to process items in parallel and also added a threshold, below which the processing of items is async
* Added a runId filter, and WIP for batchId filter
* WIP triggerAll
* Add new batch methods for triggering multiple different tasks in a single batch
* Disabled switch styling
* Batch filtering, force child runs to show if filtering by batch/run
* Added schedule ID filtering
* Force child runs to show when filtering by scheduleId, for consistency
* realtime: allow setting enabled: false on useApiClient
* Batches page
* Always complete batches, not only batchTriggerAndWait in deployed tasks
* Add batch.retrieve and allow filtering by batch in runs.list
* Renamed pending to “In progress”
* Tidied up the table a bit
* Deal with old batches: “Legacy batch”
* Added the Batch to the run inspector
* Fixed the migration that created the new idempotency key index on BatchTaskRun
* Fixed the name of the idempotencyKeyExpiresAt option and now default idempotency key TTL is 30 days, not 24 hours
* Timezone fix: wrong month in Usage page dropdown
* The DateField now defaults to local time, but can be overriden to use utc with an option
* Don’t allow the task icon to get squished
* BatchFilters removed unused imports
* In the batch filtering, use `id` instead of `batchId` in the URL
* BatchFilters: we don’t need a child tasks hidden input field
* Creates some common filter components/functions
* Fix for batchVersion check when filtering by batch status
* Add additional logging around telemetry and more attributes for trigger spans
* Show clear button for specific id filters
* Batch list: only allow environments that are part of this project
* Unnecessary optional chain
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add JSDocs
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Stop using Response errors when validating API Keys, instead introduce a new "Result" type that has success and failure conditions. Adding in a way to progressively adopt because this touches everything.
* Make sure authenticateApiRequestWithFailure always returns a result
* Fixed task trigger queue update logic to only update an existing queue if the concurrency limit changes, instead of on every single trigger task call
* Fix bad update from another branch
* WIP realtime streams
* Handle realtime with large payloads or outputs #1451
* feat: optimize Redis stream handling with batching
Add STREAM_ORIGIN to environment schema. Improve performance in
RealtimeStreams by using TextDecoderStream for simpler text
decoding and implementing batching of XADD commands for Redis
streams. Limit stream size using MAXLEN option. Update
environment variable repository with new variable type. Adjust
import statements for Redis key and value types.
* 🔧 chore: add dev dependencies for bundle analysis
* add metadata tests and a few more utilties
* Add stream tests and improve streaming
* Added AI tool tasks, descriptions to tasks
* Use the config file path to determine the workingDir, then the package.json path
* Remove stream test files
* useTaskTrigger react hook that allows triggering a task from the client
* Add streaming support for the realtime react hooks
* Add ability to stream results after useTaskTrigger
* Improve the stream throttling
* Use the runId as the ID key to bust the cache after triggering
* Upgrade to to the latest electric sql client and server
* Make realtime server backwards compat with 3.1.2 release
* Pass the runId into useRealtimeRun
* Fix scopes when specifiying reading all runs
* WIP @trigger.dev/rsc package
* Various fixes and accepted recommendations by CodeRabbit
* Regenerate pnpm lock file
* A couple tweaks to rsc and give up on rendering react in tasks for now
* Add changeset
* Remove triggerRequest from the useEffect deps
* Improve realtime & frontend authentication errors
* Fixed authorization tests
* Remove unnecessary log
* Add metadata.stream limits and improve the metadata streams structure
* Streams can now have up to 2500 entries
* Various coderabbit fixes
* additional react-hooks jsdocs
* Updated styling
* Updated the main Table component styles to the new design
* Run page table layout style updates
* Updated row styling and new scrolling behaviour
* Fixed bulk action bar no appearing
* Improved table component styles
* Pagination arrows now chevrons
* Added link to Runs docs page
* Small popover tweak
* Moved the extra Test button out of Tables and into the RunsTable
* WIP refactoring the sticky table cell
* Support for the existing way we have menus in the sticky table cells
* Created a specific Docs style button
* Improved dialog trigger button styles
* Improved docs button styles
* Updated Docs buttons to the new style in the tooltips
* Adding a Test button on hover
* Removed the page padding and made table scroll
* Fixed table scrolling
* cleaned up imports
* WIP adding a test button that links to the test page
* added a docs button to the header
* Much nicer display bar for showing how many schedules you have (and it works with the updated table)
* Fixed the button types in the schedules info panel
* Fixed button icon colour
* Removed the pagination as it’s not being used
* Delete modal button position updated
* Fixed padding on recent payloads panel
* Fixed button icon colours
* Better style for active table row
* Table scrolling now behaves properly
* environment buttons now have a hover state
* Bigger Run test bar to match the Run page
* Small layout fixes to the schedule side panel
* Test page now has link to docs
* fixed text alignment with table headers
* API keys page now has full width tables
* Fix typo and modal layout
* Table uses new sticky button type
* Tidy imports
* Env var table now full width
* Form buttons have a border top
* Updated button styles
* Updated blank state message style to be consistent
* Removed unnecessary form button border
* Alerts page tables now full width
* Allowing the Detail cell to be styled
* Updated the page structure to be the same as the scheduled page – with better progress indicator
* Concurrency limits page now full width
* Fixed function name label extending too far
* Improved the button sizes on the replay modal
* Added a minimal style to the info panels
* Make sure you can always see the info and unlock staging info when the table scrolls
* Info panel style updated to match the others
* Removed the duplicate header
* Usage page tables now full width
* Personal access token page using full width tables
* Small CSS fixes
* Deployments page table now full width
* Deploy page scrolls properly and pagination stays on screen if table is scrollable
* TableCellMenu using the correct popover
* Added missing button variant
* Button group is now named so it doesn’t cause conflicts
* Added docs link to the Run page
* Typo
* Added a link to the test page if you have no runs
* Make the table sticky cell buttons more obvious
* Env var page now using the new popover menu
* Improved the menu items
* Copy tweak
* Made the icons bright on hover
* Updated the cancel run dialog
* Improved the API keys sticky menu behaviour
* Alerts table menu using the new popover
* Correct sized buttons in the Env Var page table menu
* Fixed weird Dialog text styling
* Schedules page blank state using correct docs button style
* Copy tweak
* Menu buttons styled correctly
* Added close buttons to the deploy dialogs
* Made the Dialog footer match the Form footer style
* Fixed feedback form buttons
* Fixed cancel run dialog
* Test page uses the taskIdentifier in the path
* Test task button working
* Added Spline component for 404 logo to remove spline logo
* Removed todo comment
* Added a docs button to the PAT page
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* feat: exit if docker buildx can't be found for self-hosted builds
* chore: tweak debug log message
* chore: improve error message based on rabbit's suggestion
* Fix parsing of project path for cli init command
* Update development instructions for cli-v3
* Create gentle-mails-sip.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* Fix: Typo on overview.mdx
the import helloWorldTask wasn't called correctly at an example
* Fix: Typo update on overview.mdx
Now the import is called `helloWorld` instead of `helloWorldTask` to be consistent with the filename as requested
* Added sync env vars docs example
* Added links in the config + deploy files
* Updated copy
* Added vercel docs cards and added them to all of the relevant docs pages
* Added sync env vars to the intro page
* Added automatically sync env vars section to nextjs guide
* Added ‘Manually’
* Removed code block and updated title
* updated formatting and added vercelsyncenvvars to the config file
* Added missing comment
* Updated imports to /core
* Added sync env var docs link and note
* Improved the VERCEL_ACCESS_TOKEN note
* Added link to vercel
* Updated docs and improved formatting
* Copy tweaks
* Added space
* Updated deploy docs
* magic links on span event errors
* prevent task monitor from processing errors handled elsewhere
* exclusively use internal error code enum for completion data
* add complete attempt service opts
* reattempts need to go via the queue for task controllers that may have exited
* only infer retry config if completed via crash or system failure
* enhance error before deciding if retriable
* retry on SIGTERM
* enable retry config helper for latest sdk
* don't retry heartbeat timeouts for now
* enable task monitor to update fatal errors
* add missing service
* update retry config since package version
* don't alter completion time when updating existing error
* refactor finalize run service
* refactor complete attempt service
* remove separate graceful exit handling
* refactor task status helpers
* clearly separate statuses in prisma schema
* all non-final statuses should be failable
* new import payload error code
* store default retry config if none set on task
* failed run service now respects retries
* fix merged task retry config indexing
* some errors should never be retried
* finalize run service takes care of acks now
* execution payload helper now with single object arg
* internal error code enum export
* unify failed and crashed run retries
* Prevent uncaught socket ack exceptions (#1415)
* catch all the remaining socket acks that could possibly throw
* wrap the remaining handlers in try catch
* New onboarding question (#1404)
* Updated “Twitter” to be “X (Twitter)”
* added Textarea to storybook
* Updated textarea styling to match input field
* WIP adding new text field to org creation page
* Added description to field
* Submit feedback to Plain when an org signs up
* Formatting improvement
* type improvement
* removed userId
* Moved submitting to Plain into its own file
* Change orgName with name
* use sendToPlain function for the help & feedback email form
* use name not orgName
* import cleanup
* Downgrading plan form uses sendToPlain
* Get the userId from requireUser only
* Added whitespace-pre-wrap to the message property on the run page
* use requireUserId
* Removed old Plain submit code
* Added a new Context page for the docs (#1416)
* Added a new context page with task context properties
* Removed code comments
* Added more crosslinks
* Fix updating many environment variables at once (#1413)
* Move code example to the side menu
* New docs example for creating a HN email summary
* doc: add instructions to create new reference project and run it locally (#1417)
* doc: add instructions to create new reference project and run it locally
* doc: Add instruction for running tunnel
* minor language improvement
* Fix several restore and resume bugs (#1418)
* try to correct resume messages with missing checkpoint
* prevent creating checkpoints for outdated task waits
* prevent creating checkpoints for outdated batch waits
* use heartbeats to check for and clean up any leftover containers
* lint
* improve exec logging
* improve resume attempt logs
* fix for resuming parents of canceled child runs
* separate SIGTERM from maybe OOM errors
* pretty errors can have magic dashboard links
* prevent uncancellable checkpoints
* simplify task run error code enum export
* grab the last, not the first child run
* Revert "prevent creating checkpoints for outdated batch waits"
This reverts commit f2b5c2ac42.
* Revert "grab the last, not the first child run"
This reverts commit 89ec5c8bfd.
* Revert "prevent creating checkpoints for outdated task waits"
This reverts commit 11066b4e74.
* more logs for resume message handling
* add magic error link comment
* add changeset
* chore: Update version for release (#1410)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.13
* capture ffmpeg oom errors
* respect maxAttempts=1 when failing before first attempt creation
* request worker exit on fatal errors
* fix error code merge
* add new error code to should retry
* pretty segfault errors
* pretty internal errors for attempt spans
* decrease oom false positives
* fix timeline event color for failed runs
* auto-retry packet import and export
* add sdk version check and complete event while completing attempt
* all internal errors become crashes by default
* use pretty error helpers exclusively
* error to debug log
* zodfetch fixes
* rename import payload to task input error
* fix true non-zero exit error display
* fix retry config parsing
* correctly mark crashes as crashed
* add changeset
* remove non-zero exit comment
* pretend we don't support default default retry configs yet
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: shubham yadav <126192924+yadavshubham01@users.noreply.github.com>
Co-authored-by: Tarun Pratap Singh <101409098+Wackyator@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* try to correct resume messages with missing checkpoint
* prevent creating checkpoints for outdated task waits
* prevent creating checkpoints for outdated batch waits
* use heartbeats to check for and clean up any leftover containers
* lint
* improve exec logging
* improve resume attempt logs
* fix for resuming parents of canceled child runs
* separate SIGTERM from maybe OOM errors
* pretty errors can have magic dashboard links
* prevent uncancellable checkpoints
* simplify task run error code enum export
* grab the last, not the first child run
* Revert "prevent creating checkpoints for outdated batch waits"
This reverts commit f2b5c2ac42.
* Revert "grab the last, not the first child run"
This reverts commit 89ec5c8bfd.
* Revert "prevent creating checkpoints for outdated task waits"
This reverts commit 11066b4e74.
* more logs for resume message handling
* add magic error link comment
* add changeset
* Updated “Twitter” to be “X (Twitter)”
* added Textarea to storybook
* Updated textarea styling to match input field
* WIP adding new text field to org creation page
* Added description to field
* Submit feedback to Plain when an org signs up
* Formatting improvement
* type improvement
* removed userId
* Moved submitting to Plain into its own file
* Change orgName with name
* use sendToPlain function for the help & feedback email form
* use name not orgName
* import cleanup
* Downgrading plan form uses sendToPlain
* Get the userId from requireUser only
* Added whitespace-pre-wrap to the message property on the run page
* use requireUserId
* Removed old Plain submit code
* Added support for the env vars `POD_EPHEMERAL_STORAGE_SIZE_LIMIT` and `POD_EPHEMERAL_STORAGE_SIZE_REQUEST`
* Reverted pnpm lockfile
* Made sure there is no diff between the main lockfile & PR's lockfile
* Not touching the storage for the index containers
* Not touching the storage for the prePull container
* Added a new dropdown help and feedback menu to the side menu
* Added a shortcut to the popover menu
* Removed dev cli connected button for now
* Contact us form uses original Feedback component to prevent broken links
* Improved the messaging when selecting different options in the email form
* buttons style tweak
* SideMenuItem supports the trailingIconClassName
* Adding a consistent focus-visible states
* Removing tooltips for now
* Squashed commit of the following:
commit 7d11123c0a
Author: Eric Goldman <eric@sequin.io>
Date: Mon Sep 30 17:54:06 2024 -0700
Add sequin guide (#1368)
Co-authored-by: James Ritchie <james@trigger.dev>
commit 8da495ac00
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Mon Sep 30 13:42:22 2024 +0100
Improve checkpoint reliability and cleanup of temp files (#1367)
* improve cleanup reliability
* improve logging
* bye-bye execa
* fix for trailing newlines
* prettier errors
* trim args and log output by default
* fix archive cleanup
* prevent potential memleak
* more cleanup debug logs
* ignore abort during cleanup
* rename checkpoint dir env var and move to helper
* add global never throw override
* add tmp cleaner
* also clean up checkpoint dir by default
* split by any whitespace, not just tabs
* only create tmp cleaner if paths to clean
commit 69ec68ee31
Author: Eric Allam <eallam@icloud.com>
Date: Sun Sep 29 19:18:39 2024 -0700
Release 3.0.9
commit a6ea8444c9
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Sun Sep 29 19:17:26 2024 -0700
chore: Update version for release (#1366)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 4c1ee3d6ea
Author: Eric Allam <eallam@icloud.com>
Date: Sun Sep 29 19:09:38 2024 -0700
fix: run metadata not working when using npx/pnpm dlx
* More support for custom-focus
* More custom focus styles added
* Support for focus-visible style for the Segmented control
* Fixed table triple dot menu z-index issue
* Improved help menu wording
* When you submit the help form, close the modal
* focus-visible style for radio buttons
* button prop is now optional in the SideMenu component
* focus styling for a text link
* Deleted unused sequin files
* Checkbox component can have its label styles
* Improved the dialog footer
* Handle sending feedback to Slack using Plain
* WIP making the modal conditional
* Moved the Plain form action into the select plan file
* removed comment
* Show a confirmation diaglog if you’re downgrading from Pro to Hobby
* Downgrading to Hobby works
* Use redirectWithErrorMessage instead of throw error
* The cancel form now submits the data correctly
* Modals don’t trigger when you upgrade
* Copy improvements
* Added a tooltip to explain the link to the pricing page
* Squashed commit of the following:
commit 886429b0ed
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:36:46 2024 +0100
Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config
commit f65157a14f
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:31:27 2024 +0100
Lockfile with run-engine removed
commit 3d67bb81e9
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:24:31 2024 +0100
Removed run-engine from the webapp package.json/tsconfig
commit d30e9713f4
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:06:04 2024 +0100
Dockerfile fix because the database package has been moved
commit f2babbf637
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 09:41:22 2024 -0700
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
commit fc609470c4
Author: Dan <8297864+D-K-P@users.noreply.github.com>
Date: Tue Oct 8 14:36:03 2024 +0100
Supabase database webhook example upgrade (#1386)
* Added overview for guides and examples section and split them all out
* New supabase guide wip
* Updated images and improved docs
* Trimmed the supabase prereqs
* Supabase guide wip
* more updates
* Replaced old database webhook guide
* Created one intro page and removed snippets
* Updated guide sidebar titles
* Code updates
* More improvements
* Updates and added images
* Compressed image
* Updated guides descriptions and edge function basic
* Removed bold
* Updated redirects
* Fixed broken links
* Updated intro
commit 07f82eae54
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 13:28:54 2024 +0100
Release 3.0.11
commit 13ebfcc5b9
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Tue Oct 8 13:24:38 2024 +0100
chore: Update version for release (#1381)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 2a04d17e1f
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:24:23 2024 +0100
Simplify showLogs expression
commit 002ae4bc0d
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:29 2024 +0100
Fix dotenv overrides for dev runs (#1388)
* override dashboard dev env vars with local .env
* add changeset
* add simple task for testing env vars
commit 047cb00417
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:05 2024 +0100
Disable schedules for deleted orgs on next tick (#1383)
* disable schedules for deleted orgs
* add debug logs
commit 2c014f725a
Author: James Ritchie <james@trigger.dev>
Date: Sun Oct 6 13:02:00 2024 -0700
Override log retention (#1385)
* set full log retention as admin
* If run.logsDeletedAt is set, don’t bother getting the trace
commit a69e04f03a
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 14:18:58 2024 +0100
Include push output in logs for self-hosted deploys (#1382)
* include push output in logs
* changeset
commit c5488df8ca
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 13:12:47 2024 +0100
Fix CLI downgrade check (#1380)
* fix downgrade detection
* remove unused semver package from webapp
* add changeset
commit 1caec27f72
Author: Eric Allam <eric@trigger.dev>
Date: Fri Oct 4 15:33:35 2024 -0700
docs: Max duration (#1379)
* maxDuration docs
* Update the init command to set the maxDuration and include a commented out maxDuration in the config file
commit e14c954e12
Author: Eric Allam <eallam@icloud.com>
Date: Fri Oct 4 15:02:05 2024 -0700
Release 3.0.10
commit 8e61f5d631
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Fri Oct 4 14:59:07 2024 -0700
chore: Update version for release (#1378)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 08db565a72
Author: Eric Allam <eallam@icloud.com>
Date: Thu Oct 3 12:38:25 2024 -0700
improve the timed out description
commit 6d0884254e
Author: Eric Allam <eric@trigger.dev>
Date: Thu Oct 3 12:43:26 2024 -0700
feat: Add maxDuration to tasks (#1377)
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
commit 665ccf88b7
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Thu Oct 3 12:33:18 2024 +0100
Update github actions and self-hosting docs
commit 1ff7b861ca
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 18:26:36 2024 -0700
Add max queue depth limits (#1376)
* Add runs to an env queue, as well as the actual queue
* Add queue size limit guard on triggering tasks
commit c531a9d986
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 15:30:39 2024 -0700
fix: cleanup ttl expire run graphile jobs (#1373)
* fix: remove ttl expire run graphile jobs when a run is started or completed
* Update expireEnqueuedRun.server.ts
commit 0bf500fa95
Author: Matt Aitken <matt@mattaitken.com>
Date: Wed Oct 2 15:30:16 2024 -0700
Prioritize finishing waited runs (#1375)
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* Squashed commit of the following:
commit 886429b0ed
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:36:46 2024 +0100
Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config
commit f65157a14f
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:31:27 2024 +0100
Lockfile with run-engine removed
commit 3d67bb81e9
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:24:31 2024 +0100
Removed run-engine from the webapp package.json/tsconfig
commit d30e9713f4
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:06:04 2024 +0100
Dockerfile fix because the database package has been moved
commit f2babbf637
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 09:41:22 2024 -0700
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
commit fc609470c4
Author: Dan <8297864+D-K-P@users.noreply.github.com>
Date: Tue Oct 8 14:36:03 2024 +0100
Supabase database webhook example upgrade (#1386)
* Added overview for guides and examples section and split them all out
* New supabase guide wip
* Updated images and improved docs
* Trimmed the supabase prereqs
* Supabase guide wip
* more updates
* Replaced old database webhook guide
* Created one intro page and removed snippets
* Updated guide sidebar titles
* Code updates
* More improvements
* Updates and added images
* Compressed image
* Updated guides descriptions and edge function basic
* Removed bold
* Updated redirects
* Fixed broken links
* Updated intro
commit 07f82eae54
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 13:28:54 2024 +0100
Release 3.0.11
commit 13ebfcc5b9
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Tue Oct 8 13:24:38 2024 +0100
chore: Update version for release (#1381)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 2a04d17e1f
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:24:23 2024 +0100
Simplify showLogs expression
commit 002ae4bc0d
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:29 2024 +0100
Fix dotenv overrides for dev runs (#1388)
* override dashboard dev env vars with local .env
* add changeset
* add simple task for testing env vars
commit 047cb00417
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:05 2024 +0100
Disable schedules for deleted orgs on next tick (#1383)
* disable schedules for deleted orgs
* add debug logs
commit 2c014f725a
Author: James Ritchie <james@trigger.dev>
Date: Sun Oct 6 13:02:00 2024 -0700
Override log retention (#1385)
* set full log retention as admin
* If run.logsDeletedAt is set, don’t bother getting the trace
commit a69e04f03a
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 14:18:58 2024 +0100
Include push output in logs for self-hosted deploys (#1382)
* include push output in logs
* changeset
commit c5488df8ca
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 13:12:47 2024 +0100
Fix CLI downgrade check (#1380)
* fix downgrade detection
* remove unused semver package from webapp
* add changeset
commit 1caec27f72
Author: Eric Allam <eric@trigger.dev>
Date: Fri Oct 4 15:33:35 2024 -0700
docs: Max duration (#1379)
* maxDuration docs
* Update the init command to set the maxDuration and include a commented out maxDuration in the config file
commit e14c954e12
Author: Eric Allam <eallam@icloud.com>
Date: Fri Oct 4 15:02:05 2024 -0700
Release 3.0.10
commit 8e61f5d631
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Fri Oct 4 14:59:07 2024 -0700
chore: Update version for release (#1378)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 08db565a72
Author: Eric Allam <eallam@icloud.com>
Date: Thu Oct 3 12:38:25 2024 -0700
improve the timed out description
commit 6d0884254e
Author: Eric Allam <eric@trigger.dev>
Date: Thu Oct 3 12:43:26 2024 -0700
feat: Add maxDuration to tasks (#1377)
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
commit 665ccf88b7
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Thu Oct 3 12:33:18 2024 +0100
Update github actions and self-hosting docs
commit 1ff7b861ca
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 18:26:36 2024 -0700
Add max queue depth limits (#1376)
* Add runs to an env queue, as well as the actual queue
* Add queue size limit guard on triggering tasks
commit c531a9d986
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 15:30:39 2024 -0700
fix: cleanup ttl expire run graphile jobs (#1373)
* fix: remove ttl expire run graphile jobs when a run is started or completed
* Update expireEnqueuedRun.server.ts
commit 0bf500fa95
Author: Matt Aitken <matt@mattaitken.com>
Date: Wed Oct 2 15:30:16 2024 -0700
Prioritize finishing waited runs (#1375)
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* Revert "Squashed commit of the following:"
This reverts commit b837b5a558dc5187d89b09135276d16f6856f455.
* Removed console logs
* cleaned up conditionals
* Unlock free plan state
* Fixed subscribe button if you’re already github verified
* Simplified the downgrade reasons logic
* made periodEnd required
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Updated intro and moved stripe webhooks into the webhooks section
* Added additional resources links to nextjs and remix
* Updated intro page
* Changed formatting to improve layout
* Removed Supabase card from frameworks
* docker provider shouldn't force simulate by default
* structured logs for all things coordinator
* attributes for handler loggers
* add changeset
* fix tmp cleaner types
* remove unused simple logger import
* fields from args should override existing fields
* improve socket logs
* change log message position for easier scanning
* more logging improvements, more metadata
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
* Added overview for guides and examples section and split them all out
* New supabase guide wip
* Updated images and improved docs
* Trimmed the supabase prereqs
* Supabase guide wip
* more updates
* Replaced old database webhook guide
* Created one intro page and removed snippets
* Updated guide sidebar titles
* Code updates
* More improvements
* Updates and added images
* Compressed image
* Updated guides descriptions and edge function basic
* Removed bold
* Updated redirects
* Fixed broken links
* Updated intro
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* improve cleanup reliability
* improve logging
* bye-bye execa
* fix for trailing newlines
* prettier errors
* trim args and log output by default
* fix archive cleanup
* prevent potential memleak
* more cleanup debug logs
* ignore abort during cleanup
* rename checkpoint dir env var and move to helper
* add global never throw override
* add tmp cleaner
* also clean up checkpoint dir by default
* split by any whitespace, not just tabs
* only create tmp cleaner if paths to clean
* Removed the inline-code accessory from the logs when calling trigger or batchTrigger from a run
* Removed re2 from the v3 catalog
* Added a Root badge to the runs list
* Keep the side panel open when switching tasks & remove links from detail panel
* Root and parent task styling inspector
* Hide the root badge if the task isn’t the root
* margin between the dev not running message
* improved spacing of items
* Improved Root badge style
* Show a table of triggered runs in the inspector
* Add parentSpanId index to the TaskRun table
* Fix for the run inspector now opening when linked from another run/replay
* Triggered runs table has a max height
* Added a description to the replay run modal and improved the styling slightly
* Only include a bottom border when the triggered run table is more than 4 items
* Improved the triggered runs table borders
* Improved the tables so they can have an optional sticky header
* Added table types to storybook
* Fix for hover states on different backgrounds & runs table select cell
* Added supabase database examples
* Copy updates
* More copy updates
* Added comment about generating the typescript types
* Updated intro table
* Improved title
* Added supabase-docs-cards to go at the bottom of the examples and guides
* merge main
* Link fix
* Fix more links
* Run metadata
* Remove metadata from context, move it to it’s own tab
* More run metadata stuff
- Add metadata to testing
- Make using metadata outside of runs a no-op
- Add docs
* Replaying should copy over the metadata
* transfer final attempt output to the task run
* A couple of minor fixes
* Use the new clientOrThrow() method everywhere
* Cleaned up the update metadata endpoint and added an API doc page for it
* Mirror task run attempt errors and output
* Add safe rootTaskRunId index and a README to @trigger.dev/database
* Associate child runs with the span ID of the span in the parent run that triggered the child run
* Update deprecation notice doc links
* Added supabase storage basic example and updated ffmpeg instructions
* Added stripe webhook example
* Copy tweaks
* Typos
* Updated stripe example to check for a specific event before triggering the task
* Simplified the storage example
* Added examples to intro page
* Added further instructions
* Code improvements
* Final tweaks
* Fix resolving external packages that are ESM only by falling back to mlly resolvePathSync. This will fix mupdf
* when publishing a prerelease and aborting, clear the git stage
* Add task run hierarchical relationships to the database
* Add depth and related runs to the retrieve run API response
* Remove prisma optimize
* restructure the migrations to create the index concurrently
* Delete these tsbuildinfo files
* Fix type error by adding depth to the run list presenter
* Cleanup the task hierarchy, share more code
* Remove some fields from the list run response
* never abort the same controller twice
* prevent uncaught exception when aborting pipe
* abort signal assertions and more logging
* never abort running pipe
* display --profile flag after init if used
* add containerfile debug logs
* log all indexing errors in case of connection issues
* specify dockerfile syntax version
* add network flag for self-hosting
* move all self-hosting tags to latest
* add self-hosting update section
* manual setup subsection
* registry flag implies push
* add changeset
* Added introduction page for guides
* New intro page for examples
* Fixed links
* Updated examples intro to include all of the new ones
* Improved FFmpeg example
* Improved the react pdf example
* Added Supabase overview page
* Updated card-supabase snippet
* Added sharp payload instructions
* Added vercel payload instructions
* Added dall-e payload instructions
* Added openai payload instructions
* Added resend payload instructions
* Made the prompts more consistent
* Minor tweaks and moved bun
* Updated links and added bun logo
---------
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
* Fixes for CLI update command, and make the hide the "whoami" command output when running in dev
* Fix an issue where a missing tsconfig.json file would throw an error on dev/deploy
* Don’t show latest CLI warning when using a prerelease
* Only print CLI update required message when update is embedded
* Strip out TRIGGER\_ keys when using syncEnvVars, to prevent deploy errors
* Fix for run filtering not working with some special characters (double encoded)
* Add the full dependentTaskAttempt to a ResumeBatchRunService log
* Added RESUME_AFTER_DEPENDENCY_WITH_ACK
* Set the delay to 5s
* If a checkpoint has been created, the coordinator won’t continue the run with RESUME_AFTER_DEPENDENCY_WITH_ACK
* If we’re keeping the run alive then set socket.data.requiresCheckpointResumeWithMessage to undefined
* Log out the data before and after setting socket.data.requiresCheckpointResumeWithMessage
* Attempt to fix false package mismatch warnings
* Add changeset
* Add ability to test update checks in prerelease packages
* Resolve the trigger.dev package based on the package.json dir
* Try this
* Don’t use the version module, just resolve the packageJson
* One more dirname
* Comment
* Remove the version export because we aren’t using it anymore
* Add triggerAndWait().unwrap() to more easily get at the output or throw the subtask error
* Add taskId and runId to SubtaskUnwrapError
* WIP docs update beta -> latest
* trigger.dev init now adds @trigger.dev/build to devDependencies
* How it works doc
* Restructure some docs and update the cli commands
* Config file docs, plus aptGet and ffmpeg extensions
* Update to latest from beta docs
* Add runtime to templates
* Add --runtime option to the init CLI command
* A bunch more doc updates after feedback
* Document triggerAndWait with unwrap and result types
* beta -> latest in the webapp
* CLI update check no longer references beta
* Add major release
* Leave changeset beta, back to normal package release
* Fixed default dirs option in init command
* exclude windows-yarn variation of cli e2e tests because it’s buggy
* Remove cache to try and fix yarn e2e test workflow errors
* Better code snippet to clarify your framework
* New page for react-pdf
* Added 3 more examples
* New react to PDF example
* Alphabeticalise the side menu
* Removed github link and tweaked title
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit e63ee9e924.
* save cookie only when id is used
* Deployment table now scrolls
* removed imports
* A lot of changes to make the inspector live too… WIP
* More major overhauls to get the synced version of the run page working…
* If a span is completed show that
* Set the debounce much lower for selecting the span view
* Load the details run inspector data on demand
* Delete the SpanPresenter
* Use the async payload because it deals with superjson
* Fixed weird merge conflict
* Share some inspector timeline components
* A couple of layout tweaks
* Improved the run inspector loading states
* Fix for paragaph errors
* Fix for focusing on a span
* Undefined typre for useSyncedShape
* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples
* Updated @electric-sql/react package to the latest
* Fix the timeline duration stretching
* Added some better error handling for the electric sync
* More logging
* Better error when there are bad responses
* Turn off resizable snapshots, there’s a bug
* Added getSpan back
* Added SpanPresenter back
* Updated to the new Electric hooks package
* Made a copy so we have the old run page and the new electric one
* Put the main eventRepository back for now
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Ignore /packages/cli-v3/src/package.json
* Added more logs when resuming a dependency, added the runId
* A task for reproducing a race condition with checkpoints
* Fix for doing remote image build when not self-hosting
* Set team members, alerts and schedule limits to 100m for self-hosting
* Import fix
* Set the checkpointEventId in marqs when the checkpoint is created for batchTriggerAndWait
This should fix a horrible race condition when at max concurrency
* WIP creating a CLI section
* WIP adding more CLI commands in a new section
* New CLI section with introduction and more CLI commands
* More improvement to the CLI options
* Each CLI command has its own page
* fixed broken links
* Removed outdated copy
* Fixed empty env vars overriding in dev runs
* Don’t import package.json anymore
* fix node10 moduleResolution in @trigger.dev/core
* Support self-hosters pushing to a custom registry when running deploy
* dev: Fixed stuck runs when a child run fails with a process exit
* Make some doc notes about known issues and docker hub private repos
* Fix --project-ref when running deploy
* Fix —config option when deploying
* Fixing the flushing/killing process with the new build system
* Add monorepo-react-email e2e test fixture
* Fix issue with emitDecoratorMetadata and tsconfigs with extends
* Got the emit decorator metadata fixture working
* Fixed typechecking yarn e2e CLI tests in monorepos
* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)
* Remote externals now powered by JSON Hero to be easier to update
* resolve config source files
* Add a —javascript option to init, defaults to typescript
* Add support for prisma typed sql
* Remove msw and retry.interceptFetch
* Add missing code to the openai retries example
* Don’t generate the v3 catalog prisma client during CI
* Fixed v3-catalog task imports
* Remove interceptor usage in task file
* Only import import-in-the-middle hook if there are instrumented packages
* Fix yarn.lock file
* When resuming a batch, only do marqs operations once
* Made TaskRunDependency clearer in the Prisma schema
* New ResumeDependentParentsService service, use it from checkpoints
* WIP on making resuming more robust
* Turn the declarative schedules off because they make debugging other runs painful
* Resuming batches when there’s an attempt is working
* If there’s no attempt then create one
* Added a log if there are no span events to complete
* If Graphile addJob doesn’t return a row, log and return undefined. No throw
* Pass prisma into the ResumeDependentParentsService
* Removed the todos
* Pass Prisma through to the checkpoint service
* Fix for not checking the batch item correctly
* Fix for when a log flush times out and the process is checkpointed
* Fix for when a log flush times out and the process is checkpointed
* Another test run that does batches with failed subtasks
* Don’t call ResumeTaskRunDependenciesService anymore (we have a new service)
* Only resume if the run is in a final state
* If an attempt doesn’t exist, fix for creating queue with sanitized name
* If DEV then don’t resume using marqs/batches. The CLI manages it
* We don’t need to check the run status again, it’s in the main function now
* Added TaskRunAttempt taskRunId index
* Only allow calling ResumeDependentParentsService with a run ID
* Put the flushing back to what it was
* Separated the triggering docs page into 2 sections
* Added link to GitHub Actions from deploy page
* Improved the batchTriggerAndWait docs for handling errors
* Added docs for using google credential files
* Improved the triggering page copy some more
* Improved the google credentials copy
* Support custom config file names & paths
* Fix entry point paths on windows
* Support custom conditions
Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.
- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds
* Add changeset
* Fix passing CLI process.env down to the task processes
* Remove unused import
* reviving the e2e CLI tests
* Another attempt at fixing windows
* yet another windows attempt (yawa)
* Output index child stdout and stderr (yawa)
* normalize import paths for windows
* Added some logging (yawa)
* normalize the loader path as well
* Added some logging to figure out why the entry points aren’t being found on windows
* Fix for entry point detection on windows
* Normalize runner import loader path
* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well
* Various cleanup after windows fixes
* Remove the webapp e2e for now
* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)
https://github.com/andrewbranch/example-subpath-exports-ts-compat
* init using templates again but downloaded from the repo this time
* Adding init schedule example
* Support for js init
* init now working with js
* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit e63ee9e924.
* save cookie only when id is used
* Deployment table now scrolls
* removed imports
* A lot of changes to make the inspector live too… WIP
* More major overhauls to get the synced version of the run page working…
* If a span is completed show that
* Set the debounce much lower for selecting the span view
* Load the details run inspector data on demand
* Delete the SpanPresenter
* Use the async payload because it deals with superjson
* Fixed weird merge conflict
* Share some inspector timeline components
* A couple of layout tweaks
* Improved the run inspector loading states
* Fix for paragaph errors
* Fix for focusing on a span
* Undefined typre for useSyncedShape
* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples
* Updated @electric-sql/react package to the latest
* Fix the timeline duration stretching
* Added some better error handling for the electric sync
* More logging
* Better error when there are bad responses
* Turn off resizable snapshots, there’s a bug
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add an error to the final attempt if there isn’t one
* Improved the checkpointResumer test task
* Using triggerAndWait or batchTriggerAndWait frees up concurrency
Normally it frees up just env and org concurrency. If it’s a recursive task then it will free up the run concurrency too (e.g. a task calling itself).
* Removed the filepath and export name from attempt spans
* Run inspector: only show an error if the run is in a finished state
* Add support + docs for custom esbuild plugins
* Fixed the missing subpath export
* Add support for specifying the prisma client generator
* Add `—env-file` support to dev and deploy
* RESUME messages were missing some data that is needed for the concurrency tracker
* Cancel heartbeats when checkpoints are created for triggerAndWait and batchTriggerAndWait
* Added some logs when canceling heartbeats
* Improved the nested dependencies test task
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Remove run count from JobPresenter
* Remove job/integration counts from org/projects query
* Removed v2 counts from ProjectPresenter query
* JobRun index idx_jobrun_organizationId_createdAt
* JobRun index idx_jobrun_versionId
* BatchTaskRunItem index taskRunAttemptId
* BatchTaskRunItem index taskRunId
* Add new ProjectAlertType ‘TASK_RUN’ and then migrate channels to it
* WIP new alerts for run failures
* Consolidate failed run status into taskStatus.ts
* Added the read replica into BaseService, could be useful
* Send task run alerts code
* Revert "Added the read replica into BaseService, could be useful"
This reverts commit cc2348a40254ea3f9e11c0feaf303ecd8794433a.
* Allow adding task run alerts in the UI
* Use task run alert not attempt alert…
* Use the primary
* tasks no longer inside a group in the side menu (and added “cron”)
* Delay using a timezone
* Added React Not Defined error to the troubleshooting page
* Improved the React common problem
* Link to v2 docs
* New Development section and entry in Common Problems
* Concurrently running the terminal
* Fixed the .env weirdness
* Added section on creating PATs for Github actions
* Improved the Machine spec and limits page
* Quick start steps now have nice images
* Added a diagram for the lifecycle functions
* Added note about onFailure
* CRON -> cron/Cron
* WIP adding more steps to the next.js guide
* WIP next.js
* WIP adding tabbed steps for pages/app router
* References to Infisical links to their homepage so it’s clearer
* WIP updating the nextjs guide
* WIP nextjs guide
* More nextjs guide steps
* More copy
* Added rate limit trouble shooting
* Removed old prisma error title
* Added secret key step
* Added a note for logging in using a specified domain if self hosting
* typo
* App router docs copy
* Deploy copy update
* Added a favicon.png to fix a docs build error
* Removed unused snippet
* Server actions now inside a tab
* Server actions + restructured the triggering section
* Added troubleshooting snippet for react event handlers
* Added a new troubleshooting snippet for ESM
* Updated old replaying image to reflect the new UI
* Removed references to reattempting
* Updated replaying from the run page
* Added a bulk replay section
* Updated the writing tasks intro page
* Removed edge runtime code for now
* Added edge runtime – it seems to just work!
* import type
* Env vars: don’t show deleted dev environments
* Concurrency limits: don’t show deleted dev environments
* Create/edit schedule: don’t show deleted dev environments
* Test for checkpoints
* Make sourceTaskAttemptId optional on resumeBatchRun
* Removed all completions/executions logic from the shared queue consumer
* Removed the sourceTaskAttemptId from ResumeBatchRunService
* Revert "Removed all completions/executions logic from the shared queue consumer"
This reverts commit d35398d50463c81a1975bb5d5bcfca66a24b8ede.
* WIP on triggerAndWait…
* Fixed triggerAndWait continuing when a checkpoint completes
* Remove the ResumeAttempt code that fails attempts (was protecting against infinite restores)
* Removed messageBody.data.completedAttemptIds.length === 0 commented out code
* Don’t ack if there’s no batchRun
* Added the marqs?.replaceMessage back in but NOT when there’s no checkpoint. More logging
This is a fix for when some attempts fail
* Improvement to the test task that now randomly fails attempts
* When a checkpoint happens, only continue the attempt if it’s in the correct state
* Changeset for rollback in branch
* Set keepRunAlive to false when the dependent task isn’t finished
* Changeset manual version (to get inline with the hotfix branch)
* Changeset: Fixes for continuing after waits
* Latest lockfile (after manual changeset version)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* WIP notes on each location where we’ll use finalize
* Initial FinalizeTaskRunService
* ExpireEnqueuedRunService uses FinalizeTaskRunService
* FailedTaskRunService uses FinalizeTaskRunService
* Allow passing in an include when finalizing the run
* CrashTaskRunService using FinalizeTaskRunService
* Remove comments
* Status is optional
* CancelAttemptService using FinalizeTaskRunService
* Import tidy
* CancelTaskRunService using FinalizeTaskRunService
* Import tidying
* CompleteAttemptService system failure switched to FinalizeTaskRunService
* Added more logging to Finalizing
* CompleteAttemptStatus COMPLETED_SUCCESSFULLY
* CompletedAttempt “SYSTEM_FAILURE”
* CompletedService final pair
* Use satisfies so we can derive types from the groups
* Only allow final states to be used with this service
* BaseService tx support, minor improvements
* tasks no longer inside a group in the side menu (and added “cron”)
* Delay using a timezone
* Added React Not Defined error to the troubleshooting page
* Improved the React common problem
* Link to v2 docs
* New Development section and entry in Common Problems
* Concurrently running the terminal
* Fixed the .env weirdness
* Added section on creating PATs for Github actions
* Improved the Machine spec and limits page
* Quick start steps now have nice images
* Added a diagram for the lifecycle functions
* Added note about onFailure
* CRON -> cron/Cron
* Updated the cli-dev steps for the concurrently package
* Fixed capital letter
* Updated diagram text
* Removed dead page
* Make the tooltip text color grey so it’s readable again the primary color
* Setup and styling for the Guides section
* Better Guides icon
* Improved the prerequisites
* WIP Nextjs guide and new folder structures
* Copy updates
* Useful next steps is now a snippet
* Better icons for the next steps section
* Renamed the “prerequisites” snippet
* Node.js guide
* Added a “Creating a project” guide
* New snippet for prerequisites
* New Remix guide
* Added Remix to the side bar
* Tweaked icons for creating a project page
* Added the hello world step to the onboarding steps
* Better “useful next steps” snippet card links
* Moved prerequisites
* Fixing links to images
* Moved v2 migration page to guides
* Removed dead link
* Attempt fix for redirect
* Fixed redirects
* Getting started section includes link to roadmap
* Removed icon from side menu
* Deleted all v2 pages (excluding updating mint.json)
* Removed v2 pages, redirects and versions from mint.json
* Deleted v2 snippets
* Deleted un-used pages
* Set of more useful coming soon snippets
* All snippets use the updated format
* updated folder “v3/“ with “/pages”
* Moved all main docs files to the route and updated the redirect
* Fixed URLs in the mdx pages to the new route path
* URL goes to the proper pricing page
* Better delayed runs image
* Attempt fix for self hosting page not redirecting
* Updated the v2 project message to include end of life
* Removed the v3 badge. Showing a v2 badge instead
* Removed the v3 badge variant
* Show a badge in the side menu if you’re paying
* Popover menu section header and button state colours now reflects the design UI
* Added a link to the blog in the end-of-life message
* Removed missing variant from badge
* Remove the old log truncation
* Added TaskRUn logsDeletedAt column
* Accurate timestamps for the run inspector
* EnsureProperty type when you want to make a single property not nullable
* No logs and upgrade messages working
* Button can be autofocused
* Replay dialog code editor is autofocused
* Fix for wrapping of span duration
* Added TaskRun completedAt column
* When finalising a run set the completedAt date
* A note to discuss whether we need to set the completedAt to null
* Remove the note in the sharedQueueConsumer
2024-08-02 10:31:38 +01:00
5166 changed files with 405547 additions and 471315 deletions
Tasks should now be much more robust and resilient to reconnects during crucial operations and other failure scenarios.
Task runs now have to signal checkpointable state prior to ALL checkpoints. This ensures flushing always happens.
All important socket.io RPCs will now be retried with backoff. Actions relying on checkpoints will be replayed if we haven't been checkpointed and restored as expected, e.g. after reconnect.
Other changes:
- Fix retry check in shared queue
- Fix env var sync spinner
- Heartbeat between retries
- Fix retry prep
- Fix prod worker no tasks detection
- Fail runs above `MAX_TASK_RUN_ATTEMPTS`
- Additional debug logs in all places
- Prevent crashes due to failed socket schema parsing
Adds support for `emitDecoratorMetadata: true` and `experimentalDecorators: true` in your tsconfig using the [`@anatine/esbuild-decorators`](https://github.com/anatine/esbuildnx/tree/main/packages/esbuild-decorators) package. This allows you to use libraries like TypeORM:
```ts orm/index.ts
import "reflect-metadata";
import { DataSource } from "typeorm";
import { Entity, Column, PrimaryColumn } from "typeorm";
@Entity()
export class Photo {
@PrimaryColumn()
id!: number;
@Column()
name!: string;
@Column()
description!: string;
@Column()
filename!: string;
@Column()
views!: number;
@Column()
isPublished!: boolean;
}
export const AppDataSource = new DataSource({
type: "postgres",
host: "localhost",
port: 5432,
username: "postgres",
password: "postgres",
database: "v3-catalog",
entities: [Photo],
synchronize: true,
logging: false,
});
```
And then in your trigger.config.ts file you can initialize the datasource using the new `init` option:
```ts trigger.config.ts
import type { TriggerConfig } from "@trigger.dev/sdk/v3";
Added config option extraCACerts to ProjectConfig type. This copies the ca file along with additionalFiles and sets NODE_EXTRA_CA_CERTS environment variable in built image as well as running the task.
Updates the `trigger`, `batchTrigger` and their `*AndWait` variants to use the first parameter for the payload/items, and the second parameter for options.
We've also changed the API of the `triggerAndWait` result. Before, if the subtask that was triggered finished with an error, we would automatically "rethrow" the error in the parent task.
Now instead we're returning a `TaskRunResult` object that allows you to discriminate between successful and failed runs in the subtask:
When using idempotency keys, triggerAndWait and batchTriggerAndWait will still work even if the existing runs have already been completed (or even partially completed, in the case of batchTriggerAndWait)
- TaskRunExecutionResult.id is now the run friendlyId, not the attempt friendlyId
- A single TaskRun can now have many batchItems, in the case of batchTriggerAndWait while using idempotency keys
- A run’s idempotencyKey is now added to the ctx as well as the TaskEvent and displayed in the span view
- When resolving batchTriggerAndWait, the runtimes no longer reject promises, leading to an error in the parent task
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Claude