Commit Graph

226 Commits

Author SHA1 Message Date
github-actions[bot] b6b74046b9 chore: Update version for release (#1310)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-17 10:44:51 +01:00
Eric Allam 2b5771f38c Remove duplicate bin definition, fixes issue #1311 2024-09-16 21:40:01 +01:00
Eric Allam de135e4885 Fix for TASK_RUN_HEARTBEAT errors in deployed and dev works 2024-09-16 19:10:19 +01:00
github-actions[bot] 499fd86cf2 chore: Update version for release (#1309)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-16 18:11:20 +01:00
Eric Allam 3aa5811790 fix 3.0.0 update warning (#1308)
* Attempt to fix false package mismatch warnings

* Add changeset

* Add ability to test update checks in prerelease packages

* Resolve the trigger.dev package based on the package.json dir

* Try this

* Don’t use the version module, just resolve the packageJson

* One more dirname

* Comment

* Remove the version export because we aren’t using it anymore
2024-09-16 18:03:16 +01:00
github-actions[bot] 45287cd655 chore: Update version for release (#1269)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-16 15:37:20 +01:00
Eric Allam cf13fbdf32 Updating docs for the move from beta -> latest (with new build system) (#1299)
* Add triggerAndWait().unwrap() to more easily get at the output or throw the subtask error

* Add taskId and runId to SubtaskUnwrapError

* WIP docs update beta -> latest

* trigger.dev init now adds @trigger.dev/build to devDependencies

* How it works doc

* Restructure some docs and update the cli commands

* Config file docs, plus aptGet and ffmpeg extensions

* Update to latest from beta docs

* Add runtime to templates

* Add --runtime option to the init CLI command

* A bunch more doc updates after feedback

* Document triggerAndWait with unwrap and result types

* beta -> latest in the webapp

* CLI update check no longer references beta

* Add major release

* Leave changeset beta, back to normal package release

* Fixed default dirs option in init command

* exclude windows-yarn variation of cli e2e tests because it’s buggy

* Remove cache to try and fix yarn e2e test workflow errors
2024-09-16 15:30:32 +01:00
Eric Allam ca2cb7298c Another attempt at fixing the yarn CI failure 2024-09-16 15:13:19 +01:00
Eric Allam a50063ce04 Always insert the dirs option when initializing a new project in the trigger.config.ts 2024-09-12 14:12:16 +01:00
Eric Allam 8578c9b281 v3: new build system fixes round 2 (#1283)
* Fixed empty env vars overriding in dev runs

* Don’t import package.json anymore

* fix node10 moduleResolution in @trigger.dev/core

* Support self-hosters pushing to a custom registry when running deploy

* dev: Fixed stuck runs when a child run fails with a process exit

* Make some doc notes about known issues and docker hub private repos

* Fix --project-ref when running deploy

* Fix —config option when deploying

* Fixing the flushing/killing process with the new build system

* Add monorepo-react-email e2e test fixture

* Fix issue with emitDecoratorMetadata and tsconfigs with extends

* Got the emit decorator metadata fixture working

* Fixed typechecking yarn e2e CLI tests in monorepos

* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)

* Remote externals now powered by JSON Hero to be easier to update

* resolve config source files

* Add a —javascript option to init, defaults to typescript

* Add support for prisma typed sql

* Remove msw and retry.interceptFetch

* Add missing code to the openai retries example

* Don’t generate the v3 catalog prisma client during CI

* Fixed v3-catalog task imports

* Remove interceptor usage in task file

* Only import import-in-the-middle hook if there are instrumented packages

* Fix yarn.lock file
2024-09-11 15:48:20 +01:00
Matt Aitken 392453e8a6 Frozen run fixes (#1286)
* When resuming a batch, only do marqs operations once

* Made TaskRunDependency clearer in the Prisma schema

* New ResumeDependentParentsService service, use it from checkpoints

* WIP on making resuming more robust

* Turn the declarative schedules off because they make debugging other runs painful

* Resuming batches when there’s an attempt is working

* If there’s no attempt then create one

* Added a log if there are no span events to complete

* If Graphile addJob doesn’t return a row, log and return undefined. No throw

* Pass prisma into the ResumeDependentParentsService

* Removed the todos

* Pass Prisma through to the checkpoint service

* Fix for not checking the batch item correctly

* Fix for when a log flush times out and the process is checkpointed

* Fix for when a log flush times out and the process is checkpointed

* Another test run that does batches with failed subtasks

* Don’t call ResumeTaskRunDependenciesService anymore (we have a new service)

* Only resume if the run is in a final state

* If an attempt doesn’t exist, fix for creating queue with sanitized name

* If DEV then don’t resume using marqs/batches. The CLI manages it

* We don’t need to check the run status again, it’s in the main function now

* Added TaskRunAttempt taskRunId index

* Only allow calling ResumeDependentParentsService with a run ID

* Put the flushing back to what it was
2024-09-09 16:03:36 +01:00
James Ritchie 022bbe1371 Fixed type issues with emails 2024-09-05 17:22:58 +01:00
Eric Allam f040417440 v3: new build system fixes (#1278)
* Support custom config file names & paths

* Fix entry point paths on windows

* Support custom conditions

Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.

- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds

* Add changeset

* Fix passing CLI process.env down to the task processes

* Remove unused import

* reviving the e2e CLI tests

* Another attempt at fixing windows

* yet another windows attempt (yawa)

* Output index child stdout and stderr (yawa)

* normalize import paths for windows

* Added some logging (yawa)

* normalize the loader path as well

* Added some logging to figure out why the entry points aren’t being found on windows

* Fix for entry point detection on windows

* Normalize runner import loader path

* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well

* Various cleanup after windows fixes

* Remove the webapp e2e for now

* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)

https://github.com/andrewbranch/example-subpath-exports-ts-compat

* init using templates again but downloaded from the repo this time

* Adding init schedule example

* Support for js init

* init now working with js

* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
2024-09-05 12:56:23 +01:00
Eric Allam c6b8a5a64b Manually update package versions to 3.0.0-beta.56 2024-09-02 13:39:50 +01:00
Eric Allam e30beb779f Add support + docs for custom esbuild plugins (#1270)
* Add support + docs for custom esbuild plugins

* Fixed the missing subpath export

* Add support for specifying the prisma client generator

* Add `—env-file` support to dev and deploy
2024-08-25 16:04:26 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
github-actions[bot] 9d529e9f17 chore: Update version for release (beta) (#1264)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-20 14:46:19 +01:00
Matt Aitken 0591db5f2b Fix for short runs inside (batch)triggerAndWait (#1263)
* Test for checkpoints

* Make sourceTaskAttemptId optional on resumeBatchRun

* Removed all completions/executions logic from the shared queue consumer

* Removed the sourceTaskAttemptId from ResumeBatchRunService

* Revert "Removed all completions/executions logic from the shared queue consumer"

This reverts commit d35398d50463c81a1975bb5d5bcfca66a24b8ede.

* WIP on triggerAndWait…

* Fixed triggerAndWait continuing when a checkpoint completes

* Remove the ResumeAttempt code that fails attempts (was protecting against infinite restores)

* Removed messageBody.data.completedAttemptIds.length === 0 commented out code

* Don’t ack if there’s no batchRun

* Added the marqs?.replaceMessage back in but NOT when there’s no checkpoint. More logging

This is a fix for when some attempts fail

* Improvement to the test task that now randomly fails attempts

* When a checkpoint happens, only continue the attempt if it’s in the correct state

* Changeset for rollback in branch

* Set keepRunAlive to false when the dependent task isn’t finished

* Changeset manual version (to get inline with the hotfix branch)

* Changeset: Fixes for continuing after waits

* Latest lockfile (after manual changeset version)
2024-08-20 13:57:08 +01:00
github-actions[bot] 7cb977ff2c chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-14 11:39:44 +01:00
nicktrn 5cf90da722 v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers
2024-08-13 11:44:40 +01:00
github-actions[bot] a49edd3251 chore: Update version for release (beta) (#1237)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-01 12:57:39 +01:00
Quetzalcoatl c1d4c04e89 add xdg command check for linux servers (#1225)
* add xdg command check for linux

* bump version of cli-v3, add changeset

* Don't bump the version

* lint and log

* prettier log message

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2024-07-30 15:15:06 +01:00
github-actions[bot] 53cf5c089d chore: Update version for release (beta) (#1233)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-24 13:55:32 +01:00
Eric Allam be3157ace7 Don’t yet auto-bundle monorepo dependencies 2024-07-23 12:39:45 +01:00
Émile Ré ae9a8b0ceb Monorepo packages automatic handling (#1208)
* Add fixture

* Update fixture name

* Update fixture to work with npm & pnpm

* Update setup/teardown and README

* Add support for monorepo to e2e suite utilities

* Use direct dependencies info in esbuild plugin

* Renaming

* Add support for yarn

* Add fixture for npm

* Remove console.log call

* Edge cases

* Remove unused options

* Fix error unknown yarn in CI

* Fix logger debug call's error field

* Add changeset

* Fix missing span end
2024-07-23 11:02:48 +01:00
Émile Ré 6e65591e87 Fix cli v3 e2e suite issues (#1207)
* Fix resolve-legacy-peer-deps when PM=pnpm

* Fix setup/teardown

* Fix skip test before installing dependencies

* Re-enable e2e suite in CI

* Add changeset
2024-07-22 16:53:45 +01:00
github-actions[bot] 8b10a592f3 chore: Update version for release (beta) (#1229)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-19 11:33:36 +01:00
github-actions[bot] e8993ee882 chore: Update version for release (beta) (#1211)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-17 14:18:40 +01:00
Matt Aitken d6c6dc993d try/catch opening the v3 CLI login URL. Fix for Ubuntu 2024-07-17 13:15:42 +01:00
nicktrn c2707800aa Improve prisma errors for missing generate step 2024-07-17 09:53:38 +01:00
nicktrn dbda820a71 v3: fix websocket timeouts (#1220)
* prevent uncaught websocket exceptions

* dev cli sends regular pings to keep connection alive

* add changeset

* fix for prod message forwarding

* add server-side websocket ping

* Revert "dev cli sends regular pings to keep connection alive"

This reverts commit 9a6a7a21433b9f0665f72be015d0ba5a8d83eba4.

* add dev cli websocket debug logs, always print errors

* actually start the ping service

* update changeset

* catch remaining dev cli message handler errors
2024-07-17 09:05:15 +01:00
Gautam Singh e417aca879 v3: add extraCACerts config (#1214)
* Added extraCACerts config to help with self signed/private ca cert chain

* changeset

* Update changeset

* resolved pr comments

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2024-07-17 08:56:52 +01:00
nicktrn 9971de6a1e Increase span attribute value length limit to 2048 2024-07-16 08:15:05 +01:00
github-actions[bot] e341df806c chore: Update version for release (beta) (#1209)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-09 14:05:49 +01:00
github-actions[bot] 2f14e5136d chore: Update version for release (beta) (#1203)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-09 12:24:09 +01:00
Eric Allam 16ad595338 v3: update @depot/cli to latest 0.0.1-cli.2.71.0 2024-07-08 19:33:15 +01:00
github-actions[bot] ad68a3cc05 chore: Update version for release (beta) (#1199)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-04 14:07:42 +01:00
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
github-actions[bot] b9b86c89a7 chore: Update version for release (beta) (#1188)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-07-01 17:16:11 +01:00
nicktrn b171fde483 Fix index failure completion override 2024-06-29 19:23:32 +01:00
nicktrn 5ae3da6b4e v3: various small fixes (#1192)
* Await file watcher cleanup in dev

* Fix artifact detection logs

* Fix next runs table when schedule disabled

* Improve OOM error messages

* Add test link to completed deployment message

* Fix OOM detection, again

* Add changeset
2024-06-29 16:48:01 +01:00
Eric Allam f565829959 v3: Copy over more of the project's package.json keys into the deployed package.json (support for custom config like zenstack) 2024-06-28 20:13:54 +01:00
Eric Allam 75ec4ac6a6 v3: postInstall config option now replaces the postinstall script found in package.json 2024-06-28 15:59:41 +01:00
nicktrn 374b6b9c0c Increase dev worker timeout 2024-06-28 15:56:51 +01:00
nicktrn d0d3a64bd6 v3: misc CLI improvements (#1173)
* prevent downgrades during update check

* detect bun and use npm instead

* detect missing tsconfig during init and print helpful error

* add changeset

* add links to dev worker started message

* allow users to add custom pkg manager args during init

* update changeset

* fix links in unsupported terminals

* deprecate terminalLink

* update changeset
2024-06-28 15:51:17 +01:00
nicktrn 568da01785 v3: improve non-zero exit errors (#1179)
* docker provider will optionally enforce machine presets

* update task monitor oom message

* add oom task to v3 catalog

* improve handling of non-zero exit errors

* add changeset
2024-06-28 15:50:04 +01:00
nicktrn c75e29a9a7 v3: add audio binaries to worker images (#1183)
* add sox

* add audiowaveform

* add changeset
2024-06-28 15:49:24 +01:00
Émile Ré e5d26bd12d Fix infisical-sdk fixture, use in config instead (#1190) 2024-06-28 09:50:24 +01:00
Émile Ré 52b6f48a94 Cli v3 e2e: fixtures (#1184)
* Run tests

* Remove useless comments

* Fix readme

* Add fixtures for config step

* Add fixture for additionalDependencies name parsing scope issue

* Add URL to commit fix

* Some fixtures will only use 1 package manager

* Add resolve-trigger-deps fixture

* use rimraf in test setup

* Add legacy-peer-deps fixture

* Use vitest fixtures

* Add nested peer dep locking issue

* Add missing peer dep resolving fixture

* Get rid of global namespace, fix parallelism

* Add console logs to debug log level

* Add changeset

* Remove comments

* Update README.md
2024-06-27 13:55:54 +01:00
github-actions[bot] 9ba2a217a4 chore: Update version for release (beta) (#1187)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-06-27 13:38:57 +01:00