Commit Graph

91 Commits

Author SHA1 Message Date
Eric Allam cf13fbdf32 Updating docs for the move from beta -> latest (with new build system) (#1299)
* Add triggerAndWait().unwrap() to more easily get at the output or throw the subtask error

* Add taskId and runId to SubtaskUnwrapError

* WIP docs update beta -> latest

* trigger.dev init now adds @trigger.dev/build to devDependencies

* How it works doc

* Restructure some docs and update the cli commands

* Config file docs, plus aptGet and ffmpeg extensions

* Update to latest from beta docs

* Add runtime to templates

* Add --runtime option to the init CLI command

* A bunch more doc updates after feedback

* Document triggerAndWait with unwrap and result types

* beta -> latest in the webapp

* CLI update check no longer references beta

* Add major release

* Leave changeset beta, back to normal package release

* Fixed default dirs option in init command

* exclude windows-yarn variation of cli e2e tests because it’s buggy

* Remove cache to try and fix yarn e2e test workflow errors
2024-09-16 15:30:32 +01:00
Eric Allam ca2cb7298c Another attempt at fixing the yarn CI failure 2024-09-16 15:13:19 +01:00
Niels 55b7ffeb72 Don't run release workflow on mdx changes (#1304) 2024-09-15 11:24:35 +01:00
Eric Allam f040417440 v3: new build system fixes (#1278)
* Support custom config file names & paths

* Fix entry point paths on windows

* Support custom conditions

Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.

- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds

* Add changeset

* Fix passing CLI process.env down to the task processes

* Remove unused import

* reviving the e2e CLI tests

* Another attempt at fixing windows

* yet another windows attempt (yawa)

* Output index child stdout and stderr (yawa)

* normalize import paths for windows

* Added some logging (yawa)

* normalize the loader path as well

* Added some logging to figure out why the entry points aren’t being found on windows

* Fix for entry point detection on windows

* Normalize runner import loader path

* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well

* Various cleanup after windows fixes

* Remove the webapp e2e for now

* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)

https://github.com/andrewbranch/example-subpath-exports-ts-compat

* init using templates again but downloaded from the repo this time

* Adding init schedule example

* Support for js init

* init now working with js

* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
2024-09-05 12:56:23 +01:00
Eric Allam 92d98e5499 Build before running typecheck now 2024-08-23 13:46:18 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
Émile Ré ae9a8b0ceb Monorepo packages automatic handling (#1208)
* Add fixture

* Update fixture name

* Update fixture to work with npm & pnpm

* Update setup/teardown and README

* Add support for monorepo to e2e suite utilities

* Use direct dependencies info in esbuild plugin

* Renaming

* Add support for yarn

* Add fixture for npm

* Remove console.log call

* Edge cases

* Remove unused options

* Fix error unknown yarn in CI

* Fix logger debug call's error field

* Add changeset

* Fix missing span end
2024-07-23 11:02:48 +01:00
Émile Ré 6e65591e87 Fix cli v3 e2e suite issues (#1207)
* Fix resolve-legacy-peer-deps when PM=pnpm

* Fix setup/teardown

* Fix skip test before installing dependencies

* Re-enable e2e suite in CI

* Add changeset
2024-07-22 16:53:45 +01:00
nicktrn 1249523922 Switch from latest to v3 tag for versioned container images 2024-07-09 10:27:35 +01:00
nicktrn 7aed154854 Update pnpm setup action 2024-07-03 17:06:57 +01:00
Eric Allam d57dec6919 Remove e2e tests from CI for now 2024-06-28 16:17:58 +01:00
Émile Ré d4ccdf7105 v3 CLI compiling E2E test suite (#1135)
* Boilerplate server-only use case

* wip: integration suite instrumentation setup

* Working poc testing compileProject

* Add pnpm script to run e2e tests only

* Use vitest globals

* Remove commented line

* Remove useless export

* Add modifier to test only one fixture project

* Handle package manager and log level choice

* Update server-only example

* Setup / teardown + split compile for package manager capabilities

* Ignore yarn files

* Fix issue with corepack, store version in engines field

* Rename test file

* Fix npm updates yarn.lock

* Move typecheking in a dedicated test

* Stop bundling the compile command to allow for more granular testing

* Put config resolving in separate test

* Add no-config test case and add test case expected errors configuration

* Add wantCompilationError option

* Add dependencies handling

* Use packageManager passed as option to resolve required deps

* Remove unused guard clauses

* Add postinstall & hash handling step

* Add worker start test

* Handle yarn.lock copy renaming on sigterm and sigkill

* Update vitest and use concurrent option

* Add a readme file

* Add CI workflow

* Fix handle cli deps

* Run cli v3 e2e tests on publish action

* Increase timeout on deps resolving step

* Add changeset

* Remove .pnp.cjs as we use yarn with nodeLinker node-modules

* Add missing .yarnrc.yml file

* No need to build CLI to run E2E tests

* Remove bun.lockb files

* Update beige-pears-explode.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-06-14 11:28:36 +01:00
nicktrn 35dbaedf69 v3: self-hosting (#1147)
* add amin email regex env var

* fix displayed init command for self-hosted setups

* shared env var to disable telemetry in cli and webapp

* pin sdk version during init

* if specified, add api url to dev command shown after init

* improve checkpoint support detection

* control forced checkpoint simulation via env var

* add public init to providers

* better checkpoint support check for coordinator

* add docker to coordinator image

* update docker provider containerfile

* bump remaining containers to node 20

* add infra image build to default publish workflow

* lockfile

* remove concurrency group from infra workflow

* add docker provider to build matrix

* fix var subst

* checkpoint test is docker specific

* enable v3 projects by default on self-hosted instances

* fix v3 setup command again

* add default posthog key

* self-hosting docs

* add latest tags to versioned infra and webapp builds

* some checkpoint errors should skip retrying

* add changeset

* shorten paragraph

* some docs updates

* update tunnelling section

* add registry setup section

* use correct cli push flag

* add checkout to v3 branch

* update the worker machine setup steps

* fix infra build

* small docs update

* remove unused feature function

* Revert "remove unused feature function"

This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.

* fix self-hosted v3 feature gate

* add note about missing arm support

* simplify helper script syntax
2024-06-10 14:13:04 +01:00
Eric Allam affc128161 Try to fix prisma generate errors 2024-04-26 18:28:17 +01:00
Eric Allam 12c83a56af Skip e2e github workflows for now because playwright is broken 2024-04-24 10:33:11 +01:00
nicktrn 1e93ec4216 Revert "change action runner location" (resolved)
This reverts commit cb27b7278a.
2024-04-24 07:45:08 +01:00
nicktrn cb27b7278a change action runner location ⬆️⬆️⬇️⬇️⬅️➡️⬅️➡️🅱️🅰️ 2024-04-23 18:13:04 +01:00
Eric Allam e398bdf859 Only use 16vcpu on e2e 2024-03-27 16:27:28 +00:00
Eric Allam 53054d0146 Enter into changesets “beta” tag for packages 2024-03-27 14:55:14 +00:00
Eric Allam 412d0149d5 Fixed release workflow 2024-03-27 12:37:23 +00:00
Eric Allam d614c8d2ca More CI fixes for node 20/pnpm 8 2024-03-27 12:33:24 +00:00
Eric Allam 4329d1130a Attempt to fix typecheck 2024-03-27 12:30:54 +00:00
nicktrn 9ecf07731a v3: small features and fixes for prod infra (#968)
* add checkpoint restore events

* fix retries.enabledInDev

* fix tsconfig paths

* fix provider build and dev

* update kubernetes provider and fix builds again

* upgrade prod base to node 20

* update infra publish workflow

* rethrow k8s errors after logging

* shorten index container names

* fix error type assertion

* rename type assertion

* remove resource limits for now

* add missing run id on create

* add push to deploy command for self-hosting

* checkpointing fixes

* update coordinator image

* ensure valid registry login

* delete checkpoint archive after successful push

* log options on error

* structured logs for socket connections

* fix structured log merge

* exit process after checkpointing

* update restore pull secret name

* append shortcode to restore names

* log handler payload

* disable post start lifecycle hook

* pass in coordinator host via volume

* replace dapi with taskinfo

* add missing restore label

* don't restart restored containers

* remove init container from create

* atomic post-completion checkpoints

* switch to run id for container names

* improve wait accuracy

* measure basic checkpoint perf

* always log disconnect reason

* use system clock to end wait spans

* checkpoint readiness and cancel signals

* restore from checkpoint events and fix statuses

* remove attempt id env var

* restore dependencies from events

* reconnect wip

* lifecycle hooks are back

* fix hooks and improve reconnect

* make docker send postStart hook

* only checkpoint for retry if large delay

* fix a few more resume issues

* lifecycle hook fixes

* skip connection handler when waiting for post start hook

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-25 16:28:56 +00:00
Matt Aitken de302858f8 v3 docs (#929)
* Started building out the docs navigation

* Drafted out a lot of the docs pages. With just titles and descriptions with a coming soon message for now.

* A ton more scaffolded pages with titles, descriptions, coming soon/incomplete docs messages

* Created the remaining page stubs

* Introduction page improvements

* Lots of docs changes, mostly the task overview page

* Moved some bits around

* Task overview fleshed out

* Triggering docs page

* Regular task page

* CLi dev docs

* Docs for debugging locally

* Environment variables docs

* Added env var images

* Deployment guides

* Updated the title of “Integrations” to “Deployment integrations” to make it less confusing

* Lots more docs pages

* Changed the language to JavaScript

* Fixed typos about queues and added a simple code sample

* Moved the local debugger docs into the CLI dev page

* Added the CLI deploy options

* Delete the local debugger page

* Updated the logging page with structured logging

* Added the new CLI deploy options

* Change to h4 for one of the options

* Errors page

* WIP on retrying

* Retrying guide

* Wait docs pages

* Queuing docs simplified, partial written

* Queuing and concurrency docs

* Run tests docs

* New combined error and retrying page

* Errors and retrying page updated

* More docs

* Added a possible configurations coming soon table

* Created a limits page

* Added warnings about how you need to get early access

* Minor fixes for the docs
2024-03-22 14:09:27 +00:00
Eric Allam affa3f25ee Trying beefier buildjet machines for CI 2024-03-19 11:38:48 +00:00
Eric Allam 6afda40c35 Add depot setup action 2024-03-19 09:24:33 +00:00
Eric Allam f26ba9a5cd Use depot.dev for docker builds (remove publishing to latest for now) 2024-03-18 20:52:08 +00:00
nicktrn 9b6f8f9238 v3: provider and coordinator updates (#927)
* add zod socket

* start using zod socket

* use zod socket for shared queue

* use shared provider shell

* update k8s provider task ops

* set otlp endpoint

* update docker actions and fix builds

* remove unused types

* update injected secret key env var name

* complete socket.io types to schemas migration

* update env example to new v3 key var

* fix coordinator build

* set task run label

* ensure attempt id is always set

* increase prod worker timeout

* set otlp endpoint for on runs

* pass otlp env var to prod worker

* auto-remove completed index jobs
2024-03-06 16:01:59 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam 933cf0f119 v3: run.isTest now propagates through child tasks and the trace (#917)
* run.isTest now propagates through child tasks and the trace

* Try to fix the pr checks docker hub rate limit error
2024-02-28 17:01:31 +00:00
Matt Aitken d04c8e89f6 Added a “repo-token” to the Install Protoc GitHub action steps
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
2024-02-19 14:15:15 +00:00
Eric Allam 6be1099a12 Don’t git submodule in docker 2024-02-12 14:53:03 +00:00
Eric Allam baf3a84cda v3 dev cli command + more (#894) 2024-02-12 14:24:32 +00:00
nicktrn 225614fea3 Feature: ARM builds (#821)
* Add native ARM runner

* Switch to arch-agnostic manifest digest

* Set correct latest tag
2024-01-02 15:48:53 +00:00
Eric Allam 17f6f29d05 Feature: Support multiple runtimes other than Node.js (#774)
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
2023-12-06 10:23:37 +00:00
Eric Allam 56ae0ed0d3 Only tag v.docker tags with the latest tag 2023-11-24 16:46:06 +00:00
Eric Allam a2f5dc8bc6 Only attempt to login to Docker if the workflow event is a push 2023-11-24 14:16:41 +00:00
Eric Allam 03403ad9cb Fix for pull request workflow (#751)
* Allow PRs that have /publish-docker in the body to cause docker images to be published

* Also ignore mdx file changes

* pr_checks now uses pull_request instead of pull_request_target
2023-11-24 11:19:06 +00:00
Eric Allam 624072c23b Update release.yml 2023-10-10 15:04:13 +01:00
Eric Allam d37576f814 Update release.yml 2023-10-10 15:03:30 +01:00
Eric Allam 8c8d1598e3 Reuse common workflow jobs in publish workflow 2023-10-09 18:30:55 +01:00
Eric Allam 646bfd37bd Pushing a commit to the branch so I can open a PR (#592)
* Pushing a commit to the branch so I can open a PR

* Couple improvements to the pr checks workflow
2023-10-09 18:28:49 +01:00
Eric Allam 8c383bdc69 Add a PR checks github workflow to run typecheck, unit tests, and e2e tests on PRs 2023-10-09 18:19:06 +01:00
Eric Allam b9ed7e2ced Allow blank issues 2023-10-04 13:21:38 +01:00
Eric Allam 486ed20ae7 Renamed the examples dir to references (true examples are in another repo and this was confusing) 2023-09-15 16:09:04 +01:00
Eric Allam f98c425186 Redirect people to discord to ask a question 2023-09-15 12:54:54 +01:00
Eric Allam 5aeef1233b A few teaks to the templates 2023-09-15 12:52:29 +01:00
Vishesh Rawal fb5f4e308f Created Temp for pull req, bugreport & feature req 2023-09-15 12:52:29 +01:00
Matt Aitken 2895ad52fd Re-run the publish workflow when the test have been updated 2023-09-13 13:56:44 +01:00
Matt Aitken 2636d9091e Release workflow: the changeset folder wasn’t set correctly 2023-09-05 15:14:46 +01:00