Commit Graph

5640 Commits

Author SHA1 Message Date
Matt Aitken c738ef39cb Fix for runs.list with from/to Date or timestamp 2024-09-16 15:15:34 +01:00
Eric Allam ca2cb7298c Another attempt at fixing the yarn CI failure 2024-09-16 15:13:19 +01:00
Niels 55b7ffeb72 Don't run release workflow on mdx changes (#1304) 2024-09-15 11:24:35 +01:00
Jacob Paris cfe0abd95e Update remix examples (#1303)
* Update remix examples

* fix: remove trailing space
2024-09-15 11:10:01 +01:00
James Ritchie c1690bd1ae New React to PDF example task (#1300)
* Better code snippet to clarify your framework

* New page for react-pdf

* Added 3 more examples

* New react to PDF example

* Alphabeticalise the side menu

* Removed github link and tweaked title
2024-09-13 16:46:52 +01:00
Matt Aitken 9a7ad926aa ElectricSQL run page (hidden page for now) (#1297)
* WIP on using react-window-splitter

* WIP with new resizable panels and SSR

* Use the cookie package

* Resizable storybook page

* Increase indexing memory limit

* Fixed v2 usage meter displaying when on paid plan (#1255)

* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects

* Concurrency page and more accurate tracking (#1252)

* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”

* v3: fix unfreezable state crashes for runs with multiple waits (#1253)

* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers

* Added more logging to TaskRunConcurrencyTracker and some more try/catches

* Call subscriber.messageDequeued in dequeueMessageInSharedQueue

* Added messageReplaced to concurrency tracking (when freezing)

* Added depenenciesToBundle guide to bundle all packages

* Include the old message data when replacing, so we get the projectId etc.

* Fix restored container names

* Fix for schedule page not scrolling

* Added a description panel to the Concurrency admin page

* chore: Update version for release (beta) (#1256)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.0.0-beta.53

* Added a note to use batchTrigger() instead of trigger()

* The latest react-window-splitter fixes the ESM issues

* Set sensible defaults for the run page

* Deployments page

* Test page

* Schedules page

* Latest version of react-window-splitter (0.2.5)

* Updated to the latest version: react-window-splitter

* Callout if runs don’t start right away now has some top margin

* Small padding fix

* styled the handle focus state

* Added isStaticAtRest prop to resizable panel

* Updated resizable storybook

* Inline code blocks behave nicer when text wraps

* Added ElectricSQL to docker-compose, available on 3060

* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend

* Use the new util

* More restructuring ready to use the trace summary from the frontend

* Using ElectricSQL for the run page data

* Min size for resizable panel on test page

* Don’t load the trace in the RunPresenter anymore

* Fix for the resizable panels on the run page

* Added overflow hidden to the panel group

* min size for the test page left hand panel

* Updated to latest window-splitter version

* Removed unused const

* One fix for client-server mismatch

* Slight improvement in the loading state

* Restructured the page so the loading is better

* Improvement to the loading states

* Improved the loading behaviour with the inspector

* WIP on auth, having problems with it

* Upgrade Remix to 2.9.1 (same as PR #1096)

* Switched structure around again so we only call the useTrace hook from the client

* Added auth to the sync

* Overscan more rows in the tree view

* Fix for TS error

* Remove duplicate import

* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"

This reverts commit e63ee9e924.

* save cookie only when id is used

* Deployment table now scrolls

* removed imports

* A lot of changes to make the inspector live too… WIP

* More major overhauls to get the synced version of the run page working…

* If a span is completed show that

* Set the debounce much lower for selecting the span view

* Load the details run inspector data on demand

* Delete the SpanPresenter

* Use the async payload because it deals with superjson

* Fixed weird merge conflict

* Share some inspector timeline components

* A couple of layout tweaks

* Improved the run inspector loading states

* Fix for paragaph errors

* Fix for focusing on a span

* Undefined typre for useSyncedShape

* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples

* Updated @electric-sql/react package to the latest

* Fix the timeline duration stretching

* Added some better error handling for the electric sync

* More logging

* Better error when there are bad responses

* Turn off resizable snapshots, there’s a bug

* Added getSpan back

* Added SpanPresenter back

* Updated to the new Electric hooks package

* Made a copy so we have the old run page and the new electric one

* Put the main eventRepository back for now

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-13 11:45:24 +01:00
Matt Aitken a66db33dee If there’s no attempt and no worker when finalizing a run, don’t try to create an attempt 2024-09-12 15:08:37 +01:00
Matt Aitken f90e23d503 Fix so failed cancels in the dashboard redirect back with an error message 2024-09-12 15:08:06 +01:00
Matt Aitken 0cc56040f3 batchTriggerAndWait checkpoint race condition when at max concurrency (#1296)
* Ignore /packages/cli-v3/src/package.json

* Added more logs when resuming a dependency, added the runId

* A task for reproducing a race condition with checkpoints

* Fix for doing remote image build when not self-hosting

* Set team members, alerts and schedule limits to 100m for self-hosting

* Import fix

* Set the checkpointEventId in marqs when the checkpoint is created for batchTriggerAndWait

This should fix a horrible race condition when at max concurrency
2024-09-12 14:27:42 +01:00
Eric Allam 67547d2521 Remove payload from task run task events 2024-09-12 14:12:16 +01:00
Eric Allam a50063ce04 Always insert the dirs option when initializing a new project in the trigger.config.ts 2024-09-12 14:12:16 +01:00
Eric Allam 8c690a960c Make sure BuildManifest is exported from @trigger.dev/build 2024-09-12 14:12:16 +01:00
D-K-P 511a24b9f3 Supabase docs improvements 2024-09-12 11:28:46 +01:00
Dan 7ca2b628f7 Added tutorial docs and images (#1295) 2024-09-12 10:10:51 +01:00
James Ritchie 9c61a25402 Docs/task examples (#1293)
* WIP first example task guide

* Added example link to github repo

* Created a top level Examples docs section

* Renamed side menu to examples for now
2024-09-11 17:37:49 +01:00
James Ritchie 972b5e34b8 Docs/more improvements (#1292)
* WIP creating a CLI section

* WIP adding more CLI commands in a new section

* New CLI section with introduction and more CLI commands

* More improvement to the CLI options

* Each CLI command has its own page

* fixed broken links

* Removed outdated copy
2024-09-11 17:14:39 +01:00
Eric Allam 47ca24aab8 Update new build sytem docs 2024-09-11 15:57:22 +01:00
Eric Allam 8578c9b281 v3: new build system fixes round 2 (#1283)
* Fixed empty env vars overriding in dev runs

* Don’t import package.json anymore

* fix node10 moduleResolution in @trigger.dev/core

* Support self-hosters pushing to a custom registry when running deploy

* dev: Fixed stuck runs when a child run fails with a process exit

* Make some doc notes about known issues and docker hub private repos

* Fix --project-ref when running deploy

* Fix —config option when deploying

* Fixing the flushing/killing process with the new build system

* Add monorepo-react-email e2e test fixture

* Fix issue with emitDecoratorMetadata and tsconfigs with extends

* Got the emit decorator metadata fixture working

* Fixed typechecking yarn e2e CLI tests in monorepos

* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)

* Remote externals now powered by JSON Hero to be easier to update

* resolve config source files

* Add a —javascript option to init, defaults to typescript

* Add support for prisma typed sql

* Remove msw and retry.interceptFetch

* Add missing code to the openai retries example

* Don’t generate the v3 catalog prisma client during CI

* Fixed v3-catalog task imports

* Remove interceptor usage in task file

* Only import import-in-the-middle hook if there are instrumented packages

* Fix yarn.lock file
2024-09-11 15:48:20 +01:00
Matt Aitken 8bb81ec73f The usage tasks table was showing double all values if you had a staging env… 🤦‍♂️ 2024-09-11 11:39:08 +01:00
Matt Aitken c46e3d528e Fix for BigInt values getting stripped from log properties 2024-09-11 11:20:22 +01:00
Matt Aitken 93bc32ca87 Checkpoint restoring should of triggerAndWait should use the child run 2024-09-10 15:46:24 +01:00
Matt Aitken 36397d0cb1 Added more logging when finalizing runs and when messages don’t exist in marqs 2024-09-10 14:52:32 +01:00
nicktrn 90bf01fb23 Decrease cpu requests by 25% 2024-09-10 14:28:05 +01:00
Matt Peck 15ed74b63e Allow custom docker network (#1291) 2024-09-09 22:54:27 +01:00
Matt Aitken 392453e8a6 Frozen run fixes (#1286)
* When resuming a batch, only do marqs operations once

* Made TaskRunDependency clearer in the Prisma schema

* New ResumeDependentParentsService service, use it from checkpoints

* WIP on making resuming more robust

* Turn the declarative schedules off because they make debugging other runs painful

* Resuming batches when there’s an attempt is working

* If there’s no attempt then create one

* Added a log if there are no span events to complete

* If Graphile addJob doesn’t return a row, log and return undefined. No throw

* Pass prisma into the ResumeDependentParentsService

* Removed the todos

* Pass Prisma through to the checkpoint service

* Fix for not checking the batch item correctly

* Fix for when a log flush times out and the process is checkpointed

* Fix for when a log flush times out and the process is checkpointed

* Another test run that does batches with failed subtasks

* Don’t call ResumeTaskRunDependenciesService anymore (we have a new service)

* Only resume if the run is in a final state

* If an attempt doesn’t exist, fix for creating queue with sanitized name

* If DEV then don’t resume using marqs/batches. The CLI manages it

* We don’t need to check the run status again, it’s in the main function now

* Added TaskRunAttempt taskRunId index

* Only allow calling ResumeDependentParentsService with a run ID

* Put the flushing back to what it was
2024-09-09 16:03:36 +01:00
Elie Steinbock e5f0aafb14 Include uploadthing in .env.example for references (#1288) 2024-09-08 10:28:16 +01:00
Elie Steinbock 05cdc6993c Remove astro as recommended extension (#1285) 2024-09-06 16:36:15 +01:00
James Ritchie 569dcf4ef4 Docs improvements (#1284)
* Separated the triggering docs page into 2 sections

* Added link to GitHub Actions from deploy page

* Improved the batchTriggerAndWait docs for handling errors

* Added docs for using google credential files

* Improved the triggering page copy some more

* Improved the google credentials copy
2024-09-06 14:26:58 +01:00
Eric Allam 87e46d497b projects now have a dedicated build server (#1281) 2024-09-06 09:36:03 +01:00
James Ritchie 022bbe1371 Fixed type issues with emails 2024-09-05 17:22:58 +01:00
James Ritchie 3e0881c2e9 Added project name to run alert email 2024-09-05 15:33:55 +01:00
Matt Aitken ec3c300cac Put the database command back in with wal_level logical 2024-09-05 14:18:21 +01:00
Matt Aitken 3f182cd145 Latest lockfile 2024-09-05 14:18:01 +01:00
Matt Aitken 09e6f58e84 Revert "ElectricSQL run page and react-window-splitter (#1277)"
This reverts commit 5231381717.

# Conflicts:
#	pnpm-lock.yaml
2024-09-05 14:13:16 +01:00
Eric Allam aa79b01f7e Update docs with new 0.0.0-prerelease-20240905123715 prerelease 2024-09-05 13:46:24 +01:00
Eric Allam f040417440 v3: new build system fixes (#1278)
* Support custom config file names & paths

* Fix entry point paths on windows

* Support custom conditions

Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.

- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds

* Add changeset

* Fix passing CLI process.env down to the task processes

* Remove unused import

* reviving the e2e CLI tests

* Another attempt at fixing windows

* yet another windows attempt (yawa)

* Output index child stdout and stderr (yawa)

* normalize import paths for windows

* Added some logging (yawa)

* normalize the loader path as well

* Added some logging to figure out why the entry points aren’t being found on windows

* Fix for entry point detection on windows

* Normalize runner import loader path

* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well

* Various cleanup after windows fixes

* Remove the webapp e2e for now

* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)

https://github.com/andrewbranch/example-subpath-exports-ts-compat

* init using templates again but downloaded from the repo this time

* Adding init schedule example

* Support for js init

* init now working with js

* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
2024-09-05 12:56:23 +01:00
Matt Aitken 5231381717 ElectricSQL run page and react-window-splitter (#1277)
* WIP on using react-window-splitter

* WIP with new resizable panels and SSR

* Use the cookie package

* Resizable storybook page

* Increase indexing memory limit

* Fixed v2 usage meter displaying when on paid plan (#1255)

* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects

* Concurrency page and more accurate tracking (#1252)

* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”

* v3: fix unfreezable state crashes for runs with multiple waits (#1253)

* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers

* Added more logging to TaskRunConcurrencyTracker and some more try/catches

* Call subscriber.messageDequeued in dequeueMessageInSharedQueue

* Added messageReplaced to concurrency tracking (when freezing)

* Added depenenciesToBundle guide to bundle all packages

* Include the old message data when replacing, so we get the projectId etc.

* Fix restored container names

* Fix for schedule page not scrolling

* Added a description panel to the Concurrency admin page

* chore: Update version for release (beta) (#1256)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.0.0-beta.53

* Added a note to use batchTrigger() instead of trigger()

* The latest react-window-splitter fixes the ESM issues

* Set sensible defaults for the run page

* Deployments page

* Test page

* Schedules page

* Latest version of react-window-splitter (0.2.5)

* Updated to the latest version: react-window-splitter

* Callout if runs don’t start right away now has some top margin

* Small padding fix

* styled the handle focus state

* Added isStaticAtRest prop to resizable panel

* Updated resizable storybook

* Inline code blocks behave nicer when text wraps

* Added ElectricSQL to docker-compose, available on 3060

* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend

* Use the new util

* More restructuring ready to use the trace summary from the frontend

* Using ElectricSQL for the run page data

* Min size for resizable panel on test page

* Don’t load the trace in the RunPresenter anymore

* Fix for the resizable panels on the run page

* Added overflow hidden to the panel group

* min size for the test page left hand panel

* Updated to latest window-splitter version

* Removed unused const

* One fix for client-server mismatch

* Slight improvement in the loading state

* Restructured the page so the loading is better

* Improvement to the loading states

* Improved the loading behaviour with the inspector

* WIP on auth, having problems with it

* Upgrade Remix to 2.9.1 (same as PR #1096)

* Switched structure around again so we only call the useTrace hook from the client

* Added auth to the sync

* Overscan more rows in the tree view

* Fix for TS error

* Remove duplicate import

* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"

This reverts commit e63ee9e924.

* save cookie only when id is used

* Deployment table now scrolls

* removed imports

* A lot of changes to make the inspector live too… WIP

* More major overhauls to get the synced version of the run page working…

* If a span is completed show that

* Set the debounce much lower for selecting the span view

* Load the details run inspector data on demand

* Delete the SpanPresenter

* Use the async payload because it deals with superjson

* Fixed weird merge conflict

* Share some inspector timeline components

* A couple of layout tweaks

* Improved the run inspector loading states

* Fix for paragaph errors

* Fix for focusing on a span

* Undefined typre for useSyncedShape

* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples

* Updated @electric-sql/react package to the latest

* Fix the timeline duration stretching

* Added some better error handling for the electric sync

* More logging

* Better error when there are bad responses

* Turn off resizable snapshots, there’s a bug

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-04 17:55:36 +01:00
Eric Allam c6b8a5a64b Manually update package versions to 3.0.0-beta.56 2024-09-02 13:39:50 +01:00
Matt Aitken 55fa386bdd Documentation edits made through Mintlify web editor 2024-08-29 21:55:10 +01:00
Eugen Istoc 7c1339c108 Update upgrading-packages.mdx (#1273)
I believe the name of the executable in the `bin` dir is called `triggerdev` instead of `trigger.dev`
2024-08-28 10:50:52 +01:00
nicktrn 4cfb52ef65 Remove packageManager override from docs package.json 2024-08-28 10:39:34 +01:00
Matt Aitken f7d32b83b1 Free up concurrency when using triggerAndWait. Improved errors on runs (#1272)
* Add an error to the final attempt if there isn’t one

* Improved the checkpointResumer test task

* Using triggerAndWait or batchTriggerAndWait frees up concurrency

Normally it frees up just env and org concurrency. If it’s a recursive task then it will free up the run concurrency too (e.g. a task calling itself).

* Removed the filepath and export name from attempt spans

* Run inspector: only show an error if the run is in a finished state
2024-08-25 20:15:45 +01:00
Matt Aitken 0979a521bd Updated docs to 0.0.0-prerelease-20240825150620 2024-08-25 16:12:55 +01:00
Eric Allam e30beb779f Add support + docs for custom esbuild plugins (#1270)
* Add support + docs for custom esbuild plugins

* Fixed the missing subpath export

* Add support for specifying the prisma client generator

* Add `—env-file` support to dev and deploy
2024-08-25 16:04:26 +01:00
Matt Aitken 99a63d4cd5 When enqueuing clear concurrency for that run (#1271)
* Better logging if a background worker isn’t created

* When enqueuing a message into the queue clear the concurrency
2024-08-25 15:56:40 +01:00
Matt Aitken 6744617879 Added @ splat route 2024-08-23 18:28:28 +01:00
Matt Aitken d62979914f Updated the new build system docs with the prerelease package 2024-08-23 14:26:56 +01:00
Matt Aitken fc02e95f2a Removed another old changeset 2024-08-23 14:20:48 +01:00
Matt Aitken 8688822723 Another fix for changeset with old package (react) 2024-08-23 14:20:36 +01:00
Matt Aitken 0d829f9530 Fixed changesets for packages no longer in the repo 2024-08-23 14:20:00 +01:00