Commit Graph

30 Commits

Author SHA1 Message Date
nicktrn 3f982ed366 docs: update email regex examples (#2749)
The previous examples lacked start and end anchors
2025-12-04 17:58:25 +00:00
Matt Aitken 68d0037e60 fix(dev): dotenv issues when getting setup with the repo (#2612)
- The .env.example was missing required Clickhouse values.
- A symlink was needed for .env from the root to the apps/webapp folder
2025-10-17 11:49:40 +01:00
Matt Aitken 630e9556b0 Bulk actions 2.0 (and switch all run listing to ClickHouse) (#2264)
* useSearchParams has

* useSearchParams has

* useSearchParams has

* Consistent way to get the run filters

* Consistent way to get the run filters

* Consistent way to get the run filters

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* WIP actions and filtering

* WIP actions and filtering

* WIP actions and filtering

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* Migrations

* Migrations

* Migrations

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* Bulk action button

* Bulk action button

* Bulk action button

* Make the next runs page the default now

* Make the next runs page the default now

* Make the next runs page the default now

* Improved the RadioButton style

* Improved the RadioButton style

* Improved the RadioButton style

* Remove the old bulk action bar

* Remove the old bulk action bar

* Remove the old bulk action bar

* More UI progress

* More UI progress

* More UI progress

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Improved the Switch secondary style

* Improved the Switch secondary style

* Improved the Switch secondary style

* Buggy filter fixes

* Buggy filter fixes

* Buggy filter fixes

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Clear button is minimal

* Clear button is minimal

* Clear button is minimal

* Using a presenter now

* Using a presenter now

* Using a presenter now

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Individually selected runs working

* Individually selected runs working

* Individually selected runs working

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Added the modal

* Added the modal

* Added the modal

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Renamed bulk action file

* Renamed bulk action file

* Renamed bulk action file

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* WIP on bulk actions page

* WIP on bulk actions page

* WIP on bulk actions page

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Added an option action completion email

* Added an option action completion email

* Added an option action completion email

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Nicer completed email

* Nicer completed email

* Nicer completed email

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Updates secondary/small switch style

* Updates secondary/small switch style

* Updates secondary/small switch style

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Aborting working

* Aborting working

* Aborting working

* Bulk action live reloading

* Bulk action live reloading

* Bulk action live reloading

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Wider bulk action onboarding

* Wider bulk action onboarding

* Wider bulk action onboarding

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Improved the accordion component hover states

* Improved the accordion component hover states

* Improved the accordion component hover states

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Fix for selecting a single run

* Fix for selecting a single run

* Fix for selecting a single run

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Animate the progress bar

* Set TZ="UTC" in the env example

* Filter summary in the bulk inspector

* Improves the pagination styling

* Improves the pagination styling

* Delete old bulk action routes

* Removed old Postgres RunListPresenter

* Retry any replication error where the message contains "timeout"

* Increase wait to make test less flaky

* The test was using run id instead of friendly id

* Safer array access

* Remove error log if there's a bad status

* Nicer frontend type safety with the bulk action and mode

* Switched a log to a debug log

* Retry replication unless the error is a known non-retry error

Flip the strategy to retry by default

* Make ClickHouse required

* Backfill run replication admin API endpoint

* Set a CLICKHOUSE_URL for unit tests

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-14 15:50:01 +01:00
nicktrn 47b19a49d4 A few small v4 fixes (#2153)
* remove unused env vars

* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)

* increase fallback branch limit to 100M

* more unused env vars and example

* hide usage page when self-hosted

* fix for init flow

* set default or concurrency to 300 to match 100 on env

* fail fast when registry env vars are empty strings
2025-06-06 10:15:07 +01:00
nicktrn e7795a06ad Fix: fixes and prerequisites for v4 self-hosting (#2150)
* remove pgadmin

* remove V3_ENABLED

* v3 is always enabled

* enfore docker machine presets by default

* rename autoremove env var

* prefix more k8s-specific env vars

* same prefix for all docker settings

* improve profile switcher copy

* supervisor can load token from file

* optional webapp worker group bootstrap

* fix error message

* fix app origin fallback for otlp endpoint

* use pnpm cache for webapp docker builds

* increase default org and env concurrency limit to 100

* optional machine preset overrides

* improve s3 pre-signing errors

* fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion

* shard unit tests

* fix for s3-compatible services

* optional object store region

* Update apps/supervisor/src/workerToken.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix DEPLOY_REGISTRY_HOST example

* fix platform mock

* remove remaining v3Enabled refs

* fix error type.. bad bot

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 18:02:38 +01:00
nicktrn f603725393 Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138)
* remove registry proxy

* remove --self-hosted flag

* automatically set network build flag

* update syncEnvVars debug log

* improve switch command

* always display deploy errors if they exist

* fix stuck deploy command after finalize error

* webapp-driven deploys, multi-platform support, lots of fixes

* add worker deployment migration

* rename image platform env var

* only try to sync parent env vars for preview deployments

* add KEEP_TMP_DIRS

* supervisor: docker api version lock, auth, multi-platform

* set image ref on create, validate digest

* use metadata for digest, fix local multi-platform builds

* print git meta branch before commit

* improve push and load flag handling

* make runs after local builds compatible with load and push

* small improvement for platform overrides

* add image platform to dequeued message

* remove deprecated init request body fields

* fix fail deployment id param

* remove build debug logs

* pass report merge with no tests

* structured run debug logs

* add required env var for tests

* should not be an error log

* add changeset
2025-06-04 15:54:10 +01:00
Erin Allison 4d2412a0fe Add Support for Alternate Email Transports (#1580)
* Add support for AWS SES and SMTP email transports

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Correct environment variable names for new email settings

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Correct option names being passed to nodemailer for SMTP

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Remove use of AWS SDK synthetic default export

It apparently causes issues when transpiled/bundled

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Add documentation for new email settings

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Move nodemailer types to devDependencies

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Adjust mail transport error handling

Gotta keep the linter happy :)

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Fix typecheck error on MailTransportOptions

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Correct environment variable usage for SMTP email transport

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

---------

Signed-off-by: Erin Allison <eallison@andrettikarting.com>
2024-12-30 17:38:17 +00:00
Matt Aitken 9a7ad926aa ElectricSQL run page (hidden page for now) (#1297)
* WIP on using react-window-splitter

* WIP with new resizable panels and SSR

* Use the cookie package

* Resizable storybook page

* Increase indexing memory limit

* Fixed v2 usage meter displaying when on paid plan (#1255)

* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects

* Concurrency page and more accurate tracking (#1252)

* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”

* v3: fix unfreezable state crashes for runs with multiple waits (#1253)

* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers

* Added more logging to TaskRunConcurrencyTracker and some more try/catches

* Call subscriber.messageDequeued in dequeueMessageInSharedQueue

* Added messageReplaced to concurrency tracking (when freezing)

* Added depenenciesToBundle guide to bundle all packages

* Include the old message data when replacing, so we get the projectId etc.

* Fix restored container names

* Fix for schedule page not scrolling

* Added a description panel to the Concurrency admin page

* chore: Update version for release (beta) (#1256)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.0.0-beta.53

* Added a note to use batchTrigger() instead of trigger()

* The latest react-window-splitter fixes the ESM issues

* Set sensible defaults for the run page

* Deployments page

* Test page

* Schedules page

* Latest version of react-window-splitter (0.2.5)

* Updated to the latest version: react-window-splitter

* Callout if runs don’t start right away now has some top margin

* Small padding fix

* styled the handle focus state

* Added isStaticAtRest prop to resizable panel

* Updated resizable storybook

* Inline code blocks behave nicer when text wraps

* Added ElectricSQL to docker-compose, available on 3060

* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend

* Use the new util

* More restructuring ready to use the trace summary from the frontend

* Using ElectricSQL for the run page data

* Min size for resizable panel on test page

* Don’t load the trace in the RunPresenter anymore

* Fix for the resizable panels on the run page

* Added overflow hidden to the panel group

* min size for the test page left hand panel

* Updated to latest window-splitter version

* Removed unused const

* One fix for client-server mismatch

* Slight improvement in the loading state

* Restructured the page so the loading is better

* Improvement to the loading states

* Improved the loading behaviour with the inspector

* WIP on auth, having problems with it

* Upgrade Remix to 2.9.1 (same as PR #1096)

* Switched structure around again so we only call the useTrace hook from the client

* Added auth to the sync

* Overscan more rows in the tree view

* Fix for TS error

* Remove duplicate import

* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"

This reverts commit e63ee9e924.

* save cookie only when id is used

* Deployment table now scrolls

* removed imports

* A lot of changes to make the inspector live too… WIP

* More major overhauls to get the synced version of the run page working…

* If a span is completed show that

* Set the debounce much lower for selecting the span view

* Load the details run inspector data on demand

* Delete the SpanPresenter

* Use the async payload because it deals with superjson

* Fixed weird merge conflict

* Share some inspector timeline components

* A couple of layout tweaks

* Improved the run inspector loading states

* Fix for paragaph errors

* Fix for focusing on a span

* Undefined typre for useSyncedShape

* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples

* Updated @electric-sql/react package to the latest

* Fix the timeline duration stretching

* Added some better error handling for the electric sync

* More logging

* Better error when there are bad responses

* Turn off resizable snapshots, there’s a bug

* Added getSpan back

* Added SpanPresenter back

* Updated to the new Electric hooks package

* Made a copy so we have the old run page and the new electric one

* Put the main eventRepository back for now

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-13 11:45:24 +01:00
Matt Aitken 09e6f58e84 Revert "ElectricSQL run page and react-window-splitter (#1277)"
This reverts commit 5231381717.

# Conflicts:
#	pnpm-lock.yaml
2024-09-05 14:13:16 +01:00
Matt Aitken 5231381717 ElectricSQL run page and react-window-splitter (#1277)
* WIP on using react-window-splitter

* WIP with new resizable panels and SSR

* Use the cookie package

* Resizable storybook page

* Increase indexing memory limit

* Fixed v2 usage meter displaying when on paid plan (#1255)

* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects

* Concurrency page and more accurate tracking (#1252)

* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”

* v3: fix unfreezable state crashes for runs with multiple waits (#1253)

* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers

* Added more logging to TaskRunConcurrencyTracker and some more try/catches

* Call subscriber.messageDequeued in dequeueMessageInSharedQueue

* Added messageReplaced to concurrency tracking (when freezing)

* Added depenenciesToBundle guide to bundle all packages

* Include the old message data when replacing, so we get the projectId etc.

* Fix restored container names

* Fix for schedule page not scrolling

* Added a description panel to the Concurrency admin page

* chore: Update version for release (beta) (#1256)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.0.0-beta.53

* Added a note to use batchTrigger() instead of trigger()

* The latest react-window-splitter fixes the ESM issues

* Set sensible defaults for the run page

* Deployments page

* Test page

* Schedules page

* Latest version of react-window-splitter (0.2.5)

* Updated to the latest version: react-window-splitter

* Callout if runs don’t start right away now has some top margin

* Small padding fix

* styled the handle focus state

* Added isStaticAtRest prop to resizable panel

* Updated resizable storybook

* Inline code blocks behave nicer when text wraps

* Added ElectricSQL to docker-compose, available on 3060

* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend

* Use the new util

* More restructuring ready to use the trace summary from the frontend

* Using ElectricSQL for the run page data

* Min size for resizable panel on test page

* Don’t load the trace in the RunPresenter anymore

* Fix for the resizable panels on the run page

* Added overflow hidden to the panel group

* min size for the test page left hand panel

* Updated to latest window-splitter version

* Removed unused const

* One fix for client-server mismatch

* Slight improvement in the loading state

* Restructured the page so the loading is better

* Improvement to the loading states

* Improved the loading behaviour with the inspector

* WIP on auth, having problems with it

* Upgrade Remix to 2.9.1 (same as PR #1096)

* Switched structure around again so we only call the useTrace hook from the client

* Added auth to the sync

* Overscan more rows in the tree view

* Fix for TS error

* Remove duplicate import

* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"

This reverts commit e63ee9e924.

* save cookie only when id is used

* Deployment table now scrolls

* removed imports

* A lot of changes to make the inspector live too… WIP

* More major overhauls to get the synced version of the run page working…

* If a span is completed show that

* Set the debounce much lower for selecting the span view

* Load the details run inspector data on demand

* Delete the SpanPresenter

* Use the async payload because it deals with superjson

* Fixed weird merge conflict

* Share some inspector timeline components

* A couple of layout tweaks

* Improved the run inspector loading states

* Fix for paragaph errors

* Fix for focusing on a span

* Undefined typre for useSyncedShape

* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples

* Updated @electric-sql/react package to the latest

* Fix the timeline duration stretching

* Added some better error handling for the electric sync

* More logging

* Better error when there are bad responses

* Turn off resizable snapshots, there’s a bug

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-04 17:55:36 +01:00
Hamza 83a35226a7 removed CLOUD_ENV from .env.example (#1230) 2024-07-24 12:02:03 +01:00
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
nicktrn 35dbaedf69 v3: self-hosting (#1147)
* add amin email regex env var

* fix displayed init command for self-hosted setups

* shared env var to disable telemetry in cli and webapp

* pin sdk version during init

* if specified, add api url to dev command shown after init

* improve checkpoint support detection

* control forced checkpoint simulation via env var

* add public init to providers

* better checkpoint support check for coordinator

* add docker to coordinator image

* update docker provider containerfile

* bump remaining containers to node 20

* add infra image build to default publish workflow

* lockfile

* remove concurrency group from infra workflow

* add docker provider to build matrix

* fix var subst

* checkpoint test is docker specific

* enable v3 projects by default on self-hosted instances

* fix v3 setup command again

* add default posthog key

* self-hosting docs

* add latest tags to versioned infra and webapp builds

* some checkpoint errors should skip retrying

* add changeset

* shorten paragraph

* some docs updates

* update tunnelling section

* add registry setup section

* use correct cli push flag

* add checkout to v3 branch

* update the worker machine setup steps

* fix infra build

* small docs update

* remove unused feature function

* Revert "remove unused feature function"

This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.

* fix self-hosted v3 feature gate

* add note about missing arm support

* simplify helper script syntax
2024-06-10 14:13:04 +01:00
Eric Allam c9ebe7f114 Update contributing guide for local v3 development 2024-05-20 15:43:31 +01:00
Matt Aitken 80997bcd6d .env.example Redis env vars are uncommented 2024-05-12 18:02:42 +01:00
Eric Allam bc020a3ffe Get the internal telemetry ready for test/prod 2024-03-28 18:12:01 +00:00
Eric Allam 719c0a0b94 v3: better clock management for spans/logs in CRIU envs (#980)
* durable clock WIP

* Fixed incorrect span timings around checkpoints by implementing a precise wall clock that resets after restores

* Add changeset
2024-03-28 14:20:44 +00:00
Eric Allam 215b60c742 v3: superjson output support (#971)
* superjson output support

* Better support for superjson in the task events, limiting output attributes

* Offload large outputs to object store (r2)

* Finishing up the offloading of large outputs/payloads to an object store
2024-03-27 11:57:22 +00:00
Eric Allam ba0ffacb15 Set the OTEL_EXPORTER_OTLP_ENDPOINT env var on dev workers 2024-03-18 20:16:25 +00:00
Eric Allam 67cac97b26 Add DEPLOY_REGISTRY_HOST to allow configuring the deploy registry via the server 2024-03-18 15:56:28 +00:00
Eric Allam c385db63be v3: cli deploy command (#931)
* WIP proxy/deploy

* WIP, registry proxy in express working

* A couple of notes, preparing for indexing

* Move the changes to prod-worker into the new file

* Deploy command working with indexing and runs (docker provider only for now)

* Removed ts-expect-error directive

* Fixed build command
2024-03-08 10:49:42 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam 07f7c65054 v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev

* Convert MarQS to using lua and dealing with concurrency

* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)

* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects

* Start adding internal telemetry support for the server

* Get env vars to work in dev and implement prisma tracing in webapp

* Cleanup telemetry and implement it in the consumer

* Implement dequeuing a message from a parent shared queue

* Implement a custom logger exporter instead of using console log exporter

* Use node instead of shell for generating protocol buffer code

* Propogate trace context into debug logs, and allow turning off logger exporter through env vars

* Switch to using baselime for internal otel data

* Make orgMember optional to fix type issues

* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade

* Removed the logger import

* Address Matt’s comments

* Addressing more of Matt’s comments

* Handle sending an execution after a websocket connection closes

* Remove auth from the env attributes to prevent obfuscation
2024-02-20 12:16:41 +00:00
Eric Allam 9cb39bf7d7 Added Redis example env vars 2023-11-29 10:34:54 +00:00
Abdou Belghalem 10e935b864 fix(TRI-1444): Allow only logins from WHITELISTED_EMAILS regex (fixes #685) (#721) 2023-11-06 09:06:54 +00:00
nicktrn 15f17d27e0 Going exponential with Linear (#478)
* Unleash GPT magic

* Clean up after GPT

* All the hooks

* Provisional integration catalog entry

* Sample webhook jobs

* Attachments with alpha warnings

* Remove some verbose logs

* Fix IP restrictions

* Remove tunnel

* Revert "Remove tunnel"

This reverts commit c5b69ce6524e3b40c26b66cdc56e087b8576b8c6.

* Resolve event name clashes

* Remove circular dependency

* Use correct payload uuid

* Schema fixes

* Fix webhook event name

* Start to Linearify catalog entry

* Remove todo

* More catalog updates

* Make OAuth work

* Rename webhook helper

* Schema juggling

* More discrimination

* Add Issue SLA event

* Simplify triggers

* Handle rate limits

* Fix Project schema

* Payload examples

* Improve event props

* Remove redundant source metadata

* One type to rule them all

* Recursive WithoutFunctions type

* Linear output serializer

* Some tasks

* Update catalog entry

* Dynamic usage sample

* Bump version

* Remove tunnel

* More tasks

* Add optional skipRetrying on runTask errors

* Fail fast on user errors

* Entity getter tasks

* Another couple of tasks

* Token to apiKey

* Sort tasks

* Add filtered issue SLA triggers

* Add docs

* Type fixes

* Job catalog examples

* Serialization helper docs

* Add changeset

* Refactor webhooks

* Enhance properties

* Pagination helper and docs

* Clean up imports

* Change misc catalog job

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-09-21 15:14:54 -07:00
nicktrn 603ffbc426 docs: add supabase guide (#297)
* docs: add supabase guide

* feat: add support for external connection poolers

* chore: use .env symlink in database package

* chore: remove database .env.example

* docs: finish supabase pooling section

* docs: fix small typo

* docs: replace prisma with app wording

* docs: fix up supabase pooling section

* fix: deleted too much in contributing docs

* docs: change pooling heading

* docs: reduce linux shaming

* docs: include direct connection url in fly.io
2023-08-11 15:51:54 +01:00
Eric Allam 36ec16babe feat: encrypt values in SecretStore using an encryption key (#240)
* feat: encrypt values in SecretStore using an encryption key and `aes-256-gcm`

* Stopped using apiCors in the connectionId API endpoint
2023-08-01 10:11:47 +01:00
Eric Allam 0b45338300 Remove the bad RESEND_API_KEY from .env.example 2023-07-17 13:00:38 +01:00
James Ritchie 00845b43b4 Renamed the .example env file and added instructions for getting your Resend API 2023-06-26 14:31:27 +01:00