Commit Graph

23 Commits

Author SHA1 Message Date
Matt Aitken 9a7ad926aa ElectricSQL run page (hidden page for now) (#1297)
* WIP on using react-window-splitter

* WIP with new resizable panels and SSR

* Use the cookie package

* Resizable storybook page

* Increase indexing memory limit

* Fixed v2 usage meter displaying when on paid plan (#1255)

* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects

* Concurrency page and more accurate tracking (#1252)

* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”

* v3: fix unfreezable state crashes for runs with multiple waits (#1253)

* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers

* Added more logging to TaskRunConcurrencyTracker and some more try/catches

* Call subscriber.messageDequeued in dequeueMessageInSharedQueue

* Added messageReplaced to concurrency tracking (when freezing)

* Added depenenciesToBundle guide to bundle all packages

* Include the old message data when replacing, so we get the projectId etc.

* Fix restored container names

* Fix for schedule page not scrolling

* Added a description panel to the Concurrency admin page

* chore: Update version for release (beta) (#1256)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.0.0-beta.53

* Added a note to use batchTrigger() instead of trigger()

* The latest react-window-splitter fixes the ESM issues

* Set sensible defaults for the run page

* Deployments page

* Test page

* Schedules page

* Latest version of react-window-splitter (0.2.5)

* Updated to the latest version: react-window-splitter

* Callout if runs don’t start right away now has some top margin

* Small padding fix

* styled the handle focus state

* Added isStaticAtRest prop to resizable panel

* Updated resizable storybook

* Inline code blocks behave nicer when text wraps

* Added ElectricSQL to docker-compose, available on 3060

* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend

* Use the new util

* More restructuring ready to use the trace summary from the frontend

* Using ElectricSQL for the run page data

* Min size for resizable panel on test page

* Don’t load the trace in the RunPresenter anymore

* Fix for the resizable panels on the run page

* Added overflow hidden to the panel group

* min size for the test page left hand panel

* Updated to latest window-splitter version

* Removed unused const

* One fix for client-server mismatch

* Slight improvement in the loading state

* Restructured the page so the loading is better

* Improvement to the loading states

* Improved the loading behaviour with the inspector

* WIP on auth, having problems with it

* Upgrade Remix to 2.9.1 (same as PR #1096)

* Switched structure around again so we only call the useTrace hook from the client

* Added auth to the sync

* Overscan more rows in the tree view

* Fix for TS error

* Remove duplicate import

* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"

This reverts commit e63ee9e924.

* save cookie only when id is used

* Deployment table now scrolls

* removed imports

* A lot of changes to make the inspector live too… WIP

* More major overhauls to get the synced version of the run page working…

* If a span is completed show that

* Set the debounce much lower for selecting the span view

* Load the details run inspector data on demand

* Delete the SpanPresenter

* Use the async payload because it deals with superjson

* Fixed weird merge conflict

* Share some inspector timeline components

* A couple of layout tweaks

* Improved the run inspector loading states

* Fix for paragaph errors

* Fix for focusing on a span

* Undefined typre for useSyncedShape

* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples

* Updated @electric-sql/react package to the latest

* Fix the timeline duration stretching

* Added some better error handling for the electric sync

* More logging

* Better error when there are bad responses

* Turn off resizable snapshots, there’s a bug

* Added getSpan back

* Added SpanPresenter back

* Updated to the new Electric hooks package

* Made a copy so we have the old run page and the new electric one

* Put the main eventRepository back for now

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-13 11:45:24 +01:00
Matt Aitken 09e6f58e84 Revert "ElectricSQL run page and react-window-splitter (#1277)"
This reverts commit 5231381717.

# Conflicts:
#	pnpm-lock.yaml
2024-09-05 14:13:16 +01:00
Matt Aitken 5231381717 ElectricSQL run page and react-window-splitter (#1277)
* WIP on using react-window-splitter

* WIP with new resizable panels and SSR

* Use the cookie package

* Resizable storybook page

* Increase indexing memory limit

* Fixed v2 usage meter displaying when on paid plan (#1255)

* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects

* Concurrency page and more accurate tracking (#1252)

* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”

* v3: fix unfreezable state crashes for runs with multiple waits (#1253)

* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers

* Added more logging to TaskRunConcurrencyTracker and some more try/catches

* Call subscriber.messageDequeued in dequeueMessageInSharedQueue

* Added messageReplaced to concurrency tracking (when freezing)

* Added depenenciesToBundle guide to bundle all packages

* Include the old message data when replacing, so we get the projectId etc.

* Fix restored container names

* Fix for schedule page not scrolling

* Added a description panel to the Concurrency admin page

* chore: Update version for release (beta) (#1256)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.0.0-beta.53

* Added a note to use batchTrigger() instead of trigger()

* The latest react-window-splitter fixes the ESM issues

* Set sensible defaults for the run page

* Deployments page

* Test page

* Schedules page

* Latest version of react-window-splitter (0.2.5)

* Updated to the latest version: react-window-splitter

* Callout if runs don’t start right away now has some top margin

* Small padding fix

* styled the handle focus state

* Added isStaticAtRest prop to resizable panel

* Updated resizable storybook

* Inline code blocks behave nicer when text wraps

* Added ElectricSQL to docker-compose, available on 3060

* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend

* Use the new util

* More restructuring ready to use the trace summary from the frontend

* Using ElectricSQL for the run page data

* Min size for resizable panel on test page

* Don’t load the trace in the RunPresenter anymore

* Fix for the resizable panels on the run page

* Added overflow hidden to the panel group

* min size for the test page left hand panel

* Updated to latest window-splitter version

* Removed unused const

* One fix for client-server mismatch

* Slight improvement in the loading state

* Restructured the page so the loading is better

* Improvement to the loading states

* Improved the loading behaviour with the inspector

* WIP on auth, having problems with it

* Upgrade Remix to 2.9.1 (same as PR #1096)

* Switched structure around again so we only call the useTrace hook from the client

* Added auth to the sync

* Overscan more rows in the tree view

* Fix for TS error

* Remove duplicate import

* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"

This reverts commit e63ee9e924.

* save cookie only when id is used

* Deployment table now scrolls

* removed imports

* A lot of changes to make the inspector live too… WIP

* More major overhauls to get the synced version of the run page working…

* If a span is completed show that

* Set the debounce much lower for selecting the span view

* Load the details run inspector data on demand

* Delete the SpanPresenter

* Use the async payload because it deals with superjson

* Fixed weird merge conflict

* Share some inspector timeline components

* A couple of layout tweaks

* Improved the run inspector loading states

* Fix for paragaph errors

* Fix for focusing on a span

* Undefined typre for useSyncedShape

* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples

* Updated @electric-sql/react package to the latest

* Fix the timeline duration stretching

* Added some better error handling for the electric sync

* More logging

* Better error when there are bad responses

* Turn off resizable snapshots, there’s a bug

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-04 17:55:36 +01:00
Hamza 83a35226a7 removed CLOUD_ENV from .env.example (#1230) 2024-07-24 12:02:03 +01:00
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
nicktrn 35dbaedf69 v3: self-hosting (#1147)
* add amin email regex env var

* fix displayed init command for self-hosted setups

* shared env var to disable telemetry in cli and webapp

* pin sdk version during init

* if specified, add api url to dev command shown after init

* improve checkpoint support detection

* control forced checkpoint simulation via env var

* add public init to providers

* better checkpoint support check for coordinator

* add docker to coordinator image

* update docker provider containerfile

* bump remaining containers to node 20

* add infra image build to default publish workflow

* lockfile

* remove concurrency group from infra workflow

* add docker provider to build matrix

* fix var subst

* checkpoint test is docker specific

* enable v3 projects by default on self-hosted instances

* fix v3 setup command again

* add default posthog key

* self-hosting docs

* add latest tags to versioned infra and webapp builds

* some checkpoint errors should skip retrying

* add changeset

* shorten paragraph

* some docs updates

* update tunnelling section

* add registry setup section

* use correct cli push flag

* add checkout to v3 branch

* update the worker machine setup steps

* fix infra build

* small docs update

* remove unused feature function

* Revert "remove unused feature function"

This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.

* fix self-hosted v3 feature gate

* add note about missing arm support

* simplify helper script syntax
2024-06-10 14:13:04 +01:00
Eric Allam c9ebe7f114 Update contributing guide for local v3 development 2024-05-20 15:43:31 +01:00
Matt Aitken 80997bcd6d .env.example Redis env vars are uncommented 2024-05-12 18:02:42 +01:00
Eric Allam bc020a3ffe Get the internal telemetry ready for test/prod 2024-03-28 18:12:01 +00:00
Eric Allam 719c0a0b94 v3: better clock management for spans/logs in CRIU envs (#980)
* durable clock WIP

* Fixed incorrect span timings around checkpoints by implementing a precise wall clock that resets after restores

* Add changeset
2024-03-28 14:20:44 +00:00
Eric Allam 215b60c742 v3: superjson output support (#971)
* superjson output support

* Better support for superjson in the task events, limiting output attributes

* Offload large outputs to object store (r2)

* Finishing up the offloading of large outputs/payloads to an object store
2024-03-27 11:57:22 +00:00
Eric Allam ba0ffacb15 Set the OTEL_EXPORTER_OTLP_ENDPOINT env var on dev workers 2024-03-18 20:16:25 +00:00
Eric Allam 67cac97b26 Add DEPLOY_REGISTRY_HOST to allow configuring the deploy registry via the server 2024-03-18 15:56:28 +00:00
Eric Allam c385db63be v3: cli deploy command (#931)
* WIP proxy/deploy

* WIP, registry proxy in express working

* A couple of notes, preparing for indexing

* Move the changes to prod-worker into the new file

* Deploy command working with indexing and runs (docker provider only for now)

* Removed ts-expect-error directive

* Fixed build command
2024-03-08 10:49:42 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam 07f7c65054 v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev

* Convert MarQS to using lua and dealing with concurrency

* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)

* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects

* Start adding internal telemetry support for the server

* Get env vars to work in dev and implement prisma tracing in webapp

* Cleanup telemetry and implement it in the consumer

* Implement dequeuing a message from a parent shared queue

* Implement a custom logger exporter instead of using console log exporter

* Use node instead of shell for generating protocol buffer code

* Propogate trace context into debug logs, and allow turning off logger exporter through env vars

* Switch to using baselime for internal otel data

* Make orgMember optional to fix type issues

* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade

* Removed the logger import

* Address Matt’s comments

* Addressing more of Matt’s comments

* Handle sending an execution after a websocket connection closes

* Remove auth from the env attributes to prevent obfuscation
2024-02-20 12:16:41 +00:00
Eric Allam 9cb39bf7d7 Added Redis example env vars 2023-11-29 10:34:54 +00:00
Abdou Belghalem 10e935b864 fix(TRI-1444): Allow only logins from WHITELISTED_EMAILS regex (fixes #685) (#721) 2023-11-06 09:06:54 +00:00
nicktrn 15f17d27e0 Going exponential with Linear (#478)
* Unleash GPT magic

* Clean up after GPT

* All the hooks

* Provisional integration catalog entry

* Sample webhook jobs

* Attachments with alpha warnings

* Remove some verbose logs

* Fix IP restrictions

* Remove tunnel

* Revert "Remove tunnel"

This reverts commit c5b69ce6524e3b40c26b66cdc56e087b8576b8c6.

* Resolve event name clashes

* Remove circular dependency

* Use correct payload uuid

* Schema fixes

* Fix webhook event name

* Start to Linearify catalog entry

* Remove todo

* More catalog updates

* Make OAuth work

* Rename webhook helper

* Schema juggling

* More discrimination

* Add Issue SLA event

* Simplify triggers

* Handle rate limits

* Fix Project schema

* Payload examples

* Improve event props

* Remove redundant source metadata

* One type to rule them all

* Recursive WithoutFunctions type

* Linear output serializer

* Some tasks

* Update catalog entry

* Dynamic usage sample

* Bump version

* Remove tunnel

* More tasks

* Add optional skipRetrying on runTask errors

* Fail fast on user errors

* Entity getter tasks

* Another couple of tasks

* Token to apiKey

* Sort tasks

* Add filtered issue SLA triggers

* Add docs

* Type fixes

* Job catalog examples

* Serialization helper docs

* Add changeset

* Refactor webhooks

* Enhance properties

* Pagination helper and docs

* Clean up imports

* Change misc catalog job

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-09-21 15:14:54 -07:00
nicktrn 603ffbc426 docs: add supabase guide (#297)
* docs: add supabase guide

* feat: add support for external connection poolers

* chore: use .env symlink in database package

* chore: remove database .env.example

* docs: finish supabase pooling section

* docs: fix small typo

* docs: replace prisma with app wording

* docs: fix up supabase pooling section

* fix: deleted too much in contributing docs

* docs: change pooling heading

* docs: reduce linux shaming

* docs: include direct connection url in fly.io
2023-08-11 15:51:54 +01:00
Eric Allam 36ec16babe feat: encrypt values in SecretStore using an encryption key (#240)
* feat: encrypt values in SecretStore using an encryption key and `aes-256-gcm`

* Stopped using apiCors in the connectionId API endpoint
2023-08-01 10:11:47 +01:00
Eric Allam 0b45338300 Remove the bad RESEND_API_KEY from .env.example 2023-07-17 13:00:38 +01:00
James Ritchie 00845b43b4 Renamed the .example env file and added instructions for getting your Resend API 2023-06-26 14:31:27 +01:00