Commit Graph

3309 Commits

Author SHA1 Message Date
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
Matt Aitken 7976d924fb Marqs rebalancing is based on the env var again 2024-07-03 09:54:50 +01:00
nicktrn 1e667ec28f v3: cluster uptime heartbeat (#1194)
* add basic uptime heartbeat

* add more heartbeat metrics
2024-07-02 12:33:12 +01:00
nicktrn cd5d2ae92b Override completion on OOM crashes 2024-07-01 13:22:35 +01:00
Eric Allam 0e77e7ef7d v3: Delayed runs and run ttl expiration (#1193)
* v3: Trigger delayed runs and reschedule them

* Create a `@trigger.dev/core/v3/schemas` export

* fixed the `@trigger.dev/core/v3/schemas` export

* Small docs tweak

* Add ttl option when triggering tasks, expire runs after ttl

Dev runs expire in 10m by default
2024-07-01 12:01:35 +01:00
Kai 76a5c6204f fix: prevent filter tooltip from opening on autofocus (#1185)
* Prevent tooltip from opening on autofocus

* Remove autofocus

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-07-01 10:50:07 +01:00
nicktrn b171fde483 Fix index failure completion override 2024-06-29 19:23:32 +01:00
nicktrn 5ae3da6b4e v3: various small fixes (#1192)
* Await file watcher cleanup in dev

* Fix artifact detection logs

* Fix next runs table when schedule disabled

* Improve OOM error messages

* Add test link to completed deployment message

* Fix OOM detection, again

* Add changeset
2024-06-29 16:48:01 +01:00
nicktrn d0d3a64bd6 v3: misc CLI improvements (#1173)
* prevent downgrades during update check

* detect bun and use npm instead

* detect missing tsconfig during init and print helpful error

* add changeset

* add links to dev worker started message

* allow users to add custom pkg manager args during init

* update changeset

* fix links in unsupported terminals

* deprecate terminalLink

* update changeset
2024-06-28 15:51:17 +01:00
nicktrn 568da01785 v3: improve non-zero exit errors (#1179)
* docker provider will optionally enforce machine presets

* update task monitor oom message

* add oom task to v3 catalog

* improve handling of non-zero exit errors

* add changeset
2024-06-28 15:50:04 +01:00
Eric Allam 4ca8887972 v3: fix issue where in progress events weren’t being shown 2024-06-26 16:15:44 +01:00
Eric Allam 34ca7667d3 Various perf improvements to prevent event loop lag (#1186)
* WIP

* Handle tasks that have failed but are being auto yielded

* Limit trace view to 25k event records, add a download run logs button

Also added two new indexes to TaskEvent:

```
/// Used on eventRepository.getTraceSummary()
  @@index([traceId, startTime])
  // Used for getting all logs for a run
  @@index([runId])
```

* perf improvements on eventRepository.getSpan()

* v2: Add a 5 minute timeout for run execution requests in dev

* v3: Include presigned urls for downloading large payloads and outputs when using runs.retrieve

* v3: better handle large task payloads and outputs

* Change to 512KB

* v2: paginate trigger schedules endpoint

* v3: add 3MB limit on batch and single payloads

* Update task payload and output limits
2024-06-26 15:22:35 +01:00
Eric Allam da6a66efff v3: fix task queues with concurrency = 0 getting cleared on task trigger 2024-06-24 16:37:30 +01:00
Eric Allam 7c36a1a4b0 v3: Adding SDK functions for triggering tasks in a typesafe way (#1177)
* v3: Adding SDK functions for triggering tasks in a typesafe way, without importing task file

* Add type usages
2024-06-24 12:37:34 +01:00
Eric Allam 65f960e883 Move the max live reload setting into an env var instead of hardcoding it 2024-06-21 12:55:33 +01:00
Eric Allam ccbeff47e6 Improve trace view performance 2024-06-21 12:46:46 +01:00
Matt Aitken 7c8f2df105 After 500 logs we stop live reloading on the run page 2024-06-21 11:26:02 +01:00
Eric Allam fd44dabfe0 Disable the event loop monitor 2024-06-20 23:56:29 +01:00
Eric Allam 5daed3f69d Turn off rebalancing 2024-06-20 23:42:54 +01:00
Eric Allam 596bf78e55 Add an event loop monitor 2024-06-20 23:26:38 +01:00
Eric Allam 6ca66b76f4 Remove unnecessary transaction from the runtime environment session handling 2024-06-20 22:47:12 +01:00
nicktrn 29ef0395ce Only import types from db in task status 2024-06-20 15:25:18 +01:00
nicktrn 55d1f8c677 v3: fix dependency checkpoint race (#1171)
* consolidate task statuses and utils

* check dependency completion when creating checkpoints

* add changeset
2024-06-20 10:10:39 +01:00
Eric Allam 233316f7e8 TaskRun.startedAt will now be lazily migrated 2024-06-18 15:49:14 +01:00
Matt Aitken b45ca4e146 If the timezone has changed then we need to reschedule the next scheduled run 2024-06-18 14:49:16 +01:00
Eric Allam 85a543d8ec v3: usage tracking (#1163)
* Starting to measure wall time and cpu time in the workers, and reporting that via otel and to completed task run attempts

* Move usage tracking outside of the executor

* WIP prod usage tracking

* WIP

* WIP custom fetch to openmeter

* Create a usage client

* WIP

* WIP

* Implement new machine preset stuff and send usage reports to OpenMeter from webapp

* WIP

* Expose usage info to the client

* Add usage and cost to TaskEvent

* Add ability to globally configure the task machine preset

* Report start run usage

* Change the machine docs to use presets

* setExpirationTime to 24h

* Removed logs

* Update machines.mdx

* Removed console.logs

* Handle revalidating JWT tokens

* Couple tweaks

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-06-18 09:40:23 +01:00
Ryan Lee 10ceb85a92 fix: cloudfront timeouts not being caught (#1162) 2024-06-14 15:11:39 +01:00
Matt Aitken c405ae7117 Schedule limits and timezone support (#1165)
* Added maximumScheduleInstancesLimit column to Org, default to 20

* Docs on the schedule limits and improved soft-limit communication

* Added limit info to the schedules list page

* Created a task that creates schedules, useful for testing

* Make deduplicationKey required when creating/updating a schedule using the SDK

* New schedule button shows an alert if you’re over the limit

* Added timezone to the form and db

* WIP on the timezone dropdown for the create/edit schedule form

* Use the new filter search for timezones

* Made the timezone dropdown faster by fixing the virtualization

* The preview table is working and added a nice message about daylight savings

* Created a page where you can view the full list of timezones

The URL is included in the error message if you send an invalid time using the SDK

* Creating tasks with the timezone

* Added timezone support the the scheduler and the schedules list

* Added timezone support to more of the schedules UI

* The timezone comes through to scheduled runs with nice JSDocs

* Allow setting the timezone from the SDK

* Always have a timezone on a schedule

* Updated jsdocs

* Updated catalog example

* Changed the column to be a string, not null. Added the timezone across the SDK

* API endpoint for getting the timezones

* Added an SDK function to get the list of timezones

* Added timezones to the docs

* Changeset: Added timezone support to schedules

* Added support for testing timezone

* Tidied up imports

* Imports

* Imports

* Update limits.mdx

* Fixed a couple type issues and use the already exported zodfetch

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-06-14 13:31:27 +01:00
nicktrn 3687fcb61e Make pod cleaner interval configurable 2024-06-14 12:12:42 +01:00
nicktrn e08b4569e5 Improve checkpoint restore logging 2024-06-13 10:18:41 +01:00
Eric Allam c9e97d6b78 Add debug logging when we skip importing a resource span 2024-06-12 17:27:12 +01:00
nicktrn 1d47cab69f Ensure rollback triggers execution of tasks waiting for deploy 2024-06-12 11:51:51 +01:00
nicktrn 68d32429b6 v3: checkpoint failover and misc fixes (#1157)
* configurable checkpoint registry namespace

* add missing task create await

* remove unused messages

* changeset

* update self-hosting docs

* capture and display stderr for failed deploys

* add missing lockfile changes

* stderr changeset

* fix cli stderr message

* update error logs label
2024-06-11 14:06:50 +01:00
Eric Allam 36ac79ac66 Make sure users cannot revoke invites for other orgs 2024-06-11 11:09:52 +01:00
nicktrn a5d8e453a5 v3: deploy rollbacks (#1154)
* add deployment rollbacks

* update test presenter

* update task list presenter

* fix test page for dev env

* read replica for the test presenter
2024-06-11 10:09:29 +01:00
Eric Allam eae294a332 Add back in the v2 timeout task thing 2024-06-10 16:06:21 +01:00
Eric Allam 465cd0335c v2: No longer eagerly timeout runs when no tasks are created 2024-06-10 14:54:22 +01:00
nicktrn 35dbaedf69 v3: self-hosting (#1147)
* add amin email regex env var

* fix displayed init command for self-hosted setups

* shared env var to disable telemetry in cli and webapp

* pin sdk version during init

* if specified, add api url to dev command shown after init

* improve checkpoint support detection

* control forced checkpoint simulation via env var

* add public init to providers

* better checkpoint support check for coordinator

* add docker to coordinator image

* update docker provider containerfile

* bump remaining containers to node 20

* add infra image build to default publish workflow

* lockfile

* remove concurrency group from infra workflow

* add docker provider to build matrix

* fix var subst

* checkpoint test is docker specific

* enable v3 projects by default on self-hosted instances

* fix v3 setup command again

* add default posthog key

* self-hosting docs

* add latest tags to versioned infra and webapp builds

* some checkpoint errors should skip retrying

* add changeset

* shorten paragraph

* some docs updates

* update tunnelling section

* add registry setup section

* use correct cli push flag

* add checkout to v3 branch

* update the worker machine setup steps

* fix infra build

* small docs update

* remove unused feature function

* Revert "remove unused feature function"

This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.

* fix self-hosted v3 feature gate

* add note about missing arm support

* simplify helper script syntax
2024-06-10 14:13:04 +01:00
Eric Allam fb52b9efea Remove redundant log (you’re welcome baselime) 2024-06-07 20:41:34 +01:00
Matt Aitken 0896b9fffc Use the read replica more (#1152)
* Switch to read replica: getEvent API endpoint

* Switch to read replica: v2 run list presenter

* Switch to read replica: Job presenter

* Switch to read replica: Job list presenter

* Switch to read replica: billing client

* Switch to read replica: OrgUsagePresenter

* Switch to read replica: OrgBillingPlanPresenter

* Switch to read replica: ScheduleListPresenter

* Switch to read replica: EventRepository taskEvent.findMany
2024-06-07 15:47:54 +01:00
Matt Aitken 3a2dd983c5 Fix for sendEvent same id causing multiple runs (#1151)
* Proof of concept

* When ingesting events, if it’s already been delivered then don’t continue

* DeliverEvent: throw AlreadyDeliveredError and don’t retry if that’s thrown

* Test for duplicate event ids

* Return the original event so sendEvent doesn’t fail, don’t enqueue

* Add AlreadyDeliveredError to the logged out message

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-06-07 15:10:53 +01:00
Eric Allam a627ca67d1 Use a different logger key for the marqs version 2024-06-07 14:19:05 +01:00
Eric Allam afc180aa70 marqs: Concurrency monitor that runs periodically and vacuums completed runs (#1150) 2024-06-07 13:36:02 +01:00
Eric Allam 393af1b7c5 Add ability to enable v2 marqs requeuing (and remove deprecated marqs visibility requeuing) 2024-06-06 17:14:15 +01:00
Eric Allam 8e8ed4a3bf Remove queueName option from ZodWorker 2024-06-06 14:49:45 +01:00
Eric Allam 8fc8f57b39 v2: MarQS powered job executions (#1149)
* WIP

* Allow marqsv2 and v2 graphile to run in parallel

* Fix missing GraphileLogger import

* Fixed heartbeat after rebase

* Replace postgres based run counters with redis ones with a backfill

* Add back in the graphile logger

* Remove duplicate visibility timeout calls

* Clamp simple weighted strategy to max of 5
2024-06-06 13:56:27 +01:00
Eric Allam 665f7c9756 v3: runs.retrieve fixed payload value 2024-06-06 13:21:59 +01:00
Matt Aitken 928a632e23 The IngestSendEvent rate limiter is now a singleton 2024-06-06 09:18:27 +01:00
Eric Allam 74db2de1bc Use graphile strategy 0 (no named queues) and remove all named queues 2024-06-05 19:08:00 +01:00
Eric Allam 93acca6c3c maxPoolSize should be +1 from concurrency 2024-06-05 17:50:15 +01:00