docs/promote-deployment
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 nicktrn
|
94fb532f5e |
Add RE2 supervisor (#1769)
* add new core server only export * exit after run if warm starts disabled * add supervisor |
||
|
Eric Allam
|
e89fb92532 |
re2: dev runs work without worker groups, fixed some type issues (#1756)
* In dev, the worker group is optional when triggering tasks (the master queue is defined by the environment). Also deprecated the TaskEvent.isDebug column and using TaskEventKind.LOG instead for debug events * Fixed a couple of type issues * More type fixes |
||
|
nicktrn
|
e97704d904 |
Run Engine 2.0 (WIP) (#1575)
* bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
nicktrn
|
7186b1e428 |
Add support for deferred checkpoints (#1721)
* add ability to delay checkpoints * optional checkpoint delays for dependency waits * prevent checkpoint creation for resumed batches * unpause after checkpoint rejected by platform * prevent checkpoint creation for resumed task waits * fix checkpoint cancellation |
||
|
nicktrn
|
36fea7d645 |
Add support for manual checkpoints (#1709)
* don't crash run on lazy attempt errors and count on retry * remove abort controller after checkpoint in all cases * duration checkpoints * manual checkpoints * add busybox to coordinator image * add async toggle to process in background * remove early returns * add changeset for manual checkpoint schema |
||
|
nicktrn
|
90593adea5 |
Fix several restore and resume bugs (#1418)
* try to correct resume messages with missing checkpoint * prevent creating checkpoints for outdated task waits * prevent creating checkpoints for outdated batch waits * use heartbeats to check for and clean up any leftover containers * lint * improve exec logging * improve resume attempt logs * fix for resuming parents of canceled child runs * separate SIGTERM from maybe OOM errors * pretty errors can have magic dashboard links * prevent uncancellable checkpoints * simplify task run error code enum export * grab the last, not the first child run * Revert "prevent creating checkpoints for outdated batch waits" This reverts commit |
||
|
nicktrn
|
bf15d8aff1 |
Improve coordinator logs and extend structured logger (#1389)
* docker provider shouldn't force simulate by default * structured logs for all things coordinator * attributes for handler loggers * add changeset * fix tmp cleaner types * remove unused simple logger import * fields from args should override existing fields * improve socket logs * change log message position for easier scanning * more logging improvements, more metadata |
||
|
nicktrn
|
8da495ac00 |
Improve checkpoint reliability and cleanup of temp files (#1367)
* improve cleanup reliability * improve logging * bye-bye execa * fix for trailing newlines * prettier errors * trim args and log output by default * fix archive cleanup * prevent potential memleak * more cleanup debug logs * ignore abort during cleanup * rename checkpoint dir env var and move to helper * add global never throw override * add tmp cleaner * also clean up checkpoint dir by default * split by any whitespace, not just tabs * only create tmp cleaner if paths to clean |
||
|
nicktrn
|
d8006e15ac |
Prevent abort signals from causing uncaught exceptions (#1320)
* never abort the same controller twice * prevent uncaught exception when aborting pipe * abort signal assertions and more logging * never abort running pipe |
||
|
Eric Allam
|
f9ec66c562 |
v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions * remove v2 only packages, will be moved to a dedicated repo * remove more v2 code and run pnpm install * use the npm yalt package in the webapp * convert @trigger.dev/core to tshy * Switch from jest to vitest in @trigger.dev/core * Fixed core test * move core-backend code into core subpath export * convert @trigger.dev/sdk to tshy * Removed hono * move core-apps to core/v3/apps, remove core-apps, start converting cli-v3 * Fix up some of the commands * cli now building and loadable * using package-json-from-dist to get package version now in core and cli * dev command WIP * cleaned up some repetition and structure of the entry point stuff * bringing back the background worker stuff * Indexing of the v3 catalog * getting closer to executing dev runs... * centralize dev logging using event emitter * Move indexing to it’s own entry point, simplify code * dev runs working * Get instrumentation to work with openai * debugging achieved internally * provide worker files as part of the worker creation on the server * support for cjs and esm javascript * Fixed timeout * worker manifest now has the config path * auto-upgrade config to non-deprecated alternatives * Adding package preview release * deployment WIP * improve the syncEnvVars output and adapt resolveEnvVars * WIP bun runtime * WIP bun support * seed tasks with the machine preset if listed in the config * deploy run executions WIP, extracted TaskRunProcess into 1 place * deployed tasks running and executing 🎉 * support for waits and better flushing & process cleanup * Fixed the heartbeating * Better warning messages * Improve and unify the indexing between dev and deploy * Support for external deps that need node-gyp to build * build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files * Add back in the main/types/module to sdk * dev no longer is Ink/React, grace period for disconnections in dev * Fix the changeset config * More changeset fixes * Remove config packages * More changeset fixes * Fixed typescript issues (needed to revert back to zod 3.22.3 * Fix pr_checks workflow * Remove the prepare script * Fixed tests and package versions * Remove cli test script * Remove packages from tailwind watch paths * Add repo to public packages * Just commit the generated files and do the building at dev time * Try and get pkg.pr.new working * Try again * Fix emitDecoratorMetadata importing named export from typescript * config file backwards compat with export const config * Fixed issue where import errors weren’t coming through * p-retry is a prod dep * typescript needs to be a prod dependency for emitDecoratorMetadata * Add better debug logging to help track down import-in-the-middle bug * An external is only considered resolvable if it resolves to the same path as the collected external * Fix runtime checks to allow >=18.20 * Move extensions to a new build package * Fixed building packages in dockerfile * Remove the e2e test from publish workflow for now * Don’t treat pkg.pr.new versions has needing upgrading * making sure config handleError works, and discovered path aliases don’t work in config files * Strip empty string env vars so they accidentally override real values * Couple of things * Update version to use preview instead of beta * Hopefully fix re-attempts with >30s delay * Match socket emit messages to current latest in main * Initial guide * Go back to beta * Go back to the preview, and update guide to use pr preview tags * Go back to beta --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
nicktrn
|
16cb1c46ac | Fix restored container names | ||
|
nicktrn
|
5cf90da722 |
v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups * write crash errors to attempt.error * make restored pod names unique per checkpoint * use last eight characters of checkpoint id instead * add more chaos monkey env vars * Ignore unfreezable states * prevent excessive queue config parsing errors * handle dependency resume edge case * better entry point logging * ignore checkpoint cancellation timeouts * add missing idempotency keys to wait for dep replays * remove checkpoints between attempts * fix retry container names on kubernetes * add changeset * fix types * bring back internal duration timers |
||
|
nicktrn
|
14c2bdf89b |
v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold * rename checkpoint threshold env var * log task monitor ignores * crash runs with unbounded attempts * fix retry check in shared queue consumer * add missing stop for env var sync spinner * prod entry point refactor * missing awaits * more verbose prod flush and exit logs * reduce checkpoint support logs * heartbeat while checkpointing between retries * dynamic coordinator config * measure lazy attempt creation time in prod * simplify delay threshold * heartbeat clarifications * crash run if it doesn't reach checkpointable state * require dynamic config threshold * fix retry prep, await previous worker kill * unify wait mechanics * fix prod worker without tasks error * ensure worker is ready to be checkpointed for dependency waits * improve worker attempt creation logging * prevent crashes caused by failed socket schema parsing * fix dynamic imports in v3 catalog * clarify attempt retry mechanics * move backoff helper to core-apps * remove core-apps barrel file * add backoff execute with callback * deprecate non-lazy attempt messages * update socket.io-client to v4.7.5 * fix socket.io types for emits with timeout * retry all the things * remove todo * fix retry restores * improve index failure logs * retry incomplete dependency waits * fix checkpoint in-progress detection * prevent losing messages during reconnect * checkpoint when greater or equal to threshold * improve handling of duration wait edge cases * add ready for lazy attempt replay * retry attempt completion * allow failing runs with unfriendly run id * fix min max jitter * cancel checkpoints on run failure * improve attempt creation errors * prevent crashing run on failed cleanup * handle at-least-once execute lazy attempt delivery * log exit code on prepare for retry * fix timeout promise * mark some things * chaos monkey superpowers * refactor checkpointer * set chaos monkey defaults * less chaos * fix backoff * handle uncaught entry point exceptions * only replay rpcs on true reconnects * allow resume unless final run status * add changeset * small fixes |