fix/core-api-request-timeout
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 nicktrn
|
3f97cf4a6b |
Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles * fix retry spans * only download debug logs if admin * add nypm changeset * pull out env override logic * use runner env gather helper * handle dev flushing failures gracefully * fix path normalization for init.ts * add logger * add execution heartbeat service * add snapshot poller service * fix poller * add changesets * create socket in constructor * enable strictPropertyInitialization * deprecate dequeue from version * start is not async * dependency injection in prep for tests * add warm start count to all controller logs * add restore count * pull out run execution logic * temp disable pre * add a controller log when starting an execution * refactor execution and squash some bugs * cleanup completed docker containers by default * execution fixes and logging improvements * don't throw afet abort cleanup * poller should use private interval * rename heartbeat service file * rename HeartbeatService to IntervalService * restore old heartbeat service but deprecate it * use the new interval service everywhere * Revert "temp disable pre" This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b. * add changeset * replace all run engine find uniques with find first |
||
|
nicktrn
|
e97704d904 |
Run Engine 2.0 (WIP) (#1575)
* bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Eric Allam
|
26f9a1e478 |
Feat: two phase deployment, version pinning (#1739)
* WIP two-phase deployments * Fix the help text * Rename TRIGGER_WORKER_VERSION to TRIGGER_VERSION * Add changeset * A few naming fixes |
||
|
Eric Allam
|
0e5ec8bfbc |
Only allow a single dev queue consumer to dequeue at once (#1737)
* Only allow a single dev queue consumer to dequeue at once * Expire dev queue consumer connections keys * Add missing return |
||
|
nicktrn
|
e0cdebbe09 | Disable excessive websocket heartbeat logs | ||
|
nicktrn
|
dbda820a71 |
v3: fix websocket timeouts (#1220)
* prevent uncaught websocket exceptions * dev cli sends regular pings to keep connection alive * add changeset * fix for prod message forwarding * add server-side websocket ping * Revert "dev cli sends regular pings to keep connection alive" This reverts commit 9a6a7a21433b9f0665f72be015d0ba5a8d83eba4. * add dev cli websocket debug logs, always print errors * actually start the ping service * update changeset * catch remaining dev cli message handler errors |
||
|
Eric Allam
|
e69ffd314a |
v3: Refactor attempt creation to be worker requested (#1077)
* WIP worker TaskRunAttempt creation
* Handling failing task runs that cannot create an attempt for whatever reason
* Move the visibility queue stuff into a graphile job
* Fixed task runs with unsanitized queue names
* “Borrow” the code from alerts PR to get self hosted deployments working
* Add an admin API endpoint to get info about the shared marqs queue
* Allow admins to view any project metrics
* start adding lazy attempts to prod
* lazy attempt creation for prod workers
* resurrect prod stack traces
* add exception event to failed run spans
* simplify dependency resumes
* fix typecheck
* fix merge
* fresh process for all attempts
* always try sigterm first
* stop heartbeat timeout on non-inplace replace message
* add missing ack on checkpoint creation service failure
* bypass dequeue for retries with running worker
* respect retry delays
* crash runs with invalid run status for execution
* remove debug logs
* fix nack message
* fix version locking
* fresh attempt processes in dev and prod
* improve handling of ipc timeouts
* consider checkpoint failures on cancellation
* add basic chaos monkey to checkpointer
* changeset
* control forced checkpoint simulation via env var
* fix merge
* kill old attempt processes before checkpointing
* detailed perf logging for checkpointing
* add coordinator otlp endpoint example
* improve prod run cancellation
* rename supports lazy attempts migration
* fix graceful exit
* fix retry mechanics
* clear paused state before retry
* remove checkpoint image after push
* crash worker on unrecoverable errors
* refactor unrecoverable error emit
* switch to do hosted busybox image
* increase wait for duration ipc timeout
* add changeset for misc fixes
* fix merge
* fix retry delay span runId
* fix dev retries
* improve prod worker logging
* log checkpoint sizes
* add lazy attempts catalog entries
* Fixed merge issue: use zodFetch, not wrapZodFetch
* Revert "Fixed merge issue: use zodFetch, not wrapZodFetch"
This reverts commit
|
||
|
Eric Allam
|
b68012f81c |
v3: Various fixes for Next.js projects and projects that use v2 and v3 together (#1051)
* Fixes an issue that was treating v2 trigger directories as v3 * Make msw a normal dependency (for now) to fix Module Not Found error in Next.js. * Extract out all the zod* stuff from core so the SDK does not import it * Add a changeset * Fixing typecheck errors in the webapp * Export the Task and TaskOptions types * Extract additional exports from core/v3 that aren’t used in the SDK * Move to our global system from AsyncLocalStorage for the current task context storage * Update the esbuild core bundling plugin for the new core v3 exports * Fix v3 CLI telemetry * Add support for tasks located in subdirectories inside trigger dirs * Remove the env var check during deploy (too many false negatives) |
||
|
Matt Aitken
|
4986bfda2e |
Scheduled tasks (#1036)
* Database schema and migrations for schedules * Added schedules to the side menu * The pagination can optionally hide the page numbers for a compactive mode * Filters for the schedule page * Added triggerSource (“STANDARD”, “SCHEDULED”) to BackgroundWorkerTask * Added the ability to disabled a LinkButton * Started work on the schedule page * Environment buttons * The new schedule form styles * Added a cxouple of extra fields * Allow a checkbox to have a rich label * Added cronstrue package to the webapp * WIP creating tasks using the form * Improved the form styling * Creating schedules is working in the UI * Minor improvements * Basic schedule table is displaying * Creating tasks with triggerSource = scheduled. Refactored how task metadata is stored and accessed to be cleaner * Resource route * Fixes in the form * Added a gap between the environments * WIP on OpenAI generating CRON expressions * AI generated CRON expressions is working * Fix for the CRON field being uneditable after an AI generation * Improvements * Table padding * useThrottle now behaves correctly * Added filtering to the schedules list * Improved the layout and fixed CRON search * Fixed pagination for the schedules list. Just use a regular Prisma query * Page size of 20 * Added links to the schedule rows * Get rid of the Last run column for now * Implement triggered scheduled tasks Also implemented superjson payloads and dev runtime environment “presence” with RuntimeEnvironmentSession * Move CronPattern and CreateSchedule into a common client-accessible file * Latest UI changes * Fix for creating a task schedule with a blank dedup key * Refactor the human to cron stuff into a separate file and use json_object OpenAI response format * Fix for trying to use a hook on the server-side… * A couple of fixes to the new schedule form * WIP on viewing a scheduled run * Make the filters all optional * Use the RunListPresenter from the schedule presenter * Display a table of runs… the wrong runs but still * Runs from the schedule * Deleting schedules from the UI * Tidied imports and fixed name of options object * Disabling a schedule * Editing schedules * Tidied imports * Added icons to the task list, needs some design love * Added a tooltip for CRON pattersn * Show the last run in the schedules table * Some tweaks * Added a placeholder to the CRON AI field * Improved the trigger source icon * Dim out disabled schedules * Scheduled tasks have the correct icon in a run * Added the task source icon to the test task list * Added the date field component to storybook * Style improvements to the date field * Implement Task Schedule API * added a medium sized variant to the date field * Fixed replay run for superjson payload types Also now linking from the replaying run to the original run (using span links) Also added a project metrics prometheus endpoint to detect the state of the queues * WIP on allowing different forms for testing * If you pass a string to prettyPrintPacket which is json or superjson, safeParse it first * Test page, deal json and superjson. Started splitting UI for schedules * WIP on schedule form * Removed the Label from the DateField * WIP on test schedule form * Fix for the runs page showing the wrong message when there are no runs from time filtering * Removed labels from the DateField * Fixes for the form * Test runs are working for schedules * Fix for nextScheduledTimestamps in triggerScheduledTask * Implement idempotency key support and fix issue with cancelled runs causing concurrency usage * Added API documentation for the schedule API * Remove log * Removed console log from runs page * Transform the recent runs test data on the server * Fix for hydration mismatch * Current date as the default for the test form * Recent payloads working * Delete schedule modal * Deal with empty strings from the form * Set the initial value for the scheduled test form * Add option to print console logs in the dev CLI locally (issue #1014) * Export queue from the SDK * Fix for schedules list when you have no schedule tasks * Blank states improved * Make task schedules more generic, to support additional schedule generators in the future * Removed log from maqrs * Removed “v3/schedules” export from the SDK --------- Co-authored-by: Eric Allam <eallam@icloud.com> Co-authored-by: James Ritchie <james@jamesritchie.co.uk> |
||
|
Eric Allam
|
a5aed0d139 |
Fix memory leak by calling punsubscribe instead of unsubscribe
Also adding a /metrics endpoint with a few custom metrics and the built-in prom-client ones |
||
|
Eric Allam
|
478ce006cb |
v3: task run status and canceling runs (#941)
* WIP task run status, revamped resuming task dependencies * Don’t select a span when toggling collapsing * Cancel runs and attempts, in prod and dev |
||
|
Eric Allam
|
ca47e6bdcd |
v3: cli cancel improvements and add back in deprecating background workers (#918)
* Show runs as cancelled when killing the dev CLI * Fixed error when waiting for a task * Deprecated workers will no longer be able to run unlocked task runs * Fixed selecting the latest background worker version in test list * Don’t propogate trace context when triggering an async task Also add batch id to the context and the task events |
||
|
Eric Allam
|
b6517af522 |
v3: Cancel dev runs when the dev CLI exits (#916)
* Cancel dev runs when the dev CLI exits * Clear out in progress attempts after cancelling them |
||
|
Eric Allam
|
1fabcf68fa | Override local env vars with env vars in the server (#914) | ||
|
Eric Allam
|
dd63fe6e9f |
v3: Implemented in-process task retries in dev w/cache support (#905)
* Implemented in-process task retries in dev w/cache support * Improve the stack trace correction |
||
|
Eric Allam
|
07f7c65054 |
v3 queue system (#903)
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev * Convert MarQS to using lua and dealing with concurrency * Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency) * Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects * Start adding internal telemetry support for the server * Get env vars to work in dev and implement prisma tracing in webapp * Cleanup telemetry and implement it in the consumer * Implement dequeuing a message from a parent shared queue * Implement a custom logger exporter instead of using console log exporter * Use node instead of shell for generating protocol buffer code * Propogate trace context into debug logs, and allow turning off logger exporter through env vars * Switch to using baselime for internal otel data * Make orgMember optional to fix type issues * Provide the CLI dev env vars through the CLI, don’t build dotenv into facade * Removed the logger import * Address Matt’s comments * Addressing more of Matt’s comments * Handle sending an execution after a websocket connection closes * Remove auth from the env attributes to prevent obfuscation |