@trigger.dev/python@4.0.0-v4-beta.24
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matt Aitken
|
630e9556b0 |
Bulk actions 2.0 (and switch all run listing to ClickHouse) (#2264)
* useSearchParams has * useSearchParams has * useSearchParams has * Consistent way to get the run filters * Consistent way to get the run filters * Consistent way to get the run filters * Initial work on the new bulk actions * Initial work on the new bulk actions * Initial work on the new bulk actions * WIP actions and filtering * WIP actions and filtering * WIP actions and filtering * Empty filter arrays are set to undefined * Empty filter arrays are set to undefined * Empty filter arrays are set to undefined * WIP prisma schema Removed extra runtimeEnvironmentId * WIP prisma schema Removed extra runtimeEnvironmentId * WIP prisma schema Removed extra runtimeEnvironmentId * Migrations * Migrations * Migrations * BulkActionGroup changed some columns around * BulkActionGroup changed some columns around * BulkActionGroup changed some columns around * New badge variant, removed unused ones * New badge variant, removed unused ones * New badge variant, removed unused ones * Bulk action button * Bulk action button * Bulk action button * Make the next runs page the default now * Make the next runs page the default now * Make the next runs page the default now * Improved the RadioButton style * Improved the RadioButton style * Improved the RadioButton style * Remove the old bulk action bar * Remove the old bulk action bar * Remove the old bulk action bar * More UI progress * More UI progress * More UI progress * Lots of UI changes to the Runs page * Lots of UI changes to the Runs page * Lots of UI changes to the Runs page * Fixed period filter resetting everything * Fixed period filter resetting everything * Fixed period filter resetting everything * Improved the Switch secondary style * Improved the Switch secondary style * Improved the Switch secondary style * Buggy filter fixes * Buggy filter fixes * Buggy filter fixes * Improved the filter display and fixed a bug with search param from object * Improved the filter display and fixed a bug with search param from object * Improved the filter display and fixed a bug with search param from object * Clear button is minimal * Clear button is minimal * Clear button is minimal * Using a presenter now * Using a presenter now * Using a presenter now * Bulk actions are created, but not actually processed (yet) * Bulk actions are created, but not actually processed (yet) * Bulk actions are created, but not actually processed (yet) * Bulk replay/cancel is working * Bulk replay/cancel is working * Bulk replay/cancel is working * Multiple fixes, added bulk column to PG * Multiple fixes, added bulk column to PG * Multiple fixes, added bulk column to PG * Bulk action run filtering working using CH * Bulk action run filtering working using CH * Bulk action run filtering working using CH * Replay setting the bulk id on the runs * Replay setting the bulk id on the runs * Replay setting the bulk id on the runs * Properly cap the time when doing a bulk action * Properly cap the time when doing a bulk action * Properly cap the time when doing a bulk action * If the bulk action isn't recent, add it to the dropdown anyway * If the bulk action isn't recent, add it to the dropdown anyway * If the bulk action isn't recent, add it to the dropdown anyway * Blank version of the bulk actions page * Blank version of the bulk actions page * Blank version of the bulk actions page * Individually selected runs working * Individually selected runs working * Individually selected runs working * Use selected mode if runs are checked * Use selected mode if runs are checked * Use selected mode if runs are checked * Added the modal * Added the modal * Added the modal * Marked the old bulk actions stuff as deprecated * Marked the old bulk actions stuff as deprecated * Marked the old bulk actions stuff as deprecated * Renamed bulk action file * Renamed bulk action file * Renamed bulk action file * Bulk run filter with the name and a default * Bulk run filter with the name and a default * Bulk run filter with the name and a default * WIP on bulk actions page * WIP on bulk actions page * WIP on bulk actions page * Updated panel, added new truncated id component * Updated panel, added new truncated id component * Updated panel, added new truncated id component * Style improvements to the radio buttons * Style improvements to the radio buttons * Style improvements to the radio buttons * Added an option action completion email * Added an option action completion email * Added an option action completion email * Adds a blank state for the bulk actions page * Adds a blank state for the bulk actions page * Adds a blank state for the bulk actions page * Nicer completed email * Nicer completed email * Nicer completed email * Don't open the bulk action panel if there are no runs * Don't open the bulk action panel if there are no runs * Don't open the bulk action panel if there are no runs * Runs blank state and bulk action accordion * Runs blank state and bulk action accordion * Runs blank state and bulk action accordion * Updates secondary/small switch style * Updates secondary/small switch style * Updates secondary/small switch style * Pagination buttons no longer split in twain (WIP) * Pagination buttons no longer split in twain (WIP) * Pagination buttons no longer split in twain (WIP) * Aborting working * Aborting working * Aborting working * Bulk action live reloading * Bulk action live reloading * Bulk action live reloading * ListPagination works correctly in all states * ListPagination works correctly in all states * ListPagination works correctly in all states * Run page, show friendlyId instead of number * Run page, show friendlyId instead of number * Run page, show friendlyId instead of number * Bulk action help open by default if you have none * Bulk action help open by default if you have none * Bulk action help open by default if you have none * Extra status filtering step because of replication delay * Extra status filtering step because of replication delay * Extra status filtering step because of replication delay * Wider bulk action onboarding * Wider bulk action onboarding * Wider bulk action onboarding * More sensible widths on the bulk action side panel * More sensible widths on the bulk action side panel * More sensible widths on the bulk action side panel * Border color tweak to the RadioButton * Border color tweak to the RadioButton * Border color tweak to the RadioButton * Improved the accordion component hover states * Improved the accordion component hover states * Improved the accordion component hover states * Updates the bulk action blank state images to the latest UI * Updates the bulk action blank state images to the latest UI * Updates the bulk action blank state images to the latest UI * Added R and C shortcuts back in * Added R and C shortcuts back in * Added R and C shortcuts back in * Fix for selecting a single run * Fix for selecting a single run * Fix for selecting a single run * Improved exit icon, added shortcut to modal * Improved exit icon, added shortcut to modal * Improved exit icon, added shortcut to modal * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Tidy imports * Fix for grid layout when 1 page of bulk actions visible * Fix for grid layout when 1 page of bulk actions visible * Fix for grid layout when 1 page of bulk actions visible * Removed the ... on the abort button * Removed the ... on the abort button * Removed the ... on the abort button * Removed the ... on the abort button * Animate the progress bar * Set TZ="UTC" in the env example * Filter summary in the bulk inspector * Improves the pagination styling * Improves the pagination styling * Delete old bulk action routes * Removed old Postgres RunListPresenter * Retry any replication error where the message contains "timeout" * Increase wait to make test less flaky * The test was using run id instead of friendly id * Safer array access * Remove error log if there's a bad status * Nicer frontend type safety with the bulk action and mode * Switched a log to a debug log * Retry replication unless the error is a known non-retry error Flip the strategy to retry by default * Make ClickHouse required * Backfill run replication admin API endpoint * Set a CLICKHOUSE_URL for unit tests --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Eric Allam
|
b119a52e08 |
Implement MFA (#2244)
* Adds a new route for logging in with mfa * New path for security page * Adds “Security” link to account side menu * Update the Switch component to allow label positions left and right * Optionally hide the Close button in the Dialog title bar * Installs `qrcode` react package for generating QR codes. * CopyButton component now takes children * New Security route for setting up MFA * Adds new OTP package for the chadcn InputOTP component * Adds new InputOTP chadcn component * Adds InputOTP chadcn component to the MFA login screen * InputOTP component supports variant styles * Improvements to form handling * Show a confirmation modal before you can disable MFA * Revert redirect back to the dashboard for now * Implement MFA enabling and disabling * Refactor and cleanup mfa management code * More cleanup * Handle errors in the management action * Implement mfa login flow * recovery code input should be password * Implement rate limiting on the mfa validation endpoint * Better error ux * Implement mfa emails and apply James' updates * Use latest @better-auth/utils * Improvements via CodeRabbit review --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Matt Aitken
|
80d449bfdb |
Preview branch alerts (#2136)
* Preview branch alerts * Show “Preview” not the branch for new alert modal |
||
|
Matt Aitken
|
7b862b9438 |
Major dashboard improvements (environment centric) (#1796)
* Delete the proxy app (was v2) * Delete RunPresenterElectric * Select the best proj/org/env * Storing current proj/env in DB. Initial selection logic working with tasks page * 2sm needed to be in the Tailwind merge list * Move the task stream route (although we don’t actually use the env for now) * Alerts moved from /v3 * API keys page moved from /v3 * Concurrency page moved from /v3 * WIP on side menu sections * Improved the accordion animation * Moved schedules from /v3 * More pages moved * Move pages working * Run page working * Schedules working * Moved deployments * Alert pages moved * Delete electric hooks, not used * Started setting up blank states * Test page working * Removed “Select task” from the test page * Some work on deployment page * Style tweaks * Redirect from project root to approriate env * Improved env selector styling * Fix for jsx errors * Better min width on env selector * Improved the env switching logic * Added deployments to env routing * Redirect deployments to the correct env * Redirect run from proj to env * JSX icon fix * Only allow single env schedules from now on * Remove env var count from the API keys page * Move improvements and redirects * Project settings moved * Fix for scroll area on test page * Tweaked the test design * Made recent payloads column narrower * Improved the test layout some more * Added org icon, new project selector menu * WIP on org switching menu * Org switching is working * New menu working well, removed old side menu items * Buttons can now have a component name or an actual component for their icons * Removed the Projects page, instead redirect appropriately * Fix for broken blank states * Minor run table improvements * Removed unused switcher log and logic * Concurrency page fix for invalid html, improved layout * Minor improvements * Moved the side menu to the project level * Improved account styling * Moved org settings pages (with redirects) * Add current plan to billing side menu link * Upgrade to get staging from env dropdown * New env badge on concurrency limits page * Show Run Engine version in span presenter * New promote icon * Concurrency limits page is the sum of engine v1 + v2 queues * Fix for missing batch import * Added currentConcurrencyOfEnvQueue function * Basic avatar setting working * Avatar setting is working * You can change the color of your icon * Avatar improvements * Bugfix for mising prop * Removed some old env badges * Fixed replaying * Removed EnvironmentLabel * Old env badge deleted, changed everywhere to the new one * Fix for Slack integration paths * Fix for waitpoint completion form moving * Bulk replay/cancel env fix * Fix for alert webhook path * Redirect projects/v3/* to project/* * Fixes for CLI redirect routes * Remove welcome email (unused) * Change how we count schedules towards your limits * Use new schedules limits when checking a schedule * Added projectId back in to task queries (indexes) * WIP dev presence * CLI modal * Moved things around and use Context * Fix for p inside p * Dev connected status on run page * Correct dev env (not a teammates) * Show disconnected message at the end * Minor tweak on project dropdown icon padding * Fix for inconsistent date format for presence * Added a message when pushing to the billing page * Center the team page * Project settings page centered * Improvements to the dev presence |
||
|
nicktrn
|
e97704d904 |
Run Engine 2.0 (WIP) (#1575)
* bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Eric Allam
|
457273e4cf | Fix email alert failures because of newlines in error messages (fixes #1745) (#1749) | ||
|
James Ritchie
|
e71df640a9 |
Adds more org names to the alert emails (#1561)
Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Erin Allison
|
4d2412a0fe |
Add Support for Alternate Email Transports (#1580)
* Add support for AWS SES and SMTP email transports Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Correct environment variable names for new email settings Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Correct option names being passed to nodemailer for SMTP Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Remove use of AWS SDK synthetic default export It apparently causes issues when transpiled/bundled Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Add documentation for new email settings Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Move nodemailer types to devDependencies Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Adjust mail transport error handling Gotta keep the linter happy :) Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Fix typecheck error on MailTransportOptions Signed-off-by: Erin Allison <eallison@andrettikarting.com> * Correct environment variable usage for SMTP email transport Signed-off-by: Erin Allison <eallison@andrettikarting.com> --------- Signed-off-by: Erin Allison <eallison@andrettikarting.com> |
||
|
James Ritchie
|
f0182abc83 |
Alert emails now contain the Org name (#1555)
* Alert emails now contain the Org name * Adds org title to the alert parameters |
||
|
Matt Aitken
|
ea23dbd297 | Throw an error if Resend responds with an error (#1540) | ||
|
Eric Allam
|
7d4708839f |
Upgrade zod to 3.23.8 (#1484)
* Upgrade to zod 3.23.8 * Add changeset |
||
|
Matt Aitken
|
f2babbf637 |
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests |