* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
Matt Aitken