* WIP
* Run queue now works with the worker queue / master queue split
* Acking should also cause the master queue to be processed
* Convert run engine tests and run engine to use runQueue changes
* Include the util files in the test tsconfig
* coordinator target should be es2020 as well
* providers target 2020
* Fix the triggerTask tests in the webapp
* v4 now working with the new worker queues, and added the legacy master queue migration stuff
* report worker queue lengths via opentelemetry metrics
* Adding lock metrics
* Release concurrency bucket metrics
* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount)
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class
* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker
* Fix CodeRabbit suggestions
* return undefined from dequeueFromWorkerQueue, not null
* Remove message from worker queue in certain circumstances when acking
* Update log
* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var
* Change how the run queue master queue consumers are disabled internally
* Fixed tests
* process the queue on nack
* Fix more tests
* Fix priority tests
* Fixed dequeueing test
* shard unit tests
* temp enable for all pushes
* fix test workflow
* update to latest vitest and only add to root package.json
* additionally use default reporter
* gather reports before uploading
* split up slow replication tests
* split up unit tests workflow
* move workflows to parent dir
* use new paths in parent workflow
* prevent artifact clashes
* we always need to create the reports dir
* speed up merge reports
* gather reports even when tests fail
* fix artifact patterns
* increase shards
* disable push trigger again
* improve dequeue snapshot test reliability
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* re2: fix @trigger.dev/core exports
* re2: WIP env based queue selection algo
* more wip
* WIP
* Get run engine tests to pass
* Adding tests for the fair dequeueing strat in the run engine
* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing
* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic
* Fixed webapp typechecks
* dev now depends on build, fixed supervisor typecheck
* Fixed run engine tests
* Fixed e2e tests
Eric Allam