Eric Allam
2b3ea692fe
* WIP * Run queue now works with the worker queue / master queue split * Acking should also cause the master queue to be processed * Convert run engine tests and run engine to use runQueue changes * Include the util files in the test tsconfig * coordinator target should be es2020 as well * providers target 2020 * Fix the triggerTask tests in the webapp * v4 now working with the new worker queues, and added the legacy master queue migration stuff * report worker queue lengths via opentelemetry metrics * Adding lock metrics * Release concurrency bucket metrics * • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter • Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount) • Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter • Updated DeleteProjectService to call the method once per environment instead of once per master queue • Simplified API by encapsulating master queue sharding logic within RunQueue class * metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker * Fix CodeRabbit suggestions * return undefined from dequeueFromWorkerQueue, not null * Remove message from worker queue in certain circumstances when acking * Update log * Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var * Change how the run queue master queue consumers are disabled internally * Fixed tests * process the queue on nack * Fix more tests * Fix priority tests * Fixed dequeueing test
Kubernetes provider
The kubernetes-provider allows the platform to be orchestrator-agnostic. The platform can perform actions such as INDEX_TASKS or INVOKE_TASK which the provider translates into Kubernetes actions.