7 Commits

Author	SHA1	Message	Date
Eric Allam	65da20c225	feat: replicate task runs to clickhouse to power dashboard improvements (#2035) ... * WIP clickhouse package with test containers setup * More clickhouse client setup now with otel and real tests, and the v1 of raw run events * Add some additional columns to raw_run_events_v1 * WIP runs dashboard service * Create a new run engine event bus event for the runs dashboard to hook into * Track run events in the run engine * make sure engine v1 runs get synced to CH * Update the attemptNumber of v3 task runs * Restructure the run events to be more sparse * emit more stuff * Setup replication package * scaffold the replication package * replication wip * resolve conflicts * more replication stuff * Add ability to drop the replication slot completely on teardown * Use the new single replacingmergetree task events table for replication * get it working * insert payloads into their own table only on insert and then join * prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh * Handover WIP and tests * Testing the replication service * Remove the runs dashboard stuff that we aren't using anymore * Added a test for large payloads * hacky typecheck fix * Fix new internal package typecheck issues and start adding telemetry to the replication service * tracing over spans, some other improvements * Improvements to the runs replication service, now ready for testing * Some fixes and cleanups * Don't need this code anymore * move transaction types into the runs replication service * only send spans where there are transaction events * A couple of suggested tweaks	2025-05-12 22:12:36 +01:00
Eric Allam	6ebd435e81	Feature: Run execution concurrency limits (#750) ... * WIP execution concurrency controls implemented via Redis - Split up resuming a run and executing a run - Added some new statuses to better show what is going on in a run - Removed preprocessing runs * WIP * Convert to using ZSETs and adding env vars * Removed unused import * Improve run number generation using advistory locks, and only on start * More execution concurrency stuff * Add support for job concurrency limits and concurrency limit groups * Create wild-swans-battle.md * Increase slots refresh timeout to 10s * Try to fix Redis connection issues * Don’t be so strict about the APP_ENV * Add the blank tls option to the normal redis client as well * Add docs	2023-11-28 16:21:06 +00:00
Eric Allam	c922d24941	Adding simulate graphile job to easily debug graceful shutdown retries in docker image	2023-10-16 16:04:05 +01:00
Eric Allam	24d892e322	Merges 573 with fix for Dockerfile (#578) ... * update prisma to 5.4.1 (#573) * Get prisma 5.4.1 to work in the Dockerfile --------- Co-authored-by: Karthik <kartim316@gmail.com>	2023-10-06 15:10:43 +01:00
nicktrn	603ffbc426	docs: add supabase guide (#297) ... * docs: add supabase guide * feat: add support for external connection poolers * chore: use .env symlink in database package * chore: remove database .env.example * docs: finish supabase pooling section * docs: fix small typo * docs: replace prisma with app wording * docs: fix up supabase pooling section * fix: deleted too much in contributing docs * docs: change pooling heading * docs: reduce linux shaming * docs: include direct connection url in fly.io	2023-08-11 15:51:54 +01:00
Eric Allam	1dfe7e0ab5	Updated dev-compose.yml and added some scripts to run it in package.json	2023-07-20 10:31:39 +01:00
Eric Allam	79a0532e1d	Handle OS signals in the docker container	2023-07-04 14:36:59 +01:00