3 Commits

Author	SHA1	Message	Date
Eric Allam	65da20c225	feat: replicate task runs to clickhouse to power dashboard improvements (#2035) ... * WIP clickhouse package with test containers setup * More clickhouse client setup now with otel and real tests, and the v1 of raw run events * Add some additional columns to raw_run_events_v1 * WIP runs dashboard service * Create a new run engine event bus event for the runs dashboard to hook into * Track run events in the run engine * make sure engine v1 runs get synced to CH * Update the attemptNumber of v3 task runs * Restructure the run events to be more sparse * emit more stuff * Setup replication package * scaffold the replication package * replication wip * resolve conflicts * more replication stuff * Add ability to drop the replication slot completely on teardown * Use the new single replacingmergetree task events table for replication * get it working * insert payloads into their own table only on insert and then join * prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh * Handover WIP and tests * Testing the replication service * Remove the runs dashboard stuff that we aren't using anymore * Added a test for large payloads * hacky typecheck fix * Fix new internal package typecheck issues and start adding telemetry to the replication service * tracing over spans, some other improvements * Improvements to the runs replication service, now ready for testing * Some fixes and cleanups * Don't need this code anymore * move transaction types into the runs replication service * only send spans where there are transaction events * A couple of suggested tweaks	2025-05-12 22:12:36 +01:00
nicktrn	49a3f72e13	Publish redis-worker and add graceful shutdown manager (#1810) ... * add shutdown manager * update ai test instructions * add shutdown timeout to redis-worker * move redis worker to packages * add unregister method * prep for publishing package * fix types * update ai files * fix cursor terminal links * prevent overly friendly ids * use structured logger * use unique shutdown handler names * rework suspend completion * add trycatch util * rework suspend restore * add http server metrics * add missing prom-client to core * add prom metrics to redis worker * bundle redis-worker * fix esm/cjs interop * remove proxy from changeset ignore and add supervisor * add pause to prerelease script for any manual edits * unregister the correct handler and add early detection * small change to http handler return * fix worker tests * fix shutdown manager tests	2025-03-21 14:53:27 +00:00
Eric Allam	d855d55ea0	re2: env based queue selection algo (#1775) ... * re2: fix @trigger.dev/core exports * re2: WIP env based queue selection algo * more wip * WIP * Get run engine tests to pass * Adding tests for the fair dequeueing strat in the run engine * Configure the new queue selection strategy in the webapp and get it all building and typechecks passing * webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic * Fixed webapp typechecks * dev now depends on build, fixed supervisor typecheck * Fixed run engine tests * Fixed e2e tests	2025-03-07 14:30:19 +00:00