Files
triggerdotdev--trigger.dev/internal-packages/replication/package.json
T
Eric Allam 65da20c225 feat: replicate task runs to clickhouse to power dashboard improvements (#2035)
* WIP clickhouse package with test containers setup

* More clickhouse client setup now with otel and real tests, and the v1 of raw run events

* Add some additional columns to raw_run_events_v1

* WIP runs dashboard service

* Create a new run engine event bus event for the runs dashboard to hook into

* Track run events in the run engine

* make sure engine v1 runs get synced to CH

* Update the attemptNumber of v3 task runs

* Restructure the run events to be more sparse

* emit more stuff

* Setup replication package

* scaffold the replication package

* replication wip

* resolve conflicts

* more replication stuff

* Add ability to drop the replication slot completely on teardown

* Use the new single replacingmergetree task events table for replication

* get it working

* insert payloads into their own table only on insert and then join

* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh

* Handover WIP and tests

* Testing the replication service

* Remove the runs dashboard stuff that we aren't using anymore

* Added a test for large payloads

* hacky typecheck fix

* Fix new internal package typecheck issues and start adding telemetry to the replication service

* tracing over spans, some other improvements

* Improvements to the runs replication service, now ready for testing

* Some fixes and cleanups

* Don't need this code anymore

* move transaction types into the runs replication service

* only send spans where there are transaction events

* A couple of suggested tweaks
2025-05-12 22:12:36 +01:00

30 lines
932 B
JSON

{
"name": "@internal/replication",
"private": true,
"version": "0.0.1",
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"type": "module",
"dependencies": {
"@internal/redis": "workspace:*",
"@internal/tracing": "workspace:*",
"@trigger.dev/core": "workspace:*",
"pg": "8.15.6",
"redlock": "5.0.0-beta.2"
},
"devDependencies": {
"@internal/testcontainers": "workspace:*",
"@vitest/coverage-v8": "^3.0.8",
"rimraf": "6.0.1",
"vitest": "^3.0.8",
"@types/pg": "8.11.14"
},
"scripts": {
"clean": "rimraf dist",
"typecheck": "tsc --noEmit -p tsconfig.build.json",
"build": "pnpm run clean && tsc -p tsconfig.build.json",
"dev": "tsc --watch -p tsconfig.build.json",
"test": "vitest --sequence.concurrent=false --no-file-parallelism",
"test:coverage": "vitest --sequence.concurrent=false --no-file-parallelism --coverage.enabled"
}
}