github-featured/triggerdotdev--trigger.dev

Files

T

Eric Allam 65da20c225 feat: replicate task runs to clickhouse to power dashboard improvements (#2035 )

* WIP clickhouse package with test containers setup

* More clickhouse client setup now with otel and real tests, and the v1 of raw run events

* Add some additional columns to raw_run_events_v1

* WIP runs dashboard service

* Create a new run engine event bus event for the runs dashboard to hook into

* Track run events in the run engine

* make sure engine v1 runs get synced to CH

* Update the attemptNumber of v3 task runs

* Restructure the run events to be more sparse

* emit more stuff

* Setup replication package

* scaffold the replication package

* replication wip

* resolve conflicts

* more replication stuff

* Add ability to drop the replication slot completely on teardown

* Use the new single replacingmergetree task events table for replication

* get it working

* insert payloads into their own table only on insert and then join

* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh

* Handover WIP and tests

* Testing the replication service

* Remove the runs dashboard stuff that we aren't using anymore

* Added a test for large payloads

* hacky typecheck fix

* Fix new internal package typecheck issues and start adding telemetry to the replication service

* tracing over spans, some other improvements

* Improvements to the runs replication service, now ready for testing

* Some fixes and cleanups

* Don't need this code anymore

* move transaction types into the runs replication service

* only send spans where there are transaction events

* A couple of suggested tweaks

2025-05-12 22:12:36 +01:00

1.7 KiB

Raw Permalink Blame History

ClickHouse Table Naming Conventions

The following document is heavily inspired by the Unkey ClickHouse naming conventions.

This document outlines the naming conventions for tables and materialized views in our ClickHouse setup. Adhering to these conventions ensures consistency, clarity, and ease of management across our data infrastructure.

General Rules

Use lowercase letters and separate words with underscores.
Avoid ClickHouse reserved words and special characters in names.
Be descriptive but concise.

Table Naming Convention

Format: [prefix]_[domain]_[description]_[version]

Prefixes

raw_: Input data tables
tmp_{yourname}_: Temporary tables for experiments, add your name, so it's easy to identify ownership.

Versioning

Version numbers: _v1, _v2, etc.

Aggregation Suffixes

For aggregated or summary tables, use suffixes like:

_per_day
_per_month
_summary

Materialized View Naming Convention

Format: [description]_[aggregation]_mv_[version]

Always suffix with mv_[version]
Include a description of the view's purpose
Add aggregation level if applicable

Examples

Raw Data Table: raw_sales_transactions_v1
Materialized View: active_users_per_day_mv_v2
Temporary Table: tmp_eric_user_analysis_v1
Aggregated Table: sales_summary_per_hour_mv_v1

Maintain consistent naming across related tables, views, and other objects:

raw_user_activity_v1
user_activity_per_day_v1
user_activity_per_day_mv_v1

By following these conventions, we ensure a clear, consistent, and scalable naming structure for our ClickHouse setup.

1.7 KiB Raw Permalink Blame History