Oskar Otwinowski
adaa8e9e30
## Summary `goose up` against `internal-packages/clickhouse/schema` panics on `main` today, so ClickHouse migrations cannot be applied from a fresh checkout. Renumbering the external deployment id migration from 040 to 041 clears it. ## Root cause Two migrations claim version 40. [#4615](https://github.com/triggerdotdev/trigger.dev/pull/4615) added `040_create_task_events_search_v2.sql`, and [#4661](https://github.com/triggerdotdev/trigger.dev/pull/4661) added `040_add_task_runs_v2_external_deployment_id.sql` a day later. #4661 was opened before #4615 merged, so 040 was genuinely free at branch time, and because the two files have different names there is no textual conflict for git or a rebase to surface. Both merged green, and no workflow in this repo runs `goose`, so the collision only shows up the first time someone actually migrates. goose parses the numeric filename prefix as the version and refuses duplicates: ``` panic: goose: duplicate version 40 detected: .../040_create_task_events_search_v2.sql .../040_add_task_runs_v2_external_deployment_id.sql ``` It aborts while collecting the directory, before executing any SQL, so nothing was half applied and there is no migration state to repair. This migration gets renumbered rather than the `task_events_search_v2` one because goose keys on the version number and not the filename: version 40 is already recorded wherever 040 has been applied, so renaming that file would re-run an applied migration. Verified with a full `goose up` against ClickHouse 26.2.19.43 (the image pinned in `internal-packages/testcontainers`): migrations apply cleanly through version 41, and `task_runs_v2.external_deployment_id` lands as `String DEFAULT ''`.
ClickHouse Table Naming Conventions
The following document is heavily inspired by the Unkey ClickHouse naming conventions.
This document outlines the naming conventions for tables and materialized views in our ClickHouse setup. Adhering to these conventions ensures consistency, clarity, and ease of management across our data infrastructure.
General Rules
- Use lowercase letters and separate words with underscores.
- Avoid ClickHouse reserved words and special characters in names.
- Be descriptive but concise.
Table Naming Convention
Format: [prefix]_[domain]_[description]_[version]
Prefixes
raw_: Input data tablestmp_{yourname}_: Temporary tables for experiments, add your name, so it's easy to identify ownership.
Versioning
- Version numbers:
_v1,_v2, etc.
Aggregation Suffixes
For aggregated or summary tables, use suffixes like:
_per_day_per_month_summary
Materialized View Naming Convention
Format: [description]_[aggregation]_mv_[version]
- Always suffix with
mv_[version] - Include a description of the view's purpose
- Add aggregation level if applicable
Examples
-
Raw Data Table:
raw_sales_transactions_v1 -
Materialized View:
active_users_per_day_mv_v2 -
Temporary Table:
tmp_eric_user_analysis_v1 -
Aggregated Table:
sales_summary_per_hour_mv_v1
Consistency Across Related Objects
Maintain consistent naming across related tables, views, and other objects:
raw_user_activity_v1user_activity_per_day_v1user_activity_per_day_mv_v1
By following these conventions, we ensure a clear, consistent, and scalable naming structure for our ClickHouse setup.