Files
triggerdotdev--trigger.dev/packages
Eric Allam bb253400a2 perf(runs-replication): Improve the CPU efficiency and throughput of the runs replication to clickhouse (#2866)
## Summary

Optimizes the runs replication service for better CPU efficiency and
throughput when inserting task runs into ClickHouse.

### Key Changes

- **Switch to compact array format** - Uses
`JSONCompactEachRowWithNames` instead of `JSONEachRow` for ClickHouse
inserts, reducing JSON serialization overhead
- **Type-safe tuple arrays** - Introduces `TaskRunInsertArray` and
`PayloadInsertArray` tuple types with compile-time column order
validation
- **Pre-sorted batch inserts** - Sorts inserts by primary key before
flushing for better ClickHouse insert performance
- **Programmatic index generation** - `TASK_RUN_INDEX` and
`PAYLOAD_INDEX` are generated from column arrays to prevent manual
synchronization errors

### Files Changed

- `runsReplicationService.server.ts` - Core optimization to use compact
array inserts
- `@internal/clickhouse` - Added `insertCompactRaw` method and tuple
types
- `taskRuns.ts` - Column definitions, index constants, and insert
functions

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 12:18:30 +00:00
..
2026-01-08 11:26:59 +00:00
2026-01-08 11:26:59 +00:00
2026-01-08 11:26:59 +00:00
2026-01-08 11:26:59 +00:00
2026-01-08 11:26:59 +00:00
2026-01-08 11:26:59 +00:00