Eric Allam
0445b8ec27
## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read.
30 lines
1.3 KiB
Diff
30 lines
1.3 KiB
Diff
diff --git a/dist/error/error.d.ts b/dist/error/error.d.ts
|
|
index fb04ebb65a8c7115c465093462edaafa14a814de..30b370bf2eb8ba7491c249db96d565a67fe367c6 100644
|
|
--- a/dist/error/error.d.ts
|
|
+++ b/dist/error/error.d.ts
|
|
@@ -7,6 +7,7 @@ interface ParsedClickHouseError {
|
|
export declare class ClickHouseError extends Error {
|
|
readonly code: string;
|
|
readonly type: string | undefined;
|
|
+ rawMessage?: string;
|
|
constructor({ message, code, type }: ParsedClickHouseError);
|
|
}
|
|
export declare function parseError(input: string | Error): ClickHouseError | Error;
|
|
diff --git a/dist/error/error.js b/dist/error/error.js
|
|
index e06fa4ab36537c2a448857c1b001e70cd771bfe5..9750bd7e1285ac8cf2c8095b592ed344126176b6 100644
|
|
--- a/dist/error/error.js
|
|
+++ b/dist/error/error.js
|
|
@@ -35,7 +35,11 @@ function parseError(input) {
|
|
const match = message.match(errorRe);
|
|
const groups = match?.groups;
|
|
if (groups) {
|
|
- return new ClickHouseError(groups);
|
|
+ const error = new ClickHouseError(groups);
|
|
+ if (!inputIsError && groups.message.startsWith("Cannot parse JSON object")) {
|
|
+ Object.defineProperty(error, "rawMessage", { value: message, enumerable: false });
|
|
+ }
|
|
+ return error;
|
|
}
|
|
else {
|
|
return inputIsError ? input : new Error(input);
|