fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759)
## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Matt Aitken
@@ -0,0 +1,21 @@
|
||||
---
|
||||
area: webapp
|
||||
type: fix
|
||||
---
|
||||
|
||||
Extend the runs-replication sanitizer (`sanitizeUnknownInPlace`) to detect
|
||||
JS Numbers that JSON-serialise as bare integer tokens outside the
|
||||
Int64..UInt64 range and replace them with their string form, so a
|
||||
following retry insert no longer trips ClickHouse's
|
||||
`INCORRECT_DATA` parser failure on `JSON(max_dynamic_paths)` columns.
|
||||
|
||||
This is the second class of poisoned-row failure that was stranding
|
||||
`scan-social-profiles` runs in `EXECUTING` on the Tasks page even after
|
||||
the UTF-16 surrogate fix (#3708 / TRI-9755). Root cause: upstream JS
|
||||
Number precision loss on a 21-digit Google Plus ID
|
||||
(`117039831458782873093` → `117039831458782870000`) — the precision-lossy
|
||||
value still serialises as a bare integer that exceeds UInt64.MAX,
|
||||
which CH's JSON column rejects with `Cannot parse JSON object here`.
|
||||
|
||||
Recovery stays purely reactive (no extra cost on the hot replication
|
||||
path); the sanitizer only runs after a ClickHouse parse-error rejection.
|
||||
@@ -7,6 +7,48 @@ import { detectBadJsonStrings } from "~/utils/detectBadJsonStrings";
|
||||
*/
|
||||
export const INVALID_UTF16_SENTINEL = "[invalid-utf16]";
|
||||
|
||||
/**
|
||||
* ClickHouse's `JSON(max_dynamic_paths)` column fits each bare-integer
|
||||
* JSON token into Int64 (signed) or UInt64 (unsigned). Bare integers
|
||||
* outside `[-2^63, 2^64 - 1]` are rejected with `INCORRECT_DATA` (no
|
||||
* silent fallback to Float64). `JSON.stringify` emits any integer-valued
|
||||
* Number with `|value| < 1e21` as a bare integer (no exponent), so any
|
||||
* JS Number above ~9.2e18 that *happens* to be integer-valued lands on
|
||||
* the wire as a token CH cannot accept.
|
||||
*
|
||||
* The fix: replace such Numbers with their string form. CH's dynamic
|
||||
* JSON column accepts a `String` subtype on the same path, so the row
|
||||
* inserts cleanly on retry. The numeric value was already
|
||||
* precision-lossy upstream (JS Number can't represent integers above
|
||||
* 2^53 faithfully), so type-flipping to string is information-preserving
|
||||
* relative to what arrived.
|
||||
*
|
||||
* Float-valued numbers (including very large ones like `1e25`) serialise
|
||||
* with an exponent and are accepted by CH at any magnitude, so they're
|
||||
* left alone.
|
||||
*/
|
||||
const UINT64_MAX = 18446744073709551615n;
|
||||
const INT64_MIN = -9223372036854775808n;
|
||||
|
||||
function isUnsafeJsonInteger(value: number): boolean {
|
||||
if (!Number.isFinite(value)) return false;
|
||||
if (!Number.isInteger(value)) return false;
|
||||
// JSON.stringify emits integer-valued Numbers as bare integer tokens
|
||||
// (no exponent) only while `|value| < 1e21`; at or above that
|
||||
// threshold `Number.prototype.toString` switches to exponential form,
|
||||
// which CH accepts as Float64 at any magnitude. So the dangerous band
|
||||
// is strictly between the Int64/UInt64 boundary and 1e21.
|
||||
if (Math.abs(value) >= 1e21) return false;
|
||||
// Compare via BigInt for exactness. The Number literal 18446744073709551615
|
||||
// is rounded to 2**64 in float64 (the float spacing near 2^64 is 2048), so a
|
||||
// direct `value > 18446744073709551615` would miss a Number whose float64
|
||||
// value is exactly 2**64 — `JSON.stringify` of that emits
|
||||
// "18446744073709552000", which exceeds UInt64.MAX and ClickHouse rejects.
|
||||
// `BigInt(value)` is safe here because we already gated on Number.isInteger.
|
||||
const asBigInt = BigInt(value);
|
||||
return asBigInt > UINT64_MAX || asBigInt < INT64_MIN;
|
||||
}
|
||||
|
||||
export type SanitizeResult = {
|
||||
/** How many rows had at least one string field replaced. */
|
||||
rowsTouched: number;
|
||||
@@ -62,6 +104,10 @@ export function sanitizeUnknownInPlace(value: unknown): { value: unknown; fixed:
|
||||
return { value, fixed: 0 };
|
||||
}
|
||||
|
||||
if (typeof value === "number" && isUnsafeJsonInteger(value)) {
|
||||
return { value: String(value), fixed: 1 };
|
||||
}
|
||||
|
||||
if (Array.isArray(value)) {
|
||||
let fixed = 0;
|
||||
for (let i = 0; i < value.length; i++) {
|
||||
|
||||
@@ -105,6 +105,104 @@ describe("sanitizeUnknownInPlace", () => {
|
||||
expect(sanitizeUnknownInPlace(null)).toEqual({ value: null, fixed: 0 });
|
||||
expect(sanitizeUnknownInPlace(undefined)).toEqual({ value: undefined, fixed: 0 });
|
||||
});
|
||||
|
||||
// ─── Out-of-range integers (TRI-9755) ──────────────────────────────────────
|
||||
// ClickHouse's JSON(max_dynamic_paths) column rejects bare integer tokens
|
||||
// outside [Int64.MIN, UInt64.MAX]. Such Numbers serialise as bare integer
|
||||
// form via JSON.stringify (no exponent, since |value| < 1e21) so they reach
|
||||
// ClickHouse as unquoted oversized ints. Sanitizer replaces them with the
|
||||
// string form, which ClickHouse's dynamic JSON column accepts as a String
|
||||
// subtype on that path.
|
||||
|
||||
it("replaces an integer-valued Number above UInt64.MAX with its string form", () => {
|
||||
// 117039831458782870000 is the actual prod value (Google Plus ID after
|
||||
// upstream JS-Number precision loss from 117039831458782873093).
|
||||
const result = sanitizeUnknownInPlace(117039831458782870000);
|
||||
expect(result.value).toBe("117039831458782870000");
|
||||
expect(result.fixed).toBe(1);
|
||||
});
|
||||
|
||||
it("catches the float64 boundary at exactly 2**64 (UInt64.MAX + 1)", () => {
|
||||
// float64 cannot represent UInt64.MAX (2^64 - 1) exactly — the literal
|
||||
// 18446744073709551615 in JS source rounds to 2^64. JSON.stringify
|
||||
// emits this Number as "18446744073709552000", which exceeds UInt64.MAX
|
||||
// and trips ClickHouse. Regression for the BigInt-based comparison;
|
||||
// a naïve `value > 18446744073709551615` would let this pass.
|
||||
const result = sanitizeUnknownInPlace(2 ** 64);
|
||||
expect(result.value).toBe("18446744073709552000");
|
||||
expect(result.fixed).toBe(1);
|
||||
});
|
||||
|
||||
it("replaces an integer-valued Number below Int64.MIN with its string form", () => {
|
||||
// -9223372036854775809 is the first failing negative; in float64 it
|
||||
// rounds to the same representation as Int64.MIN (-9223372036854775808),
|
||||
// but for completeness we check a clearly-out-of-range negative.
|
||||
const result = sanitizeUnknownInPlace(-1e20);
|
||||
expect(result.value).toBe("-100000000000000000000");
|
||||
expect(result.fixed).toBe(1);
|
||||
});
|
||||
|
||||
it("leaves safe integers and boundary values untouched", () => {
|
||||
// 42 — safe integer
|
||||
expect(sanitizeUnknownInPlace(42)).toEqual({ value: 42, fixed: 0 });
|
||||
// Number.MAX_SAFE_INTEGER (2^53 - 1) — JSON.stringify still emits as integer
|
||||
expect(sanitizeUnknownInPlace(Number.MAX_SAFE_INTEGER)).toEqual({
|
||||
value: Number.MAX_SAFE_INTEGER,
|
||||
fixed: 0,
|
||||
});
|
||||
// 2^63 (Int64.MAX + 1) — still fits in UInt64, CH accepts it
|
||||
expect(sanitizeUnknownInPlace(2 ** 63)).toEqual({ value: 2 ** 63, fixed: 0 });
|
||||
});
|
||||
|
||||
it("leaves non-integer numbers untouched (floats, NaN, Infinity)", () => {
|
||||
// Numbers with a fractional part — emitted with `.` in JSON
|
||||
expect(sanitizeUnknownInPlace(3.14)).toEqual({ value: 3.14, fixed: 0 });
|
||||
// Very large float-form (>= 1e21) — JSON.stringify uses exponent form,
|
||||
// CH parses as Float64 successfully
|
||||
expect(sanitizeUnknownInPlace(1e25)).toEqual({ value: 1e25, fixed: 0 });
|
||||
// NaN / Infinity — JSON.stringify emits `null`, so harmless on the wire
|
||||
expect(sanitizeUnknownInPlace(Number.NaN)).toEqual({ value: Number.NaN, fixed: 0 });
|
||||
expect(sanitizeUnknownInPlace(Number.POSITIVE_INFINITY)).toEqual({
|
||||
value: Number.POSITIVE_INFINITY,
|
||||
fixed: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("finds an oversized integer nested deep inside the actual scan-social-profiles shape", () => {
|
||||
const row = {
|
||||
output: {
|
||||
data: {
|
||||
profiles: [
|
||||
{ module: "linktree", query: "x@example.com" },
|
||||
{
|
||||
module: "poshmark",
|
||||
spec_format: [
|
||||
{
|
||||
platform_variables: [
|
||||
{
|
||||
key: "gp_id",
|
||||
proper_key: "Gp Id",
|
||||
// The actual prod value — bare JSON integer > UInt64.MAX
|
||||
value: 117039831458782870000,
|
||||
type: "int",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
};
|
||||
const result = sanitizeUnknownInPlace(row);
|
||||
expect(result.fixed).toBe(1);
|
||||
expect(
|
||||
(row.output.data.profiles[1].spec_format![0].platform_variables[0] as any).value
|
||||
).toBe("117039831458782870000");
|
||||
// Untouched neighbours
|
||||
expect(row.output.data.profiles[0].module).toBe("linktree");
|
||||
expect(row.output.data.profiles[1].spec_format![0].platform_variables[0].type).toBe("int");
|
||||
});
|
||||
});
|
||||
|
||||
describe("sanitizeRows", () => {
|
||||
@@ -158,4 +256,36 @@ describe("sanitizeRows", () => {
|
||||
expect(result.rowsTouched).toBe(1);
|
||||
expect(result.fieldsSanitized).toBe(2);
|
||||
});
|
||||
|
||||
it("counts surrogate fixes and out-of-range integer fixes together (TRI-9755)", () => {
|
||||
const rows = [
|
||||
{
|
||||
id: "r0",
|
||||
attributes: {
|
||||
surrogate: `bad ${HIGH_SURROGATE}`,
|
||||
bigint: 117039831458782870000,
|
||||
clean: "fine",
|
||||
safe: 42,
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "r1",
|
||||
attributes: {
|
||||
bigint: -1e20,
|
||||
clean: "still fine",
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "r2",
|
||||
attributes: { clean: "no fixes needed" },
|
||||
},
|
||||
];
|
||||
const result = sanitizeRows(rows);
|
||||
expect(result.rowsTouched).toBe(2);
|
||||
expect(result.fieldsSanitized).toBe(3);
|
||||
expect(rows[0].attributes.surrogate).toBe(INVALID_UTF16_SENTINEL);
|
||||
expect(rows[0].attributes.bigint).toBe("117039831458782870000");
|
||||
expect(rows[0].attributes.safe).toBe(42);
|
||||
expect(rows[1].attributes.bigint).toBe("-100000000000000000000");
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user