fix(webapp): restore Postgres fallback for non-ClickHouse OTLP spans (#3803)

## Problem

On environments where runs carry a Postgres-backed `taskEventStore`
value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints
(`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500.

**Root cause:** The org-scoped ClickHouse factory introduced in a recent
PR routes all OTLP spans through `getEventRepositoryForOrganizationSync`
→ `buildEventRepository`. That function only handles `"clickhouse"` and
`"clickhouse_v2"` store values and throws `Unknown ClickHouse event
repository store: <value>` for anything else. The throw occurred inside
the grouping loop of `#exportEvents`, unwinding the entire method and
returning 500 for the whole batch.

The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as
non-retryable and drops the batch — causing real span loss.

**Fix:** Guard the `getEventRepositoryForOrganizationSync` call in
`#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2`
store values. All other values are routed directly to the Postgres
`eventRepository`, matching the guard pattern already present in
`resolveEventRepositoryForStore` and `getEventRepositoryForStore` in
`eventRepository/index.server.ts`.

The ClickHouse factory call is also wrapped in a try/catch that falls
back to Postgres so any unexpected store value in a future OTLP batch
degrades gracefully instead of failing the whole request.

## Changes

- `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing
guard and try/catch fallback in `#exportEvents`

## Testing

The `eventRepository/index.server.ts` module already has the same guard
pattern thoroughly covered. The fix brings `#exportEvents` into
alignment with that existing, tested pattern. Manual verification:
confirm OTLP batches containing Postgres-store spans return 200 and
route to the correct repository.
This commit is contained in:
Matt Aitken
2026-06-02 16:19:51 +01:00
committed by GitHub
parent 4b78d7e84e
commit 6961004a73
2 changed files with 19 additions and 4 deletions
@@ -0,0 +1,6 @@
---
area: webapp
type: fix
---
Fixes OTLP ingest endpoints returning HTTP 500 for runs on environments that use a Postgres-backed task event store. This caused the OpenTelemetry collector to drop entire span batches as non-retryable, resulting in real span loss.
+13 -4
View File
@@ -24,6 +24,7 @@ import type { ClickhouseFactory } from "~/services/clickhouse/clickhouseFactory.
import { clickhouseFactory } from "~/services/clickhouse/clickhouseFactoryInstance.server";
import { generateSpanId } from "./eventRepository/common.server";
import { eventRepository } from "./eventRepository/eventRepository.server";
import type {
CreatableEventKind,
CreatableEventStatus,
@@ -120,10 +121,18 @@ class OTLPExporter {
const routeKey = `${event.organizationId}\0${taskEventStore}`;
let resolved = routeCache.get(routeKey);
if (!resolved) {
resolved = this._clickhouseFactory.getEventRepositoryForOrganizationSync(
taskEventStore,
event.organizationId
);
// Non-ClickHouse stores (taskEvent / taskEventPartitioned) are Postgres-backed.
// The ClickHouse factory only handles clickhouse/clickhouse_v2 and throws otherwise.
if (taskEventStore !== "clickhouse" && taskEventStore !== "clickhouse_v2") {
// Non-ClickHouse stores (taskEvent / taskEventPartitioned) are Postgres-backed.
// The ClickHouse factory only handles clickhouse/clickhouse_v2 and throws otherwise.
resolved = { key: "postgres:default", repository: eventRepository };
} else {
resolved = this._clickhouseFactory.getEventRepositoryForOrganizationSync(
taskEventStore,
event.organizationId
);
}
routeCache.set(routeKey, resolved);
}