feat(webapp): add CLICKHOUSE_READER_URL to route reads to a replica

Read-only clients (logs, query, admin, runsList, engine, realtime) now fall back X_CLICKHOUSE_URL ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL, and the events client uses the reader/writer split so trace/span/log reads hit the replica while event + log inserts stay on the writer. Set CLICKHOUSE_READER_URL once to move all reads off the primary. Writes (events, replication, sessions_replication, standard) always stay on CLICKHOUSE_URL. No-op when CLICKHOUSE_READER_URL is unset.
feat(webapp): add runsList clickhouse client for dedicated runs list reads
2026-06-30 11:36:13 +01:00 · 2026-06-30 11:16:59 +01:00 · 2026-06-23 13:07:34 +01:00
11 changed files with 196 additions and 27 deletions
@@ -0,0 +1,6 @@
+---
+area: webapp
+type: feature
+---
+
+Add an "Override region" option to the bulk replay action so replayed runs can be routed to a chosen region, defaulting to keeping each run in its original region.
@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Add `CLICKHOUSE_READER_URL` to route ClickHouse reads to a read replica while writes stay on `CLICKHOUSE_URL`. Optional; defaults to `CLICKHOUSE_URL`.
@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Add `RUNS_LIST_CLICKHOUSE_URL` to send runs list queries to a separate ClickHouse instance. Defaults to `CLICKHOUSE_URL`.
@@ -1627,6 +1627,11 @@ const EnvironmentSchema = z

    // Clickhouse
    CLICKHOUSE_URL: z.string(),
+    // Optional read replica endpoint. Read-only clients (logs, query, admin, runsList,
+    // engine, realtime) and the events client's READ path default to this when their own
+    // URL is unset; writes always stay on CLICKHOUSE_URL. Set once to move all reads to a
+    // replica. Must share storage with the CLICKHOUSE_URL warehouse.
+    CLICKHOUSE_READER_URL: z.string().optional(),
    CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
    CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
    CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
@@ -1653,13 +1658,13 @@ const EnvironmentSchema = z
    LOGS_CLICKHOUSE_URL: z
      .string()
      .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),

    // Query page ClickHouse limits (for TSQL queries)
    QUERY_CLICKHOUSE_URL: z
      .string()
      .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
    QUERY_CLICKHOUSE_MAX_EXECUTION_TIME: z.coerce.number().int().default(10),
    QUERY_CLICKHOUSE_MAX_MEMORY_USAGE: z.coerce.number().int().default(1_073_741_824), // 1GB in bytes
    QUERY_CLICKHOUSE_MAX_AST_ELEMENTS: z.coerce.number().int().default(4_000_000),
@@ -1678,7 +1683,7 @@ const EnvironmentSchema = z
    ADMIN_CLICKHOUSE_URL: z
      .string()
      .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),

    EVENTS_CLICKHOUSE_URL: z
      .string()
@@ -1696,7 +1701,7 @@ const EnvironmentSchema = z
    RUN_ENGINE_CLICKHOUSE_URL: z
      .string()
      .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
    RUN_ENGINE_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
    RUN_ENGINE_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
    RUN_ENGINE_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(5),
@@ -1708,7 +1713,7 @@ const EnvironmentSchema = z
    REALTIME_BACKEND_NATIVE_CLICKHOUSE_URL: z
      .string()
      .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
    REALTIME_BACKEND_NATIVE_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
    REALTIME_BACKEND_NATIVE_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
    REALTIME_BACKEND_NATIVE_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
@@ -1716,6 +1721,20 @@ const EnvironmentSchema = z
      .enum(["log", "error", "warn", "info", "debug"])
      .default("info"),
    REALTIME_BACKEND_NATIVE_CLICKHOUSE_COMPRESSION_REQUEST: z.string().default("1"),
+    // Dedicated ClickHouse pool for the runs list (dashboard + API). Lets us point
+    // the highest-traffic read path at a read replica without moving ingest/replication
+    // writes off CLICKHOUSE_URL. Falls back to CLICKHOUSE_URL when unset.
+    RUNS_LIST_CLICKHOUSE_URL: z
+      .string()
+      .optional()
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
+    RUNS_LIST_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
+    RUNS_LIST_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
+    RUNS_LIST_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
+    RUNS_LIST_CLICKHOUSE_LOG_LEVEL: z
+      .enum(["log", "error", "warn", "info", "debug"])
+      .default("info"),
+    RUNS_LIST_CLICKHOUSE_COMPRESSION_REQUEST: z.string().default("1"),
    EVENTS_CLICKHOUSE_BATCH_SIZE: z.coerce.number().int().default(1000),
    EVENTS_CLICKHOUSE_FLUSH_INTERVAL_MS: z.coerce.number().int().default(1000),
    METRICS_CLICKHOUSE_BATCH_SIZE: z.coerce.number().int().default(10000),
@@ -269,7 +269,7 @@ export class ApiRunListPresenter extends BasePresenter {
        options.machines = searchParams["filter[machine]"];
      }

-      const clickhouse = await clickhouseFactory.getClickhouseForOrganization(organizationId, "standard");
+      const clickhouse = await clickhouseFactory.getClickhouseForOrganization(organizationId, "runsList");
      const presenter = new NextRunListPresenter(this._replica, clickhouse);

      logger.debug("Calling RunListPresenter", { options });
@@ -95,7 +95,7 @@ export const loader = async ({ request, params }: LoaderFunctionArgs) => {

  const clickhouse = await clickhouseFactory.getClickhouseForOrganization(
    project.organizationId,
-    "standard"
+    "runsList"
  );
  const presenter = new NextRunListPresenter($replica, clickhouse);
  const list = presenter.call(project.organizationId, environment.id, {
@@ -40,6 +40,7 @@ import { InputGroup } from "~/components/primitives/InputGroup";
 import { Label } from "~/components/primitives/Label";
 import { Paragraph } from "~/components/primitives/Paragraph";
 import { RadioGroup, RadioGroupItem } from "~/components/primitives/RadioButton";
+import { Select, SelectItem } from "~/components/primitives/Select";
 import { type TaskRunListSearchFilters } from "~/components/runs/v3/RunFilters";
 import { useEnvironment } from "~/hooks/useEnvironment";
 import { useOptimisticLocation } from "~/hooks/useOptimisticLocation";
@@ -51,37 +52,45 @@ import { redirectWithErrorMessage, redirectWithSuccessMessage } from "~/models/m
 import { findProjectBySlug } from "~/models/project.server";
 import { findEnvironmentBySlug } from "~/models/runtimeEnvironment.server";
 import { CreateBulkActionPresenter } from "~/presenters/v3/CreateBulkActionPresenter.server";
+import { RegionsPresenter } from "~/presenters/v3/RegionsPresenter.server";
 import { RUNS_BULK_INSPECTOR_UI_SEARCH_PARAMS } from "~/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs._index/shouldRevalidateRunsList";
 import { logger } from "~/services/logger.server";
-import { requireUserId } from "~/services/session.server";
+import { requireUser, requireUserId } from "~/services/session.server";
 import { cn } from "~/utils/cn";
 import { EnvironmentParamSchema, v3BulkActionPath } from "~/utils/pathBuilder";
 import { BulkActionService } from "~/v3/services/bulk/BulkActionV2.server";

 export async function loader({ request, params }: LoaderFunctionArgs) {
-  const userId = await requireUserId(request);
+  const user = await requireUser(request);

  const { organizationSlug, projectParam, envParam } = EnvironmentParamSchema.parse(params);

-  const project = await findProjectBySlug(organizationSlug, projectParam, userId);
+  const project = await findProjectBySlug(organizationSlug, projectParam, user.id);
  if (!project) {
    throw new Response("Not Found", { status: 404 });
  }

-  const environment = await findEnvironmentBySlug(project.id, envParam, userId);
+  const environment = await findEnvironmentBySlug(project.id, envParam, user.id);
  if (!environment) {
    throw new Response("Not Found", { status: 404 });
  }

  const presenter = new CreateBulkActionPresenter();
-  const data = await presenter.call({
-    organizationId: project.organizationId,
-    projectId: project.id,
-    environmentId: environment.id,
-    request,
-  });
+  const [data, regionsResult] = await Promise.all([
+    presenter.call({
+      organizationId: project.organizationId,
+      projectId: project.id,
+      environmentId: environment.id,
+      request,
+    }),
+    new RegionsPresenter().call({
+      userId: user.id,
+      projectSlug: projectParam,
+      isAdmin: user.admin || user.isImpersonating,
+    }),
+  ]);

-  return typedjson(data);
+  return typedjson({ ...data, regions: regionsResult.regions });
 }

 export const CreateBulkActionSearchParams = z.object({
@@ -89,6 +98,10 @@ export const CreateBulkActionSearchParams = z.object({
  action: BulkActionAction.default("cancel"),
 });

+// Sentinel for the "Override region" dropdown meaning "keep each run's original
+// region". Normalized to `undefined` in the action so the service never sees it.
+const REPLAY_REGION_NO_OVERRIDE_VALUE = "__no_override__";
+
 export const CreateBulkActionPayload = z.discriminatedUnion("mode", [
  z.object({
    mode: z.literal("selected"),
@@ -99,6 +112,7 @@ export const CreateBulkActionPayload = z.discriminatedUnion("mode", [
      return [];
    }, z.array(z.string())),
    title: z.string().optional(),
+    region: z.string().optional(),
    failedRedirect: z.string(),
    emailNotification: z.preprocess((value) => value === "on", z.boolean()),
  }),
@@ -106,6 +120,7 @@ export const CreateBulkActionPayload = z.discriminatedUnion("mode", [
    mode: z.literal("filter"),
    action: BulkActionAction,
    title: z.string().optional(),
+    region: z.string().optional(),
    failedRedirect: z.string(),
    emailNotification: z.preprocess((value) => value === "on", z.boolean()),
  }),
@@ -138,6 +153,12 @@ export async function action({ params, request }: ActionFunctionArgs) {
    return redirectWithErrorMessage("/", request, "Invalid bulk action");
  }

+  // "Don't override" keeps each run's original region — drop it so it isn't
+  // stored as a real override.
+  if (submission.value.region === REPLAY_REGION_NO_OVERRIDE_VALUE) {
+    submission.value.region = undefined;
+  }
+
  const service = new BulkActionService();
  const [error, result] = await tryCatch(
    service.create(
@@ -212,6 +233,23 @@ export function CreateBulkActionInspector({
  const impactedCountElement =
    mode === "selected" ? selectedItems.size : <EstimatedCount count={data?.count} />;

+  // Region is a replay-only override and only applies to deployed environments.
+  // The default keeps each run in its original region so a bulk action spanning
+  // multiple regions doesn't silently re-route runs.
+  const regions = data?.regions ?? [];
+  const showRegion =
+    action === "replay" && environment.type !== "DEVELOPMENT" && regions.length > 1;
+  const regionItems = [
+    { value: REPLAY_REGION_NO_OVERRIDE_VALUE, label: "Don't override", isDefault: false },
+    ...regions.map((r) => ({
+      // masterQueue is the region routing key the replay resolves against
+      // (WorkerGroupService matches regionOverride on masterQueue); name is display only.
+      value: r.masterQueue,
+      label: r.description ? `${r.name} — ${r.description}` : r.name,
+      isDefault: r.isDefault,
+    })),
+  ];
+
  return (
    <Form
      method="post"
@@ -342,6 +380,34 @@ export function CreateBulkActionInspector({
                />
              </RadioGroup>
            </InputGroup>
+            {showRegion && (
+              <InputGroup>
+                <Label htmlFor="region">Override region</Label>
+                {/* Our Select primitive uses Ariakit, which treats value={undefined}
+                    as uncontrolled and keeps stale state when switching environments.
+                    The key forces a remount so it reinitializes with the default value. */}
+                <Select
+                  key={`bulk-region-${environment.id}`}
+                  name="region"
+                  variant="tertiary/medium"
+                  dropdownIcon
+                  items={regionItems}
+                  defaultValue={REPLAY_REGION_NO_OVERRIDE_VALUE}
+                  text={(value) => regionItems.find((r) => r.value === value)?.label}
+                >
+                  {regionItems.map((r) => (
+                    <SelectItem key={r.value} value={r.value}>
+                      {r.label}
+                      {r.isDefault ? " (default)" : ""}
+                    </SelectItem>
+                  ))}
+                </Select>
+                <Hint>
+                  By default each run is replayed in its original region. Select a region to run
+                  them all there instead.
+                </Hint>
+              </InputGroup>
+            )}
            <InputGroup>
              <Label>Preview</Label>
              <BulkActionFilterSummary
@@ -67,7 +67,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {

  const clickhouse = await clickhouseFactory.getClickhouseForOrganization(
    project.organizationId,
-    "standard"
+    "runsList"
  );
  const runsRepository = new RunsRepository({ clickhouse, prisma: $replica });

@@ -34,7 +34,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {

  const clickhouse = await clickhouseFactory.getClickhouseForOrganization(
    project.organizationId,
-    "standard"
+    "runsList"
  );
  const runsRepository = new RunsRepository({ clickhouse, prisma: $replica });

@@ -242,6 +242,36 @@ function initializeRealtimeClickhouseClient(): ClickHouse {
  });
 }

+/** Runs list reads — dashboard + API (`RUNS_LIST_CLICKHOUSE_URL`);
+ *  falls back to the default client if unset. */
+const defaultRunsListClickhouseClient = singleton(
+  "runsListClickhouseClient",
+  initializeRunsListClickhouseClient
+);
+
+function initializeRunsListClickhouseClient(): ClickHouse {
+  if (!env.RUNS_LIST_CLICKHOUSE_URL) {
+    return defaultClickhouseClient;
+  }
+
+  const url = new URL(env.RUNS_LIST_CLICKHOUSE_URL);
+  url.searchParams.delete("secure");
+
+  return new ClickHouse({
+    url: url.toString(),
+    name: "runs-list-clickhouse",
+    keepAlive: {
+      enabled: env.RUNS_LIST_CLICKHOUSE_KEEP_ALIVE_ENABLED === "1",
+      idleSocketTtl: env.RUNS_LIST_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS,
+    },
+    logLevel: env.RUNS_LIST_CLICKHOUSE_LOG_LEVEL,
+    compression: {
+      request: env.RUNS_LIST_CLICKHOUSE_COMPRESSION_REQUEST === "1",
+    },
+    maxOpenConnections: env.RUNS_LIST_CLICKHOUSE_MAX_OPEN_CONNECTIONS,
+  });
+}
+
 /** Task events (`EVENTS_CLICKHOUSE_URL`); not exported — accessed via factory. */
 const defaultEventsClickhouseClient = singleton(
  "eventsClickhouseClient",
@@ -253,12 +283,10 @@ function initializeEventsClickhouseClient(): ClickHouse {
    throw new Error("EVENTS_CLICKHOUSE_URL is not set");
  }

-  const url = new URL(env.EVENTS_CLICKHOUSE_URL);
-  url.searchParams.delete("secure");
+  const writerUrl = new URL(env.EVENTS_CLICKHOUSE_URL);
+  writerUrl.searchParams.delete("secure");

-  return new ClickHouse({
-    url: url.toString(),
-    name: "task-events",
+  const commonConfig = {
    keepAlive: {
      enabled: env.EVENTS_CLICKHOUSE_KEEP_ALIVE_ENABLED === "1",
      idleSocketTtl: env.EVENTS_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS,
@@ -268,6 +296,29 @@ function initializeEventsClickhouseClient(): ClickHouse {
      request: env.EVENTS_CLICKHOUSE_COMPRESSION_REQUEST === "1",
    },
    maxOpenConnections: env.EVENTS_CLICKHOUSE_MAX_OPEN_CONNECTIONS,
+  };
+
+  // This client both inserts events and reads traces/spans/logs. When a reader replica
+  // is configured, split it so queries hit the replica while inserts stay on the writer.
+  if (env.CLICKHOUSE_READER_URL) {
+    const readerUrl = new URL(env.CLICKHOUSE_READER_URL);
+    readerUrl.searchParams.delete("secure");
+
+    if (readerUrl.toString() !== writerUrl.toString()) {
+      return new ClickHouse({
+        ...commonConfig,
+        writerName: "task-events-writer",
+        writerUrl: writerUrl.toString(),
+        readerName: "task-events-reader",
+        readerUrl: readerUrl.toString(),
+      });
+    }
+  }
+
+  return new ClickHouse({
+    ...commonConfig,
+    name: "task-events",
+    url: writerUrl.toString(),
  });
 }

@@ -289,7 +340,8 @@ export type ClientType =
  | "query"
  | "admin"
  | "engine"
-  | "realtime";
+  | "realtime"
+  | "runsList";

 function buildOrgClickhouseClient(url: string, clientType: ClientType): ClickHouse {
  const parsed = new URL(url);
@@ -379,6 +431,7 @@ function buildOrgClickhouseClient(url: string, clientType: ClientType): ClickHou
    case "standard":
    case "query":
    case "admin":
+    case "runsList":
      return new ClickHouse({
        url: parsed.toString(),
        name,
@@ -446,6 +499,8 @@ export class ClickhouseFactory {
          return defaultRunEngineClickhouseClient;
        case "realtime":
          return defaultRealtimeClickhouseClient;
+        case "runsList":
+          return defaultRunsListClickhouseClient;
      }
    }

@@ -37,6 +37,12 @@ export class BulkActionService extends BaseService {
  ) {
    const filters = await getFilters(payload, request);

+    // Region is a replay-only override that re-routes the replayed runs. It's
+    // stored alongside the run-list filters under a dedicated key so it isn't
+    // mistaken for a `regions` selection filter when the params are parsed.
+    const replayRegion = payload.action === "replay" ? payload.region : undefined;
+    const params = replayRegion ? { ...filters, replayRegion } : filters;
+
    // Count the runs that will be affected by the bulk action
    const clickhouse = await clickhouseFactory.getClickhouseForOrganization(organizationId, "standard");
    const runsRepository = new RunsRepository({
@@ -61,7 +67,7 @@ export class BulkActionService extends BaseService {
        userId,
        name: payload.title,
        type: payload.action === "cancel" ? BulkActionType.CANCEL : BulkActionType.REPLAY,
-        params: filters,
+        params,
        queryName: "bulk_action_v1",
        totalCount: count,
        completionNotification:
@@ -141,6 +147,10 @@ export class BulkActionService extends BaseService {
    // 2. Parse the params
    const rawParams = group.params && typeof group.params === "object" ? group.params : {};
    const finalizeRun = "finalizeRun" in rawParams && (rawParams as any).finalizeRun === true;
+    const replayRegion =
+      "replayRegion" in rawParams && typeof (rawParams as any).replayRegion === "string"
+        ? (rawParams as any).replayRegion
+        : undefined;
    const filters = parseRunListInputOptions({
      organizationId: group.project.organizationId,
      projectId: group.projectId,
@@ -248,6 +258,7 @@ export class BulkActionService extends BaseService {
              replayService.call(run, {
                bulkActionId: bulkActionId,
                triggerSource: "dashboard",
+                region: replayRegion,
              })
            );
            if (error) {
Author	SHA1	Message	Date
nicktrn	99ef921dc1	feat(webapp): add CLICKHOUSE_READER_URL to route reads to a replica 🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 10m37s Details 🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / units (push) Failing after 10m53s Details 🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled Details Read-only clients (logs, query, admin, runsList, engine, realtime) now fall back X_CLICKHOUSE_URL ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL, and the events client uses the reader/writer split so trace/span/log reads hit the replica while event + log inserts stay on the writer. Set CLICKHOUSE_READER_URL once to move all reads off the primary. Writes (events, replication, sessions_replication, standard) always stay on CLICKHOUSE_URL. No-op when CLICKHOUSE_READER_URL is unset.	2026-06-30 11:36:13 +01:00
nicktrn	aefa861723	feat(webapp): add runsList clickhouse client for dedicated runs list reads Runs list reads (dashboard list, runs list API, live reload, child-status counts) went through the shared standard client (CLICKHOUSE_URL). Add a dedicated runsList client type backed by RUNS_LIST_CLICKHOUSE_URL so this high-traffic read path can target a read replica without moving ingest or replication writes off CLICKHOUSE_URL. Falls back to CLICKHOUSE_URL when unset, so it is a no-op unless configured.	2026-06-30 11:16:59 +01:00
Matt Aitken	17990e0f6c	feat(webapp): add region override to the bulk replay action 🚀 Publish Trigger.dev Docker / units (push) Failing after 10m57s Details 🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 10m58s Details 🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped Details 🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled Details When replaying runs in bulk from a deployed environment, you can now choose which region the replayed runs run in. The inspector shows an "Override region" dropdown that defaults to "Don't override", which keeps each run in its original region, so replaying a selection that spans multiple regions doesn't silently re-route anything. Pick a region and every matched run is replayed there instead.	2026-06-23 13:07:34 +01:00