Files
Daniel Sutton 1801b0e80b feat(webapp,docker): run-ops boot interlocks and migrations at N databases (#4780)
## Summary

The run-ops boot interlocks and the migration entrypoint each assume
exactly two run-ops
databases. This generalizes them to any number, so a deployment that
configures
`RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two
stores: no two stores
may point at one database, every store that owns its own database must
replicate to
ClickHouse, and every store must have its schema migrated.

With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check
over a two-element set is
the pairwise compare it replaces, replication coverage is the check it
was, and the entrypoint
runs the same two migration invocations.

A shard may declare `aliasOf: "new"`, which shares an existing store's
client by reference. An
aliased shard is not its own database, so it is exempt from the
distinctness check and needs no
replication slot of its own. Every check keys that exemption on the
declared field, never on
client object identity: two client objects can sit over one database,
which identity comparison
cannot see.

## Design

**Distinctness.** `probeDistinctDatabases` compared two URLs. It now
delegates to
`probeDistinctStores`, which reads every fingerprint in parallel and
groups them by system
identifier and database name. Any two stores under one key refuse the
boot. The old pairwise
entry point stays, so its existing container tests are the proof that
set uniqueness over one
pair gives the verdict it gave before. Fail-closed is unchanged: a probe
that cannot answer
returns not-distinct, because "distinct" is a positive claim a failed
probe cannot support.

**Co-residency.** The advisory runs once per store against the control
plane. The legacy
emission keeps its exact call shape and its untagged metric series, so
an existing dashboard
does not change. Each shard emits its own point carrying its shard key.
Every store emits
before any enforcement throw, so one offending store never costs another
store its metric.

**Replication.** `buildReplicationSources` appends one source per shard
that owns its own
database, taking the slot, publication and origin generation its
descriptor declares.
`assertReplicationCoversSplit` then requires a source per such shard.

That check also closes a hole it inherited. The descriptor parser
validates uniqueness among
shards only, so a shard could take the slot name, publication name or
origin generation of the
legacy or the new source. The replication service does validate this,
but it throws from its
constructor, and the caller reaches that constructor only after shutting
the bootstrap instance
down:

```ts
if (sources.length > 1) {
  await service.shutdown();                       // legacy stream stops here
  service = new RunsReplicationService({ ... });   // throws: duplicate slotName
}
```

The throw was not a `SplitReplicationMisconfiguredError`, so the process
stayed up with no
replication at all, legacy included, behind one logged line. That is the
silent ClickHouse
under-count the error exists to prevent. The check now runs at the boot
gate, before anything is
torn down, and raises a subclass the existing exit path already
recognizes. A correct deployment
already satisfies it, because two consumers on one WAL slot is a data
race that cannot work.

**Migrations.** Every shard runs the identical schema, so a new shard is
the existing migrations
against a new DSN. The runner image has no `jq`, so a small node script
prints one DSN per line
and the entrypoint loops over them. The loop is a `for` and not a `while
read` pipeline: a
pipeline subshell swallows a failed migration on any iteration but the
last, which would let a
broken shard boot. Tracing stays off across the capture and the loop,
because `set -x` prints an
assignment and a DSN carries credentials.

Verified end to end against real Postgres containers for the fingerprint
probes, and against the
real shell block with a stubbed migration command: an aliased shard is
skipped, `directUrl` wins
over `url`, a failing shard stops the container on the first failure,
and a malformed descriptor
stops it before it migrates anything.

Stacked on #4764.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-26 13:50:20 +01:00

96 lines
3.4 KiB
TypeScript

/**
* isSplitEnabled() is the Wave-0 gate. The entire migration/routing/FK-drop family
* MUST be unreachable when this returns false. Default is false (single-DB). Never
* infer split-vs-single from URL string-equality — distinctness is proven by the
* runtime sentinel.
*/
import { env } from "~/env.server";
import { logger } from "~/services/logger.server";
import { probeDistinctStores as defaultProbe } from "./distinctDbSentinel.server";
import { nonAliasedShards, type ShardTarget } from "~/v3/runOpsShards.server";
export type SplitModeConfig = {
flagEnabled: boolean;
legacyUrl?: string;
newUrl?: string;
/** Gen-2 shards that own their own database. Empty (the default) is today's gen-1 pair. */
shards?: ShardTarget[];
};
export type SplitModeDeps = {
probe?: typeof defaultProbe;
logger?: { warn: (msg: string, meta?: Record<string, unknown>) => void };
};
export async function computeSplitEnabled(
config: SplitModeConfig,
deps: SplitModeDeps = {}
): Promise<boolean> {
// Hard gate #1: explicit positive opt-in. OFF by default -> never probe.
if (!config.flagEnabled) {
return false;
}
// Both URLs are required to even consider a split.
if (!config.legacyUrl || !config.newUrl) {
deps.logger?.warn(
"RUN_OPS_SPLIT_ENABLED is on but RUN_OPS_LEGACY_DATABASE_URL / RUN_OPS_DATABASE_URL are not both set; staying single-DB."
);
return false;
}
// Hard gate #2: runtime sentinel must confirm physically-distinct DBs. At N stores this is set
// uniqueness over every store that owns its own database, not a compare of the gen-1 pair. An
// aliased shard is already absent from `shards` — it shares its target's client by reference.
const probe = deps.probe ?? defaultProbe;
const targets = [
{ id: "legacy", url: config.legacyUrl },
{ id: "new", url: config.newUrl },
...(config.shards ?? []).map((shard) => ({ id: `shard-${shard.key}`, url: shard.url })),
];
const result = await probe(targets, { logger: deps.logger });
return result.distinct === true;
}
export type SplitRealtimeInterlockConfig = {
splitEnabled: boolean;
nativeRealtimeEnabled: boolean;
};
/**
* Boot-time realtime interlock (pure predicate). Split mode puts NEW-resident
* (run-ops id) runs on the dedicated run-ops DB, but Electric replicates only from the
* control-plane DB — with the native realtime backend OFF those runs are invisible
* and every realtime subscription hangs. Refuse split unless native is on; split-off
* is always allowed regardless of the realtime backend.
*/
export function assertSplitRealtimeInterlock(config: SplitRealtimeInterlockConfig): void {
if (!config.splitEnabled) {
return;
}
if (!config.nativeRealtimeEnabled) {
throw new Error(
"RUN_OPS_SPLIT_ENABLED is on but the native realtime backend (REALTIME_BACKEND_NATIVE_ENABLED) is not enabled — Electric cannot serve NEW-resident runs; refusing to enable split."
);
}
}
let cached: Promise<boolean> | undefined;
export function isSplitEnabled(): Promise<boolean> {
if (!cached) {
cached = computeSplitEnabled(
{
flagEnabled: env.RUN_OPS_SPLIT_ENABLED,
legacyUrl: env.RUN_OPS_LEGACY_DATABASE_URL,
newUrl: env.RUN_OPS_DATABASE_URL,
shards: nonAliasedShards(env.RUN_OPS_SHARDS),
},
{ logger }
);
}
return cached;
}
function __resetSplitModeCacheForTests(): void {
cached = undefined;
}