Files
triggerdotdev--trigger.dev/docker/scripts/entrypoint.sh
T
Daniel Sutton 1801b0e80b feat(webapp,docker): run-ops boot interlocks and migrations at N databases (#4780)
## Summary

The run-ops boot interlocks and the migration entrypoint each assume
exactly two run-ops
databases. This generalizes them to any number, so a deployment that
configures
`RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two
stores: no two stores
may point at one database, every store that owns its own database must
replicate to
ClickHouse, and every store must have its schema migrated.

With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check
over a two-element set is
the pairwise compare it replaces, replication coverage is the check it
was, and the entrypoint
runs the same two migration invocations.

A shard may declare `aliasOf: "new"`, which shares an existing store's
client by reference. An
aliased shard is not its own database, so it is exempt from the
distinctness check and needs no
replication slot of its own. Every check keys that exemption on the
declared field, never on
client object identity: two client objects can sit over one database,
which identity comparison
cannot see.

## Design

**Distinctness.** `probeDistinctDatabases` compared two URLs. It now
delegates to
`probeDistinctStores`, which reads every fingerprint in parallel and
groups them by system
identifier and database name. Any two stores under one key refuse the
boot. The old pairwise
entry point stays, so its existing container tests are the proof that
set uniqueness over one
pair gives the verdict it gave before. Fail-closed is unchanged: a probe
that cannot answer
returns not-distinct, because "distinct" is a positive claim a failed
probe cannot support.

**Co-residency.** The advisory runs once per store against the control
plane. The legacy
emission keeps its exact call shape and its untagged metric series, so
an existing dashboard
does not change. Each shard emits its own point carrying its shard key.
Every store emits
before any enforcement throw, so one offending store never costs another
store its metric.

**Replication.** `buildReplicationSources` appends one source per shard
that owns its own
database, taking the slot, publication and origin generation its
descriptor declares.
`assertReplicationCoversSplit` then requires a source per such shard.

That check also closes a hole it inherited. The descriptor parser
validates uniqueness among
shards only, so a shard could take the slot name, publication name or
origin generation of the
legacy or the new source. The replication service does validate this,
but it throws from its
constructor, and the caller reaches that constructor only after shutting
the bootstrap instance
down:

```ts
if (sources.length > 1) {
  await service.shutdown();                       // legacy stream stops here
  service = new RunsReplicationService({ ... });   // throws: duplicate slotName
}
```

The throw was not a `SplitReplicationMisconfiguredError`, so the process
stayed up with no
replication at all, legacy included, behind one logged line. That is the
silent ClickHouse
under-count the error exists to prevent. The check now runs at the boot
gate, before anything is
torn down, and raises a subclass the existing exit path already
recognizes. A correct deployment
already satisfies it, because two consumers on one WAL slot is a data
race that cannot work.

**Migrations.** Every shard runs the identical schema, so a new shard is
the existing migrations
against a new DSN. The runner image has no `jq`, so a small node script
prints one DSN per line
and the entrypoint loops over them. The loop is a `for` and not a `while
read` pipeline: a
pipeline subshell swallows a failed migration on any iteration but the
last, which would let a
broken shard boot. Tracing stays off across the capture and the loop,
because `set -x` prints an
assignment and a DSN carries credentials.

Verified end to end against real Postgres containers for the fingerprint
probes, and against the
real shell block with a stubbed migration command: an aliased shard is
skipped, `directUrl` wins
over `url`, a failing shard stops the container on the first failure,
and a malformed descriptor
stops it before it migrates anything.

Stacked on #4764.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-26 13:50:20 +01:00

141 lines
5.5 KiB
Bash
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/bin/sh
set -xe
if [ -n "$DATABASE_HOST" ]; then
scripts/wait-for-it.sh ${DATABASE_HOST} -- echo "database is up"
fi
if [ "$SKIP_POSTGRES_MIGRATIONS" != "1" ]; then
echo "Running prisma migrations"
pnpm --filter @trigger.dev/database db:migrate:deploy
echo "Prisma migrations done"
else
echo "SKIP_POSTGRES_MIGRATIONS=1, skipping Postgres migrations."
fi
# Run-ops split: migrate the dedicated NEW run-ops database only when it is configured. Single-DB
# installs never set the URL, so this is a no-op there.
{ set +x; } 2>/dev/null
if [ -n "$RUN_OPS_DATABASE_URL" ]; then
set -x
if [ "$SKIP_RUN_OPS_MIGRATIONS" != "1" ]; then
echo "Running run-ops migrations"
pnpm --filter @internal/run-ops-database db:migrate:deploy
echo "Run-ops migrations done"
else
echo "SKIP_RUN_OPS_MIGRATIONS=1, skipping run-ops migrations."
fi
else
set -x
echo "RUN_OPS_DATABASE_URL not set, skipping run-ops migrations."
fi
# Run-ops split: keep the legacy runs DB's schema current by applying the full @trigger.dev/database
# migrations to it too, pointed at its direct (non-pooled) URL. Only runs when that URL is configured;
# installs that never set it skip this entirely.
{ set +x; } 2>/dev/null
if [ -n "$RUN_OPS_LEGACY_DIRECT_URL" ]; then
set -x
if [ "$SKIP_RUN_OPS_LEGACY_MIGRATIONS" != "1" ]; then
echo "Running legacy run-ops migrations"
# Subshell with tracing off so `set -x` does not print the DSN (with credentials) to the logs.
(set +x; DATABASE_URL="$RUN_OPS_LEGACY_DIRECT_URL" DIRECT_URL="$RUN_OPS_LEGACY_DIRECT_URL" pnpm --filter @trigger.dev/database db:migrate:deploy)
echo "Legacy run-ops migrations done"
else
echo "SKIP_RUN_OPS_LEGACY_MIGRATIONS=1, skipping legacy run-ops migrations."
fi
else
set -x
echo "RUN_OPS_LEGACY_DIRECT_URL not set, skipping legacy run-ops migrations."
fi
# Run-ops shards: migrate every gen-2 shard that owns its own database. Each shard runs the
# identical schema, so this is the existing run-ops migrations against a new DSN. An aliased shard is
# skipped by the DSN script: it IS its target's database. Installs that never set RUN_OPS_SHARDS
# skip this entirely.
{ set +x; } 2>/dev/null
if [ -n "$RUN_OPS_SHARDS" ]; then
set -x
if [ "$SKIP_RUN_OPS_SHARD_MIGRATIONS" != "1" ]; then
echo "Running run-ops shard migrations"
# Tracing stays OFF from here to the end of the loop: `set -x` prints an assignment, so
# capturing a DSN under tracing would put the credentials in the logs.
{ set +x; } 2>/dev/null
# A malformed descriptor exits 1 here, so the container stops before it migrates anything.
shard_dsns=$(node scripts/runOpsShardDsns.mjs)
# A `for` loop and NOT `... | while read`: a pipeline subshell would swallow a failed migration
# on any iteration but the last. Here `set -e` stops the boot on the first shard that fails.
# The whole loop runs in a subshell, so the IFS and `set -f` changes need no restore and cannot
# leak into the rest of the entrypoint. IFS is newline-only so a DSN is never split on other
# whitespace, and `set -f` stops a DSN query string (it holds `?`) from acting as a glob.
(
IFS='
'
set -f
for shard_dsn in $shard_dsns; do
# Tracing stays off so `set -x` never prints the DSN (with credentials) to the logs.
RUN_OPS_DATABASE_URL="$shard_dsn" DIRECT_URL="$shard_dsn" pnpm --filter @internal/run-ops-database db:migrate:deploy
done
)
set -x
echo "Run-ops shard migrations done"
else
echo "SKIP_RUN_OPS_SHARD_MIGRATIONS=1, skipping run-ops shard migrations."
fi
else
set -x
echo "RUN_OPS_SHARDS not set, skipping run-ops shard migrations."
fi
if [ "$SKIP_DASHBOARD_AGENT_MIGRATIONS" != "1" ]; then
echo "Running dashboard agent migrations"
pnpm --filter @internal/dashboard-agent-db db:migrate:deploy
echo "Dashboard agent migrations done"
else
echo "SKIP_DASHBOARD_AGENT_MIGRATIONS=1, skipping dashboard agent migrations."
fi
{ set +x; } 2>/dev/null
if [ -n "$CLICKHOUSE_URL" ] && [ "$SKIP_CLICKHOUSE_MIGRATIONS" != "1" ]; then
# Run ClickHouse migrations
echo "Running ClickHouse migrations..."
export GOOSE_DRIVER=clickhouse
# Ensure secure=true is in the connection string
if echo "$CLICKHOUSE_URL" | grep -q "secure="; then
# secure parameter already exists, use as is
export GOOSE_DBSTRING="$CLICKHOUSE_URL"
elif echo "$CLICKHOUSE_URL" | grep -q "?"; then
# URL has query parameters, append secure=true
export GOOSE_DBSTRING="${CLICKHOUSE_URL}&secure=true"
else
# URL has no query parameters, add secure=true
export GOOSE_DBSTRING="${CLICKHOUSE_URL}?secure=true"
fi
export GOOSE_MIGRATION_DIR=/triggerdotdev/internal-packages/clickhouse/schema
/usr/local/bin/goose up
echo "ClickHouse migrations complete."
elif [ "$SKIP_CLICKHOUSE_MIGRATIONS" = "1" ]; then
echo "SKIP_CLICKHOUSE_MIGRATIONS=1, skipping ClickHouse migrations."
else
echo "CLICKHOUSE_URL not set, skipping ClickHouse migrations."
fi
set -x
# Copy over required prisma files
cp internal-packages/database/prisma/schema.prisma apps/webapp/prisma/
cp node_modules/@prisma/engines/*.node apps/webapp/prisma/
cd /triggerdotdev/apps/webapp
# Decide how much old-space memory Node should get.
# Use $NODE_MAX_OLD_SPACE_SIZE if its set; otherwise fall back to 8192.
MAX_OLD_SPACE_SIZE="${NODE_MAX_OLD_SPACE_SIZE:-8192}"
echo "Setting max old space size to ${MAX_OLD_SPACE_SIZE}"
NODE_PATH='/triggerdotdev/node_modules/.pnpm/node_modules' exec dumb-init node --max-old-space-size=${MAX_OLD_SPACE_SIZE} ./build/server.js