Enabling Docker took three steps because the production image runs as `node` and
the socket is mode 660 root:docker, so a matching group was required. The group id
is host-specific — 0 on Docker Desktop, commonly 999 on Debian/Ubuntu, 993 on
Amazon Linux 2023 — and a wrong value is a silent EACCES that shows up as "Docker
never appears in the dashboard". There is no value that is right everywhere, so
there is no default to ship.
The container can read it off the socket. The runner stage no longer sets
`USER node`; a new entrypoint starts as root, takes the socket's group with
`stat -c %g`, joins it, and execs the command through `su-exec node`. The app
process is still `node` and `tini` is still PID 1. With no socket mounted it
drops to `node` and does nothing else, and a deployment that pins `user:` is
left alone.
`group_add` and `DOCKER_GID` are gone from all five compose files, `.env.example`,
the docs in four languages, the dashboard guide (three steps to two) and the
not-configured API message. Uncommenting one volume line is now the whole edit.
The mount stays opt-in: the socket is root-equivalent on the host, and on by
default would hand every deployment's backend the ability to take over its host
whether or not it uses compute.
Verified on a build of `target: runner` — no socket: uid 1000; socket at gid 0:
joins root, writable; socket at gid 999 with the group already present: joins it,
writable; and the old `USER node` path with no entrypoint: denied, which is the
failure the removed step existed to avoid. Then end to end from that image
against a live database: provider ready, metadata slice present, a service
created and serving on its published port.
Also batches the compute logs limiter test. 120 sequential requests overran the
10s timeout on a busy machine and 120 simultaneous ones had connections dropped;
ten in flight is neither, and the assertion is by count rather than position.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add umask 077 so pg_dump output and backups/ are not world-readable.
chmod +x deploy/backup.sh after the HTTPS fetch path in setup.sh so
INSFORGE_NO_GIT installs can run ./deploy/backup.sh.
Ship a maintained backup script that pg_dump's Postgres and copies .env,
include it in the setup.sh sparse checkout manifest, and point the
deployment security guide at the script instead of an inline copy-paste.
Both pages start ~/insforge on the default ports, then hand project1 those same
ports in the multi-instance section. Anyone reading either page end-to-end brings
up the second one into a bind failure.
Same note in both, since the README carries a copy of this section.
`cp .env.example .env.project1` gave every instance the placeholder secrets the
file ships — the same published JWT_SECRET on each, plus POSTGRES_PASSWORD=postgres
and ROOT_ADMIN_PASSWORD=change-this-password. The note below it asked readers to
change JWT_SECRET and ROOT_ADMIN_PASSWORD, which is two of the six and reads as
hardening rather than as "these are public".
Running setup.sh into a directory each generates a real set per instance, and
matches what the README already says. Verified: two directories get different
secrets, neither placeholder, and compose resolves them as separate projects.
The --env-file form goes with it — one .env per directory means plain
`docker compose` from inside it, which is what COMPOSE_FILE is set up for.
- "re-running leaves an existing .env untouched" was wrong: pin_compose_file
adds COMPOSE_FILE, and repoints it when it still names the development stack.
Fixed in setup.sh too — its own header and success message carried the same
sentence, and a script that contradicts the README is worse than either.
- The storage section put both overlay choices in one env block. They are shell
assignments to the same variable, so copying it verbatim selects RustFS and
silently discards the MinIO line. Split, with a note that only one survives.
- The source build passes -f explicitly, which overrides COMPOSE_FILE — so the
COMPOSE_FILE instructions above do not apply to it. Said so, and pointed at
further -f flags instead.
- prod.yml also reads ACCESS_API_KEY and ACCESS_ANON_KEY, which the secret list
omitted. Left empty the backend generates a pair only it knows, which is safe
but leaves you unable to authenticate against your own install.
- MD040 on the two project .env fences.
Verified both deploy sources on a cold EC2 host (AL2023, Docker 25.0.14, the prod
compose built from this branch with the socket mount and group_add uncommented as
the file instructs) and fixed what the fourth review round found.
Live results: with the images deleted first, the image path pulled and served on the
published loopback port while refusing the host's private IP, and the source path
built through the classic builder and ran the tag it produced -- the case BuildKit
cannot do at all. Paging a 12-line backlog at limit=5 from an old cursor delivered
1-5, 6-10, 11-12, then empty, which is the log fix confirmed against a real daemon.
The three guards behaved: 413 naming the knob, 429 for a concurrent upload, 400 for
a bad dockerfile path. Migrations re-ran twice with provider and ingress both
NOT NULL, prune held at two images, reconcile corrected nothing.
What the live run could not prove: I tried to stall an upload to exercise the idle
watchdog and failed to construct a stall -- curl sent its partial body and saw EOF,
and a truncated 2048-byte tar still holds the whole Dockerfile, so it built fine.
That branch needs socket-level control, which is a unit test.
So: two tests that drive a real listener with an underfilled Content-Length. One
drops the socket while the request is parked in express.raw and asserts the slot is
handed back; one sets the timeout to its smallest value and asserts the watchdog
destroys the request and frees the slot. Both verified by reverting the branch.
The 30s timeout was hard-coded while the size limit beside it was config-driven; it
is now COMPUTE_BUILD_UPLOAD_IDLE_TIMEOUT (seconds, default 30), forwarded through
all five compose files.
Two earlier calls of mine were wrong and are corrected here. COMPUTE_ISOLATE_NETWORK
used `=== 'true'`, which I dismissed twice by checking the codebase-wide convention;
app.config.ts has its own parseEnvBool accepting 1/true/yes/on six lines above, so
mine was the outlier, and a knob that silently ignores `=1` fails in the unsafe
direction. And following my own DOCKER_GID instruction on the live host appended a
duplicate key beside .env.example's placeholder -- compose takes the last so it
worked, but the instruction now says to fill in the existing line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six real findings out of 30 comments. The best was a critique of a test added in
the previous commit.
The new first-page log test asserted a shape no daemon produces: it mocked a
five-line body for a request carrying `tail=2`, then asserted the oldest two.
`tail=2` makes a real daemon return the newest two, so the trimming branch is
unreachable on a first page -- the product code was right, the test was theatre
and would have passed through a regression. Replaced with a mock that derives its
response from the request (`tail` -> newest N, `since` -> everything after that
second): one test for first-page semantics, one that pages a five-line backlog at
limit 2 and asserts every line arrives exactly once. Restoring the old
tail-plus-slice(-limit) code makes it fail with ['line-3','line-4'] while 0-2
never appear.
endpointUrl was persisted only alongside a replacement. That holds for Docker,
where `ingress` is in the spec hash so an in-place update cannot move the
published port, but nothing in the provider contract ties `endpointUrl` to a new
instance id -- a driver that re-publishes in place would have its URL dropped.
A build context was buffered whole before any concurrency check. The route
borrowed maxJsonBodySize (100mb) and the driver's build cap only rejects once the
body is in memory, so N concurrent uploads cost N contexts; on a t4g.nano's
~418MB that is an OOM rather than a 429. Adds COMPUTE_BUILD_MAX_CONTEXT (default
64mb, forwarded through all five compose files for the same reason
DOCKER_SOCKET_PATH had to be) and a gate ahead of express.raw that turns away a
second upload without reading it. The gate calls req.resume(): answering while
the request body sits unread stalls the connection, which a hanging test found
and which is equally true in production.
Built images leaked when the build succeeded but the deploy failed, because the
prune came after updateService. Moved into a finally. Docker refuses to delete an
image a container is using (409, already logged and skipped), so this cannot pull
the running image out from under a failed deploy.
Reconcile's status-correction write was unscoped. healMachineGone already carries
`AND provider_instance_id = $2`, but the correction UPDATE beside it did not, so a
stale `stopped` reading could overwrite the `running` a concurrent deploy had just
committed. Same guard added.
The `dockerfile` query parameter reached the daemon unvalidated. Now rejects
absolute paths, `..` segments, empty values, and anything over 255 characters with
a 400 that says what to fix.
Also moves the mock provider's capability restore into beforeEach, so a test that
throws mid-body cannot leak its override into the rest of the file.
Eighteen new tests, including the first HTTP-boundary coverage for
POST /:id/build -- body parse, content type, tenant check, builder failure, every
`dockerfile` shape, and the concurrency gate. Both new guards were verified by
reverting the fix and watching the test fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight fixes from the PR review round.
resolveNetwork() cached its own failure: the catch wrote null into the same
field as the success path, so one daemon blip during the first launch detached
every later container from the project network until the process restarted --
the exact outcome the lookup exists to prevent.
getLogs paired `tail` with `since`. `tail` keeps the newest N, so 500 lines
since the cursor with tail=100 returned the newest 100 and then advanced the
cursor past the other 400, which no later request could reach. `tail` is now
sent only on the first page; a resume sends `since` alone and returns the
oldest `limit`, so the next page picks up where the last stopped. Verified
against a real daemon that logs come back oldest-first and that `since`
without `tail` returns the whole window.
A recreate left status stale: the replacement branch persisted the new instance
id and endpoint URL but not the status, so stop -> redeploy reported 'stopped'
while the container ran.
Migration 064 applied NOT NULL to `provider` with the single
ADD COLUMN IF NOT EXISTS ... NOT NULL DEFAULT form, which applies neither the
backfill nor the constraint when the column already exists -- the failure mode
the comment below it explains for `ingress` and defends against with a split
add/backfill/constrain. Both columns now use that pattern. Reproduced against a
pre-existing nullable column: the old file left it nullable with a NULL row;
the new one constrains and backfills, idempotently.
DOCKER_SOCKET_PATH existed only in .env.example -- no compose file passed it,
so overriding it did nothing. It now reaches the backend and drives both sides
of the (still commented) mount, which is also what makes rootless Docker and
Podman configurable.
The DOCKER_GID instruction showed a bare shell assignment, which Compose never
sees; it now says the value has to land in .env. The cold-host verification
round only worked because that was done by hand.
COMPUTE_DOMAIN was missing from the prod file and all three deploy variants, so
`host` ingress could not produce a URL on any path a self-hoster uses.
A failed heal aborted the reconcile sweep: healMachineGone was awaited inside
the catch, so a DB error escaped the loop and abandoned every remaining row.
Six new unit tests cover the behavioural four. The compose changes are asserted
against `docker compose config --format json` on all five files rather than by
grep, which is how a hunk previously landed in the wrong service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rollback section and setup.sh's header had grown more explanation than
instruction. Section 16.2 is four numbered steps now, with one sentence on the
consequence of skipping the last one; the three paragraphs justifying the
approach are gone. setup.sh's header keeps the usage lines and the environment
variables and drops the reasoning behind them, and the same trim runs through the
comments inside — 259 lines to 243.
INSFORGE_REF now says what it accepts: a tag, branch or commit. It takes all
three (raw.githubusercontent.com resolves each, and git clone --branch covers the
first two), and the previous wording implied releases only.
Also removes a stray blank line and shortens the two access-key comments in
.env.example, left over from the variables that came out of this branch.
Re-ran the setup.sh matrix after the trim, since comment edits in a shell script
are not as safe as they look: both acquisition modes, ref pinning in each, all
three guards, and the fail-closed path — 10 checks, no failures. The structural
checks pass too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two P1s from cubic, both in the commit before this one.
`INSFORGE_NO_GIT=1` with `INSFORGE_REF` set — the combination the header
documents — failed with "fatal: not a git repository". Moving checkout_ref after
the guard put it on the HTTPS path too, where there is nothing to fetch into and
the files were already retrieved at that ref. It returns early when NO_GIT is
set. Pinning to v2.2.9 over HTTPS now lands that release's files, which is
verifiable: its compose file still names postgres-all.
The working-tree guard tested `-d "$TARGET/.git"`. In a linked worktree .git is a
file, so the guard read as "no repository here" for exactly the tree it most
needs to catch — and this is the second time in this series that a .git test had
to change from -d to -e.
deploy/setup.sh also comes out of FILES. The script belongs in the checkout so it
travels with it for the update procedure, which is a git-path concern; on the
HTTPS path the caller already holds the script, and any ref older than this file
404s on it and fails the whole fetch. The sparse patterns add it, FILES does not.
And scripts/check-setup-sh.sh, run in CI, asserts the structure. Every rule in it
guards an invariant that was broken after being established: secrets written with
set_var rather than gen_secret (twice), an operation inserted ahead of the guard,
a .git test narrowed to -d, a git call left on the no-git path. None of those show
up in a diff review or a happy-path run.
The checks are mutation-tested: each of the seven rules was verified to fail when
the corresponding bug is reintroduced into a copy of the tree, and to pass on the
tree as it stands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Self-hosted InsForge runs its data locally but has only ever been able to run
custom containers on Fly.io — an operator's Postgres and storage sit on their box
while their compute sits in someone else's account. This adds a Docker driver
behind the compute API that already exists, so containers run on the operator's
own host.
Phase 0 — provider-neutral seam
Fly-specific naming (app names, 6PN network, .fly.dev URLs) moves out of the
service layer into shared helpers. Providers now declare a ComputeCapabilities
descriptor instead of the service hardcoding Fly's answers, which is what made a
second driver impossible to add without lying somewhere.
Migration 064 renames fly_app_id/fly_machine_id to provider_app_id/
provider_instance_id and adds `provider` and `ingress` columns. `provider` is
load-bearing: without it an operator who enables Docker leaves Fly-backed rows
that reconcile would try to heal against the wrong daemon.
Drivers coexist. A service is managed by the driver that created it, routed by
its `provider` column, so COMPUTE_DRIVER selects the default for *new* services
rather than restricting the deployment — an operator with Fly services keeps them
running and manageable while new work lands locally.
Phase 1 — the driver
Dependency-free client over the unix socket (node's http supports socketPath).
Full lifecycle, plus the parts that are easy to get silently wrong: the log
stream's 8-byte frame headers are demuxed, and the forward cursor is two-part
because `since` accepts integer seconds only and is inclusive — a nanosecond
watermark dedupes, the request floors it to seconds.
Every container carries insforge.managed/project labels and every read and write
filters on them. This is not defensive politeness: from inside a container with
the socket mounted, an unfiltered listing includes Postgres and the backend
itself, so an unscoped stop takes out the database.
The driver constructs each container spec itself and never forwards a
caller-supplied HostConfig, so there is no field through which a leaked API key
could request --privileged or a host bind mount.
Ingress is per-service and defaults to none, because most compute takes no
inbound traffic at all. Published ports bind loopback by default — Docker's own
default is 0.0.0.0 and [::], i.e. the whole internet.
Startup runs a preflight probe and a reconcile sweep. exited maps to stopped
regardless of exit code: a host reboot leaves SIGKILLed containers at 137, and
treating that as failure would mark a chunk of services failed after every boot.
Phase 2 — source build
Uploaded context tarball streamed straight to the daemon's build endpoint.
BuildKit is used without decoding its protobuf progress frames, which is
affordable because a failed build reports the error as plain JSON in the body —
and it returns HTTP 200 while doing so, so status alone must not be trusted.
Superseded images are pruned after each successful deploy.
Known v1 limitation: no persistent volumes (the Fly path has none either).
Container state survives restarts and host reboots but not a redeploy, which
recreates the container. Use the project's Postgres or Storage for data to keep.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs from the round of review on the last commit, both mine, both
reintroducing something this series already fixed once.
greptile caught the ordering: checkout_ref ran inside the acquisition branch,
before the guard that refuses to act on a development checkout. So
`INSFORGE_REF=v2.2.9 sh deploy/setup.sh <a dev clone>` detached that clone's
HEAD at the ref and *then* exited with an error — the guard reporting damage it
was placed there to prevent. It runs after the guard now, and a dev clone comes
out still on its branch.
coderabbit caught the second: I wrote the two access keys with
`set_var ACCESS_API_KEY "ik_$(openssl rand -hex 20)"` rather than through
gen_secret, so a failing openssl left `ACCESS_API_KEY=ik_` — non-empty, so
Compose passes it, and the backend seeds three characters as the instance's
superadmin key. That is the fail-open path gen_secret exists to close, and worse
than the placeholder it replaced. gen_secret takes an optional prefix now and
applies it on the far side of the emptiness check.
Also: RAW derives from an SSH remote as well, since `git@github.com:owner/repo`
left the no-git path with a URL curl cannot fetch.
Verified: a dev clone keeps its branch when the ref is rejected, a stubbed-out
openssl leaves no .env behind, both remote forms produce the right raw host, and
a normal install still lands twelve files with generated prefixed keys.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both guards said the target "is a full checkout, not a self-hosting one", which
assumes the git working tree it found is a clone of this repository. Someone who
runs this at the root of their own application's repository hits the same guard
and reads a sentence about a checkout they do not have.
The messages name the consequence instead: fetching would overwrite tracked
files including uncommitted changes, and a sparse checkout applied there would
empty the tree of everything outside this script's file list.
Demonstrated the case the no-git guard closes: with it removed, running against a
git working tree replaced a modified, uncommitted functions/server.ts with the
version from the network and left eight untracked files behind. curl does not
ask, warn, or back anything up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cubic caught a P0 I introduced. `.env.example` carried
`ACCESS_API_KEY=ik_your-api-key-here-32-chars-minimum`, and setup.sh copies that
template verbatim — so the moment this branch made the compose file pass the
variable through, every fresh install seeded a superadmin key printed in a public
file. That is the same shape as the placeholder secrets I spent this series
removing, and I added it.
The template's two keys are empty now, and setup.sh generates them alongside the
other four secrets. Generating rather than leaving them empty is deliberate: the
backend would make its own, but then only it would know them, and an install with
no credentials to hand out cannot be driven by the CLI or any script.
Verified on a running stack: the generated key answers 200 on
/api/database/tables and the placeholder answers 401.
Also from that round of review:
- INSFORGE_REF was honoured only by the first clone, so re-running against an
existing checkout with a different ref reported success and changed nothing —
reported separately by three reviewers. It fetches and detaches onto the ref
now, and git refuses rather than discarding local edits.
- The no-git path skipped the guard that refuses to act on a development clone,
where it would have curl-overwritten tracked files in place with no prompt.
- NO_GIT is initialized, so an inherited variable of that name from the caller's
environment cannot skip acquisition and leave the rest running against
whatever directory invoked the script.
- RAW derives from INSFORGE_REPO, so pointing at a fork no longer fetches the
official files while cloning nothing.
- The header's example pinned v2.2.9, the one ref this PR documents as carrying
the broken postgres-all config. It reads vX.Y.Z.
- functions/examples/ is absent where the old directory pattern swept it in.
Noted as deliberate: the runtime loads server.ts, and demo functions are not
something a self-host install needs on disk.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
INSFORGE_OSS_VERSION and INSFORGE_DEPLOYMENT_METHOD do not earn their place. A
caller can already override an image and add environment variables the compose
file never mentions, by appending an overlay to COMPOSE_FILE — the mechanism the
storage overlays use, documented in .env.example, needing nothing from here:
services:
insforge:
image: ghcr.io/insforge/insforge-oss:v2.2.9
That is strictly better than the variable it replaces: it pins any image in the
stack, not only this one.
I also overstated the case for the version variable. I said a rollback that
edited the compose file would leave the install unable to update, because
`git merge --ff-only` refuses when a tracked file has local changes. It refuses
only when upstream also changed that line; otherwise the merge succeeds and
keeps the edit. The overlay avoids the question entirely by being a file of the
operator's own, and section 16 now teaches that instead.
Dropping the telemetry stamp also avoids a behaviour change nobody asked for:
with a `docker-compose` default, every existing self-host would have started
reporting `docker-compose` where it reports `docker` today.
ACCESS_API_KEY and ACCESS_ANON_KEY stay, for a reason that has nothing to do
with any caller: .env.example documents both as settable, and this compose file
silently dropped them, so a self-hoster who set them saw no effect. That is the
same inconsistency as section 5.4 telling people to set AWS_* variables the file
does not pass.
Verified on a running stack: the overlay pins insforge-oss to v2.2.9, adds
INSFORGE_DEPLOYMENT_METHOD which the base file never names, and the injected
ACCESS_API_KEY answers 200 on /api/database/tables.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`insforge local start` carries its own 266-line compose template, a copy of
this one that has already drifted: the connection-pool alignment added here
yesterday never reached it. The two files describe the same stack — same
services, and 6 environment variables apart out of ~50 — so the copy exists
only because three values it needs are not variables here.
They are now, each defaulting to what this file already did:
- `INSFORGE_OSS_VERSION` pins the insforge-oss image, empty meaning latest.
Named for the image rather than the stack because that is what it pins;
Postgres, PostgREST and Deno carry their own upstream versions in this file.
- `ACCESS_API_KEY` / `ACCESS_ANON_KEY` reach the backend, which seeds from them
when set and generates its own when not. Self-hosters could not previously
drive their instance from a machine that already knew the keys — the CLI
needs it, and so does anyone scripting against their own install.
- `INSFORGE_DEPLOYMENT_METHOD` defaults to docker-compose, which is what this
file is. A caller wrapping it says what it is instead, which is how telemetry
tells the deployment paths apart.
setup.sh grows two ways to be driven:
- `INSFORGE_REF` stays on one ref instead of tracking main. Self-hosting had no
way to pin a release at all.
- `INSFORGE_NO_GIT=1`, or no git on PATH, fetches the files over HTTPS instead
of cloning. 34KB against 47MB for the repository tarball, and precise where a
tar glob is not — `*/functions` also matches backend/src/.../functions, which
extracted 400KB of unrelated files when I tried it. This loses the update
path, which needs a checkout to diff and merge into.
Both modes read one FILES list, so they cannot disagree about what the stack
needs. That makes the sparse patterns file-level rather than the directory-level
ones they were: a release that mounts a new file adds it there, which the
comment above the list already required.
Verified on both paths: each lands the same twelve files in the same layout, the
compose file parses with its relative mounts resolving, and a stack started from
the result reports 2.2.9 with the image pinned to v2.2.9, the injected API key
answering 200 on /api/database/tables, INSFORGE_DEPLOYMENT_METHOD arriving as
cli-local, insforge_pg_utils preloaded, and CREATE POLICY as project_admin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tested on real instances of both platforms, because the mandatory-variable
change from the last two commits had never been through one.
Dokploy honours `${VAR:?message}` exactly as intended: the deploy stops and
the operator's log carries each message verbatim.
Coolify does not. Its parser treats `:?` like `:-`, seeding the environment
row with the message text as the variable's value — so the deploy either starts
on a known string or, as it did here, dies on an unrelated error: my message
contained an apostrophe ("platform's"), which broke the single-quoted postgres
command line and produced 'services[postgres].command' invalid command line
string. Measured on Coolify 4.1.2.
So Dokploy keeps the mandatory form and Coolify goes back to defaults, now
spelled REPLACE_ME_* so the value is self-describing wherever an operator sees
it. The messages lost their apostrophes and parentheses either way — a message
that can be substituted into a command line has no business containing quotes.
The uniform fix remains the one @jwfing proposed: have the backend refuse to
boot on known placeholder secrets. That covers Coolify, Zeabur and cloud at
once, where a compose-level guard only reaches the platforms whose parser
respects it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The steady trickle of small findings has one cause: I have been fixing this
corpus with grep-driven sweeps, and every pattern has been narrower than the
corpus. 30 of these files are near-duplicate translated prose, so each logical
fix landed in twelve files and missed four — the variants being case
(`~/InsForge` vs `~/insforge`), indentation (fences nested in list items),
translated phrasings that do not match the English anchor, and code comments
rather than prose. Reviewers then found the remainder, one batch at a time.
So this pass wrote the invariants down and checked all 34 changed deploy files
against them mechanically. Thirteen rules, ten violations, all fixed:
- Azure's update commands used `~/InsForge` while its setup step creates
`~/insforge` — case-sensitive on Linux, so the documented update fails before
it starts. Four locales.
- The image-only compose probed `pg_isready -U postgres` with POSTGRES_USER
configurable one line above, so overriding the user left Postgres permanently
unhealthy and, since insforge gates on it, never started the backend. The same
class of bug as the postgrest healthcheck, in the same file. Verified: with
POSTGRES_USER=ifadmin the container now reaches healthy.
- The update procedure's review diff covered `deploy` and `.env.example` but not
`functions/`, which is in the sparse checkout and which the Deno service
executes. Four locales.
- Section 5.2 still told the reader to set POSTGRES_PASSWORD, two sections after
4.2 started the stack — and Postgres only reads it at cluster init, so the
instruction could not work. It says so now. Four locales.
- setup.sh only added COMPOSE_FILE when absent, so a .env copied from the
template by hand kept pointing at the development stack, whose files the
checkout does not contain. It now repoints that exact value, and leaves
anything else — including a hand-appended storage overlay — alone.
- MD040 on two fences, and the update prose that called an update "a pull and
restart" when the checkout carries Postgres's config and the Deno functions.
The audit also caught something no reviewer did: .env.example's COMPOSE_FILE has
to stay equal to what Compose discovers on its own in that directory, which
changed when docker-compose.override.yml was deleted earlier in this branch.
It is equal — verified byte-for-byte against auto-discovery — and setup.sh's
self-heal constant now has to match it, with both sides commented.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
From cubic's pass, after reproducing each:
- setup.sh's migration branch exits before the fresh-install path's `chmod 600`,
so an install whose old .env was group-readable stayed that way. It now
chmods right after the move; verified by handing it a 644 file.
- The multi-instance section had `cp .env.example .env.projectN` with no further
instruction, and .env.example now ships COMPOSE_FILE's development value —
a file the sparse checkout does not contain. Each copy needs the production
path.
- ROOT_ADMIN_PASSWORD is mandatory in the platform files now, alongside
JWT_SECRET and ENCRYPTION_KEY. Unlike POSTGRES_PASSWORD there is no migration
hazard: auth.service.ts hashes it from the environment on every boot rather
than persisting a record, so changing it takes effect on restart. A dashboard
reachable from the internet on `changeme123` is the most directly exploitable
of the three placeholders.
- The zh, zh-Hant and es guides' section 5.5 still listed AWS_S3_BUCKET/AWS_*,
which the compose file does not pass, while English had moved to S3_*. The
note I added to 5.4 pointed at "the S3_* variables in 5.5" — which did not
exist in those three. Aligned on the names the compose actually reads.
- GCE's backup snippet still said "run from deploy/docker-compose/" in a code
comment, which the earlier path sweep did not match.
- Containarium's agent flow wrote the whole of ~/insforge/.env, overwriting the
secrets setup.sh had just generated two steps earlier.
Also updates the backend skill note, which claimed the two platform files differ
only in INSFORGE_DEPLOYMENT_METHOD; the build context differs too, since Coolify
builds with --project-directory and Dokploy does not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second round of @jwfing's review raised the deno env asymmetry between the
three compose files. It is real, but the resolution goes the other way: the
platform files passed AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_BUCKET,
AWS_REGION and S3_ENDPOINT_URL to the deno service, and functions/ reads none
of them — zero references in either server.ts or worker-template.js. Nor could
function code pick them up: the worker is constructed with `env: false`
(server.ts:170), the two-tier sandbox from #1137, and secrets reach it through
the message payload instead. So they were dead config rather than exposure.
Removed, and the deno service's variables now match across all three files.
Also from that review: the platform paths have no equivalent of setup.sh's
secret generation, so an operator who skips the environment step gets a stack
running on `change-this-jwt-secret-min-32-characters` — a value published in
this file. Removing the defaults does not help, because the backend takes
`process.env.JWT_SECRET || ''` (app.config.ts:145) and would boot on an empty
one. Compose's mandatory-variable form does: the deploy now fails with "set
JWT_SECRET (openssl rand -hex 32) in this platform's environment settings".
Scoped to JWT_SECRET and ENCRYPTION_KEY. POSTGRES_PASSWORD keeps its default on
purpose — Postgres fixes the password at cluster init, so making it mandatory
would invite an operator with an existing install to set a value the cluster
does not have and lock the backend out of its own database. The blast radius for
the two that changed is small: Coolify's file is new in this PR, and Dokploy's
moved out of the repository root, so existing installs have to revisit their
compose path either way.
jwfing's third suggestion — a CI tripwire for config drift — stays a follow-up.
The invariant that the two platform files change together is recorded in the
backend skill instead, per an earlier decision to prefer a documented
convention over a linter here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
From @jwfing's review:
- The image-only compose was the only one of the four stacks without
PGRST_DB_POOL. PostgREST defaults to a pool of 10 while the backend's
postgrestMaxSockets defaults to 50 (app.config.ts:202) — exactly the
misalignment .env.example warns about, with queuing pushed into PostgREST
under load. It now sets the pool and passes the three socket knobs through,
matching the platform files and docker-compose.prod.yml.
- setup.sh notes why non-cone mode is deliberate. git's own docs call it
deprecated, and there is no cone-mode way to say "these root files but not
those" — cone always adds every root-level file, which would put the
development compose file in a self-hoster's checkout. With COMPOSE_FILE unset
for any reason, `docker compose up` would then build from source and start
dev servers rather than failing with "no configuration file provided". (For
the record, git 2.50.1 prints no runtime warning yet.)
- .env.example's bundled-store example still passed -f docker-compose.prod.yml,
a file setup.sh does not check out. It points at COMPOSE_FILE now, matching
the guidance higher up in the same file.
- The image-only stack's deno service runs as the image default while the
platform files drop to uid 1000, which now says why: doing it here needs the
cache volume chowned, and a fresh named volume takes its ownership from the
image, which needs a build step this stack avoids.
From cubic's review, reproduced first: a failing or missing openssl left
`JWT_SECRET=` in .env, and Compose treats an empty value as unset, so
`${JWT_SECRET:-dev-secret-please-change-in-production}` handed out the
placeholder. Secret generation now aborts instead — and removes the
half-written .env, because the re-run path leaves an existing one untouched,
which would have meant the secrets were never generated at all. chmod 600 also
moved ahead of the writes rather than after them.
Verified: with openssl stubbed to fail, the script aborts with no .env left
behind, and a re-run in the same directory produces a complete 600 file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review on #1887 caught two real gaps.
The platform files dropped VERCEL_TOKEN, VERCEL_TEAM_ID and VERCEL_PROJECT_ID
— the only three variables the image-only compose passes that they did not —
so site deployments and custom domains ran without credentials. Both guides
tell the reader that .env.example lists every supported variable, which made
that silent.
None of the cloud guides re-ran setup.sh after `git pull` in their update
sequence. A release that adds a file the compose reads also adds its path to
the script, and without re-applying, the merge lands the file in git but never
in a sparse working tree — the stack then starts against a file that is not
there. Only the security guide had this step.
While checking those: sixteen prose references still pointed readers at
`~/insforge/deploy/docker-compose` as the directory to run commands from. Four
were the Azure guide's `~/InsForge/...`, which the earlier case-sensitive sweep
missed.
Also settles the Coolify verification gap this PR shipped with. Coolify binds
domains per compose service through `docker_compose_domains` on the
application, which was null, which is why its auto-assigned FQDN returned 404.
Set to bind insforge, a real deploy answers /api/health over the domain, and an
edge function created and invoked through it returns the worker's body — as
does hitting http://deno:7133 directly, so service-name resolution works under
Coolify's networking. Verified with real secrets: PATCH, not POST, is what
updates an environment variable Coolify already discovered from the compose
file.
Vale flagged three words in the new guides. Coolify joins Dokploy and Zeabur in
the vocabulary; "undecryptable" and "preloads" are rewritten rather than added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- The quickstart is the setup.sh one-liner, with a download-and-read variant
for anyone who will not pipe a script to a shell. The old "do the same by
hand" block reproduced the script's sparse-checkout list, which had already
drifted once when the list gained deploy/setup.sh and the copy did not.
- Every command runs from the checkout root. The four cloud guides (AWS EC2,
Azure VM, GCE, Containarium) cloned the full monorepo — 1,782 files, 57.6MB —
and then ran the image-only compose file anyway; none reference a build step.
They use setup.sh now, which also stops walking the reader through inventing
JWT_SECRET by hand or pasting a placeholder over a generated secret.
- New Coolify and Dokploy guides. Both platforms had compose files in the repo
and no documentation; Coolify appeared once, in a list of telemetry values.
- COMPOSE_PROJECT_NAME is documented with its consequence: any .env on the host
carrying the same name means `docker compose up` there adopts and recreates
that project's containers.
- self-host-storage.mdx covers the image-only stack, which it never did —
appending an overlay to COMPOSE_FILE rather than passing -f flags.
- Section 5.4 no longer tells self-hosters to set AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY; the compose file passes neither.
- The automated backup script pointed -f at a compose file the checkout does
not contain, and -f alone loads neither COMPOSE_FILE nor
COMPOSE_PROJECT_NAME from .env, so its `exec` would not have found the
container.
Registered in docs/deployment/README.md and docs.json's English tree, locale
trees regenerated with scripts/build-docs-langs.py. The i18n parity check
reports the same 24 pre-existing gaps as before.
Not verified: the domain-assignment step in the two platform guides. Coolify's
auto-assigned application FQDN returns 404 because a compose deployment binds
domains per service, and I found no API field for that; the Dokploy stack was
checked over the container network. Both steps follow the platforms' own docs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docker-compose.dokploy.yml sat at the repository root on postgres-all, so RLS
on managed tables was broken there for the same reason it was everywhere else.
Neither platform can fix that by mounting: Coolify creates file bind mounts as
directories (coollabsio/coolify#3375, open for years) and Dokploy re-clones
code/ on every deploy, which makes a mount into the repository go stale and
otherwise needs File Mounts wired up by hand in the UI. Both build Postgres
from deploy/Dockerfile.postgres instead — four lines, FROM the same published
base image plus 5,352 bytes of configuration, so the config cannot fall behind
the code the way a prebuilt image can. Zeabur keeps its inlined configs; a
marketplace template is PREBUILT and has no build context.
The two files carry identical service definitions apart from two lines, both
load-bearing:
- INSFORGE_DEPLOYMENT_METHOD, which telemetry reads as an artifact stamp — its
own comment notes Dokploy is identified this way because it injects nothing
into containers.
- the build context. Coolify builds with `--project-directory <repo root>`, so
Compose resolves a relative context against the root; Dokploy passes no
--project-directory, so Compose uses the compose file's own directory. A real
Coolify deploy with `context: ../..` failed with "resolve : lstat /deploy: no
such file or directory" — it had resolved to `/`.
Neither publishes a port, so both headers name the step that makes the stack
reachable: assign a domain to the insforge service on port 7130, then point
API_BASE_URL at it. Without it the deploy succeeds and nothing answers.
docker-compose.override.yml is deleted. It was added to bake configuration for
Coolify, not as a development convenience, and it only ever added `build:` to
services that already declared `image:` — which Compose skips when the image is
present, so it never ran. Its postgres build duplicated what the base compose
already mounts.
Verified on real instances of both: Coolify 4.1.2 deploy finished and Dokploy
0.29.14 composeStatus done, each with four containers up, insforge_pg_utils
preloaded, CREATE POLICY on storage.objects as project_admin, 63 migrations,
the repo's 7,488-byte worker-template.js in the deno container running as uid
1000, and on Dokploy an edge function created and invoked end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Self-hosting ran ghcr.io/insforge/postgres-all, which bakes its configuration
in. The published image's postgresql.conf dates from May and its
shared_preload_libraries omits insforge_pg_utils — the hook that lets
project_admin create policies on tables it does not own — so RLS on managed
tables fails on every self-hosted install, while this repo's own config has
carried the hook, its grant-role GUCs and insforge.internal_schemas for
months. Baked configuration cannot track a moving schema. The compose file now
runs the base image and mounts the three files from the checkout.
Same split, same result, for edge functions: ghcr.io/insforge/deno-runtime is
built in InsForge/deploy-provider, and hashing every historical revision of
functions/server.ts and functions/worker-template.js against the image's
copies matches none of them. It is 92 lines behind on server.ts and 175 on
worker-template.js, and where this repo registers self.onmessage before the
top-level await (d12e69c3b, "fix 504 race") the image still assigns it inline.
The stack now runs the official denoland/deno image with functions/ mounted
from the checkout, read-only and with --no-lock, since Deno otherwise tries to
write deno.lock back into it and the container restart-loops.
With a checkout part of self-hosting rather than an optional convenience:
- deploy/setup.sh sparse-checks out the ten files the stack reads (~700KB over
the network, no build step) and generates JWT_SECRET, ENCRYPTION_KEY,
ROOT_ADMIN_PASSWORD and POSTGRES_PASSWORD at mode 600. Postgres reads that
last one only at cluster init, so it has to be settled before first boot;
it defaulted to "postgres" and was left to a manual checklist item.
Re-running the script after `git merge` is what picks up files a release
adds — a merge lands them in git but not in a sparse working tree. It
refuses to touch a full development clone, where the sparse-checkout would
empty the working tree.
- One .env.example, at the repo root. The copy under deploy/docker-compose/
had drifted: the S3 storage settings went undocumented there for months.
- .env sits beside it, and COMPOSE_FILE inside it points Compose at
deploy/docker-compose/docker-compose.yml, so every command runs from the
checkout root. Compose reads .env, and COMPOSE_FILE within it, from the
directory you are in, while the compose file's relative mounts still resolve
against its own directory. The template ships the development value, which
resolves byte-identically to today's auto-discovery; setup.sh rewrites the
line. Shipping the production value there would have silently repointed
every developer's `docker compose up` at published images.
- Installs predating this layout keep .env beside the compose file, where
Compose ignores it — secrets that no longer reach Postgres read as a lost
database. setup.sh moves it. Those installs have no COMPOSE_PROJECT_NAME,
and Compose derives "docker-compose" for them from the compose file's
directory, the same name they already run under, so their volumes stay
attached.
PGDATA is pinned to the subdirectory postgres-all baked into its own ENV.
Changing it makes Postgres initdb an empty cluster beside the real one, which
reads as total data loss.
Dockerfile.deno did not build: the base image already ships a deno user at uid
1000, so `addgroup -S deno` exits 1. Nothing caught it because the override
only added `build:` to services that already declared `image:`, and Compose
skips building when the image is present — inert for anyone who had pulled it,
broken for anyone who had not.
postgrest's healthcheck is gone, and insforge depends on it with
service_started. On amd64 that image is 24 entries — bin/postgrest and a
certificate directory, no shell — so a CMD-SHELL probe reports unhealthy
forever and `docker compose up -d` ends with "dependency failed to start:
container postgrest-1 is unhealthy". The backend never starts. The same probe
and condition are on main; it went unnoticed because the arm64 image for that
tag is a full Debian build with a shell, so the stack comes up on an Apple
Silicon laptop and fails on the Linux servers people self-host on.
Verified on an amd64 EC2 instance following the guide as written: stack up,
/api/health 200 at 2.2.9, 63 migrations, insforge_pg_utils preloaded, CREATE
POLICY on storage.objects as project_admin, and an edge function created
through the API and invoked end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
insforge.policy_grant_role and insforge.extension_grant_role both already default
to 'project_admin' inside the extension (insforge_pg_utils.c:74-84, :98-103), so
behaviour is unchanged. Pinned explicitly because the default lives in a compiled
.so, invisible to a reader of this file, and a rebuild could shift permission
behaviour with no diff in any config repo.
extension_grant_role was declared nowhere and matters more than it looks: the hook
runs CREATE/DROP EXTENSION as the bootstrap superuser, a broader grant than the
policy one.
Both copies updated; deploy/zeabur/template.yml inlines this file as a YAML
template string.
Verified on ghcr.io/insforge/postgres:v15.13.4 that both report
source=configuration file with value project_admin, and that the preload list is
byte-identical to origin/main.
An earlier revision of this PR also preloaded pg_net. Dropped: cloud has had it
preloaded for seven months with pg_net.database_name unset, meaning async SQL HTTP
never worked there and nobody reported it — so there is no demand to serve, and
preloading costs a permanent background worker per instance.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The WAL settings were annotated "Enable logical replication for Logflare".
Logflare was removed in 2025-10 (dc953b7a3), so the comments pointed at a
dependency that no longer exists.
Settings unchanged. wal_level stays on `logical` because changing it requires a
restart, and nothing consumes it today — realtime uses pg_notify (migration 017),
and backend/src has no references to replication slots, publications, logical
decoding, pgoutput or wal2json.
Both copies updated; deploy/zeabur/template.yml inlines this file as a YAML
template string.
Verified: non-comment settings are byte-identical to origin/main in both files
(diffed with comments stripped), the zeabur template block keeps its 20-space
indentation, prettier passes, and a Postgres started on the edited conf reports
wal_level=logical with unchanged shared_preload_libraries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image-only compose file starts a deno container but never tells the backend
where it is. DENO_RUNTIME_URL is set in docker-compose.yml, docker-compose.prod.yml
and docker-compose.dokploy.yml — only this file omits it, so the backend falls
back to its default of http://localhost:7133 (app.config.ts), which inside the
insforge container is the container itself, where nothing listens.
Effect for anyone self-hosting via the documented download-and-run path:
`functions deploy` reports success, and every invoke returns 502 with
`request to http://localhost:7133/... failed`. Edge functions have never worked
on this path.
Verified on this stack: the backend container now reports
DENO_RUNTIME_URL=http://deno:7133 and `wget http://deno:7133/health` from inside
it returns the runtime's ok response. The full deploy → invoke chain was verified
separately against the same variable value, where invoke returns the function's
own JSON instead of 502.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file pinned insforge-oss:v1.5.0 on 2026-02-02 and nobody touched the line
again. Since then there have been 36 releases and 48 migrations, and the file
itself was edited 17 times — telemetry, payments, S3 backends, healthchecks —
without the tag ever moving. Anyone installing by following the documented
"download the compose file and run it" path has been getting a February backend.
Nothing automated reads this file, which is why it went unnoticed: CI and E2E use
the root docker-compose.yml, which builds from source. The other two images here
were already on :latest and never rotted — only the pinned one did, so it goes to
:latest too rather than to a new version that would rot the same way.
Affects only self-hosters who follow the download-and-run path (README quickstart,
deploy/docker-deploy.md, and the zh / zh-Hant deployment guides). Cloud is
unaffected — it resolves versions through INSFORGE_OSS_VER on the instance, not
from this file. Existing running deployments are unaffected until they re-download
the compose file; when they do, boot-time migrate:up applies the pending
migrations in order, which is supported but not reversible, so a pg_dump first is
worth mentioning in the release notes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Verb-prefixed boolean per convention (matches S3_FORCE_PATH_STYLE);
avoids reading as a URL-valued variable like S3_ENDPOINT_URL. Internal
field renamed to s3UsePresignedUrls. The flag was introduced in this PR
so there is no compatibility surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-hosting no longer uses the misleading AWS_ prefix for what is a
generic S3-compatible configuration:
- Config loader prefers S3_BUCKET / S3_REGION; AWS_S3_BUCKET / AWS_REGION
remain as fallbacks (cloud provisioning sets them, existing .env files
keep working). Internal field renamed awsRegion -> s3Region.
- Self-host compose files (deploy, dokploy) expose a pure S3_* block;
the AWS-specific vars (CloudFront, AWS credentials) stay cloud-only.
Root dev/prod compose keep AWS_* passthrough for cloud parity and
CloudWatch, with the neutral names added.
- Overlays, docs, .env.example, error messages, and dashboard copy
(all locales) switched to the S3_* names, with AWS_* documented as
cloud-project/legacy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-hosters can now back InsForge Storage with any S3-compatible store
and use the S3 gateway, previously blocked by deployment wiring:
- Add S3_PRESIGNED_URLS flag (default true, no behavior change). When
false the S3 provider returns backend-proxied direct strategies —
same contract as the local provider — so stores on private networks
(bundled MinIO/RustFS) or without POST-policy support (R2) work
without being exposed to browsers.
- Stream proxy-mode downloads through the backend with Range/206/416
support instead of buffering whole objects.
- Pass the storage env block through to the insforge service in the
prod, dokploy, and deploy compose files (previously silently ignored).
- Add turn-key docker-compose.minio.yml / docker-compose.rustfs.yml
overlays: internal-only store, bucket init, proxy mode preset.
- Gate the dashboard S3 Configuration tab on backend capability
(available field on GET /api/storage/s3/config) instead of cloud
hostname, so self-hosters can mint gateway access keys.
- Docs: new deployment/self-host-storage guide; update .env.example,
security guide, and S3-compatibility page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The schedules module's encrypt_headers/decrypt_headers functions require
the Postgres GUC app.encryption_key, but the deploy/docker-compose and
dokploy compose files only passed ENCRYPTION_KEY as an env var without
setting the GUC. This caused schedules with headers to fail with:
'Encryption key app.encryption_key is not set'.
Mirrors the pattern from the root docker-compose.yml and prod.yml which
already set the GUC via 'postgres -c app.encryption_key=...'.
Closes#1763
The release workflow moves the latest tag on every clean release tag
push, so new template deployments always pull the newest release
without a per-release template bump.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- postgresql.conf: correct stale "migration 055" → 056 and drop the obsolete
"_-prefixed schemas handled separately" note (that rule was removed).
- resolvePostgrestSchema: reject a blank/repeated ?schema= or array-valued
profile header with 400 instead of silently falling back to `public`, and
re-forward the normalized profile header so it cannot disagree with the
schemaName used for metadata lookups.
- Tests for the new guards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sources the internal-schema deny-list from a single insforge.internal_schemas
GUC (postgresql.conf) instead of hardcoding it in is_exposed_schema, so it has
one editable source of truth and can change without a migration. The literal
list stays inline only as a fallback when the GUC is unset.
Adds native PostgREST schema selection to the records and RPC proxy routes:
an explicit ?schema= is desugared into Accept-Profile (reads) / Content-Profile
(writes & RPC) and stripped from the forwarded query, a client-sent profile
header is honored as-is, otherwise PostgREST's default schema applies. The
records route now threads the resolved schema into getColumnTypeMap so body
filtering uses the right table's column types instead of always public.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The anon key was a never-expiring HS256 JWT signed with the shared
JWT_SECRET: impossible to rotate without invalidating every user session,
impossible to revoke, and carrying a fake identity through the user token
path. Replace it with an opaque anon_<hex> key following the same design
as the ik_ admin API key (random, timing-safe compare, rotation with
grace period), validated at the gateway and exchanged for an internal
anon JWT before reaching PostgREST.
- SecretService: generateAnonKey/verifyAnonKey (cache-backed hot path),
rotateAnonKey (7-day default grace), initializeAnonKey (migrates
legacy JWT-format ANON_KEY values in place)
- auth middleware: verifyUser dispatches on credential shape
(ik_/anon_/JWT) and fails closed — an expired user JWT returns 401
instead of silently downgrading to anon
- PostgREST proxy: forwardAsAnon swaps the opaque key for an
internally-minted role:anon JWT (records + rpc routes)
- routes: GET /api/auth/keys/anon (new), POST /api/auth/tokens/anon now
returns the opaque key (deprecated), POST /api/secrets/anon-key/rotate
with edge-function redeploy trigger
- socket manager: accepts the opaque anon key for realtime connections
- dashboard: fetches anonKey from the new endpoint
- legacy anon JWTs keep verifying through the JWT path (cannot be
revoked anyway); new clients only ever see the opaque key
Lyu