Files
Daniel Sutton f98e303292 feat(webapp): resolve which shard an environment mints run roots into (#4755)
## Summary

Adds the shard-selection stage of run-id minting.
`resolveMintShard(env)` returns which run-ops database an environment
mints its new run roots into: the active shard list, then a fleet-wide
override, then a per-environment or per-organization pin, then a
rendezvous hash of the environment id.

That half is inert. Nothing calls `resolveMintShard`, no deployment has
any of the new flags set, and an empty active list returns the current
answer without reading anything.

**The other half is not inert, and it is where review effort belongs.**
To stamp a grace window this needs a read-then-write under a lock, so it
rewrites the global feature-flag write path that `runOpsMintKind`
already depends on in production. See below.

## Placement

Resolution reads the active list from a global flag, applies the grace
window, and then picks:

- a fleet-wide override if one is set, which is how a cutover completes
without visiting each organization. `new` holds the whole fleet on the
current id format.
- otherwise a per-environment or per-organization pin. `new` holds one
organization back while the rest move, which is how a canary works.
- otherwise a rendezvous hash, so adding a shard moves only about
1/(N+1) of environments and removing one moves only its own.

Two hash details are load-bearing. Scores are 64-bit `sha256(envId \0
key)`, because a 32-bit score collides at our environment count and an
undetected tie would resolve by iteration order. The parsed key list is
sorted, because otherwise two deployments listing the same shards in a
different CSV order would place environments differently.

A pin or override naming a shard that has left the active list falls
through to the hash and reports once. Honouring it would leak the drain
the active list exists to perform, and throwing would fail triggers
whenever a pinned shard drains.

## Why the active list is a flag and not an environment variable

A deploy rolls for hours, so two pods hold two different environment
values at the same time. A list held in the environment therefore splits
the fleet for the length of the rollout, with new pods placing an
environment on one shard and old pods on another. A grace window
measured in seconds cannot cover that, and the same knob times the
existing mint-kind flip so it cannot simply be lengthened. An
environment variable also cannot record its own flip time, and an
operator cannot know a rollout's end in advance.

So the list, its grace stamp and the override are global flags, written
server-side against the control-plane clock under an advisory lock. This
branch adds no environment variables.

## The write path, which is live

Stamping generalises to any number of graced flag groups in one
transaction under one lock. That has three consequences a reviewer
should look at directly:

- It closes a real bug. `runOpsMintKind` is an editable control on the
global flags page, and that page previously wrote it with a bare upsert:
no lock, no stamp. An operator flipping mint kind through the UI got an
ungraced flip, so every pod crossed the cutover at a different moment.
Verified against a running instance, before and after.
- A graced group is all-or-nothing. Submitting its primary writes the
group with a fresh stamp; omitting it deletes the primary and its stamp
together, because a stamp left without its primary keeps being served
and would mint into a shard just removed.
- The advisory lock takes the previous id as well as the current one, in
a fixed order, so writers on an older release still serialise during a
rollout. The legacy id can be dropped one release after this ships.

This folds with #4751 rather than replacing it: its `unlockLockedFlags`
rule decides what the sweep may delete, and the graced groups keep their
stamp under the lock. Both sets of tests pass.

## Notes for review

Determinism is a property of the pure core for fixed inputs. The wrapper
supplies the clock, the same split `effectiveMintKind` already uses. A
failed read of the list falls back to the current id format rather than
guessing.

Six flags appear in the admin pages immediately. The two pins are
per-organization, so they render read-only on the global page. The list,
its stamp and the override are deployment-wide, so they render read-only
in the organization dialog.

Nothing bounds the active list against shards that actually exist. That
is safe while nothing mints, but the change that carries a shard key
into an id must land after the shard descriptors bound the list, or
bound it itself.
2026-08-24 15:23:58 +01:00
..