8 Commits

Author	SHA1	Message	Date
Eric Allam	d3906241a5	feat(webapp): read realtime run rows from the primary, not the replica (#4378) ... ## Summary The realtime runs feed hydrates run rows from read replicas, which means it needs a replica-lag gate to avoid serving a run's previous state right after a write. Setting `REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from each run store's primary instead, so there is no lag to gate against: no probe, no wake delay, no stale-read retries. Off by default, so nothing changes unless you set it. ## Design The run stores already decide replica-vs-primary from the *brand* on the read client they are handed: a branded replica keeps the read on the owning store's replica, an unbranded writer escalates it to that store's own primary. So this is a one-line choice at the hydrator, and it stays correct across topologies. With the run-ops split on, each leg lands on its own writer and the caller's client is never forwarded across databases; with the split off, it is the single database's primary. ```ts const runReader = new RunHydrator({ readClient: runReadsFromPrimary ? prisma : $replica, runStore, }); ``` The same flag skips constructing the lag estimator, since probing a replica the feed no longer reads would be measuring the wrong thing. Independently, `AuroraReplicaLagSource` detected Aurora by letting `aurora_replica_status()` fail, on the assumption that the app-level catch made that free. It isn't: an unresolvable function is a query error the driver reports to the error log on every sample, so a non-Aurora replica produced a continuous stream of error events while the estimator quietly fell through to its next candidate. It now resolves the function with `to_regproc` and memoizes the answer, so the unparseable call never reaches the wire.	2026-07-26 19:43:41 +01:00
Daniel Sutton	092b9ef07a	fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154) ... ## Problem The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**. The supervisor writes the run id into the Kubernetes pod name (`runner-<id>`), and pod names must be DNS-1123 labels (lowercase `[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and **those runs never launch** (they loop in `PENDING_EXECUTING` until the heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding collides distinct ids and destroys sort order. ## Fix: change the encoding, not the structure Mint a **26-char lowercase base32hex** run id: ``` run_<24-char base32hex core><region char><version char> [ 6-byte ms timestamp ][ 9 CSPRNG bytes ] ``` - **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase, order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding. Hand-rolled encode/decode (no new dependency). - **48-bit ms timestamp** in the leading bytes → plain string sort == creation order at millisecond resolution. - **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no retry loop). - **region / version** are raw positional chars (read via one `charAt` before decoding/routing), version = `"1"`. DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** — `runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still recovers the exact id incl. region+version. ## Residency discriminator: length → version char `classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a third format. It now discriminates on the **version char at a fixed position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex alphabet) → NEW; everything else → LEGACY. Total, never throws. The `Residency` (NEW/LEGACY) contract the routing store consumes is unchanged; the `"ksuid"` `ResidencyKind` label is retained only because it's the persisted `runOpsMintKsuid` feature-flag value. ## Scope / verification - Generator + discriminator in `@trigger.dev/core` isomorphic; mint path + all id-shape call sites swept (~40 webapp files); changeset added (`@trigger.dev/core` patch). - Core unit tests (encode/decode round-trip + property, generator shape, ms sort-order incl. intra-second, parse partitioned-vs-legacy, firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp typechecks; format/lint clean. ## Open decisions (flagged, not silently chosen) 1. **Backward-compat**: existing 27-char base62 KSUID runs now classify LEGACY. On test cloud these are the broken/looping runs that never completed, so this is acceptable — but worth a conscious call before prod. No transitional length-recognition added (keeps the discriminator clean). 2. **Storage collation**: the sort guarantee is byte-order — if the run-ops id column is `TEXT` with default locale collation it's silently not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the run-ops schema. 3. **Region sourcing** wiring — see `regionCharForRegion` / `REGION_CODES`. --- ## ⚠️ Required migration — deploy in lockstep This PR renames a persisted feature-flag key/value and an env var. These are **not** changed by the code alone and must be migrated when this deploys, or affected orgs silently fall back to `cuid` minting (no crash — `defaultValue: "cuid"`): 1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` → `RUN_OPS_MINT_ENABLED` (carry the value over). 2. **DB** `organization.featureFlags`: migrate both the key and value together: - key `runOpsMintKsuid` → `runOpsMintKind` - value `"ksuid"` → `"runOpsId"` Until an org's flag row is migrated, its `runOpsMintKind` lookup misses and it mints `cuid` (legacy) — so no NEW-store ids for that org until the data lands. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 10:05:54 +01:00
Daniel Sutton	0a51341347	feat(run-ops): read presenters — de-join control-plane relations + read-through hydration (#4122)	2026-07-03 20:37:59 +01:00
Daniel Sutton	84f3e1b39c	feat(run-ops): webapp write path — trigger/batch minting, idempotency routing, run lifecycle (#4118) ... ## What Routes the webapp write path through the run-ops split seam: trigger/batch minting, idempotency-key resolution, and the run-lifecycle services now determine residency and dispatch writes to the correct store. - **Trigger & batch** (`runEngine/services/triggerTask.server.ts`, `batchTrigger.server.ts`, `createBatch.server.ts`, `streamBatchItems.server.ts`, `v3/services/batchTriggerV3.server.ts`): mint ids with the run-ops-aware minting and route creation/streaming through the store; batch children inherit the parent's residency. - **Idempotency** (`runEngine/concerns/idempotencyKeys.server.ts` + new `idempotencyResidency.server.ts`): idempotency-key lookup/dedup is residency-aware so a keyed retrigger resolves against the store that owns the original run. - **Run lifecycle services** (`createCheckpoint`, `createTaskRunAttempt`, `enqueueDelayedRun`, `expireEnqueuedRun`, `finalizeTaskRun`, `resumeBatchRun`, `cancelDevSessionRuns`, `executeTasksWaitingForDeploy`, `triggerFailedTask`): resolve their target run through the store rather than a fixed client. - **Reads that fan out from writes** (`runsRepository` + `clickhouseRunsRepository`, `BulkActionV2` + batch read-through, realtime `sessions`/`runReader`, alerts `deliverAlert`/`performTaskRunAlerts`): route through the read-through resolver. - `9535ae63d` — resolves the parent run through an injectable run store in `TriggerFailedTaskService`. - `bf8f7c881` — drops the "known-migrated" concept from write-path and read repos; residency is id-shape only. - `515b897ea` — self-defaults `resolveWaitpointThroughReadThrough` to the safe run-ops clients. ## Why PR6 of the run-ops split stack. This is the write-path counterpart to the read foundation in the previous PRs: with it in place, both reads and writes route through the seam. Additive when the split is disabled (id-shape resolution collapses to the control-plane client); behavior-changing on the minting, idempotency, and lifecycle paths when enabled. ## Tests Large new/expanded vitest suite under `apps/webapp/test/` and colocated service tests: trigger-task and batch-trigger store routing, residency inheritance, idempotency dedup residency + legacy-authority, bulk-action read routing, cancel-dev-session routing, alerts store routing, runs-repository read-through, realtime session/run-reader read-through and stream-registration routing, and the waitpoint read-through default. Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4117** (`runops/pr05-webapp-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-03 18:52:08 +01:00
Chris Arderne	4fde283e76	chore: format and lint webapp also (#4056) ... #3977 added formatting and linting everywhere else. This extends it to the webapp.	2026-06-26 13:02:53 +01:00
Daniel Sutton	65c545da4e	refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990) ... ## Summary Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`, `findRuns`) and routes every Postgres read of `TaskRun` through them, mirroring how writes already go through the store. Behavior-preserving: each relocated read keeps its exact query, field selection, and database client (writer, replica, or transaction). This lets `TaskRun` reads be retargeted to a different backing store later without touching call sites. Stacked on #3981 (the write adapter); that PR is the base of this one. ## Scope In scope: the run engine, webapp services, presenters, and route loaders. Three reads that pulled `TaskRun` in through a parent model's relation `include` (alert delivery, batch results, attempt-dependency cancellation) are decomposed to fetch the run(s) through the store and stitch them back, since a relation include would not follow `TaskRun` to a new table. Left reading the existing table (out of scope): the legacy MarQS paths, the legacy trigger idempotency read, and one raw-SQL recovery script (commented for revisiting at cutover). ## Notes Reads default to the read replica; callers pass the writer or a transaction client wherever the original read did, so writer-vs-replica behavior is unchanged.	2026-06-22 10:02:57 +01:00
Eric Allam	954ee5c572	fix(webapp): deliver realtime changes with current content when the read replica lags (#3910) ... ## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself.	2026-06-12 07:34:50 +01:00
Eric Allam	f9d57d3bd5	feat(webapp): add a new backend for the realtime runs feed (#3864) ... ## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default.	2026-06-11 07:56:10 +01:00