Eric Allam
954ee5c572
## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself.
Webapp tests
Three suites live in this directory.
Unit tests — *.test.ts
Run with pnpm test from apps/webapp. Default vitest pickup. No
container setup. Run on every PR via unit-tests-webapp.yml.
Smoke e2e — *.e2e.test.ts
End-to-end auth baseline that proves the route auth plumbing is wired up.
Each file spins up its own webapp + Postgres + Redis container in
beforeAll (~30s startup). Vitest config: vitest.e2e.config.ts. Run on
every PR via e2e-webapp.yml.
cd apps/webapp
pnpm exec vitest --config vitest.e2e.config.ts
Comprehensive auth e2e — *.e2e.full.test.ts
The full RBAC auth matrix — every route family with explicit pass/fail scenarios. See TRI-8731 for the parent ticket and TRI-8732 onwards for each family's coverage spec.
Architecture: one container reused across the whole suite via
vitest.e2e.full.config.ts's globalSetup. Test files share the server
through getTestServer() from helpers/sharedTestServer.ts. Each test
seeds its own resources so order doesn't matter.
Layout:
| File | Top-level describe | Family subtasks |
|---|---|---|
auth-api.e2e.full.test.ts |
API |
TRI-8733 trigger, TRI-8734 run resource, TRI-8735 run mutations, TRI-8736 run lists, TRI-8737 batches, TRI-8738 prompts, TRI-8739 deployments + query, TRI-8740 waitpoints + input streams, TRI-8741 PAT |
auth-dashboard.e2e.full.test.ts |
Dashboard |
TRI-8742 admin pages |
auth-cross-cutting.e2e.full.test.ts |
Cross-cutting |
TRI-8743 deleted projects / revoked keys / expired JWTs / env mismatch / force-fallback toggle |
Adding a new family: pick the relevant file, add a nested describe
block. Inside, seed your own fixtures via the helpers and hit the shared
server.
describe("Trigger task", () => {
const server = getTestServer();
it("missing Authorization → 401", async () => {
const res = await server.webapp.fetch("/api/v1/tasks/x/trigger", { method: "POST", body: "{}" });
expect(res.status).toBe(401);
});
});
CI: e2e-webapp-auth-full.yml. Triggers on workflow_dispatch,
nightly schedule, and PRs touching auth-relevant paths (route builders,
rbac.server.ts, apiAuth.server.ts, apiroutes, the suite itself).
Run locally:
cd apps/webapp
pnpm exec vitest --config vitest.e2e.full.config.ts