Matt Aitken
ef7b3aaf38
## Summary A run's finish commit and its follow-up side effects (completing the associated waitpoint, waking blocked parents, releasing the queue slot, nudging batch completion) are separate writes across Postgres and Redis. If a database error landed between them, the child run was already finished, so the runner's retries hit the "Run is already finished" guard and the completion signal was lost for good. A parent blocked on `triggerAndWait` or `batchTriggerAndWait` then stayed waiting forever. TTL expiry had the same shape: its worker retry returned early on a non-pending run, and the batch expiry path swallowed a failed waitpoint-job enqueue. ## Fix Every finalizing path (attempt success, permanent failure, cancellation, TTL expiry) now enqueues a durable `ensureRunFinalized` job before the finish commit, and acks it once the inline side effects all succeed. In steady state the guard never executes; the cost is one Redis enqueue and ack per completion. When the inline path dies in between, the guard fires after a short delay and re-derives everything from current state: it releases the run's queue message and concurrency slot, completes a still-pending associated waitpoint from the run row's output or error, re-runs the blocked-run fan-out (covering a lost unblock enqueue even after the waitpoint committed), and re-schedules the batch completion check. Every leg is idempotent, so racing the inline path is a no-op. The job retries with a capped backoff for roughly five weeks before dead-lettering, so it outlives any database outage while a genuinely poisoned item still becomes visible. Cancellation gets special handling: CANCELED is the only terminal run status where execution can still be in flight, so the guard only re-delivers for a canceled run once its execution snapshot is FINISHED, re-arming itself until then rather than resuming the parent while the child is still winding down. A `finalization_rederivations` counter increments whenever the guard actually re-delivers a lost signal; it should stay at zero in a healthy system. Tests cover six shapes: waitpoint completion lost after the finish commit, unblock fan-out lost after the waitpoint completed, a failed guard enqueue failing the completion request with nothing committed, a stale guard held back during an in-flight cancellation, waitpoint completion lost during TTL expiry, and the happy path where the guard is acked and never runs. Known accepted edge: a guard re-run after a partial inline completion can re-emit a cached-run completion event for the same span; this only happens during failure recovery and is bounded to duplicate trace events.
Server Changes
This directory tracks changes to server-only components (webapp, supervisor, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
When to add a file
These entries are user-facing release notes, not a catalog of every change. The test is "would a user or customer care about this change?", not "did I touch a server app?". Add one only when a server-only change is something a user would notice, act on, or want to hear about (a feature, a bug fix they could have hit, a behavior or performance change they would feel). Skip it for internal-only or admin-only changes, refactors, test-only changes, chores, and performance tuning with no user-visible effect. Anyone who wants the exact history reads the commits. When in doubt, ask a maintainer rather than adding a note by default.
Server-only PRs: If your PR only changes apps/webapp/, apps/supervisor/, or other server components (and does NOT change anything in packages/) AND the change is user-facing, add a .server-changes/ file.
Mixed PRs (both packages and server): the changeset covers it, so no .server-changes/ file is needed. If the package change is internal and needs no changeset but the server change is user-facing, add a .server-changes/ file for it.
Package-only PRs: Just add a changeset as usual, when the change is user-facing.
File format
Create a markdown file with a descriptive name:
.server-changes/fix-batch-queue-stalls.md
With this format:
---
area: webapp
type: fix
---
Speed up batch queue processing by removing stalls and fixing retry race
Fields
- area (required):
webapp|supervisor - type (required):
feature|fix|improvement|breaking
Description
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
Writing guidance
These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
- Write for the user, not the reviewer. Lead with what the user notices or has to do. If a reader who doesn't know the codebase can't tell what changed for them, rewrite it.
- One sentence is usually enough. The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
- Describe behavior, not implementation. Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
- Never name internal tools or infra. Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
Before / after:
- ❌ "The image verification step now parses the manifest's layer media types and returns a new result the finalizer rejects." (describes the wiring; a user can't act on it)
- ✅ "Deploying with an outdated CLI could produce an image that fails to start on every run. These deploys are now stopped before going live, with a message asking you to upgrade the CLI and re-deploy." (what the user sees and does)
Lifecycle
- Engineer adds a
.server-changes/file in their PR - Files accumulate on
mainas PRs merge - The changeset release PR includes these in its summary
- After the release merges, CI cleans up the consumed files
Examples
New feature:
---
area: webapp
type: feature
---
TRQL query language and the Query page
Bug fix:
---
area: webapp
type: fix
---
Fix schedule limit counting for orgs with custom limits
Improvement:
---
area: webapp
type: improvement
---
Use the replica for API auth queries to reduce primary load