Eric Allam
1ab5066ed0
## Summary Batch triggers that use per-item idempotency keys could take seconds instead of milliseconds when the target task had a large run history. This keeps the idempotency lookup fast regardless of how many runs a task has accumulated. ## Root cause The batch path checks which items already have runs by looking up their idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large `TaskRun` table Postgres underestimates the row count of a specific `(environment, task)` pair, so once the `IN` list grows past a handful of keys it stops doing per-key index probes and instead scans every run for that `(environment, task)` and filters the keys in memory. The cost is then flat and large regardless of how many keys are being checked, and a routine `ANALYZE` does not correct the estimate at that table size. ## Fix Look each idempotency key up on its own, batched into a `UNION ALL` of point lookups (chunked, run with bounded concurrency). Each branch is an equality on all three columns of the unique index, so the planner can only do a per-key index probe and can never fall back to the range scan. Same results, same columns, confined to the batch trigger path.
Server Changes
This directory tracks changes to server-only components (webapp, supervisor, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
When to add a file
Server-only PRs: If your PR only changes apps/webapp/, apps/supervisor/, or other server components (and does NOT change anything in packages/), add a .server-changes/ file.
Mixed PRs (both packages and server): Just add a changeset as usual. No .server-changes/ file needed — the changeset covers it.
Package-only PRs: Just add a changeset as usual.
File format
Create a markdown file with a descriptive name:
.server-changes/fix-batch-queue-stalls.md
With this format:
---
area: webapp
type: fix
---
Speed up batch queue processing by removing stalls and fixing retry race
Fields
- area (required):
webapp|supervisor - type (required):
feature|fix|improvement|breaking
Description
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
Writing guidance
These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
- Write for the user, not the reviewer. Lead with what the user notices or has to do. If a reader who doesn't know the codebase can't tell what changed for them, rewrite it.
- One sentence is usually enough. The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
- Describe behavior, not implementation. Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
- Never name internal tools or infra. Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
Before / after:
- ❌ "The image verification step now parses the manifest's layer media types and returns a new result the finalizer rejects." (describes the wiring; a user can't act on it)
- ✅ "Deploying with an outdated CLI could produce an image that fails to start on every run. These deploys are now stopped before going live, with a message asking you to upgrade the CLI and re-deploy." (what the user sees and does)
Lifecycle
- Engineer adds a
.server-changes/file in their PR - Files accumulate on
mainas PRs merge - The changeset release PR includes these in its summary
- After the release merges, CI cleans up the consumed files
Examples
New feature:
---
area: webapp
type: feature
---
TRQL query language and the Query page
Bug fix:
---
area: webapp
type: fix
---
Fix schedule limit counting for orgs with custom limits
Improvement:
---
area: webapp
type: improvement
---
Use the replica for API auth queries to reduce primary load