Eric Allam
4fd7cc0f55
## What
The `billingLimit.reconcileTick` worker calls
`getOrgIdsWithBillingPauseSource()` on
`BILLING_LIMIT_RECONCILE_INTERVAL_MS` (~every 90s) to find which orgs
currently have billing-limit-paused environments. Two problems:
1. `RuntimeEnvironment.pauseSource` had no index, so `WHERE pauseSource
= 'BILLING_LIMIT'` was a **sequential scan of the whole table** on the
control-plane primary, every tick.
2. Prisma `distinct` dedups **after** fetching, so it read every paused
row (thousands) to produce a handful of distinct org ids.
This PR:
- Adds a **partial index** on `RuntimeEnvironment (pauseSource,
organizationId) WHERE pauseSource IS NOT NULL`. Nearly all rows have
`pauseSource = null`, so the index stays tiny. Second column lets the DB
satisfy the distinct-org lookup from the index. Defined in SQL (Prisma
can't express partial indexes), matching the existing partial-unique
indexes on this model.
- Switches the query from `findMany({ distinct })` to
`groupBy(["organizationId"])`, pushing DISTINCT into the DB so it
returns only the distinct orgs.
## Evidence
**Correctness** — colocated `postgresTest` (testcontainers, no mocks):
multiple `BILLING_LIMIT` envs in one org collapse to one org id,
`pauseSource = null` envs are excluded, each org id returned once. 5/5
tests in `billingLimitReconciliation.test.ts` pass.
**Plan change** — `EXPLAIN ANALYZE` on a synthetic table (200k rows,
5,250 `BILLING_LIMIT` across ~40 orgs, mirroring the test-side numbers
from the investigation):
| | Before (no index) | After (partial index) |
|---|---|---|
| Plan | Seq Scan (194,750 rows removed by filter) | Bitmap Index Scan
on partial index |
| Buffers | 1355 | 51 (index 6 + heap 45) |
| Exec time | 6.06 ms | 0.59 ms |
Index size 56 kB vs table 11 MB. The key win: cost now scales with the
paused-env count, not total table size, which matters most on prod where
the table is far larger.
## Rollout & rollback
- **Index**: `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, in its own
migration file. Pre-apply the index manually on the control-plane
primary before deploying the migration (the migration is a no-op if the
index already exists).
- **Query change** is behavior-equivalent (same distinct org set), so no
flag needed.
- **Rollback**: revert the deploy and drop the index. No data migration
either direction.
## Notes / limitations
- The planner uses a Bitmap Heap Scan, so `organizationId` is still read
from the heap (45 blocks for the matched rows only, not the whole
table). A pure index-only scan isn't chosen for the bitmap path; the
second index column keeps that open for the index-scan path at
negligible cost.
refs TRI-13169
Server Changes
This directory tracks changes to server-only components (webapp, supervisor, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
When to add a file
Server-only PRs: If your PR only changes apps/webapp/, apps/supervisor/, or other server components (and does NOT change anything in packages/), add a .server-changes/ file.
Mixed PRs (both packages and server): Just add a changeset as usual. No .server-changes/ file needed — the changeset covers it.
Package-only PRs: Just add a changeset as usual.
File format
Create a markdown file with a descriptive name:
.server-changes/fix-batch-queue-stalls.md
With this format:
---
area: webapp
type: fix
---
Speed up batch queue processing by removing stalls and fixing retry race
Fields
- area (required):
webapp|supervisor - type (required):
feature|fix|improvement|breaking
Description
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
Writing guidance
These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
- Write for the user, not the reviewer. Lead with what the user notices or has to do. If a reader who doesn't know the codebase can't tell what changed for them, rewrite it.
- One sentence is usually enough. The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
- Describe behavior, not implementation. Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
- Never name internal tools or infra. Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
Before / after:
- ❌ "The image verification step now parses the manifest's layer media types and returns a new result the finalizer rejects." (describes the wiring; a user can't act on it)
- ✅ "Deploying with an outdated CLI could produce an image that fails to start on every run. These deploys are now stopped before going live, with a message asking you to upgrade the CLI and re-deploy." (what the user sees and does)
Lifecycle
- Engineer adds a
.server-changes/file in their PR - Files accumulate on
mainas PRs merge - The changeset release PR includes these in its summary
- After the release merges, CI cleans up the consumed files
Examples
New feature:
---
area: webapp
type: feature
---
TRQL query language and the Query page
Bug fix:
---
area: webapp
type: fix
---
Fix schedule limit counting for orgs with custom limits
Improvement:
---
area: webapp
type: improvement
---
Use the replica for API auth queries to reduce primary load