Eric Allam
c6f0769299
## Summary The logs search page (behind a feature flag) ran ClickHouse out of memory when browsing back over long time ranges. This keeps it within bounded memory and fixes a pagination bug that could skip or duplicate rows at a page boundary. ## Fix Memory: the list query reads in sort-key order, which opens one read stream per part in the window, and on object storage those per-part read buffers dominate peak memory, so it scaled with the number of parts scanned. Two changes bound it: - The logs ClickHouse client caps the per-part read buffers via new env-tunable settings. The object-storage-only setting is opt-in, so it is never sent to a ClickHouse version that lacks it. - Recent-first window narrowing: rows come back newest first, so the presenter probes the most recent window and only widens toward the full requested range when a page is short. A busy environment fills a page from a few recent parts instead of scanning the whole range; a quiet one still returns every row in a couple of cheap reads. Correctness: the keyset cursor ordered on (triggered_timestamp, trace_id), which is not unique because the spans of a trace share both, so rows at a tie could be skipped or duplicated across pages. The cursor and ORDER BY now include span_id, and the cursor is versioned so stale cursors reset to the first page. Guards: the effective page size is capped, and the existing per-query memory limit lets a pathological wide browse fail with an error instead of taking the node down. ## ClickHouse 26.2 The memory fix relies on lazy materialization deferring the wide attributes column to the output rows, which only holds on 26.x. Cloud already runs 26.2, so this moves the dev stack, testcontainers, and CI to match. The ClickHouse test suite passes on 26.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Server Changes
This directory tracks changes to server-only components (webapp, supervisor, coordinator, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
When to add a file
Server-only PRs: If your PR only changes apps/webapp/, apps/supervisor/, apps/coordinator/, or other server components (and does NOT change anything in packages/), add a .server-changes/ file.
Mixed PRs (both packages and server): Just add a changeset as usual. No .server-changes/ file needed — the changeset covers it.
Package-only PRs: Just add a changeset as usual.
File format
Create a markdown file with a descriptive name:
.server-changes/fix-batch-queue-stalls.md
With this format:
---
area: webapp
type: fix
---
Speed up batch queue processing by removing stalls and fixing retry race
Fields
- area (required):
webapp|supervisor|coordinator|kubernetes-provider|docker-provider - type (required):
feature|fix|improvement|breaking
Description
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
Writing guidance
These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
- One sentence is usually enough. The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
- Describe behavior, not implementation. Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
- Never name internal tools or infra. Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
Lifecycle
- Engineer adds a
.server-changes/file in their PR - Files accumulate on
mainas PRs merge - The changeset release PR includes these in its summary
- After the release merges, CI cleans up the consumed files
Examples
New feature:
---
area: webapp
type: feature
---
TRQL query language and the Query page
Bug fix:
---
area: webapp
type: fix
---
Fix schedule limit counting for orgs with custom limits
Improvement:
---
area: webapp
type: improvement
---
Use the replica for API auth queries to reduce primary load