* fix(loader): seek taxonomy-filtered listings via a denormalized pivot
Denormalize the filter+sort columns onto content_taxonomies, mirror the
ec_* sort indexes onto the pivot, and drive taxonomy-filtered listings
from the pivot with an authoritative ec_* re-check. Fixes full-collection
scans (~75k D1 rows read for a one-row page) on selective terms (#1834).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: update query-count snapshots
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* fix(ci): allow fork checkout in auto-format-apply workflow
actions/checkout now refuses to check out fork PR code from a
workflow_run trigger unless allow-unsafe-pr-checkout is set, which broke
the formatted-patch push-back for fork PRs (e.g. run 29045432430). This
job never executes the fork's code -- it only applies an inert diff
artifact and pushes back via an app token, same shape as the already-
merged fix for query-counts-apply.yml (#1816). Added the opt-in flag
with a comment explaining why it's safe.
* ci: update query-count snapshots
---------
Co-authored-by: Matt Kane <mkane@cloudflare.com>
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* fix(taxonomies): count only publicly visible entries in term counts (#581)
Term usage counts aggregated the content_taxonomies pivot directly, so
drafts, scheduled-future, and trashed entries inflated the counts shown
in the Categories/Tags widgets, on term pages, and in the admin term
list. All three paths now share a single UNION ALL count query that
joins the content tables, filters with buildStatusCondition (published
or scheduled-and-due) and deleted_at IS NULL, and scopes to the
taxonomy's declared collections.
Consolidates the fixes proposed in #1839, #822, and #593.
* ci: update query-count snapshots
* fix(taxonomies): include collection scope in term-count request-cache key
Per-locale rows of the same taxonomy def can drift in their declared
collections, and a future caller may pass a narrower scope; keying the
request cache by taxonomy name alone let the first scope poison the
entry for the rest of the request. Raised by review on #1892.
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* perf(core): fold byline + taxonomy hydration into the content query
loadEntry/loadCollection now hydrate bylines and taxonomy terms via correlated JSON-array subqueries in the content query instead of two follow-up round trips. Per-page query counts drop substantially (demo article 24->17, index 11->6).
- Dialect-aware aggregation (SQLite json_group_array/json_object string; Postgres json_agg/json_build_object parsed JSON), validated on both.
- Terms correlate on the entry's own locale (#1441); credits sorted in the consumer (dialect-neutral).
- Entries with byline custom fields or a missing/locale-mismatched explicit credit fall back to the existing query path.
* ci: update query-count snapshots
* fix(core): address fold-hydration review findings
- Carry avatarMediaId in the folded byline JSON; it is a required
BylineSummary field templates read to render author avatars, so the
fold dropping it broke avatars on the fast path.
- Fall back to the byline query path when the custom-field probe throws
a non-missing-table error, instead of serving folded bylines with
empty customFields.
- Prime the per-entry term request cache from the folded-terms fast path
(new primeFoldedEntryTerms helper), so subsequent getEntryTerms calls
hit the cache instead of issuing an N+1 query.
- Regenerate the SQLite query-counts snapshot to match the emitted SQL.
- Add a public-shape test (avatarMediaId, term-cache priming).
* ci: update query-count snapshots
* perf(core): drop the taxonomy-defs lookup from folded-term priming
The previous priming seeded [] for taxonomies applicable to the
collection but absent from the entry, which required a getTaxonomyDefs
query on every fold render. That added one round trip to routes that
never call getEntryTerms for an absent taxonomy (contributors, about,
rss, search), eroding the fold's win.
Prime only the wildcard and present-taxonomy keys, straight from the
folded data with no DB lookup. getEntryTerms(id, absentTaxonomy) falls
through to its own cached query instead, which is the rarer case.
* ci: update query-count snapshots
* chore: regenerate D1 query-text snapshot for the hydration fold
The D1 query-text snapshot was last written in #1580 and still showed
the pre-fold path (separate byline and term queries). CI regenerates it
but only commits the count snapshots, so it never got refreshed. Counts
are unchanged; this just updates the recorded SQL to the folded query.
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
The per-query log (EMDASH_QUERY_LOG=1) flushed its recorder when middleware
returned, i.e. when the response headers were ready but before the body
streamed. Queries issued by components during streaming were appended to the
recorder but never emitted, so the query-count harness only saw pre-header
queries. Astro 7's queued rendering moves more queries into the streaming
phase, which made the gap obvious (the post-detail fixture route reported 9
queries while the request actually ran 24).
Flush the recorder when the body finishes streaming instead (in the same
stream-end transform that already snapshots the metrics), and keep a fallback
flush in the middleware finally for bodyless responses (redirects, 304s). The
flush is now idempotent so the two paths can't double-emit.
Also adds a companion query-text snapshot to the harness
(query-counts.queries.{target}.json): a per-route map of the actual SQL to its
occurrence count, so a count change shows which query appeared or vanished, not
just that the number moved. Snapshots regenerated for both targets now reflect
full per-request query counts.
* perf(core): cut redundant queries on content pages
- Cache getWidgetArea per request (it was the only content helper not
request-cached).
- Fetch taxonomy term usage-counts once per request via a shared
request-cached aggregate, instead of re-running the full content_taxonomies
GROUP BY for every taxonomy widget (Categories + Tags each ran it).
- Make getTermsForEntries cache-aware: reuse already-hydrated per-entry terms
from the request cache and only query the misses. Returns private copies so
callers mutating the result can't poison the cache, and orders the miss path
by label to match the hydration primer.
All same-shape, all backends. Adds regression tests for cache reuse and
mutation-safety. Updates the sqlite query-count snapshot (post route -1).
* ci: update query-count snapshots
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* perf: resolve byline avatar storage key during hydration
Byline content-credit hydration now LEFT JOINs the media table so
entry.data.bylines carry avatarStorageKey/avatarAlt, letting list pages
build a direct avatar URL without a per-author MediaRepository.findById
(an N+1). The fields are additive/optional and null on the plain byline
finders. Inferred author bylines (findByUserIds) get the same join.
Also adds opt-in seed support for byline avatars (bylines[].avatar,
hermetic: no download) and a perf-fixture route plus query-count
snapshots measuring the optimized vs naive per-avatar-lookup paths.
* fix: harden seed avatar validation and cleanup per PR review
- deleteMediaRow is now best-effort (logs+swallows) so cleanup failure
can't mask the original byline-write error (emdashbot)
- reject leading/trailing whitespace on avatar storageKey/filename/mimeType,
which are used verbatim in the media lookup (Copilot)
- correct docstrings: storage keys aren't constrained to {ulid}{ext}, and
the new BylineSummary fields are null (not absent) on plain finders (Copilot)
* fix(templates): pin @astrojs/cloudflare to 13.5.1
13.5.2 introduced an astroFrontmatterScanPlugin that breaks any
`virtual:*` module import inside .astro frontmatter (e.g. astro-iconset,
astro-expressive-code). 13.5.0/13.5.1 also expose a cascade-reload bug
in workerd that surfaces as "Astro is not defined" or
"chunk-XXX.js does not exist" on the first request after a clean cache,
unless every transitive dep is pre-bundled.
13.5.1 escapes both: no frontmatter scan, and its dep-resolution fixes
keep the cascade quiet for the deps emdash actually uses. Pinning the
catalog to exact `13.5.1` (no caret) propagates to all four
`templates/*-cloudflare/package.json` files via the catalog sync, so
`create-emdash` scaffolded sites install a known-good combination.
13.5.1 declares peerDependencies.astro: ^6.3.0, so bump the catalog
astro range to ^6.3.0 to satisfy it.
Add a picomatch override to ^4.0.4 so the workspace's transitive
graph dedups to a single version. Without it, `@rollup/pluginutils`
pulls picomatch@4.0.3 on a sibling path that the adapter's
`astro > picomatch` pre-bundle doesn't cover, and workerd hits raw
CJS at module load. End-user installs happened to dedup correctly,
but local dev in this repo did not.
* fix(playground): bind SESSION KV id for wrangler preview deploys
@astrojs/cloudflare 13.2+ writes the auto-provisioned SESSION binding
into both the top-level config and previews.kv_namespaces of the built
wrangler.json, but with no id field. wrangler preview reads from
previews.kv_namespaces and forwards namespace_id verbatim; unlike
wrangler deploy it has no provisionBindings path. The API rejects the
binding-without-id with error 10021.
Declare the SESSION KV id explicitly in the playground's wrangler.jsonc
at both locations so the adapter's customizer skips re-injecting and
the user-supplied id propagates to both. Uses the existing
emdash-playground-session namespace already bound to the production
worker.
* fix(deps): bump @astrojs/node catalog to ^10.1.1
@astrojs/node@10.0.0 calls app.getAdapterLogger() at server start,
which doesn't exist on the new Astro v6 App class. Crashes any node
adapter consumer (including fixtures/perf-site, used by the query-
counts CI workflow):
TypeError: app.getAdapterLogger is not a function
at createAppHandler
at createStandaloneHandler
@astrojs/node@10.1.1 updates to the v6 Adapter API. The astro@^6.3.0
catalog bump that #1116 introduced (required by @astrojs/cloudflare
13.5.1) made this surface.
* ci: update query-count snapshots
* fix(deps): bump @astrojs/cloudflare to 13.5.3, drop picomatch override
Replaces the exact 13.5.1 pin with ^13.5.3, the upstream-shipped fix
(withastro/astro#16801) that reverts the broken esbuild scan plugin
from 13.5.2.
The picomatch override is also no longer needed. The override was
defending against a workspace-only dual-resolve (picomatch@4.0.3 and
@4.0.4 both installed). With the catalog now on @cloudflare/vite-plugin
^1.36.3 and @astrojs/cloudflare 13.5.3, transitive consumers dedup
cleanly to a single 4.0.4 and the workerd dev runtime no longer hits
the unbundled CJS path.
Temporarily exempts astro and @astrojs/cloudflare from
minimumReleaseAge so the catalog can pick up the fix versions that are
still inside the 24h supply-chain cooldown. The exclusions should be
removed once the cooldown passes.
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* perf(core): dedupe taxonomy-def and posts-list fetches per request
Two intra-request duplicate-fetch patterns showed up on the perf-fixture
post-detail render:
1. `_emdash_taxonomy_defs` was fetched three times per render — once
unfiltered (during entry-term hydration) and twice with `WHERE name=?`
(one per taxonomy widget). Have `getTaxonomyDef(name)` peek the
already-cached full list before falling through to a narrower query.
2. `getEmDashCollection("posts", { limit: N })` was issued three times at
different small limits (4 from the page body, 5 from RecentPosts, no
limit from Archives) — each duplicate dragged byline + term hydration
along with it. Bucket small limits up to a shared minimum (10) so
sibling widgets at slightly different sizes share one fetch + slice
to their requested size in the wrapper. Cursor-paginated calls are
exempt; nextCursor is recomputed from the slice boundary.
Trade-off: bucketed fetches over-fetch up to ~10 entries (and the +1
over-fetch the loader already does) for callers who would otherwise have
asked for fewer. On the perf fixture this nets out to fewer queries
overall because the bucket is shared. For a page with a single small-
limit query and no sibling widgets the bucket fetches more rows than
strictly needed, but stays at the same query count.
Cuts the post-detail snapshot from 33 to 27 queries (sqlite target).
* ci: update query-count snapshots
* fix(core): align bucket-slice cursor with loader's encoding
Three fixes for the bucket-then-slice path in getEmDashCollection:
- encodeEntryCursor now skips invalid orderBy field names via the same
FIELD_NAME_PATTERN check the loader's getPrimarySort uses, so a malformed
first key falls back to created_at instead of producing a cursor against
a phantom column.
- For date columns, encodeEntryCursor now reads the raw stored string from
a hidden CURSOR_RAW_VALUES symbol the loader stashes on each entry,
rather than round-tripping through new Date().toISOString(). The
round-trip mints `.000Z` for ISO strings without milliseconds, which
would lex-mismatch the stored value and re-include the cursor row on
the next page.
- sliceCollectionResult docstring updated to match the actual behavior
(early-return for entries already within the limit, shallow-copy only
on truncation).
Adds tests pinning the bucket → slice → re-paginate round-trip, including
exact cursor equivalence with a direct loader call at the same limit.
Resolves Copilot review comments on PR #840.
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* perf: cache site:* settings prefix scan across requests
Site settings change rarely but were re-fetched on every public page
load. A globalThis-scoped holder caches the resolved settings until the
next site:* write bumps its version, dropping one prefix-scan query per
public route. Cross-isolate staleness is bounded by isolate lifetime.
* fix(core): invalidate site-settings cache on partial-write failure
OptionsRepository.setMany iterates row-by-row without a transaction. If
a write fails mid-loop, earlier rows are committed but
invalidateSiteSettingsCache() never ran — leaving the globalThis cache
pointing at pre-write data while the DB has post-write data on some
rows. Wrapping the call in try/finally guarantees invalidation runs
whether the write succeeded or threw.
Adversarial-review finding from PR #800-class perf branch.
* ci: update query-count snapshots
* ci: update query-count snapshots
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* perf(core): drop redundant author_id IN lookup after list fetch
The byline hydration helper used to refetch `author_id` from the content
table for every entry without explicit byline credits, even though the
loader already exposes `data.authorId` on each row. Pass it through from
the call site so the inferred-byline fallback uses the value already in
hand. Saves ~30 queries across the perf-fixture suite, including 4 on
post-detail.
* ci: update query-count snapshots
---------
Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>
* fix(ci): query-counts-apply workflow no longer silently fails
The Apply workflow has been silently broken for ~2 days, never pushing
auto-updated snapshots to PRs that drifted. Two distinct bugs:
(1) Same-repo PRs: the artifact was unpacked into
$GITHUB_WORKSPACE/artifact, but the later actions/checkout step
defaults to clean: true and wipes the workspace, deleting our
payload before the Apply step's cp could read it. Stage under
$RUNNER_TEMP/qc-artifact instead, which sits outside the workspace.
(2) Fork PRs: the cross-check used listPullRequestsAssociatedWithCommit
against the base repo, which doesn't reliably surface fork PRs from
that view, so valid fork artifacts were rejected as 'not associated
with commit'. Replace with pulls.get(prNumber) and verify
pr.head.sha === workflow_run.head_sha — equivalent tamper-resistance
(a forged pr-number won't match), works for forks.
Add pull-requests: read so pulls.get works when the default token
permissions are restricted.
* ci: reconcile query-count snapshot baseline
The Apply workflow has been silently broken for ~2 days, so PRs that
landed in that window (#820, #816, #811) merged with stale snapshots
in the repo. CI's Measure on this PR shows the real per-route counts
on top of current main: uniform +1 cold and +1 warm on all public
routes except /pages/about and /rss.xml. This is pre-existing drift,
not caused by this PR — it's the natural consequence of the Apply
workflow failing to commit the auto-updated snapshots that the
Measure step correctly produced on those merged PRs.
Apply CI's regenerated snapshots from this PR's Measure run so main
ends up in a consistent state once this lands. Future PRs will then
get their own real drift signal rather than inheriting this baseline
correction.
* perf: drop has-any probes, join widget_areas + widgets
Two query-count reductions on the request hot path:
- Widget areas now fetch in one query instead of two — area lookup +
widget fetch collapse into a single left join. Saves one query per
<WidgetArea> rendered.
- Drop the hasAnyBylines / hasAnyTermAssignments probes. They fired
on every hydration call to save a single query on sites with zero
bylines/terms — exactly backwards: the batch join already handles
empty sites for the same cost. Pre-migration DBs (tables missing)
are still handled via an isMissingTableError catch. Saves two
queries per render on pages that hydrate both.
invalidateBylineCache and invalidateTermCache are kept as no-op
exports so existing callers (seed apply, admin routes) compile
unchanged.
Snapshot deltas on the fixture post-detail page:
sqlite /posts/[slug] cold: 34 -> 32, warm: 34 -> 32
d1 /posts/[slug] cold: 43 -> 39, warm: 33 -> 31
Other routes: -1 each from the widget JOIN.
* address Copilot review on the perf cleanup
- widgets: include `created_at` in the left-join select and the
constructed WidgetRow, so the cast is actually sound. No other
caller needs additional fields but WidgetRow declares created_at
as non-null.
- taxonomies: drop the stale "no assignments short-circuit" claim
from the getAllTermsForEntries doc comment — that behavior was
removed with the probe.
* perf: dedup repeat DB queries within a single render
Two classes of fix:
1. `hasBylinesSingleton` and `hasTermAssignmentsSingleton` were module-
scoped, but the bundler duplicates those modules across chunks so
each chunk got its own local singleton — the probe ran per chunk
instead of per worker. Stored on globalThis with a Symbol key (same
pattern as request-context.ts) so all chunks share one value.
2. Wrapped `getCollectionInfo`, `getTaxonomyDef`, `getTaxonomyTerms`,
and `getEmDashCollection` in the request-scoped cache so two callers
with the same args in the same render share one query.
Fixture snapshot deltas:
sqlite post detail: 37 → 34 cold, 35 → 34 warm
d1 post detail: 45 → 43 cold
* fix(query-cache): make collection cache key insertion-order-stable
JSON.stringify is sensitive to object key order, so two callers
passing semantically identical filters in different key orders
would miss the cache and fire duplicate queries. Build the key
from fixed top-level fields, sort where-clause keys (their order
is irrelevant), and preserve orderBy key order (that's the sort
priority — reordering would be semantically different).
Flagged by Copilot review.
* feat: add query-count perf harness + instrumentation
Opt-in Kysely log hook gated behind EMDASH_QUERY_LOG=1 emits per-request
NDJSON on stdout so a harness can count DB queries per route. Zero
overhead when disabled. Exposed at emdash/database/instrumentation so
@emdash-cms/cloudflare can wire the same hook into its per-request D1
session Kysely.
Adds fixtures/perf-site (minimal blog-style fixture, dual sqlite/d1
config), scripts/query-counts.mjs (pnpm query-counts), committed
snapshot files for both targets, and a CI job that runs both.
* fix(perf): invoke emdash CLI directly in query-counts harness
pnpm exec emdash fails in CI because bin symlinks aren't linked for
workspace-local packages (see scripts/relink-bins-if-needed.mjs, which
early-exits under CI). Invoke the built CLI entry by absolute path
instead so the harness works in both CI and local dev.
* ci(perf): build all packages for query-counts job
The fixture config imports from @emdash-cms/cloudflare for the d1 path,
so `pnpm run --filter emdash... build` (which only walks emdash's
deps, not its dependents) leaves cloudflare unbuilt and astro fails
to resolve the import when loading the config.
* fix(perf): wait for TCP port instead of parsing stdout for ready
The ready-regex approach was fragile — in CI, the cloudflare adapter's
dev mode wraps output in [vite] prefixes and the "ready in" line
sometimes never matches (observed on the D1 seed step: typegen POST
succeeded but ready timeout still fired).
TCP-connect is the real question anyway ("is the server accepting
connections?"). It also doesn't warm a fresh workerd isolate —
workerd defers isolate creation to the first HTTP request — so the
per-route cold-isolate measurement stays honest.
* fix(perf): seed D1 before building for preview
`astro dev` (the seed step) leaves .wrangler/deploy/ without the
build-time config.json that cloudflare adapter's preview requires, so
running `astro build` after the seed is what makes the subsequent
`astro preview` spins work.
logelog