Round 3 adversarial-panel fixes for PR #999:
- Snapshot writes now go through a per-queryKey file plus a CAS re-read
guard before rename, closing the last-writer-wins race where a slower
recompute against an older corpus could clobber a fresher save, and
where distinct queryKeys used to evict each other's single shared slot
(B-G1).
- cline-cli's discoverSessions now fingerprints the growing
<sessionId>.messages.json sibling instead of the static
<sessionId>.json metadata file, so new turns in a live session are no
longer invisible to computeCorpusFingerprint (C-G1).
- computeCorpusFingerprint folds in computeEnvFingerprint per discovered
provider, so non-discovery env vars like CODEBURN_CURSOR_MAX_BUBBLES
and KIMI_MODEL_NAME can no longer serve a stale-forever snapshot (A-G1).
- collectFilesRecursive now resolves symlinked subdirectories and
recurses into them (with a visited-inode guard against cycles) instead
of misclassifying them as leaf files (C-G2).
- Added tests/session-cache-status-snapshot.test.ts covering concurrent
writers against the snapshot file, mirroring session-cache-shards.test.ts's
existing coverage for the main cache (D-G9).
Spent credits are finite nanoAiu / 1e9. Token-priced costUSD does not
fill the Copilot bar. Thread nanoAiu through cachedCallToApiCall with
no cache-version bump.
Resolves the conflicts #946 accumulated while it was in validation. Eight
files conflicted; the session-store accounting is unchanged.
src/daily-cache.ts — version collision. This PR minted 25 when main was at
24; #1056 (`codex-auto-review` pricing) then spent 25 on main. The bump moves
to 26/MIN 26 and daily-cache.v26.json, with main's full comment ladder kept as
the foundation and this PR's paragraph rewritten to name 26 and record the
collision. PENDING_REDERIVE_PROVIDERS and the B1 migration semantics from
b6481c19 carry over intact, retargeted at 26.
src/models.ts, src/parser.ts, src/audit-report.ts, src/models-report.ts —
#1075/#1078 replaced the per-site "reasoning is already inside output" tests
with billableOutputTokens() and REASONING_INCLUDED_IN_OUTPUT. This PR had
added copilot to that case at three sites independently. Union: all three
sites take main's helper call verbatim, and copilot joins claude and codex in
the set — same accounting this PR shipped, now through main's single source of
truth. It also reaches parser.ts activeGeneratedTokens (a fourth site, from
#1079), which is the same correction: a copilot supplementary call carries
reasoning with output 0, so counting it as generated repeats the per-turn
output. The audit legend already said so on this side.
src/providers/copilot.ts — comment-only. #1054's lastEventTimestamp-first
shutdown fallback was derived from this branch, so the code was already
identical on both sides: the `shutdownTimestamp` expression and the
`copilot:<sid>:shutdown:<model>:<n>` key are byte-for-byte main's. Both
rationales are kept (leg-collapse on date, and residual anchoring).
src/session-cache.ts — PROVIDER_PARSE_VERSIONS.copilot takes this PR's
`-session-store-v3` suffix; main's #1051 note about why a fingerprint change
is expensive is kept above it. Codex keeps main's #1092 suffix chain untouched.
src/main.ts — #1067 deleted the unreachable live dailyMap fallback that this
PR had taught behavioral weight. Main's deletion wins; the now-unused
isBehavioralTurn import goes with it.
tests/parser.test.ts — import union.
Also: scripts/upgrade-path/run.mjs NEW_DAILY_CACHE -> daily-cache.v26.json,
CODEBURN_COPILOT_SESSION_STORE_DB added to the #1064 env-isolation CLEARED
list, and the CHANGELOG entry's stale "v21" corrected to v26.
Verified: tsc clean; 3132 tests pass across 223 files; test:locks 26/26;
verify:upgrade PASSED, re-deriving daily-cache.v26.json and holding durable
copilot history across the bump. #1054's regression ("keeps three stampless
shutdown legs as :n keys with lastEventTimestamp") passes on the merged tree.
Real-corpus A/B against origin/main over 2026-07-01..2026-08-22: codex, grok,
kimicode and opencode byte-identical in export, audit and models; claude drifts
only monotonically with run order (a live session writing transcripts, confirmed
by interleaving four runs). This machine has no copilot data, so the copilot
recovery semantics rest on the suites and the upgrade-path corpus.
Mixed-version binaries were clobbering the unsuffixed Codex, Cursor,
and Antigravity result files and each re-parsed the whole corpus.
Write *-results.v<n>.json like the daily cache. Leave the unsuffixed
file for older binaries; adopt a matching-version copy once.
#1070 is hygiene, not a leak. User docs no longer name
deriveTraceId or list session ids under What is NOT sent.
Code comment matches the wire key. Class test pins every
attribution span: no ai.session_id, join is shared traceId.
Attribution spans already share deriveTraceId(sessionId) with
usage spans. Emitting the raw session id was redundant and
undid the pre-attribution wire rule that session id is hash
input only. Receivers upsert session spans by that keyed
traceId. Usage spans never had the field. Docs match.
Merge resolution
- daily-cache: main shipped v20 with #1040, so 21 is now the first FREE number
rather than one dodging an unmerged head. Both version notes kept, ours on
top. The re-derivation seed stays one below current (20), now the shipped
predecessor rather than a draft.
- upgrade-path: expected daily cache filename stays v21.
The rollup is cumulative across resume legs — already handled, now pinned
Measured on a real 3-leg session (CLI 1.0.80): every counter in a leg includes
the legs before it, and the last leg of a complete session equals its store-row
total to the nano. The parser has always emitted `cumulative - previous
cumulative` per model, so the interval arithmetic downstream already consumes
per-leg claims; it was the PR body and the provider doc that described the raw
journal as per-leg and misled the reader. Doc corrected, and (c3) pins it end
to end: a complete 3-leg session serves 900 (its last leg / store total), not
1,400 (the sum of its legs); an uncovered one serves 900 exactly once; and a
leg reporting LESS than its predecessor is taken as a fresh epoch rather than
clamped to a negative delta, so an older per-leg CLI never loses a leg.
initiator = 'compaction' replaces the timestamp heuristic where it exists
The summarization request does write its own assistant_usage_events row, and
newer stores label it. The label is now read — schema-adaptively, and the
enrichment select is a graduated chain so a store carrying the billing columns
but not `initiator` keeps its billing metadata instead of falling all the way
back to the base select. Two uses: the row is subtracted from the leg it
belongs to even though it commits before the compaction stamp, and it is kept
out of per-turn pairing since it has no assistant.message to pair with.
Optional twice over — absent on older stores, NULL on 1,504 of 2,509 rows on a
real one — so (c5) pins the labelled path at 350, the identical UNLABELLED
fixture at 400 (the documented one-request-per-compaction over-serve, which no
timestamp rule can close: the request that triggered the compaction completes
immediately before it too), and a compaction row never stealing a pairing
partner.
Attribution invariant, standing guard for the 1.8x report
(c4) runs `codeburn audit`'s own two numbers — attributed vs recomputed —
over all four combinations of the three representations a session can be
written in, at the magnitudes of a real reported day (gpt-5.6-terra, 146 rows,
input 17,792 / cache write 501,395 / cache read 12,097,364 / output 63,344 /
reasoning 24,831, billed $4.47). Every shape reconciles at 1.000, and the
covered case lands on $4.4687 = what the tokens price at = what GitHub billed,
with output 63,344 rather than the pre-fix 88,175 that re-priced reasoning on
top of itself.
upgrade-path: a day KEY is not the unit of never-lose
A parse change that re-dates a call to its true day legitimately empties one
day and fills its neighbour, token for token — observed on a real cache where
2026-08-08's single call moved to 08-07 exactly. The aging step now compares a
+/-1 day WINDOW rather than the day alone, so only a window that shrinks is a
loss, and an emptied day key is reported rather than failed.
RECONCILE_SETTLE_MS keeps 24h, with the measurement that argues against it
recorded at the constant and in the provider doc: across 91 real sessions zero
rows landed after shutdown (median -0.1s, max -0.0s). One machine, one CLI
version — the number to beat is seconds, not hours, once a second machine
agrees.
B-1 The freeze was one-directional and left the reverse hole wide open. Rows
sync; at the 90-day durable age-out the cached rows are pruned, the rollup
stops being dropped and serves again under a key that was never sent; the
session is long past settle, so it pushes an aggregate on top of the
per-request spans the receiver already holds.
Reframed around the thing that actually matters: a copilot session's
input/cache leaves this machine in one of two SHAPES — the raw rollup
(`copilot:<sid>:shutdown:`), or reconciled output (rows plus
`:shutdown-residual:`, which are disjoint by construction and together are
exactly the rollup re-expressed). The receiver must never hold both. Whichever
shape a session was first synced in, it stays in, and the other is frozen for
that session permanently — in both directions. Growth WITHIN the sent shape is
untouched, because same-shape output is additive, never substitutive: a
resumed session's new rows and residuals still push if rows were sent, a new
leg's rollup still pushes if rollups were.
This also corrects the previous commit, which classed a residual as aggregate
and so froze it for a session whose rows had gone out — the residual is
reconciled output and belongs with the rows.
B-2 A session with an unparseable timestamp was held forever while the CLI
promised it would "push once it settles". Nothing could ever settle it. Settle
now reads the newest moment a session can be SHOWN active; a session with
nothing datable at all is sent rather than held. Mixed sessions still settle on
their datable stamps.
B-3 A stamp implausibly far in the future is broken data, not evidence the
session is live, so it no longer counts toward that newest moment. One
year-ahead row can no longer hold a month-old session hostage. Ordinary clock
skew is absorbed by a one-hour grace in both directions and still reads as
live.
B-4 The residual dedup key carried the leg's POSITION. Legs sort across every
cached file for a session, so an earlier leg arriving later renumbers every
residual after it — and a renamed key is a span the receiver takes a second
time, since there is no retraction for a usage span. Keyed by the leg's own
instant instead: append-only files mean it never moves, and the equal-timestamp
coalescing above makes it unique per leg. Residuals are new in this change and
have never shipped, so no CACHE_VERSION concern.
B-5 The dry-run "already synced" count now subtracts frozen too, matching the
nothing-to-push line.
Tests: symmetric freeze from the rows side (rollup frozen, residual and new
rows still sent, per-turn untouched); all-unparseable sent, partly-unparseable
still held; a year-ahead stamp ignored while a one-minute skew still holds;
residual keys named after their leg instant with the positional names gone. The
key-prefix pin picks up the residual's new tail. (c2) pins the property rather
than one insertion scenario, because a second file's leg currently collides on
the rollup's own dedup key before it can reach the residual sweep — the
reachable repro would prove nothing about the next one, and that is stated at
the test.
Flakes: (f2) and (sc) re-bucket each run 10x isolated and 10x with four busy
loops pinning cores — 40/40 clean. Neither has clock or ordering dependence at
its margins: afterEach removes both tmpHome and the cache dir, and the two
relative timestamps in the re-bucket test are 35 days apart so they cannot
share a month. The likeliest cause of the transient failures is this branch's
own commits rewriting src/parser.ts under a concurrent vitest. (f2) gains a
self-check that its fingerprint sentinel really differs from the computed one,
which is the one way it could have passed while exercising nothing.
F1 — the upgrade-path guard was measuring the wrong layer. providerDetails
come from the menubar payload, and on a version bump adoptOlderDailyCaches
carries the superseded daily file forward as the baseline for exactly the days
this scenario creates, so a session-cache loss was mostly masked: reverting the
carry-forward showed a 10% dip where the real loss is 50%. It now sums
export.json's per-call copilot records (straight out of the parse), deletes the
baseline daily cache before the upgrade run so nothing else can answer, and
asserts EQUALITY rather than "did not shrink" — the failure mode on the other
side of the fix is a double. Proven: reverting the carry-forward now reports
80 calls -> 40, LOST history, and fails. (The double direction is asserted but
not demonstrated; a faithful re-keying simulation is collapsed by serve-time
dedup before it can be counted, which is noted at the step.)
F2 — sync was forward-only in the wrong direction. A receiver that took
`copilot:<sid>:shutdown:<model>:<n>` rollup spans from 0.9.20 keeps them
forever (append-once, no retraction for a usage span) and would now also
receive `copilot-store:` rows for the same sessions: permanent doubling on
every synced receiver, on the first push after upgrade. Sessions whose rollup
is already in the ledger are now FROZEN — their rows and residuals never go
out, and the receiver keeps the older, lossier number. A bounded under-count
in place of an unbounded over-count. Per-turn output spans are untouched
either way (the rollup never held output, reconciliation never touches them),
and sessions new to sync take the per-row path in full. `frozen` is reported
by --dry-run and the nothing-to-push line; docs/sync/README.md gains an FAQ
pointing at `codeburn sync reset --confirm` with the caveat that the receiver
must be cleared too.
F3 — docs/providers/copilot.md gains a Sharp edges section saying plainly that
a day sealed on a short store snapshot is a permanent UNDER-REPORT the
watermark advances past, not a stall, and that both realizable causes are now
closed (read ordering, compaction anchor).
F4 — the copilot dedup-key shapes are pinned by reading them back out of the
source, minted keys and discriminator prefixes alike, with the reason: the
durable union recognises history by key, so a changed prefix is a
CACHE_VERSION bump, not a parse-version bump.
F5 — the carry-forward comment now states its price: a bump no longer
re-derives a call it carried forward, so cost-shaped fixes land (cost is
recomputed from cached usage) but metadata and day-attribution fixes reach
only newly parsed calls.
F6 — the durable union filters at CALL level. A turn holding one cached call
beside a new one used to be dropped whole; nothing enforced the one-call-
per-turn assumption that made that safe.
F7 — the retain-source fence no longer skips a store it cannot fingerprint.
Present-but-unreadable defers, matching the classification path; only a
genuinely absent one stays a skip.
The CLI's session.shutdown rollup RESETS its counters at a successful
in-session compaction, so a leg containing one describes only its
post-compaction requests. Running the store-row subtraction from the previous
leg cancelled that leg's usage against the whole pre-compaction conversation
and left the residual short by exactly that much. With a complete store the
floor hid it; with a partial snapshot it was a permanent undercount, and a day
sealed on it never corrected. That is the half of the reviewer's blocker 2 the
read-ordering fence could not reach: no timing rule helps when the rows are
simply not there yet, and rollup > rows-in-interval is indistinguishable from
legitimate partial coverage.
Anchor at the compaction instead. Pre-compaction rows still SERVE; they just
stop cancelling usage the rollup never claimed. The maintainer's repro (row A
100 pre-compaction, sole rollup 200 for post-compaction request B, B's row
absent) now totals 300 instead of 200, and still totals 300 once B lands.
Event names taken from @github/copilot 1.0.80, not guessed:
session.compaction_start (nothing needed from it) and
session.compaction_complete, whose `success` is read and every other field
ignored — the payload also carries the summarization call's own usage, token
counts, model and trigger. Only success:true anchors; failed or absent falls
back to the previous leg; last compaction in a leg wins. The stamp rides on
the cached rollup call as `compactedAt` (add-only, validated optional).
Accepted, bounded over-serve, documented at the doc site: if the summarization
call writes its own store row it sits just before the compaction stamp, so it
falls outside the interval while the post-reset rollup counts it — one request
per compaction can serve twice. Not subtracted from compactionTokensUsed,
because the request that TRIGGERED the compaction also completes immediately
before it: any grace window wide enough to catch the summarization row also
catches a real request and turns an over-serve into a loss. The row writer is
native, so the bundle cannot settle it; a real store can.
(c1) pins all four shapes: the repro at 300 (and again through the cache), no
double once the row lands, a failed compaction ignored, and a leg with no
compaction unchanged. The existing 107 copilot tests stay green.
The sent-ledger is append-once and the OTLP span id derives from the same
deduplication key, so the pipeline assumes a served call is immutable: same
key, same value, forever. Copilot's serve-time reconciliation is the first
producer that breaks that (#988), three ways inside one session — a shutdown
residual shrinks as the store rows covering it land, a rollup is dropped once
rows cover its leg, and an unpaired row becomes supplementary when its journal
call appears. Sent at an intermediate state, the receiver keeps that state
forever AND receives what supersedes it. Local reports re-reconcile every pass
and were never affected; this is only about what leaves the machine.
Value-versioned keys were the other option, but usage spans have no upsert
contract — receivers sum them, so a superseding span adds rather than
replaces. (Attribution can do it because its spans are documented as upserts
by `ai.session_id`.) Giving usage spans one is a receiver protocol change.
So: hold, do not re-key. Every input to the reconciliation is written during
the session, so a session quiet for 24 hours cannot reconcile further and its
first send is also its last word. The whole session is held, not just its
residual — holding the residual alone still ships a row whose pairing can
flip. Nothing is dropped: `held` is reported by --dry-run and by the
nothing-to-push line, and the next push after the window sends it.
Scoped by provider because copilot is the only reconciling producer; every
other provider's calls go out unchanged and undelayed.
Resolved against main's newer invariants:
- daily-cache: DAILY_CACHE_VERSION/MIN_SUPPORTED -> 21. Main shipped 19
(#1015 grok); 18 was burned by this branch's earlier public head and 20 is
claimed by the unmerged #1040, and isMigratableCache would adopt either as
finalized without re-deriving. Header keeps main's v19 note and adds v21's.
- session-cache: PROVIDER_PARSE_VERSIONS keeps main's grok/dsh entries and
appends this branch's `-session-store-v2` to main's copilot value.
- parser: durable age-out takes main's orphan-only rule (#992).
retainWhilePresent is now redundant under it rather than load-bearing.
- audit-report: both imports (isBehavioralCall + sanitizeModelForDisplay).
- tests/parser (e), (l), (sc): re-pinned to the orphan-only age-out. A
still-discovered >90d copilot events.jsonl now keeps serving its per-turn
output alongside the store rows; the (sc) age-out subject is now an orphan.
- daily-cache-version-rederivation: seed 20 so the adjacent-draft-version
case is what the test pins.
- scripts/upgrade-path: expected daily cache filename -> v21.
Resolves the single conflict in src/main.ts: upstream added the
sanitizeModelForDisplay import (models.js) alongside this branch's
own additions to the same import line (the four pricing-config-hash
getters, computeCorpusFingerprint, isSessionHydrationComplete).
Both sets are disjoint and both are used elsewhere in the file
(sanitizeModelForDisplay at line 2209, the fix imports throughout
the status-snapshot query-key/gating logic) — merged as a union,
no logic changed. src/parser.ts and src/session-cache.ts auto-merged
cleanly.
Responds to ozymandiashh's CHANGES_REQUESTED review on #999.
Structural: kept the disk status-snapshot, scoped to serve's documented
CLI-spawn-fallback path, after bringing its invalidation to parity with
serve's own in-process memo (see docs/design/perf-cache-fix.md for the full
reconciliation reasoning against #972's serve.ts/watcher invalidation,
already merged into this branch's ancestry).
Correctness fixes:
1. queryKey now includes the same four pricing-config hashes parser.ts's
own cacheKey deliberately includes (proxyPaths/modelAliases/
priceOverrides/localModelSavings), so an edited pricing config
invalidates the snapshot instead of waiting for an unrelated corpus
change.
2. computeCorpusFingerprint now gives network providers (e.g.
vercel-gateway) a now-stamped synthetic fingerprint entry every call,
mirroring parseAllSessions' own treatment of the same sources, instead
of letting them contribute nothing to the hash.
3. computeCorpusFingerprint generically expands any directory-shaped
non-Claude source (e.g. mistral-vibe) via a new collectFilesRecursive
helper, instead of only Claude getting leaf-file expansion.
4. saveStatusSnapshot is now gated on isSessionHydrationComplete(), so a
degraded read-only parse can no longer be checkpointed as authoritative.
5. The settle-window defer in loadStatusSnapshot now anchors on wall-clock
time since the fingerprint FIRST mismatched (persisted via a new
mismatchFirstSeenAt field) instead of the corpus-wide newest mtime,
which fix#2 can otherwise keep perpetually "recent."
Also fixes a 6th instance of the same defect class as #1, found while
root-causing a test failure: the query key omitted the active display
currency, so switching currencies kept serving stale numbers. Updates
tests/parse-workers.test.ts, whose "warm" assertion assumed every
menubar-json call does a full parse — no longer true within the settle
window after this PR's own snapshot feature.
The month-scoped load a ranged query takes is a behaviour change on a warm
cache with no way back except deleting it. Drop the scope in loadCache, the
one place every caller (including the resident serve) routes through, so a
suspect scoped read can be compared against a full one in place.
Read policy only: deliberately not in PROVIDER_ENV_VARS, so setting or
unsetting it invalidates nothing.
The daily-cache re-derivation test seeded v18, a version that only ever
existed as an unreleased draft of this change. Seed the shipped v17 so the
test models the 17 -> 19 upgrade path users actually hit, and rename it: the
bump re-derives every day for every provider, not just Grok, because the
daily cache has no per-provider invalidation. The Grok day stays as the
fixture since Grok is what the bump exists to correct.
The changelog entry now says outright that Grok totals change materially on
upgrade (150K -> 96.3M cache-read tokens on a 568-session corpus), that a
turn without a turn_completed record inside an otherwise-covered session is
dropped rather than estimated, and that the one-time daily re-derivation
reads the warm session cache and keeps the superseded file. The
context-bloat denominator fix moves to Fixed and names the providers it
corrects.
docs/providers/grok.md gets the same undercount warning in the token model
and a matching entry under Quirks.
README gains a Windows section next to the macOS and GNOME ones, a download
badge on the menubar card, and an honest note that the Tauri tray builds on
Linux but is unreleased there. docs/architecture.md picks up windows/ in the
surfaces diagram and gets a section covering the crate layout, the PATH and
System32 spawn rules, and the Claude quota parity.
`codeburn menubar` on Windows now points at the windows-v release page instead
of failing with "macOS only". The generalized installer from the source branch
is not brought over: it is 1182 commits behind this file and would drop the
proxy support, retry/backoff, checksum and bundle verification, and persistent
CLI path handling that landed since.
Groups sessions by their opening block (whitespace/ANSI-normalized, hashed
over the first 2 KB) and flags a block of at least 1.5 KB that opens five or
more sessions. Class nudge: CodeBurn will not move the user's own text into
CLAUDE.md, so the fix asks Claude to give the block a permanent home. Only
the repeats count as savings, sized from the block's bytes because provider
usage is per API call and cannot isolate the paste. The opener comes from
the session scan that already runs, so nothing extra is read.
Adds docs/optimize.md (what optimize scans, the three classes, the exact
files --apply may touch plus undo, measured vs estimated, the health
grade bands, the --yes CLAUDE.md guardrail), links it from the README
waste section, and corrects the detector count in docs/architecture.md
(14 -> 19).
Grok CLI writes a turn_completed update carrying a full usage object --
inputTokens, outputTokens, cachedReadTokens, cacheCreationTokens,
reasoningTokens -- into the same updates.jsonl the parser already reads. We
ignored it and reconstructed an estimate from _meta.totalTokens, a running
context-size counter that rides on unrelated events, with a
total < prevTotal * 0.5 reset as the turn boundary.
On the cache-heavy session reported in #998 that reconstruction captured about
1.4% of the real cache-read volume and roughly 6% of the day's tokens, while
over-counting output about fivefold. cacheCreationInputTokens and
reasoningTokens were hardcoded to zero regardless of what the session held.
The parser now reads turn_completed.usage, keyed by the record's snake_case
prompt_id so a re-emitted turn cannot double count, and sums across turns.
Two decompositions matter, both derivable from the reported numbers:
totalTokens equals inputTokens + outputTokens exactly, so cachedReadTokens and
cacheCreationTokens are subsets of input and are subtracted out per record
before pricing, matching the cache-exclusive convention codex and copilot
already use; and reasoningTokens is a subset of output.
That second one needs care, because the repo contract is the opposite of
Grok's: ParsedProviderCall.reasoningTokens is exclusive of outputTokens
everywhere, and every consumer sums the two -- tests/providers/kiro.test.ts
says so outright. So reasoning is clamped to the reported output and output is
emitted without it, and the downstream sum reconstructs Grok's number. Without
the clamp a record with reasoning > output produced a negative output and left
the pipeline pricing reasoning instead.
Multi-model attribution is deliberately out of scope. modelUsage only selects a
priced attribution id; a session that used two models is priced at one rate.
Splitting per model was tried and dropped: chooseAuthoritativeModel's
priced-id fallback exists to avoid a truthful-but-$0 row when modelUsage names
an id this checkout cannot price, and per-model pricing loses it -- the
reporter's own session collapsed from $1.20 to near zero the moment a second
id appeared.
When no valid completed record exists -- older Grok CLI versions -- the old
heuristic still runs, unchanged. The decision is taken from the deduplicated
records rather than latched per line, so a superseded or all-zero record cannot
flip a session off the heuristic and drop it. A session only partly covered by
turn_completed records keeps costIsEstimated: true rather than presenting
itself as fully provider-measured.
costUsdTicks is deliberately not read. Its scale is undocumented, and guessing
it would fabricate spend.
Bumps the grok parse version, and DAILY_CACHE_VERSION with
MIN_SUPPORTED_VERSION together, since the daily cache serves every day before
today and retains ten years. Moving them in lockstep is what keeps the
carry-forward lossless: the filename is version-suffixed, so the old file stays
on disk and is adopted for days no source can still re-derive.
Separately, detectContextBloat divided by outputTokens alone. Reasoning is
stored beside output for every reasoning-bearing provider, so the detector saw
a fraction of the generated tokens and invented high-impact findings -- a
session whose provider-reported ratio is 20:1, below the 25:1 threshold, was
reported as 133:1 with 710K tokens of claimed savings. It now uses the same
output + reasoning sum the reports use, which fixes codex, kiro, hermes, qwen
and cursor-agent too.
Reported in #998.
The Copilot CLI and the GitHub Copilot desktop app both write
~/.copilot/session-store.db unconditionally; its assistant_usage_events
table holds one row per API request. Until now input/cache tokens for
these surfaces came only from the session.shutdown rollups in
events.jsonl, which are written only on clean shutdown (a crash loses
the whole leg's input/cache accounting) and lump each session leg into
one per-model total. The rollup also RESETS its counters at in-session
compaction (traced on a clean single-process 107-request session whose
sole rollup covered exactly its five post-compaction requests), so even
cleanly-closed long sessions were truncated; on a long-history machine
the store recovered ~35% of real Copilot spend lost to crashes and
compaction resets. The DB rows are per-request, crash-proof, and carry
real timestamps.
The store's input_tokens is cache-INCLUSIVE (input + cache_read +
cache_write), the same convention as the shutdown rollups — verified
against each row's token_details_json and by reconciling per-session
sums against the CLI's own footers and rollups across two machines
(1,380+ rows, 8 models, CLI 1.0.70–1.0.79, schema_version 6): every
divergence was a rollup gap. Emitted calls mirror the shutdown-call
contract: input/cache/reasoning only, output 0 — per-turn output stays
owned by the events.jsonl assistant.message calls.
Rollup-vs-store precedence is RECONCILED at serve time, per
(session, model), and only there. Both representations always parse and
cache; parseProviderSources aggregates the cached calls and, wherever
store rows exist for a (session, model), drops the rollup calls and
serves the rows plus per-leg RESIDUAL calls: each rollup leg subtracts
only the rows in its own interval — rows commit strictly before their
leg's shutdown line, so a leg at time T covers exactly the rows in
(previous leg's T, T] — and any remainder (per token component, floored
at zero) serves once at that leg's own timestamp. A store missing
requests a leg covered — adopted mid-session, rows pruned before ever
being read — therefore still serves that tail exactly once ON THAT
LEG'S DAY, a crash-tail row the rollup never saw can never cancel it,
and a complete store serves pure per-request granularity with every
residual retired to zero. The decision reads only cached contents, never discovery:
deleting or resetting the store changes nothing served, so finalized
daily history can never flip on an absence epoch; cached rows of a
deleted store remain the record until the 90-day orphan age-out (which
exempts still-discovered paths). The serve set is the one coherent
snapshot — nothing a writer does between discovery and a parse can
change what one pass sees — and read-time precedence heals persisted
duplication (stale epochs, runtimes without node:sqlite, restored
files) instead of preserving it, following the buildDurablePeriod
pattern.
Store rows and rollups carry supplementary accounting weight. A rollup
(or its residual) is aggregate accounting, never a request: zero
api-call/model-call/turn weight, tokens and cost fully retained. A
store row is one real request, but when it pairs with a served per-turn
call it is supplementary too; rows pair with same-model per-turn calls
by timestamp adjacency (monotone matching, tight 2-minute window — the
two are written at the same completion moment, and a wide window would
let a crash-only row pair against a neighbor whose own row is missing),
computed once over the FULL serve set so a date-range boundary that
separates a row from its call cannot double the request across adjacent
day queries. Only the unpaired rows — store-only requests, exactly
where crash-lost requests sit — count. Supplementary-only turns fold
into the nearest behavioral turn within 30 minutes; with no behavioral
turn to fold into they stay separate weightless turns, each on its own
day, with apiCalls 0 — and the session emission gate admits
usage-bearing zero-call sessions. The weight
propagates into the daily cache: aggregateProjectsIntoDays applies the
same rule to every calls counter and category-turn count it seals, so
v19 history and live summaries can never disagree about what was a
request.
A changed source whose read defers on the busy shape (locked, EACCES,
corrupt mid-replace — discovery still emits the source; only true
absence or a schema mismatch reads as absent) now marks session
hydration incomplete, so the daily backfill holds its watermark instead
of finalizing a day the deferred rows never reached; an unchanged
unreadable store defers nothing. The verdict travels with its result —
the 180s memo and the serve burst-reuse restore the hydration verdict
their cached data was parsed under, so a memoized partial parse cannot
inherit a later parse's complete — and a discovered source whose
FINGERPRINT cannot be read (EACCES on a present file) defers instead of
silently skipping, while a genuinely deleted file stays a silent skip. Copilot reasoning tokens are no longer
double-billed at the report layer: they are a subset of the output the
per-turn calls already price, and copilot joins claude in the
reasoning-inside-output case of the query-time cost recompute.
Store dedup keys are content-discriminated —
copilot-store:<sid>:<rowId>:<fnv1a64(created_at|tokens|model)> —
because AUTOINCREMENT prevents id reuse only within one database
lifetime: a same-path DB reset reusing row ids now mints new keys
instead of the durable union swallowing the new usage, while a
byte-identical re-insert still collapses (64-bit: 32-bit FNV
collisions between plausible token tuples are constructible). Every
call of a session serves under one project label resolved at serve
time — the session-state-derived label when the serve set knows it,
else the store rows' own — so neither rows cached before events.jsonl
existed nor an events.jsonl orphaned by a session-state prune can
split the session across two grouping keys.
CODEBURN_COPILOT_SESSION_STORE_DB is read but deliberately NOT
fingerprinted, per the #927 ruling (any copilot fingerprint change
drops cached entries whose path still exists, destroying pruned history
only the cache holds); the read is allowlisted in the #927 guard, and
serve-time reconciliation makes repointing safe without a fingerprint —
the new store's rows parse on sight and the old path's entries persist
as durable orphans. The copilot parse version appends session-store-v2
and the daily cache bumps v17 → v19: per-day attribution, call counts
and costs all change against pre-store builds. 19, not 18: an earlier
pushed head of this PR already claimed v18 under different accounting,
and the carry-forward would adopt those days as finalized without
re-deriving them.
Verified by A/B on snapshots of two real stores, a live SIGKILL crash
test (row present, no rollup, tokens recovered exactly), live resumes
whose warm-cache deltas matched new rows to the token, upgrade-healing
at 4,800-session scale, and serve-level regressions pinning every
maintainer finding from six review rounds: the rows-then-shutdown race,
stale-cache healing, age-out exemption, absence-epoch identity,
progressive row landing with residual retirement, behavioral weight
across all four pinned scenarios, the hydration fence, project
unification in both directions, the same-path reset, mixed
coverage (crash tail vs covered-leg gap), multi-leg residual day
attribution, range-invariant pairing, memo-scoped hydration verdicts,
and the fingerprint-failure fence.
docs/providers/NEW_PROVIDER.md items the PR had not reached yet, plus the two
surfaces that are functional rather than cosmetic:
- docs/providers/dsh.md and its row in the provider index, documenting the
storage layout, the JSONL-backend-only scope (the opt-in SQLite persistence
backend is not read), and that DSH is a developer preview whose format
version 0 implies no compatibility.
- CHANGELOG entry under Unreleased.
- README provider count 40 -> 41 and a data-locations row.
- app/package.json: $HOME/.dsh in the snap personal-files allowlist, without
which the Linux snap build cannot read DSH sessions at all.
- UsageDataChangeGuard: the DSH sessions root, without which the menubar never
notices a new session and does not refresh.
- Bumps the dsh parse version, since the parser's attribution changed.
Three fixes from review, all measured on this box.
The workload gate was files OR bytes. The files arm is wrong: 250 pending files
holding 117 KB between them spawned 5 threads and ran ~5% SLOWER than serial,
and a file count only starts paying for itself around 400. Gate on bytes alone;
the count still takes max(files / 50, bytes / 200 MB), so a few hundred huge
rollouts keep their threads.
The flat 256 MB per-worker memory budget was contradicted by the Codex workload:
a 260 MB rollout peaks near 430 MB in its worker, linearly across the pool. It is
now derived per parse as clamp(256 MB, 2 x average pending file + 128 MB, 1 GB),
which leaves a corpus of small Claude transcripts where it was and stops
over-subscribing on rollouts. The parent's buffer of up to pool.size finished
results is part of that peak and is named in the comment.
The worker/file pairing at both install sites was positional, guarded only by
position (Claude) or a path membership check (Codex). Each worker now echoes its
path and the parent asserts it, outside the per-file try: a misalignment would
install one session's turns under another's path -- a wrong number nobody would
ever notice -- so it fails the run rather than being swallowed as a parse
failure. On the Claude side that meant hoisting the whole worker-result block
above the try, which is safe because an append never consumes a result in either
its shortcut or its straddled-fallthrough case.
The comment at the install site claimed only that an overlapping worker result
'is discarded'. State why the empty-set result is installable at all — an empty
id intersection is proof a serial parse would have dropped nothing — and why the
tempting shortcut is wrong: parsedTurnsToCachedTurns delta-encodes gitBranch
across turns, so dropping one turn changes whether a LATER turn carries a
gitBranch key. Overlap discards the whole file, never individual turns.
Tests: the end-to-end determinism check now runs both parses over the SAME
corpus, so cache shard BODIES are compared byte for byte instead of just their
keys, and a new resumed-session fixture (a transcript restating another file's
message ids, in both filename orders) makes install order decide the answer.
Verified by mutation: removing the discard guard fails it, and yielding worker
results out of order fails it.
CODEBURN_VERBOSE now reports how many worker results were re-parsed in-process
on id overlap, which is what the new test asserts on. The worker bundle's source
map is excluded from the published package (-1.8 MB).
os.freemem() reports free pages on macOS, not available memory: on an idle
128 GB machine it reads a few hundred MB, so the 2 GB gate switched the worker
pool on and off between runs on the platform the desktop app ships to. The gate
and the budget now use process.availableMemory() (cgroup/rlimit-aware in a
container), falling back to os.totalmem(): serial under 4 GB available, budget
min(0.25 * available, 2 GB). An 8 GB box earns 8 threads, a 4 GB box none.
The verbose line now carries every decision input — cores, available GB, pending
files and bytes — on both the gate and the go path, so one support log explains
itself.
flatSlice returned strings within the bound unchanged, but provider adapters
pre-truncate with .slice(0, 500) before the cache site, so those views still
pinned their parent buffers. Always flatten; the round-trip is ~150ns per turn.
Use utf16le so lone surrogates survive the copy.
Cache the canonical-path Promise instead of the resolved value so calls in one
Promise.all batch share a single walk.
Document the one-time kiro re-parse and worktree regrouping.
vitest's default glob reached the Electron app's specs under app/, which carry their
own vitest config and their own jsdom in app/node_modules. From a root install that
fails with ERR_MODULE_NOT_FOUND: jsdom, so the command CONTRIBUTING documents and the
one RELEASING.md names as the pre-release gate both error out.
Move the scoping CI already applies into package.json: test runs tests/ minus the
parallelism-sensitive cache-refresh-lock suites, test:locks runs those three serially,
test:watch keeps watch mode at the same scope. The first two are byte-identical to the
invocations .github/workflows/tests.yml spells out, so the workflow can be pointed at
the scripts to stop the two drifting apart again; that edit is left out of this PR so it
needs no workflow permissions. test plus test:locks together still cover all 192 files
under tests/.
Scoping the script changes what a trailing path argument means: vitest ORs positional
filters, so 'npm test -- tests/providers/hermes.test.ts' would no longer narrow to that
file, it would run the whole suite. Rewrite those to 'npx vitest run <path>' everywhere
they appear - four provider guides and the MCP design plan, thirteen lines in all.
Also refresh the stale test docs: 42 files/568 tests (now 192 under tests/), the
per-directory counts, the line claiming vitest does not run in CI which stopped being
true when tests.yml landed, and the provider test-gap list, which still named
antigravity and gemini after both gained test files.
Record the cache-refresh-lock naming convention in CONTRIBUTING, since the split makes
it load-bearing: a lock test that misses the prefix runs under the full worker pool and
flakes, and one that matches it but is absent from test:locks never runs at all.
OpenClaude is a Claude Code fork routing to any LLM; transcripts are
Claude-Code-schema JSONL under ~/.openclaude/projects/<slug>/<uuid>.jsonl
with replay.json siblings skipped. Only usage-bearing assistant lines
become calls; sidechain lines are counted as real spend; costs are always
computed (the transcript reports none) through the shared tables.
Real local testing: sessions generated with the actual CLI against
DeepSeek (deepseek-chat), parsed end to end.
.nvmrc matches the engines floor and the appx pin (22.13.0), taming the
package-lock churn from contributors on drifting node/npm versions. The
checklist distills the house rules new-provider PRs keep relearning:
product split, cache-key coupling, reported-cost presence semantics,
defensive parsing, probeRoots for doctor, and the real-local-testing bar.
The Cline CLI (npm `cline`, 3.x) stores sessions as
<sessions>/<id>/<id>.json + <id>.messages.json. The existing `cline`
provider only discovers tasks/<id>/ui_messages.json, so every CLI session
was silently reported as $0.00 — no warning, not even under --verbose.
Adds `cline-cli` as its own provider rather than a third root on `cline`,
leaving the shared Cline-family parser (Roo Code, KiloCode, IBM Bob)
untouched. It mirrors the CLI's own root resolution
(CLINE_SESSION_DATA_DIR -> CLINE_DATA_DIR -> CLINE_DIR -> ~/.cline),
implements probeRoots() so `doctor` can tell "not installed" from "wrong
override", emits one call per assistant message's `metrics` block, and
falls back to the session rollup when a session carries none. The
fallback reads `usage`, not `aggregateUsage`, which folds in spawned
subagents that are themselves separate session directories.
Two supporting changes, both required for CLI costs to report correctly:
- parser.ts re-priced cline-cli calls from tokens because the provider
was not on the reported-cost allowlist, inflating a real 12-session
local sample from $1.11 to $3.92.
- session-cache.ts gains the matching PROVIDER_ENV_VARS entry (so a
changed override invalidates) and a `reported-cost-v1` parse version
(so sessions cached before the allowlist fix re-parse once instead of
being re-priced forever).
Cost is treated as metered only when actually present and non-negative,
so a metered $0 stays reported while a missing or negative cost falls
back to token pricing — applied identically on the per-message and
rollup paths. Timestamps promote a seconds-resolution value rather than
silently landing in 1970, matching the guard kiro.ts uses.
CLINE_DIR / CLINE_DATA_DIR / CLINE_SESSION_DATA_DIR are added to the test
env-isolation list so a developer's real sessions cannot bleed into
fixtures.
The VS Code variant discovery bug reported alongside this in #874 is
deliberately NOT fixed here — it shipped in #882.
Verified against 18 real local sessions: 142 calls, 4,934,762 input /
224,561 output tokens, and a cost matching the CLI's own metered total to
the cent. `codeburn doctor` reports "Cline CLI OK".
Refs: #874
Codex session discovery required `payload.originator` to start with
"codex" (case-insensitive). `originator` is a free-form client identity
string, not a format marker: any tool driving `codex app-server` writes
structurally identical rollouts under ~/.codex/sessions with its own
value ("t3code_desktop", "JetBrains.IntelliJ IDEA", ...). Those sessions
were silently dropped from every report, and each past fix only admitted
one more spelling.
Gate on structure instead: a first line that parses as JSON, has
type === "session_meta", and carries a plain-object payload. Foreign and
malformed files are still rejected. Directory ownership decides the
provider — codex.ts is the only provider that reads ~/.codex, and the
walk only visits rollout-*.jsonl under the strict YYYY/MM/DD path or
archived_sessions/ — so no double counting is possible. `originator` is
still parsed onto the meta entry; nothing downstream reads it.
Bump the daily cache to v16. Historical days are served from that cache
(usage-aggregator only recomputes today) and retention is ten years, so
without a bump an upgrading user with a warm cache keeps the pre-fix
rollups forever: discovery reruns, so the session COUNT moves, while
cost and calls stay frozen — a self-contradicting report that reads as
"fixed". Measured on a fixture with two same-day rollouts, one
codex-cli and one t3code_desktop:
pristine main, fresh cache cost 4.55 calls 1 sessions 1
this branch, main's warm cache cost 4.55 calls 1 sessions 2 (was)
this branch, main's warm cache cost 18.2 calls 2 sessions 2 (now)
this branch, fresh cache cost 18.2 calls 2 sessions 2 (truth)
CODEX_CACHE_VERSION and PROVIDER_PARSE_VERSIONS.codex deliberately stay
put: both caches are keyed per file path and are written only after a
successful parse, so a file rejected at discovery has no entry to
invalidate. Verified on the fixture above — main's codex-results.json
and session-cache.v7.json hold only the first-party rollout, and reusing
them unchanged still yields the correct total.
Harden `payload.cwd` while admitting unverified clients. It is declared
`string` but comes straight off JSON.parse, and a number/object/array
threw "cwd.replace is not a function" out of sanitizeProject; the throw
escaped discoverSessions into safeDiscoverSessions, which returns [] for
the WHOLE provider, so one malformed file made every Codex report read
zero. Guarded in discovery (falls back to the `unknown` project) and on
the parse side, where a non-string cwd would otherwise ride into
projectPath/workingDirectory and reach the parser's path helpers.
Closes#873, closes#626.
Resham Joshi