Files
Matthew Kelch a5a0ddaeac feat(copilot): read per-request input/cache from session-store.db
The Copilot CLI and the GitHub Copilot desktop app both write
~/.copilot/session-store.db unconditionally; its assistant_usage_events
table holds one row per API request. Until now input/cache tokens for
these surfaces came only from the session.shutdown rollups in
events.jsonl, which are written only on clean shutdown (a crash loses
the whole leg's input/cache accounting) and lump each session leg into
one per-model total. The rollup also RESETS its counters at in-session
compaction (traced on a clean single-process 107-request session whose
sole rollup covered exactly its five post-compaction requests), so even
cleanly-closed long sessions were truncated; on a long-history machine
the store recovered ~35% of real Copilot spend lost to crashes and
compaction resets. The DB rows are per-request, crash-proof, and carry
real timestamps.

The store's input_tokens is cache-INCLUSIVE (input + cache_read +
cache_write), the same convention as the shutdown rollups — verified
against each row's token_details_json and by reconciling per-session
sums against the CLI's own footers and rollups across two machines
(1,380+ rows, 8 models, CLI 1.0.70–1.0.79, schema_version 6): every
divergence was a rollup gap. Emitted calls mirror the shutdown-call
contract: input/cache/reasoning only, output 0 — per-turn output stays
owned by the events.jsonl assistant.message calls.

Rollup-vs-store precedence is RECONCILED at serve time, per
(session, model), and only there. Both representations always parse and
cache; parseProviderSources aggregates the cached calls and, wherever
store rows exist for a (session, model), drops the rollup calls and
serves the rows plus per-leg RESIDUAL calls: each rollup leg subtracts
only the rows in its own interval — rows commit strictly before their
leg's shutdown line, so a leg at time T covers exactly the rows in
(previous leg's T, T] — and any remainder (per token component, floored
at zero) serves once at that leg's own timestamp. A store missing
requests a leg covered — adopted mid-session, rows pruned before ever
being read — therefore still serves that tail exactly once ON THAT
LEG'S DAY, a crash-tail row the rollup never saw can never cancel it,
and a complete store serves pure per-request granularity with every
residual retired to zero. The decision reads only cached contents, never discovery:
deleting or resetting the store changes nothing served, so finalized
daily history can never flip on an absence epoch; cached rows of a
deleted store remain the record until the 90-day orphan age-out (which
exempts still-discovered paths). The serve set is the one coherent
snapshot — nothing a writer does between discovery and a parse can
change what one pass sees — and read-time precedence heals persisted
duplication (stale epochs, runtimes without node:sqlite, restored
files) instead of preserving it, following the buildDurablePeriod
pattern.

Store rows and rollups carry supplementary accounting weight. A rollup
(or its residual) is aggregate accounting, never a request: zero
api-call/model-call/turn weight, tokens and cost fully retained. A
store row is one real request, but when it pairs with a served per-turn
call it is supplementary too; rows pair with same-model per-turn calls
by timestamp adjacency (monotone matching, tight 2-minute window — the
two are written at the same completion moment, and a wide window would
let a crash-only row pair against a neighbor whose own row is missing),
computed once over the FULL serve set so a date-range boundary that
separates a row from its call cannot double the request across adjacent
day queries. Only the unpaired rows — store-only requests, exactly
where crash-lost requests sit — count. Supplementary-only turns fold
into the nearest behavioral turn within 30 minutes; with no behavioral
turn to fold into they stay separate weightless turns, each on its own
day, with apiCalls 0 — and the session emission gate admits
usage-bearing zero-call sessions. The weight
propagates into the daily cache: aggregateProjectsIntoDays applies the
same rule to every calls counter and category-turn count it seals, so
v19 history and live summaries can never disagree about what was a
request.

A changed source whose read defers on the busy shape (locked, EACCES,
corrupt mid-replace — discovery still emits the source; only true
absence or a schema mismatch reads as absent) now marks session
hydration incomplete, so the daily backfill holds its watermark instead
of finalizing a day the deferred rows never reached; an unchanged
unreadable store defers nothing. The verdict travels with its result —
the 180s memo and the serve burst-reuse restore the hydration verdict
their cached data was parsed under, so a memoized partial parse cannot
inherit a later parse's complete — and a discovered source whose
FINGERPRINT cannot be read (EACCES on a present file) defers instead of
silently skipping, while a genuinely deleted file stays a silent skip. Copilot reasoning tokens are no longer
double-billed at the report layer: they are a subset of the output the
per-turn calls already price, and copilot joins claude in the
reasoning-inside-output case of the query-time cost recompute.

Store dedup keys are content-discriminated —
copilot-store:<sid>:<rowId>:<fnv1a64(created_at|tokens|model)> —
because AUTOINCREMENT prevents id reuse only within one database
lifetime: a same-path DB reset reusing row ids now mints new keys
instead of the durable union swallowing the new usage, while a
byte-identical re-insert still collapses (64-bit: 32-bit FNV
collisions between plausible token tuples are constructible). Every
call of a session serves under one project label resolved at serve
time — the session-state-derived label when the serve set knows it,
else the store rows' own — so neither rows cached before events.jsonl
existed nor an events.jsonl orphaned by a session-state prune can
split the session across two grouping keys.

CODEBURN_COPILOT_SESSION_STORE_DB is read but deliberately NOT
fingerprinted, per the #927 ruling (any copilot fingerprint change
drops cached entries whose path still exists, destroying pruned history
only the cache holds); the read is allowlisted in the #927 guard, and
serve-time reconciliation makes repointing safe without a fingerprint —
the new store's rows parse on sight and the old path's entries persist
as durable orphans. The copilot parse version appends session-store-v2
and the daily cache bumps v17 → v19: per-day attribution, call counts
and costs all change against pre-store builds. 19, not 18: an earlier
pushed head of this PR already claimed v18 under different accounting,
and the carry-forward would adopt those days as finalized without
re-deriving them.

Verified by A/B on snapshots of two real stores, a live SIGKILL crash
test (row present, no rollup, tokens recovered exactly), live resumes
whose warm-cache deltas matched new rows to the token, upgrade-healing
at 4,800-session scale, and serve-level regressions pinning every
maintainer finding from six review rounds: the rows-then-shutdown race,
stale-cache healing, age-out exemption, absence-epoch identity,
progressive row landing with residual retirement, behavioral weight
across all four pinned scenarios, the hydration fence, project
unification in both directions, the same-path reset, mixed
coverage (crash tail vs covered-leg gap), multi-leg residual day
attribution, range-invariant pairing, memo-scoped hydration verdicts,
and the fingerprint-failure fence.
2026-08-17 17:52:01 -04:00

363 lines
13 KiB
TypeScript

import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { mkdtemp, readFile, readdir, rm } from 'fs/promises'
import { join } from 'path'
import { tmpdir } from 'os'
import { exportCsv, exportJson, type PeriodExport } from '../src/export.js'
import type { ProjectSummary } from '../src/types.js'
let tmpDir: string
beforeEach(async () => {
tmpDir = await mkdtemp(join(tmpdir(), 'export-test-'))
})
afterEach(async () => {
await rm(tmpDir, { recursive: true, force: true })
})
function makeProject(projectPath: string, agentType?: string): ProjectSummary {
return {
project: projectPath,
projectPath,
sessions: [
{
sessionId: 'sess-001',
project: projectPath,
agentType,
firstTimestamp: '2026-04-14T10:00:00Z',
lastTimestamp: '2026-04-14T10:01:00Z',
totalCostUSD: 1.23,
totalInputTokens: 100,
totalOutputTokens: 50,
totalCacheReadTokens: 0,
totalCacheWriteTokens: 0,
apiCalls: 1,
turns: [
{
userMessage: '=SUM(1,2)',
timestamp: '2026-04-14T10:00:00Z',
sessionId: 'sess-001',
category: 'coding',
retries: 0,
hasEdits: true,
assistantCalls: [
{
provider: 'claude',
model: '+danger-model',
usage: {
inputTokens: 100,
outputTokens: 50,
cacheCreationInputTokens: 0,
cacheReadInputTokens: 0,
cachedInputTokens: 0,
reasoningTokens: 0,
webSearchRequests: 0,
},
costUSD: 1.23,
tools: ['Read'],
mcpTools: [],
skills: [],
subagentTypes: [],
hasAgentSpawn: false,
hasPlanMode: false,
speed: 'standard',
timestamp: '2026-04-14T10:00:00Z',
bashCommands: ['@malicious'],
deduplicationKey: 'dedup-1',
},
],
},
],
modelBreakdown: {
'+danger-model': {
calls: 1,
costUSD: 1.23,
tokens: {
inputTokens: 100,
outputTokens: 50,
cacheCreationInputTokens: 0,
cacheReadInputTokens: 0,
cachedInputTokens: 0,
reasoningTokens: 0,
webSearchRequests: 0,
},
},
},
toolBreakdown: {
Read: { calls: 1 },
},
mcpBreakdown: {},
bashBreakdown: {
'@malicious': { calls: 1 },
},
categoryBreakdown: {
coding: { turns: 1, costUSD: 1.23, retries: 0, editTurns: 1, oneShotTurns: 1 },
debugging: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
feature: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
refactoring: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
testing: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
exploration: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
planning: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
delegation: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
git: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
'build/deploy': { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
conversation: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
brainstorming: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
general: { turns: 0, costUSD: 0, retries: 0, editTurns: 0, oneShotTurns: 0 },
},
skillBreakdown: {},
},
],
totalCostUSD: 1.23,
totalApiCalls: 1,
}
}
describe('exportCsv', () => {
it('prefixes formula-like cells to prevent CSV injection', async () => {
const periods: PeriodExport[] = [
{
label: '30 Days',
projects: [makeProject('=cmd,calc')],
},
]
const outputPath = join(tmpDir, 'report.csv')
const folder = await exportCsv(periods, outputPath)
// exportCsv now writes a folder of clean one-table-per-file CSVs, so the formula-prefix
// guard is scattered across files. Concatenate them for the assertion surface.
const [projects, models, shell] = await Promise.all([
readFile(join(folder, 'projects.csv'), 'utf-8'),
readFile(join(folder, 'models.csv'), 'utf-8'),
readFile(join(folder, 'shell-commands.csv'), 'utf-8'),
])
const content = projects + models + shell
expect(content).toContain("\"'=cmd,calc\"")
expect(content).toContain("'+danger-model")
expect(content).toContain("'@malicious")
})
it('escapes tab and carriage-return prefixes in CSV cells', async () => {
const periods: PeriodExport[] = [
{
label: '30 Days',
projects: [makeProject('\tcmd'), makeProject('\rcmd')],
},
]
const outputPath = join(tmpDir, 'tab-cr.csv')
const folder = await exportCsv(periods, outputPath)
const projects = await readFile(join(folder, 'projects.csv'), 'utf-8')
expect(projects).toContain("'\tcmd")
expect(projects).toContain("'\rcmd")
})
it('includes per-model efficiency metrics', async () => {
const periods: PeriodExport[] = [
{
label: '30 Days',
projects: [makeProject('app')],
},
]
const outputPath = join(tmpDir, 'models.csv')
const folder = await exportCsv(periods, outputPath)
const models = await readFile(join(folder, 'models.csv'), 'utf-8')
expect(models).toContain('Edit Turns')
expect(models).toContain('One-shot Rate (%)')
expect(models).toContain('Retries/Edit')
expect(models).toContain('Cost/Edit')
expect(models).toContain(',1,100,0,')
})
it('does not crash when periods array is empty', async () => {
const outputPath = join(tmpDir, 'empty.csv')
const folder = await exportCsv([], outputPath)
const entries = await readdir(folder)
expect(entries.length).toBeGreaterThanOrEqual(0)
})
it('describes detail files without hardcoding a 30-day window', async () => {
const periods: PeriodExport[] = [
{
label: '2026-04-07 to 2026-04-10',
projects: [makeProject('app')],
},
]
const outputPath = join(tmpDir, 'custom.csv')
const folder = await exportCsv(periods, outputPath)
const readme = await readFile(join(folder, 'README.txt'), 'utf-8')
expect(readme).toContain('selected detail period')
expect(readme).not.toContain('30-day window')
})
it('writes MCP server usage to mcp.csv', async () => {
const project = makeProject('app')
project.sessions[0]!.mcpBreakdown = { node_repl: { calls: 5 } }
const periods: PeriodExport[] = [{ label: '30 Days', projects: [project] }]
const folder = await exportCsv(periods, join(tmpDir, 'mcp.csv'))
const mcp = await readFile(join(folder, 'mcp.csv'), 'utf-8')
expect(mcp).toContain('Server,Calls,Share (%)')
expect(mcp).toContain('node_repl,5,100')
})
it('writes optional subagentType and model fields to per-call records.csv', async () => {
const periods: PeriodExport[] = [{ label: '30 Days', projects: [makeProject('app', 'planner')] }]
const folder = await exportCsv(periods, join(tmpDir, 'records.csv'))
const records = await readFile(join(folder, 'records.csv'), 'utf-8')
const lines = records.trimEnd().split('\n')
expect(lines[0]).toContain('subagentType,model')
expect(lines[1]).toContain('planner')
expect(lines[1]).toContain("'+danger-model")
})
it('keeps supplementary accounting rows in records.csv and marks them', async () => {
const project = makeProject('app')
const turn = project.sessions[0]!.turns[0]!
turn.assistantCalls.push({
...turn.assistantCalls[0]!,
supplementaryAccounting: true,
costUSD: 0.5,
deduplicationKey: 'dedup-supp',
})
const periods: PeriodExport[] = [{ label: '30 Days', projects: [project] }]
const folder = await exportCsv(periods, join(tmpDir, 'records.csv'))
const lines = (await readFile(join(folder, 'records.csv'), 'utf-8')).trimEnd().split('\n')
// The column exists on every row (undefined on normal ones), so rowsToCsv —
// which reads headers off the first row — always emits it.
expect(lines[0]!.endsWith(',supplementary')).toBe(true)
expect(lines[1]!.endsWith(',1.23,0,')).toBe(true)
expect(lines[2]!.endsWith(',0.5,0,true')).toBe(true)
expect(lines).toHaveLength(3)
})
it('counts only behavioral turns in the sessions.csv Turns column', async () => {
const project = makeProject('app')
const session = project.sessions[0]!
session.turns.push({
...session.turns[0]!,
assistantCalls: [{
...session.turns[0]!.assistantCalls[0]!,
supplementaryAccounting: true,
deduplicationKey: 'dedup-supp',
}],
})
const periods: PeriodExport[] = [{ label: '30 Days', projects: [project] }]
const folder = await exportCsv(periods, join(tmpDir, 'sessions.csv'))
const [header, row] = (await readFile(join(folder, 'sessions.csv'), 'utf-8')).split('\n')
const turns = row!.split(',')[header!.split(',').indexOf('Turns')]
// Two raw turns, one of them accounting-only.
expect(turns).toBe('1')
})
it('adds optional subagentType and unambiguous model fields to sessions.csv', async () => {
const periods: PeriodExport[] = [{ label: '30 Days', projects: [makeProject('app', 'planner')] }]
const folder = await exportCsv(periods, join(tmpDir, 'sessions.csv'))
const sessions = await readFile(join(folder, 'sessions.csv'), 'utf-8')
expect(sessions.split('\n')[0]).toBe('Project,Session ID,Started At,Cost (USD),Saved (USD),API Calls,Turns,subagentType,model')
expect(sessions.split('\n')[1]).toContain('planner')
expect(sessions.split('\n')[1]).toContain("'+danger-model")
})
})
describe('exportJson', () => {
it('adds per-call records with optional subagentType and model fields', async () => {
const periods: PeriodExport[] = [{
label: '30 Days',
projects: [makeProject('agent-project', 'planner'), makeProject('main-project')],
}]
const outputPath = join(tmpDir, 'records.json')
const saved = await exportJson(periods, outputPath)
const data = JSON.parse(await readFile(saved, 'utf-8'))
expect(data.records[0]).toMatchObject({
project: 'agent-project',
subagentType: 'planner',
model: '+danger-model',
inputTokens: 100,
outputTokens: 50,
cost: 1.23,
})
expect(data.records[1]).toMatchObject({ project: 'main-project', model: '+danger-model' })
expect(data.records[1]).not.toHaveProperty('subagentType')
expect(data.sessions[0]).toMatchObject({ subagentType: 'planner', model: '+danger-model' })
expect(data.sessions[1]).toMatchObject({ model: '+danger-model' })
expect(data.sessions[1]).not.toHaveProperty('subagentType')
})
it('keeps supplementary-accounting tokens/cost in daily rows without counting them as calls', async () => {
const project = makeProject('app')
const turn = project.sessions[0]!.turns[0]!
turn.assistantCalls.push({
...turn.assistantCalls[0]!,
supplementaryAccounting: true,
usage: { ...turn.assistantCalls[0]!.usage, inputTokens: 40, outputTokens: 0, cacheReadInputTokens: 900 },
costUSD: 0.5,
deduplicationKey: 'dedup-supp',
})
const periods: PeriodExport[] = [{ label: '30 Days', projects: [project] }]
const saved = await exportJson(periods, join(tmpDir, 'supp.json'))
const data = JSON.parse(await readFile(saved, 'utf-8'))
expect(data.periods[0].daily).toHaveLength(1)
expect(data.periods[0].daily[0]).toMatchObject({
'API Calls': 1,
'Input Tokens': 140,
'Cache Read Tokens': 900,
'Cost (USD)': 1.73,
})
})
it('marks supplementary records and omits the key on normal ones', async () => {
const project = makeProject('app')
const turn = project.sessions[0]!.turns[0]!
turn.assistantCalls.push({
...turn.assistantCalls[0]!,
supplementaryAccounting: true,
costUSD: 0.5,
deduplicationKey: 'dedup-supp',
})
const periods: PeriodExport[] = [{ label: '30 Days', projects: [project] }]
const saved = await exportJson(periods, join(tmpDir, 'supp-records.json'))
const data = JSON.parse(await readFile(saved, 'utf-8'))
expect(data.records).toHaveLength(2)
expect(data.records[0]).not.toHaveProperty('supplementary')
expect(data.records[1]).toMatchObject({ supplementary: true, cost: 0.5 })
})
it('includes an mcp section with per-server usage', async () => {
const project = makeProject('app')
project.sessions[0]!.mcpBreakdown = { node_repl: { calls: 3 }, github: { calls: 1 } }
const periods: PeriodExport[] = [{ label: '30 Days', projects: [project] }]
const outputPath = join(tmpDir, 'export.json')
const saved = await exportJson(periods, outputPath)
const data = JSON.parse(await readFile(saved, 'utf-8'))
expect(Array.isArray(data.mcp)).toBe(true)
expect(data.mcp).toEqual([
{ Server: 'node_repl', Calls: 3, 'Share (%)': 75 },
{ Server: 'github', Calls: 1, 'Share (%)': 25 },
])
})
})