docs(solutions): compound the checkpointed host-judgment protocol learning (#858)
Validate / mcp-tests (push) Has been cancelled
Scorecard / Scorecard analysis (push) Has been cancelled
Security / Dependency audit (push) Has been cancelled
Security / Dependency review (push) Has been cancelled
Security / Secret scan (push) Has been cancelled
Security / SAST scan (push) Has been cancelled
Validate / tests (push) Has been cancelled
Validate / eval (push) Has been cancelled
GitHub Actions Security Analysis with zizmor 🌈 / zizmor (push) Has been cancelled

New architecture-patterns doc: five design conventions for the
checkpointed discovery protocol (identity/TTL binding, lossless-state vs
fenced-digest split, fail-closed empty state, provenance enforcement,
guarded writes + stale-sibling invalidation), with the meta-lesson that
an external LLM API call inside an engine whose invoker IS an LLM is an
architecture smell. 56/58 claims grounded against the tree by the
validation pass; the two exceptions were corrected in place. CONCEPTS.md
gains Handoff checkpoint and refreshes Junk shape (host verdict vs
heuristic flag) and Covered (judge is now the hosting model).


Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-07-21 14:49:15 -07:00
committed by GitHub
parent 51781fe02d
commit 01aef34ca4
2 changed files with 386 additions and 2 deletions
+6 -2
View File
@@ -50,6 +50,10 @@ The topic-less research mode: instead of researching a named topic, it finds wha
A named candidate topic produced by Discovery's listing sweep: clustered items from the river feeds, given a short searchable name plus a Junk shape flag and a content-worthiness score that blends into its seed rank. On protocol runs the hosting model judges all three via the judgments file - the engine's deterministic heuristics only fill rows the host left absent; on headless one-shot runs deterministic distillation supplies the name and junk flag and no worthiness signal exists. A Nomination is only a candidate - its blended seed rank decides which topics deserve an Enrichment pass and the display order of survivors; the Confidence floor judgment and the displayed velocity score are computed from the enriched evidence, never the seed score. The Nomination's name doubles as its Enrichment pass search query and its research handoff, so naming happens before enrichment, never at render time.
### Handoff checkpoint
The persisted state that lets Discovery's protocol pause for host judgment and resume in a later invocation: the nominations bundle (the full judge pool with its seed evidence, written by leg 1, awaiting the host's judgments) and the pending report (the enriched, floored, ranked round written by leg 2, awaiting the host's angles). A checkpoint is identity-bound - every host-written file must echo the checkpoint's bundle id, and a mismatch is a fix-the-id-and-retry error, never a redo of the expensive leg - and time-bound by its own TTL so a stale round is rejected rather than resumed. Checkpoints also carry provenance the resume legs enforce: mock and real state never cross, degraded sweep coverage survives into later legs instead of reading as clean, and an explicitly scoped store is the only place its checkpoints are looked for. Structurally empty checkpoint state is treated as corruption and fails closed - it never becomes an authoritative-looking empty result.
### Enrichment pass
A full research-pipeline run executed on one Nomination's topic name during Discovery. This is what gives a trend card the whole multi-source corpus (community comments, prediction markets, keyword-driven sources that have no hot-list of their own) instead of thin listing evidence. Enrichment passes run in parallel against a wall-clock budget; a pass that fails or outruns the budget downgrades its topic to nomination-only evidence, never fails the run.
@@ -64,7 +68,7 @@ The honest empty outcome of a Discovery run in which zero topics cleared the Con
### Junk shape
A classification applied to a Nomination whose leading item reads as a help-me question, beginner ask, or personal musing rather than a story - the post shapes that engagement alone cannot distinguish from news. Junk shape does not exclude a topic outright; it removes the Confidence floor's single-source bypass so the topic surfaces only with independent seed-source corroboration.
A classification applied to a Nomination whose leading item reads as a help-me question, beginner ask, or personal musing rather than a story - the post shapes that engagement alone cannot distinguish from news. Its force depends on who flagged it: a host junk verdict is authoritative and excludes the Nomination from Enrichment passes outright (it can still appear as the weak signal in a Nothing-solid brief), while a heuristic junk shape on a row the host never judged only removes the Confidence floor's single-source bypass, so that topic surfaces solely with independent seed-source corroboration.
### Topic queue
@@ -74,7 +78,7 @@ Identity in the queue is annotate-only: a new topic name that closely matches an
### Covered
The user-set status on a Topic queue row meaning "I already produced content for this story." Set by marking a topic covered by its exact name; surfaced is the only other status. A resurfacing never un-covers a row, and a new name that fuzzily matches a Covered row is born Covered - so the mark survives the LLM judge renaming the same story across runs instead of silently re-pitching it.
The user-set status on a Topic queue row meaning "I already produced content for this story." Set by marking a topic covered by its exact name; surfaced is the only other status. A resurfacing never un-covers a row, and a new name that fuzzily matches a Covered row is born Covered - so the mark survives the judge (now the hosting model) renaming the same story across runs instead of silently re-pitching it.
## Flagged ambiguities
@@ -0,0 +1,380 @@
---
title: "Checkpointed discovery protocol: five design conventions for host-side LLM judgment"
date: 2026-07-21
category: architecture-patterns
module: discovery-checkpoint-protocol
problem_type: architecture_pattern
component: tooling
severity: high
applies_when:
- "The product's primary consumer is a frontier reasoning model invoking the tool as an agent skill, not a traditional programmatic API client"
- "A pipeline stage needs semantic judgment (naming, classification, worthiness scoring) that only an LLM can supply"
- "Building a keyless or free-tier path where a silent heuristic fallback would degrade output quality without disclosing that an API key was assumed"
- "A CLI or script needs to persist state across multiple invocations while the host model performs judgment in between (checkpoint-and-resume design)"
- "An existing skill law or convention already establishes that host-side reasoning replaces engine-side API keys, and a new pipeline stage needs the same treatment"
symptoms:
- "v3.17.0 silently fell back to deterministic topic naming and junk heuristics for keyless users when the engine's own LLM judge was unavailable"
- "An API key was the de facto front door to real discovery judgment, contradicting the skill's keyless-path promise and its own LAW 7 host-is-the-reasoning-model precedent"
root_cause: wrong_api
resolution_type: code_fix
tags:
- "discovery-protocol"
- "host-judged-protocol"
- "checkpoint-files"
- "law-11"
- "keyless-path"
- "nominations-bundle"
- "provenance-enforcement"
- "bundle-id-ttl"
related_components:
- "skills/last30days/scripts/lib/discovery_handoff.py"
- "skills/last30days/scripts/last30days.py"
- "skills/last30days/SKILL.md"
- "tests/test_discover_handoff.py"
- "tests/test_discover_mode.py"
---
# Checkpointed discovery protocol: five design conventions for host-side LLM judgment
## Context
An Agent Skill's primary consumer is a frontier reasoning model: the engine
(`skills/last30days/scripts/last30days.py`) is invoked by Claude Code, Codex,
Gemini, or another agent runtime that read SKILL.md. v3.17.0 (PR #852) forgot
that and shipped judgment as an engine-side LLM pass: `lib/discovery_judge.py`
(since deleted by PR #856; the path is historical) resolved a reasoning
provider across Gemini/OpenAI/xAI/OpenRouter keys and,
per its own contract, never raised - "No provider, a failed call, or a
malformed payload logs a warning and returns None, and the caller falls back"
to deterministic heuristics. Every keyless user silently got the degraded
branch: heuristic topic names like "120k 1,600 ESP32s" and zero content
angles, with no signal that a better path existed.
PR #856 (v3.18.0) deleted the engine judge outright (CHANGELOG.md:18-20) and
replaced it with a three-command host-judged protocol, mandated by SKILL.md
LAW 11 "YOU ARE THE JUDGE" (skills/last30days/SKILL.md:233): the pipeline
pauses at its judgment points and persists versioned checkpoint files that
the hosting model judges between invocations. Leg 1 (`--discover
--nominate-only`) sweeps and writes the nominations bundle; the host writes a
judgments file; leg 2 (`--discover --judgments <file>`) resumes, deep-enriches,
and writes the pending report; the host writes an angles file; leg 3
(`--discover --finalize [--angles <file>]`) renders offline. The contracts
live in `skills/last30days/scripts/lib/discovery_handoff.py` (module
docstring, lines 1-18), the leg handlers in
`skills/last30days/scripts/last30days.py:1716-1986`, and the resumed pipeline
math in `skills/last30days/scripts/lib/pipeline.py:1525-1697`.
This doc records the five conventions that make a checkpoint protocol safe:
identity/TTL binding, the lossless-state-vs-capped-digest split, fail-closed
parsing of empty state, provenance enforcement across invocations, and
guarded writes plus stale-sibling invalidation.
## Guidance
### 1. Checkpoints are identity-bound and time-bound
Every host-authored file must echo the checkpoint's identity. Leg 1 mints a
random `bundle_id` (`discovery_handoff.py:270`), prints it in the digest, and
both host files (judgments, angles) must carry it back.
`_require_bundle_binding` (`discovery_handoff.py:638-673`) enforces the echo
and its error names BOTH ids and the cheap remedy:
```python
raise HandoffContractError(
f"The {label} file is bound to bundle_id {file_bundle_id!r} but the "
f"{noun} is {bundle.bundle_id!r}. {location_label}:\n"
f"{_searched_lines(searched)}\n"
f"Correct the bundle_id field in your {label} file to "
f"{bundle.bundle_id!r} and re-run this same leg."
)
```
WHY the remedy split matters: a mismatched echo means the host copied the
wrong id into an otherwise-current file, so the fix is edit-one-field and
retry THIS leg - never the expensive re-sweep (`_RESWEEP_REMEDY`,
`discovery_handoff.py:43`) or resume (`_RESUME_REMEDY`, lines 48-51)
remedies, which belong to missing/stale state. On the finalize leg the
message deliberately names the pending report, not the bundle, so the host's
retry is not misdirected (lines 653-664). `HandoffContractError` maps to
exit 2 in one place (`last30days.py:1984-1986`).
Time binding is a dedicated module constant with a deliberate non-reuse
comment (`discovery_handoff.py:32-36`):
```python
# How long a nominations bundle stays valid. Deliberately a module constant
# and NOT the LAST30DAYS_REPORT_CACHE_TTL_SECONDS env knob: a user who
# lowered the report-cache TTL for drill freshness must not shrink the
# window a host has to author judgments.
DISCOVERY_HANDOFF_TTL_SECONDS = 3600.0
```
Staleness is checked in the shared envelope validator via
`env.is_timestamp_fresh` (`discovery_handoff.py:430-436`, `env.py:121`), and
the pending report gets a FRESH TTL clock stamped at leg-2 write time
(`last30days.py:1854-1856`) because leg 2 started a new authoring window.
WHY: an unrelated cache knob silently shrinking the host's judging window is
exactly the class of cross-feature coupling a checkpoint file must not have.
### 2. One checkpoint, two audiences, hard split: lossless resume state vs capped fenced digest
The nominations bundle serves the engine and the host, and the two halves
have opposite rules.
Engine half: the FULL judge pool with complete seed items, serialized
losslessly (`schema.py:848-852` states the contract;
`schema.nomination_to_dict`, `schema.py:917-932`, round-trips every item).
Leg 2's floor/velocity/entity math must score identically to a
single-process run: `_floor_survivor_records` is "Shared verbatim by
run_discover (one-shot) and run_discover_resume (protocol leg 2) so floor
semantics can never drift between the paths" (`pipeline.py:1199-1215`), and
velocity scores against the bundle's momentum window, never the resume-time
clock (`pipeline.py:1558-1561`). A capped bundle would silently starve
downgraded-topic scoring: host-junk rows, and heuristic-junk fallback rows
below the seed-source floor, never
get an enrichment pass, so their weak-signal velocity and the
seed-source-corroboration floor count are computed purely from bundle seed
items (`pipeline.py:1584-1593`, `rerank.py:110-136`) - truncate the items and
those rows under-count sources and engagement with no error anywhere. Parity
is test-pinned: `tests/test_discover_handoff.py:232`
(`test_parity_floor_and_velocity_inputs_survive_round_trip`) asserts
velocity, engagement totals, source sets, and entity-disambiguation inputs
(title + snippet) recompute identically after the round trip.
Host half: `build_host_digest` (`discovery_handoff.py:915-976`) is capped
(`_DIGEST_TITLE_MAX_CHARS`/`_DIGEST_SNIPPET_MAX_CHARS`/`_DIGEST_COMMENT_MAX_CHARS`,
lines 66-68) and its evidence lines ride inside the untrusted-content fence:
```python
if evidence_lines:
lines.append("")
lines.append(rerank._fenced_untrusted_content("\n".join(evidence_lines)))
```
That is the exact fence the rerank judge uses (`rerank.py:305-312`), and the
deleted engine judge fenced the same evidence the same way (v3.17.0's
`discovery_judge.py` imported `_fenced_untrusted_content` for both of its
prompts). Dropping the fencing during the rewrite was a caught regression;
the fence is now pinned by
`tests/test_discover_handoff.py:755`
(`test_digest_fences_untrusted_evidence_like_the_engine_judge`): scraped
titles/snippets/comments inside the fence, structural lines (ids, sources,
signal, bundle path) outside it. Host-supplied text going the other
direction is capped too - names at 96 chars, angles at 200
(`discovery_handoff.py:53-62`), "ported from the retired engine-judge pass"
because names become search queries and angles render verbatim on cards.
### 3. Engine-written checkpoints parse strict-at-top, lenient-per-row, but FAIL CLOSED on structurally empty state
The readers are strict at the top level (readable, JSON object, right kind,
right schema version, bundle_id present, within TTL - all in
`_parse_handoff_envelope`, `discovery_handoff.py:381-437`) and lenient per
row: one corrupt nomination row is warned and skipped, never fatal
(`discovery_handoff.py:469-487`). But leniency has a floor. A non-list
`nominations` value raises (`discovery_handoff.py:460-466`), and zero valid
parsed rows raises too (`discovery_handoff.py:506-514`):
```python
if not nominations:
# Leg 1 never writes an empty bundle (a zero-nomination sweep
# short-circuits with no bundle file), so an empty or all-invalid
# nominations array is corrupt state: fail closed, never hand the
# resume leg a silently empty pool.
raise HandoffContractError(...)
```
WHY: without this, a corrupt bundle flows into leg 2 as an empty pool, which
floors to zero survivors and renders an authoritative-looking "Nothing solid
this window" brief - a green result manufactured from broken state. PR
#856's review caught this
empty-pool-renders-authoritative-empty-result failure; it is now pinned by
`tests/test_discover_handoff.py:439` (all rows malformed) and `:457` (empty
list). The invariant that makes fail-closed valid: leg 1 short-circuits a
zero-nomination sweep to the nothing-solid brief and writes NO bundle
(`last30days.py:1748-1750`), so an empty pool on disk is always corruption,
never a legitimate outcome.
### 4. Cross-invocation state carries provenance, and the resume legs enforce it
Three kinds of provenance ride the checkpoint files:
Mock parity. Both checkpoints stamp `mock` at write time
(`discovery_handoff.py:307`, `last30days.py:1861`), and every resume leg
runs `_require_discover_mock_parity` (`last30days.py:1506-1532`): mock-born
state is rejected by a real run and real state by a `--mock` run, in both
cases exit 2 with a fix-the-flag or fresh-sweep remedy. WHY: "mock-born
state finalized by a real run would fake a real brief from fixture data, and
real state finalized by --mock would silently drop the round's queue write."
Sweep coverage. Leg 1 serializes the sweep's per-source outcome map into the
bundle (`discovery_handoff.py:308-311`), leg 2 restores it into its report
(`pipeline.py:1681-1692`), and leg 3 inherits it through the pending report,
so degraded coverage "survives the protocol instead of silently reading as
clean" (`discovery_handoff.py:115-118`). Every leg terminal - one-shot and
all three legs - exits through the ONE shared strict-exit helper,
`_discovery_strict_exit_code` (`last30days.py:1481-1503`, called at
1703, 1750, 1797, 1839, 1895, 1967), which turns `LAST30DAYS_STRICT_EXIT`
plus any non-clean source outcome into exit 3. The PR #856 review validated
this as a P1: before the fix, protocol legs silently exited 0 on degraded
sweeps because the status map was dropped between legs.
Store scoping. An explicit `--save-dir` is the SOLE handoff store.
`_search_paths` (`discovery_handoff.py:211-227`) returns "ONLY the save dir
when one was supplied, else the config dir", mirroring `_scoped_store_db`
(`last30days.py:432-437`): "a handoff file in the config dir must never
silently satisfy a save-dir run." The second validated P1 of the review:
with a fallback chain, a missing pending file in the save dir would let a
bare `--finalize` quietly consume the config-dir store's pending report and
finalize another store's run.
### 5. Guard the write after the expensive work, and invalidate stale siblings on fresh rounds
Both engine checkpoint writes happen after minutes of paid-for work (a sweep;
a deep enrichment pass), so an OSError there is converted to the typed
contract error, never a traceback (`discovery_handoff.py:329-334` for the
bundle; `last30days.py:1869-1877` for the pending report):
```python
except OSError as exc:
# A locked/read-only/full disk is the protocol's clean exit-2 path,
# never a traceback.
raise HandoffContractError(
f"Could not write nominations bundle {path}: {exc}"
) from exc
```
This is the repo's guarded-write convention (same shape as the discovery
queue's guarded end-of-run write) applied to checkpoints, and it is pinned by
`tests/test_discover_handoff.py:472`.
Fresh rounds invalidate stale siblings. A new leg-1 bundle starts a NEW
protocol round, so any pending report left by a prior round is deleted
alongside it (`last30days.py:1783-1788`); a leg 2 that ends nothing-solid
wrote no pending file this round, so it also unlinks any stale one
(`last30days.py:1830-1837`). WHY: without the unlinks, an unbound bare
`--finalize` inside the TTL could re-serve the PREVIOUS round's report as if
it belonged to the current sweep. The deliberate exception proves the rule:
a SUCCESSFUL finalize leaves the pending file in place
(`last30days.py:1905-1911`) so a retry with a corrected angles file keeps
working, and idempotency comes from replaying the leg-2 `run_ref` into the
queue (`last30days.py:1958-1964`) rather than from deleting state.
## Why This Matters
The architecture smell this pattern removes: an external LLM API call inside
an engine whose invoker IS an LLM. That shape fails three ways at once. It
adds cost (a second metered model where a capable one is already in the
loop). It forks quality silently between keyed and keyless users - v3.17.0's
judge never raised on a missing provider, so keyless users got heuristic
names and no angles with zero indication anything was degraded, on the
skill's PRIMARY invocation path. And it produces a strictly worse judge: the
budget-priced engine-side model (flash-lite class, batched, no session
context) judged evidence the frontier host model could have judged directly.
The checkpoint protocol is the general remedy shape: pause the pipeline at
each judgment point, persist versioned, identity-bound, TTL-bound state, let
the host judge between invocations, and validate every resume so stale or
mismatched state becomes a clean exit 2 with a named remedy instead of
silent wrong output. LAW 11's framing (SKILL.md:233) is the contract in one
line: "You do not need an API key ... you ARE the reasoning model." The
one-shot path prints a loud note pointing at the protocol
(`pipeline.py:1421-1433`) precisely so a reasoning-model host can never
mistake heuristic output for a capability ceiling.
The five conventions are what make the pause safe. Splitting one pipeline
into three processes creates every classic distributed-state hazard in
miniature - stale state, cross-round state, cross-store state, fixture/real
crosses, silently-empty state, lost coverage warnings - and each convention
above closes one of them.
## When to Apply
Apply this pattern when:
- A CLI or engine embedded in an Agent Skill needs semantic judgment
(naming, junk filtering, scoring, prose authoring) in the middle of an
otherwise deterministic pipeline - the host model is the judge; checkpoint
around the judgment points.
- You are about to add an LLM provider key, client, or "reasoning provider"
resolution to an engine whose invoker is already a reasoning model - that
is the smell; reach for the protocol instead.
- An existing engine-side LLM pass has a "silent heuristic fallback" - the
keyless majority is getting invisible degraded output today.
Do NOT apply it when:
- No reasoning model is in the loop. The one-shot cron/scripted path keeps
the single-process pipeline deliberately (`run_discover`,
`pipeline.py:1384`, and the degradation rule in SKILL.md:398): a
checkpoint pause with nobody to judge is just a hang.
- The judgment is expressible as a deterministic rule - the junk-shape
heuristics and the confidence floor stayed engine-side because they need
no model at all.
## Examples
The three-command sequence as SKILL.md ships it (skills/last30days/SKILL.md:318-399),
one identical `--save-dir` threaded through all three legs:
```bash
# Leg 1 - sweep and nominate (global trending; domain runs pass the domain
# phrase as the --discover argument on this leg only):
python3 scripts/last30days.py --discover --nominate-only \
--save-dir="$HOME/Documents/Last30Days"
# stdout: judging digest + bundle path + bundle_id. Host reads the bundle
# file, then writes judgments.json:
# {"bundle_id": "<echoed>", "judgments": [
# {"id": "n1", "name": "Gemma 4 chat templates", "junk": false, "worthiness": 85},
# {"id": "n2", "name": "Beginner asks how to deploy", "junk": true, "worthiness": 10}]}
# Leg 2 - resume with judgments; deep per-topic research (several minutes):
python3 scripts/last30days.py --discover --judgments judgments.json \
--save-dir="$HOME/Documents/Last30Days"
# stdout ends with angle inputs keyed by surviving id. Host writes
# angles.json: {"bundle_id": "<same>", "angles": [
# {"id": "n1", "podcast": "<hook>", "x_article": "<hook>"}]}
# Leg 3 - finalize offline: apply angles, render, record the topic queue.
python3 scripts/last30days.py --discover --finalize --angles angles.json \
--emit=compact --save-dir="$HOME/Documents/Last30Days"
```
Failure-mode walkthrough (mismatched then stale checkpoint):
1. The host echoes a bundle_id from an earlier round into judgments.json and
runs leg 2. `_require_bundle_binding` raises; the CLI prints
`[last30days] The judgments file is bound to bundle_id 'aaaa...' but the
current nominations bundle is 'bbbb...'` plus the searched location, and
exits 2. Remedy as printed: correct the `bundle_id` field and re-run leg 2.
The expensive sweep is NOT redone - the bundle on disk is still current.
2. The host instead waits 90 minutes before judging. The envelope check
(`discovery_handoff.py:430-436`) finds `generated_at` outside the 3600s
TTL and exits 2: the bundle "is stale ... the momentum window it captured
has moved on. Run a fresh `--discover --nominate-only` re-sweep." Here the
expensive leg IS the remedy, because the state itself expired - the
protocol never asks for the expensive path when a cheap edit fixes the
problem, and never accepts cheap edits when the data has aged out.
The deterministic end-to-end twin of the whole sequence is pinned in CI:
`tests/test_discover_mode.py:2439`
(`test_discovery_cli_full_mock_protocol_three_legs_end_to_end`).
## Related
- `docs/solutions/architecture-patterns/discovery-topic-queue-design-conventions.md` -
same feature family, the queue side: the persistent topic queue leg 3
writes into (idempotently, under the leg-2 `run_ref`), including the
guarded-write convention this protocol reuses.
- `docs/solutions/design-patterns/ranked-output-confidence-floor-honest-empty-state.md` -
the confidence-floor semantics the protocol preserves verbatim across the
process split (`_floor_survivor_records` shared by both paths), including
seed-source corroboration for junk shapes.
- `docs/solutions/logic-errors/non-daemon-executor-threads-defeat-wall-clock-budget.md` -
the enrichment wall-clock budget pattern leg 2's deep tier extends
(`RESUME_DEEP_ENRICH_BUDGET_SECONDS` 450s via
`LAST30DAYS_ENRICH_BUDGET_SECONDS`, `pipeline.py:1492-1508`; workers stay
daemon threads and never touch disk - the pending report is ONE post-loop
write from the main thread, `last30days.py:1867-1871`).
- PR #856 (protocol, engine-judge removal), PR #852 (the v3.17.0 engine
judge this replaced), CHANGELOG.md v3.18.0 / v3.17.0 entries.
- SKILL.md LAW 11 and the Step 1 DISCOVERY branch (skills/last30days/SKILL.md:233, 314-399).