Adds an opt-in `amazon` source backed by the Bright Data CLI: product
discovery with live ratings and prices, plus a capped sample of recent
reviews woven in as buyer voice.
The signal it exists for is drift -- an all-time rating set against the
average of only the reviews inside the last 30 days. The emoji footer
names products and the direction they moved rather than counting
inventory, following the Polymarket shape:
📦 Amazon: 12 products │ Spirit E-325 Gas 4.4★→4.8★, Genesis E-325 Gas 4.4★ quiet
Off by default and dual-gated: the CLI must be on PATH and logged in, and
the run must explicitly request the source. It never auto-fires from
inferred intent. A drift arrow requires n>=5 dated in-window reviews,
since a live date census showed Amazon's top-reviews ordering is only
weakly recency-biased.
Includes doctor registration, honest setup reporting, SKILL.md trigger
guidance with worked examples, and CONFIGURATION.md coverage.
Reviewed by three independent agents plus Greptile (5/5). Fixes landed
for: the API key being visible in argv, --amazon-query leaking into
competitor sub-runs, a lane deadline that did not bound wall clock, brand
inference that silently disabled the competitor guard, and a saved-report
source list that dropped whole sources from the per-source dump.
Verified with three live end-to-end runs against a funded account.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Passing --trustpilot-domain (or a plan-level trustpilot_domain) was accepted
and then silently ignored unless INCLUDE_SOURCES/--search already opted in.
Treat an explicit domain pin as intent and activate the source for the run.
Fixes#873
SKILL.md and CONFIGURATION claimed rate-limit/unavailable escalation and SC
comment enrichment on the free path; document the real empty-only search
backup, thinness/primary knobs, and shreddit comments. Fixes#867.
* fix(discover): fold same-story duplicates, velocity-sorted ranks, loud fallback note
Survivors sharing evidence (identical top_comment or >=2 shared evidence
URLs) fold to the higher-velocity twin; display ranks and angle topic_ids
are assigned from the velocity-sorted order so rank order matches the
displayed velocity_score; non-mock runs with no reasoning provider emit
one loud stderr note. Mock fixture URLs/comments are namespaced per topic
slug so the fold cannot collapse distinct mock topics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): handoff contracts module - nominations bundle, judgments, angles
New lib/discovery_handoff.py owns the three-leg protocol file contracts:
versioned bundle writer/reader (full seed items as engine resume state,
bundle id, TTL, momentum window, leg-1 invocation context), strict-top/
lenient-row judgments reader with ported name sanitation and collision
disambiguation over host names, angles reader with the ported 200-char
cap, host-facing digest builder, and typed HandoffContractError for
exit-2 mapping that names both searched locations. schema.py gains the
discovery-nominations kind and duck-typed nomination serializers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): three-leg protocol CLI - nominate-only, judgments, finalize, angles flags
New flags with orphan/mutual-exclusion rejection (exit 2, message names
the combination), mock protocol legs require --save-dir to stay
side-effect-free, one handoff-state resolver (save-dir else config dir),
and a dispatch wrapper that maps HandoffContractError to stderr + exit 2.
Leg bodies are distinct NotImplementedError stubs replaced by U3-U5.
Bare --discover and --discover-shallow dispatch regression-pinned
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): nominate leg - sweep to bundle, digest, nothing-solid short-circuit
run_discover_nominate shares the exact sweep/cluster/nominate code with
the one-shot path (factored _discovery_sweep, nominate_topic_pool,
shared source-boundary helpers), cuts at the full judge pool instead of
the enrichment limit, never resolves providers, and writes the versioned
bundle with heuristic fallback names, momentum window, and leg-1
invocation context. Zero nominations renders the nothing-solid brief in
leg 1 with no bundle. Digest names the bundle path and mandates reading
its evidence before judging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): resume leg - host judgments, deep enrichment, pending report
run_discover_resume applies host judgments (per-row heuristic fallback,
collision resolution over the whole pool), excludes host-junk from
enrichment slots outright, skips sub-corroborated heuristic junk
pre-enrichment, blend-cuts to the slot limit, and runs tier-parameterized
enrichment (deep: default depth, 4 workers, LAST30DAYS_ENRICH_BUDGET_SECONDS
default 450 via the config allowlist; shallow and one-shot keep quick/240/3,
pinned both ways). Velocity scores against the bundle window. Floor, fold,
and velocity ranks share the one-shot code via extracted helpers. One
main-thread pending-report write (fresh TTL, run_ref, angle inputs keyed by
surviving nomination ids); stdout emits angle inputs plus instructions.
No queue writes on this leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): finalize leg - host angles, render, idempotent queue write
--discover --finalize loads the pending report (typed errors name both
searched locations and the resume-leg remedy; TTL from the leg-2 write),
applies host angles by nomination id, renders through the one-shot's
emit flow with O_EXCL artifact saving, and records the topic queue once,
guarded. record_discovery_surfacing gains a per-run idempotency guard:
a matched row already stamped with this run_ref neither increments nor
re-annotates, so finalize retries render byte-identical output. Mock
finalize stays queue-free. The full mock three-leg sequence is pinned
end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover)!: delete the engine-side LLM judge - the host is the judge
Removes lib/discovery_judge.py and every provider touch in the discovery
path: no resolve_runtime in run_discover, no stage-1 verdict blending,
no stage-2 angle pass. One-shot discovery always names via topic_shape
heuristics, ranks velocity-only, renders no angles, and emits one loud
note pointing at the SKILL.md host-judged protocol (no provider-key
advice). Mock guard tightened to no-provider-client-constructed across
all legs; grep-level pins keep the judge from coming back. providers.py
untouched for the normal pipeline. Coverage 87.53% vs the 84 floor.
Keyed one-shot cron users lose LLM naming/angles by design; the
protocol replaces them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(skill): LAW 11 - YOU ARE THE JUDGE; three-command discovery protocol in SKILL.md
New LAW 11 with the LAW-7-style anticipated-misread note (the one-shot
heuristics stderr line is a skipped-protocol signal, never a capability
constraint) and a pre-Bash self-check. Step 1 DISCOVERY branch rewritten
as the three-leg protocol: nominate-only (180s), judge via mktemp
heredoc judgments file, resume (600s), host angles file, finalize (60s)
relay-verbatim; identical --save-dir threaded through all legs;
fail-twice degradation to bare one-shot; shallow tier for sub-8-minute
shell caps. CONFIGURATION.md documents the four new flags; CONCEPTS.md
Discovery/Nomination name the host as judge; README trending mentions
updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* refactor(discover): consolidate handoff validation, fix leg-3 binding error text
Simplify pass on the branch: shared _parse_handoff_envelope for the two
engine-written files, one _search_paths helper, shared _known_rows gate
for host-file rows, schema._source_status_from_dict for both report
deserializers, schema._utc_now reuse, _discover_domain helper. Real fix:
_require_bundle_binding now names the pending report and the resume-leg
remedy when the finalize leg's angles file mismatches - it previously
pointed the host's retry at the nominations bundle; regression-pinned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* fix(review): handoff contract hardening - save-dir isolation, junk type-gate, fail-closed bundle, fenced digest
Explicit save-dir is now the single handoff store (no config-dir
fallback; matches scoped-db semantics and SKILL.md's own contract);
judgments junk accepts only real booleans (null/"false" fall back
per-row); non-list or zero-valid-row nominations fail closed instead of
rendering nothing-solid; bundle writes raise the protocol's exit-2 error
on OSError instead of a traceback; bundle_id mismatch remedy now says
fix-the-id-and-retry-this-leg; digest evidence rides inside the
untrusted-content fence the deleted judge used; SKILL.md documents the
leg-2 budget knob and data-not-instructions rule. Lenient-row paths
regression-pinned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* fix(review): protocol leg semantics - degraded-state plumbing, retry history, guard hoist, mock parity
Bundle and pending report now carry the leg-1 sweep's source_status and
a mock flag; the resume report restores degraded-source warnings and one
shared strict-exit helper gates every leg terminal (nothing-solid paths
included) exactly like the one-shot. Finalize retries reconstruct
pre-run queue history instead of dropping Pipeline lines; --as-of and
html-emit guards hoisted to all discover invocations; malformed pending
bodies and unwritable pending writes become clean exit-2 contract
errors; fresh rounds invalidate stale pending files; the same-story fold
re-scans to a fixpoint so three-way chains collapse; mock/real handoff
state cannot cross legs. Duplicated render/save and queue-warning blocks
extracted into shared helpers. Coverage 87.68%.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* docs(changelog): unreleased entries for host-judged discovery protocol
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* docs(changelog): fill PR number for host-judged discovery entries
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SKILL.md doctor section + the source_status note, CONFIGURATION.md health
section (new commands, corrected network-note, LAST30DAYS_DOCTOR_PROBE_TIMEOUT),
and a CHANGELOG Unreleased entry for the audit redesign, post-mortem,
live probe, CLI health, new sources, sub-lanes, and the Threads fix.
* feat(discover): U1 nominate stage with keyword-gate toggle for global vs domain
Extract the discovery fetch+normalize sweep into a reusable nominate_candidates()
and add a keyword_gate param to _fetch_discovery_source. Domain discovery keeps the
gate on; global trending (no domain) turns it off so the river feeds' own hot
ranking is the signal. run_discover now delegates its fetch loop to
nominate_candidates(keyword_gate=True), preserving existing behavior.
* feat(discover): U2 nominate_topics - named, seed-ranked candidate topics
Extract the cluster/rank/name block from run_discover into nominate_topics(),
returning Nomination objects (name, seed velocity score, cluster items, leader
summary). This is the contract between the nominate stage and the upcoming
enrichment fan-out: names deduped casefold, ranked by cheap seed velocity,
never padded past the evidence. run_discover consumes nominations with
identical output to before.
* feat(discover): U3 enrichment fan-out - full pipeline pass per nomination
enrich_nominations() runs the real research pipeline (run() with
internal_subrun=True, the comparison-mode lane) on each nominated topic in a
bounded ThreadPoolExecutor against a wall-clock batch budget. A topic whose
sub-run raises is kept as nomination-only with the error recorded; topics
unfinished at budget expiry are likewise dropped to nomination-only. The batch
never raises and preserves nomination order. This is what buys discovery the
whole multi-source corpus (Reddit+comments, X, YouTube, Techmeme, arXiv, HN,
Polymarket, web) with zero bespoke fetch code.
* feat(discover): U4 confidence floor - 'nothing solid' beats ranked noise
THE bug fix for the 2026-07-12 junk-trend regression (five 1-like tweets
ranked as a 'sports' trend list). passes_discovery_floor() gates every topic:
an absolute engagement floor kills junk outright, then a topic clears via
independent cross-source confirmation OR a genuinely strong single-source
spike (a 1,600-point HN thread is a story; a 30-upvote meme is not).
run_discover gains enrich= (full pipeline pass per nomination via U3) and
judges each topic on its enriched corpus when available, seed evidence
otherwise. Zero floor-clearing topics -> DiscoveryReport.outcome
'nothing-solid' with the strongest sub-floor name in weak_signal, exported
through to_discovery_export. Regression test: the junk corpus now returns an
honest empty result.
* feat(discover): U5 global trending mode - bare --discover with no domain
--discover DOMAIN is now nargs-optional: bare --discover sweeps every river
feed's own hot list (r/all, HN front page, Digg) with the keyword gate off -
the feeds' hot ranking is the signal and the confidence floor culls junk. X
sits out of the global nominate stage (its search lane needs a keyword) and
joins per-topic at enrichment. Dispatch keys on 'flag present' so the empty
domain is never mistaken for 'flag absent'.
Discovery now enriches by default (full research pass per nomination);
--discover-shallow is the fast escape hatch that ranks on listing evidence
only, still floored. Global saves slug as 'trending'.
* feat(discover): U6 rich trend cards + honest nothing-solid rendering
DiscoveryTopic gains top_comment (strongest verbatim community comment from
the enriched corpus, per-platform vote-normalized, with attribution) and
corroboration_count; both exported in to_discovery_export. Renderer upgrades:
'# Trending now' header for global runs, a 'confirmed across N sources' badge
on the momentum line, a '**Community voice:**' quote line on enriched cards,
and an explicit nothing-solid empty state naming the closest weak signal
instead of the old generic 'no clusters survived' line.
* feat(discover): U7 Techmeme + arXiv reach discovery through enrichment
Enrichment sub-runs get requested_sources=None by default, so every available
source - Techmeme, arXiv, YouTube, Polymarket, community comments - joins each
nominated topic's research pass despite having no river feed of its own. An
explicit user --search boundary now holds through enrichment too (the CLI
threads the original pre-narrowed list as enrich_requested_sources). The
keyword-source nominate gap (X/Techmeme/arXiv have no front-page lane) is
documented at the planner.
* docs(discover): U8 route global vs domain trending; document two-stage flow
SKILL.md discovery branch now routes bare trending asks (/last30days trending,
'what's hot') to bare --discover and domain asks to --discover DOMAIN,
documents the nominate-enrich timing (10-minute Bash timeout), the
--discover-shallow fast path, and pins 'Nothing solid this window' as a valid
verbatim pass-through outcome (never retry or fabricate around it).
CONFIGURATION.md and the JSON export reference document the new flag surface
and contract fields (outcome, weak_signal, top_comment, corroboration_count).
* fix(discover): strip quote chars from community-voice bodies to avoid doubled quotes
Live global-trending verification surfaced comments whose bodies start with a
quote character rendering as doubled quotes inside the card's wrapping quotes.
* fix(discover): address PR #816 review - real budget enforcement + shallow-flag guard
P1: enrichment workers now run as daemon threads behind a semaphore instead of
a ThreadPoolExecutor. Executor threads are non-daemon and joined at interpreter
shutdown, so one hung sub-run could keep the process alive long after its topic
was downgraded to nomination-only. Daemon workers make the wall-clock budget
real; abandonment is safe because internal_subrun passes write nothing to disk
and every fetch layer carries its own timeout.
P2: --discover-shallow without --discover now errors (exit 2) instead of
silently no-opping into a full research run.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(sources): honor INCLUDE_SOURCES for xiaohongshu and dripstack
Both sources were per-run-request-only, so a persisted
INCLUDE_SOURCES=xiaohongshu or dripstack in .env was silently ignored -
the exact silent-ignore class documented from #707/#708 - and the only
persistent route (LAST30DAYS_DEFAULT_SEARCH) replaces the whole source
set instead of adding to it. Both gates now follow the established
LinkedIn/Perplexity/Trustpilot opt-in pattern: active when the source
is in INCLUDE_SOURCES or explicitly requested, otherwise off, with the
xiaohongshu availability probe still firing only after an opt-in.
Default behavior is unchanged; tests lock the checkbox on, the default
off, and the probe never firing without opt-in.
* fix(sources): strip whitespace in INCLUDE_SOURCES tokens
INCLUDE_SOURCES=linkedin, dripstack (space after the comma) produced a
' dripstack' token that failed the exact membership checks, silently
dropping the persisted opt-in - a pre-existing wart for every
INCLUDE_SOURCES consumer that this PR's hoisted computation now fixes
for all of them.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* Add DripStack to related skills
DripStack is a complementary AI agent skill for premium financial
newsletter research via micropayments. Same audience, different data
layer — last30days covers public/social signal, DripStack covers
paywalled financial signal.
* Add DripStack as a source: premium financial newsletter search
DripStack indexes paid Substack newsletters, analyst writeups, and
financial podcasts. The search endpoint is free and public (no API key
required) — returns article metadata with relevance-scored snippets.
Integration:
- New source module: scripts/lib/dripstack.py
- Normalizer registered in normalize.py
- Wired into pipeline.py (always available, like GitHub/HN)
- Planner assigns it to opinion/prediction intents alongside StockTwits
- Mock data for tests
- SKILL.md display name mapping
- CONFIGURATION.md source table entry
Signal: complementary to StockTwits (retail sentiment) and Polymarket
(prediction odds) — DripStack covers what professional analysts and paid
newsletter authors are writing about. Publication attribution (e.g.
'SemiAnalysis', 'Bloomberg') is high-credibility signal for synthesis.
* fix(dripstack): requested-only gating, shared http path, window filtering
Maintainer hardening on #791 before merge (owner decision: every new
source ships default-off with an explicit toggle):
- DripStack activates only when explicitly requested (--search dripstack
or LAST30DAYS_DEFAULT_SEARCH), mirroring the Xiaohongshu guard; default
runs send zero traffic to dripstack.xyz and their available_sources set
is unchanged.
- All requests route through the shared lib/http.py choke point so
capture/replay, fixtures, and the failure taxonomy apply.
- Results honor the 30-day window at fetch (dated items outside the
window are dropped; undated items are kept, not guessed).
- Docs describe the opt-in; the README Related Skills block recommending
a separate skill install is removed (out of scope for a source PR).
- tests/test_dripstack.py covers gating, choke point, windowing, failure,
and normalization.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(sources): local corpus source - your own files as a ranked signal
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: defang corpus sentinels, matching-window snippets, exclusion-aware hosted gate, traversal notes, bounded discovery
* fix: keep absolute local paths out of corpus notes and coverage diagnostics
* fix(corpus): keep raw exception text out of coverage notes
OS and subprocess errors embed the failing absolute path in str(exc),
and scan/cache notes flow into source_status detail rendered in
coverage diagnostics outside the private corpus block. Notes now carry
the error's strerror (or class name) instead, so a permission failure
or file race can no longer leak a local path from a private run.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(grounding): per-claim freshness verdicts (--verify-freshness)
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: prefer active markets in mixed-event verification, plan-only cookies for cached verification
* fix: verify event identity on slug-fallback refetch before assigning verdicts
* test: use a real outcome datum key in slug-identity test
* fix(freshness): carry values in stale detail and announce zero-claim passes
A stale verdict's machine-readable detail said only that the value
moved; the rendered table already showed was/now, so agent-JSON
consumers saw less than readers. Compose the detail as
"moved: <original> -> <current>", formatting each value the way the
claim text does (percent for Polymarket/StockTwits, thousands
separators for stars).
A verification pass that extracts zero claims used to complete
silently, which misled a live post-hoc run. _verify_report_set now
prints one stderr note when no verified report produced any verdict,
aggregated across main and entity reports so multi-report passes note
it once; the drill path routes through the same helper.
* fix(freshness): verify star facts attached by candidate enrichment
A GitHub-flavored run produced zero verdicts live: star enrichment
attaches metadata["github_stars"] to candidates after reranking, but
claim extraction read only item-level engagement, and those candidates'
primary items are typically non-GitHub sources.
Extraction now emits one repo-keyed star claim per enriched repo
(skipping repos already claimed at item level). These claims cannot
ride the item-source dispatch - they would land unsupported as
"reddit"/"x" claims - so verify_report routes the repo-slug shape
straight to the GitHub refetcher, bypassing the grounding-item lookup
and the per-source outcome gate (the datum came from enrichment, not
the github search source). refetch_datum accepts an owner/repo slug as
the datum key, and the snapshot cache keys these claims by repo so
multiple candidates citing one repo share one request.
* fix(freshness): review fixes - honest evidence, joinable export, rendered detail
Four review findings on the hardening pass:
- Unsupported verdicts no longer fabricate evidence: evidence_url and
evidence_timestamp stay empty when no fresh evidence was obtained;
provenance remains on source_url/source_timestamp.
- Agent export results now carry candidate_id so consumers can join
freshness_verdicts to the result they annotate (schema_version 1.2,
golden snapshot and docs updated).
- The rendered verdict table shows the verifier's detail - formatted
movement on stale rows, the un-checkable reason on unsupported rows -
instead of reconstructing raw values.
- Item-level star dedup is scoped per candidate: another candidate's
item-level claim no longer suppresses an enriched candidate's own
verdict and inline flag; both share one repo snapshot.
* fix(freshness): require event-id identity on Polymarket slug fallback
The slug-fallback refetch verified the response slug but accepted a
slug-matched event whose id differed from the cached item's event id,
so a re-created event reusing a slug could produce current/stale
verdicts from another market's prices. When the cached item carries a
real numeric event id, the fallback now requires the response id to
match it (synthetic PM<N> parse ids carry no identity and skip the
check); mismatches raise and degrade to an unsupported verdict.
* fix(freshness): fail closed when the slug fallback has no event identity
A cached item with neither metadata.event_id nor a real numeric item id
(the synthetic PM<N> parse fallback) let the slug fallback accept
whatever event currently owns the slug, so a recreated or loosely
matched event could still drive a current/stale verdict. With no
identity to verify against, the refetch now raises before any request
and the claim degrades to unsupported.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(library): FTS search over saved briefs with self-citing library context
* fix: address self-review findings
* fix: safety note on search output, reject --output, repopulate FTS after table loss
* fix: per-library search index and scoped briefing archive for --save-dir libraries
* fix(library): keep scoped runs out of the shared store and index
Three leaks between --save-dir scopes and the shared research store:
- library search with --save-dir merged shared-store sightings into a
scoped search, so one client's completed-run snippets could surface
in another client's results. Scoped searches now read only a store
inside the save dir (usually absent).
- The passive "From your library" context had the same fallback to
the shared store on scoped runs; it now scopes identically.
- The markdown save hook synced the shared library.db from a scoped
scan, and sync's stale-row prune could delete other scopes' (and the
default library's) rows. Scoped saves now sync a per-directory index
with the same paths scoped search uses.
* fix(store): write scoped-run findings to the save-dir store
Scoped reads (library search, library context) were fixed to stay
inside the save dir, but --store on a --save-dir run still wrote
findings through the shared research.db - so a scoped client's own
sightings never appeared in its scoped searches while remaining
visible to unscoped ones. persist_report now routes store access
through a scoped_db context to <save-dir>/research.db, the same path
the scoped read side uses; unscoped runs keep the shared store.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(library): publish saved research as index and Atom feed
Scan saved reports and briefing archives into deterministic, topic-grouped library pages with stable Atom entries. Add explicit opt-in multi-document publishing, publish-safe social IDs, slash-command guidance, configuration docs, and regression coverage.
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: keep per-suffix reports distinct in library identity
* fix: scope briefing archive to the selected library, back up hand-written index/feed files
* fix(library): prune only marker-bearing briefs and never clobber a backup
A refresh deleted any orphaned briefs/ page whose filename looked
generated, so a hand-written page with a generated-looking name could
be removed. Rendered briefs now embed an ownership comment marker and
the prune requires it before unlink; name shape alone is no longer
grounds for deletion.
The hand-written index/feed backup used replace(), so a second refresh
overwrote the first .bak. Backups now pick the first free .bak/.bakN
name instead.
* fix(library): preserve hand-edited brief pages on refresh
The orphan prune checks the generator marker, but the brief write path
replaced briefs/<name>.html for current entries without any ownership
check, silently destroying a hand-edited or hand-written page whose
name matches a current report. Every library-feed write (briefs, index,
feed) now goes through one preservation helper: a page without the
generator marker is moved to the first free .bak/.bakN name before the
generated page is written.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(pipeline): add --discover mode for topic-less trending discovery
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: boundary-aware category match, additive-only discovery totals, preserve discovery intent through onboarding
* fix: visible default-search fallback, non-plural domain anchors, clean X backend fallback
* fix(discover): apply domain filter to Digg listing clusters
The Digg source is an AI-focused leaderboard feed, so a discovery sweep
on a non-AI domain surfaced off-domain stories (observed live: a crypto
sweep returning model-release coverage). Filter parsed Digg clusters
with the same title-level _matches_discovery_domain check the adjacent
Hacker News branch already applies; an all-filtered result stays a
clean no-results outcome.
* fix(discover): surface the producing X backend's own error as partial
Failing over past a dead backend is a clean outcome, but when the
backend that actually produced items also returned an error, the sweep
is degraded; returning a clean state hid that from source_status and
strict-exit runs. Keep the producing backend's error; earlier fallback
errors remain stderr observability only.
* fix(discover): hold the configured source boundary instead of widening
When the configured source filter (LAST30DAYS_DEFAULT_SEARCH or
--search) contains no discovery-capable source, the sweep silently
widened to all discovery feeds with only a warning, querying sources
the user had filtered out. Discovery now exits 2 with the unsupported
sources and the discovery-capable set named, for both the config and
explicit-flag paths.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(schema): versioned agent JSON export profile with --json-profile raw fallback
* fix: address self-review findings
* fix: exclude author-reach fields from headline engagement (stocktwits followers)
* fix: fail loudly when the evaluator receives the agent profile instead of raw Report
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(pipeline): typed per-run source outcomes in the report
* fix: address self-review findings
* fix: propagate failure capture to nested pools, surface authed GitHub failures, classify wrapped timeouts, opt-in strict exit
* fix: pick the most specific captured failure, not the last-appended one
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Patch release carrying the Hermes-scan fix (#768): the install-time scanner
verdict drops from DANGEROUS to CAUTION (0 CRITICAL). Cutting this release
rebuilds the last30days.skill asset with the fix, which is what Hermes installs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QgVqyQ8nwZL6opLtnNEMAm
* fix(hermes-scan): eliminate CRITICAL findings so install verdict is caution
The Hermes install-time scanner (skills_guard.py) returned DANGEROUS and hard-
blocked `hermes skills install mvanhorn/last30days-skill` (community + dangerous;
--force powerless), per issue #513. The 14 CRITICAL findings were all false
positives on legitimate code:
- 7 python_environ_get_secret: os.environ.get("...API_KEY") credential reads
-> routed through env.read_secret_env(name) so no secret-shaped literal sits
inside an os.environ.get(...) call. Behaviour identical.
- 3 ruby_env_secret: a Ruby ENV[] rule firing case-insensitively on Python
`env[key] = value` -> rewritten as env.update({key: value}).
- 2 env_exfil_httpx: http.get(..., headers={"X-Api-Key": token}) in xquik.py
-> headers extracted to a local var off the call line.
- 1 ruby_env_secret in vendored bird-search cookies.js -> vendored tree
excluded via .skillignore (third-party node_modules analog; still installed).
- 1 deception_hide: a SKILL.md line "do not tell the user..." -> reworded to
positive framing with identical meaning.
Verdict now caution (0 CRITICAL, verified against the real skills_guard.py);
--force installs. SAFE/no-force is not cleanly reachable because oversized_skill
(HIGH, 1.6MB > 1MB limit) would require .skillignore-ing ~500KB of runtime code.
All changes are behavior-preserving; full test suite green (2 pre-existing
network-dependent GitHub-auth failures unrelated). Baseline + plan under
tests/hermes/ and docs/plans/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QgVqyQ8nwZL6opLtnNEMAm
* test(hermes-scan): add regression guard asserting zero CRITICAL scan findings
Self-contained replica of skills_guard.py's CRITICAL-severity rules; scans the
skill subtree (honoring .skillignore) and fails if any blocking pattern
reappears, so a future edit can't silently re-block community installs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QgVqyQ8nwZL6opLtnNEMAm
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(health): dependency probe taxonomy with reinstall prescriptions
Uniform ok/missing/broken/timeout probes for yt-dlp, PP CLIs, node,
ffmpeg. broken = resolves on PATH but cannot exec (the #692 stale-shim
false-negative); prescriptions keyed by owning package manager;
off-PATH-but-on-disk reports missing with a PATH fix. (U1 of 5)
* feat(backends): chain descriptors with predicted will-use resolution
Alternative mode (X/YouTube/web) probes all candidates side-effect-free
then picks first fully-usable, so an unauthenticated preferred backend
cannot shadow a working fallback; Reddit renders conditional
public-default + ScrapeCreators-backfill wording instead of a fake
winner. Chains import env.py's own definitions; paid lanes probe key
presence only, guarded by no-network tests. (U2 of 5)
* feat(prescriptions): single fix-remediation registry shared by nudges
(source, failure) -> cause + natural-language fix + exact CLI fix +
config anchor, seeded with the known credential/tool failure inventory.
quality_nudge composes its fix text from registry entries (wording
byte-identical); backends.py setup-command prescriptions embed the same
strings, so doctor and nudges cannot drift. (U3 of 5)
* feat(doctor): unified doctor command with tier rollup and frozen aliases
/last30days doctor aggregates dependency probes, backend descriptors,
prescriptions, diagnose data, and the permission summary into one
grouped report (text + --json). Tier rollup per the plan's table;
per-source exception isolation; predicted will-use backend for chains;
no secrets, no cookie reads, no network; exit 0. --diagnose/--preflight
shapes characterization-frozen (pre-v3.9.0 baseline, re-record on
landing) with explicit MCP-passthrough and available_sources consumer
tests. (U4 of 5)
* feat(doctor): TTL cache, --cached read path, SKILL.md standing rule
Doctor persists its JSON beside last-run.json (doctor-cache.json);
--cached serves within LAST30DAYS_DOCTOR_TTL (900s default, registered
in env; 0 disables) and falls through live on stale/corrupt/absent.
Explicit doctor always runs live and refreshes. SKILL.md gains doctor
triggers plus the cached pre-research rule (2 lines); CONFIGURATION.md
documents the command, cache, TTL, and pin vars. (U5 of 5)
* refactor(doctor): simplification pass across the doctor stack
Concurrent per-source probes (order-preserving pool.map, worst case now
bounded by slowest probe); memoized xurl availability (kills a double
whoami spawn); INCLUDE_SOURCES token parsing matches the pipeline gate;
shared timestamp-freshness helper; tier/status constants unified;
public health accessors replace private cross-module reads; explicit
DependencyProbe.off_path replaces detail-string sniffing.
* fix(review): truthful probes and cache trust (wave 1)
Doctor's X probe is now genuinely local-only: xurl checked via its
token store, never whoami; the safe/diagnose path can no longer reach
the network through any of three vectors (forbid-network tests lock
it). Bird checks cookies before node so unconfigured X reports off,
not error. Cache layer: schema stamp, full shape validation with live
fall-through (fixes the reproduced KeyError crash), config fingerprint
invalidation, generated_at/from_cache staleness signals, and cache
write failures now warn on stderr.
* fix(review): single-source-of-truth pass (wave 2)
pipeline.py and providers.py read env's pin/floor constants instead of
restating the raw strings (parity tests guard it); env grows public
include_sources()/is_setup_complete() wrappers ending doctor's private
reaches; health.py and setup_wizard share one installer candidate-dir
list, restoring the Windows PrintingPress dir the mirror had dropped
(the documented Digg off-PATH failure mode), with Windows regression
tests.
* fix(review): doctor output fidelity (wave 3)
requires names the actually-failed backend; digg broken/timeout gets
reinstall-framed prescriptions; ok-tier fixes render in text (YouTube
transcription backstop now visible + covered); SKILL.md frontmatter
gains health-check discovery keywords and the standing rule is marked
mandatory, both locked by contract tests.
* fix(doctor): rebase onto v3.10.0 baseline + Greptile review fixes
Re-record the diagnose/preflight characterization snapshots against the
committed v3.10.0 baseline (new sources, x_pending_browser_auth key, PP
CLI external commands) per the planned re-record path; thread the
no-network gate through main's new x_pending precompute so the safe
path stays provably offline; tolerant alt_cli lookup in quality_nudge
(Greptile P1) with an arity pin test; autouse probe-cache reset fixture
(Greptile P2).
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Matt Van Horn