Adds an opt-in `amazon` source backed by the Bright Data CLI: product
discovery with live ratings and prices, plus a capped sample of recent
reviews woven in as buyer voice.
The signal it exists for is drift -- an all-time rating set against the
average of only the reviews inside the last 30 days. The emoji footer
names products and the direction they moved rather than counting
inventory, following the Polymarket shape:
📦 Amazon: 12 products │ Spirit E-325 Gas 4.4★→4.8★, Genesis E-325 Gas 4.4★ quiet
Off by default and dual-gated: the CLI must be on PATH and logged in, and
the run must explicitly request the source. It never auto-fires from
inferred intent. A drift arrow requires n>=5 dated in-window reviews,
since a live date census showed Amazon's top-reviews ordering is only
weakly recency-biased.
Includes doctor registration, honest setup reporting, SKILL.md trigger
guidance with worked examples, and CONFIGURATION.md coverage.
Reviewed by three independent agents plus Greptile (5/5). Fixes landed
for: the API key being visible in argv, --amazon-query leaking into
competitor sub-runs, a lane deadline that did not bound wall clock, brand
inference that silently disabled the competitor guard, and a saved-report
source list that dropped whole sources from the per-source dump.
Verified with three live end-to-end runs against a funded account.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Passing --trustpilot-domain (or a plan-level trustpilot_domain) was accepted
and then silently ignored unless INCLUDE_SOURCES/--search already opted in.
Treat an explicit domain pin as intent and activate the source for the run.
Fixes#873
SKILL.md and CONFIGURATION claimed rate-limit/unavailable escalation and SC
comment enrichment on the free path; document the real empty-only search
backup, thinness/primary knobs, and shreddit comments. Fixes#867.
* fix(discover): fold same-story duplicates, velocity-sorted ranks, loud fallback note
Survivors sharing evidence (identical top_comment or >=2 shared evidence
URLs) fold to the higher-velocity twin; display ranks and angle topic_ids
are assigned from the velocity-sorted order so rank order matches the
displayed velocity_score; non-mock runs with no reasoning provider emit
one loud stderr note. Mock fixture URLs/comments are namespaced per topic
slug so the fold cannot collapse distinct mock topics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): handoff contracts module - nominations bundle, judgments, angles
New lib/discovery_handoff.py owns the three-leg protocol file contracts:
versioned bundle writer/reader (full seed items as engine resume state,
bundle id, TTL, momentum window, leg-1 invocation context), strict-top/
lenient-row judgments reader with ported name sanitation and collision
disambiguation over host names, angles reader with the ported 200-char
cap, host-facing digest builder, and typed HandoffContractError for
exit-2 mapping that names both searched locations. schema.py gains the
discovery-nominations kind and duck-typed nomination serializers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): three-leg protocol CLI - nominate-only, judgments, finalize, angles flags
New flags with orphan/mutual-exclusion rejection (exit 2, message names
the combination), mock protocol legs require --save-dir to stay
side-effect-free, one handoff-state resolver (save-dir else config dir),
and a dispatch wrapper that maps HandoffContractError to stderr + exit 2.
Leg bodies are distinct NotImplementedError stubs replaced by U3-U5.
Bare --discover and --discover-shallow dispatch regression-pinned
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): nominate leg - sweep to bundle, digest, nothing-solid short-circuit
run_discover_nominate shares the exact sweep/cluster/nominate code with
the one-shot path (factored _discovery_sweep, nominate_topic_pool,
shared source-boundary helpers), cuts at the full judge pool instead of
the enrichment limit, never resolves providers, and writes the versioned
bundle with heuristic fallback names, momentum window, and leg-1
invocation context. Zero nominations renders the nothing-solid brief in
leg 1 with no bundle. Digest names the bundle path and mandates reading
its evidence before judging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): resume leg - host judgments, deep enrichment, pending report
run_discover_resume applies host judgments (per-row heuristic fallback,
collision resolution over the whole pool), excludes host-junk from
enrichment slots outright, skips sub-corroborated heuristic junk
pre-enrichment, blend-cuts to the slot limit, and runs tier-parameterized
enrichment (deep: default depth, 4 workers, LAST30DAYS_ENRICH_BUDGET_SECONDS
default 450 via the config allowlist; shallow and one-shot keep quick/240/3,
pinned both ways). Velocity scores against the bundle window. Floor, fold,
and velocity ranks share the one-shot code via extracted helpers. One
main-thread pending-report write (fresh TTL, run_ref, angle inputs keyed by
surviving nomination ids); stdout emits angle inputs plus instructions.
No queue writes on this leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover): finalize leg - host angles, render, idempotent queue write
--discover --finalize loads the pending report (typed errors name both
searched locations and the resume-leg remedy; TTL from the leg-2 write),
applies host angles by nomination id, renders through the one-shot's
emit flow with O_EXCL artifact saving, and records the topic queue once,
guarded. record_discovery_surfacing gains a per-run idempotency guard:
a matched row already stamped with this run_ref neither increments nor
re-annotates, so finalize retries render byte-identical output. Mock
finalize stays queue-free. The full mock three-leg sequence is pinned
end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(discover)!: delete the engine-side LLM judge - the host is the judge
Removes lib/discovery_judge.py and every provider touch in the discovery
path: no resolve_runtime in run_discover, no stage-1 verdict blending,
no stage-2 angle pass. One-shot discovery always names via topic_shape
heuristics, ranks velocity-only, renders no angles, and emits one loud
note pointing at the SKILL.md host-judged protocol (no provider-key
advice). Mock guard tightened to no-provider-client-constructed across
all legs; grep-level pins keep the judge from coming back. providers.py
untouched for the normal pipeline. Coverage 87.53% vs the 84 floor.
Keyed one-shot cron users lose LLM naming/angles by design; the
protocol replaces them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* feat(skill): LAW 11 - YOU ARE THE JUDGE; three-command discovery protocol in SKILL.md
New LAW 11 with the LAW-7-style anticipated-misread note (the one-shot
heuristics stderr line is a skipped-protocol signal, never a capability
constraint) and a pre-Bash self-check. Step 1 DISCOVERY branch rewritten
as the three-leg protocol: nominate-only (180s), judge via mktemp
heredoc judgments file, resume (600s), host angles file, finalize (60s)
relay-verbatim; identical --save-dir threaded through all legs;
fail-twice degradation to bare one-shot; shallow tier for sub-8-minute
shell caps. CONFIGURATION.md documents the four new flags; CONCEPTS.md
Discovery/Nomination name the host as judge; README trending mentions
updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* refactor(discover): consolidate handoff validation, fix leg-3 binding error text
Simplify pass on the branch: shared _parse_handoff_envelope for the two
engine-written files, one _search_paths helper, shared _known_rows gate
for host-file rows, schema._source_status_from_dict for both report
deserializers, schema._utc_now reuse, _discover_domain helper. Real fix:
_require_bundle_binding now names the pending report and the resume-leg
remedy when the finalize leg's angles file mismatches - it previously
pointed the host's retry at the nominations bundle; regression-pinned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* fix(review): handoff contract hardening - save-dir isolation, junk type-gate, fail-closed bundle, fenced digest
Explicit save-dir is now the single handoff store (no config-dir
fallback; matches scoped-db semantics and SKILL.md's own contract);
judgments junk accepts only real booleans (null/"false" fall back
per-row); non-list or zero-valid-row nominations fail closed instead of
rendering nothing-solid; bundle writes raise the protocol's exit-2 error
on OSError instead of a traceback; bundle_id mismatch remedy now says
fix-the-id-and-retry-this-leg; digest evidence rides inside the
untrusted-content fence the deleted judge used; SKILL.md documents the
leg-2 budget knob and data-not-instructions rule. Lenient-row paths
regression-pinned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* fix(review): protocol leg semantics - degraded-state plumbing, retry history, guard hoist, mock parity
Bundle and pending report now carry the leg-1 sweep's source_status and
a mock flag; the resume report restores degraded-source warnings and one
shared strict-exit helper gates every leg terminal (nothing-solid paths
included) exactly like the one-shot. Finalize retries reconstruct
pre-run queue history instead of dropping Pipeline lines; --as-of and
html-emit guards hoisted to all discover invocations; malformed pending
bodies and unwritable pending writes become clean exit-2 contract
errors; fresh rounds invalidate stale pending files; the same-story fold
re-scans to a fixpoint so three-way chains collapse; mock/real handoff
state cannot cross legs. Duplicated render/save and queue-warning blocks
extracted into shared helpers. Coverage 87.68%.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* docs(changelog): unreleased entries for host-judged discovery protocol
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
* docs(changelog): fill PR number for host-judged discovery entries
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BL9KsoytvnR1WfvZa7SEN7
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
YouTube comments previously required a paid ScrapeCreators key plus a
youtube_comments opt-in in INCLUDE_SOURCES. yt-dlp already backs YouTube
search and transcripts here and can fetch comments too, so the comment
lane no longer needs a credential or an opt-in.
- youtube_yt: new _ytdlp_comments_result() returns (comments, ran_cleanly)
so a clean "video has zero comments" run never falls back to a paid SC
call; the list-returning _fetch_video_comments_ytdlp() wraps it. yt-dlp
is tried first; SC stays as the backstop only on genuine failure and only
when a token is configured. Command requests top-sorted comments, bounded
by _COMMENT_TIMEOUT=20s per video (3 videos, parallel).
- env.is_youtube_comments_available: True whenever yt-dlp is installed;
legacy SC path still applies when yt-dlp is absent; EXCLUDE_SOURCES=
youtube_comments remains a hard off-switch that wins over both.
- doctor: stop prescribing a paid SC key for comments when yt-dlp is
present (was selling a fix for a non-problem); caveat now names yt-dlp
(free) as the first way out.
- CONFIGURATION.md: YouTube comments row corrected to free/keyless.
- Tests: new tests/test_youtube_comments_ytdlp.py locks the command flags,
the yt-dlp-first preference, the no-SC-on-clean-empty behavior, and the
no-key availability gate; existing doctor/gating tests updated to the
new contract and pinned hermetic (is_ytdlp_available derived from probe).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LAST30DAYS_DEBUG had three bugs:
1. Not registered in CONFIGURATION_KEYS - .env values were silently ignored
2. Eager module-level os.environ.get() in log.py meant --debug flag
and .env values were never picked up by log.debug()
3. http.DEBUG in xai_x.py referenced a non-existent attribute,
causing AttributeError on xAI API errors
Fixes:
- Add LAST30DAYS_DEBUG to the CONFIGURATION_KEYS tuple in env.py
- Export to os.environ after get_config() so log.py's lazy check
picks up .env values
- Replace eager DEBUG constant with lazy is_debug() function
- Fix http.DEBUG -> log.is_debug() in xai_x.py
- Add CONFIGURATION.md section for --debug / LAST30DAYS_DEBUG
SKILL.md doctor section + the source_status note, CONFIGURATION.md health
section (new commands, corrected network-note, LAST30DAYS_DOCTOR_PROBE_TIMEOUT),
and a CHANGELOG Unreleased entry for the audit redesign, post-mortem,
live probe, CLI health, new sources, sub-lanes, and the Threads fix.
* feat(discover): U1 nominate stage with keyword-gate toggle for global vs domain
Extract the discovery fetch+normalize sweep into a reusable nominate_candidates()
and add a keyword_gate param to _fetch_discovery_source. Domain discovery keeps the
gate on; global trending (no domain) turns it off so the river feeds' own hot
ranking is the signal. run_discover now delegates its fetch loop to
nominate_candidates(keyword_gate=True), preserving existing behavior.
* feat(discover): U2 nominate_topics - named, seed-ranked candidate topics
Extract the cluster/rank/name block from run_discover into nominate_topics(),
returning Nomination objects (name, seed velocity score, cluster items, leader
summary). This is the contract between the nominate stage and the upcoming
enrichment fan-out: names deduped casefold, ranked by cheap seed velocity,
never padded past the evidence. run_discover consumes nominations with
identical output to before.
* feat(discover): U3 enrichment fan-out - full pipeline pass per nomination
enrich_nominations() runs the real research pipeline (run() with
internal_subrun=True, the comparison-mode lane) on each nominated topic in a
bounded ThreadPoolExecutor against a wall-clock batch budget. A topic whose
sub-run raises is kept as nomination-only with the error recorded; topics
unfinished at budget expiry are likewise dropped to nomination-only. The batch
never raises and preserves nomination order. This is what buys discovery the
whole multi-source corpus (Reddit+comments, X, YouTube, Techmeme, arXiv, HN,
Polymarket, web) with zero bespoke fetch code.
* feat(discover): U4 confidence floor - 'nothing solid' beats ranked noise
THE bug fix for the 2026-07-12 junk-trend regression (five 1-like tweets
ranked as a 'sports' trend list). passes_discovery_floor() gates every topic:
an absolute engagement floor kills junk outright, then a topic clears via
independent cross-source confirmation OR a genuinely strong single-source
spike (a 1,600-point HN thread is a story; a 30-upvote meme is not).
run_discover gains enrich= (full pipeline pass per nomination via U3) and
judges each topic on its enriched corpus when available, seed evidence
otherwise. Zero floor-clearing topics -> DiscoveryReport.outcome
'nothing-solid' with the strongest sub-floor name in weak_signal, exported
through to_discovery_export. Regression test: the junk corpus now returns an
honest empty result.
* feat(discover): U5 global trending mode - bare --discover with no domain
--discover DOMAIN is now nargs-optional: bare --discover sweeps every river
feed's own hot list (r/all, HN front page, Digg) with the keyword gate off -
the feeds' hot ranking is the signal and the confidence floor culls junk. X
sits out of the global nominate stage (its search lane needs a keyword) and
joins per-topic at enrichment. Dispatch keys on 'flag present' so the empty
domain is never mistaken for 'flag absent'.
Discovery now enriches by default (full research pass per nomination);
--discover-shallow is the fast escape hatch that ranks on listing evidence
only, still floored. Global saves slug as 'trending'.
* feat(discover): U6 rich trend cards + honest nothing-solid rendering
DiscoveryTopic gains top_comment (strongest verbatim community comment from
the enriched corpus, per-platform vote-normalized, with attribution) and
corroboration_count; both exported in to_discovery_export. Renderer upgrades:
'# Trending now' header for global runs, a 'confirmed across N sources' badge
on the momentum line, a '**Community voice:**' quote line on enriched cards,
and an explicit nothing-solid empty state naming the closest weak signal
instead of the old generic 'no clusters survived' line.
* feat(discover): U7 Techmeme + arXiv reach discovery through enrichment
Enrichment sub-runs get requested_sources=None by default, so every available
source - Techmeme, arXiv, YouTube, Polymarket, community comments - joins each
nominated topic's research pass despite having no river feed of its own. An
explicit user --search boundary now holds through enrichment too (the CLI
threads the original pre-narrowed list as enrich_requested_sources). The
keyword-source nominate gap (X/Techmeme/arXiv have no front-page lane) is
documented at the planner.
* docs(discover): U8 route global vs domain trending; document two-stage flow
SKILL.md discovery branch now routes bare trending asks (/last30days trending,
'what's hot') to bare --discover and domain asks to --discover DOMAIN,
documents the nominate-enrich timing (10-minute Bash timeout), the
--discover-shallow fast path, and pins 'Nothing solid this window' as a valid
verbatim pass-through outcome (never retry or fabricate around it).
CONFIGURATION.md and the JSON export reference document the new flag surface
and contract fields (outcome, weak_signal, top_comment, corroboration_count).
* fix(discover): strip quote chars from community-voice bodies to avoid doubled quotes
Live global-trending verification surfaced comments whose bodies start with a
quote character rendering as doubled quotes inside the card's wrapping quotes.
* fix(discover): address PR #816 review - real budget enforcement + shallow-flag guard
P1: enrichment workers now run as daemon threads behind a semaphore instead of
a ThreadPoolExecutor. Executor threads are non-daemon and joined at interpreter
shutdown, so one hung sub-run could keep the process alive long after its topic
was downgraded to nomination-only. Daemon workers make the wall-clock budget
real; abandonment is safe because internal_subrun passes write nothing to disk
and every fetch layer carries its own timeout.
P2: --discover-shallow without --discover now errors (exit 2) instead of
silently no-opping into a full research run.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(sources): honor INCLUDE_SOURCES for xiaohongshu and dripstack
Both sources were per-run-request-only, so a persisted
INCLUDE_SOURCES=xiaohongshu or dripstack in .env was silently ignored -
the exact silent-ignore class documented from #707/#708 - and the only
persistent route (LAST30DAYS_DEFAULT_SEARCH) replaces the whole source
set instead of adding to it. Both gates now follow the established
LinkedIn/Perplexity/Trustpilot opt-in pattern: active when the source
is in INCLUDE_SOURCES or explicitly requested, otherwise off, with the
xiaohongshu availability probe still firing only after an opt-in.
Default behavior is unchanged; tests lock the checkbox on, the default
off, and the probe never firing without opt-in.
* fix(sources): strip whitespace in INCLUDE_SOURCES tokens
INCLUDE_SOURCES=linkedin, dripstack (space after the comma) produced a
' dripstack' token that failed the exact membership checks, silently
dropping the persisted opt-in - a pre-existing wart for every
INCLUDE_SOURCES consumer that this PR's hoisted computation now fixes
for all of them.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* Add DripStack to related skills
DripStack is a complementary AI agent skill for premium financial
newsletter research via micropayments. Same audience, different data
layer — last30days covers public/social signal, DripStack covers
paywalled financial signal.
* Add DripStack as a source: premium financial newsletter search
DripStack indexes paid Substack newsletters, analyst writeups, and
financial podcasts. The search endpoint is free and public (no API key
required) — returns article metadata with relevance-scored snippets.
Integration:
- New source module: scripts/lib/dripstack.py
- Normalizer registered in normalize.py
- Wired into pipeline.py (always available, like GitHub/HN)
- Planner assigns it to opinion/prediction intents alongside StockTwits
- Mock data for tests
- SKILL.md display name mapping
- CONFIGURATION.md source table entry
Signal: complementary to StockTwits (retail sentiment) and Polymarket
(prediction odds) — DripStack covers what professional analysts and paid
newsletter authors are writing about. Publication attribution (e.g.
'SemiAnalysis', 'Bloomberg') is high-credibility signal for synthesis.
* fix(dripstack): requested-only gating, shared http path, window filtering
Maintainer hardening on #791 before merge (owner decision: every new
source ships default-off with an explicit toggle):
- DripStack activates only when explicitly requested (--search dripstack
or LAST30DAYS_DEFAULT_SEARCH), mirroring the Xiaohongshu guard; default
runs send zero traffic to dripstack.xyz and their available_sources set
is unchanged.
- All requests route through the shared lib/http.py choke point so
capture/replay, fixtures, and the failure taxonomy apply.
- Results honor the 30-day window at fetch (dated items outside the
window are dropped; undated items are kept, not guessed).
- Docs describe the opt-in; the README Related Skills block recommending
a separate skill install is removed (out of scope for a source PR).
- tests/test_dripstack.py covers gating, choke point, windowing, failure,
and normalization.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(sources): local corpus source - your own files as a ranked signal
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: defang corpus sentinels, matching-window snippets, exclusion-aware hosted gate, traversal notes, bounded discovery
* fix: keep absolute local paths out of corpus notes and coverage diagnostics
* fix(corpus): keep raw exception text out of coverage notes
OS and subprocess errors embed the failing absolute path in str(exc),
and scan/cache notes flow into source_status detail rendered in
coverage diagnostics outside the private corpus block. Notes now carry
the error's strerror (or class name) instead, so a permission failure
or file race can no longer leak a local path from a private run.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(grounding): per-claim freshness verdicts (--verify-freshness)
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: prefer active markets in mixed-event verification, plan-only cookies for cached verification
* fix: verify event identity on slug-fallback refetch before assigning verdicts
* test: use a real outcome datum key in slug-identity test
* fix(freshness): carry values in stale detail and announce zero-claim passes
A stale verdict's machine-readable detail said only that the value
moved; the rendered table already showed was/now, so agent-JSON
consumers saw less than readers. Compose the detail as
"moved: <original> -> <current>", formatting each value the way the
claim text does (percent for Polymarket/StockTwits, thousands
separators for stars).
A verification pass that extracts zero claims used to complete
silently, which misled a live post-hoc run. _verify_report_set now
prints one stderr note when no verified report produced any verdict,
aggregated across main and entity reports so multi-report passes note
it once; the drill path routes through the same helper.
* fix(freshness): verify star facts attached by candidate enrichment
A GitHub-flavored run produced zero verdicts live: star enrichment
attaches metadata["github_stars"] to candidates after reranking, but
claim extraction read only item-level engagement, and those candidates'
primary items are typically non-GitHub sources.
Extraction now emits one repo-keyed star claim per enriched repo
(skipping repos already claimed at item level). These claims cannot
ride the item-source dispatch - they would land unsupported as
"reddit"/"x" claims - so verify_report routes the repo-slug shape
straight to the GitHub refetcher, bypassing the grounding-item lookup
and the per-source outcome gate (the datum came from enrichment, not
the github search source). refetch_datum accepts an owner/repo slug as
the datum key, and the snapshot cache keys these claims by repo so
multiple candidates citing one repo share one request.
* fix(freshness): review fixes - honest evidence, joinable export, rendered detail
Four review findings on the hardening pass:
- Unsupported verdicts no longer fabricate evidence: evidence_url and
evidence_timestamp stay empty when no fresh evidence was obtained;
provenance remains on source_url/source_timestamp.
- Agent export results now carry candidate_id so consumers can join
freshness_verdicts to the result they annotate (schema_version 1.2,
golden snapshot and docs updated).
- The rendered verdict table shows the verifier's detail - formatted
movement on stale rows, the un-checkable reason on unsupported rows -
instead of reconstructing raw values.
- Item-level star dedup is scoped per candidate: another candidate's
item-level claim no longer suppresses an enriched candidate's own
verdict and inline flag; both share one repo snapshot.
* fix(freshness): require event-id identity on Polymarket slug fallback
The slug-fallback refetch verified the response slug but accepted a
slug-matched event whose id differed from the cached item's event id,
so a re-created event reusing a slug could produce current/stale
verdicts from another market's prices. When the cached item carries a
real numeric event id, the fallback now requires the response id to
match it (synthetic PM<N> parse ids carry no identity and skip the
check); mismatches raise and degrade to an unsupported verdict.
* fix(freshness): fail closed when the slug fallback has no event identity
A cached item with neither metadata.event_id nor a real numeric item id
(the synthetic PM<N> parse fallback) let the slug fallback accept
whatever event currently owns the slug, so a recreated or loosely
matched event could still drive a current/stale verdict. With no
identity to verify against, the refetch now raises before any request
and the claim degrades to unsupported.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(library): FTS search over saved briefs with self-citing library context
* fix: address self-review findings
* fix: safety note on search output, reject --output, repopulate FTS after table loss
* fix: per-library search index and scoped briefing archive for --save-dir libraries
* fix(library): keep scoped runs out of the shared store and index
Three leaks between --save-dir scopes and the shared research store:
- library search with --save-dir merged shared-store sightings into a
scoped search, so one client's completed-run snippets could surface
in another client's results. Scoped searches now read only a store
inside the save dir (usually absent).
- The passive "From your library" context had the same fallback to
the shared store on scoped runs; it now scopes identically.
- The markdown save hook synced the shared library.db from a scoped
scan, and sync's stale-row prune could delete other scopes' (and the
default library's) rows. Scoped saves now sync a per-directory index
with the same paths scoped search uses.
* fix(store): write scoped-run findings to the save-dir store
Scoped reads (library search, library context) were fixed to stay
inside the save dir, but --store on a --save-dir run still wrote
findings through the shared research.db - so a scoped client's own
sightings never appeared in its scoped searches while remaining
visible to unscoped ones. persist_report now routes store access
through a scoped_db context to <save-dir>/research.db, the same path
the scoped read side uses; unscoped runs keep the shared store.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(library): publish saved research as index and Atom feed
Scan saved reports and briefing archives into deterministic, topic-grouped library pages with stable Atom entries. Add explicit opt-in multi-document publishing, publish-safe social IDs, slash-command guidance, configuration docs, and regression coverage.
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: keep per-suffix reports distinct in library identity
* fix: scope briefing archive to the selected library, back up hand-written index/feed files
* fix(library): prune only marker-bearing briefs and never clobber a backup
A refresh deleted any orphaned briefs/ page whose filename looked
generated, so a hand-written page with a generated-looking name could
be removed. Rendered briefs now embed an ownership comment marker and
the prune requires it before unlink; name shape alone is no longer
grounds for deletion.
The hand-written index/feed backup used replace(), so a second refresh
overwrote the first .bak. Backups now pick the first free .bak/.bakN
name instead.
* fix(library): preserve hand-edited brief pages on refresh
The orphan prune checks the generator marker, but the brief write path
replaced briefs/<name>.html for current entries without any ownership
check, silently destroying a hand-edited or hand-written page whose
name matches a current report. Every library-feed write (briefs, index,
feed) now goes through one preservation helper: a page without the
generator marker is moved to the first free .bak/.bakN name before the
generated page is written.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(pipeline): add --discover mode for topic-less trending discovery
* fix: address self-review findings
* fix: address round-2 residual (surgical round)
* fix: boundary-aware category match, additive-only discovery totals, preserve discovery intent through onboarding
* fix: visible default-search fallback, non-plural domain anchors, clean X backend fallback
* fix(discover): apply domain filter to Digg listing clusters
The Digg source is an AI-focused leaderboard feed, so a discovery sweep
on a non-AI domain surfaced off-domain stories (observed live: a crypto
sweep returning model-release coverage). Filter parsed Digg clusters
with the same title-level _matches_discovery_domain check the adjacent
Hacker News branch already applies; an all-filtered result stays a
clean no-results outcome.
* fix(discover): surface the producing X backend's own error as partial
Failing over past a dead backend is a clean outcome, but when the
backend that actually produced items also returned an error, the sweep
is degraded; returning a clean state hid that from source_status and
strict-exit runs. Keep the producing backend's error; earlier fallback
errors remain stderr observability only.
* fix(discover): hold the configured source boundary instead of widening
When the configured source filter (LAST30DAYS_DEFAULT_SEARCH or
--search) contains no discovery-capable source, the sweep silently
widened to all discovery feeds with only a warning, querying sources
the user had filtered out. Discovery now exits 2 with the unsupported
sources and the discovery-capable set named, for both the config and
explicit-flag paths.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(schema): versioned agent JSON export profile with --json-profile raw fallback
* fix: address self-review findings
* fix: exclude author-reach fields from headline engagement (stocktwits followers)
* fix: fail loudly when the evaluator receives the agent profile instead of raw Report
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(pipeline): typed per-run source outcomes in the report
* fix: address self-review findings
* fix: propagate failure capture to nested pools, surface authed GitHub failures, classify wrapped timeouts, opt-in strict exit
* fix: pick the most specific captured failure, not the last-appended one
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(health): dependency probe taxonomy with reinstall prescriptions
Uniform ok/missing/broken/timeout probes for yt-dlp, PP CLIs, node,
ffmpeg. broken = resolves on PATH but cannot exec (the #692 stale-shim
false-negative); prescriptions keyed by owning package manager;
off-PATH-but-on-disk reports missing with a PATH fix. (U1 of 5)
* feat(backends): chain descriptors with predicted will-use resolution
Alternative mode (X/YouTube/web) probes all candidates side-effect-free
then picks first fully-usable, so an unauthenticated preferred backend
cannot shadow a working fallback; Reddit renders conditional
public-default + ScrapeCreators-backfill wording instead of a fake
winner. Chains import env.py's own definitions; paid lanes probe key
presence only, guarded by no-network tests. (U2 of 5)
* feat(prescriptions): single fix-remediation registry shared by nudges
(source, failure) -> cause + natural-language fix + exact CLI fix +
config anchor, seeded with the known credential/tool failure inventory.
quality_nudge composes its fix text from registry entries (wording
byte-identical); backends.py setup-command prescriptions embed the same
strings, so doctor and nudges cannot drift. (U3 of 5)
* feat(doctor): unified doctor command with tier rollup and frozen aliases
/last30days doctor aggregates dependency probes, backend descriptors,
prescriptions, diagnose data, and the permission summary into one
grouped report (text + --json). Tier rollup per the plan's table;
per-source exception isolation; predicted will-use backend for chains;
no secrets, no cookie reads, no network; exit 0. --diagnose/--preflight
shapes characterization-frozen (pre-v3.9.0 baseline, re-record on
landing) with explicit MCP-passthrough and available_sources consumer
tests. (U4 of 5)
* feat(doctor): TTL cache, --cached read path, SKILL.md standing rule
Doctor persists its JSON beside last-run.json (doctor-cache.json);
--cached serves within LAST30DAYS_DOCTOR_TTL (900s default, registered
in env; 0 disables) and falls through live on stale/corrupt/absent.
Explicit doctor always runs live and refreshes. SKILL.md gains doctor
triggers plus the cached pre-research rule (2 lines); CONFIGURATION.md
documents the command, cache, TTL, and pin vars. (U5 of 5)
* refactor(doctor): simplification pass across the doctor stack
Concurrent per-source probes (order-preserving pool.map, worst case now
bounded by slowest probe); memoized xurl availability (kills a double
whoami spawn); INCLUDE_SOURCES token parsing matches the pipeline gate;
shared timestamp-freshness helper; tier/status constants unified;
public health accessors replace private cross-module reads; explicit
DependencyProbe.off_path replaces detail-string sniffing.
* fix(review): truthful probes and cache trust (wave 1)
Doctor's X probe is now genuinely local-only: xurl checked via its
token store, never whoami; the safe/diagnose path can no longer reach
the network through any of three vectors (forbid-network tests lock
it). Bird checks cookies before node so unconfigured X reports off,
not error. Cache layer: schema stamp, full shape validation with live
fall-through (fixes the reproduced KeyError crash), config fingerprint
invalidation, generated_at/from_cache staleness signals, and cache
write failures now warn on stderr.
* fix(review): single-source-of-truth pass (wave 2)
pipeline.py and providers.py read env's pin/floor constants instead of
restating the raw strings (parity tests guard it); env grows public
include_sources()/is_setup_complete() wrappers ending doctor's private
reaches; health.py and setup_wizard share one installer candidate-dir
list, restoring the Windows PrintingPress dir the mirror had dropped
(the documented Digg off-PATH failure mode), with Windows regression
tests.
* fix(review): doctor output fidelity (wave 3)
requires names the actually-failed backend; digg broken/timeout gets
reinstall-framed prescriptions; ok-tier fixes render in text (YouTube
transcription backstop now visible + covered); SKILL.md frontmatter
gains health-check discovery keywords and the standing rule is marked
mandatory, both locked by contract tests.
* fix(doctor): rebase onto v3.10.0 baseline + Greptile review fixes
Re-record the diagnose/preflight characterization snapshots against the
committed v3.10.0 baseline (new sources, x_pending_browser_auth key, PP
CLI external commands) per the planned re-record path; thread the
no-network gate through main's new x_pending precompute so the safe
path stays provably offline; tolerant alt_cli lookup in quality_nudge
(Greptile P1) with an arity pin test; autouse probe-cache reset fixture
(Greptile P2).
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(techmeme): window results to real dates, drop useless sync, tolerate old-binary prose
The techmeme-pp-cli search command hits Techmeme's live archive (back to
~2005), not the synced cache, and until now the adapter stamped every hit
with today's date - so a Dec 2022 Parler headline could appear in a
last-30-days report as current news (observed in the 2026-07-04 Kanye West
run). Fixed CLI binaries now emit a per-record ISO date; the adapter windows
records to the research range on that date, keeps undated records as
honestly dateless (date_confidence: low downstream), deletes the pre-search
sync that search never read, and parses old binaries' zero-hit prose
('No results for ...') as an empty result set instead of a decode error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWuSdMPdQLnAeh65wTeG3L
* fix(techmeme): apply review findings - cap priority for dated records, windowing-inactive hint
Review pass caught: undated archive hits could evict confirmed in-window
stories from the depth cap (the inline comment overclaimed 'stale records
never consume cap slots' - only dated ones); and old-binary degraded mode
was invisible. Dated in-window records now take cap slots first, undated
fill the remainder; when zero records carry usable dates the adapter logs a
windowing-inactive hint with the upgrade command. Plus boundary/non-string/
whitespace date tests, the dropped-records log assertion, and a docstring
note on the deliberate divergence from arxiv.py's drop-on-unparseable
policy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWuSdMPdQLnAeh65wTeG3L
* fix(techmeme): correct npm package in the upgrade hint
Greptile review: the windowing-inactive hint named @mvanhorn/printing-press;
the repo's canonical installer (setup_wizard.PRINTING_PRESS_NPM and every
CONFIGURATION.md example) is @mvanhorn/printing-press-library with
--cli-only. A user running the hint verbatim would have hit the wrong
package.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWuSdMPdQLnAeh65wTeG3L
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(instagram): wire Instagram comments via ScrapeCreators
U1: instagram.enrich_with_comments + _fetch_post_comments hit
GET /v2/instagram/post/comments, sorted by comment_like_count;
env.is_instagram_comments_available gate (key + instagram_comments in
INCLUDE_SOURCES); pipeline enriches the Instagram source when available.
Mirrors the TikTok-comments implementation.
* feat(instagram): full vote-weighting participation for IG comments
U2: comment_like_count is remapped to the shared score field; a dedicated
_instagram_engagement mirrors _tiktok_engagement with the 0.10 top-comment
carve-out so highly-liked IG comments lift item ranking; render gains the
instagram vote label (likes) + min-score threshold. _VOTE_LOG_REFERENCE
already had instagram. IG comment votes now weight ranking/Best-Takes/render
like YouTube/TikTok.
* feat(render): rank-based cross-platform comment diversity
U3: _render_top_comments interleaves comments round-robin by within-platform
rank (every platform's #1, then every #2, then #3) instead of a global vote-
magnitude sort. Top-3-of-each-platform outranks 4th-of-any and each platform's
#1 is guaranteed a slot -- a viral platform can no longer sweep the list. The
cross-platform list drops the per-platform absolute floor (min_score=0) so a
less-watched video's killer low-vote comment surfaces too; the per-candidate
card still applies the floor. Vote strength only orders within a platform.
* feat(nux): comments-by-default Step 5 tier copy
U4: recommended tier is now posts AND top comments for TikTok + Instagram plus
YouTube comments (INCLUDE_SOURCES=tiktok,instagram,youtube_comments,
tiktok_comments,instagram_comments); Everything adds Threads + Pinterest. No
posts-only tier. Operator-approved copy; mirrored in the non-modal 5b flow.
* feat(nux): accurate ScrapeCreators offer copy + fix stale docstrings
U5: Step 4 offer states comments are on by default (posts AND top comments +
YouTube comments) and describes the real auto-enrichment (Reddit public+SC
merged, YouTube search backstop) instead of vague 'backup'. Fixes the
is_youtube_comments_available docstring that claimed comments are excluded
from the Recommended tier. (Xiaohongshu was never labeled ScrapeCreators in
the docs; no change needed there.)
* docs: comments-by-default tier + instagram_comments in AGENTS/CONFIGURATION
U6: AGENTS.md onboarding note now describes comments-on-by-default (Recommended
tier), IG-comments parity, and the rank-based diversity selection.
CONFIGURATION.md documents instagram_comments/tiktok_comments INCLUDE_SOURCES
keys and the new default string; corrects the youtube_comments row and the
Step 5 tier prose.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(nux): make first-run welcome and GitHub device code deterministic
The v3.9.3 prose fixes for the welcome and the GitHub device code failed a
real cold run: the model skipped the welcome despite 'REQUIRED FIRST', and
never surfaced the device code (backgrounding + poll orchestration). Move
both from model-obeyed prose to engine-driven behavior.
- Welcome: new 'last30days.py --welcome' command prints the engine-owned
welcome; SKILL.md Step 1 relays it verbatim. Single source of truth; the
model reliably relays command output even when it skips prose.
- GitHub device code: split setup --github into --github-start (submit, copy
to clipboard, print the code to stdout, open browser, return immediately)
and --github-poll (wait, fetch, persist). run_github_start keeps the
existing-key short-circuit; setup --github still chains both for back-compat.
- Code printed to stdout as a plain line so a foreground caller can't miss it;
clipboard claim only made when pbcopy succeeded.
Updates onboarding contract tests, adds tests/test_welcome.py, updates
AGENTS.md and CONFIGURATION.md. Full suite green (coverage 84.06%).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBTxHmgTPg5xsgyUbgNdiz
* fix(nux): honest clipboard_ok in poll + in-memory handle for one-shot
Greptile findings on the device-auth split:
- run_github_poll hardcoded clipboard_ok=True, so the periodic poll reminder
would falsely claim the code was on the clipboard on non-macOS or when
pbcopy failed. Now clipboard_ok is carried in the handle (file + in-memory)
and read back, defaulting to False when absent.
- run_full_device_auth (one-shot --github) relied on the persisted handle
file, so a failed handle write stranded it. Extracted _start_device_flow
returning (public, handle); the one-shot now passes the handle to poll
in-memory (device_code never enters the public stdout dict), and
run_github_start persists the file only for the separate --github-poll
process.
Adds tests for both. Full suite green (coverage 84.09%).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBTxHmgTPg5xsgyUbgNdiz
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds an optional hosted-backend path: when both LAST30DAYS_API_KEY and
LAST30DAYS_API_BASE are set (and --mock is not passed), research runs
through the configured remote API instead of local sources - submit,
poll with progress on stderr, render the server's report. The endpoint
comes only from LAST30DAYS_API_BASE; there is no built-in default, so
with either variable unset the engine runs local sources unchanged.
Claude-Session: https://claude.ai/code/session_012gvxSQgfjp6RDyv6726VeB
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(trustpilot): resolve name->domain, warm session pre-fanout, cap to one fetch
Trustpilot returned 0 items on company topics: the engine passed the raw
topic name to a CLI keyed by domain (info ThriftBooks -> HTTP 404), and N
parallel subqueries each raced their own Chrome WAF-cookie harvest.
- --trustpilot-domain flag (verbatim, bypasses the brand-shape gate; flows
into competitors-plan sub-runs with provenance: user-set is final,
resolved hints retry via search on a miss)
- name->domain resolution via the CLI's search, cached per topic; name-match
mandatory, ambiguous multi-hit falls back rather than misattributing
- ensure_session_ready: one process-global, lock-serialized auth status /
auth login before the fan-out; brand-gated so generic topics never launch
Chrome; logs structured status strings only
- MAX_SOURCE_FETCHES trustpilot=1: N streams used identical identifiers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhBKEWrZV9cpfzNu7rmRzR
* feat(resolve): auto-resolve a Trustpilot domain hint from official-site URLs
Headless --auto-resolve runs fill args.trustpilot_domain (hint provenance)
from news/handle search-result URLs whose registrable label matches the
topic. Hints are tier-2: the engine retries via CLI search when they miss.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhBKEWrZV9cpfzNu7rmRzR
* docs(skill): Step 0.5d Trustpilot-domain resolution + CONFIGURATION.md mirror
Pre-flight checklist row, Step 0.5d resolution subsection, Resolved-block
Trustpilot line, per-entity trustpilot_domain in --competitors-plan, and
the CONFIGURATION.md source-table row reflecting domain resolution and the
pre-fan-out session warm-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhBKEWrZV9cpfzNu7rmRzR
* fix(review): harden trustpilot resolution per 10-reviewer code review
- exclude trustpilot from the thin-source retry: it returns at most one
item by design, so '<3 items' re-fetched it after every success --
bypassing the fetch cap and re-resolving without --trustpilot-domain
(lookalike-misattribution path) [cross-model adversarial, verified]
- move the session warm-up from pipeline.run's pre-fan-out slot into
search_trustpilot's first fetch: never delays other sources' streams,
never fires when the plan fetches no trustpilot [reliability, adversarial]
- replace the warm-up's boolean-forever flag with a 240s monotonic TTL:
long-lived host processes re-check after the token window and retry a
previously failed login [cross-model adversarial, correctness]
- only USER-set domains bypass the brand-shape gate; an auto-resolved hint
no longer widens activation to generic topics [security]
- do not cache transient search errors as permanent negative resolutions
[cross-model adversarial, correctness]
- bound the hint-retry chain: skip when the first lookup already consumed
a full single-call budget [reliability]
- SKILL.md: --trustpilot-domain in the engine-command flag list
[project-standards], vs-mode per-entity lookup type 5 + main-topic
outer-flag clarification [agent-native, correctness]
- tests: thin-retry exclusion, main()-level flag plumbing, warm-up-at-
first-touch, TTL lapse, hint-on-generic-topic quiet, transient-error
not cached [testing]
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhBKEWrZV9cpfzNu7rmRzR
* fix(trustpilot): address Greptile P2s - degenerate payloads uncached, retry timer after warm-up
- an empty-stdout CLI response ({}) is a degenerate payload, not a
definitive no-match: skip the cache write so it retries next lookup
(a well-formed empty hits list still caches)
- start the hint-retry budget timer after the warm-up so a slow Chrome
harvest cannot consume the retry budget of a fast info call
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhBKEWrZV9cpfzNu7rmRzR
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(setup-wizard): scan Chrome/Chromium before Safari during onboarding
The wizard forced FROM_BROWSER=firefox,safari, so Chrome was never tried
and macOS users logged into X in Chrome fell through to Safari, whose
binarycookies read requires Full Disk Access — a dead-end. Chrome reads
via the Keychain with no FDA. Scan the Chromium family first (explicit
comma list preserves order; 'auto' is silent-first and would not).
Pin FROM_BROWSER only for a silent winner (firefox/safari); a Chromium
winner is left unpinned so steady-state runs never re-trigger the
Keychain prompt (U1 / R1 / R2 / OQ2 conservative default).
* fix(setup-wizard): surface device code on stdout, validate it, and short-circuit re-registration
Three fixes to the ScrapeCreators GitHub signup:
- U4: emit a structured device_code_ready line to stdout as soon as the
code is available, so a backgrounded caller can show it immediately
instead of the user seeing only a spinner until the 300s process exits.
- U5: validate user_code against ^[0-9A-Z]{4}-[0-9A-Z]{4}$ before copying,
labeling, or emitting it; a key-shaped value is never mislabeled as a
GitHub code. Scrub the incomplete-response log so a returning account's
raw key can't land in logs.
- U6: an already-saved SCRAPECREATORS_API_KEY short-circuits to
status=already_registered with no device dance; the CLI now masks the
key for every status that carries one, not just success, so it never
reaches captured stdout unmasked.
OQ3 conservative default: no key-shaped-user_code branch is built; the
.env-present check is the primary already-registered mechanism.
* fix(skill): Chrome-first cookie copy, honest gh signup, clearer sources, new-source surfacing
SKILL.md NUX copy, both Modal and Non-Modal flows:
- U2: cookie consent leads with Chrome (Keychain 'Always Allow' cue);
Full Disk Access reframed as the Safari-only fallback.
- U3: StockTwits surfaced as a ticker/crypto conditional source; a
power-user pointer to the Manual Setup guide for credential-gated
opt-ins (LinkedIn, Bluesky, Perplexity) - Threads/Pinterest stay out
of onboarding per the contract.
- U4 (orchestration): run setup --github in the background and poll
stdout for the device_code_ready line so the code is shown immediately
instead of a spinner; parse the LAST JSON line for final status; add a
foreground fallback for hosts that can't background.
- U7: remove the false 'GitHub CLI ~2 seconds - no browser' promise;
honest device-code copy; qualitative credit contrast (numberless,
OQ1 fallback); already_registered + timeout/denied recovery.
- U8: clearer TikTok/Instagram opt-in (per-run cost vs backup-only).
Contract test locks the new copy (Chrome-first, StockTwits, no false
promise, device_code_ready in both flows, already_registered).
* fix(skill): drop StockTwits from the setup welcome
StockTwits needs zero setup (keyless) and only auto-fires on ticker/crypto
topics, so advertising it in the first-run welcome is noise for the typical
user whose topic is non-financial. It self-activates when relevant; the
welcome should show what needs a setup action or is broadly always-on.
Contract test now locks it OUT of the welcome.
* fix(skill): restore StockTwits bullet in the setup welcome
Keep StockTwits surfaced in the welcome as a ticker/crypto-gated source
(sits with arXiv/Techmeme as specialized auto-on). Reverts the prior
removal per maintainer call; contract test locks it back in.
* fix(sources): make Threads, YouTube comments, and Pinterest real INCLUDE_SOURCES opt-ins
The onboarding 'Recommended vs Everything' tiers were a no-op: a
ScrapeCreators key auto-ran TikTok, Instagram, Threads, and YouTube
comments regardless of INCLUDE_SOURCES, and Pinterest's opt-in only read
the per-run --sources list (a persisted INCLUDE_SOURCES=pinterest never
activated it). Gate the three Everything-tier sources properly:
- Threads: gated in available_sources on INCLUDE_SOURCES (perplexity
pattern); is_threads_available stays an availability-only predicate.
- YouTube comments: is_youtube_comments_available flipped from opt-out
(default-on) to opt-in, mirroring is_tiktok_comments_available.
- Pinterest: available_sources gate now honors a persisted
INCLUDE_SOURCES=pinterest, not only requested_sources.
TikTok/Instagram stay auto-on with the key (the Recommended tier). New
TestScrapeCreatorsTierGating locks the tiers; YouTube-comments and
Threads tests reversed/clarified to the new contract.
Existing key-holders lose auto-on Threads/YouTube-comments until they
opt into Everything (additive sources; accepted, no migration).
* fix(skill): real Recommended vs Everything ScrapeCreators tiers + define backups
Step 4 offer defines 'backups' inline (keeps Reddit/YouTube working at
rate limits) instead of the undefined 'ScrapeCreators backups' jargon,
and drops the now-false 'YouTube comments auto-on' claim. Step 5 becomes
two honest tiers:
- Recommended: TikTok + Instagram (auto with key) + backups; writes no
INCLUDE_SOURCES.
- Everything: also Threads, Pinterest, YouTube/TikTok/Instagram comments;
writes the full INCLUDE_SOURCES list (incl. tiktok,instagram so
quality_nudge doesn't treat them as excluded).
Mirrored into the Non-Modal flow (new step 5b).
Contract test resliced: Threads/Pinterest asserted present in the Step 5
Everything option and absent from the welcome/offer (reversing the old
blanket 'never in onboarding' lock); the Everything INCLUDE_SOURCES write
is locked. AGENTS.md and CONFIGURATION.md updated to match (Threads/
Pinterest/YouTube-comments are opt-in, offered in the Everything tier).
* test: address Greptile review nits on PR #744
- test_already_registered_status_handled now also asserts the prose flow
(the handling was added to both flows but only the modal was checked).
- Rename mock_pbcopy -> mock_subprocess_run and add a note clarifying it
patches subprocess.run (the pbcopy call), per the review.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat: add StockTwits as a source, gated to ticker/crypto topics
StockTwits is cashtag-native and carries self-reported Bullish/Bearish
tags, making it uniquely good at one thing the other sources can't
quantify for a stock/crypto query: a sentiment ratio and retail volume
on a specific symbol. On a "ServiceNow stock" run today the engine leans
on the product community (r/servicenow) for lack of an investor-voice
source; StockTwits fills that gap.
Design:
- New lib/stocktwits.py: public API (no auth), symbol resolution
(cashtags, crypto .X aliases, name->symbol search), stream fetch with
cursor pagination + date filter, and a bull/bear aggregate.
- Gating is a single chokepoint: run() sets config["_financial_topic"]
via stocktwits.is_financial_topic(topic); available_sources() only
registers "stocktwits" when that flag is set. Because
eligible_sources = available ∩ capabilities, the planner can never
assign it to a non-financial topic (verified: "Kanye West",
"Apple pie recipe" -> no symbol, source absent).
- Items normalize like every other source; per-message sentiment + the
symbol-level aggregate ride in metadata for synthesis to cite.
- Planner: capability entry + added to breaking_news/prediction/opinion
priorities.
Output is a direction/volume signal only (retail-skewed, self-reported
tags) - not analysis. Documented in the module and CONFIGURATION.md.
Tests: tests/test_stocktwits.py (hermetic; HTTP patched) covers the
gate, name-resolution firing only for financial topics, parsing,
sentiment aggregation, normalize wiring, and the pipeline availability
gate. No network in tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Tighten the financial-topic gate and cap per-run fetches
- Drop ambiguous words (share/token/coin/bull/bear) from _FINANCE_HINTS:
they misfired on general topics like 'share files' or 'bear attacks'
and could inject stock chatter into non-financial runs
- Add unambiguous crypto names (bitcoin, btc, ethereum, solana, dogecoin,
cardano, xrp) so 'bitcoin price' gates without a cashtag; short
colliding aliases (eth, sol, ada, doge) stay out of the gate
- Register stocktwits in MAX_SOURCE_FETCHES so multiple subqueries don't
re-fetch the same symbol stream and burn the unauthenticated IP quota
- Reword the stale wiring docstring; negative and positive gate tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBTxHmgTPg5xsgyUbgNdiz
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(arxiv): add arXiv as a default-on research source
Mirrors the digg adapter: lib/arxiv.py shells out to arxiv-pp-cli with a
quoted, relevance-sorted query and a 365-day recency cutoff (research does
not trend daily; the cutoff also keeps arXiv quiet on non-research topics
where the only keyword match is a stale paper). Registered across pipeline
(MOCK_AVAILABLE_SOURCES, available_sources gate on which(arxiv-pp-cli),
binary map, dispatch, mock fixture), normalize, planner, signals, render
(emoji 📄), and ui. 12 unit tests.
* feat(techmeme): add Techmeme as a default-on tech-news source
lib/techmeme.py shells out to techmeme-pp-cli: ensures a one-time cache sync
per run, then searches via 'search <topic> --json' (not --agent, which implies
--compact and blanked records on pre-PR-1383 binaries). Filters publication-
name header rows, dates headlines to the sync time (the cache is the current
news cycle), and ranks on topic relevance plus source quality. Registered
across pipeline, normalize, planner, signals, render (emoji 📰), and ui.
11 unit tests.
* feat(trustpilot): add Trustpilot as a default-on brand-sentiment source
lib/trustpilot.py shells out to trustpilot-pp-cli 'info <company>' for
TrustScore + AI review summary. Three default-on safety gates: (1) brand-shape
gate -- the CLI is invoked only for company-like topics (domain token or a
<=2-word capitalized proper noun), so it stays quiet and never harvests Chrome
on generic topics; (2) LAST30DAYS_TRUSTPILOT_NO_BROWSER opt-out for automated
contexts; (3) graceful degradation to empty on any CLI failure. Registered
across pipeline (dispatch passes config), normalize, planner, signals
(reviews engagement), render (emoji ⭐), and ui. 19 unit tests.
* feat(setup): auto-install arxiv/techmeme/trustpilot from Printing Press
Generalize the digg install path into _install_pp_cli(slug, bin_name) and
install_default_pp_sources(), mirroring digg's action taxonomy
(already_installed | installed | installed_off_path | install_failed | no_npx)
and PATH-candidate detection. run_auto_setup now best-effort-installs the three
new default-on sources via npx @mvanhorn/printing-press-library install <slug>
--cli-only and reports per-CLI status under results['pp_sources']. 7 new tests;
updated the digg install test to assert_any_call (digg is now one of several
install calls).
* fix(techmeme): drop unsupported --max-results flag, cap client-side
Live evaluation surfaced 'Error: unknown flag: --max-results' on every
techmeme search -- the search subcommand has no result-limit flag, so the flag
broke the source entirely (0 records). Remove it and apply the depth cap
client-side after parsing. Verified live: searches now return real headlines.
* feat(sources): document the three default-on sources + add eval harness
SKILL.md auto-setup list now names arXiv (📄), Techmeme (📰), and Trustpilot
(⭐) with their gating notes. test_new_sources_eval.py drives the three across
five diverse real topics: a deterministic fire/quiet gating matrix (offline,
CI-safe) plus an opt-in LAST30DAYS_EVAL_LIVE mode that exercises the real CLIs
and asserts the negative controls stay quiet.
* fix(review): address code-review findings (opt-out, sync race, gating edges, docs)
- P1 Trustpilot browser opt-out was inert: LAST30DAYS_TRUSTPILOT_NO_BROWSER is
read from config, but config is built from an env.py allowlist that omitted
it, so the documented kill-switch never fired in production. Add an
os.environ fallback in _harvest_allowed and add the key to env.py's
allowlist; new test exercises the env-var path env.get_config uses.
- P1 CONFIGURATION.md: document the three new sources and the
LAST30DAYS_TRUSTPILOT_NO_BROWSER env var (AGENTS.md hard rule).
- P2 Techmeme sync: replace the unlocked _SYNCED bool with a lock + monotonic
TTL, fixing the concurrent-double-sync race under ThreadPoolExecutor fan-out
and the never-reset staleness in long-lived/multi-report processes.
- P2 Techmeme: test the client-side depth cap (the compensating half of the
--max-results removal) plus sync-failure swallow and headline-word boundary.
- P3 arXiv: one-day future-date grace (same-UTC-day papers no longer dropped),
empty-phrase guard (quote-only topic no longer searches all:""), and
end-to-end _run_cli tests (envelope flatten, non-zero exit, bad JSON).
Accepted residual (follow-up): adapter-level URL-scheme guard parity with digg
-- the actual XSS surface (HTML output) is already centrally guarded by
html_render._is_safe_link_url, and these sources emit URLs from trusted
first-party CLIs scraping known domains.
* change(trustpilot): make Trustpilot opt-in instead of default-on
Trustpilot's headless-Chrome WAF-cookie harvest is heavier than the zero-auth
arXiv/Techmeme sources, so gate it behind INCLUDE_SOURCES=trustpilot (the same
opt-in pattern as perplexity/linkedin) rather than activating on binary
presence. The setup wizard no longer auto-installs its binary (off-by-default
source); opting in installs it on demand. arXiv and Techmeme remain default-on.
Docs (SKILL.md, CONFIGURATION.md) updated; new tests assert off-by-default,
on-with-INCLUDE_SOURCES, and on-when-requested.
* fix(review): exclude single-word tech names from Trustpilot brand gate
Greptile P1: is_brand_shaped admitted bare capitalized language/framework/tool
names (Python, React, Docker, Rust, Linux, Swift, Java) because _GENERIC_TOKENS
only covered AI/query vocabulary. An opted-in user querying a single tech term
would trigger the Chrome WAF harvest and risk surfacing an unrelated company's
reviews. Add a _TECH_TOKENS exclusion set; explicit company intent still
resolves via the domain branch (e.g. docker.com). Parametrized tests lock the
behavior.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
The ScrapeCreators /v1/linkedin/search/posts parser was keyed on field
names the live API doesn't return (`text`/`date`), so every post was
dropped (10 raw -> 0 items). Map the real fields (`description`,
`datePublished`) into the existing fallback chains; the unit tests now
encode the real response shape with a regression guard for this exact bug.
Also honor "a LinkedIn article is high signal": articles never appear in
post search (only `/posts/` URLs come back), so add a bounded profile-
enrichment lane. On person topics (a returned post author matches the
topic) we make one `/v1/linkedin/profile` call and surface the author's
`articles[]` as high-signal items (relevance 0.9, ranked above posts).
Keyword topics make no profile call.
Off-by-default opt-in gating (INCLUDE_SOURCES=linkedin) is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012iWk4GFaH6dHjUEwsZneiB
- Gate linkedin in available_sources() behind INCLUDE_SOURCES/requested_sources,
matching the perplexity opt-in pattern, so existing SCRAPECREATORS_API_KEY
holders don't silently start spending credits on LinkedIn.
- Add instagram-style graceful date-range filtering to
parse_linkedin_response(), with fallback to keep all results if the
filter would otherwise return zero.
- Remove unused sys import in linkedin.py.
- Document linkedin in CONFIGURATION.md's source table and SKILL.md's
tags list, power-user opt-in note, and ACTIVE_SOURCES_LIST mapping.
- Add TestLinkedinAvailability (pipeline) and TestDateRangeFiltering
(linkedin) test coverage.
Matt Van Horn