feat(pipeline): --discover mode for topic-less trending discovery (#801)

* feat(pipeline): add --discover mode for topic-less trending discovery

* fix: address self-review findings

* fix: address round-2 residual (surgical round)

* fix: boundary-aware category match, additive-only discovery totals, preserve discovery intent through onboarding

* fix: visible default-search fallback, non-plural domain anchors, clean X backend fallback

* fix(discover): apply domain filter to Digg listing clusters

The Digg source is an AI-focused leaderboard feed, so a discovery sweep
on a non-AI domain surfaced off-domain stories (observed live: a crypto
sweep returning model-release coverage). Filter parsed Digg clusters
with the same title-level _matches_discovery_domain check the adjacent
Hacker News branch already applies; an all-filtered result stays a
clean no-results outcome.

* fix(discover): surface the producing X backend's own error as partial

Failing over past a dead backend is a clean outcome, but when the
backend that actually produced items also returned an error, the sweep
is degraded; returning a clean state hid that from source_status and
strict-exit runs. Keep the producing backend's error; earlier fallback
errors remain stderr observability only.

* fix(discover): hold the configured source boundary instead of widening

When the configured source filter (LAST30DAYS_DEFAULT_SEARCH or
--search) contains no discovery-capable source, the sweep silently
widened to all discovery feeds with only a warning, querying sources
the user had filtered out. Discovery now exits 2 with the unsupported
sources and the discovery-capable set named, for both the config and
explicit-flag paths.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
Matt Van Horn
2026-07-12 07:27:39 -07:00
committed by GitHub
parent ffa2e31e9b
commit 298310ca34
15 changed files with 1608 additions and 12 deletions
+1
View File
@@ -44,6 +44,7 @@ The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path(
- `--save-dir <path>` - one-off output location. **Flag wins over env var.** If neither flag nor env var is set, the engine does not write a file (DB persistence is independent — see `LAST30DAYS_STORE` below).
- `--output <file>` - write the rendered output to an exact file path, using the format selected by `--emit`.
- `--json-profile {agent,raw}` - select the research JSON shape used with `--emit=json`. `agent` is the default, versioned workflow contract; `raw` preserves the full internal `Report` dump for debugging and power users. See the [JSON export reference](docs/reference/json-export.md).
- `--discover <domain>` - topic-less trending discovery. Sweeps rising/top-week Reddit listings (category-mapped communities, with r/all as the uncategorized fallback), Hacker News front/best stories, Digg AI 1000 clusters when `digg-pp-cli` is on PATH, and broad X activity when an X backend is authenticated, then returns 5-10 engagement-velocity-ranked topics. Run without a positional topic; it is mutually exclusive with `--drill`. `--emit=json` uses the separate versioned discovery contract documented in the [JSON export reference](docs/reference/json-export.md).
- `--drill <target>` - deep follow-up over the fresh `~/.config/last30days/last-report.json` cache. Accepts a 1-based index (`--drill "cluster 3"` or `--drill "3"`) or a fuzzy cluster title/entity description. It re-fetches only sources that contributed to the matched cluster, enables their deep comment/transcript enrichment paths, merges/dedupes the evidence, and replaces the cache so drills can chain. Run it without a positional topic; if the cache is absent or expired, run a normal research pass first.
- `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`).
- `--no-browser-cookies` - hard-disable browser-cookie extraction for this run, even when `FROM_BROWSER` is configured. MCP and folder-mode hosts use this for safe defaults.