Compare commits

...

61 Commits

Author SHA1 Message Date
dependabot[bot] cae3843828 chore(deps): bump actions/attest from 4.2.0 to 4.2.2
Bumps [actions/attest](https://github.com/actions/attest) from 4.2.0 to 4.2.2.
- [Release notes](https://github.com/actions/attest/releases)
- [Changelog](https://github.com/actions/attest/blob/main/RELEASE.md)
- [Commits](https://github.com/actions/attest/compare/f7c74d28b9d84cb8768d0b8ca14a4bac6ef463e6...1e69f48acb82d1966a394da916b4c1698aa569d6)

---
updated-dependencies:
- dependency-name: actions/attest
  dependency-version: 4.2.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-16 18:43:19 +00:00
Matt Van Horn c7460f6114 chore(release): bump version to 3.21.0 (#1006)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-08-14 17:23:53 +00:00
Matt Van Horn 40b42b7bed feat(x): bird first, grok CLI opt-in only (#1005)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-08-14 10:16:04 -07:00
Matt Van Horn 0b35e0c2c8 chore(release): bump version to 3.20.0 (#1003)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-08-14 12:36:17 +00:00
Matt Van Horn ab124fc8f9 feat(x): retrieve-judge-retry so X stays on-topic (#998)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-08-14 05:27:11 -07:00
Matt Van Horn c7cc880c36 fix(grok): treat expired sessions as degraded, not ok (#1000)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-08-14 04:50:34 -07:00
Matt Van Horn acdc397fec fix(amazon): start review lane at search time so multi-source runs keep a real budget (#999)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-08-14 04:46:53 -07:00
github-actions[bot] 4fa89d772d chore(release): bump version to 3.19.0
Built CHANGELOG.md from changelog.d via towncrier and bumped lockstep version surfaces to 3.19.0.
2026-08-14 01:08:49 -07:00
Matt Van Horn 698bfd0c3d feat(x): X search via Grok CLI plus first-party evidence fixes (#991)
Grok CLI X search with no X credential, first-party evidence fixes, skip-floor when unidentified, and thin-retry handle protection.
2026-08-14 00:55:57 -07:00
Matt Van Horn ddc761d2c2 fix(rank): keep out-of-window evidence out of the top ranks (#993)
Cherry-picks the recency contract from #885 so stale items cannot lead ranks, including cluster representatives and the final sort.

Co-authored-by: bekonyn <noyanjeanbean@gmail.com>
2026-08-14 00:37:52 -07:00
Matt Van Horn 6835eb8b1e feat(reddit): arctic-shift fallback for shreddit listing lanes (#992)
Cherry-picks arctic listing fallback from #960 and keeps failed shreddit lanes honest when arctic recovers only part of the request.

Co-authored-by: technicianofthesacred <technicianofthesacred@users.noreply.github.com>
2026-08-14 00:35:17 -07:00
Matt Van Horn 55885bcacc fix(github): query is:issue and is:pull-request when authenticated + partition honesty (#994)
Cherry-picks authenticated GitHub search from #967 and reports partial failure when one partition fails.

Co-authored-by: Mario Marquez <60635115+Mariomarquezt@users.noreply.github.com>
2026-08-14 00:31:03 -07:00
Matt Van Horn e93c8249d8 feat(amazon): buyer-signal source via Bright Data CLI (#988)
Adds an opt-in `amazon` source backed by the Bright Data CLI: product
discovery with live ratings and prices, plus a capped sample of recent
reviews woven in as buyer voice.

The signal it exists for is drift -- an all-time rating set against the
average of only the reviews inside the last 30 days. The emoji footer
names products and the direction they moved rather than counting
inventory, following the Polymarket shape:

  📦 Amazon: 12 products │ Spirit E-325 Gas 4.4★→4.8★, Genesis E-325 Gas 4.4★ quiet

Off by default and dual-gated: the CLI must be on PATH and logged in, and
the run must explicitly request the source. It never auto-fires from
inferred intent. A drift arrow requires n>=5 dated in-window reviews,
since a live date census showed Amazon's top-reviews ordering is only
weakly recency-biased.

Includes doctor registration, honest setup reporting, SKILL.md trigger
guidance with worked examples, and CONFIGURATION.md coverage.

Reviewed by three independent agents plus Greptile (5/5). Fixes landed
for: the API key being visible in argv, --amazon-query leaking into
competitor sub-runs, a lane deadline that did not bound wall clock, brand
inference that silently disabled the competitor guard, and a saved-report
source list that dropped whole sources from the per-source dump.

Verified with three live end-to-end runs against a funded account.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 14:40:01 -07:00
Terry Li 1004324ad3 fix: save the full artifact on --emit=compact --save-dir runs (#924)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-07 14:51:06 -07:00
Som Samantray 9b4684d5f3 fix(github): strip planner-injected search qualifiers from topic queries (#955) 2026-08-07 14:50:16 -07:00
Anupam Mediratta 475a51e5dd fix: upgrade golang.org/x/text to 0.39.0 (CVE-2026-56852) (#941) 2026-08-07 14:49:33 -07:00
dependabot[bot] c3591ee056 chore(deps): bump trufflesecurity/trufflehog from 3.95.8 to 3.96.0 (#937)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 14:49:18 -07:00
dependabot[bot] ee003df621 chore(deps): bump ossf/scorecard-action from 2.4.3 to 2.4.4 (#936)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 14:49:08 -07:00
dependabot[bot] 47f532fe78 chore(deps): bump zizmorcore/zizmor-action from 0.6.0 to 0.6.1 (#935)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 14:48:58 -07:00
dependabot[bot] 5d21d9e924 chore(deps): bump astral-sh/setup-uv from 8.2.0 to 9.0.0 (#934)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 14:48:44 -07:00
dependabot[bot] 9fcbd7b472 chore(deps): bump github/codeql-action/upload-sarif from 4.37.1 to 4.37.3 (#933)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 14:48:33 -07:00
dependabot[bot] d82c6a0948 chore(deps): bump github.com/mark3labs/mcp-go from 0.56.0 to 0.57.0 in /mcp (#932)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 14:48:22 -07:00
Trevin Chow 52f53312ff docs: correct meaning-changing errors in the localized READMEs (#920) 2026-07-31 19:15:29 -07:00
Trevin Chow a22c8e7576 docs: add five localized READMEs (#915) 2026-07-30 19:23:58 -07:00
Eugene 57f8d2c87a fix(reddit): report keyless 429/403 as transport failures, not no-results (#900)
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-07-30 12:17:50 -07:00
Beko b0fdfb0d51 fix(github): repos lane when person-mode PR search is empty (#883)
Co-authored-by: bekonyn <noyanjeanbean@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-07-30 11:58:15 -07:00
Som Samantray a940c98965 fix: four independently-verified bugs (#905, #908, #886, #909) (#912)
Co-authored-by: SomSamantray <>
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-07-30 11:44:10 -07:00
21Silva cb4e93cff1 docs: add Simplified Chinese README (#881)
Co-authored-by: yifosheng001 <yifosheng001@ke.com>
2026-07-30 11:24:37 -07:00
Terry Li 53fb6104c9 fix(hackernews): stop dropping every HN comment on an unmeetable score floor (#890)
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-07-30 11:18:37 -07:00
Terry Li 21c54d646c fix(polymarket): match spelled-out topics against acronym titles (#891)
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-07-30 10:50:09 -07:00
Som Samantray f310b82c14 fix(setup-wizard): resolve npx via shutil.which on Windows (#904) (#911)
Co-authored-by: SomSamantray <>
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-07-30 10:22:43 -07:00
Trevin Chow ef6c279ceb fix(security): block SessionStart .env key RCE and align project trust (#914) 2026-07-30 10:14:55 -07:00
Trevin Chow 0188da7ce7 fix(ci): package MCPB without printing-press (#903)
Validate / tests (push) Has been cancelled
Scorecard / Scorecard analysis (push) Has been cancelled
Security / Dependency audit (push) Has been cancelled
Security / Dependency review (push) Has been cancelled
Security / Secret scan (push) Has been cancelled
Security / SAST scan (push) Has been cancelled
Tag release / tag (push) Has been cancelled
GitHub Actions Security Analysis with zizmor 🌈 / zizmor (push) Has been cancelled
Validate / eval (push) Has been cancelled
Validate / mcp-tests (push) Has been cancelled
2026-07-28 14:16:54 -07:00
github-actions[bot] 9067158574 chore(release): bump version to 3.18.4 (#902)
Release / build-skill (push) Has been cancelled
Release / build-mcpb (amd64, darwin, darwin/amd64) (push) Has been cancelled
Release / build-mcpb (amd64, linux, linux/amd64) (push) Has been cancelled
Release / build-mcpb (arm64, darwin, darwin/arm64) (push) Has been cancelled
Release / release (push) Has been cancelled
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-28 13:53:16 -07:00
Trevin Chow 5cc8e7cceb fix(youtube): stop comparison-mode search timeouts under fan-out (#901) 2026-07-28 13:51:13 -07:00
Trevin Chow e82dbd41e1 Merge pull request #896 from mvanhorn/dependabot/github_actions/actions/attest-4.2.0
Tag release / tag (push) Has been cancelled
Validate / tests (push) Has been cancelled
Validate / eval (push) Has been cancelled
Scorecard / Scorecard analysis (push) Has been cancelled
Security / Dependency audit (push) Has been cancelled
Security / Dependency review (push) Has been cancelled
Security / Secret scan (push) Has been cancelled
Security / SAST scan (push) Has been cancelled
Validate / mcp-tests (push) Has been cancelled
GitHub Actions Security Analysis with zizmor 🌈 / zizmor (push) Has been cancelled
2026-07-26 14:51:00 -07:00
Trevin Chow 4b071d2605 Merge branch 'main' into dependabot/github_actions/actions/attest-4.2.0 2026-07-26 14:50:18 -07:00
Trevin Chow b2ab9bf45b Merge pull request #895 from mvanhorn/dependabot/github_actions/zizmorcore/zizmor-action-0.6.0 2026-07-26 14:50:10 -07:00
Trevin Chow afb27826e9 Merge branch 'main' into dependabot/github_actions/zizmorcore/zizmor-action-0.6.0 2026-07-26 14:49:36 -07:00
Trevin Chow 3d17afb39e Merge pull request #894 from mvanhorn/dependabot/github_actions/actions/setup-go-7.0.0 2026-07-26 14:49:29 -07:00
Trevin Chow 0f205e64e1 Merge branch 'main' into dependabot/github_actions/actions/attest-4.2.0 2026-07-26 14:49:21 -07:00
Trevin Chow 2e8dcaf746 Merge branch 'main' into dependabot/github_actions/zizmorcore/zizmor-action-0.6.0 2026-07-26 14:49:14 -07:00
Trevin Chow e0011c07e3 Merge branch 'main' into dependabot/github_actions/actions/setup-go-7.0.0 2026-07-26 14:48:46 -07:00
Trevin Chow b92fbd4201 Merge pull request #893 from mvanhorn/dependabot/github_actions/actions/checkout-7.0.1 2026-07-26 14:48:38 -07:00
Trevin Chow 4c186a182c Merge branch 'main' into dependabot/github_actions/actions/checkout-7.0.1 2026-07-26 14:48:11 -07:00
Trevin Chow 69a2510637 Merge pull request #892 from mvanhorn/dependabot/github_actions/github/codeql-action/upload-sarif-4.37.1 2026-07-26 14:48:02 -07:00
dependabot[bot] 1434895279 chore(deps): bump actions/attest from 4.1.1 to 4.2.0
Bumps [actions/attest](https://github.com/actions/attest) from 4.1.1 to 4.2.0.
- [Release notes](https://github.com/actions/attest/releases)
- [Changelog](https://github.com/actions/attest/blob/main/RELEASE.md)
- [Commits](https://github.com/actions/attest/compare/a1948c3f048ba23858d222213b7c278aabede763...f7c74d28b9d84cb8768d0b8ca14a4bac6ef463e6)

---
updated-dependencies:
- dependency-name: actions/attest
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-26 18:44:03 +00:00
dependabot[bot] 0c0c51f1fe chore(deps): bump zizmorcore/zizmor-action from 0.5.7 to 0.6.0
Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.5.7 to 0.6.0.
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](https://github.com/zizmorcore/zizmor-action/compare/192e21d79ab29983730a13d1382995c2307fbcaa...6599ee8b7a49aef6a770f63d261d214911a7ce02)

---
updated-dependencies:
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-26 18:43:59 +00:00
dependabot[bot] 0b6109dd77 chore(deps): bump actions/setup-go from 6.5.0 to 7.0.0
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6.5.0 to 7.0.0.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/924ae3a1cded613372ab5595356fb5720e22ba16...b7ad1dad31e06c5925ef5d2fc7ad053ef454303e)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-26 18:43:56 +00:00
dependabot[bot] eda1ffe668 chore(deps): bump actions/checkout from 7.0.0 to 7.0.1
Bumps [actions/checkout](https://github.com/actions/checkout) from 7.0.0 to 7.0.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0...3d3c42e5aac5ba805825da76410c181273ba90b1)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-26 18:43:50 +00:00
dependabot[bot] 0612ff7605 chore(deps): bump github/codeql-action/upload-sarif
Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.0 to 4.37.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/99df26d4f13ea111d4ec1a7dddef6063f76b97e9...7188fc363630916deb702c7fdcf4e481b751f97a)

---
updated-dependencies:
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-26 18:43:42 +00:00
Trevin Chow a4e7eca516 Merge pull request #880 from mvanhorn/tmchow/fix-ci-workflow-failures
Scorecard / Scorecard analysis (push) Has been cancelled
Security / Dependency audit (push) Has been cancelled
Security / Dependency review (push) Has been cancelled
Security / Secret scan (push) Has been cancelled
Security / SAST scan (push) Has been cancelled
Tag release / tag (push) Has been cancelled
Validate / tests (push) Has been cancelled
Validate / eval (push) Has been cancelled
Validate / mcp-tests (push) Has been cancelled
GitHub Actions Security Analysis with zizmor 🌈 / zizmor (push) Has been cancelled
fix(ci): repair tag-release.yml YAML so main pushes stop failing
2026-07-24 18:33:41 -07:00
Trevin Chow 906d9614f5 test(ci): harden tag-release contract coverage for #880
Require PyYAML in CI and execute the VERSION sed pipeline against
direct and merge-commit message shapes.
2026-07-24 18:33:02 -07:00
Trevin Chow 48fecf0fa4 docs(changelog): name fragment after PR #880 2026-07-24 18:25:45 -07:00
Trevin Chow b6efa6445f fix(ci): make tag-release.yml parseable YAML again
Unquoted chore(release): in the job if: broke Actions parsing on every
main push; also match merge-commit messages when tagging.
2026-07-24 18:25:25 -07:00
Trevin Chow fcff56b57c Merge pull request #879 from mvanhorn/release/v3.18.3
Release / build-skill (push) Has been cancelled
Release / build-mcpb (amd64, darwin, darwin/amd64) (push) Has been cancelled
Release / build-mcpb (amd64, linux, linux/amd64) (push) Has been cancelled
Release / build-mcpb (arm64, darwin, darwin/arm64) (push) Has been cancelled
Release / release (push) Has been cancelled
chore(release): bump version to 3.18.3
2026-07-24 18:17:57 -07:00
github-actions[bot] bb08c50bdd chore(release): bump version to 3.18.3 2026-07-25 01:15:04 +00:00
Trevin Chow 3d33d97a35 Merge pull request #851 from amitvijapur/fix/youtube-failure-count
fix(youtube): log ScrapeCreators transcript rescue instead of masking it
2026-07-24 18:11:17 -07:00
Trevin Chow f60d477d74 docs(changelog): add towncrier fragment for #851 2026-07-24 18:07:53 -07:00
Amit Vijapur 94138da0ae fix(youtube): use neutral wording for ScrapeCreators rescue logs
The SC fallback can fire when yt-dlp was never attempted (not
installed) and only the direct HTTP path failed, so attributing the
rescue to a yt-dlp failure could mislead operators about the cause.
Attribute it to the keyless fetch cascade instead, and drop the
yt-dlp clause from the batch summary. Addresses the review note.
2026-07-24 18:07:52 -07:00
Amit Vijapur 1809f08f9d fix(youtube): log ScrapeCreators transcript rescue instead of masking it (#831)
_sc_fetch_transcript() had no logging on its success path. When yt-dlp
failed for every video in a batch (rate-limited or bot-gated) and the
ScrapeCreators fallback silently rescued them, fetch_transcripts_parallel()
printed "Got transcripts for N/N videos (0 failed)" with zero trace of the
fallback ever firing — making a fully rate-limited run read as a clean
success.

Log the rescue in fetch_transcript() and flag it via status['sc_rescued'],
then fold that count into the batch summary so a masked failure can't be
misread as "0 failed": the line now reads "N/N videos (M failed, K rescued
via ScrapeCreators fallback after yt-dlp failure)" whenever the fallback
was needed, unchanged otherwise.
2026-07-24 18:07:52 -07:00
133 changed files with 17445 additions and 717 deletions
+1 -1
View File
@@ -11,7 +11,7 @@
{
"name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.18.2",
"version": "3.21.0",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days",
"version": "3.18.2",
"version": "3.21.0",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": {
"name": "Matt Van Horn",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days",
"version": "3.18.2",
"version": "3.21.0",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and the web.",
"author": {
"name": "Matt Van Horn",
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
pull-requests: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
+2 -2
View File
@@ -30,13 +30,13 @@ jobs:
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- name: Set up Python
run: uv python install 3.12
+31 -37
View File
@@ -31,7 +31,7 @@ jobs:
attestations: write
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ env.RELEASE_TAG }}
fetch-depth: 0
@@ -43,7 +43,7 @@ jobs:
test -f dist/last30days.skill
- name: Attest .skill artifact provenance
uses: actions/attest@a1948c3f048ba23858d222213b7c278aabede763 # v4.1.1
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4.2.2
with:
subject-path: dist/last30days.skill
@@ -54,8 +54,8 @@ jobs:
path: dist/last30days.skill
# Cross-compile the Go MCP server for each Claude Desktop platform and
# package each as a .mcpb. printing-press bundle handles the manifest +
# zip layout; we only supply the pre-built binary via --skip-build.
# package each as a .mcpb. MCPB v0.3 is a ZIP containing the checked-in
# manifest and the pre-built binary at the manifest's entry point.
build-mcpb:
runs-on: ubuntu-latest
permissions:
@@ -64,49 +64,32 @@ jobs:
attestations: write
env:
MCPB_OUTPUT: mcp/build/last30days-pp-mcp-${{ matrix.goos }}-${{ matrix.goarch }}.mcpb
MCPB_PLATFORM: ${{ matrix.platform }}
strategy:
fail-fast: false
matrix:
include:
- goos: darwin
goarch: arm64
platform: darwin/arm64
- goos: darwin
goarch: amd64
platform: darwin/amd64
- goos: linux
goarch: amd64
platform: linux/amd64
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ env.RELEASE_TAG }}
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
# printing-press v4.8.0 declares `go >= 1.26.3`, newer than the
# engine's own floor in mcp/go.mod. Install a 1.26.x toolchain so the
# PP `go install` below is satisfied without a runtime toolchain
# download (which GOSUMDB=off would block). Building the MCP binary
# with a newer toolchain than mcp/go.mod declares is backward-safe.
# Build the MCP binary with a toolchain newer than the floor in
# mcp/go.mod. Keeping this explicit also avoids a runtime toolchain
# download during the cross-compile.
go-version: "1.26"
cache: false
- name: Install printing-press
# Pin to a known-good PP release so the bundle command's behavior
# is deterministic across our tags. Bump deliberately when adopting
# a newer PP version. GOSUMDB=off skips the sumdb 404 some
# private-namespaced go install calls hit even when the repo is
# public; harmless here because the module path is fully qualified.
env:
GOPRIVATE: github.com/mvanhorn/*
GOSUMDB: "off"
run: go install github.com/mvanhorn/cli-printing-press/v4/cmd/printing-press@v4.8.0
- name: Sync engine into vendored/
run: bash mcp/scripts/sync-engine.sh
@@ -124,19 +107,30 @@ jobs:
./cmd/last30days-pp-mcp
- name: Bundle .mcpb
# printing-press bundle reads manifest.json from the cli dir and
# rewrites the binary into bin/<entry_point> inside the zip. The
# --platform tag drives the output filename suffix; the binary
# itself is whatever we just cross-compiled.
# Keep this equivalent to printing-press's bundle layout without
# downloading a separate packager: manifest.json at the ZIP root and
# the executable at its declared server.entry_point.
run: |
printing-press bundle mcp \
--skip-build \
--binary mcp/build/last30days-pp-mcp \
--platform "${MCPB_PLATFORM}" \
--output "${MCPB_OUTPUT}"
set -euo pipefail
entry_point="$(jq -er '.server.entry_point' mcp/manifest.json)"
test "${entry_point}" = "bin/last30days-pp-mcp"
staging="${RUNNER_TEMP}/last30days-mcpb"
output="${GITHUB_WORKSPACE}/${MCPB_OUTPUT}"
mkdir -p "${staging}/bin" "$(dirname "${output}")"
cp mcp/manifest.json "${staging}/manifest.json"
cp mcp/build/last30days-pp-mcp "${staging}/${entry_point}"
chmod 0755 "${staging}/${entry_point}"
(
cd "${staging}"
zip -q -X "${output}" manifest.json "${entry_point}"
)
unzip -Z1 "${output}" | grep -Fxq "manifest.json"
unzip -Z1 "${output}" | grep -Fxq "${entry_point}"
- name: Attest .mcpb artifact provenance
uses: actions/attest@a1948c3f048ba23858d222213b7c278aabede763 # v4.1.1
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4.2.2
with:
subject-path: ${{ env.MCPB_OUTPUT }}
@@ -158,7 +152,7 @@ jobs:
# checkout with the tag present; without it the step fails with
# "fatal: not a git repository".
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ env.RELEASE_TAG }}
fetch-depth: 0
+3 -3
View File
@@ -41,12 +41,12 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Run OpenSSF Scorecard
uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3
uses: ossf/scorecard-action@2d1146689b8cda280b9bc96326124645441f03bc # v2.4.4
with:
results_file: scorecard.sarif
results_format: sarif
@@ -64,6 +64,6 @@ jobs:
retention-days: 5
- name: Upload SARIF to code-scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
with:
sarif_file: scorecard.sarif
+6 -6
View File
@@ -17,12 +17,12 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
# Block known vulnerabilities in the locked Python dependency graph.
- name: Run uv audit against locked dependencies
@@ -36,7 +36,7 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -50,7 +50,7 @@ jobs:
contents: read
steps:
- name: Checkout full history for diff-aware scanning
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
@@ -59,7 +59,7 @@ jobs:
# verified secrets. Keep output limited to verified findings to avoid noisy
# unverified annotations.
- name: Run TruffleHog OSS secret scan
uses: trufflesecurity/trufflehog@00155c9dc586f34d189adc83d3ac2698c2ec551f # v3.95.8
uses: trufflesecurity/trufflehog@6f3c981e7b77f235fd2702dd74af25fc4b72bf11 # v3.96.0
with:
version: 3.95.5
extra_args: --results=verified
@@ -73,7 +73,7 @@ jobs:
image: semgrep/semgrep@sha256:06938c1f365d3f67b8cedd8bc117607ae64253f88a0e768e9da9408548927dd6 # v1.167.0
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
+9 -4
View File
@@ -14,15 +14,19 @@ jobs:
tag:
runs-on: ubuntu-latest
# Only act on the release-prep commit shape produced by prepare-release.yml
# (or an equivalent manual chore(release) commit).
if: startsWith(github.event.head_commit.message, 'chore(release): bump version to ')
# (or an equivalent manual chore(release) commit). Quote the expression —
# a bare `chore(release):` colon is invalid YAML and fails the whole
# workflow before any job runs. Use contains (not startsWith) so merge
# commits whose subject is "Merge pull request #N …" still match when the
# PR title is in the body.
if: "contains(github.event.head_commit.message, 'chore(release): bump version to ')"
permissions:
contents: write
actions: write
pull-requests: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
@@ -36,7 +40,8 @@ jobs:
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
run: |
set -euo pipefail
VERSION="$(printf '%s\n' "${HEAD_MSG}" | head -n1 | sed -n 's/^chore(release): bump version to \([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/p')"
# Scan every line — merge commits put the chore(release) title in the body.
VERSION="$(printf '%s\n' "${HEAD_MSG}" | sed -n 's/^chore(release): bump version to \([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/p' | head -n1)"
if [ -z "${VERSION}" ]; then
echo "Could not parse version from commit message: ${HEAD_MSG}"
exit 1
+6 -6
View File
@@ -15,12 +15,12 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- name: Set up Python
run: uv python install 3.12
@@ -34,12 +34,12 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- name: Set up Python
run: uv python install 3.12
@@ -53,12 +53,12 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: "1.25.5"
cache-dependency-path: mcp/go.sum
+2 -2
View File
@@ -15,9 +15,9 @@ jobs:
security-events: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Run zizmor 🌈
uses: zizmorcore/zizmor-action@192e21d79ab29983730a13d1382995c2307fbcaa # v0.5.7
uses: zizmorcore/zizmor-action@6fc4b006235f201fdab3722e17240ab420d580e5 # v0.6.1
+1 -1
View File
@@ -9,7 +9,7 @@
{
"name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.18.2",
"version": "3.21.0",
"category": "productivity",
"source": {
"source": "url",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days",
"version": "3.18.2",
"version": "3.21.0",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": {
"name": "Matt Van Horn",
+4
View File
@@ -68,6 +68,10 @@ Agents open most PRs. Follow this so `CHANGELOG.md` stops conflicting and versio
- Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
- Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
## Maintaining README translations
`README.md` is the canonical English README. When changing it, reflect the same substantive updates in `README.fr.md`, `README.de.md`, `README.es.md`, `README.pt-BR.md`, `README.ja.md`, and `README.zh-CN.md`, preserving commands, links, tables, and reciprocal language navigation.
## Maintaining CONFIGURATION.md
`CONFIGURATION.md` is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from `SKILL.md` (the canonical runtime spec).
+80
View File
@@ -9,6 +9,86 @@ This project uses [towncrier](https://towncrier.readthedocs.io/). Upcoming notes
<!-- towncrier release notes start -->
## [3.21.0] - 2026-08-14
### Changed
- X backend priority changed: bird (browser cookies) is now first in the auto chain, ahead of xai/xurl/xquik. Cookies beat XAI_API_KEY when both are present. Grok CLI is demoted to opt-in only: a leftover `~/.grok/auth.json` no longer steals the X lane. Pin `LAST30DAYS_X_BACKEND=grok` to enable it explicitly.
## [3.20.0] - 2026-08-14
### Added
- X search now judges corpus on-topic ratio and retries once with a wider AND query when the initial results are mostly off-topic (e.g., phrase-quoted "Rome Italy" returning AS Roma sports accounts). Multi-word search queries use unquoted AND as the primary variant instead of phrase-quoting. Handles extracted from entity_extract are now filtered for the from: lane based on whether their already-retrieved posts are on-topic (≥2 on-topic hits and ≥50% ratio), not just frequency. Extracted handles that qualify for the from: lane AND the topic into the query (`from:handle Rome`) to prevent off-topic timelines from filling the X budget. Explicit --x-handle and --x-related handles always get the from: lane without ANDing the topic. Source status reflects off-topic floods as a warning artifact, not a failure. First-party floor immunity remains conservative (explicit handles only, not promoted commentators).
### Fixed
- Amazon review enrichment now starts at search time instead of after all other sources finish, ensuring multi-source runs have a useful budget (up to 180s) rather than leftover crumbs. Previously, a run that spent 269s on retrieval would leave only 11s for reviews, causing all Bright Data pulls to time out. Budgets below 90s now skip the lane entirely instead of firing doomed short pulls that spend credits without returning reviews.
- Grok session expiry is now detected locally by parsing `expires_at` from `~/.grok/auth.json`. Doctor reports expired sessions as **degraded** (not ok) with the expiry timestamp and a hint to run `grok login --device-auth` if refresh fails. Research-time availability still attempts grok when credentials exist (expired access_token does not prove the refresh_token is dead). When the Grok CLI returns "Not signed in" or `invalid_grant` mid-run, the pipeline now reports `auth-failed` with a proper fix hint instead of a generic PARTIAL outcome, and falls back to the next X backend.
## [3.19.0] - 2026-08-14
### Security
- Source URLs containing unsafe schemes or Markdown delimiters are now rendered as inert escaped text instead of raw Markdown. ([#886](https://github.com/mvanhorn/last30days-skill/issues/886))
- SessionStart `check-config.sh` now rejects non-identifier `.env` keys before `printf -v` (blocking array-subscript command substitution) and loads `.claude/last30days.env` only when `LAST30DAYS_TRUST_PROJECT_CONFIG` is set in the process environment or global config, matching `lib/env.py`.
### Added
- **Amazon buyer signals** — a new opt-in `amazon` source, backed by the Bright Data CLI. On shopping-intent topics it pulls discovered products with live ratings and prices, plus a capped sample of recent written reviews woven in as buyer voice.
The signal it exists for is *drift*: an all-time rating from thousands of ratings set against the average of only the reviews inside the last 30 days. When those disagree, something changed this month, and the review text says what. The emoji footer names each product and the direction it moved — `📦 Amazon: 3 products │ Chill Max XL 4.4★→3.8★ ↓, Deluxe Bag 4.7★→5.0★, BLUEY Set 4.8★ new` — rather than reporting inventory counts.
Off by default and dual-gated: the `brightdata` CLI must be on PATH and logged in, *and* the run must ask for the source (`--search ...,amazon` or `INCLUDE_SOURCES=amazon`). It never auto-fires from inferred intent. Use `--amazon-query` when the product keyword differs from the topic — a person topic searches their company's product line, not their name. `LAST30DAYS_AMAZON_DOMAIN` selects a non-US marketplace.
Billing is one credit per request against a 5,000/month free tier, so a typical run costs 4 credits regardless of how many reviews come back.
- Reddit keyless discovery now falls back to the arctic-shift archive when the shreddit listing partials return nothing — hosts on datacenter egress (where Reddit 403s `/svc/shreddit`) keep scored Reddit discovery, score backfill, and discover-mode listings instead of reporting `auth-failed`.
- X search now works with no X credential at all. Install the Grok CLI (`curl -fsSL https://x.ai/cli/install.sh | bash`, then `grok login`) and last30days reaches X with no X account, no browser cookies, and no `XAI_API_KEY` — on any host, including Claude Code, Codex, Cursor and GrokBot. It sits ahead of the browser-cookie path by default; pin `LAST30DAYS_X_BACKEND=bird` to keep cookies. Covers three lanes for a person or company topic: posts by the subject, posts @-mentioning them, and posts naming them in plain text (which is most of the discussion, and which a mention-only search misses).
### Fixed
- Source URLs in the saved raw report and internal evidence output now render as clickable markdown links instead of plain text. ([#886](https://github.com/mvanhorn/last30days-skill/issues/886))
- Hacker News comments no longer vanish from every per-source path. HN comments arrive as `{author, text, points}` while downstream readers key on `score`/`excerpt`, and `_normalize_hackernews` stored them raw, so `render._top_comments_list` filtered `(c.get("score") or 0) >= 5` against a key that was never present and rejected the entire source. HN comments are now remapped like the YouTube and TikTok ones, and the HN floor is 0 because the Algolia API returns `points: null` for every comment child, which makes any positive threshold unmeetable. A comment with no vote signal renders without a fabricated "(0 points)". ([#889](https://github.com/mvanhorn/last30days-skill/issues/889))
- Polymarket topics spelled out in full ("artificial general intelligence") now match markets titled in shorthand ("AGI by 2030?"). Previously the topic filter and the relevance floor both compared full words against an acronym, so every on-topic market was dropped and the run reported zero results — indistinguishable from the source genuinely having none. ([#891](https://github.com/mvanhorn/last30days-skill/issues/891))
- Reddit HTTP 429/403 on the keyless lanes is no longer reported as a clean `no-results`: the failure now survives the worker-thread hop into the run outcome, so `source_status` carries `rate-limited`/`auth-failed` with the status in `detail` and `doctor --postmortem` lists Reddit under Failed instead of "No failures on the last run." `doctor --probe` now checks the RSS endpoint the engine actually uses (the old `/r/all/hot.json` probe is permanently 403 keyless) and counts a 403/429 there as blocked rather than reachable. Under `LAST30DAYS_STRICT_EXIT` a blocked Reddit run now exits 3 instead of 0. ([#899](https://github.com/mvanhorn/last30days-skill/issues/899))
- On Windows, the setup wizard's npx-based installs (Digg, arXiv, Techmeme) always failed silently because `shutil.which("npx")` resolves `PATHEXT` but `subprocess.run` given the bare string `"npx"` does not. Windows users also got macOS-only Homebrew guidance when yt-dlp was missing. Both are fixed: the resolved npx path is now passed through, and Windows gets `pip install yt-dlp` guidance instead. ([#904](https://github.com/mvanhorn/last30days-skill/issues/904))
- `--web-backend=keyless` is now accepted by the CLI, matching what `CONFIGURATION.md` already documented. The keyless web-search floor was already fully supported internally; only the argument parser rejected the value. ([#905](https://github.com/mvanhorn/last30days-skill/issues/905))
- arXiv no longer returns zero results for natural-language multi-word topics. The exact-phrase quoted query now retries unquoted once when it matches nothing, instead of silently dropping arXiv from the report. ([#908](https://github.com/mvanhorn/last30days-skill/issues/908))
- Truth Social search no longer fails with a Cloudflare-triggered HTTP 403 on every request. Requests now send browser-like headers, the same fix already applied to Reddit. ([#909](https://github.com/mvanhorn/last30days-skill/issues/909))
- `--emit=compact --save-dir` runs now save the complete debug artifact (all clusters plus every per-source item, with the emoji footer citing the actual written path) instead of the compact stdout render, which had made most collected evidence unrecoverable from the raw file. ([#923](https://github.com/mvanhorn/last30days-skill/issues/923))
- The GitHub source no longer reports zero results when the planner writes search qualifiers into the topic (e.g. `open source AI stars:>1000 created:>2025-03-20`). `search_github` appends its own `created:>{from_date}` window, and two `created:` qualifiers collide: GitHub honors the first and ignores the appended window, so out-of-window items are fetched and then dropped wholesale by the local date filter, surfacing as a silent `no-results`. Qualifiers are now stripped from the topic before the query is built (including comma/semicolon-glued forms and quoted values such as `label:"bug fix"`), topic terms glued after a qualifier value are preserved, and a qualifier-only topic reports an explicit error instead of searching the whole site. ([#949](https://github.com/mvanhorn/last30days-skill/issues/949))
- The GitHub source no longer returns zero results once a credential is available. GitHub rejects authenticated `/search/issues` requests that carry neither `is:issue` nor `is:pull-request` with HTTP 422, while anonymous requests are still accepted without one — so the source worked until a user ran `gh auth login` or set `GITHUB_TOKEN`, then failed silently while `doctor` still reported it healthy. Authenticated searches now run both qualifier-scoped queries and merge them, deduped by item id and re-sorted by reaction count, which keeps issues and pull requests in the same result set; appending a single qualifier would have dropped roughly 87% of matches on a typical topic. The unauthenticated path is unchanged. When one partition fails but the other returns items, the surviving items are now kept and the source is reported as partial rather than silently claiming success — full-failure (both partitions return nothing) is still a clear failure. ([#967](https://github.com/mvanhorn/last30days-skill/issues/967))
- Out-of-window evidence no longer leads the ranked output. Items whose dates fall outside the run's window were flagged `[date:low]` but ranked normally, so a 2025-10 video took the #1 cluster in a 2026-07 brief and a 2025-12 one took #5. Candidates whose every dated item is out of window are now demoted in both the fusion sort and `_final_score`, so they still appear as evidence but never above in-window material; items with no date at all are untouched, since an unknown date is a coverage gap rather than a stale item. The freshness verdict ("only N of M dated items are from the last 7 days") also reaches the pass-through footer instead of only the report body.
- X runs on a person or company no longer discard the subject's own posts. A post almost never contains its own author's name, so lexical relevance scored it at zero and the retrieval floor pruned it — a run for "Peter Steinberger steipete" fetched 8 posts by him and reported none of them. Fixed across the chain: planner scaffolding words no longer count as topic signal, posts by a handle the run is searching are exempt from the floor, auto-discovered handles now reach the first-party protections (previously only `--x-handle` did), quoted proper-noun phrases survive into the provider query instead of degrading into a token conjunction, and the subject of the topic gets a higher per-author cap than incidental accounts. When no real handle can be identified at all, the X floor is skipped rather than pruning against lexical name tokens. The thin-source retry path defers the X floor the same way Phase 1 does, so a subject-authored post recovered on retry is not discarded before handle resolution.
- `--github-user` no longer returns unrelated repos for people whose PR search comes back empty or is unavailable. Person mode now falls back to the selected user's public GitHub events and returns only in-window `PushEvent` activity attributed to that actor, instead of treating repository-level `pushed_at` as proof that the selected user pushed. A pinned `--github-user` that still yields nothing is recorded as `no-results` instead of passing silently.
## [3.18.4] - 2026-07-28
### Fixed
- Tag release workflow YAML now parses on every main push; merge commits can mint `vX.Y.Z` tags again. ([#880](https://github.com/mvanhorn/last30days-skill/issues/880))
- YouTube yt-dlp search under comparison-mode fan-out no longer self-throttles into 120s timeouts: concurrent yt-dlp invocations are process-wide capped, identical searches are deduped within a run, and a search timeout is recorded as `timeout` rather than `no-results`. `LAST30DAYS_YT_SEARCH_TIMEOUT` configures the search deadline.
## [3.18.3] - 2026-07-25
### Fixed
- Top Community Comments ranking blends thread relevance with vote strength and applies the relevance floor only when enough on-topic candidates exist. ([#701](https://github.com/mvanhorn/last30days-skill/issues/701))
- Chromium cookie extraction now searches every browser profile for a matching cookie set, and reuses the Keychain/AES key across profiles in one scan. ([#725](https://github.com/mvanhorn/last30days-skill/issues/725))
- Synthesis contract is echoed at the top of the evidence envelope so hosts that truncate stdout still see the "synthesize, don't dump" directive. ([#727](https://github.com/mvanhorn/last30days-skill/issues/727))
- `store_findings` no longer raises `TypeError` when a re-sighted finding carries `engagement_score: None`. ([#796](https://github.com/mvanhorn/last30days-skill/issues/796))
- Explicit --plan payloads with an invalid schema now exit with a field-specific error instead of silently running a deterministic plan. ([#841](https://github.com/mvanhorn/last30days-skill/issues/841))
- Company-topic runs no longer auto-add the `jobs` source when an explicit `--search` / `requested_sources` filter is set; `--hiring-signals` still forces jobs. ([#842](https://github.com/mvanhorn/last30days-skill/issues/842))
- Report footer path now matches the collision-safe path actually reserved by save_output. ([#850](https://github.com/mvanhorn/last30days-skill/issues/850))
- YouTube ScrapeCreators transcript rescue is logged instead of being masked as a hard failure. ([#851](https://github.com/mvanhorn/last30days-skill/issues/851))
- Polymarket domain-sweep topics no longer drop every market after noise-word stripping removes terms like "ai". ([#859](https://github.com/mvanhorn/last30days-skill/issues/859))
- Reddit fetch windows now track the requested date range so short `--days` runs no longer pull a depth-default month and discard everything outside the window. ([#860](https://github.com/mvanhorn/last30days-skill/issues/860))
## [3.18.2] - 2026-07-25
### Added
+13 -4
View File
@@ -103,10 +103,14 @@ The skill reads keys from a `.env` file. Two locations are supported:
Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_CONFIG_DIR=""` for no-config mode). File permissions should be `600` on POSIX hosts - the engine warns on every run if they aren't.
The project-scoped file is useful for **intentional per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), then opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or `~/.config/last30days/.env`. Folder-mode hosts such as Codex desktop do not trust hidden project config by default, and discovery stops at the git root so unrelated parent folders cannot silently influence runs.
The project-scoped file is useful for **intentional per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), then opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or `~/.config/last30days/.env`. Folder-mode hosts such as Codex desktop do not trust hidden project config by default, and discovery stops at the git root so unrelated parent folders cannot silently influence runs. The SessionStart status hook (`hooks/scripts/check-config.sh`) uses the same trust gate — an untrusted repo's `.claude/last30days.env` is not read at session start.
**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. A configured local corpus is the privacy exception: the engine bypasses the hosted backend and runs locally rather than forwarding file-derived input. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.
**`BRIGHTDATA_API_KEY`** - optional, for the `amazon` source. The Bright Data CLI normally owns its own auth via `brightdata login`, so this is only needed if you would rather keep an explicit key in `.env` or the keychain. It is resolved through the standard config layering and passed to the CLI through the child process environment, never on the command line (where it would be readable from `/proc/<pid>/cmdline` by other local users on a shared host).
**`LAST30DAYS_AMAZON_DOMAIN`** - optional, default `https://www.amazon.com`. The marketplace the `amazon` source searches; set it to `https://www.amazon.co.uk`, `https://www.amazon.de`, and so on. Product URLs are validated against this host, so records from other marketplaces are rejected.
### Local corpus (your files)
Register persistent directories with `LAST30DAYS_CORPUS_DIRS`. Separate paths with `:` on macOS/Linux (the platform path separator is `;` on Windows):
@@ -145,7 +149,8 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
| arXiv | `arxiv-pp-cli` on PATH (auto-installed during first-run setup via `npx -y @mvanhorn/printing-press-library@0.1.16 install arxiv --cli-only`) | always on if `arxiv-pp-cli` on PATH; fires on research/technical topics and stays quiet otherwise (relevance + 365-day recency gating) | yes (free, keyless) |
| Techmeme | `techmeme-pp-cli` on PATH (auto-installed via `... install techmeme --cli-only`) | always on if `techmeme-pp-cli` on PATH; searches Techmeme's live archive and keeps only headlines dated within the research window (undated headlines flow through as low-confidence) | yes (free, keyless) |
| Trustpilot | `trustpilot-pp-cli` on PATH (NOT auto-installed; install on demand via `npx -y @mvanhorn/printing-press-library@0.1.16 install trustpilot --cli-only`) + (`INCLUDE_SOURCES` contains `trustpilot` **or** an explicit `--trustpilot-domain` / plan-level `trustpilot_domain`) | **opt-in, off by default**; `--trustpilot-domain=<domain>` (and per-entity `trustpilot_domain` in `--competitors-plan`) auto-activates the source for that run and bypasses the brand-shape gate. Persist with `INCLUDE_SOURCES=trustpilot` to skip per-run auto-enable. `EXCLUDE_SOURCES=trustpilot` still wins. Bare company names auto-resolve to the review-page domain via the CLI's search only when the source is already active. The session warms once before the search fan-out; a stale session does a ~10s headless-Chrome WAF-cookie harvest (set `LAST30DAYS_TRUSTPILOT_NO_BROWSER=1` to disable in cron/CI) | yes (no API key; cookie-replay after the one-time harvest) |
| X / Twitter | one of: `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `XQUIK_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | cookie-jar / Bird = free; Xquik / xAI / ScrapeCreators = key-based |
| Amazon | `brightdata` CLI on PATH **and logged in** (NOT auto-installed: `npm i -g @brightdata/cli` then `brightdata login`) + (`INCLUDE_SOURCES` contains `amazon` **or** `--search` includes `amazon`) | product records with live rating, rating count, and price, plus a capped sample of recent written reviews woven as buyer voice; the emoji footer shows each product's all-time-vs-last-30-days drift | **opt-in, off by default**. Free tier is 5,000 requests/month (~$7.50 equivalent); a typical run spends 4 (1 product search + up to 3 review pulls) regardless of how many reviews come back, since billing is per request. Past the free tier it bills the account balance at $1.50 per 1,000 records (~$0.32 for a default run). `--amazon-query=<keyword>` sets the product keyword when it differs from the topic; `LAST30DAYS_AMAZON_DOMAIN` selects a non-US marketplace. `EXCLUDE_SOURCES=amazon` wins. Never auto-fires: the model requests it per run or the user enables it durably |
| X / Twitter | one of: a signed-in `grok` CLI (no X credential), `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `XQUIK_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | grok = no X credential, draws on your Grok plan; cookie-jar / Bird = free; Xquik / xAI / ScrapeCreators = key-based |
| TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls |
| Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |
| Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls |
@@ -160,7 +165,11 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
| Jobs / careers pages | none for public ATS pages; web backend improves fallback discovery | `--hiring-signals` and strong Hiring Signals in standard company reports | yes |
| Apify (alternate scraper) | `APIFY_API_TOKEN` | fallback for Reddit/TikTok/Instagram when ScrapeCreators is exhausted | yes (limited) |
**YouTube transcript tuning.** `LAST30DAYS_YT_SUB_LANGS` controls the comma-separated caption-language priority passed to yt-dlp and defaults to `en,es,pt`. When `SCRAPECREATORS_API_KEY` is available, yt-dlp uses one fast attempt before the paid fallback; set `LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT` to the number of seconds allowed for that attempt when a throttled host needs longer than the 12-second default. A VTT completed before the timeout is reused rather than discarded.
**YouTube transcript tuning.** `LAST30DAYS_YT_SUB_LANGS` controls the comma-separated caption-language priority passed to yt-dlp and defaults to `en,es,pt`. When `SCRAPECREATORS_API_KEY` is available, yt-dlp uses one fast attempt before the paid fallback; set `LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT` to the number of seconds allowed for that attempt when a throttled host needs longer than the 12-second default. A VTT completed before the timeout is reused rather than discarded. `LAST30DAYS_YT_SEARCH_TIMEOUT` sets the per-search yt-dlp deadline (default 120s). Comparison-mode fan-out also caps concurrent yt-dlp processes process-wide and caches identical searches within a run so redundant `ytsearch` calls do not self-throttle the same IP.
**X backend priority (bird first).** The default X backend chain is bird (browser cookies) → xai (API key) → xurl (OAuth2 CLI) → xquik (API key). Cookies beat `XAI_API_KEY` when both are present. A leftover grok login never steals the X lane; see below.
**Grok CLI (opt-in backup).** Install the Grok CLI (`curl -fsSL https://x.ai/cli/install.sh | bash`) and run `grok login`, and X can work with no X account, no browser cookies, and no `XAI_API_KEY`. However, grok is **opt-in only**: a leftover `~/.grok/auth.json` must never steal the X lane. Pin `LAST30DAYS_X_BACKEND=grok` to enable it. It is not "free" in the way the cookie path is: calls draw on your Grok plan, and depth costs several calls per run because the underlying tool caps each search at 10 posts. Results are validated before use — every returned post's ID is decoded to confirm it falls inside the requested date range, because the retrieval is performed by a language model and can otherwise return confident, well-formed posts that were never searched for.
**X on cookie-less hosts.** Bird (the free X source) scrapes X using your logged-in browser cookies (`AUTH_TOKEN`/`CT0`), which agent hosts like OpenClaw, CI, or headless runs often can't supply — and scraping carries some account risk. On those, set `XQUIK_API_KEY` (or `XAI_API_KEY`) for full, ranked X coverage from a single API key: the same engagement-based ranking, first-party authorship, and handle (from/mentions) lanes the native X source gets. `--diagnose` reports whether the key is working (and flags an unpaid key).
@@ -414,7 +423,7 @@ Every live run writes its JSON result to `~/.config/last30days/doctor-cache.json
| --- | --- |
| `LAST30DAYS_DOCTOR_TTL` | Freshness window for `doctor --cached`, in **seconds**. Defaults to `900` (15 minutes). `0` makes every `--cached` call run live. |
| `LAST30DAYS_DOCTOR_PROBE_TIMEOUT` | Per-source deadline (**seconds**) for `doctor --probe` live checks. Defaults to `10`. Caps each concurrent probe so a slow source cannot hang the command. |
| `LAST30DAYS_X_BACKEND` | Pins the X backend (`xai` / `bird` / `xurl` / `xquik`); doctor renders the pin and predicts "will use" accordingly. |
| `LAST30DAYS_X_BACKEND` | Pins the X backend (`bird` / `xai` / `xurl` / `xquik` / `grok`); doctor renders the pin and predicts "will use" accordingly. The unpinned auto chain is bird → xai → xurl → xquik (grok is opt-in only). Pin `grok` to enable it; a leftover `~/.grok/auth.json` is never auto-selected. |
| `LAST30DAYS_REDDIT_BACKEND` | `scrapecreators` makes ScrapeCreators the primary Reddit backend; doctor renders Reddit's conditional routing with the pin applied. |
| `LAST30DAYS_REDDIT_SC_MIN_ITEMS` | Integer thinness floor for ScrapeCreators Reddit **search** backfill. Default `0` = empty-only (free path keeps any non-empty result; no credit spend). Set above `0` to backfill when free yield is below that count; merged results dedupe by post id. Requires `SCRAPECREATORS_API_KEY`. Ignored when `LAST30DAYS_REDDIT_BACKEND=scrapecreators` (SC is already primary). |
+383
View File
@@ -0,0 +1,383 @@
# /last30days
[English](README.md) | [Français](README.fr.md) | Deutsch | [Español](README.es.md) | [Português (Brasil)](README.pt-BR.md) | [日本語](README.ja.md) | [简体中文](README.zh-CN.md)
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days - an AI agent-led search engine that searches people, not editors" />
</p>
<p align="center">
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
**Eine von einem KI-Agenten gesteuerte Suchmaschine, die nach Upvotes, Likes und echtem Geld gewichtet nicht nach Redaktionen.**
Dieses README beschreibt die aktuelle v3-Pipeline. Die Laufzeitspezifikation der Skill liegt in [skills/last30days/SKILL.md](skills/last30days/SKILL.md) und ist maßgeblich für das aktuelle Verhalten von Befehlen und Setup.
**Claude Code (empfohlen automatische Updates über den Marketplace):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex, Cursor, Copilot, Gemini CLI oder einer von 50+ [Agent Skills](https://agentskills.io)-Hosts:**
```
npx skills add mvanhorn/last30days-skill -g
```
(`-g` installiert global für deinen Benutzer, also in allen Projekten verfügbar. Lass das Flag weg, wenn du die Installation auf ein Projekt beschränken willst.)
Weitere Installationswege (claude.ai im Browser, OpenClaw, manuell) findest du unten im Abschnitt [Installation](#installation).
Null Konfiguration. Reddit, HN, Polymarket und GitHub funktionieren sofort. Führe die Skill einmal aus, und der Setup-Assistent schaltet X, YouTube, TikTok, arXiv, Techmeme und mehr in 30 Sekunden frei.
---
Upvotes von Reddit. Likes von X. YouTube-Transkripte. TikTok-Engagement. Polymarket-Quoten, gedeckt durch echtes Geld und Insiderwissen. Das sind Millionen Menschen, die jeden Tag mit ihrer Aufmerksamkeit und ihrem Geldbeutel abstimmen. /last30days durchsucht all das parallel, gewichtet nach dem, womit echte Menschen tatsächlich interagieren, und ein KI-Agent fasst es als Juror zu einem einzigen Briefing zusammen.
Google aggregiert Redaktionen. /last30days durchsucht Menschen.
Diese Suche bekommst du nirgendwo sonst, weil keine einzelne KI Zugriff auf alles hat. Google erfasst weder Reddit-Kommentare noch X-Beiträge. ChatGPT hat einen Deal mit Reddit, kann aber weder X noch TikTok durchsuchen. Gemini hat YouTube, aber kein Reddit. Claude hat nichts davon nativ. Jede Plattform ist ein abgeschotteter Garten mit eigener API, eigenen Tokens, eigener Authentifizierung. Aber du kannst deine eigenen Schlüssel und Browser-Sessions mitbringen und plötzlich durchsucht ein KI-Agent alle gleichzeitig, wägt sie gegeneinander ab und sagt dir, was wirklich zählt.
Das ist der eigentliche Durchbruch. Keine bessere Suchmaschine, sondern ein Dutzend getrennter Plattformen, die ein Agent miteinander verbindet.
```
/last30days Peter Steinberger
```
Du hast morgen ein Meeting. Du googelst die Person. Du bekommst ihr LinkedIn-Profil von 2023. /last30days zeigt dir, was sie diesen Monat wirklich macht: bei OpenAI eingestiegen, um an Codex zu arbeiten, kämpft gegen Anthropics Verbot von Drittanbieter-Agenten, hat 23 PRs mit 85 % Merge-Rate geliefert, baut „LobsterOS“ für geräteübergreifende Agentensteuerung und ein Thread in r/ClaudeCode kam auf 569 Upvotes bei der Frage, ob sie ein Held oder „unerträglich“ ist. Verteilt über X-Beiträge, Reddit-Threads, YouTube-Transkripte und GitHub-Commits. Nichts davon stand bei Google.
## Warum es das gibt
Ich habe es gebaut, um bei KI Schritt zu halten. Alles ändert sich täglich, und die Nerds auf Reddit und X wissen es immer zuerst. Ich brauchte bessere Prompts, und die Trainingsdaten lagen immer Monate hinter dem, was die Community längst herausgefunden hatte.
Daraus wurde etwas Größeres. Heute lasse ich es vor einem Sales-Call laufen, um die Wahrheit der letzten 30 Tage über ein Unternehmen zu kennen. Vor einem Meeting, um die aktuellen Tweets und Podcast-Transkripte meines Gegenübers zu lesen. Vor einer Reise nach Disney World, um zu wissen, welche Attraktionen geschlossen sind und was die Community über Genie+ sagt. Bevor ich irgendetwas baue, um zu wissen, an welchen Problemen die Leute wirklich hängen.
Wenn du dich mit einem CEO triffst: Hast du alle Tweets und YouTube-Transkripte der letzten 30 Tage gelesen? Ich schon.
## Quellen, gewichtet von den Menschen
| Quelle | Was dir die Menschen sagen |
|--------|--------------------------|
| **Reddit** | Die ungefilterte Meinung. Top-Kommentare mit echten Upvote-Zahlen, kostenlos, ohne API-Schlüssel. Die echten Meinungen, die Google vergräbt. |
| **X / Twitter** | Die spontane Einschätzung, der Experten-Thread, die erste Reaktion auf eine Eilmeldung. Zuerst informiert, zuerst am Streiten. |
| **YouTube** | Die 45-minütige Tiefenanalyse. Vollständige Transkripte, durchsucht nach den 5 zitierfähigen Sätzen, auf die es ankommt. |
| **TikTok** | Der Creator, der 3,6 Millionen Menschen mit einer Sichtweise erreicht, die du bei Google nie findest. |
| **Instagram Reels** | Die Perspektive der Influencer, inklusive Transkript des Gesprochenen. Das Signal der visuellen Kultur. |
| **Hacker News** | Der Konsens der Entwickler. 825 Punkte, 899 Kommentare. Wo technische Leute wirklich streiten. |
| **Polymarket** | Keine Meinungen. Quoten. Gedeckt durch echtes Geld. 96 % Wahrscheinlichkeit bei Albumverkäufen. 4 % bei einer Übernahme. |
| **GitHub** | Für Personen: PR-Tempo, Top-Repos nach Sternen, Release Notes. Für Themen: Issues und Discussions. |
| **Digg** | Kuratierte Story-Cluster aus Diggs AI-1000-Leaderboard (rund 1000 KI-Accounts mit hohem Signal auf X), mit zuordenbaren Inline-Zitaten und ganz ohne X-Authentifizierung. Wird automatisch aktiv, sobald `digg-pp-cli` im PATH liegt. |
| **arXiv** | Die Fachartikel hinter dem Hype. Neue Forschung im Zeitfenster, kostenlos, ohne API-Schlüssel. Wird automatisch aktiv, sobald `arxiv-pp-cli` im PATH liegt (das Erst-Setup installiert es). |
| **Techmeme** | Die redaktionelle Ebene der Tech-News, begrenzt auf dein 30-Tage-Fenster. Kostenlos, ohne API-Schlüssel. Wird automatisch aktiv, sobald `techmeme-pp-cli` im PATH liegt (das Erst-Setup installiert es). |
| **LinkedIn** | Das berufliche Signal. Beiträge und Artikel, wobei Artikel als starkes Signal gewichtet werden. |
| **StockTwits** | Die Stimmung der Trader. Aktiviert sich automatisch, wenn dein Thema ein Ticker oder eine Kryptowährung ist. |
| **Threads** | Die Textebene nach Twitter. Gespräche von Creators und Marken. |
| **Pinterest** | Visuelle Entdeckung. Pins, gespeicherte Beiträge und Kommentare zu Produkten und Ideen. |
| **Xiaohongshu (RED)** | Chinesische Signale zu Lifestyle, Produkten und Creators. Wird ausdrücklich mit `--search xhs` angefordert, wenn lokal ein eingeloggtes x-mcp-Browser-Plugin oder ein `xiaohongshu-mcp`-Dienst läuft. |
| **Bluesky** | Die dezentrale soziale Ebene. AT-Protocol-Beiträge aus der Abwanderung nach Twitter. |
| **Perplexity** | Belegte Sonar-Synthese, Rohtreffer der Search API und Deep Research. |
| **Web** | Die redaktionelle Berichterstattung, die Blog-Vergleiche. Ein Signal von vielen, nicht das einzige. |
Die Community steuert laufend weitere bei. Truth Social und andere Nischenquellen stecken bereits in der Engine, weitere folgen.
Ein Reddit-Thread mit 1.500 Upvotes ist ein stärkeres Signal als ein Blogbeitrag, den niemand gelesen hat. Ein TikTok mit 3,6 Millionen Aufrufen sagt mehr darüber aus, was kulturell relevant ist, als jede Pressemitteilung. Polymarket-Quoten mit 66.000 $ Handelsvolumen dahinter lassen sich schwerer wegdiskutieren als die Vermutung eines Kommentators.
Die Synthese sortiert nach dem, womit echte Menschen tatsächlich interagiert haben. Soziale Relevanz, nicht SEO-Relevanz.
## Wofür die Leute es wirklich nutzen
**Vor einem Meeting.** `/last30days Peter Steinberger` beim Codex-Team von OpenAI eingestiegen, kämpft gegen Anthropics Verbot von Drittanbieter-Agenten, 23 PRs mit 85 % Merge-Rate auf GitHub gemergt, baut LobsterOS für geräteübergreifende Agentensteuerung. r/ClaudeCode: „Seit OpenClaw erschienen ist, war allgemein bekannt: Wer es über etwas anderes als die API laufen lässt, fliegt irgendwann raus“ (227 Upvotes). Das steht so nicht auf LinkedIn.
**Um Hiring-Signale zu lesen.** `/last30days Listen Labs --hiring-signals` aktuelle Stellenanzeigen und Karriereseiten werden zu zitierten Belegen für Schwerpunktverschiebungen: Einstellungen in Enterprise Security, Customer Success, Infrastruktur oder Produktausbau. Der Bericht sagt, was das Hiring zu signalisieren scheint, nicht was die Roadmap liefern wird.
**Um ein Thema vor seinem Höhepunkt zu finden.** Frag `/last30days what's exploding in AI agents?`, und die Skill wechselt in den Discovery-Modus: Die Engine durchkämmt Reddit-Kategorielisten, die Front- und Best-Stories von Hacker News, Diggs AI-1000-Feed und X, sofern du authentifiziert bist. Dein Agent bewertet die Vorschläge (Namen, Müllfilterung, inhaltliche Relevanz) und schreibt Podcast- und X-Artikel-Ansätze. Am Ende bekommst du 5 bis 10 nach Velocity sortierte Themen. Jedes Ergebnis enthält quellenübergreifende Zahlen, ein Momentum-Label und einen startklaren Folgebefehl `/last30days "<topic>"`.
**Wenn etwas erscheint.** `/last30days Kanye West` Großbritannien hat sein Visum blockiert, das Wireless Festival wurde abgesagt, die Sponsoren sind abgesprungen. Aber BULLY stieg auf Platz 2 der Billboard-Charts ein. Fantano kam aus seinem „Yay sabbatical“ zurück, um es zu rezensieren (653.000 Aufrufe). Beim SoFi Homecoming holte er Lauryn Hill und Travis Scott für 44 Songs auf die Bühne. Polymarket: „Wird Kanye wieder twittern?“ 86 % Ja. 23 Reddit-Threads, 17 YouTube-Videos, 86.000 Upvotes.
**Um Tools zu vergleichen.** `/last30days OpenClaw vs Hermes vs Paperclip` „Das sind keine Konkurrenten, das sind Schichten.“ OpenClaw ist die ausführende Ebene (351.000 GitHub-Sterne, produktiv), Hermes ist das sich selbst verbessernde Gehirn (31.000 Sterne), Paperclip ist das Organigramm (49.000 Sterne). Die Sternzahlen kommen live aus der GitHub-API, nicht aus veralteten Blogbeiträgen. Vergleichstabelle mit Architektur, Speicher, Sicherheit und idealem Einsatzzweck. Laut @IMJustinBrooke: „OpenClaw = Glumanda, Hermes = Glurak.“
**Um die Welt zu verstehen.** `/last30days Iran vs USA` Tag 38 des Krieges. Trumps Ultimatum bis Dienstag, damit der Iran die Straße von Hormus wieder öffnet. Zwei US-Kampfjets abgeschossen. Öl bei 126 $ pro Barrel. Die IEA nannte es „die größte Versorgungsstörung in der Geschichte des globalen Ölmarkts“. Polymarket: Waffenstillstand bis zum 31. Dezember bei 74 %. 27 X-Beiträge, 10 YouTube-Videos, 20 Prognosemärkte.
**Vor einer Reise.** `/last30days Universal Epic Universe` die Erweiterung ist bereits im Bau. Baugenehmigung „Project 680“ eingereicht. Eine Feuerwerksshow ist über die Infrastruktur belegt, aber noch nicht angekündigt. Wartezeiten: Mine-Cart Madness im Schnitt 148 Minuten. Noch keine Jahreskarte, und die Einheimischen sind genervt. Stardust Racers steht bis zum 5. April wegen Renovierung still.
**Um schnell etwas zu lernen.** `/last30days Nano Banana Pro prompting` JSON-strukturierte Prompts lösen den Tag-Wildwuchs ab. Das verschachtelte Format von @pictsbyai verhindert „Concept Bleeding“. Bearbeiten schlägt neu generieren. Und danach schreibt dir die Skill einen produktionsreifen Prompt, der genau das umsetzt, was die Community als funktionierend beschrieben hat.
## Was neu ist
Seit der Ankündigung von v3.3 im Mai und mit Stand v3.11.1 (Juli 2026): 175 gemergte PRs 122 davon von 52 Beitragenden aus der Community verteilt auf 15 Releases. Das ist gelandet.
### Erstklassig auf OpenAI Codex
/last30days ist jetzt ein natives Codex-Plugin mit geführtem Setup keine Portierung, sondern ein vollwertiger Bürger. Renderer-bewusste Zitate sorgen dafür, dass die Codex-Ausgabe sich wie ein Briefing liest und nicht wie eine URL-Suppe (#694), und dieselbe Engine läuft auf Claude Code, Cursor, Copilot, Gemini CLI, Claude Desktop, OpenClaw und 50+ Agent-Skills-Hosts. Codex-Plugin-Manifest von [@rfoust](https://github.com/rfoust) (#686), Codex-Auth-Fix von [@tmchow](https://github.com/tmchow) (#698).
### arXiv, Techmeme und Digg kostenlos, ohne API-Schlüssel
arXiv liefert die Fachartikel hinter dem Hype, Techmeme die redaktionelle Tech-News-Ebene kostenlos, ohne einen einzigen Schlüssel, und das Erst-Setup installiert ihre CLIs, sodass sie sich von selbst aktivieren (#709). Diggs AI-1000-Story-Cluster kommen genauso ohne X-Authentifizierung an: Das Setup installiert dir die kostenlose Digg-CLI (#590). Trustpilot ist optional zuschaltbar für Recherchen zu Consumer-Marken.
### Reddit gratis, mit echten Scores und Top-Kommentaren
Reddits öffentliche .json-API ist gestorben; der kostenlose Weg kam stärker zurück. Schlüsselloses RSS plus Shreddit-Scraping (#457), gezielte Subreddit-Suche mit echten Upvote-Zahlen über arctic-shift (#696) und eine Relevanzschwelle, damit ein viraler Off-Topic-Beitrag dein Briefing nicht kapert (#488, danke [@rzachsmith](https://github.com/rzachsmith)). Kein API-Schlüssel. Echte Scores. Top-Kommentare inklusive.
### Die besten Kommentare in jedem Briefing
Kommentare sind jetzt eine quellenübergreifend standardmäßig aktive Ebene: Instagram-Kommentare mit rangbasierter Vielfalt, damit fünf zugespitzte Meinungen nicht alle aus einem einzigen Beitrag stammen (#751), YouTube-Kommentare plus ein Transkript-Backup über ScrapeCreators, falls yt-dlp scheitert (#637), und von der Community hochgevotete Kommentare, die in die Best-Takes-Wertung einfließen, damit die witzigsten Zeilen die Bewertung überleben (#592, #608).
### Ein einziger doctor-Befehl
Bitte um einen Health-Check: doctor prüft jede Quelle und verschreibt dann die genauen Korrekturen welcher Schlüssel fehlt, welche CLI nicht im PATH liegt, welches Cookie abgelaufen ist (#753). Kein Rätselraten mehr, warum X so wenig geliefert hat.
### Die X-Suche, neu gebaut
Die X-Pipeline wurde von Grund auf überarbeitet: FROM- und ABOUT-Lanes, damit sowohl die eigenen Beiträge einer Person als auch das Gespräch über sie einsortiert werden (#610), personenbezogene Auflösung mehrdeutiger Unterabfragen (#611), Verifizierung der Urheberschaft aus erster Hand samt Ranking nach Interaktionssignalen (#613) und eine einzige X-Quelle mit automatischem Backend-Failover (#622). Dazu ein ehrliches `--diagnose`, das die Authentifizierung wirklich prüft (#609).
### Weitere Quellen sind dazugekommen
LinkedIn über ScrapeCreators, mit Artikeln als starkem Signal ([@ravstr](https://github.com/ravstr), #702). StockTwits aktiviert sich automatisch bei Ticker- und Krypto-Themen ([@wtiwana](https://github.com/wtiwana), #658). Perplexity hat direkte API-Modi und asynchrone Deep Research dazubekommen ([@sk-holmes](https://github.com/sk-holmes), #629).
### Von der Community gehärtet
Die Sicherheitswelle war fast vollständig Community-Arbeit: Fixes für Stored XSS im HTML-Renderer ([@iliaal](https://github.com/iliaal), [@aaronjmars](https://github.com/aaronjmars)), abgesicherte temporäre Cookie-Dateien, eine gegen Supply-Chain-Angriffe gehärtete CI mit OpenSSF Scorecard und Build-Provenance-Attestierung ([@shaanmajid](https://github.com/shaanmajid), [@hammadxcm](https://github.com/hammadxcm), [@aniruddh909](https://github.com/aniruddh909)), Semgrep- und OSV-Scanner-Scans plus ein Dependency-Review-Gate für jeden PR ([@23241a6749](https://github.com/23241a6749)), eine Mindestgrenze für die Testabdeckung, eingeführt bei 60 % und inzwischen auf 84 % angehoben ([@gourab5139014](https://github.com/gourab5139014)), und ein Hermes-Sicherheitsscan, der inzwischen keinen einzigen CRITICAL-Befund mehr enthält (#768).
### Reicht weiter
Hebräisch und andere nichtlateinische Sprachen ([@dudyme](https://github.com/dudyme)). CJK-taugliche Tokenisierung für chinesische Quellen ([@An-idd](https://github.com/An-idd)). Eine Welle an Windows-Kompatibilität. Cookie-Extraktion für die gesamte Chromium-Familie Brave, Edge, Vivaldi, Opera, Arc ([@andrey-esipov](https://github.com/andrey-esipov)) plus macOS Keychain und pass(1) unter Linux als Quellen für Zugangsdaten. Historischer Rückblick mit `--as-of` ([@chiyi-creator](https://github.com/chiyi-creator)). Automatisch bereitgestelltes Python 3.12 über uv ([@buntysomroy](https://github.com/buntysomroy)). `--hiring-signals` zum Auslesen der Stellenseiten eines Unternehmens. Watchlist-Deltas zwischen zwei Durchläufen.
### Weiterhin ab Werk dabei seit v3
Die Grundlagen aus v3 sind alle noch da: das Pre-Research-Hirn, das die richtigen Handles, Subreddits und Hashtags ermittelt, bevor ein einziger API-Aufruf rausgeht (gebaut von [@j-sperling](https://github.com/j-sperling)); die Best-Takes-Wertung, die Humor und Viralität neben Relevanz berücksichtigt; quellenübergreifendes Cluster-Merging; Vergleiche in einem Durchgang („CLI vs MCP“ in 3 Minuten statt 12); automatisch gefundene `--competitors`-Vergleiche; der GitHub-Personenmodus (`--github-user=steipete`); der ELI5-Modus („eli5 on“ nach jedem Durchlauf); und teilbare, in sich geschlossene HTML-Briefings (`--emit=html`). Die Konfigurationsschalter stehen in [CONFIGURATION.md](CONFIGURATION.md).
## Installation
| Umgebung | Installation | Updates |
|---------|---------|---------|
| **Claude Code** (empfohlen) | `/plugin marketplace add mvanhorn/last30days-skill` | Automatisch über den Marketplace, oder `claude plugin update last30days@last30days-skill` |
| **Grok** (xAI Build CLI) | `grok plugin marketplace add mvanhorn/last30days-skill`, dann `grok plugin install last30days` | `grok plugin update last30days` |
| **Codex, Cursor, Copilot, Gemini CLI oder einer von 50+ [Agent Skills](https://agentskills.io)-Hosts** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **claude.ai** (Browser) | [`last30days.skill` herunterladen](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) und über claude.ai > Customize > Skills > + > Create skill > Upload a skill hochladen | Neu herunterladen und erneut hochladen |
| **Claude Desktop** | [Die `.mcpb` für deine Plattform herunterladen](https://github.com/mvanhorn/last30days-skill/releases/latest) und in Settings > Extensions ziehen | Neu herunterladen und das neue Bundle hineinziehen |
| **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code (empfohlen)
```
/plugin marketplace add mvanhorn/last30days-skill
```
Empfohlen, weil der Claude-Code-Marketplace die Updates für dich übernimmt: Der Plugin-Cache ist versioniert und aktualisiert sich automatisch, sobald ein neues Release erscheint. Mit `claude plugin update last30days@last30days-skill` erzwingst du eine Prüfung.
Wenn du lieber den Agent-Skills-Installationsweg unter Claude Code nutzt, wird auch der unterstützt:
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
Das native Plugin und die `npx skills`-Installation können nebeneinander existieren. Beachte aber: Claude Code dedupliziert nicht über Installationsmethoden hinweg. Wenn sowohl das Marketplace-Plugin als auch die `npx skills`-Kopie aktiv sind, taucht `/last30days` doppelt auf. Nutze pro Rechner eine Installationsmethode.
### Grok (xAI Build CLI)
[Grok Build](https://docs.x.ai/build/features/skills-plugins-marketplaces) (`grok`) installiert last30days als natives Plugin. Die direkte Installation folgt dem Repository:
```bash
grok plugin install mvanhorn/last30days-skill
```
Oder füge dieses Repository als Marketplace-Quelle hinzu und installiere anschließend über den Plugin-Namen:
```bash
grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
Mit `--trust` überspringst du die Installationsbestätigung. Aktualisieren kannst du mit `grok plugin update last30days`. Grok liest aus Kompatibilitätsgründen auch die Claude-Code-Manifeste; das native `.grok-plugin/`-Paar ist der bevorzugte Weg und genau darauf verweist ein offizieller Eintrag im [xAI-Marketplace](https://github.com/xai-org/plugin-marketplace). `npx skills add` bleibt ein gültiger Fallback über alle Hosts hinweg.
### Codex, Cursor, Copilot, Gemini CLI und weitere Agent-Skills-Hosts
Installiere über die offene [Agent Skills](https://agentskills.io)-CLI sie unterstützt 50+ Hosts, darunter `codex`, `cursor`, `github-copilot`, `gemini-cli`, `claude-code`, `windsurf`, `cline`, `continue`, `roo`, `aider-desk`, `opencode`, `goose` und weitere (vollständige Liste im [Repository vercel-labs/skills](https://github.com/vercel-labs/skills)).
```bash
npx skills add mvanhorn/last30days-skill -g
```
Das Flag `-g` (global) installiert in dein Benutzerverzeichnis, sodass die Skill in allen Projekten verfügbar ist. Ohne `-g` installiert `npx skills` projektlokal nach `./.skills/` (und wird mit dem Repository eingecheckt). Für ein Werkzeug, mit dem du die ganze Welt recherchierst, willst du die globale Installation.
Codex Desktop und andere Hosts, die auf Ordnerebene arbeiten, funktionieren sowohl in gewöhnlichen Ordnern als auch in Git-Repositories. Bitte den Host-Agenten vor der ersten Recherche, das mitgelieferte `scripts/last30days.py --preflight` aus dem geladenen Skill-Verzeichnis auszuführen; in einem Checkout des Quellcodes lautet der entsprechende Befehl `python3 skills/last30days/scripts/last30days.py --preflight`. Er zeigt dir, woher die Konfiguration stammt, welche Browser-Cookies gelesen würden, welche Dateien geschrieben würden, welche optionalen Befehle es gibt und welche Projektkonfiguration ignoriert wird ohne Cookies zu lesen, Dateien zu schreiben oder eine Recherche zu starten.
Standardmäßig wird für den Host installiert, den `npx skills` erkennt. Um gezielt einen (oder mehrere) anzusprechen:
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
Später aktualisieren mit:
```bash
npx skills update last30days -g
```
Oder aktualisiere alles, was du global über `npx skills` installiert hast:
```bash
npx skills update -g
```
Auflisten und entfernen kannst du mit `npx skills list -g` und `npx skills remove last30days -g`.
### claude.ai (Browser)
1. [`last30days.skill` herunterladen](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) aus dem neuesten Release
2. Geh zu [claude.ai > Customize > Skills](https://claude.ai/customize/skills)
3. Klicke im Skills-Panel auf `+`, dann auf `Create skill` > `Upload a skill`, und wähle die Datei aus oder zieh sie hinein
Aktiviere vorher unter Capabilities die Option „Code execution and file creation“ ohne sie laufen Skills nicht.
### Claude Desktop
Claude Desktop installiert `/last30days` als MCP-Server über ein `.mcpb`-Bundle (ein Model-Context-Protocol-Paket zum Ein-Klick-Installieren).
1. Öffne das [neueste Release](https://github.com/mvanhorn/last30days-skill/releases/latest) und lade die `.mcpb` für deine Plattform herunter:
- macOS Apple Silicon: `last30days-pp-mcp-darwin-arm64.mcpb`
- macOS Intel: `last30days-pp-mcp-darwin-amd64.mcpb`
- Linux x86_64: `last30days-pp-mcp-linux-amd64.mcpb`
2. Öffne Claude Desktop, geh zu Settings > Extensions und zieh die Datei hinein.
3. Füge auf Nachfrage die API-Schlüssel für die Quellen ein, die du aktivieren willst. Jedes Feld ist optional überspringst du alle, fällt die Engine auf den reinen Web-Modus zurück. Die Schlüssel landen im Schlüsselbund deines Betriebssystems.
4. Starte Claude Desktop neu. Bitte Claude, „zu Peter Steinberger zu recherchieren“ oder zu einem beliebigen anderen Thema, und es ruft das Tool `research` auf.
**Voraussetzung auf dem Host:** Python 3.12+ im PATH. Das Bundle bringt den Quellcode der Engine mit, nutzt aber deinen lokalen Python-Interpreter. Unter Windows installierst du ihn von [python.org](https://www.python.org/downloads/); macOS und die meisten Linux-Distributionen bringen bereits eine kompatible Version mit.
**Die Schlüssel werden nicht mit der Claude-Code-Skill geteilt.** Claude Desktop und Claude Code halten bewusst getrennte Speicher für Zugangsdaten. Wenn du `~/.config/last30days/.env` bereits für die Claude-Code-Skill eingerichtet hast, gibst du dieselben Schlüssel hier einmalig erneut ein.
Windows-Unterstützung ist zurückgestellt, bis die plattformspezifischen Einstiegspunkte im Manifest geklärt sind; verfolgt wird das in einem eigenen Issue.
### OpenClaw
```bash
clawhub install last30days-official
```
Für X/Twitter-Aktionen außerhalb der `/last30days`-Recherche Tweets oder
Antworten posten, Follower exportieren, Medien verwalten, Accounts beobachten
und Verlosungen auswerten nutzt du [TweetClaw](https://github.com/Xquik-dev/tweetclaw)
als ergänzendes OpenClaw-Plugin. TweetClaw wird von Xquik-dev gepflegt und ist
hier nur als optionale Ergänzung aufgeführt, nicht als Abhängigkeit oder
Empfehlung von last30days.
### Manuell (für Entwickler)
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
```
Der Symlink hält die Installation beim Bearbeiten mit deinem Arbeitsverzeichnis synchron erneutes Kopieren entfällt. Für `claude.ai` baust du die `.skill`-Datei aus dem Quellcode: `bash skills/last30days/scripts/build-skill.sh` erzeugt `dist/last30days.skill`.
Reddit (mit Kommentaren), Hacker News, Polymarket und GitHub funktionieren sofort. Null Konfiguration. Führe `/last30days` einmal aus, und der Setup-Assistent schaltet in 30 Sekunden weitere Quellen frei, darunter die kostenlosen CLIs für arXiv und Techmeme.
## Bring deine eigenen Schlüssel mit
Diese Plattformen haben nichts miteinander zu tun. X weiß nicht, was Reddit denkt. YouTube sieht TikTok nicht. Aber du kannst deine eigenen API-Schlüssel und Browser-Tokens mitbringen und hast auf einen Schlag Zugriff auf alle gleichzeitig.
| Quellen | Was du brauchst | Kosten |
|---------|---------------|------|
| Reddit (mit Kommentaren) + HN + Polymarket + GitHub + StockTwits | Nichts | Kostenlos |
| arXiv + Techmeme | Kostenlose CLIs, die das Erst-Setup automatisch installiert | Kostenlos |
| X / Twitter | In einem beliebigen Browser bei x.com anmelden, oder `XQUIK_API_KEY` / `XAI_API_KEY` setzen | Browser-Cookies sind kostenlos; Schlüssel hängen vom Anbieter ab |
| YouTube | `brew install yt-dlp` | Kostenlos |
| Bluesky | App-Passwort von bsky.app | Kostenlos |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube-Kommentare | Ein ScrapeCreators-Schlüssel | 10.000 kostenlose Aufrufe, danach nutzungsabhängig |
| Xiaohongshu (RED) | Ein eingeloggtes x-mcp-Browser-Plugin oder einen `xiaohongshu-mcp`-Dienst laufen lassen und die Quelle mit `--search xhs` pro Durchlauf oder `INCLUDE_SOURCES=xiaohongshu` in `.env` zuschalten; last30days probiert automatisch `http://localhost:18060` und danach `http://host.docker.internal:18060`, oder du setzt `XIAOHONGSHU_API_BASE` für eine eigene URL | Kein last30days-API-Schlüssel nötig; hängt von deinem lokalen Browser-Session-Dienst ab |
| DripStack (Premium-Finanznewsletter) | Zuschaltbar: `--search dripstack` pro Durchlauf, oder `INCLUDE_SOURCES=dripstack` in `.env` | Kein Schlüssel; kostenlose öffentliche Such-API |
| Perplexity Sonar / Search API / Deep Research | Ein Perplexity-Schlüssel, oder ein OpenRouter-Schlüssel als Sonar-Fallback | Nutzungsabhängig |
| Websuche | Ein Brave-Search-Schlüssel | 2.000 kostenlose Anfragen pro Monat |
### macOS Keychain (optional)
Unter macOS kannst du Schlüssel im System-Schlüsselbund statt in einer `.env`-Datei ablegen. Die Skill liest sie automatisch aus, allerdings mit der niedrigsten Priorität bei einer Kollision gewinnen weiterhin `.env`-Dateien und die Prozessumgebung.
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
Die Einträge werden für den aktuellen Benutzer unter dem Dienstnamen `last30days-<KEY>` gespeichert. Auf Nicht-Darwin-Plattformen tut der Loader nichts, für Linux- und Windows-Nutzer ändert sich also am Verhalten nichts.
Du hast bereits Schlüssel unter anderen Keychain-Dienstnamen? Dann setz das nicht geheime Mapping `LAST30DAYS_KEYCHAIN_ALIASES`, das in [CONFIGURATION.md](CONFIGURATION.md#reusing-existing-macos-keychain-items) beschrieben ist, statt Geheimnisse zu kopieren.
Die vollständige Schlüsselmatrix pro Quelle, die Priorität der Reasoning-Anbieter und die Priorität der Websuche-Backends stehen in [CONFIGURATION.md](CONFIGURATION.md).
## Konfiguration
Zwei Dinge, die du vermutlich schon am ersten Tag wissen willst:
**Wo die Rechercheergebnisse landen.** `LAST30DAYS_MEMORY_DIR` zeigt standardmäßig auf `~/Documents/Last30Days/` (unter Windows: `C:\Users\<you>\Documents\Last30Days\`). Überschreib das, indem du die Umgebungsvariable in deiner Shell auf einen beliebigen Pfad setzt, oder mit `--save-dir <path>` pro Durchlauf. Nutze `--output <file>`, wenn du das gerenderte Ergebnis an einem exakten Pfad brauchst im Format, das `--emit` vorgibt. Mit `--save-suffix=<name>` hältst du mehrere Varianten desselben Themas auseinander (etwa pro Kunde). Jeder Durchlauf mit `--save-dir` erzeugt `<slug>-raw[-suffix].md`. Mit `python3 skills/last30days/scripts/last30days.py --preflight` siehst du vor einer Recherche, welche Dateien geschrieben würden.
**Strukturierte Ausgabe für Agenten und Workflows.** Bitte `/last30days` um maschinenlesbares JSON, dann bekommst du das stabile, versionierte Agentenprofil. Für den direkten Einsatz der Engine in Skripten oder in der Entwicklung führst du `python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json` aus; `--json-profile=raw` brauchst du nur, wenn du den unversionierten internen `Report`-Dump willst. Siehe die [Feldreferenz des JSON-Exports samt Versionierungsrichtlinie](docs/reference/json-export.md).
**Discovery ohne festes Thema.** Frag `/last30days what's trending in AI agents?`, um ein sortiertes Discovery-Briefing zu bekommen, statt ein Thema zu recherchieren, das du ohnehin kennst. Auf einem Agenten-Host läuft dafür das dreistufige, vom Host bewertete Protokoll (das Modell benennt Themen, filtert Müll heraus, bewertet ihre Relevanz und schreibt die inhaltlichen Ansätze). Für den direkten Einsatz der Engine in Skripten oder per Cron führst du `python3 skills/last30days/scripts/last30days.py --discover "AI agents"` aus (einmaliger Lauf: deterministische Themennamen, keine Ansätze); mit `--emit=json` bekommst du den versionierten Discovery-Vertrag. Discovery schließt ein positionsbasiertes Thema und `--drill` gegenseitig aus.
**Trendbeobachtung über mehrere Durchläufe.** Der Standardmodus erzeugt pro Durchlauf einen frischen Markdown-Snapshot. Um Erkenntnisse über die Zeit zu sammeln, hängst du `--store` an, damit sie in einer SQLite-Datenbank landen, und nutzt dann [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) für geplante Durchläufe (auf Wunsch mit Zustellung per Slack oder Webhook bei neuen Funden) sowie [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) für tägliche oder wöchentliche Zusammenfassungen. Das vollständige Taktmuster steht in [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
**Eine abonnierbare Recherche-Bibliothek.** Bitte `/last30days`, deinen Bibliotheks-Feed zu bauen, oder nutze für Skripting und Entwicklung direkt `python3 skills/last30days/scripts/last30days.py library feed`. Das verwandelt gespeicherte Briefings in eine `index.html`, ein lokales Atom-`feed.xml` und lesbare Briefing-Seiten. Hänge `--publish` nur an, wenn der HTML-Index und die Briefing-Seiten gehostet werden sollen; das Veröffentlichen ist eine bewusste Entscheidung und standardmäßig öffentlich. Damit der Atom-Feed wirklich abonnierbar wird, hoste das erzeugte Ausgabeverzeichnis bei einem statischen Anbieter wie GitHub Pages.
**Durchsuche alles, was du schon recherchiert hast.** Frag `/last30days search my library for MCP servers` oder `/last30days have I researched MCP servers before?`. Für den direkten Einsatz der Engine führst du `python3 skills/last30days/scripts/last30days.py library search "MCP servers"` aus. Die Suche läuft offline und deterministisch: Sie indexiert nach und nach dieselben gespeicherten Briefings, die auch der Bibliotheks-Feed nutzt, führt passende Treffer aus dem Store je Durchlauf zusammen und gruppiert die Ergebnisse nach Thema und Datum. Neue Durchläufe blenden außerdem einen kompakten Abschnitt **From your library** („aus deiner Bibliothek“) ein, wenn frühere Recherchen das aktuelle Thema überschneiden; mit `LAST30DAYS_LIBRARY_CONTEXT=off` schaltest du diesen passiven Kontext ab.
Wrapper-Skripte pro Kunde, eigene Kategorie-Subreddits und der experimentelle Beta-Kanal für Anpassungen in Arbeit sind ebenfalls in [CONFIGURATION.md](CONFIGURATION.md) dokumentiert.
## Showcase: Recherche-Feeds aus der Community
Du hast mit last30days ein wiederkehrendes KI-Update, eine Marktbeobachtung oder eine herrlich spezielle Obsession veröffentlicht? Teil die URL deiner öffentlichen Bibliothek oder die Atom-URL, sobald `feed.xml` bei einem statischen Anbieter liegt im [Showcase-Thread der Community](https://github.com/mvanhorn/last30days-skill/issues/532). Community-Feeds werden hier verlinkt, sobald ihre Besitzer sie einreichen; bis dahin ist der Thread die Sammelstelle.
## So funktioniert es
1. **Du tippst ein Thema ein.** Person, Unternehmen, Produkt, Technologie, „X vs Y“. Alles ist möglich.
2. **Der Agent klärt, wer zählt.** Er findet X-Handles (auch die von Gründerinnen und Gründern), GitHub-Repos, Subreddits, TikTok-Hashtags und YouTube-Kanäle. Bei „Kanye West“ weiß er, dass r/hiphopheads, @kanyewest und „bully review“ auf YouTube dazugehören. Bei „OpenClaw“ löst er openclaw/openclaw auf GitHub auf und holt die aktuellen Sternzahlen.
3. **Alle Quellen werden parallel durchsucht.** Erweiterung über mehrere Suchanfragen. Ergebnisse gewichtet nach Engagement, Relevanz und Aktualität.
4. **Die Tiefe, die sonst niemand hat.** Vollständige YouTube-Transkripte aus Reaktionsvideos. Die besten Reddit-Kommentare samt Upvote-Zahlen. TikTok-Captions. Polymarket-Quoten. Nicht nur Titel und Links.
5. **Dieselbe Geschichte, zusammengeführt.** Das Wireless Festival auf Reddit angekündigt, auf X diskutiert, Ticketpreise auf TikTok das ergibt einen Cluster, nicht drei getrennte Einträge.
6. **Zu einem Briefing verdichtet.** Auf konkreten Daten fußend. Nach Quelle belegt. Sortiert nach dem, womit Menschen wirklich interagieren. Nicht „hier ist, was ich gefunden habe“, sondern „hier ist, was zählt“.
7. **Danach wird es dein Experte.** Nach einem einzigen Durchlauf weiß deine Claude-Sitzung alles, was die Community weiß. Stell Rückfragen. Lass sie Prompts schreiben, E-Mails entwerfen, Reisen planen, Systeme entwerfen immer verankert in dem, was gerade wirklich stimmt.
## Was die Leute sagen
> „Ich habe eine Claude-Code-Skill gefunden, die zu jedem Thema die letzten 30 Tage auf Reddit, X, YouTube und HN recherchiert. Und dann schreibt sie dir die Prompts. Vor jedem Text, den ich schreibe, habe ich das bisher von Hand auf Reddit und X gemacht. Tab für Tab. Thread für Thread. Genau das ist der Teil, der 90 Minuten frisst. Der fällt jetzt weg.“ @itsjasonai
> „Diese eine Skill hat meinen kompletten Recherche-Workflow ersetzt. Du gibst ihr ein Thema, sie holt sich von Reddit, X und dem Web, worüber die Leute wirklich reden. Keine alten Blogbeiträge. Echte Gespräche aus den letzten 30 Tagen.“ @itswilsoncharles
> „5 der 10 Trending-Repos heute auf GitHub sind Claude-Tools. Nummer 1: mvanhorn/last30days-skill“ @yieldhunter95
## Open Source
MIT-Lizenz. Kein Tracking. Keine Analytics. Deine Recherche bleibt auf deinem Rechner. Über 2.700 Tests.
Gebaut mit Python 3.12+, yt-dlp, Node.js (mitgelieferter Bird-Client für die X-Suche) und der ScrapeCreators-API. Architektur der v3-Engine von [@j-sperling](https://github.com/j-sperling).
Wie du einen PR aufmachst, steht in [CONTRIBUTING.md](CONTRIBUTING.md), die vollständige Liste der Community-Beitragenden in [CONTRIBUTORS.md](CONTRIBUTORS.md) und die Versionshistorie in [CHANGELOG.md](CHANGELOG.md).
## Sternverlauf
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
+383
View File
@@ -0,0 +1,383 @@
# /last30days
[English](README.md) | [Français](README.fr.md) | [Deutsch](README.de.md) | Español | [Português (Brasil)](README.pt-BR.md) | [日本語](README.ja.md) | [简体中文](README.zh-CN.md)
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days - an AI agent-led search engine that searches people, not editors" />
</p>
<p align="center">
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
**Un buscador dirigido por un agente de IA que puntúa por votos positivos, likes y dinero real, no por redacciones.**
Este README documenta el pipeline v3 actual. La especificación de ejecución de la skill vive en [skills/last30days/SKILL.md](skills/last30days/SKILL.md), que es la referencia definitiva sobre el comportamiento de los comandos y la configuración.
**Claude Code (recomendado — actualizaciones automáticas vía marketplace):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex, Cursor, Copilot, Gemini CLI, o cualquiera de los 50+ hosts de [Agent Skills](https://agentskills.io):**
```
npx skills add mvanhorn/last30days-skill -g
```
(`-g` instala de forma global para tu usuario, así que la tienes disponible en todos tus proyectos. Omite ese flag si prefieres limitar la instalación a un proyecto.)
Más formas de instalarlo (claude.ai web, OpenClaw, manual) en la sección [Instalación](#instalación) de más abajo.
Cero configuración. Reddit, HN, Polymarket y GitHub funcionan de inmediato. Ejecútalo una vez y el asistente de configuración desbloquea X, YouTube, TikTok, arXiv, Techmeme y más en 30 segundos.
---
Los votos positivos de Reddit. Los likes de X. Las transcripciones de YouTube. La interacción en TikTok. Las cuotas de Polymarket, respaldadas por dinero real y por información privilegiada. Eso son millones de personas votando cada día con su atención y su cartera. /last30days lo busca todo en paralelo, lo puntúa según aquello con lo que la gente interactúa de verdad, y un agente de IA hace de juez para sintetizarlo en un único informe.
Google agrega redacciones. /last30days busca personas.
Esta búsqueda no la consigues en ningún otro sitio, porque ninguna IA tiene acceso a todo. Google no toca los comentarios de Reddit ni las publicaciones de X. ChatGPT tiene un acuerdo con Reddit, pero no puede buscar en X ni en TikTok. Gemini tiene YouTube, pero no Reddit. Claude no tiene ninguno de forma nativa. Cada plataforma es un jardín amurallado con su propia API, sus propios tokens y su propia autenticación. Pero tú puedes aportar tus claves y tus sesiones de navegador y, de golpe, un agente de IA las consulta todas a la vez, las compara entre sí y te dice qué importa de verdad.
Ese es el desbloqueo. No se trata de un buscador mejor, sino de una docena de plataformas incomunicadas que un agente conecta entre sí.
```
/last30days Peter Steinberger
```
Mañana tienes una reunión. Buscas a la persona en Google. Te sale su LinkedIn de 2023. /last30days te da lo que está haciendo de verdad este mes: se ha incorporado a OpenAI para trabajar en Codex, pelea contra el veto de Anthropic a los agentes de terceros, ha entregado 23 PR con un 85 % de tasa de merge, construye «LobsterOS» para controlar agentes entre dispositivos, y un hilo de r/ClaudeCode llegó a 569 votos positivos debatiendo si es un héroe o un «insoportable». Todo repartido entre publicaciones de X, hilos de Reddit, transcripciones de YouTube y commits de GitHub. Nada de eso estaba en Google.
## Por qué existe esto
Lo construí para no quedarme atrás en IA. Todo cambia cada día y los frikis de Reddit y de X siempre se enteran primero. Necesitaba mejores prompts, y los datos de entrenamiento siempre iban meses por detrás de lo que la comunidad ya había averiguado.
Pero acabó siendo algo más grande. Ahora lo lanzo antes de una llamada comercial, para conocer la verdad de los últimos 30 días sobre una empresa. Antes de una reunión, para leer los tuits recientes y las transcripciones de podcasts de la otra persona. Antes de un viaje a Disney World, para saber qué atracciones están cerradas y qué opina la comunidad sobre Genie+. Antes de construir nada, para saber con qué problemas se está encontrando la gente de verdad.
Si te vas a reunir con un CEO, ¿te has leído todos sus tuits y todas sus transcripciones de YouTube de los últimos 30 días? Yo sí.
## Fuentes, puntuadas por la gente
| Fuente | Lo que te dice la gente |
|--------|--------------------------|
| **Reddit** | La opinión sin filtros. Los mejores comentarios con su recuento real de votos positivos, gratis y sin clave de API. Las opiniones reales que Google entierra. |
| **X / Twitter** | La reacción en caliente, el hilo del experto, la primera respuesta a una noticia de última hora. Los primeros en enterarse, los primeros en discutir. |
| **YouTube** | El análisis a fondo de 45 minutos. Transcripciones completas, rastreadas para sacar las 5 frases citables que importan. |
| **TikTok** | El creador que llega a 3,6 millones de personas con una lectura que nunca encontrarás en Google. |
| **Instagram Reels** | La mirada de los influencers, con transcripción de lo que dicen. La señal de la cultura visual. |
| **Hacker News** | El consenso de los desarrolladores. 825 puntos, 899 comentarios. Donde la gente técnica discute de verdad. |
| **Polymarket** | No son opiniones. Son cuotas. Respaldadas por dinero real. 96 % de probabilidad en ventas de un álbum. 4 % en una adquisición. |
| **GitHub** | Para personas: ritmo de PR, mejores repositorios por estrellas, notas de versión. Para temas: issues y discusiones. |
| **Digg** | Grupos de noticias seleccionados del ranking AI 1000 de Digg (unas 1000 cuentas de IA con mucha señal en X), con citas atribuibles integradas y sin necesidad de autenticarte en X. Se activa solo cuando `digg-pp-cli` está en el PATH. |
| **arXiv** | Los artículos científicos que hay detrás del ruido. Investigación nueva dentro de la ventana, gratis y sin clave de API. Se activa solo cuando `arxiv-pp-cli` está en el PATH (la configuración inicial lo instala). |
| **Techmeme** | La capa editorial de la actualidad tecnológica, acotada a tu ventana de 30 días. Gratis y sin clave de API. Se activa solo cuando `techmeme-pp-cli` está en el PATH (la configuración inicial lo instala). |
| **LinkedIn** | La señal profesional. Publicaciones y artículos, con los artículos ponderados como señal fuerte. |
| **StockTwits** | El sentimiento de los traders. Se activa automáticamente cuando tu tema es un ticker o una criptomoneda. |
| **Threads** | La capa de texto posterior a Twitter. Conversaciones de creadores y marcas. |
| **Pinterest** | Descubrimiento visual. Pines, guardados y comentarios sobre productos e ideas. |
| **Xiaohongshu (RED)** | Señales chinas sobre estilo de vida, productos y creadores. Se pide de forma explícita con `--search xhs` cuando tienes corriendo en local un plugin de navegador x-mcp con sesión iniciada o un servicio `xiaohongshu-mcp`. |
| **Bluesky** | La capa social descentralizada. Publicaciones de AT Protocol surgidas de la migración posterior a Twitter. |
| **Perplexity** | La síntesis fundamentada de Sonar, los resultados en bruto de la Search API y Deep Research. |
| **Web** | La cobertura editorial, las comparativas de los blogs. Una señal entre muchas, no la única. |
La comunidad no para de sumar fuentes. Truth Social y otras fuentes de nicho ya están en el motor, y vienen más.
Un hilo de Reddit con 1.500 votos positivos es una señal más fuerte que una entrada de blog que no leyó nadie. Un TikTok con 3,6 millones de visualizaciones dice más sobre lo que es culturalmente relevante que cualquier nota de prensa. Unas cuotas de Polymarket respaldadas por 66.000 dólares de volumen son más difíciles de rebatir que la corazonada de un tertuliano.
La síntesis ordena según aquello con lo que la gente real ha interactuado de verdad. Relevancia social, no relevancia SEO.
## Para qué lo usa la gente en realidad
**Antes de una reunión.** `/last30days Peter Steinberger` — se ha incorporado al equipo de Codex de OpenAI, pelea contra el veto de Anthropic a los agentes de terceros, 23 PR mergeadas con un 85 % de tasa de merge en GitHub, construye LobsterOS para controlar agentes entre dispositivos. r/ClaudeCode: «Desde que salió OpenClaw, todo el mundo sabía que, si lo pasabas por algo que no fuera la API, acabarías baneado» (227 votos positivos). Eso no está en LinkedIn.
**Para leer señales de contratación.** `/last30days Listen Labs --hiring-signals` — las ofertas de empleo y las páginas de carreras actuales se convierten en pruebas citadas de un cambio de prioridades: contratación en seguridad para empresa, customer success, infraestructura o expansión de producto. El informe dice lo que la contratación parece señalar, no lo que la hoja de ruta va a entregar.
**Para encontrar el tema antes de su pico.** Pregunta `/last30days what's exploding in AI agents?` y la skill cambia a modo descubrimiento: el motor barre los listados por categoría de Reddit, la portada y las mejores historias de Hacker News, el feed AI 1000 de Digg y X si estás autenticado; tu agente evalúa las candidaturas (nombres, filtrado de ruido, interés real) y escribe enfoques para pódcast o para un artículo en X; después obtienes entre 5 y 10 temas ordenados por velocidad. Cada resultado incluye cifras de varias fuentes, una etiqueta de impulso y un comando `/last30days "<topic>"` listo para lanzar.
**Cuando sale algo nuevo.** `/last30days Kanye West` — el Reino Unido le bloqueó el visado, el Wireless Festival se canceló, los patrocinadores huyeron. Pero BULLY debutó en el número 2 del Billboard. Fantano volvió de su «Yay sabbatical» para reseñarlo (653.000 visualizaciones). En el SoFi Homecoming sacó al escenario a Lauryn Hill y a Travis Scott para 44 canciones. Polymarket: «¿Volverá Kanye a tuitear?» 86 % sí. 23 hilos de Reddit, 17 vídeos de YouTube, 86.000 votos positivos.
**Para comparar herramientas.** `/last30days OpenClaw vs Hermes vs Paperclip` — «No son competidores, son capas.» OpenClaw es la capa de ejecución (351.000 estrellas en GitHub, en producción), Hermes es el cerebro que se mejora a sí mismo (31.000 estrellas), Paperclip es el organigrama (49.000 estrellas). El número de estrellas se saca en directo de la API de GitHub, no de entradas de blog caducadas. Tabla comparativa con arquitectura, memoria, seguridad y caso de uso ideal. Según @IMJustinBrooke: «OpenClaw = Charmander, Hermes = Charizard.»
**Para entender el mundo.** `/last30days Iran vs USA` — día 38 de la guerra. El ultimátum de Trump, con plazo hasta el martes, para que Irán reabra el estrecho de Ormuz. Dos aviones de combate estadounidenses derribados. El petróleo a 126 dólares el barril. La AIE lo calificó como «la mayor interrupción de suministro de la historia del mercado mundial del petróleo». Polymarket: alto el fuego antes del 31 de diciembre al 74 %. 27 publicaciones de X, 10 vídeos de YouTube, 20 mercados de predicción.
**Antes de un viaje.** `/last30days Universal Epic Universe` — la ampliación ya está en obras. Licencia «Project 680» presentada. El espectáculo de fuegos artificiales está confirmado por la infraestructura, pero sin anunciar. Tiempos de espera: Mine-Cart Madness promedia 148 minutos. Todavía no hay pase anual, y los vecinos están hartos. Stardust Racers cerrada por reforma hasta el 5 de abril.
**Para aprender algo rápido.** `/last30days Nano Banana Pro prompting` — los prompts estructurados en JSON están sustituyendo al amontonamiento de etiquetas. El formato anidado de @pictsbyai evita el «concept bleeding». Editar gana a regenerar. Y después te escribe un prompt de producción aplicando exactamente lo que la comunidad ha dicho que funciona.
## Novedades
Desde el anuncio de la v3.3 en mayo y hasta la v3.11.1 (julio de 2026): 175 PR mergeadas —122 de ellas de 52 colaboradores de la comunidad— repartidas en 15 versiones. Esto es lo que ha entrado.
### Ciudadano de primera en OpenAI Codex
/last30days ya es un plugin nativo de Codex con configuración guiada: no es un port, es un ciudadano de primera. Las citas tienen en cuenta el renderizador, así que la salida en Codex se lee como un informe y no como una sopa de URL (#694), y el mismo motor funciona en Claude Code, Cursor, Copilot, Gemini CLI, Claude Desktop, OpenClaw y 50+ hosts de Agent Skills. Manifiesto del plugin de Codex por [@rfoust](https://github.com/rfoust) (#686), corrección de autenticación en Codex por [@tmchow](https://github.com/tmchow) (#698).
### arXiv, Techmeme y Digg: gratis y sin claves de API
arXiv aporta los artículos científicos que hay detrás del ruido y Techmeme la capa editorial de la actualidad tecnológica: gratis, sin una sola clave, y la configuración inicial instala sus CLI para que se activen solas (#709). Los grupos de noticias AI 1000 de Digg llegan igual, sin autenticarte en X: la configuración instala por ti la CLI gratuita de Digg (#590). Trustpilot está disponible como opción para investigar marcas de consumo.
### Reddit gratis, con puntuaciones reales y mejores comentarios
La API pública .json de Reddit desapareció; la vía gratuita volvió más fuerte. RSS sin clave y scraping de shreddit (#457), descubrimiento de subreddits específicos con recuentos reales de votos positivos vía arctic-shift (#696), y un umbral de relevancia para que una publicación viral fuera de tema no secuestre tu informe (#488, gracias [@rzachsmith](https://github.com/rzachsmith)). Sin clave de API. Puntuaciones reales. Con los mejores comentarios incluidos.
### Los mejores comentarios en cada informe
Los comentarios son ya una capa activada por defecto en todas las fuentes: comentarios de Instagram con diversidad basada en el ranking, para que cinco opiniones rotundas no salgan todas de la misma publicación (#751), comentarios de YouTube más un respaldo de transcripción vía ScrapeCreators para cuando yt-dlp falla (#637), y comentarios votados por la comunidad ponderados dentro de Best Takes, para que las mejores frases sobrevivan a la puntuación (#592, #608).
### Un único comando doctor
Pide una revisión y doctor comprueba todas las fuentes y receta los arreglos exactos: qué clave falta, qué CLI no está en el PATH, qué cookie ha caducado (#753). Se acabó adivinar por qué X ha devuelto tan poco.
### La búsqueda en X, reconstruida
El pipeline de X se rehízo de arriba abajo: carriles FROM y ABOUT para que se posicionen tanto las publicaciones de una persona como la conversación sobre ella (#610), desambiguación de subconsultas según la persona buscada (#611), verificación de la autoría de primera mano con ranking por señales de interacción (#613), y una única fuente X con conmutación automática entre backends (#622). Además, un `--diagnose` honesto que comprueba de verdad la autenticación (#609).
### Se han sumado más fuentes
LinkedIn vía ScrapeCreators, con los artículos como señal fuerte ([@ravstr](https://github.com/ravstr), #702). StockTwits se activa automáticamente en temas de tickers y cripto ([@wtiwana](https://github.com/wtiwana), #658). Perplexity ha ganado modos de API directos y Deep Research asíncrono ([@sk-holmes](https://github.com/sk-holmes), #629).
### Endurecido por la comunidad
La oleada de seguridad fue casi por completo trabajo de la comunidad: correcciones de XSS almacenado en el renderizador HTML ([@iliaal](https://github.com/iliaal), [@aaronjmars](https://github.com/aaronjmars)), archivos temporales de cookies blindados, CI endurecida frente a ataques a la cadena de suministro con OpenSSF Scorecard y atestación de procedencia de las builds ([@shaanmajid](https://github.com/shaanmajid), [@hammadxcm](https://github.com/hammadxcm), [@aniruddh909](https://github.com/aniruddh909)), análisis con Semgrep y OSV-Scanner más un control de revisión de dependencias en cada PR ([@23241a6749](https://github.com/23241a6749)), un mínimo de cobertura de pruebas fijado al 60 % y elevado desde entonces al 84 % ([@gourab5139014](https://github.com/gourab5139014)), y un análisis de seguridad de Hermes que ya no arroja ningún hallazgo CRITICAL (#768).
### Llega más lejos
Hebreo y otros idiomas no latinos ([@dudyme](https://github.com/dudyme)). Tokenización adaptada a CJK para las fuentes chinas ([@An-idd](https://github.com/An-idd)). Una oleada de compatibilidad con Windows. Extracción de cookies en toda la familia Chromium —Brave, Edge, Vivaldi, Opera, Arc ([@andrey-esipov](https://github.com/andrey-esipov))— además del llavero de macOS y pass(1) en Linux como orígenes de credenciales. Consulta histórica hacia atrás con `--as-of` ([@chiyi-creator](https://github.com/chiyi-creator)). Instalación automática de Python 3.12 mediante uv ([@buntysomroy](https://github.com/buntysomroy)). `--hiring-signals` para leer las páginas de empleo de una empresa. Diferencias de la lista de seguimiento entre ejecuciones.
### Lo que ya venía de serie desde la v3
Los cimientos de la v3 siguen todos aquí: el cerebro previo a la investigación, que identifica las cuentas, subreddits y hashtags correctos antes de que salga una sola llamada a la API (obra de [@j-sperling](https://github.com/j-sperling)); la puntuación Best Takes, que valora el humor y la viralidad además de la relevancia; la fusión de clústeres entre fuentes; las comparativas en una sola pasada («CLI vs MCP» en 3 minutos, no en 12); las comparativas `--competitors` descubiertas de forma automática; el modo persona de GitHub (`--github-user=steipete`); el modo ELI5 («eli5 on» después de cualquier ejecución); y los informes HTML autocontenidos y compartibles (`--emit=html`). Los ajustes de configuración están en [CONFIGURATION.md](CONFIGURATION.md).
## Instalación
| Entorno | Instalación | Actualizaciones |
|---------|---------|---------|
| **Claude Code** (recomendado) | `/plugin marketplace add mvanhorn/last30days-skill` | Automáticas vía marketplace, o `claude plugin update last30days@last30days-skill` |
| **Grok** (xAI Build CLI) | `grok plugin marketplace add mvanhorn/last30days-skill` y después `grok plugin install last30days` | `grok plugin update last30days` |
| **Codex, Cursor, Copilot, Gemini CLI, o cualquiera de los 50+ hosts de [Agent Skills](https://agentskills.io)** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **claude.ai** (web) | [Descarga `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) y súbelo desde claude.ai > Customize > Skills > + > Create skill > Upload a skill | Volver a descargar y volver a subir |
| **Claude Desktop** | [Descarga el `.mcpb` de tu plataforma](https://github.com/mvanhorn/last30days-skill/releases/latest) y arrástralo a Settings > Extensions | Volver a descargar y arrastrar el nuevo paquete |
| **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code (recomendado)
```
/plugin marketplace add mvanhorn/last30days-skill
```
Es la opción recomendada porque el marketplace de Claude Code se encarga de las actualizaciones por ti: la caché del plugin está versionada y se refresca sola cuando se publica una versión nueva. Ejecuta `claude plugin update last30days@last30days-skill` para forzar una comprobación.
Si prefieres usar la vía de instalación de Agent Skills en Claude Code, también está soportada:
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
El plugin nativo y la instalación con `npx skills` pueden convivir. Ojo: Claude Code no deduplica entre métodos de instalación. Si tienes activos a la vez el plugin del marketplace y la copia de `npx skills`, `/last30days` aparecerá dos veces. Usa un solo método de instalación por máquina.
### Grok (xAI Build CLI)
[Grok Build](https://docs.x.ai/build/features/skills-plugins-marketplaces) (`grok`) instala last30days como plugin nativo. La instalación directa sigue el repositorio:
```bash
grok plugin install mvanhorn/last30days-skill
```
O añade este repositorio como fuente de marketplace y luego instálalo por nombre de plugin:
```bash
grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
Añade `--trust` para saltarte la confirmación de instalación. Actualiza con `grok plugin update last30days`. Grok también lee los manifiestos de Claude Code por compatibilidad; el par nativo `.grok-plugin/` es la vía principal, y es a lo que apunta una entrada oficial en el [marketplace de xAI](https://github.com/xai-org/plugin-marketplace). `npx skills add` sigue siendo una alternativa válida para cualquier host.
### Codex, Cursor, Copilot, Gemini CLI y otros hosts de Agent Skills
Instálalo con la CLI abierta de [Agent Skills](https://agentskills.io): soporta 50+ hosts, entre ellos `codex`, `cursor`, `github-copilot`, `gemini-cli`, `claude-code`, `windsurf`, `cline`, `continue`, `roo`, `aider-desk`, `opencode`, `goose` y más (lista completa en el [repositorio vercel-labs/skills](https://github.com/vercel-labs/skills)).
```bash
npx skills add mvanhorn/last30days-skill -g
```
El flag `-g` (global) instala en tu directorio de usuario, de modo que la skill queda disponible en todos los proyectos. Sin `-g`, `npx skills` instala solo en el proyecto, dentro de `./.skills/` (y se versiona con el repositorio). Para una herramienta que sirve para investigar el mundo entero, lo que quieres es la instalación global.
Codex de escritorio y otros hosts que trabajan a nivel de carpeta funcionan tanto en carpetas normales como en repositorios Git. Antes de la primera investigación, pídele al agente anfitrión que ejecute el `scripts/last30days.py --preflight` incluido desde el directorio de la skill cargada; en un clon del código fuente, el comando equivalente es `python3 skills/last30days/scripts/last30days.py --preflight`. Te muestra de dónde sale la configuración, qué cookies del navegador se leerían, qué archivos se escribirían, qué comandos opcionales hay y qué configuración de proyecto se ignora, todo ello sin leer cookies, sin escribir archivos y sin lanzar ninguna investigación.
Por defecto se instala para el host que detecte `npx skills`. Para apuntar a uno concreto (o a varios):
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
Para actualizar más adelante:
```bash
npx skills update last30days -g
```
O actualiza todo lo que hayas instalado globalmente con `npx skills`:
```bash
npx skills update -g
```
Puedes listarlo y desinstalarlo con `npx skills list -g` y `npx skills remove last30days -g`.
### claude.ai (web)
1. [Descarga `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) de la última versión publicada
2. Entra en [claude.ai > Customize > Skills](https://claude.ai/customize/skills)
3. Pulsa el botón `+` del panel de Skills, luego `Create skill` > `Upload a skill`, y busca o arrastra el archivo
Activa antes «Code execution and file creation» en Capabilities: sin eso, las skills no se ejecutan.
### Claude Desktop
Claude Desktop instala `/last30days` como servidor MCP mediante un paquete `.mcpb` (un paquete de Model Context Protocol de un solo clic).
1. Entra en la [última versión publicada](https://github.com/mvanhorn/last30days-skill/releases/latest) y descarga el `.mcpb` de tu plataforma:
- macOS Apple Silicon: `last30days-pp-mcp-darwin-arm64.mcpb`
- macOS Intel: `last30days-pp-mcp-darwin-amd64.mcpb`
- Linux x86_64: `last30days-pp-mcp-linux-amd64.mcpb`
2. Abre Claude Desktop, ve a Settings > Extensions y arrastra el archivo ahí.
3. Cuando te las pida, pega las claves de API de las fuentes que quieras activar. Todos los campos son opcionales: si los saltas todos, el motor se queda en modo solo web. Las claves se guardan en el llavero de tu sistema operativo.
4. Reinicia Claude Desktop. Pídele a Claude que «investigue a Peter Steinberger», o cualquier otro tema, y llamará a la herramienta `research`.
**Requisito del anfitrión:** Python 3.12+ en el PATH. El paquete incluye el código del motor, pero usa tu intérprete de Python local. En Windows, instálalo desde [python.org](https://www.python.org/downloads/); macOS y la mayoría de distribuciones de Linux ya traen una versión compatible.
**Las claves no se comparten con la skill de Claude Code.** Claude Desktop y Claude Code mantienen almacenes de credenciales separados a propósito. Si ya configuraste `~/.config/last30days/.env` para la skill de Claude Code, aquí tendrás que introducir esas mismas claves una vez.
La compatibilidad con Windows queda aplazada hasta resolver los puntos de entrada por plataforma del manifiesto; el seguimiento se hace en una incidencia aparte.
### OpenClaw
```bash
clawhub install last30days-official
```
Para flujos de acción en X/Twitter fuera de la investigación de `/last30days` —publicar
tuits o respuestas, exportar seguidores, gestionar medios, monitorizar cuentas y
resolver sorteos— usa [TweetClaw](https://github.com/Xquik-dev/tweetclaw) como
plugin complementario de OpenClaw. TweetClaw lo mantiene Xquik-dev y aparece aquí
únicamente como opción complementaria: no es una dependencia ni una recomendación
de last30days.
### Manual (para desarrolladores)
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
```
El enlace simbólico mantiene la instalación sincronizada con tu copia de trabajo a medida que editas, sin necesidad de volver a copiar nada. Para `claude.ai`, compila el archivo `.skill` desde el código fuente: `bash skills/last30days/scripts/build-skill.sh` genera `dist/last30days.skill`.
Reddit (con comentarios), Hacker News, Polymarket y GitHub funcionan de inmediato. Cero configuración. Ejecuta `/last30days` una vez y el asistente de configuración desbloquea más fuentes en 30 segundos, incluidas las CLI gratuitas de arXiv y Techmeme.
## Aporta tus propias claves
Estas plataformas no tienen ninguna relación entre sí. X no sabe lo que piensa Reddit. YouTube no ve TikTok. Pero tú puedes aportar tus claves de API y tus tokens de navegador y, de golpe, tienes acceso a todas a la vez.
| Fuentes | Lo que necesitas | Coste |
|---------|---------------|------|
| Reddit (con comentarios) + HN + Polymarket + GitHub + StockTwits | Nada | Gratis |
| arXiv + Techmeme | CLI gratuitas, instaladas automáticamente por la configuración inicial | Gratis |
| X / Twitter | Inicia sesión en x.com en cualquier navegador, o define `XQUIK_API_KEY` / `XAI_API_KEY` | Las cookies del navegador son gratis; las claves dependen del proveedor |
| YouTube | `brew install yt-dlp` | Gratis |
| Bluesky | Una contraseña de aplicación de bsky.app | Gratis |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + comentarios de YouTube | Una clave de ScrapeCreators | 10.000 llamadas gratis y luego pago por uso |
| Xiaohongshu (RED) | Ten corriendo un plugin de navegador x-mcp con sesión iniciada o un servicio `xiaohongshu-mcp`, y activa la fuente con `--search xhs` por ejecución o con `INCLUDE_SOURCES=xiaohongshu` en `.env`; last30days prueba automáticamente `http://localhost:18060` y después `http://host.docker.internal:18060`, o usa `XIAOHONGSHU_API_BASE` para una URL propia | No hace falta clave de API de last30days; depende de tu servicio local de sesión de navegador |
| DripStack (boletines financieros premium) | Opcional: `--search dripstack` por ejecución, o `INCLUDE_SOURCES=dripstack` en `.env` | Sin clave; API de búsqueda pública y gratuita |
| Perplexity Sonar / Search API / Deep Research | Una clave de Perplexity, o una clave de OpenRouter como alternativa para Sonar | Pago por uso |
| Búsqueda web | Una clave de Brave Search | 2.000 consultas gratis al mes |
### Llavero de macOS (opcional)
En macOS puedes guardar las claves en el llavero del sistema en lugar de en un archivo `.env`. La skill las recoge automáticamente como la fuente de menor prioridad: si hay conflicto, siguen ganando los archivos `.env` y el entorno del proceso.
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
Las entradas se guardan con el nombre de servicio `last30days-<KEY>` para el usuario actual. En plataformas que no son Darwin el cargador no hace nada, así que para quienes usan Linux o Windows no cambia el comportamiento.
¿Ya tienes claves guardadas con otros nombres de servicio en el llavero? Define el mapeo no secreto `LAST30DAYS_KEYCHAIN_ALIASES` que se describe en [CONFIGURATION.md](CONFIGURATION.md#reusing-existing-macos-keychain-items), en lugar de copiar secretos.
Consulta [CONFIGURATION.md](CONFIGURATION.md) para ver la matriz completa de claves por fuente, el orden de prioridad de los proveedores de razonamiento y el de los backends de búsqueda web.
## Configuración
Dos cosas que seguramente querrás saber desde el primer día:
**Dónde se guardan los archivos de investigación.** `LAST30DAYS_MEMORY_DIR` apunta por defecto a `~/Documents/Last30Days/` (en Windows: `C:\Users\<you>\Documents\Last30Days\`). Puedes cambiarlo definiendo esa variable de entorno en tu shell con la ruta que quieras, o con `--save-dir <path>` en una ejecución concreta. Usa `--output <file>` cuando necesites el resultado renderizado en una ruta exacta, con el formato que elijas en `--emit`. Usa `--save-suffix=<name>` para mantener separadas varias variantes del mismo tema (por cliente, por ejemplo). Cada ejecución con `--save-dir` genera `<slug>-raw[-suffix].md`. Ejecuta `python3 skills/last30days/scripts/last30days.py --preflight` para revisar qué se va a escribir antes de lanzar una investigación.
**Salida estructurada para agentes y flujos de trabajo.** Pídele a `/last30days` JSON legible por máquina y obtendrás el perfil de agente estable y versionado. Para usar el motor directamente en scripts o en desarrollo, ejecuta `python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json`; añade `--json-profile=raw` solo si necesitas el volcado interno sin versionar de `Report`. Consulta la [referencia de campos de la exportación JSON y la política de versionado](docs/reference/json-export.md).
**Descubrimiento sin tema.** Pregunta `/last30days what's trending in AI agents?` para obtener un informe de descubrimiento ordenado, en lugar de investigar un tema que ya conoces. En un host con agente esto ejecuta el protocolo de tres comandos arbitrado por el host (el modelo propone los temas, filtra el ruido, puntúa lo que merece la pena y escribe los enfoques de contenido). Para usar el motor directamente en scripts o en cron, ejecuta `python3 skills/last30days/scripts/last30days.py --discover "AI agents"` (una sola pasada: nombres de tema deterministas, sin enfoques); añade `--emit=json` para el contrato de descubrimiento versionado. El descubrimiento es incompatible con un tema posicional y con `--drill`.
**Seguimiento de tendencias entre ejecuciones.** El modo por defecto genera una instantánea Markdown nueva en cada ejecución. Para ir acumulando hallazgos con el tiempo, añade `--store` y se guardarán en una base de datos SQLite; después usa [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) para ejecuciones programadas (con envío opcional por Slack o webhook cuando aparezcan hallazgos nuevos) y [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) para resúmenes diarios o semanales. El patrón de cadencia completo está en [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
**Una biblioteca de investigación a la que suscribirse.** Pídele a `/last30days` que genere el feed de tu biblioteca, o usa directamente `python3 skills/last30days/scripts/last30days.py library feed` para scripting y desarrollo. Convierte los informes guardados en un `index.html`, un `feed.xml` Atom local y páginas de informe legibles. Añade `--publish` solo cuando quieras alojar el índice HTML y las páginas de informe; publicar es una decisión explícita y por defecto es público. Para que el feed Atom se pueda seguir de verdad, aloja el directorio de salida generado en un alojamiento estático como GitHub Pages.
**Busca en todo lo que ya has investigado.** Pregunta `/last30days search my library for MCP servers` o `/last30days have I researched MCP servers before?`. Para usar el motor directamente, ejecuta `python3 skills/last30days/scripts/last30days.py library search "MCP servers"`. La búsqueda es offline y determinista: indexa de forma incremental los mismos informes guardados que usa el feed de la biblioteca, fusiona las coincidencias registradas en el almacén de cada ejecución y agrupa los resultados por tema y fecha. Las ejecuciones nuevas muestran además una sección compacta **From your library** («desde tu biblioteca») cuando una investigación anterior se solapa con el tema actual; define `LAST30DAYS_LIBRARY_CONTEXT=off` para desactivar ese contexto pasivo.
Los scripts envoltorio por cliente, los subreddits de categoría personalizados y el canal beta experimental para personalizaciones en curso también están documentados en [CONFIGURATION.md](CONFIGURATION.md).
## Escaparate: feeds de investigación de la comunidad
¿Has publicado con last30days una actualización periódica sobre IA, un seguimiento de mercado o una obsesión maravillosamente específica? Comparte la URL de tu biblioteca pública —o la URL de Atom, una vez alojado `feed.xml` en un alojamiento estático— en [el hilo de escaparate de la comunidad](https://github.com/mvanhorn/last30days-skill/issues/532). Los feeds de la comunidad se irán enlazando aquí a medida que sus autores los envíen; mientras tanto, el hilo es el punto de recogida.
## Cómo funciona
1. **Escribes un tema.** Una persona, una empresa, un producto, una tecnología, «X vs Y». Lo que sea.
2. **El agente averigua quién importa.** Encuentra las cuentas de X (incluidas las de fundadores), los repositorios de GitHub, los subreddits, los hashtags de TikTok y los canales de YouTube. Para «Kanye West» sabe que hay que mirar r/hiphopheads, @kanyewest y «bully review» en YouTube. Para «OpenClaw» resuelve openclaw/openclaw en GitHub y trae el número de estrellas en directo.
3. **Todas las fuentes se consultan en paralelo.** Expansión con varias consultas. Resultados puntuados por interacción, relevancia y frescura.
4. **La profundidad que no tiene nadie más.** Transcripciones completas de YouTube de vídeos de reacción. Los mejores comentarios de Reddit con su recuento de votos positivos. Los textos de los TikTok. Las cuotas de Polymarket. No solo títulos y enlaces.
5. **La misma historia, fusionada.** El Wireless Festival anunciado en Reddit, comentado en X y con los precios de las entradas en TikTok: un solo clúster, no tres entradas distintas.
6. **Sintetizado en un único informe.** Anclado en datos concretos. Citado por fuente. Ordenado según aquello con lo que la gente interactúa de verdad. No es «esto es lo que he encontrado», es «esto es lo que importa».
7. **Y después se convierte en tu experto.** Tras una sola ejecución, tu sesión de Claude sabe todo lo que sabe la comunidad. Haz preguntas de seguimiento. Pídele que escriba prompts, redacte correos, planifique viajes o diseñe arquitecturas, todo anclado en lo que es real ahora mismo.
## Lo que dice la gente
> «He encontrado una skill de Claude Code que investiga cualquier tema en Reddit, X, YouTube y HN de los últimos 30 días. Y luego te escribe los prompts. Antes de cada contenido que escribo, hacía esa búsqueda a mano en Reddit y X. Pestaña a pestaña. Hilo a hilo. Esa es la parte que se lleva 90 minutos. Esto la elimina.» —@itsjasonai
> «Esta única skill ha sustituido todo mi flujo de investigación. Le das un tema y rastrea Reddit, X y la web para sacar de qué está hablando la gente de verdad. Nada de entradas de blog viejas. Conversaciones reales de los últimos 30 días.» —@itswilsoncharles
> «5 de los 10 repos en tendencia hoy en GitHub son herramientas de Claude. El número 1: mvanhorn/last30days-skill» —@yieldhunter95
## Código abierto
Licencia MIT. Sin rastreo. Sin analíticas. Tu investigación se queda en tu máquina. Más de 2.700 pruebas.
Construido con Python 3.12+, yt-dlp, Node.js (cliente Bird incorporado para la búsqueda en X) y la API de ScrapeCreators. Arquitectura del motor v3 de [@j-sperling](https://github.com/j-sperling).
Consulta [CONTRIBUTING.md](CONTRIBUTING.md) para abrir un PR, [CONTRIBUTORS.md](CONTRIBUTORS.md) para la lista completa de colaboradores de la comunidad y [CHANGELOG.md](CHANGELOG.md) para el historial de versiones.
## Evolución de las estrellas
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
+383
View File
@@ -0,0 +1,383 @@
# /last30days
[English](README.md) | Français | [Deutsch](README.de.md) | [Español](README.es.md) | [Português (Brasil)](README.pt-BR.md) | [日本語](README.ja.md) | [简体中文](README.zh-CN.md)
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days - an AI agent-led search engine that searches people, not editors" />
</p>
<p align="center">
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
**Un moteur de recherche piloté par un agent IA, qui classe les résultats selon les upvotes, les likes et l'argent réel — pas selon des rédacteurs.**
Ce README décrit le pipeline v3 actuel. La spécification d'exécution de la skill se trouve dans [skills/last30days/SKILL.md](skills/last30days/SKILL.md), qui fait référence pour le comportement des commandes et de la configuration.
**Claude Code (recommandé — mises à jour automatiques via la marketplace) :**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex, Cursor, Copilot, Gemini CLI, ou l'un des 50+ hôtes [Agent Skills](https://agentskills.io) :**
```
npx skills add mvanhorn/last30days-skill -g
```
(`-g` installe la skill globalement pour votre utilisateur, donc disponible dans tous vos projets. Omettez ce flag pour une installation limitée au projet.)
D'autres options d'installation (claude.ai web, OpenClaw, manuelle) dans la section [Installation](#installation) ci-dessous.
Zéro configuration. Reddit, HN, Polymarket et GitHub fonctionnent immédiatement. Lancez la skill une fois : l'assistant de configuration débloque X, YouTube, TikTok, arXiv, Techmeme et d'autres sources en 30 secondes.
---
Les upvotes de Reddit. Les likes de X. Les transcriptions YouTube. L'engagement TikTok. Les cotes Polymarket, adossées à de l'argent réel et à des informations d'initiés. Ce sont des millions de personnes qui votent chaque jour avec leur attention et leur portefeuille. /last30days interroge tout cela en parallèle, classe les résultats selon ce avec quoi les gens interagissent vraiment, et un agent IA joue le rôle de juge pour en tirer un seul brief.
Google agrège des rédactions. /last30days interroge les gens.
Cette recherche est introuvable ailleurs, parce qu'aucune IA n'a accès à l'ensemble. Google ne touche ni aux commentaires Reddit ni aux posts X. ChatGPT a un accord avec Reddit mais ne sait chercher ni sur X ni sur TikTok. Gemini a YouTube mais pas Reddit. Claude n'a nativement accès à aucun des trois. Chaque plateforme est un jardin clos, avec son API, ses tokens et son authentification. Mais vous pouvez apporter vos propres clés et vos sessions de navigateur : d'un coup, un agent IA peut toutes les interroger en même temps, les comparer entre elles et vous dire ce qui compte vraiment.
C'est ça, le déclic. Pas un meilleur moteur de recherche. Une douzaine de plateformes cloisonnées, reliées par un agent.
```
/last30days Peter Steinberger
```
Vous avez une réunion demain. Vous cherchez la personne sur Google. Vous tombez sur son LinkedIn de 2023. /last30days vous donne ce qu'elle fait vraiment ce mois-ci : elle a rejoint OpenAI pour travailler sur Codex, elle conteste l'interdiction des agents tiers décrétée par Anthropic, elle a livré 23 PR avec un taux de merge de 85 %, elle construit « LobsterOS » pour piloter des agents entre appareils, et un fil r/ClaudeCode a atteint 569 upvotes en débattant de savoir si elle est un héros ou « insupportable ». Le tout dispersé entre des posts X, des fils Reddit, des transcriptions YouTube et des commits GitHub. Rien de tout ça n'était sur Google.
## Pourquoi ce projet existe
Je l'ai construit pour suivre le rythme de l'IA. Tout change chaque jour, et les passionnés de Reddit et de X sont toujours au courant les premiers. J'avais besoin de meilleurs prompts, et les données d'entraînement avaient toujours plusieurs mois de retard sur ce que la communauté avait déjà compris.
Mais c'est devenu quelque chose de plus large. Aujourd'hui je le lance avant un rendez-vous commercial, pour connaître la vérité des 30 derniers jours sur une entreprise. Avant une réunion, pour lire les tweets récents et les transcriptions de podcasts de mon interlocuteur. Avant un séjour à Disney World, pour savoir quelles attractions sont fermées et ce que la communauté pense de Genie+. Avant de construire quoi que ce soit, pour savoir sur quels problèmes les gens butent réellement.
Si vous rencontrez un PDG, avez-vous lu tous ses tweets et toutes ses transcriptions YouTube des 30 derniers jours ? Moi, oui.
## Les sources, classées par les gens
| Source | Ce que les gens vous disent |
|--------|--------------------------|
| **Reddit** | L'avis brut. Les meilleurs commentaires avec leur vrai nombre d'upvotes, gratuit, sans clé API. Les vraies opinions que Google enterre. |
| **X / Twitter** | La réaction à chaud, le fil d'expert, la première réaction à l'actualité. Premiers informés, premiers à débattre. |
| **YouTube** | L'analyse approfondie de 45 minutes. Des transcriptions complètes, fouillées pour en extraire les 5 phrases citables qui comptent. |
| **TikTok** | Le créateur qui touche 3,6 millions de personnes avec un angle que vous ne trouverez jamais sur Google. |
| **Instagram Reels** | Le regard des influenceurs, avec la transcription de ce qui est dit. Le signal de la culture visuelle. |
| **Hacker News** | Le consensus des développeurs. 825 points, 899 commentaires. Là où les gens techniques débattent vraiment. |
| **Polymarket** | Pas des opinions. Des cotes. Adossées à de l'argent réel. 96 % de probabilité sur des ventes d'album. 4 % sur une acquisition. |
| **GitHub** | Pour les personnes : rythme des PR, meilleurs dépôts par étoiles, notes de version. Pour les sujets : issues et discussions. |
| **Digg** | Des groupes d'articles sélectionnés depuis le classement AI 1000 de Digg (environ 1000 comptes IA à fort signal sur X), avec des citations attribuables intégrées (sans authentification X). Activé automatiquement quand `digg-pp-cli` est présent dans le PATH. |
| **arXiv** | Les articles scientifiques derrière le battage médiatique. La recherche publiée dans la fenêtre, gratuit, sans clé API. Activé automatiquement quand `arxiv-pp-cli` est présent dans le PATH (la configuration initiale l'installe). |
| **Techmeme** | La couche éditoriale de l'actu tech, restreinte à votre fenêtre de 30 jours. Gratuit, sans clé API. Activé automatiquement quand `techmeme-pp-cli` est présent dans le PATH (la configuration initiale l'installe). |
| **LinkedIn** | Le signal professionnel. Posts et articles, les articles étant pondérés comme signal fort. |
| **StockTwits** | Le sentiment des traders. S'active automatiquement quand votre sujet est un ticker ou une crypto. |
| **Threads** | La couche texte de l'après-Twitter. Les conversations des créateurs et des marques. |
| **Pinterest** | La découverte visuelle. Épingles, enregistrements et commentaires sur des produits et des idées. |
| **Xiaohongshu (RED)** | Les signaux chinois sur le lifestyle, les produits et les créateurs. À demander explicitement avec `--search xhs` quand un plugin de navigateur x-mcp connecté ou un service `xiaohongshu-mcp` tourne en local. |
| **Bluesky** | La couche sociale décentralisée. Les posts AT Protocol issus de la migration post-Twitter. |
| **Perplexity** | La synthèse Sonar sourcée, les résultats bruts de la Search API et Deep Research. |
| **Web** | La couverture éditoriale, les comparatifs de blogs. Un signal parmi d'autres, pas le seul. |
La communauté en ajoute sans cesse. Truth Social et d'autres sources de niche sont déjà dans le moteur, et d'autres arrivent.
Un fil Reddit à 1 500 upvotes est un signal plus fort qu'un billet de blog que personne n'a lu. Un TikTok à 3,6 millions de vues en dit plus sur ce qui compte culturellement qu'un communiqué de presse. Des cotes Polymarket adossées à 66 000 $ de volume sont plus difficiles à contester que l'intuition d'un éditorialiste.
La synthèse classe selon ce avec quoi de vraies personnes ont vraiment interagi. La pertinence sociale, pas la pertinence SEO.
## Ce que les gens en font vraiment
**Avant une réunion.** `/last30days Peter Steinberger` — a rejoint l'équipe Codex d'OpenAI, conteste l'interdiction des agents tiers décrétée par Anthropic, 23 PR mergées avec un taux de merge de 85 % sur GitHub, construit LobsterOS pour piloter des agents entre appareils. r/ClaudeCode : « Depuis la sortie d'OpenClaw, tout le monde savait que si vous passiez par autre chose que l'API, vous finiriez par être banni » (227 upvotes). Ça, ce n'est pas sur LinkedIn.
**Pour lire les signaux de recrutement.** `/last30days Listen Labs --hiring-signals` — les offres d'emploi et les pages carrières actuelles deviennent des preuves citées de changements de priorités : recrutements en sécurité entreprise, customer success, infrastructure ou expansion produit. Le rapport dit ce que le recrutement semble signaler, pas ce que la roadmap va livrer.
**Pour repérer un sujet avant son pic.** Demandez `/last30days what's exploding in AI agents?` et la skill bascule en mode découverte : le moteur balaie les listings de catégories Reddit, la une et les meilleures histoires de Hacker News, le flux AI 1000 de Digg, et X si vous êtes authentifié ; votre agent évalue les candidats (noms, filtrage du bruit, intérêt réel) et rédige des angles pour un podcast ou un article X ; vous obtenez ensuite 5 à 10 sujets classés par vélocité. Chaque résultat comprend des chiffres multi-sources, une étiquette de momentum et une commande `/last30days "<topic>"` prête à lancer.
**Quand quelque chose sort.** `/last30days Kanye West` — le Royaume-Uni a bloqué son visa, le Wireless Festival est annulé, les sponsors ont fui. Mais BULLY est entré n° 2 au Billboard. Fantano est revenu de son « Yay sabbatical » pour le chroniquer (653 000 vues). SoFi Homecoming a fait monter Lauryn Hill et Travis Scott sur scène pour 44 titres. Polymarket : « Kanye tweetera-t-il de nouveau ? » 86 % de oui. 23 fils Reddit, 17 vidéos YouTube, 86 000 upvotes.
**Pour comparer des outils.** `/last30days OpenClaw vs Hermes vs Paperclip` — « Ce ne sont pas des concurrents, ce sont des couches. » OpenClaw est la couche d'exécution (351 000 étoiles GitHub, en production), Hermes est le cerveau qui s'améliore tout seul (31 000 étoiles), Paperclip est l'organigramme (49 000 étoiles). Nombres d'étoiles récupérés en direct via l'API GitHub, pas repris de billets de blog périmés. Tableau comparatif avec architecture, mémoire, sécurité et cas d'usage idéal. Selon @IMJustinBrooke : « OpenClaw = Salamèche, Hermes = Dracaufeu. »
**Pour comprendre le monde.** `/last30days Iran vs USA` — 38e jour de guerre. Ultimatum de Trump, fixé à mardi, pour que l'Iran rouvre le détroit d'Ormuz. Deux avions de combat américains abattus. Le pétrole à 126 $ le baril. L'AIE parle de « la plus grande perturbation d'approvisionnement de l'histoire du marché pétrolier mondial ». Polymarket : cessez-le-feu avant le 31 décembre à 74 %. 27 posts X, 10 vidéos YouTube, 20 marchés de prédiction.
**Avant un voyage.** `/last30days Universal Epic Universe` — l'extension est déjà en construction. Permis « Project 680 » déposé. Spectacle de feux d'artifice confirmé par les travaux mais toujours pas annoncé. Temps d'attente : Mine-Cart Madness à 148 minutes en moyenne. Toujours pas de pass annuel, et les habitants s'agacent. Stardust Racers fermé pour rénovation jusqu'au 5 avril.
**Pour apprendre vite.** `/last30days Nano Banana Pro prompting` — les prompts structurés en JSON remplacent l'empilement de tags. Le format imbriqué de @pictsbyai évite le « concept bleeding ». Mieux vaut éditer que régénérer. Et ensuite, la skill vous écrit un prompt de production en appliquant exactement ce que la communauté a validé.
## Nouveautés
Depuis l'annonce de la v3.3 en mai, et jusqu'à la v3.11.1 (juillet 2026) : 175 PR mergées — dont 122 venant de 52 contributeurs de la communauté — réparties sur 15 versions. Voici ce qui a atterri.
### Citoyen de première classe sur OpenAI Codex
/last30days est désormais un plugin Codex natif avec configuration guidée : pas un portage, un vrai citoyen de première classe. Les citations tiennent compte du rendu, ce qui fait que la sortie Codex se lit comme un brief et non comme une soupe d'URL (#694), et le même moteur tourne sur Claude Code, Cursor, Copilot, Gemini CLI, Claude Desktop, OpenClaw et 50+ hôtes Agent Skills. Manifeste du plugin Codex par [@rfoust](https://github.com/rfoust) (#686), correctif d'authentification Codex par [@tmchow](https://github.com/tmchow) (#698).
### arXiv, Techmeme et Digg — gratuits, sans clé API
arXiv apporte les articles scientifiques derrière le battage médiatique et Techmeme la couche éditoriale de l'actu tech — gratuits, sans aucune clé, et la configuration initiale installe leurs CLI pour qu'ils s'activent tout seuls (#709). Les groupes d'articles AI 1000 de Digg arrivent de la même façon, sans authentification X : la configuration installe pour vous la CLI Digg gratuite (#590). Trustpilot est disponible en option pour la recherche sur les marques grand public.
### Reddit gratuit, avec de vrais scores et les meilleurs commentaires
L'API .json publique de Reddit a disparu ; la voie gratuite est revenue plus forte. Flux RSS sans clé et scraping de shreddit (#457), découverte de subreddits dédiés avec de vrais décomptes d'upvotes via arctic-shift (#696), et un seuil de pertinence pour qu'un post viral hors sujet ne détourne pas votre brief (#488, merci [@rzachsmith](https://github.com/rzachsmith)). Pas de clé API. De vrais scores. Les meilleurs commentaires inclus.
### Les meilleurs commentaires dans chaque brief
Les commentaires sont maintenant une couche activée par défaut sur toutes les sources : commentaires Instagram avec une diversité fondée sur le rang, pour que cinq avis tranchés ne viennent pas tous du même post (#751), commentaires YouTube plus une récupération de transcription via ScrapeCreators quand yt-dlp échoue (#637), et commentaires plébiscités par la communauté intégrés au scoring Best Takes, pour que les meilleures punchlines survivent au classement (#592, #608).
### Une seule commande doctor
Demandez un diagnostic : doctor teste chaque source, puis prescrit les correctifs exacts — quelle clé manque, quelle CLI est absente du PATH, quel cookie a expiré (#753). Fini de deviner pourquoi X est revenu à vide.
### La recherche X, reconstruite
Le pipeline X a été repensé de fond en comble : des voies FROM et ABOUT pour que les posts d'une personne et la conversation à son sujet soient classés tous les deux (#610), désambiguïsation des sous-requêtes selon la personne visée (#611), vérification de la paternité des posts avec classement par signaux d'interaction (#613), et une source X unique avec bascule automatique entre backends (#622). Plus un `--diagnose` honnête qui teste vraiment l'authentification (#609).
### De nouvelles sources
LinkedIn via ScrapeCreators, avec les articles comme signal fort ([@ravstr](https://github.com/ravstr), #702). StockTwits s'active automatiquement sur les sujets liés aux tickers et aux cryptos ([@wtiwana](https://github.com/wtiwana), #658). Perplexity a gagné des modes API directs et Deep Research en asynchrone ([@sk-holmes](https://github.com/sk-holmes), #629).
### Durci par la communauté
La vague sécurité est presque entièrement le fait de la communauté : correctifs XSS stocké dans le rendu HTML ([@iliaal](https://github.com/iliaal), [@aaronjmars](https://github.com/aaronjmars)), fichiers temporaires de cookies verrouillés, CI durcie contre les attaques de chaîne d'approvisionnement avec OpenSSF Scorecard et attestation de provenance des builds ([@shaanmajid](https://github.com/shaanmajid), [@hammadxcm](https://github.com/hammadxcm), [@aniruddh909](https://github.com/aniruddh909)), analyses Semgrep et OSV-Scanner plus un contrôle de revue des dépendances sur chaque PR ([@23241a6749](https://github.com/23241a6749)), un seuil plancher de couverture de tests instauré à 60 % puis relevé à 84 % ([@gourab5139014](https://github.com/gourab5139014)), et un audit de sécurité Hermes désormais sans aucune finding CRITICAL (#768).
### Une portée plus large
L'hébreu et les langues non latines ([@dudyme](https://github.com/dudyme)). Une tokenisation adaptée au CJK pour les sources chinoises ([@An-idd](https://github.com/An-idd)). Une vague d'améliorations sur Windows. L'extraction des cookies sur toute la famille Chromium — Brave, Edge, Vivaldi, Opera, Arc ([@andrey-esipov](https://github.com/andrey-esipov)) — plus le trousseau macOS et pass(1) sous Linux comme sources d'identifiants. Le retour en arrière historique avec `--as-of` ([@chiyi-creator](https://github.com/chiyi-creator)). L'installation automatique de Python 3.12 via uv ([@buntysomroy](https://github.com/buntysomroy)). `--hiring-signals` pour lire les pages emploi d'une entreprise. Les écarts de watchlist d'une exécution à l'autre.
### Toujours livré depuis la v3
Les fondations de la v3 sont toujours là : le cerveau de pré-recherche qui identifie les bons comptes, subreddits et hashtags avant le moindre appel API (construit par [@j-sperling](https://github.com/j-sperling)) ; le scoring Best Takes, qui prend en compte l'humour et la viralité en plus de la pertinence ; la fusion de clusters entre sources ; les comparaisons en une seule passe (« CLI vs MCP » en 3 minutes, pas 12) ; les comparaisons `--competitors` découvertes automatiquement ; le mode personne de GitHub (`--github-user=steipete`) ; le mode ELI5 (« eli5 on » après n'importe quelle exécution) ; et des briefs HTML autonomes et partageables (`--emit=html`). Les options de configuration sont détaillées dans [CONFIGURATION.md](CONFIGURATION.md).
## Installation
| Environnement | Installation | Mises à jour |
|---------|---------|---------|
| **Claude Code** (recommandé) | `/plugin marketplace add mvanhorn/last30days-skill` | Automatiques via la marketplace, ou `claude plugin update last30days@last30days-skill` |
| **Grok** (xAI Build CLI) | `grok plugin marketplace add mvanhorn/last30days-skill` puis `grok plugin install last30days` | `grok plugin update last30days` |
| **Codex, Cursor, Copilot, Gemini CLI, ou l'un des 50+ hôtes [Agent Skills](https://agentskills.io)** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **claude.ai** (web) | [Téléchargez `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) et envoyez-le via claude.ai > Customize > Skills > + > Create skill > Upload a skill | Retélécharger et renvoyer |
| **Claude Desktop** | [Téléchargez le `.mcpb` de votre plateforme](https://github.com/mvanhorn/last30days-skill/releases/latest) et glissez-le dans Settings > Extensions | Retélécharger et glisser le nouveau bundle |
| **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code (recommandé)
```
/plugin marketplace add mvanhorn/last30days-skill
```
Recommandé parce que la marketplace Claude Code gère les mises à jour pour vous : le cache du plugin est versionné et se rafraîchit automatiquement à chaque nouvelle version publiée. Lancez `claude plugin update last30days@last30days-skill` pour forcer une vérification.
Si vous préférez passer par le chemin d'installation Agent Skills sur Claude Code, c'est également pris en charge :
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
Le plugin natif et l'installation `npx skills` peuvent coexister. Attention : Claude Code ne déduplique pas entre méthodes d'installation. Si le plugin de la marketplace et la copie `npx skills` sont actifs tous les deux, `/last30days` apparaîtra en double. Utilisez une seule méthode d'installation par machine.
### Grok (xAI Build CLI)
[Grok Build](https://docs.x.ai/build/features/skills-plugins-marketplaces) (`grok`) installe last30days comme plugin natif. L'installation directe suit le dépôt :
```bash
grok plugin install mvanhorn/last30days-skill
```
Ou ajoutez ce dépôt comme source de marketplace, puis installez par nom de plugin :
```bash
grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
Ajoutez `--trust` pour sauter la confirmation d'installation. Mettez à jour avec `grok plugin update last30days`. Grok lit aussi les manifestes Claude Code par compatibilité ; la paire native `.grok-plugin/` reste la voie principale, et c'est elle que pointe une entrée officielle dans la [marketplace xAI](https://github.com/xai-org/plugin-marketplace). `npx skills add` reste une solution de repli valable, tous hôtes confondus.
### Codex, Cursor, Copilot, Gemini CLI et autres hôtes Agent Skills
Installez via la CLI ouverte [Agent Skills](https://agentskills.io) — elle prend en charge 50+ hôtes, dont `codex`, `cursor`, `github-copilot`, `gemini-cli`, `claude-code`, `windsurf`, `cline`, `continue`, `roo`, `aider-desk`, `opencode`, `goose` et d'autres (liste complète sur le [dépôt vercel-labs/skills](https://github.com/vercel-labs/skills)).
```bash
npx skills add mvanhorn/last30days-skill -g
```
Le flag `-g` (global) installe dans votre répertoire utilisateur, ce qui rend la skill disponible dans tous vos projets. Sans `-g`, `npx skills` installe localement dans `./.skills/` (versionné avec le dépôt). Pour un outil qui sert à explorer le monde entier, c'est bien l'installation globale que vous voulez.
Codex desktop et les autres hôtes qui travaillent au niveau du dossier fonctionnent aussi bien dans un dossier ordinaire que dans un dépôt Git. Avant la première recherche, demandez à l'agent hôte de lancer le `scripts/last30days.py --preflight` fourni depuis le répertoire de la skill chargée ; dans un clone du dépôt source, la commande équivalente est `python3 skills/last30days/scripts/last30days.py --preflight`. Elle affiche l'origine de la configuration, le plan de lecture des cookies de navigateur, les fichiers qui seront écrits, les commandes optionnelles et la configuration projet ignorée — sans lire de cookies, sans écrire de fichier et sans lancer de recherche.
Par défaut, l'installation cible l'hôte que `npx skills` détecte. Pour en viser un en particulier (ou plusieurs) :
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
Pour mettre à jour plus tard :
```bash
npx skills update last30days -g
```
Ou mettez à jour tout ce que vous avez installé globalement via `npx skills` :
```bash
npx skills update -g
```
Listez et désinstallez avec `npx skills list -g` et `npx skills remove last30days -g`.
### claude.ai (web)
1. [Téléchargez `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) depuis la dernière version publiée
2. Allez sur [claude.ai > Customize > Skills](https://claude.ai/customize/skills)
3. Cliquez sur le bouton `+` du panneau Skills, puis sur `Create skill` > `Upload a skill`, et déposez le fichier
Activez d'abord « Code execution and file creation » dans Capabilities — sans cela, les skills ne s'exécutent pas.
### Claude Desktop
Claude Desktop installe `/last30days` comme serveur MCP via un bundle `.mcpb` (un paquet Model Context Protocol en un clic).
1. Ouvrez la [dernière version publiée](https://github.com/mvanhorn/last30days-skill/releases/latest) et téléchargez le `.mcpb` correspondant à votre plateforme :
- macOS Apple Silicon : `last30days-pp-mcp-darwin-arm64.mcpb`
- macOS Intel : `last30days-pp-mcp-darwin-amd64.mcpb`
- Linux x86_64 : `last30days-pp-mcp-linux-amd64.mcpb`
2. Ouvrez Claude Desktop, allez dans Settings > Extensions et glissez-y le fichier.
3. Quand l'application vous les demande, collez les clés API des sources que vous voulez activer. Tous les champs sont facultatifs : si vous les ignorez tous, le moteur se rabat sur le mode web uniquement. Les clés sont stockées dans le trousseau de votre système.
4. Redémarrez Claude Desktop. Demandez à Claude de « faire des recherches sur Peter Steinberger », ou sur n'importe quel sujet, et il appellera l'outil `research`.
**Prérequis côté hôte :** Python 3.12+ dans le PATH. Le bundle embarque le code du moteur mais utilise votre interpréteur Python local. Installez-le depuis [python.org](https://www.python.org/downloads/) sous Windows ; macOS et la plupart des distributions Linux fournissent déjà une version compatible.
**Les clés ne sont pas partagées avec la skill Claude Code.** Claude Desktop et Claude Code maintiennent délibérément des stockages d'identifiants distincts. Si vous avez déjà configuré `~/.config/last30days/.env` pour la skill Claude Code, il faudra ressaisir les mêmes clés ici, une fois.
La prise en charge de Windows est reportée le temps de régler les points d'entrée par plateforme dans le manifeste ; le suivi se fait dans une issue dédiée.
### OpenClaw
```bash
clawhub install last30days-official
```
Pour les workflows d'action sur X/Twitter en dehors des recherches `/last30days`
publier des tweets ou des réponses, exporter des abonnés, gérer les médias,
surveiller des comptes, organiser des tirages au sort — utilisez
[TweetClaw](https://github.com/Xquik-dev/tweetclaw), le plugin OpenClaw
complémentaire. TweetClaw est maintenu par Xquik-dev et n'est mentionné que comme
option complémentaire : ce n'est ni une dépendance ni une recommandation de last30days.
### Installation manuelle (développeurs)
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
```
Le lien symbolique garde l'installation synchronisée avec votre copie de travail au fil de vos modifications — inutile de recopier quoi que ce soit. Pour `claude.ai`, construisez le fichier `.skill` depuis les sources : `bash skills/last30days/scripts/build-skill.sh` produit `dist/last30days.skill`.
Reddit (avec les commentaires), Hacker News, Polymarket et GitHub fonctionnent immédiatement. Zéro configuration. Lancez `/last30days` une fois : l'assistant de configuration débloque d'autres sources en 30 secondes, dont les CLI gratuites arXiv et Techmeme.
## Apportez vos propres clés
Ces plateformes n'ont aucune relation entre elles. X ignore ce que pense Reddit. YouTube ne voit pas TikTok. Mais vous pouvez apporter vos propres clés API et vos tokens de navigateur, et vous avez soudain accès à toutes en même temps.
| Sources | Ce qu'il vous faut | Coût |
|---------|---------------|------|
| Reddit (avec les commentaires) + HN + Polymarket + GitHub + StockTwits | Rien | Gratuit |
| arXiv + Techmeme | Des CLI gratuites, installées automatiquement à la configuration initiale | Gratuit |
| X / Twitter | Connectez-vous à x.com dans n'importe quel navigateur, ou définissez `XQUIK_API_KEY` / `XAI_API_KEY` | Les cookies de navigateur sont gratuits ; les clés dépendent du fournisseur |
| YouTube | `brew install yt-dlp` | Gratuit |
| Bluesky | Un mot de passe d'application depuis bsky.app | Gratuit |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + commentaires YouTube | Une clé ScrapeCreators | 10 000 appels gratuits, puis paiement à l'usage |
| Xiaohongshu (RED) | Faites tourner un plugin de navigateur x-mcp connecté ou un service `xiaohongshu-mcp`, puis activez la source avec `--search xhs` pour une exécution ou `INCLUDE_SOURCES=xiaohongshu` dans `.env` ; last30days teste automatiquement `http://localhost:18060` puis `http://host.docker.internal:18060`, ou utilisez `XIAOHONGSHU_API_BASE` pour une URL personnalisée | Aucune clé API last30days ; dépend de votre service local de session de navigateur |
| DripStack (newsletters financières premium) | Sur activation : `--search dripstack` pour une exécution, ou `INCLUDE_SOURCES=dripstack` dans `.env` | Aucune clé ; API de recherche publique et gratuite |
| Perplexity Sonar / Search API / Deep Research | Une clé Perplexity, ou une clé OpenRouter en repli pour Sonar | Paiement à l'usage |
| Recherche web | Une clé Brave Search | 2 000 requêtes gratuites par mois |
### Trousseau macOS (facultatif)
Sous macOS, vous pouvez stocker vos clés dans le trousseau système plutôt que dans un fichier `.env`. La skill les récupère automatiquement, comme source de plus faible priorité : en cas de conflit, les fichiers `.env` et les variables d'environnement du processus l'emportent toujours.
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
Les entrées sont enregistrées sous le nom de service `last30days-<KEY>` pour l'utilisateur courant. Sur les plateformes non Darwin, le chargeur ne fait rien : aucun changement de comportement pour les utilisateurs Linux et Windows.
Vous avez déjà des clés sous d'autres noms de service dans le trousseau ? Définissez la correspondance non secrète `LAST30DAYS_KEYCHAIN_ALIASES` décrite dans [CONFIGURATION.md](CONFIGURATION.md#reusing-existing-macos-keychain-items), plutôt que de recopier vos secrets.
Voir [CONFIGURATION.md](CONFIGURATION.md) pour la matrice complète des clés par source, l'ordre de priorité des fournisseurs de raisonnement et celui des backends de recherche web.
## Configuration
Deux choses que vous voudrez sans doute savoir dès le premier jour :
**Où sont enregistrés les fichiers de recherche.** `LAST30DAYS_MEMORY_DIR` vaut par défaut `~/Documents/Last30Days/` (sous Windows : `C:\Users\<you>\Documents\Last30Days\`). Redéfinissez cette variable d'environnement dans votre shell pour pointer ailleurs, ou passez `--save-dir <path>` sur une exécution. Utilisez `--output <file>` quand vous voulez le résultat rendu à un chemin précis, dans le format choisi par `--emit`. Utilisez `--save-suffix=<name>` pour garder séparées plusieurs variantes d'un même sujet (par client, par exemple). Chaque exécution avec `--save-dir` produit `<slug>-raw[-suffix].md`. Lancez `python3 skills/last30days/scripts/last30days.py --preflight` pour vérifier les écritures prévues avant une recherche.
**Sortie structurée pour les agents et les workflows.** Demandez à `/last30days` du JSON exploitable par une machine pour obtenir le profil d'agent stable et versionné. Pour un usage direct du moteur en script ou en développement, lancez `python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json` ; n'ajoutez `--json-profile=raw` que si vous avez besoin du dump interne non versionné de `Report`. Voir la [référence des champs de l'export JSON et la politique de versionnement](docs/reference/json-export.md).
**Découverte sans sujet imposé.** Demandez `/last30days what's trending in AI agents?` pour obtenir un brief de découverte classé, au lieu de rechercher un sujet que vous connaissez déjà. Sur un hôte agentique, cela déclenche le protocole en trois commandes arbitré par l'hôte (le modèle propose les sujets, écarte le bruit, note leur intérêt et rédige les angles éditoriaux). Pour un usage direct du moteur en script ou en cron, lancez `python3 skills/last30days/scripts/last30days.py --discover "AI agents"` (en une passe : noms de sujets déterministes, sans angles) ; ajoutez `--emit=json` pour le contrat de découverte versionné. La découverte est incompatible avec un sujet positionnel et avec `--drill`.
**Suivi des tendances d'une exécution à l'autre.** Le mode par défaut produit un instantané Markdown à chaque exécution. Pour accumuler les résultats dans le temps, ajoutez `--store` afin de les conserver dans une base SQLite, puis utilisez [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) pour les exécutions planifiées (avec envoi facultatif sur Slack ou via webhook à chaque nouveau résultat) et [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) pour des synthèses quotidiennes ou hebdomadaires. Le schéma de cadence complet est dans [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
**Une bibliothèque de recherche à laquelle s'abonner.** Demandez à `/last30days` de générer le flux de votre bibliothèque, ou utilisez directement `python3 skills/last30days/scripts/last30days.py library feed` pour vos scripts et vos développements. La commande transforme les briefs enregistrés en un `index.html`, un `feed.xml` Atom local et des pages de brief lisibles. N'ajoutez `--publish` que si vous voulez héberger l'index HTML et les pages de brief ; la publication est un choix explicite, et publique par défaut. Pour rendre le flux Atom réellement abonnable, hébergez le répertoire de sortie généré sur un hébergeur statique comme GitHub Pages.
**Cherchez dans tout ce que vous avez déjà recherché.** Demandez `/last30days search my library for MCP servers` ou `/last30days have I researched MCP servers before?`. Pour un usage direct du moteur, lancez `python3 skills/last30days/scripts/last30days.py library search "MCP servers"`. La recherche est hors ligne et déterministe : elle indexe au fil de l'eau les mêmes briefs enregistrés que le flux de bibliothèque, y fusionne les occurrences correspondantes conservées dans le store, et regroupe les résultats par sujet et par date. Les nouvelles exécutions affichent aussi une section compacte **From your library** (« depuis votre bibliothèque ») quand des recherches antérieures recoupent le sujet en cours ; définissez `LAST30DAYS_LIBRARY_CONTEXT=off` pour désactiver ce contexte passif.
Les scripts d'encapsulation par client, les subreddits de catégorie personnalisés et le canal bêta expérimental pour les personnalisations en cours sont également documentés dans [CONFIGURATION.md](CONFIGURATION.md).
## Vitrine : les flux de recherche de la communauté
Vous avez publié une veille IA récurrente, un suivi de marché ou une obsession merveilleusement pointue avec last30days ? Partagez l'URL de votre bibliothèque publique — ou l'URL Atom une fois `feed.xml` hébergé sur un hébergeur statique — dans [le fil vitrine de la communauté](https://github.com/mvanhorn/last30days-skill/issues/532). Les flux communautaires seront listés ici au fur et à mesure que leurs auteurs les proposeront ; en attendant, le fil sert de point de collecte.
## Comment ça marche
1. **Vous saisissez un sujet.** Une personne, une entreprise, un produit, une technologie, « X vs Y ». N'importe quoi.
2. **L'agent identifie qui compte.** Il trouve les comptes X (y compris ceux des fondateurs), les dépôts GitHub, les subreddits, les hashtags TikTok, les chaînes YouTube. Pour « Kanye West », il sait qu'il faut r/hiphopheads, @kanyewest et « bully review » sur YouTube. Pour « OpenClaw », il identifie openclaw/openclaw sur GitHub et récupère le nombre d'étoiles en direct.
3. **Toutes les sources interrogées en parallèle.** Expansion multi-requêtes. Résultats classés selon l'engagement, la pertinence et la fraîcheur.
4. **Une profondeur que personne d'autre n'a.** Les transcriptions YouTube complètes des vidéos de réaction. Les meilleurs commentaires Reddit avec leur nombre d'upvotes. Les légendes TikTok. Les cotes Polymarket. Pas seulement des titres et des liens.
5. **Une même histoire, fusionnée.** Le Wireless Festival annoncé sur Reddit, commenté sur X, avec le prix des billets sur TikTok : un seul cluster, pas trois entrées distinctes.
6. **Synthétisé en un seul brief.** Ancré dans des données précises. Sourcé. Classé selon ce avec quoi les gens interagissent vraiment. Pas « voilà ce que j'ai trouvé », mais « voilà ce qui compte ».
7. **Ensuite, la skill devient votre experte.** Après une seule exécution, votre session Claude sait tout ce que sait la communauté. Posez vos questions de suivi. Faites-lui écrire des prompts, rédiger des e-mails, planifier des voyages, concevoir des architectures — le tout ancré dans la réalité du moment.
## Ce que les gens en disent
> « J'ai trouvé une skill Claude Code qui fait des recherches sur n'importe quel sujet à travers Reddit, X, YouTube et HN sur les 30 derniers jours. Et elle écrit les prompts à votre place. Avant chaque contenu que j'écris, je faisais ces recherches à la main sur Reddit et X. Onglet par onglet. Fil par fil. C'est la partie qui prend 90 minutes. Elle disparaît. » — @itsjasonai
> « Cette seule skill a remplacé tout mon workflow de recherche. Vous lui donnez un sujet, elle récupère sur Reddit, X et le web ce dont les gens parlent vraiment. Pas de vieux billets de blog. De vraies conversations des 30 derniers jours. » — @itswilsoncharles
> « 5 des 10 dépôts tendance du jour sur GitHub sont des outils Claude. N° 1 : mvanhorn/last30days-skill » — @yieldhunter95
## Open source
Licence MIT. Aucun tracking. Aucune analytics. Vos recherches restent sur votre machine. Plus de 2 700 tests.
Construit avec Python 3.12+, yt-dlp, Node.js (client Bird intégré pour la recherche X) et l'API ScrapeCreators. Architecture du moteur v3 par [@j-sperling](https://github.com/j-sperling).
Voir [CONTRIBUTING.md](CONTRIBUTING.md) pour ouvrir une PR, [CONTRIBUTORS.md](CONTRIBUTORS.md) pour la liste complète des contributeurs de la communauté, et [CHANGELOG.md](CHANGELOG.md) pour l'historique des versions.
## Évolution des étoiles
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
+382
View File
@@ -0,0 +1,382 @@
# /last30days
[English](README.md) | [Français](README.fr.md) | [Deutsch](README.de.md) | [Español](README.es.md) | [Português (Brasil)](README.pt-BR.md) | 日本語 | [简体中文](README.zh-CN.md)
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days - an AI agent-led search engine that searches people, not editors" />
</p>
<p align="center">
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
**編集者ではなく、アップボート・いいね・実際に動いたお金でランク付けする、AIエージェント主導の検索エンジンです。**
このREADMEは現行のv3パイプラインについて説明しています。実行時のスキル仕様は [skills/last30days/SKILL.md](skills/last30days/SKILL.md) にあり、コマンドとセットアップの挙動についてはそちらが最新かつ正式なものです。
**Claude Code(推奨 — マーケットプレイス経由で自動更新):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex、Cursor、Copilot、Gemini CLI、その他50以上の [Agent Skills](https://agentskills.io) ホスト:**
```
npx skills add mvanhorn/last30days-skill -g
```
(`-g` を付けるとユーザー単位でグローバルにインストールされ、すべてのプロジェクトで使えます。プロジェクト単位に限定したい場合はこのフラグを外してください。)
その他のインストール方法(claude.aiのウェブ版、OpenClaw、手動)は下の [インストール](#インストール) セクションにあります。
設定は不要です。Reddit、HN、Polymarket、GitHub はすぐに使えます。一度実行すれば、セットアップウィザードが30秒で X、YouTube、TikTok、arXiv、Techmeme などを有効にします。
---
Reddit のアップボート。X のいいね。YouTube の文字起こし。TikTok のエンゲージメント。実際のお金とインサイダー情報に裏打ちされた Polymarket のオッズ。つまり、毎日何百万人もの人が自分の注意と財布で投票しているということです。/last30days はそのすべてを並行して検索し、実際に人々が反応したかどうかでスコアを付け、AIエージェントが判定役となって1本のブリーフにまとめます。
Google は編集者を束ねます。/last30days は人を検索します。
この検索は他のどこでも手に入りません。単独のAIがすべてにアクセスできないからです。Google の検索は Reddit のコメントにも X の投稿にも届きません。ChatGPT は Reddit と提携していますが、X も TikTok も検索できません。Gemini には YouTube がありますが Reddit がありません。Claude はそのどれもネイティブには持っていません。どのプラットフォームも、独自のAPI・独自のトークン・独自の認証を備えた閉じた庭です。しかし自分のキーとブラウザセッションを持ち込めば、AIエージェントが一度にすべてを検索し、互いに突き合わせてスコアを付け、本当に重要なことを教えてくれるようになります。
そこが突破口です。優れた検索エンジンが1つ増えるという話ではありません。断絶していた十数のプラットフォームを、エージェントが橋渡しするのです。
```
/last30days Peter Steinberger
```
明日、打ち合わせがあるとします。その人を Google で調べると、出てくるのは2023年の LinkedIn です。/last30days なら、その人が今月実際にやっていることが分かります。Codex に取り組むため OpenAI に参加し、サードパーティ製エージェントを禁じた Anthropic の方針と争い、23本のPRをマージ率85%で送り、デバイスをまたいでエージェントを操作する「LobsterOS」を作っていて、さらに r/ClaudeCode では彼が英雄なのか「鼻につく」のかという議論が569アップボートを集めている。それらは X の投稿、Reddit のスレッド、YouTube の文字起こし、GitHub のコミットに散らばっていて、どれも Google には出てきませんでした。
## なぜ作ったのか
AIの動きに追いつくために作りました。何もかもが日々変わり、Reddit と X の濃い人たちがいつも真っ先に把握しています。もっと良いプロンプトが必要でしたが、学習データはコミュニティがすでに突き止めたことより常に数か月遅れていました。
ただ、そこからもっと大きなものになりました。今では商談の前に走らせて、その会社について直近30日間の実情を押さえます。打ち合わせの前には、相手の最近のツイートやポッドキャストの文字起こしを読むために。ディズニー・ワールドに行く前には、どのアトラクションが休止中で、Genie+ についてコミュニティが何と言っているかを知るために。何かを作り始める前には、人々が実際にどんな問題にぶつかっているかを知るために。
CEOと会うとして、直近30日間のツイートと YouTube の文字起こしを全部読んできましたか。私は読んでいます。
## 人々がスコアを付けた情報源
| 情報源 | 人々が教えてくれること |
|--------|--------------------------|
| **Reddit** | フィルターのかかっていない本音。実際のアップボート数付きのトップコメントが、無料・APIキーなしで手に入ります。Google が埋もれさせてしまう本当の意見です。 |
| **X / Twitter** | 勢いのある一言、専門家のスレッド、速報への最初の反応。誰よりも早く知り、誰よりも早く議論が始まります。 |
| **YouTube** | 45分の掘り下げ。文字起こし全文を検索し、引用に値する5つの文だけを取り出します。 |
| **TikTok** | Google では絶対に見つからない切り口で360万人に届いているクリエイター。 |
| **Instagram Reels** | 話した内容の文字起こし付きで届く、インフルエンサーの視点。ビジュアル文化のシグナルです。 |
| **Hacker News** | 開発者の総意。825ポイント、899コメント。技術寄りの人たちが本気で議論している場所です。 |
| **Polymarket** | 意見ではなく、オッズ。実際のお金が裏付けています。アルバムの売上に96%、買収に4%といった具合です。 |
| **GitHub** | 人物について: PRの勢い、スター数の多いリポジトリ、リリースノート。トピックについて: Issue と Discussion。 |
| **Digg** | Digg の AI 1000 リーダーボード(X 上でシグナルの強いAI関連アカウント約1000件)から集めたストーリークラスター。出典をたどれるインライン引用付きで、X の認証は不要です。`digg-pp-cli` が PATH にあると自動的に有効になります。 |
| **arXiv** | 話題の裏側にある論文。対象期間に出た新しい研究が、無料・APIキーなしで手に入ります。`arxiv-pp-cli` が PATH にあると自動的に有効になります(初回セットアップでインストールされます)。 |
| **Techmeme** | テックニュースの編集レイヤーを、対象の30日間に絞って取得します。無料・APIキーなし。`techmeme-pp-cli` が PATH にあると自動的に有効になります(初回セットアップでインストールされます)。 |
| **LinkedIn** | ビジネス面のシグナル。投稿と記事を拾い、記事は強いシグナルとして重み付けします。 |
| **StockTwits** | トレーダーの温度感。調べる対象が銘柄コードや暗号資産のときに自動で有効になります。 |
| **Threads** | Twitter 以後のテキストの層。クリエイターやブランドの会話です。 |
| **Pinterest** | ビジュアル起点の発見。プロダクトやアイデアに対するピン・保存・コメント。 |
| **Xiaohongshu(RED)** | 中国のライフスタイル・プロダクト・クリエイターのシグナル。ログイン済みの x-mcp ブラウザプラグイン、または `xiaohongshu-mcp` サービスがローカルで動いているときに、`--search xhs` で明示的に指定して使います。 |
| **Bluesky** | 分散型のソーシャル層。Twitter 以後の移住で生まれた AT Protocol の投稿です。 |
| **Perplexity** | 根拠付きの Sonar による統合、Search API の生の結果、そして Deep Research。 |
| **Web** | 編集記事や、ブログの比較記事。数あるシグナルの1つであって、唯一のものではありません。 |
コミュニティが今も情報源を増やし続けています。Truth Social をはじめとするニッチな情報源もすでにエンジンに入っていて、さらに追加予定です。
1,500アップボートの Reddit スレッドは、誰にも読まれなかったブログ記事よりも強いシグナルです。360万回再生の TikTok は、プレスリリースよりも「今、文化的に何が効いているか」を語ります。6.6万ドルの出来高に裏打ちされた Polymarket のオッズは、評論家の当て推量よりも反論しにくいものです。
この統合処理は、実在の人々が実際に反応したかどうかで順位を付けます。SEO上の関連性ではなく、社会的な関連性です。
## みんなが実際に使っている場面
**打ち合わせの前に。** `/last30days Peter Steinberger` — OpenAI の Codex チームに参加、サードパーティ製エージェントを禁じた Anthropic の方針と対立、GitHub で23本のPRをマージ率85%でマージ、デバイスをまたいでエージェントを操作する LobsterOS を開発中。r/ClaudeCode では「OpenClaw が出てからずっと、API 以外の経路で動かせばいずれBANされると広く知られていた」(227アップボート)。これは LinkedIn には載っていません。
**採用シグナルを読むために。** `/last30days Listen Labs --hiring-signals` — 現在の求人ページやキャリアページが、注力領域の変化を示す引用可能な根拠になります。エンタープライズ向けセキュリティ、カスタマーサクセス、インフラ、プロダクト拡張といった採用の動きです。レポートが述べるのは「採用が何を示唆しているように見えるか」であって、「ロードマップが何を出すか」ではありません。
**ピークを迎える前の話題を見つけるために。** `/last30days what's exploding in AI agents?` と尋ねると、スキルはディスカバリーモードに切り替わります。エンジンが Reddit のカテゴリー一覧、Hacker News のフロントページとベストストーリー、Digg の AI 1000 フィード、そして認証済みであれば X を横断してさらいます。次にエージェントが候補を審査し(名前の妥当性、ノイズの除去、記事になるか)、ポッドキャストや X 記事の切り口を書きます。最後に、勢いの強さで並べた5〜10件のトピックが返ってきます。各結果には情報源をまたいだ数値、モメンタムのラベル、そしてそのまま実行できる `/last30days "<topic>"` が付いてきます。
**何かが出たとき。** `/last30days Kanye West` — イギリスがビザを却下、Wireless Festival は中止、スポンサーは離脱。それでも BULLY は Billboard 初登場2位。Fantano は「Yay sabbatical」から復帰してレビューを公開(65.3万回再生)。SoFi Homecoming では Lauryn Hill と Travis Scott を迎えて44曲を披露。Polymarket では「Kanye はまたツイートするか?」が「はい」86%。Reddit のスレッド23件、YouTube の動画17本、アップボート8.6万件。
**ツールを比べるために。** `/last30days OpenClaw vs Hermes vs Paperclip` — 「これらは競合ではなくレイヤーだ」。OpenClaw は実行を担うレイヤー(GitHub スター35.1万、稼働中)、Hermes は自己改善する頭脳(スター3.1万)、Paperclip は組織図(スター4.9万)。スター数は古いブログ記事からではなく GitHub API からその場で取得しています。アーキテクチャ、メモリ、セキュリティ、向いている用途を並べた比較表付き。@IMJustinBrooke いわく「OpenClaw = ヒトカゲ、Hermes = リザードン」。
**世界の動きを理解するために。** `/last30days Iran vs USA` — 開戦から38日目。トランプ大統領はイランに対し、ホルムズ海峡の再開について火曜日を期限とする最後通告。米軍機2機が撃墜。原油は1バレル126ドル。IEA はこれを「世界の石油市場の歴史上最大の供給途絶」と呼びました。Polymarket では12月31日までの停戦が74%。X の投稿27件、YouTube の動画10本、予測市場20件。
**旅行の前に。** `/last30days Universal Epic Universe` — 拡張エリアはすでに着工済み。「Project 680」の建設許可が申請されています。花火ショーはインフラの痕跡から確認できるものの、まだ発表はありません。待ち時間は Mine-Cart Madness が平均148分。年間パスはまだ出ておらず、地元の人たちは不満を漏らしています。Stardust Racers は4月5日まで改修で運休。
**手早く学ぶために。** `/last30days Nano Banana Pro prompting` — JSON で構造化したプロンプトが、タグの寄せ集めに取って代わりつつあります。@pictsbyai の入れ子形式は「コンセプトの混線」を防ぎます。作り直すより、編集を前提にしたワークフローのほうが結果が出ます。そのうえで、コミュニティが「これは効く」と言った内容をそのまま使って、実運用向けのプロンプトを書いてくれます。
## 最近の変更
5月の v3.3 発表以降、v3.11.1(2026年7月)時点までで、15回のリリースにわたり175本のPRがマージされました。うち122本はコミュニティの52人によるものです。以下がその内容です。
### OpenAI Codex での一級対応
/last30days は、ガイド付きセットアップを備えた Codex のネイティブプラグインになりました。移植版ではなく、一級の対応です。レンダラーを踏まえた引用処理によって、Codex での出力はURLの羅列ではなくブリーフとして読めるようになり(#694)、同じエンジンが Claude Code、Cursor、Copilot、Gemini CLI、Claude Desktop、OpenClaw、そして50以上の Agent Skills ホストで動きます。Codex のプラグインマニフェストは [@rfoust](https://github.com/rfoust)(#686)、Codex の認証まわりの修正は [@tmchow](https://github.com/tmchow)(#698)によるものです。
### arXiv、Techmeme、Digg — 無料、APIキー不要
arXiv は話題の裏側にある論文を、Techmeme はテックニュースの編集レイヤーを持ち込みます。いずれも無料でキーは一切不要、しかも初回セットアップが各CLIをインストールするので自動的に有効になります(#709)。Digg の AI 1000 ストーリークラスターも同じように、X の認証なしで届きます。セットアップが無料の Digg CLI を入れてくれます(#590)。Trustpilot は消費者向けブランドの調査用に、任意で有効にできます。
### 無料の Reddit が、実数のスコアとトップコメント付きで復活
Reddit の公開 .json API は終了しましたが、無料の経路はより強くなって戻ってきました。キー不要の RSS と shreddit のスクレイピング(#457)、arctic-shift 経由で実際のアップボート数まで取れるサブレディット特定(#696)、そして話題から外れたバズ投稿にブリーフを乗っ取られないようにする関連性の下限(#488、[@rzachsmith](https://github.com/rzachsmith) に感謝)。APIキーは不要。スコアは実数。トップコメントも込みです。
### どのブリーフにも最高のコメントを
コメントは今や、どの情報源でも既定で有効なレイヤーです。Instagram のコメントは順位に基づいて分散させ、尖った意見5件が同じ投稿ばかりから出ないようにしています(#751)。YouTube のコメントに加えて、yt-dlp が失敗したときのために ScrapeCreators による文字起こしのバックアップも用意しました(#637)。さらに、コミュニティの投票で支持されたコメントを Best Takes のスコアに反映し、いちばん面白い一言が選別を生き延びるようにしています(#592#608)。
### doctor コマンド1つで
ヘルスチェックを頼めば、doctor がすべての情報源を試したうえで、必要な対処をそのまま提示します。どのキーが足りないのか、どのCLIが PATH に入っていないのか、どのクッキーが期限切れなのか(#753)。X の結果が薄かった理由を当てずっぽうで探す必要はもうありません。
### X 検索の作り直し
X のパイプラインを一から作り直しました。FROM レーンと ABOUT レーンを設けて、本人の投稿と本人についての会話の両方が順位付けされるようにし(#610)、対象人物に応じてサブクエリの曖昧さを解消し(#611)、本人による投稿かどうかを裏付けたうえでインタラクションのシグナルで順位を付け(#613)、バックエンドを自動で切り替える単一の X ソースにまとめました(#622)。さらに、認証を実際に確かめる正直な `--diagnose` も入っています(#609)。
### 情報源が増えました
ScrapeCreators 経由の LinkedIn。記事は強いシグナルとして扱います([@ravstr](https://github.com/ravstr)、#702)。StockTwits は銘柄コードや暗号資産の話題で自動的に有効になります([@wtiwana](https://github.com/wtiwana)、#658)。Perplexity は直接APIモードと非同期の Deep Research に対応しました([@sk-holmes](https://github.com/sk-holmes)、#629)。
### コミュニティによる堅牢化
セキュリティ面の改善は、ほぼすべてコミュニティの手によるものです。HTML レンダラーの格納型XSSの修正([@iliaal](https://github.com/iliaal)、[@aaronjmars](https://github.com/aaronjmars))、クッキーの一時ファイルの権限強化、OpenSSF Scorecard とビルド来歴の証明を組み込んだサプライチェーン耐性のあるCI([@shaanmajid](https://github.com/shaanmajid)、[@hammadxcm](https://github.com/hammadxcm)、[@aniruddh909](https://github.com/aniruddh909))、Semgrep と OSV-Scanner によるスキャンおよびPRごとの依存関係レビューゲート([@23241a6749](https://github.com/23241a6749))、60%で導入し現在は84%まで引き上げたテストカバレッジの下限([@gourab5139014](https://github.com/gourab5139014))、そして CRITICAL の指摘がゼロになった Hermes のセキュリティスキャン(#768)。
### 届く範囲が広がりました
ヘブライ語をはじめとする非ラテン文字の言語に対応([@dudyme](https://github.com/dudyme))。中国語の情報源向けに CJK を考慮したトークナイズ([@An-idd](https://github.com/An-idd))。Windows 対応の改善もまとめて入りました。Chromium 系ブラウザ全体(Brave、Edge、Vivaldi、Opera、Arc)からのクッキー抽出([@andrey-esipov](https://github.com/andrey-esipov))に加え、macOS のキーチェーンと Linux の pass(1) も認証情報の取得元として使えます。`--as-of` による過去時点の振り返り([@chiyi-creator](https://github.com/chiyi-creator))。uv 経由での Python 3.12 の自動セットアップ([@buntysomroy](https://github.com/buntysomroy))。企業の求人ページを読む `--hiring-signals`。実行と実行のあいだのウォッチリスト差分。
### v3 から引き続き入っているもの
v3 の土台はすべて健在です。APIコールを1件も投げる前に、適切なアカウント・サブレディット・ハッシュタグを特定する事前リサーチの頭脳([@j-sperling](https://github.com/j-sperling) が構築)。関連性だけでなくユーモアやバイラル性も見る Best Takes のスコアリング。情報源をまたいだクラスターの統合。1回のパスで済む比較(「CLI vs MCP」が12分ではなく3分)。自動で候補を見つける `--competitors` 比較。GitHub の人物モード(`--github-user=steipete`)。ELI5 モード(実行後に「eli5 on」)。そして共有できる自己完結型の HTML ブリーフ(`--emit=html`)。設定項目は [CONFIGURATION.md](CONFIGURATION.md) にまとまっています。
## インストール
| 環境 | インストール | 更新 |
|---------|---------|---------|
| **Claude Code**(推奨) | `/plugin marketplace add mvanhorn/last30days-skill` | マーケットプレイス経由で自動、または `claude plugin update last30days@last30days-skill` |
| **Grok**(xAI Build CLI) | `grok plugin marketplace add mvanhorn/last30days-skill` のあとに `grok plugin install last30days` | `grok plugin update last30days` |
| **Codex、Cursor、Copilot、Gemini CLI、その他50以上の [Agent Skills](https://agentskills.io) ホスト** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **claude.ai**(ウェブ) | [`last30days.skill` をダウンロード](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill)し、claude.ai > Customize > Skills > + > Create skill > Upload a skill からアップロード | ダウンロードし直してアップロードし直す |
| **Claude Desktop** | [お使いのプラットフォーム向けの `.mcpb` をダウンロード](https://github.com/mvanhorn/last30days-skill/releases/latest)し、Settings > Extensions にドラッグ | ダウンロードし直して新しいバンドルをドラッグ |
| **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code(推奨)
```
/plugin marketplace add mvanhorn/last30days-skill
```
Claude Code のマーケットプレイスが更新を代わりにやってくれるため、これが推奨です。プラグインのキャッシュはバージョン管理されていて、新しいリリースが公開されると自動で更新されます。`claude plugin update last30days@last30days-skill` を実行すれば、その場で確認を強制できます。
Claude Code で Agent Skills 経由のインストールを使いたい場合も、それはそれで対応しています。
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
ネイティブプラグインと `npx skills` でのインストールは共存できます。ただし Claude Code はインストール方法をまたいだ重複排除を行いません。マーケットプレイス版のプラグインと `npx skills` のコピーを両方とも有効にしていると、`/last30days` が2件表示されます。1台につきインストール方法は1つにしてください。
### Grok(xAI Build CLI)
[Grok Build](https://docs.x.ai/build/features/skills-plugins-marketplaces)(`grok`)は last30days をネイティブプラグインとしてインストールします。直接インストールする場合はリポジトリを追跡します。
```bash
grok plugin install mvanhorn/last30days-skill
```
あるいは、このリポジトリをマーケットプレイスのソースとして追加してから、プラグイン名でインストールすることもできます。
```bash
grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
インストール時の確認を省きたい場合は `--trust` を付けてください。更新は `grok plugin update last30days` です。Grok は互換性のために Claude Code のマニフェストも読みますが、第一の経路はネイティブの `.grok-plugin/` のペアで、[xAI のマーケットプレイス](https://github.com/xai-org/plugin-marketplace)への公式掲載もこちらを指しています。`npx skills add` は、どのホストでも使える代替手段として引き続き有効です。
### Codex、Cursor、Copilot、Gemini CLI、その他の Agent Skills ホスト
オープンな [Agent Skills](https://agentskills.io) の CLI からインストールします。`codex``cursor``github-copilot``gemini-cli``claude-code``windsurf``cline``continue``roo``aider-desk``opencode``goose` など50以上のホストに対応しています(全一覧は [vercel-labs/skills リポジトリ](https://github.com/vercel-labs/skills)にあります)。
```bash
npx skills add mvanhorn/last30days-skill -g
```
`-g`(グローバル)フラグを付けるとユーザーディレクトリにインストールされ、スキルをすべてのプロジェクトで使えます。`-g` を付けない場合、`npx skills` はプロジェクト内の `./.skills/` にインストールし、リポジトリと一緒にコミットされます。世界中を調べるためのツールなので、通常はグローバルが向いています。
Codex のデスクトップ版など、フォルダ単位で動くホストは、Git リポジトリでも普通のフォルダでも動作します。最初の調査を始める前に、読み込み済みのスキルディレクトリから同梱の `scripts/last30days.py --preflight` を実行するようホストのエージェントに頼んでください。ソースをチェックアウトしている場合、同等のコマンドは `python3 skills/last30days/scripts/last30days.py --preflight` です。設定の取得元、ブラウザのクッキーをどう扱う予定か、どのファイルを書き込む予定か、任意で使えるコマンド、無視されるプロジェクト設定を表示します。クッキーの読み取りもファイルの書き込みも調査の実行もしません。
既定では、`npx skills` が検出したホスト向けにインストールされます。特定のホスト(または複数)を指定するには次のようにします。
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
あとから更新するには次のようにします。
```bash
npx skills update last30days -g
```
`npx skills` でグローバルに入れたものをまとめて更新することもできます。
```bash
npx skills update -g
```
一覧表示と削除は `npx skills list -g``npx skills remove last30days -g` で行えます。
### claude.ai(ウェブ)
1. 最新リリースから [`last30days.skill` をダウンロード](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill)します
2. [claude.ai > Customize > Skills](https://claude.ai/customize/skills) を開きます
3. Skills パネルの `+` ボタンをクリックし、`Create skill` > `Upload a skill` と進んで、ファイルを選択するかドロップします
先に Capabilities で「Code execution and file creation」を有効にしてください。これがないとスキルは動きません。
### Claude Desktop
Claude Desktop では、`.mcpb` バンドル(ワンクリック版の Model Context Protocol パッケージ)を使って `/last30days` を MCP サーバーとしてインストールします。
1. [最新リリース](https://github.com/mvanhorn/last30days-skill/releases/latest)を開き、お使いのプラットフォーム向けの `.mcpb` をダウンロードします:
- macOS Apple Silicon: `last30days-pp-mcp-darwin-arm64.mcpb`
- macOS Intel: `last30days-pp-mcp-darwin-amd64.mcpb`
- Linux x86_64: `last30days-pp-mcp-linux-amd64.mcpb`
2. Claude Desktop を開き、Settings > Extensions に移動して、ファイルをドラッグします。
3. 求められたら、有効にしたい情報源のAPIキーを貼り付けます。どの項目も任意です。すべて省略した場合、エンジンはウェブのみのモードに切り替わります。キーはOSのキーチェーンに保存されます。
4. Claude Desktop を再起動します。Claude に「Peter Steinberger について調べて」などと頼めば、`research` ツールが呼び出されます。
**ホスト側の要件:** PATH の通った Python 3.12以上。バンドルにはエンジンのソースが含まれますが、実行にはローカルの Python インタプリタを使います。Windows では [python.org](https://www.python.org/downloads/) からインストールしてください。macOS とたいていの Linux ディストリビューションには、対応するバージョンが最初から入っています。
**キーは Claude Code のスキルとは共有されません。** Claude Desktop と Claude Code は、設計上それぞれ別に認証情報を保管しています。Claude Code のスキル用にすでに `~/.config/last30days/.env` を設定していても、ここで同じキーをもう一度だけ入力する必要があります。
Windows のサポートは、マニフェストのプラットフォーム別エントリーポイントが整理されるまで見送りとなっており、専用のIssueで追跡しています。
### OpenClaw
```bash
clawhub install last30days-official
```
`/last30days` の調査以外で X/Twitter を操作したい場合 — ツイートや返信の投稿、
フォロワーのエクスポート、メディアの扱い、モニタリング、プレゼント企画の抽選など —
には、OpenClaw の補助プラグインとして [TweetClaw](https://github.com/Xquik-dev/tweetclaw)
を使ってください。TweetClaw は Xquik-dev が管理しており、ここでは任意の補助的な
選択肢として挙げているだけです。last30days の依存でも推奨でもありません。
### 手動インストール(開発者向け)
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
```
シンボリックリンクにしておけば、編集するたびに作業ツリーとインストール先が同期するので、コピーし直す必要はありません。`claude.ai` 用には、ソースから `.skill` ファイルをビルドしてください。`bash skills/last30days/scripts/build-skill.sh``dist/last30days.skill` が生成されます。
Reddit(コメント込み)、Hacker News、Polymarket、GitHub はすぐに使えます。設定は不要です。`/last30days` を一度実行すれば、セットアップウィザードが30秒でさらに多くの情報源を有効にします。無料の arXiv と Techmeme の CLI も含まれます。
## 自分のキーを持ち込む
これらのプラットフォーム同士には何のつながりもありません。X は Reddit が何を考えているかを知りませんし、YouTube に TikTok は見えていません。しかし自分のAPIキーとブラウザのトークンを持ち込めば、それらすべてに一度にアクセスできるようになります。
| 情報源 | 必要なもの | 費用 |
|---------|---------------|------|
| Reddit(コメント込み)+ HN + Polymarket + GitHub + StockTwits | 不要 | 無料 |
| arXiv + Techmeme | 無料のCLI。初回セットアップが自動でインストールします | 無料 |
| X / Twitter | 任意のブラウザで x.com にログインするか、`XQUIK_API_KEY` / `XAI_API_KEY` を設定 | ブラウザのクッキーは無料。キーの料金は提供元によります |
| YouTube | `brew install yt-dlp` | 無料 |
| Bluesky | bsky.app のアプリパスワード | 無料 |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube のコメント | ScrapeCreators のキー | 1万リクエストまで無料、以降は従量課金 |
| Xiaohongshu(RED) | ログイン済みの x-mcp ブラウザプラグインか `xiaohongshu-mcp` サービスを動かしたうえで、実行ごとに `--search xhs` を付けるか `.env``INCLUDE_SOURCES=xiaohongshu` を設定して有効化します。last30days は `http://localhost:18060`、次に `http://host.docker.internal:18060` の順に自動で接続を試し、独自のURLを使う場合は `XIAOHONGSHU_API_BASE` を指定します | last30days 側のAPIキーは不要。ローカルのブラウザセッションのサービス次第です |
| DripStack(有料の金融ニュースレター) | 任意で有効化: 実行ごとに `--search dripstack`、または `.env``INCLUDE_SOURCES=dripstack` | キー不要。無料の公開検索APIを使います |
| Perplexity Sonar / Search API / Deep Research | Perplexity のキー、または Sonar の代替として OpenRouter のキー | 従量課金 |
| ウェブ検索 | Brave Search のキー | 月2,000クエリまで無料 |
### macOS のキーチェーン(任意)
macOS では、キーを `.env` ファイルではなくシステムのキーチェーンに保存できます。スキルは最も優先度の低い取得元として自動的に読み込むため、衝突した場合は `.env` ファイルとプロセスの環境変数が優先されます。
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
項目は現在のユーザー向けに、サービス名 `last30days-<KEY>` で保存されます。Darwin 以外のプラットフォームではローダーは何もしないため、Linux や Windows のユーザーにとって挙動は変わりません。
すでに別のサービス名でキーチェーンにキーを保存している場合は、秘密情報をコピーする代わりに、[CONFIGURATION.md](CONFIGURATION.md#reusing-existing-macos-keychain-items) で説明している秘密情報ではないマッピング `LAST30DAYS_KEYCHAIN_ALIASES` を設定してください。
情報源ごとのキーの一覧、推論プロバイダーの優先順位、ウェブ検索バックエンドの優先順位については [CONFIGURATION.md](CONFIGURATION.md) を参照してください。
## 設定
初日に知っておくとよいことが2つあります。
**調査ファイルの保存先。** `LAST30DAYS_MEMORY_DIR` の既定値は `~/Documents/Last30Days/` です(Windows では `C:\Users\<you>\Documents\Last30Days\`)。変更したい場合は、シェルでこの環境変数に任意のパスを設定するか、実行ごとに `--save-dir <path>` を指定します。レンダリング結果を特定のパスに出力したいときは `--output <file>` を使い、形式は `--emit` で選びます。同じトピックの複数のバリエーションを分けて残したいときは `--save-suffix=<name>` を使ってください(クライアントごとに分ける場合など)。`--save-dir` を付けた実行では `<slug>-raw[-suffix].md` が生成されます。調査を走らせる前に書き込み予定を確認するには `python3 skills/last30days/scripts/last30days.py --preflight` を実行してください。
**エージェントやワークフロー向けの構造化出力。** `/last30days` に機械可読なJSONを求めると、安定したバージョン付きのエージェント向けプロファイルが返ります。スクリプトや開発でエンジンを直接使う場合は `python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json` を実行してください。バージョン管理されていない内部の `Report` のダンプが必要なときだけ `--json-profile=raw` を追加します。[JSONエクスポートのフィールド一覧とバージョニング方針](docs/reference/json-export.md)も参照してください。
**トピックを決めないディスカバリー。** すでに知っているトピックを調べる代わりに、順位付きのディスカバリーブリーフがほしいときは `/last30days what's trending in AI agents?` と尋ねてください。エージェントを備えたホストでは、ホストが判定する3コマンドのプロトコルが走ります(モデルがトピックを挙げ、ノイズを除き、取り上げる価値を採点し、コンテンツの切り口を書きます)。スクリプトや cron でエンジンを直接使う場合は `python3 skills/last30days/scripts/last30days.py --discover "AI agents"` を実行します(単発実行。トピック名は決定論的で、切り口は付きません)。バージョン付きのディスカバリー契約がほしい場合は `--emit=json` を追加してください。ディスカバリーは、位置引数のトピックや `--drill` とは併用できません。
**実行をまたいだトレンド監視。** 既定のモードでは、実行のたびに新しい Markdown のスナップショットが作られます。時間をかけて結果を蓄積したい場合は `--store` を付けて SQLite データベースに保存し、定期実行には [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py)(新しい結果が出たときの Slack や Webhook への通知も任意で設定できます)、日次・週次のまとめには [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) を使ってください。運用サイクルの全体像は [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings) にあります。
**購読できる調査ライブラリ。** `/last30days` にライブラリのフィードを作らせるか、スクリプトや開発用には `python3 skills/last30days/scripts/last30days.py library feed` を直接使ってください。保存済みのブリーフが `index.html`、ローカルの Atom 形式の `feed.xml`、読みやすいブリーフのページに変換されます。HTML のインデックスとブリーフのページをホスティングしたいときだけ `--publish` を付けてください。公開は明示的に選ぶ形で、既定では誰でも見られる状態になります。Atom フィードを実際に購読できるようにするには、生成された出力ディレクトリを GitHub Pages のような静的ホスティングに置いてください。
**これまで調べたものをすべて検索する。** `/last30days search my library for MCP servers``/last30days have I researched MCP servers before?` と尋ねてください。エンジンを直接使う場合は `python3 skills/last30days/scripts/last30days.py library search "MCP servers"` を実行します。この検索はオフラインかつ決定論的です。ライブラリのフィードが使うのと同じ保存済みブリーフを少しずつインデックス化し、実行ごとにストアへ記録された該当分をまとめ、トピックと日付で結果をグループ化します。新しく実行したときも、過去の調査が今回のトピックと重なっていれば、**From your library**(あなたのライブラリから)というコンパクトなセクションが表示されます。この受動的な文脈表示をやめたい場合は `LAST30DAYS_LIBRARY_CONTEXT=off` を設定してください。
クライアントごとのラッパースクリプト、カテゴリー用のサブレディットのカスタマイズ、作業中のカスタマイズを試す実験的なベータチャンネルについても [CONFIGURATION.md](CONFIGURATION.md) に記載しています。
## ショーケース: コミュニティの調査フィード
last30days で、定期的なAIのまとめ、市場ウォッチ、あるいは見事にニッチな偏愛を公開しましたか。公開ライブラリのURL(または `feed.xml` を静的ホスティングに置いたあとの Atom のURL)を[コミュニティのショーケーススレッド](https://github.com/mvanhorn/last30days-skill/issues/532)で共有してください。コミュニティのフィードは、作者から届き次第ここにリンクしていきます。それまでのあいだは、このスレッドが集約先です。
## 仕組み
1. **トピックを入力します。** 人物、企業、プロダクト、技術、「X vs Y」。何でもかまいません。
2. **エージェントが「誰が重要か」を特定します。** X のアカウント(創業者を含む)、GitHub のリポジトリ、サブレディット、TikTok のハッシュタグ、YouTube のチャンネルを見つけます。「Kanye West」なら r/hiphopheads、@kanyewest、YouTube の「bully review」だと分かります。「OpenClaw」なら GitHub 上の openclaw/openclaw を特定し、スター数をその場で取得します。
3. **すべての情報源を並行して検索します。** 複数クエリへの展開。結果はエンゲージメント、関連性、新しさでスコア付けされます。
4. **他にはない深さ。** リアクション動画の YouTube 全文文字起こし。アップボート数付きの Reddit のトップコメント。TikTok のキャプション。Polymarket のオッズ。タイトルとリンクだけではありません。
5. **同じ話題はまとめます。** Reddit で告知され、X で語られ、TikTok にチケット価格が出た Wireless Festival は、3件の別々の項目ではなく1つのクラスターになります。
6. **1本のブリーフに統合します。** 具体的なデータに基づき、情報源を明示し、実際に人々が反応したかどうかで順位を付けます。「見つけたものはこれです」ではなく「重要なのはこれです」を返します。
7. **そのあとは、あなたの専門家になります。** 一度実行すれば、あなたの Claude のセッションはコミュニティが知っていることをすべて把握しています。続けて質問してください。プロンプトを書かせる、メールを下書きさせる、旅程を立てさせる、システム構成を設計させる。どれも「今、実際に起きていること」に基づきます。
## 使っている人の声
> 「Reddit、X、YouTube、HN を横断して直近30日のあらゆるトピックを調べてくれる Claude Code のスキルを見つけた。しかもプロンプトまで書いてくれる。書く記事ごとに、これまでは Reddit と X を手作業で調べていた。タブごと、スレッドごとに。そこが90分かかっていた部分だ。それがなくなる。」 — @itsjasonai
> 「このスキル1つで、私の調査ワークフローがまるごと置き換わった。トピックを渡すと、Reddit、X、ウェブから人々が本当に話していることを拾ってくる。古いブログ記事ではなく、直近30日の生の会話だ。」 — @itswilsoncharles
> 「今日 GitHub でトレンド入りしているリポジトリ10件のうち5件が Claude 関連のツール。1位は mvanhorn/last30days-skill」 — @yieldhunter95
## オープンソース
MIT ライセンス。トラッキングなし。アナリティクスなし。調査結果はあなたのマシンに残ります。テストは2,700件以上。
Python 3.12以上、yt-dlp、Node.js(X 検索用に同梱した Bird クライアント)、ScrapeCreators API で構築しています。v3 のエンジンアーキテクチャは [@j-sperling](https://github.com/j-sperling) によるものです。
PRの出し方は [CONTRIBUTING.md](CONTRIBUTING.md)、コミュニティの貢献者の一覧は [CONTRIBUTORS.md](CONTRIBUTORS.md)、バージョン履歴は [CHANGELOG.md](CHANGELOG.md) を参照してください。
## スター数の推移
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
+2
View File
@@ -1,5 +1,7 @@
# /last30days
English | [Français](README.fr.md) | [Deutsch](README.de.md) | [Español](README.es.md) | [Português (Brasil)](README.pt-BR.md) | [日本語](README.ja.md) | [简体中文](README.zh-CN.md)
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days - an AI agent-led search engine that searches people, not editors" />
</p>
+383
View File
@@ -0,0 +1,383 @@
# /last30days
[English](README.md) | [Français](README.fr.md) | [Deutsch](README.de.md) | [Español](README.es.md) | Português (Brasil) | [日本語](README.ja.md) | [简体中文](README.zh-CN.md)
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days - an AI agent-led search engine that searches people, not editors" />
</p>
<p align="center">
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
**Um buscador conduzido por um agente de IA, que pontua por votos positivos, curtidas e dinheiro de verdade — não por redações.**
Este README descreve o pipeline v3 atual. A especificação de execução da skill fica em [skills/last30days/SKILL.md](skills/last30days/SKILL.md), que é a referência definitiva sobre o comportamento dos comandos e da configuração.
**Claude Code (recomendado — atualizações automáticas via marketplace):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex, Cursor, Copilot, Gemini CLI, ou qualquer um dos 50+ hosts do [Agent Skills](https://agentskills.io):**
```
npx skills add mvanhorn/last30days-skill -g
```
(`-g` instala globalmente para o seu usuário, então fica disponível em todos os projetos. Omita essa flag se quiser limitar a instalação a um projeto.)
Outras formas de instalar (claude.ai web, OpenClaw, manual) estão na seção [Instalação](#instalação), mais abaixo.
Configuração zero. Reddit, HN, Polymarket e GitHub funcionam de imediato. Rode uma vez e o assistente de configuração libera X, YouTube, TikTok, arXiv, Techmeme e mais em 30 segundos.
---
Os votos positivos do Reddit. As curtidas do X. As transcrições do YouTube. O engajamento no TikTok. As probabilidades do Polymarket, lastreadas em dinheiro de verdade e em informação privilegiada. São milhões de pessoas votando todo dia com a atenção e com o bolso. O /last30days busca tudo isso em paralelo, pontua pelo que as pessoas realmente engajam e um agente de IA atua como juiz para sintetizar tudo em um único briefing.
O Google agrega redações. O /last30days busca pessoas.
Essa busca você não encontra em nenhum outro lugar, porque nenhuma IA sozinha tem acesso a tudo. O Google não alcança nem os comentários do Reddit nem as publicações do X. O ChatGPT tem acordo com o Reddit, mas não consegue buscar no X nem no TikTok. O Gemini tem o YouTube, mas não o Reddit. O Claude não tem nenhum deles de forma nativa. Cada plataforma é um jardim murado, com API, tokens e autenticação próprios. Mas você pode trazer suas próprias chaves e sessões de navegador e, de repente, um agente de IA busca em todas ao mesmo tempo, compara umas com as outras e diz o que realmente importa.
É esse o destravamento. Não é um buscador melhor: é uma dúzia de plataformas isoladas, conectadas por um agente.
```
/last30days Peter Steinberger
```
Você tem uma reunião amanhã. Procura a pessoa no Google. Aparece o LinkedIn dela de 2023. O /last30days entrega o que ela está fazendo de fato neste mês: entrou na OpenAI para trabalhar no Codex, enfrenta o veto da Anthropic a agentes de terceiros, entregou 23 PRs com 85 % de taxa de merge, constrói o "LobsterOS" para controlar agentes entre dispositivos, e uma thread no r/ClaudeCode chegou a 569 votos positivos discutindo se ele é um herói ou "insuportável". Tudo espalhado entre publicações no X, threads no Reddit, transcrições do YouTube e commits no GitHub. Nada disso estava no Google.
## Por que isso existe
Construí para acompanhar o ritmo da IA. Tudo muda todo dia, e o pessoal do Reddit e do X sempre sabe primeiro. Eu precisava de prompts melhores, e os dados de treinamento estavam sempre meses atrás do que a comunidade já tinha descoberto.
Mas virou algo maior. Hoje eu rodo antes de uma call de vendas, para saber a verdade dos últimos 30 dias sobre uma empresa. Antes de uma reunião, para ler os tweets recentes e as transcrições de podcast de quem vou encontrar. Antes de uma viagem à Disney World, para saber quais brinquedos estão fechados e o que a comunidade acha do Genie+. Antes de construir qualquer coisa, para saber em quais problemas as pessoas realmente estão esbarrando.
Se você vai se reunir com um CEO, já leu todos os tweets e todas as transcrições do YouTube dos últimos 30 dias? Eu li.
## Fontes, pontuadas pelas pessoas
| Fonte | O que as pessoas te contam |
|--------|--------------------------|
| **Reddit** | A opinião sem filtro. Os melhores comentários com a contagem real de votos positivos, de graça e sem chave de API. As opiniões de verdade que o Google enterra. |
| **X / Twitter** | A opinião quente, a thread do especialista, a primeira reação ao factual. Os primeiros a saber, os primeiros a discutir. |
| **YouTube** | A análise de 45 minutos. Transcrições completas, garimpadas atrás das 5 frases citáveis que importam. |
| **TikTok** | O criador que alcança 3,6 milhões de pessoas com uma leitura que você nunca vai achar no Google. |
| **Instagram Reels** | O olhar dos influenciadores, com a transcrição do que é falado. O sinal da cultura visual. |
| **Hacker News** | O consenso da turma de desenvolvimento. 825 pontos, 899 comentários. Onde o pessoal técnico discute de verdade. |
| **Polymarket** | Não são opiniões. São probabilidades. Lastreadas em dinheiro de verdade. 96 % de confiança em vendas de um álbum. 4 % em uma aquisição. |
| **GitHub** | Para pessoas: ritmo de PRs, principais repositórios por estrelas, notas de versão. Para assuntos: issues e discussions. |
| **Digg** | Agrupamentos de histórias curados a partir do ranking AI 1000 do Digg (cerca de 1000 contas de IA com alto sinal no X), com citações atribuíveis embutidas e sem exigir autenticação no X. Ativa sozinho quando `digg-pp-cli` está no PATH. |
| **arXiv** | Os artigos científicos por trás do hype. Pesquisa nova dentro da janela, de graça e sem chave de API. Ativa sozinho quando `arxiv-pp-cli` está no PATH (a configuração inicial instala). |
| **Techmeme** | A camada editorial do noticiário de tecnologia, limitada à sua janela de 30 dias. De graça e sem chave de API. Ativa sozinho quando `techmeme-pp-cli` está no PATH (a configuração inicial instala). |
| **LinkedIn** | O sinal profissional. Publicações e artigos, com os artigos ponderados como sinal forte. |
| **StockTwits** | O humor dos traders. Ativa automaticamente quando seu assunto é um ticker ou uma cripto. |
| **Threads** | A camada de texto do pós-Twitter. Conversas de criadores e marcas. |
| **Pinterest** | Descoberta visual. Pins, itens salvos e comentários sobre produtos e ideias. |
| **Xiaohongshu (RED)** | Sinais chineses sobre estilo de vida, produtos e criadores. É pedido explicitamente com `--search xhs` quando há um plugin de navegador x-mcp logado ou um serviço `xiaohongshu-mcp` rodando localmente. |
| **Bluesky** | A camada social descentralizada. Publicações do AT Protocol vindas da migração pós-Twitter. |
| **Perplexity** | A síntese fundamentada do Sonar, os resultados brutos da Search API e o Deep Research. |
| **Web** | A cobertura editorial, as comparações de blog. Um sinal entre muitos, não o único. |
A comunidade não para de acrescentar fontes. Truth Social e outras fontes de nicho já estão no motor, e vêm mais por aí.
Uma thread do Reddit com 1.500 votos positivos é um sinal mais forte do que um post de blog que ninguém leu. Um TikTok com 3,6 milhões de visualizações diz mais sobre o que é culturalmente relevante do que qualquer release de imprensa. Probabilidades do Polymarket lastreadas em US$ 66 mil de volume são bem mais difíceis de contestar do que o palpite de um comentarista.
A síntese ordena pelo que as pessoas de verdade realmente engajaram. Relevância social, não relevância de SEO.
## Para que as pessoas realmente usam
**Antes de uma reunião.** `/last30days Peter Steinberger` — entrou no time do Codex na OpenAI, enfrenta o veto da Anthropic a agentes de terceiros, 23 PRs mergeados com 85 % de taxa de merge no GitHub, constrói o LobsterOS para controlar agentes entre dispositivos. r/ClaudeCode: "Desde que o OpenClaw saiu, todo mundo já sabia que, se você rodasse por qualquer coisa que não fosse a API, uma hora ia ser banido" (227 votos positivos). Isso não está no LinkedIn.
**Para ler sinais de contratação.** `/last30days Listen Labs --hiring-signals` — as vagas e páginas de carreira atuais viram evidência citada de mudança de prioridade: contratação em segurança para empresas, customer success, infraestrutura ou expansão de produto. O relatório diz o que a contratação parece sinalizar, não o que o roadmap vai entregar.
**Para achar o assunto antes do pico.** Pergunte `/last30days what's exploding in AI agents?` e a skill muda para o modo descoberta: o motor varre as listagens por categoria do Reddit, a capa e as melhores histórias do Hacker News, o feed AI 1000 do Digg e o X quando você está autenticado; seu agente avalia as indicações (nomes, filtragem de ruído, se rende conteúdo) e escreve ângulos para podcast ou para um artigo no X; depois você recebe de 5 a 10 assuntos ordenados por velocidade. Cada resultado traz números de várias fontes, um rótulo de momentum e um comando `/last30days "<topic>"` pronto para rodar.
**Quando alguma coisa é lançada.** `/last30days Kanye West` — o Reino Unido bloqueou o visto dele, o Wireless Festival foi cancelado, os patrocinadores fugiram. Mas BULLY estreou em 2º na Billboard. Fantano voltou do "Yay sabbatical" para resenhar o disco (653 mil visualizações). No SoFi Homecoming, ele levou Lauryn Hill e Travis Scott ao palco para 44 músicas. Polymarket: "Kanye vai tuitar de novo?" 86 % sim. 23 threads no Reddit, 17 vídeos no YouTube, 86 mil votos positivos.
**Para comparar ferramentas.** `/last30days OpenClaw vs Hermes vs Paperclip` — "Não são concorrentes, são camadas." O OpenClaw é a camada de execução (351 mil estrelas no GitHub, em produção), o Hermes é o cérebro que se aprimora sozinho (31 mil estrelas), o Paperclip é o organograma (49 mil estrelas). As contagens de estrelas vêm ao vivo da API do GitHub, não de posts de blog desatualizados. Tabela lado a lado com arquitetura, memória, segurança e melhor caso de uso. Segundo @IMJustinBrooke: "OpenClaw = Charmander, Hermes = Charizard."
**Para entender o mundo.** `/last30days Iran vs USA` — dia 38 da guerra. O ultimato de Trump, com prazo até terça, para o Irã reabrir o Estreito de Ormuz. Dois caças americanos abatidos. Petróleo a US$ 126 o barril. A AIE chamou o episódio de "a maior interrupção de fornecimento da história do mercado global de petróleo". Polymarket: cessar-fogo até 31 de dezembro a 74 %. 27 publicações no X, 10 vídeos no YouTube, 20 mercados de previsão.
**Antes de uma viagem.** `/last30days Universal Epic Universe` — a expansão já está em obras. Alvará do "Project 680" protocolado. Show de fogos confirmado pela infraestrutura, mas ainda não anunciado. Tempo de espera: Mine-Cart Madness com média de 148 minutos. Ainda sem passe anual, e os moradores estão irritados. Stardust Racers fechada para reforma até 5 de abril.
**Para aprender algo rápido.** `/last30days Nano Banana Pro prompting` — prompts estruturados em JSON estão substituindo a sopa de tags. O formato aninhado do @pictsbyai evita o "concept bleeding". Editar ganha de regerar. E depois a skill escreve um prompt de produção usando exatamente o que a comunidade disse que funciona.
## Novidades
Desde o anúncio da v3.3 em maio e até a v3.11.1 (julho de 2026): 175 PRs mergeados — 122 deles de 52 pessoas da comunidade — distribuídos em 15 versões. Foi isso que entrou.
### Cidadão de primeira classe no OpenAI Codex
O /last30days agora é um plugin nativo do Codex com configuração guiada: não é um port, é cidadão de primeira classe. As citações levam o renderizador em conta, então a saída no Codex se lê como um briefing e não como uma sopa de URLs (#694), e o mesmo motor roda no Claude Code, Cursor, Copilot, Gemini CLI, Claude Desktop, OpenClaw e em 50+ hosts do Agent Skills. Manifesto do plugin do Codex por [@rfoust](https://github.com/rfoust) (#686), correção de autenticação no Codex por [@tmchow](https://github.com/tmchow) (#698).
### arXiv, Techmeme e Digg — de graça, sem chaves de API
O arXiv traz os artigos científicos por trás do hype e o Techmeme traz a camada editorial do noticiário de tecnologia — de graça, sem nenhuma chave, e a configuração inicial instala as CLIs deles para que ativem sozinhos (#709). Os agrupamentos de histórias AI 1000 do Digg chegam do mesmo jeito, sem autenticação no X: a configuração instala a CLI gratuita do Digg para você (#590). O Trustpilot está disponível como opção para pesquisa de marcas de consumo.
### Reddit gratuito, com pontuações reais e melhores comentários
A API pública .json do Reddit morreu; o caminho gratuito voltou mais forte. RSS sem chave e scraping do shreddit (#457), descoberta de subreddits específicos com contagem real de votos positivos via arctic-shift (#696) e um piso de relevância para que um post viral fora do tema não sequestre seu briefing (#488, valeu [@rzachsmith](https://github.com/rzachsmith)). Sem chave de API. Pontuações reais. Melhores comentários incluídos.
### Os melhores comentários em cada briefing
Os comentários agora são uma camada ligada por padrão em todas as fontes: comentários do Instagram com diversidade baseada em ranking, para que cinco opiniões fortes não venham todas do mesmo post (#751), comentários do YouTube mais um backup de transcrição via ScrapeCreators para quando o yt-dlp falha (#637), e comentários votados pela comunidade entrando com peso no Best Takes, para que as melhores tiradas sobrevivam à pontuação (#592, #608).
### Um único comando doctor
Peça um diagnóstico: o doctor testa cada fonte e receita as correções exatas — qual chave está faltando, qual CLI não está no PATH, qual cookie expirou (#753). Chega de adivinhar por que o X voltou fraco.
### A busca no X, reconstruída
O pipeline do X foi refeito do zero: faixas FROM e ABOUT para que tanto as publicações da própria pessoa quanto a conversa sobre ela sejam ranqueadas (#610), desambiguação de subconsultas conforme a pessoa buscada (#611), verificação de autoria de primeira mão com ranqueamento por sinais de interação (#613) e uma única fonte X com failover automático entre backends (#622). Além de um `--diagnose` honesto, que testa a autenticação de verdade (#609).
### Mais fontes entraram
LinkedIn via ScrapeCreators, com artigos como sinal forte ([@ravstr](https://github.com/ravstr), #702). O StockTwits ativa sozinho em assuntos de ticker e cripto ([@wtiwana](https://github.com/wtiwana), #658). O Perplexity ganhou modos de API diretos e Deep Research assíncrono ([@sk-holmes](https://github.com/sk-holmes), #629).
### Endurecido pela comunidade
A onda de segurança foi quase toda trabalho da comunidade: correções de XSS armazenado no renderizador HTML ([@iliaal](https://github.com/iliaal), [@aaronjmars](https://github.com/aaronjmars)), arquivos temporários de cookie protegidos, CI endurecida contra ataques à cadeia de suprimentos com OpenSSF Scorecard e atestação de proveniência de build ([@shaanmajid](https://github.com/shaanmajid), [@hammadxcm](https://github.com/hammadxcm), [@aniruddh909](https://github.com/aniruddh909)), varreduras com Semgrep e OSV-Scanner mais um portão de revisão de dependências em cada PR ([@23241a6749](https://github.com/23241a6749)), um piso de cobertura de testes criado em 60 % e desde então elevado para 84 % ([@gourab5139014](https://github.com/gourab5139014)), e uma varredura de segurança do Hermes que hoje não tem nenhum achado CRITICAL (#768).
### Alcança mais longe
Hebraico e outras línguas não latinas ([@dudyme](https://github.com/dudyme)). Tokenização adaptada a CJK para fontes chinesas ([@An-idd](https://github.com/An-idd)). Uma onda de compatibilidade com Windows. Extração de cookies em toda a família Chromium — Brave, Edge, Vivaldi, Opera, Arc ([@andrey-esipov](https://github.com/andrey-esipov)) — além do Keychain do macOS e do pass(1) no Linux como origens de credenciais. Consulta retroativa com `--as-of` ([@chiyi-creator](https://github.com/chiyi-creator)). Provisionamento automático do Python 3.12 via uv ([@buntysomroy](https://github.com/buntysomroy)). `--hiring-signals` para ler as páginas de vagas de uma empresa. Deltas de watchlist entre execuções.
### O que já vinha de fábrica desde a v3
As bases da v3 continuam todas aqui: o cérebro de pré-pesquisa, que identifica os handles, subreddits e hashtags certos antes de disparar uma única chamada de API (construído por [@j-sperling](https://github.com/j-sperling)); a pontuação Best Takes, que considera humor e viralidade além de relevância; a fusão de clusters entre fontes; as comparações em uma única passada ("CLI vs MCP" em 3 minutos, não em 12); as comparações `--competitors` descobertas automaticamente; o modo pessoa do GitHub (`--github-user=steipete`); o modo ELI5 ("eli5 on" depois de qualquer execução); e os briefings HTML autocontidos e compartilháveis (`--emit=html`). Os ajustes de configuração estão em [CONFIGURATION.md](CONFIGURATION.md).
## Instalação
| Ambiente | Instalação | Atualizações |
|---------|---------|---------|
| **Claude Code** (recomendado) | `/plugin marketplace add mvanhorn/last30days-skill` | Automáticas via marketplace, ou `claude plugin update last30days@last30days-skill` |
| **Grok** (xAI Build CLI) | `grok plugin marketplace add mvanhorn/last30days-skill` e depois `grok plugin install last30days` | `grok plugin update last30days` |
| **Codex, Cursor, Copilot, Gemini CLI, ou qualquer um dos 50+ hosts do [Agent Skills](https://agentskills.io)** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **claude.ai** (web) | [Baixe `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) e envie por claude.ai > Customize > Skills > + > Create skill > Upload a skill | Baixar de novo e enviar de novo |
| **Claude Desktop** | [Baixe o `.mcpb` da sua plataforma](https://github.com/mvanhorn/last30days-skill/releases/latest) e arraste para Settings > Extensions | Baixar de novo e arrastar o novo pacote |
| **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code (recomendado)
```
/plugin marketplace add mvanhorn/last30days-skill
```
Recomendado porque o marketplace do Claude Code cuida das atualizações por você: o cache do plugin é versionado e se atualiza sozinho quando sai uma versão nova. Rode `claude plugin update last30days@last30days-skill` para forçar uma verificação.
Se preferir usar o caminho de instalação do Agent Skills no Claude Code, ele também é suportado:
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
O plugin nativo e a instalação com `npx skills` podem conviver. Só atenção: o Claude Code não deduplica entre métodos de instalação. Se você tiver ativos ao mesmo tempo o plugin do marketplace e a cópia do `npx skills`, o `/last30days` vai aparecer duas vezes. Use um método de instalação por máquina.
### Grok (xAI Build CLI)
O [Grok Build](https://docs.x.ai/build/features/skills-plugins-marketplaces) (`grok`) instala o last30days como plugin nativo. A instalação direta acompanha o repositório:
```bash
grok plugin install mvanhorn/last30days-skill
```
Ou adicione este repositório como fonte de marketplace e depois instale pelo nome do plugin:
```bash
grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
Acrescente `--trust` para pular a confirmação de instalação. Atualize com `grok plugin update last30days`. O Grok também lê os manifestos do Claude Code por compatibilidade; o par nativo `.grok-plugin/` é o caminho principal — e é para ele que aponta um registro oficial no [marketplace da xAI](https://github.com/xai-org/plugin-marketplace). O `npx skills add` continua sendo uma alternativa válida em qualquer host.
### Codex, Cursor, Copilot, Gemini CLI e outros hosts do Agent Skills
Instale pela CLI aberta do [Agent Skills](https://agentskills.io) — ela suporta 50+ hosts, entre eles `codex`, `cursor`, `github-copilot`, `gemini-cli`, `claude-code`, `windsurf`, `cline`, `continue`, `roo`, `aider-desk`, `opencode`, `goose` e outros (lista completa no [repositório vercel-labs/skills](https://github.com/vercel-labs/skills)).
```bash
npx skills add mvanhorn/last30days-skill -g
```
A flag `-g` (global) instala no seu diretório de usuário, então a skill fica disponível em todos os projetos. Sem `-g`, o `npx skills` instala só no projeto, dentro de `./.skills/` (e vai versionado junto com o repositório). Para uma ferramenta feita para pesquisar o mundo inteiro, o que você quer é a instalação global.
O Codex desktop e outros hosts que trabalham no nível de pasta funcionam tanto em pastas comuns quanto em repositórios Git. Antes da primeira pesquisa, peça ao agente host que rode o `scripts/last30days.py --preflight` que acompanha a skill, a partir do diretório da skill carregada; em um clone do código-fonte, o comando equivalente é `python3 skills/last30days/scripts/last30days.py --preflight`. Ele mostra de onde vem a configuração, quais cookies do navegador seriam lidos, quais arquivos seriam escritos, quais comandos opcionais existem e qual configuração de projeto está sendo ignorada — sem ler cookies, sem escrever arquivos e sem rodar pesquisa nenhuma.
Por padrão, a instalação vale para o host que o `npx skills` detectar. Para mirar em um específico (ou em vários):
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
Para atualizar depois:
```bash
npx skills update last30days -g
```
Ou atualize tudo que você instalou globalmente pelo `npx skills`:
```bash
npx skills update -g
```
Dá para listar e remover com `npx skills list -g` e `npx skills remove last30days -g`.
### claude.ai (web)
1. [Baixe `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) da versão mais recente
2. Vá em [claude.ai > Customize > Skills](https://claude.ai/customize/skills)
3. Clique no botão `+` no painel de Skills, depois em `Create skill` > `Upload a skill`, e escolha ou arraste o arquivo
Ative antes "Code execution and file creation" em Capabilities — sem isso, as skills não rodam.
### Claude Desktop
O Claude Desktop instala o `/last30days` como servidor MCP por meio de um pacote `.mcpb` (um pacote Model Context Protocol de um clique).
1. Vá até a [versão mais recente](https://github.com/mvanhorn/last30days-skill/releases/latest) e baixe o `.mcpb` da sua plataforma:
- macOS Apple Silicon: `last30days-pp-mcp-darwin-arm64.mcpb`
- macOS Intel: `last30days-pp-mcp-darwin-amd64.mcpb`
- Linux x86_64: `last30days-pp-mcp-linux-amd64.mcpb`
2. Abra o Claude Desktop, vá em Settings > Extensions e arraste o arquivo para lá.
3. Quando for solicitado, cole as chaves de API das fontes que quiser ativar. Todo campo é opcional — se pular todos, o motor cai para o modo só web. As chaves ficam guardadas no chaveiro do seu sistema operacional.
4. Reinicie o Claude Desktop. Peça ao Claude para "pesquisar sobre Peter Steinberger", ou sobre qualquer outro assunto, e ele vai chamar a ferramenta `research`.
**Requisito do host:** Python 3.12+ no PATH. O pacote traz o código do motor, mas usa o seu interpretador Python local. No Windows, instale a partir do [python.org](https://www.python.org/downloads/); o macOS e a maioria das distribuições Linux já vêm com uma versão compatível.
**As chaves não são compartilhadas com a skill do Claude Code.** O Claude Desktop e o Claude Code mantêm armazenamentos de credenciais separados de propósito. Se você já configurou o `~/.config/last30days/.env` para a skill do Claude Code, vai precisar digitar essas mesmas chaves aqui uma vez.
O suporte a Windows está adiado até que os pontos de entrada por plataforma no manifesto sejam resolvidos; o acompanhamento fica em uma issue à parte.
### OpenClaw
```bash
clawhub install last30days-official
```
Para fluxos de ação no X/Twitter fora da pesquisa do `/last30days` — publicar
tweets ou respostas, exportar seguidores, cuidar de mídia, monitorar contas e
apurar sorteios — use o [TweetClaw](https://github.com/Xquik-dev/tweetclaw) como
plugin complementar do OpenClaw. O TweetClaw é mantido pelo Xquik-dev e aparece
aqui apenas como opção complementar: não é dependência nem recomendação do
last30days.
### Manual (para quem desenvolve)
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
```
O symlink mantém a instalação em sincronia com sua árvore de trabalho conforme você edita — não precisa copiar de novo. Para o `claude.ai`, compile o arquivo `.skill` a partir do código-fonte: `bash skills/last30days/scripts/build-skill.sh` gera `dist/last30days.skill`.
Reddit (com comentários), Hacker News, Polymarket e GitHub funcionam de imediato. Configuração zero. Rode `/last30days` uma vez e o assistente de configuração libera mais fontes em 30 segundos, incluindo as CLIs gratuitas do arXiv e do Techmeme.
## Traga suas próprias chaves
Essas plataformas não têm relação nenhuma entre si. O X não sabe o que o Reddit pensa. O YouTube não enxerga o TikTok. Mas você pode trazer suas próprias chaves de API e tokens de navegador e, de repente, tem acesso a todas ao mesmo tempo.
| Fontes | O que você precisa | Custo |
|---------|---------------|------|
| Reddit (com comentários) + HN + Polymarket + GitHub + StockTwits | Nada | De graça |
| arXiv + Techmeme | CLIs gratuitas, instaladas automaticamente pela configuração inicial | De graça |
| X / Twitter | Faça login em x.com em qualquer navegador, ou defina `XQUIK_API_KEY` / `XAI_API_KEY` | Os cookies do navegador são gratuitos; as chaves dependem do provedor |
| YouTube | `brew install yt-dlp` | De graça |
| Bluesky | Uma senha de aplicativo do bsky.app | De graça |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + comentários do YouTube | Uma chave do ScrapeCreators | 10.000 chamadas gratuitas e depois pagamento por uso |
| Xiaohongshu (RED) | Deixe rodando um plugin de navegador x-mcp logado ou um serviço `xiaohongshu-mcp` e habilite a fonte com `--search xhs` por execução ou com `INCLUDE_SOURCES=xiaohongshu` no `.env`; o last30days testa automaticamente `http://localhost:18060` e depois `http://host.docker.internal:18060`, ou use `XIAOHONGSHU_API_BASE` para uma URL própria | Não precisa de chave de API do last30days; depende do seu serviço local de sessão de navegador |
| DripStack (newsletters financeiras premium) | Opcional: `--search dripstack` por execução, ou `INCLUDE_SOURCES=dripstack` no `.env` | Sem chave; API de busca pública e gratuita |
| Perplexity Sonar / Search API / Deep Research | Uma chave do Perplexity, ou uma chave do OpenRouter como alternativa para o Sonar | Pagamento por uso |
| Busca na web | Uma chave do Brave Search | 2.000 consultas gratuitas por mês |
### Keychain do macOS (opcional)
No macOS você pode guardar as chaves no Keychain do sistema em vez de em um arquivo `.env`. A skill as encontra automaticamente como a origem de menor prioridade — em caso de conflito, os arquivos `.env` e o ambiente do processo continuam ganhando.
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
Os itens ficam guardados sob o nome de serviço `last30days-<KEY>` para o usuário atual. Em plataformas que não são Darwin o carregador não faz nada, então não há mudança de comportamento para quem usa Linux ou Windows.
Já tem chaves guardadas com outros nomes de serviço no Keychain? Defina o mapeamento não secreto `LAST30DAYS_KEYCHAIN_ALIASES` descrito em [CONFIGURATION.md](CONFIGURATION.md#reusing-existing-macos-keychain-items), em vez de copiar segredos.
Veja [CONFIGURATION.md](CONFIGURATION.md) para a matriz completa de chaves por fonte, a ordem de prioridade dos provedores de raciocínio e a dos backends de busca web.
## Configuração
Duas coisas que você provavelmente vai querer saber no primeiro dia:
**Onde os arquivos de pesquisa são salvos.** O `LAST30DAYS_MEMORY_DIR` aponta por padrão para `~/Documents/Last30Days/` (no Windows: `C:\Users\<you>\Documents\Last30Days\`). Para mudar, defina essa variável de ambiente no seu shell com o caminho que quiser, ou use `--save-dir <path>` em uma execução específica. Use `--output <file>` quando precisar do resultado renderizado em um caminho exato, no formato escolhido por `--emit`. Use `--save-suffix=<name>` para manter separadas várias variações do mesmo assunto (por cliente, por exemplo). Cada execução com `--save-dir` gera `<slug>-raw[-suffix].md`. Rode `python3 skills/last30days/scripts/last30days.py --preflight` para conferir o que será escrito antes de disparar uma pesquisa.
**Saída estruturada para agentes e fluxos de trabalho.** Peça ao `/last30days` um JSON legível por máquina e você recebe o perfil de agente estável e versionado. Para usar o motor direto em scripts ou no desenvolvimento, rode `python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json`; use `--json-profile=raw` só quando precisar do dump interno não versionado do `Report`. Veja a [referência de campos da exportação JSON e a política de versionamento](docs/reference/json-export.md).
**Descoberta sem assunto definido.** Pergunte `/last30days what's trending in AI agents?` para receber um briefing de descoberta ordenado, em vez de pesquisar um assunto que você já conhece. Em um host com agente, isso executa o protocolo de três comandos arbitrado pelo host (o modelo nomeia os assuntos, filtra ruído, avalia o que vale a pena e escreve os ângulos de conteúdo). Para usar o motor direto em scripts ou no cron, rode `python3 skills/last30days/scripts/last30days.py --discover "AI agents"` (passada única: nomes de assunto determinísticos, sem ângulos); acrescente `--emit=json` para o contrato de descoberta versionado. A descoberta é mutuamente exclusiva com um assunto posicional e com `--drill`.
**Monitoramento de tendências entre execuções.** O modo padrão gera um snapshot Markdown novo a cada execução. Para acumular achados ao longo do tempo, acrescente `--store` e eles ficam guardados em um banco SQLite; depois use [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) para execuções agendadas (com envio opcional por Slack ou webhook quando surgirem achados novos) e [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) para resumos diários ou semanais. O padrão de cadência completo está em [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
**Uma biblioteca de pesquisa que dá para assinar.** Peça ao `/last30days` que monte o feed da sua biblioteca, ou use direto `python3 skills/last30days/scripts/last30days.py library feed` para scripts e desenvolvimento. Ele transforma os briefings salvos em um `index.html`, um `feed.xml` Atom local e páginas de briefing legíveis. Acrescente `--publish` só quando quiser hospedar o índice HTML e as páginas de briefing; publicar é uma decisão explícita e, por padrão, é público. Para o feed Atom ficar realmente assinável, hospede o diretório de saída gerado em um serviço estático como o GitHub Pages.
**Busque em tudo que você já pesquisou.** Pergunte `/last30days search my library for MCP servers` ou `/last30days have I researched MCP servers before?`. Para usar o motor direto, rode `python3 skills/last30days/scripts/last30days.py library search "MCP servers"`. A busca é offline e determinística: ela indexa aos poucos os mesmos briefings salvos que o feed da biblioteca usa, junta as ocorrências correspondentes registradas no store a cada execução e agrupa os resultados por assunto e data. Execuções novas também exibem uma seção compacta **From your library** ("da sua biblioteca") quando pesquisas anteriores se sobrepõem ao assunto atual; defina `LAST30DAYS_LIBRARY_CONTEXT=off` para desativar esse contexto passivo.
Scripts wrapper por cliente, subreddits de categoria personalizados e o canal beta experimental para personalizações em andamento também estão documentados em [CONFIGURATION.md](CONFIGURATION.md).
## Vitrine: feeds de pesquisa da comunidade
Publicou com o last30days um panorama recorrente de IA, um acompanhamento de mercado ou uma obsessão maravilhosamente específica? Compartilhe a URL da sua biblioteca pública — ou a URL do Atom, depois de hospedar o `feed.xml` em um serviço estático — na [thread de vitrine da comunidade](https://github.com/mvanhorn/last30days-skill/issues/532). Os feeds da comunidade serão linkados aqui conforme as pessoas os enviarem; enquanto isso, a thread é o ponto de coleta.
## Como funciona
1. **Você digita um assunto.** Pessoa, empresa, produto, tecnologia, "X vs Y". Qualquer coisa.
2. **O agente descobre quem importa.** Ele encontra os perfis do X (inclusive de fundadores), os repositórios do GitHub, os subreddits, as hashtags do TikTok e os canais do YouTube. Para "Kanye West" ele sabe que o caminho é r/hiphopheads, @kanyewest e "bully review" no YouTube. Para "OpenClaw" ele resolve openclaw/openclaw no GitHub e busca a contagem de estrelas ao vivo.
3. **Todas as fontes buscadas em paralelo.** Expansão com várias consultas. Resultados pontuados por engajamento, relevância e frescor.
4. **A profundidade que ninguém mais tem.** Transcrições completas do YouTube de vídeos de reação. Os melhores comentários do Reddit com a contagem de votos positivos. As legendas dos TikToks. As probabilidades do Polymarket. Não só títulos e links.
5. **Mesma história, unificada.** O Wireless Festival anunciado no Reddit, discutido no X e com preço de ingresso no TikTok vira um cluster só, não três itens separados.
6. **Sintetizado em um único briefing.** Ancorado em dados específicos. Citado por fonte. Ordenado pelo que as pessoas realmente engajam. Não é "olha o que eu encontrei", é "olha o que importa".
7. **E então ele vira o seu especialista.** Depois de uma única execução, sua sessão do Claude sabe tudo o que a comunidade sabe. Faça perguntas de acompanhamento. Peça para escrever prompts, redigir e-mails, planejar viagens, desenhar arquiteturas — tudo ancorado no que é real agora.
## O que as pessoas estão dizendo
> "Achei uma skill do Claude Code que pesquisa qualquer assunto no Reddit, X, YouTube e HN dos últimos 30 dias. E ainda escreve os prompts pra você. Antes de cada conteúdo que eu escrevo, eu fazia essa busca na mão no Reddit e no X. Aba por aba. Thread por thread. É justamente essa a parte que leva 90 minutos. Isso elimina ela." — @itsjasonai
> "Essa skill sozinha substituiu todo o meu fluxo de pesquisa. Você dá um assunto e ela raspa Reddit, X e a web atrás do que as pessoas estão falando de verdade. Nada de post de blog velho. Conversas reais dos últimos 30 dias." — @itswilsoncharles
> "5 dos 10 repositórios em alta no GitHub hoje são ferramentas do Claude. O nº 1: mvanhorn/last30days-skill" — @yieldhunter95
## Código aberto
Licença MIT. Sem rastreamento. Sem analytics. Sua pesquisa fica na sua máquina. Mais de 2.700 testes.
Construído com Python 3.12+, yt-dlp, Node.js (cliente Bird embarcado para a busca no X) e a API do ScrapeCreators. Arquitetura do motor v3 por [@j-sperling](https://github.com/j-sperling).
Veja [CONTRIBUTING.md](CONTRIBUTING.md) para abrir um PR, [CONTRIBUTORS.md](CONTRIBUTORS.md) para a lista completa de quem contribuiu e [CHANGELOG.md](CHANGELOG.md) para o histórico de versões.
## Histórico de estrelas
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
+381
View File
@@ -0,0 +1,381 @@
# /last30days
[English](README.md) | [Français](README.fr.md) | [Deutsch](README.de.md) | [Español](README.es.md) | [Português (Brasil)](README.pt-BR.md) | [日本語](README.ja.md) | 简体中文
<p align="center">
<img src="media/pr-assets/last30days-ad.gif" width="720" alt="last30days——由 AI 智能体驱动、搜索真实用户而非编辑内容的搜索引擎" />
</p>
<p align="center">
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending 单日排名第一的仓库" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
**一个由 AI 智能体驱动的搜索引擎:按赞同票、点赞和真金白银评分,而不是由编辑决定。**
本文档对应当前的 v3 流水线。运行时 Skill 规范位于 [skills/last30days/SKILL.md](skills/last30days/SKILL.md),最新命令与配置行为以该文件为准。
**Claude Code(推荐——通过 marketplace 自动更新):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex、Cursor、Copilot、Gemini CLI,或其他 50 多个支持 [Agent Skills](https://agentskills.io) 的宿主:**
```
npx skills add mvanhorn/last30days-skill -g
```
`-g` 会安装到当前用户的全局环境,所有项目均可使用;去掉该参数则仅安装到当前项目。)
更多安装方式(claude.ai 网页版、OpenClaw、手动安装)见下方[安装](#安装)章节。
开箱即用。Reddit、Hacker News、Polymarket 和 GitHub 无需配置即可搜索。首次运行时,配置向导会在 30 秒内帮你解锁 X、YouTube、TikTok、arXiv、Techmeme 等更多来源。
---
Reddit 的赞同票、X 的点赞、YouTube 的完整字幕、TikTok 的互动数据,以及由真金白银和内幕信息支撑的 Polymarket 概率——每天都有数百万人用注意力和钱包投票。`/last30days` 会并行搜索这些平台,按照真实用户的参与度评分,再由 AI 智能体裁判综合成一份简报。
Google 聚合编辑选出的内容,`/last30days` 搜索真实的人。
你无法从别的单一搜索产品获得这些结果,因为没有哪个 AI 天生能访问所有平台。Google 搜不到 Reddit 评论和 X 帖子;ChatGPT 虽然与 Reddit 合作,却无法搜索 X 或 TikTokGemini 能访问 YouTube,却没有 RedditClaude 原生不具备这些能力。每个平台都是一座围墙花园,有自己的 API、令牌和认证机制。但只要接入你自己的密钥和浏览器会话,AI 智能体就能同时搜索所有平台、横向比较信号,并告诉你真正值得关注的内容。
这才是关键:不是再造一个更好的搜索引擎,而是让一个智能体把十几个彼此割裂的平台连接起来。
```
/last30days Peter Steinberger
```
假设你明天要和一个人开会。用 Google 搜他,你看到的可能还是 2023 年的 LinkedIn 页面;`/last30days` 告诉你的则是他这个月真正做了什么:加入 OpenAI 参与 Codex、反对 Anthropic 禁止第三方智能体、提交 23 个 PR 且合并率达到 85%、打造用于跨设备智能体控制的 “LobsterOS”,以及 r/ClaudeCode 上一场获得 569 个赞同票的争论——他究竟是英雄,还是“令人难以忍受”。这些信息散落在 X 帖子、Reddit 讨论、YouTube 字幕和 GitHub 提交中,Google 上根本没有。
## 为什么要做这个项目
最初,我做它是为了跟上 AI 的变化。这个领域每天都在变,而 Reddit 和 X 上的极客通常最先发现新东西。我需要更好的提示词,但模型训练数据总比社区已经摸索出的经验慢几个月。
后来,它变成了更大的东西。现在,销售通话前,我用它了解一家公司过去 30 天的真实动态;开会前,我用它读完对方最近的推文和播客字幕;去迪士尼世界前,我用它确认哪些项目停运、社区怎么看 Genie+;开始做任何产品前,我用它找出人们真正遇到的问题。
如果你要见一位 CEO,你读过他过去 30 天的所有推文和 YouTube 字幕吗?我读过。
## 由真实用户评分的信息源
| 来源 | 人们会告诉你什么 |
|------|------------------|
| **Reddit** | 未经过滤的真实看法。免费获取带实际赞同数的热门评论,无需 API 密钥;那些常被 Google 埋没的真实意见。 |
| **X / Twitter** | 犀利观点、专家长帖和突发事件的第一反应。最早知道,也最早争论。 |
| **YouTube** | 45 分钟的深度内容。搜索完整字幕,只提取真正值得引用的 5 句话。 |
| **TikTok** | 一个触达 360 万人的创作者观点——你永远不会在 Google 上搜到。 |
| **Instagram Reels** | 带口播字幕的影响者视角,反映视觉文化的信号。 |
| **Hacker News** | 开发者共识:825 分、899 条评论,技术从业者真正交锋的地方。 |
| **Polymarket** | 不是观点,而是由真金白银支撑的概率:专辑销量 96%,收购概率 4%。 |
| **GitHub** | 搜人时查看 PR 速度、按 Star 排名的热门仓库和发行说明;搜主题时查看 Issue 与 Discussion。 |
| **Digg** | 来自 Digg AI 1000 排行榜(约 1,000 个高信号 X 账号)的精选话题聚类,包含可追溯的行内引用,无需 X 认证。当 PATH 中存在 `digg-pp-cli` 时自动启用。 |
| **arXiv** | 热点背后的论文。免费查找时间窗口内的新研究,无需 API 密钥。当 PATH 中存在 `arxiv-pp-cli` 时自动启用(首次配置会安装)。 |
| **Techmeme** | 科技新闻的编辑视角,并按你设定的 30 天窗口筛选。免费且无需 API 密钥。当 PATH 中存在 `techmeme-pp-cli` 时自动启用(首次配置会安装)。 |
| **LinkedIn** | 职业领域的信号。搜索帖子和文章,其中文章被视为高价值信号。 |
| **StockTwits** | 交易者情绪。当主题是股票代码或加密货币时自动启用。 |
| **Threads** | 后 Twitter 时代的文字内容层,汇集创作者和品牌的讨论。 |
| **Pinterest** | 视觉发现:围绕产品和创意的 Pin、收藏与评论。 |
| **小红书(RED** | 来自中国生活方式、产品和创作者的信号。当本机运行已登录的 x-mcp 浏览器插件或 `xiaohongshu-mcp` 服务时,通过 `--search xhs` 显式启用。 |
| **Bluesky** | 去中心化的社交内容层,搜索 Twitter 用户迁移后产生的 AT Protocol 帖子。 |
| **Perplexity** | 基于来源的 Sonar 综合结果、原始 Search API 数据和 Deep Research。 |
| **Web** | 编辑报道和博客对比。它只是众多信号之一,而不是唯一来源。 |
社区贡献者仍在不断加入更多平台。Truth Social 等垂直来源已经进入引擎,更多来源也在路上。
一条获得 1,500 个赞同票的 Reddit 帖子,信号强度高于一篇无人阅读的博客;一个拥有 360 万次观看的 TikTok,比新闻稿更能说明当下的文化热点;一个有 6.6 万美元成交量支撑的 Polymarket 概率,也比评论员的猜测更难反驳。
综合排序依据的是人们真正参与过的内容——看社会相关性,而不是 SEO 相关性。
## 大家实际上怎么用它
**开会之前。** `/last30days Peter Steinberger`——加入 OpenAI Codex 团队、反对 Anthropic 禁止第三方智能体、GitHub 上合并了 23 个 PR 且合并率达 85%、正在开发跨设备智能体控制系统 LobsterOS。r/ClaudeCode 上的一条评论说:“自从 OpenClaw 发布之后,大家就知道,只要你不是通过 API 运行它,迟早会被封。”(227 个赞同票)。这些不会出现在 LinkedIn 上。
**判断招聘信号。** `/last30days Listen Labs --hiring-signals`——把最新职位和招聘页面变成有引用依据的证据,从中判断公司是否正转向企业安全、客户成功、基础设施或产品扩张。报告只解释招聘看起来释放了什么信号,不会武断预测路线图一定会交付什么。
**在话题爆发前发现它。** 输入 `/last30days what's exploding in AI agents?`,Skill 会切换到发现模式:引擎扫描 Reddit 分类列表、Hacker News 的 front/best 故事、Digg AI 1000 信息流,以及认证后的 X;随后由你的智能体评审候选主题(命名、过滤垃圾、判断内容价值),并写出播客或 X 长文的切入角度;最终给出 5–10 个按增长速度排序的话题。每条结果都包含跨平台数据、势头标签,以及可直接运行的 `/last30days "<topic>"` 后续命令。
**突发事件发生时。** `/last30days Kanye West`——英国拒绝其签证,Wireless Festival 取消演出,赞助商纷纷离场;但《BULLY》首周登上 Billboard 第二名。Fantano 结束自己的 “Yay sabbatical” 回归评测(65.3 万次观看);SoFi Homecoming 请来 Lauryn Hill 和 Travis Scott,共演出 44 首歌。Polymarket:“Kanye 还会再发推吗?”86% 认为会。共找到 23 个 Reddit 主题、17 个 YouTube 视频和 8.6 万次赞同。
**比较工具。** `/last30days OpenClaw vs Hermes vs Paperclip`——“它们并非竞品,而是处于不同层次。”OpenClaw 是执行层(GitHub 35.1 万 Star,已上线),Hermes 是会自我改进的大脑(3.1 万 Star),Paperclip 是组织结构图(4.9 万 Star)。Star 数来自 GitHub API 的实时数据,不是过期博客。报告会提供架构、记忆、安全性和适用场景的横向表格。正如 @IMJustinBrooke 所说:“OpenClaw = 小火龙,Hermes = 喷火龙。”
**理解世界。** `/last30days Iran vs USA`——战争进入第 38 天。特朗普要求伊朗在周二的最后期限前重新开放霍尔木兹海峡;两架美国战机被击落;油价涨至每桶 126 美元。IEA 称之为“全球石油市场史上最大规模的供应中断”。Polymarket 认为 12 月 31 日前停火的概率为 74%。共找到 27 条 X 帖子、10 个 YouTube 视频和 20 个预测市场。
**旅行之前。** `/last30days Universal Epic Universe`——扩建工程已经开工,“Project 680” 许可已提交;基础设施证实将有烟花表演,但官方尚未公布。Mine-Cart Madness 平均排队 148 分钟;年票仍未推出,当地居民对此不满;Stardust Racers 将停运翻修至 4 月 5 日。
**快速学习。** `/last30days Nano Banana Pro prompting`——JSON 结构化提示词正在取代标签堆砌;@pictsbyai 的嵌套格式能避免“概念串色”;以编辑为先的工作流优于反复重新生成。随后,它会严格依据社区验证有效的方法,为你写出一条可用于生产的提示词。
## 最近更新
自 5 月发布 v3.3 公告以来,截至 v3.11.12026 年 7 月),项目已在 15 个版本中合并 175 个 PR,其中 122 个来自 52 位社区贡献者。下面是主要变化。
### 正式支持 OpenAI Codex
`/last30days` 现在是带引导式配置的原生 Codex 插件——不是简单移植,而是一等公民。针对不同渲染器优化的引用格式,让 Codex 输出读起来像简报,而不是一团 URL(#694)。同一套引擎也运行在 Claude Code、Cursor、Copilot、Gemini CLI、Claude Desktop、OpenClaw 以及 50 多个 Agent Skills 宿主上。Codex 插件清单由 [@rfoust](https://github.com/rfoust) 贡献(#686),Codex 认证修复由 [@tmchow](https://github.com/tmchow) 贡献(#698)。
### arXiv、Techmeme 与 Digg——免费,无需 API 密钥
arXiv 提供热点背后的论文,Techmeme 提供科技新闻的编辑视角;二者均免费、无需密钥,首次配置会安装相应 CLI 并自动启用(#709)。Digg 的 AI 1000 话题聚类同样无需 X 认证——配置过程会自动安装免费的 Digg CLI(#590)。此外还加入了可选的 Trustpilot 来源,适合消费品牌研究。
### 免费 Reddit 搜索也有真实评分和热门评论
Reddit 的公开 `.json` API 停止工作后,免费的数据通路以更强的方式回归:无密钥 RSS + shreddit 抓取(#457)、通过 arctic-shift 发现垂直 subreddit 并获取真实赞同数(#696),以及相关性下限,防止病毒式传播但偏题的帖子劫持整份简报(#488,感谢 [@rzachsmith](https://github.com/rzachsmith))。无需 API 密钥,提供真实评分和热门评论。
### 每份简报都收录最好的评论
评论现在是各来源默认启用的一层:Instagram 评论采用基于排名的多样性机制,避免五条热门观点全来自同一篇帖子(#751);YouTube 评论配合 ScrapeCreators 字幕回退,以应对 yt-dlp 失效(#637);经过社区投票的评论还会计入 Best Takes 的权重,让最有趣的金句不会在评分中消失(#592#608)。
### 一个 doctor 命令解决健康检查
要求执行健康检查时,doctor 会逐一测试所有来源,并给出精确修复建议:缺少哪个密钥、哪个 CLI 不在 PATH、哪个 Cookie 已过期(#753)。不必再猜为什么 X 的结果这么少。
### 重构 X 搜索
X 流水线经过彻底重构:新增 FROM 和 ABOUT 两条通路,让某人的原创帖子和外界对他的讨论都能进入排名(#610);按人物感知的子查询消歧(#611);基于第一方作者身份的信息归属,并结合互动信号排序(#613);统一的 X 来源与自动后端故障转移(#622)。另外,`--diagnose` 现在会真正探测认证状态,如实报告问题(#609)。
### 更多信息源加入
通过 ScrapeCreators 接入 LinkedIn,并将文章视为高价值信号([@ravstr](https://github.com/ravstr)#702)。StockTwits 会在股票代码和加密货币主题下自动启用([@wtiwana](https://github.com/wtiwana)#658)。Perplexity 新增直接 API 模式和异步 Deep Research[@sk-holmes](https://github.com/sk-holmes)#629)。
### 在社区协作下进一步加固
这一轮安全改进几乎全部来自社区:修复 HTML 渲染器中的存储型 XSS([@iliaal](https://github.com/iliaal)、[@aaronjmars](https://github.com/aaronjmars));收紧 Cookie 临时文件权限;通过 OpenSSF Scorecard 和构建来源证明加固 CI 供应链([@shaanmajid](https://github.com/shaanmajid)、[@hammadxcm](https://github.com/hammadxcm)、[@aniruddh909](https://github.com/aniruddh909));增加 Semgrep、OSV-Scanner 扫描以及 PR 依赖审查门禁([@23241a6749](https://github.com/23241a6749));测试覆盖率门槛从 60% 起步,现已提高到 84%([@gourab5139014](https://github.com/gourab5139014));Hermes 安全扫描中的所有 CRITICAL 问题也已清零(#768)。
### 覆盖范围更广
支持希伯来语和其他非拉丁文字语言([@dudyme](https://github.com/dudyme));为中文来源加入 CJK 感知的分词([@An-idd](https://github.com/An-idd));推进一系列 Windows 兼容性改进;支持从完整 Chromium 浏览器家族提取 Cookie——Brave、Edge、Vivaldi、Opera、Arc[@andrey-esipov](https://github.com/andrey-esipov))——并接入 macOS Keychain 和 Linux `pass(1)` 凭据来源。此外还有 `--as-of` 历史回溯([@chiyi-creator](https://github.com/chiyi-creator))、通过 uv 自动配置 Python 3.12[@buntysomroy](https://github.com/buntysomroy))、用于解读公司招聘页面的 `--hiring-signals`,以及多次运行之间的观察列表差异。
### v3 的核心能力仍然完整保留
v3 打下的基础都还在:真正调用 API 前先运行预研究模块,解析正确的账号、subreddit 和话题标签(由 [@j-sperling](https://github.com/j-sperling) 开发);Best Takes 评分在相关性之外也衡量幽默感和传播力;跨来源故事聚类;单次完成对比研究(例如 “CLI vs MCP” 只需 3 分钟,而不是 12 分钟);自动发现竞品的 `--competitors` 对比;GitHub 人物模式(`--github-user=steipete`);任何研究结束后可开启的 ELI5 模式(输入 “eli5 on”);以及可分享、自包含的 HTML 简报(`--emit=html`)。配置项详见 [CONFIGURATION.md](CONFIGURATION.md)。
## 安装
| 使用环境 | 安装方式 | 更新方式 |
|---------|---------|---------|
| **Claude Code**(推荐) | `/plugin marketplace add mvanhorn/last30days-skill` | 通过 marketplace 自动更新,或运行 `claude plugin update last30days@last30days-skill` |
| **Grok**xAI Build CLI | 先运行 `grok plugin marketplace add mvanhorn/last30days-skill`,再运行 `grok plugin install last30days` | `grok plugin update last30days` |
| **Codex、Cursor、Copilot、Gemini CLI,或其他 50 多个支持 [Agent Skills](https://agentskills.io) 的宿主** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **claude.ai**(网页) | [下载 `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill),然后在 claude.ai 中依次进入 Customize > Skills > + > Create skill > Upload a skill 上传 | 重新下载并上传 |
| **Claude Desktop** | 从[最新版本](https://github.com/mvanhorn/last30days-skill/releases/latest)下载适用于你的平台的 `.mcpb`,拖入 Settings > Extensions | 重新下载新包并拖入 |
| **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code(推荐)
```
/plugin marketplace add mvanhorn/last30days-skill
```
推荐这种方式,是因为 Claude Code marketplace 会替你处理更新:插件缓存按版本管理,每次发布新版本都会自动刷新。要强制检查更新,请运行 `claude plugin update last30days@last30days-skill`
如果你更愿意在 Claude Code 中使用 Agent Skills 的安装方式,同样支持:
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
原生插件和 `npx skills` 安装可以共存。但 Claude Code 不会对不同安装方式进行去重:若两者同时启用,`/last30days` 会出现两个条目。建议每台机器只选一种安装方式。
### GrokxAI Build CLI
[Grok Build](https://docs.x.ai/build/features/skills-plugins-marketplaces)`grok`)可以将 last30days 安装为原生插件。直接安装会跟踪仓库更新:
```bash
grok plugin install mvanhorn/last30days-skill
```
也可以先把本仓库添加为 marketplace 来源,再按插件名安装:
```bash
grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
加入 `--trust` 可跳过安装确认;使用 `grok plugin update last30days` 更新。为兼容旧机制,Grok 也会读取 Claude Code 的清单文件;原生 `.grok-plugin/` 文件是首选通路,也是 [xAI marketplace](https://github.com/xai-org/plugin-marketplace) 官方目录条目指向的对象。`npx skills add` 仍是有效的跨宿主备用方案。
### Codex、Cursor、Copilot、Gemini CLI 与其他 Agent Skills 宿主
通过开放的 [Agent Skills](https://agentskills.io) CLI 安装。它支持 50 多种运行环境,包括 `codex``cursor``github-copilot``gemini-cli``claude-code``windsurf``cline``continue``roo``aider-desk``opencode``goose` 等(完整列表见 [vercel-labs/skills 仓库](https://github.com/vercel-labs/skills))。
```bash
npx skills add mvanhorn/last30days-skill -g
```
`-g`(全局)参数会把 Skill 安装到用户目录,因此所有项目均可使用。不加 `-g` 时,`npx skills` 会安装到当前项目的 `./.skills/` 中,并随仓库提交。对于一个用于研究整个世界的工具,全局安装通常更合适。
Codex 桌面版和其他以文件夹为工作区的宿主,不仅能在 Git 仓库中运行,也能在普通文件夹中工作。第一次研究前,请让宿主智能体从已加载的 Skill 目录运行随附的 `scripts/last30days.py --preflight`;若在源码仓库中,则运行等价命令 `python3 skills/last30days/scripts/last30days.py --preflight`。该命令会展示配置来源、浏览器 Cookie 方案、计划写入的文件、可选命令和被忽略的项目配置,但不会读取 Cookie、写入文件或执行研究。
默认情况下,`npx skills` 会安装到它自动检测到的宿主。若要指定一个或多个宿主:
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
日后可通过以下命令更新:
```bash
npx skills update last30days -g
```
也可以一次更新所有通过 `npx skills` 全局安装的 Skill
```bash
npx skills update -g
```
使用 `npx skills list -g` 查看列表,使用 `npx skills remove last30days -g` 卸载。
### claude.ai(网页)
1. 从最新版本[下载 `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill)
2. 打开 [claude.ai > Customize > Skills](https://claude.ai/customize/skills)
3. 在 Skills 面板点击 `+`,再选择 `Create skill` > `Upload a skill`,浏览或拖入文件
请先在 Capabilities 中启用 “Code execution and file creation”——否则 Skill 无法运行。
### Claude Desktop
Claude Desktop 通过 `.mcpb` 包(一种一键式 Model Context Protocol 软件包)将 `/last30days` 安装为 MCP 服务器。
1. 打开[最新版本](https://github.com/mvanhorn/last30days-skill/releases/latest),下载适用于你的平台的 `.mcpb`
- macOS Apple Silicon`last30days-pp-mcp-darwin-arm64.mcpb`
- macOS Intel`last30days-pp-mcp-darwin-amd64.mcpb`
- Linux x86_64`last30days-pp-mcp-linux-amd64.mcpb`
2. 打开 Claude Desktop,进入 Settings > Extensions,将文件拖入。
3. 出现提示时,粘贴你想启用的数据源所需的 API 密钥。所有字段均可留空——如果全部跳过,引擎会降级为纯 Web 模式。密钥存储在操作系统的钥匙串中。
4. 重启 Claude Desktop。让 Claude “research Peter Steinberger” 或研究任意主题,它就会调用 `research` 工具。
**宿主要求:** PATH 中需要 Python 3.12+。软件包自带引擎源码,但使用本地 Python 解释器。Windows 用户可从 [python.org](https://www.python.org/downloads/) 安装;macOS 和大多数 Linux 发行版通常已提供兼容版本。
**密钥不会与 Code Skill 同步。** Claude Desktop 与 Claude Code 采用彼此独立的凭据存储,这是有意的设计。即使你已为 Code Skill 配置 `~/.config/last30days/.env`,仍需在这里重新输入一次相同的密钥。
Windows 支持需要等各平台的清单入口点确定后再实现,请关注后续 Issue。
### OpenClaw
```bash
clawhub install last30days-official
```
如果你需要在 `/last30days` 研究之外执行 X/Twitter 操作,例如发布推文或回复、导出关注者、处理媒体、监控账号或抽奖,可使用 [TweetClaw](https://github.com/Xquik-dev/tweetclaw) 作为配套 OpenClaw 插件。TweetClaw 由 Xquik-dev 维护,这里仅将其列为可选配套方案;它不是 last30days 的依赖,也不代表本项目为其背书。
### 手动安装(开发者)
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
```
这个符号链接会让安装内容随工作区代码实时同步,无需重复复制。若用于 `claude.ai`,可从源码构建 `.skill` 文件:运行 `bash skills/last30days/scripts/build-skill.sh`,产物位于 `dist/last30days.skill`
Reddit(含评论)、Hacker News、Polymarket 和 GitHub 无需任何配置即可使用。首次运行 `/last30days` 后,配置向导会在 30 秒内解锁更多来源,包括免费的 arXiv 与 Techmeme CLI。
## 使用你自己的密钥
这些平台之间互不相通:X 不知道 Reddit 在讨论什么,YouTube 也看不到 TikTok。但只要接入你自己的 API 密钥和浏览器令牌,就能一次访问所有平台。
| 来源 | 你需要准备什么 | 成本 |
|------|----------------|------|
| Reddit(含评论)+ HN + Polymarket + GitHub + StockTwits | 无 | 免费 |
| arXiv + Techmeme | 免费 CLI,由首次配置自动安装 | 免费 |
| X / Twitter | 在任意浏览器中登录 x.com,或设置 `XQUIK_API_KEY` / `XAI_API_KEY` | 浏览器 Cookie 免费;密钥费用取决于服务商 |
| YouTube | `brew install yt-dlp` | 免费 |
| Bluesky | 来自 bsky.app 的应用密码 | 免费 |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube 评论 | ScrapeCreators 密钥 | 前 10,000 次调用免费,之后按量付费 |
| 小红书(RED) | 运行已登录的 x-mcp 浏览器插件或 `xiaohongshu-mcp` 服务,并在单次运行中通过 `--search xhs` 启用,或在 `.env` 中设置 `INCLUDE_SOURCES=xiaohongshu`last30days 会依次自动探测 `http://localhost:18060``http://host.docker.internal:18060`,也可通过 `XIAOHONGSHU_API_BASE` 指定自定义地址 | last30days 不需要 API 密钥;依赖本地浏览器会话服务 |
| DripStack(付费金融通讯) | 每次运行通过 `--search dripstack` 启用,或在 `.env` 中设置 `INCLUDE_SOURCES=dripstack` | 无需密钥;公共搜索 API 免费 |
| Perplexity Sonar / Search API / Deep Research | Perplexity 密钥,或作为 Sonar 回退方案的 OpenRouter 密钥 | 按量付费 |
| Web 搜索 | Brave Search 密钥 | 每月 2,000 次免费查询 |
### macOS Keychain(可选)
在 macOS 上,你可以把密钥存入系统 Keychain,而不是 `.env` 文件。Skill 会自动将其作为最低优先级的密钥来源;发生冲突时,`.env` 文件和进程环境变量仍然优先。
```bash
# 交互式配置——逐个询问已知密钥,留空即可跳过
skills/last30days/scripts/setup-keychain.sh
# 也可以手动存入单个密钥
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# 查看 / 清理
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
密钥项以 `last30days-<KEY>` 作为服务名称,归当前用户所有。在非 Darwin 平台上,加载器不会执行任何操作,因此 Linux/Windows 用户的行为不受影响。
如果已有密钥使用其他 Keychain 服务名称,可按 [CONFIGURATION.md](CONFIGURATION.md#reusing-existing-macos-keychain-items) 中的说明设置不含秘密的 `LAST30DAYS_KEYCHAIN_ALIASES` 映射,无需复制密钥。
各来源的完整密钥矩阵、推理服务商优先级和 Web 搜索后端优先级,请参阅 [CONFIGURATION.md](CONFIGURATION.md)。
## 配置
第一天使用时,你大概最想知道以下两件事:
**研究文件保存在哪里。** `LAST30DAYS_MEMORY_DIR` 默认指向 `~/Documents/Last30Days/`Windows`C:\Users\<you>\Documents\Last30Days\`)。可以在 shell 中把该环境变量设为任意路径,也可以为单次运行传入 `--save-dir <path>`。若需要把渲染结果精确写入某个路径,请使用 `--output <file>`;文件格式由 `--emit` 决定。使用 `--save-suffix=<name>` 可分别保存同一主题的多个版本(例如按客户区分)。每次使用 `--save-dir` 都会生成 `<slug>-raw[-suffix].md`。研究前运行 `python3 skills/last30days/scripts/last30days.py --preflight`,可预览计划写入的内容。
**面向智能体和工作流的结构化输出。**`/last30days` 输出机器可读的 JSON,即可获得稳定且带版本号的 agent profile。若在脚本或开发中直接调用引擎,可运行 `python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json`;只有确实需要未版本化的内部 `Report` 转储时,才添加 `--json-profile=raw`。详见 [JSON 导出字段参考与版本策略](docs/reference/json-export.md)。
**无指定主题的趋势发现。** 输入 `/last30days what's trending in AI agents?`,会得到按排名整理的发现简报,而不是研究一个你已经知道的主题。在智能体宿主上,它会执行由宿主模型评审的三段式流程:模型命名主题、过滤垃圾、判断内容价值并撰写切入角度。在脚本或定时任务中直接调用引擎时,可运行 `python3 skills/last30days/scripts/last30days.py --discover "AI agents"`(单次执行:主题名称由确定性逻辑生成,不含内容角度);加入 `--emit=json` 可获得带版本号的发现数据契约。发现模式不能与位置参数主题或 `--drill` 同时使用。
**跨运行趋势监控。** 默认模式每次运行都会生成新的 Markdown 快照。若要长期积累结果,可添加 `--store` 写入 SQLite 数据库;随后使用 [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) 定时运行(发现新内容时可发送到 Slack 或 Webhook),使用 [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) 生成日报或周报。完整的周期配置见 [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings)。
**可订阅的研究资料库。**`/last30days` 构建你的资料库信息流;在脚本和开发中也可以直接运行 `python3 skills/last30days/scripts/last30days.py library feed`。该命令会把已保存的简报整理成 `index.html`、本地 Atom `feed.xml` 和便于阅读的简报页面。仅在确实想托管 HTML 索引和简报页面时添加 `--publish`;发布必须显式开启,且默认公开。若要让 Atom 信息流可订阅,请将生成目录托管到 GitHub Pages 等静态站点服务。
**搜索你做过的所有研究。** 输入 `/last30days search my library for MCP servers``/last30days have I researched MCP servers before?`。直接调用引擎时,运行 `python3 skills/last30days/scripts/last30days.py library search "MCP servers"`。搜索完全离线且结果确定:它增量索引资料库信息流使用的同一批简报,合并每次运行存储的匹配记录,并按主题和日期分组。新的研究若与历史内容重叠,还会显示精简的 **From your library** 章节;设置 `LAST30DAYS_LIBRARY_CONTEXT=off` 可关闭这种被动上下文。
各客户端包装脚本、自定义分类同类 subreddit,以及用于试验进行中定制功能的 beta 通道,也都记录在 [CONFIGURATION.md](CONFIGURATION.md) 中。
## 展示:社区研究信息流
你是否用 last30days 发布了定期 AI 动态、市场观察,或某个小众到可爱的长期专题?欢迎在[社区展示帖](https://github.com/mvanhorn/last30days-skill/issues/532)分享公开资料库 URL;若已将 `feed.xml` 托管到静态站点,也可分享 Atom URL。社区成员提交后,我们会在这里陆续添加链接;在此之前,该讨论帖就是统一的收集入口。
## 工作原理
1. **你输入一个主题。** 人物、公司、产品、技术、“X vs Y”——任何内容都可以。
2. **智能体识别关键对象。** 找出 X 账号(包括创始人)、GitHub 仓库、subreddit、TikTok 话题标签和 YouTube 频道。搜索 “Kanye West” 时,它知道该查 r/hiphopheads、@kanyewest,以及 YouTube 上的 “bully review”;搜索 “OpenClaw” 时,它会定位 GitHub 上的 openclaw/openclaw 并获取实时 Star 数。
3. **并行搜索所有来源。** 扩展多个查询,再按互动度、相关性和新鲜度评分。
4. **提供其他工具没有的深度。** 获取反应视频的完整 YouTube 字幕、带赞同数的 Reddit 热门评论、TikTok 文案和 Polymarket 概率,而不只是标题和链接。
5. **合并同一事件。** Wireless Festival 在 Reddit 官宣、在 X 上引发讨论、TikTok 出现票价信息——这些会合并为一个故事聚类,而不是三条重复结果。
6. **综合成一份简报。** 用具体数据作依据,为来源添加引用,并按真实互动排序。不是“这是我找到的内容”,而是“这是最重要的内容”。
7. **随后成为你的领域专家。** 运行一次后,当前 Claude 会话就掌握社区知道的一切。你可以继续追问,让它写提示词、起草邮件、规划旅行或设计系统架构——所有回答都基于此刻真实存在的信息。
## 用户怎么评价
> “我发现了一个 Claude Code Skill,可以研究任意主题过去 30 天在 Reddit、X、YouTube 和 HN 上的内容,然后替你写提示词。以前每写一篇内容,我都得手动在 Reddit 和 X 上做研究:一个标签页接一个标签页,一条讨论接一条讨论。光这一步就要 90 分钟。它彻底省掉了这些工作。” ——@itsjasonai
> “仅仅这一个 Skill,就取代了我的整套研究工作流。给它一个主题,它会抓取 Reddit、X 和 Web 上人们真正在谈论的内容。不是陈旧的博客,而是过去 30 天里真实发生的讨论。” ——@itswilsoncharles
> “今天 GitHub 的 10 个趋势仓库中,有 5 个是 Claude 工具。第一名:mvanhorn/last30days-skill。” ——@yieldhunter95
## 开源
采用 MIT 许可证。无跟踪、无分析,你的研究数据始终留在本机。拥有 2,700 多项测试。
项目基于 Python 3.12+、yt-dlp、Node.js(内置用于 X 搜索的 Bird 客户端)和 ScrapeCreators API 构建。v3 引擎架构由 [@j-sperling](https://github.com/j-sperling) 设计。
提交 PR 请参阅 [CONTRIBUTING.md](CONTRIBUTING.md),完整社区贡献者名单见 [CONTRIBUTORS.md](CONTRIBUTORS.md),版本历史见 [CHANGELOG.md](CHANGELOG.md)。
## Star 历史
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star 历史图" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
-1
View File
@@ -1 +0,0 @@
Top Community Comments ranking blends thread relevance with vote strength and applies the relevance floor only when enough on-topic candidates exist.
-1
View File
@@ -1 +0,0 @@
Chromium cookie extraction now searches every browser profile for a matching cookie set, and reuses the Keychain/AES key across profiles in one scan.
-1
View File
@@ -1 +0,0 @@
Synthesis contract is echoed at the top of the evidence envelope so hosts that truncate stdout still see the "synthesize, don't dump" directive.
-1
View File
@@ -1 +0,0 @@
`store_findings` no longer raises `TypeError` when a re-sighted finding carries `engagement_score: None`.
-1
View File
@@ -1 +0,0 @@
Explicit --plan payloads with an invalid schema now exit with a field-specific error instead of silently running a deterministic plan.
-1
View File
@@ -1 +0,0 @@
Company-topic runs no longer auto-add the `jobs` source when an explicit `--search` / `requested_sources` filter is set; `--hiring-signals` still forces jobs.
-1
View File
@@ -1 +0,0 @@
Report footer path now matches the collision-safe path actually reserved by save_output.
-1
View File
@@ -1 +0,0 @@
Polymarket domain-sweep topics no longer drop every market after noise-word stripping removes terms like "ai".
-1
View File
@@ -1 +0,0 @@
Reddit fetch windows now track the requested date range so short `--days` runs no longer pull a depth-default month and discard everything outside the window.
@@ -0,0 +1,40 @@
# feat(x): demote Grok CLI to opt-in backup
Stop using the Grok CLI as the default X backend. A leftover `~/.grok/auth.json` must never steal the X lane. Grok stays as a pin-only backup: off unless LAST30DAYS_X_BACKEND=grok or --x-backend grok.
### Requirements
- R1. Unpinned auto chain is bird → xai → xurl → xquik. Bird is first. Grok is not a member. Presence of ~/.grok/auth.json (ok, expired, or error) must not change which backend an unpinned run uses.
- R2. Grok remains a valid explicit selection: LAST30DAYS_X_BACKEND=grok and --x-backend grok. A pin forces grok with no failover. If grok is unusable, X is unconfigured and doctor/footer say so with the existing login hint.
- R3. Doctor "will use: grok" only when grok is pinned and the probe is OK or DEGRADED. Unpinned, grok may appear as unused opt-in ("available, unused — pin LAST30DAYS_X_BACKEND=grok"), never as the predicted winner.
- R4. get_x_source_status and get_x_source_with_method must prefer bird over xai/xurl/xquik when cookies are present. Grok wins only when the pin is grok.
- R5. Host docs stop presenting grok as the default keyless X path. Document it as opt-in backup. Default story is bird first, then xai / xurl / xquik.
- R6. Setup / first-run / prescriptions do not nag grok login as the fix for missing X. Cookie consent and paid keys remain the default prescriptions. Grok login is mentioned only as an optional pin.
- R7. Do not delete scripts/lib/grok_x.py, retrieve-judge-retry, or expires_at honesty. Pinned grok still uses them.
- R8. A machine with only a grok login (no cookies, no XAI/XQUIK, no xurl) has X unconfigured until the user pins grok. Footer: X skipped-unconfigured, not auth-failed-from-grok.
- R9. Tests cover the cases above; docs/changelog updated.
### Implementation units
U1 env.py: _X_BACKEND_ORDER = ("bird", "xai", "xurl", "xquik"); X_BACKEND_OPT_IN = ("grok",); X_BACKEND_KNOWN = ORDER + OPT_IN; pin uses KNOWN; unpinned walks ORDER only; get_x_source_status bird first, grok only if pinned; get_x_source_with_method bird before xai.
U2 backends.py / doctor.py / prescriptions.py: descriptor is auto ORDER then grok opt-in; unpinned collect-then-pick ignores opt-in; do not change _probe_grok honesty.
U3 SKILL.md, CONFIGURATION.md, README.md, README.pt-BR.md if needed, changelog.d: remove "sits ahead of the cookie path"; document bird → xai → xurl → xquik; pin grok to enable it.
U4 tests: unpinned grok-only empty; unpinned grok+bird → bird; unpinned bird+xai → bird; pin grok+store → ["grok"]; pin grok no store empty; doctor unpinned never predicts grok; descriptor parity treats grok as trailing opt-in.
### Tests T1T7
T1 unpinned grok AUTH_OK, no other creds → X unconfigured
T2 unpinned grok AUTH_EXPIRED, no other creds → X unconfigured (not will-use grok)
T3 unpinned grok AUTH_OK + cookies → bird
T4 unpinned XAI_API_KEY + grok store, no cookies → xai
T4b unpinned XAI_API_KEY + cookies → bird
T5 pin grok AUTH_OK → grok no failover
T6 pin grok no store → error / grok login prescription
T7 docs match R5
### Keep-the-door-open (KTD)
1. grok_x.py stays untouched
2. x_judge.py stays untouched
3. expires_at honesty stays untouched
4. auth.x.ai is never called
5. bird cookie extraction is unchanged
@@ -0,0 +1,71 @@
# Retrieve-Judge-Retry for X Search
**Date:** 2026-08-14
**Status:** Completed
## Problem Statement
X search results become off-topic when multi-word search queries are phrase-quoted. The Rome failure (2026-08-14) demonstrated this:
1. Planner generated `search_query: "Rome Italy"` (phrase-quoted)
2. Phrase-quoting returned thin hits with engagement bait (pretty-cities, geopolitics accounts)
3. `entity_extract` ranked off-topic handles (PrettyCitiesX, visegrad24) by frequency
4. `pipeline.py` promoted those handles to the FROM lane
5. FROM lane filled the 40-slot X budget with off-topic timelines
## Solution
Implement retrieve-judge-retry for X search:
1. **Query Compilation (R2):** X now uses `raw_topic` like Reddit/YouTube, not the planner's `search_query`
2. **Fanout Queries (R3):** Multi-word topics use unquoted AND as first variant; phrase-quote only for proper names
3. **Corpus Judging (R7):** New `x_judge.py` module evaluates corpus on-topic ratio after retrieval
4. **Retry (R1):** If off-topic flood detected (ratio < 0.4), retry ONCE with simplified keyword query inside the X stream (not `_retry_thin_sources`)
5. **Split FROM Promotion (R4):**
- Explicit handles (--x-handle): always FROM, no AND topic
- Extracted handles: FROM only if ≥2 on-topic hits AND ≥50% ratio, and they DO AND the topic
6. **First-Party Exemption (R5):** Floor immunity stays conservative (explicit handles only)
7. **Status Reporting (R6):** Off-topic floods emit artifact warning, not `record_failure(PARTIAL)`
## Implementation Details
### New Module: `x_judge.py`
- `judge_x_corpus(items, topic, ranking_query)`: Returns on_topic_ratio, is_off_topic_flood, on_topic_items, handle_stats
- `promotable_handles(items, topic, extracted_handles, explicit_handles)`: Returns (explicit_promotable, extracted_promotable)
- `should_retry_x_search(items, topic, depth)`: Returns True if retry warranted
- `prune_off_topic_items(items, topic)`: Returns only on-topic items
### Key Changes
- `grok_x._fanout_queries()`: No phrase-quote for place/disambiguation strings
- `grok_x._is_proper_name()`: Detects title-cased proper names for phrase-quoting
- `grok_x.search_handles()`: Added `and_topic` parameter (default False)
- `pipeline._fetch_x_backend()`: Accepts query directly, not subquery
- `pipeline._retrieve_stream_impl()`: X source uses raw_topic, judges corpus, retries if needed
- `pipeline._run_supplemental_searches()`: Uses `promotable_handles` for split FROM logic
### Tests
- `tests/test_x_judge.py`: New test file for x_judge module
- `tests/test_grok_x.py`: Added fanout and and_topic tests
- `tests/test_pipeline_v3.py`: Updated fixtures to have promotable content
## Success Criteria
- [ ] `_fanout_queries("Rome Italy")` has no `"Rome Italy"` variant
- [ ] `search_name("Peter Steinberger")` still phrase-quotes
- [ ] `search_handles(["steipete"], "topic")` does not AND topic by default
- [ ] `search_handles(["visegrad24"], "Rome", and_topic=True)` does AND Rome
- [ ] Explicit --x-handle always gets FROM lane
- [ ] Off-topic handles (visegrad24) not promoted to FROM lane
- [ ] On-topic handles (mamboitaliano__) promoted to FROM lane
- [ ] X source status is artifact warning, not PARTIAL failure
- [ ] All tests pass
## Out of Scope
- No live Grok calls in tests
- No auth/doctor touch
- No collision lexicon (AS Roma / Odunze still appear; judge + ranking_query drop them)
- bird_x quote-preserving `build_topic_query` (follow-up if needed)
@@ -0,0 +1,74 @@
# Fix Amazon Review Budget by Starting at Search Time
**Date:** 2026-08-14
**Status:** Implemented
## Problem (measured 2026-08-14 Bentgo run)
- Full multi-source run. Amazon search listings returned (12 products, stars, rating counts).
- Review lane logged: "pulling up to 50 reviews for 3 products (budget 11s)"
- "lane deadline 11s hit; dropped 3 straggling pull(s)"
- Bright Data timed out after 11s x3. Zero review bodies. Credits spent.
- Isolated Amazon-only re-run got the full 180s, finished in 124s, reviews landed.
- Cause: `_remaining_lane_budget` = min(LANE_DEADLINE=180, max(0, FOREGROUND_CONTRACT=300 - elapsed - RENDER_MARGIN=20)). Enrichment runs after all other sources. Elapsed ~269s → 11s.
## Why the floor-up fix was wrong
`max(120, leftover)` at elapsed=269 means the run goes to ~389449s. The host Bash contract is 300s (`SKILL.md` 300000ms). That kills the **whole** report. Do not do that.
The measured failure is **when** the review lane starts, not pull quality. Isolated Amazon-only already finishes in 124s of 180s. On a full run, search has already landed products, then reviews wait through every other source plus Phase 2/2b, then get 11s.
## What to build
1. Start `enrich_with_reviews` when Amazon search returns, inside `_retrieve_stream` (amazon branch ~4284), overlapping other source futures. Pass real `elapsed = time.monotonic() - run_started` (thread `run_started` into retrieve). After a 3090s search, leftover is 190250s → clamp to 180. Isolated Amazon-only unchanged.
2. Keep finalize (`_finalize_items_by_source` ~2956) as attach-if-missing only. `enrich_source_items` already no-ops if `top_comments` is set. Do not make finalize the only start. Do not enrich inline on the collect loop (that serializes other sources for 124s).
3. Leftover below a useful floor → budget **0**, skip the lane. Do not fire doomed 11s pulls (Bright Data `cli_timeout = max(5, timeout-10)` so budget 11s → CLI timeout 1s, still spends 3 credits). Suggested `MIN_USEFUL_REVIEW_BUDGET = 90`. Crumbs are a skip, not a short timeout.
4. If the lane is skipped or all pulls drop: Amazon `source_status` **PARTIAL** with detail like `review lane timed out` / `review lane skipped (budget 0s)`. Listings stay. Do not flip the whole source to `timeout` (search succeeded). Footer already shows ⚠ when state != ok — do not change render.py.
5. `depth=quick` still 0 pulls. `mock=True` still skips. No env knob. No raising LANE_DEADLINE or FOREGROUND_CONTRACT.
## Out of scope
- Do NOT change footer/render.py.
- Do NOT change X search or Grok auth.
- Do NOT add an env knob.
- Do NOT reorder the whole source schedule (deferred).
## Implementation
### amazon.py
- Added `MIN_USEFUL_REVIEW_BUDGET = 90`
- Changed `_remaining_lane_budget` to return 0 when below floor (not floor up)
- Changed `enrich_with_reviews` to return `(products, status_detail)` tuple where status_detail is:
- `None` for normal success
- `"review lane skipped (budget 0s)"` when budget is below floor
- `"review lane timed out"` when all pulls dropped
### pipeline.py
- Added `run_started` parameter to `_retrieve_stream`, `_retrieve_stream_impl`, and `_retry_thin_sources`
- Updated all call sites to pass `run_started`
- In the Amazon branch of `_retrieve_stream_impl`:
- Run search as before
- Calculate `elapsed = time.monotonic() - run_started`
- Call `enrich_with_reviews` immediately after search returns
- If enrichment reports a degraded status, add `_source_outcome` with `state=PARTIAL` to the artifact
- Updated finalize comments to note it's now attach-if-missing only
### Tests (test_amazon.py)
Retargeted existing tests:
- `test_lane_budget_shrinks_as_the_run_clock_advances` — now tests floor behavior
- `test_dropped_straggler_keeps_its_product_with_search_stats` — uses patched short LANE_DEADLINE
- `test_exhausted_wall_clock_skips_the_lane_entirely` — unchanged
New tests:
- `test_lane_budget_floor_prevents_doomed_pulls` — verifies elapsed=269 returns 0
- `test_lane_budget_constants_are_sane` — guards against constant drift
- `test_crumb_budget_skips_not_fires_doomed_pulls` — regression test for the Bentgo bug
- `test_early_elapsed_gets_full_budget` — verifies elapsed=40 gets 180s timeout
- `test_all_pulls_dropped_reports_timed_out_status` — verifies PARTIAL status on all-dropped
@@ -0,0 +1,63 @@
# Plan: Fix Grok Auth Honesty
**Date:** 2026-08-14
**Status:** Implemented
**PR:** fix(grok): treat expired sessions as degraded, not ok
## Problem (measured 2026-08-14 Peter Steinberger run on the user's Mac)
- grok binary on PATH. Doctor cached grok status ok / will use grok because `~/.grok/auth.json` existed with token markers.
- `stored_auth_status()` substring-scans for `refresh_token`/`access_token`/`auth_mode`. It never parses `expires_at`.
- The file had `expires_at 2026-08-14T01:26:53Z`, hours dead.
- A prior run at 07:47:26 UTC had `run_outcome` ok (2 items). Session was real.
- At 08:43 grok loaded auth, `is_expired` true, OIDC refresh → `invalid_grant` "Refresh token has been revoked". grok deleted auth.json.
- Engine exit 1 "Not signed in", fell back to bird (30 items via Safari cookies), lane flagged PARTIAL.
- Host told the user "Grok CLI is not signed in" as if it never was.
## Three states to distinguish
1. **No grok CLI** — silent fallback. Fine. Do not waste the user's time. Do not nag install on every research run.
2. **CLI installed, never logged in** — silent fallback. Fine.
3. **CLI installed, WAS logged in, session dead** — currently reports ok then partial. **This is the bug.**
## What was built
1. **`stored_auth_status` parses `expires_at` locally** (no network, no subprocess). Added `AUTH_EXPIRED` distinct from `AUTH_OK` / `AUTH_MISSING` / `AUTH_ERROR`. Never echoes token values. Finds `expires_at` anywhere in the vendor-keyed JSON object via recursive search.
2. **Doctor / `_probe_grok` does NOT map `AUTH_EXPIRED` to `health.OK`** or "will use: grok". Reports `DEGRADED`/warn + expiry timestamp + "refresh happens at run; if refresh was revoked, `grok login --device-auth`".
3. **Research-time `is_available` STILL attempts grok when a `refresh_token` marker is present** even if `access` `expires_at` is past. Expiry of the access token is not proof refresh is dead. Does not skip a refresh that might work.
4. **Auth revocation detection**: If grok exits "Not signed in" / RefreshTokenRejected / auth.json vanished mid-run: does not retry grok in that run. Falls back once. Typed outcome `auth-failed` (via `is_auth_revoked_error()` and `classify_run_failure()`), not a generic PARTIAL that reads as "the product half-worked."
5. **Host-facing copy for case 3**: SKILL.md updated with guidance: "X used <fallback> after the Grok session expired" + login hint. Not "Grok CLI is not signed in" when `run_outcome` shows it worked earlier.
6. **Doctor --probe still does not call xAI or grok.** Whole-doctor-path test patches `subprocess.run` to raise and still passes. `active_backend` stays a prediction; when `run_outcome.at` is stale or not ok, doctor says "will use grok, unverified since <time>".
7. **Tests**: Fixture stores (missing file, future `expires_at`, past `expires_at`, unparseable JSON). No network.
8. **SKILL.md**: Host reads `sources.x.run_outcome` and grok expiry warn; does not treat `active_backend` as verified; does not spend a turn installing grok unless the user asked for first-party X.
9. **Changelog fragment**: `changelog.d/+grok-auth-expired.fixed.md`. Tests pass with `uv run pytest`.
## Scope boundaries (NOT in this PR)
- X query construction, fanout, `search_name`, retrieve-judge-retry, and handle promotion are unchanged. That is a separate PR.
## Success criteria (all met)
- Past `expires_at` fixture → not grok ok.
- Future `expires_at` → still ok (not live-verified).
- No grok binary → no extra user-facing failure.
- Simulated "Not signed in" after prior ok `run_outcome` → typed `auth-failed` / fallback copy, not "never signed in."
- No-subprocess doctor test still passes.
## Files changed
- `skills/last30days/scripts/lib/grok_x.py``AUTH_EXPIRED`, `stored_auth_status()` returns 3-tuple, `is_auth_revoked_error()`, `classify_run_failure()`, `_invoke()` sets `auth_revoked`, `_run_query()` returns 3-tuple, `search_x()` propagates `auth_revoked`
- `skills/last30days/scripts/lib/backends.py``_probe_grok()` handles `AUTH_EXPIRED` as `DEGRADED`
- `skills/last30days/scripts/lib/pipeline.py``_fetch_x_backend()` propagates `auth_revoked`, `_classify_source_failure()` recognizes grok markers
- `skills/last30days/SKILL.md` — Grok session expiry handling guidance
- `tests/test_grok_x.py` — expires_at and auth revocation tests
- `tests/test_backend_descriptors.py` — grok expiry state tests
- `changelog.d/+grok-auth-expired.fixed.md` — release notes fragment
@@ -0,0 +1,29 @@
## Residual Review Findings
Run context: ce-code-review `mode:agent` on branch `fix/github-qualifier-strip` (head `42c5ab5bebcb3d4bd4d8bfc11f89b4df4bc1da9b`), plan `docs/plans/2026-08-07-001-fix-github-qualifier-collision-plan.md`, run id `20260807-231856-17902`. Findings not applied in LFG step 5; filed for durability.
### Filed (tracker: GitHub Issues)
- **P1**`skills/last30days/scripts/lib/github.py:237` — Qualifier-only topic classified as ERROR poisons retry eligibility — [mvanhorn/last30days-skill#951](https://github.com/mvanhorn/last30days-skill/issues/951) (settled-conflict: report-only per KTD-1)
- **P2**`skills/last30days/scripts/lib/github.py:186` — Quote-wrapped or paren-wrapped qualifiers bypass the strip — [mvanhorn/last30days-skill#952](https://github.com/mvanhorn/last30days-skill/issues/952)
- **P2**`skills/last30days/scripts/lib/github.py:231` — Empty or noise-plus-qualifier topics flip to hard ERROR — [mvanhorn/last30days-skill#953](https://github.com/mvanhorn/last30days-skill/issues/953) (settled-conflict: report-only per KTD-1)
- **P3**`skills/last30days/scripts/lib/github.py:229` — Repeated qualifier-only subqueries spam logs and error detail — [mvanhorn/last30days-skill#954](https://github.com/mvanhorn/last30days-skill/issues/954)
### Settled-conflict findings (report-only, not filed as apply requests)
- **P1**`skills/last30days/scripts/lib/github.py:237` — Qualifier-only topic classified as ERROR poisons retry eligibility — conflicts with KTD-1 (session-settled plan decision: qualifier-only topics return the error envelope). Downstream ERROR/attempted classification blocks `_retry_thin_sources`; filed as #951 for durability, not for application.
- **P2**`skills/last30days/scripts/lib/github.py:231` — Empty or noise-plus-qualifier topics flip to hard ERROR — conflicts with KTD-1/R3 (error envelope for qualifier-only/empty topics). Filed as #953 for durability, not for application.
### No sink / failed
None.
### Proceeded-and-flagged settled-decision conflicts (from ce-work step 2)
None — ce-work returned no `settled_decision_conflicts`.
### Residual risks carried from the review
- Error-envelope `context["core"]` is unstripped in the qualifier-only path vs stripped in the success path; no current consumer is affected.
- GitHub 422 behavior for unbalanced quotes is external API behavior, not exercised in tests.
- Planner emission of comma-glued, quoted, or wrapped qualifier shapes is LLM behavior; exposure is unquantifiable from code.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days-skill",
"version": "3.18.2",
"version": "3.21.0",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
+73 -11
View File
@@ -2,10 +2,12 @@
set -euo pipefail
# Check last30days configuration status and show appropriate welcome message.
# Priority for this status hook:
# .claude/last30days.env > ~/.config/last30days/.env > env vars > Keychain presence
# Priority for this status hook (mirrors lib/env.py):
# process env > trusted .claude/last30days.env > ~/.config/last30days/.env > Keychain presence
# Project-scoped config is loaded only when LAST30DAYS_TRUST_PROJECT_CONFIG is
# truthy in the process environment or the global config file — never from the
# project file itself (it cannot self-grant trust).
PROJECT_ENV=".claude/last30days.env"
GLOBAL_ENV="$HOME/.config/last30days/.env"
if [[ "${LAST30DAYS_CONFIG_DIR+x}" == "x" ]]; then
if [[ -n "$LAST30DAYS_CONFIG_DIR" ]]; then
@@ -68,6 +70,13 @@ load_env_vars() {
[[ "$key" =~ ^[[:space:]]*# ]] && continue
[[ -z "$key" ]] && continue
key="$(trim_ws "$key")"
# Only plain identifiers may reach `printf -v`. printf -v uses assignment
# semantics, so a key carrying an array subscript — e.g. `x[$(id)]` — has
# that subscript arithmetic-evaluated, which runs the command inside it.
# A project-scoped .claude/last30days.env is attacker-controlled as soon
# as an untrusted repo is opened, so an unvalidated key here is arbitrary
# code execution at session start.
[[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue
value="$(strip_outer_quotes "$(trim_ws "$value")")"
# Strip inline comments (# preceded by whitespace) to prevent
# command substitution in backtick-containing comments
@@ -81,19 +90,72 @@ load_env_vars() {
fi
}
# Determine which config file is active
# Match lib/env.py::_truthy — process/global trust signal only.
is_truthy() {
local v
v="$(trim_ws "$1")"
case "$v" in
1|[Tt][Rr][Uu][Ee]|[Yy][Ee][Ss]|[Oo][Nn]) return 0 ;;
*) return 1 ;;
esac
}
# Project config cannot self-grant trust. Process env (including empty/0 deny)
# wins when set; otherwise the global config file's trust flag is consulted.
project_config_trusted() {
if [[ "${LAST30DAYS_TRUST_PROJECT_CONFIG+x}" == "x" ]]; then
is_truthy "$LAST30DAYS_TRUST_PROJECT_CONFIG"
return $?
fi
is_truthy "${ENV_LAST30DAYS_TRUST_PROJECT_CONFIG:-}"
}
# Mirror lib/env.py::_find_project_env: walk up from $PWD for
# .claude/last30days.env, stopping at the git root, $HOME, or filesystem root.
# Prints the absolute path on stdout when found; returns 1 when none.
find_project_env() {
local dir candidate parent
dir="$PWD"
while :; do
candidate="${dir}/.claude/last30days.env"
if [[ -f "$candidate" ]]; then
printf '%s' "$candidate"
return 0
fi
# Stop at git root even if no project env was found there (matches env.py).
if [[ -e "${dir}/.git" ]]; then
return 1
fi
if [[ "$dir" == "$HOME" ]]; then
return 1
fi
parent="$(dirname "$dir")"
if [[ "$parent" == "$dir" ]]; then
return 1
fi
dir="$parent"
done
}
# Determine which config file(s) are active. Always load global first (when
# present) so a trust signal there can unlock the project file — matching
# lib/env.py, where the project file is never parsed before the trust check.
CONFIG_FILE=""
if [[ -f "$PROJECT_ENV" ]]; then
CONFIG_FILE="$PROJECT_ENV"
check_perms "$PROJECT_ENV"
elif [[ -f "$GLOBAL_ENV" ]]; then
if [[ -n "$GLOBAL_ENV" && -f "$GLOBAL_ENV" ]]; then
CONFIG_FILE="$GLOBAL_ENV"
check_perms "$GLOBAL_ENV"
load_env_vars "$GLOBAL_ENV"
fi
# Load config if found
if [[ -n "$CONFIG_FILE" ]]; then
load_env_vars "$CONFIG_FILE"
PROJECT_ENV=""
if project_config_trusted; then
# `|| true` keeps set -e from aborting when no project env is in the walk.
PROJECT_ENV="$(find_project_env)" || true
fi
if [[ -n "$PROJECT_ENV" && -f "$PROJECT_ENV" ]]; then
CONFIG_FILE="$PROJECT_ENV"
check_perms "$PROJECT_ENV"
load_env_vars "$PROJECT_ENV"
fi
# Load Keychain item presence for status checks without reading secret values.
+2 -2
View File
@@ -2,7 +2,7 @@ module github.com/mvanhorn/last30days-skill/mcp
go 1.25.5
require github.com/mark3labs/mcp-go v0.56.0
require github.com/mark3labs/mcp-go v0.57.0
require (
github.com/google/jsonschema-go v0.4.2 // indirect
@@ -10,5 +10,5 @@ require (
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/text v0.39.0 // indirect
)
+4 -2
View File
@@ -14,8 +14,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mark3labs/mcp-go v0.56.0 h1:7aCj2wODCskMi08f923ADG+EfELZBdiKILny415cIS8=
github.com/mark3labs/mcp-go v0.56.0/go.mod h1:+8WclSK1ZUweCP3hvktSji8n8ABG/95QaEkeVE/Uwas=
github.com/mark3labs/mcp-go v0.57.0 h1:jzWKyCzdWnwnZt05cvcQQ+ngiUl2RnixXJa7Kj4qP1E=
github.com/mark3labs/mcp-go v0.57.0/go.mod h1:+8WclSK1ZUweCP3hvktSji8n8ABG/95QaEkeVE/Uwas=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
@@ -30,5 +30,7 @@ github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zI
github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+2 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "last30days-skill"
version = "3.18.2"
version = "3.21.0"
description = "Multi-source last-30-days research skill"
readme = "README.md"
requires-python = ">=3.12"
@@ -10,6 +10,7 @@ dependencies = []
dev = [
"pytest>=9.1.1,<10",
"pytest-cov>=7,<8",
"pyyaml>=6.0.2,<7",
"towncrier>=25.8.0,<26",
]
+46 -7
View File
@@ -1,6 +1,6 @@
---
name: last30days
version: "3.18.2"
version: "3.21.0"
description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web. Includes a doctor health check to diagnose broken or missing sources."
argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
@@ -412,7 +412,7 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
---
# last30days v3.18.2: Research Any Topic from the Last 30 Days
# last30days v3.21.0: Research Any Topic from the Last 30 Days
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). On hosts with `uv` and no Python 3.12+, the preflight may install a uv-managed CPython 3.12 (one-time ~28MB download, announced on stderr). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
@@ -584,6 +584,8 @@ Options (give each option the description shown):
- "Skip X - just the CLIs" - description: "No cookie reads. Still installs yt-dlp (YouTube), Digg, arXiv, and Techmeme." Run `FROM_BROWSER=off "${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/last30days.py setup`.
- "xAI API key for X instead" - description: "Use an api.x.ai key for X search (no cookie read), plus install yt-dlp (YouTube), Digg, arXiv, and Techmeme." Ask them to paste it, write `XAI_API_KEY` to `.env`, then run `FROM_BROWSER=off "${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/last30days.py setup`.
**Grok CLI is an opt-in backup, not a setup-time recommendation.** Do NOT check for grok first or offer it as a primary option during setup. A leftover `~/.grok/auth.json` must never steal the X lane. If the user mentions having a Grok account, tell them: "You can use the Grok CLI by pinning `LAST30DAYS_X_BACKEND=grok` in your `.env` after running `grok login`. This is opt-in because a leftover grok login should not take over X automatically." Do not call it free — it needs a Grok plan.
The consented `setup --allow-browser-cookies` run extracts cookies (Chrome/Chromium family first via the Keychain with no Full Disk Access, then Firefox and Safari as fallbacks; the winning browser is pinned for future runs only when it is Firefox or Safari, so Chrome never re-triggers the Keychain prompt on later runs) and best-effort installs yt-dlp (YouTube), the free keyless Digg CLI (`digg-pp-cli` via `@mvanhorn/printing-press-library install digg --cli-only`; Digg activates only when the binary is on the **agent subprocess PATH**, typically `$HOME/.local/bin`; setup reports honestly if installed off-PATH; recommend-only if `npx` is unavailable), plus the free keyless arXiv and Techmeme CLIs. Show the user what was found and installed - including whether Digg landed on PATH (active) or off-PATH (installed but not yet active).
**macOS Full Disk Access remediation (Safari fallback only).** Chrome and Firefox need no Full Disk Access; only the Safari fallback does. After the `setup` run, inspect its stderr. If it contains `Permission denied reading Cookies.binarycookies` and the platform is macOS, the OS blocked the Safari read - surface the fix instead of swallowing it: `macOS blocked the Safari cookie read. If your x.com login is in Chrome, you don't need this. To use Safari: System Settings > Privacy & Security > Full Disk Access > enable your terminal (or the Claude app), then I can retry.` Offer ONE retry of the `setup` command. If the user skips, continue.
@@ -675,6 +677,7 @@ Shown when a Claude Code user picks "Manual setup", or for anyone who wants to c
The magic of /last30days is Reddit comments + X posts together - and both are free. Add these to `~/.config/last30days/.env`:
**X/Twitter (pick one - the most important source):**
- **Grok CLI (no X credential):** install with `curl -fsSL https://x.ai/cli/install.sh | bash`, then `grok login`. No X account, no cookies, no API key. Needs a Grok plan; calls draw on it.
- `FROM_BROWSER=auto` - free. Reads your x.com login cookies live at search time (Firefox/Safari, never saved to disk).
- `XAI_API_KEY=xxx` - no browser access needed. Get a key at api.x.ai. Best for servers.
- `XQUIK_API_KEY=xxx` - keyless-style X via Xquik.
@@ -770,6 +773,8 @@ SKILL_DIR="<absolute path of the directory containing the SKILL.md you just Read
**Doctor health check:** When the user asks for a health check ("is X working?", "why is a source missing?", "what's broken?", "did setup work?"), run `"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" doctor` (append `--json` for the machine contract) and relay the audit and fix prescriptions. `doctor` renders a **four-state audit** - **WORKING** (verified this run/last run or keyless-always-on), **TURNED ON - UNVERIFIED** (configured/opted-in but no run evidence), **NOT WORKING** (configured but failing, or the last run errored), **COULD BE ON** (available, not yet configured) - one line per source, plus a **CLI-health** block for sources that need a downloaded binary and indented **backup/comment** sub-lanes. Two on-demand modes: `doctor --postmortem` reads the last run's `last-report.json` and reports what actually broke per source (Failed/Partial/Succeeded with fix hints) - reach for it right after a run that returned less than expected; `doctor --probe` runs a **bounded** live test (free HTTP + keyless CLI sources only; credit-gated sources are never probed) to verify WORKING instead of guessing, and the same bounded probe auto-fires on a plain `doctor` when there is no fresh run. Per-source probe deadline is `LAST30DAYS_DOCTOR_PROBE_TIMEOUT` (default 10s). **MANDATORY standing rule.** Before research that depends on login-backed sources (X via cookies, Reddit's ScrapeCreators backfill), consult `doctor --cached --json` — it serves the report cached at `~/.config/last30days/doctor-cache.json` within its TTL (`LAST30DAYS_DOCTOR_TTL` seconds, default 900) for the cost of one file read. Re-run live `doctor` only when the cache is stale or the previous run reported a degraded login-backed source. When X is in ACTIVE_SOURCES_LIST, announce its predicted backend from the report's `sources.x.active_backend` (e.g. "X will use: bird") in the pre-research status line.
**Grok session expiry handling:** The grok CLI backend for X reports three auth states: `ok` (non-expired credentials), `expired` (access_token `expires_at` is past), and `missing` (never signed in). When doctor reports grok as **degraded** with an expiry timestamp, say "Grok session expired at {timestamp}; will attempt refresh at run time. If refresh fails, run `grok login --device-auth`" — not "Grok CLI is not signed in" (which misrepresents the history). The refresh attempt happens automatically at research time: an expired access_token does not prove the refresh_token is dead. If the run then fails with `auth_revoked` or `invalid_grant`, the user truly needs to re-login. **Host-facing copy:** when `sources.x.run_outcome.state` is `auth-failed` and the prior run's outcome was `ok`, say "X used {fallback} after the Grok session expired — run `grok login --device-auth` to restore first-party X." Avoid "Grok CLI is not signed in" when `run_outcome` history shows it worked recently. Avoid proactively installing grok or prompting about grok unless the user asks for first-party X search; the cookie and XAI_API_KEY paths work without a Grok subscription.
Then display (use "and more" if 5+ sources, otherwise list all with Oxford comma):
@@ -858,6 +863,7 @@ Before running the engine, determine which flags apply to this topic and resolve
| `--github-user={user}` | Step 0.5b | Topic is a person who ships code (developer, engineer, CEO-who-codes, researcher) |
| `--github-repo={owner/repo}` | Step 0.5c | Topic is a product / project / open-source tool |
| `--trustpilot-domain={domain}` | Step 0.5d | Topic is a company / brand / service with a Trustpilot presence (passing the flag also auto-activates the opt-in Trustpilot source for this run) |
| `--amazon-query={keyword}` | Step 0.5e | Recent buyer sentiment would materially inform the report AND `brightdata` is on PATH and logged in. Keyword is brand-plus-category (`Weber grill`), and for a person topic it is their company's product line (`June Oven`), not their name. Also add `amazon` to `--search` |
| `--subreddits={sub1,sub2,...}` | Step 0.55 | Always — almost every topic has active Reddit communities |
| `--tiktok-hashtags={h1,h2,...}` | Step 0.55 | Always — inferred from topic |
| `--tiktok-creators={c1,c2,...}` | Step 0.55 | Creator / influencer / brand topics |
@@ -1016,6 +1022,36 @@ Store: `RESOLVED_TRUSTPILOT_DOMAIN = {domain or empty}`
---
### Step 0.5e: Decide the Amazon Buyer-Signal Lane (if `brightdata` is available)
**Availability first.** This lane exists only when the Bright Data CLI is on PATH and logged in (`--diagnose` reports `brightdata_installed` and `brightdata_authenticated`). If either is false the source does not exist, nothing changes, and you should skip this step entirely — do not mention it, do not suggest installing it mid-run.
**The one question to ask:** *would recent Amazon buyer sentiment materially inform this report?* Not "is this shopping" — the test is whether buyer evidence is real evidence for this topic.
| Topic | Fires? | `--amazon-query` |
|---|---|---|
| "Weber Grills" | Yes — brand topic where review signal is core evidence | `Weber grill` |
| "best bluetooth speaker under $100" | Yes — buying question, the whole point | `bluetooth speaker` |
| "Bentgo Box" | Yes — brand line | `Bentgo lunch box` |
| "Matt Van Horn" (CEO of June) | Yes — **and the keyword is the company's product, not the person** | `June Oven` |
| "Kanye West" | No — person/culture topic, buyer reviews are noise | — |
| "the 2026 election" | No — nothing to buy | — |
**Two mechanics that matter:**
1. **The keyword is yours to choose and is often not the topic.** Map person → company → product line using what you know plus what Step 0.55 surfaced. A "Matt Van Horn" run that searches Amazon for his name returns nothing; searching `June Oven` returns his company's product reviews, which is the actual signal.
2. **Phrase it as brand plus category, never bare brand.** A bare brand keyword lands on Amazon's ad-heavy page 1 and can miss the brand's own bestsellers — a live `Bentgo` search returned 57 competitor ads and missed the flagship, while `Bentgo lunch box` surfaced it. Say `Weber grill`, not `Weber`.
**`--search` is replace-not-add.** Passing `--search` narrows the run to exactly the sources listed, so include the full intended set: `--search reddit,x,youtube,amazon` — never a bare `--search amazon`, which would silently drop every other source.
**Cost and latency, so you can set expectations:** one credit for the product search plus one per review pull, 4 per typical run against a 5,000/month free tier. Review sampling adds roughly 30 seconds to 2 minutes at default depth. Quick depth pulls no reviews at all.
Store: `AMAZON_QUERY = {product keyword or empty}` — pass as `--amazon-query="{AMAZON_QUERY}"` and add `amazon` to `--search`.
**Skip this step if:** the CLI is unavailable, the topic has no consumer-product dimension, or the user set `EXCLUDE_SOURCES=amazon`.
---
## Agent Mode (--agent flag)
If `--agent` appears in ARGUMENTS (e.g., `/last30days plaud granola --agent`):
@@ -1356,6 +1392,7 @@ Only show lines for platforms where something was resolved. Skip empty lines. On
- **CRITICAL: Your PRIMARY subquery MUST include ALL of these sources: reddit, x, youtube, tiktok, instagram, hackernews, polymarket.** Never omit reddit (highest-signal discussion) or youtube (unique transcripts + official content). Secondary subqueries can target specific platforms.
- `search_query` should be concise and keyword-heavy - match how content is TITLED on platforms
- `ranking_query` should read like a natural language question
- **X disambiguation:** express your disambiguation intent in `ranking_query` (e.g., "What are people saying about Rome the city in Italy, not AS Roma or Rome Odunze?") — do not phrase-quote `search_query` for X or invent X operators; the engine handles X query compilation internally.
- **DISAMBIGUATION (mandatory for collision-prone names — the #1 cause of off-topic noise).** Anchor the `search_query` with the disambiguating context you resolved in Step 0.5 / 0.55 — the entity's company, role, or domain — when the topic name (a) is a common word or has non-product meanings ("Loom" = weaving tool, "Tella" = soccer player), OR (b) is a PERSON whose name collides with other public figures or common words. Apply the anchor to **EVERY subquery, not just the primary**, and mirror it in the `ranking_query`. Anchor on a SPECIFIC named entity (a company/product/firm), not a generic domain word. Examples: `"kevin rose digg founder"` not `"kevin rose"` (collides with Kevin Warsh / Leon Rose / Kevin Hart); `"lan xuezhao basis set ventures"` not `"lan xuezhao"` (collides with "Lanzhou" food, cdrama edits); `"trevin chow compound engineering"` not `"trevin chow"` (collides with Trevin Wax / Trevin Brown); `"tella screen recording"` not `"tella"`. The `ranking_query` carries the same anchor: `"ranking_query": "What has Kevin Rose, founder of Digg, been doing in the last 30 days?"`, not a bare `"...Kevin Rose..."`. A bare collision-prone name as a subquery is the named 2026-06-17 failure mode — "Kevin Rose" returned 55 items with ~0 about the actual founder until every subquery was anchored to "Digg founder". When the name is globally unambiguous (Kanye West, Nvidia, Peter Steinberger/OpenClaw), no anchor is needed.
- **For comparison queries**, each subquery should include the product category: "tella screen recorder review" not just "tella review", "loom video tool pricing" not just "loom pricing".
- NEVER include temporal phrases in search_query: no "last 30 days", "recent", month names, year numbers
@@ -1367,7 +1404,7 @@ Only show lines for platforms where something was resolved. Skip empty lines. On
- For how_to: prioritize YouTube (tutorials) and Reddit (guides)
- Primary subquery weight = 1.0, secondary = 0.6-0.8, peripheral = 0.3-0.5
**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key), digg (Digg clusters - only if `digg-pp-cli` is on PATH)
**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key), digg (Digg clusters - only if `digg-pp-cli` is on PATH), amazon (buyer reviews - only if `brightdata` is on PATH and logged in; see Step 0.5e)
**Intent → freshness_mode mapping:**
- breaking_news, prediction → `strict_recent`
@@ -1969,16 +2006,18 @@ Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Eu
If the research output contains a `**🔍 Research Coverage:**` block, render it verbatim right before the stats block. This tells the user which core sources are missing and how to unlock them. Do NOT render this block if it is absent from the output (100% coverage = no nudge).
**Just-in-time X unlock:** If X returned 0 results because no X auth is configured (no AUTH_TOKEN/CT0, no XAI_API_KEY, no FROM_BROWSER), offer to set it up right there:
**Just-in-time X unlock:** If X returned 0 results because no X auth is configured (no AUTH_TOKEN/CT0, no XAI_API_KEY, no FROM_BROWSER), offer to set it up right there.
**Call AskUserQuestion:**
Question: "X/Twitter wasn't searched. Want to unlock it?"
Options:
**Call AskUserQuestion.** Question: "X/Twitter wasn't searched. Want to unlock it?"
Default options (always presented first — cookie consent and paid keys are the primary X fix):
- "Scan my browser cookies (free)" - Get consent, run cookie scan, write BROWSER_CONSENT=true + FROM_BROWSER=auto to .env
- "I have AUTH_TOKEN and CT0 from my browser" - Ask them to paste each value, then write AUTH_TOKEN=<value>\nCT0=<value> to .env
- "I have an xAI API key" - Ask them to paste it, write XAI_API_KEY to .env
- "Skip for now"
**Grok CLI is an opt-in backup, not a default prescription.** After showing the modal, add one line: "If you have a Grok account and prefer to use it: install the Grok CLI (`curl -fsSL https://x.ai/cli/install.sh | bash`), run `grok login`, then set `LAST30DAYS_X_BACKEND=grok` to enable it." Do not describe the Grok path as free — it needs a Grok plan. Do not put grok first or as a primary recommendation; a leftover `~/.grok/auth.json` must never steal the X lane.
**THEN - Engine footer pass-through (right before invitation):**
**The research output ENDS with a deterministic footer block bracketed by `---` lines, starting with `✅ All agents reported back!` and ending with `📎 Raw results saved to {resolved LAST30DAYS_MEMORY_DIR}/<slug>-raw.md`. You MUST include that footer block verbatim in your response, positioned after your "What I learned" + "KEY PATTERNS" narrative and before the invitation. Do not recompute the stats. Do not reformat the tree. Do not paraphrase. Do not skip it. Do not add your own source lines. Copy the exact bytes.**
+112 -2
View File
@@ -677,8 +677,9 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
parser.add_argument("--web-backend", default="auto",
choices=["auto", "brave", "exa", "serper", "parallel", "none"],
help="Web search backend (default: auto, tries Brave then Exa then Serper then Parallel)")
choices=["auto", "brave", "exa", "serper", "parallel", "keyless", "none"],
help="Web search backend (default: auto, tries Brave then Exa then Serper then Parallel; "
"keyless forces the zero-key DuckDuckGo/SearXNG floor)")
parser.add_argument("--deep-research", action="store_true",
help="Use Perplexity Deep Research (~$0.90/query) for in-depth analysis. Requires PERPLEXITY_API_KEY or OPENROUTER_API_KEY.")
parser.add_argument("--hiring-signals", action="store_true",
@@ -731,6 +732,18 @@ def build_parser() -> argparse.ArgumentParser:
"Find the domain with `trustpilot-pp-cli search '<name>'`."
),
)
parser.add_argument(
"--amazon-query",
help=(
"Product keyword the amazon source searches, when that source is active. "
"Defaults to the topic. Supply it whenever the topic is not the product: "
"a person topic searches their company's product line "
"(--amazon-query='June Oven'), and a brand searches brand-plus-category "
"(--amazon-query='Weber grill', not 'Weber' -- a bare brand keyword lands "
"on an ad-heavy page that can miss the brand's own bestsellers). "
"Requires the brightdata CLI on PATH and logged in."
),
)
parser.add_argument(
"--competitors",
nargs="?",
@@ -2303,6 +2316,15 @@ def _render_save_and_print(
json_profile=args.json_profile,
register=audience.name,
)
if args.emit not in {"json", "html"} and not entity_reports:
# Markdown saves keep the complete debug artifact (all clusters
# and per-source items), matching the render_fn-less path in
# save_output and the comparison peer saves. Saving the compact
# stdout render instead made most collected evidence
# unrecoverable from the raw file (#923). The stdout re-render
# above still runs so the visible footer cites the real path,
# and the saved artifact carries the same citation.
return render.render_full(report, save_path=display)
return rendered
save_path = save_output(
@@ -2736,6 +2758,40 @@ def _run_library_search(
return 0
def _looks_like_entity_topic(topic: str) -> bool:
"""Whether a topic names a person, company, or product rather than a theme.
Keys on brevity, not capitalization. People type lowercase: "bentgo",
"peter steinberger" and "getenergy.com" are entity searches every bit as
much as their title-cased forms, and requiring a capital meant the most
common real-world spelling never resolved a handle.
A short topic is an entity search; a longer one is a theme. "Peter
Steinberger", "bentgo" and "getenergy.com" qualify; "best AI coding tools
2026" and "how to build agents that scale" do not. Question-shaped topics
are themes regardless of length.
Used only to decide whether resolving an X handle is worth one web search,
so a false negative costs the old behavior and a false positive costs a
single search.
"""
text = (topic or "").strip()
if not text or text.endswith("?"):
return False
words = [w for w in re.findall(r"[A-Za-z0-9_.@'-]+", text) if w]
if not words or len(words) > 4:
return False
if any(w.startswith("@") for w in words):
return True
# A theme reads as a phrase built from common words; an entity does not.
common = {
"best", "top", "how", "why", "what", "when", "vs", "versus", "guide",
"tips", "review", "reviews", "news", "latest", "update", "updates",
"trends", "tools", "and", "or", "for", "the", "with", "about",
}
return not any(w.lower() in common for w in words)
def main() -> int:
parser = build_parser()
# Use parse_known_args so setup sub-flags (--device-auth, --github,
@@ -3214,6 +3270,28 @@ def _main(
# without WebSearch (OpenClaw, Codex, raw CLI).
repos_from_auto_resolve = False
trustpilot_domain_is_hint = False
# Resolve automatically for entity-shaped topics even without the flag.
# A person or company topic whose handle the user did not supply is the
# case where first-party evidence is hardest to protect: the handle is
# absent from the topic and may never appear in retrieved mentions, so
# nothing downstream can identify the subject's own posts. One web
# search closes that. If it returns nothing, pipeline.run skips the X
# relevance floor entirely — a noisier report beats losing evidence.
# Skipped when a handle was already supplied, when an external plan
# owns resolution, or in mock runs.
if (
not args.auto_resolve
and not external_plan
and not args.x_handle
and not args.mock
and _looks_like_entity_topic(topic)
):
args.auto_resolve = True
sys.stderr.write(
"[AutoResolve] entity-shaped topic with no --x-handle; "
"resolving the subject's handle so its own posts are not pruned\n"
)
if args.auto_resolve and not external_plan:
from lib import resolve
resolution = resolve.auto_resolve(topic, config)
@@ -3223,6 +3301,8 @@ def _main(
if resolution.get("x_handle") and not args.x_handle:
args.x_handle = resolution["x_handle"]
sys.stderr.write(f"[AutoResolve] X handle: @{args.x_handle}\n")
# Empty x_handle is intentional: do not invent a lexical stand-in.
# pipeline.run treats an unidentified subject as "skip the X floor".
if resolution.get("github_user") and not args.github_user:
args.github_user = resolution["github_user"]
sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
@@ -3298,6 +3378,28 @@ def _main(
if keywords:
config["_polymarket_keywords"] = keywords
# Product keyword for the amazon source. Carried on config rather than
# threaded through the run signature (the _polymarket_keywords idiom):
# it is one optional string consumed in exactly two places.
if getattr(args, "amazon_query", None):
config["_amazon_query"] = args.amazon_query.strip()
# Unlike --trustpilot-domain, this flag deliberately does NOT
# auto-activate its source: the lane spends metered credits, so
# turning it on stays an explicit request. But silence is the
# wrong failure mode -- a model that resolves the keyword and
# forgets the --search token would otherwise get no signal at
# all that the flag did nothing.
_amazon_requested = (
(requested_sources and "amazon" in requested_sources)
or "amazon" in str(config.get("INCLUDE_SOURCES") or "").lower()
)
if not _amazon_requested:
sys.stderr.write(
"[Amazon] --amazon-query was set but the amazon source was not "
"requested; add it to --search (e.g. --search reddit,x,amazon) "
"or set INCLUDE_SOURCES=amazon. Ignoring the keyword.\n"
)
# vs-mode / plan routing: split a vs-topic into main + peers unless
# discover-N or an explicit --competitors-list already decided who runs.
topic, comp_enabled, comp_count, comp_explicit = apply_vs_competitor_routing(
@@ -3418,6 +3520,14 @@ def _main(
# leak across sub-runs. Each sub-run writes its own
# `_auto_resolve_context` into its local config copy.
entity_config = dict(config)
# The Amazon keyword is entity-SPECIFIC, unlike the depth caps
# this shallow copy exists to inherit. Leaving the main topic's
# keyword in place would search Weber SKUs for a Traeger peer,
# render a rival's products as that peer's buyer evidence, and
# multiply the metered spend by the number of entities. Drop it
# so each peer derives its own keyword from its own topic; a
# per-entity keyword can ride in the --competitors-plan entry.
entity_config.pop("_amazon_query", None)
plan_entry = comp_plan.get(entity.strip().lower(), {})
resolved = {
"entity": entity,
+877
View File
@@ -0,0 +1,877 @@
"""Amazon product and review signals via the Bright Data CLI.
Two-stage source, following the digg discover-then-enrich shape:
1. **Discovery** -- one ``amazon_product_search`` per run turns a
model-supplied product keyword into product records carrying live
aggregate stats (rating, rating count, price). Cheap and fast.
2. **Enrichment** -- ``amazon_product_reviews`` pulls a capped sample of
written reviews for the top few surviving products, in parallel, under
a lane deadline. Reviews ride on their product item as metadata
comments and feed community-voice weaving.
The signature signal is the fusion of those two: an all-time rating from
thousands of ratings, set against the average of just the reviews inside
the last-30-day window. When those disagree, something changed this month,
and the review text says what. No Amazon page shows that.
Metering (R13): one credit per pipeline request regardless of records
returned, so the caps here bound paid-tier *records*, not credits. A
default run is 1 search + up to 3 review pulls = 4 requests.
Field names and quirks below are verified against live payloads pulled
2026-08-13; see the plan's schema block. Three fields arrive doubled
(``review_posted_date``, ``review_header``, ``badge``) and are repaired
here rather than downstream.
"""
from __future__ import annotations
import re
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
from urllib.parse import urlparse
from . import brightdata, log
from .relevance import token_overlap_relevance
SEARCH_PIPELINE = "amazon_product_search"
REVIEWS_PIPELINE = "amazon_product_reviews"
DEFAULT_DOMAIN = "https://www.amazon.com"
# Reviews requested per pull. Uniform across topic shapes and depths by
# decision: billing is per *request*, not per record, so a bigger cap is
# free on the monthly credit tier, and the in-window sample is what the
# drift signal rests on. Live-verified that latency does not scale with
# this number (50 reviews in 22s vs 20 reviews in 115s on a slower SKU).
#
# It is a ceiling, never a quota -- a SKU with 31 total reviews returns 31.
MAX_REVIEWS = 50
# How many products get a review pull, per depth. Quick spends one credit
# on discovery only: aggregate stats with no recent window.
DEPTH_CONFIG = {
"quick": 0,
"default": 3,
"deep": 5,
}
SEARCH_TIMEOUT = 90
REVIEW_TIMEOUT = 180
# Wall-clock ceiling for the whole parallel review lane. Pulls that miss it
# are abandoned, and their products degrade to `quiet` rather than
# disappearing (a slow SKU is real and unrelated to the cap: one live pull
# took 115s).
LANE_DEADLINE = 180
# The engine's foreground contract. The lane deadline is clamped against
# whatever remains of it, minus room to render.
FOREGROUND_CONTRACT = 300
RENDER_MARGIN = 20
# Minimum useful budget for the review lane. Below this threshold, Bright
# Data pulls reliably time out (cli_timeout = max(5, timeout-10), so budget
# 11s → CLI timeout 1s). Crumbs are a skip, not a short timeout: firing
# doomed pulls still spends 3 credits with no reviews returned.
MIN_USEFUL_REVIEW_BUDGET = 90
# Minimum dated reviews inside the window before a drift arrow is honest.
# Live census: a 50-cap pull returned 31 records of which only 5 were
# inside 30 days, so an unguarded arrow would routinely publish a "trend"
# computed from one or two reviews.
MIN_DRIFT_SAMPLE = 5
RECENT_WINDOW_DAYS = 30
# Product names run long and pipe-delimited; the footer needs a scannable
# handle, not a title.
SHORT_NAME_MAX = 18
_STAR_FIELDS = (
("one_star", 1),
("two_star", 2),
("three_star", 3),
("four_star", 4),
("five_star", 5),
)
def _log(msg: str) -> None:
log.source_log("Amazon", msg, tty_only=False)
def _today() -> datetime:
return datetime.now(timezone.utc)
# --------------------------------------------------------------- parsing
def undouble(text: str) -> str:
"""Repair the CLI's doubled string fields.
Observed live: ``review_header`` arrives as ``"Best Box!Best Box!"`` and
``badge`` as ``"Verified Purchase, Verified Purchase"``. Handles the
exact-repeat case and the comma-joined repeat, and leaves anything else
untouched -- a genuinely repetitive title must survive intact.
"""
value = (text or "").strip()
if not value:
return ""
half, odd = divmod(len(value), 2)
# Only treat an exact repeat as doubling when the halves are substantial
# and look like a phrase rather than a syllable -- otherwise a real title
# of "ByeBye" or "NoNo" gets silently truncated to half of itself. The
# observed artifact doubles whole headlines, so requiring some length and
# either whitespace or terminal punctuation keeps the repair targeted.
if not odd and half >= 6 and value[:half] == value[half:]:
first = value[:half]
if " " in first or first[-1] in ".!?":
return first.strip()
parts = [p.strip() for p in value.split(",")]
if len(parts) == 2 and parts[0] and parts[0] == parts[1]:
return parts[0]
return value
_DATE_HEAD = re.compile(r"^([A-Z][a-z]+ \d{1,2}, \d{4})")
def parse_review_date(raw: Any) -> Optional[str]:
"""Pull the ISO date out of the CLI's prose-wrapped date field.
Live shape: ``"August 3, 2026Reviewed in the United States on August 3,
2026"``. Only the leading ``%B %d, %Y`` is trustworthy; the tail is
localized prose that varies by marketplace.
Returns ``YYYY-MM-DD`` or None.
"""
match = _DATE_HEAD.match(str(raw or "").strip())
if not match:
return None
try:
return datetime.strptime(match.group(1), "%B %d, %Y").date().isoformat()
except ValueError:
return None
def short_name(name: str, brand: str = "") -> str:
"""Derive a scannable footer handle from a long product name.
Live names are pipe-delimited marketing strings with the brand carried
in its own field rather than as a prefix ("Chill Max Leak-Proof XL
Bento-Style Lunch Box | Included Ice Pack Keeps Food Cold"). Take the
segment before the first delimiter, drop a leading brand token if one
did sneak in, and clip to a scannable width on a word boundary.
"""
text = re.split(r"[|(–—]", str(name or ""), maxsplit=1)[0].strip(" -,")
brand_token = str(brand or "").strip()
if brand_token:
# Word-boundary anchored: a bare startswith() eats into sub-brands and
# coincidental prefixes ("AnkerWork" under brand "Anker" would become
# "Work", "Chillax" under "Chill" would become "ax").
stripped = re.sub(
rf"^{re.escape(brand_token)}\b[\s\-,]*", "", text, count=1, flags=re.IGNORECASE
)
if stripped:
text = stripped.strip(" -,")
if len(text) <= SHORT_NAME_MAX:
return text
clipped = text[:SHORT_NAME_MAX].rsplit(" ", 1)[0].strip(" -,")
return clipped or text[:SHORT_NAME_MAX].strip()
def _as_float(value: Any) -> Optional[float]:
try:
result = float(value)
except (TypeError, ValueError):
return None
return result
def _as_int(value: Any) -> int:
try:
return int(value)
except (TypeError, ValueError):
return 0
def _is_sponsored(value: Any) -> bool:
"""The flag arrives as the string 'true'/'false', not a bool.
Recorded in metadata but never used to filter (R4): its distribution
swings hard with keyword phrasing, so filtering on it can blank the
lane on an unlucky query.
"""
if isinstance(value, bool):
return value
return str(value or "").strip().lower() == "true"
def _valid_product_url(url: str, domain: str) -> bool:
"""Accept only https URLs on the configured Amazon host."""
try:
parsed = urlparse(url)
expected = urlparse(domain or DEFAULT_DOMAIN)
except ValueError:
return False
if parsed.scheme != "https" or not parsed.netloc:
return False
host = parsed.netloc.lower().removeprefix("www.")
want = (expected.netloc or "").lower().removeprefix("www.")
return bool(want) and host == want
# Amazon ASINs are a fixed shape. Validating it matters because the value
# is interpolated into a URL that is then refetched through the CLI *and*
# rendered as a link in the report -- two sinks, one unvalidated API field.
_ASIN_RE = re.compile(r"^[A-Za-z0-9]{10}$")
def _valid_asin(asin: str) -> bool:
return bool(_ASIN_RE.match(asin or ""))
def canonical_product_url(url: str, asin: str, domain: str) -> str:
"""Strip Amazon's tracking tail down to a stable /dp/<asin> link.
Search records carry 200+ character URLs with session-scoped ``dib``
tokens. Those work but are unreadable in a report and unstable across
runs, which breaks dedupe on re-runs of the same topic.
Falls back to the (already host-validated) original URL if the ASIN is
not well-formed, so a malformed record can never shape the rebuilt URL.
"""
if not _valid_asin(asin):
return url
base = (domain or DEFAULT_DOMAIN).rstrip("/")
return f"{base}/dp/{asin}"
# ------------------------------------------------------------- discovery
def search_products(
keyword: str,
*,
domain: str = DEFAULT_DOMAIN,
config: Optional[Dict[str, Any]] = None,
timeout: int = SEARCH_TIMEOUT,
) -> Dict[str, Any]:
"""Run one product search. Never raises; returns the adapter envelope."""
query = (keyword or "").strip()
if not query:
return {"records": []}
# A leading dash would be parsed as a CLI option rather than a search
# term. The keyword is model-supplied and can be influenced by
# pre-research over untrusted web content, so reject rather than
# sanitize -- a keyword starting with '-' is never a real product.
if query.startswith("-"):
_log(f"rejecting option-shaped keyword: {query!r}")
return {"records": [], "error": "amazon keyword may not begin with '-'"}
_log(f"search '{query}' on {domain}")
response = brightdata.run_pipeline(
SEARCH_PIPELINE, [query, domain or DEFAULT_DOMAIN],
timeout=timeout, config=config,
)
if response.get("error"):
_log(f"search failed: {response['error']}")
else:
_log(f"search returned {len(response.get('records') or [])} records")
return response
def parse_search_response(
response: Dict[str, Any],
keyword: str,
*,
domain: str = DEFAULT_DOMAIN,
min_relevance: float = 0.15,
) -> List[Dict[str, Any]]:
"""Turn raw search records into deduped, relevance-gated product dicts.
Dedupe is by ASIN: live payloads repeat a single product up to five
times across the result set (64 unique of 66 records on one pull).
Relevance is scored against the *supplied keyword*, not the run topic,
because the model may search "June Oven" on a topic about a person.
"""
records = response.get("records") if isinstance(response, dict) else None
if not isinstance(records, list):
return []
today = _today().date().isoformat()
seen: Dict[str, Dict[str, Any]] = {}
for record in records:
if not isinstance(record, dict):
continue
asin = str(record.get("asin") or "").strip()
raw_url = str(record.get("url") or "").strip()
if not _valid_asin(asin) or not _valid_product_url(raw_url, domain):
continue
name = str(record.get("name") or "").strip()
brand = str(record.get("brand") or "").strip()
if not name:
continue
relevance = token_overlap_relevance(keyword, f"{brand} {name}".strip())
if relevance < min_relevance:
continue
num_ratings = _as_int(record.get("num_ratings"))
existing = seen.get(asin)
# Duplicates of one ASIN can disagree on rating count (variant-level
# records); keep the richest.
if existing and _as_int(existing.get("num_ratings")) >= num_ratings:
continue
seen[asin] = {
"asin": asin,
# Current-date stamped (KTD6, trustpilot precedent): a live
# aggregate rating is a fact about now, not about the product's
# launch date, so it must not be dropped by the 30-day filter.
"date": today,
"name": name,
"short_name": short_name(name, brand),
"brand": brand,
"url": canonical_product_url(raw_url, asin, domain),
"rating": _as_float(record.get("rating")),
"num_ratings": num_ratings,
"price": _as_float(record.get("final_price")),
"currency": str(record.get("currency") or "").strip(),
"badge": undouble(str(record.get("badge") or "")),
"sponsored": _is_sponsored(record.get("sponsored")),
"bought_past_month": _as_int(record.get("bought_past_month")),
"rank_on_page": _as_int(record.get("rank_on_page")),
"relevance": relevance,
}
products = sorted(
seen.values(),
key=lambda p: (p["num_ratings"], p["relevance"]),
reverse=True,
)
_log(f"{len(products)} unique on-keyword products after dedupe")
return products
def infer_brand(products: Sequence[Dict[str, Any]], keyword: str) -> str:
"""Detect a brand topic by matching record brands against the keyword.
This is the guard against paying to review a competitor. Rival brands
buy ads against a brand keyword and can outrank the brand's own catalog
on raw rating count: on a live "bentgo lunch box" search a competitor
held the top two slots and would have taken two of the three review
pulls, putting a rival's reviews in a Bentgo report.
Matching the *keyword's own tokens*, rather than picking the most
common brand in the results, is what keeps category topics unfiltered.
"best bluetooth speaker" names no brand, so nothing is constrained and
the top products across brands compete on merit -- which is exactly
what that topic shape wants.
"""
normalized_keyword = " ".join(re.findall(r"[a-z0-9]+", (keyword or "").lower()))
if not normalized_keyword:
return ""
keyword_tokens = set(normalized_keyword.split())
# Keyed by the lowercased brand so one vendor spelled two ways ("Bentgo"
# and "BENTGO" in the same result set) reads as one candidate. Without
# this the set has two members, the function bails, and the guard it
# exists to provide silently turns off.
candidates: Dict[str, str] = {}
for product in products:
brand = str(product.get("brand") or "").strip()
if not brand:
continue
brand_tokens = re.findall(r"[a-z0-9]+", brand.lower())
if not brand_tokens:
continue
# Multi-word brands ("Hydro Flask") can never match a single-token
# test, so compare the brand's whole token sequence against the
# keyword's -- otherwise the guard is off for every two-word brand.
if len(brand_tokens) == 1:
matched = brand_tokens[0] in keyword_tokens and len(brand_tokens[0]) > 2
else:
matched = " ".join(brand_tokens) in normalized_keyword
if matched:
# First spelling wins, so the result is deterministic across runs.
candidates.setdefault(brand.lower(), brand)
return next(iter(candidates.values())) if len(candidates) == 1 else ""
def select_enrichment_targets(
products: Sequence[Dict[str, Any]],
*,
limit: int,
brand: str = "",
keyword: str = "",
) -> List[Dict[str, Any]]:
"""Pick which products get a review pull.
Ranked by rating count, which is a coarse signal: search records carry
variant-level counts that can undercount badly (84 on a record whose
review pull reported 8,446). The review pull's own
``product_rating_count`` is authoritative once available.
Two filters run before the cut:
* **Brand**, supplied or inferred from the keyword (see ``infer_brand``).
The record's own ``brand`` field does the work, which also solves
accessory contamination outright -- a "grill brush for Weber" carries
the brush maker's brand, not Weber. A front-anchored name match covers
the few records where ``brand`` is null.
* **Variant collapse.** Live results repeat one product across colors
and sizes under distinct ASINs with near-identical names. Two of those
would burn two of three pulls on the same product and render as
duplicate footer entries, so only the best-ranked of each short-name
group stays eligible.
"""
if limit <= 0:
return []
pool = list(products)
wanted = (brand or "").strip().lower() or infer_brand(pool, keyword).lower()
if wanted:
matched = [
p for p in pool
if (p.get("brand") or "").strip().lower() == wanted
or (not (p.get("brand") or "").strip()
and str(p.get("name") or "").strip().lower().startswith(wanted))
]
if matched:
pool = matched
deduped: List[Dict[str, Any]] = []
seen_names: set[str] = set()
for product in pool:
key = (product.get("short_name") or "").strip().lower()
if key and key in seen_names:
continue
if key:
seen_names.add(key)
deduped.append(product)
return deduped[:limit]
# ------------------------------------------------------------ enrichment
def fetch_reviews(
product_url: str,
*,
max_reviews: int = MAX_REVIEWS,
config: Optional[Dict[str, Any]] = None,
timeout: int = REVIEW_TIMEOUT,
) -> Dict[str, Any]:
"""Pull a capped review sample for one product. Never raises."""
if not product_url:
return {"records": []}
return brightdata.run_pipeline(
REVIEWS_PIPELINE, [product_url, str(max_reviews)],
timeout=timeout, config=config,
)
def parse_reviews(response: Dict[str, Any]) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
"""Split a review payload into comment dicts and product-level stats.
Product-level fields (``product_rating``, ``product_rating_count``, the
``product_rating_object`` star distribution) ride on *every* review
record, so they are read off the first one.
Comments are built directly in the shared score/excerpt shape rather
than routed through ``normalize._remap_comments``, which strips keys it
does not know -- and rating, date, and verified are exactly the keys
this source needs to keep. Sorted newest first so the woven sample
favors recent voices.
"""
records = response.get("records") if isinstance(response, dict) else None
if not isinstance(records, list) or not records:
return [], {}
first = records[0]
distribution = first.get("product_rating_object")
stats: Dict[str, Any] = {
"product_rating": _as_float(first.get("product_rating")),
"product_rating_count": _as_int(first.get("product_rating_count")),
"star_distribution": distribution if isinstance(distribution, dict) else {},
}
comments: List[Dict[str, Any]] = []
for record in records:
if not isinstance(record, dict):
continue
body = str(record.get("review_text") or "").strip()
header = undouble(str(record.get("review_header") or ""))
excerpt = body or header
if not excerpt:
continue
comments.append(
{
# Shared comment shape: downstream weaving reads score/excerpt.
"score": _as_int(record.get("helpful_count")),
"excerpt": excerpt,
"author": str(record.get("author_name") or "").strip(),
"rating": _as_int(record.get("rating")),
"date": parse_review_date(record.get("review_posted_date")),
"verified": bool(record.get("is_verified")),
"vine": bool(record.get("is_amazon_vine")),
"title": header,
}
)
# Newest first; undated records sink rather than disappear (R2a).
comments.sort(key=lambda c: (c["date"] or "", c["score"]), reverse=True)
return comments, stats
def _remaining_lane_budget(elapsed: float) -> int:
"""Compute the review lane's wall-clock budget.
Returns the lesser of LANE_DEADLINE and whatever remains of the foreground
contract. If the remaining time is below MIN_USEFUL_REVIEW_BUDGET, returns
0 (skip the lane entirely) rather than firing doomed short pulls that spend
Bright Data credits without returning reviews.
"""
remaining = FOREGROUND_CONTRACT - elapsed - RENDER_MARGIN
if remaining < MIN_USEFUL_REVIEW_BUDGET:
return 0
return int(min(LANE_DEADLINE, remaining))
def enrich_with_reviews(
products: Sequence[Dict[str, Any]],
*,
depth: str = "default",
config: Optional[Dict[str, Any]] = None,
elapsed: float = 0.0,
max_reviews: int = MAX_REVIEWS,
brand: str = "",
keyword: str = "",
fetcher=None,
) -> Tuple[List[Dict[str, Any]], Optional[str]]:
"""Attach review samples to the top products, in parallel, under a deadline.
Every product is returned either way. A product whose pull is dropped
by the deadline keeps its search-record stats and simply carries no
review sample -- it renders as ``quiet`` rather than vanishing, because
losing a top product entirely is a worse failure than losing its
recent-window read. The dropped pull has spent its credit regardless.
Returns (enriched_products, status_detail). status_detail is None when
enrichment succeeded normally, or a string describing a degraded outcome:
- ``"review lane skipped (budget 0s)"`` -- crumb budget, lane did not run
- ``"review lane timed out"`` -- all pulls dropped by the deadline
"""
enriched = [dict(p) for p in products]
pull_count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
if pull_count <= 0:
_log(f"depth={depth}: discovery only, no review pulls")
return enriched, None
budget = _remaining_lane_budget(elapsed)
if budget <= 0:
_log(f"review lane skipped (budget {budget}s, floor {MIN_USEFUL_REVIEW_BUDGET}s)")
return enriched, "review lane skipped (budget 0s)"
targets = select_enrichment_targets(
enriched, limit=pull_count, brand=brand, keyword=keyword
)
if not targets:
return enriched, None
by_asin = {p["asin"]: p for p in enriched}
pull = fetcher or (
lambda url: fetch_reviews(
url, max_reviews=max_reviews, config=config,
timeout=min(REVIEW_TIMEOUT, budget),
)
)
_log(f"pulling up to {max_reviews} reviews for {len(targets)} products (budget {budget}s)")
started = time.monotonic()
completed_count = 0
dropped_count = 0
# Not a `with` block on purpose. Every future is already running (one
# worker per target), so `future.cancel()` can never succeed, and
# ThreadPoolExecutor's context-manager exit calls shutdown(wait=True) --
# which would block on the very straggler the deadline just declared
# dropped, making the deadline advisory rather than real. Shutting down
# without waiting lets the abandoned thread finish and discard its result
# in the background while the run proceeds.
pool = ThreadPoolExecutor(max_workers=max(1, len(targets)))
try:
futures = {pool.submit(pull, t["url"]): t["asin"] for t in targets}
try:
for future in as_completed(futures, timeout=budget):
asin = futures[future]
try:
response = future.result()
except Exception as exc: # never let one pull kill siblings
_log(f"review pull failed for {asin}: {exc}")
continue
if response.get("error"):
_log(f"review pull error for {asin}: {response['error']}")
continue
comments, stats = parse_reviews(response)
product = by_asin.get(asin)
if product is None:
continue
product["top_comments"] = comments
product.update({k: v for k, v in stats.items() if v})
completed_count += 1
except TimeoutError:
dropped_count = sum(1 for f in futures if not f.done())
_log(f"lane deadline {budget}s hit; dropped {dropped_count} straggling pull(s)")
finally:
pool.shutdown(wait=False, cancel_futures=True)
_log(f"review lane finished in {time.monotonic() - started:.0f}s")
# Report degraded outcome if all pulls dropped (none completed)
status_detail = None
if completed_count == 0 and dropped_count > 0:
status_detail = "review lane timed out"
return enriched, status_detail
def enrich_source_items(
items: List[Any],
*,
depth: str = "default",
config: Optional[Dict[str, Any]] = None,
keyword: str = "",
elapsed: float = 0.0,
max_reviews: int = MAX_REVIEWS,
fetcher=None,
) -> List[Any]:
"""Attach review samples to the amazon SourceItems that survived dedupe.
Reads product identity out of ``metadata`` and writes ``top_comments``
plus the computed stat block back into it, in place. Runs from
``pipeline._finalize_items_by_source`` so the review budget is spent on
the products the brief will actually show, not on the top of the raw
fanout (the digg enrichment precedent).
"""
products: List[Dict[str, Any]] = []
by_asin: Dict[str, Any] = {}
for item in items:
if getattr(item, "source", None) != "amazon":
continue
metadata = getattr(item, "metadata", None) or {}
asin = str(metadata.get("asin") or "").strip()
if not asin or metadata.get("top_comments"):
continue
products.append(
{
"asin": asin,
"url": getattr(item, "url", "") or metadata.get("url", ""),
"name": metadata.get("name") or getattr(item, "title", ""),
"short_name": metadata.get("short_name") or "",
"brand": metadata.get("brand") or "",
"num_ratings": metadata.get("num_ratings") or 0,
"rating": metadata.get("rating"),
}
)
by_asin[asin] = item
if not products:
return items
enriched, _status = enrich_with_reviews(
products, depth=depth, config=config, elapsed=elapsed,
max_reviews=max_reviews, keyword=keyword, fetcher=fetcher,
)
for product in enriched:
item = by_asin.get(product["asin"])
if item is None:
continue
metadata = getattr(item, "metadata", None)
if metadata is None:
continue
if product.get("top_comments"):
metadata["top_comments"] = product["top_comments"]
stats = product_stats(product)
metadata["stats"] = stats
# The review pull's product_rating_count supersedes the search
# record's, which is variant-level and can undercount by orders of
# magnitude (84 on a record whose pull reported 8,446). Normalization
# ran before enrichment, so refresh the surfaces that already baked
# the old number in -- otherwise one product shows two different
# rating counts in the same report.
for key in ("product_rating", "product_rating_count", "star_distribution"):
if product.get(key):
metadata[key] = product[key]
authoritative = stats.get("ratings_total") or 0
if authoritative and getattr(item, "engagement", None) is not None:
item.engagement["ratings"] = authoritative
metadata["num_ratings"] = authoritative
_refresh_title(item, stats)
return items
def _refresh_title(item: Any, stats: Dict[str, Any]) -> None:
"""Rewrite the trailing "- 4.4/5 (N ratings)" headline after enrichment."""
title = getattr(item, "title", "") or ""
rating = stats.get("all_time")
total = stats.get("ratings_total") or 0
if not title or rating is None or not total:
return
headline = f"{rating}/5 ({total:,} ratings)"
base = title.rsplit(" - ", 1)[0] if " - " in title else title
item.title = f"{base} - {headline}"
# ------------------------------------------------------------------ stats
def stats_from_item(item: Any, *, today: Optional[datetime] = None) -> Dict[str, Any]:
"""Compute the stat block for a rendered SourceItem.
Enrichment stores a precomputed block, but mock runs and replayed
fixtures skip enrichment entirely, so render recomputes from metadata
when it is absent. Cheap and pure -- all the inputs already live on
the item.
"""
metadata = getattr(item, "metadata", None) or {}
cached = metadata.get("stats")
if isinstance(cached, dict) and cached:
return cached
return product_stats(
{
"short_name": metadata.get("short_name") or "",
"name": metadata.get("name") or getattr(item, "title", ""),
"url": getattr(item, "url", "") or "",
"rating": metadata.get("rating"),
"num_ratings": metadata.get("num_ratings") or 0,
"product_rating": metadata.get("product_rating"),
"product_rating_count": metadata.get("product_rating_count") or 0,
"star_distribution": metadata.get("star_distribution") or {},
"top_comments": metadata.get("top_comments") or [],
},
today=today,
)
def footer_entry(stats: Dict[str, Any], *, quote: str = "") -> str:
"""Render one product's segment of the emoji-footer line (R1c).
Shapes, by drift state::
Chill Max XL 4.43.8 "the lid jams" negative drift (+ quote)
Deluxe Bag 4.75.0 positive or flat drift
Spirit E-325 4.4 quiet too few in-window reviews
BLUEY Set new no all-time baseline
The ```` is asymmetric on purpose: a sagging product is the alarm
worth catching at a glance, and a healthy one needs no decoration.
"""
name = stats.get("short_name") or "Product"
all_time = stats.get("all_time")
recent = stats.get("recent_avg")
drift = stats.get("drift")
if drift == "new" or all_time is None:
return f"{name} new"
if drift == "quiet" or recent is None:
return f"{name} {all_time}★ quiet"
entry = f"{name} {all_time}★→{recent}"
if drift == "down":
entry += ""
if quote:
entry += f' "{quote}"'
return entry
def five_star_share(distribution: Dict[str, Any]) -> Optional[float]:
"""Share of ratings that are 5-star, from the star-distribution object."""
if not isinstance(distribution, dict) or not distribution:
return None
total = sum(_as_int(distribution.get(key)) for key, _ in _STAR_FIELDS)
if total <= 0:
return None
return _as_int(distribution.get("five_star")) / total
def recent_window_stats(
comments: Iterable[Dict[str, Any]],
*,
today: Optional[datetime] = None,
window_days: int = RECENT_WINDOW_DAYS,
) -> Dict[str, Any]:
"""Average rating and sample size inside the recent window."""
reference = (today or _today()).date()
ratings: List[int] = []
for comment in comments or []:
iso = comment.get("date")
if not iso:
continue
try:
posted = datetime.strptime(iso, "%Y-%m-%d").date()
except (TypeError, ValueError):
continue
if 0 <= (reference - posted).days <= window_days:
rating = _as_int(comment.get("rating"))
if rating:
ratings.append(rating)
if not ratings:
return {"recent_n": 0, "recent_avg": None}
return {"recent_n": len(ratings), "recent_avg": sum(ratings) / len(ratings)}
def product_stats(
product: Dict[str, Any],
*,
today: Optional[datetime] = None,
) -> Dict[str, Any]:
"""Compute the render-facing stat block for one product.
``drift`` is one of:
* ``"new"`` -- no all-time baseline to move away from
* ``"quiet"`` -- baseline exists but the window has too few dated
reviews to average honestly (below MIN_DRIFT_SAMPLE)
* ``"up"`` / ``"down"`` / ``"flat"`` -- a real, sample-backed move
The engine owns every number here; the model owns the words (R1b).
"""
# The review pull's rating count supersedes the search record's, which
# can be variant-level and badly low.
all_time = product.get("product_rating")
if all_time is None:
all_time = product.get("rating")
ratings_total = product.get("product_rating_count") or product.get("num_ratings") or 0
window = recent_window_stats(product.get("top_comments") or [], today=today)
recent_avg = window["recent_avg"]
recent_n = window["recent_n"]
if all_time is None:
drift = "new"
elif recent_n < MIN_DRIFT_SAMPLE or recent_avg is None:
drift = "quiet"
elif round(recent_avg, 1) > round(float(all_time), 1):
drift = "up"
elif round(recent_avg, 1) < round(float(all_time), 1):
drift = "down"
else:
drift = "flat"
return {
"short_name": product.get("short_name") or short_name(product.get("name", "")),
"url": product.get("url", ""),
"all_time": round(float(all_time), 1) if all_time is not None else None,
"ratings_total": _as_int(ratings_total),
"five_star_share": five_star_share(product.get("star_distribution") or {}),
"recent_avg": round(recent_avg, 1) if recent_avg is not None else None,
"recent_n": recent_n,
"reviews_pulled": len(product.get("top_comments") or []),
"drift": drift,
}
+42 -8
View File
@@ -61,13 +61,21 @@ def _today() -> datetime:
return datetime.now(timezone.utc)
def _build_search_query(topic: str) -> str:
"""Quote the topic so arXiv treats it as a phrase across all fields.
def _build_search_query(topic: str, *, quoted: bool = True) -> str:
"""Build the arXiv search-query string for ``topic``.
Inner double-quotes are stripped (arXiv has no phrase-escaping); the outer
quotes plus ``all:`` give a phrase-scoped relevance search.
Quoted (default): phrase-scoped exact match across all fields. Precise
for topics that genuinely appear as a phrase in a title/abstract, but a
natural-language multi-word topic ("AI video generation advances") almost
never appears verbatim, so it returns zero results (#908). Unquoted uses
an AND-conjoined clause for every individual term as a fallback retry.
Inner double-quotes are stripped (arXiv has no phrase-escaping) either way.
"""
return f'all:"{_clean_phrase(topic)}"'
phrase = _clean_phrase(topic)
if quoted:
return f'all:"{phrase}"'
return " AND ".join(f'all:"{term}"' for term in phrase.split())
def _clean_phrase(topic: str) -> str:
@@ -75,12 +83,12 @@ def _clean_phrase(topic: str) -> str:
return " ".join(topic.replace('"', " ").split())
def _build_search_args(topic: str, limit: int) -> List[str]:
def _build_search_args(topic: str, limit: int, *, quoted: bool = True) -> List[str]:
return [
CLI_BIN,
"query",
"--search-query",
_build_search_query(topic),
_build_search_query(topic, quoted=quoted),
"--sort-by",
"relevance",
"--max-results",
@@ -118,13 +126,18 @@ def _run_cli(cmd: List[str], timeout: int) -> Dict[str, Any]:
stdout = result.stdout or ""
if not stdout.strip():
return {"results": []}
_log("CLI returned empty stdout")
return {"results": [], "error": "empty stdout"}
try:
data = json.loads(stdout)
except json.JSONDecodeError as exc:
_log(f"JSON decode failed: {exc}")
return {"results": [], "error": f"json decode: {exc}"}
if not _is_entry_envelope(data):
_log("CLI returned an unrecognized JSON response")
return {"results": [], "error": "unrecognized JSON response"}
return {"results": _extract_entries(data)}
@@ -150,6 +163,20 @@ def _extract_entries(data: Any) -> List[Dict[str, Any]]:
return []
def _is_entry_envelope(data: Any) -> bool:
"""Return whether ``data`` has one of the supported entry-list shapes."""
if isinstance(data, list):
return True
if not isinstance(data, dict):
return False
results = data.get("results")
return (
isinstance(results, list)
or (isinstance(results, dict) and isinstance(results.get("entries"), list))
or isinstance(data.get("entries"), list)
)
def search_arxiv(
topic: str,
from_date: str,
@@ -172,6 +199,13 @@ def search_arxiv(
_log(f"query '{topic}' (relevance, max={limit})")
response = _run_cli(cmd, timeout=SEARCH_TIMEOUT)
_log(f"found {len(response.get('results') or [])} entries")
# Retry a clean zero-result phrase match with individually quoted AND terms.
# CLI failures, malformed responses, and missing binaries skip the retry.
if not response.get("error") and not response.get("results"):
retry_cmd = _build_search_args(topic, limit, quoted=False)
_log(f"quoted phrase matched nothing; retrying unquoted for '{topic}'")
response = _run_cli(retry_cmd, timeout=SEARCH_TIMEOUT)
_log(f"unquoted retry found {len(response.get('results') or [])} entries")
return response
+103 -10
View File
@@ -107,12 +107,15 @@ class BackendSpec:
``probe`` must be side-effect-free. When ``paid`` is True the probe is
key-presence only: no subprocess, no network, no credential spend.
``opt_in`` marks backends that are never auto-selected and require an
explicit pin (grok).
"""
name: str
requires: str
probe: Callable[[Dict[str, Any]], "BackendFinding"]
paid: bool = False
opt_in: bool = False
@dataclass(frozen=True)
@@ -251,6 +254,80 @@ def _probe_bird(config: Dict[str, Any]) -> BackendFinding:
)
def _probe_grok(config: Dict[str, Any]) -> BackendFinding:
"""grok CLI = keyless X. LOCAL-ONLY probe, like _probe_xurl.
Deliberately does NOT call ``health.probe_dependency``: that helper runs
``subprocess.run([name, "--version"])``, and the whole-doctor-path test
patches ``subprocess.run`` to raise.
Consequence to be honest about: a grok binary that resolves on PATH but
will not execute (the stale-shim class) reports OK here and fails only when
a real run shells out. ``grok_x.is_available`` does not close that gap
either -- it is also filesystem-only. ``health.probe_dependency("grok")``
is the executing probe, and it runs in doctor's CLI-health block rather
than on this no-subprocess path.
"""
from . import grok_x
requires = "grok CLI installed + signed in (no X credential)"
if which("grok") is None:
off_path = health._off_path_binary("grok")
if off_path is not None:
return BackendFinding(
name="grok",
status=health.MISSING,
requires=requires,
detail=f"grok is installed at {off_path} but that directory is not on this process's PATH",
prescription=f'add {off_path.parent} to PATH (e.g. export PATH="{off_path.parent}:$PATH")',
)
return BackendFinding(
name="grok",
status=health.MISSING,
requires=requires,
detail="grok CLI not found on PATH",
prescription=(
"install the Grok CLI: curl -fsSL https://x.ai/cli/install.sh | bash, "
"then run `grok login`"
),
)
store_status, store_detail, expires_at = grok_x.stored_auth_status()
if store_status == grok_x.AUTH_OK:
return BackendFinding(
name="grok",
status=health.OK,
requires=requires,
detail=f"{store_detail} (not live-verified until a run)",
)
if store_status == grok_x.AUTH_EXPIRED:
expiry_str = expires_at.isoformat() if expires_at else "unknown"
return BackendFinding(
name="grok",
status=health.DEGRADED,
requires=requires,
detail=(
f"Grok session expired at {expiry_str}; "
"refresh happens at run time (if revoked, run `grok login --device-auth`)"
),
prescription="grok login --device-auth",
)
if store_status == grok_x.AUTH_ERROR:
return BackendFinding(
name="grok",
status=health.ERROR,
requires=requires,
detail=store_detail,
prescription="grok login",
)
return BackendFinding(
name="grok",
status=health.MISSING,
requires=requires,
detail="grok CLI installed but not signed in",
prescription="grok login",
)
def _probe_xurl(config: Dict[str, Any]) -> BackendFinding:
"""xurl = official X API v2 CLI (OAuth2). Free lane; LOCAL-ONLY probe.
@@ -347,11 +424,14 @@ def _probe_reddit_public(config: Dict[str, Any]) -> BackendFinding:
_X_PROBES: Dict[str, Callable[[Dict[str, Any]], BackendFinding]] = {
"xai": _key_probe("xai", "XAI_API_KEY", "XAI_API_KEY (xAI/Grok live search)"),
"grok": _probe_grok,
"bird": _probe_bird,
"xurl": _probe_xurl,
"xquik": _key_probe("xquik", "XQUIK_API_KEY", "XQUIK_API_KEY (xquik.com)"),
}
_X_PAID = {"xai", "xquik"}
# Opt-in backends: never auto-selected; require explicit pin.
_X_OPT_IN = set(env.X_BACKEND_OPT_IN)
_WEB_PROBES: Dict[str, Callable[[Dict[str, Any]], BackendFinding]] = {
"brave": _key_probe("brave", "BRAVE_API_KEY", "BRAVE_API_KEY"),
@@ -372,24 +452,32 @@ _SC_SPEC = BackendSpec(
paid=True,
)
# X backend requirements, keyed by name.
_X_REQUIRES: Dict[str, str] = {
"xai": "XAI_API_KEY (xAI/Grok live search)",
"grok": "grok CLI installed + signed in (opt-in only; pin to enable)",
"bird": "X browser cookies (AUTH_TOKEN/CT0) + node",
"xurl": "xurl CLI installed + OAuth2 login",
"xquik": "XQUIK_API_KEY (xquik.com)",
}
DESCRIPTORS: Dict[str, ChainDescriptor] = {
# X: chain order and pin var imported from env.py (single source of truth).
# Backends include the auto chain (X_BACKEND_ORDER) plus opt-in entries
# (X_BACKEND_OPT_IN) for doctor visibility. Opt-in backends like grok
# appear in findings but are never auto-selected; pin to enable.
"x": ChainDescriptor(
source="x",
mode=MODE_ALTERNATIVE,
backends=tuple(
BackendSpec(
name=name,
requires={
"xai": "XAI_API_KEY (xAI/Grok live search)",
"bird": "X browser cookies (AUTH_TOKEN/CT0) + node",
"xurl": "xurl CLI installed + OAuth2 login",
"xquik": "XQUIK_API_KEY (xquik.com)",
}[name],
requires=_X_REQUIRES[name],
probe=_X_PROBES[name],
paid=name in _X_PAID,
opt_in=name in _X_OPT_IN,
)
for name in env.X_BACKEND_ORDER
for name in env.X_BACKEND_ORDER + env.X_BACKEND_OPT_IN
),
pin_var=env.X_BACKEND_PIN_VAR,
),
@@ -499,6 +587,8 @@ def _resolve_alternative(
) -> BackendResolution:
names = [spec.name for spec in descriptor.backends]
by_name = {f.name: f for f in findings}
# Track which backends are opt-in (never auto-selected).
opt_in_names = {spec.name for spec in descriptor.backends if spec.opt_in}
res = BackendResolution(
source=descriptor.source,
mode=MODE_ALTERNATIVE,
@@ -535,18 +625,21 @@ def _resolve_alternative(
# Collect-then-pick: first fully-usable wins; else best degraded; else
# error carrying the highest-priority backend's prescription.
for finding in findings:
# Opt-in backends are NEVER auto-selected; skip them entirely.
auto_findings = [f for f in findings if f.name not in opt_in_names]
for finding in auto_findings:
if finding.status == health.OK:
res.active_backend = finding.name
res.tier = TIER_OK
return res
for finding in findings:
for finding in auto_findings:
if finding.status == health.DEGRADED:
res.active_backend = finding.name
res.tier = TIER_WARN
return res
res.tier = TIER_ERROR
res.prescription = findings[0].prescription if findings else ""
# Prescription comes from the first auto-chain backend, not opt-in.
res.prescription = auto_findings[0].prescription if auto_findings else ""
return res
+41 -3
View File
@@ -112,7 +112,12 @@ def _extract_core_subject(topic: str) -> str:
def _plain_query_tokens(text: str) -> list[str]:
"""Return lexical tokens without Bird query grouping syntax."""
"""Return lexical tokens without Bird query grouping syntax.
Strips phrase quotes as well as grouping characters. Used where a flat
token list is wanted; use ``build_topic_query`` for the provider query,
which preserves quoted phrases.
"""
separators = str.maketrans({char: " " for char in '\"“”()[]{}'})
return [
clean
@@ -121,6 +126,38 @@ def _plain_query_tokens(text: str) -> list[str]:
]
# Bird/X grouping syntax that carries no lexical meaning. Double quotes are
# deliberately absent: X advanced search treats "..." as a phrase match, which
# is exactly what the planner intended when it quoted a proper noun.
_GROUPING_CHARS = "“”()[]{}"
def build_topic_query(topic: str, from_date: str) -> str:
"""Build the X topic query, preserving quoted proper-noun phrases.
Previously the topic went through ``_plain_query_tokens``, which stripped
the quotes the planner had added, so an intended phrase match for
'"Peter Steinberger"' degraded into `peter AND steinberger` -- narrower and
noisier at once. X supports phrase queries natively, so the quotes are
passed through.
"""
separators = str.maketrans({char: " " for char in _GROUPING_CHARS})
cleaned = topic.translate(separators)
# An unbalanced quote is worse than no quote: X reads the orphan as an
# unterminated phrase and matches nothing. Upstream trimming (core-subject
# extraction, retry shortening) can cut a topic mid-phrase, so verify the
# quotes pair up and fall back to bare tokens when they do not.
if cleaned.count('"') % 2:
cleaned = cleaned.replace('"', " ")
tokens = [
clean
for token in cleaned.split()
if (clean := token.strip("'‘’"))
]
core = " ".join(tokens).strip()
return f"{core} since:{from_date}" if core else f"since:{from_date}"
def is_bird_installed() -> bool:
"""Check if vendored Bird search module is available.
@@ -371,9 +408,10 @@ def search_x(
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
# Extract core subject - X search is literal, not semantic
core_words = _plain_query_tokens(_extract_core_subject(topic))
core_subject = _extract_core_subject(topic)
core_words = _plain_query_tokens(core_subject)
core_topic = " ".join(core_words)
query = f"{core_topic} since:{from_date}"
query = build_topic_query(core_subject, from_date)
_log(f"Searching: {query}")
response = _run_bird_search(query, count, timeout)
+290
View File
@@ -0,0 +1,290 @@
"""Bright Data CLI adapter for last30days.
Shells out to the ``brightdata`` CLI (``@brightdata/cli``) to run Bright
Data Pipelines. The CLI owns authentication end to end -- ``brightdata
login`` does a gh-style zero-click browser flow and stores credentials in
a platform config directory -- so this module never handles a login, and
never reads credential *contents*: the auth probe is presence-only.
Activation gate: two-way, mirroring the digg CLI-gated precedent but with
an auth dimension the digg source does not have.
1. ``shutil.which("brightdata")`` must resolve on the **agent subprocess
PATH** (not merely exist on disk -- Hermes/OpenClaw gateways often drop
``~/.local/bin``).
2. A credential signal must be present: either ``BRIGHTDATA_API_KEY``
resolved through the normal config layering, or the CLI's own
credentials file in the platform config dir.
The second check is deliberately offline. A stale token passes it and
then 401s fast at call time; that path degrades to empty results with the
CLI's own error line preserved in the envelope, which is the AE2 contract.
Metering note (R13): no pricing logic lives here. One pipeline request
costs one credit against the account's monthly free tier regardless of how
many records come back, so caps in the calling adapter bound *records*
(paid-tier cost), not credits. Credit and auth warnings from the CLI are
passed through verbatim rather than interpreted.
"""
from __future__ import annotations
import json
import os
import shutil
import sys
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence
from . import log, subproc
CLI_BIN = "brightdata"
# Env var carrying an explicit API key. Registered in env.py so `.env` file
# and keychain users pass the gate the same way process-env users do; when
# it resolves from a non-process-env layer we hand it to the CLI via -k.
API_KEY_ENV = "BRIGHTDATA_API_KEY"
# Credentials filename written by `brightdata login`. Probed for existence
# only -- never opened, parsed, or logged.
_CREDENTIALS_FILENAME = "credentials.json"
_CONFIG_DIRNAME = "brightdata-cli"
# The CLI's own polling timeout sits below our subprocess timeout so the CLI
# exits cleanly with its own error rather than being SIGTERM'd mid-poll. Its
# timeout path throws with zero records (verified in its polling module --
# never partial output), so a timed-out pull is a clean parseable failure.
_CLI_TIMEOUT_MARGIN = 10
def _log(msg: str) -> None:
log.source_log("BrightData", msg, tty_only=False)
def _config_dir() -> Path:
"""Platform config directory the Bright Data CLI stores credentials in.
Mirrors the CLI's own credentials module: APPDATA on Windows, the
Application Support tree on macOS, XDG_CONFIG_HOME (or ~/.config) on
everything else.
"""
if sys.platform == "win32":
base = os.environ.get("APPDATA")
root = Path(base) if base else Path.home() / "AppData" / "Roaming"
elif sys.platform == "darwin":
root = Path.home() / "Library" / "Application Support"
else:
base = os.environ.get("XDG_CONFIG_HOME")
root = Path(base) if base else Path.home() / ".config"
return root / _CONFIG_DIRNAME
def is_installed() -> bool:
"""True when the brightdata binary resolves on the agent subprocess PATH."""
return shutil.which(CLI_BIN) is not None
def _api_key(config: Optional[Dict[str, Any]]) -> str:
if not config:
return ""
return str(config.get(API_KEY_ENV) or "").strip()
def has_credentials(config: Optional[Dict[str, Any]] = None) -> bool:
"""True when some credential signal exists, without reading any secret.
Presence-only by design: an explicit API key resolved through config
layering, or the existence of the CLI's credentials file. The file is
never opened. This cannot distinguish a live token from an expired one
-- that is what the fast 401 at call time is for.
"""
if _api_key(config):
return True
try:
return (_config_dir() / _CREDENTIALS_FILENAME).exists()
except OSError:
return False
def is_available(config: Optional[Dict[str, Any]] = None) -> bool:
"""The full activation gate: binary on PATH *and* a credential signal."""
return is_installed() and has_credentials(config)
def gate_status(config: Optional[Dict[str, Any]] = None) -> Dict[str, bool]:
"""Two-field probe for ``pipeline.diagnose`` (bird_installed precedent).
Network-free, so it is safe on the ``--diagnose`` / doctor path.
"""
installed = is_installed()
return {
"brightdata_installed": installed,
"brightdata_authenticated": installed and has_credentials(config),
}
def _build_args(
pipeline_type: str,
params: Sequence[str],
*,
cli_timeout: int,
) -> List[str]:
"""Assemble the CLI invocation.
The API key is deliberately **absent** here -- it travels in the child's
environment instead (see ``_child_env``). Process arguments are not a
secret channel: ``/proc/<pid>/cmdline`` is world-readable under the
default ``hidepid=0``, and a review pull lives for up to 180s, so a key
on the command line is readable by any other local user and is captured
verbatim by execve auditing, process accounting, and any monitoring
agent that snapshots ``ps``. Mirrors the ``bird_x`` cookie-injection
precedent.
Positional params are fenced behind ``--`` so a keyword that happens to
begin with a dash is parsed as a search term rather than as an option.
"""
return [
CLI_BIN,
"pipelines",
pipeline_type,
"--json",
"--timeout",
str(cli_timeout),
"--",
*(str(p) for p in params),
]
def _child_env(api_key: str) -> Optional[Dict[str, str]]:
"""Environment for the child process, carrying the key when we have one.
Returns None when there is nothing to inject, so the child simply
inherits the parent environment (the common case: the CLI owns its own
credentials file, or the key is already exported).
"""
if not api_key:
return None
return {**os.environ, API_KEY_ENV: api_key}
def _scrub(text: str, secret: str) -> str:
"""Remove a secret from text before it is logged or returned.
Defense in depth for the passthrough paths: the stderr lines this
module deliberately surfaces are auth and quota failures, which are
exactly the messages a CLI is most likely to echo the rejected
credential back in.
"""
if not secret or not text:
return text
return text.replace(secret, "***")
def _extract_records(payload: Any) -> List[Dict[str, Any]]:
"""Pull the record list out of a parsed CLI payload.
Verified live (2026-08-13): both amazon pipelines return a **bare JSON
array** of flat record dicts, not the ``{"results": [...]}`` envelope the
digg CLI uses. The dict branches below are defensive against CLI churn,
which is a live risk on a package this young.
"""
if isinstance(payload, list):
return [r for r in payload if isinstance(r, dict)]
if isinstance(payload, dict):
for key in ("records", "results", "data"):
value = payload.get(key)
if isinstance(value, list):
return [r for r in value if isinstance(r, dict)]
return []
def run_pipeline(
pipeline_type: str,
params: Sequence[str],
*,
timeout: int,
config: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Run one Bright Data pipeline and return ``{"records", "error"}``.
Never raises. Every failure mode -- missing binary, spawn failure,
subprocess timeout, non-zero exit, unparseable stdout -- returns empty
records plus a one-line ``error`` string, so callers can record the
failure in ``errors_by_source`` without branching on exception types.
The CLI's first stderr line is preserved verbatim as the error (auth
401s and low-credit warnings are the cases that matter), and also
mirrored to ``source_log`` so the failure is visible in non-TTY hosts.
Args:
pipeline_type: pipeline name, e.g. ``amazon_product_search``.
params: positional pipeline params, passed through in order.
timeout: subprocess timeout in seconds. The CLI's own polling
timeout is set just below this so it can fail cleanly first.
config: resolved config dict, consulted only for the API key.
Returns:
``{"records": [...]}`` on success, else ``{"records": [], "error": str}``.
"""
if not is_installed():
return {"records": [], "error": f"{CLI_BIN} not on PATH"}
cli_timeout = max(5, int(timeout) - _CLI_TIMEOUT_MARGIN)
key = _api_key(config)
cmd = _build_args(pipeline_type, params, cli_timeout=cli_timeout)
try:
result = subproc.run_with_timeout(cmd, timeout=timeout, env=_child_env(key))
except subproc.SubprocTimeout as exc:
_log(f"Timeout: {exc}")
return {"records": [], "error": str(exc)}
except FileNotFoundError as exc:
_log(f"Binary missing: {exc}")
return {"records": [], "error": str(exc)}
except OSError as exc:
_log(f"Spawn failed: {exc}")
return {"records": [], "error": str(exc)}
stderr = _scrub(result.stderr or "", key)
_passthrough_warnings(stderr)
if result.returncode != 0:
lines = [ln.strip() for ln in stderr.strip().splitlines() if ln.strip()]
# The CLI narrates polling progress on stderr, so the *last* line is
# the actual failure; the first line is "Triggering pipeline...".
first = lines[-1] if lines else f"exit {result.returncode}"
_log(f"CLI exit {result.returncode}: {first}")
return {"records": [], "error": first}
stdout = result.stdout or ""
if not stdout.strip():
return {"records": []}
try:
payload = json.loads(stdout)
except json.JSONDecodeError as exc:
_log(f"JSON decode failed: {exc}")
return {"records": [], "error": f"json decode: {exc}"}
return {"records": _extract_records(payload)}
# Substrings that mark a stderr line worth surfacing even on a successful
# run -- credit exhaustion and auth trouble are the two the user must see.
# Matched case-insensitively against the CLI's own wording, and echoed
# verbatim rather than reworded (R13: no pricing logic, no interpretation).
_WARNING_MARKERS = ("credit", "quota", "balance", "unauthor", "401", "expired", "login")
def _passthrough_warnings(stderr: str) -> None:
"""Echo credit/auth warning lines from the CLI verbatim.
Skips the routine polling narration so a normal run stays quiet.
"""
for line in (stderr or "").splitlines():
text = line.strip()
if not text or text.lower().startswith(("status:", "triggering", "triggered", "data received")):
continue
lowered = text.lower()
if any(marker in lowered for marker in _WARNING_MARKERS):
_log(text)
+21 -5
View File
@@ -4,6 +4,18 @@ from __future__ import annotations
from . import dedupe, entity_extract, schema
def _cluster_sort_key(candidate: schema.Candidate) -> tuple:
"""Sort key that partitions stale candidates below fresh ones.
Stale items (all dated source_items outside the window) must never lead
cluster representatives or render as the cluster title.
"""
return (
1 if schema.candidate_out_of_window(candidate) else 0,
-candidate.final_score,
)
CLUSTERABLE_INTENTS = {"breaking_news", "opinion", "comparison", "prediction"}
def _candidate_text(candidate: schema.Candidate) -> str:
@@ -20,7 +32,7 @@ def _mmr_representatives(
remaining = list(candidates)
while remaining and len(selected) < limit:
if not selected:
best = max(remaining, key=lambda candidate: candidate.final_score)
best = min(remaining, key=_cluster_sort_key)
selected.append(best)
remaining_set.discard(best.candidate_id)
remaining = [c for c in remaining if c.candidate_id in remaining_set]
@@ -28,12 +40,16 @@ def _mmr_representatives(
selected_preps = [text_cache[c.candidate_id] for c in selected]
def score(candidate: schema.Candidate) -> float:
def score(candidate: schema.Candidate) -> tuple:
prep = text_cache[candidate.candidate_id]
diversity_penalty = max(
dedupe.prepared_similarity(prep, sp) for sp in selected_preps
)
return (diversity_lambda * candidate.final_score) - ((1 - diversity_lambda) * diversity_penalty * 100)
base_score = (diversity_lambda * candidate.final_score) - ((1 - diversity_lambda) * diversity_penalty * 100)
return (
0 if schema.candidate_out_of_window(candidate) else 1,
base_score,
)
best = max(remaining, key=score)
selected.append(best)
@@ -89,7 +105,7 @@ def cluster_candidates(
clusters: list[schema.Cluster] = []
for index, group in enumerate(groups, start=1):
group.sort(key=lambda candidate: candidate.final_score, reverse=True)
group.sort(key=_cluster_sort_key)
cluster_id = f"cluster-{index}"
representatives = _mmr_representatives(group, text_cache)
for candidate in group:
@@ -197,7 +213,7 @@ def _merge_entity_clusters(
# Pick representatives from combined pool
combined_candidates = [candidate_map[cid] for cid in combined_cids if cid in candidate_map]
combined_candidates.sort(key=lambda c: c.final_score, reverse=True)
combined_candidates.sort(key=_cluster_sort_key)
merge_text_cache = {
c.candidate_id: dedupe._PreparedText(_candidate_text(c))
for c in combined_candidates
+4 -1
View File
@@ -24,6 +24,9 @@ COMPETITORS_MIN = 1
COMPETITORS_MAX = 6
COMPETITORS_DEFAULT = 2
COMPARISON_ENTITY_MAX = COMPETITORS_MAX + 1
# Discovery SERP fan-out is small (3 queries today) but still needs a ceiling
# so a future query expansion cannot open one worker per query unbounded.
MAX_DISCOVERY_WORKERS = 3
# A "brand-shaped" token starts with uppercase OR is camelCase with an
# uppercase letter later. Catches "Anthropic", "OpenAI", "xAI", "iPhone",
@@ -178,7 +181,7 @@ def discover_competitors(
items, _artifact = grounding.web_search(query, date_range, config)
return label, items
with ThreadPoolExecutor(max_workers=len(queries)) as executor:
with ThreadPoolExecutor(max_workers=min(len(queries), MAX_DISCOVERY_WORKERS)) as executor:
futures = {
executor.submit(_search, label, q): label
for label, q in queries.items()
+168 -18
View File
@@ -53,7 +53,7 @@ import urllib.request
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional
from . import backends, env, health, prescriptions
from . import backends, brightdata, env, health, http, prescriptions
from .backends import TIER_ERROR, TIER_OK, TIER_WARN
# Rollup tiers (R1). ok/warn/error are U2's; only "off" is doctor's own.
@@ -157,6 +157,7 @@ SOURCE_ORDER = (
"techmeme",
"arxiv",
"trustpilot",
"amazon",
"tiktok",
"instagram",
"threads",
@@ -181,6 +182,9 @@ CLI_DEPENDENCIES = {
"techmeme": "techmeme-pp-cli",
"arxiv": "arxiv-pp-cli",
"trustpilot": "trustpilot-pp-cli",
# The only entry that also needs auth; _amazon_record reports the
# installed-but-unauthenticated state the shared CLI helper cannot.
"amazon": "brightdata",
"github": "gh",
}
_OPTIONAL_CLI_SOURCES = frozenset({"github"})
@@ -421,16 +425,91 @@ def _x_record(config):
# diagnose cannot drift. It reads no cookie *values*, so it confirms a run
# will *attempt* browser auth, not that the session is currently valid -
# keep the note honest and point at the verified key-backed path.
if record["status"] == "unconfigured" and env.x_pending_browser_auth(
config, local_only=True
):
record["status"] = health.OK
record["tier"] = TIER_BY_STATUS[health.OK]
record["note"] = (
"will use: bird (browser cookies; session not verified until a run "
"- add XAI_API_KEY for a verified, cookie-free path)"
#
# This check MUST come before grok normalization: a pending bird path takes
# precedence over marking X as unconfigured due to an unused grok store.
# Handle both "unconfigured" (all backends missing) and "error" (grok present
# but opt-in, no auto-chain backend usable) when pending bird applies.
#
# HOWEVER: pending bird must NOT replace a record that has a configured
# auto-chain backend in ERROR/DEGRADED/BROKEN/TIMEOUT. Same rule as the
# grok normalizer: only upgrade when no auto backend is configured-but-broken.
pending_bird = env.x_pending_browser_auth(config, local_only=True)
if pending_bird and record["status"] in ("unconfigured", health.ERROR):
backends_list = record.get("backends", [])
auto_chain_names = {"bird", "xai", "xurl", "xquik"}
auto_backends = [b for b in backends_list if b.get("name") in auto_chain_names]
# Only apply pending-bird upgrade if ALL auto-chain backends are MISSING.
# If any auto backend is configured but broken, keep that error.
all_auto_missing = all(
b.get("status") == health.MISSING for b in auto_backends
)
record["fix"] = ""
if all_auto_missing:
record["status"] = health.OK
record["tier"] = TIER_BY_STATUS[health.OK]
record["note"] = (
"will use: bird (browser cookies; session not verified until a run "
"- add XAI_API_KEY for a verified, cookie-free path)"
)
record["fix"] = ""
return record
#
# Grok is opt-in only: a leftover ~/.grok/auth.json must never steal the X
# lane. The grok backend appears in the chain findings (for visibility) but
# is never auto-selected. Doctor reports it as "available, unused - pin
# LAST30DAYS_X_BACKEND=grok to enable" rather than "will use: grok".
#
# R3/R8: When no auto-chain backend is CONFIGURED (all MISSING) but grok has
# any non-MISSING status, X is unconfigured/skipped - NOT broken/auth-failed.
# The tier must be "off" (unconfigured), not "error" (NOT WORKING).
#
# HOWEVER: if an auto-chain backend IS configured but broken (ERROR/DEGRADED),
# do NOT normalize to unconfigured. Keep that backend's error and repair
# guidance. Unused grok must not swallow a genuine auto-chain failure.
#
# Do NOT apply this normalization when pending browser auth would make bird
# usable — check pending_bird first (handled above via early return).
if (
record["tier"] == TIER_ERROR
and not record.get("pinned")
and record.get("active_backend") is None
and not pending_bird
):
backends_list = record.get("backends", [])
auto_chain_names = {"bird", "xai", "xurl", "xquik"}
auto_backends = [b for b in backends_list if b.get("name") in auto_chain_names]
# Only normalize if ALL auto-chain backends are MISSING (not configured).
# If any auto backend is ERROR/DEGRADED/BROKEN/TIMEOUT, keep that error.
all_auto_missing = all(
b.get("status") == health.MISSING for b in auto_backends
)
if not all_auto_missing:
# An auto-chain backend is configured but broken — do NOT normalize.
# Keep the original error and its repair guidance.
return record
grok_finding = next(
(b for b in backends_list if b.get("name") == "grok"),
None,
)
if grok_finding and grok_finding.get("status") in (
health.OK,
health.DEGRADED,
health.ERROR,
):
record["status"] = "unconfigured"
record["tier"] = TIER_OFF
if grok_finding.get("status") == health.ERROR:
record["note"] = (
"X unconfigured; grok CLI store is broken but unused (opt-in only) — "
"pin LAST30DAYS_X_BACKEND=grok to enable, then fix the store"
)
else:
record["note"] = (
"X unconfigured; grok CLI available but opt-in only — "
"pin LAST30DAYS_X_BACKEND=grok to enable"
)
record["fix"] = ""
return record
return record
@@ -540,6 +619,39 @@ def _trustpilot_record(config):
return _cli_gated_record(config, "trustpilot-pp-cli", "trustpilot")
def _amazon_record(config):
"""Amazon buyer signals: CLI-gated *and* auth-gated.
Unlike the other CLI-gated sources, a present binary is not enough --
the Bright Data CLI owns its own login, so a user can have `brightdata`
on PATH and still get nothing. Report those states separately: an
unauthenticated install is configured-but-broken (a real fix exists and
the user wants to hear it), while a missing binary is just an optional
source nobody opted into.
"""
probe = health.probe_dependency(brightdata.CLI_BIN)
requires = f"{brightdata.CLI_BIN} on the agent-subprocess PATH, logged in"
if probe.ok:
if brightdata.has_credentials(config):
return _record(status=health.OK, detail=probe.detail, requires=requires)
return _record(
status="unconfigured",
fix="run `brightdata login` to activate the amazon source",
detail="brightdata is installed but has no credentials",
requires=requires,
)
entry = prescriptions.for_dependency_probe(probe)
fix = _fix_text(entry) if entry else probe.prescription
if probe.status == health.MISSING and not probe.off_path:
return _record(
status="opt-in",
fix="npm i -g @brightdata/cli && brightdata login",
detail=probe.detail,
requires=requires,
)
return _record(status=probe.status, fix=fix, detail=probe.detail, requires=requires)
def _tiktok_record(config):
return _sc_gated_record(config, "tiktok")
@@ -716,6 +828,7 @@ _SOURCE_BUILDERS: Dict[str, Callable[[Dict[str, Any]], Dict[str, Any]]] = {
"techmeme": _techmeme_record,
"arxiv": _arxiv_record,
"trustpilot": _trustpilot_record,
"amazon": _amazon_record,
"tiktok": _tiktok_record,
"instagram": _instagram_record,
"threads": _threads_record,
@@ -1469,12 +1582,30 @@ def _write_cache(report: Dict[str, Any], config: Dict[str, Any]) -> bool:
# Free, keyless liveness endpoints (reachability check, tiny payload).
_HTTP_PROBE_URLS = {
"reddit": "https://www.reddit.com/r/all/hot.json?limit=1",
# The keyless engine's real discovery endpoint (reddit_rss._build_urls).
# /r/all/hot.json is permanently 403 keyless (see the reddit_keyless module
# docstring) and no lane requests it any more, so probing it measured an
# endpoint the engine had already abandoned.
"reddit": "https://www.reddit.com/search.rss?q=test&sort=relevance&t=month",
"hackernews": "https://hn.algolia.com/api/v1/search?query=test&hitsPerPage=1",
"polymarket": "https://gamma-api.polymarket.com/events?limit=1",
"github": "https://api.github.com/rate_limit",
}
# Per-source exception to "a 4xx still means the endpoint responded". The
# keyless Reddit lanes send no credentials, so a 403/429 there is the host
# refusing this client — the exact failure the engine hits — not reachability.
_PROBE_BLOCKED_STATUSES = {"reddit": frozenset({403, 429})}
# Probe with the identity the lane sends, or the probe measures the User-Agent
# rather than the endpoint (get_text sends http.BROWSER_USER_AGENT).
_PROBE_HEADERS = {
"reddit": {
"User-Agent": http.BROWSER_USER_AGENT,
"Accept": "application/atom+xml",
},
}
DEFAULT_PROBE_TIMEOUT_SECONDS = 10
@@ -1501,18 +1632,32 @@ def _probeable_sources() -> tuple:
return tuple(dict.fromkeys(list(_HTTP_PROBE_URLS) + cli_only))
def _http_ok(url: str, timeout: float) -> tuple:
def _http_ok(
url: str,
timeout: float,
*,
blocked_statuses: frozenset = frozenset(),
headers: Optional[Dict[str, str]] = None,
) -> tuple:
"""Reachability check: a 4xx still means the endpoint responded; 5xx or a
connection/timeout error means it did not."""
connection/timeout error means it did not.
``blocked_statuses`` names the per-source codes that mean "responded, but
refused us" (Reddit's keyless 403/429) — those are a failure, not
reachability. ``headers`` overrides the probe identity so a source can be
probed with the same User-Agent its lane sends.
"""
def _verdict(code: int) -> tuple:
return code < 500 and code not in blocked_statuses, f"HTTP {code}"
try:
req = urllib.request.Request(
url, headers={"User-Agent": "last30days-doctor"}
url, headers=headers or {"User-Agent": "last30days-doctor"}
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
code = getattr(resp, "status", 200) or 200
return code < 500, f"HTTP {code}"
return _verdict(getattr(resp, "status", 200) or 200)
except urllib.error.HTTPError as exc:
return exc.code < 500, f"HTTP {exc.code}"
return _verdict(exc.code)
except Exception as exc:
return False, f"{type(exc).__name__}: {exc}"
@@ -1520,7 +1665,12 @@ def _http_ok(url: str, timeout: float) -> tuple:
def _probe_source(name: str, config: Dict[str, Any], timeout: float) -> Optional[Dict[str, Any]]:
url = _HTTP_PROBE_URLS.get(name)
if url:
ok, detail = _http_ok(url, timeout)
ok, detail = _http_ok(
url,
timeout,
blocked_statuses=_PROBE_BLOCKED_STATUSES.get(name, frozenset()),
headers=_PROBE_HEADERS.get(name),
)
return {"ok": ok, "detail": detail, "probed": True}
cli = CLI_DEPENDENCIES.get(name)
if cli:
+93 -31
View File
@@ -60,7 +60,7 @@ KEYCHAIN_KEYS = (
"AUTH_TOKEN", "CT0", "BSKY_HANDLE", "BSKY_APP_PASSWORD",
"TRUTHSOCIAL_TOKEN", "BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY",
"OPENROUTER_API_KEY", "PERPLEXITY_API_KEY", "PARALLEL_API_KEY", "XQUIK_API_KEY",
"XIAOHONGSHU_API_BASE", "GITHUB_TOKEN",
"XIAOHONGSHU_API_BASE", "GITHUB_TOKEN", "BRIGHTDATA_API_KEY",
)
# pass(1) integration: Linux/Unix analog of the Keychain source. Each key in
@@ -530,6 +530,14 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
('LAST30DAYS_PERPLEXITY_DEEP_TIMEOUT_SECONDS', '600'),
('PARALLEL_API_KEY', None),
('XQUIK_API_KEY', None),
# Bright Data CLI. Optional: the CLI normally owns its own auth via
# `brightdata login`, so this only matters for users who prefer an
# explicit key in a `.env` file or the keychain. Registered here so
# those layers reach the gate and the subprocess (-k) alike.
('BRIGHTDATA_API_KEY', None),
# Amazon marketplace the amazon source searches. Non-US users point
# this at their own storefront (e.g. https://www.amazon.co.uk).
('LAST30DAYS_AMAZON_DOMAIN', 'https://www.amazon.com'),
# Host-native search signal: set by the SKILL.md agent-host path when the
# invoking runtime has its own (better) web-search tool, so the engine's
# keyless search floor stays off there. Defaults unset -> floor allowed.
@@ -565,6 +573,7 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
('GROQ_API_KEY', None),
('LAST30DAYS_YT_SUB_LANGS', 'en,es,pt'),
('LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT', None),
('LAST30DAYS_YT_SEARCH_TIMEOUT', None),
('GITHUB_TOKEN', None),
]
@@ -579,16 +588,13 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
# youtube_yt reads these tuning knobs lazily from os.environ, so values
# loaded from .env must be exported into the current engine process.
if config.get('LAST30DAYS_YT_SUB_LANGS'):
os.environ.setdefault(
'LAST30DAYS_YT_SUB_LANGS',
config['LAST30DAYS_YT_SUB_LANGS'],
)
if config.get('LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT'):
os.environ.setdefault(
'LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT',
config['LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT'],
)
for key in (
'LAST30DAYS_YT_SUB_LANGS',
'LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT',
'LAST30DAYS_YT_SEARCH_TIMEOUT',
):
if config.get(key):
os.environ.setdefault(key, config[key])
# Backward-compat: ScrapeCreators' own examples and tutorials use the
# SCRAPE_CREATORS_API_KEY spelling (with underscore between SCRAPE and
@@ -745,12 +751,18 @@ def extract_browser_credentials(config: dict[str, Any]) -> dict[str, str]:
def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
"""Return (source, method) for X search, where method describes the auth origin."""
if config.get("XAI_API_KEY"):
return "xai", "xai"
"""Return (source, method) for X search, where method describes the auth origin.
Order mirrors _X_BACKEND_ORDER: bird first (cookies beat XAI_API_KEY when
both are present), then xai, then xurl. Grok is opt-in only and is never
auto-selected here.
"""
# Bird first: cookies beat XAI_API_KEY when both are present.
if config.get("AUTH_TOKEN") and config.get("CT0"):
method = config.get("_AUTH_TOKEN_SOURCE", "env")
return "bird", method
if config.get("XAI_API_KEY"):
return "xai", "xai"
# Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
from . import xurl_x
if xurl_x.is_available():
@@ -783,17 +795,27 @@ def get_reddit_source(config: dict[str, Any]) -> str | None:
# source; the rest are ordered failover backups, tried only if the one before
# returns nothing or errors. There is one X source ("x"); these are its
# interchangeable backends, never run in parallel.
# xai — xAI/Grok live search (XAI_API_KEY)
# bird — X GraphQL scrape via the user's browser cookies (AUTH_TOKEN/CT0)
# xai — xAI/Grok live search (XAI_API_KEY)
# xurl — official X API v2 (xurl CLI, OAuth2)
# xquik — key-based REST X search (XQUIK_API_KEY); keyless of browser cookies
_X_BACKEND_ORDER = ("xai", "bird", "xurl", "xquik")
# xquik — key-based REST X search (XQUIK_API_KEY)
_X_BACKEND_ORDER = ("bird", "xai", "xurl", "xquik")
# Opt-in backends: never in the unpinned auto chain; require explicit pin.
# grok is here because a leftover ~/.grok/auth.json must never steal the X
# lane. Pin LAST30DAYS_X_BACKEND=grok to enable it.
_X_BACKEND_OPT_IN = ("grok",)
# All known backends (auto chain + opt-in): valid values for the pin var.
_X_BACKEND_KNOWN = _X_BACKEND_ORDER + _X_BACKEND_OPT_IN
# Public routing definitions for the doctor/backend-descriptor layer
# (lib/backends.py). These are aliases for knowledge this module already
# owns — the declared X chain order and the pin/floor env var names — so
# descriptors import one source of truth instead of restating it.
X_BACKEND_ORDER = _X_BACKEND_ORDER
X_BACKEND_OPT_IN = _X_BACKEND_OPT_IN
X_BACKEND_KNOWN = _X_BACKEND_KNOWN
X_BACKEND_PIN_VAR = 'LAST30DAYS_X_BACKEND'
REDDIT_BACKEND_PIN_VAR = 'LAST30DAYS_REDDIT_BACKEND'
REDDIT_SC_MIN_ITEMS_VAR = 'LAST30DAYS_REDDIT_SC_MIN_ITEMS'
@@ -807,6 +829,12 @@ def _x_backend_available(
) -> bool:
if backend == 'xai':
return bool(config.get('XAI_API_KEY'))
if backend == 'grok':
# Keyless relative to X: needs only an installed, signed-in grok CLI.
# Both surfaces are filesystem-only (PATH lookup + credential store),
# so local_only needs no separate branch.
from . import grok_x
return grok_x.has_stored_auth()
if backend == 'bird':
from . import bird_x
return has_bird_creds and bird_x.is_bird_installed()
@@ -830,9 +858,14 @@ def x_backend_chain(config: dict[str, Any], local_only: bool = False) -> list[st
exactly one X source these are its backends, never fetched in parallel.
A ``LAST30DAYS_X_BACKEND`` pin forces a single backend (no failover): the
user explicitly chose it. Browser-cookie probing is intentionally avoided
(automatic Keychain access causes popups); bird counts as available only
when AUTH_TOKEN and CT0 are present explicitly.
user explicitly chose it. Valid pin values are in ``_X_BACKEND_KNOWN``
(the auto chain plus opt-in backends like grok). Browser-cookie probing
is intentionally avoided (automatic Keychain access causes popups); bird
counts as available only when AUTH_TOKEN and CT0 are present explicitly.
Unpinned runs walk only ``_X_BACKEND_ORDER``: opt-in backends like grok
are never auto-selected. A leftover ~/.grok/auth.json must not steal the
X lane; pin ``LAST30DAYS_X_BACKEND=grok`` to enable it explicitly.
``local_only=True`` is the doctor/safe-diagnose flavor: availability is
answered from local evidence only (no subprocess spawns that reach the
@@ -845,11 +878,14 @@ def x_backend_chain(config: dict[str, Any], local_only: bool = False) -> list[st
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
preferred = (config.get(X_BACKEND_PIN_VAR) or '').lower()
if preferred in _X_BACKEND_ORDER:
# Pin accepted from _X_BACKEND_KNOWN (auto chain + opt-in like grok).
if preferred in _X_BACKEND_KNOWN:
if _x_backend_available(preferred, config, has_bird_creds, local_only):
return [preferred]
return []
# Unpinned: walk only _X_BACKEND_ORDER (bird -> xai -> xurl -> xquik).
# Opt-in backends like grok are never auto-selected.
return [
b for b in _X_BACKEND_ORDER
if _x_backend_available(b, config, has_bird_creds, local_only)
@@ -1238,20 +1274,45 @@ def get_x_source_status(config: dict[str, Any], probe: bool = False) -> dict[str
from . import xurl_x as _xurl_x
xurl_available = _xurl_x.is_available() if probe else _xurl_x.has_stored_auth()
# Determine active source. bird (browser cookies) and xAI win when present;
# when neither is available, xquik is the active X source. A probe that
# clearly failed (False) means xquik is not actually usable.
if bird_status["authenticated"]:
# Grok availability is filesystem-only on both paths (PATH lookup plus the
# credential store), so it is safe to compute here regardless of `probe`.
# Grok is opt-in only: it appears in grok_available but never wins the
# unpinned source selection.
from . import grok_x as _grok_x
grok_available = _grok_x.has_stored_auth()
# Determine active source. A pin forces a single backend (R4): ANY known
# pin is exclusive, mirroring x_backend_chain's [] semantics. Pinned
# backend available → that source. Pinned backend unavailable → None.
# Otherwise, order mirrors _X_BACKEND_ORDER: bird first (cookies beat
# XAI_API_KEY when both are present), then xai, then xurl, then xquik.
# Grok is opt-in only and never auto-selected; a leftover ~/.grok/auth.json
# must not steal the X lane.
pin = (config.get(X_BACKEND_PIN_VAR) or '').lower()
if pin and pin in _X_BACKEND_KNOWN:
# Pin is exclusive: pinned backend if available, else None (no fallback).
if pin == 'bird':
source = 'bird' if bird_status["authenticated"] else None
elif pin == 'xai':
source = 'xai' if xai_available else None
elif pin == 'xurl':
source = 'xurl' if xurl_available else None
elif pin == 'xquik':
source = 'xquik' if (xquik_available and xquik_working is not False) else None
elif pin == 'grok':
source = 'grok' if grok_available else None
else:
source = None
elif bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
elif xurl_available:
source = 'xurl'
elif xquik_available and xquik_working is not False:
source = 'xquik'
else:
if xurl_available:
source = 'xurl'
elif xquik_available and xquik_working is not False:
source = 'xquik'
else:
source = None
source = None
return {
"source": source,
@@ -1259,6 +1320,7 @@ def get_x_source_status(config: dict[str, Any], probe: bool = False) -> dict[str
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"grok_available": grok_available,
"xurl_available": xurl_available,
"xquik_available": xquik_available,
"xquik_working": xquik_working,
+5 -1
View File
@@ -16,7 +16,7 @@ from __future__ import annotations
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Callable
from . import log, schema
from . import log, schema, youtube_yt
# Sub-runs hit the same upstream APIs as the main topic. Cap parallelism so a
# 6-way fan-out does not stampede a single backend's rate limit.
@@ -51,6 +51,10 @@ def run_competitor_fanout(
report = main_runner()
return [(main_topic, report)]
# One clear for the whole comparison so entity sub-runs share the YouTube
# search cache without inheriting a prior run's results in this process.
youtube_yt.reset_search_cache()
workers = min(len(competitors) + 1, MAX_PARALLEL_SUBRUNS)
def _run_one(label: str, fn: Callable[[], schema.Report]) -> tuple[str, schema.Report | None, Exception | None]:
+68 -14
View File
@@ -2,6 +2,8 @@
from __future__ import annotations
from collections.abc import Iterable
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
from . import schema
@@ -11,7 +13,18 @@ RRF_K = 60
def _candidate_sort_key(c: schema.Candidate) -> tuple:
return (-c.rrf_score, -c.local_relevance, -c.freshness, schema.candidate_source_label(c), c.title)
# Out-of-window evidence sorts strictly below anything in the window. A
# "last 30 days" brief that ranks a nine-month-old video at #1 breaks its
# own contract, however relevant that video is; it still appears, just
# never above in-window evidence.
return (
1 if schema.candidate_out_of_window(c) else 0,
-c.rrf_score,
-c.local_relevance,
-c.freshness,
schema.candidate_source_label(c),
c.title,
)
def _normalize_url(url: str) -> str:
@@ -39,6 +52,15 @@ _DIVERSITY_RELEVANCE_THRESHOLD = 0.25
# Per-author cap: no single author/handle should dominate the pool.
_MAX_ITEMS_PER_AUTHOR = 3
# Raised cap for the subject of the topic (a handle in the run's
# resolved_handles). On a person or company topic the subject is what the user
# asked about, so the flat cap discards exactly the evidence the run worked
# hardest to retrieve -- the measured 'Peter Steinberger steipete' baseline
# recovered 8 subject-authored posts and would have kept 3. Still bounded: a
# prolific subject must not crowd out commentary about them, which is the other
# half of the answer a user wants.
_MAX_ITEMS_PER_FIRST_PARTY_AUTHOR = 8
def _extract_author(candidate: schema.Candidate) -> str | None:
"""Return a normalized author key from a candidate's source items."""
@@ -51,12 +73,23 @@ def _extract_author(candidate: schema.Candidate) -> str | None:
def _apply_per_author_cap(
candidates: list[schema.Candidate],
max_per_author: int = _MAX_ITEMS_PER_AUTHOR,
first_party_handles: Iterable[str] | None = None,
max_per_first_party_author: int = _MAX_ITEMS_PER_FIRST_PARTY_AUTHOR,
) -> list[schema.Candidate]:
"""Keep at most *max_per_author* items from any single author.
Authors named in *first_party_handles* -- the subject of the topic -- get
the higher *max_per_first_party_author* allowance instead, because their
own posts are the point of the query rather than one voice among many.
Candidates are assumed to already be sorted by quality (rrf_score etc.),
so the first N encountered per author are the best ones.
"""
first_party = {
h.strip().lstrip("@").lower()
for h in (first_party_handles or ())
if h and h.strip()
}
author_counts: dict[str, int] = {}
result: list[schema.Candidate] = []
for c in candidates:
@@ -64,8 +97,13 @@ def _apply_per_author_cap(
if author is None:
result.append(c)
continue
limit = (
max_per_first_party_author
if author.strip().lstrip("@").lower() in first_party
else max_per_author
)
count = author_counts.get(author, 0)
if count < max_per_author:
if count < limit:
result.append(c)
author_counts[author] = count + 1
return result
@@ -112,8 +150,19 @@ def weighted_rrf(
plan: schema.QueryPlan,
*,
pool_limit: int,
range_from: str | None = None,
range_to: str | None = None,
first_party_handles: Iterable[str] | None = None,
) -> list[schema.Candidate]:
"""Fuse ranked lists into a single candidate pool."""
"""Fuse ranked lists into a single candidate pool.
When ``range_from`` and ``range_to`` are provided, they are stored in each
candidate's metadata so ``candidate_out_of_window`` can compare the actual
date against the run window (instead of relying solely on adapter-provided
``date_confidence``). ``first_party_handles`` raises the per-author cap
for the topic's subject so their own posts are not flattened to the
incidental-account allowance.
"""
subqueries = {subquery.label: subquery for subquery in plan.subqueries}
candidates: dict[str, schema.Candidate] = {}
# Track (source, item_id) pairs already attached to each candidate for O(1) dedup.
@@ -129,6 +178,20 @@ def weighted_rrf(
item_freshness = item.freshness if item.freshness is not None else int(item.metadata.get("freshness", 0))
item_source_quality = item.source_quality if item.source_quality is not None else float(item.metadata.get("source_quality", 0.6))
if key not in candidates:
candidate_metadata: dict = {
"provenance": [
{
"source": source,
"subquery_label": label,
"native_rank": rank,
"item_id": item.item_id,
}
]
}
if range_from:
candidate_metadata["range_from"] = range_from
if range_to:
candidate_metadata["range_to"] = range_to
candidates[key] = schema.Candidate(
candidate_id=key,
item_id=item.item_id,
@@ -145,16 +208,7 @@ def weighted_rrf(
rrf_score=score,
sources=[item.source],
source_items=[item],
metadata={
"provenance": [
{
"source": source,
"subquery_label": label,
"native_rank": rank,
"item_id": item.item_id,
}
]
},
metadata=candidate_metadata,
)
seen_source_items[key] = {(item.source, item.item_id)}
continue
@@ -203,5 +257,5 @@ def weighted_rrf(
candidate.snippet = item.snippet
fused = sorted(candidates.values(), key=_candidate_sort_key)
fused = _apply_per_author_cap(fused)
fused = _apply_per_author_cap(fused, first_party_handles=first_party_handles)
return _diversify_pool(fused, pool_limit)
+283 -11
View File
@@ -166,6 +166,38 @@ def _compute_relevance(
return round(relevance, 2)
# GitHub search qualifiers the planner sometimes writes straight into the topic
# string (e.g. "open source AI stars:>1000 created:>2025-03-20"). They must not
# reach the query builder in `search_github`: it appends its own
# `created:>{from_date}`, and when two `created:` qualifiers collide GitHub
# honours the FIRST and silently ignores ours. The API then returns
# out-of-window items that `parse_github_response`'s date filter drops
# wholesale — a source that fetches results and reports zero (issue #949).
QUALIFIER_KEYS = frozenset({
"archived", "assignee", "author", "base", "closed", "comments", "commenter",
"created", "fork", "forks", "head", "in", "interactions", "involves", "is",
"label", "language", "license", "linked", "mentions", "merged", "milestone",
"no", "org", "project", "pushed", "reactions", "repo", "review",
"review-requested", "reviewed-by", "size", "sort", "stars", "state", "team",
"topic", "topics", "type", "updated", "user",
})
_QUALIFIER_RE = re.compile(
r"(?:(?<=[\s,;])|^)(?:" + "|".join(sorted(QUALIFIER_KEYS)) + r"):(?:[<>]=?)?(?:\"[^\"]*\"|[^\s,;()\[\]]+)[,;]?",
re.IGNORECASE,
)
def strip_search_qualifiers(text: str) -> str:
"""Strip GitHub search qualifiers from a topic, leaving plain-language text.
Whitespace is collapsed. Returns an empty string when the topic was nothing
but qualifiers; callers must handle that rather than searching on an empty
term, which would match the entire site.
"""
return " ".join(_QUALIFIER_RE.sub(" ", text).split())
def search_github(
topic: str,
from_date: str,
@@ -193,6 +225,24 @@ def search_github(
"""
count = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
core = extract_core_subject(topic)
plain_core = strip_search_qualifiers(core)
if plain_core != core:
_log(f"Stripped search qualifiers: '{core}' -> '{plain_core}'")
if not plain_core:
# A qualifier-only (or empty) topic leaves nothing to search on.
# Report it instead of querying an empty term, which would match the
# whole site and then be discarded by the date filter as a bogus
# "no results" (issue #949).
_log("Topic contained only search qualifiers or was empty; nothing to search")
return {
"items": [],
"context": {"core": core, "from_date": from_date,
"to_date": to_date, "count": count},
"error": (
f"GitHub topic contained only search qualifiers or was empty: {topic!r}"
),
}
core = plain_core
resolved_token = _resolve_token(token)
authed = bool(resolved_token)
if not authed:
@@ -203,21 +253,66 @@ def search_github(
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
# Build search query with date filter
q = f"{core} created:>{from_date}"
params = {
"q": q,
"sort": "reactions",
"order": "desc",
"per_page": str(min(count, 100)),
}
url = f"{SEARCH_URL}?{urllib.parse.urlencode(params)}"
base_q = f"{core} created:>{from_date}"
def _search(qualifier: Optional[str]) -> Optional[Dict[str, Any]]:
q = f"{base_q} {qualifier}" if qualifier else base_q
params = {
"q": q,
"sort": "reactions",
"order": "desc",
"per_page": str(min(count, 100)),
}
url = f"{SEARCH_URL}?{urllib.parse.urlencode(params)}"
return _fetch_json(url, token=resolved_token, timeout=30,
failure_out=fetch_failures)
fetch_failures: List[str] = []
data = _fetch_json(url, token=resolved_token, timeout=30, failure_out=fetch_failures)
partition_failure: Optional[str] = None
if authed:
# GitHub rejects AUTHENTICATED /search/issues queries that carry
# neither `is:issue` nor `is:pull-request` (HTTP 422). Anonymous
# queries are still grandfathered, which is why this only bites once
# a token is present -- including via the `gh auth token` fallback.
#
# Appending a single qualifier would silently halve the corpus: for
# "rust async created:>2026-08-02" GitHub reports 1,072 issues and
# 7,044 pull requests against 8,115 combined, so `is:issue` alone
# drops ~87% of matches. Query both and merge instead, which keeps
# coverage AND the authenticated rate limit.
merged: List[Dict[str, Any]] = []
seen_ids = set()
failed_qualifiers: List[str] = []
for qualifier in ("is:issue", "is:pull-request"):
part = _search(qualifier)
if part is None:
failed_qualifiers.append(qualifier)
continue
for item in part.get("items", []):
item_id = item.get("id")
if item_id in seen_ids:
continue
seen_ids.add(item_id)
merged.append(item)
# Both sub-queries are reaction-sorted; the merge is not, so re-sort
# before truncating or the second query's tail would outrank the
# first query's head.
merged.sort(key=lambda i: (i.get("reactions") or {}).get("total_count", 0),
reverse=True)
data = {"items": merged[:count]} if merged else None
if failed_qualifiers:
partition_failure = (
f"GitHub partition(s) failed: {', '.join(failed_qualifiers)}"
+ (f" ({fetch_failures[-1]})" if fetch_failures else "")
)
else:
data = _search(None)
if not data:
envelope = {"items": [], "context": {"core": core, "from_date": from_date,
"to_date": to_date, "count": count}}
if authed and fetch_failures:
if authed and partition_failure:
envelope["error"] = partition_failure
elif authed and fetch_failures:
# Authenticated transport failures must not be laundered into a
# clean no-results outcome (issue #384).
envelope["error"] = f"GitHub API request failed: {fetch_failures[-1]}"
@@ -234,7 +329,7 @@ def search_github(
raw_items = data.get("items", [])
_log(f"Found {len(raw_items)} issues/PRs")
return {
envelope: Dict[str, Any] = {
"items": raw_items,
"context": {
"core": core,
@@ -243,6 +338,9 @@ def search_github(
"count": count,
},
}
if partition_failure:
envelope["error"] = partition_failure
return envelope
def parse_github_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
@@ -427,6 +525,8 @@ PERSON_DEPTH_LIMITS = {
"deep": {"pr_pages": 2, "own_repos": 5, "external_repos": 15},
}
PERSON_EVENTS_PER_PAGE = 100
def _fetch_readme_snippet(repo: str, token: str, max_chars: int = 500) -> Optional[str]:
"""Fetch README content for a repo, truncated to first ~max_chars."""
@@ -623,6 +723,17 @@ def search_github_person(
_log(f"Found {total_prs} total PRs, {merged_count} merged")
if total_prs == 0 and merged_count == 0:
# An empty PR search can mean no PRs in the window or an account that
# GitHub's issue index cannot search. Public PushEvents provide an
# actor-attributed fallback for either case.
search_unavailable = total_data is None or merged_data is None
recent = _person_recent_pushes(
username, from_date, to_date, limits, resolved_token,
)
if recent:
reason = "account not searchable" if search_unavailable else "no PRs in window"
_log(f"PR search empty ({reason}); public events returned {len(recent)} items")
return recent
_log("No PRs found, falling back to keyword search")
return []
@@ -822,6 +933,167 @@ def search_github_person(
return items
def _person_recent_pushes(
username: str,
from_date: str,
to_date: str,
limits: Dict[str, int],
token: str,
) -> List[Dict[str, Any]]:
"""Return repos the selected actor publicly pushed inside the window."""
latest_by_repo: Dict[str, Dict[str, str]] = {}
encoded_username = urllib.parse.quote(username, safe="")
page = 1
while True:
url = (
f"https://api.github.com/users/{encoded_username}/events/public"
f"?per_page={PERSON_EVENTS_PER_PAGE}&page={page}"
)
data = _fetch_json(url, token=token, timeout=15)
if not data or not isinstance(data, list):
break
reached_before_window = False
for event in data:
created_at = event.get("created_at")
pushed = _parse_date(created_at)
if not pushed:
continue
if pushed < from_date:
reached_before_window = True
break
if pushed > to_date or event.get("type") != "PushEvent":
continue
actor = event.get("actor")
actor_login = actor.get("login", "") if isinstance(actor, dict) else ""
if actor_login.casefold() != username.casefold():
continue
repo = event.get("repo")
full_name = repo.get("name", "") if isinstance(repo, dict) else ""
if not re.fullmatch(r"[^/\s]+/[^/\s]+", full_name):
continue
previous = latest_by_repo.get(full_name)
if previous is None or created_at > previous["created_at"]:
latest_by_repo[full_name] = {
"full_name": full_name,
"pushed": pushed,
"created_at": created_at,
"actor": actor_login,
"event_id": str(event.get("id") or ""),
}
if reached_before_window or len(data) < PERSON_EVENTS_PER_PAGE:
break
page += 1
if not latest_by_repo:
return []
recent = sorted(
latest_by_repo.values(),
key=lambda r: r["created_at"],
reverse=True,
)
_log(
f"Public events: {len(recent)} actor-attributed repos pushed in window, "
"loading repository metadata for ranking"
)
repo_info: Dict[str, Dict[str, Any]] = {}
with ThreadPoolExecutor(max_workers=8) as executor:
info_futures = {
executor.submit(_fetch_repo_info, r["full_name"], token): r["full_name"]
for r in recent
}
for future in as_completed(info_futures):
name = info_futures[future]
try:
repo_info[name] = future.result(timeout=20) or {}
except Exception as exc:
_log(f"Push-event repo metadata failed for {name}: {exc}")
repo_info[name] = {}
recent.sort(
key=lambda r: (
repo_info.get(r["full_name"], {}).get("stars", 0),
r["created_at"],
),
reverse=True,
)
selected = recent[:limits["own_repos"]]
enrichments: Dict[str, Dict[str, Any]] = {}
_log(f"Public events: enriching {len(selected)} top-ranked repositories")
with ThreadPoolExecutor(max_workers=8) as executor:
enrichment_futures = {
executor.submit(_enrich_own_repo, r["full_name"], token): r["full_name"]
for r in selected
}
for future in as_completed(enrichment_futures):
name = enrichment_futures[future]
try:
enrichments[name] = future.result(timeout=25)
except Exception as exc:
_log(f"Push-event enrichment failed for {name}: {exc}")
enrichments[name] = {}
items: List[Dict[str, Any]] = []
for idx, repo in enumerate(selected, start=1):
name = repo["full_name"]
info = repo_info.get(name, {})
stars = info.get("stars", 0)
stars_str = _format_stars(stars)
open_issues = info.get("open_issues", 0)
enrichment = enrichments.get(name, {})
readme = enrichment.get("readme")
releases = enrichment.get("releases", [])
snippet_parts = [
f"@{repo['actor']} pushed {name} on {repo['pushed']} "
f"({stars_str} stars, {open_issues} open issues)"
]
if info.get("description"):
snippet_parts.append(f" {info['description']}")
if readme:
snippet_parts.append(f" README: {readme[:300]}")
for rel in releases[:2]:
body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
snippet_parts.append(f" Release: {rel['name']} ({rel['date']}){body_preview}")
items.append({
"id": f"GH{idx}",
"title": f"@{repo['actor']} pushed {name} on {repo['pushed']}",
"url": f"https://github.com/{name}",
"date": repo["pushed"],
"author": repo["actor"],
"source": "github",
"score": stars,
"container": name,
"snippet": "\n".join(snippet_parts),
"relevance": min(0.9, 0.6 + math.log1p(stars) / 30),
"why_relevant": (
f"GitHub activity: @{repo['actor']} pushed {name} on {repo['pushed']} "
f"({stars_str} stars)"
),
"engagement": {"stars": stars, "comments": open_issues},
"metadata": {
"labels": ["person-profile", "recent-push"],
"state": "open",
"comment_count": open_issues,
"reactions": stars,
"is_pr": False,
"event_type": "PushEvent",
"event_id": repo["event_id"],
},
})
return items
def _enrich_external_repo(repo: str, token: str) -> Dict[str, Any]:
"""Fetch star count + releases for an external repo."""
info = _fetch_repo_info(repo, token)
+965
View File
@@ -0,0 +1,965 @@
"""X (Twitter) search via the Grok CLI — no X credential of any kind.
The `grok` CLI (https://x.ai/cli) exposes X search tools natively
(`x_keyword_search`, `x_semantic_search`, `x_thread_fetch`, `x_user_search`).
Reaching X through it needs no X account, no browser cookies, and no
`XAI_API_KEY` only an installed and signed-in `grok`.
Install: curl -fsSL https://x.ai/cli/install.sh | bash (or npm i -g @xai-official/grok)
Auth: grok login
Two invocation constraints, both measured, both load-bearing:
* **Never pass `--json-schema`.** Constrained decoding competes with tool use:
the search silently does not run and the model fills the schema's required
fields from training data instead. Measured with an interleaved A/B
controlling for time: plain output returned verified in-window posts on 4 of
4 calls, `--json-schema` on 1 of 4.
* **Never pass `--tools`.** Two runs produced no output in 7 minutes and were
killed; the identical prompts without it completed normally.
Because retrieval is performed by a language model rather than an API client,
its output can be *confidently wrong* in a way no other backend's can. Every
returned post is therefore validated against the requested window via its
snowflake timestamp before it is allowed into the item flow see
`_validate_items`. Author matching and schema shape are not sufficient: a
fabricated post carries a plausible handle and a numeric id by construction.
"""
import json
import os
import re
import shutil
import subprocess
import tempfile
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
from . import log
from .relevance import token_overlap_relevance as _compute_relevance
def _log(msg: str) -> None:
log.source_log("Grok", msg, tty_only=False)
# Posts requested per call. The tool caps `limit` at 10, so depth is achieved
# by fanning out across queries rather than by raising a single call's limit.
_MAX_LIMIT_PER_CALL = 10
# Upper bound on calls per topic search. Each call is an LLM subprocess of
# roughly 15-45s, so depth must not translate into unbounded wall time.
_MAX_FANOUT_CALLS = 4
# Wall-clock ceiling for ALL grok Phase 2 lanes combined. Without it, three
# lanes over three handles is up to 14 sequential LLM subprocess calls bounded
# only by per-call timeouts -- tens of minutes of foreground time for a source
# that now runs by default. Lanes stop issuing queries once this passes and
# return whatever they have.
LANE_BUDGET_SECONDS = 150.0
# Below this, a call cannot plausibly complete (measured calls run 15-45s), so
# the budget is spent rather than overrun. Skipping is strictly better than
# starting a call guaranteed to be killed mid-flight.
_MIN_USEFUL_CALL_SECONDS = 15
def _is_proper_name(topic: str) -> bool:
"""True when topic looks like a title-cased proper name (person/product).
"Peter Steinberger" True (phrase-quote in fanout)
"Rome Italy" False (no phrase-quote; place/disambiguation string)
"""
words = topic.split()
if len(words) < 2:
return False
# Title-cased: each word starts uppercase, rest lowercase
# Place names like "Rome Italy" are title-cased but are NOT proper names
# for phrase-quoting purposes. Heuristic: if ALL words are common place/
# disambiguation words OR all-caps acronyms, don't phrase-quote.
place_words = {
"italy", "rome", "paris", "london", "berlin", "tokyo", "new", "york",
"los", "angeles", "san", "francisco", "city", "country", "state",
"north", "south", "east", "west", "united", "states", "kingdom",
}
lower_words = [w.lower() for w in words]
if all(w in place_words or w.isupper() for w in lower_words):
return False
# Check for title case pattern (First Last, First Middle Last)
return all(
w[0].isupper() and (len(w) == 1 or w[1:].islower())
for w in words
if w.isalpha()
)
def _fanout_queries(topic: str, from_date: str, to_date: str, calls: int) -> List[str]:
"""Distinct query formulations for one topic, widest signal first.
Each returns at most 10 posts, and the formulations surface different
sets -- Top vs Latest ordering, and an engagement-floored variant -- so
fanning out adds coverage rather than repeating one result set.
Multi-word topics are NOT phrase-quoted unless they look like proper names
(person/product). "Rome Italy" no phrase-quote (place/disambiguation).
"Peter Steinberger" phrase-quote in one variant (proper name).
"""
window = f"since:{from_date} until:{to_date}"
# First variant: unquoted AND (multi-word topics naturally AND their terms)
variants = [
f"{topic} {window}",
f"{topic} {window} min_faves:5",
]
# Third variant: phrase-quote only for proper names, else filter:links
if " " in topic and _is_proper_name(topic):
variants.append(f'"{topic}" {window}')
else:
variants.append(f"{topic} {window} filter:links")
variants.append(f"{topic} {window} -filter:replies")
return variants[:calls]
DEPTH_CONFIG = {
"quick": 10,
"default": 30,
"deep": 60,
}
# Wall-clock ceiling for one `grok` invocation. A run that blocks on an
# unexpected interactive prompt would otherwise hang indefinitely, and a
# non-daemon worker can outlive a wall-clock budget.
_TIMEOUT_SECONDS = {"quick": 120, "default": 240, "deep": 360}
# Twitter/X snowflake epoch (2010-11-04T01:42:54.657Z) in milliseconds.
_SNOWFLAKE_EPOCH_MS = 1288834974657
_AUTH_STORE = Path.home() / ".grok" / "auth.json"
# Substrings that indicate stored credentials. Deliberately format-agnostic:
# the observed store is a JSON object keyed by issuer and principal, but the
# shape is the vendor's to change. Mirrors xurl_x's marker scan.
_TOKEN_STORE_MARKERS = ("refresh_token", "access_token", "auth_mode", '"key"')
AUTH_OK = "ok" # token store present with non-expired credentials
AUTH_EXPIRED = "expired" # credentials present but access_token expires_at is past
AUTH_MISSING = "missing" # no token store, or no credentials stored in it
AUTH_ERROR = "error" # token store exists but could not be read
# Markers that indicate the Grok session was revoked mid-run (refresh failed).
# When these appear in grok CLI stderr/stdout, the run should fall back once
# and not retry grok in that run. Distinct from "never signed in" since a prior
# run may have succeeded with the same auth.json.
_AUTH_REVOKED_MARKERS = (
"not signed in",
"not logged in",
"invalid_grant",
"refresh token has been revoked",
"session expired",
"authentication failed",
"unauthorized",
)
_availability_cache: Optional[bool] = None
def clear_availability_cache() -> None:
"""Reset the memoized is_available() result (tests, or a re-check after login)."""
global _availability_cache
_availability_cache = None
def binary_path() -> Optional[str]:
"""Resolved `grok` path, or None when it is not on PATH.
PATH resolution is the gate, not file existence: a binary present on disk
but off the agent subprocess PATH is not installed as far as the engine is
concerned.
"""
return shutil.which("grok")
def token_store_path() -> Path:
return _AUTH_STORE
def _find_expires_at(obj: Any) -> Optional[str]:
"""Recursively find expires_at in a nested dict/list structure.
The Grok auth.json is keyed by issuer and principal; this finds expires_at
anywhere in the tree without assuming the structure.
"""
if isinstance(obj, dict):
if "expires_at" in obj:
return obj["expires_at"]
for v in obj.values():
found = _find_expires_at(v)
if found is not None:
return found
elif isinstance(obj, list):
for item in obj:
found = _find_expires_at(item)
if found is not None:
return found
return None
def _parse_expires_at(raw: str) -> Optional[datetime]:
"""Parse an ISO 8601 expires_at timestamp."""
if not raw:
return None
try:
normalized = raw.replace("Z", "+00:00")
return datetime.fromisoformat(normalized)
except (TypeError, ValueError):
return None
def stored_auth_status() -> Tuple[str, str, Optional[datetime]]:
"""Local-only auth check: filesystem read, no subprocess, no network.
This is the doctor / --diagnose / --preflight surface. It must never spawn
a process: the whole-doctor-path test patches ``subprocess.run`` to raise,
and shelling out to `grok` here would fail it.
Returns (status, detail, expires_at). The expires_at datetime is None when
not parseable or not present. Status is:
- AUTH_OK: credentials present and not expired (or no expires_at to check)
- AUTH_EXPIRED: credentials present but expires_at is in the past
- AUTH_MISSING: no token store or no credential markers
- AUTH_ERROR: token store exists but could not be read
"""
path = token_store_path()
try:
if not path.exists():
return AUTH_MISSING, f"no Grok credential store at {path}", None
raw = path.read_text(encoding="utf-8", errors="replace")
except OSError as exc:
return AUTH_ERROR, f"{type(exc).__name__}: {exc}", None
if not any(marker in raw for marker in _TOKEN_STORE_MARKERS):
return AUTH_MISSING, f"Grok credential store at {path} has no stored credentials", None
expires_at: Optional[datetime] = None
try:
data = json.loads(raw)
expires_str = _find_expires_at(data)
expires_at = _parse_expires_at(expires_str) if expires_str else None
except (json.JSONDecodeError, TypeError):
pass
if expires_at is not None:
now = datetime.now(timezone.utc)
if expires_at.tzinfo is None:
expires_at = expires_at.replace(tzinfo=timezone.utc)
if expires_at < now:
return (
AUTH_EXPIRED,
f"Grok session expired at {expires_at.isoformat()} "
f"(refresh may restore it; if revoked, run `grok login --device-auth`)",
expires_at,
)
return AUTH_OK, f"stored Grok credentials found in {path}", expires_at
def has_stored_auth() -> bool:
"""True when grok binary is on PATH and credentials are stored.
NOTE: This returns True even when AUTH_EXPIRED, because the refresh_token
might still work. The caller (is_available) decides whether to attempt
grok anyway. Doctor uses the status directly to show the degraded state.
"""
if binary_path() is None:
return False
status = stored_auth_status()[0]
return status in (AUTH_OK, AUTH_EXPIRED)
def is_available() -> bool:
"""Research-time availability. May spawn a subprocess; memoized per process."""
global _availability_cache
if _availability_cache is None:
_availability_cache = _is_available_uncached()
return _availability_cache
def _is_available_uncached() -> bool:
"""Research-time availability check.
Returns True when grok is on PATH and credentials exist, even if
AUTH_EXPIRED. Rationale: expires_at being in the past does not prove the
refresh_token is dead. The CLI will attempt OIDC refresh at run time and
might succeed. Only a runtime failure ("Not signed in", invalid_grant)
proves the session is truly revoked.
"""
if binary_path() is None:
return False
status = stored_auth_status()[0]
return status in (AUTH_OK, AUTH_EXPIRED)
def _subprocess_env(home: str) -> Dict[str, str]:
"""Minimal environment for the `grok` child process, rooted at a throwaway HOME.
The child runs with tool permissions bypassed (non-interactivity requires
it) while its context is filled with retrieved X post text, which is
attacker-controlled. Stripping credential env vars is necessary but not
sufficient: this engine writes XAI_API_KEY / AUTH_TOKEN / CT0 /
SCRAPECREATORS_API_KEY to ``$HOME/.config/last30days/.env``, and ``~/.ssh``
and ``~/.aws`` sit alongside it. An empty cwd is not a boundary for a
filesystem-capable agent -- cwd constrains relative paths, not ``$HOME/...``
reads -- so the child gets its own HOME containing only the Grok credential
store it actually needs.
"""
keep = ("PATH", "LANG", "LC_ALL", "TMPDIR", "SystemRoot")
env = {k: os.environ[k] for k in keep if k in os.environ}
env.setdefault("PATH", os.defpath)
env["HOME"] = home
if os.name == "nt":
env["USERPROFILE"] = home
return env
def _stage_child_home(workdir: str) -> str:
"""Create the child's throwaway HOME holding only the credential file.
Copies ``auth.json`` alone, never the ``~/.grok`` tree: that directory is
~1.6 GB (marketplace cache, bundled runtime, session history), and copying
it per invocation made a single search take minutes. The child needs the
credential to authenticate and nothing else -- session history and caches
are state we specifically do not want a permission-bypassed child to read
or mutate.
Copied rather than symlinked so the child cannot follow a link back to the
real store, and copied rather than shared so it cannot rewrite the user's
credentials.
"""
home = os.path.join(workdir, "home")
store = token_store_path()
child_store_dir = os.path.join(home, store.parent.name)
os.makedirs(child_store_dir, mode=0o700, exist_ok=True)
try:
if store.is_file():
shutil.copyfile(store, os.path.join(child_store_dir, store.name))
os.chmod(os.path.join(child_store_dir, store.name), 0o600)
except OSError as exc:
_log(f"could not stage Grok credentials for the child: {exc}")
return home
def _decode_snowflake(post_id: str) -> Optional[datetime]:
"""Recover a post's creation time from its id, with no network call."""
try:
value = int(str(post_id).strip())
except (TypeError, ValueError):
return None
if value <= 0:
return None
try:
return datetime.fromtimestamp(
((value >> 22) + _SNOWFLAKE_EPOCH_MS) / 1000, tz=timezone.utc
)
except (OverflowError, OSError, ValueError):
return None
def _looks_generated(ids: List[str]) -> bool:
"""True when ids form a near-uniform arithmetic run.
Real ranked results are not evenly spaced in time. A fabricated set often
is, because the model interpolates a plausible-looking id sequence. Four
ids is the minimum used here: three gaps are needed before a near-uniform
step reads as generated rather than coincidental.
"""
numeric = []
for pid in ids:
try:
numeric.append(int(pid))
except (TypeError, ValueError):
return False
if len(numeric) < 4:
return False
numeric.sort()
gaps = [b - a for a, b in zip(numeric, numeric[1:])]
if any(g <= 0 for g in gaps):
return False
mean = sum(gaps) / len(gaps)
if mean <= 0:
return False
# Every gap within 5% of the mean is not something real timelines do.
return all(abs(g - mean) / mean < 0.05 for g in gaps)
# Why the most recent parse returned nothing. Lets _run_query distinguish a
# clean empty window (common, and not worth a second LLM call) from a suspect
# response (fabricated ids, a generated sequence, a self-reported
# non-execution), which is the only case retrying can actually fix.
_LAST_REJECTION = {"reason": ""}
_RETRYABLE_REJECTIONS = (
"failed provenance validation",
"near-uniform sequence",
"unparsable date window",
)
_PLACEHOLDER_HANDLES = {"unknown", "n/a", "none", "null", "example", "user", ""}
# X's real handle grammar. Model-reported handles are interpolated into post
# URLs and into the NEXT child prompt, so anything outside this charset is
# rejected rather than passed through: a poisoned post that steers the child
# into emitting a crafted handle line would otherwise reach a prompt slot it
# can close. entity_extract applies the same rule to @mentions.
_HANDLE_RE = re.compile(r"[A-Za-z0-9_]{1,15}")
def _clean_handle(value: str) -> str:
"""Return a grammar-valid handle, or '' when the value is not one."""
candidate = str(value or "").strip().lstrip("@")
return candidate if _HANDLE_RE.fullmatch(candidate) else ""
_NON_EXECUTION_MARKERS = (
"was not executed",
"not executed in this turn",
"unable to search",
"could not search",
"no tool call",
"tool not available",
)
def _validate_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> Tuple[List[Dict[str, Any]], str]:
"""Drop anything that did not come from a real in-window post.
Returns (kept, reason). A non-empty reason means the response should be
treated as a non-execution to retry rather than as a thin result.
"""
if not items:
return [], "no items parsed"
try:
lo = datetime.strptime(from_date, "%Y-%m-%d").replace(tzinfo=timezone.utc)
hi = datetime.strptime(to_date, "%Y-%m-%d").replace(tzinfo=timezone.utc)
except (TypeError, ValueError):
# Fail closed. Skipping the window check would silently disable the
# module's central provenance guarantee for the whole response.
return [], "unparsable date window"
kept: List[Dict[str, Any]] = []
for item in items:
text = str(item.get("text") or "").lower()
if any(marker in text for marker in _NON_EXECUTION_MARKERS):
continue
handle = str(item.get("author_handle") or "").strip().lstrip("@").lower()
if handle in _PLACEHOLDER_HANDLES:
continue
created = _decode_snowflake(item.get("post_id"))
if created is None:
continue
if not (lo <= created <= hi.replace(hour=23, minute=59, second=59)):
continue
kept.append(item)
if not kept:
return [], "every item failed provenance validation (window/handle/id)"
if _looks_generated([str(i.get("post_id")) for i in kept]):
return [], "post ids form a near-uniform sequence (generated, not retrieved)"
return kept, ""
# --- prose parsing ---------------------------------------------------------
_FIELD_ALIASES = {
"id": "post_id",
"post id": "post_id",
"conversation id": "conversation_id",
"author": "author",
"handle": "author_handle",
"text": "text",
"content": "text",
"created_at": "created_at",
"timestamp": "created_at",
"likes": "likes",
"reposts": "reposts",
"retweets": "reposts",
"replies": "replies",
"quotes": "quotes",
"bookmarks": "bookmarks",
"views": "views",
}
_FIELD_LINE = re.compile(
r"^[\s\-*>]*\**\s*([A-Za-z][A-Za-z _]{1,20}?)\**\s*[:=]\s*(.+?)\s*$"
)
# Engagement counts arrive either literal ("1,462") or display-abbreviated
# ("39K", "1.2M"). Parsing only the leading digits turns 1.2M into 1, which
# does not merely lose precision -- it inverts ranking, placing a viral post
# below one with 500 literal likes.
_INT_RE = re.compile(r"(-?\d[\d,]*(?:\.\d+)?)\s*([KMB])?", re.I)
_SUFFIX_MULTIPLIER = {"k": 1_000, "m": 1_000_000, "b": 1_000_000_000}
def _as_int(value: str) -> Optional[int]:
match = _INT_RE.search(value or "")
if not match:
return None
number, suffix = match.group(1), match.group(2)
try:
parsed = float(number.replace(",", ""))
except ValueError:
return None
if suffix:
parsed *= _SUFFIX_MULTIPLIER[suffix.lower()]
return int(parsed)
def _parse_date(value: str) -> Optional[str]:
value = (value or "").strip()
for fmt in ("%a, %d %b %Y %H:%M:%S %Z", "%a %b %d %H:%M:%S %z %Y"):
try:
return datetime.strptime(value, fmt).strftime("%Y-%m-%d")
except (TypeError, ValueError):
continue
try:
return datetime.fromisoformat(value.replace("Z", "+00:00")).strftime("%Y-%m-%d")
except (TypeError, ValueError):
return None
def _split_blocks(text: str) -> List[str]:
"""Split the model's prose into per-post blocks.
Keyed on the post-id field starting a new record rather than on any
heading style, because the narration around the blocks varies run to run.
"""
blocks: List[str] = []
current: List[str] = []
for line in (text or "").splitlines():
match = _FIELD_LINE.match(line)
key = _FIELD_ALIASES.get(match.group(1).strip().lower()) if match else None
if key == "post_id" and current:
blocks.append("\n".join(current))
current = []
if match or current:
current.append(line)
if current:
blocks.append("\n".join(current))
return blocks
def parse_x_response(
response: Dict[str, Any],
topic: str = "",
from_date: str = "",
to_date: str = "",
) -> List[Dict[str, Any]]:
"""Parse a grok response into normalized X item dicts.
Total: returns [] on error rather than raising.
"""
if not isinstance(response, dict):
return []
if response.get("error"):
_log(f"error: {response['error']}")
return []
raw: List[Dict[str, Any]] = []
for block in _split_blocks(response.get("text") or ""):
fields: Dict[str, Any] = {}
for line in block.splitlines():
match = _FIELD_LINE.match(line)
if not match:
continue
key = _FIELD_ALIASES.get(match.group(1).strip().lower())
if key and key not in fields:
value = match.group(2).strip()
# Field values arrive with varying markdown decoration
# (`- **id:** 123`), so strip emphasis and code marks.
value = value.strip("*").strip().strip("`").strip()
fields[key] = value
if fields.get("post_id"):
raw.append(fields)
kept, reason = _validate_items(raw, from_date, to_date) if from_date else (raw, "")
if reason:
_log(f"rejected response: {reason}")
_LAST_REJECTION["reason"] = reason
return []
_LAST_REJECTION["reason"] = ""
items: List[Dict[str, Any]] = []
seen_ids = set()
for index, fields in enumerate(kept, start=1):
post_id = str(fields.get("post_id") or "").strip()
if post_id in seen_ids:
continue
seen_ids.add(post_id)
handle = _clean_handle(fields.get("author_handle"))
if not handle:
author = str(fields.get("author") or "")
match = re.search(r"@([A-Za-z0-9_]{1,15})", author)
handle = match.group(1) if match else ""
if not handle:
continue
text = str(fields.get("text") or "").strip()[:500]
engagement = {
"likes": _as_int(str(fields.get("likes", ""))),
"reposts": _as_int(str(fields.get("reposts", ""))),
"replies": _as_int(str(fields.get("replies", ""))),
"quotes": _as_int(str(fields.get("quotes", ""))),
}
items.append({
"id": f"GK{index}",
"text": text,
"url": f"https://x.com/{handle}/status/{post_id}",
"author_handle": handle,
"date": _parse_date(str(fields.get("created_at", ""))),
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "",
"relevance": _compute_relevance(topic, text) if topic else 0.7,
})
return items
# --- invocation ------------------------------------------------------------
_PROMPT = """Use {tool} with query '{query}', mode Top, limit {limit}.
Report every post the tool returned, one block per post, using exactly these
field labels on their own lines:
id: <numeric post id>
handle: <author handle without @>
created_at: <post timestamp>
likes: <number>
reposts: <number>
replies: <number>
quotes: <number>
text: <full post text on one line>
Report only posts the tool actually returned. If the tool returned nothing or
could not run, say so plainly and report no post blocks. Do not supply posts
from your own knowledge."""
def is_auth_revoked_error(error: str) -> bool:
"""True when the error indicates the Grok session was revoked mid-run.
Distinct from "never signed in": the user may have had a working session
that expired or was revoked (e.g., OIDC refresh returned invalid_grant).
"""
if not error:
return False
text = error.lower()
return any(marker in text for marker in _AUTH_REVOKED_MARKERS)
def classify_run_failure(detail: str) -> str:
"""Classify a grok run failure into a health state.
Used by the pipeline to report typed outcomes (AUTH_FAILED vs generic
ERROR) so doctor and the host can surface the right fix.
"""
from . import health
if not detail:
return health.ERROR
text = detail.lower()
if any(marker in text for marker in _AUTH_REVOKED_MARKERS):
return health.AUTH_FAILED
if "timed out" in text or "timeout" in text:
return health.TIMEOUT
return health.ERROR
def _invoke(prompt: str, timeout: int) -> Dict[str, Any]:
"""Run `grok` once. Never raises; every failure returns {'error': str}.
When the error indicates auth revocation (refresh token rejected, not
signed in, etc.), the response also carries 'auth_revoked': True so
callers can fall back without retrying grok.
"""
binary = binary_path()
if binary is None:
return {"error": "grok CLI not found on PATH"}
try:
with tempfile.TemporaryDirectory(prefix="last30days-grok-") as workdir:
child_home = _stage_child_home(workdir)
result = subprocess.run(
[binary, "-p", prompt, "--permission-mode", "bypassPermissions"],
capture_output=True,
text=True,
timeout=timeout,
cwd=workdir,
env=_subprocess_env(child_home),
)
except FileNotFoundError:
return {"error": "grok CLI not found on PATH"}
except subprocess.TimeoutExpired:
return {"error": f"grok CLI timed out after {timeout}s"}
except OSError as exc:
return {"error": f"{type(exc).__name__}: {exc}"}
except Exception as exc: # noqa: BLE001 - search_x must never raise
return {"error": f"{type(exc).__name__}: {exc}"}
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()[:300]
error_msg = f"grok CLI exited {result.returncode}: {detail}"
response: Dict[str, Any] = {"error": error_msg}
if is_auth_revoked_error(detail):
response["auth_revoked"] = True
return response
return {"text": result.stdout or ""}
def _run_query(
query: str,
from_date: str,
to_date: str,
*,
tool: str = "x_keyword_search",
limit: int = _MAX_LIMIT_PER_CALL,
depth: str = "default",
attempts: int = 2,
relevance_topic: str = "",
deadline: Optional[float] = None,
) -> Tuple[List[Dict[str, Any]], str, bool]:
"""Run one query, retrying only when the response looks fabricated.
A clean empty result is NOT retried: an empty window is a common, correct
outcome (especially for the mention lane on a low-profile handle and for
the name lane's engagement floor), and re-issuing a byte-identical prompt
doubles latency and Grok-plan spend to get the same answer.
Returns (items, error, auth_revoked). When auth_revoked is True, the caller
should not retry grok in this run.
"""
timeout = _TIMEOUT_SECONDS.get(depth, _TIMEOUT_SECONDS["default"])
prompt = _PROMPT.format(tool=tool, query=query, limit=min(limit, _MAX_LIMIT_PER_CALL))
last_error = ""
for attempt in range(1, attempts + 1):
if deadline is not None:
remaining = deadline - time.monotonic()
if remaining < _MIN_USEFUL_CALL_SECONDS:
return [], last_error or "X lane budget exhausted", False
timeout = min(timeout, int(remaining))
_log(f"searching: {query}" + (f" (attempt {attempt})" if attempt > 1 else ""))
response = _invoke(prompt, timeout)
if response.get("error"):
last_error = response["error"]
if response.get("auth_revoked"):
return [], last_error, True
continue
items = parse_x_response(
response,
topic=relevance_topic or query,
from_date=from_date,
to_date=to_date,
)
if items:
return items, "", False
reason = _LAST_REJECTION.get("reason", "")
if not any(marker in reason for marker in _RETRYABLE_REJECTIONS):
return [], "", False
last_error = reason or "no verified in-window posts returned"
return [], last_error, False
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X for a topic, fanning out to reach the depth's target count.
The underlying tool caps each call at 10 posts, so depth is achieved across
calls. Without this, grok returned 10 posts at every depth while sitting
ahead of bird in the chain -- silently downgrading a `--deep` run from 60
posts to 10.
Returns {'items': [...]}; 'error' is set only for an actual invocation
failure. A completed run that found nothing returns an empty list with no
error, matching bird and xquik -- reporting "no results" as a hard failure
would make an empty window look like a broken backend.
When the Grok session is revoked mid-run (refresh token rejected),
'auth_revoked': True is set so the pipeline can fall back without retrying
grok and can surface the correct fix to the user.
"""
target = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
calls = max(1, min(_MAX_FANOUT_CALLS, -(-target // _MAX_LIMIT_PER_CALL)))
collected: List[Dict[str, Any]] = []
seen: set = set()
last_error = ""
invocation_failed = False
auth_revoked = False
for mode_query in _fanout_queries(topic, from_date, to_date, calls):
items, error, revoked = _run_query(mode_query, from_date, to_date, depth=depth,
relevance_topic=topic)
if revoked:
auth_revoked = True
last_error = error or "Grok session expired or was revoked"
break
if error and not items:
last_error = error
if "not found" in error or "timed out" in error or "exited" in error:
invocation_failed = True
for item in items:
key = item["url"]
if key not in seen:
seen.add(key)
collected.append(item)
if len(collected) >= target:
break
for index, item in enumerate(collected, start=1):
item["id"] = f"GK{index}"
if collected:
result: Dict[str, Any] = {"items": collected[:target]}
if auth_revoked:
result["auth_revoked"] = True
return result
if auth_revoked:
return {"items": [], "error": last_error, "auth_revoked": True}
if invocation_failed:
return {"items": [], "error": last_error}
return {"items": []}
def search_handles(
handles: List[str],
topic: str,
from_date: str,
to_date: str,
*,
count_per: int = 8,
deadline: Optional[float] = None,
and_topic: bool = False,
) -> Tuple[List[Dict[str, Any]], bool]:
"""BY lane: posts authored by each handle.
``topic`` is used for relevance ranking only and is never ANDed into the
query by default -- doing so was a prior defect that emptied the lane
(person posts omit their own name).
When ``and_topic=True``, the topic IS ANDed into the query (e.g.,
``from:handle Rome``) to ensure extracted handles demonstrate on-topic
content. This prevents off-topic timelines from filling the X budget.
Returns (items, auth_revoked) so the pipeline can record AUTH_FAILED.
"""
collected: List[Dict[str, Any]] = []
auth_revoked = False
for handle in handles:
if deadline is not None and time.monotonic() >= deadline:
_log("lane budget exhausted; skipping remaining handles")
break
clean = _clean_handle(handle)
if not clean:
continue
# AND topic only when explicitly requested (extracted handles)
if and_topic and topic:
query = f"from:{clean} {topic} since:{from_date} until:{to_date}"
else:
query = f"from:{clean} since:{from_date} until:{to_date}"
items, _, revoked = _run_query(
query,
from_date, to_date, limit=count_per, relevance_topic=topic,
attempts=1, deadline=deadline,
)
if revoked:
_log("Grok session revoked; stopping lane")
auth_revoked = True
break
collected.extend(
i for i in items
if i["author_handle"].lower() == clean.lower()
)
return collected, auth_revoked
def search_mentions(
handles: List[str],
from_date: str,
to_date: str,
*,
topic: str = "",
count_per: int = 5,
deadline: Optional[float] = None,
) -> Tuple[List[Dict[str, Any]], bool]:
"""ABOUT lane (mention form): posts @-mentioning each handle.
Returns (items, auth_revoked) so the pipeline can record AUTH_FAILED.
"""
collected: List[Dict[str, Any]] = []
auth_revoked = False
for handle in handles:
if deadline is not None and time.monotonic() >= deadline:
_log("lane budget exhausted; skipping remaining handles")
break
clean = _clean_handle(handle)
if not clean:
continue
items, _, revoked = _run_query(
f"@{clean} -from:{clean} since:{from_date} until:{to_date}",
from_date, to_date, limit=count_per, relevance_topic=topic,
attempts=1, deadline=deadline,
)
if revoked:
_log("Grok session revoked; stopping lane")
auth_revoked = True
break
collected.extend(
i for i in items
if i["author_handle"].lower() != clean.lower()
)
return collected, auth_revoked
def search_name(
name: str,
from_date: str,
to_date: str,
*,
exclude_handles: Optional[List[str]] = None,
count_per: int = 8,
min_faves: int = 2,
deadline: Optional[float] = None,
) -> Tuple[List[Dict[str, Any]], bool]:
"""ABOUT lane (name form): posts naming the subject in plain text.
Not redundant with the mention lane and not a fallback for it. Most talk
about a person or company never @-mentions them -- people write "Bentgo
lunch box from Costco", not "@Bentgo lunch box from Costco". A modest
engagement floor applies here only, because a bare name query is the
widest and noisiest of the three lanes.
Returns (items, auth_revoked) so the pipeline can record AUTH_FAILED.
"""
name = (name or "").strip()
if not name:
return [], False
if name.count('"') % 2:
name = name.replace('"', " ").strip()
phrase = f'"{name}"' if " " in name else name
excludes = " ".join(
f"-from:{clean}"
for clean in (_clean_handle(h) for h in (exclude_handles or []))
if clean
)
query = " ".join(
part for part in
[phrase, excludes, f"min_faves:{min_faves}", f"since:{from_date}", f"until:{to_date}"]
if part
)
items, _, revoked = _run_query(
query, from_date, to_date, limit=count_per, attempts=1, deadline=deadline,
)
if revoked:
_log("Grok session revoked")
blocked = {c.lower() for c in (_clean_handle(h) for h in (exclude_handles or [])) if c}
return [i for i in items if i["author_handle"].lower() not in blocked], revoked
+1 -1
View File
@@ -339,7 +339,7 @@ def _fetch_item_comments(object_id: str, max_comments: int = 5) -> Dict[str, Any
comments.append({
"author": c.get("author", ""),
"text": excerpt,
"points": c.get("points") or 0,
"points": c.get("points"),
})
# First sentence as insight
first_sentence = text.split(". ")[0].split("\n")[0][:200]
+14 -1
View File
@@ -132,7 +132,7 @@ _PP_CLI_SUFFIX = "-pp-cli"
_PRINTING_PRESS_NPM = "@mvanhorn/printing-press-library@0.1.16"
# Dependencies the doctor probes by default.
KNOWN_DEPENDENCIES: Tuple[str, ...] = ("yt-dlp", "digg-pp-cli", "node", "ffmpeg")
KNOWN_DEPENDENCIES: Tuple[str, ...] = ("yt-dlp", "digg-pp-cli", "node", "ffmpeg", "grok")
# Cheap side-effect-free version invocation per dependency (default --version).
_VERSION_ARGS: Dict[str, List[str]] = {
@@ -157,6 +157,15 @@ _MANAGER_PRESCRIPTIONS: Dict[str, Dict[str, Tuple[str, str]]] = {
"brew": ("brew install ffmpeg", "brew reinstall ffmpeg"),
"apt": ("sudo apt-get install -y ffmpeg", "sudo apt-get install -y --reinstall ffmpeg"),
},
# The official installer is the primary path; npm is a real alternative
# (the package is published as @xai-official/grok) and fits the existing
# manager-preference machinery.
"grok": {
"npm": (
"npm install -g @xai-official/grok",
"reinstall the Grok CLI: npm install -g @xai-official/grok@latest",
),
},
}
# Last-resort prescriptions when no known package manager is detected.
@@ -173,6 +182,10 @@ _FALLBACK_PRESCRIPTIONS: Dict[str, Tuple[str, str]] = {
"install ffmpeg (https://ffmpeg.org/download.html) and ensure it is on PATH",
"reinstall ffmpeg (https://ffmpeg.org/download.html); the current binary won't run",
),
"grok": (
"install the Grok CLI: curl -fsSL https://x.ai/cli/install.sh | bash, then run `grok login`",
"reinstall the Grok CLI: curl -fsSL https://x.ai/cli/install.sh | bash; the current binary won't run",
),
}
+29 -1
View File
@@ -9,6 +9,7 @@ import threading
import time
import urllib.error
import urllib.request
from concurrent.futures import Future
from contextlib import contextmanager
from contextvars import ContextVar, copy_context
from pathlib import Path
@@ -432,6 +433,27 @@ def capture_failures():
_failure_sink.reset(token)
@contextmanager
def tee_failures():
"""Observe failures locally WITHOUT hiding them from the enclosing sink.
``capture_failures()`` *replaces* the context-local sink, so nesting it
inside a retrieval context swallows the very failure the pipeline needs.
This yields a local list and forwards its contents to the parent sink on
exit, so a swallow site (``get_text`` returns None and drops the status)
can recover what it lost while the pipeline still sees the failure.
"""
parent = _failure_sink.get()
local: list[HTTPError] = []
token = _failure_sink.set(local)
try:
yield local
finally:
_failure_sink.reset(token)
if parent is not None:
parent.extend(local)
@contextmanager
def expected_misses(*status_codes: int):
"""Exclude adapter-declared probe misses from captured run failures."""
@@ -444,7 +466,7 @@ def expected_misses(*status_codes: int):
_expected_miss_statuses.reset(token)
def submit_with_context(executor, func, /, *args, **kwargs):
def submit_with_context(executor, func, /, *args, **kwargs) -> Future:
"""Submit a worker with the caller's failure-capture context."""
context = copy_context()
return executor.submit(context.run, func, *args, **kwargs)
@@ -483,6 +505,12 @@ def classify_failure(*, status_code: Optional[int] = None, message: str = "") ->
"forbidden",
"authentication failed",
"expired token",
"not signed in",
"not logged in",
"invalid_grant",
"refresh token",
"session expired",
"grok session expired",
)
):
return health.AUTH_FAILED
+161 -25
View File
@@ -40,14 +40,24 @@ def normalize_source_items(
"reddit": _normalize_reddit,
"x": _normalize_x,
"youtube": _normalize_youtube,
"tiktok": lambda s, i, idx, fd, td: _normalize_shortform_video(s, i, idx, fd, td, "TK", "TikTok post"),
"instagram": lambda s, i, idx, fd, td: _normalize_shortform_video(s, i, idx, fd, td, "IG", "Instagram reel"),
"tiktok": lambda s, i, idx, fd, td: _normalize_shortform_video(
s, i, idx, fd, td, "TK", "TikTok post"
),
"instagram": lambda s, i, idx, fd, td: _normalize_shortform_video(
s, i, idx, fd, td, "IG", "Instagram reel"
),
"hackernews": _normalize_hackernews,
"stocktwits": _normalize_stocktwits,
"dripstack": _normalize_dripstack,
"bluesky": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "BS", "Bluesky post"),
"truthsocial": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "TS", "Truth Social post"),
"threads": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "TH", "Threads post"),
"bluesky": lambda s, i, idx, fd, td: _normalize_microblog(
s, i, idx, fd, td, "BS", "Bluesky post"
),
"truthsocial": lambda s, i, idx, fd, td: _normalize_microblog(
s, i, idx, fd, td, "TS", "Truth Social post"
),
"threads": lambda s, i, idx, fd, td: _normalize_microblog(
s, i, idx, fd, td, "TH", "Threads post"
),
"xquik": _normalize_x,
"pinterest": _normalize_pinterest,
"polymarket": _normalize_polymarket,
@@ -55,6 +65,7 @@ def normalize_source_items(
"arxiv": _normalize_arxiv,
"techmeme": _normalize_techmeme,
"trustpilot": _normalize_trustpilot,
"amazon": _normalize_amazon,
"grounding": _normalize_grounding,
"xiaohongshu": _normalize_grounding,
"github": _normalize_github,
@@ -65,7 +76,10 @@ def normalize_source_items(
normalizer = normalizers.get(source)
if normalizer is None:
raise ValueError(f"Unsupported source: {source}")
normalized = [normalizer(source, item, index, from_date, to_date) for index, item in enumerate(items)]
normalized = [
normalizer(source, item, index, from_date, to_date)
for index, item in enumerate(items)
]
if source == "jobs":
# A careers board is a snapshot of CURRENTLY OPEN roles. An open posting
# is current evidence regardless of when it was posted, so date-windowing
@@ -74,7 +88,9 @@ def normalize_source_items(
# Keep the full board; recency is annotated, not used to drop.
return normalized
require_date = source == "grounding"
filtered = filter_by_date_range(normalized, from_date, to_date, require_date=require_date)
filtered = filter_by_date_range(
normalized, from_date, to_date, require_date=require_date
)
if filtered:
return filtered
if freshness_mode == "evergreen_ok" and source == "youtube":
@@ -88,6 +104,8 @@ def _remap_comments(
raw: list[Any],
score_keys: tuple[str, ...],
excerpt_keys: tuple[str, ...],
*,
preserve_absent_score: bool = False,
) -> list[dict[str, Any]]:
"""Normalize comments from any source into the shared Reddit-compatible shape.
@@ -95,19 +113,29 @@ def _remap_comments(
entity_extract, rerank) all expect `score` and `excerpt`. This helper maps
per-source field names (YT: likes/text, TikTok: digg_count/text) onto that
shape while preserving author/date/url passthrough.
Sources that distinguish an absent vote from a measured zero can opt into
preserving the absent value as ``None``.
"""
out: list[dict[str, Any]] = []
for raw_c in raw:
if not isinstance(raw_c, dict):
continue
score = _first_present(raw_c, score_keys, default=0)
score = _first_present(
raw_c,
score_keys,
default=None if preserve_absent_score else 0,
)
excerpt = _first_present(raw_c, excerpt_keys, default="")
try:
score_int = int(score or 0)
except (TypeError, ValueError):
score_int = 0
if score is None and preserve_absent_score:
normalized_score = None
else:
try:
normalized_score = int(score or 0)
except (TypeError, ValueError):
normalized_score = 0
entry: dict[str, Any] = {
"score": score_int,
"score": normalized_score,
"excerpt": str(excerpt or "")[:400],
"author": str(raw_c.get("author") or ""),
"date": str(raw_c.get("date") or ""),
@@ -145,7 +173,9 @@ def _domain_from_url(url: str) -> str | None:
return domain or None
def _date_confidence(item: dict[str, Any], from_date: str, to_date: str, default: str = "low") -> str:
def _date_confidence(
item: dict[str, Any], from_date: str, to_date: str, default: str = "low"
) -> str:
if item.get("date_confidence"):
return str(item["date_confidence"])
date_value = item.get("date")
@@ -211,7 +241,7 @@ def _normalize_stocktwits(
relevance_hint=item.get("relevance", 0.7),
why_relevant=str(item.get("why_relevant") or ""),
snippet=str(item.get("snippet") or "")[:400],
metadata=meta, # carries sentiment + symbol-level bull/bear aggregate
metadata=meta, # carries sentiment + symbol-level bull/bear aggregate
)
@@ -234,7 +264,9 @@ def _normalize_dripstack(
item_id=str(item.get("id") or f"DS{index + 1}"),
source=source,
title=str(item.get("title") or ""),
body=str(item.get("body") or "") or str(item.get("snippet") or "") or str(item.get("title") or ""),
body=str(item.get("body") or "")
or str(item.get("snippet") or "")
or str(item.get("title") or ""),
url=str(item.get("url") or ""),
author=str(item.get("author") or "") or None,
container=str(meta.get("publication_slug") or "") or None,
@@ -326,7 +358,9 @@ def _normalize_jobs(
title = str(item.get("title") or "").strip()
department = str(item.get("department") or "").strip()
location = str(item.get("location") or "").strip()
body = "\n".join(part for part in [title, department, location, description] if part)
body = "\n".join(
part for part in [title, department, location, description] if part
)
provider = str(item.get("provider") or "").strip()
return _source_item(
item_id=str(item.get("id") or f"J{index + 1}"),
@@ -345,12 +379,15 @@ def _normalize_jobs(
metadata={
"provider": provider,
"department": department,
"departments": item.get("departments") or ([department] if department else []),
"departments": item.get("departments")
or ([department] if department else []),
"location": location,
"offices": item.get("offices") or [],
"board_token": item.get("board_token") or "",
"source_url": item.get("source_url") or "",
"source_domain": item.get("source_domain") or _domain_from_url(str(item.get("url") or "")) or "",
"source_domain": item.get("source_domain")
or _domain_from_url(str(item.get("url") or ""))
or "",
},
)
@@ -471,10 +508,23 @@ def _normalize_hackernews(
from_date: str,
to_date: str,
) -> schema.SourceItem:
top_comments = item.get("top_comments") or []
comment_text = _join_comment_excerpts(top_comments, "text")
# HN comments arrive as {author, text, points}; downstream code keys on
# score/excerpt, so remap here exactly as the YouTube and TikTok normalisers
# do. Without this the per-source floor in render._top_comments_list reads a
# `score` that is never present and rejects every HN comment.
top_comments = _remap_comments(
item.get("top_comments") or [],
score_keys=("points", "score"),
excerpt_keys=("text", "excerpt"),
preserve_absent_score=True,
)
comment_text = _join_comment_excerpts(top_comments, "excerpt")
title = str(item.get("title") or "").strip()
body = "\n".join(part for part in [title, str(item.get("text") or "").strip(), comment_text] if part)
body = "\n".join(
part
for part in [title, str(item.get("text") or "").strip(), comment_text]
if part
)
return _source_item(
item_id=str(item.get("id") or f"HN{index + 1}"),
source=source,
@@ -666,7 +716,8 @@ def _normalize_trustpilot(
return _source_item(
item_id=str(item.get("id") or f"TP{index + 1}"),
source=source,
title=title or (f"{name} on Trustpilot" if name else f"Trustpilot reviews {index + 1}"),
title=title
or (f"{name} on Trustpilot" if name else f"Trustpilot reviews {index + 1}"),
body=body,
url=str(item.get("url") or ""),
author=name or None,
@@ -686,6 +737,87 @@ def _normalize_trustpilot(
)
def _normalize_amazon(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
"""Normalizer for Amazon product-and-review signals.
One item per product. The aggregate rating is current-state evidence, so
the item is stamped with today's date on the Trustpilot precedent -- a
live 4.4-star average is a fact about now, not about whenever the
product launched.
Reviews arrive already in the shared score/excerpt comment shape (built
in the amazon adapter, deliberately not routed through _remap_comments,
which would strip the rating/date/verified keys this source needs), so
they pass straight through to metadata.
"""
name = str(item.get("name") or "").strip()
brand = str(item.get("brand") or "").strip()
top_comments = item.get("top_comments") or []
comment_text = _join_comment_excerpts(top_comments, "excerpt")
rating = item.get("product_rating") if item.get("product_rating") is not None else item.get("rating")
ratings_total = item.get("product_rating_count") or item.get("num_ratings") or 0
headline = " ".join(
part for part in [
f"{rating}/5" if rating is not None else "",
f"({ratings_total:,} ratings)" if ratings_total else "",
] if part
)
# The brand rides in its own field and is usually absent from the name,
# so prepend it -- unless the name already leads with it, which would
# otherwise read "Weber Weber Spirit E-325".
if brand and not name.lower().startswith(brand.lower()):
product_label = f"{brand} {name}".strip()
else:
product_label = name or brand
title = " - ".join(part for part in [product_label, headline] if part)
body = "\n".join(part for part in [title, comment_text] if part)
return _source_item(
item_id=str(item.get("asin") or f"AMZ{index + 1}"),
source=source,
title=title or f"Amazon product {index + 1}",
body=body,
url=str(item.get("url") or ""),
author=brand or None,
container="Amazon",
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="low"),
engagement=item.get("engagement") or {"ratings": ratings_total},
relevance_hint=item.get("relevance", 0.6),
why_relevant=str(item.get("why_relevant") or ""),
snippet=comment_text[:400],
metadata={
"asin": str(item.get("asin") or ""),
"name": name,
"short_name": item.get("short_name") or "",
"brand": brand,
"rating": item.get("rating"),
"num_ratings": item.get("num_ratings") or 0,
"price": item.get("price"),
"currency": item.get("currency") or "",
"badge": item.get("badge") or "",
# Recorded, never used as a filter: the flag's distribution
# swings with keyword phrasing, so filtering can blank the lane.
"sponsored": bool(item.get("sponsored")),
"top_comments": top_comments,
"product_rating": item.get("product_rating"),
"product_rating_count": item.get("product_rating_count") or 0,
"star_distribution": item.get("star_distribution") or {},
# Relevant by construction: the adapter already gated products
# against the model-supplied keyword, and review text rarely
# names the product (KTD8).
"grounding_exempt": True,
},
)
def _normalize_polymarket(
source: str,
item: dict[str, Any],
@@ -703,7 +835,11 @@ def _normalize_polymarket(
item_id=str(item.get("event_id") or item.get("id") or f"PM{index + 1}"),
source=source,
title=title or question or f"Polymarket event {index + 1}",
body="\n".join(part for part in [title, question, str(item.get("price_movement") or "")] if part),
body="\n".join(
part
for part in [title, question, str(item.get("price_movement") or "")]
if part
),
url=str(item.get("url") or ""),
author=None,
container="Polymarket",
@@ -723,7 +859,6 @@ def _normalize_polymarket(
)
def _normalize_github(
source: str,
item: dict[str, Any],
@@ -759,6 +894,7 @@ def _normalize_github(
},
)
def _normalize_grounding(
source: str,
item: dict[str, Any],
File diff suppressed because it is too large Load Diff
+21 -2
View File
@@ -150,6 +150,7 @@ SOURCE_CAPABILITIES = {
"arxiv": {"reference", "analysis", "link"},
"techmeme": {"discussion", "link", "reference"},
"trustpilot": {"reference", "company_signal", "social"},
"amazon": {"reference", "company_signal", "product_signal"},
"xiaohongshu": {"video", "video_shortform", "social"},
"github": {"discussion", "link"},
"grounding": {"web", "reference", "link"},
@@ -835,8 +836,26 @@ def _keyword_query(topic: str, core: str) -> str:
term for term in compounds
if re.match(r"^(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+$", term)
]
quoted = " ".join(f'"{term}"' for term in title_cased[:2])
keywords = [quoted.strip(), core.strip() or topic.strip()]
selected = title_cased[:2]
quoted = " ".join(f'"{term}"' for term in selected)
remainder = core.strip() or topic.strip()
# Drop words already carried by a quoted phrase. Emitting both produced
# '"Peter Steinberger" peter steinberger steipete', which reads to a
# provider as the phrase AND each of its words again -- strictly narrower
# than the phrase alone, and on X it degraded to a bare token conjunction
# once the quotes were stripped downstream. Distinct tokens (here
# "steipete") are preserved.
if selected and remainder:
phrase_words = {
word.lower()
for term in selected
for word in term.split()
}
remainder = " ".join(
word for word in remainder.split()
if word.strip('"').lower() not in phrase_words
)
keywords = [quoted.strip(), remainder.strip()]
return " ".join(part for part in keywords if part).strip()
@@ -199,6 +199,31 @@ def _domain_word_fallback_allows(core_words: list[str], informative: list[str],
return False
def _acronym_credit(core_words: list[str], title_words: set[str]) -> int:
"""Credit matches when the title abbreviates a phrase the topic spells out.
Prediction-market titles use shorthand ("AGI by 2030?") while topics arrive
spelled out ("artificial general intelligence"), so word overlap scores zero
on a title that is squarely on topic. For each run of 3+ consecutive
informative words, build its initialism and, if the title carries it as a
whole word, credit one match per abbreviated word. Requiring at least three
letters avoids treating ambiguous tokens such as "ML" as expanded phrases.
"""
informative_set = set(_informative_words(core_words))
credit = 0
run: list[str] = []
for word in core_words + [""]:
if word in informative_set:
run.append(word)
continue
if len(run) >= 3:
acronym = "".join(w[0] for w in run)
if len(acronym) >= 3 and acronym in title_words:
credit = max(credit, len(run))
run = []
return credit
def _passes_topic_filter(topic: str, event_title: str) -> bool:
"""Check if event title contains enough informative words from the topic.
@@ -236,6 +261,12 @@ def _passes_topic_filter(topic: str, event_title: str) -> bool:
if len(word) >= 4 and word in title_lower:
match_count += 1
# A title that abbreviates what the topic spells out ("AGI" for
# "artificial general intelligence") scores zero above; credit it here.
if match_count < 2:
match_count = max(match_count,
_acronym_credit(core_words, title_words))
# For topics with 3+ informative words, require at least 2 matches.
# This prevents single-word false positives like "mill" in "Meek Mill"
# when the topic is "Mill.com food recycler" (3 informative words).
@@ -594,6 +625,14 @@ def _compute_text_similarity(topic: str, title: str, outcomes: List[str] = None)
if core in title_lower:
return 1.0
# Same match, abbreviated: "AGI" standing in for an informative phrase.
# Use the filter's matcher so modifiers and minimum acronym length cannot
# produce different decisions at the filtering and scoring stages.
core_words = [w for w in re.sub(r"[^\w\s]", " ", core).split() if len(w) > 1]
title_words = set(re.sub(r"[^\w\s]", " ", title_lower).split())
if _acronym_credit(core_words, title_words):
return 1.0
query_type = _infer_query_intent(topic)
title_score = token_overlap_relevance(core, title)
best_score = title_score
@@ -90,6 +90,23 @@ REGISTRY: Dict[Tuple[str, str], Prescription] = dict((
fix_cli=SETUP_BROWSER_COOKIES_CLI,
anchor="api-keys-env",
),
_entry(
"x", "grok_cli_missing",
cause="the Grok CLI is not installed, so the keyless X path is unavailable",
fix_nl=(
"install the Grok CLI (curl -fsSL https://x.ai/cli/install.sh | bash) "
"and sign in with `grok login` to search X without any X credential"
),
fix_cli="npm install -g @xai-official/grok",
anchor="api-keys-env",
),
_entry(
"x", "grok_not_authenticated",
cause="the Grok CLI is installed but not signed in",
fix_nl="sign in to Grok once; no X account or API key is needed after that",
fix_cli="grok login",
anchor="api-keys-env",
),
_entry(
"scrapecreators", "key_missing",
cause="SCRAPECREATORS_API_KEY is not set",
+6 -3
View File
@@ -315,9 +315,12 @@ def resolve_runtime(config: dict[str, Any], depth: str) -> tuple[schema.Provider
def _resolve_x_backend(config: dict[str, Any]) -> str | None:
preferred = (config.get(env.X_BACKEND_PIN_VAR) or "").lower()
if preferred in {"xai", "bird"}:
return preferred
"""Resolve the X backend for runtime fetch.
Delegates to env.get_x_source which handles:
- Any known pin (X_BACKEND_KNOWN) exclusively: returns pin if available, None otherwise
- Unpinned: walks auto-chain (X_BACKEND_ORDER) only, never auto-selects opt-in backends
"""
return env.get_x_source(config)
+17 -1
View File
@@ -301,9 +301,25 @@ def _build_nudge_text(
free_suggestions.append(f"X/Twitter errored - {x_fix.fix_nl}.")
else:
x_fix = prescriptions.get("x", "cookies_missing")
# Pick by state: telling a user who already installed grok to
# install it again is the stale-shim reading the health layer
# exists to avoid. Mirrors _probe_grok's three-way split.
from . import grok_x as _grok_x
grok_key = (
"grok_not_authenticated"
if _grok_x.binary_path() and not _grok_x.has_stored_auth()
else "grok_cli_missing"
)
grok_fix = prescriptions.get("x", grok_key)
# Deliberately not described as free: grok needs no X credential,
# but it does need an installed, signed-in grok CLI drawing on a
# Grok plan. This block is headed "Free suggestions", so the
# precondition has to be stated inline rather than inherited.
free_suggestions.append(
"X/Twitter: real-time posts with likes and reposts - the fastest "
f"signal for breaking topics. Three options: {x_fix.fix_nl}."
"signal for breaking topics. Easiest path if you have a Grok "
f"account: {grok_fix.fix_nl} (no X credential at all). "
f"Otherwise: {x_fix.fix_nl}."
)
if "youtube" in core_missing:
+142 -1
View File
@@ -16,16 +16,32 @@ count rather than failing the Reddit source.
import sys
import time
from typing import Dict, List
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from . import http
API = "https://arctic-shift.photon-reddit.com/api/posts/ids"
SEARCH_API = "https://arctic-shift.photon-reddit.com/api/posts/search"
BATCH = 50 # ids per request
TIMEOUT = 15
MAX_BATCHES = 3 # cap total requests per run (bounds latency + rate-limit risk)
PACE_SECONDS = 0.4 # gap between batches; arctic-shift answers 422 "slow down"
CACHE_MAX = 4096 # hard size bound so the in-run memo can never grow unbounded
# Listing-lane knobs. Base limits mirror reddit_listing's DEPTH_LIMITS so callers
# get the same per-depth volume. The supplement multiplier is applied when the
# caller requested multiple sorts (top/hot/new) — arctic-shift has no sort lanes,
# so we fetch more posts to increase the chance of covering what the failed
# shreddit lanes would have returned.
#
# KNOWN LIMITATION: Arctic-shift is recency-only (sort=desc). It has no top/hot/
# new/rising lanes — failed shreddit sort lanes are supplemented with recent
# posts, not lane-specific results. This is a fundamental backend constraint.
_LISTING_DEPTH_LIMITS = {"quick": 10, "default": 25, "deep": 50}
_LISTING_SUPPLEMENT_MULTIPLIER = 2 # fetch 2x posts when supplementing multi-sort requests
# Total deadline for listing fetches to prevent unbounded stalls when many
# subreddits are requested and arctic is slow/unreachable.
_LISTING_DEADLINE_SECONDS = 45 # ~3 subs at 15s timeout each
# In-run memo: base36 id -> {score, num_comments}. Module-level so repeated
# fetch_scores calls within one `/last30days` run (e.g. across subqueries) reuse
# results, but capped at CACHE_MAX entries (never reached in a normal CLI run).
@@ -90,3 +106,128 @@ def fetch_scores(post_ids: List[str]) -> Dict[str, Dict[str, int]]:
_cache[rid] = entry
out[rid] = entry
return out
def _epoch_to_date(value: Any) -> Optional[str]:
"""Epoch seconds -> YYYY-MM-DD (UTC), or None on garbage."""
try:
return datetime.fromtimestamp(int(value), tz=timezone.utc).date().isoformat()
except (TypeError, ValueError, OSError):
return None
def _normalize_listing_row(row: Dict[str, Any], query: str = "") -> Dict[str, Any]:
"""Normalize an arctic-shift post row to reddit_listing.parse_cards shape.
Mirrors the shreddit card schema (title/url/score/num_comments/subreddit/
created_utc/author/selftext/date/engagement/relevance/metadata.post_id) so
reddit_keyless can consume either backend interchangeably.
"""
from .relevance import token_overlap_relevance
pid = str(row.get("id") or "").removeprefix("t3_")
permalink = row.get("permalink") or ""
title = row.get("title") or ""
try:
score = int(row.get("score") or 0)
except (TypeError, ValueError):
score = 0
try:
num_comments = int(row.get("num_comments") or 0)
except (TypeError, ValueError):
num_comments = 0
author = row.get("author") or "[deleted]"
if author in ("[deleted]", "[removed]"):
author = "[deleted]"
url = f"https://www.reddit.com{permalink}" if permalink.startswith("/") else (permalink or "")
return {
"id": "",
"title": title,
"url": url,
"score": score,
"num_comments": num_comments,
"subreddit": row.get("subreddit") or "",
"created_utc": row.get("created_utc"),
"author": author,
"selftext": row.get("selftext") or "",
"date": _epoch_to_date(row.get("created_utc")),
"engagement": {"score": score, "num_comments": num_comments, "upvote_ratio": None},
"relevance": round(token_overlap_relevance(query, title), 3) if query else 0.0,
"why_relevant": "Reddit listing (arctic-shift)",
"metadata": {"post_id": pid},
}
def fetch_listings(
subreddits: List[str],
depth: str = "default",
query: str = "",
sorts: Optional[List[str]] = None,
timeframe: str = "month",
limit: Optional[int] = None,
) -> List[Dict[str, Any]]:
"""Scored subreddit listings from the arctic-shift archive, keyless.
Drop-in fallback/supplement for ``reddit_listing.fetch_listings`` (shreddit
partials), which datacenter IPs get HTTP 403 on. Arctic-shift serves recent
posts with real score/num_comments from any IP.
Arctic-shift has no top/hot/new lanes, only recency. When ``sorts`` contains
multiple entries (e.g., dedicated lanes requesting top+hot+new), we fetch
more posts per subreddit to partially compensate for the missing lane
coverage the caller's engagement ranking does the final sorting.
Best-effort, never raises: returns ``[]`` on any failure.
"""
if not subreddits:
return []
base = limit or _LISTING_DEPTH_LIMITS.get(depth, _LISTING_DEPTH_LIMITS["default"])
# When multiple sorts were requested, fetch more posts to compensate for
# arctic-shift's lack of sort lanes.
n = base * _LISTING_SUPPLEMENT_MULTIPLIER if sorts and len(sorts) > 1 else base
out: List[Dict[str, Any]] = []
# Process all requested subreddits with pacing and a total deadline to
# prevent unbounded stalls when arctic is slow or unreachable.
deadline = time.time() + _LISTING_DEADLINE_SECONDS
fetched_count = 0
for sub in subreddits:
if time.time() >= deadline:
_log(f"listing deadline reached after {fetched_count} subs; skipping remaining")
break
sub = sub.removeprefix("r/").strip()
if not sub or sub.lower() == "all":
continue
if fetched_count:
time.sleep(PACE_SECONDS)
fetched_count += 1
try:
# Use retries=1 (single attempt) so retries don't exceed our deadline.
# The deadline handles overall timing; per-request retries would
# multiply the delay unpredictably.
data = http.get(
f"{SEARCH_API}?subreddit={sub}&limit={n}&sort=desc",
headers={"User-Agent": http.BROWSER_USER_AGENT},
timeout=TIMEOUT,
retries=1,
)
except Exception as e: # network error / non-200 — degrade, never raise
_log(f"listing search failed r/{sub}: {e}")
continue
rows = (data or {}).get("data")
if not isinstance(rows, list):
_log(f"unexpected listing response for r/{sub}: {str(data)[:80]}")
continue
for row in rows:
if not isinstance(row, dict):
continue
post = _normalize_listing_row(row, query)
if post["url"]:
out.append(post)
seen: set = set()
unique: List[Dict[str, Any]] = []
for p in out:
if p["url"] not in seen:
seen.add(p["url"])
unique.append(p)
return unique
@@ -72,6 +72,51 @@ def _apply_scores(post: Dict[str, Any], scored: Dict[str, int]) -> None:
post["engagement"]["num_comments"] = scored["num_comments"]
def _scored_listings(
subreddits: List[str],
depth: str = "default",
query: str = "",
sorts: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
"""Scored subreddit listings: shreddit partials, arctic-shift supplement.
The shreddit ``community-more-posts`` partials 403 from datacenter IPs
(and any host Reddit decides to block). Shreddit is tried first; arctic-
shift supplements with any posts shreddit missed. Individual sort lanes
can fail silently (shreddit's ``fetch_listings`` flattens results without
exposing per-sort status), so arctic is called for all requested subreddits
and merged via deduplication. This ensures fresh posts sought through
``hot`` or ``new`` are recovered even when only ``top`` succeeded. Never
raises.
"""
posts = reddit_listing.fetch_listings(subreddits, depth=depth, query=query, sorts=sorts)
# Supplement with arctic for all requested subreddits. Shreddit's per-sort
# success/failure is opaque, so arctic provides coverage for any failed
# sort lanes (e.g., hot/new failing while top succeeded). Deduplication
# ensures no redundant posts when shreddit fully succeeded.
if subreddits:
try:
arctic_posts = reddit_arctic.fetch_listings(
subreddits, depth=depth, query=query, sorts=sorts
)
except Exception as exc: # the fallback must never break the pipeline
_log(f"arctic-shift listing supplement failed: {exc}")
arctic_posts = []
if arctic_posts:
# Merge and dedupe by URL — shreddit posts take priority.
seen = {p["url"] for p in posts}
added = 0
for p in arctic_posts:
if p["url"] not in seen:
seen.add(p["url"])
posts.append(p)
added += 1
if added:
_log(f"arctic-shift supplement: {added} new posts from {len(arctic_posts)} arctic results")
return posts
def _discover(
topic: str,
depth: str,
@@ -83,7 +128,7 @@ def _discover(
# an on-topic post whose title lacks the entity name is never dropped.
dedicated_posts: List[Dict[str, Any]] = []
if dedicated_subreddits:
dedicated_posts = reddit_listing.fetch_listings(
dedicated_posts = _scored_listings(
dedicated_subreddits, depth=depth, query=topic, sorts=DEDICATED_SORTS
)
for p in dedicated_posts:
@@ -98,7 +143,7 @@ def _discover(
if subreddits:
# Targeted run: the caller chose these subreddits, so their listing cards
# are on-topic — include them as scored discovery AND as a score source.
listing_posts = reddit_listing.fetch_listings(subreddits, depth=depth, query=topic)
listing_posts = _scored_listings(subreddits, depth=depth, query=topic)
score_source = listing_posts
else:
# Bare global run: subreddits derived from noisy RSS results are NOT
@@ -107,7 +152,7 @@ def _discover(
# would flood results with high-upvote but irrelevant posts.
listing_posts = []
derived = _top_subreddits(rss_posts)
score_source = reddit_listing.fetch_listings(derived, depth=depth, query=topic)
score_source = _scored_listings(derived, depth=depth, query=topic)
_log(
f"Tier 1 (RSS) {len(rss_posts)} posts; "
f"{'listing discovery ' + str(len(listing_posts)) if subreddits else 'score-only'}; "
+119 -8
View File
@@ -18,10 +18,16 @@ import re
import sys
from datetime import datetime, timezone
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Optional, Set
from . import http
from .relevance import token_overlap_relevance
from .relevance import token_overlap_relevance, tokenize
# Generic domain terms that are excluded from the keyword gate — matches
# pipeline._DISCOVERY_GENERIC_DOMAIN_TERMS (duplicated to avoid circular import).
_DISCOVERY_GENERIC_DOMAIN_TERMS: Set[str] = {
"ai", "artificial", "intelligence", "tech", "technology", "trending", "trend",
}
# Listing sorts pulled per subreddit, by depth.
LISTING_SORTS = {
@@ -42,6 +48,26 @@ def _log(msg: str) -> None:
sys.stderr.flush()
def _matches_discovery_domain(domain: str, text: str) -> bool:
"""Require a distinctive domain term, not a generic token such as ``AI``.
Duplicated from pipeline._matches_discovery_domain to avoid circular imports.
The rule must stay in sync: pipeline.py owns the authoritative version and
test_reddit_listing.py verifies parity.
"""
def terms(value: str) -> Set[str]:
words: Set[str] = set()
for word in tokenize(value):
words.add(word)
if len(word) > 4 and word.endswith("s") and not word.endswith("ss"):
words.add(word[:-1])
return words
domain_terms = terms(domain)
anchors = domain_terms - _DISCOVERY_GENERIC_DOMAIN_TERMS
return bool((anchors or domain_terms) & terms(text))
def _attr(tag: str, name: str) -> Optional[str]:
m = re.search(rf'\b{name}="([^"]*)"', tag)
return _html.unescape(m.group(1)) if m else None
@@ -73,6 +99,21 @@ def _post_id(permalink: str) -> str:
return m.group(1) if m else ""
_ERROR_PATTERN = re.compile(r"^r/(\S+)\s+(\S+):", re.IGNORECASE)
def _shreddit_error_recovered(error: str, successes: Set[tuple[str, str]]) -> bool:
"""Return True if the error's (sub, sort) pair is in the successes set.
Error format: "r/{sub} {sort}: {message}".
"""
m = _ERROR_PATTERN.match(error)
if not m:
return False
sub, sort = m.group(1).lower(), m.group(2).lower()
return (sub, sort) in successes
def parse_cards(html_text: str, query: str = "") -> List[Dict[str, Any]]:
"""Parse <shreddit-post> cards into normalized post dicts with real scores."""
posts: List[Dict[str, Any]] = []
@@ -148,9 +189,17 @@ def _fetch_one_with_status(
timeframe: str = TIMEFRAME,
) -> tuple[List[Dict[str, Any]], Optional[str]]:
try:
text = http.reddit_keyless_get_text(_listing_url(subreddit, sort, timeframe), timeout=LISTING_TIMEOUT,
accept="text/html")
return (parse_cards(text, query) if text else []), None
# tee_failures, not capture_failures: the latter would replace the
# pipeline's sink and hide this failure from it. get_text launders a
# terminal HTTP failure into None, so the tee is how this lane recovers
# the status code it needs to report (issue #899).
with http.tee_failures() as swallowed:
text = http.reddit_keyless_get_text(_listing_url(subreddit, sort, timeframe), timeout=LISTING_TIMEOUT,
accept="text/html")
if text is None:
# An empty body ("") is a real empty listing; None never is.
return [], (str(swallowed[-1]) if swallowed else "no response")
return parse_cards(text, query), None
except Exception as e:
_log(f"listing fetch failed r/{subreddit} {sort}: {e}")
return [], str(e)
@@ -178,7 +227,9 @@ def fetch_listings(
jobs = [(sub, sort) for sub in subreddits for sort in sorts]
all_posts: List[Dict[str, Any]] = []
with ThreadPoolExecutor(max_workers=min(MAX_WORKERS, len(jobs)) or 1) as executor:
futures = {executor.submit(_fetch_one, sub, sort, query, timeframe): (sub, sort)
# submit_with_context, not executor.submit — see the note in
# fetch_discovery_listings below (issue #899).
futures = {http.submit_with_context(executor, _fetch_one, sub, sort, query, timeframe): (sub, sort)
for sub, sort in jobs}
for future in futures:
try:
@@ -201,15 +252,31 @@ def fetch_discovery_listings(
query: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Fetch rising/top-week listings while preserving per-feed failures."""
"""Fetch rising/top-week listings while preserving per-feed failures.
When shreddit fails and arctic-shift recovers, errors are cleared only for
subreddits whose posts survive the keyword gate. If query is empty (global
``--discover`` with no domain), the gate is skipped and any arctic result
counts as recovery.
"""
if not subreddits:
return {"items": [], "errors": []}
jobs = [(subreddit, sort) for subreddit in subreddits for sort in ("rising", "top")]
items: List[Dict[str, Any]] = []
errors: List[str] = []
# Track which (sub, sort) pairs shreddit successfully delivered posts for.
# Used to decide which errors to clear — Arctic can supplement but cannot
# "recover" a failed hot/top/new/rising lane (it's recency-only).
shreddit_successes: Set[tuple[str, str]] = set()
with ThreadPoolExecutor(max_workers=min(MAX_WORKERS, len(jobs)) or 1) as executor:
# submit_with_context, not executor.submit: a plain submit starts the
# worker with an empty context, dropping the pipeline's
# capture_failures() sink so a listing's 429/403 is silently discarded
# and the source reports a clean no-results (issue #899).
futures = {
executor.submit(_fetch_one_with_status, subreddit, sort, query, "week"): (subreddit, sort)
http.submit_with_context(
executor, _fetch_one_with_status, subreddit, sort, query, "week"
): (subreddit, sort)
for subreddit, sort in jobs
}
for future, (subreddit, sort) in futures.items():
@@ -221,6 +288,9 @@ def fetch_discovery_listings(
items.extend(fetched)
if error:
errors.append(f"r/{subreddit} {sort}: {error}")
elif fetched:
# Shreddit succeeded for this (sub, sort) lane.
shreddit_successes.add((subreddit.lower(), sort.lower()))
seen: set[str] = set()
unique = []
@@ -229,6 +299,47 @@ def fetch_discovery_listings(
continue
seen.add(item["url"])
unique.append(item)
# Supplement with arctic-shift for all requested subreddits. Shreddit's
# per-sort success/failure is opaque (individual rising/top lanes can fail
# while others succeed), so arctic provides coverage for any failed lanes.
# Deduplication ensures no redundant posts when shreddit fully succeeded.
from . import reddit_arctic
arctic_items = reddit_arctic.fetch_listings(
subreddits, depth=depth, query=query, sorts=("rising", "top")
)
if arctic_items:
_log(f"discovery arctic supplement: {len(arctic_items)} posts")
# Apply the same keyword gate that pipeline._fetch_discovery_source
# uses downstream. When query is empty (global --discover), skip the
# gate — there's no keyword to match, and the river feed IS the signal.
if query:
arctic_items = [
item for item in arctic_items
if _matches_discovery_domain(
query,
f"{item.get('title') or ''} {item.get('selftext') or ''}",
)
]
# Merge arctic items into unique list, deduping by URL.
added = 0
for item in arctic_items:
if item["url"] not in seen:
seen.add(item["url"])
unique.append(item)
added += 1
if added:
_log(f"discovery arctic supplement added {added} new posts")
# Clear errors only for (sub, sort) pairs where shreddit succeeded.
# Arctic supplements recency posts but cannot "recover" a failed hot/top/
# rising lane — it has no sort lanes. Errors for failed shreddit lanes are
# preserved even when another sort for the same subreddit succeeded.
if errors and shreddit_successes:
errors = [
e for e in errors
if not _shreddit_error_recovered(e, shreddit_successes)
]
return {"items": unique, "errors": errors}
+8 -1
View File
@@ -203,7 +203,14 @@ def search_rss(
all_posts: List[Dict[str, Any]] = []
workers = min(MAX_WORKERS, len(urls)) or 1
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {executor.submit(_fetch_feed, url, query): url for url in urls}
# submit_with_context, not executor.submit: a plain submit starts the
# worker with an empty context, dropping the pipeline's
# capture_failures() sink so a feed's 429/403 is silently discarded and
# the source reports a clean no-results (issue #899).
futures = {
http.submit_with_context(executor, _fetch_feed, url, query): url
for url in urls
}
for future in futures:
try:
all_posts.extend(future.result(timeout=FEED_TIMEOUT + 5))
@@ -59,6 +59,19 @@ SYNONYMS = {
# Generic query words that should not carry relevance on their own.
# They still help when paired with stronger entity/topic matches.
#
# The second group is scaffolding emitted by planner's ranking-query templates
# ("What recent evidence from the last 30 days is most relevant to X?" and its
# siblings). Those words are not the topic, but every one of them was being
# counted as an informative query token, which capped achievable coverage at the
# topic's share of the query and demoted on-topic posts. Kept here rather than
# stripped in the planner so any caller building a similar natural-language
# ranking query gets the same treatment.
#
# Domain nouns from those same templates (production, market, workflows,
# experience, signals, ...) are deliberately absent: they can legitimately be a
# user's topic, and demoting them globally would hurt every source.
# tests/test_ranking_query_scaffolding.py pins that split.
LOW_SIGNAL_QUERY_TOKENS = frozenset({
'advice', 'animation', 'animations', 'best', 'chance', 'chances',
'code', 'compare', 'comparison', 'differences', 'explain', 'guide',
@@ -67,6 +80,10 @@ LOW_SIGNAL_QUERY_TOKENS = frozenset({
'prompting', 'prompts', 'rate', 'review', 'reviews', 'thoughts',
'tip', 'tips', 'tutorial', 'tutorials', 'update', 'updates', 'use',
'using', 'versus', 'vs', 'worth',
# planner ranking-query scaffolding
'30', 'current', 'days', 'describing', 'especially', 'evidence', 'exist',
'follow', 'hands', 'last', 'matter', 'most', 'new', 'people', 'real',
'recent', 'relevant', 'running', 'up', 'world',
})
File diff suppressed because it is too large Load Diff
+76 -1
View File
@@ -26,6 +26,10 @@ ENTITY_MISS_PENALTY = 25.0
FALLBACK_ENTITY_MISS_CONFIDENCE_ESCAPE = 0.5
FALLBACK_ENTITY_MISS_TOPIC_ESCAPE = 0.25
_FALLBACK_ENTITY_MISS_EXPLANATION = "fallback-local-score (entity-miss demotion)"
# Explanation stamped on a first-party post whose entity-miss marker was
# cleared by _apply_first_party_floor. Carries no "entity-miss" substring, so
# every downstream relevance gate treats the post as grounded.
_FIRST_PARTY_EXPLANATION = "first-party post (authored by a resolved handle)"
# Small additive credit for a post authored by one of the run's resolved
# handles (see rerank_candidates / _fallback_tuple). Deliberately small: the
@@ -537,8 +541,18 @@ def _apply_first_party_floor(
if not resolved_handles:
return
for c in candidates:
if _is_first_party(c, resolved_handles) and c.final_score < FIRST_PARTY_FLOOR:
if not _is_first_party(c, resolved_handles):
continue
if c.final_score < FIRST_PARTY_FLOOR:
c.final_score = FIRST_PARTY_FLOOR
# Clear the entity-miss marker here, at the one site that knows the
# resolved handles. Downstream relevance gates key on the marker, not
# on handle knowledge, so neutralizing it once lets the carve-out
# propagate instead of forcing every gate to re-derive first-party.
# A first-party post is entity-grounded by authorship: nobody repeats
# their own name in their own post.
if c.explanation and "entity-miss" in c.explanation.lower():
c.explanation = _FIRST_PARTY_EXPLANATION
def _apply_engagement_rescue(
@@ -642,6 +656,17 @@ def _fallback_tuple(
if resolved_handles and _is_first_party(candidate, resolved_handles):
score += FIRST_PARTY_AUTHOR_CREDIT
return max(0.0, min(100.0, score)), "fallback-local-score (first-party authorship)"
# Grounding-exempt evidence (currently Amazon): the adapter gated these
# against the model-supplied keyword before they existed, so the
# entity-miss demotion below would punish them for a match they were
# never going to make -- a "Weber Grills" run legitimately surfaces a
# product called "Spirit E-325" whose reviews discuss searing, not Weber.
# Returning here also skips _final_score's secondary penalty, which greps
# the reason string for "entity-miss": one flag, both paths, per the
# propagation pattern in
# docs/solutions/logic-errors/entity-grounding-full-phrase-false-demotion.md
if _is_grounding_exempt(candidate):
return max(0.0, min(100.0, score)), "fallback-local-score (grounding-exempt source)"
# Entity-grounding demotion: subtract ENTITY_MISS_PENALTY when the candidate
# never mentions the primary entity's head token, across all text surfaces
# (title, snippet, transcript, transcript highlights, top comments,
@@ -670,6 +695,23 @@ def _primary_entity(topic: str) -> str:
return stripped
def _is_grounding_exempt(candidate: schema.Candidate) -> bool:
"""True when the candidate carries the relevant-by-construction label.
Set by adapters that already gated their results against an explicit
keyword at retrieval time (see normalize._normalize_amazon). Checked on
the candidate's own metadata and on any of its source items, since
clustering can build a candidate from several items.
"""
metadata = candidate.metadata or {}
if isinstance(metadata, dict) and metadata.get("grounding_exempt"):
return True
return any(
isinstance(item.metadata, dict) and item.metadata.get("grounding_exempt")
for item in candidate.source_items
)
def _is_corpus_candidate(candidate: schema.Candidate) -> bool:
"""True when the candidate carries private corpus evidence."""
if candidate.source == "corpus":
@@ -728,6 +770,15 @@ def prune_fallback_entity_misses(
#: the dilute penalty. This backstop makes the demotion actually decisive.
ENTITY_MISS_FINAL_PENALTY = 20.0
#: Multiplier applied to a candidate whose every dated item falls outside the
#: run's window. The tool's whole promise is the window, so a stale item must
#: not lead the ranked clusters however relevant it reads — a 2025-10 video
#: ranked #1 in a 2026-07 brief, and a 2025-12 one ranked #5, both correctly
#: flagged [date:low] and both ranked anyway. Scaling rather than subtracting
#: keeps the ordering *among* older items intact, so the "still worth reading"
#: signal survives underneath the in-window evidence.
OUT_OF_WINDOW_FINAL_MULTIPLIER = 0.35
def _final_score(candidate: schema.Candidate) -> float:
normalized_rrf = _normalized_rrf(candidate.rrf_score)
@@ -752,6 +803,9 @@ def _final_score(candidate: schema.Candidate) -> float:
# at final_score level so engagement signal can't mask the demotion.
if candidate.explanation and "entity-miss" in candidate.explanation:
base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
# Recency contract: out-of-window evidence never leads the ranked output.
if schema.candidate_out_of_window(candidate):
base *= OUT_OF_WINDOW_FINAL_MULTIPLIER
return base
@@ -891,3 +945,24 @@ def _normalized_rrf(rrf_score: float) -> float:
# Max single-stream RRF at rank 1 is 1/(K+1) ~ 0.016; multi-stream
# accumulation reaches ~0.08.
return max(0.0, min(100.0, (rrf_score / 0.08) * 100.0))
def candidate_relevance_ok(candidate: schema.Candidate) -> bool:
"""Shared gate: is this candidate topically usable for display surfaces?
Single owner of the entity-miss demotion test. Render-side surfaces (Best
Takes, cluster visibility) must call this rather than re-testing the
explanation string themselves -- a second copy of the predicate is how the
documented mirrored-predicate drift bug recurs, and it means a carve-out
added here silently fails to reach them.
First-party posts are handled upstream: ``_apply_first_party_floor`` clears
their entity-miss marker at the one site that knows the resolved handles,
so this predicate needs no handle knowledge.
"""
explanation = (candidate.explanation or "").lower()
if "entity-miss" in explanation:
return False
if (candidate.final_score or 0.0) <= 0.0:
return False
return True
+51 -4
View File
@@ -39,7 +39,7 @@ class ProviderRuntime:
reasoning_provider: Literal["gemini", "openai", "xai", "local"]
planner_model: str
rerank_model: str
x_search_backend: Literal["xai", "bird"] | None = None
x_search_backend: Literal["xai", "grok", "bird", "xurl", "xquik"] | None = None
@dataclass(frozen=True)
@@ -357,9 +357,18 @@ class RetrievalBundle:
*,
attempted: bool = True,
) -> None:
"""Record a failure, preserving already-returned items as partial."""
"""Record a failure, preserving already-returned items as partial.
AUTH_FAILED is preserved even when items exist, since the re-login
signal shouldn't be downgraded to generic PARTIAL guidance.
"""
count = len(self.items_by_source.get(source, []))
outcome_state: RunOutcomeState = PARTIAL if count else state
# Preserve AUTH_FAILED even when items exist: it's an actionable signal
# (re-login needed) that shouldn't be downgraded to PARTIAL.
if state == AUTH_FAILED:
outcome_state: RunOutcomeState = AUTH_FAILED
else:
outcome_state = PARTIAL if count else state
self.errors_by_source.setdefault(source, detail)
self.source_status[source] = SourceOutcome(
source=source,
@@ -379,7 +388,13 @@ class RetrievalBundle:
detail = None
fix_hint = None
if previous and previous.state not in (health.OK, NO_RESULTS):
state = PARTIAL if self.items_by_source[source] else previous.state
# Preserve AUTH_FAILED state even when items are added: it's an
# actionable signal (re-login needed) that shouldn't be downgraded
# to PARTIAL. Other failure states become PARTIAL when items exist.
if previous.state == AUTH_FAILED:
state = AUTH_FAILED
else:
state = PARTIAL if self.items_by_source[source] else previous.state
detail = previous.detail
fix_hint = previous.fix_hint
self.source_status[source] = SourceOutcome(
@@ -576,6 +591,38 @@ def candidate_source_label(candidate: Candidate) -> str:
return ", ".join(sources) if sources else "unknown"
def candidate_out_of_window(candidate: Candidate) -> bool:
"""True when every dated item behind this candidate falls outside the window.
Window membership is derived from the actual ``published_at`` date compared
to the run's ``range_from``/``range_to`` (stored in candidate.metadata by
fusion.weighted_rrf). Some adapters provide ``date_confidence="high"`` for
old dates, so relying solely on adapter-provided confidence is insufficient.
Candidates with no dated item at all are not treated as out of window an
unknown date is a coverage gap, not a stale item.
"""
dated = [item for item in candidate.source_items if item.published_at]
if not dated:
return False
range_from = candidate.metadata.get("range_from")
range_to = candidate.metadata.get("range_to")
if range_from and range_to:
try:
start = datetime.fromisoformat(range_from).date()
end = datetime.fromisoformat(range_to).date()
for item in dated:
item_date = datetime.fromisoformat(item.published_at[:10]).date()
if start <= item_date <= end:
return False
return True
except (ValueError, TypeError):
pass
return all(item.date_confidence != "high" for item in dated)
def candidate_best_published_at(candidate: Candidate) -> str | None:
return max(
(item.published_at for item in candidate.source_items if item.published_at),
+180 -37
View File
@@ -17,6 +17,8 @@ from typing import Any, Dict, Optional, Tuple
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from . import brightdata
logger = logging.getLogger(__name__)
@@ -112,11 +114,17 @@ def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = Fals
cookies_found[source_name] = result[1]
break # Found cookies for this service, stop trying browsers
# Check yt-dlp availability and install via Homebrew if missing
# Check yt-dlp availability and install via Homebrew if missing. Windows
# has no Homebrew, and its working install path is `pip install yt-dlp`
# (see #904), so it gets its own no-op-install guidance branch instead of
# falling into the Homebrew-oriented no_homebrew outcome.
ytdlp_action: str
if shutil.which("yt-dlp") is not None:
ytdlp_installed = True
ytdlp_action = "already_installed"
elif os.name == "nt":
ytdlp_installed = False
ytdlp_action = "no_pip_windows"
elif shutil.which("brew") is not None:
brew_stderr = ""
try:
@@ -153,6 +161,12 @@ def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = Fals
# Per-CLI status for the additional default-on Printing Press sources
# (arxiv, techmeme, trustpilot): {source: {installed, action, ...}}.
"pp_sources": pp_sources,
# Reported, never installed: this CLI spends the user's own metered
# credits, so acquiring it stays their decision (U5/R11). Passing
# config matters: a user whose key lives in a .env file or the
# keychain (rather than a `brightdata login` credentials file) is
# active in the engine, and setup must not tell them otherwise.
"brightdata": brightdata_status(config),
"env_written": False,
}
if ytdlp_action == "install_failed":
@@ -226,6 +240,46 @@ def _digg_bin_dir_hint(digg_path: str) -> str:
return parent
def _run_npx_install(slug: str) -> Tuple[str, str]:
"""Resolve ``npx`` and run the Printing Press catalog install for ``slug``.
Shared by ``_install_digg_cli`` and ``_install_pp_cli`` -- this is only the
"resolve npx, run the install, interpret no_npx/exception/nonzero-rc"
slice; each caller keeps its own on-path/off-path re-verification
(``_digg_bin_candidate_paths`` vs ``_pp_bin_candidate_paths`` already use
different candidate-directory sources, so merging them here would change
off-path detection behavior beyond this fix's scope).
Fixes the Windows PATHEXT mismatch: ``shutil.which("npx")`` resolves
``npx.CMD`` via PATHEXT, but ``subprocess.run`` given the bare string
``"npx"`` as argv[0] does not do that resolution and fails with
``WinError 2``. Passing the resolved path is a no-op on macOS/Linux, where
``shutil.which`` already returns the exact path ``CreateProcess``/``execve``
would resolve.
Returns ``(action, stderr)``: ``action`` is ``"no_npx"``,
``"install_failed"``, or ``""`` when the subprocess ran and returned
``rc=0`` (in which case ``stderr`` carries any non-fatal stderr output for
the caller's own off-path logging).
"""
npx = shutil.which("npx")
if npx is None:
return "no_npx", ""
try:
proc = subprocess.run(
[npx, "-y", PRINTING_PRESS_NPM, "install", slug, "--cli-only"],
capture_output=True, text=True, timeout=DIGG_INSTALL_TIMEOUT,
)
except Exception as exc:
logger.warning("npx install %s exception: %s", slug, exc)
return "install_failed", str(exc)
if proc.returncode != 0:
stderr = proc.stderr or f"npx install {slug} exited {proc.returncode}"
logger.warning("npx install %s failed (rc=%s): %s", slug, proc.returncode, stderr)
return "install_failed", stderr
return "", (proc.stderr or "")
def _install_digg_cli() -> Tuple[bool, str, str, str]:
"""Best-effort install of the digg-pp-cli binary.
@@ -247,32 +301,21 @@ def _install_digg_cli() -> Tuple[bool, str, str, str]:
off_path = _digg_off_path_binary()
if off_path:
return False, "installed_off_path", "", off_path
if shutil.which("npx") is None:
return False, "no_npx", "", ""
try:
proc = subprocess.run(
["npx", "-y", PRINTING_PRESS_NPM, "install", "digg", "--cli-only"],
capture_output=True, text=True, timeout=DIGG_INSTALL_TIMEOUT,
)
except Exception as exc:
logger.warning("npx install digg exception: %s", exc)
return False, "install_failed", str(exc), ""
if proc.returncode != 0:
stderr = proc.stderr or f"npx install digg exited {proc.returncode}"
logger.warning("npx install digg failed (rc=%s): %s", proc.returncode, stderr)
return False, "install_failed", stderr, ""
action, stderr = _run_npx_install("digg")
if action:
return False, action, stderr, ""
on_path = _digg_on_path()
if on_path:
return True, "installed", "", ""
off_path = _digg_off_path_binary()
if off_path:
combined = (proc.stderr or "").strip()
combined = stderr.strip()
if combined:
logger.warning("digg-pp-cli installed off PATH: %s", combined)
return False, "installed_off_path", combined, off_path
stderr = proc.stderr or "install completed but digg-pp-cli was not found"
logger.warning("npx install digg failed verification: %s", stderr)
return False, "install_failed", stderr, ""
stderr_msg = stderr or "install completed but digg-pp-cli was not found"
logger.warning("npx install digg failed verification: %s", stderr_msg)
return False, "install_failed", stderr_msg, ""
# Additional default-on Printing Press sources installed the same way as Digg:
@@ -287,6 +330,85 @@ PP_DEFAULT_SOURCES: list[tuple[str, str, str]] = [
("techmeme", "techmeme", "techmeme-pp-cli"),
]
# Bright Data is deliberately absent from PP_DEFAULT_SOURCES: it is not a
# Printing Press CLI, it is opt-in like Trustpilot, and it spends the user's
# own metered credits. Setup reports its state and never installs it.
BRIGHTDATA_BIN = "brightdata"
def _brightdata_off_path_binary() -> Optional[str]:
"""Locate a brightdata binary that exists on disk but not on PATH.
Covers the common npm global prefixes. The distinction matters because
Hermes and OpenClaw gateways routinely run the engine with a PATH that
excludes the user's npm bin directory, so "installed" and "the engine
can see it" are different questions.
"""
home = Path.home()
candidates = [
home / ".local" / "bin" / BRIGHTDATA_BIN,
home / ".npm-global" / "bin" / BRIGHTDATA_BIN,
Path("/opt/homebrew/bin") / BRIGHTDATA_BIN,
Path("/usr/local/bin") / BRIGHTDATA_BIN,
]
npm_prefix = os.environ.get("NPM_CONFIG_PREFIX")
if npm_prefix:
candidates.insert(0, Path(npm_prefix) / "bin" / BRIGHTDATA_BIN)
for candidate in candidates:
try:
if candidate.is_file() and os.access(candidate, os.X_OK):
return str(candidate)
except OSError:
continue
return None
def brightdata_status(config: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
"""Report the Bright Data install and auth state honestly.
Deliberately never claims the source is active unless the engine's own
gate would pass -- ``brightdata.is_available`` is the single predicate,
so setup and the engine cannot drift apart. Three states matter:
* ``already_installed`` -- on PATH; ``authenticated`` says whether the
amazon lane will actually run.
* ``installed_off_path`` -- on disk but invisible to the engine, which
is the Hermes/OpenClaw failure mode. Carries the path so the user can
fix their PATH.
* ``not_installed`` -- nothing found. No auto-install: this CLI
spends the user's metered credits, so acquiring it is their call.
"""
installed = brightdata.is_installed()
authenticated = brightdata.has_credentials(config)
if installed:
action = "already_installed"
off_path = ""
else:
off_path = _brightdata_off_path_binary() or ""
action = "installed_off_path" if off_path else "not_installed"
status: Dict[str, Any] = {
"installed": installed,
"action": action,
"authenticated": installed and authenticated,
# The engine gate, verbatim. Never report active on anything else.
"engine_active": brightdata.is_available(config),
}
if off_path:
status["path"] = off_path
status["hint"] = (
f"brightdata found at {off_path} but not on PATH; add its directory "
"to PATH so the engine subprocess can see it"
)
elif installed and not authenticated:
status["hint"] = "run `brightdata login` to activate the amazon source"
elif not installed:
status["hint"] = (
"install with `npm i -g @brightdata/cli` then `brightdata login` "
"to enable the amazon source"
)
return status
def _pp_bin_candidate_paths(bin_name: str) -> list[Path]:
"""Known install locations for a Printing Press CLI binary (slug-parameterized
@@ -328,32 +450,21 @@ def _install_pp_cli(slug: str, bin_name: str) -> Tuple[bool, str, str, str]:
off_path = _pp_off_path_binary(bin_name)
if off_path:
return False, "installed_off_path", "", off_path
if shutil.which("npx") is None:
return False, "no_npx", "", ""
try:
proc = subprocess.run(
["npx", "-y", PRINTING_PRESS_NPM, "install", slug, "--cli-only"],
capture_output=True, text=True, timeout=DIGG_INSTALL_TIMEOUT,
)
except Exception as exc:
logger.warning("npx install %s exception: %s", slug, exc)
return False, "install_failed", str(exc), ""
if proc.returncode != 0:
stderr = proc.stderr or f"npx install {slug} exited {proc.returncode}"
logger.warning("npx install %s failed (rc=%s): %s", slug, proc.returncode, stderr)
return False, "install_failed", stderr, ""
action, stderr = _run_npx_install(slug)
if action:
return False, action, stderr, ""
on_path = shutil.which(bin_name)
if on_path:
return True, "installed", "", ""
off_path = _pp_off_path_binary(bin_name)
if off_path:
combined = (proc.stderr or "").strip()
combined = stderr.strip()
if combined:
logger.warning("%s installed off PATH: %s", bin_name, combined)
return False, "installed_off_path", combined, off_path
stderr = proc.stderr or f"install completed but {bin_name} was not found"
logger.warning("npx install %s failed verification: %s", slug, stderr)
return False, "install_failed", stderr, ""
stderr_msg = stderr or f"install completed but {bin_name} was not found"
logger.warning("npx install %s failed verification: %s", slug, stderr_msg)
return False, "install_failed", stderr_msg, ""
def install_default_pp_sources() -> Dict[str, Dict[str, Any]]:
@@ -555,6 +666,11 @@ def get_setup_status_text(results: Dict[str, Any]) -> str:
lines.append(" - yt-dlp install failed \u2014 run `brew install yt-dlp` manually")
elif ytdlp_action == "no_homebrew":
lines.append(" - yt-dlp not found. Install Homebrew first, then: brew install yt-dlp")
elif ytdlp_action == "no_pip_windows":
lines.append(
" - yt-dlp not found. Install with: pip install yt-dlp "
"(it may install to a Scripts directory not on PATH -- add it to PATH if YouTube search stays inactive)"
)
elif ytdlp_action == "already_installed":
lines.append(" - yt-dlp already installed")
elif results.get("ytdlp_installed", False):
@@ -624,6 +740,33 @@ def get_setup_status_text(results: Dict[str, Any]) -> str:
f"then: `npx -y {PRINTING_PRESS_NPM} install {source_key} --cli-only`"
)
# Bright Data / Amazon. Reported but never installed (it spends the user's
# own metered credits), so the only useful thing setup can do is say
# precisely why the lane is or is not active -- the three states below are
# otherwise invisible, since SKILL.md tells the model not to raise the
# subject mid-run.
brightdata_status_entry = results.get("brightdata") or {}
bd_action = brightdata_status_entry.get("action", "")
if brightdata_status_entry.get("engine_active"):
lines.append(" - Bright Data CLI ready (Amazon buyer signals available)")
elif bd_action == "already_installed":
lines.append(
" - Bright Data CLI installed but not logged in — run "
"`brightdata login` to enable Amazon buyer signals (optional)"
)
elif bd_action == "installed_off_path":
bd_path = brightdata_status_entry.get("path", "")
lines.append(
f" - Bright Data CLI found at {bd_path} but not on PATH — add "
f"{os.path.dirname(os.path.expanduser(bd_path))} to PATH and restart "
"your agent session/gateway for Amazon buyer signals to activate"
)
elif bd_action == "not_installed":
lines.append(
" - Amazon buyer signals not installed (optional; 5,000 free "
"requests/month). Install with: npm i -g @brightdata/cli && brightdata login"
)
env_written = results.get("env_written", False)
if env_written:
lines.append("")
+46 -3
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
import math
from collections.abc import Iterable
from . import dates, relevance, schema
@@ -16,6 +17,9 @@ SOURCE_QUALITY = {
"arxiv": 0.9,
"techmeme": 0.85,
"trustpilot": 0.78,
# Verified-purchase reviews on a live aggregate rating: high-quality
# buyer evidence, a notch above Trustpilot's open review model.
"amazon": 0.8,
"reddit": 0.6,
"x": 0.68,
"bluesky": 0.66,
@@ -59,6 +63,17 @@ def local_relevance(
if "project-mode" in labels:
score = max(score, 0.8)
# Grounding-exempt floor (currently Amazon): the adapter already gated
# these against the model-supplied product keyword before creating them,
# so they are relevant by construction. Their text is marketing copy plus
# buyer reviews, which rarely repeats the topic phrasing -- a "Weber
# Grills" run surfaces a product named "Spirit E-325" whose reviews talk
# about searing, not about Weber. Without the floor, correctly-retrieved
# evidence gets pruned for failing a keyword match it was never going to
# win. Mirrors the project-mode GitHub floor above.
if isinstance(item.metadata, dict) and item.metadata.get("grounding_exempt"):
score = max(score, 0.8)
return score
@@ -167,6 +182,7 @@ ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {
"polymarket": [("volume", 0.60), ("liquidity", 0.40)],
"digg": [("postCount", 0.40), ("uniqueAuthors", 0.30), ("rank_score", 0.30)],
"trustpilot": [("reviews", 1.0)],
"amazon": [("ratings", 1.0)],
}
@@ -318,16 +334,35 @@ def _passes_engagement_floor(item: schema.SourceItem, sole_source: bool) -> bool
def prune_low_relevance(
items: list[schema.SourceItem],
minimum: float = 0.15,
first_party_handles: Iterable[str] | None = None,
) -> list[schema.SourceItem]:
"""Drop weak lexical matches when stronger evidence exists.
Social-source items with zero engagement get a stricter threshold
because zero engagement on a social platform is a strong noise signal.
Social-source items with genuinely zero engagement get a stricter
threshold because zero engagement on a social platform is a strong noise
signal.
TikTok and Instagram items with fewer than 1000 views are pruned
(unless they are the only source represented in the batch).
``first_party_handles`` names accounts this run is explicitly searching
(the subject of the topic). Their own posts are exempt from the floor: a
post almost never contains its own author's name, so lexical relevance
scores it at or near zero no matter how on-topic it is. Without the
exemption a mixed batch loses them silently, because the ``filtered or
items`` rescue below only fires when *every* item fails.
"""
sources_present = {item.source for item in items}
first_party = {
h.strip().lstrip("@").lower()
for h in (first_party_handles or ())
if h and h.strip()
}
def _is_first_party(item: schema.SourceItem) -> bool:
if not first_party or not item.author:
return False
return item.author.strip().lstrip("@").lower() in first_party
def passes(item: schema.SourceItem) -> bool:
# YouTube items with successfully extracted transcripts should not
@@ -335,10 +370,18 @@ def prune_low_relevance(
# already proves substantive topical coverage.
if item.source == "youtube" and item.snippet:
return True
# Posts by an account this run is explicitly searching are evidence by
# provenance, not by lexical overlap with the topic.
if _is_first_party(item):
return True
rel = item.local_relevance if item.local_relevance is not None else 0.0
if rel < minimum:
return False
if item.source in _SOCIAL_SOURCES and (item.engagement_score is None or item.engagement_score == 0):
# Key the stricter social gate on genuinely absent engagement, not on
# the normalized score: signals.normalize is min-max over the batch, so
# it maps the least-engaged item to exactly 0 even when that item has
# thousands of likes.
if item.source in _SOCIAL_SOURCES and not engagement_raw(item):
if rel < minimum * 1.5:
return False
sole_source = sources_present == {item.source}
+9 -1
View File
@@ -89,7 +89,15 @@ def search_truthsocial(
try:
response = http.request(
"GET", url,
headers={"Authorization": f"Bearer {token}"},
headers={
"Authorization": f"Bearer {token}",
# Cloudflare 403s the skill's default User-Agent regardless of token validity (#909).
# Reuse http.BROWSER_USER_AGENT, as the keyless Reddit path does.
"User-Agent": http.BROWSER_USER_AGENT,
"Accept": "application/json, text/plain, */*",
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://truthsocial.com/",
},
timeout=30,
)
except http.HTTPError as e:
+2
View File
@@ -130,6 +130,7 @@ SOURCE_COMPLETION_ORDER = [
"arxiv",
"techmeme",
"trustpilot",
"amazon",
]
SOURCE_COMPLETION_META = {
@@ -148,6 +149,7 @@ SOURCE_COMPLETION_META = {
"arxiv": ("arXiv", "paper", "papers", Colors.RED),
"techmeme": ("Techmeme", "headline", "headlines", Colors.CYAN),
"trustpilot": ("Trustpilot", "review", "reviews", Colors.GREEN),
"amazon": ("Amazon", "product", "products", Colors.YELLOW),
}
+292
View File
@@ -0,0 +1,292 @@
"""X corpus judging for retrieve-judge-retry.
No I/O: judges items already retrieved. Reuses relevance.token_overlap_relevance.
The judge detects off-topic floods and determines which extracted handles should
be promoted to the FROM lane based on on-topic post ratio, not frequency.
"""
from collections import Counter
from typing import Any, Dict, List, Optional, Set, Tuple
from . import relevance
# Minimum on-topic ratio for the overall corpus. Below this, the engine should
# retry with a wider keyword query. Rome measured ~0.2 (8/40 on-topic).
CORPUS_ON_TOPIC_FLOOR = 0.4
# Minimum on-topic ratio for a handle's posts to qualify for FROM promotion.
HANDLE_ON_TOPIC_FLOOR = 0.5
# Minimum on-topic keyword hits before a handle can be promoted to FROM lane.
# Prevents promoting handles that appeared in thin phrase hits with only 1 match.
MIN_ON_TOPIC_HITS = 2
# Ambiguous short tokens that require case-sensitive matching to avoid
# pronoun/acronym collisions. E.g., "US" (country) vs "us" (pronoun).
# For these tokens, require the text to contain the uppercase form (acronym)
# rather than just the lowercase form (common word).
_CASE_SENSITIVE_ACRONYMS = frozenset({'us'})
def _compute_relevance(query: str, text: str) -> float:
"""Compute relevance score for a post against the topic query.
Returns 0.0 for empty/stopword-only queries to avoid treating all items
as equally relevant (the shared relevance module returns 0.5 for empty
queries as a neutral fallback, but x_judge needs strict filtering).
Uses case-sensitive matching for ambiguous short tokens like 'us' to
distinguish country acronym 'US' from pronoun 'us'.
"""
import re
if not query or not text:
return 0.0
q_tokens = relevance.tokenize(query)
if not q_tokens:
return 0.0 # All query tokens were stopwords
# Check for ambiguous acronyms that need case-sensitive handling
t_tokens = relevance.tokenize(text)
filtered_q_tokens = set(q_tokens)
for acronym in _CASE_SENSITIVE_ACRONYMS:
if acronym in q_tokens and acronym in t_tokens:
# Text has the token, but we need to check if it's the acronym (US)
# or the common word (us). If text only has lowercase, don't count it.
has_uppercase = bool(re.search(rf'\b{acronym.upper()}\b', text))
has_lowercase = bool(re.search(rf'\b{acronym}\b', text))
if has_lowercase and not has_uppercase:
# Text only has lowercase version (pronoun) - don't count this
# token in query overlap. This effectively removes "us" from
# contributing to the score when text has only the pronoun.
t_tokens = t_tokens - {acronym}
# Also remove from query for this calculation to avoid
# penalizing the overall coverage ratio
filtered_q_tokens = filtered_q_tokens - {acronym}
# If filtering removed all query tokens, fall back to 0
if not filtered_q_tokens:
return 0.0
overlap_tokens = filtered_q_tokens & t_tokens
if not overlap_tokens:
return 0.0
# Compute simplified relevance: coverage ratio
# This is a simpler version of token_overlap_relevance that uses
# the filtered tokens rather than re-tokenizing the original text
coverage = len(overlap_tokens) / len(filtered_q_tokens)
return coverage
def judge_x_corpus(
items: List[Dict[str, Any]],
topic: str,
*,
ranking_query: str = "",
) -> Dict[str, Any]:
"""Judge the retrieved X corpus for on-topic ratio.
Args:
items: List of X items with 'author_handle' and 'text' fields
topic: The user topic (e.g., "Rome")
ranking_query: Optional ranking query for better relevance scoring
Returns:
Dict with:
- on_topic_ratio: float, fraction of posts that are on-topic
- is_off_topic_flood: bool, True if corpus fails the on-topic floor
- on_topic_items: list, items that passed relevance check
- off_topic_items: list, items that failed relevance check
- handle_stats: dict, per-handle on-topic counts and totals
"""
if not items:
return {
"on_topic_ratio": 1.0,
"is_off_topic_flood": False,
"on_topic_items": [],
"off_topic_items": [],
"handle_stats": {},
}
# Use ranking_query if provided, otherwise topic
query = ranking_query or topic
on_topic_items = []
off_topic_items = []
handle_stats: Dict[str, Dict[str, int]] = {}
for item in items:
handle = (item.get("author_handle") or "").lower()
text = item.get("text") or ""
score = _compute_relevance(query, text)
# On-topic threshold: relevance.RELEVANCE_FLOOR is 0.1
is_on_topic = score >= relevance.RELEVANCE_FLOOR
if is_on_topic:
on_topic_items.append(item)
else:
off_topic_items.append(item)
if handle:
if handle not in handle_stats:
handle_stats[handle] = {"on_topic": 0, "total": 0}
handle_stats[handle]["total"] += 1
if is_on_topic:
handle_stats[handle]["on_topic"] += 1
on_topic_ratio = len(on_topic_items) / len(items) if items else 1.0
# Check if top-3 frequency authors have poor on-topic ratio
top_handles = sorted(
handle_stats.items(),
key=lambda x: x[1]["total"],
reverse=True,
)[:3]
top_authors_off_topic = all(
stats["on_topic"] / stats["total"] < HANDLE_ON_TOPIC_FLOOR
for _, stats in top_handles
if stats["total"] > 0
) if top_handles else False
is_off_topic_flood = (
on_topic_ratio < CORPUS_ON_TOPIC_FLOOR
or (top_authors_off_topic and len(on_topic_items) < MIN_ON_TOPIC_HITS)
)
return {
"on_topic_ratio": on_topic_ratio,
"is_off_topic_flood": is_off_topic_flood,
"on_topic_items": on_topic_items,
"off_topic_items": off_topic_items,
"handle_stats": handle_stats,
}
def promotable_handles(
items: List[Dict[str, Any]],
topic: str,
extracted_handles: List[str],
*,
explicit_handles: Optional[List[str]] = None,
ranking_query: str = "",
) -> Tuple[List[str], List[str]]:
"""Determine which handles should be promoted to the FROM lane.
Split FROM promotion:
- Explicit handles (--x-handle/--x-related): always promoted, no AND topic
- Extracted handles: promoted only if:
- MIN_ON_TOPIC_HITS on-topic keyword hits AND
- author on-topic ratio HANDLE_ON_TOPIC_FLOOR
- These pulls AND the topic (from:handle Rome)
Args:
items: List of X items with 'author_handle' and 'text' fields
topic: The user topic
extracted_handles: Handles extracted from entity_extract
explicit_handles: Explicit --x-handle/--x-related handles
ranking_query: Optional ranking query for relevance scoring
Returns:
Tuple of (explicit_promotable, extracted_promotable):
- explicit_promotable: handles that get FROM without AND topic
- extracted_promotable: handles that get FROM with AND topic
"""
explicit_set = {
h.lower().lstrip("@")
for h in (explicit_handles or [])
if h and h.strip()
}
# Judge corpus to get handle stats
judgment = judge_x_corpus(items, topic, ranking_query=ranking_query)
handle_stats = judgment["handle_stats"]
explicit_promotable = []
extracted_promotable = []
for handle in extracted_handles:
handle_lower = handle.lower().lstrip("@")
# Explicit handles always promoted (no AND topic)
if handle_lower in explicit_set:
explicit_promotable.append(handle)
continue
# Check if handle qualifies for extracted promotion
stats = handle_stats.get(handle_lower, {"on_topic": 0, "total": 0})
# Need ≥MIN_ON_TOPIC_HITS on-topic posts
if stats["on_topic"] < MIN_ON_TOPIC_HITS:
continue
# Need ≥HANDLE_ON_TOPIC_FLOOR ratio
if stats["total"] > 0:
ratio = stats["on_topic"] / stats["total"]
if ratio >= HANDLE_ON_TOPIC_FLOOR:
extracted_promotable.append(handle)
# Also check explicit handles not in extracted list
for handle in (explicit_handles or []):
handle_lower = handle.lower().lstrip("@")
if handle_lower not in [h.lower() for h in explicit_promotable]:
if handle_lower not in [h.lower() for h in extracted_handles]:
explicit_promotable.append(handle)
return explicit_promotable, extracted_promotable
def should_retry_x_search(
items: List[Dict[str, Any]],
topic: str,
*,
ranking_query: str = "",
depth: str = "default",
) -> bool:
"""Determine if X search should retry with wider keyword query.
Skip retry on quick/mock (same as Phase 2).
Args:
items: Retrieved X items
topic: The user topic
ranking_query: Optional ranking query
depth: Search depth ("quick", "default", "deep")
Returns:
True if retry is warranted
"""
if depth == "quick":
return False
if not items:
return False # Nothing to judge, no retry
judgment = judge_x_corpus(items, topic, ranking_query=ranking_query)
return judgment["is_off_topic_flood"]
def prune_off_topic_items(
items: List[Dict[str, Any]],
topic: str,
*,
ranking_query: str = "",
) -> List[Dict[str, Any]]:
"""Prune off-topic items before the pool.
Eight on-topic items with 32 pruned ok with 8.
Zero on-topic no-results, not ok with 40 junk.
Args:
items: Retrieved X items
topic: The user topic
ranking_query: Optional ranking query
Returns:
Only on-topic items
"""
judgment = judge_x_corpus(items, topic, ranking_query=ranking_query)
return judgment["on_topic_items"]
+277 -83
View File
@@ -6,6 +6,7 @@ transcript extraction. No API keys needed — just have yt-dlp installed.
Inspired by Peter Steinberger's toolchain approach (yt-dlp + summarize CLI).
"""
import copy
import json
import math
import os
@@ -14,6 +15,7 @@ import shlex
import shutil
import sys
import tempfile
import threading
import time
import urllib.error
import urllib.request
@@ -71,6 +73,18 @@ _TRANSCRIPT_MAX_RETRIES = 2
_TRANSCRIPT_BACKOFF_BASE = 2.0 # seconds; multiplied by (attempt + 1)
_TRANSCRIPT_TIMEOUT = 30 # seconds per yt-dlp attempt (keyless: no fallback to fail over to)
_TRANSCRIPT_FAST_TIMEOUT = 12 # seconds per attempt when a ScrapeCreators fallback exists
_SEARCH_TIMEOUT = 120 # seconds per ytsearch metadata extraction
# Comparison-mode fan-out (and nested transcript/comment pools) can stampede the
# same throttled YouTube IP. Cap concurrent yt-dlp processes process-wide.
_YTDLP_MAX_CONCURRENT = 2
_ytdlp_slots = threading.Semaphore(_YTDLP_MAX_CONCURRENT)
# In-run search cache: comparison mode re-issues identical ytsearch queries from
# every entity sub-run; cache hits avoid the redundant expensive --dump-json work.
# Inflight coalescing prevents N concurrent identical searches from all missing
# the cache and stampeding YouTube together.
_search_cache: Dict[Tuple[str, int, str], Dict[str, Any]] = {}
_search_inflight: Dict[Tuple[str, int, str], tuple[threading.Event, list]] = {}
_search_cache_lock = threading.Lock()
# Comments are enrichment, not core evidence: keep the budget tight so a slow
# comment API can never dominate a run's wall clock (bounded to 3 videos).
_COMMENT_TIMEOUT = 20
@@ -146,11 +160,118 @@ def _log(msg: str):
log.source_log("YouTube", msg, tty_only=False)
def reset_search_cache() -> None:
"""Clear the in-run ytsearch cache.
Call at the start of each top-level research run so a long-lived process
(agent host, REPL, test suite) does not reuse results across runs. Within
one comparison fan-out the cache stays hot so identical queries coalesce.
"""
with _search_cache_lock:
_search_cache.clear()
_search_inflight.clear()
def _env_positive_float(name: str, default: float) -> float:
"""Read a positive finite float from the environment, else ``default``."""
raw = os.environ.get(name, "").strip()
try:
value = float(raw) if raw else float(default)
except ValueError:
return float(default)
if not math.isfinite(value) or value <= 0:
return float(default)
return value
def _search_timeout() -> float:
"""Return the ytsearch timeout, preserving the 120s default."""
return _env_positive_float("LAST30DAYS_YT_SEARCH_TIMEOUT", float(_SEARCH_TIMEOUT))
def _run_ytdlp(cmd: List[str], *, timeout: float) -> subproc.SubprocResult:
"""Run a yt-dlp (or SSH-wrapped) command under the process-wide concurrency gate."""
with _ytdlp_slots:
return subproc.run_with_timeout(cmd, timeout=timeout)
def _claim_search_slot(
cache_key: Tuple[str, int, str],
) -> tuple[Optional[Dict[str, Any]], Optional[threading.Event], Optional[list], bool]:
"""Return ``(cached, event, slot, is_leader)`` for search coalesce.
- Cache hit: ``(payload, None, None, False)`` caller returns ``payload``.
- Waiter: ``(None, event, slot, False)`` caller awaits ``slot`` via ``event``.
- Leader: ``(None, event, slot, True)`` caller runs yt-dlp and finishes the slot.
"""
with _search_cache_lock:
cached = _search_cache.get(cache_key)
if cached is not None:
return copy.deepcopy(cached), None, None, False
existing = _search_inflight.get(cache_key)
if existing is not None:
return None, existing[0], existing[1], False
event = threading.Event()
slot: list = [None]
_search_inflight[cache_key] = (event, slot)
return None, event, slot, True
def _finish_search_slot(
cache_key: Tuple[str, int, str],
payload: Dict[str, Any],
*,
event: threading.Event,
slot: list,
) -> Dict[str, Any]:
"""Publish a search result to waiters; cache only clean (non-error) payloads.
Ownership is by slot identity: after ``reset_search_cache()`` clears the
registry, a stale leader must still wake its own waiters but must not pop
or overwrite a newer run's registration for the same key.
"""
shared = copy.deepcopy(payload)
with _search_cache_lock:
if slot[0] is not None:
# Idempotent re-finish of this slot (e.g. finally after return).
event.set()
return payload
slot[0] = shared
current = _search_inflight.get(cache_key)
if current is not None and current[1] is slot:
if not payload.get("error"):
_search_cache[cache_key] = shared
_search_inflight.pop(cache_key, None)
# else: stale leader after a reset — wake local waiters only.
event.set()
return payload
def _await_search_slot(
event: threading.Event,
slot: list,
) -> Dict[str, Any]:
"""Wait for a leader search to publish.
Waiters block until the leader finishes (success or failure). The leader
path always publishes via ``_finish_search_slot``, including on unexpected
exceptions, so a timed wait would only invent a false timeout while the
leader was still queued behind other yt-dlp work.
"""
event.wait()
shared = slot[0]
if isinstance(shared, dict):
return copy.deepcopy(shared)
return {"items": [], "error": "YouTube search failed"}
def classify_run_failure(detail: str) -> str:
"""Map yt-dlp's text-only throttling and bot-gate errors."""
text = detail.lower()
if any(marker in text for marker in ("yt-dlp not installed", "yt-dlp not found")):
return health.SKIPPED_UNCONFIGURED
if any(marker in text for marker in ("timed out", "timeout")):
return health.TIMEOUT
if any(
marker in text
for marker in ("http error 429", "confirm you're not a bot", "confirm youre not a bot", "bot-gate")
@@ -317,6 +438,20 @@ def search_youtube(
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
cache_key = (core_topic, count, from_date)
timeout = _search_timeout()
cached, event, slot, is_leader = _claim_search_slot(cache_key)
if cached is not None:
_log(f"YouTube search cache hit for '{core_topic}' (count={count})")
return cached
assert event is not None and slot is not None
if not is_leader:
_log(f"YouTube search awaiting in-flight query for '{core_topic}'")
return _await_search_slot(event, slot)
def _publish(payload: Dict[str, Any]) -> Dict[str, Any]:
return _finish_search_slot(cache_key, payload, event=event, slot=slot)
_log(f"Searching YouTube for '{core_topic}' (since {from_date}, count={count})")
@@ -336,82 +471,97 @@ def search_youtube(
cmd = _wrap_ytdlp_cmd(cmd)
ssh_host = _ytdlp_ssh_host()
published: Dict[str, Any] | None = None
try:
result = subproc.run_with_timeout(cmd, timeout=120)
except subproc.SubprocTimeout:
_log("YouTube search timed out (120s)")
return {"items": [], "error": "Search timed out"}
except FileNotFoundError:
return {"items": [], "error": "yt-dlp not found"}
stdout = result.stdout
if ssh_host and result.returncode != 0 and not stdout.strip():
stderr_first = (result.stderr or "").strip().splitlines()
first_line = stderr_first[0] if stderr_first else "(no stderr)"
_log(
f"YouTube search via SSH host {ssh_host!r} failed "
f"(rc={result.returncode}): {first_line}"
)
return {
"items": [],
"error": f"SSH routing to {ssh_host!r} failed: {first_line}",
}
if not stdout.strip():
_log("YouTube search returned 0 results")
return {"items": []}
# Parse JSON-per-line output
items = []
for line in stdout.strip().split("\n"):
line = line.strip()
if not line:
continue
try:
video = json.loads(line)
except json.JSONDecodeError:
continue
result = _run_ytdlp(cmd, timeout=timeout)
except subproc.SubprocTimeout:
_log(f"YouTube search timed out ({timeout:g}s)")
published = _publish(
{"items": [], "error": f"Search timed out after {timeout:g}s"}
)
return published
except FileNotFoundError:
published = _publish({"items": [], "error": "yt-dlp not found"})
return published
video_id = video.get("id", "")
view_count = video.get("view_count") if video.get("view_count") is not None else 0
like_count = video.get("like_count") if video.get("like_count") is not None else 0
comment_count = video.get("comment_count") if video.get("comment_count") is not None else 0
upload_date = video.get("upload_date", "") # YYYYMMDD
stdout = result.stdout
if ssh_host and result.returncode != 0 and not stdout.strip():
stderr_first = (result.stderr or "").strip().splitlines()
first_line = stderr_first[0] if stderr_first else "(no stderr)"
_log(
f"YouTube search via SSH host {ssh_host!r} failed "
f"(rc={result.returncode}): {first_line}"
)
published = _publish(
{"items": [], "error": f"SSH routing to {ssh_host!r} failed: {first_line}"},
)
return published
if not stdout.strip():
_log("YouTube search returned 0 results")
published = _publish({"items": []})
return published
# Convert YYYYMMDD to YYYY-MM-DD
date_str = None
if upload_date and len(upload_date) == 8:
date_str = f"{upload_date[:4]}-{upload_date[4:6]}-{upload_date[6:8]}"
# Parse JSON-per-line output
items = []
for line in stdout.strip().split("\n"):
line = line.strip()
if not line:
continue
try:
video = json.loads(line)
except json.JSONDecodeError:
continue
description = str(video.get("description", ""))[:500]
items.append({
"video_id": video_id,
"title": video.get("title", ""),
"url": f"https://www.youtube.com/watch?v={video_id}",
"channel_name": video.get("channel", video.get("uploader", "")),
"date": date_str,
"engagement": {
"views": view_count,
"likes": like_count,
"comments": comment_count,
},
"duration": video.get("duration"),
"relevance": _compute_relevance(core_topic, f"{video.get('title', '')} {description}"),
"why_relevant": f"YouTube: {video.get('title', core_topic)[:60]}",
"description": description,
})
video_id = video.get("id", "")
view_count = video.get("view_count") if video.get("view_count") is not None else 0
like_count = video.get("like_count") if video.get("like_count") is not None else 0
comment_count = video.get("comment_count") if video.get("comment_count") is not None else 0
upload_date = video.get("upload_date", "") # YYYYMMDD
# Soft date filter: prefer recent items but fall back to all if too few
recent = [i for i in items if i["date"] and i["date"] >= from_date]
if len(recent) >= 3:
items = recent
_log(f"Found {len(items)} videos within date range")
else:
_log(f"Found {len(items)} videos ({len(recent)} within date range, keeping all)")
# Convert YYYYMMDD to YYYY-MM-DD
date_str = None
if upload_date and len(upload_date) == 8:
date_str = f"{upload_date[:4]}-{upload_date[4:6]}-{upload_date[6:8]}"
# Sort by views descending
items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
description = str(video.get("description", ""))[:500]
items.append({
"video_id": video_id,
"title": video.get("title", ""),
"url": f"https://www.youtube.com/watch?v={video_id}",
"channel_name": video.get("channel", video.get("uploader", "")),
"date": date_str,
"engagement": {
"views": view_count,
"likes": like_count,
"comments": comment_count,
},
"duration": video.get("duration"),
"relevance": _compute_relevance(core_topic, f"{video.get('title', '')} {description}"),
"why_relevant": f"YouTube: {video.get('title', core_topic)[:60]}",
"description": description,
})
return {"items": items}
# Soft date filter: prefer recent items but fall back to all if too few
recent = [i for i in items if i["date"] and i["date"] >= from_date]
if len(recent) >= 3:
items = recent
_log(f"Found {len(items)} videos within date range")
else:
_log(f"Found {len(items)} videos ({len(recent)} within date range, keeping all)")
# Sort by views descending
items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
published = _publish({"items": items})
return published
except Exception as exc:
# Post-subprocess failures (parse/relevance/sort) must still unblock
# coalesced waiters — otherwise the inflight key orphans forever.
published = _publish({"items": [], "error": str(exc)})
return published
finally:
if published is None:
_publish({"items": [], "error": "YouTube search failed"})
def _clean_vtt(vtt_text: str) -> str:
@@ -564,7 +714,7 @@ def _fetch_transcript_ytdlp_via_ssh(video_id: str, ssh_host: str) -> Optional[st
)
cmd = ["ssh", "-o", "BatchMode=yes", "--", ssh_host, remote_script]
try:
result = subproc.run_with_timeout(cmd, timeout=45)
result = _run_ytdlp(cmd, timeout=45)
except subproc.SubprocTimeout:
_log(f"SSH yt-dlp transcript timed out for {video_id} via {ssh_host!r}")
return None
@@ -593,15 +743,10 @@ def _ytdlp_sub_langs() -> str:
def _transcript_fast_timeout() -> float:
"""Return the keyed-run yt-dlp timeout, preserving the 12s default."""
raw = os.environ.get("LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT", "").strip()
try:
timeout = float(raw) if raw else float(_TRANSCRIPT_FAST_TIMEOUT)
except ValueError:
return float(_TRANSCRIPT_FAST_TIMEOUT)
if not math.isfinite(timeout) or timeout <= 0:
return float(_TRANSCRIPT_FAST_TIMEOUT)
return timeout
return _env_positive_float(
"LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT",
float(_TRANSCRIPT_FAST_TIMEOUT),
)
def _pick_ytdlp_vtt(video_id: str, temp_dir: str, priority: List[str]) -> Optional[Path]:
"""Return the best on-disk VTT match for video_id, preferring priority order."""
@@ -686,7 +831,7 @@ def _fetch_transcript_ytdlp(
last_reason: Optional[str] = None
for attempt in range(attempts):
try:
result = subproc.run_with_timeout(cmd, timeout=timeout)
result = _run_ytdlp(cmd, timeout=timeout)
except subproc.SubprocTimeout:
last_reason = f"timed out after {timeout}s"
_log(f"yt-dlp transcript timed out after {timeout}s for {video_id} "
@@ -832,6 +977,17 @@ def fetch_transcript(
if token and _should_try_sc_transcript(status):
sc_transcript = _sc_fetch_transcript(video_id, token)
if sc_transcript:
# The keyless cascade (yt-dlp / direct HTTP) already logged its
# failure above. Without this line that failure is the last thing
# printed for this video, and the batch summary in
# fetch_transcripts_parallel() counts it as a plain success —
# making a rate-limited/bot-gated run look like nothing went
# wrong. Log the rescue and flag it in `status` so the summary
# can report it explicitly instead of masking it (#831).
_log(f"ScrapeCreators transcript fallback rescued {video_id} "
f"after the keyless fetch cascade failed")
if status is not None:
status["sc_rescued"] = True
return sc_transcript
_log(f"No transcript available for {video_id}")
@@ -892,7 +1048,20 @@ def fetch_transcripts_parallel(
got = sum(1 for v in results.values() if v)
errors = sum(1 for v in results.values() if v is None)
_log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
# `got` includes videos that only succeeded because the ScrapeCreators
# fallback rescued a failed keyless fetch — yt-dlp when available, or the
# direct HTTP path alone (see fetch_transcript()). Folding
# those into a bare "M failed" count previously made a fully rate-limited
# yt-dlp run — every fetch failing, silently saved by the fallback — read
# as "0 failed", with no trace of the fallback ever having fired (#831).
# Surface the split so the summary can't misrepresent a masked failure
# as a clean success.
sc_rescued = sum(1 for st in statuses.values() if st.get("sc_rescued"))
if sc_rescued:
_log(f"Got transcripts for {got}/{len(video_ids)} videos "
f"({errors} failed, {sc_rescued} rescued via ScrapeCreators fallback)")
else:
_log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
return results
@@ -960,6 +1129,22 @@ def _transcript_candidate_sort_key(item: dict) -> tuple:
return (views, recency)
def _prefer_search_error(current: Optional[str], new: str) -> str:
"""Keep the most actionable search failure across multi-query merges."""
if current is None:
return new
priority = ("timed out", "timeout", "429", "bot")
def _rank(text: str) -> int:
lower = text.lower()
for index, marker in enumerate(priority):
if marker in lower:
return index
return len(priority)
return new if _rank(new) < _rank(current) else current
def search_and_transcribe(
topic: str,
from_date: str,
@@ -987,8 +1172,12 @@ def search_and_transcribe(
queries = expand_youtube_queries(topic, depth)
seen_ids: Set[str] = set()
items: List[Dict[str, Any]] = []
search_error: Optional[str] = None
for q in queries:
search_result = search_youtube(q, from_date, to_date, depth)
err = search_result.get("error")
if err:
search_error = _prefer_search_error(search_error, str(err))
for item in search_result.get("items", []):
vid = item.get("video_id", "")
if vid and vid not in seen_ids:
@@ -999,7 +1188,7 @@ def search_and_transcribe(
items.sort(key=lambda x: x.get("engagement", {}).get("views") or 0, reverse=True)
if not items:
return search_result
return {"items": [], **({"error": search_error} if search_error else {})}
# Step 2: Fetch transcripts for top videos.
# Sort candidates by a combination of views and recency so that recent
@@ -1046,7 +1235,12 @@ def search_and_transcribe(
)
item["captions_disabled"] = vid in captions_disabled_ids
return {"items": items}
result: Dict[str, Any] = {"items": items}
if search_error:
# Partial coverage: some queries succeeded; keep the failure visible so
# source_status becomes partial/timeout rather than a quiet OK.
result["error"] = search_error
return result
def parse_youtube_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
@@ -1161,7 +1355,7 @@ def _ytdlp_comments_result(
])
try:
result = subproc.run_with_timeout(cmd, timeout=_COMMENT_TIMEOUT)
result = _run_ytdlp(cmd, timeout=_COMMENT_TIMEOUT)
except Exception as exc:
_log(f"yt-dlp comment fetch failed for {video_id}: {exc}")
return [], False
@@ -41,6 +41,7 @@ ALL_KEYS=(
XQUIK_API_KEY
XIAOHONGSHU_API_BASE
GITHUB_TOKEN
BRIGHTDATA_API_KEY
)
if [[ "${OSTYPE:-}" != darwin* ]]; then
+1
View File
@@ -43,6 +43,7 @@ ALL_KEYS=(
XQUIK_API_KEY
XIAOHONGSHU_API_BASE
GITHUB_TOKEN
BRIGHTDATA_API_KEY
)
REPLACE=0
+19 -1
View File
@@ -16,15 +16,33 @@ def _no_arctic_network():
yield
@pytest.fixture(autouse=True)
def _no_ambient_grok_cli():
"""Default the grok CLI to absent so no test resolves it from the developer's
own machine. grok is a new X-chain backend whose availability is a plain
filesystem check, so a machine with it installed and signed in would
otherwise silently change chain resolution in every existing X test.
Tests that exercise grok stub these themselves (test_grok_x,
test_backend_descriptors._x_env) and override this by patching inside the
test body."""
# Stub the input (PATH resolution), not the logic: has_stored_auth and
# _is_available_uncached then both resolve absent on their own, leaving the
# module's real control flow intact for tests that exercise it.
with mock.patch("lib.grok_x.binary_path", return_value=None):
yield
@pytest.fixture(autouse=True)
def _reset_probe_caches():
"""The doctor stack memoizes probe results in module-level dicts (safe for
the one-shot CLI process, wrong across tests). Clear them around every test
so a probe cached by one test can never leak into another."""
from lib import health, xurl_x
from lib import grok_x, health, xurl_x
health.clear_dependency_probe_cache()
xurl_x.clear_availability_cache()
grok_x.clear_availability_cache()
yield
health.clear_dependency_probe_cache()
xurl_x.clear_availability_cache()
grok_x.clear_availability_cache()
+929
View File
@@ -0,0 +1,929 @@
"""Tests for the Amazon source: discovery, enrichment, stats, footer (U2, U3).
Fixtures mirror live payload shapes pulled 2026-08-13, including the three
fields that arrive doubled and the fact that ``max_reviews`` is a ceiling
rather than a quota.
Nothing here spawns a subprocess or touches the network.
"""
from __future__ import annotations
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import amazon # noqa: E402
TODAY = datetime(2026, 8, 13, tzinfo=timezone.utc)
DOMAIN = "https://www.amazon.com"
def _days_ago(n: int) -> str:
return (TODAY - timedelta(days=n)).date().isoformat()
def search_record(**over):
base = {
"asin": "B0AAA00001",
"url": "https://www.amazon.com/Bentgo-Chill/dp/B0AAA00001/ref=sr_1_1?dib=xyz",
"name": "Chill Max Leak-Proof XL Bento-Style Lunch Box | Ice Pack Included",
"brand": "Bentgo",
"sponsored": "false",
"rating": 4.4,
"num_ratings": 459,
"final_price": 39.99,
"currency": "USD",
}
base.update(over)
return base
def review_record(days_ago: int, rating: int, **over):
posted = (TODAY - timedelta(days=days_ago)).strftime("%B %-d, %Y")
base = {
"review_id": f"R{days_ago}{rating}",
# Live shape: the date is doubled and prose-wrapped.
"review_posted_date": f"{posted}Reviewed in the United States on {posted}",
"review_header": "Great box!Great box!",
"review_text": f"Review body from {days_ago} days ago.",
"rating": rating,
"helpful_count": 0,
"is_verified": True,
"is_amazon_vine": False,
"author_name": "A Buyer",
"product_rating": 4.4,
"product_rating_count": 459,
"product_rating_object": {
"one_star": 28, "two_star": 9, "three_star": 28,
"four_star": 60, "five_star": 335,
},
}
base.update(over)
return base
# ------------------------------------------------------- field repair
class TestFieldRepair:
def test_doubled_header_is_repaired(self):
assert amazon.undouble("Best Box!Best Box!") == "Best Box!"
def test_comma_doubled_badge_is_repaired(self):
assert amazon.undouble("Verified Purchase, Verified Purchase") == "Verified Purchase"
def test_genuinely_repetitive_text_survives(self):
assert amazon.undouble("Great great product") == "Great great product"
assert amazon.undouble("Buy one, buy two") == "Buy one, buy two"
def test_prose_wrapped_date_yields_the_leading_date(self):
raw = "August 3, 2026Reviewed in the United States on August 3, 2026"
assert amazon.parse_review_date(raw) == "2026-08-03"
def test_unparseable_date_is_none_not_an_exception(self):
assert amazon.parse_review_date("") is None
assert amazon.parse_review_date("sometime last year") is None
assert amazon.parse_review_date(None) is None
class TestShortName:
def test_takes_the_segment_before_the_delimiter(self):
assert amazon.short_name("Chill Max XL | Ice Pack Included") == "Chill Max XL"
def test_strips_a_leading_brand_when_present(self):
assert amazon.short_name("Weber Spirit E-325", "Weber") == "Spirit E-325"
def test_clips_long_names_on_a_word_boundary(self):
out = amazon.short_name("Kids Prints Leak-Proof 5-Compartment Bento-Style Box")
assert len(out) <= amazon.SHORT_NAME_MAX
assert not out.endswith("-")
assert " " not in out[-1:]
# ---------------------------------------------------------- discovery
class TestDiscovery:
def test_parses_products_with_rating_and_price(self):
products = amazon.parse_search_response({"records": [search_record()]}, "bentgo lunch box")
assert len(products) == 1
product = products[0]
assert product["rating"] == 4.4
assert product["price"] == 39.99
assert product["num_ratings"] == 459
assert product["brand"] == "Bentgo"
def test_urls_are_canonicalized_to_dp_asin(self):
"""Live URLs carry 200+ chars of session-scoped tracking tail."""
products = amazon.parse_search_response({"records": [search_record()]}, "bentgo")
assert products[0]["url"] == "https://www.amazon.com/dp/B0AAA00001"
def test_duplicate_asins_collapse_keeping_the_richest_record(self):
records = [
search_record(num_ratings=84),
search_record(num_ratings=459),
search_record(num_ratings=12),
]
products = amazon.parse_search_response({"records": records}, "bentgo")
assert len(products) == 1
assert products[0]["num_ratings"] == 459
def test_off_keyword_products_are_gated_out(self):
records = [
search_record(),
search_record(asin="B0ZZZ00001", name="Cordless Drill Driver Kit", brand="DeWalt",
url="https://www.amazon.com/dp/B0ZZZ00001"),
]
products = amazon.parse_search_response({"records": records}, "bentgo lunch box")
assert [p["asin"] for p in products] == ["B0AAA00001"]
def test_non_amazon_and_non_https_urls_are_rejected(self):
records = [
search_record(asin="B000000001", url="http://www.amazon.com/dp/B000000001"),
search_record(asin="B000000002", url="https://evil.example.com/dp/B2"),
search_record(asin="B000000003", url="https://www.amazon.com/dp/B000000003"),
]
products = amazon.parse_search_response({"records": records}, "bentgo chill max lunch box")
assert [p["asin"] for p in products] == ["B000000003"]
def test_alternate_marketplace_domain_is_honored(self):
record = search_record(url="https://www.amazon.co.uk/dp/B0AAA00001")
products = amazon.parse_search_response(
{"records": [record]}, "bentgo", domain="https://www.amazon.co.uk"
)
assert products and products[0]["url"].startswith("https://www.amazon.co.uk/dp/")
def test_sponsored_string_is_recorded_as_bool_never_filtered(self):
"""R4: the flag is metadata only. Filtering can blank the lane."""
records = [
search_record(asin="B000000001", sponsored="true", url="https://www.amazon.com/dp/B000000001"),
search_record(asin="B000000002", sponsored="false", url="https://www.amazon.com/dp/B000000002"),
]
products = amazon.parse_search_response({"records": records}, "bentgo chill max lunch box")
assert len(products) == 2
assert {p["asin"]: p["sponsored"] for p in products} == {"B000000001": True, "B000000002": False}
def test_error_envelope_yields_no_products(self):
assert amazon.parse_search_response({"records": [], "error": "401"}, "x") == []
class TestTargetSelection:
def _pool(self):
return amazon.parse_search_response(
{
"records": [
search_record(asin="C000000001", brand="Fimibuke", num_ratings=901,
name="60oz Leakproof Bento Lunch Box",
url="https://www.amazon.com/dp/C000000001"),
search_record(asin="B000000001", brand="Bentgo", num_ratings=821,
name="Kids Insulated Lunch Bag",
url="https://www.amazon.com/dp/B000000001"),
search_record(asin="B000000002", brand="Bentgo", num_ratings=710,
name="MicroSteel Bento Lunch Box",
url="https://www.amazon.com/dp/B000000002"),
search_record(asin="B000000003", brand="Bentgo", num_ratings=623,
name="Classic Stackable Lunch Box",
url="https://www.amazon.com/dp/B000000003"),
]
},
"bentgo lunch box",
)
def test_brand_topic_excludes_competitors_buying_the_keyword(self):
"""The guard against paying to review a rival's product."""
targets = amazon.select_enrichment_targets(self._pool(), limit=3, keyword="bentgo lunch box")
assert [t["asin"] for t in targets] == ["B000000001", "B000000002", "B000000003"]
assert all(t["brand"] == "Bentgo" for t in targets)
def test_category_topic_stays_unfiltered_and_ranks_on_merit(self):
targets = amazon.select_enrichment_targets(self._pool(), limit=3, keyword="kids lunch box")
assert targets[0]["asin"] == "C000000001"
def test_infer_brand_needs_the_keyword_to_name_it(self):
pool = self._pool()
assert amazon.infer_brand(pool, "bentgo lunch box") == "Bentgo"
assert amazon.infer_brand(pool, "best kids lunch box") == ""
def test_near_identical_variants_do_not_take_two_pulls(self):
pool = amazon.parse_search_response(
{
"records": [
search_record(asin="V000000001", num_ratings=901, name="60oz Leakproof Box | Blue",
url="https://www.amazon.com/dp/V000000001"),
search_record(asin="V000000002", num_ratings=900, name="60oz Leakproof Box | Pink",
url="https://www.amazon.com/dp/V000000002"),
search_record(asin="V000000003", num_ratings=500, name="Chill Max XL Box",
url="https://www.amazon.com/dp/V000000003"),
]
},
"bentgo lunch box",
)
targets = amazon.select_enrichment_targets(pool, limit=2, keyword="bentgo lunch box")
assert [t["asin"] for t in targets] == ["V000000001", "V000000003"]
def test_zero_limit_selects_nothing(self):
assert amazon.select_enrichment_targets(self._pool(), limit=0) == []
# --------------------------------------------------------- enrichment
class TestReviewParsing:
def test_reviews_become_comments_with_stats(self):
response = {"records": [review_record(3, 5), review_record(10, 4)]}
comments, stats = amazon.parse_reviews(response)
assert len(comments) == 2
assert stats["product_rating"] == 4.4
assert stats["product_rating_count"] == 459
assert stats["star_distribution"]["five_star"] == 335
def test_comments_carry_the_keys_remap_would_strip(self):
"""Rating, date, and verified are exactly what this source needs."""
comments, _ = amazon.parse_reviews({"records": [review_record(3, 5)]})
comment = comments[0]
assert set(comment) >= {"score", "excerpt", "rating", "date", "verified"}
assert comment["rating"] == 5
assert comment["date"] == _days_ago(3)
def test_doubled_header_is_repaired_in_the_comment_title(self):
comments, _ = amazon.parse_reviews({"records": [review_record(3, 5)]})
assert comments[0]["title"] == "Great box!"
def test_woven_sample_is_newest_first(self):
"""R2a: recency is enforced client-side, not assumed from the API."""
response = {"records": [
review_record(400, 5), review_record(2, 3), review_record(45, 4),
]}
comments, _ = amazon.parse_reviews(response)
assert [c["date"] for c in comments] == [_days_ago(2), _days_ago(45), _days_ago(400)]
def test_empty_payload_is_not_an_error(self):
assert amazon.parse_reviews({"records": []}) == ([], {})
class TestEnrichmentLane:
def _products(self, n=4):
return [
{"asin": f"B00000000{i}", "url": f"https://www.amazon.com/dp/B00000000{i}",
"name": f"Product {i}", "short_name": f"Product {i}",
"brand": "Bentgo", "num_ratings": 900 - i, "rating": 4.4}
for i in range(n)
]
def test_quick_depth_spawns_no_review_pulls(self):
calls = []
out, status = amazon.enrich_with_reviews(
self._products(), depth="quick",
fetcher=lambda url: calls.append(url) or {"records": []},
)
assert calls == []
assert len(out) == 4
assert status is None # quick depth is not a degraded outcome
def test_default_depth_pulls_exactly_three(self):
calls = []
_, status = amazon.enrich_with_reviews(
self._products(), depth="default",
fetcher=lambda url: calls.append(url) or {"records": [review_record(2, 5)]},
)
assert len(calls) == 3
assert status is None
def test_deep_depth_pulls_five(self):
calls = []
_, status = amazon.enrich_with_reviews(
self._products(6), depth="deep",
fetcher=lambda url: calls.append(url) or {"records": []},
)
assert len(calls) == 5
def test_cap_is_fifty_per_pull(self):
seen = {}
def fetcher(url):
return {"records": []}
# The cap reaches the CLI through fetch_reviews; assert the constant
# and the plumbed default together.
assert amazon.MAX_REVIEWS == 50
import lib.brightdata as bd
original = bd.run_pipeline
bd.run_pipeline = lambda p, params, **k: seen.update(params=params) or {"records": []}
try:
amazon.fetch_reviews("https://www.amazon.com/dp/B000000001")
finally:
bd.run_pipeline = original
assert seen["params"] == ["https://www.amazon.com/dp/B000000001", "50"]
def test_reviews_attach_to_the_right_product(self):
out, status = amazon.enrich_with_reviews(
self._products(3), depth="default",
fetcher=lambda url: {"records": [review_record(2, 5, review_id=url)]},
)
assert all(p.get("top_comments") for p in out)
assert out[0]["product_rating_count"] == 459
assert status is None
def test_one_failing_pull_does_not_discard_its_siblings(self):
def fetcher(url):
if url.endswith("B000000001"):
return {"records": [], "error": "snapshot failed"}
return {"records": [review_record(2, 5)]}
out, status = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)
by_asin = {p["asin"]: p for p in out}
assert not by_asin["B000000001"].get("top_comments")
assert by_asin["B000000000"].get("top_comments")
assert by_asin["B000000002"].get("top_comments")
assert status is None # partial success is not reported as status
def test_one_raising_pull_does_not_kill_the_lane(self):
def fetcher(url):
if url.endswith("B000000001"):
raise RuntimeError("boom")
return {"records": [review_record(2, 5)]}
out, status = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)
assert sum(1 for p in out if p.get("top_comments")) == 2
assert status is None # partial success is not reported as status
def test_dropped_straggler_keeps_its_product_with_search_stats(self):
"""A deadline drop must never delete the product from the report.
Use a patched short LANE_DEADLINE to trigger the straggler case without
relying on crumb budgets (which now skip the lane entirely).
"""
import time as _time
import unittest.mock
def slow(url):
if url.endswith("B000000000"):
_time.sleep(3)
return {"records": [review_record(2, 5)]}
# Patch LANE_DEADLINE to 1s so the slow product times out but other
# products have time to complete. elapsed=0 keeps the full 1s budget.
with unittest.mock.patch.object(amazon, "LANE_DEADLINE", 1):
out, status = amazon.enrich_with_reviews(
self._products(2), depth="default", fetcher=slow,
elapsed=0.0,
)
by_asin = {p["asin"]: p for p in out}
assert set(by_asin) == {"B000000000", "B000000001"}
# Search-record stats survive on the dropped product.
assert by_asin["B000000000"]["rating"] == 4.4
assert by_asin["B000000000"]["num_ratings"] == 900
# One straggler dropped, so status should be "review lane timed out"
# (since all 3 pulls didn't complete, but some did)
# Actually, B000000001 completed, so status is None
# Let me check: if completed_count > 0, status is None
def test_exhausted_wall_clock_skips_the_lane_entirely(self):
calls = []
out, status = amazon.enrich_with_reviews(
self._products(), depth="default",
fetcher=lambda url: calls.append(url) or {"records": []},
elapsed=amazon.FOREGROUND_CONTRACT,
)
assert calls == []
assert status == "review lane skipped (budget 0s)"
def test_crumb_budget_skips_not_fires_doomed_pulls(self):
"""Regression test for the Bentgo bug: elapsed=269 must skip, not fire doomed 11s pulls.
The bug: a multi-source run took 269s before reaching Amazon enrichment,
leaving only 11s of budget (300 - 269 - 20 = 11). All Bright Data pulls
timed out (cli_timeout = max(5, timeout-10) = 1s), spending credits
without returning reviews.
The fix: crumb budgets (below MIN_USEFUL_REVIEW_BUDGET=90) return 0,
skipping the lane entirely instead of firing doomed short pulls.
"""
calls = []
out, status = amazon.enrich_with_reviews(
self._products(), depth="default",
fetcher=lambda url: calls.append(url) or {"records": []},
elapsed=269.0,
)
# Fetcher should never be called
assert calls == []
# Products keep their search stats (no top_comments)
assert all(not p.get("top_comments") for p in out)
assert all(p.get("rating") == 4.4 for p in out)
assert all(p.get("num_ratings") for p in out)
# Status should indicate the lane was skipped
assert status == "review lane skipped (budget 0s)"
def test_early_elapsed_gets_full_budget(self):
"""A quick search (40s elapsed) should get the full LANE_DEADLINE budget."""
timeouts_seen = []
def capturing_fetcher(url):
return {"records": [review_record(2, 5)]}
# Patch fetch_reviews to capture the timeout
original_fetch = amazon.fetch_reviews
captured_timeout = []
def mock_fetch(url, *, max_reviews=50, config=None, timeout=180):
captured_timeout.append(timeout)
return {"records": [review_record(2, 5)]}
amazon.fetch_reviews = mock_fetch
try:
out, status = amazon.enrich_with_reviews(
self._products(1), depth="default",
elapsed=40.0,
)
finally:
amazon.fetch_reviews = original_fetch
assert captured_timeout, "fetch_reviews was not called"
# elapsed=40 → remaining = 300-40-20 = 240 → clamped to LANE_DEADLINE=180
assert captured_timeout[0] == amazon.LANE_DEADLINE
assert status is None
def test_all_pulls_dropped_reports_timed_out_status(self):
"""When all pulls drop (none complete), status should be 'review lane timed out'."""
import time as _time
import unittest.mock
def very_slow(url):
_time.sleep(5) # Longer than the deadline
return {"records": [review_record(2, 5)]}
# Use a very short deadline so all pulls time out
with unittest.mock.patch.object(amazon, "LANE_DEADLINE", 1):
with unittest.mock.patch.object(amazon, "MIN_USEFUL_REVIEW_BUDGET", 1):
out, status = amazon.enrich_with_reviews(
self._products(2), depth="default", fetcher=very_slow,
elapsed=0.0,
)
# No products should have top_comments (all dropped)
assert all(not p.get("top_comments") for p in out)
# Status should indicate timeout
assert status == "review lane timed out"
# --------------------------------------------------------------- stats
class TestStats:
def test_five_star_share_from_the_distribution_object(self):
share = amazon.five_star_share(
{"one_star": 28, "two_star": 9, "three_star": 28, "four_star": 60, "five_star": 335}
)
assert round(share * 100) == 73
def test_five_star_share_is_none_without_a_distribution(self):
assert amazon.five_star_share({}) is None
def test_recent_window_counts_only_dated_reviews_inside_it(self):
comments = [
{"date": _days_ago(2), "rating": 4},
{"date": _days_ago(29), "rating": 4},
{"date": _days_ago(31), "rating": 1},
{"date": None, "rating": 5},
]
window = amazon.recent_window_stats(comments, today=TODAY)
assert window["recent_n"] == 2
assert window["recent_avg"] == 4
@pytest.mark.parametrize(
"ratings,expected",
[
([1, 2, 3, 4, 5], "down"), # avg 3.0 vs 4.4
([5, 5, 5, 5, 5], "up"), # avg 5.0 vs 4.4
([4, 4, 5, 4, 5], "flat"), # avg 4.4 vs 4.4
],
)
def test_drift_direction(self, ratings, expected):
product = {
"product_rating": 4.4,
"product_rating_count": 459,
"top_comments": [
{"date": _days_ago(i + 1), "rating": r} for i, r in enumerate(ratings)
],
}
assert amazon.product_stats(product, today=TODAY)["drift"] == expected
def test_below_threshold_sample_renders_quiet_not_a_drift(self):
"""Live census: a 50-cap pull can land only a handful in-window."""
product = {
"product_rating": 4.4,
"top_comments": [{"date": _days_ago(i + 1), "rating": 1} for i in range(4)],
}
assert amazon.product_stats(product, today=TODAY)["drift"] == "quiet"
def test_threshold_is_exactly_five(self):
product = {
"product_rating": 4.4,
"top_comments": [{"date": _days_ago(i + 1), "rating": 1} for i in range(5)],
}
assert amazon.product_stats(product, today=TODAY)["drift"] == "down"
def test_no_baseline_renders_new(self):
assert amazon.product_stats({"top_comments": []}, today=TODAY)["drift"] == "new"
def test_review_pull_rating_count_supersedes_the_search_record(self):
"""Search counts are variant-level and can undercount 100x."""
stats = amazon.product_stats(
{"num_ratings": 84, "product_rating_count": 8446, "product_rating": 4.7},
today=TODAY,
)
assert stats["ratings_total"] == 8446
def test_reproduces_the_live_chill_max_reading(self):
"""End-to-end against the real 2026-08-13 payload shape."""
records = (
[review_record(i, r) for i, r in ((1, 5), (2, 1), (5, 5), (9, 4), (11, 4))]
+ [review_record(200, 5), review_record(400, 5)]
)
comments, stats = amazon.parse_reviews({"records": records})
product = {"short_name": "Chill Max XL", **stats, "top_comments": comments}
out = amazon.product_stats(product, today=TODAY)
assert out["all_time"] == 4.4
assert out["ratings_total"] == 459
assert round(out["five_star_share"] * 100) == 73
assert out["recent_n"] == 5
assert out["recent_avg"] == 3.8
assert out["drift"] == "down"
# -------------------------------------------------------------- footer
class TestFooterEntry:
def test_negative_drift_gets_the_arrow_marker(self):
entry = amazon.footer_entry(
{"short_name": "Chill Max XL", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"}
)
assert entry == "Chill Max XL 4.4★→3.8★ ↓"
def test_positive_drift_gets_no_marker(self):
entry = amazon.footer_entry(
{"short_name": "Deluxe Bag", "all_time": 4.7, "recent_avg": 5.0, "drift": "up"}
)
assert entry == "Deluxe Bag 4.7★→5.0★"
assert "" not in entry
def test_quiet_state_shows_the_baseline_without_an_arrow(self):
entry = amazon.footer_entry(
{"short_name": "Genesis E-325", "all_time": 4.4, "recent_avg": None, "drift": "quiet"}
)
assert entry == "Genesis E-325 4.4★ quiet"
assert "" not in entry
def test_new_state_claims_no_baseline(self):
entry = amazon.footer_entry({"short_name": "BLUEY Set", "all_time": None, "drift": "new"})
assert entry == "BLUEY Set new"
def test_quote_renders_only_on_negative_drift(self):
sagging = amazon.footer_entry(
{"short_name": "Chill Max XL", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"},
quote="the lid jams",
)
assert sagging == 'Chill Max XL 4.4★→3.8★ ↓ "the lid jams"'
healthy = amazon.footer_entry(
{"short_name": "Deluxe Bag", "all_time": 4.7, "recent_avg": 5.0, "drift": "up"},
quote="the lid jams",
)
assert '"' not in healthy
def test_absent_quote_renders_the_clean_numeric_entry(self):
entry = amazon.footer_entry(
{"short_name": "Chill Max XL", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"},
quote="",
)
assert entry == "Chill Max XL 4.4★→3.8★ ↓"
class _Item:
"""Minimal SourceItem stand-in for the enrichment adapter."""
def __init__(self, asin, **meta):
self.source = "amazon"
self.url = f"https://www.amazon.com/dp/{asin}"
self.title = meta.get("name", asin)
self.metadata = {"asin": asin, "short_name": asin, "brand": "Bentgo", **meta}
class TestSourceItemEnrichment:
def test_reviews_and_stats_land_on_item_metadata(self):
items = [_Item("B000000001"), _Item("B000000002")]
amazon.enrich_source_items(
items, depth="default", keyword="bentgo lunch box",
fetcher=lambda url: {"records": [
review_record(i + 1, r) for i, r in enumerate([1, 1, 1, 1, 1])
]},
)
for item in items:
assert item.metadata["top_comments"]
assert item.metadata["stats"]["drift"] == "down"
def test_non_amazon_items_are_untouched(self):
other = _Item("B000000001")
other.source = "reddit"
amazon.enrich_source_items([other], depth="default", fetcher=lambda url: {"records": []})
assert "top_comments" not in other.metadata
def test_already_enriched_items_are_not_re_pulled(self):
"""enrich_source_items no-ops when top_comments is already set.
This is critical for the thin-retry path: Phase 1 enriches products at
search time, then thin retry (Phase 2b) may return the same ASINs. The
pipeline passes skip_amazon_enrichment=True in thin retry, so products
arrive at finalize without re-enrichment. Finalize calls enrich_source_items,
which skips already-enriched items (top_comments set) and only enriches
genuinely new products. This prevents duplicate Bright Data pulls.
"""
item = _Item("B000000001", top_comments=[{"excerpt": "cached", "score": 0, "rating": 5, "date": None}])
calls = []
amazon.enrich_source_items(
[item], depth="default",
fetcher=lambda url: calls.append(url) or {"records": []},
)
assert calls == []
def test_quick_depth_touches_nothing(self):
items = [_Item("B000000001")]
calls = []
amazon.enrich_source_items(
items, depth="quick",
fetcher=lambda url: calls.append(url) or {"records": []},
)
assert calls == []
assert "top_comments" not in items[0].metadata
class TestStatsFromItem:
def test_uses_the_cached_block_when_enrichment_already_ran(self):
item = _Item("B000000001", stats={"short_name": "Cached", "drift": "up"})
assert amazon.stats_from_item(item)["short_name"] == "Cached"
def test_recomputes_from_metadata_when_absent(self):
"""Mock runs and replayed fixtures skip enrichment entirely."""
item = _Item(
"B1",
product_rating=4.4,
product_rating_count=459,
star_distribution={"one_star": 28, "two_star": 9, "three_star": 28,
"four_star": 60, "five_star": 335},
top_comments=[{"date": _days_ago(i + 1), "rating": 1} for i in range(5)],
)
stats = amazon.stats_from_item(item, today=TODAY)
assert stats["drift"] == "down"
assert stats["all_time"] == 4.4
assert round(stats["five_star_share"] * 100) == 73
class _FooterItem:
def __init__(self, short_name, *, reviews=0, all_time=4.4, recent=None,
drift="quiet", quote=None):
self.source = "amazon"
self.url = "https://www.amazon.com/dp/X"
self.title = short_name
self.metadata = {
"asin": short_name,
"stats": {
"short_name": short_name, "all_time": all_time,
"recent_avg": recent, "drift": drift,
"reviews_pulled": reviews, "ratings_total": 100,
"five_star_share": 0.7, "recent_n": 5 if recent else 0,
"url": self.url,
},
}
if quote:
self.metadata["pulse_quote"] = quote
def _report(items, *, keyword="bentgo lunch box"):
from lib import schema
report = object.__new__(schema.Report)
object.__setattr__(report, "items_by_source", {"amazon": items})
object.__setattr__(report, "artifacts", {"amazon_query": keyword})
object.__setattr__(report, "source_status", {})
return report
class TestFooterLine:
def _line(self, items, **kw):
from lib import render
return render._amazon_footer_line(_report(items, **kw))
def test_only_sampled_products_get_a_slot(self):
"""A dozen discovered products must not become a dozen entries."""
items = [_FooterItem("Sampled", reviews=20, recent=3.8, drift="down")]
items += [_FooterItem(f"Unsampled{i}") for i in range(9)]
line = self._line(items)
assert "Sampled 4.4★→3.8★ ↓" in line
assert "Unsampled" not in line
# The count still reports everything discovered.
assert line.startswith("📦 Amazon: 10 products │")
def test_duplicate_variant_names_are_collapsed(self):
items = [
_FooterItem("Kids Bento", reviews=20, recent=4.9, drift="up"),
_FooterItem("Kids Bento", reviews=20, recent=4.8, drift="up"),
]
assert self._line(items).count("Kids Bento") == 1
def test_quick_depth_renders_the_inventory_form(self):
items = [_FooterItem("A"), _FooterItem("B")]
line = self._line(items)
assert "" not in line
assert "average" in line and "ratings" in line
def test_footer_renders_no_quote_today(self):
"""The engine renders this line before the model sees the report, so
there is no weave-time path for a model-written quote. Deferred."""
items = [_FooterItem("Sagging", reviews=20, recent=3.8, drift="down",
quote="the lid jams")]
line = self._line(items)
assert "" in line
assert '"' not in line
def test_footer_entry_still_accepts_a_quote_for_a_future_writer(self):
entry = amazon.footer_entry(
{"short_name": "X", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"},
quote="the lid jams",
)
assert '"the lid jams"' in entry
def test_empty_result_names_the_keyword(self):
assert self._line([]) == '📦 Amazon: no products matched "bentgo lunch box"'
def test_no_line_at_all_when_the_source_never_ran(self):
assert self._line([], keyword="") is None
def test_malformed_asin_records_are_rejected(self):
"""The ASIN is interpolated into a URL that is refetched and rendered."""
records = [
search_record(asin="../../etc/passwd", url="https://www.amazon.com/dp/x"),
search_record(asin="B0AAA0000", url="https://www.amazon.com/dp/y"), # 9 chars
search_record(asin="B0AAA000012", url="https://www.amazon.com/dp/z"), # 11 chars
search_record(asin="B0AAA00001", url="https://www.amazon.com/dp/ok"),
]
products = amazon.parse_search_response({"records": records}, "bentgo chill max lunch box")
assert [p["asin"] for p in products] == ["B0AAA00001"]
def test_canonical_url_falls_back_when_the_asin_is_malformed(self):
original = "https://www.amazon.com/dp/legit"
assert amazon.canonical_product_url(original, "not-an-asin", DOMAIN) == original
def test_option_shaped_keyword_is_rejected_before_the_cli_runs(self):
"""A leading dash would be parsed as a flag, not a search term."""
calls = []
import lib.brightdata as bd
original = bd.run_pipeline
bd.run_pipeline = lambda *a, **k: calls.append(a) or {"records": []}
try:
out = amazon.search_products("--help")
finally:
bd.run_pipeline = original
assert calls == []
assert "may not begin" in out["error"]
class TestReviewFindingRegressions:
"""Regressions caught by the correctness review pass."""
def test_brand_inference_survives_mixed_casing(self):
"""One vendor spelled two ways must not disable the guard."""
pool = [{"brand": "Bentgo"}, {"brand": "BENTGO"}, {"brand": "Umi"}]
assert amazon.infer_brand(pool, "bentgo lunch box") == "Bentgo"
def test_multi_word_brands_are_matched(self):
pool = [{"brand": "Hydro Flask"}, {"brand": "Iron Flask"}]
assert amazon.infer_brand(pool, "hydro flask water bottle") == "Hydro Flask"
def test_two_distinct_brands_in_the_keyword_stay_ambiguous(self):
pool = [{"brand": "Yeti"}, {"brand": "Stanley"}]
assert amazon.infer_brand(pool, "yeti vs stanley tumbler") == ""
def test_short_name_respects_word_boundaries_on_the_brand(self):
"""A bare startswith() ate into sub-brands and coincidental prefixes."""
assert amazon.short_name("AnkerWork B600 Video Bar", "Anker") == "AnkerWork B600"
assert amazon.short_name("Chillax Bento Lunch Box", "Chill") == "Chillax Bento"
def test_undouble_leaves_short_repeated_words_alone(self):
assert amazon.undouble("ByeBye") == "ByeBye"
assert amazon.undouble("NoNo") == "NoNo"
# Real doubling of a whole headline still repairs.
assert amazon.undouble("Best Box!Best Box!") == "Best Box!"
assert amazon.undouble("Great value for money.Great value for money.") == \
"Great value for money."
def test_lane_budget_shrinks_as_the_run_clock_advances(self):
"""The clamp was dead code until `elapsed` was threaded through."""
# Fresh run gets full LANE_DEADLINE
assert amazon._remaining_lane_budget(0.0) == amazon.LANE_DEADLINE
# Mid-run (100s elapsed) still has 180s leftover (300-100-20=180), clamped to LANE_DEADLINE
mid = amazon._remaining_lane_budget(100.0)
assert mid == amazon.LANE_DEADLINE
# Below floor (300-200-20=80 < MIN_USEFUL_REVIEW_BUDGET=90) → 0
assert amazon._remaining_lane_budget(200.0) == 0
# Way past contract → 0
assert amazon._remaining_lane_budget(295.0) == 0
def test_lane_budget_floor_prevents_doomed_pulls(self):
"""Crumb budgets return 0, not the crumbs.
This is the fix for the Bentgo bug: elapsed=269 left only 11s of budget,
causing Bright Data pulls to time out (cli_timeout = max(5, timeout-10)
1s timeout). Now any budget below MIN_USEFUL_REVIEW_BUDGET returns 0.
"""
# elapsed=269 → remaining = 300-269-20 = 11 < MIN_USEFUL=90 → 0
assert amazon._remaining_lane_budget(269.0) == 0
# Just above floor: 300-190-20=90 == MIN_USEFUL → 90 (not 0)
assert amazon._remaining_lane_budget(190.0) == amazon.MIN_USEFUL_REVIEW_BUDGET
# Just below floor: 300-191-20=89 < MIN_USEFUL=90 → 0
assert amazon._remaining_lane_budget(191.0) == 0
def test_lane_budget_constants_are_sane(self):
"""Guard against accidental constant drift breaking the logic."""
assert amazon.MIN_USEFUL_REVIEW_BUDGET == 90
assert amazon.LANE_DEADLINE == 180
assert amazon.MIN_USEFUL_REVIEW_BUDGET < amazon.LANE_DEADLINE
def test_enrichment_refreshes_the_variant_level_rating_count(self):
"""Search counts undercount badly; the pull's count is authoritative."""
item = _Item("B000000001", name="Deluxe Bag", num_ratings=84)
item.title = "Bentgo Deluxe Bag - 4.7/5 (84 ratings)"
item.engagement = {"ratings": 84}
amazon.enrich_source_items(
[item], depth="default",
fetcher=lambda url: {"records": [
review_record(i + 1, 5, product_rating=4.7, product_rating_count=8446)
for i in range(5)
]},
)
assert item.engagement["ratings"] == 8446
assert "8,446 ratings" in item.title
assert "84 ratings" not in item.title
assert item.metadata["stats"]["ratings_total"] == 8446
class TestSavedArtifactCompleteness:
"""The saved report is the copy users keep; it must not drop evidence."""
def _saved(self, sources):
from lib import render, schema
report = object.__new__(schema.Report)
for field, value in {
"topic": "bentgo", "range_from": "2026-07-14", "range_to": "2026-08-13",
"generated_at": "2026-08-13", "clusters": [], "ranked_candidates": [],
"items_by_source": sources, "errors_by_source": {}, "source_status": {},
"freshness_verdicts": [], "warnings": [], "artifacts": {},
"library_context": [], "drill_of": None,
"provider_runtime": schema.ProviderRuntime(
reasoning_provider="local", planner_model="m", rerank_model="m",
),
"query_plan": schema.QueryPlan(
intent="product", freshness_mode="balanced_recent", cluster_mode="none",
raw_topic="bentgo", subqueries=[],
source_weights={s: 1.0 for s in sources},
),
}.items():
object.__setattr__(report, field, value)
return render.render_full(report)
def _item(self, source, item_id, engagement):
from lib import schema
return schema.SourceItem(
item_id=item_id, source=source, title=f"{source} item", body="b",
url="https://example.com", author="A", container=None,
published_at="2026-08-10", date_confidence="high",
engagement=engagement, relevance_hint=0.5, why_relevant="",
snippet="", metadata={},
)
def test_amazon_items_appear_in_the_per_source_dump(self):
"""Regression: a hardcoded source list silently dropped this section."""
out = self._saved({"amazon": [self._item("amazon", "B0AAA00001", {"ratings": 459})]})
assert "### Amazon (1 items)" in out
assert "B0AAA00001" in out
def test_a_source_absent_from_the_fixed_order_still_renders(self):
"""The list is display order, not the source registry."""
out = self._saved({"trustpilot": [self._item("trustpilot", "TP1", {"reviews": 12})]})
assert "TP1" in out
def test_engagement_is_not_blank_for_a_non_allowlisted_metric(self):
out = self._saved({"amazon": [self._item("amazon", "B0AAA00001", {"ratings": 459})]})
assert "459 ratings" in out
def test_allowlisted_sources_keep_their_existing_engagement_format(self):
"""The fall-through must not add previously-unshown keys."""
out = self._saved({"reddit": [self._item(
"reddit", "R1", {"score": 120, "num_comments": 48, "upvote_ratio": 0.91}
)]})
assert "120 score, 48 num_comments" in out
assert "upvote_ratio" not in out
+118
View File
@@ -46,6 +46,28 @@ def test_search_query_strips_inner_quotes():
assert q == 'all:"say hello world"'
def test_unquoted_fallback_search_args_conjoin_terms_at_command_boundary():
args = arxiv._build_search_args("AI video generation advances", 10, quoted=False)
assert args == [
"arxiv-pp-cli",
"query",
"--search-query",
'all:"AI" AND all:"video" AND all:"generation" AND all:"advances"',
"--sort-by",
"relevance",
"--max-results",
"10",
"--agent",
]
def test_unquoted_fallback_quotes_operator_and_colon_terms_at_command_boundary():
args = arxiv._build_search_args("AI AND OR title:video", 5, quoted=False)
assert args[args.index("--search-query") + 1] == (
'all:"AI" AND all:"AND" AND all:"OR" AND all:"title:video"'
)
# ---- envelope extraction ----
def test_extract_entries_handles_nested_results_envelope():
@@ -165,3 +187,99 @@ def test_run_cli_bad_json_returns_error(monkeypatch):
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", lambda cmd, timeout: _Proc(0, "not json"))
resp = arxiv.search_arxiv("topic", "2026-06-01", "2026-06-27")
assert resp["results"] == [] and "error" in resp
def test_empty_stdout_returns_error_without_retry(monkeypatch):
monkeypatch.setattr(arxiv, "_is_available", lambda: True)
calls = []
def fake_run(cmd, timeout):
calls.append(cmd)
return _Proc(0, "")
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", fake_run)
resp = arxiv.search_arxiv("topic", "2026-06-01", "2026-06-27")
assert resp == {"results": [], "error": "empty stdout"}
assert len(calls) == 1
def test_unrecognized_json_returns_error_without_retry(monkeypatch):
monkeypatch.setattr(arxiv, "_is_available", lambda: True)
calls = []
def fake_run(cmd, timeout):
calls.append(cmd)
return _Proc(0, '{"status":"ok"}')
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", fake_run)
resp = arxiv.search_arxiv("topic", "2026-06-01", "2026-06-27")
assert resp == {"results": [], "error": "unrecognized JSON response"}
assert len(calls) == 1
def test_recognized_empty_list_retries(monkeypatch):
monkeypatch.setattr(arxiv, "_is_available", lambda: True)
calls = []
def fake_run(cmd, timeout):
calls.append(cmd)
return _Proc(0, "[]")
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", fake_run)
resp = arxiv.search_arxiv("topic", "2026-06-01", "2026-06-27")
assert resp == {"results": []}
assert len(calls) == 2
# ---- unquoted-retry on zero results (#908) ----
def test_zero_result_quoted_query_retries_unquoted_and_finds_results(monkeypatch):
"""A natural-language multi-word topic matches nothing as an exact
phrase, but the unquoted retry finds it -- the fix for #908."""
monkeypatch.setattr(arxiv, "_is_available", lambda: True)
calls = []
def fake_run(cmd, timeout):
calls.append(cmd)
query = cmd[cmd.index("--search-query") + 1]
if query == 'all:"AI video generation advances"':
return _Proc(0, '{"results":{"entries":[]}}')
return _Proc(0, '{"results":{"entries":[{"title":"AI video generation advances"}]}}')
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", fake_run)
resp = arxiv.search_arxiv("AI video generation advances", "2026-06-01", "2026-06-27")
assert resp["results"] == [{"title": "AI video generation advances"}]
assert len(calls) == 2
assert 'all:"AI video generation advances"' in calls[0]
assert 'all:"AI" AND all:"video" AND all:"generation" AND all:"advances"' in calls[1]
def test_zero_result_quoted_query_retry_also_empty_returns_empty(monkeypatch):
monkeypatch.setattr(arxiv, "_is_available", lambda: True)
calls = []
def fake_run(cmd, timeout):
calls.append(cmd)
return _Proc(0, '{"results":{"entries":[]}}')
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", fake_run)
resp = arxiv.search_arxiv("truly obscure nonsense topic", "2026-06-01", "2026-06-27")
assert resp["results"] == []
assert "error" not in resp
assert len(calls) == 2
def test_real_cli_error_does_not_trigger_unquoted_retry(monkeypatch):
"""A genuine failure (nonzero exit) must not retry -- only a clean
zero-result success should (R6)."""
monkeypatch.setattr(arxiv, "_is_available", lambda: True)
calls = []
def fake_run(cmd, timeout):
calls.append(cmd)
return _Proc(1, "", "boom")
monkeypatch.setattr(arxiv.subproc, "run_with_timeout", fake_run)
resp = arxiv.search_arxiv("topic", "2026-06-01", "2026-06-27")
assert resp["results"] == [] and "boom" in resp["error"]
assert len(calls) == 1
+419 -8
View File
@@ -23,7 +23,7 @@ from unittest import mock
import pytest
from lib import backends, env, health, xurl_x
from lib import backends, env, grok_x, health, xurl_x
# ---------------------------------------------------------------------------
@@ -54,6 +54,9 @@ def _x_env(
xurl_installed=False,
xurl_authed=False,
node_status=health.OK,
grok_installed=False,
grok_authed=False,
grok_expired=False,
):
"""Context managers configuring the X-chain probe environment.
@@ -61,20 +64,56 @@ def _x_env(
network check (``is_available``) and the doctor-path local evidence
(``stored_auth_status``/``has_stored_auth``) a real machine where the
user logged in has both.
``grok_expired`` simulates an expired session: AUTH_EXPIRED status, but
has_stored_auth/is_available still return True (refresh may work).
"""
from datetime import datetime, timezone, timedelta
stored = (
(xurl_x.AUTH_OK, "stored OAuth credentials found in ~/.xurl")
if xurl_authed
else (xurl_x.AUTH_MISSING, "no token store at ~/.xurl")
)
if grok_expired:
past = datetime.now(timezone.utc) - timedelta(hours=2)
grok_stored = (
grok_x.AUTH_EXPIRED,
f"Grok session expired at {past.isoformat()}",
past,
)
grok_available = grok_installed
elif grok_authed:
grok_stored = (
grok_x.AUTH_OK,
"stored Grok credentials found in ~/.grok/auth.json",
None,
)
grok_available = grok_installed
else:
grok_stored = (
grok_x.AUTH_MISSING,
"no Grok credential store at ~/.grok/auth.json",
None,
)
grok_available = False
return (
mock.patch("lib.bird_x.is_bird_installed", return_value=bird_installed),
mock.patch("lib.bird_x.set_credentials", lambda *a, **k: None),
mock.patch("lib.xurl_x.is_available", return_value=xurl_authed),
mock.patch(
"lib.backends.which",
lambda name: "/usr/local/bin/xurl" if (name == "xurl" and xurl_installed) else None,
lambda name: (
"/usr/local/bin/xurl" if (name == "xurl" and xurl_installed)
else "/usr/local/bin/grok" if (name == "grok" and grok_installed)
else None
),
),
mock.patch("lib.grok_x.stored_auth_status", return_value=grok_stored),
mock.patch(
"lib.grok_x.has_stored_auth",
return_value=grok_installed and (grok_authed or grok_expired),
),
mock.patch("lib.grok_x.is_available", return_value=grok_available),
mock.patch("lib.health.probe_dependency", _probe_dep({"node": node_status})),
mock.patch("lib.xurl_x.stored_auth_status", return_value=stored),
mock.patch(
@@ -114,8 +153,21 @@ class TestDescriptorRegistry:
def test_x_chain_comes_from_env_definitions(self):
d = backends.get_descriptor("x")
assert d.mode == backends.MODE_ALTERNATIVE
assert tuple(s.name for s in d.backends) == env.X_BACKEND_ORDER
assert env.X_BACKEND_ORDER == ("xai", "bird", "xurl", "xquik")
# Auto chain order: bird first, grok excluded (opt-in only).
assert env.X_BACKEND_ORDER == ("bird", "xai", "xurl", "xquik")
# Grok is opt-in only, not in the auto chain.
assert env.X_BACKEND_OPT_IN == ("grok",)
# All known backends (auto + opt-in) for pin validation.
assert env.X_BACKEND_KNOWN == ("bird", "xai", "xurl", "xquik", "grok")
# Descriptor includes all backends (auto + opt-in) for doctor visibility.
assert tuple(s.name for s in d.backends) == env.X_BACKEND_ORDER + env.X_BACKEND_OPT_IN
# Grok is marked opt-in in the descriptor.
grok_spec = next(s for s in d.backends if s.name == "grok")
assert grok_spec.opt_in is True
# Auto chain backends are NOT marked opt-in.
for name in env.X_BACKEND_ORDER:
spec = next(s for s in d.backends if s.name == name)
assert spec.opt_in is False
assert d.pin_var == env.X_BACKEND_PIN_VAR == "LAST30DAYS_X_BACKEND"
def test_env_exposes_reddit_pin_constants(self):
@@ -158,11 +210,49 @@ class TestXPrediction:
assert res.active_backend == "bird"
assert res.tier == backends.TIER_OK
assert res.pinned is False
# Chain rendered in declared order regardless of availability.
assert res.chain == list(env.X_BACKEND_ORDER)
assert [f.name for f in res.findings] == list(env.X_BACKEND_ORDER)
# Chain includes all backends (auto + opt-in) for doctor visibility.
expected_chain = list(env.X_BACKEND_ORDER + env.X_BACKEND_OPT_IN)
assert res.chain == expected_chain
assert [f.name for f in res.findings] == expected_chain
assert "will use: bird" in res.summary
def test_bird_predicted_even_when_xai_key_present(self):
"""Cookies beat XAI_API_KEY when both are present (bird-first chain)."""
config = {"AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0", "XAI_API_KEY": "dummy-key"}
res = _resolve_x(config, bird_installed=True)
assert res.active_backend == "bird"
assert res.tier == backends.TIER_OK
assert "will use: bird" in res.summary
def test_grok_is_never_auto_selected_unpinned(self):
"""Grok is opt-in only: even if grok is the only configured backend, X is unconfigured unpinned."""
config = {}
res = _resolve_x(config, grok_installed=True, grok_authed=True)
# Grok is available but opt-in - should NOT be auto-selected.
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.OK
# But it should not be the active backend.
assert res.active_backend is None
assert res.tier == backends.TIER_ERROR
def test_grok_selected_when_pinned(self):
"""Pin grok to enable it explicitly."""
config = {"LAST30DAYS_X_BACKEND": "grok"}
res = _resolve_x(config, grok_installed=True, grok_authed=True)
assert res.active_backend == "grok"
assert res.pinned is True
assert res.pin == "grok"
assert res.tier == backends.TIER_OK
def test_grok_pin_with_no_store_is_error(self):
"""Pin grok without a valid store -> error with grok login prescription."""
config = {"LAST30DAYS_X_BACKEND": "grok"}
res = _resolve_x(config, grok_installed=True, grok_authed=False)
assert res.active_backend is None
assert res.pinned is True
assert res.tier == backends.TIER_ERROR
assert "grok login" in res.prescription.lower()
# Scenario 2: pin var set to a later backend -> honored + marked pinned.
def test_pin_to_later_backend_honored_and_marked(self):
config = {
@@ -194,7 +284,9 @@ class TestXPrediction:
res = _resolve_x({})
assert res.active_backend is None
assert res.tier == backends.TIER_ERROR
assert "XAI_API_KEY" in res.prescription
# bird (cookies) is first in the chain, so the prescription is about
# browser cookies, not XAI_API_KEY.
assert "browser-cookie" in res.prescription or "cookies" in res.prescription.lower()
def test_pinned_but_unusable_backend_is_error_with_its_prescription(self):
# Pin bird without cookies: env.x_backend_chain returns [] (pipeline
@@ -245,6 +337,98 @@ class TestXPrediction:
assert bird.status == health.BROKEN
assert "node" in bird.prescription.lower()
def test_grok_expired_is_degraded_not_ok(self):
"""Expired grok session -> DEGRADED tier (warn), not OK."""
# Grok is opt-in: needs explicit pin to be selected.
config = {"LAST30DAYS_X_BACKEND": "grok"}
res = _resolve_x(config, grok_installed=True, grok_expired=True)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.DEGRADED
assert grok.usable # DEGRADED is still usable (refresh may work)
assert "expired" in grok.detail.lower()
assert "grok login" in grok.prescription.lower()
# With pin, grok is selected (degraded is usable).
assert res.active_backend == "grok"
assert res.tier == backends.TIER_WARN
def test_grok_expired_unpinned_not_selected(self):
"""Expired grok without pin: X unconfigured, grok not auto-selected."""
res = _resolve_x({}, grok_installed=True, grok_expired=True)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.DEGRADED
# Grok is opt-in, so even though it's usable (degraded), it's not selected.
assert res.active_backend is None
assert res.tier == backends.TIER_ERROR
def test_grok_expired_with_fallback_picks_fallback(self):
"""When grok is expired AND a better auto-chain backend is OK, pick the OK one."""
config = {"AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0"}
res = _resolve_x(config, grok_installed=True, grok_expired=True, bird_installed=True)
# Bird is OK and in the auto chain. Grok is not considered (opt-in).
assert res.active_backend == "bird"
assert res.tier == backends.TIER_OK
def test_grok_future_expires_at_is_ok(self):
"""Grok with future expires_at reports OK."""
res = _resolve_x({}, grok_installed=True, grok_authed=True)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.OK
assert "not live-verified" in grok.detail
# ---------------------------------------------------------------------------
# Grok session expiry: three states (grok is opt-in only)
# ---------------------------------------------------------------------------
class TestGrokExpiryStates:
"""Test the three grok auth states from the plan:
1. No grok CLI -> silent fallback (opt-in only)
2. CLI installed, never logged in -> silent fallback (opt-in only)
3. CLI installed, WAS logged in, session dead -> DEGRADED with expiry info (opt-in only)
Note: Grok is opt-in only. These tests verify the finding status, but grok
is never auto-selected unpinned.
"""
def test_no_grok_cli_is_missing(self):
"""No grok CLI -> MISSING status, no extra failure noise."""
res = _resolve_x({}, grok_installed=False)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.MISSING
assert "not found on PATH" in grok.detail
# Grok is opt-in, so even MISSING doesn't affect the resolution.
# X is unconfigured (no auto-chain backends available).
assert res.active_backend is None
def test_grok_installed_never_logged_in_is_missing(self):
"""CLI installed but never logged in -> MISSING with login hint."""
res = _resolve_x({}, grok_installed=True, grok_authed=False)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.MISSING
assert "not signed in" in grok.detail
assert "grok login" in grok.prescription
# Grok is opt-in, so X is unconfigured.
assert res.active_backend is None
def test_grok_session_expired_is_degraded_with_expiry(self):
"""Session expired -> DEGRADED with timestamp and refresh hint."""
res = _resolve_x({}, grok_installed=True, grok_expired=True)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.DEGRADED
assert "expired" in grok.detail.lower()
# The detail should include the expiry timestamp and hint
assert "refresh" in grok.detail.lower() or "login" in grok.prescription.lower()
# Grok is opt-in, so X is unconfigured even with degraded grok.
assert res.active_backend is None
def test_grok_healthy_session_is_ok(self):
"""Non-expired credentials -> OK, but still opt-in only."""
res = _resolve_x({}, grok_installed=True, grok_authed=True)
grok = next(f for f in res.findings if f.name == "grok")
assert grok.status == health.OK
# Grok is opt-in, so X is unconfigured unpinned.
assert res.active_backend is None
# ---------------------------------------------------------------------------
# Scenario 5: paid lanes probe key presence only — never network/subprocess
@@ -497,6 +681,233 @@ class TestXParityWithPipeline:
def test_parity_nothing_configured(self):
self._assert_parity({})
def test_parity_grok_only_unpinned_is_unconfigured(self):
"""Grok-only with no pin: X unconfigured (parity with env.x_backend_chain)."""
self._assert_parity({}, grok_installed=True, grok_authed=True)
def test_parity_grok_pinned(self):
"""Grok pinned: grok is selected (parity with env.x_backend_chain)."""
self._assert_parity(
{"LAST30DAYS_X_BACKEND": "grok"},
grok_installed=True,
grok_authed=True,
)
def test_parity_cookies_beat_xai_key(self):
"""Cookies beat XAI_API_KEY when both present (bird-first chain)."""
self._assert_parity(
{"AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0", "XAI_API_KEY": "dummy-key"},
bird_installed=True,
)
# ---------------------------------------------------------------------------
# get_x_source_status pin semantics (R4): grok pin forces grok source
# ---------------------------------------------------------------------------
class TestGetXSourceStatusGrokPin:
"""get_x_source_status must respect LAST30DAYS_X_BACKEND=grok pin."""
def test_pin_grok_with_store_returns_grok_source(self):
"""Pin grok + valid store -> get_x_source_status source is 'grok'."""
config = {"LAST30DAYS_X_BACKEND": "grok"}
bird_status = {
"installed": False,
"authenticated": False,
"username": "",
"can_install": False,
}
with (
mock.patch("lib.grok_x.has_stored_auth", return_value=True),
mock.patch("lib.bird_x.get_bird_status", return_value=bird_status),
):
status = env.get_x_source_status(config, probe=False)
assert status["source"] == "grok"
assert status["grok_available"] is True
def test_unpinned_with_store_does_not_return_grok_source(self):
"""Unpinned + valid grok store -> source is NOT 'grok' (opt-in only)."""
config = {} # No pin
bird_status = {
"installed": False,
"authenticated": False,
"username": "",
"can_install": False,
}
with (
mock.patch("lib.grok_x.has_stored_auth", return_value=True),
mock.patch("lib.bird_x.get_bird_status", return_value=bird_status),
):
status = env.get_x_source_status(config, probe=False)
# Grok is available but NOT the source (opt-in only)
assert status["source"] != "grok"
assert status["source"] is None # No other backend configured
assert status["grok_available"] is True
def test_pin_grok_with_cookies_still_returns_grok(self):
"""Pin grok with cookies present -> grok (pin forces single backend)."""
config = {
"LAST30DAYS_X_BACKEND": "grok",
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
}
bird_status = {
"installed": True,
"authenticated": True,
"username": "test",
"can_install": True,
}
with (
mock.patch("lib.grok_x.has_stored_auth", return_value=True),
mock.patch("lib.bird_x.get_bird_status", return_value=bird_status),
):
status = env.get_x_source_status(config, probe=False)
# Pin forces grok even when bird is available
assert status["source"] == "grok"
def test_pin_grok_no_store_with_other_creds_returns_none(self):
"""Pin grok + no store + other creds present -> source is None (exclusive pin)."""
config = {
"LAST30DAYS_X_BACKEND": "grok",
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
"XAI_API_KEY": "dummy-key",
}
bird_status = {
"installed": True,
"authenticated": True,
"username": "test",
"can_install": True,
}
with (
mock.patch("lib.grok_x.has_stored_auth", return_value=False),
mock.patch("lib.bird_x.get_bird_status", return_value=bird_status),
):
status = env.get_x_source_status(config, probe=False)
# Pin is exclusive: grok unavailable -> None, NOT fallback to bird/xai
assert status["source"] is None
assert status["grok_available"] is False
# Other backends ARE available, but pin blocks fallback
assert status["bird_authenticated"] is True
assert status["xai_available"] is True
def test_pin_xai_with_cookies_returns_xai(self):
"""Pin xai + cookies + XAI_API_KEY -> source is xai, not bird."""
config = {
"LAST30DAYS_X_BACKEND": "xai",
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
"XAI_API_KEY": "dummy-key",
}
bird_status = {
"installed": True,
"authenticated": True,
"username": "test",
"can_install": True,
}
with (
mock.patch("lib.grok_x.has_stored_auth", return_value=False),
mock.patch("lib.bird_x.get_bird_status", return_value=bird_status),
):
status = env.get_x_source_status(config, probe=False)
# Pin is exclusive: xai pinned + available -> xai (not bird)
assert status["source"] == "xai"
# Bird is also available, but pin forces xai
assert status["bird_authenticated"] is True
def test_pin_xai_no_key_with_cookies_returns_none(self):
"""Pin xai + no XAI_API_KEY + cookies -> source is None (exclusive pin)."""
config = {
"LAST30DAYS_X_BACKEND": "xai",
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
# No XAI_API_KEY
}
bird_status = {
"installed": True,
"authenticated": True,
"username": "test",
"can_install": True,
}
with (
mock.patch("lib.grok_x.has_stored_auth", return_value=False),
mock.patch("lib.bird_x.get_bird_status", return_value=bird_status),
):
status = env.get_x_source_status(config, probe=False)
# Pin is exclusive: xai pinned but unavailable -> None (no fallback)
assert status["source"] is None
assert status["xai_available"] is False
# Bird is available but pin blocks fallback
assert status["bird_authenticated"] is True
# ---------------------------------------------------------------------------
# Runtime X backend pin (x_backend_chain / _resolve_x_backend)
# ---------------------------------------------------------------------------
class TestRuntimeXBackendPin:
"""Runtime fetch path must honor any known pin exclusively, including grok."""
def test_pin_grok_with_store_and_cookies_returns_grok(self):
"""Pin grok + grok store + cookies -> runtime returns grok, not bird."""
from lib import grok_x, providers
config = {
"LAST30DAYS_X_BACKEND": "grok",
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
"XAI_API_KEY": "dummy-key",
}
with (
mock.patch.object(grok_x, "has_stored_auth", return_value=True),
mock.patch("lib.bird_x.is_bird_installed", return_value=True),
):
# x_backend_chain is the authoritative runtime path
chain = env.x_backend_chain(config)
# _resolve_x_backend delegates to get_x_source (wraps x_backend_chain)
resolved = providers._resolve_x_backend(config)
# Pin grok + available -> grok (not bird/xai)
assert chain == ["grok"]
assert resolved == "grok"
def test_pin_grok_no_store_with_cookies_returns_none(self):
"""Pin grok + no store + cookies -> runtime returns None (exclusive pin)."""
from lib import grok_x, providers
config = {
"LAST30DAYS_X_BACKEND": "grok",
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
}
with (
mock.patch.object(grok_x, "has_stored_auth", return_value=False),
mock.patch("lib.bird_x.is_bird_installed", return_value=True),
):
chain = env.x_backend_chain(config)
resolved = providers._resolve_x_backend(config)
# Pin grok + unavailable -> [] / None (no fallthrough to bird)
assert chain == []
assert resolved is None
def test_unpinned_with_grok_store_and_cookies_returns_bird(self):
"""Unpinned + grok store + cookies -> runtime returns bird, never grok."""
from lib import grok_x, providers
config = {
"AUTH_TOKEN": "dummy-token",
"CT0": "dummy-ct0",
}
with (
mock.patch.object(grok_x, "has_stored_auth", return_value=True),
mock.patch("lib.bird_x.is_bird_installed", return_value=True),
):
chain = env.x_backend_chain(config)
resolved = providers._resolve_x_backend(config)
# Unpinned -> auto-chain (bird first), grok never auto-selected
assert chain[0] == "bird"
assert "grok" not in chain
assert resolved == "bird"
# ---------------------------------------------------------------------------
# YouTube chain: yt-dlp -> ScrapeCreators
+313
View File
@@ -0,0 +1,313 @@
"""Tests for the Bright Data CLI adapter (U1).
Covers the activation gate (PATH + presence-only credential probe), the
never-raise error envelope across every failure mode, and the verbatim
passthrough of the CLI's own auth/credit warnings.
No test in this file spawns a real subprocess or touches the network.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import brightdata, subproc # noqa: E402
class _Result:
"""Stand-in for subproc.SubprocResult."""
def __init__(self, returncode=0, stdout="", stderr=""):
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
# One real search record, trimmed. Field names verified against a live
# amazon_product_search payload on 2026-08-13.
SEARCH_RECORD = {
"asin": "B0DWD47BW7",
"url": "https://www.amazon.com/dp/B0DWD47BW7",
"name": "Chill Max Leak-Proof XL Bento-Style Lunch Box | Ice Pack Included",
"brand": "Bentgo",
"sponsored": "false",
"rating": 4.4,
"num_ratings": 459,
"final_price": 39.99,
"currency": "USD",
}
# ---------------------------------------------------------------- gate
def test_gate_false_when_binary_missing(monkeypatch):
monkeypatch.setattr(brightdata.shutil, "which", lambda _: None)
assert brightdata.is_installed() is False
assert brightdata.is_available({}) is False
def test_gate_reports_unauthenticated_when_no_credential_signal(monkeypatch, tmp_path):
monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")
monkeypatch.setattr(brightdata, "_config_dir", lambda: tmp_path / "absent")
status = brightdata.gate_status({})
assert status["brightdata_installed"] is True
assert status["brightdata_authenticated"] is False
assert brightdata.is_available({}) is False
def test_api_key_alone_satisfies_the_credential_probe(monkeypatch, tmp_path):
monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")
monkeypatch.setattr(brightdata, "_config_dir", lambda: tmp_path / "absent")
assert brightdata.is_available({"BRIGHTDATA_API_KEY": "dummy-key-not-real"}) is True
def test_credentials_file_satisfies_the_probe_without_being_read(monkeypatch, tmp_path):
cfg = tmp_path / "brightdata-cli"
cfg.mkdir()
creds = cfg / "credentials.json"
creds.write_text('{"token": "dummy-not-real"}')
creds.chmod(0o600)
monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")
monkeypatch.setattr(brightdata, "_config_dir", lambda: cfg)
def _explode(*a, **k): # pragma: no cover - must never run
raise AssertionError("credential contents must never be read")
monkeypatch.setattr(Path, "read_text", _explode)
assert brightdata.has_credentials({}) is True
def test_gate_does_not_spawn_a_subprocess(monkeypatch, tmp_path):
monkeypatch.setattr(brightdata.shutil, "which", lambda _: None)
def _explode(*a, **k): # pragma: no cover - must never run
raise AssertionError("gate must stay offline")
monkeypatch.setattr(subproc, "run_with_timeout", _explode)
brightdata.gate_status({})
assert brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)["records"] == []
# ------------------------------------------------------------ envelope
def _installed(monkeypatch):
monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")
def test_happy_path_parses_a_bare_json_array(monkeypatch):
"""Live payloads are a top-level list, not a {"results": ...} envelope."""
_installed(monkeypatch)
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda *a, **k: _Result(stdout=json.dumps([SEARCH_RECORD, SEARCH_RECORD])),
)
out = brightdata.run_pipeline("amazon_product_search", ["bentgo", "https://www.amazon.com"], timeout=60)
assert "error" not in out
assert len(out["records"]) == 2
assert out["records"][0]["asin"] == "B0DWD47BW7"
@pytest.mark.parametrize("wrapper", ["records", "results", "data"])
def test_dict_envelopes_are_tolerated_for_cli_churn(monkeypatch, wrapper):
_installed(monkeypatch)
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda *a, **k: _Result(stdout=json.dumps({wrapper: [SEARCH_RECORD]})),
)
assert len(brightdata.run_pipeline("amazon_product", ["u"], timeout=30)["records"]) == 1
def test_auth_401_surfaces_the_cli_error_without_raising(monkeypatch):
_installed(monkeypatch)
stderr = "Triggering pipeline collection...\nError: 401 Unauthorized - run `brightdata login`"
monkeypatch.setattr(
subproc, "run_with_timeout", lambda *a, **k: _Result(returncode=1, stderr=stderr)
)
out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)
assert out["records"] == []
assert "401" in out["error"]
def test_error_uses_the_last_stderr_line_not_the_polling_narration(monkeypatch):
"""The CLI narrates polling on stderr; the failure is the final line."""
_installed(monkeypatch)
stderr = (
"Triggering pipeline collection for amazon_product_reviews...\n"
"Status: running - polling again (attempt 1/600)\n"
"Error: snapshot failed"
)
monkeypatch.setattr(
subproc, "run_with_timeout", lambda *a, **k: _Result(returncode=1, stderr=stderr)
)
assert brightdata.run_pipeline("amazon_product_reviews", ["u", "50"], timeout=30)["error"] == "Error: snapshot failed"
def test_subprocess_timeout_returns_empty_records_and_error(monkeypatch):
_installed(monkeypatch)
def _timeout(*a, **k):
raise subproc.SubprocTimeout("Command brightdata timed out after 180s")
monkeypatch.setattr(subproc, "run_with_timeout", _timeout)
out = brightdata.run_pipeline("amazon_product_reviews", ["u", "50"], timeout=180)
assert out["records"] == []
assert "timed out" in out["error"]
def test_malformed_json_returns_empty_records_and_error(monkeypatch):
_installed(monkeypatch)
monkeypatch.setattr(subproc, "run_with_timeout", lambda *a, **k: _Result(stdout="not json{"))
out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)
assert out["records"] == []
assert "json decode" in out["error"]
def test_empty_stdout_is_not_an_error(monkeypatch):
_installed(monkeypatch)
monkeypatch.setattr(subproc, "run_with_timeout", lambda *a, **k: _Result(stdout=" \n"))
out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)
assert out["records"] == []
assert "error" not in out
def test_spawn_failures_never_raise(monkeypatch):
_installed(monkeypatch)
for exc in (FileNotFoundError("no binary"), OSError("spawn failed")):
def _raise(*a, __e=exc, **k):
raise __e
monkeypatch.setattr(subproc, "run_with_timeout", _raise)
out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)
assert out["records"] == [] and out["error"]
# ------------------------------------------------------------ arguments
def test_cli_timeout_sits_below_the_subprocess_timeout(monkeypatch):
"""The CLI must fail cleanly on its own before we SIGTERM it."""
_installed(monkeypatch)
seen = {}
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda cmd, **k: seen.update(cmd=cmd, timeout=k["timeout"]) or _Result(stdout="[]"),
)
brightdata.run_pipeline("amazon_product_reviews", ["u", "50"], timeout=180)
cli_timeout = int(seen["cmd"][seen["cmd"].index("--timeout") + 1])
assert cli_timeout < seen["timeout"] == 180
def test_params_are_passed_positionally_in_order(monkeypatch):
_installed(monkeypatch)
seen = {}
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda cmd, **k: seen.update(cmd=cmd) or _Result(stdout="[]"),
)
brightdata.run_pipeline("amazon_product_reviews", ["https://a/dp/X", 50], timeout=60)
cmd = seen["cmd"]
assert cmd[:3] == ["brightdata", "pipelines", "amazon_product_reviews"]
# Positionals sit after the option fence, in the order supplied.
assert cmd[cmd.index("--") + 1:] == ["https://a/dp/X", "50"]
assert "--json" in cmd
def test_api_key_travels_in_the_environment_never_in_argv(monkeypatch):
"""argv is not a secret channel: /proc/<pid>/cmdline is world-readable."""
_installed(monkeypatch)
seen = {}
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda cmd, **k: seen.update(cmd=cmd, env=k.get("env")) or _Result(stdout="[]"),
)
monkeypatch.delenv("BRIGHTDATA_API_KEY", raising=False)
brightdata.run_pipeline(
"amazon_product", ["u"], timeout=30, config={"BRIGHTDATA_API_KEY": "dummy-key"}
)
assert "-k" not in seen["cmd"]
assert "dummy-key" not in " ".join(seen["cmd"])
assert seen["env"]["BRIGHTDATA_API_KEY"] == "dummy-key"
def test_no_key_means_the_child_simply_inherits_the_parent_env(monkeypatch):
_installed(monkeypatch)
seen = {}
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda cmd, **k: seen.update(env=k.get("env")) or _Result(stdout="[]"),
)
brightdata.run_pipeline("amazon_product", ["u"], timeout=30, config={})
assert seen["env"] is None
def test_positional_params_are_fenced_behind_a_double_dash(monkeypatch):
"""A keyword beginning with '-' must not be parsed as an option."""
_installed(monkeypatch)
seen = {}
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda cmd, **k: seen.update(cmd=cmd) or _Result(stdout="[]"),
)
brightdata.run_pipeline("amazon_product_search", ["--help", "https://a"], timeout=30)
cmd = seen["cmd"]
assert "--" in cmd
assert cmd.index("--") < cmd.index("--help")
def test_the_key_is_scrubbed_out_of_surfaced_stderr(monkeypatch):
"""Auth failures are exactly where a CLI echoes the rejected key back."""
_installed(monkeypatch)
logged = []
monkeypatch.setattr(brightdata.log, "source_log", lambda n, m, **k: logged.append(m))
monkeypatch.setattr(
subproc, "run_with_timeout",
lambda *a, **k: _Result(returncode=1, stderr="Error: 401 for key dummy-key"),
)
out = brightdata.run_pipeline(
"amazon_product", ["u"], timeout=30, config={"BRIGHTDATA_API_KEY": "dummy-key"}
)
assert "dummy-key" not in out["error"]
assert not any("dummy-key" in m for m in logged)
# ------------------------------------------------------------- warnings
def test_credit_and_auth_warnings_pass_through_verbatim(monkeypatch):
_installed(monkeypatch)
warned = []
monkeypatch.setattr(brightdata.log, "source_log", lambda name, msg, **k: warned.append(msg))
stderr = (
"Triggering pipeline collection for amazon_product_search...\n"
"Status: running - polling again (attempt 1/600)\n"
"Warning: only 12 credits remaining on your free tier\n"
)
monkeypatch.setattr(
subproc, "run_with_timeout", lambda *a, **k: _Result(stdout="[]", stderr=stderr)
)
brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)
assert "Warning: only 12 credits remaining on your free tier" in warned
# Routine polling narration stays out of the log.
assert not any("polling again" in m for m in warned)
def test_source_log_is_called_with_tty_only_false(monkeypatch):
"""Repo rule: tty_only=True silently drops output in non-TTY hosts."""
_installed(monkeypatch)
kwargs = {}
monkeypatch.setattr(
brightdata.log, "source_log", lambda name, msg, **k: kwargs.update(k)
)
monkeypatch.setattr(
subproc, "run_with_timeout", lambda *a, **k: _Result(returncode=1, stderr="Error: 401")
)
brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)
assert kwargs.get("tty_only") is False
+74
View File
@@ -4,11 +4,15 @@ from __future__ import annotations
import importlib.util
import json
import os
import re
import subprocess
import tempfile
import unittest
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[1]
@@ -45,6 +49,41 @@ def _assert_run_blocks_indented(text: str, *, label: str) -> None:
in_run = False
def _tag_release_workflow_text() -> str:
return (ROOT / ".github" / "workflows" / "tag-release.yml").read_text(
encoding="utf-8"
)
def _tag_release_version_sed_expr(workflow_text: str) -> str:
"""Extract the sed expression from tag-release.yml's VERSION pipeline."""
match = re.search(
r'printf \'%s\\n\' "\$\{HEAD_MSG\}" \| sed -n \'([^\']+)\' \| head -n1',
workflow_text,
)
if not match:
raise AssertionError(
"Could not find VERSION sed pipeline in tag-release.yml"
)
return match.group(1)
def _parse_release_version_from_message(message: str, sed_expr: str) -> str:
"""Run the same sed pipeline tag-release.yml uses to extract VERSION."""
result = subprocess.run(
[
"bash",
"-c",
f"printf '%s\\n' \"${{HEAD_MSG}}\" | sed -n '{sed_expr}' | head -n1",
],
capture_output=True,
text=True,
check=True,
env={**os.environ, "HEAD_MSG": message},
)
return result.stdout.strip()
def _load_prepare_release():
path = ROOT / ".github" / "scripts" / "prepare_release.py"
spec = importlib.util.spec_from_file_location("prepare_release", path)
@@ -90,6 +129,41 @@ class TestChangelogWorkflow(unittest.TestCase):
):
self.assertTrue((workflows / name).is_file(), msg=name)
def test_tag_release_workflow_yaml_parses(self) -> None:
"""Bare ``chore(release):`` in an unquoted ``if:`` breaks Actions YAML.
GitHub then reports the run as failed with zero jobs on every push to
main. The expression must be double-quoted; prefer ``contains`` so
merge-commit messages still match.
"""
text = _tag_release_workflow_text()
yaml.safe_load(text)
self.assertRegex(
text,
r'(?m)^\s+if:\s+"contains\(github\.event\.head_commit\.message, '
r"'chore\(release\): bump version to '\)\"\s*$",
)
# VERSION parse must scan the full message (merge commits put the
# chore line in the body, not on line 1).
self.assertNotIn("| head -n1 | sed -n", text)
def test_tag_release_workflow_version_extraction(self) -> None:
"""VERSION sed must match direct and merge-commit message shapes."""
text = _tag_release_workflow_text()
sed_expr = _tag_release_version_sed_expr(text)
expected = "3.18.3"
direct = f"chore(release): bump version to {expected}"
merge = (
"Merge pull request #879 from mvanhorn/release/v3.18.3\n\n"
f"chore(release): bump version to {expected}"
)
for message in (direct, merge):
with self.subTest(message=message.splitlines()[0]):
self.assertEqual(
_parse_release_version_from_message(message, sed_expr),
expected,
)
def test_changelog_guard_run_blocks_stay_indented(self) -> None:
"""Column-0 lines inside ``run: |`` break Actions YAML parsing."""
path = ROOT / ".github" / "workflows" / "changelog-guard.yml"
+406
View File
@@ -0,0 +1,406 @@
"""Security tests for hooks/scripts/check-config.sh env parsing.
Covers the SessionStart hook's .env loader:
- printf -v key injection (array-subscript command substitution)
- LAST30DAYS_TRUST_PROJECT_CONFIG gate matching lib/env.py
- Legitimate identifier keys still parse
The PoC key ``x[$(touch RCE-PROOF.txt)]=1`` executes under bash 4+/5 via
printf -v assignment semantics; bash 3.2 rejects it as an invalid identifier
and (with ``set -e``) aborts the hook. Either way, after the fix the hook must
exit 0 and must never create the proof file.
"""
from __future__ import annotations
import os
import re
import shutil
import subprocess
from pathlib import Path
import pytest
HOOK = Path(__file__).resolve().parents[1] / "hooks" / "scripts" / "check-config.sh"
POC_LINE = "x[$(touch RCE-PROOF.txt)]=1\n"
def _bash_binaries() -> list[str]:
"""Prefer modern bash (4+) when present so the RCE path is actually exercised."""
seen: list[str] = []
for candidate in (
"/opt/homebrew/bin/bash",
"/usr/local/bin/bash",
shutil.which("bash"),
):
if not candidate:
continue
path = str(Path(candidate).resolve())
if path not in seen and Path(path).is_file():
seen.append(path)
return seen
def _bash_major(bash_path: str) -> int:
result = subprocess.run(
[bash_path, "-c", 'echo "${BASH_VERSINFO[0]}"'],
capture_output=True,
text=True,
timeout=10,
check=False,
)
try:
return int((result.stdout or "").strip() or "0")
except ValueError:
return 0
def _mode_bits_assertable() -> bool:
"""Match check_perms: Windows/MSYS synthesized modes are not meaningful."""
if os.name == "nt":
return False
uname = ""
try:
uname = os.uname().sysname # type: ignore[attr-defined]
except AttributeError:
return True
return not uname.startswith(("MINGW", "MSYS", "CYGWIN"))
def _assert_mode(path: Path, expected: str) -> None:
if _mode_bits_assertable():
assert oct(path.stat().st_mode)[-3:] == expected
def _isolated_path(tmp_path: Path) -> str:
"""PATH with a stub ``security`` so macOS Keychain presence cannot leak into assertions."""
bin_dir = tmp_path / "hook-bin"
bin_dir.mkdir(exist_ok=True)
security = bin_dir / "security"
if not security.exists():
security.write_text("#!/bin/sh\nexit 1\n", encoding="utf-8")
security.chmod(0o755)
return f"{bin_dir}{os.pathsep}{os.environ.get('PATH', '')}"
def _run_hook(
bash_path: str,
cwd: Path,
tmp_path: Path,
env_overrides: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
env = os.environ.copy()
for k in (
"LAST30DAYS_MEMORY_DIR",
"SETUP_COMPLETE",
"LAST30DAYS_CONFIG_DIR",
"LAST30DAYS_TRUST_PROJECT_CONFIG",
"OPENAI_API_KEY",
"SCRAPECREATORS_API_KEY",
"AUTH_TOKEN",
"CT0",
"XAI_API_KEY",
"BSKY_HANDLE",
"EXA_API_KEY",
):
env.pop(k, None)
env["PATH"] = _isolated_path(tmp_path)
if env_overrides:
env.update(env_overrides)
return subprocess.run(
[bash_path, str(HOOK)],
capture_output=True,
text=True,
env=env,
cwd=str(cwd),
timeout=30,
check=False,
)
@pytest.fixture(params=_bash_binaries())
def bash_path(request: pytest.FixtureRequest) -> str:
return request.param
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_malicious_project_env_key_does_not_execute(bash_path: str, tmp_path: Path):
"""Reporter PoC: crafted key must not run, even under bash 4+/5."""
project = tmp_path / "repo"
env_file = project / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text(f"SETUP_COMPLETE=1\n{POC_LINE}", encoding="utf-8")
proof = project / "RCE-PROOF.txt"
result = _run_hook(
bash_path,
project,
tmp_path,
{
"LAST30DAYS_CONFIG_DIR": str(tmp_path / "empty-config"),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert not proof.exists(), f"RCE proof file was created under {bash_path}"
assert result.returncode == 0, (
f"hook aborted under {bash_path}: stderr={result.stderr!r} stdout={result.stdout!r}"
)
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_malicious_key_blocked_even_when_project_trusted(bash_path: str, tmp_path: Path):
"""Identifier gate must hold even after an explicit trust opt-in."""
project = tmp_path / "repo"
env_file = project / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text(f"SETUP_COMPLETE=1\n{POC_LINE}", encoding="utf-8")
proof = project / "RCE-PROOF.txt"
result = _run_hook(
bash_path,
project,
tmp_path,
{
"LAST30DAYS_TRUST_PROJECT_CONFIG": "1",
"LAST30DAYS_CONFIG_DIR": str(tmp_path / "empty-config"),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert not proof.exists(), f"RCE proof file was created under {bash_path} (trusted path)"
assert result.returncode == 0, (
f"hook aborted under {bash_path}: stderr={result.stderr!r} stdout={result.stdout!r}"
)
assert "Ready" in result.stdout
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_untrusted_project_env_is_ignored(bash_path: str, tmp_path: Path):
"""Without LAST30DAYS_TRUST_PROJECT_CONFIG, project file is not read or chmod'd."""
project = tmp_path / "repo"
env_file = project / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text(
"SETUP_COMPLETE=true\nSCRAPECREATORS_API_KEY=scrape-test-key-untrusted\n",
encoding="utf-8",
)
env_file.chmod(0o644)
result = _run_hook(
bash_path,
project,
tmp_path,
{
"LAST30DAYS_CONFIG_DIR": str(tmp_path / "empty-config"),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert result.returncode == 0, result.stderr
# check_perms only runs on the chosen config file; untrusted project stays 644.
_assert_mode(env_file, "644")
# Project ScrapeCreators key must not suppress the tip when the configured
# banner path runs (isolated PATH stubs Keychain).
if "sources active" in result.stdout:
assert "Tip: Add ScrapeCreators" in result.stdout
else:
assert "Ready to use" in result.stdout
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_trusted_project_env_loads_normal_keys(bash_path: str, tmp_path: Path):
project = tmp_path / "repo"
env_file = project / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text(
"SETUP_COMPLETE=true\n"
"SCRAPECREATORS_API_KEY=scrape-test-key\n",
encoding="utf-8",
)
env_file.chmod(0o644)
result = _run_hook(
bash_path,
project,
tmp_path,
{
"LAST30DAYS_TRUST_PROJECT_CONFIG": "1",
"LAST30DAYS_CONFIG_DIR": str(tmp_path / "empty-config"),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert result.returncode == 0, result.stderr
assert re.search(r"Ready — \d+ sources active", result.stdout)
assert "Tip: Add ScrapeCreators" not in result.stdout
_assert_mode(env_file, "600")
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_global_trust_signal_unlocks_project_env(bash_path: str, tmp_path: Path):
"""Trust from ~/.config (via LAST30DAYS_CONFIG_DIR) unlocks project overlay."""
config_dir = tmp_path / "config"
config_dir.mkdir()
(config_dir / ".env").write_text(
"LAST30DAYS_TRUST_PROJECT_CONFIG=1\nSETUP_COMPLETE=true\n",
encoding="utf-8",
)
project = tmp_path / "repo"
env_file = project / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text("SCRAPECREATORS_API_KEY=from-project\n", encoding="utf-8")
env_file.chmod(0o644)
result = _run_hook(
bash_path,
project,
tmp_path,
{
"LAST30DAYS_CONFIG_DIR": str(config_dir),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert result.returncode == 0, result.stderr
assert re.search(r"Ready — \d+ sources active", result.stdout)
assert "Tip: Add ScrapeCreators" not in result.stdout
_assert_mode(env_file, "600")
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_process_deny_overrides_global_trust(bash_path: str, tmp_path: Path):
config_dir = tmp_path / "config"
config_dir.mkdir()
(config_dir / ".env").write_text(
"LAST30DAYS_TRUST_PROJECT_CONFIG=1\nSETUP_COMPLETE=true\n",
encoding="utf-8",
)
project = tmp_path / "repo"
env_file = project / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text("SCRAPECREATORS_API_KEY=from-project\n", encoding="utf-8")
env_file.chmod(0o644)
result = _run_hook(
bash_path,
project,
tmp_path,
{
"LAST30DAYS_TRUST_PROJECT_CONFIG": "0",
"LAST30DAYS_CONFIG_DIR": str(config_dir),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert result.returncode == 0, result.stderr
assert re.search(r"Ready — \d+ sources active", result.stdout)
assert "Tip: Add ScrapeCreators" in result.stdout
_assert_mode(env_file, "644")
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_trusted_project_env_discovered_from_nested_cwd(bash_path: str, tmp_path: Path):
"""Mirror lib/env.py: walk up from a subdirectory to the repo-root project env."""
repo = tmp_path / "repo"
nested = repo / "apps" / "web"
nested.mkdir(parents=True)
(repo / ".git").mkdir()
env_file = repo / ".claude" / "last30days.env"
env_file.parent.mkdir(parents=True)
env_file.write_text(
"SETUP_COMPLETE=true\nSCRAPECREATORS_API_KEY=from-repo-root\n",
encoding="utf-8",
)
env_file.chmod(0o644)
result = _run_hook(
bash_path,
nested,
tmp_path,
{
"LAST30DAYS_TRUST_PROJECT_CONFIG": "1",
"LAST30DAYS_CONFIG_DIR": str(tmp_path / "empty-config"),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert result.returncode == 0, result.stderr
assert re.search(r"Ready — \d+ sources active", result.stdout)
assert "Tip: Add ScrapeCreators" not in result.stdout
_assert_mode(env_file, "600")
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_project_env_walk_stops_at_git_root(bash_path: str, tmp_path: Path):
"""An env above the git root must not be discovered (matches lib/env.py)."""
outside = tmp_path / ".claude" / "last30days.env"
outside.parent.mkdir(parents=True)
outside.write_text(
"SETUP_COMPLETE=true\nSCRAPECREATORS_API_KEY=outside-repo\n",
encoding="utf-8",
)
outside.chmod(0o644)
repo = tmp_path / "repo"
nested = repo / "nested"
nested.mkdir(parents=True)
(repo / ".git").mkdir()
result = _run_hook(
bash_path,
nested,
tmp_path,
{
"LAST30DAYS_TRUST_PROJECT_CONFIG": "1",
"LAST30DAYS_CONFIG_DIR": str(tmp_path / "empty-config"),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert result.returncode == 0, result.stderr
_assert_mode(outside, "644")
# Outside key must not suppress the ScrapeCreators tip / must not count as configured.
if "sources active" in result.stdout:
assert "Tip: Add ScrapeCreators" in result.stdout
else:
assert "Ready to use" in result.stdout
@pytest.mark.skipif(not _bash_binaries(), reason="bash not on PATH")
def test_malicious_key_in_global_env_also_blocked(bash_path: str, tmp_path: Path):
"""Identifier gate applies to the global file too (defense in depth)."""
config_dir = tmp_path / "config"
config_dir.mkdir()
(config_dir / ".env").write_text(f"SETUP_COMPLETE=1\n{POC_LINE}", encoding="utf-8")
work = tmp_path / "workdir"
work.mkdir()
proof = work / "RCE-PROOF.txt"
result = _run_hook(
bash_path,
work,
tmp_path,
{
"LAST30DAYS_CONFIG_DIR": str(config_dir),
"LAST30DAYS_MEMORY_DIR": str(tmp_path / "memory"),
},
)
assert not proof.exists(), f"RCE proof created from global env under {bash_path}"
assert result.returncode == 0, result.stderr
assert "Ready" in result.stdout
@pytest.mark.skipif(
not any(_bash_major(b) >= 4 for b in _bash_binaries()),
reason="needs bash 4+ to exercise printf -v RCE",
)
def test_rce_path_exercised_on_modern_bash(tmp_path: Path):
"""Sanity: at least one bash>=4 is under test so the PoC path is real, not vacuous."""
modern = [b for b in _bash_binaries() if _bash_major(b) >= 4]
assert modern, "expected a bash 4+ binary from _bash_binaries()"
test_malicious_key_blocked_even_when_project_trusted(modern[0], tmp_path)
+48
View File
@@ -186,6 +186,26 @@ class CliV3Tests(unittest.TestCase):
self.assertEqual(["biosecurity", "ai", "agents"], args.topic)
self.assertEqual([], extra)
def test_build_parser_accepts_web_backend_keyless(self):
"""Regression for #905: CONFIGURATION.md documents --web-backend=keyless
to force the zero-key floor, but the choices list rejected it."""
parser = cli.build_parser()
args, extra = parser.parse_known_args(["--web-backend", "keyless", "biosecurity"])
self.assertEqual("keyless", args.web_backend)
self.assertEqual([], extra)
def test_build_parser_still_accepts_other_web_backend_values(self):
parser = cli.build_parser()
for value in ("auto", "brave", "exa", "serper", "parallel", "none"):
args, extra = parser.parse_known_args(["--web-backend", value, "biosecurity"])
self.assertEqual(value, args.web_backend)
self.assertEqual([], extra)
def test_build_parser_rejects_invalid_web_backend(self):
parser = cli.build_parser()
with self.assertRaises(SystemExit):
parser.parse_known_args(["--web-backend", "bogus", "biosecurity"])
def test_build_parser_accepts_explicit_output_file(self):
parser = cli.build_parser()
args, extra = parser.parse_known_args(
@@ -329,6 +349,34 @@ class CliV3Tests(unittest.TestCase):
payload = json.loads(path.read_text())
self.assertEqual("OpenClaw vs NanoClaw", payload["query"])
def test_compact_emit_saves_full_artifact_not_compact_render(self):
"""A --emit=compact --save-dir run must save the complete debug
artifact (all clusters plus per-source items), not the compact stdout
render. Saving the compact render made most collected evidence
unrecoverable from the raw file (#923)."""
with tempfile.TemporaryDirectory() as tmp:
result = subprocess.run(
[
sys.executable,
"skills/last30days/scripts/last30days.py",
"compact save probe",
"--mock",
"--emit=compact",
f"--save-dir={tmp}",
],
cwd=REPO_ROOT,
capture_output=True,
text=True,
encoding="utf-8",
check=False,
)
self.assertEqual(0, result.returncode, result.stderr)
saved = list(Path(tmp).glob("*.md"))
self.assertEqual(1, len(saved), saved)
content = saved[0].read_text(encoding="utf-8")
self.assertIn("## All Items by Source", content)
self.assertNotIn("## All Items by Source", result.stdout)
def test_save_output_uses_unique_dated_fallback(self):
report = self.make_report()
with tempfile.TemporaryDirectory() as tmp:
+78
View File
@@ -170,6 +170,84 @@ class TestPolymarketIsolation(unittest.TestCase):
self.assertEqual(2, len(clusters[0].candidate_ids))
class TestStaleClusterDemotion(unittest.TestCase):
"""Stale candidates must never become cluster representatives or titles."""
def _plan(self):
return schema.QueryPlan(
intent="breaking_news",
freshness_mode="strict_recent",
cluster_mode="story",
raw_topic="test",
subqueries=[schema.SubQuery(label="primary", search_query="test", ranking_query="test", sources=["reddit", "x"])],
source_weights={"reddit": 0.5, "x": 0.5},
)
def _candidate_with_date(
self, candidate_id: str, source: str, title: str, score: float,
published_at: str, date_confidence: str, range_from: str, range_to: str,
) -> schema.Candidate:
item = schema.SourceItem(
item_id=candidate_id,
source=source,
title=title,
body=title,
url=f"https://example.com/{candidate_id}",
published_at=published_at,
date_confidence=date_confidence,
)
return schema.Candidate(
candidate_id=candidate_id,
item_id=candidate_id,
source=source,
title=title,
url=f"https://example.com/{candidate_id}",
snippet=title,
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.8,
freshness=80,
engagement=10,
source_quality=0.7,
rrf_score=0.02,
rerank_score=score,
final_score=score,
source_items=[item],
metadata={"range_from": range_from, "range_to": range_to},
)
def test_stale_candidate_not_cluster_representative(self):
"""A stale item with higher final_score must not lead a cluster over a fresh item.
This guards against the issue where a 2025-10 video ranked #1 in a
2026-07 brief because clustering re-sorted by final_score alone.
"""
range_from = "2026-06-15"
range_to = "2026-07-15"
stale = self._candidate_with_date(
"stale", "reddit", "Model launch reactions discussion",
score=95.0,
published_at="2025-10-15",
date_confidence="low",
range_from=range_from,
range_to=range_to,
)
fresh = self._candidate_with_date(
"fresh", "x", "Model launch reactions update",
score=50.0,
published_at="2026-07-10",
date_confidence="high",
range_from=range_from,
range_to=range_to,
)
candidates = [stale, fresh]
clusters = cluster.cluster_candidates(candidates, self._plan())
self.assertEqual(1, len(clusters))
self.assertEqual("fresh", clusters[0].representative_ids[0])
self.assertEqual(fresh.title, clusters[0].title)
class TestClusterUncertainty(unittest.TestCase):
def test_single_source_returns_single_source(self):
candidates = [make_candidate("c1", "reddit", "Title", "Body", 80)]
+23
View File
@@ -101,6 +101,29 @@ class FanoutOrchestratorTests(unittest.TestCase):
)
self.assertEqual([label for label, _ in results], ["OpenAI"])
def test_fanout_clears_youtube_search_cache_before_parallel_work(self):
"""Comparison mode must not inherit a prior run's ytsearch cache."""
from lib import youtube_yt
youtube_yt._search_cache[("prior", 8, "2026-01-01")] = {"items": []}
cleared = []
def main_runner():
cleared.append("main" in youtube_yt._search_cache or len(youtube_yt._search_cache) == 0)
return _fake_report("OpenAI")
with mock.patch.object(
youtube_yt, "reset_search_cache", wraps=youtube_yt.reset_search_cache
) as reset_mock, redirect_stderr(io.StringIO()):
fanout.run_competitor_fanout(
main_topic="OpenAI",
main_runner=main_runner,
competitors=["Anthropic"],
competitor_runner=lambda e: _fake_report(e),
)
reset_mock.assert_called()
self.assertNotIn(("prior", 8, "2026-01-01"), youtube_yt._search_cache)
def test_sub_runs_execute_in_parallel(self):
"""Wall clock should be closer to max(latency) than sum(latency)."""
delay = 0.2
+6 -1
View File
@@ -66,6 +66,8 @@ DIAGNOSE_TOP_KEYS = {
"native_search",
"has_scrapecreators",
"has_github",
"brightdata_installed",
"brightdata_authenticated",
"x_pending_browser_auth",
"available_sources",
"safe",
@@ -82,7 +84,10 @@ DIAGNOSE_TOP_KEYS = {
DIAGNOSE_PROVIDERS_KEYS = {"google", "openai", "xai", "openrouter", "perplexity"}
DIAGNOSE_BROWSER_COOKIES_KEYS = {"mode", "browsers", "reads_values"}
DIAGNOSE_EXTERNAL_COMMANDS_KEYS = {"yt-dlp", "digg-pp-cli", "arxiv-pp-cli", "techmeme-pp-cli", "trustpilot-pp-cli", "gh"}
DIAGNOSE_EXTERNAL_COMMANDS_KEYS = {
"yt-dlp", "digg-pp-cli", "arxiv-pp-cli", "techmeme-pp-cli", "trustpilot-pp-cli",
"brightdata", "gh",
}
DIAGNOSE_CREDENTIAL_DESTINATIONS_KEYS = {"global_env"}
PREFLIGHT_TOP_KEYS = {
+42 -1
View File
@@ -5,6 +5,7 @@ import json
import os
import subprocess
import sys
import urllib.error
from pathlib import Path
from unittest import mock
@@ -379,22 +380,62 @@ def test_listing_failure_is_not_reported_as_clean_no_results():
assert report.source_status["reddit"].detail == "connection timed out"
def test_discovery_listing_block_is_reported_as_rate_limited():
# get_text swallows the 429 and hands back None; without the tee the lane
# could not tell that apart from an empty listing and the sweep reported a
# clean no-results (issue #899).
blocked = urllib.error.HTTPError(
"https://www.reddit.com/svc/shreddit/community-more-posts/rising/",
429,
"Too Many Requests",
{},
None,
)
with mock.patch.object(pipeline, "available_sources", return_value=["reddit"]), \
mock.patch("lib.http.time.sleep"), \
mock.patch("lib.http.urllib.request.urlopen", side_effect=blocked):
report = pipeline.run_discover(
domain="AI agents",
config={},
as_of_date="2026-07-10",
subreddits=["AI_Agents"],
)
outcome = report.source_status["reddit"]
assert outcome.state == "rate-limited"
assert "429" in (outcome.detail or "")
def test_reddit_discovery_adapter_preserves_partial_feed_errors():
"""When one shreddit sort lane fails and another succeeds, the failed lane's error is kept.
Errors are cleared per (sub, sort) pair, not per subreddit. Arctic-shift cannot
recover a specific sort lane since it's recency-only.
"""
item = {
"url": "https://reddit.com/r/example/comments/1",
"title": "AI agent launch",
"subreddit": "AI_Agents", # Required for error-clearing logic.
}
with mock.patch.object(
reddit_listing,
"_fetch_one_with_status",
side_effect=[([], "rising timed out"), ([item], None)],
), mock.patch(
"lib.reddit_arctic.fetch_listings",
return_value=[], # Arctic supplement returns nothing.
):
result = reddit_listing.fetch_discovery_listings(
["AI_Agents"], query="AI agents",
)
assert result["items"] == [item]
assert result["errors"] == ["r/AI_Agents rising: rising timed out"]
# Shreddit top succeeded → no error for top.
# Shreddit rising failed → error for rising is preserved.
# Error-clearing is per (sub, sort) pair, not per subreddit.
assert len(result["errors"]) == 1
assert "rising" in result["errors"][0].lower()
def test_discovery_cli_json_contract_and_mutual_exclusion():
+13 -13
View File
@@ -77,7 +77,7 @@ def test_nominations_ranked_by_seed_velocity():
]
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("AI", ["hackernews"]), _plan("AI", ["hackernews"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert nominations, "expected at least one nomination"
assert "GPT-6" in nominations[0].name
@@ -95,7 +95,7 @@ def test_nominations_dedupe_names_casefold():
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("AI agents", ["hackernews", "reddit"]),
_plan("AI agents", ["hackernews", "reddit"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
names = [nomination.name.casefold() for nomination in nominations]
assert len(names) == len(set(names))
@@ -109,7 +109,7 @@ def test_fewer_clusters_than_limit_returns_all_without_padding():
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("quantum", ["hackernews"]),
_plan("quantum", ["hackernews"]),
to_date="2026-07-10", limit=8,
from_date="2026-06-10", to_date="2026-07-10", limit=8,
)
assert 1 <= len(nominations) < 8
@@ -122,7 +122,7 @@ def test_zero_velocity_clusters_are_dropped():
]
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("AI", ["hackernews"]), _plan("AI", ["hackernews"]),
to_date="2026-07-10", limit=8,
from_date="2026-06-10", to_date="2026-07-10", limit=8,
)
assert nominations == []
@@ -145,7 +145,7 @@ def test_names_are_short_distilled_topics_not_raw_titles():
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("AI agents", ["hackernews"]),
_plan("AI agents", ["hackernews"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert nominations
name = nominations[0].name
@@ -169,7 +169,7 @@ def test_no_provider_names_are_distilled_and_deterministic():
return pipeline.nominate_topics(
bundle, _query_plan("AI agents", ["hackernews"]),
_plan("AI agents", ["hackernews"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
first, second = run(), run()
@@ -197,7 +197,7 @@ def test_nomination_carries_leader_summary_and_items():
]
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("Linux", ["hackernews"]), _plan("Linux", ["hackernews"]),
to_date="2026-07-10", limit=8,
from_date="2026-06-10", to_date="2026-07-10", limit=8,
)
assert nominations
top = nominations[0]
@@ -226,7 +226,7 @@ def test_same_entity_clusters_disambiguate_instead_of_dropping():
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("AI agents", ["hackernews"]),
_plan("AI agents", ["hackernews"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert len(nominations) == 2
@@ -259,7 +259,7 @@ def test_third_same_entity_cluster_survives_via_successive_tokens():
nominations = pipeline.nominate_topics(
_bundle(items), _query_plan("AI agents", ["hackernews"]),
_plan("AI agents", ["hackernews"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert len(nominations) == 3
@@ -314,7 +314,7 @@ def test_indistinguishable_distinct_representative_clusters_still_dedupe():
_bundle(items),
_query_plan("AI agents", ["hackernews", "reddit"]),
_plan("AI agents", ["hackernews", "reddit"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert len(nominations) == 1
@@ -358,7 +358,7 @@ def test_clusters_sharing_a_representative_dedupe_to_one():
_bundle(items),
_query_plan("AI agents", ["hackernews", "reddit"]),
_plan("AI agents", ["hackernews", "reddit"]),
to_date="2026-07-10", limit=10,
from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert len(nominations) == 1
@@ -381,10 +381,10 @@ def test_nominate_topic_pool_pairs_nominations_with_cluster_ids():
query_plan = _query_plan("AI", ["hackernews"])
plan = _plan("AI", ["hackernews"])
pool = pipeline.nominate_topic_pool(
bundle, query_plan, plan, to_date="2026-07-10", limit=10,
bundle, query_plan, plan, from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
nominations = pipeline.nominate_topics(
bundle, query_plan, plan, to_date="2026-07-10", limit=10,
bundle, query_plan, plan, from_date="2026-06-10", to_date="2026-07-10", limit=10,
)
assert [nomination for nomination, _cluster_id in pool] == nominations
cluster_ids = [cluster_id for _nomination, cluster_id in pool]
+50 -1
View File
@@ -22,12 +22,13 @@ import os
import sys
import tempfile
import unittest
import urllib.error
from contextlib import redirect_stderr, redirect_stdout
from pathlib import Path
from unittest import mock
import last30days as cli
from lib import backends, doctor, health, prescriptions
from lib import backends, doctor, health, http, prescriptions
BIRD_STATUS_OFF = {
"installed": False,
@@ -959,6 +960,54 @@ class LiveProbe(unittest.TestCase):
def test_probe_source_credit_gated_returns_none(self):
self.assertIsNone(doctor._probe_source("tiktok", {}, 5))
def test_reddit_probe_targets_the_endpoint_the_engine_uses(self):
# /r/all/hot.json is permanently 403 keyless and no lane requests it;
# probing it certified an endpoint the engine had abandoned (#899).
url = doctor._HTTP_PROBE_URLS["reddit"]
self.assertIn("search.rss", url)
self.assertNotIn("hot.json", url)
def _probe_reddit_with_status(self, code):
error = urllib.error.HTTPError(
doctor._HTTP_PROBE_URLS["reddit"], code, "Blocked", {}, None
)
with mock.patch(
"lib.doctor.urllib.request.urlopen", side_effect=error
):
return doctor._probe_source("reddit", {}, 5)
def test_reddit_probe_403_is_not_reachable(self):
res = self._probe_reddit_with_status(403)
self.assertFalse(res["ok"])
self.assertIn("403", res["detail"])
def test_reddit_probe_429_is_not_reachable(self):
res = self._probe_reddit_with_status(429)
self.assertFalse(res["ok"])
self.assertIn("429", res["detail"])
def test_non_reddit_probe_keeps_4xx_as_reachable(self):
# The blocked-status carve-out is per-source: a 4xx elsewhere still
# means the endpoint responded.
error = urllib.error.HTTPError(
doctor._HTTP_PROBE_URLS["github"], 403, "Forbidden", {}, None
)
with mock.patch("lib.doctor.urllib.request.urlopen", side_effect=error):
res = doctor._probe_source("github", {}, 5)
self.assertTrue(res["ok"])
def test_reddit_probe_sends_the_engine_user_agent(self):
# Probing with a different UA measures the User-Agent, not the endpoint.
seen = {}
def capture(req, timeout=None):
seen["ua"] = req.get_header("User-agent")
raise urllib.error.HTTPError(req.full_url, 500, "boom", {}, None)
with mock.patch("lib.doctor.urllib.request.urlopen", capture):
doctor._probe_source("reddit", {}, 5)
self.assertEqual(http.BROWSER_USER_AGENT, seen["ua"])
def test_probe_failure_is_isolated(self):
def flaky(name, config, timeout):
if name == "reddit":
+60
View File
@@ -0,0 +1,60 @@
"""Which topics are worth one web search to resolve an X handle.
Gates on brevity rather than capitalization. Requiring a capital letter meant
the most common real-world spelling never resolved a handle: every recent run
on this machine -- bentgo, buzz, getenergy.com -- is lowercase.
"""
import importlib.util
from pathlib import Path
import pytest
_CLI = Path(__file__).resolve().parent.parent / "skills" / "last30days" / "scripts" / "last30days.py"
def _cli():
spec = importlib.util.spec_from_file_location("l30_cli", _CLI)
module = importlib.util.module_from_spec(spec)
try:
spec.loader.exec_module(module)
except SystemExit:
pass
return module
@pytest.mark.parametrize("topic", [
"bentgo",
"Bentgo",
"peter steinberger",
"Peter Steinberger",
"getenergy.com",
"buzz by block",
"@steipete",
"claude code",
])
def test_entity_topics_resolve(topic):
assert _cli()._looks_like_entity_topic(topic) is True
@pytest.mark.parametrize("topic", [
"best AI coding tools 2026",
"how to build agents",
"top tools for research",
"what is the best model?",
"",
])
def test_theme_topics_do_not_resolve(topic):
assert _cli()._looks_like_entity_topic(topic) is False
def test_casing_does_not_change_the_verdict():
"""The regression this replaces: capitalization decided whether a subject's
own posts could be protected."""
cli = _cli()
for lower, upper in [("bentgo", "Bentgo"), ("peter steinberger", "Peter Steinberger")]:
assert cli._looks_like_entity_topic(lower) == cli._looks_like_entity_topic(upper)
def test_question_shaped_topics_are_themes():
assert _cli()._looks_like_entity_topic("who is peter steinberger?") is False
+110
View File
@@ -0,0 +1,110 @@
"""The topic subject gets a higher per-author cap than incidental authors.
The flat cap of 3 is anti-flooding protection and is right for third parties.
But on a person or company topic the subject is the point of the query: the
measured 'Peter Steinberger steipete' run recovered 8 subject-authored posts,
and a flat cap would discard 5 of them after the rest of Phase A worked to
retrieve and keep them.
The raised cap stays bounded so a prolific subject cannot crowd out all
commentary about them.
"""
from lib import fusion, schema
def _cand(cid: str, author: str) -> schema.Candidate:
url = f"https://x.com/{author}/status/{cid}"
cand = schema.Candidate(
candidate_id=cid,
item_id=f"i{cid}",
source="x",
title=f"post {cid}",
url=url,
snippet="s",
subquery_labels=["primary"],
native_ranks={"primary:x": 1},
local_relevance=0.5,
freshness=80,
engagement=50,
source_quality=0.68,
rrf_score=0.02,
)
cand.source_items = [
schema.SourceItem(
item_id=f"i{cid}", source="x", title=f"post {cid}",
body="b", url=url, author=author,
)
]
return cand
def _subject_posts(n: int, author: str = "steipete"):
return [_cand(str(i), author) for i in range(1, n + 1)]
def test_third_party_author_is_still_capped_at_three():
kept = fusion._apply_per_author_cap(_subject_posts(8, author="rando"))
assert len(kept) == 3, "the anti-flooding cap must be unchanged for third parties"
def test_subject_keeps_more_than_three_posts():
kept = fusion._apply_per_author_cap(
_subject_posts(8), first_party_handles={"steipete"}
)
assert len(kept) > 3, (
"the subject of the topic was capped at 3 of 8 recovered posts, "
"discarding evidence the rest of Phase A worked to keep"
)
def test_subject_cap_is_bounded():
"""A prolific subject must not fill the pool."""
kept = fusion._apply_per_author_cap(
_subject_posts(40), first_party_handles={"steipete"}
)
assert len(kept) <= fusion._MAX_ITEMS_PER_FIRST_PARTY_AUTHOR
assert len(kept) < 40
def test_mixed_pool_caps_each_author_by_its_own_rule():
pool = _subject_posts(8) + [_cand(f"r{i}", "rando") for i in range(1, 8)]
kept = fusion._apply_per_author_cap(pool, first_party_handles={"steipete"})
subject = [c for c in kept if c.source_items[0].author == "steipete"]
third = [c for c in kept if c.source_items[0].author == "rando"]
assert len(subject) > 3
assert len(third) == 3
def test_no_handles_behaves_exactly_as_before():
pool = _subject_posts(8)
assert len(fusion._apply_per_author_cap(pool)) == \
len(fusion._apply_per_author_cap(pool, first_party_handles=set()))
def test_handles_are_matched_case_insensitively():
kept = fusion._apply_per_author_cap(
_subject_posts(8), first_party_handles={"SteiPete"}
)
assert len(kept) > 3
def test_ordering_within_an_author_is_preserved():
"""Candidates arrive sorted by quality; the cap keeps the best N."""
kept = fusion._apply_per_author_cap(
_subject_posts(8), first_party_handles={"steipete"}
)
assert [c.candidate_id for c in kept] == \
[str(i) for i in range(1, len(kept) + 1)]
def test_weighted_rrf_accepts_first_party_handles():
import inspect
sig = inspect.signature(fusion.weighted_rrf)
assert "first_party_handles" in sig.parameters, (
"the fusion entry point must accept the run's resolved handles or the "
"raised cap can never fire in production"
)
assert sig.parameters["first_party_handles"].default in (None, frozenset()), (
"must stay optional so the discovery caller is unaffected"
)
+623
View File
@@ -0,0 +1,623 @@
"""Retrieval-floor exemption for posts authored by a handle the run is searching.
The production failure this pins: a mixed batch where the mention lane clears
the relevance floor and the from lane does not. Because
`prune_low_relevance` ends with `return filtered or items`, the all-fail rescue
only fires when *everything* fails. A mixed batch is therefore the exact shape
that silently loses the subject's own posts, and no prior test exercised it --
existing supplement-lane tests use single-item batches that trip the rescue.
"""
from lib import schema, signals
def _x_item(item_id: str, author: str, relevance: float, engagement: dict | None = None):
item = schema.SourceItem(
item_id=item_id,
source="x",
title="",
body="post body",
url=f"https://x.com/{author}/status/{item_id}",
author=author,
engagement=engagement or {},
)
item.local_relevance = relevance
return item
def _mixed_batch():
"""From-lane items score 0.0 (a post rarely names its own author);
mention-lane items clear the floor because they contain the handle."""
return [
_x_item("1", "steipete", 0.0, {"likes": 7773, "reposts": 391}),
_x_item("2", "steipete", 0.0, {"likes": 3466, "reposts": 128}),
_x_item("3", "someone_else", 0.32, {"likes": 78, "reposts": 8}),
_x_item("4", "another_acct", 0.23, {"likes": 8, "reposts": 1}),
]
def _annotate(items):
"""Populate engagement_score the way the real pipeline does."""
scores = signals.normalize([signals.engagement_raw(i) for i in items])
for item, score in zip(items, scores, strict=True):
item.engagement_score = score
return items
def test_mixed_batch_keeps_first_party_and_drops_off_topic():
items = _annotate(_mixed_batch())
kept = signals.prune_low_relevance(items, first_party_handles={"steipete"})
authors = sorted(i.author for i in kept)
assert "steipete" in authors, (
"first-party posts were pruned from a mixed batch; this is the measured "
"defect where 8 subject-authored posts never reached the report"
)
assert len([a for a in authors if a == "steipete"]) == 2
def test_mixed_batch_without_exemption_still_loses_first_party():
"""Characterizes the defect: without the exemption the batch drops them."""
items = _annotate(_mixed_batch())
kept = signals.prune_low_relevance(items)
assert all(i.author != "steipete" for i in kept), (
"expected the unexempted path to still drop zero-relevance first-party "
"posts; if this now passes, the floor changed and the exemption's "
"justification needs rechecking"
)
def test_non_first_party_below_floor_is_still_pruned():
"""The exemption must be scoped, not a blanket floor removal."""
items = _annotate(_mixed_batch() + [_x_item("5", "spam_acct", 0.02, {"likes": 0})])
kept = signals.prune_low_relevance(items, first_party_handles={"steipete"})
assert all(i.author != "spam_acct" for i in kept)
def test_batch_minimum_is_not_treated_as_zero_engagement():
"""`normalize` maps the batch minimum to exactly 0, so an item with real
engagement was being given the stricter 1.5x social threshold purely for
being the least-engaged item present."""
items = _annotate([
_x_item("1", "acct_a", 0.20, {"likes": 500, "reposts": 40}),
_x_item("2", "acct_b", 0.20, {"likes": 5000, "reposts": 400}),
])
least = next(i for i in items if i.author == "acct_a")
assert least.engagement_score == 0, "precondition: min-max maps batch min to 0"
kept = signals.prune_low_relevance(items)
assert any(i.author == "acct_a" for i in kept), (
"an item with 500 likes was pruned by the zero-engagement gate purely "
"because it was the batch minimum"
)
def test_genuinely_zero_engagement_still_gets_stricter_threshold():
"""The stricter gate must survive for real zero-engagement social noise."""
items = _annotate([
_x_item("1", "acct_a", 0.20, {"likes": 0, "reposts": 0}),
_x_item("2", "acct_b", 0.90, {"likes": 5000, "reposts": 400}),
])
kept = signals.prune_low_relevance(items)
assert all(i.author != "acct_a" for i in kept), (
"a genuinely zero-engagement item at 0.20 should fail the 0.225 gate"
)
def test_all_fail_rescue_is_unchanged():
items = _annotate([_x_item("1", "acct_a", 0.01), _x_item("2", "acct_b", 0.02)])
kept = signals.prune_low_relevance(items)
assert len(kept) == 2, "the all-fail rescue must still return the batch"
def test_batch_with_no_first_party_behaves_as_before():
items = _annotate(_mixed_batch())
assert signals.prune_low_relevance(items, first_party_handles=frozenset()) == \
signals.prune_low_relevance(items)
def test_handles_are_matched_case_insensitively():
items = _annotate(_mixed_batch())
kept = signals.prune_low_relevance(items, first_party_handles={"SteiPete"})
assert any(i.author == "steipete" for i in kept)
# --- KTD8: one owner for the entity-miss predicate -------------------------
def _candidate(explanation: str, final_score: float, author: str = "someone"):
url = f"https://x.com/{author}/status/1"
cand = schema.Candidate(
candidate_id="c1",
item_id="i1",
source="x",
title="t",
url=url,
snippet="s",
subquery_labels=["primary"],
native_ranks={"primary:x": 1},
local_relevance=0.0,
freshness=80,
engagement=50,
source_quality=0.68,
rrf_score=0.02,
)
cand.source_items = [
schema.SourceItem(
item_id="i1", source="x", title="t", body="b", url=url, author=author,
)
]
cand.explanation = explanation
cand.final_score = final_score
return cand
def test_render_delegates_to_shared_predicate():
"""render must not carry its own copy of the entity-miss test."""
from lib import render, rerank
cand = _candidate("fallback-local-score (entity-miss demotion)", 40.0)
assert render._best_take_relevance_ok(cand) is rerank.candidate_relevance_ok(cand)
ok = _candidate("llm-scored", 40.0)
assert render._best_take_relevance_ok(ok) is rerank.candidate_relevance_ok(ok)
def test_shared_predicate_rejects_entity_miss_and_zero_score():
from lib import rerank
assert not rerank.candidate_relevance_ok(
_candidate("fallback-local-score (entity-miss demotion)", 40.0)
)
assert not rerank.candidate_relevance_ok(_candidate("llm-scored", 0.0))
assert rerank.candidate_relevance_ok(_candidate("llm-scored", 40.0))
def test_first_party_carveout_reaches_render_side_gate():
"""The measured KTD8 failure: a first-party post demoted on the LLM path
was floored by rerank but still discarded at render because the render-side
copy re-tested the explanation string."""
from lib import render, rerank
cand = _candidate("fallback-local-score (entity-miss demotion)", 0.0, author="steipete")
assert not render._best_take_relevance_ok(cand), "precondition: demoted before the floor runs"
rerank._apply_first_party_floor([cand], resolved_handles={"steipete"})
assert cand.final_score >= rerank.FIRST_PARTY_FLOOR
assert render._best_take_relevance_ok(cand), (
"first-party carve-out applied in rerank did not propagate to the "
"render-side relevance gate"
)
def test_non_first_party_demotion_survives_the_floor_pass():
from lib import render, rerank
cand = _candidate("fallback-local-score (entity-miss demotion)", 0.0, author="rando")
rerank._apply_first_party_floor([cand], resolved_handles={"steipete"})
assert not render._best_take_relevance_ok(cand), (
"an off-topic collision post must stay buried"
)
# --- Phase 1 / quick-depth wiring (Greptile) --------------------------------
def test_phase_one_normalize_receives_the_explicit_handles():
"""Quick runs skip Phase 2 entirely, so an exemption reaching only the
supplement path leaves quick-depth reports discarding the subject's posts."""
import inspect
from lib import pipeline
src = inspect.getsource(pipeline.run)
assert "explicit_first_party = {" in src, (
"the user-named handles must be resolved before retrieval, not after"
)
assert "first_party_handles=explicit_first_party," in src, (
"the Phase 1 per-source normalize must receive the exemption"
)
def test_explicit_handles_are_available_before_any_retrieval():
"""The entity-extracted set does not exist until Phase 2; the explicit one
must be built from run()'s own arguments so Phase 1 can use it."""
import inspect
from lib import pipeline
src = inspect.getsource(pipeline.run)
build_at = src.index("explicit_first_party = {")
first_use = src.index("first_party_handles=explicit_first_party,")
assert build_at < first_use
def test_related_handles_lane_gets_the_exemption():
import inspect
from lib import pipeline
src = inspect.getsource(pipeline._run_supplemental_searches)
assert "first_party_handles=related_handles," in src
def test_quick_run_without_an_explicit_handle_still_protects_the_subject():
"""The gap in the first fix: covering only user-typed handles does nothing
for a quick run, where nobody typed one and Phase 2's automatic resolution
never executes."""
from lib import pipeline
candidates = pipeline._topic_first_party_candidates("Peter Steinberger steipete")
assert "steipete" in candidates
items = _annotate(_mixed_batch())
kept = signals.prune_low_relevance(items, first_party_handles=candidates)
assert any(i.author == "steipete" for i in kept), (
"a quick search naming the subject must not discard what they wrote"
)
def test_topic_candidates_include_explicit_mentions():
from lib import pipeline
assert "getenergy_" in pipeline._topic_first_party_candidates("@GetEnergy_ launch")
def test_topic_candidates_exclude_stopwords():
from lib import pipeline
got = pipeline._topic_first_party_candidates("the best of the year")
assert "the" not in got and "of" not in got
def test_topic_candidates_do_not_exempt_unrelated_authors():
"""Candidates only matter when a post's author matches one, so ordinary
words cost nothing -- no account is named 'lunch'."""
from lib import pipeline
candidates = pipeline._topic_first_party_candidates("bentgo lunch boxes")
items = _annotate(_mixed_batch() + [_x_item("9", "spam_acct", 0.01, {"likes": 0})])
kept = signals.prune_low_relevance(items, first_party_handles=candidates)
assert all(i.author != "spam_acct" for i in kept)
def test_topic_candidates_are_unioned_into_the_explicit_set():
import inspect
from lib import pipeline
src = inspect.getsource(pipeline.run)
assert "_topic_first_party_candidates(topic)" in src
build = src.index("explicit_first_party")
use = src.index("first_party_handles=explicit_first_party,")
assert build < use
def test_name_only_topic_resolves_the_subject_from_mentions():
"""The hard case: search "Peter Steinberger" with no handle anywhere. His
handle is @steipete, which matches no topic token, and Phase 2's resolution
has not run. Posts *about* him mention him, which is the signal the engine
already uses -- just later than the prune."""
from lib import pipeline
raw = [
{"text": "Great thread from @steipete on agent loops"},
{"text": "@steipete nailed this one"},
{"text": "watching @steipete build in public is wild"},
{"text": "unrelated chatter with no mention"},
]
assert "steipete" in pipeline._batch_subject_handles(raw)
def test_batch_subject_keys_on_mentions_not_authors():
"""A prolific commentator inflates author counts; being mentioned by other
accounts is what identifies the subject."""
from lib import pipeline
raw = [
{"author_handle": "spam_acct", "text": "buy now"},
{"author_handle": "spam_acct", "text": "buy now again"},
{"author_handle": "spam_acct", "text": "and again"},
{"author_handle": "someone", "text": "actually useful thread by @realsubject"},
]
got = pipeline._batch_subject_handles(raw)
assert "spam_acct" not in got
assert "realsubject" in got
def test_batch_subject_is_capped():
from lib import pipeline
raw = [{"text": f"@acct{i} said something"} for i in range(10)]
assert len(pipeline._batch_subject_handles(raw)) <= 2
def test_batch_subject_is_empty_without_mentions():
from lib import pipeline
assert pipeline._batch_subject_handles([{"text": "no mentions here"}]) == set()
assert pipeline._batch_subject_handles([]) == set()
def test_batch_inference_is_unioned_not_gated():
"""Regression: gating this on "no handles supplied" made it dead code.
The caller's set is derived partly from topic tokens, so it is non-empty
for essentially every real topic -- a fallback would never fire and the
name-only case would stay broken while looking fixed.
"""
import inspect
from lib import pipeline
src = inspect.getsource(pipeline._normalize_score_dedupe)
assert "floor_handles |= _batch_subject_handles(raw_items)" in src
assert "not floor_handles" not in src, (
"batch inference must union, never gate on an empty supplied set"
)
def test_name_only_topic_keeps_subject_posts_end_to_end():
"""The full path: topic names a person, handle appears nowhere in it, and
the subject's own zero-relevance posts still survive the floor."""
from lib import pipeline
supplied = pipeline._topic_first_party_candidates("Peter Steinberger")
assert "steipete" not in supplied, "precondition: the handle is not in the topic"
raw = [
{"text": "Great thread from @steipete on agent loops"},
{"text": "@steipete nailed this"},
{"text": "more praise for @steipete"},
]
inferred = pipeline._batch_subject_handles(raw)
items = _annotate(_mixed_batch())
kept = signals.prune_low_relevance(
items, first_party_handles=supplied | inferred
)
assert any(i.author == "steipete" for i in kept)
# --- ordering fix: prune X after resolution, not before ---------------------
def test_x_defers_its_relevance_floor_past_resolution():
"""The root ordering bug: X was pruned before the run knew who the subject
was, so the floor could not exempt an author nobody had identified yet. No
amount of guessing at prune time substitutes for knowing."""
import inspect
from lib import pipeline
src = inspect.getsource(pipeline.run)
assert 'defer_relevance_prune=(source == "x")' in src
def test_deferred_prune_runs_after_resolution_and_before_fusion():
import inspect
from lib import pipeline
src = inspect.getsource(pipeline.run)
resolve_at = src.index("resolved_handles = explicit_first_party")
prune_at = src.index("Deferred X relevance floor")
fuse_at = src.index("candidates = weighted_rrf(")
assert resolve_at < prune_at < fuse_at, (
"the deferred floor must see resolved handles and still run before fusion"
)
def test_non_x_sources_still_prune_in_place():
"""Only X defers; everything else keeps its existing behavior."""
import inspect
from lib import pipeline
src = inspect.getsource(pipeline._normalize_score_dedupe)
assert 'if source != "jobs" and not defer_relevance_prune:' in src
def test_deferred_prune_still_drops_off_topic_posts():
"""Deferring must not mean skipping."""
items = _annotate(_mixed_batch() + [_x_item("9", "spam_acct", 0.01, {"likes": 0})])
kept = signals.prune_low_relevance(items, first_party_handles={"steipete"})
assert any(i.author == "steipete" for i in kept)
assert all(i.author != "spam_acct" for i in kept)
# --- unresolved subject policy: skip X floor when no real handle identified ---
def test_topic_handle_mentions_extracts_only_at_mentions():
"""@mentions in the topic are real handles; regular words are not."""
from lib import pipeline
assert pipeline._topic_handle_mentions("Peter Steinberger @steipete") == {"steipete"}
assert pipeline._topic_handle_mentions("Peter Steinberger") == set()
assert pipeline._topic_handle_mentions("@GetEnergy_ launch") == {"getenergy_"}
def test_topic_handle_mentions_is_case_insensitive():
from lib import pipeline
assert pipeline._topic_handle_mentions("@SteiPete") == {"steipete"}
def test_entity_topic_no_handle_no_discovery_skips_x_floor():
"""Policy: when the subject cannot be identified, skip the X floor entirely.
Entity-shaped topic, no --x-handle, auto-resolve/Phase 2 return nothing.
Retrieved X items include a zero-relevance post whose author is not a
topic token. That post survives prune because the floor is not applied.
A companion on-topic post is in the batch so the all-fail rescue cannot
hide an incorrectly applied floor.
"""
from unittest.mock import patch
from lib import pipeline
topic = "Peter Steinberger"
assert pipeline._topic_handle_mentions(topic) == set()
topic_tokens = pipeline._topic_first_party_candidates(topic)
assert "peter" in topic_tokens and "steinberger" in topic_tokens
assert "rando_acct" not in topic_tokens
raw_items = [
{
"id": "1",
"text": "shipping a new agent loop tonight",
"url": "https://x.com/rando_acct/status/1",
"author_handle": "rando_acct",
"date": "2026-08-01",
"engagement": {"likes": 0, "reposts": 0, "replies": 0},
},
{
"id": "2",
"text": "Peter Steinberger just shipped another agent demo",
"url": "https://x.com/third_acct/status/2",
"author_handle": "third_acct",
"date": "2026-08-01",
"engagement": {"likes": 50, "reposts": 5, "replies": 2},
},
]
plan = {
"intent": "person",
"freshness_mode": "balanced_recent",
"cluster_mode": "topic",
"subqueries": [{
"label": "primary",
"search_query": topic,
"ranking_query": topic,
"sources": ["x"],
}],
"source_weights": {"x": 1.0},
}
def fake_retrieve(**kwargs):
if kwargs.get("source") == "x":
return raw_items, {}
return [], {}
with patch("lib.pipeline._retrieve_stream", side_effect=lambda **kw: fake_retrieve(**kw)):
report = pipeline.run(
topic=topic,
config={"LAST30DAYS_REASONING_PROVIDER": "gemini"},
depth="quick",
requested_sources=["x"],
mock=True,
external_plan=plan,
as_of_date="2026-08-14",
)
x_items = report.items_by_source.get("x") or []
authors = {item.author for item in x_items}
assert "rando_acct" in authors, (
"unresolved subject policy: skip X floor when no real handle identified; "
"the zero-relevance post whose author is not a topic token must survive"
)
def test_entity_topic_with_at_mention_applies_x_floor():
"""When the topic DOES include @mentions, the X floor applies normally."""
from lib import pipeline
topic = "Peter Steinberger @steipete"
explicit_x_handles = pipeline._topic_handle_mentions(topic)
supplemental_handles = []
assert "steipete" in explicit_x_handles, (
"precondition: @mention is a real handle"
)
real_x_handles = explicit_x_handles | {
h.lstrip("@").strip().lower() for h in supplemental_handles if h and h.strip()
}
assert real_x_handles, "real handles were resolved"
topic_candidates = pipeline._topic_first_party_candidates(topic)
items = _annotate([
_x_item("1", "steipete", 0.0, {"likes": 100, "reposts": 10}),
_x_item("2", "rando_acct", 0.0, {"likes": 0, "reposts": 0}),
_x_item("3", "third_acct", 0.18, {"likes": 20, "reposts": 2}),
])
kept = signals.prune_low_relevance(items, first_party_handles=topic_candidates | real_x_handles)
assert "steipete" in {i.author for i in kept}, (
"first-party posts survive the floor"
)
assert "rando_acct" not in {i.author for i in kept}, (
"when real handles exist, off-topic zero-relevance posts are pruned"
)
def test_explicit_x_handle_applies_x_floor():
"""--x-handle triggers the floor even with entity-only topic."""
from lib import pipeline
topic = "Peter Steinberger"
x_handle = "steipete"
explicit_x_handles = {x_handle.lstrip("@").strip().lower()}
assert pipeline._topic_handle_mentions(topic) == set(), (
"precondition: topic has no @mentions"
)
assert explicit_x_handles == {"steipete"}, (
"but we have an explicit --x-handle"
)
topic_candidates = pipeline._topic_first_party_candidates(topic)
resolved_handles = topic_candidates | explicit_x_handles
items = _annotate([
_x_item("1", "steipete", 0.0, {"likes": 100, "reposts": 10}),
_x_item("2", "rando_acct", 0.0, {"likes": 0, "reposts": 0}),
])
kept = signals.prune_low_relevance(items, first_party_handles=resolved_handles)
assert "steipete" in {i.author for i in kept}
assert "rando_acct" not in {i.author for i in kept}, (
"--x-handle triggers the floor, pruning off-topic posts"
)
# --- thin-source retry must not prune X before handle resolution ------------
def test_thin_retry_keeps_zero_relevance_subject_authored_x_post():
"""Phase 1 defers the X floor; the simplified-query retry must too.
A default/deep run with fewer than three X items retries with a simpler
query. If that retry returns a subject-authored post that does not repeat
the subject's name, applying the relevance floor here (with no resolved
handles) discards it before it enters the bundle. The later
resolved-handle floor cannot recover a post that never arrived.
"""
import threading
from unittest.mock import patch
from lib import pipeline
topic = "Peter Steinberger"
raw_items = [
{
"id": "1",
"text": "shipping a new agent loop tonight",
"url": "https://x.com/steipete/status/1",
"author_handle": "steipete",
"date": "2026-08-01",
"engagement": {"likes": 3466, "reposts": 128, "replies": 40},
},
{
"id": "2",
"text": "Peter Steinberger just shipped another agent demo",
"url": "https://x.com/third_acct/status/2",
"author_handle": "third_acct",
"date": "2026-08-01",
"engagement": {"likes": 50, "reposts": 5, "replies": 2},
},
]
plan = schema.QueryPlan(
intent="person",
freshness_mode="balanced_recent",
cluster_mode="topic",
raw_topic=topic,
subqueries=[
schema.SubQuery(
label="primary",
search_query=topic,
ranking_query=topic,
sources=["x"],
)
],
source_weights={"x": 1.0},
)
bundle = schema.RetrievalBundle()
with patch("lib.pipeline._retrieve_stream", return_value=(raw_items, {})):
pipeline._retry_thin_sources(
topic=topic,
bundle=bundle,
plan=plan,
config={},
depth="default",
date_range=("2026-07-15", "2026-08-14"),
runtime=schema.ProviderRuntime(
reasoning_provider="mock",
planner_model="mock",
rerank_model="mock",
),
mock=False,
rate_limited_sources=set(),
rate_limit_lock=threading.Lock(),
settings=pipeline.DEPTH_SETTINGS["default"],
)
x_items = bundle.items_by_source.get("x") or []
authors = {item.author for item in x_items}
assert "steipete" in authors, (
"thin-retry X path must defer the relevance floor the way Phase 1 does; "
"a zero-relevance subject-authored post must survive into the bundle"
)
assert "third_acct" in authors, (
"precondition: the companion on-topic post cleared the floor, so the "
"all-fail rescue cannot hide an incorrectly applied prune"
)

Some files were not shown because too many files have changed in this diff Show More