Compare commits

...

40 Commits

Author SHA1 Message Date
jakevin aae6e823b4 chore(release): 1.7.22 (#1586)
Release / release (push) Has been cancelled
E2E Headed Chrome / e2e-headed (macos-latest) (push) Has been cancelled
E2E Headed Chrome / e2e-headed (ubuntu-latest) (push) Has been cancelled
External CLI ergonomics + two adapter envelope/auth fixes.

- feat(external): longbridge CLI passthrough (#1584)
- feat(external-cli): brand alias rendering for ntn/dws/wecom-cli (#1585)
- fix(boss): map code=24 → AuthRequiredError (#1573)
- fix(weibo): unwrap page.evaluate envelope in read adapters (#1568)
2026-05-15 17:30:34 +08:00
jakevin 3f62cc45bf feat(external-cli): render brand alias for ambiguous executable names (#1585)
`ntn`, `dws`, and `wecom-cli` are opaque executable names — users seeing them
in `opencli list` or root help have no way to know they correspond to Notion,
DingTalk Workspace, and 企业微信. Repurpose the existing `package` field to
double as a human-readable brand label, so help output renders as
`ntn(notion)`, `dws(DingTalk Workspace)`, `wecom-cli(企业微信)`.

- `src/external-clis.yaml`: add `package:` to ntn / dws / wecom-cli
- `src/external.ts`: update JSDoc on `package` to cover both upstream
  distribution names (tg-cli, discord-cli) and brand labels (notion, 企业微信)
- `src/cli.ts:629` (`opencli list`): use `formatExternalCliLabel` so the
  listing matches root help, which already used it
- `src/external.test.ts`: regression test for brand-alias labels

Verification:
- npx vitest run --project unit src/external.test.ts: 9/9 pass
- npm run typecheck: clean
- npm run build: 813 manifest entries
- Smoke: `opencli list` and `opencli --help` both render the new labels
2026-05-15 16:37:27 +08:00
jakevin b6f352b318 feat(external): add longbridge cli (#1584) 2026-05-15 16:27:08 +08:00
Benjamin Liu dadf01b56f fix(weibo): unwrap page.evaluate envelope in read adapters (#1568)
* fix(weibo): unwrap page.evaluate envelope in read adapters (#1567)

`page.evaluate(...)` returns a `{ session, data }` envelope rather than
the raw IIFE return value, so all weibo cookie-strategy read adapters
silently dropped their results on v1.7.19:

- `getSelfUid` returned the envelope object instead of the uid string,
  so `'10001' + uid` produced `'10001[object Object]'` and every
  feed/me/favorites request hit a broken list_id.
- `feed`, `hot`, `comments`, `search`, `favorites` did `Array.isArray`
  on the envelope (always false) and returned `[]`.
- `me`, `user`, `post` returned the envelope wrapper itself instead of
  the inner profile/post object.

Same pattern as #1561 for xiaohongshu/rednote. Adds an
`unwrapEvaluateResult` helper to `clis/weibo/utils.js` (kept local
rather than cross-importing from `xiaohongshu/search.js` since weibo
is an unrelated site) and wraps every `await page.evaluate(...)` in
the 8 read adapters plus the two helper calls in `getSelfUid`.

Skipped `publish.js` (write command, out of scope for this read fix).

Verified live:
- `opencli weibo hot --limit 3` returns 3 real trending items
- `opencli weibo feed --limit 3` returns 3 timeline posts with
  correct `https://weibo.com/<uid>/<mblogid>` URLs (proves
  `getSelfUid` unwrap works)
- `opencli weibo me` returns the logged-in profile object
- All 20 weibo unit tests pass (6 new for `unwrapEvaluateResult`)

* fix(weibo): fail typed on malformed evaluate payloads

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 22:34:43 +08:00
Benjamin Liu 1239798d04 fix(boss): map code=24 (identity mismatch) to AuthRequiredError (#1573)
Recruiter-only BOSS commands (recommend, joblist, stats, resume, mark,
exchange, invite, greet, batchgreet) returned a generic
`COMMAND_EXEC: 请切换身份后再试 (code=24)` when called from a job-seeker
account. The original error hid the actionable bit: this command set
needs a recruiter (BOSS-side) account.

chatlist / chatmsg already special-case code=24 by falling back to the
geek-side fetch when --side=auto. Recruiter-only commands have no
geek-side equivalent and were just leaking the raw API code.

Fix: add a `checkRecruiterSide` step inside `assertOk` that maps
code=24 to AuthRequiredError with a clear message. All 9 recruiter-only
commands inherit it through their existing `bossFetch` calls; no
adapter-level changes needed. chatlist / chatmsg are unaffected because
they use `allowNonZero: true` and never hit the auto-error path.

Closes #1572.
2026-05-14 22:26:28 +08:00
jakevin 9ccc896585 chore(release): 1.7.21 (#1571)
E2E Headed Chrome / e2e-headed (macos-latest) (push) Has been cancelled
E2E Headed Chrome / e2e-headed (ubuntu-latest) (push) Has been cancelled
Release / release (push) Has been cancelled
2026-05-14 19:36:41 +08:00
jakevin 1a69f40a80 fix(social): use ephemeral adapter site sessions (#1569) 2026-05-14 19:22:42 +08:00
J.Chen 300607f692 fix(facebook/feed): add fallback extraction for empty article nodes (#1538)
* fix(facebook/feed): add fallback extraction for empty article nodes

Add fallback extraction for Facebook feed posts when [role=article] nodes exist but contain empty text. Includes diagnostic errors, content/author cleanup, nested-container dedupe, and an evaluate-script syntax regression test.

* fix(facebook): bound feed fallback extraction

* fix(facebook): keep feed fallback available after chrome articles

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 18:42:50 +08:00
J.Chen 42b5a4e68d feat(boss): support job-seeker chatlist and chatmsg (#1539)
* feat(boss): support job-seeker chatlist and chatmsg

* fix(boss): type chat-side failure boundaries

* fix(boss): guard malformed chat API payloads

---------

Co-authored-by: Jeff Chen <jeff@adtiming.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 18:41:02 +08:00
jakevin bccd275d66 test(extension): cover adapter group tiebreaker (#1566) 2026-05-14 18:04:07 +08:00
胡大头 edfa5f0da3 feat: add DuckDuckGo, Brave, and Yahoo web search adapters (#1546)
* feat: add DuckDuckGo, Brave, and Yahoo web search adapters

Add three new search engine adapters with browser-based DOM extraction:

- duckduckgo/search: Search DuckDuckGo via html.duckduckgo.com
  Supports region, time filters, and XHR-based pagination (--offset)
- duckduckgo/suggest: Search suggestion autocomplete (no browser needed)
- brave/search: Search Brave Search via search.brave.com
  Supports GET-based pagination (--offset)
- yahoo/search: Search Yahoo (Bing-powered) via search.yahoo.com
  Supports GET-based pagination (--page)

All search adapters use Strategy.PUBLIC with browser:true, navigating
the target site and extracting results via page.evaluate() DOM queries.
Includes full test coverage (16 tests).

* fix: use clampInt from shared utils and add adapter docs

- Replace Math.max/Math.min patterns with clampInt() from _shared/common.js
  to pass the typed-error-lint gate (4 silent-clamp violations resolved)
- Add adapter documentation for duckduckgo, brave, and yahoo to fix
  the doc-coverage CI check
- Regenerate cli-manifest.json and typed-error-lint-baseline.json

* fix: avoid silent-column-drop overlap in brave/yahoo extractors

Change buildExtractorJs to return arrays instead of objects whose keys
matched columns. This prevents silent-column-drop audit false positives
as per opencli-adapter-author conventions.

* fix(search): tighten browser search adapters

* chore(search): drop baseline churn

* fix(duckduckgo): execute search extractor safely

* fix(yahoo): reject unsafe redirect targets

---------

Co-authored-by: huzekang <huzekang@opencode.ai>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 17:54:34 +08:00
J.Chen 16b02bcc58 fix(extension): reuse existing adapter tab group (#1541)
* fix(extension): reuse existing adapter tab group

* fix(extension): choose best existing adapter group

---------

Co-authored-by: Jeff Chen <jeff@adtiming.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 17:15:39 +08:00
Iris Chen 5af2ff1d6c fix(xiaohongshu,rednote): unwrap page.evaluate envelope in search adapter (#1561)
* fix(xiaohongshu,rednote): unwrap page.evaluate envelope in search adapter

`page.evaluate(...)` returns a `{ session, data }` envelope rather than
the raw IIFE return value, but the search adapters were calling
`Array.isArray(payload)` directly on the envelope. `Array.isArray` is
always false on the envelope, so every search result was silently
dropped — status=success, exit 0, empty array, no error.

The rednote adapter had this same bug; both share `buildSearchExtractJs`
from `xiaohongshu/search.js`.

Introduces `unwrapEvaluateResult(payload)` as a shared helper in
`clis/xiaohongshu/search.js` (re-exported via the existing import line
from `rednote/search.js`). The helper is a defensive ternary: it
unwraps when payload looks like an envelope with an array `.data`,
otherwise it passes the value through unchanged. This keeps the change
back-compat with bridge versions that return the raw value, and
preserves the existing `Array.isArray(payload)` typecheck at each call
site.

Verified manually against `opencli xiaohongshu search "补墙洞"` (a query
known to return 20+ results in a logged-in browser tab): previously
`[]`, now returns the expected ranked rows with all declared columns
(`rank, title, author, likes, published_at, url`) populated.

Adds 5 unit tests for `unwrapEvaluateResult` covering raw array passthrough,
envelope unwrap, non-envelope object passthrough, null/undefined safety,
and the "data is not an array" guard. The existing 19 search tests in
`clis/xiaohongshu/search.test.js` still pass — the unwrap is invisible
to the existing mocks which already return raw arrays.

* fix(xhs): unwrap search evaluate envelopes

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 17:05:58 +08:00
jakevin 9c25bc7009 fix(ci): add Windows native binding lock entries (#1563) 2026-05-14 16:45:00 +08:00
jakevin 8c88a3cbf3 chore(release): 1.7.20 (#1562)
E2E Headed Chrome / e2e-headed (macos-latest) (push) Has been cancelled
E2E Headed Chrome / e2e-headed (ubuntu-latest) (push) Has been cancelled
Release / release (push) Has been cancelled
2026-05-14 16:33:19 +08:00
jakevin 9c4f4a3d30 fix(cli): show external CLI package aliases (#1560) 2026-05-14 16:28:18 +08:00
jakevin 29c135b656 refactor(notion): replace built-in CDP adapter with external ntn CLI (#1559)
* refactor(notion): replace built-in CDP adapter with external ntn CLI

Notion has shipped an official CLI at https://ntn.dev. It uses the
public Notion API (blocks / databases / properties / comments) instead
of reverse-engineering the Desktop UI, so it survives Notion app
updates and exposes a wider command surface than the in-tree adapter
could.

Changes:
  - `src/external-clis.yaml` — register `ntn` as first-class external CLI
    (binary `ntn`, homepage ntn.dev, install via the shell-pipe script
    on mac/linux)
  - `clis/notion/` — entire directory removed (8 commands: status /
    search / read / new / write / sidebar / favorites / export)
  - `docs/adapters/desktop/notion.md` — removed
  - `docs/.vitepress/config.mts` — drop nav entry
  - `docs/adapters/index.md` — drop adapter row
  - `README.md` / `README.zh-CN.md` — drop notion from feature lines,
    drop adapter table row, add `ntn` to CLI hub examples
  - `docs/index.md` / `docs/zh/index.md` / `docs/guide/getting-started.md`
    — drop notion from electron-control feature copy
  - `skills/opencli-usage/SKILL.md` — drop notion from electron list
  - `cli-manifest.json` — rebuilt with --allow-removals=8

Migration for users:
  `curl -fsSL https://ntn.dev | bash`  (or `opencli external install ntn`)
  Then use `opencli ntn <command>` in place of `opencli notion <command>`.

Rationale: the in-tree adapter was reverse-engineered against Notion
Desktop CDP and shipped only 8 commands. The official CLI gives users
the full Notion API surface and reduces our maintenance burden to zero.
Same pattern as gh / obsidian / lark-cli / tg-cli / discord-cli / wx-cli.

Verification:
  - `npx tsc --noEmit` clean
  - `npx vitest run --project unit` → 1091/1 skipped
  - `npm run build` (with --allow-removals=8) — manifest 809 entries
  - grep notion in user-facing docs (README / docs / skills) — only
    descriptive mentions remain in non-blocking places (comparison /
    site-recon / electron how-to / design doc), no broken adapter
    references

* fix(notion): align ntn external migration

* docs(notion): clarify ntn manual install
2026-05-14 16:12:17 +08:00
jakevin 7edf53783f fix(daemon): report unknown browser command results (#1558) 2026-05-14 14:30:13 +08:00
jakevin af7b94152f feat(twitter): add extractMedia parity to bookmarks + bookmark-folder (#1555)
Mirrors PR #1464 (list-tweets) and the timeline/search/tweets/likes/thread
family: spread `...extractMedia(legacy)` into the row and surface
`has_media` + `media_urls` columns. Pure parity, no behavior change for
existing callers — media keys do not collide with the original columns.

- bookmarks.js: import `extractMedia` from ./shared.js, spread into
  extractBookmarkTweet row, append columns, export __test__.
- bookmark-folder.js: same change on extractFolderTweet, export
  extractFolderTweet via __test__.
- bookmarks.test.js (new): baseline + photo + video + entities-only
  fallback + dedup + envelope + empty-envelope (8 tests).
- bookmark-folder.test.js: update existing baseline expectation with
  has_media/media_urls, add 3 new media tests (photo / mp4 / no-media).
- cli-manifest.json: regenerated; only the two `columns` entries change.

Reverse-validated: tests fail when extractMedia spread is removed.

Audits unchanged: typed-error-lint 189/189, silent-column-drop 102/103
(pre-existing main resolution noted but not consumed here).
2026-05-14 14:24:31 +08:00
Ocean 6b26aedd56 feat(twitter/list-tweets): include media via extractMedia (parity with timeline/search) (#1464)
* feat(twitter/list-tweets): include media via extractMedia (parity with timeline/search)

list-tweets was the only X recall path that dropped media. timeline.js and
search.js both call extractMedia(legacy) and emit has_media/media_urls;
list-tweets returned only text fields, so downstream consumers (e.g.
ml-scout's rate UI) couldn't render image/video thumbnails on tweets pulled
from a list timeline.

Changes:
- Import extractMedia from ./shared.js
- Spread extractMedia(legacy) into extractTimelineTweet return
- Add has_media, media_urls to columns array (--format columns parity)
- Update unit test to assert the new shape; add coverage for photo and
  video extraction

* chore(manifest): rebuild cli-manifest.json for list-tweets media columns

---------

Co-authored-by: ml-scout <ml-scout@anthropic.com>
2026-05-14 14:01:34 +08:00
jakevin 68b18cdbcd fix(extension): coalesce daemon websocket connects (#1554) 2026-05-14 13:57:38 +08:00
J.Chen cddc84776c docs(browser): clarify named session lifecycle (#1542)
* docs(browser): clarify named session lifecycle

* docs(browser): clarify owned versus bound sessions

---------

Co-authored-by: Jeff Chen <jeff@adtiming.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-14 13:38:41 +08:00
J.Chen 4f5fcd9acb fix(extension): keep active daemon websocket
Keep stale Browser Bridge WebSocket events from clobbering the active daemon connection.\n\nCo-authored-by: Jeff Chen <jeff@adtiming.com>
2026-05-14 13:38:05 +08:00
jakevin 40b2f75098 feat(external)!: drop -cli suffix from tg/discord/wx subcommand names (#1544)
The opencli external-CLI name is the user-typed subcommand; the binary is
what gets executed. The convention everywhere else (`gh`, `docker`,
`obsidian`, `vercel`, `dws`) is `name == binary`. Three entries violated
the convention: `tg-cli` / `discord-cli` / `wx-cli` registered an
opencli name with a `-cli` suffix that does NOT exist on the binary,
forcing the awkward double-prefix `opencli discord-cli dc` instead of
`opencli discord dc`.

The README's example column already showed the desired form
(`opencli tg search`, `opencli discord recent`, `opencli wx search`) —
only the yaml registration was out of sync.

Renames in `src/external-clis.yaml`:

* `name: tg-cli`     → `name: tg`       (binary: `tg`)
* `name: discord-cli`→ `name: discord`  (binary: `discord`)
* `name: wx-cli`     → `name: wx`       (binary: `wx`)

The `binary`, `homepage`, and `install` fields are unchanged — the
underlying packages (`kabi-tg-cli`, `kabi-discord-cli`, `@jackwener/wx-cli`)
keep their published names.

Other entries left as-is: `lark-cli`, `wecom-cli`, and `dws` already have
`name == binary` (their actual binaries are `lark-cli`, `wecom-cli`, `dws`).

BREAKING CHANGE: `opencli tg-cli ...`, `opencli discord-cli ...`,
`opencli wx-cli ...` no longer resolve. Use `opencli tg ...`,
`opencli discord ...`, `opencli wx ...` instead. The feature is recent
(shipped 2026-05) so impact is expected to be minimal.
2026-05-14 04:13:39 +08:00
jakevin feab24f76c chore(release): 1.7.19 (#1543)
E2E Headed Chrome / e2e-headed (macos-latest) (push) Has been cancelled
E2E Headed Chrome / e2e-headed (ubuntu-latest) (push) Has been cancelled
Release / release (push) Has been cancelled
2026-05-14 02:33:56 +08:00
jakevin 8ef7e903b8 feat(twitter): default tweets to logged-in user + fix sibling envelope-unwrap silent bug (#1531)
* feat(twitter): default tweets to logged-in user + fix sibling envelope-unwrap silent bug

Primary: make `opencli twitter tweets` default to the logged-in user
when no username is given, so agents can pull their own posts without
needing to know their own handle. Mirrors the existing self-detection
pattern in twitter/profile and twitter/likes (AppTabBar_Profile_Link
probe on /home, then UserByScreenName lookup). Description + help
string now mention the default so agents discover it.

Consistency pass — profile/likes/following/followers: the
self-detection in these four siblings was silently broken because
page.evaluate() primitive returns come back through the CDP bridge
wrapped as `{session: 'site:twitter', data: '/<handle>'}` (same
envelope root cause as #1525). They called `.replace()` directly on
the envelope object → TypeError surfaced as AUTH_REQUIRED 'Could not
detect logged-in user', even for logged-in users. Wrap each probe
with unwrapBrowserResult so the bare href string survives. Also:
- Add an explicit page.goto('/home') + page.wait(primaryColumn)
  before the probe in likes/following so the AppTabBar sidebar is
  guaranteed rendered (framework pre-nav lands on bare x.com without
  the sidebar mounted).
- following.js: switch its probe from the function-literal form
  `() => {...}` to a template-string. Confirmed live: function-literal
  silently drops primitive returns entirely — bridge returns
  `{session}` with no `data` field at all, while template-string
  returns `{session, data}` as expected.

Out of scope (pre-existing, flagged as follow-up): likes/following
have additional downstream evaluate paths (userId/GraphQL fetch) that
still drop or envelope their results; they return [] or
'Could not find user' even after this PR. Same daemon-side bug class
as #1525.

Live-verified:
  opencli twitter tweets --limit 2     → own tweets (@jakevin7)
  opencli twitter profile              → own profile

Tests 227/227, audits typed-error-lint 189 + silent-column-drop 103
unchanged, manifest stable at 816 entries.

* fix(twitter): validate self-detected handles

* fix(twitter): unwrap downstream self evaluate results
2026-05-13 22:51:48 +08:00
ppop123 7c5bafd49b fix(twitter): repair list-add / list-tweets / lists / following after 2026-05 changes (#1503)
* fix(twitter): unwrap page.evaluate primitive returns in lists/list-tweets/following

The opencli >=1.7.x browser bridge wraps page.evaluate's primitive return
values as { session, data: <value> }. Adapters that destructure .data
inline (e.g. data.queryId, data.viewer) keep working because the wrapper
spreads object-typed responses to the top level, but ones that consume
the return value as a bare string broke:

- twitter list-tweets: the dynamically resolved queryId (a string) became
  {session, data:"..."}. Interpolating that into the GraphQL URL produced
  /i/api/graphql/[object Object]/ListLatestTweetsTimeline, giving "HTTP
  400: queryId may have expired".
- twitter lists: same on ListsManagementPageTimeline queryId.
- twitter following: same shape bug on the href read from the profile
  link, producing "TypeError: href.replace is not a function" when no
  --user is given.

Add a small unwrap() helper at each call site so primitive returns are
extracted from the wrapper before use. Object-typed GraphQL responses
are left as-is since they rely on spread semantics.

* fix(twitter): rewrite list-add to use ListAddMember GraphQL mutation

In 2026-05 X replaced the "Add/remove from Lists" modal dialog with a
full-page route (/i/lists/add_member). The previous UI flow no longer
works:

  Save button not found in dialog (X expected text Save/Done).
  Dialog structure may have changed.

The mutation that the dialog used to fire (ListAddMember) is still the
right primitive — and the surrounding adapter already calls X GraphQL
APIs directly to resolve userId and verify member_count. Drop the UI
flow entirely and call ListAddMember directly via fetch in the page
context.

Wins:
- Works again on current X UI (verified 2026-05-12 on x.com).
- ~10x faster: no goto-profile + click-caret + scroll-dialog round trips.
- One less moving piece — no dependency on Chrome extension's nativeClick
  for this command.

Implementation notes:
- LIST_ADD_MEMBER_QUERY_ID is a 2026-05 fallback; resolveTwitterQueryId
  does live lookup from the loaded client-web bundle, matching the
  pattern already used elsewhere in the twitter clis.
- X's ListAddMember response routinely contains a non-fatal partial
  decode error on default_banner_media_results (code 214, Validation /
  BadRequestError) alongside a fully populated data.list. We treat the
  call as failed only when data.list / member_count is missing, and
  ignore decode-flavored errors confined to banner fields.
- Same opencli >=1.7.x { session, data } primitive-wrap behavior that
  the previous commit addressed applies here: userId from the
  UserByScreenName call needs unwrap before being interpolated into
  the mutation body, otherwise X parses "[object Object]" as user_id
  and returns "strconv.ParseInt ... invalid syntax".

Verified flows:
- noop (already a member) → status: noop, member_count unchanged.
- new add (e.g. @AnthropicAI on a fresh list) → status: success,
  member_count incremented.

Trade-off: rejection signals (e.g. X declining to add @deepseek_ai)
look indistinguishable from noop at the response level, since X returns
HTTP 200 with member_count unchanged. Documented in the success message.

* fix(twitter): integrate list media and harden list-add

---------

Co-authored-by: wangyan <wy@wang-yan-Air.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-13 22:35:57 +08:00
jakevin f66996a148 fix(twitter): raise cursor pagination guard
Fix hidden pagination ceilings across Twitter cursor-pagination adapters.\n\nCo-authored-by: Mingming Lou <1109198+lmmsoft@users.noreply.github.com>
2026-05-13 22:14:05 +08:00
jakevin 4fac911425 feat(zhihu): add answer-detail to fetch a single answer's full content (#1528)
* feat(zhihu): add answer-detail to fetch a single answer's full content

The existing `zhihu answer` adapter is a write (post an answer); the
listing `zhihu question` truncates each answer's body to 200 chars.
There was no way to fetch one specific answer's full content by id.

New read adapter `zhihu answer-detail`:

- Accepts a bare numeric answer id, a typed target `answer:<qid>:<aid>`,
  or a full Zhihu answer URL (the form you paste from a browser).
- Calls `/api/v4/answers/<aid>?include=content,voteup_count,...,question`
  inside the cookie-bearing page context (Strategy.COOKIE).
- Returns a single row with id / author / votes / comments /
  question_id / question_title / url / created_at / updated_at /
  content. The content column is the full stripped answer body by
  default — no silent truncation. `--max-content N` is an opt-in user
  cap (mirroring the wikipedia `page` flag), and `--max-content 0`
  (the default) means "no cap, full content".

Important precision note: Zhihu answer ids since 2024 routinely
exceed `Number.MAX_SAFE_INTEGER` (the test fixture uses the real id
`1937205528846655537`). `data.id` is round-tripped through browser
`JSON.parse` and would round to `1937205528846655500`, so the adapter
deliberately ignores `data.id` for the canonical row id and anchors
it to the already-validated input string instead. A regression test
locks this contract in by mocking `data.id = 0` and asserting the row
still carries the parsed input id.

Typed errors: bad input → INVALID_INPUT; 401/403 → AuthRequiredError;
other HTTP / null → FETCH_ERROR. No silent fallbacks, no sentinel
strings.

Live-verified against the example URL — fetched 5547 votes / 165
comments / 1937205528846655537-end-to-end. 16 unit tests, audits
unchanged (typed-error-lint 189/189, silent-column-drop 103/103),
manifest 816→817.

* fix(zhihu): tighten answer-detail contracts
2026-05-13 21:47:37 +08:00
xcd_git b52da639a3 fix(google-scholar/search): wrap evaluate return to fix serialization (#1525)
* fix(google-scholar/search): wrap evaluate return to fix serialization

Same issue as google/search: page.evaluate() serializes JS arrays as
plain objects across the CDP boundary, causing Array.isArray() to
return false. The adapter silently returned [] instead of results.

Also replace fixed page.wait(3) with selector-based wait for
.gs_r.gs_or.gs_scl with a 3s fallback.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(google-scholar): type search evaluate payload

* chore: rerun google scholar search checks

---------

Co-authored-by: cxiao <chuda.xiao@wuerzburg-dynamics.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-13 19:07:39 +08:00
Joseph赛博阿隆 b1dca04ddd fix(xiaohongshu): parseLikes should handle 2.1w / 1.5万 / 1.2k shortforms (#1504)
* fix(xiaohongshu): parseLikes should handle 2.1w / 1.5万 / 1.2k shortforms

Xiaohongshu renders top-popular comment like-counts as shortened
strings like '2.1w' / '1.1万' / '1.2k' once they exceed ~10 000.
The previous parseLikes only matched bare digits via /^\d+$/ and
silently returned 0 for any shortform, which inverted the sort
order: the highest-liked comments (often 10k+) ranked last while
mid-tier comments with plain numeric counts (e.g. 7569) appeared
on top.

Repro on any popular xiaohongshu thread (>10 000 likes on a top
comment): with --format json the most-upvoted parent rows show
"likes": 0.

This patch keeps the original fast path for plain integers and
adds a single regex for the well-known shortform suffixes:

  - w / 万 -> *10000
  - k / 千 -> *1000
  - trailing '+' tolerated (e.g. '999+')
  - unknown shapes still fall back to 0 (no behavior change)

Note: parseLikes runs inside the IIFE injected via page.evaluate(),
so the existing comments.test.js mock harness (which stubs
evaluate's return value directly) does not exercise it. A future
refactor that exports parseLikes for direct testing would be a
separate change.

Affects both top-level comments and 楼中楼 sub-replies (same
helper).

* fix(xiaohongshu): parse comment like shortforms safely

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-13 18:47:20 +08:00
jakevin f481585ba1 chore: drop util.styleText to support Node v20+ (#1524)
* chore: drop util.styleText to support Node v20+

util.styleText was added in Node v21.7.0 / v20.12.0. v21.0.0-v21.6.x and
v20.0.0-v20.11.x throw `SyntaxError: ... styleText` at startup because the
import resolves before any user code runs (a real user reported this on
v21.2.0).

OpenCLI is primarily agent-facing — terminal colors are noise to consumers,
and the [OK] / [WARN] / [FAIL] / ℹ / ⚠ / ✖ markers we already write carry
the semantic info that colors only repeated. Strip styleText entirely from
logger / output / doctor / tui / update-check / cli / download/progress /
commands/daemon and clean up the resulting awkward `${'literal'}` template
fragments. engines.node now reads ">=20.0.0".

This removes the Node-version coupling that A/B fixes would only have
papered over.

* fix(runtime): truly support Node v20+ by aligning guard + undici

Follow-up to the styleText removal: declaring engines.node >=20.0.0 is
not enough on its own. Two coupled barriers remained:

- src/runtime-detect.ts: MIN_SUPPORTED_NODE_MAJOR = 21 explicitly
  rejected v20 at startup
- undici@^8.0.2 declares engines.node >=22.19.0; Node 20/21 crash on
  webidl.util.markAsUncloneable before any user code runs

Lower the guard to 20 and downgrade undici to ^6.25.0 (engines >=18.17,
retains Agent / EnvHttpProxyAgent / fetch / Dispatcher). Smoke-tested
--help / doctor / list on Node v20.0.0, v21.2.0, v22.22.2. 213/213
targeted unit tests pass.
2026-05-13 18:33:21 +08:00
lenovobenben 723f2b9147 feat(zhihu): paginate question answers and recommendations (#1517)
* feat(zhihu): paginate question answers and recommendations

* fix(zhihu): drop Math.min limit clamp and 'unknown' sentinel

Two audit-driven fixes on top of feat/zhihu-pagination-recommend:

1. question.js: replace `Math.min(answerLimit, 20)` with a named
   constant `ZHIHU_PAGE_SIZE = 20`. The Zhihu API caps `limit` at 20
   per request anyway, and the pagination loop already trims to the
   user-requested `answerLimit` via `answers.length >= answerLimit`,
   so the Math.min silent-clamp was both unnecessary and tripped the
   silent-clamp audit. Updates the existing unit test to expect the
   API-max page size in the fetch URL with an explanatory comment.

2. recommend.js: rebuild the dedup key without the `'unknown'`
   sentinel. The old form `\`\${target.type || 'unknown'}:\${target.id}\``
   collapsed distinct typed items into the same bucket whenever
   `target.type` was missing, and tripped the silent-sentinel audit.
   New form: prefer `type:targetId`, fall back to `__feed:item.id`,
   and when neither id is available keep the row but skip dedup
   (surfacing potentially-duplicate items beats silently dropping
   them).

Audits unchanged (typed-error-lint 189/189, silent-column-drop
103/103). All 88 zhihu tests pass.

---------

Co-authored-by: lihaidong <lihaidong@kingsoft.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-13 18:28:44 +08:00
Benjamin Liu 2babed84e9 fix(xiaohongshu+rednote/search): fall back to href-based note cards when section.note-item class is dropped (#1506) (#1507)
Issue #1506 reports `opencli xiaohongshu search` returning `[]` even though
the page visibly has results. Trace evidence: xhs ships a render variant
where each note card is a bare `<section>` (no `note-item` class), so
the three `section.note-item` selectors in this file all match zero
elements.

Three call sites in the shared search IIFEs now use the same defensive
selector strategy: try the legacy `section.note-item` class first, then
fall back to any `<section>` that wraps a `/search_result/...` or
`/explore/...` link. The change is in the xiaohongshu file so the
rednote adapter (which imports `buildSearchExtractJs` and
`buildScrollUntilJs` from here) picks it up automatically.

Extraction-side title selector also gets a fallback: when no
`.title` / `.note-title` element matches, read the first `<span>`
inside the search-result link, which is where the bare-section render
puts the caption per the trace.

## Verification

`npx vitest run --project adapter clis/xiaohongshu/`: 105/105 green
(existing test suite unchanged, passes on both legacy and fallback paths).

Live verify on rednote (same code path, account-safe):

```
$ opencli rednote search "美食" --limit 3 -f json
[ {rank:1, title:"在朋友家吃过一次..."}, {rank:2, title:"我的15💰晚餐..."}, {rank:3, title:"干净饮食🫛..."} ]
```

Legacy `section.note-item` path is exercised here (rednote still renders
the class) and returns identical row shape to before the fix, confirming
no regression on the working path.

Live verify on xiaohongshu cannot be performed here (no logged-in xhs
session on the test machine; xhs account-ban risk per the project's
operational guidance). The fix is structural: the new `<section>` shape
the issue reporter traced is reachable through the fallback, and the
existing test fixture keeps the legacy path green.

`npx tsc --noEmit` clean. `npm run build` 815 manifest entries unchanged
shape. `silent-column-drop` / `typed-error-lint` baselines unchanged.

Closes #1506
Refs #1500
2026-05-13 18:24:55 +08:00
陈家名 a6ca53c7cf fix: clamp download progress percentages (#1520)
* fix: clamp download progress percentages

* test(download): cover unknown progress total

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-13 18:24:26 +08:00
jakevin c3912d8e5c feat(reddit/read): --expand-more via /api/morechildren + 7-kind typed errors (#1492)
* feat(reddit/read): add --expand-more via /api/morechildren + 7-kind discriminated union

PR B of the rdt-cli parity follow-up (after PR #1491, see #1481 thread).
Closes the second-largest gap: Reddit's "[+N more replies]" stubs were
opaque markers in the comment tree. With --expand-more, the adapter
follows them by POST-ing the t1 ids to /api/morechildren.json, then
re-threads the returned things back into the tree by parent_id before
walking it.

New args:

- `--expand-more` (bool, default false) — turn on stub expansion.
- `--expand-rounds <N>` (int, default 2, range [1, 5]) — Reddit returns
  fresh "more" stubs at the expansion depth boundary, so up to N rounds
  are run. Strictly validated via `parseExpandRounds` — out-of-range
  raises ArgumentError BEFORE `page.goto`, no silent clamp.

Boy-Scout: the in-browser script now returns a 7-kind discriminated
union instead of a flat row array (matching the PR #1428 / #1491
sediment). Each kind maps 1:1 to a typed error on the Node side:

  - `inaccessible` → EmptyResultError
      401/403/404 on /comments/<id>.json (post-specific access, not
      session-level auth — applies the PR #1491 review-side sediment
      "inaccessible-resource vs session-auth").
  - `auth`         → AuthRequiredError
      401/403 on /api/morechildren (expand-write endpoints often demand
      a logged-in session even when the read endpoint is anonymous).
  - `http`         → CommandExecutionError
  - `malformed`    → CommandExecutionError
      200 with unexpected envelope shape — schema drift, not empty.
  - `parser-drift` → CommandExecutionError
      tree had t1 entries but the walker produced no rows (PR #1491
      review-side sediment "post-construction 0 rows + pre-walk
      non-empty = parser drift, not legitimate empty").
  - `expand-failed`→ CommandExecutionError
      /api/morechildren returned a non-empty json.errors array.
  - `ok`           → returns rows[].

Intermediate keys (kind / detail / httpStatus / where / rows /
expandMeta) deliberately avoid the declared columns (type / author /
score / text) per the PR #1329 silent-column-drop sediment.

Tests:
  clis/reddit/read.test.js — 11 tests
    - Adapter shape (browser / siteSession / columns / args)
    - --expand-more / --expand-rounds present with correct types/defaults
    - parseExpandRounds default / range / non-integer rejection
    - Pre-navigation validation (bad --expand-rounds doesn't reach goto)
    - kind=ok happy path (POST + L0 rows)
    - 6-kind error → typed error mapping
    - Unknown envelope shape → CommandExecutionError
    - Evaluate script embeds expandMore/expandRounds/sort/limit literals
    - Evaluate script contains /api/morechildren POST scaffolding
    - Evaluate script never names declared columns as intermediate keys

Full reddit suite 48/48; full project 3402/3402.

Audits: typed-error-lint 189/189 (0 new), silent-column-drop 103/103
(0 new). Manifest 815 → 815 (existing read entry gets 2 new args).

Existing --limit / --depth / --replies / --max-length keep their
original Math.max-style behaviour (grandfathered in the baseline);
only the new --expand-rounds flag fails fast per the typed-errors
standard.

Refs: https://github.com/jackwener/rdt-cli (browse.read --expand-more)

* fix(reddit): preserve expanded comment tree order

* fix(reddit): fail on partial morechildren expansion
2026-05-13 18:13:11 +08:00
darthjaja 67599ea67c fix(twitter): repair search and tweets readback (#1512)
* fix(twitter): repair search and tweets readback

* fix(twitter): prefer baked operation features when bundle parse returns empty

The bundle parser in resolveTwitterOperationMetadata locates the queryId via
`queryId:"..."` inside a ~2500-char snippet around the operationName marker,
then independently extracts `featureSwitches:[...]` and `fieldToggles:[...]`
via separate regexes. When minification rearranges the snippet (or the
snippet window truncates before the array), either regex can miss while
queryId still resolves; keysToFlags(undefined) then returns {}.

sanitizeTwitterOperationMetadata previously accepted any object as
features / fieldToggles, including {}. Twitter's GraphQL endpoint rejects
SearchTimeline / UserTweets requests with empty features (HTTP 400),
surfacing a misleading "queryId may have expired" error — the queryId is
fresh; only the feature flags are missing.

Guard against this by deferring to the baked fallback whenever the resolved
map is empty. Adds a JSDOM-free unit test that, reverse-validated, fails on
the un-fixed code with the exact silent-fallback shape.

Refs PR #1512

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-13 18:11:41 +08:00
darthjaja f321a6096d fix(twitter): make reply submission robust (#1511) 2026-05-13 18:10:19 +08:00
xcd_git 59ebf551f0 fix(google/search): wrap evaluate return value in object to fix serialization (#1523)
page.evaluate() serializes JS arrays as plain objects, causing
Array.isArray() to return false and the adapter to throw NOT_FOUND
even when results exist. Wrap the return value in {items: results}
and extract via wrapper.items to avoid the type check issue.

Co-authored-by: cxiao <chuda.xiao@wuerzburg-dynamics.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-13 18:05:03 +08:00
jakevin 04a57029b3 ci(adapter-test): gate adapter-test off pull_request trigger (#1522)
Per @WAWQAQ direction (DM): trim PR-time CI to fast-feedback only.
adapter-test (~30-60s) is the next-largest PR wait after e2e-headed
(which #1521 just removed). Adapter authors typically run focused tests
locally before pushing (`npm run test:adapter`); CI duplication adds
queue latency without catching new classes of bugs.

PR-time CI surface now:
  - typecheck / unit (~1 min)
  - lint gates (typed-error / silent-column-drop)
  - build × 3 platforms

Adapter test guards (still strict):
  - push to main / dev
  - nightly cron
  - workflow_dispatch (manual when an adapter-heavy PR really wants the
    signal before merge)

Same gate as smoke-test (`if: github.event_name == 'push' || schedule
|| workflow_dispatch`) for consistency.
2026-05-13 17:55:23 +08:00
160 changed files with 8277 additions and 3656 deletions
+5 -1
View File
@@ -110,8 +110,12 @@ jobs:
- name: Run unit tests under Bun
run: bun vitest run --project unit --reporter=verbose
# Adapter tests are pure unit tests — OS doesn't affect results.
# Adapter tests are pure unit tests — OS doesn't affect results. Gated off
# `pull_request` to keep PR CI under ~2 minutes; adapter authors run focused
# tests locally before pushing, and `push` to main / nightly cron / manual
# dispatch still guard the merged state.
adapter-test:
if: github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
needs: build
steps:
+94 -13
View File
@@ -1,27 +1,108 @@
# Changelog
## Unreleased
## [1.7.22](https://github.com/jackwener/opencli/compare/v1.7.21...v1.7.22) (2026-05-15)
### Bug Fixes
* **browser** — `page.evaluate()` / `evaluateInFrame()` now return the user JavaScript value directly. Browser Bridge `exec` previously routed through a shared `pageScopedResult` helper that spread / wrapped the lease's `session` into the result `data`, contaminating arbitrary user returns: array / primitive returns came back as `{ session, data }` envelopes, and plain-object returns had an extra `session` key injected (overwriting any user `session` field). `google search` and `xiaohongshu search` were the visible repro — Chrome rendered results correctly but adapters extracted an empty array. Fixed in extension 1.0.14 by reverting `pageScopedResult` to its pre-1461 form (`{ id, ok, data, page }`); no client-side unwrap is needed.
* **google/search** — wait for `#rso a h3` before extracting, falling back to the existing fixed wait. On Chrome 148 + Linux Wayland the DOM can settle before SERP anchors are populated, making extraction return empty even with the envelope bug fixed.
* **xiaohongshu/search** — extract initially visible cards before scrolling, then merge post-scroll rows by URL. Xiaohongshu's virtualized masonry layout can evict the initial cards from the DOM after scroll, so the previous always-scroll-then-extract flow could lose the top results.
External CLI ergonomics + two adapter envelope/auth fixes. New `longbridge` external CLI entry; `opencli list` / root help now render human-readable brand labels for executables whose bare name is ambiguous.
### Features
* **browser** — add `page.evaluate(fn, ...args)` for type-safe browser-context evaluation with JSON-serialized arguments. String evaluation remains supported, but new adapter code should use function form to avoid implicit `wrapForEval` auto-IIFE magic.
* **external** — add the Longbridge CLI as a built-in external CLI passthrough (`opencli longbridge ...`) for Longbridge OpenAPI market data, account, and trading commands. ([#1584](https://github.com/jackwener/opencli/issues/1584))
* **external-cli** — render brand alias `name(package)` in `opencli list` and root help when the bare executable name is ambiguous. Built-in entries `ntn``ntn(notion)`, `dws``dws(DingTalk Workspace)`, `wecom-cli``wecom-cli(企业微信)` now self-explain in help output. `package` field is repurposed to cover both upstream distribution names (e.g. `tg-cli`) and human-readable brand labels (e.g. `notion`, `企业微信`). ([#1585](https://github.com/jackwener/opencli/issues/1585))
### ⚠ BREAKING CHANGES
### Bug Fixes
* **browser** — replace the `--session <name>` flag with a `<session>` positional argument that immediately follows `browser`. `opencli browser work click 12` instead of `opencli browser --session work click 12`; `opencli browser work bind` instead of `opencli browser bind --session work`. Required-flag semantics are now encoded structurally as a positional, matching the Docker/git convention for required operation-target identifiers. The internal `--session` flag is preserved for the daemon protocol and for direct `program.parseAsync` callers but is no longer part of the user-facing surface.
* **env** — remove `OPENCLI_KEEP_TAB`. The flag was a debugging shortcut, not a config dimension: `--keep-tab true|false` on the command line is the single source of truth, and adapter `siteSession: 'persistent'` already pins persistent site tabs as a hard constraint. Removing the env eliminates a globally-leaking process state that overrode every browser command in the shell.
* **extension** — remove the internal `surface\\0session` command-session backdoor. Browser Bridge commands now route only through structured `session` + `surface` fields; lease-key strings remain an extension-internal registry detail.
* **boss** — map `code=24` (identity mismatch) to `AuthRequiredError` so re-login is signaled instead of surfacing as a generic API error. ([#1573](https://github.com/jackwener/opencli/issues/1573))
* **weibo** — unwrap Browser Bridge `page.evaluate` envelopes in read adapters. ([#1568](https://github.com/jackwener/opencli/issues/1568))
## [1.7.21](https://github.com/jackwener/opencli/compare/v1.7.20...v1.7.21) (2026-05-14)
Adapter polish release: new web search adapters, better Browser Bridge tab group reuse, and social adapters returning to one-shot tab leases. Extension package version is bumped to 1.0.15 for the Browser Bridge fix.
### Features
* **search** — add DuckDuckGo, Brave, and Yahoo web search adapters. ([#1546](https://github.com/jackwener/opencli/issues/1546))
* **boss** — support job-seeker `chatlist` and `chatmsg` adapters. ([#1539](https://github.com/jackwener/opencli/issues/1539))
### Bug Fixes
* **extension** — reuse existing `OpenCLI Adapter` tab groups before creating new ones, including cross-window discovery, legacy `OpenCLI` title fallback, and deterministic candidate selection. ([#1541](https://github.com/jackwener/opencli/issues/1541))
* **twitter, reddit** — default browser-backed social adapters back to ephemeral tab leases. Twitter/X and Reddit commands now release their site tab after each run while keeping the shared Adapter window available for reuse; persistent sessions remain reserved for AI/chat-style adapters that need long-lived conversation state. ([#1569](https://github.com/jackwener/opencli/issues/1569))
* **xiaohongshu, rednote** — unwrap Browser Bridge `page.evaluate` envelopes in search adapters. ([#1561](https://github.com/jackwener/opencli/issues/1561))
* **facebook/feed** — add fallback extraction for empty article nodes. ([#1538](https://github.com/jackwener/opencli/issues/1538))
### Internal
* **extension 1.0.14** — `pageScopedResult` no longer injects `session` into `data`. The field had no consumers and contaminated `exec` results with arbitrary user-JS shapes; routing-relevant identity is already exposed via `Result.page`.
* **extension 1.0.13** — remove the internal command-session lease-key backdoor.
* **ci** — add Windows native binding lockfile entries for Rolldown/Rollup optional packages. ([#1563](https://github.com/jackwener/opencli/issues/1563))
* **extension** — add regression coverage for the adapter tab group `groupId` tiebreaker. ([#1566](https://github.com/jackwener/opencli/issues/1566))
## [1.7.20](https://github.com/jackwener/opencli/compare/v1.7.19...v1.7.20) (2026-05-14)
External CLI surface cleanup + Browser Bridge WebSocket lifecycle hardening. Two BREAKING changes around external CLIs: built-in `tg`/`discord`/`wx` (was `tg-cli`/`discord-cli`/`wx-cli`) now match their real binary names, and Notion's in-tree CDP adapter is replaced by the official `ntn` external CLI.
### ⚠ BREAKING CHANGES
* **notion** — remove the in-tree `clis/notion/` CDP-on-Desktop adapter (8 commands: `status` / `search` / `read` / `new` / `write` / `sidebar` / `favorites` / `export`). Notion has shipped an official CLI at <https://ntn.dev>, registered as a first-class external CLI in `external-clis.yaml`. Migration: install `ntn` from <https://ntn.dev> (`curl -fsSL https://ntn.dev | bash`), then use `opencli ntn <command>`. Auto-install is intentionally not configured because the official installer is a shell script while OpenCLI external installs run shell-free command strings. The official CLI uses the public Notion API rather than reverse-engineering the Desktop UI, so it survives Notion app updates and exposes a wider command surface (blocks / databases / properties / comments) than the reverse-engineered adapter could. ([#1559](https://github.com/jackwener/opencli/issues/1559))
* **external** — drop the `-cli` suffix from built-in external CLI subcommand names. `opencli tg-cli`, `opencli discord-cli`, `opencli wx-cli` are now `opencli tg`, `opencli discord`, `opencli wx`, matching the real binary names that those tools install as. Root help still shows the package lineage as `tg(tg-cli)` / `discord(discord-cli)` / `wx(wx-cli)`. ([#1544](https://github.com/jackwener/opencli/issues/1544))
### Features
* **twitter** — `bookmarks` and `bookmark-folder` now include media via `extractMedia`, reaching parity with `timeline` / `search`. ([#1555](https://github.com/jackwener/opencli/issues/1555))
* **twitter/list-tweets** — include media via `extractMedia` (parity with `timeline` / `search`). ([#1464](https://github.com/jackwener/opencli/issues/1464))
### Bug Fixes
* **daemon** — report ambiguous browser command outcomes with a distinct `command_result_unknown` errorCode and `503` when the extension WebSocket drops between command dispatch and result delivery. `sendCommandRaw()` treats this code as hard non-retryable, so write-side commands (`navigate` / `click` / `type` / `eval`) won't be silently re-issued and double-executed. Daemon exposes a `commandResultUnknown` counter on `/status` for future observability. ([#1558](https://github.com/jackwener/opencli/issues/1558))
* **extension** — keep active daemon WebSocket; stale sockets no longer clobber active connection (`onopen` / `onclose` / `onmessage` are all gated by `ws !== thisWs` short-circuit), and `safeSend` only fires when `readyState === OPEN`. ([#1540](https://github.com/jackwener/opencli/issues/1540))
* **extension** — coalesce concurrent daemon WebSocket connects via an in-flight promise. Startup / keepalive / reconnect triggering `connect()` during the daemon-probe or context-lookup async gap no longer creates duplicate real WebSocket connections. ([#1554](https://github.com/jackwener/opencli/issues/1554))
* **external** — distinguish external CLI executable names from distribution/project names in root help. Built-in aliases such as `tg`, `discord`, `wx` remain the callable `opencli <name> ...` entrypoints while help renders `tg(tg-cli)`, `discord(discord-cli)`, `wx(wx-cli)` to show their package lineage. ([#1560](https://github.com/jackwener/opencli/issues/1560))
### Docs
* **browser** — clarify named session lifecycle in the Browser Bridge guide. ([#1542](https://github.com/jackwener/opencli/issues/1542))
## [1.7.19](https://github.com/jackwener/opencli/compare/v1.7.18...v1.7.19) (2026-05-14)
Major hotfix + simplification batch. Extension bumped to 1.0.14. Node floor lowered to v20 so the long tail of Node v20v21.6 users no longer crashes at module load. `opencli browser` user surface replaces required-flag `--session <name>` with a `<session>` positional. `page.evaluate(fn, ...args)` adds a type-safe alternative to the implicit auto-IIFE string form. Twitter cursor pagination no longer silently caps at ~500 items.
### ⚠ BREAKING CHANGES
* **browser** — replace the `--session <name>` flag with a `<session>` positional argument that immediately follows `browser`. `opencli browser work click 12` instead of `opencli browser --session work click 12`; `opencli browser work bind` instead of `opencli browser bind --session work`. Required-flag semantics are now encoded structurally as a positional, matching the Docker/git convention for required operation-target identifiers. The internal `--session` flag is preserved for the daemon protocol and for direct `program.parseAsync` callers but is no longer part of the user-facing surface. ([#1505](https://github.com/jackwener/opencli/issues/1505))
* **env** — remove `OPENCLI_KEEP_TAB`. The flag was a debugging shortcut, not a config dimension: `--keep-tab true|false` on the command line is the single source of truth, and adapter `siteSession: 'persistent'` already pins persistent site tabs as a hard constraint. Removing the env eliminates a globally-leaking process state that overrode every browser command in the shell. ([#1509](https://github.com/jackwener/opencli/issues/1509))
* **extension** — remove the internal `surface\\0session` command-session backdoor. Browser Bridge commands now route only through structured `session` + `surface` fields; lease-key strings remain an extension-internal registry detail. ([#1510](https://github.com/jackwener/opencli/issues/1510))
### Features
* **browser** — add `page.evaluate(fn, ...args)` for type-safe browser-context evaluation with JSON-serialized arguments. String evaluation remains supported, but new adapter code should use function form to avoid implicit `wrapForEval` auto-IIFE magic. ([#1508](https://github.com/jackwener/opencli/issues/1508))
* **twitter** — default `tweets` command to the logged-in user when `user` is omitted, and fix the sibling envelope-unwrap silent bug. ([#1531](https://github.com/jackwener/opencli/issues/1531))
* **zhihu** — add `answer-detail` to fetch a single answer's full content. ([#1528](https://github.com/jackwener/opencli/issues/1528))
* **zhihu** — paginate question answers and recommendations. ([#1517](https://github.com/jackwener/opencli/issues/1517))
* **reddit/read** — `--expand-more` via `/api/morechildren` + 7-kind typed errors. ([#1492](https://github.com/jackwener/opencli/issues/1492))
* **reddit** — add `whoami`, `home`, `subreddit-info` read commands. ([#1491](https://github.com/jackwener/opencli/issues/1491))
* **ctrip** — add `hotel-search` + flight browser-mode commands. ([#1489](https://github.com/jackwener/opencli/issues/1489))
### Bug Fixes
* **browser** — `page.evaluate()` / `evaluateInFrame()` now return the user JavaScript value directly. Browser Bridge `exec` previously routed through a shared `pageScopedResult` helper that spread / wrapped the lease's `session` into the result `data`, contaminating arbitrary user returns: array / primitive returns came back as `{ session, data }` envelopes, and plain-object returns had an extra `session` key injected (overwriting any user `session` field). `google search` and `xiaohongshu search` were the visible repro — Chrome rendered results correctly but adapters extracted an empty array. Fixed in extension 1.0.14 by reverting `pageScopedResult` to its pre-1461 form (`{ id, ok, data, page }`); no client-side unwrap is needed. ([#1518](https://github.com/jackwener/opencli/issues/1518))
* **twitter** — raise fixed cursor-pagination caps in `bookmarks` / `likes` / `tweets` / `timeline` / `bookmark-folder` / `list-tweets` / `search` / `following`. The old `i < 5` / `i < 10` literals and following's `Math.ceil(limit / 50) + 2` formula imposed hidden result ceilings below `--limit`; the loop now treats the page count as a high runaway guard while `--limit` and cursor exhaustion control normal pagination. ([#1532](https://github.com/jackwener/opencli/issues/1532))
* **twitter** — repair `list-add` / `list-tweets` / `lists` / `following` after 2026-05 site changes. ([#1503](https://github.com/jackwener/opencli/issues/1503))
* **twitter** — repair `search` and `tweets` readback. ([#1512](https://github.com/jackwener/opencli/issues/1512))
* **twitter** — make reply submission robust. ([#1511](https://github.com/jackwener/opencli/issues/1511))
* **google/search** — wait for `#rso a h3` before extracting, falling back to the existing fixed wait. On Chrome 148 + Linux Wayland the DOM can settle before SERP anchors are populated, making extraction return empty even with the envelope bug fixed. ([#1518](https://github.com/jackwener/opencli/issues/1518))
* **google/search** — wrap evaluate return value in object to fix serialization. ([#1523](https://github.com/jackwener/opencli/issues/1523))
* **google-scholar/search** — wrap evaluate return to fix serialization. ([#1525](https://github.com/jackwener/opencli/issues/1525))
* **xiaohongshu/search** — extract initially visible cards before scrolling, then merge post-scroll rows by URL. Xiaohongshu's virtualized masonry layout can evict the initial cards from the DOM after scroll, so the previous always-scroll-then-extract flow could lose the top results. ([#1518](https://github.com/jackwener/opencli/issues/1518))
* **xiaohongshu** — `parseLikes` handles `2.1w` / `1.5万` / `1.2k` shortforms. ([#1504](https://github.com/jackwener/opencli/issues/1504))
* **xiaohongshu+rednote/search** — fall back to href-based note cards when `section.note-item` class is dropped. ([#1507](https://github.com/jackwener/opencli/issues/1507))
* **xueqiu** — `kline` / `earnings-date` format dates in Asia/Shanghai instead of UTC. ([#1498](https://github.com/jackwener/opencli/issues/1498))
* **download** — clamp progress percentages. ([#1520](https://github.com/jackwener/opencli/issues/1520))
### Internal
* **runtime** — lower the Node floor to `>=20.0.0`. Three coupled changes: drop all `util.styleText()` usage (added in Node v21.7.0 / v20.12.0; previously crashed v21.0v21.6 at module load), downgrade `undici` from `^8.0.2` (engines `>=22.19.0`) to `^6.25.0` (engines `>=18.17`, retains `Agent` / `EnvHttpProxyAgent` / `fetch`), and lower `MIN_SUPPORTED_NODE_MAJOR` from 21 to 20 so the startup guard matches the declared `engines.node`. Smoke-tested on v20.0.0 / v21.2.0 / v22.22.2. The semantic markers (`[OK]` / `[WARN]` / `[FAIL]` / `` / `⚠` / `✖`) keep their meaning; ANSI colors were redundant for the primarily agent-facing CLI. ([#1524](https://github.com/jackwener/opencli/issues/1524))
* **extension 1.0.14** — `pageScopedResult` no longer injects `session` into `data`. The field had no consumers and contaminated `exec` results with arbitrary user-JS shapes; routing-relevant identity is already exposed via `Result.page`. ([#1518](https://github.com/jackwener/opencli/issues/1518))
* **extension 1.0.13** — remove the internal command-session lease-key backdoor. ([#1510](https://github.com/jackwener/opencli/issues/1510))
* **ci** — drop `e2e-headed` and `adapter-test` from `pull_request` triggers (kept on `push` to main / nightly / `workflow_dispatch`). PR-time CI now targets ~2 min wall-time. ([#1521](https://github.com/jackwener/opencli/issues/1521), [#1522](https://github.com/jackwener/opencli/issues/1522))
* **scripts** — auto-refresh `dist/` before `build-manifest`. ([#1490](https://github.com/jackwener/opencli/issues/1490))
## [1.7.18](https://github.com/jackwener/opencli/compare/v1.7.17...v1.7.18) (2026-05-12)
+12 -9
View File
@@ -14,17 +14,17 @@ OpenCLI gives you one surface for three different kinds of automation:
- **Let AI Agents operate any website** — install the `opencli-adapter-author` skill in your AI agent (Claude Code, Cursor, etc.), and it can navigate, click, type/fill, extract, and inspect any page through your logged-in browser via `opencli browser` primitives.
- **Write new adapters** end-to-end with `opencli browser` + the `opencli-adapter-author` skill, which guides from first recon through field decoding, code, and `opencli browser verify`.
It also works as a **CLI hub** for local tools such as `gh`, `docker`, `tg-cli`, `discord-cli`, `wx-cli`, and other binaries you register yourself, plus **desktop app adapters** for Electron apps like Cursor, Codex, Antigravity, ChatGPT, and Notion.
It also works as a **CLI hub** for local tools such as `gh`, `docker`, `longbridge`, `tg`, `discord`, `wx`, `ntn` (Notion), and other binaries you register yourself, plus **desktop app adapters** for Electron apps like Cursor, Codex, Antigravity, and ChatGPT.
## Highlights
- **Desktop App Control** — Drive Electron apps (Cursor, Codex, ChatGPT, Notion, etc.) directly from the terminal via CDP.
- **Desktop App Control** — Drive Electron apps (Cursor, Codex, ChatGPT, etc.) directly from the terminal via CDP.
- **Browser Automation for AI Agents** — Install the `opencli-adapter-author` skill, and your AI agent can operate any website: navigate, click, type/fill, extract, screenshot — all through your logged-in Chrome session.
- **Multi-profile Browser Bridge** — Install the extension in each Chrome profile you want to use, then route commands with `--profile`, `OPENCLI_PROFILE`, or `opencli profile use`.
- **Website → CLI** — Turn any website into a deterministic CLI: 100+ site surfaces are already registered, or write your own with the `opencli-adapter-author` skill + `opencli browser verify`.
- **Account-safe** — Reuses Chrome/Chromium logged-in state; your credentials never leave the browser.
- **AI Agent ready** — One skill takes you from site recon through API discovery, field decoding, adapter writing, and verification.
- **CLI Hub** — Discover, auto-install, and passthrough commands to any external CLI (gh, docker, obsidian, tg-cli, discord-cli, wx-cli, etc).
- **CLI Hub** — Discover, auto-install, and passthrough commands to any external CLI (gh, docker, obsidian, tg, discord, wx, etc).
- **Zero LLM cost** — No tokens consumed at runtime. Run 10,000 times and pay nothing.
- **Deterministic** — Same command, same output schema, every time. Pipeable, scriptable, CI-friendly.
@@ -181,7 +181,7 @@ When the site you need is not yet covered, use the `opencli-adapter-author` skil
OpenCLI is not only for websites. It can also:
- expose local binaries like `gh`, `docker`, `obsidian`, `tg-cli`, `discord-cli`, `wx-cli`, or custom tools through `opencli <tool> ...`
- expose local binaries like `gh`, `docker`, `obsidian`, `tg`, `discord`, `wx`, or custom tools through `opencli <tool> ...`
- control Electron desktop apps through dedicated adapters and CDP-backed integrations
## Prerequisites
@@ -283,19 +283,21 @@ To load the source Browser Bridge extension:
## CLI Hub
OpenCLI acts as a universal hub for your existing command-line tools — unified discovery, pure passthrough execution, and auto-install (if a tool isn't installed, OpenCLI runs `brew install <tool>` automatically before re-running the command).
OpenCLI acts as a universal hub for your existing command-line tools — unified discovery, pure passthrough execution, and auto-install when a safe package-manager command is configured.
| External CLI | Description | Example |
|--------------|-------------|---------|
| **gh** | GitHub CLI | `opencli gh pr list --limit 5` |
| **obsidian** | Obsidian vault management | `opencli obsidian search query="AI"` |
| **docker** | Docker | `opencli docker ps` |
| **longbridge** | Longbridge CLI — market data, account management, and trading via Longbridge OpenAPI | `opencli longbridge quote TSLA.US --format json` |
| **ntn** | Notion CLI — official Notion API CLI for pages, databases, blocks, search, comments | `opencli ntn pages list` |
| **lark-cli** | Lark/Feishu — messages, docs, calendar, tasks, 200+ commands | `opencli lark-cli calendar +agenda` |
| **dws** | DingTalk — cross-platform CLI for DingTalk's full suite, designed for humans and AI agents | `opencli dws msg send --to user "hello"` |
| **wecom-cli** | WeCom/企业微信 — CLI for WeCom open platform, for humans and AI agents | `opencli wecom-cli msg send --to user "hello"` |
| **tg-cli** | Telegram — local-first sync, search, and export via MTProto for AI agents | `opencli tg search "AI news" -f json` |
| **discord-cli** | Discord — local-first sync, search, and export via SQLite for AI agents | `opencli discord recent --channel general` |
| **wx-cli** | WeChat — query local WeChat data: sessions, messages, search, contacts, export | `opencli wx search "OpenCLI"` |
| **tg(tg-cli)** | Telegram — local-first sync, search, and export via MTProto for AI agents | `opencli tg search "AI news" -f json` |
| **discord(discord-cli)** | Discord — local-first sync, search, and export via SQLite for AI agents | `opencli discord recent --channel general` |
| **wx(wx-cli)** | WeChat — query local WeChat data: sessions, messages, search, contacts, export | `opencli wx search "OpenCLI"` |
| **vercel** | Vercel — deploy projects, manage domains, env vars, logs | `opencli vercel deploy --prod` |
**Register your own** — add any local CLI so AI agents can discover it via `opencli list`:
@@ -304,6 +306,8 @@ OpenCLI acts as a universal hub for your existing command-line tools — unified
opencli external register mycli
```
**Manual install** — some external CLIs use official shell-script installers rather than shell-free package-manager commands. For `ntn`, install from <https://ntn.dev> first, then run `opencli ntn ...`.
### Desktop App Adapters
Control Electron desktop apps directly from the terminal. Each adapter has its own detailed documentation:
@@ -315,7 +319,6 @@ Control Electron desktop apps directly from the terminal. Each adapter has its o
| **Antigravity** | Control Antigravity Ultra from terminal | [Doc](./docs/adapters/desktop/antigravity.md) |
| **ChatGPT App** | Automate ChatGPT macOS desktop app | [Doc](./docs/adapters/desktop/chatgpt-app.md) |
| **ChatWise** | Multi-LLM client (GPT-4, Claude, Gemini) | [Doc](./docs/adapters/desktop/chatwise.md) |
| **Notion** | Search, read, write Notion pages | [Doc](./docs/adapters/desktop/notion.md) |
| **Discord** | Discord Desktop — messages, channels, servers | [Doc](./docs/adapters/desktop/discord.md) |
| **Doubao** | Control Doubao AI desktop app via CDP | [Doc](./docs/adapters/desktop/doubao-app.md) |
+10 -10
View File
@@ -14,16 +14,16 @@ OpenCLI 可以用同一套 CLI 做三类事情:
- **让 AI Agent 操作任意网站**:在你的 AI AgentClaude Code、Cursor 等)中安装 `opencli-adapter-author` skill,Agent 就能用你的已登录浏览器导航、点击、输入/填充、提取任意网页内容。
- **把新网站写成 CLI**:用 `opencli browser` 原语 + `opencli-adapter-author` skill,从站点侦察、API 发现、字段解码到 `opencli browser verify` 一条龙。
除了网站能力,OpenCLI 还是一个 **CLI 枢纽**:你可以把 `gh``docker``tg-cli``discord-cli``wx-cli` 等本地工具统一注册到 `opencli` 下,也可以通过桌面端适配器控制 Cursor、Codex、Antigravity、ChatGPT、Notion 等 Electron 应用。
除了网站能力,OpenCLI 还是一个 **CLI 枢纽**:你可以把 `gh``docker``longbridge``tg``discord``wx``ntn`Notion等本地工具统一注册到 `opencli` 下,也可以通过桌面端适配器控制 Cursor、Codex、Antigravity、ChatGPT 等 Electron 应用。
## 亮点
- **桌面应用控制** — 通过 CDP 直接在终端驱动 Electron 应用(Cursor、Codex、ChatGPT、Notion 等)。
- **桌面应用控制** — 通过 CDP 直接在终端驱动 Electron 应用(Cursor、Codex、ChatGPT 等)。
- **AI Agent 浏览器自动化** — 安装 `opencli-adapter-author` skill,你的 AI Agent 就能操作任意网站:导航、点击、输入/填充、提取、截图——全部通过你的已登录 Chrome 会话完成。
- **网站 → CLI** — 把任何网站变成确定性 CLI:100+ 站点能力已注册,或用 `opencli-adapter-author` skill + `opencli browser verify` 自己写。
- **账号安全** — 复用 Chrome/Chromium 登录态,凭证永远不会离开浏览器。
- **面向 AI Agent** — 一个 skill 带你走完站点侦察、API 发现、字段解码、适配器编写、验证的全流程。
- **CLI 枢纽** — 统一发现、自动安装、纯透传任何外部 CLI(gh、docker、obsidian、tg-cli、discord-cli、wx-cli 等)。
- **CLI 枢纽** — 统一发现、自动安装、纯透传任何外部 CLI(gh、docker、obsidian、tg、discord、wx 等)。
- **零 LLM 成本** — 运行时不消耗模型 token,跑 10,000 次也不花一分钱。
- **确定性输出** — 相同命令,相同输出结构,每次一致。可管道、可脚本、CI 友好。
@@ -165,7 +165,7 @@ Agent 在内部自动处理所有 `opencli browser` 命令——你只需用自
OpenCLI 不只是网站 CLI,还可以:
- 统一代理本地二进制工具,例如 `gh``docker``obsidian``tg-cli``discord-cli``wx-cli`
- 统一代理本地二进制工具,例如 `gh``docker``obsidian``tg``discord``wx`
- 通过专门适配器和 CDP 集成控制 Electron 桌面应用
## 前置要求
@@ -241,7 +241,6 @@ npm link
| **chatwise** | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` | 桌面端 |
| **doubao** | `status` `new` `send` `read` `ask` `history` `detail` `meeting-summary` `meeting-transcript` | 浏览器 |
| **doubao-app** | `status` `new` `send` `read` `ask` `screenshot` `dump` | 桌面端 |
| **notion** | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` | 桌面端 |
| **discord-app** | `status` `send` `read` `channels` `servers` `search` `members` | 桌面端 |
| **v2ex** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | 公开 / 浏览器 |
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `comments` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 浏览器 |
@@ -333,17 +332,19 @@ OpenCLI 也可以作为你现有命令行工具的统一入口,负责发现、
| **gh** | GitHub CLI | `opencli gh pr list --limit 5` |
| **obsidian** | Obsidian 仓库管理 | `opencli obsidian search query="AI"` |
| **docker** | Docker 命令行工具 | `opencli docker ps` |
| **longbridge** | Longbridge CLI — 通过 Longbridge OpenAPI 获取行情、账户和交易能力 | `opencli longbridge quote TSLA.US --format json` |
| **ntn** | Notion CLI — 基于官方 Notion API 的页面、数据库、块、搜索、评论命令 | `opencli ntn pages list` |
| **lark-cli** | 飞书 CLI — 消息、文档、日历、任务,200+ 命令 | `opencli lark-cli calendar +agenda` |
| **dws** | 钉钉 CLI — 钉钉全套产品能力的跨平台命令行工具,支持人类和 AI Agent 使用 | `opencli dws msg send --to user "hello"` |
| **wecom-cli** | 企业微信 CLI — 企业微信开放平台命令行工具,支持人类和 AI Agent 使用 | `opencli wecom-cli msg send --to user "hello"` |
| **tg-cli** | Telegram CLI — 基于 MTProto 的本地优先同步、搜索、导出,面向 AI Agent | `opencli tg search "AI news" -f json` |
| **discord-cli** | Discord CLI — 基于 SQLite 的本地优先同步、搜索、导出,面向 AI Agent | `opencli discord recent --channel general` |
| **wx-cli** | 微信本地数据 CLI — 会话、聊天记录、搜索、联系人、导出 | `opencli wx search "OpenCLI"` |
| **tg(tg-cli)** | Telegram CLI — 基于 MTProto 的本地优先同步、搜索、导出,面向 AI Agent | `opencli tg search "AI news" -f json` |
| **discord(discord-cli)** | Discord CLI — 基于 SQLite 的本地优先同步、搜索、导出,面向 AI Agent | `opencli discord recent --channel general` |
| **wx(wx-cli)** | 微信本地数据 CLI — 会话、聊天记录、搜索、联系人、导出 | `opencli wx search "OpenCLI"` |
| **vercel** | Vercel — 部署项目、管理域名、环境变量、日志 | `opencli vercel deploy --prod` |
**零配置透传**:OpenCLI 会把你的输入原样转发给底层二进制,保留原生 stdout / stderr 行为。
**自动安装**:如果你运行 `opencli gh ...` 时系统中还没有 `gh`,OpenCLI 会优先尝试通过系统包管理器安装,然后自动重试命令
**自动安装**:如果某个外部 CLI 配置了安全的包管理器安装命令,OpenCLI 会优先尝试安装后再执行;`ntn` 的官方安装方式是 shell 脚本,请先按 <https://ntn.dev> 手动安装
**注册自定义本地 CLI**
@@ -362,7 +363,6 @@ opencli register mycli
| **Antigravity** | 在终端直接控制 Antigravity Ultra | [Doc](./docs/adapters/desktop/antigravity.md) |
| **ChatGPT App** | 自动化操作 ChatGPT macOS 桌面客户端 | [Doc](./docs/adapters/desktop/chatgpt-app.md) |
| **ChatWise** | 多 LLM 客户端(GPT-4、Claude、Gemini | [Doc](./docs/adapters/desktop/chatwise.md) |
| **Notion** | 搜索、读取、写入 Notion 页面 | [Doc](./docs/adapters/desktop/notion.md) |
| **Discord** | Discord 桌面版 — 消息、频道、服务器 | [Doc](./docs/adapters/desktop/discord.md) |
| **Doubao** | 通过 CDP 控制豆包桌面应用 | [Doc](./docs/adapters/desktop/doubao-app.md) |
+348 -257
View File
@@ -3484,7 +3484,7 @@
{
"site": "boss",
"name": "chatlist",
"description": "BOSS直聘查看聊天列表(招聘端)",
"description": "BOSS直聘查看聊天列表(招聘端/求职端",
"access": "read",
"domain": "www.zhipin.com",
"strategy": "cookie",
@@ -3509,12 +3509,26 @@
"type": "str",
"default": "0",
"required": false,
"help": "Filter by job ID (0=all)"
"help": "Filter by job ID (0=all, boss side only)"
},
{
"name": "side",
"type": "str",
"default": "auto",
"required": false,
"help": "Identity side: auto (default), boss (recruiter), or geek (job-seeker)",
"choices": [
"auto",
"boss",
"geek"
]
}
],
"columns": [
"name",
"company",
"job",
"title",
"last_msg",
"last_time",
"uid",
@@ -3528,7 +3542,7 @@
{
"site": "boss",
"name": "chatmsg",
"description": "BOSS直聘查看与候选人的聊天消息",
"description": "BOSS直聘查看聊天消息历史(招聘端/求职端)",
"access": "read",
"domain": "www.zhipin.com",
"strategy": "cookie",
@@ -3547,6 +3561,18 @@
"default": 1,
"required": false,
"help": "Page number"
},
{
"name": "side",
"type": "str",
"default": "auto",
"required": false,
"help": "Identity side: auto (default), boss (recruiter), or geek (job-seeker)",
"choices": [
"auto",
"boss",
"geek"
]
}
],
"columns": [
@@ -4009,6 +4035,47 @@
"sourceFile": "boss/stats.js",
"navigateBefore": false
},
{
"site": "brave",
"name": "search",
"description": "Search Brave Search",
"access": "read",
"domain": "search.brave.com",
"strategy": "public",
"browser": true,
"args": [
{
"name": "keyword",
"type": "str",
"required": true,
"positional": true,
"help": "Search query"
},
{
"name": "limit",
"type": "int",
"default": 10,
"required": false,
"help": "Number of results per page (max 18)"
},
{
"name": "offset",
"type": "int",
"default": 0,
"required": false,
"help": "Page offset (0, 1, 2...). Brave returns ~18 results per page"
}
],
"columns": [
"rank",
"title",
"url",
"snippet"
],
"type": "js",
"modulePath": "brave/search.js",
"sourceFile": "brave/search.js"
},
{
"site": "chaoxing",
"name": "assignments",
@@ -8704,6 +8771,93 @@
"sourceFile": "douyin/videos.js",
"navigateBefore": "https://creator.douyin.com"
},
{
"site": "duckduckgo",
"name": "search",
"description": "Search DuckDuckGo",
"access": "read",
"domain": "html.duckduckgo.com",
"strategy": "public",
"browser": true,
"args": [
{
"name": "keyword",
"type": "str",
"required": true,
"positional": true,
"help": "Search query"
},
{
"name": "limit",
"type": "int",
"default": 10,
"required": false,
"help": "Number of results per page (1-10). For multi-page, use --offset"
},
{
"name": "offset",
"type": "int",
"default": 0,
"required": false,
"help": "Result offset for pagination (0, 10, 20...). Uses XHR POST internally"
},
{
"name": "region",
"type": "str",
"required": false,
"help": "Region code (e.g. jp-jp, us-en, cn-zh). Default: all regions"
},
{
"name": "time",
"type": "str",
"required": false,
"help": "Time range: d (day), w (week), m (month), y (year)"
}
],
"columns": [
"rank",
"title",
"url",
"snippet",
"displayUrl",
"icon",
"resultType"
],
"type": "js",
"modulePath": "duckduckgo/search.js",
"sourceFile": "duckduckgo/search.js"
},
{
"site": "duckduckgo",
"name": "suggest",
"description": "DuckDuckGo search suggestions",
"access": "read",
"domain": "duckduckgo.com",
"strategy": "public",
"browser": false,
"args": [
{
"name": "keyword",
"type": "str",
"required": true,
"positional": true,
"help": "Search query prefix"
},
{
"name": "limit",
"type": "int",
"default": 8,
"required": false,
"help": "Max number of suggestions"
}
],
"columns": [
"phrase"
],
"type": "js",
"modulePath": "duckduckgo/suggest.js",
"sourceFile": "duckduckgo/suggest.js"
},
{
"site": "eastmoney",
"name": "announcement",
@@ -9387,7 +9541,7 @@
"type": "js",
"modulePath": "facebook/feed.js",
"sourceFile": "facebook/feed.js",
"navigateBefore": "https://www.facebook.com"
"navigateBefore": false
},
{
"site": "facebook",
@@ -15753,181 +15907,6 @@
"sourceFile": "notebooklm/summary.js",
"navigateBefore": false
},
{
"site": "notion",
"name": "export",
"description": "Export the current Notion page as Markdown",
"access": "read",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [
{
"name": "output",
"type": "str",
"required": false,
"help": "Output file (default: /tmp/notion-export.md)"
}
],
"columns": [
"Status",
"File"
],
"type": "js",
"modulePath": "notion/export.js",
"sourceFile": "notion/export.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "favorites",
"description": "List pages from the Notion Favorites section in the sidebar",
"access": "read",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [],
"columns": [
"Index",
"Title",
"Icon"
],
"type": "js",
"modulePath": "notion/favorites.js",
"sourceFile": "notion/favorites.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "new",
"description": "Create a new page in Notion",
"access": "write",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [
{
"name": "title",
"type": "str",
"required": false,
"positional": true,
"help": "Page title (optional)"
}
],
"columns": [
"Status"
],
"type": "js",
"modulePath": "notion/new.js",
"sourceFile": "notion/new.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "read",
"description": "Read the content of the currently open Notion page",
"access": "read",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [],
"columns": [
"Title",
"Content"
],
"type": "js",
"modulePath": "notion/read.js",
"sourceFile": "notion/read.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "search",
"description": "Search pages and databases in Notion via Quick Find (Cmd+P)",
"access": "read",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [
{
"name": "query",
"type": "str",
"required": true,
"positional": true,
"help": "Search query"
}
],
"columns": [
"Index",
"Title"
],
"type": "js",
"modulePath": "notion/search.js",
"sourceFile": "notion/search.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "sidebar",
"description": "List pages and databases from the Notion sidebar",
"access": "read",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [],
"columns": [
"Index",
"Title"
],
"type": "js",
"modulePath": "notion/sidebar.js",
"sourceFile": "notion/sidebar.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "status",
"description": "Check active CDP connection to Notion Desktop",
"access": "read",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [],
"columns": [
"Status",
"Url",
"Title"
],
"type": "js",
"modulePath": "notion/status.js",
"sourceFile": "notion/status.js",
"navigateBefore": true
},
{
"site": "notion",
"name": "write",
"description": "Append text content to the currently open Notion page",
"access": "write",
"domain": "localhost",
"strategy": "ui",
"browser": true,
"args": [
{
"name": "text",
"type": "str",
"required": true,
"positional": true,
"help": "Text to append to the page"
}
],
"columns": [
"Status"
],
"type": "js",
"modulePath": "notion/write.js",
"sourceFile": "notion/write.js",
"navigateBefore": true
},
{
"site": "nowcoder",
"name": "companies",
@@ -19165,8 +19144,7 @@
"type": "js",
"modulePath": "reddit/comment.js",
"sourceFile": "reddit/comment.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19196,8 +19174,7 @@
"type": "js",
"modulePath": "reddit/frontpage.js",
"sourceFile": "reddit/frontpage.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19229,8 +19206,7 @@
"type": "js",
"modulePath": "reddit/home.js",
"sourceFile": "reddit/home.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19299,8 +19275,7 @@
"type": "js",
"modulePath": "reddit/popular.js",
"sourceFile": "reddit/popular.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19352,6 +19327,20 @@
"default": 2000,
"required": false,
"help": "Max characters per comment body (min 100)"
},
{
"name": "expand-more",
"type": "bool",
"default": false,
"required": false,
"help": "Follow Reddit \"more comments\" stubs by calling /api/morechildren.json"
},
{
"name": "expand-rounds",
"type": "int",
"default": 2,
"required": false,
"help": "Max expansion passes when --expand-more is on (15; each round can fan out new \"more\" stubs)"
}
],
"columns": [
@@ -19363,8 +19352,7 @@
"type": "js",
"modulePath": "reddit/read.js",
"sourceFile": "reddit/read.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19430,8 +19418,7 @@
"type": "js",
"modulePath": "reddit/save.js",
"sourceFile": "reddit/save.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19460,8 +19447,7 @@
"type": "js",
"modulePath": "reddit/saved.js",
"sourceFile": "reddit/saved.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19519,8 +19505,7 @@
"type": "js",
"modulePath": "reddit/search.js",
"sourceFile": "reddit/search.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19570,8 +19555,7 @@
"type": "js",
"modulePath": "reddit/subreddit.js",
"sourceFile": "reddit/subreddit.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19597,8 +19581,7 @@
"type": "js",
"modulePath": "reddit/subreddit-info.js",
"sourceFile": "reddit/subreddit-info.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19631,8 +19614,7 @@
"type": "js",
"modulePath": "reddit/subscribe.js",
"sourceFile": "reddit/subscribe.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19665,8 +19647,7 @@
"type": "js",
"modulePath": "reddit/upvote.js",
"sourceFile": "reddit/upvote.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19695,8 +19676,7 @@
"type": "js",
"modulePath": "reddit/upvoted.js",
"sourceFile": "reddit/upvoted.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19722,8 +19702,7 @@
"type": "js",
"modulePath": "reddit/user.js",
"sourceFile": "reddit/user.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19758,8 +19737,7 @@
"type": "js",
"modulePath": "reddit/user-comments.js",
"sourceFile": "reddit/user-comments.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19795,8 +19773,7 @@
"type": "js",
"modulePath": "reddit/user-posts.js",
"sourceFile": "reddit/user-posts.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "reddit",
@@ -19814,8 +19791,7 @@
"type": "js",
"modulePath": "reddit/whoami.js",
"sourceFile": "reddit/whoami.js",
"navigateBefore": "https://reddit.com",
"siteSession": "persistent"
"navigateBefore": "https://reddit.com"
},
{
"site": "rednote",
@@ -22603,8 +22579,7 @@
"type": "js",
"modulePath": "twitter/article.js",
"sourceFile": "twitter/article.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -22697,13 +22672,14 @@
"retweets",
"bookmarks",
"created_at",
"url"
"url",
"has_media",
"media_urls"
],
"type": "js",
"modulePath": "twitter/bookmark-folder.js",
"sourceFile": "twitter/bookmark-folder.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -22723,8 +22699,7 @@
"type": "js",
"modulePath": "twitter/bookmark-folders.js",
"sourceFile": "twitter/bookmark-folders.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -22758,13 +22733,14 @@
"retweets",
"bookmarks",
"created_at",
"url"
"url",
"has_media",
"media_urls"
],
"type": "js",
"modulePath": "twitter/bookmarks.js",
"sourceFile": "twitter/bookmarks.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -22838,8 +22814,7 @@
"type": "js",
"modulePath": "twitter/download.js",
"sourceFile": "twitter/download.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -22899,8 +22874,7 @@
"type": "js",
"modulePath": "twitter/followers.js",
"sourceFile": "twitter/followers.js",
"navigateBefore": true,
"siteSession": "persistent"
"navigateBefore": true
},
{
"site": "twitter",
@@ -22935,8 +22909,7 @@
"type": "js",
"modulePath": "twitter/following.js",
"sourceFile": "twitter/following.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23036,8 +23009,7 @@
"type": "js",
"modulePath": "twitter/likes.js",
"sourceFile": "twitter/likes.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23150,13 +23122,14 @@
"retweets",
"replies",
"created_at",
"url"
"url",
"has_media",
"media_urls"
],
"type": "js",
"modulePath": "twitter/list-tweets.js",
"sourceFile": "twitter/list-tweets.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23185,8 +23158,7 @@
"type": "js",
"modulePath": "twitter/lists.js",
"sourceFile": "twitter/lists.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23215,8 +23187,7 @@
"type": "js",
"modulePath": "twitter/notifications.js",
"sourceFile": "twitter/notifications.js",
"navigateBefore": true,
"siteSession": "persistent"
"navigateBefore": true
},
{
"site": "twitter",
@@ -23244,7 +23215,9 @@
"columns": [
"status",
"message",
"text"
"text",
"id",
"url"
],
"type": "js",
"modulePath": "twitter/post.js",
@@ -23284,8 +23257,7 @@
"type": "js",
"modulePath": "twitter/profile.js",
"sourceFile": "twitter/profile.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23372,7 +23344,8 @@
"columns": [
"status",
"message",
"text"
"text",
"url"
],
"type": "js",
"modulePath": "twitter/reply.js",
@@ -23460,7 +23433,7 @@
"description": "Search Twitter/X for tweets, with optional --from / --has / --exclude / --product filters mapped to X's search operators",
"access": "read",
"domain": "x.com",
"strategy": "intercept",
"strategy": "cookie",
"browser": true,
"args": [
{
@@ -23553,8 +23526,7 @@
"type": "js",
"modulePath": "twitter/search.js",
"sourceFile": "twitter/search.js",
"navigateBefore": true,
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23600,8 +23572,7 @@
"type": "js",
"modulePath": "twitter/thread.js",
"sourceFile": "twitter/thread.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23654,8 +23625,7 @@
"type": "js",
"modulePath": "twitter/timeline.js",
"sourceFile": "twitter/timeline.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -23682,13 +23652,12 @@
"type": "js",
"modulePath": "twitter/trending.js",
"sourceFile": "twitter/trending.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
"name": "tweets",
"description": "Fetch a Twitter user's most recent tweets (chronological, excludes pinned)",
"description": "Fetch a Twitter user's most recent tweets (chronological, excludes pinned; defaults to the logged-in user when no username is given)",
"access": "read",
"domain": "x.com",
"strategy": "cookie",
@@ -23697,9 +23666,9 @@
{
"name": "username",
"type": "string",
"required": true,
"required": false,
"positional": true,
"help": "Twitter screen name (with or without @)"
"help": "Twitter screen name (with or without @). Defaults to the logged-in user when omitted."
},
{
"name": "limit",
@@ -23733,8 +23702,7 @@
"type": "js",
"modulePath": "twitter/tweets.js",
"sourceFile": "twitter/tweets.js",
"navigateBefore": "https://x.com",
"siteSession": "persistent"
"navigateBefore": "https://x.com"
},
{
"site": "twitter",
@@ -26851,6 +26819,47 @@
"sourceFile": "xueqiu/watchlist.js",
"navigateBefore": "https://xueqiu.com"
},
{
"site": "yahoo",
"name": "search",
"description": "Search Yahoo (powered by Bing)",
"access": "read",
"domain": "search.yahoo.com",
"strategy": "public",
"browser": true,
"args": [
{
"name": "keyword",
"type": "str",
"required": true,
"positional": true,
"help": "Search query"
},
{
"name": "limit",
"type": "int",
"default": 7,
"required": false,
"help": "Number of results per page (max 7)"
},
{
"name": "page",
"type": "int",
"default": 1,
"required": false,
"help": "Page number (1, 2, 3...). Yahoo returns ~7 results per page"
}
],
"columns": [
"rank",
"title",
"url",
"snippet"
],
"type": "js",
"modulePath": "yahoo/search.js",
"sourceFile": "yahoo/search.js"
},
{
"site": "yahoo-finance",
"name": "quote",
@@ -28195,6 +28204,47 @@
"sourceFile": "zhihu/answer.js",
"navigateBefore": "https://www.zhihu.com"
},
{
"site": "zhihu",
"name": "answer-detail",
"description": "知乎单个回答完整内容(按 answer ID 获取)",
"access": "read",
"domain": "www.zhihu.com",
"strategy": "cookie",
"browser": true,
"args": [
{
"name": "id",
"type": "str",
"required": true,
"positional": true,
"help": "Answer ID, full Zhihu answer URL, or typed target (answer:<qid>:<aid>)"
},
{
"name": "max-content",
"type": "int",
"default": 0,
"required": false,
"help": "Optional cap on stripped content length in characters (0 = no truncation, return the full answer)"
}
],
"columns": [
"id",
"author",
"votes",
"comments",
"question_id",
"question_title",
"url",
"created_at",
"updated_at",
"content"
],
"type": "js",
"modulePath": "zhihu/answer-detail.js",
"sourceFile": "zhihu/answer-detail.js",
"navigateBefore": "https://www.zhihu.com"
},
{
"site": "zhihu",
"name": "collection",
@@ -28529,7 +28579,18 @@
"type": "int",
"default": 5,
"required": false,
"help": "Number of answers"
"help": "Number of answers (max 1000; use normal-sized requests)"
},
{
"name": "sort",
"type": "str",
"default": "default",
"required": false,
"help": "Answer order: default or created",
"choices": [
"default",
"created"
]
}
],
"columns": [
@@ -28543,6 +28604,36 @@
"sourceFile": "zhihu/question.js",
"navigateBefore": "https://www.zhihu.com"
},
{
"site": "zhihu",
"name": "recommend",
"description": "知乎首页推荐",
"access": "read",
"domain": "www.zhihu.com",
"strategy": "cookie",
"browser": true,
"args": [
{
"name": "limit",
"type": "int",
"default": 20,
"required": false,
"help": "Number of items to return (max 1000; use normal-sized requests)"
}
],
"columns": [
"rank",
"type",
"title",
"author",
"votes",
"url"
],
"type": "js",
"modulePath": "zhihu/recommend.js",
"sourceFile": "zhihu/recommend.js",
"navigateBefore": "https://www.zhihu.com"
},
{
"site": "zhihu",
"name": "search",
+70
View File
@@ -0,0 +1,70 @@
import { ArgumentError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
export function requireSearchQuery(value, label = 'keyword') {
const query = String(value ?? '').trim();
if (!query) {
throw new ArgumentError(`${label} cannot be empty`);
}
return query;
}
export function requireBoundedInteger(value, defaultValue, min, max, label) {
const raw = value ?? defaultValue;
const parsed = typeof raw === 'number' ? raw : Number(raw);
if (!Number.isInteger(parsed)) {
throw new ArgumentError(`${label} must be an integer between ${min} and ${max}, got ${JSON.stringify(value)}`);
}
if (parsed < min || parsed > max) {
throw new ArgumentError(`${label} must be between ${min} and ${max}, got ${parsed}`);
}
return parsed;
}
export function requireNonNegativeInteger(value, defaultValue, label) {
const raw = value ?? defaultValue;
const parsed = typeof raw === 'number' ? raw : Number(raw);
if (!Number.isInteger(parsed) || parsed < 0) {
throw new ArgumentError(`${label} must be a non-negative integer, got ${JSON.stringify(value)}`);
}
return parsed;
}
export function unwrapBrowserResult(value) {
if (value && typeof value === 'object' && !Array.isArray(value) && 'session' in value && 'data' in value) {
return value.data;
}
return value;
}
export function requireRows(value, label) {
const rows = unwrapBrowserResult(value);
if (!Array.isArray(rows)) {
throw new CommandExecutionError(`${label} returned an unexpected payload shape; expected an array of result rows.`);
}
return rows;
}
export function toHttpsUrl(value, baseUrl) {
const raw = String(value ?? '').trim();
if (!raw) return '';
try {
const url = new URL(raw, baseUrl);
if (url.protocol !== 'http:' && url.protocol !== 'https:') return '';
return url.href;
} catch {
return '';
}
}
export function emptySearchResults(site, query) {
return new EmptyResultError(`${site} search`, `No ${site} results matched "${query}".`);
}
export async function runBrowserStep(label, fn) {
try {
return await fn();
} catch (error) {
if (error?.code || error?.name === 'ArgumentError') throw error;
throw new CommandExecutionError(`${label} failed: ${error?.message ?? error}`);
}
}
+96 -14
View File
@@ -1,10 +1,60 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { requirePage, navigateToChat, fetchFriendList } from './utils.js';
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import {
requirePage, navigateToChat, navigateToGeekChat,
fetchFriendList, fetchGeekFriendLabelList, fetchGeekFriendInfoList,
readEncryptSystemId, assertOk, IDENTITY_MISMATCH_CODE,
readPositiveInteger,
} from './utils.js';
function formatMsgTime(ms) {
if (!ms) return '';
return new Date(ms).toLocaleString('zh-CN');
}
function mapBossRow(f) {
return {
name: f.name || '',
company: '',
job: f.jobName || '',
title: '',
last_msg: f.lastMessageInfo?.text || '',
last_time: f.lastTime || '',
uid: f.encryptUid || '',
security_id: f.securityId || '',
};
}
async function buildGeekRows(page, limit) {
const encryptSystemId = await readEncryptSystemId(page);
const labelList = await fetchGeekFriendLabelList(page, { encryptSystemId });
if (labelList.length === 0) {
return [];
}
const slicedLabels = labelList.slice(0, limit);
const friendIds = slicedLabels.map((f) => f.friendId).filter(Boolean);
const enriched = await fetchGeekFriendInfoList(page, friendIds);
const enrichMap = new Map(enriched.map((f) => [String(f.friendId ?? f.uid), f]));
return slicedLabels.map((f) => {
const e = enrichMap.get(String(f.friendId)) || {};
return {
name: e.name || f.name || '',
company: e.brandName || f.brandName || '',
job: e.jobName || f.jobName || '',
title: e.bossTitle || f.bossTitle || '',
last_msg: e.lastMessageInfo?.showText || e.lastMsg || f.lastMsg || '',
last_time: e.lastTime || formatMsgTime(e.lastMessageInfo?.msgTime) || formatMsgTime(f.updateTime) || '',
uid: e.encryptUid || f.encryptFriendId || String(e.uid ?? e.friendId ?? f.friendId ?? ''),
security_id: e.securityId || '',
};
});
}
cli({
site: 'boss',
name: 'chatlist',
access: 'read',
description: 'BOSS直聘查看聊天列表(招聘端)',
description: 'BOSS直聘查看聊天列表(招聘端/求职端',
domain: 'www.zhipin.com',
strategy: Strategy.COOKIE,
navigateBefore: false,
@@ -12,23 +62,55 @@ cli({
args: [
{ name: 'page', type: 'int', default: 1, help: 'Page number' },
{ name: 'limit', type: 'int', default: 20, help: 'Number of results' },
{ name: 'job-id', default: '0', help: 'Filter by job ID (0=all)' },
{ name: 'job-id', default: '0', help: 'Filter by job ID (0=all, boss side only)' },
{ name: 'side', default: 'auto', choices: ['auto', 'boss', 'geek'], help: 'Identity side: auto (default), boss (recruiter), or geek (job-seeker)' },
],
columns: ['name', 'job', 'last_msg', 'last_time', 'uid', 'security_id'],
columns: ['name', 'company', 'job', 'title', 'last_msg', 'last_time', 'uid', 'security_id'],
func: async (page, kwargs) => {
requirePage(page);
const limit = readPositiveInteger(kwargs.limit, 'chatlist --limit', 20, 100);
const pageNum = readPositiveInteger(kwargs.page, 'chatlist --page', 1);
const side = kwargs.side || 'auto';
if (side === 'boss') {
await navigateToChat(page);
const friends = await fetchFriendList(page, {
pageNum,
jobId: kwargs['job-id'] || '0',
});
if (friends.length === 0)
throw new EmptyResultError('boss chatlist', 'No recruiter-side chat sessions were returned.');
return friends.slice(0, limit).map(mapBossRow);
}
if (side === 'geek') {
await navigateToGeekChat(page);
const rows = await buildGeekRows(page, limit);
if (rows.length === 0)
throw new EmptyResultError('boss chatlist', 'No job-seeker-side chat sessions were returned.');
return rows;
}
// auto: try recruiter first, fall back to geek on identity mismatch
await navigateToChat(page);
const friends = await fetchFriendList(page, {
pageNum: kwargs.page || 1,
const bossResult = await fetchFriendList(page, {
pageNum,
jobId: kwargs['job-id'] || '0',
allowNonZero: true,
});
return friends.slice(0, kwargs.limit || 20).map((f) => ({
name: f.name || '',
job: f.jobName || '',
last_msg: f.lastMessageInfo?.text || '',
last_time: f.lastTime || '',
uid: f.encryptUid || '',
security_id: f.securityId || '',
}));
if (Array.isArray(bossResult)) {
if (bossResult.length === 0)
throw new EmptyResultError('boss chatlist', 'No recruiter-side chat sessions were returned.');
return bossResult.slice(0, limit).map(mapBossRow);
}
if (bossResult.code === IDENTITY_MISMATCH_CODE) {
await navigateToGeekChat(page);
const rows = await buildGeekRows(page, limit);
if (rows.length === 0)
throw new EmptyResultError('boss chatlist', 'No job-seeker-side chat sessions were returned.');
return rows;
}
assertOk(bossResult);
throw new CommandExecutionError('Boss chatlist returned an unexpected response');
},
});
+211
View File
@@ -0,0 +1,211 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import './chatlist.js';
const BOSS_FRIEND = {
name: '张三',
jobName: '后端工程师',
lastMessageInfo: { text: '你好' },
lastTime: '2024-01-01 10:00',
encryptUid: 'enc-boss-uid',
securityId: 'boss-sec-id',
};
const GEEK_LABEL_FRIEND = {
friendId: 12345,
name: '李四',
brandName: '字节跳动',
jobName: '产品经理',
bossTitle: 'HR',
lastMsg: '感谢投递',
updateTime: 1704067200000,
encryptFriendId: 'enc-geek-uid',
};
const GEEK_ENRICHED = {
friendId: 12345,
uid: 99999,
name: '李四',
brandName: '字节跳动',
jobName: '产品经理',
bossTitle: 'HR总监',
encryptUid: 'enc-geek-uid',
securityId: 'geek-sec-id',
lastMessageInfo: { showText: '感谢投递', msgTime: 1704067200000 },
lastTime: '2024-01-01',
};
function createPageMock(evaluateImpl) {
return {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockImplementation(evaluateImpl),
};
}
describe('boss chatlist', () => {
const command = getRegistry().get('boss/chatlist');
it('--side boss preserves existing behavior with 8-column output', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
}
return {};
});
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'boss' });
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/chat/index'));
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
name: '张三',
company: '',
job: '后端工程师',
title: '',
last_msg: '你好',
uid: 'enc-boss-uid',
security_id: 'boss-sec-id',
});
});
it('--side geek maps enriched getGeekFriendList data into 8 columns', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_ENRICHED] } };
}
return {};
});
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' });
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/geek/chat'));
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
name: '李四',
company: '字节跳动',
job: '产品经理',
title: 'HR总监',
uid: 'enc-geek-uid',
security_id: 'geek-sec-id',
});
});
it('--side geek falls back to label fields when enrichment has no match', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [] } };
}
return {};
});
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' });
expect(rows).toHaveLength(1);
expect(rows[0].name).toBe('李四');
expect(rows[0].company).toBe('字节跳动');
expect(rows[0].security_id).toBe('');
});
it('rejects invalid --limit before navigating', async () => {
const page = createPageMock(async () => ({}));
await expect(
command.func(page, { page: 1, limit: 0, 'job-id': '0', side: 'geek' })
).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
});
it('--side geek reports a true empty chat list as EmptyResultError', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [] } };
}
return {};
});
await expect(
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' })
).rejects.toBeInstanceOf(EmptyResultError);
});
it('treats malformed geek enrichment payload as CommandExecutionError', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: {} };
}
return {};
});
await expect(
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' })
).rejects.toBeInstanceOf(CommandExecutionError);
});
it('treats null Boss API payload as CommandExecutionError', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) return null;
return {};
});
await expect(
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'boss' })
).rejects.toBeInstanceOf(CommandExecutionError);
});
it('maps expired Boss cookies to AuthRequiredError', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 7, message: 'Cookie 已过期' };
}
return {};
});
await expect(
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'boss' })
).rejects.toBeInstanceOf(AuthRequiredError);
});
it('--side auto falls back to geek when recruiter returns code 24', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 24, message: '请切换身份后再试' };
}
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_ENRICHED] } };
}
return {};
});
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'auto' });
expect(rows).toHaveLength(1);
expect(rows[0].company).toBe('字节跳动');
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/geek/chat'));
});
it('--side auto uses recruiter results when code 0 and does not call geek API', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
}
return {};
});
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'auto' });
expect(rows).toHaveLength(1);
expect(rows[0].name).toBe('张三');
const evaluateCalls = page.evaluate.mock.calls.map((c) => c[0]);
expect(evaluateCalls.some((s) => s.includes('geekFilterByLabel'))).toBe(false);
});
it('registers --side as a choices-constrained arg defaulting to auto', () => {
const sideArg = command.args.find((a) => a.name === 'side');
expect(sideArg?.choices).toEqual(['auto', 'boss', 'geek']);
expect(sideArg?.default).toBe('auto');
});
});
+98 -24
View File
@@ -1,10 +1,72 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { requirePage, navigateToChat, bossFetch, findFriendByUid } from './utils.js';
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import {
requirePage, navigateToChat, navigateToGeekChat,
bossFetch, findFriendByUid, findGeekFriendByUid,
fetchGeekHistoryMsg, readEncryptSystemId,
assertOk, IDENTITY_MISMATCH_CODE,
readPositiveInteger, readRequiredString,
} from './utils.js';
const TYPE_MAP = {
1: '文本', 2: '图片', 3: '招呼', 4: '简历', 5: '系统',
6: '名片', 7: '语音', 8: '视频', 9: '表情',
};
function mapBossMsg(m, friend) {
const fromObj = m.from || {};
const isSelf = typeof fromObj === 'object' ? fromObj.uid !== friend.uid : false;
return {
from: isSelf ? '我' : (typeof fromObj === 'object' ? fromObj.name : friend.name),
type: TYPE_MAP[m.type] || `其他(${m.type})`,
text: m.text || m.body?.text || '',
time: m.time ? new Date(m.time).toLocaleString('zh-CN') : '',
};
}
function mapGeekMsg(m, friend) {
const fromUid = m.from && m.from.uid;
const isFromBoss = fromUid != null && String(fromUid) === String(friend.uid);
return {
from: isFromBoss ? '对方' : '我',
type: TYPE_MAP[m.type] || `其他(${m.type})`,
text: m.text || m.body?.text || m.body?.content || m.body?.showText ||
JSON.stringify(m.body || {}).slice(0, 120),
time: m.time ? new Date(m.time).toLocaleString('zh-CN') : '',
};
}
async function bossChatMsg(page, kwargs, existingFriend) {
const friend = existingFriend ?? await findFriendByUid(page, kwargs.uid);
if (!friend) throw new EmptyResultError('boss chatmsg', '未找到该候选人');
if (!friend.securityId) throw new CommandExecutionError('该聊天缺少 securityId,无法获取历史消息');
const gid = friend.uid;
const securityId = encodeURIComponent(friend.securityId);
const msgUrl = `https://www.zhipin.com/wapi/zpchat/boss/historyMsg?gid=${gid}&securityId=${securityId}&page=${kwargs.page}&c=20&src=0`;
const msgData = await bossFetch(page, msgUrl);
const messages = msgData.zpData?.messages ?? msgData.zpData?.historyMsgList;
if (!Array.isArray(messages)) {
throw new CommandExecutionError('Boss recruiter history response did not include a message list');
}
if (messages.length === 0) {
throw new EmptyResultError('boss chatmsg', 'Boss returned no messages for this chat.');
}
return messages.map((m) => mapBossMsg(m, friend));
}
async function geekChatMsg(page, kwargs, encryptSystemId) {
const friend = await findGeekFriendByUid(page, kwargs.uid, { encryptSystemId });
if (!friend) throw new EmptyResultError('boss chatmsg', '未找到该聊天(geek 侧)');
if (!friend.securityId) throw new CommandExecutionError('该聊天缺少 securityId,无法获取历史消息');
const messages = await fetchGeekHistoryMsg(page, friend, { page: kwargs.page });
return messages.map((m) => mapGeekMsg(m, friend));
}
cli({
site: 'boss',
name: 'chatmsg',
access: 'read',
description: 'BOSS直聘查看与候选人的聊天消息',
description: 'BOSS直聘查看聊天消息历史(招聘端/求职端)',
domain: 'www.zhipin.com',
strategy: Strategy.COOKIE,
navigateBefore: false,
@@ -12,32 +74,44 @@ cli({
args: [
{ name: 'uid', required: true, positional: true, help: 'Encrypted UID (from chatlist)' },
{ name: 'page', type: 'int', default: 1, help: 'Page number' },
{ name: 'side', default: 'auto', choices: ['auto', 'boss', 'geek'], help: 'Identity side: auto (default), boss (recruiter), or geek (job-seeker)' },
],
columns: ['from', 'type', 'text', 'time'],
func: async (page, kwargs) => {
requirePage(page);
const uid = readRequiredString(kwargs.uid, 'chatmsg uid');
const pageNum = readPositiveInteger(kwargs.page, 'chatmsg --page', 1);
const normalizedKwargs = { ...kwargs, uid, page: pageNum };
const side = kwargs.side || 'auto';
if (side === 'boss') {
await navigateToChat(page);
return await bossChatMsg(page, normalizedKwargs);
}
if (side === 'geek') {
await navigateToGeekChat(page);
const encryptSystemId = await readEncryptSystemId(page);
return await geekChatMsg(page, normalizedKwargs, encryptSystemId);
}
// auto: try recruiter first, fall back to geek when not found or identity mismatch
await navigateToChat(page);
const friend = await findFriendByUid(page, kwargs.uid);
if (!friend)
throw new Error('未找到该候选人');
const gid = friend.uid;
const securityId = encodeURIComponent(friend.securityId);
const msgUrl = `https://www.zhipin.com/wapi/zpchat/boss/historyMsg?gid=${gid}&securityId=${securityId}&page=${kwargs.page}&c=20&src=0`;
const msgData = await bossFetch(page, msgUrl);
const TYPE_MAP = {
1: '文本', 2: '图片', 3: '招呼', 4: '简历', 5: '系统',
6: '名片', 7: '语音', 8: '视频', 9: '表情',
};
const messages = msgData.zpData?.messages || msgData.zpData?.historyMsgList || [];
return messages.map((m) => {
const fromObj = m.from || {};
const isSelf = typeof fromObj === 'object' ? fromObj.uid !== friend.uid : false;
return {
from: isSelf ? '我' : (typeof fromObj === 'object' ? fromObj.name : friend.name),
type: TYPE_MAP[m.type] || '其他(' + m.type + ')',
text: m.text || m.body?.text || '',
time: m.time ? new Date(m.time).toLocaleString('zh-CN') : '',
};
});
const bossResult = await findFriendByUid(page, uid, { allowNonZero: true });
if (bossResult?.friend) {
return await bossChatMsg(page, normalizedKwargs, bossResult.friend);
}
// Not found or identity mismatch — check for hard errors before falling back
if (bossResult?.code && bossResult.code !== 0 && bossResult.code !== IDENTITY_MISMATCH_CODE) {
assertOk(bossResult);
}
// Fall back to geek side
await navigateToGeekChat(page);
const encryptSystemId = await readEncryptSystemId(page);
const geekFriend = await findGeekFriendByUid(page, uid, { encryptSystemId });
if (!geekFriend) throw new EmptyResultError('boss chatmsg', 'uid 在招聘端与求职端聊天列表中均未找到');
if (!geekFriend.securityId) throw new CommandExecutionError('该聊天缺少 securityId,无法获取历史消息');
const messages = await fetchGeekHistoryMsg(page, geekFriend, { page: pageNum });
return messages.map((m) => mapGeekMsg(m, geekFriend));
},
});
+230
View File
@@ -0,0 +1,230 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import './chatmsg.js';
const BOSS_FRIEND = {
uid: 12345,
encryptUid: 'enc-boss-uid',
securityId: 'boss-sec-id',
name: '候选人甲',
};
const BOSS_MSGS = [
{ type: 1, text: 'Hello', from: { uid: 99999, name: 'HR' }, time: 1704067200000 },
{ type: 1, text: '感谢', from: { uid: 12345, name: '候选人甲' }, time: 1704067201000 },
];
const GEEK_FRIEND_LABEL = {
friendId: 11111,
encryptFriendId: 'enc-geek-uid',
name: 'Boss张',
brandName: '公司A',
};
const GEEK_FRIEND_ENRICHED = {
friendId: 11111,
uid: 67890,
encryptUid: 'enc-geek-uid',
securityId: 'geek-sec-id',
name: 'Boss张',
};
const GEEK_MSGS = [
{ type: 1, text: '欢迎投递', received: true, time: 1704067200000, from: { uid: 67890, name: 'Boss张' } },
{ type: 1, text: '谢谢', received: true, time: 1704067201000, from: { uid: 99999, name: '我' } },
];
function createPageMock(evaluateImpl) {
return {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockImplementation(evaluateImpl),
};
}
describe('boss chatmsg', () => {
const command = getRegistry().get('boss/chatmsg');
it('rejects empty uid before navigating', async () => {
const page = createPageMock(async () => ({}));
await expect(
command.func(page, { uid: ' ', page: 1, side: 'geek' })
).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
});
it('rejects invalid --page before navigating', async () => {
const page = createPageMock(async () => ({}));
await expect(
command.func(page, { uid: 'enc-geek-uid', page: 0, side: 'geek' })
).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
});
it('--side boss preserves existing behavior', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
}
if (script.includes('boss/historyMsg')) {
return { code: 0, zpData: { messages: BOSS_MSGS } };
}
return {};
});
const rows = await command.func(page, { uid: 'enc-boss-uid', page: 1, side: 'boss' });
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/chat/index'));
expect(rows).toHaveLength(2);
expect(rows[0].from).toBe('我');
expect(rows[1].from).toBe('候选人甲');
});
it('--side geek calls historyMsg with bossId, securityId, page, c=20, src=0', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
}
if (script.includes('geek/historyMsg')) {
return { code: 0, zpData: { messages: GEEK_MSGS } };
}
return {};
});
await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' });
const historyScript = page.evaluate.mock.calls.find((c) => c[0].includes('geek/historyMsg'))?.[0];
expect(historyScript).toBeDefined();
expect(historyScript).toContain('bossId=67890');
expect(historyScript).toContain('securityId=');
expect(historyScript).toContain('page=1');
expect(historyScript).toContain('c=20');
expect(historyScript).toContain('src=0');
});
it('--side geek uses from.uid to determine direction, not received flag', async () => {
// Both messages have received:true (mirrors real geek historyMsg API behaviour)
// Direction is determined by whether m.from.uid matches the boss's uid (67890)
const msgsAllReceived = [
{ type: 1, text: '欢迎投递', received: true, time: 1704067200000, from: { uid: 67890, name: 'Boss张' } },
{ type: 1, text: '谢谢', received: true, time: 1704067201000, from: { uid: 99999, name: '我' } },
];
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
}
if (script.includes('geek/historyMsg')) {
return { code: 0, zpData: { messages: msgsAllReceived } };
}
return {};
});
const rows = await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' });
// from.uid=67890 matches friend.uid=67890 → boss sent it → '对方'
expect(rows[0].from).toBe('对方');
// from.uid=99999 does not match → geek sent it → '我'
expect(rows[1].from).toBe('我');
});
it('non-text message body does not crash and produces truncated JSON', async () => {
const nonTextMsg = { type: 99, received: true, time: 1704067200000, body: { action: 'resume_request', detail: 'X' } };
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
}
if (script.includes('geek/historyMsg')) {
return { code: 0, zpData: { messages: [nonTextMsg] } };
}
return {};
});
const rows = await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' });
expect(rows).toHaveLength(1);
expect(rows[0].text).toContain('resume_request');
});
it('--side auto falls back to geek when recruiter returns code 24', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 24, message: '请切换身份后再试' };
}
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
}
if (script.includes('geek/historyMsg')) {
return { code: 0, zpData: { messages: GEEK_MSGS } };
}
return {};
});
const rows = await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'auto' });
expect(rows).toHaveLength(2);
expect(rows[0].from).toBe('对方');
});
it('--side geek throws when uid is not found in geek chat list', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [] } };
}
return {};
});
await expect(
command.func(page, { uid: 'unknown-uid', page: 1, side: 'geek' })
).rejects.toBeInstanceOf(EmptyResultError);
});
it('--side boss maps expired cookies to AuthRequiredError', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 7, message: 'Cookie 已过期' };
}
return {};
});
await expect(
command.func(page, { uid: 'enc-boss-uid', page: 1, side: 'boss' })
).rejects.toBeInstanceOf(AuthRequiredError);
});
it('--side boss treats missing history list as parser drift', async () => {
const page = createPageMock(async (script) => {
if (script.includes('getBossFriendListV2')) {
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
}
if (script.includes('boss/historyMsg')) {
return { code: 0, zpData: {} };
}
return {};
});
await expect(
command.func(page, { uid: 'enc-boss-uid', page: 1, side: 'boss' })
).rejects.toBeInstanceOf(CommandExecutionError);
});
it('--side geek reports an empty history as EmptyResultError', async () => {
const page = createPageMock(async (script) => {
if (script.includes('document.cookie')) return 'test-enc-sys-id';
if (script.includes('geekFilterByLabel')) {
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
}
if (script.includes('getGeekFriendList.json')) {
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
}
if (script.includes('geek/historyMsg')) {
return { code: 0, zpData: { messages: [] } };
}
return {};
});
await expect(
command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' })
).rejects.toBeInstanceOf(EmptyResultError);
});
});
+257 -12
View File
@@ -1,8 +1,11 @@
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
// ── Constants ───────────────────────────────────────────────────────────────
const BOSS_DOMAIN = 'www.zhipin.com';
const CHAT_URL = `https://${BOSS_DOMAIN}/web/chat/index`;
const COOKIE_EXPIRED_CODES = new Set([7, 37]);
const COOKIE_EXPIRED_MSG = 'Cookie 已过期!请在当前 Chrome 浏览器中重新登录 BOSS 直聘。';
const RECRUITER_ONLY_MSG = '该命令仅支持招聘端(BOSS 端)账号,请使用招聘者账号登录后重试。';
const DEFAULT_TIMEOUT = 15_000;
// ── Core helpers ────────────────────────────────────────────────────────────
/**
@@ -10,7 +13,24 @@ const DEFAULT_TIMEOUT = 15_000;
*/
export function requirePage(page) {
if (!page)
throw new Error('Browser page required');
throw new CommandExecutionError('Browser page required');
}
export function readPositiveInteger(raw, name, fallback, max) {
const value = raw === undefined || raw === null || raw === '' ? fallback : Number(raw);
if (!Number.isInteger(value) || value < 1) {
throw new ArgumentError(`boss ${name} must be a positive integer`);
}
if (max !== undefined && value > max) {
throw new ArgumentError(`boss ${name} must be <= ${max}`);
}
return value;
}
export function readRequiredString(raw, name) {
const value = String(raw ?? '').trim();
if (!value) {
throw new ArgumentError(`boss ${name} cannot be empty`);
}
return value;
}
/**
* Navigate to BOSS chat page and wait for it to settle.
@@ -33,19 +53,37 @@ export async function navigateTo(page, url, waitSeconds = 1) {
*/
export function checkAuth(data) {
if (COOKIE_EXPIRED_CODES.has(data.code)) {
throw new Error(COOKIE_EXPIRED_MSG);
throw new AuthRequiredError(BOSS_DOMAIN, COOKIE_EXPIRED_MSG);
}
}
/**
* Map BOSS code=24 ("请切换身份后再试") to a typed AuthRequiredError.
* Recruiter-only commands (recommend, joblist, stats, resume, mark,
* exchange, invite, greet, batchgreet) have no geek-side equivalent;
* surfacing this as a generic COMMAND_EXEC hides what the user must do.
* chatlist / chatmsg avoid this path by using `allowNonZero: true` and
* branching to the geek-side fetch when they see code 24.
*/
function checkRecruiterSide(data) {
if (data.code === IDENTITY_MISMATCH_CODE) {
throw new AuthRequiredError(BOSS_DOMAIN, RECRUITER_ONLY_MSG);
}
}
/**
* Throw if the API response is not code 0.
* Checks for cookie expiry first, then throws with the provided message.
* Checks for cookie expiry first, then identity mismatch, then throws
* with the provided message.
*/
export function assertOk(data, errorPrefix) {
if (!data || typeof data !== 'object') {
throw new CommandExecutionError(`${errorPrefix ? `${errorPrefix}: ` : ''}Boss API returned malformed response`);
}
if (data.code === 0)
return;
checkAuth(data);
checkRecruiterSide(data);
const prefix = errorPrefix ? `${errorPrefix}: ` : '';
throw new Error(`${prefix}${data.message || 'Unknown error'} (code=${data.code})`);
throw new CommandExecutionError(`${prefix}${data.message || 'Unknown error'} (code=${data.code})`);
}
/**
* Make a credentialed XHR request via page.evaluate().
@@ -80,7 +118,19 @@ export async function bossFetch(page, url, opts = {}) {
});
}
`;
const data = await page.evaluate(script);
let data;
try {
data = await page.evaluate(script);
} catch (error) {
if (error instanceof AuthRequiredError || error instanceof CommandExecutionError) {
throw error;
}
const message = error instanceof Error ? error.message : String(error);
throw new CommandExecutionError(`Boss API request failed: ${message}`);
}
if (!data || typeof data !== 'object') {
throw new CommandExecutionError('Boss API returned malformed response');
}
// Auto-check auth unless caller opts out
if (!opts.allowNonZero && data.code !== 0) {
assertOk(data);
@@ -95,8 +145,13 @@ export async function fetchFriendList(page, opts = {}) {
const pageNum = opts.pageNum ?? 1;
const jobId = opts.jobId ?? '0';
const url = `https://${BOSS_DOMAIN}/wapi/zprelation/friend/getBossFriendListV2.json?page=${pageNum}&status=0&jobId=${jobId}`;
const data = await bossFetch(page, url);
return data.zpData?.friendList || [];
const data = await bossFetch(page, url, { allowNonZero: opts.allowNonZero });
if (opts.allowNonZero && data.code !== 0) return data;
const list = data.zpData?.friendList;
if (!Array.isArray(list)) {
throw new CommandExecutionError('Boss friend list response did not include zpData.friendList');
}
return list;
}
/**
* Fetch the recommended candidates (greetRecSortList).
@@ -104,7 +159,11 @@ export async function fetchFriendList(page, opts = {}) {
export async function fetchRecommendList(page) {
const url = `https://${BOSS_DOMAIN}/wapi/zprelation/friend/greetRecSortList`;
const data = await bossFetch(page, url);
return data.zpData?.friendList || [];
const list = data.zpData?.friendList;
if (!Array.isArray(list)) {
throw new CommandExecutionError('Boss recommend response did not include zpData.friendList');
}
return list;
}
/**
* Find a friend by encryptUid, searching through friend list and optionally greet list.
@@ -115,10 +174,14 @@ export async function findFriendByUid(page, encryptUid, opts = {}) {
const checkGreetList = opts.checkGreetList ?? false;
// Search friend list pages
for (let p = 1; p <= maxPages; p++) {
const friends = await fetchFriendList(page, { pageNum: p });
const result = await fetchFriendList(page, { pageNum: p, allowNonZero: opts.allowNonZero });
if (opts.allowNonZero && !Array.isArray(result)) {
return { friend: null, code: result.code };
}
const friends = Array.isArray(result) ? result : [];
const found = friends.find((f) => f.encryptUid === encryptUid);
if (found)
return found;
return opts.allowNonZero ? { friend: found, code: 0 } : found;
if (friends.length === 0)
break;
}
@@ -127,9 +190,9 @@ export async function findFriendByUid(page, encryptUid, opts = {}) {
const greetList = await fetchRecommendList(page);
const found = greetList.find((f) => f.encryptUid === encryptUid);
if (found)
return found;
return opts.allowNonZero ? { friend: found, code: 0 } : found;
}
return null;
return opts.allowNonZero ? { friend: null, code: 0 } : null;
}
// ── UI automation helpers ───────────────────────────────────────────────────
/**
@@ -221,3 +284,185 @@ export function verbose(msg) {
console.error(`[opencli:boss] ${msg}`);
}
}
// ── Geek-side helpers ────────────────────────────────────────────────────────
export const IDENTITY_MISMATCH_CODE = 24;
const GEEK_CHAT_URL = `https://${BOSS_DOMAIN}/web/geek/chat`;
/**
* Navigate to the job-seeker chat page.
* Establishes the cookie + JS-global context needed for geek-side API calls.
*/
export async function navigateToGeekChat(page, waitSeconds = 2) {
await page.goto(GEEK_CHAT_URL);
await page.wait({ time: waitSeconds });
}
/**
* Read the encryptSystemId value required by the geek-side list API.
* Strategy (in order):
* 1. Vue app state / Pinia stores / $route.query (Option 1 — runtime source)
* 2. performance.getEntriesByType('resource') — parse from geekFilterByLabel URL
* that the page itself already issued (Option 2 — most deterministic)
* 3. cookie, inline <script> SSR state, known window globals, localStorage (fallbacks)
* Returns empty string if nothing is found; the API may still succeed without it.
* Caller must have navigated to the geek chat page first.
*/
export async function readEncryptSystemId(page) {
const result = await page.evaluate(`
(() => {
// 1. Vue app state / Pinia / $route.query
// The chat component reads encryptSystemId from the app runtime to build
// its own geekFilterByLabel request, so the value lives in the Vue tree.
try {
const appEl = document.querySelector('#app') || document.querySelector('[data-v-app]');
const vueApp = appEl && (appEl.__vue_app__ || appEl._vei);
if (vueApp) {
// 1a. Pinia stores (Vue 3 standard state management on BOSS直聘)
const pinia = vueApp.config && vueApp.config.globalProperties.$pinia;
if (pinia && pinia.state && pinia.state.value) {
for (const store of Object.values(pinia.state.value)) {
try {
const flat = JSON.stringify(store);
if (flat.includes('encryptSystemId')) {
const m = flat.match(/"encryptSystemId":"([^"]+)"/);
if (m) return m[1];
}
} catch (_) {}
}
}
// 1b. Vue Router current route query
const router = vueApp.config && vueApp.config.globalProperties.$router;
const query = router && router.currentRoute && router.currentRoute.value && router.currentRoute.value.query;
if (query && query.encryptSystemId) return query.encryptSystemId;
}
} catch (_) {}
// 2. Performance resource entries — the page already issued geekFilterByLabel
// with encryptSystemId in the URL; read it back from the resource timing API.
try {
const entries = performance.getEntriesByType('resource');
for (const entry of entries) {
if (!entry.name.includes('geekFilterByLabel')) continue;
const u = new URL(entry.name);
const v = u.searchParams.get('encryptSystemId');
if (v) return v;
}
} catch (_) {}
// 3. cookie
try {
const m = document.cookie.match(/encryptSystemId=([^;]+)/i);
if (m) return decodeURIComponent(m[1]);
} catch (_) {}
// 4. inline <script> SSR state (Nuxt embeds server state here)
try {
for (const s of document.querySelectorAll('script:not([src])')) {
const t = s.textContent || '';
if (!t.includes('encryptSystemId')) continue;
const m = t.match(/"encryptSystemId":"([^"]+)"/);
if (m) return m[1];
}
} catch (_) {}
// 5. known BOSS / Nuxt window globals
const KNOWN = [
'__NUXT__', '__INITIAL_STATE__', '__ZP_INFO__', '__BOSS_ZP__',
'pageGlobalVar', 'ZP_DATA', '__ZP_DATA__', '__PAGE_DATA__',
];
for (const k of KNOWN) {
const obj = window[k];
if (!obj || typeof obj !== 'object') continue;
try {
const flat = JSON.stringify(obj);
if (!flat.includes('encryptSystemId')) continue;
const m = flat.match(/"encryptSystemId":"([^"]+)"/);
if (m) return m[1];
} catch (_) {}
}
// 6. localStorage
try {
for (let i = 0; i < localStorage.length; i++) {
const k = localStorage.key(i);
if (!k) continue;
if (k.toLowerCase().includes('encryptsystemid')) {
const v = localStorage.getItem(k);
if (v) return v;
}
const v = localStorage.getItem(k) || '';
if (v.includes('encryptSystemId')) {
const m = v.match(/"encryptSystemId":"([^"]+)"/);
if (m) return m[1];
}
}
} catch (_) {}
return '';
})()
`);
return result || '';
}
/**
* Fetch the job-seeker chat list (brief info, no securityId).
* Use fetchGeekFriendInfoList to enrich with securityId before calling chatmsg.
*/
export async function fetchGeekFriendLabelList(page, opts = {}) {
const labelId = opts.labelId ?? 0;
const encryptSystemId = opts.encryptSystemId ?? '';
const url = `https://${BOSS_DOMAIN}/wapi/zprelation/friend/geekFilterByLabel?labelId=${labelId}&encryptSystemId=${encodeURIComponent(encryptSystemId)}`;
const data = await bossFetch(page, url, { allowNonZero: opts.allowNonZero });
if (opts.allowNonZero && data.code !== 0) return data;
const list = data.zpData?.friendList;
if (!Array.isArray(list)) {
throw new CommandExecutionError('Boss geek chat list response did not include zpData.friendList');
}
return list;
}
/**
* Enrich a batch of geek friends with full fields including securityId.
* Processes in batches of 50 to avoid oversized request bodies.
*/
export async function fetchGeekFriendInfoList(page, friendIds = []) {
if (!friendIds.length) return [];
const BATCH_SIZE = 50;
const results = [];
for (let i = 0; i < friendIds.length; i += BATCH_SIZE) {
const batch = friendIds.slice(i, i + BATCH_SIZE).map(String);
const body = `friendIds=${batch.join(',')}`;
const data = await bossFetch(page, `https://${BOSS_DOMAIN}/wapi/zprelation/friend/getGeekFriendList.json`, {
method: 'POST',
body,
});
const batchResult = data.zpData?.result;
if (!Array.isArray(batchResult)) {
throw new CommandExecutionError('Boss geek friend enrichment response did not include zpData.result');
}
results.push(...batchResult);
}
return results;
}
/**
* Find a geek-side friend by encrypted uid.
* Merges label-list and enriched data; returns null if not found.
*/
export async function findGeekFriendByUid(page, encryptUid, opts = {}) {
const labelList = await fetchGeekFriendLabelList(page, { encryptSystemId: opts.encryptSystemId });
const candidate = labelList.find((f) => f.encryptFriendId === encryptUid ||
String(f.uid) === String(encryptUid) ||
String(f.friendId) === String(encryptUid));
if (!candidate) return null;
const enriched = await fetchGeekFriendInfoList(page, [candidate.friendId]);
return { ...candidate, ...(enriched[0] || {}) };
}
/**
* Fetch message history for a geek-side chat.
* friend must have .uid (boss's numeric id) and .securityId.
*/
export async function fetchGeekHistoryMsg(page, friend, opts = {}) {
const pageNum = opts.page ?? 1;
const bossId = friend.uid;
const securityId = encodeURIComponent(friend.securityId || '');
const url = `https://${BOSS_DOMAIN}/wapi/zpchat/geek/historyMsg?bossId=${bossId}&securityId=${securityId}&page=${pageNum}&c=20&src=0`;
const data = await bossFetch(page, url);
const messages = data.zpData?.messages ?? data.zpData?.historyMsgList;
if (!Array.isArray(messages)) {
throw new CommandExecutionError('Boss geek history response did not include a message list');
}
if (messages.length === 0) {
throw new EmptyResultError('boss chatmsg', 'Boss returned no messages for this chat.');
}
return messages;
}
+34
View File
@@ -0,0 +1,34 @@
import { describe, expect, it } from 'vitest';
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { assertOk } from './utils.js';
describe('assertOk', () => {
it('returns silently on code 0', () => {
expect(() => assertOk({ code: 0 })).not.toThrow();
});
it('maps expired cookie codes (7, 37) to AuthRequiredError', () => {
expect(() => assertOk({ code: 7, message: 'expired' })).toThrow(AuthRequiredError);
expect(() => assertOk({ code: 37, message: 'expired' })).toThrow(AuthRequiredError);
});
it('maps code 24 (identity mismatch) to AuthRequiredError with recruiter-only hint', () => {
try {
assertOk({ code: 24, message: '请切换身份后再试' });
throw new Error('assertOk should have thrown');
} catch (err) {
expect(err).toBeInstanceOf(AuthRequiredError);
expect(String(err.message)).toContain('招聘端');
}
});
it('falls through to CommandExecutionError for other non-zero codes', () => {
expect(() => assertOk({ code: 99, message: 'something else' }))
.toThrow(CommandExecutionError);
});
it('throws CommandExecutionError on malformed (non-object) response', () => {
expect(() => assertOk(null)).toThrow(CommandExecutionError);
expect(() => assertOk('not-an-object')).toThrow(CommandExecutionError);
});
});
+80
View File
@@ -0,0 +1,80 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import {
emptySearchResults,
requireBoundedInteger,
requireNonNegativeInteger,
requireRows,
requireSearchQuery,
runBrowserStep,
toHttpsUrl,
} from '../_shared/search-adapter.js';
function buildExtractorJs(limit) {
return `
(function() {
var results = [];
var seen = {};
var items = document.querySelectorAll('.snippet');
for (var i = 0; i < items.length; i++) {
if (results.length >= ${limit}) break;
var el = items[i];
if (el.classList.contains('standalone') || el.classList.contains('ad')) continue;
var titleEl = el.querySelector('.search-snippet-title');
var snippetEl = el.querySelector('.generic-snippet .content');
var linkEl = el.querySelector('.result-content a');
if (!titleEl) continue;
var title = titleEl.textContent.trim();
var href = linkEl ? linkEl.getAttribute('href') || '' : '';
var snippet = snippetEl ? snippetEl.textContent.trim() : '';
if (!title || !href || seen[href]) continue;
if (href.indexOf('/') === 0) continue;
seen[href] = true;
results.push([title, href, snippet]);
}
return results;
})()`;
}
const command = cli({
site: 'brave',
name: 'search',
access: 'read',
description: 'Search Brave Search',
domain: 'search.brave.com',
strategy: Strategy.PUBLIC,
browser: true,
args: [
{ name: 'keyword', positional: true, required: true, help: 'Search query' },
{ name: 'limit', type: 'int', default: 10, help: 'Number of results per page (max 18)' },
{ name: 'offset', type: 'int', default: 0, help: 'Page offset (0, 1, 2...). Brave returns ~18 results per page' },
],
columns: ['rank', 'title', 'url', 'snippet'],
func: async (page, kwargs) => {
const limit = requireBoundedInteger(kwargs.limit, 10, 1, 18, '--limit');
const query = requireSearchQuery(kwargs.keyword);
const keyword = encodeURIComponent(query);
const offset = requireNonNegativeInteger(kwargs.offset, 0, '--offset');
let url = `https://search.brave.com/search?q=${keyword}`;
if (offset > 0) url += `&offset=${offset}`;
await runBrowserStep('brave search navigation', () => page.goto(url));
try {
await page.wait({ selector: '.snippet', timeout: 10 });
} catch {
await page.wait(3).catch(function() {});
}
const raw = await runBrowserStep('brave search extraction', () => page.evaluate(buildExtractorJs(limit)));
const results = requireRows(raw, 'brave search');
if (results.length === 0) {
throw emptySearchResults('Brave', query);
}
const rows = results
.map(function(r, index) {
return { rank: index + 1 + offset * 18, title: r[0], url: toHttpsUrl(r[1], 'https://search.brave.com'), snippet: r[2] };
})
.filter((row) => row.url);
if (rows.length === 0) throw emptySearchResults('Brave', query);
return rows;
},
});
export const __test__ = { command };
+76
View File
@@ -0,0 +1,76 @@
import { describe, it, expect, vi } from 'vitest';
const { __test__ } = await import('./search.js');
const command = __test__.command;
function createPageMock(evaluateResult = []) {
return {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue(evaluateResult),
};
}
describe('brave search', () => {
it('should register as a valid command', () => {
expect(command).toBeDefined();
expect(command.site).toBe('brave');
expect(command.name).toBe('search');
expect(command.access).toBe('read');
expect(command.browser).toBe(true);
expect(command.strategy).toBe('public');
expect(command.domain).toBe('search.brave.com');
});
it('should define keyword positional arg', () => {
const kwArg = command.args.find(a => a.name === 'keyword');
expect(kwArg).toBeDefined();
expect(kwArg.positional).toBe(true);
expect(kwArg.required).toBe(true);
});
it('should define limit arg with default 10', () => {
const limitArg = command.args.find(a => a.name === 'limit');
expect(limitArg).toBeDefined();
expect(limitArg.type).toBe('int');
expect(limitArg.default).toBe(10);
});
it('should define output columns', () => {
expect(command.columns).toContain('rank');
expect(command.columns).toContain('title');
expect(command.columns).toContain('url');
expect(command.columns).toContain('snippet');
});
it('rejects empty query, invalid limit, and invalid offset before navigation', async () => {
const page = createPageMock();
await expect(command.func(page, { keyword: '', limit: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
await expect(command.func(page, { keyword: 'opencli', limit: 19 })).rejects.toMatchObject({ code: 'ARGUMENT' });
await expect(command.func(page, { keyword: 'opencli', limit: 5, offset: -1 })).rejects.toMatchObject({ code: 'ARGUMENT' });
expect(page.goto).not.toHaveBeenCalled();
});
it('unwraps browser envelopes and returns ranked HTTPS rows', async () => {
const page = createPageMock({
session: 'site:brave',
data: [['OpenCLI', 'https://github.com/jackwener/OpenCLI', 'CLI browser tooling']],
});
await expect(command.func(page, { keyword: 'opencli', limit: 1, offset: 1 })).resolves.toEqual([{
rank: 19,
title: 'OpenCLI',
url: 'https://github.com/jackwener/OpenCLI',
snippet: 'CLI browser tooling',
}]);
});
it('fails typed instead of silently returning [] for malformed extraction payloads', async () => {
const page = createPageMock({ rows: [] });
await expect(command.func(page, { keyword: 'opencli', limit: 1 })).rejects.toMatchObject({
code: 'COMMAND_EXEC',
message: expect.stringContaining('payload shape'),
});
});
});
+131
View File
@@ -0,0 +1,131 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { ArgumentError } from '@jackwener/opencli/errors';
import {
emptySearchResults,
requireBoundedInteger,
requireNonNegativeInteger,
requireRows,
requireSearchQuery,
runBrowserStep,
toHttpsUrl,
} from '../_shared/search-adapter.js';
function decodeDdgUrl(href) {
if (!href) return '';
try {
const url = new URL(href, 'https://duckduckgo.com');
const uddg = url.searchParams.get('uddg');
return toHttpsUrl(uddg || href, 'https://duckduckgo.com');
} catch {
return '';
}
}
function buildExtractFn(limit) {
return 'function(doc){' +
'var r=[];var seen={};var items=doc.querySelectorAll(".result");' +
'for(var i=0;i<items.length;i++){' +
'if(r.length>=' + limit + ')break;' +
'var el=items[i];var te=el.querySelector(".result__a");' +
'var se=el.querySelector(".result__snippet");' +
'var ue=el.querySelector(".result__url");' +
'var ie=el.querySelector(".result__icon__img");' +
'var cls=el.className||"";var rt="web";' +
'if(cls.indexOf("result--ad")!==-1||cls.indexOf("result--ads")!==-1||cls.indexOf("badge--ad")!==-1)continue;' +
'if(!te)continue;' +
'var t=(te.textContent||"").trim();' +
'var h=te.getAttribute("href")||"";' +
'var sn=se?(se.textContent||"").trim():"";' +
'var du=ue?(ue.textContent||"").trim():"";' +
'var ic=ie?(ie.getAttribute("src")||""):"";' +
'if(cls.indexOf("news-result")!==-1)rt="news";' +
'else if(cls.indexOf("video-result")!==-1)rt="video";' +
'else if(cls.indexOf("image-result")!==-1)rt="image";' +
'if(!t||!h||seen[h])continue;seen[h]=true;' +
'r.push([t,h,sn,du,ic,rt]);' +
'}return r;}';
}
function buildExtractorJs(limit) {
return '(' + buildExtractFn(limit) + '(document))';
}
function buildPaginateJs(limit, keyword, offset, region) {
var params = 'q=' + encodeURIComponent(keyword) + '&s=' + offset + '&v=l&o=json';
if (region) params += '&kl=' + encodeURIComponent(region);
return (
'new Promise(function($r){' +
'var x=new XMLHttpRequest();' +
'x.open("POST","/html/",true);' +
'x.setRequestHeader("Content-Type","application/x-www-form-urlencoded");' +
'x.onload=function(){' +
'try{var d=new DOMParser().parseFromString(x.responseText,"text/html");' +
'$r(' + buildExtractFn(limit) + '(d));' +
'}catch(e){$r({error:"parse",message:String(e&&e.message||e)})}' +
'};' +
'x.onerror=function(){$r({error:"network"})};' +
'x.send("' + params + '");' +
'})'
);
}
const command = cli({
site: 'duckduckgo',
name: 'search',
access: 'read',
description: 'Search DuckDuckGo',
domain: 'html.duckduckgo.com',
strategy: Strategy.PUBLIC,
browser: true,
args: [
{ name: 'keyword', positional: true, required: true, help: 'Search query' },
{ name: 'limit', type: 'int', default: 10, help: 'Number of results per page (1-10). For multi-page, use --offset' },
{ name: 'offset', type: 'int', default: 0, help: 'Result offset for pagination (0, 10, 20...). Uses XHR POST internally' },
{ name: 'region', help: 'Region code (e.g. jp-jp, us-en, cn-zh). Default: all regions' },
{ name: 'time', help: 'Time range: d (day), w (week), m (month), y (year)' },
],
columns: ['rank', 'title', 'url', 'snippet', 'displayUrl', 'icon', 'resultType'],
func: async (page, kwargs) => {
const limit = requireBoundedInteger(kwargs.limit, 10, 1, 10, '--limit');
const keyword = requireSearchQuery(kwargs.keyword);
const offset = requireNonNegativeInteger(kwargs.offset, 0, '--offset');
if (offset % 10 !== 0) {
throw new ArgumentError('--offset must be a multiple of 10 for DuckDuckGo HTML pagination');
}
if (kwargs.time && !/^(d|w|m|y)$/.test(String(kwargs.time))) {
throw new ArgumentError('--time must be one of d, w, m, or y');
}
let url = `https://html.duckduckgo.com/html/?q=${encodeURIComponent(keyword)}`;
if (kwargs.region) url += `&kl=${encodeURIComponent(String(kwargs.region))}`;
if (kwargs.time) url += `&df=${encodeURIComponent(String(kwargs.time))}`;
await runBrowserStep('duckduckgo search navigation', () => page.goto(url));
try {
await page.wait({ selector: '.result', timeout: 8 });
} catch {
await page.wait(3).catch(function() {});
}
var raw;
if (offset === 0) {
raw = await runBrowserStep('duckduckgo search extraction', () => page.evaluate(buildExtractorJs(limit)));
} else {
raw = await runBrowserStep('duckduckgo search pagination extraction', () => page.evaluate(buildPaginateJs(limit, keyword, offset, kwargs.region)));
}
const rows = requireRows(raw, 'duckduckgo search');
if (rows.length === 0) {
throw emptySearchResults('DuckDuckGo', keyword);
}
return rows.map(function(r, index) {
return {
rank: index + 1 + offset,
title: r[0],
url: decodeDdgUrl(r[1]),
snippet: r[2],
displayUrl: r[3],
icon: r[4],
resultType: r[5],
};
}).filter((row) => row.url);
},
});
export const __test__ = { command };
+128
View File
@@ -0,0 +1,128 @@
import { describe, it, expect, vi } from 'vitest';
import { JSDOM } from 'jsdom';
const { __test__ } = await import('./search.js');
const command = __test__.command;
function createPageMock(evaluateResult = []) {
return {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue(evaluateResult),
};
}
describe('duckduckgo search', () => {
it('should register as a valid command', () => {
expect(command).toBeDefined();
expect(command.site).toBe('duckduckgo');
expect(command.name).toBe('search');
expect(command.access).toBe('read');
expect(command.browser).toBe(true);
expect(command.strategy).toBe('public');
expect(command.domain).toBe('html.duckduckgo.com');
});
it('should define keyword positional arg', () => {
const kwArg = command.args.find(a => a.name === 'keyword');
expect(kwArg).toBeDefined();
expect(kwArg.positional).toBe(true);
expect(kwArg.required).toBe(true);
});
it('should define limit arg with default 10', () => {
const limitArg = command.args.find(a => a.name === 'limit');
expect(limitArg).toBeDefined();
expect(limitArg.type).toBe('int');
expect(limitArg.default).toBe(10);
});
it('should define columns for output', () => {
expect(command.columns).toContain('rank');
expect(command.columns).toContain('title');
expect(command.columns).toContain('url');
expect(command.columns).toContain('snippet');
expect(command.columns).toContain('displayUrl');
expect(command.columns).toContain('icon');
expect(command.columns).toContain('resultType');
});
it('rejects empty query and out-of-range pagination before navigation', async () => {
const page = createPageMock();
await expect(command.func(page, { keyword: ' ', limit: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
await expect(command.func(page, { keyword: 'opencli', limit: 11 })).rejects.toMatchObject({ code: 'ARGUMENT' });
await expect(command.func(page, { keyword: 'opencli', limit: 5, offset: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
expect(page.goto).not.toHaveBeenCalled();
});
it('decodes DuckDuckGo redirect URLs and assigns listing rank', async () => {
const page = createPageMock([
[
'OpenCLI',
'/l/?uddg=https%3A%2F%2Fgithub.com%2Fjackwener%2FOpenCLI',
'CLI browser tooling',
'github.com/jackwener/OpenCLI',
'',
'web',
],
]);
await expect(command.func(page, { keyword: 'opencli', limit: 1 })).resolves.toEqual([{
rank: 1,
title: 'OpenCLI',
url: 'https://github.com/jackwener/OpenCLI',
snippet: 'CLI browser tooling',
displayUrl: 'github.com/jackwener/OpenCLI',
icon: '',
resultType: 'web',
}]);
});
it('executes the DOM extractor, filters ads, and returns canonical rows', async () => {
const dom = new JSDOM(`
<div class="result result--ad">
<a class="result__a" href="https://ads.example/">Sponsored result</a>
</div>
<div class="result">
<a class="result__a" href="/l/?uddg=https%3A%2F%2Fexample.com%2Farticle">Organic result</a>
<a class="result__snippet">Organic snippet</a>
<span class="result__url">example.com/article</span>
<img class="result__icon__img" src="https://icons.duckduckgo.com/ip3/example.com.ico">
</div>
`);
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn(async (source) => Function('document', `return ${source};`)(dom.window.document)),
};
await expect(command.func(page, { keyword: 'opencli', limit: 5 })).resolves.toEqual([{
rank: 1,
title: 'Organic result',
url: 'https://example.com/article',
snippet: 'Organic snippet',
displayUrl: 'example.com/article',
icon: 'https://icons.duckduckgo.com/ip3/example.com.ico',
resultType: 'web',
}]);
});
it('unwraps browser envelopes for paginated extraction', async () => {
const page = createPageMock({ session: 'site:duckduckgo', data: [
['Result', 'https://example.com/', 'snippet', 'example.com', '', 'web'],
] });
const result = await command.func(page, { keyword: 'opencli', limit: 1, offset: 10 });
expect(result[0]).toMatchObject({ rank: 11, url: 'https://example.com/' });
});
it('fails typed instead of returning [] for malformed extraction payloads', async () => {
const page = createPageMock({ rows: [] });
await expect(command.func(page, { keyword: 'opencli', limit: 1 })).rejects.toMatchObject({
code: 'COMMAND_EXEC',
message: expect.stringContaining('payload shape'),
});
});
});
+45
View File
@@ -0,0 +1,45 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { requireBoundedInteger, requireSearchQuery } from '../_shared/search-adapter.js';
const command = cli({
site: 'duckduckgo',
name: 'suggest',
access: 'read',
description: 'DuckDuckGo search suggestions',
domain: 'duckduckgo.com',
strategy: Strategy.PUBLIC,
browser: false,
args: [
{ name: 'keyword', positional: true, required: true, help: 'Search query prefix' },
{ name: 'limit', type: 'int', default: 8, help: 'Max number of suggestions' },
],
columns: ['phrase'],
func: async (kwargs) => {
const limit = requireBoundedInteger(kwargs.limit, 8, 1, 20, '--limit');
const keyword = encodeURIComponent(requireSearchQuery(kwargs.keyword));
const url = `https://duckduckgo.com/ac/?q=${keyword}&type=list`;
let resp;
try {
resp = await fetch(url);
} catch (err) {
throw new CommandExecutionError(`DuckDuckGo suggest request failed: ${err instanceof Error ? err.message : String(err)}`);
}
if (!resp.ok) {
throw new CommandExecutionError(`DuckDuckGo suggest returned HTTP ${resp.status}`);
}
let data;
try {
data = await resp.json();
} catch (err) {
throw new CommandExecutionError(`DuckDuckGo suggest returned malformed JSON: ${err?.message ?? err}`);
}
const phrases = Array.isArray(data) && data.length > 1 && Array.isArray(data[1]) ? data[1] : [];
return phrases
.filter((phrase) => typeof phrase === 'string' && phrase.trim())
.slice(0, limit)
.map(function(p) { return { phrase: p }; });
},
});
export const __test__ = { command };
+66
View File
@@ -0,0 +1,66 @@
import { afterEach, describe, it, expect, vi } from 'vitest';
const { __test__ } = await import('./suggest.js');
const command = __test__.command;
afterEach(() => {
vi.restoreAllMocks();
});
describe('duckduckgo suggest', () => {
it('should register as a valid command', () => {
expect(command).toBeDefined();
expect(command.site).toBe('duckduckgo');
expect(command.name).toBe('suggest');
expect(command.access).toBe('read');
expect(command.browser).toBe(false);
expect(command.strategy).toBe('public');
});
it('should define keyword positional arg', () => {
const kwArg = command.args.find(a => a.name === 'keyword');
expect(kwArg).toBeDefined();
expect(kwArg.positional).toBe(true);
expect(kwArg.required).toBe(true);
});
it('should define limit arg with default 8', () => {
const limitArg = command.args.find(a => a.name === 'limit');
expect(limitArg).toBeDefined();
expect(limitArg.default).toBe(8);
});
it('should define phrase column', () => {
expect(command.columns).toEqual(['phrase']);
});
it('rejects empty query and invalid limit before fetch', async () => {
const fetchSpy = vi.spyOn(globalThis, 'fetch');
await expect(command.func({ keyword: '', limit: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
await expect(command.func({ keyword: 'opencli', limit: 21 })).rejects.toMatchObject({ code: 'ARGUMENT' });
expect(fetchSpy).not.toHaveBeenCalled();
});
it('returns filtered suggestion rows from the public API payload', async () => {
vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: true,
json: async () => ['open', ['opencli', '', 'open source']],
});
await expect(command.func({ keyword: 'open', limit: 3 })).resolves.toEqual([
{ phrase: 'opencli' },
{ phrase: 'open source' },
]);
});
it('maps fetch and malformed JSON failures to typed command errors', async () => {
vi.spyOn(globalThis, 'fetch').mockRejectedValueOnce(new Error('offline'));
await expect(command.func({ keyword: 'open', limit: 3 })).rejects.toMatchObject({ code: 'COMMAND_EXEC' });
vi.spyOn(globalThis, 'fetch').mockResolvedValueOnce({
ok: true,
json: async () => { throw new Error('bad json'); },
});
await expect(command.func({ keyword: 'open', limit: 3 })).rejects.toMatchObject({ code: 'COMMAND_EXEC' });
});
});
+299 -54
View File
@@ -1,60 +1,305 @@
import { cli } from '@jackwener/opencli/registry';
cli({
site: 'facebook',
name: 'feed',
access: 'read',
description: 'Get your Facebook news feed',
domain: 'www.facebook.com',
args: [
{ name: 'limit', type: 'int', default: 10, help: 'Number of posts' },
],
columns: ['index', 'author', 'content', 'likes', 'comments', 'shares'],
pipeline: [
{ navigate: { url: 'https://www.facebook.com/', settleMs: 4000 } },
{ evaluate: `(() => {
const limit = \${{ args.limit }};
const posts = document.querySelectorAll('[role="article"]');
return Array.from(posts)
.filter(el => {
const text = el.textContent.trim();
// Filter out "People you may know" suggestions (both CN and EN)
return text.length > 30 &&
!text.startsWith('可能认识') &&
!text.startsWith('People you may know') &&
!text.startsWith('People You May Know');
})
.slice(0, limit)
.map((el, i) => {
// Author from header link
const headerLink = el.querySelector('h2 a, h3 a, h4 a, strong a');
const author = headerLink ? headerLink.textContent.trim() : '';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
// Post text: grab visible spans, filter noise
const spans = Array.from(el.querySelectorAll('div[dir="auto"]'))
.map(s => s.textContent.trim())
.filter(t => t.length > 10 && t.length < 500);
const content = spans.length > 0 ? spans[0] : '';
const FACEBOOK_HOME = 'https://www.facebook.com/';
const MAX_LIMIT = 50;
// Engagement: find like/comment/share counts (CN + EN)
const allText = el.textContent;
const likesMatch = allText.match(/所有心情:([\\d,.\\s]*[\\d万亿KMk]+)/) ||
allText.match(/All:\\s*([\\d,.KMk]+)/) ||
allText.match(/([\\d,.KMk]+)\\s*(?:likes?|reactions?)/i);
const commentsMatch = allText.match(/([\\d,.]+\\s*[万亿]?)\\s*条评论/) ||
allText.match(/([\\d,.KMk]+)\\s*comments?/i);
const sharesMatch = allText.match(/([\\d,.]+\\s*[万亿]?)\\s*次分享/) ||
allText.match(/([\\d,.KMk]+)\\s*shares?/i);
function requireLimit(value) {
const n = Number(value);
if (!Number.isInteger(n) || n < 1 || n > MAX_LIMIT) {
throw new ArgumentError(`facebook feed --limit must be an integer between 1 and ${MAX_LIMIT}`);
}
return n;
}
function unwrapBrowserResult(value) {
if (value && typeof value === 'object' && 'data' in value) {
return value.data;
}
return value;
}
function buildFeedExtractScript(limit) {
return `(() => {
const limit = ${limit};
function clean(value) {
return String(value || '').replace(/\\s+/g, ' ').trim();
}
function textOf(el) {
return clean(el && el.textContent);
}
function labelOf(el) {
return clean(el && el.getAttribute && el.getAttribute('aria-label'));
}
function isAuthPage() {
const path = window.location && window.location.pathname ? window.location.pathname : '';
const body = textOf(document.body);
return /^\\/(login|checkpoint)(\\/|$|\\.php)/.test(path)
|| /^(Log in to Facebook|Facebook登录|登录 Facebook)/i.test(body)
|| /You must log in to continue/i.test(body);
}
function isExplicitEmptyFeed() {
const body = textOf(document.body);
return /No posts available|Nothing to show|暂无动态|没有更多动态|还没有帖子/i.test(body);
}
function isSuggestionOrChrome(text) {
return /^(People you may know|People You May Know|可能认识的人?|你可能认识的人?)/i.test(text)
|| /^(Suggested for you|Suggested Groups|推荐小组|推荐内容)/i.test(text);
}
function isSponsored(text) {
return /(^|\\s)(Sponsored|赞助|广告)(\\s|$)/i.test(text);
}
function isActionText(text) {
return /^(Like|Comment|Share|Send|Follow|赞|评论|分享|发送|关注)$/i.test(text);
}
function isMetricText(text) {
return /^(All:|所有心情:)/i.test(text)
|| /\\b(likes?|reactions?|comments?|shares?)\\b/i.test(text)
|| /(条评论|次分享)$/.test(text);
}
function isTimestampText(text) {
return /^(\\d+\\s*(s|m|h|d|w|mo|yr|min|sec|second|minute|hour|day|week|month|year)s?|Just now|Yesterday|刚刚|昨天|\\d+小时|\\d+天)(\\s*[·•.])?$/i.test(text);
}
function postUrlFrom(root) {
const links = Array.from(root.querySelectorAll('a[href]'));
for (const link of links) {
const href = link.href || link.getAttribute('href') || '';
if (/\\/posts\\/|\\/permalink\\.php|\\/story\\.php|\\/photo\\/\\?fbid=|\\/groups\\/[^/]+\\/posts\\//i.test(href)) {
return href;
}
}
return '';
}
function actionKinds(root) {
const kinds = new Set();
for (const el of root.querySelectorAll('[aria-label]')) {
const label = labelOf(el);
if (/^(Like|赞)$/i.test(label)) kinds.add('like');
if (/^(Comment|评论)$/i.test(label)) kinds.add('comment');
if (/^(Share|分享)$/i.test(label)) kinds.add('share');
}
return kinds;
}
function visibleBlocks(root) {
const seen = new Set();
return Array.from(root.querySelectorAll('[dir="auto"]'))
.map(textOf)
.filter((text) => {
if (!text || text.length > 600 || seen.has(text)) return false;
seen.add(text);
return true;
});
}
function findAuthor(root) {
const links = [
root.querySelector('h2 a[href], h3 a[href], h4 a[href], strong a[href]'),
...Array.from(root.querySelectorAll('a[role="link"][href]')),
].filter(Boolean);
for (const link of links) {
const text = textOf(link);
const href = link.href || link.getAttribute('href') || '';
if (text.length > 1 && text.length <= 80
&& !isActionText(text)
&& !isMetricText(text)
&& !isTimestampText(text)
&& !/\\/groups\\/|\\/watch\\/|\\/reel\\/|\\/events\\/|\\/friends\\//i.test(href)) {
return text;
}
}
return '';
}
function contentBlocks(root, author) {
return visibleBlocks(root).filter((text) => {
if (text === author) return false;
if (text.length <= 10) return false;
if (isSuggestionOrChrome(text) || isSponsored(text)) return false;
if (isActionText(text) || isMetricText(text) || isTimestampText(text)) return false;
if (/^(See more|查看更多|更多)$/i.test(text)) return false;
return true;
});
}
function extractPost(root, index) {
const fullText = textOf(root);
if (!fullText || isSuggestionOrChrome(fullText) || isSponsored(fullText)) return null;
const author = findAuthor(root);
const blocks = contentBlocks(root, author);
const content = clean(blocks.join(' '));
const postUrl = postUrlFrom(root);
const kinds = actionKinds(root);
if (!author && !content) return null;
if (!content && !postUrl && kinds.size < 2) return null;
const likesMatch = fullText.match(/所有心情:\\s*(\\d[\\d,.\\s万亿KMk]*)/)
|| fullText.match(/All:\\s*(\\d[\\d,.KMk]*)/)
|| fullText.match(/(\\d[\\d,.KMk]*)\\s*(?:likes?|reactions?)/i);
const commentsMatch = fullText.match(/([\\d,.]+\\s*[万亿]?)\\s*条评论/)
|| fullText.match(/(\\d[\\d,.KMk]*)\\s*comments?/i);
const sharesMatch = fullText.match(/([\\d,.]+\\s*[万亿]?)\\s*次分享/)
|| fullText.match(/(\\d[\\d,.KMk]*)\\s*shares?/i);
return {
index: i + 1,
index,
author: author.substring(0, 50),
content: content.replace(/\\n/g, ' ').substring(0, 120),
likes: likesMatch ? likesMatch[1] : '-',
comments: commentsMatch ? commentsMatch[1] : '-',
shares: sharesMatch ? sharesMatch[1] : '-',
content: content.substring(0, 120),
likes: likesMatch ? clean(likesMatch[1]) : '-',
comments: commentsMatch ? clean(commentsMatch[1]) : '-',
shares: sharesMatch ? clean(sharesMatch[1]) : '-',
};
});
})()
` },
],
});
}
function primaryContainers() {
return Array.from(document.querySelectorAll('[role="article"]'))
.filter((el) => textOf(el).length > 30);
}
function fallbackContainers() {
const main = document.querySelector('[role="main"]');
if (!main) return [];
const buttons = Array.from(main.querySelectorAll('[aria-label="Like"], [aria-label="赞"], [aria-label="Comment"], [aria-label="评论"], [aria-label="Share"], [aria-label="分享"]'));
const seen = new WeakSet();
const containers = [];
for (const button of buttons) {
let node = button.parentElement;
for (let depth = 0; depth < 16 && node && node !== main && node !== document.body; depth += 1, node = node.parentElement) {
const text = textOf(node);
const kinds = actionKinds(node);
const blocks = visibleBlocks(node);
const hasPostEvidence = Boolean(postUrlFrom(node)) || blocks.some((block) => block.length > 20 && !isActionText(block) && !isMetricText(block));
if (text.length >= 80 && kinds.has('like') && (kinds.has('comment') || kinds.has('share')) && hasPostEvidence) {
if (!seen.has(node)) {
seen.add(node);
containers.push(node);
}
break;
}
}
}
return containers;
}
function dedupe(containers) {
const seen = new Set();
const result = [];
for (const node of containers) {
const key = postUrlFrom(node) || contentBlocks(node, findAuthor(node)).join('|').substring(0, 200);
if (!key || seen.has(key)) continue;
seen.add(key);
result.push(node);
}
return result;
}
if (isAuthPage()) return { status: 'auth', rows: [], diagnostics: {} };
const primary = primaryContainers();
const combined = dedupe([...primary, ...fallbackContainers()]);
const rows = [];
for (const container of combined) {
const row = extractPost(container, rows.length + 1);
if (row) rows.push(row);
if (rows.length >= limit) break;
}
return {
status: rows.length ? 'ok' : (isExplicitEmptyFeed() ? 'empty' : 'no_rows'),
rows,
diagnostics: {
articleCount: document.querySelectorAll('[role="article"]').length,
primaryCount: primary.length,
fallbackActionCount: document.querySelectorAll('[role="main"] [aria-label="Like"], [role="main"] [aria-label="赞"], [role="main"] [aria-label="Comment"], [role="main"] [aria-label="评论"]').length,
mainTextLength: textOf(document.querySelector('[role="main"]')).length,
},
};
})()`;
}
async function getFacebookFeed(page, kwargs) {
const limit = requireLimit(kwargs.limit ?? 10);
try {
await page.goto(FACEBOOK_HOME, { settleMs: 4000 });
} catch (err) {
throw new CommandExecutionError(
`Failed to navigate to facebook feed: ${err instanceof Error ? err.message : err}`,
'Check that facebook.com is reachable and the browser extension is connected.',
);
}
let payload;
try {
payload = unwrapBrowserResult(await page.evaluate(buildFeedExtractScript(limit)));
} catch (err) {
throw new CommandExecutionError(
`Failed to read facebook feed: ${err instanceof Error ? err.message : err}`,
'Facebook may not have rendered or the feed markup may have changed.',
);
}
if (!payload || typeof payload !== 'object' || !Array.isArray(payload.rows)) {
throw new CommandExecutionError('facebook feed returned malformed extraction payload');
}
if (payload.status === 'auth') {
throw new AuthRequiredError('www.facebook.com', 'Open Chrome and log in to Facebook before retrying.');
}
if (payload.rows.length > 0) {
return payload.rows;
}
if (payload.status === 'empty') {
throw new EmptyResultError('facebook feed', 'Facebook did not show any feed posts for this account.');
}
const diagnostics = payload.diagnostics || {};
if (diagnostics.articleCount || diagnostics.fallbackActionCount || diagnostics.mainTextLength > 200) {
throw new CommandExecutionError(
'facebook feed page rendered but no feed rows could be extracted',
`Diagnostics: articles=${diagnostics.articleCount || 0}, actions=${diagnostics.fallbackActionCount || 0}, mainTextLength=${diagnostics.mainTextLength || 0}.`,
);
}
throw new EmptyResultError('facebook feed', 'No Facebook feed content was visible in the current browser session.');
}
const command = {
site: 'facebook',
name: 'feed',
access: 'read',
description: 'Get your Facebook news feed',
domain: 'www.facebook.com',
strategy: Strategy.COOKIE,
browser: true,
navigateBefore: false,
args: [
{ name: 'limit', type: 'int', default: 10, help: 'Number of posts' },
],
columns: ['index', 'author', 'content', 'likes', 'comments', 'shares'],
func: getFacebookFeed,
};
cli(command);
export const __test__ = {
buildFeedExtractScript,
command,
getFacebookFeed,
requireLimit,
};
+169
View File
@@ -0,0 +1,169 @@
import { describe, expect, it, vi } from 'vitest';
import { JSDOM } from 'jsdom';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { getRegistry } from '@jackwener/opencli/registry';
import { __test__ } from './feed.js';
function runExtract(html, limit = 10, url = 'https://www.facebook.com/') {
const dom = new JSDOM(html, { url });
return Function('window', 'document', `return ${__test__.buildFeedExtractScript(limit)};`)(dom.window, dom.window.document);
}
function createPage(payload) {
return {
goto: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue(payload),
};
}
describe('facebook feed', () => {
it('registers the feed command with the existing row contract', () => {
const cmd = getRegistry().get('facebook/feed');
expect(cmd).toBeDefined();
expect(cmd.columns).toEqual(['index', 'author', 'content', 'likes', 'comments', 'shares']);
});
it('extracts existing role=article feed rows', () => {
const payload = runExtract(`
<main role="main">
<div role="article">
<h2><a href="https://www.facebook.com/alice">Alice Example</a></h2>
<div dir="auto">This is a normal Facebook feed post with enough text to extract.</div>
<span>All: 12</span>
<span>3 comments</span>
<span>2 shares</span>
<div aria-label="Like"></div><div aria-label="Comment"></div>
</div>
</main>
`);
expect(payload.status).toBe('ok');
expect(payload.rows).toEqual([{
index: 1,
author: 'Alice Example',
content: 'This is a normal Facebook feed post with enough text to extract.',
likes: '12',
comments: '3',
shares: '2',
}]);
});
it('falls back from empty article nodes to action-bounded feed containers', () => {
const payload = runExtract(`
<main role="main">
<div role="article"></div>
<section>
<div>
<h2><a href="https://www.facebook.com/bob/posts/123">Bob Builder</a></h2>
<div dir="auto">Fallback post body from a Facebook feed card with empty article text.</div>
<a href="https://www.facebook.com/bob/posts/123">Permalink</a>
<span>All: 1.2K</span>
<span>4 comments</span>
<span>1 shares</span>
<div><button aria-label="Like">Like</button><button aria-label="Comment">Comment</button></div>
</div>
</section>
</main>
`);
expect(payload.status).toBe('ok');
expect(payload.rows).toEqual([{
index: 1,
author: 'Bob Builder',
content: 'Fallback post body from a Facebook feed card with empty article text.',
likes: '1.2K',
comments: '4',
shares: '1',
}]);
});
it('does not turn suggestions or side chrome action buttons into feed rows', () => {
const payload = runExtract(`
<main role="main">
<aside>
<h2>People you may know</h2>
<div dir="auto">Charlie Suggested</div>
<div dir="auto">Add friend from suggested people card with plenty of text.</div>
<button aria-label="Like">Like</button>
<button aria-label="Comment">Comment</button>
</aside>
<nav>
<div dir="auto">Navigation item with a Like button but not a feed post.</div>
<button aria-label="Like">Like</button>
<button aria-label="Comment">Comment</button>
</nav>
</main>
`);
expect(payload.status).toBe('no_rows');
expect(payload.rows).toEqual([]);
});
it('still considers bounded fallback rows when article nodes are suggestion chrome', () => {
const payload = runExtract(`
<main role="main">
<div role="article">
<h2>People you may know</h2>
<div dir="auto">Suggested profile card with enough text to look article-like.</div>
<button aria-label="Like">Like</button>
<button aria-label="Comment">Comment</button>
</div>
<section>
<div>
<h2><a href="https://www.facebook.com/dana/posts/456">Dana Poster</a></h2>
<div dir="auto">Fallback feed post should still be extracted after suggestion articles are filtered.</div>
<a href="https://www.facebook.com/dana/posts/456">Permalink</a>
<button aria-label="Like">Like</button>
<button aria-label="Comment">Comment</button>
</div>
</section>
</main>
`, 1);
expect(payload.status).toBe('ok');
expect(payload.rows).toEqual([{
index: 1,
author: 'Dana Poster',
content: 'Fallback feed post should still be extracted after suggestion articles are filtered.',
likes: '-',
comments: '-',
shares: '-',
}]);
});
it('reports auth pages from the browser extractor', () => {
const payload = runExtract('<main role="main">Log in to Facebook</main>', 10, 'https://www.facebook.com/login/');
expect(payload.status).toBe('auth');
expect(payload.rows).toEqual([]);
});
it('validates limit before browser navigation', async () => {
const page = createPage({ status: 'ok', rows: [] });
await expect(__test__.command.func(page, { limit: 0 })).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
});
it('maps browser envelopes and returns extracted rows', async () => {
const page = createPage({ session: 'site:facebook', data: { status: 'ok', rows: [{ index: 1, author: 'A', content: 'Body', likes: '-', comments: '-', shares: '-' }] } });
await expect(__test__.command.func(page, { limit: 1 })).resolves.toEqual([{
index: 1,
author: 'A',
content: 'Body',
likes: '-',
comments: '-',
shares: '-',
}]);
});
it('maps auth, real empty, parser drift, and malformed payloads to typed errors', async () => {
await expect(__test__.command.func(createPage({ status: 'auth', rows: [] }), { limit: 1 }))
.rejects.toBeInstanceOf(AuthRequiredError);
await expect(__test__.command.func(createPage({ status: 'empty', rows: [] }), { limit: 1 }))
.rejects.toBeInstanceOf(EmptyResultError);
await expect(__test__.command.func(createPage({ status: 'no_rows', rows: [], diagnostics: { articleCount: 1, fallbackActionCount: 2, mainTextLength: 500 } }), { limit: 1 }))
.rejects.toBeInstanceOf(CommandExecutionError);
await expect(__test__.command.func(createPage({ rows: null }), { limit: 1 }))
.rejects.toBeInstanceOf(CommandExecutionError);
});
});
+20 -5
View File
@@ -1,4 +1,5 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { clampInt, requireNonEmptyQuery } from '../_shared/common.js';
cli({
@@ -18,12 +19,17 @@ cli({
const limit = clampInt(kwargs.limit, 10, 1, 20);
const query = requireNonEmptyQuery(kwargs.query);
await page.goto(`https://scholar.google.com/scholar?q=${encodeURIComponent(query)}&hl=zh-CN`);
await page.wait(3);
const data = await page.evaluate(`
try {
await page.wait({ selector: '.gs_r.gs_or.gs_scl', timeout: 5 });
} catch {
await page.wait(3);
}
const wrapper = await page.evaluate(`
(() => {
const normalize = v => (v || '').replace(/\\s+/g, ' ').trim();
const results = [];
for (const el of document.querySelectorAll('.gs_r.gs_or.gs_scl')) {
const resultCards = Array.from(document.querySelectorAll('.gs_r.gs_or.gs_scl'));
for (const el of resultCards) {
const container = el.querySelector('.gs_ri') || el;
const titleEl = container.querySelector('.gs_rt a, h3 a');
const title = normalize(titleEl?.textContent);
@@ -50,9 +56,18 @@ cli({
});
if (results.length >= ${limit}) break;
}
return results;
return { items: results, resultCount: resultCards.length };
})()
`);
return Array.isArray(data) ? data : [];
if (!wrapper || typeof wrapper !== 'object' || !Array.isArray(wrapper.items)) {
throw new CommandExecutionError('Google Scholar search returned an unexpected payload shape');
}
if (wrapper.items.length === 0) {
if (Number(wrapper.resultCount) > 0) {
throw new CommandExecutionError('Google Scholar result cards were present but no rows could be extracted');
}
throw new EmptyResultError('google-scholar/search', 'Try a different query or check whether Google Scholar returned a CAPTCHA.');
}
return wrapper.items;
},
});
+35 -2
View File
@@ -1,4 +1,5 @@
import { describe, expect, it, vi } from 'vitest';
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { getRegistry } from '@jackwener/opencli/registry';
import './search.js';
@@ -25,14 +26,46 @@ describe('google-scholar search command', () => {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue([]),
evaluate: vi.fn().mockResolvedValue({ items: [{ rank: 1, title: 'Paper' }], resultCount: 1 }),
};
await command.func(page, { query: 'transformer' });
const rows = await command.func(page, { query: 'transformer' });
const script = page.evaluate.mock.calls[0][0];
expect(script).toContain("document.querySelectorAll('.gs_r.gs_or.gs_scl')");
expect(script).not.toContain(".gs_r.gs_or.gs_scl, .gs_ri");
expect(script).toContain("const container = el.querySelector('.gs_ri') || el");
expect(script).toContain('return { items: results, resultCount: resultCards.length }');
expect(rows).toEqual([{ rank: 1, title: 'Paper' }]);
});
it('throws typed empty when Scholar returns no result cards', async () => {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue({ items: [], resultCount: 0 }),
};
await expect(command.func(page, { query: 'no results expected' })).rejects.toThrow(EmptyResultError);
});
it('throws command execution when result cards exist but parser extracts no rows', async () => {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue({ items: [], resultCount: 2 }),
};
await expect(command.func(page, { query: 'parser drift' })).rejects.toThrow(CommandExecutionError);
});
it('throws command execution for malformed evaluate payloads instead of treating them as empty', async () => {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue({ items: { rank: 1 }, resultCount: 1 }),
};
await expect(command.func(page, { query: 'bad payload' })).rejects.toThrow(CommandExecutionError);
});
});
+5 -4
View File
@@ -39,12 +39,12 @@ cli({
catch {
await page.wait(2);
}
const results = await page.evaluate(`
const wrapper = await page.evaluate(`
(function() {
var results = [];
var seenUrls = {};
var rso = document.querySelector('#rso');
if (!rso) return results;
if (!rso) return {items: results};
// -- Featured snippet (scoped to #rso to avoid matching unrelated elements) --
var featuredEl = rso.querySelector('.xpdopen .hgKElc')
@@ -126,10 +126,11 @@ cli({
}
}
return results;
return {items: results};
})()
`);
if (!Array.isArray(results) || results.length === 0) {
const results = (wrapper && wrapper.items) || [];
if (results.length === 0) {
throw new CliError('NOT_FOUND', 'No search results found', 'Try a different keyword or check for CAPTCHA');
}
return results;
-32
View File
@@ -1,32 +0,0 @@
import * as fs from 'node:fs';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const exportCommand = cli({
site: 'notion',
name: 'export',
access: 'read',
description: 'Export the current Notion page as Markdown',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [
{ name: 'output', required: false, help: 'Output file (default: /tmp/notion-export.md)' },
],
columns: ['Status', 'File'],
func: async (page, kwargs) => {
const outputPath = kwargs.output || '/tmp/notion-export.md';
const result = await page.evaluate(`
(function() {
const titleEl = document.querySelector('[data-block-id] [placeholder="Untitled"], h1.notion-title, [class*="title"]');
const title = titleEl ? (titleEl.textContent || '').trim() : document.title;
const frame = document.querySelector('.notion-page-content, [class*="page-content"], main');
const content = frame ? (frame.innerText || '').trim() : document.body.innerText;
return { title, content };
})()
`);
const md = `# ${result.title}\n\n${result.content}`;
fs.writeFileSync(outputPath, md);
return [{ Status: 'Success', File: outputPath }];
},
});
-85
View File
@@ -1,85 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const favoritesCommand = cli({
site: 'notion',
name: 'favorites',
access: 'read',
description: 'List pages from the Notion Favorites section in the sidebar',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [],
columns: ['Index', 'Title', 'Icon'],
func: async (page) => {
const items = await page.evaluate(`
(function() {
const results = [];
// Strategy 1: Use Notion's own class 'notion-outliner-bookmarks-header-container'
const headerContainer = document.querySelector('.notion-outliner-bookmarks-header-container');
if (headerContainer) {
// Walk up to the section parent that wraps header + items
let section = headerContainer.parentElement;
if (section && section.children.length === 1) section = section.parentElement;
if (section) {
const treeItems = section.querySelectorAll('[role="treeitem"]');
treeItems.forEach((item) => {
// Title text is in a div.notranslate sibling of the icon area
const titleEl = item.querySelector('div.notranslate:not(.notion-record-icon)');
const title = titleEl
? titleEl.textContent.trim()
: (item.textContent || '').trim().substring(0, 80);
// Icon/emoji is in the notion-record-icon element
const iconEl = item.querySelector('.notion-record-icon');
const icon = iconEl ? iconEl.textContent.trim().substring(0, 4) : '';
if (title && title.length > 0) {
results.push({ Index: results.length + 1, Title: title, Icon: icon || '📄' });
}
});
}
}
// Strategy 2: Fallback — find "Favorites" text node and walk DOM
if (results.length === 0) {
const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT, null);
let node;
let favEl = null;
while (node = walker.nextNode()) {
const text = node.textContent.trim();
if (text === 'Favorites' || text === '收藏' || text === '收藏夹') {
favEl = node.parentElement;
break;
}
}
if (favEl) {
let section = favEl;
for (let i = 0; i < 6; i++) {
const p = section.parentElement;
if (!p || p === document.body) break;
const treeItems = p.querySelectorAll(':scope > [role="treeitem"]');
if (treeItems.length > 0) { section = p; break; }
section = p;
}
const treeItems = section.querySelectorAll('[role="treeitem"]');
treeItems.forEach((item) => {
const text = (item.textContent || '').trim().substring(0, 120);
if (text && text.length > 1 && !text.match(/^(Favorites|收藏夹?)$/)) {
results.push({ Index: results.length + 1, Title: text, Icon: '📄' });
}
});
}
}
return results;
})()
`);
if (items.length === 0) {
return [{ Index: 0, Title: 'No favorites found. Make sure sidebar is visible and you have favorites.', Icon: '⚠️' }];
}
return items;
},
});
-35
View File
@@ -1,35 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const newCommand = cli({
site: 'notion',
name: 'new',
access: 'write',
description: 'Create a new page in Notion',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [
{ name: 'title', required: false, positional: true, help: 'Page title (optional)' },
],
columns: ['Status'],
func: async (page, kwargs) => {
const title = kwargs.title;
// Cmd+N creates a new page in Notion
const isMac = process.platform === 'darwin';
await page.pressKey(isMac ? 'Meta+N' : 'Control+N');
await page.wait(1);
// If title is provided, type it into the title field
if (title) {
await page.evaluate(`
(function(t) {
const titleEl = document.querySelector('[placeholder="Untitled"], [data-content-editable-leaf] [placeholder]');
if (titleEl) {
titleEl.focus();
document.execCommand('insertText', false, t);
}
})(${JSON.stringify(title)})
`);
await page.wait(0.5);
}
return [{ Status: title ? `Created page: ${title}` : 'New blank page created' }];
},
});
-31
View File
@@ -1,31 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const readCommand = cli({
site: 'notion',
name: 'read',
access: 'read',
description: 'Read the content of the currently open Notion page',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [],
columns: ['Title', 'Content'],
func: async (page) => {
const result = await page.evaluate(`
(function() {
// Get the page title
const titleEl = document.querySelector('[data-block-id] [placeholder="Untitled"], .notion-page-block .notranslate, h1.notion-title, [class*="title"]');
const title = titleEl ? (titleEl.textContent || '').trim() : document.title;
// Get the page content — Notion renders blocks in a frame
const frame = document.querySelector('.notion-page-content, [class*="page-content"], .layout-content, main');
const content = frame ? (frame.innerText || frame.textContent || '').trim() : '';
return { title, content };
})()
`);
return [{
Title: result.title || 'Untitled',
Content: result.content || '(empty page)',
}];
},
});
-47
View File
@@ -1,47 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const searchCommand = cli({
site: 'notion',
name: 'search',
access: 'read',
description: 'Search pages and databases in Notion via Quick Find (Cmd+P)',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [{ name: 'query', required: true, positional: true, help: 'Search query' }],
columns: ['Index', 'Title'],
func: async (page, kwargs) => {
const query = kwargs.query;
// Open Quick Find
const isMac = process.platform === 'darwin';
await page.pressKey(isMac ? 'Meta+P' : 'Control+P');
await page.wait(0.5);
// Type the search query
await page.evaluate(`
(function(q) {
const input = document.querySelector('input[placeholder*="Search"], input[type="text"]');
if (input) {
const setter = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set;
setter.call(input, q);
input.dispatchEvent(new Event('input', { bubbles: true }));
}
})(${JSON.stringify(query)})
`);
await page.wait(1.5);
// Scrape results
const results = await page.evaluate(`
(function() {
const items = document.querySelectorAll('[role="option"], [class*="searchResult"], [class*="quick-find"] [role="button"]');
return Array.from(items).slice(0, 20).map((item, i) => ({
Index: i + 1,
Title: (item.textContent || '').trim().substring(0, 120),
}));
})()
`);
// Close Quick Find
await page.pressKey('Escape');
if (results.length === 0) {
return [{ Index: 0, Title: `No results for "${query}"` }];
}
return results;
},
});
-42
View File
@@ -1,42 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const sidebarCommand = cli({
site: 'notion',
name: 'sidebar',
access: 'read',
description: 'List pages and databases from the Notion sidebar',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [],
columns: ['Index', 'Title'],
func: async (page) => {
const items = await page.evaluate(`
(function() {
const results = [];
// Notion sidebar items
const selectors = [
'[class*="sidebar"] [role="treeitem"]',
'[class*="sidebar"] a',
'.notion-sidebar [role="button"]',
'nav [role="treeitem"]',
];
for (const sel of selectors) {
const nodes = document.querySelectorAll(sel);
if (nodes.length > 0) {
nodes.forEach((n, i) => {
const text = (n.textContent || '').trim().substring(0, 100);
if (text && text.length > 1) results.push({ Index: i + 1, Title: text });
});
break;
}
}
return results;
})()
`);
if (items.length === 0) {
return [{ Index: 0, Title: 'No sidebar items found. Toggle the sidebar first.' }];
}
return items;
},
});
-17
View File
@@ -1,17 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const statusCommand = cli({
site: 'notion',
name: 'status',
access: 'read',
description: 'Check active CDP connection to Notion Desktop',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [],
columns: ['Status', 'Url', 'Title'],
func: async (page) => {
const url = await page.evaluate('window.location.href');
const title = await page.evaluate('document.title');
return [{ Status: 'Connected', Url: url, Title: title }];
},
});
-41
View File
@@ -1,41 +0,0 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
export const writeCommand = cli({
site: 'notion',
name: 'write',
access: 'write',
description: 'Append text content to the currently open Notion page',
domain: 'localhost',
strategy: Strategy.UI,
browser: true,
args: [{ name: 'text', required: true, positional: true, help: 'Text to append to the page' }],
columns: ['Status'],
func: async (page, kwargs) => {
const text = kwargs.text;
// Focus the page body and move to the end
await page.evaluate(`
(function(text) {
// Find the editable area in Notion
const editables = document.querySelectorAll('.notion-page-content [contenteditable="true"], [class*="page-content"] [contenteditable="true"]');
let target = editables.length > 0 ? editables[editables.length - 1] : null;
if (!target) {
// Fallback: just find any contenteditable
const all = document.querySelectorAll('[contenteditable="true"]');
target = all.length > 0 ? all[all.length - 1] : null;
}
if (!target) throw new Error('Could not find editable area in Notion page');
target.focus();
// Move to end
const sel = window.getSelection();
sel.selectAllChildren(target);
sel.collapseToEnd();
document.execCommand('insertText', false, text);
})(${JSON.stringify(text)})
`);
await page.wait(0.5);
return [{ Status: 'Text appended successfully' }];
},
});
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'post-id', type: 'string', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
{ name: 'text', type: 'string', required: true, positional: true, help: 'Comment text' },
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 15 },
],
-1
View File
@@ -23,7 +23,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 25, help: `Number of posts (1${REDDIT_HOME_MAX_LIMIT})` },
],
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 20 },
],
+402 -57
View File
@@ -5,9 +5,98 @@
* - Top-K comments by score at each level
* - Configurable depth and replies-per-level
* - Indented output showing conversation threads
* - Optional --expand-more to follow Reddit's "more comments" stubs via
* /api/morechildren.json (rdt-cli parity, PR B of #1481 follow-up)
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
const REDDIT_EXPAND_ROUNDS_MIN = 1;
const REDDIT_EXPAND_ROUNDS_MAX = 5;
const DEFAULT_EXPAND_ROUNDS = 2;
const REDDIT_POST_ID_RE = /^[a-z0-9]+$/i;
function normalizeBareRedditPostId(value) {
const postId = String(value || '').trim();
if (!REDDIT_POST_ID_RE.test(postId)) {
throw new ArgumentError(
'Post ID must be a Reddit post id, t3_ fullname, or reddit.com post URL.',
'Use a bare post id like 1abc123, a fullname like t3_1abc123, or a full Reddit post URL.',
);
}
return postId.toLowerCase();
}
export function normalizeRedditPostId(value) {
const raw = String(value || '').trim();
if (!raw) {
throw new ArgumentError(
'Post ID is required.',
'Use a bare post id like 1abc123, a fullname like t3_1abc123, or a full Reddit post URL.',
);
}
const fullname = raw.match(/^t3_([a-z0-9]+)$/i);
if (fullname) return normalizeBareRedditPostId(fullname[1]);
if (/^https?:\/\//i.test(raw)) {
let parsed;
try {
parsed = new URL(raw);
} catch {
throw new ArgumentError(`Invalid Reddit post URL: ${raw}`);
}
const host = parsed.hostname.toLowerCase();
if (parsed.protocol !== 'https:' || (host !== 'reddit.com' && !host.endsWith('.reddit.com'))) {
throw new ArgumentError(
'Post URL must be an https reddit.com URL.',
'Use a URL like https://www.reddit.com/r/sub/comments/1abc123/title_slug/',
);
}
const parts = parsed.pathname.split('/').filter(Boolean);
const commentsIndex = parts.indexOf('comments');
const postIndex = commentsIndex + 1;
if (commentsIndex < 0 || parts.length <= postIndex) {
throw new ArgumentError(
'Post URL must include the target post id.',
'Use a URL like https://www.reddit.com/r/sub/comments/1abc123/title_slug/',
);
}
if (parts.length > postIndex + 3) {
throw new ArgumentError(
'Post URL must end at the post slug or comment permalink id.',
'Remove extra path segments after the post slug or comment id.',
);
}
if (parts.length === postIndex + 3) normalizeBareRedditPostId(parts[postIndex + 2]);
return normalizeBareRedditPostId(parts[postIndex]);
}
if (raw.includes('/') || raw.startsWith('t1_')) {
throw new ArgumentError(
'Post ID must be a Reddit post id, t3_ fullname, or reddit.com post URL.',
'Use a bare post id like 1abc123, a fullname like t3_1abc123, or a full Reddit post URL.',
);
}
return normalizeBareRedditPostId(raw);
}
export function parseExpandRounds(raw) {
if (raw === undefined || raw === null || raw === '') return DEFAULT_EXPAND_ROUNDS;
const n = Number(raw);
if (
!Number.isFinite(n) || !Number.isInteger(n)
|| n < REDDIT_EXPAND_ROUNDS_MIN || n > REDDIT_EXPAND_ROUNDS_MAX
) {
throw new ArgumentError(
`expand-rounds must be an integer in [${REDDIT_EXPAND_ROUNDS_MIN}, ${REDDIT_EXPAND_ROUNDS_MAX}].`,
`Got: ${raw}`,
);
}
return n;
}
cli({
site: 'reddit',
name: 'read',
@@ -16,7 +105,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'post-id', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or full URL' },
{ name: 'sort', default: 'best', help: 'Comment sort: best, top, new, controversial, old, qa' },
@@ -24,80 +112,314 @@ cli({
{ name: 'depth', type: 'int', default: 2, help: 'Max reply depth (1=no replies, 2=one level of replies, etc.)' },
{ name: 'replies', type: 'int', default: 5, help: 'Max replies shown per comment at each level (sorted by score)' },
{ name: 'max-length', type: 'int', default: 2000, help: 'Max characters per comment body (min 100)' },
{
name: 'expand-more',
type: 'bool',
default: false,
help: 'Follow Reddit "more comments" stubs by calling /api/morechildren.json',
},
{
name: 'expand-rounds',
type: 'int',
default: DEFAULT_EXPAND_ROUNDS,
help: `Max expansion passes when --expand-more is on (${REDDIT_EXPAND_ROUNDS_MIN}${REDDIT_EXPAND_ROUNDS_MAX}; each round can fan out new "more" stubs)`,
},
],
columns: ['type', 'author', 'score', 'text'],
func: async (page, kwargs) => {
// Note: --limit / --depth / --replies / --max-length keep their original
// Math.max-style behaviour for backward compatibility (grandfathered in
// the typed-error-lint baseline). The new --expand-rounds argument is
// strictly validated via parseExpandRounds — no silent clamp.
const sort = kwargs.sort ?? 'best';
const limit = Math.max(1, kwargs.limit ?? 25);
const maxDepth = Math.max(1, kwargs.depth ?? 2);
const maxReplies = Math.max(1, kwargs.replies ?? 5);
const maxLength = Math.max(100, kwargs['max-length'] ?? 2000);
const expandMore = Boolean(kwargs['expand-more']);
const expandRounds = parseExpandRounds(kwargs['expand-rounds']);
const postId = normalizeRedditPostId(kwargs['post-id']);
await page.goto('https://www.reddit.com');
const data = await page.evaluate(`
// The in-browser script returns a discriminated union so we can map
// each failure mode to its proper typed error on the Node side
// (page.evaluate boundary can't carry typed error instances). Kinds:
// - inaccessible: 401/403/404 on /comments/<id>.json (post-specific,
// not session auth — same session works for other posts)
// - auth: /api/morechildren.json 401/403 (session-level on
// the write-like expand endpoint — see two-pronged auth detection
// sediment from PR #1428)
// - http: 5xx or other non-ok
// - malformed: 200 but Reddit shape is unexpected (schema drift)
// - parser-drift: tree non-empty but walk produced 0 rows
// - expand-failed: morechildren returned errors
// - ok: rows array
//
// Intermediate keys (`rows` / `detail` / `httpStatus` / `where`)
// deliberately avoid the declared columns (`type`/`author`/`score`/
// `text`) to sidestep the silent-column-drop audit (PR #1329).
const result = await page.evaluate(`
(async function() {
var postId = ${JSON.stringify(kwargs['post-id'])};
var urlMatch = postId.match(/comments\\/([a-z0-9]+)/);
if (urlMatch) postId = urlMatch[1];
var postId = ${JSON.stringify(postId)};
var linkFullname = 't3_' + postId;
var sort = ${JSON.stringify(sort)};
var limit = ${limit};
var maxDepth = ${maxDepth};
var maxReplies = ${maxReplies};
var maxLength = ${maxLength};
var expandMore = ${JSON.stringify(expandMore)};
var expandRounds = ${expandRounds};
// Request more from API than top-level limit to get inline replies
// depth param tells Reddit how deep to inline replies vs "more" stubs
// ---------------------------------------------------------------
// Step 1: fetch the post + initial comment tree
// ---------------------------------------------------------------
// Request more from API than top-level limit to get inline replies.
// depth param tells Reddit how deep to inline replies vs "more" stubs.
var apiLimit = Math.max(limit * 3, 100);
var res = await fetch(
'/comments/' + postId + '.json?sort=' + sort + '&limit=' + apiLimit + '&depth=' + (maxDepth + 1) + '&raw_json=1',
{ credentials: 'include' }
);
if (!res.ok) return { error: 'Reddit API returned HTTP ' + res.status };
if (res.status === 401 || res.status === 403 || res.status === 404) {
return { kind: 'inaccessible', detail: 'Reddit post ' + postId + ' is not accessible (HTTP ' + res.status + ').' };
}
if (!res.ok) {
return { kind: 'http', httpStatus: res.status, where: '/comments/' + postId + '.json' };
}
var data;
try { data = await res.json(); } catch(e) { return { error: 'Failed to parse response' }; }
if (!Array.isArray(data) || data.length < 2) return { error: 'Unexpected response format' };
var results = [];
// Post
var post = data[0] && data[0].data && data[0].data.children && data[0].data.children[0] && data[0].data.children[0].data;
if (post) {
var body = post.selftext || '';
if (body.length > maxLength) body = body.slice(0, maxLength) + '\\n... [truncated]';
results.push({
type: 'POST',
author: post.author || '[deleted]',
score: post.score || 0,
text: post.title + (body ? '\\n\\n' + body : '') + (post.url && !post.is_self ? '\\n' + post.url : ''),
});
try { data = await res.json(); } catch (e) {
return { kind: 'malformed', detail: 'Failed to parse Reddit /comments/' + postId + '.json response: ' + (e && e.message || e) };
}
if (!Array.isArray(data) || data.length < 2) {
return { kind: 'malformed', detail: 'Reddit /comments/' + postId + '.json had unexpected envelope shape (length ' + (Array.isArray(data) ? data.length : typeof data) + ').' };
}
// Recursive comment walker
// depth 0 = top-level comments; maxDepth is exclusive,
// so --depth 1 means top-level only, --depth 2 means one reply level, etc.
var post = data[0] && data[0].data && data[0].data.children && data[0].data.children[0] && data[0].data.children[0].data;
if (!post) {
return { kind: 'malformed', detail: 'Reddit /comments/' + postId + '.json had no post body.' };
}
var topListing = data[1] && data[1].data && Array.isArray(data[1].data.children) ? data[1].data.children : null;
if (!topListing) {
return { kind: 'malformed', detail: 'Reddit /comments/' + postId + '.json had no comment listing.' };
}
// ---------------------------------------------------------------
// Step 2: optionally follow "more" stubs via /api/morechildren.json
// ---------------------------------------------------------------
// Each "more" thing has a .data.children array (t1 ids to fetch).
// The morechildren API returns a FLAT list of things; we re-thread
// them by parent_id (either t3_<postId> for top-level or t1_<id>
// for nested). Each round may surface new "more" stubs (because
// expansion is bounded by Reddit's depth param), so we iterate up
// to expandRounds times.
var expandMeta = { rounds: 0, fetched: 0, capped: false, errors: [] };
if (expandMore) {
// Index every existing t1 node so we can splice replies onto it.
var t1Index = {};
function indexT1(arr) {
if (!Array.isArray(arr)) return;
for (var i = 0; i < arr.length; i++) {
var node = arr[i];
if (node && node.kind === 't1' && node.data && node.data.id) {
t1Index[node.data.name || ('t1_' + node.data.id)] = node;
if (node.data.replies && node.data.replies.data && node.data.replies.data.children) {
indexT1(node.data.replies.data.children);
}
}
}
}
indexT1(topListing);
// Collect "more" stubs (with non-empty children) from anywhere in
// the tree. Each stub knows its host array via a closure-bound
// reference we attach.
function collectMoreStubs(parentArr, parentT1) {
var out = [];
if (!Array.isArray(parentArr)) return out;
for (var i = 0; i < parentArr.length; i++) {
var n = parentArr[i];
if (!n || !n.data) continue;
if (n.kind === 'more' && Array.isArray(n.data.children) && n.data.children.length > 0) {
out.push({ stub: n, hostArr: parentArr, hostT1: parentT1 });
} else if (n.kind === 't1' && n.data.replies && n.data.replies.data && n.data.replies.data.children) {
var nested = collectMoreStubs(n.data.replies.data.children, n);
for (var k = 0; k < nested.length; k++) out.push(nested[k]);
}
}
return out;
}
for (var r = 0; r < expandRounds; r++) {
var stubs = collectMoreStubs(topListing, null);
if (stubs.length === 0) break;
// Build the union of t1 ids to request this round. Reddit's
// morechildren API caps at ~100 ids per call; batch accordingly.
var allIds = [];
for (var s = 0; s < stubs.length; s++) {
var st = stubs[s].stub;
for (var c = 0; c < st.data.children.length; c++) allIds.push(st.data.children[c]);
}
if (allIds.length === 0) break;
// dedupe preserving order
var seen = {};
var uniqIds = [];
for (var j = 0; j < allIds.length; j++) {
if (!seen[allIds[j]]) { seen[allIds[j]] = 1; uniqIds.push(allIds[j]); }
}
var fetchedThings = [];
var batchSize = 100;
var batchFailed = false;
for (var b = 0; b < uniqIds.length; b += batchSize) {
var batch = uniqIds.slice(b, b + batchSize);
var body = 'api_type=json'
+ '&link_id=' + encodeURIComponent(linkFullname)
+ '&children=' + encodeURIComponent(batch.join(','))
+ '&sort=' + encodeURIComponent(sort)
+ '&raw_json=1';
var mcRes;
try {
mcRes = await fetch('/api/morechildren', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: body,
});
} catch (e) {
return { kind: 'expand-failed', detail: 'morechildren request threw: ' + (e && e.message || e), expandMeta: expandMeta };
}
if (mcRes.status === 401 || mcRes.status === 403) {
return { kind: 'auth', detail: '/api/morechildren returned HTTP ' + mcRes.status + ' (write/expand likely requires login)' };
}
if (!mcRes.ok) {
return { kind: 'http', httpStatus: mcRes.status, where: '/api/morechildren (round ' + (r + 1) + ', batch ' + ((b / batchSize) + 1) + ')' };
}
var mcData;
try { mcData = await mcRes.json(); } catch (e) {
return { kind: 'malformed', detail: 'Failed to parse /api/morechildren response: ' + (e && e.message || e) };
}
var errs = mcData && mcData.json && mcData.json.errors;
if (Array.isArray(errs) && errs.length > 0) {
return { kind: 'expand-failed', detail: 'Reddit /api/morechildren rejected: ' + errs.map(function(e) { return e.join(': '); }).join('; '), expandMeta: expandMeta };
}
var things = mcData && mcData.json && mcData.json.data && mcData.json.data.things;
if (!Array.isArray(things)) {
return { kind: 'malformed', detail: '/api/morechildren returned no things array.' };
}
for (var t = 0; t < things.length; t++) fetchedThings.push(things[t]);
}
expandMeta.rounds = r + 1;
expandMeta.fetched += fetchedThings.length;
var fetchedById = {};
for (var t = 0; t < fetchedThings.length; t++) {
var thing = fetchedThings[t];
if (!thing || !thing.data) continue;
if (thing.data.id) fetchedById[thing.data.id] = thing;
if (thing.data.name) fetchedById[thing.data.name] = thing;
}
var inserted = {};
function thingKey(thing) {
return thing && thing.data && (thing.data.name || (thing.kind + '_' + thing.data.id));
}
// Replace each collected stub in-place so expansion preserves the
// surrounding tree order instead of appending fetched comments at
// the end of the parent array.
for (var s = 0; s < stubs.length; s++) {
var rec = stubs[s];
var idx = rec.hostArr.indexOf(rec.stub);
if (idx < 0) continue;
var expectedParent = rec.hostT1
? (rec.hostT1.data.name || ('t1_' + rec.hostT1.data.id))
: linkFullname;
var replacements = [];
for (var c = 0; c < rec.stub.data.children.length; c++) {
var childId = rec.stub.data.children[c];
var replacement = fetchedById[childId] || fetchedById['t1_' + childId];
if (!replacement || !replacement.data) {
expandMeta.errors.push('missing: ' + childId + ' parent=' + expectedParent);
continue;
}
var key = thingKey(replacement);
if (key && inserted[key]) continue;
if (replacement.data.parent_id !== expectedParent) {
expandMeta.errors.push('orphan: ' + (replacement.data.id || '?') + ' parent=' + (replacement.data.parent_id || '?'));
continue;
}
replacements.push(replacement);
if (key) inserted[key] = 1;
if (replacement.kind === 't1' && replacement.data && replacement.data.id) {
t1Index[replacement.data.name || ('t1_' + replacement.data.id)] = replacement;
}
}
rec.hostArr.splice(idx, 1, ...replacements);
}
for (var t = 0; t < fetchedThings.length; t++) {
var unplaced = fetchedThings[t];
var unplacedKey = thingKey(unplaced);
if (unplacedKey && inserted[unplacedKey]) continue;
if (unplaced && unplaced.data) {
expandMeta.errors.push('unplaced: ' + (unplaced.data.id || '?') + ' parent=' + (unplaced.data.parent_id || '?'));
continue;
}
}
if (r + 1 >= expandRounds) {
// If after the last round there are still "more" stubs, mark capped.
var remaining = collectMoreStubs(topListing, null);
if (remaining.length > 0) expandMeta.capped = true;
}
}
if (expandMeta.errors.length > 0) {
return { kind: 'expand-failed', detail: 'Reddit /api/morechildren returned unplaceable comments: ' + expandMeta.errors.slice(0, 5).join('; '), expandMeta: expandMeta };
}
}
// ---------------------------------------------------------------
// Step 3: walk the (possibly augmented) tree into indented rows
// ---------------------------------------------------------------
var rows = [];
// Post header row.
var body = post.selftext || '';
if (body.length > maxLength) body = body.slice(0, maxLength) + '\\n... [truncated]';
rows.push({
type: 'POST',
author: post.author || '[deleted]',
score: post.score || 0,
text: post.title + (body ? '\\n\\n' + body : '') + (post.url && !post.is_self ? '\\n' + post.url : ''),
});
// Recursive comment walker.
function walkComment(node, depth) {
if (!node || node.kind !== 't1') return;
var d = node.data;
var body = d.body || '';
if (body.length > maxLength) body = body.slice(0, maxLength) + '...';
var cBody = d.body || '';
if (cBody.length > maxLength) cBody = cBody.slice(0, maxLength) + '...';
// Indent prefix: apply to every line so multiline bodies stay aligned
var indent = '';
for (var i = 0; i < depth; i++) indent += ' ';
var prefix = depth === 0 ? '' : indent + '> ';
var indentedBody = depth === 0
? body
: body.split('\\n').map(function(line) { return prefix + line; }).join('\\n');
? cBody
: cBody.split('\\n').map(function(line) { return prefix + line; }).join('\\n');
results.push({
rows.push({
type: depth === 0 ? 'L0' : 'L' + depth,
author: d.author || '[deleted]',
score: d.score || 0,
text: indentedBody,
});
// Count all available replies (for accurate "more" count)
var t1Children = [];
var moreCount = 0;
if (d.replies && d.replies.data && d.replies.data.children) {
@@ -111,13 +433,12 @@ cli({
}
}
// At depth cutoff: don't recurse, but show all replies as hidden
if (depth + 1 >= maxDepth) {
var totalHidden = t1Children.length + moreCount;
if (totalHidden > 0) {
var cutoffIndent = '';
for (var j = 0; j <= depth; j++) cutoffIndent += ' ';
results.push({
rows.push({
type: 'L' + (depth + 1),
author: '',
score: '',
@@ -127,19 +448,17 @@ cli({
return;
}
// Sort by score descending, take top N
t1Children.sort(function(a, b) { return (b.data.score || 0) - (a.data.score || 0); });
var toProcess = Math.min(t1Children.length, maxReplies);
for (var i = 0; i < toProcess; i++) {
walkComment(t1Children[i], depth + 1);
}
// Show hidden count (skipped replies + "more" stubs)
var hidden = t1Children.length - toProcess + moreCount;
if (hidden > 0) {
var moreIndent = '';
for (var j = 0; j <= depth; j++) moreIndent += ' ';
results.push({
rows.push({
type: 'L' + (depth + 1),
author: '',
score: '',
@@ -148,24 +467,24 @@ cli({
}
}
// Walk top-level comments
var topLevel = data[1].data.children || [];
var t1TopLevel = [];
for (var i = 0; i < topLevel.length; i++) {
if (topLevel[i].kind === 't1') t1TopLevel.push(topLevel[i]);
for (var i = 0; i < topListing.length; i++) {
if (topListing[i].kind === 't1') t1TopLevel.push(topListing[i]);
}
// Top-level are already sorted by Reddit (sort param), take top N
// Detect parser drift: tree had content but the walker produced nothing.
// We must check this AFTER the walk because top-level may be only "more"
// stubs (legitimate empty case for a brand-new post).
var preWalkSize = topListing.length;
for (var i = 0; i < Math.min(t1TopLevel.length, limit); i++) {
walkComment(t1TopLevel[i], 0);
}
// Count remaining
var moreTopLevel = topLevel.filter(function(c) { return c.kind === 'more'; })
var moreTopLevel = topListing.filter(function(c) { return c.kind === 'more'; })
.reduce(function(sum, c) { return sum + (c.data.count || 0); }, 0);
var hiddenTopLevel = Math.max(0, t1TopLevel.length - limit) + moreTopLevel;
if (hiddenTopLevel > 0) {
results.push({
rows.push({
type: '',
author: '',
score: '',
@@ -173,15 +492,41 @@ cli({
});
}
return results;
// If we produced nothing beyond the POST row but the comment listing
// wasn't empty, that's parser drift (e.g. Reddit changed t1/more
// schema). Surface as CommandExecutionError on the Node side.
if (rows.length <= 1 && preWalkSize > 0 && t1TopLevel.length > 0) {
return { kind: 'parser-drift', detail: 'Reddit comment listing for post ' + postId + ' had ' + t1TopLevel.length + ' t1 entries but walker produced no rows.' };
}
return { kind: 'ok', rows: rows, expandMeta: expandMeta };
})()
`);
if (!data || typeof data !== 'object')
throw new CommandExecutionError('Failed to fetch post data');
if (!Array.isArray(data) && data.error)
throw new CommandExecutionError(data.error);
if (!Array.isArray(data))
throw new CommandExecutionError('Unexpected response');
return data;
if (!result || typeof result !== 'object') {
throw new CommandExecutionError('Reddit /comments fetch returned no result envelope.');
}
if (result.kind === 'inaccessible') {
throw new EmptyResultError(result.detail);
}
if (result.kind === 'auth') {
throw new AuthRequiredError('reddit.com', result.detail);
}
if (result.kind === 'http') {
throw new CommandExecutionError(`HTTP ${result.httpStatus} from ${result.where}`);
}
if (result.kind === 'malformed') {
throw new CommandExecutionError(result.detail);
}
if (result.kind === 'parser-drift') {
throw new CommandExecutionError(result.detail);
}
if (result.kind === 'expand-failed') {
throw new CommandExecutionError(result.detail);
}
if (result.kind !== 'ok' || !Array.isArray(result.rows)) {
throw new CommandExecutionError(`Unexpected result from reddit read: ${JSON.stringify(result)}`);
}
return result.rows;
},
});
+317 -14
View File
@@ -1,20 +1,163 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { normalizeRedditPostId, parseExpandRounds } from './read.js';
import './read.js';
function makePage(result) {
return {
goto: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue(result),
};
}
function redditPostEnvelope(children) {
return [
{
data: {
children: [{
data: {
title: 'Post title',
selftext: '',
author: 'op',
score: 10,
is_self: true,
},
}],
},
},
{ data: { children } },
];
}
function commentThing(id, body, parent = 't3_abc123', score = 1) {
return {
kind: 't1',
data: {
id,
name: `t1_${id}`,
parent_id: parent,
author: id,
score,
body,
replies: '',
},
};
}
function moreThing(id, children, parent = 't3_abc123', count = children.length) {
return {
kind: 'more',
data: { id, parent_id: parent, children, count },
};
}
function jsonResponse(payload, status = 200) {
return {
ok: status >= 200 && status < 300,
status,
json: vi.fn().mockResolvedValue(payload),
};
}
function makeRuntimePage(fetchImpl) {
return {
goto: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn(async (script) => {
const previousFetch = globalThis.fetch;
globalThis.fetch = fetchImpl;
try {
return await eval(script);
} finally {
globalThis.fetch = previousFetch;
}
}),
};
}
describe('reddit read adapter', () => {
const command = getRegistry().get('reddit/read');
it('opts into the Reddit persistent site session', () => {
it('uses an ephemeral Reddit site tab by default', () => {
expect(command?.browser).toBe(true);
expect(command?.siteSession).toBe('persistent');
expect(command?.siteSession).toBeUndefined();
expect(command?.columns).toEqual(['type', 'author', 'score', 'text']);
});
it('returns threaded rows from the browser-evaluated payload', async () => {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue([
it('exposes the new --expand-more / --expand-rounds args', () => {
const argNames = command.args.map((a) => a.name);
expect(argNames).toContain('expand-more');
expect(argNames).toContain('expand-rounds');
const expandMore = command.args.find((a) => a.name === 'expand-more');
expect(expandMore.type).toBe('bool');
expect(expandMore.default).toBe(false);
const rounds = command.args.find((a) => a.name === 'expand-rounds');
expect(rounds.type).toBe('int');
expect(rounds.default).toBe(2);
});
describe('normalizeRedditPostId', () => {
it('accepts bare ids, t3 fullnames, and exact reddit post URLs', () => {
expect(normalizeRedditPostId('1AbC23')).toBe('1abc23');
expect(normalizeRedditPostId('t3_1AbC23')).toBe('1abc23');
expect(normalizeRedditPostId('https://www.reddit.com/r/opencli/comments/1abc23/title_slug/?sort=top')).toBe('1abc23');
expect(normalizeRedditPostId('https://www.reddit.com/r/opencli/comments/1abc23/title_slug/okf3s7u/?context=3')).toBe('1abc23');
expect(normalizeRedditPostId('https://old.reddit.com/comments/1abc23/title_slug/')).toBe('1abc23');
});
it('rejects invalid or structurally loose post identities before navigation', () => {
for (const bad of [
'',
't1_okf3s7u',
'https://reddit.com.evil.com/r/opencli/comments/1abc23/title_slug/',
'http://www.reddit.com/r/opencli/comments/1abc23/title_slug/',
'https://www.reddit.com/r/opencli/comments/',
'https://www.reddit.com/r/opencli/comments/1abc23/title_slug/okf3s7u/evil',
'not/a/post',
]) {
expect(() => normalizeRedditPostId(bad)).toThrow(ArgumentError);
}
});
});
describe('parseExpandRounds', () => {
it('returns the default for absent input but throws on out-of-range / non-integer', () => {
expect(parseExpandRounds(undefined)).toBe(2);
expect(parseExpandRounds(null)).toBe(2);
expect(parseExpandRounds('')).toBe(2);
expect(parseExpandRounds(1)).toBe(1);
expect(parseExpandRounds(5)).toBe(5);
for (const bad of [0, -1, 6, 1.5, NaN, 'abc']) {
expect(() => parseExpandRounds(bad)).toThrow(ArgumentError);
}
});
});
it('rejects a bad --expand-rounds BEFORE navigating', async () => {
const page = makePage({ kind: 'ok', rows: [] });
await expect(command.func(page, { 'post-id': 'abc123', 'expand-rounds': 99 }))
.rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('rejects a bad post identity BEFORE navigating', async () => {
const page = makePage({ kind: 'ok', rows: [] });
await expect(command.func(page, { 'post-id': 'https://evil.test/r/x/comments/abc/title/' }))
.rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('returns rows when the evaluate script reports kind=ok', async () => {
const page = makePage({
kind: 'ok',
rows: [
{ type: 'POST', author: 'alice', score: 10, text: 'Title' },
{ type: 'L0', author: 'bob', score: 5, text: 'Comment' },
]),
};
],
expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] },
});
const result = await command.func(page, { 'post-id': 'abc123', limit: 5 });
expect(page.goto).toHaveBeenCalledWith('https://www.reddit.com');
expect(result).toEqual([
@@ -22,11 +165,171 @@ describe('reddit read adapter', () => {
{ type: 'L0', author: 'bob', score: 5, text: 'Comment' },
]);
});
it('surfaces adapter-level API errors clearly', async () => {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn().mockResolvedValue({ error: 'Reddit API returned HTTP 403' }),
};
await expect(command.func(page, { 'post-id': 'abc123' })).rejects.toThrow('Reddit API returned HTTP 403');
it('maps the five failure kinds to the right typed errors', async () => {
await expect(command.func(makePage({ kind: 'inaccessible', detail: 'post 403' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(EmptyResultError);
await expect(command.func(makePage({ kind: 'auth', detail: 'morechildren 401' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(AuthRequiredError);
await expect(command.func(makePage({ kind: 'http', httpStatus: 503, where: '/comments/abc.json' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(CommandExecutionError);
await expect(command.func(makePage({ kind: 'malformed', detail: 'no comment listing' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(CommandExecutionError);
await expect(command.func(makePage({ kind: 'parser-drift', detail: 'walker drift' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(CommandExecutionError);
await expect(command.func(makePage({ kind: 'expand-failed', detail: 'morechildren errors' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(CommandExecutionError);
});
it('throws CommandExecutionError on an unknown envelope shape (no kind)', async () => {
await expect(command.func(makePage({ random: 'stuff' }), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(CommandExecutionError);
await expect(command.func(makePage(null), { 'post-id': 'abc123' }))
.rejects.toBeInstanceOf(CommandExecutionError);
});
it('embeds expandMore=false by default and inlines flags into the evaluate script', async () => {
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] } });
await command.func(page, { 'post-id': 'xyz', sort: 'top', limit: 3 });
const script = page.evaluate.mock.calls[0][0];
expect(script).toContain('var expandMore = false');
expect(script).toContain('var expandRounds = 2');
expect(script).toContain('var sort = "top"');
expect(script).toContain('var limit = 3');
expect(script).toContain('var postId = "xyz"');
});
it('embeds expandMore=true and the requested expandRounds when --expand-more is on', async () => {
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 3, fetched: 12, capped: true, errors: [] } });
await command.func(page, { 'post-id': 'xyz', 'expand-more': true, 'expand-rounds': 3 });
const script = page.evaluate.mock.calls[0][0];
expect(script).toContain('var expandMore = true');
expect(script).toContain('var expandRounds = 3');
// The /api/morechildren request body construction must be present in
// the evaluate script (round-trips the link_id + children CSV).
expect(script).toContain("'/api/morechildren'");
expect(script).toContain("'api_type=json'");
expect(script).toContain("encodeURIComponent(linkFullname)");
expect(script).toContain("encodeURIComponent(batch.join(','))");
});
it('normalizes a full reddit URL before building the browser script', async () => {
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] } });
await command.func(page, { 'post-id': 'https://www.reddit.com/r/python/comments/1abc23/title_slug/' });
const script = page.evaluate.mock.calls[0][0];
expect(script).toContain('var postId = "1abc23"');
expect(script).not.toContain('postIdRaw.match');
});
it('expands morechildren in the original tree position instead of appending to the parent', async () => {
const fetchMock = vi.fn(async (url) => {
if (String(url).startsWith('/comments/')) {
return jsonResponse(redditPostEnvelope([
commentThing('a', 'A'),
moreThing('more_top', ['b', 'c']),
commentThing('d', 'D'),
]));
}
if (String(url) === '/api/morechildren') {
return jsonResponse({
json: {
errors: [],
data: { things: [commentThing('b', 'B'), commentThing('c', 'C')] },
},
});
}
throw new Error(`unexpected URL ${url}`);
});
const page = makeRuntimePage(fetchMock);
const result = await command.func(page, {
'post-id': 'abc123',
'expand-more': true,
limit: 10,
replies: 10,
});
expect(result.map((row) => row.author)).toEqual(['op', 'a', 'b', 'c', 'd']);
expect(fetchMock).toHaveBeenCalledWith(
'/api/morechildren',
expect.objectContaining({
method: 'POST',
body: expect.stringContaining('link_id=t3_abc123'),
}),
);
});
it('fails expand-more when Reddit returns a child that cannot be placed in the requested tree', async () => {
const fetchMock = vi.fn(async (url) => {
if (String(url).startsWith('/comments/')) {
return jsonResponse(redditPostEnvelope([moreThing('more_top', ['b'])]));
}
if (String(url) === '/api/morechildren') {
return jsonResponse({
json: {
errors: [],
data: { things: [commentThing('b', 'B', 't3_other')] },
},
});
}
throw new Error(`unexpected URL ${url}`);
});
const page = makeRuntimePage(fetchMock);
await expect(command.func(page, {
'post-id': 'abc123',
'expand-more': true,
})).rejects.toBeInstanceOf(CommandExecutionError);
});
it('fails expand-more when Reddit omits a requested child instead of silently dropping the stub', async () => {
const fetchMock = vi.fn(async (url) => {
if (String(url).startsWith('/comments/')) {
return jsonResponse(redditPostEnvelope([moreThing('more_top', ['b', 'c'])]));
}
if (String(url) === '/api/morechildren') {
return jsonResponse({
json: {
errors: [],
data: { things: [commentThing('b', 'B')] },
},
});
}
throw new Error(`unexpected URL ${url}`);
});
const page = makeRuntimePage(fetchMock);
await expect(command.func(page, {
'post-id': 'abc123',
'expand-more': true,
})).rejects.toBeInstanceOf(CommandExecutionError);
});
it('uses 5-kind discriminated union keys that DO NOT collide with declared columns', () => {
// Read the evaluate template once to assert the intermediate keys we
// return on the browser side never name any of `type` / `author` /
// `score` / `text` (the declared columns) — that pattern would
// trigger the silent-column-drop audit.
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] } });
return command.func(page, { 'post-id': 'xyz' }).then(() => {
const script = page.evaluate.mock.calls[0][0];
// Each return shape uses kind / detail / httpStatus / where /
// rows / expandMeta. None overlap with the four declared
// columns. The walker IS allowed to push column-shaped row
// objects into `rows` — that's the final shape, not an
// intermediate one.
expect(script).toContain("kind: 'inaccessible'");
expect(script).toContain("kind: 'auth'");
expect(script).toContain("kind: 'http'");
expect(script).toContain("kind: 'malformed'");
expect(script).toContain("kind: 'parser-drift'");
expect(script).toContain("kind: 'expand-failed'");
expect(script).toContain("kind: 'ok'");
});
});
});
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'post-id', type: 'string', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
{ name: 'undo', type: 'boolean', default: false, help: 'Unsave instead of save' },
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 15 },
],
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'query', type: 'string', required: true, positional: true, help: 'Reddit search query' },
{
-1
View File
@@ -32,7 +32,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'name', type: 'string', required: true, positional: true, help: 'Subreddit name (no `r/` prefix needed)' },
],
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'name', type: 'string', required: true, positional: true, help: 'Subreddit name (no `r/` prefix; e.g. `python`)' },
{
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'subreddit', type: 'string', required: true, positional: true, help: 'Subreddit name (e.g. python)' },
{ name: 'undo', type: 'boolean', default: false, help: 'Unsubscribe instead of subscribe' },
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'post-id', type: 'string', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
{ name: 'direction', type: 'string', default: 'up', help: 'Vote direction: up, down, none' },
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 15 },
],
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'username', type: 'string', required: true, positional: true, help: 'Reddit username (no `u/` prefix needed)' },
{ name: 'limit', type: 'int', default: 15 },
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'username', type: 'string', required: true, positional: true, help: 'Reddit username (no `u/` prefix needed)' },
{ name: 'limit', type: 'int', default: 15 },
-1
View File
@@ -7,7 +7,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'username', type: 'string', required: true, positional: true, help: 'Reddit username (no `u/` prefix needed)' },
],
-1
View File
@@ -9,7 +9,6 @@ cli({
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [],
columns: ['field', 'value'],
func: async (page) => {
+65
View File
@@ -31,6 +31,14 @@ function createPageMock(evaluateResult) {
getCookies: vi.fn().mockResolvedValue([{ name: 'sid', value: 'secret', domain: 'www.rednote.com' }]),
};
}
function createSearchPageMock(evaluateResults) {
const page = createPageMock(undefined);
page.evaluate = vi.fn();
for (const result of evaluateResults) {
page.evaluate.mockResolvedValueOnce(result);
}
return page;
}
describe('rednote note URL identity', () => {
const download = getRegistry().get('rednote/download');
@@ -130,6 +138,63 @@ describe('rednote argument validation', () => {
});
});
describe('rednote search browser-bridge envelopes', () => {
const search = getRegistry().get('rednote/search');
it('unwraps login-wall wait result envelopes before auth handling', async () => {
const page = createSearchPageMock([
{ session: 'site:rednote', data: 'login_wall' },
]);
await expect(search.func(page, { query: 'tesla', limit: 5 })).rejects.toMatchObject({
code: 'AUTH_REQUIRED',
message: expect.stringContaining('blocked behind a login wall'),
});
expect(page.evaluate).toHaveBeenCalledTimes(1);
});
it('unwraps search extraction envelopes and preserves rednote row shape', async () => {
const url = 'https://www.rednote.com/search_result/68e90be80000000004022e66?xsec_token=test-token';
const page = createSearchPageMock([
'content',
1,
{
session: 'site:rednote',
data: [{
title: 'rednote result',
author: 'author',
likes: '12',
url,
author_url: 'https://www.rednote.com/user/profile/u1',
}],
},
]);
await expect(search.func(page, { query: 'tesla', limit: 1 })).resolves.toEqual([{
rank: 1,
title: 'rednote result',
author: 'author',
likes: '12',
published_at: '2025-10-10',
url,
author_url: 'https://www.rednote.com/user/profile/u1',
}]);
});
it('fails typed instead of silently returning [] for malformed extraction payloads', async () => {
const page = createSearchPageMock([
'content',
1,
{ session: 'site:rednote', data: { rows: [] } },
]);
await expect(search.func(page, { query: 'tesla', limit: 1 })).rejects.toMatchObject({
code: 'COMMAND_EXEC',
message: expect.stringContaining('payload shape'),
});
});
});
describe('rednote Pinia store failures', () => {
it('maps feed store read failure to CommandExecutionError', async () => {
const command = getRegistry().get('rednote/feed');
+11 -5
View File
@@ -6,8 +6,8 @@
* 1:1 comparison between the two frontends.
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
import { buildScrollUntilJs, buildSearchExtractJs, noteIdToDate } from '../xiaohongshu/search.js';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { buildScrollUntilJs, buildSearchExtractJs, noteIdToDate, unwrapEvaluateResult } from '../xiaohongshu/search.js';
function parseLimit(raw) {
const parsed = Number(raw);
@@ -19,6 +19,13 @@ function parseLimit(raw) {
}
return parsed;
}
function requireSearchRows(payload) {
const rows = unwrapEvaluateResult(payload);
if (!Array.isArray(rows)) {
throw new CommandExecutionError('Unexpected Rednote search extraction payload shape; expected an array of rows.');
}
return rows;
}
/**
* Wait for search results or login wall using MutationObserver (max 5s).
@@ -78,7 +85,7 @@ cli({
const limit = parseLimit(kwargs.limit ?? 20);
const keyword = encodeURIComponent(kwargs.query);
await page.goto(`https://www.rednote.com/search_result?keyword=${keyword}&source=web_search_result_notes`);
const waitResult = await page.evaluate(WAIT_FOR_CONTENT_JS);
const waitResult = unwrapEvaluateResult(await page.evaluate(WAIT_FOR_CONTENT_JS));
if (waitResult === 'login_wall') {
throw new AuthRequiredError('www.rednote.com', 'Rednote search results are blocked behind a login wall');
}
@@ -87,8 +94,7 @@ cli({
// `autoScroll({ times: 2 })` capped extraction at ~13 notes regardless
// of `--limit`.
await page.evaluate(buildScrollUntilJs(limit));
const payload = await page.evaluate(buildSearchExtractJs('www.rednote.com'));
const data = Array.isArray(payload) ? payload : [];
const data = requireSearchRows(await page.evaluate(buildSearchExtractJs('www.rednote.com')));
return data
.filter((item) => item.title)
.slice(0, limit)
-1
View File
@@ -11,7 +11,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'tweet-id', type: 'string', positional: true, required: true, help: 'Tweet ID or URL containing the article' },
],
+8 -5
View File
@@ -1,7 +1,7 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
import { resolveTwitterQueryId } from './shared.js';
import { extractMedia, resolveTwitterQueryId } from './shared.js';
// Companion to bookmark-folders.js: reads tweets inside a single folder.
// X exposes folder contents through a separate timeline operation
@@ -11,6 +11,7 @@ import { resolveTwitterQueryId } from './shared.js';
const OPERATION_NAME = 'BookmarkFolderTimeline';
const FALLBACK_QUERY_ID = '13H7EUATwethsj_jZ6QQAQ';
const FOLDER_ID_PATTERN = /^[A-Za-z0-9_-]+$/;
const MAX_PAGINATION_PAGES = 100;
const FEATURES = {
rweb_video_screen_enabled: false,
@@ -53,7 +54,7 @@ function buildFolderTimelineUrl(queryId, folderId, count, cursor) {
+ `&features=${encodeURIComponent(JSON.stringify(FEATURES))}`;
}
function extractFolderTweet(result, seen) {
export function extractFolderTweet(result, seen) {
if (!result) return null;
const tw = result.tweet || result;
const legacy = tw.legacy || {};
@@ -71,6 +72,7 @@ function extractFolderTweet(result, seen) {
bookmarks: legacy.bookmark_count || 0,
created_at: legacy.created_at || '',
url: screenName ? `https://x.com/${screenName}/status/${tw.rest_id}` : `https://x.com/i/status/${tw.rest_id}`,
...extractMedia(legacy),
};
}
@@ -122,13 +124,12 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'folder-id', positional: true, type: 'string', required: true, help: 'Folder id from `opencli twitter bookmark-folders`.' },
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of bookmarks to return (default 20).' },
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the folder by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the API\'s native (saved-time) ordering.' },
],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url'],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url', 'has_media', 'media_urls'],
func: async (page, kwargs) => {
const folderId = String(kwargs['folder-id'] || '').trim();
if (!folderId || !FOLDER_ID_PATTERN.test(folderId)) {
@@ -158,7 +159,8 @@ cli({
const allTweets = [];
const seen = new Set();
let cursor = null;
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
const fetchCount = Math.min(100, limit - allTweets.length + 10);
const apiUrl = buildFolderTimelineUrl(queryId, folderId, fetchCount, cursor);
const data = await page.evaluate(`async () => {
@@ -182,6 +184,7 @@ cli({
export const __test__ = {
parseBookmarkFolderTimeline,
extractFolderTweet,
buildFolderTimelineUrl,
FOLDER_ID_PATTERN,
};
+59 -1
View File
@@ -2,7 +2,7 @@ import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { __test__ } from './bookmark-folder.js';
const { parseBookmarkFolderTimeline, buildFolderTimelineUrl, FOLDER_ID_PATTERN } = __test__;
const { parseBookmarkFolderTimeline, extractFolderTweet, buildFolderTimelineUrl, FOLDER_ID_PATTERN } = __test__;
describe('twitter bookmark-folder URL builder', () => {
it('embeds the folder id and count in the variables payload', () => {
@@ -97,6 +97,8 @@ describe('twitter bookmark-folder timeline parser', () => {
bookmarks: 3,
created_at: 'Tue Mar 17 09:00:00 +0000 2026',
url: 'https://x.com/alice/status/1',
has_media: false,
media_urls: [],
},
]);
expect(nextCursor).toBe('NEXT_CURSOR');
@@ -247,6 +249,62 @@ describe('twitter bookmark-folder timeline parser', () => {
it('returns empty array + null cursor for unknown envelope', () => {
expect(parseBookmarkFolderTimeline({}, new Set())).toEqual({ tweets: [], nextCursor: null });
});
it('includes photo media URLs from extended_entities', () => {
const tweet = extractFolderTweet({
rest_id: '101',
legacy: {
full_text: 'pic folder tweet',
extended_entities: {
media: [
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/abc.jpg' },
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/def.jpg' },
],
},
},
core: { user_results: { result: { legacy: { screen_name: 'eve' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls).toEqual([
'https://pbs.twimg.com/media/abc.jpg',
'https://pbs.twimg.com/media/def.jpg',
]);
});
it('extracts mp4 variant URL for video media', () => {
const tweet = extractFolderTweet({
rest_id: '102',
legacy: {
full_text: 'video folder tweet',
extended_entities: {
media: [{
type: 'video',
media_url_https: 'https://pbs.twimg.com/amplify_video_thumb/thumb.jpg',
video_info: {
variants: [
{ content_type: 'application/x-mpegURL', url: 'https://video.twimg.com/playlist.m3u8' },
{ content_type: 'video/mp4', bitrate: 832000, url: 'https://video.twimg.com/low.mp4' },
{ content_type: 'video/mp4', bitrate: 2176000, url: 'https://video.twimg.com/high.mp4' },
],
},
}],
},
},
core: { user_results: { result: { legacy: { screen_name: 'frank' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls?.[0]).toMatch(/\.mp4$/);
});
it('returns has_media false / media_urls empty when no media present', () => {
const tweet = extractFolderTweet({
rest_id: '103',
legacy: { full_text: 'text only', favorite_count: 0, retweet_count: 0, bookmark_count: 0 },
core: { user_results: { result: { legacy: { screen_name: 'gail' } } } },
}, new Set());
expect(tweet?.has_media).toBe(false);
expect(tweet?.media_urls).toEqual([]);
});
});
describe('twitter bookmark-folder id validation', () => {
-1
View File
@@ -77,7 +77,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [],
columns: ['id', 'name', 'items', 'created_at'],
func: async (page) => {
+12 -5
View File
@@ -1,7 +1,9 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { extractMedia } from './shared.js';
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
const BOOKMARKS_QUERY_ID = 'Fy0QMy4q_aZCpkO0PnyLYw';
const MAX_PAGINATION_PAGES = 100;
const FEATURES = {
rweb_video_screen_enabled: false,
profile_label_improvements_pcf_label_in_post_enabled: true,
@@ -41,7 +43,7 @@ function buildBookmarksUrl(count, cursor) {
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
+ `&features=${encodeURIComponent(JSON.stringify(FEATURES))}`;
}
function extractBookmarkTweet(result, seen) {
export function extractBookmarkTweet(result, seen) {
if (!result)
return null;
const tw = result.tweet || result;
@@ -63,9 +65,10 @@ function extractBookmarkTweet(result, seen) {
bookmarks: legacy.bookmark_count || 0,
created_at: legacy.created_at || '',
url: `https://x.com/${screenName}/status/${tw.rest_id}`,
...extractMedia(legacy),
};
}
function parseBookmarks(data, seen) {
export function parseBookmarks(data, seen) {
const tweets = [];
let nextCursor = null;
const instructions = data?.data?.bookmark_timeline_v2?.timeline?.instructions
@@ -105,12 +108,11 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of bookmarks to return (default 20).' },
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the bookmarks by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the API\'s native (saved-time) ordering.' },
],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url'],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url', 'has_media', 'media_urls'],
func: async (page, kwargs) => {
const limit = kwargs.limit || 20;
const cookies = await page.getCookies({ url: 'https://x.com' });
@@ -150,7 +152,8 @@ cli({
const allTweets = [];
const seen = new Set();
let cursor = null;
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
const fetchCount = Math.min(100, limit - allTweets.length + 10);
const apiUrl = buildBookmarksUrl(fetchCount, cursor).replace(BOOKMARKS_QUERY_ID, queryId);
const data = await page.evaluate(`async () => {
@@ -172,3 +175,7 @@ cli({
return applyTopByEngagement(trimmed, kwargs['top-by-engagement']);
},
});
export const __test__ = {
parseBookmarks,
extractBookmarkTweet,
};
+205
View File
@@ -0,0 +1,205 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './bookmarks.js';
const { parseBookmarks, extractBookmarkTweet } = __test__;
describe('twitter bookmarks parser', () => {
it('extracts a baseline tweet with no media (has_media false, media_urls empty)', () => {
const tweet = extractBookmarkTweet({
rest_id: '1',
legacy: {
full_text: 'plain bookmark',
favorite_count: 5,
retweet_count: 1,
bookmark_count: 2,
created_at: 'Wed Apr 16 10:00:00 +0000 2026',
},
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
}, new Set());
expect(tweet).toEqual({
id: '1',
author: 'alice',
name: 'Alice',
text: 'plain bookmark',
likes: 5,
retweets: 1,
bookmarks: 2,
created_at: 'Wed Apr 16 10:00:00 +0000 2026',
url: 'https://x.com/alice/status/1',
has_media: false,
media_urls: [],
});
});
it('includes photo media URLs from extended_entities', () => {
const tweet = extractBookmarkTweet({
rest_id: '101',
legacy: {
full_text: 'pic bookmark',
extended_entities: {
media: [
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/abc.jpg' },
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/def.jpg' },
],
},
},
core: { user_results: { result: { legacy: { screen_name: 'bob' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls).toEqual([
'https://pbs.twimg.com/media/abc.jpg',
'https://pbs.twimg.com/media/def.jpg',
]);
});
it('extracts mp4 variant URL for video media', () => {
const tweet = extractBookmarkTweet({
rest_id: '102',
legacy: {
full_text: 'video bookmark',
extended_entities: {
media: [{
type: 'video',
media_url_https: 'https://pbs.twimg.com/amplify_video_thumb/thumb.jpg',
video_info: {
variants: [
{ content_type: 'application/x-mpegURL', url: 'https://video.twimg.com/playlist.m3u8' },
{ content_type: 'video/mp4', bitrate: 832000, url: 'https://video.twimg.com/low.mp4' },
{ content_type: 'video/mp4', bitrate: 2176000, url: 'https://video.twimg.com/high.mp4' },
],
},
}],
},
},
core: { user_results: { result: { legacy: { screen_name: 'carol' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls?.[0]).toMatch(/\.mp4$/);
});
it('falls back to entities.media when extended_entities is absent', () => {
const tweet = extractBookmarkTweet({
rest_id: '103',
legacy: {
full_text: 'entities-only media',
entities: {
media: [{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/legacy.jpg' }],
},
},
core: { user_results: { result: { legacy: { screen_name: 'dave' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls).toEqual(['https://pbs.twimg.com/media/legacy.jpg']);
});
it('prefers note_tweet text over truncated full_text', () => {
const tweet = extractBookmarkTweet({
rest_id: '2',
legacy: { full_text: 'short text…', favorite_count: 0, retweet_count: 0, bookmark_count: 0 },
note_tweet: { note_tweet_results: { result: { text: 'full long-form text body' } } },
core: { user_results: { result: { core: { screen_name: 'erin' } } } },
}, new Set());
expect(tweet?.text).toBe('full long-form text body');
});
it('deduplicates tweets across the seen Set', () => {
const data = {
data: {
bookmark_timeline_v2: {
timeline: {
instructions: [{
entries: [
{
entryId: 'tweet-3',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '3',
legacy: { full_text: 'first', favorite_count: 0, retweet_count: 0, bookmark_count: 0 },
core: { user_results: { result: { legacy: { screen_name: 'frank' } } } },
},
},
},
},
},
{
entryId: 'tweet-3-dup',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '3',
legacy: { full_text: 'duplicate' },
core: { user_results: { result: { legacy: { screen_name: 'frank' } } } },
},
},
},
},
},
],
}],
},
},
},
};
const seen = new Set();
const { tweets } = parseBookmarks(data, seen);
expect(tweets).toHaveLength(1);
expect(tweets[0].text).toBe('first');
});
it('extracts cursor + tweets from the bookmark_timeline_v2 envelope', () => {
const data = {
data: {
bookmark_timeline_v2: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'tweet-4',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '4',
legacy: {
full_text: 'envelope tweet',
favorite_count: 1,
retweet_count: 0,
bookmark_count: 0,
extended_entities: {
media: [{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/x.jpg' }],
},
},
core: { user_results: { result: { legacy: { screen_name: 'gina' } } } },
},
},
},
},
},
{
entryId: 'cursor-bottom-Y',
content: { __typename: 'TimelineTimelineCursor', cursorType: 'Bottom', value: 'NEXT' },
},
],
},
],
},
},
},
};
const { tweets, nextCursor } = parseBookmarks(data, new Set());
expect(tweets).toHaveLength(1);
expect(tweets[0].id).toBe('4');
expect(tweets[0].has_media).toBe(true);
expect(tweets[0].media_urls).toEqual(['https://pbs.twimg.com/media/x.jpg']);
expect(nextCursor).toBe('NEXT');
});
it('returns empty tweets + null cursor for unknown envelope', () => {
expect(parseBookmarks({}, new Set())).toEqual({ tweets: [], nextCursor: null });
});
});
-1
View File
@@ -15,7 +15,6 @@ cli({
description: 'Download Twitter/X media (images and videos). Provide either <username> to scan a profile\'s media tab, or --tweet-url to download a single tweet.',
domain: 'x.com',
strategy: Strategy.COOKIE,
siteSession: 'persistent',
args: [
{ name: 'username', positional: true, help: 'Twitter username (with or without @) to scan their /media tab. Either <username> or --tweet-url is required.' },
{ name: 'tweet-url', help: 'Single tweet URL to download. Use this OR <username>, not both required at once.' },
+20 -6
View File
@@ -1,5 +1,6 @@
import { ArgumentError, AuthRequiredError, selectorError, EmptyResultError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { normalizeTwitterScreenName, unwrapBrowserResult } from './shared.js';
/**
* Extract follower rows from Twitter/X follower-list SPA cells.
@@ -72,7 +73,7 @@ async function extractFollowersFromDOM(page) {
}
function normalizeScreenName(value) {
return String(value ?? '').trim().replace(/^\/+/, '').replace(/^@+/, '');
return normalizeTwitterScreenName(value);
}
cli({
@@ -83,7 +84,6 @@ cli({
domain: 'x.com',
strategy: Strategy.UI,
browser: true,
siteSession: 'persistent',
args: [
{
name: 'user',
@@ -103,18 +103,27 @@ cli({
throw new ArgumentError('limit must be a positive integer');
}
let targetUser = normalizeScreenName(kwargs.user);
const rawUser = String(kwargs.user ?? '').trim();
let targetUser = normalizeScreenName(rawUser);
if (rawUser && !targetUser) {
throw new ArgumentError('twitter followers user must be a valid Twitter/X handle', 'Example: opencli twitter followers @elonmusk --limit 100');
}
if (!targetUser) {
await page.goto('https://x.com/home');
await page.wait({ selector: '[data-testid="primaryColumn"]' });
const href = await page.evaluate(`() => {
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
// unwrap so the href string is usable downstream.
const href = unwrapBrowserResult(await page.evaluate(`() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
}`);
if (!href) {
}`));
if (!href || typeof href !== 'string') {
throw new AuthRequiredError('x.com', 'Could not find logged-in user profile link. Are you logged in?');
}
targetUser = normalizeScreenName(href);
if (!targetUser) {
throw new AuthRequiredError('x.com', 'Could not find logged-in user profile link. Are you logged in?');
}
}
if (!targetUser) {
throw new ArgumentError('twitter followers user cannot be empty', 'Example: opencli twitter followers @elonmusk --limit 100');
@@ -173,3 +182,8 @@ cli({
return allFollowers.slice(0, limit);
}
});
export const __test__ = {
extractFollowersFromDOM,
normalizeScreenName,
};
+44
View File
@@ -0,0 +1,44 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
import { __test__ } from './followers.js';
describe('twitter followers command', () => {
it('normalizes exact profile handles and rejects route-like hrefs', () => {
expect(__test__.normalizeScreenName('@viewer')).toBe('viewer');
expect(__test__.normalizeScreenName('/viewer')).toBe('viewer');
expect(__test__.normalizeScreenName('https://x.com/viewer')).toBe('viewer');
expect(__test__.normalizeScreenName('/home')).toBe('');
expect(__test__.normalizeScreenName('/viewer/extra')).toBe('');
});
it('rejects invalid explicit users before navigation', async () => {
const command = getRegistry().get('twitter/followers');
const page = {
goto: vi.fn(),
wait: vi.fn(),
evaluate: vi.fn(),
};
await expect(command.func(page, { user: 'viewer/extra', limit: 10 })).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
expect(page.wait).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('rejects non-profile AppTabBar hrefs instead of navigating to route followers', async () => {
const command = getRegistry().get('twitter/followers');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn(async (script) => {
if (String(script).includes('AppTabBar_Profile_Link')) return '/home';
throw new Error(`Unexpected evaluate: ${String(script).slice(0, 80)}`);
}),
};
await expect(command.func(page, { limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
expect(page.goto).not.toHaveBeenCalledWith('https://x.com/home/followers');
});
});
+32 -16
View File
@@ -1,10 +1,11 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { resolveTwitterQueryId, sanitizeQueryId } from './shared.js';
import { normalizeTwitterScreenName, resolveTwitterQueryId, sanitizeQueryId, unwrapBrowserResult } from './shared.js';
import { TWITTER_BEARER_TOKEN } from './utils.js';
const FOLLOWING_QUERY_ID = 'zx6e-TLzRkeDO_a7p4b3JQ'; // Following fallback
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
const MAX_PAGINATION_PAGES = 100;
const FEATURES = {
rweb_video_screen_enabled: false,
@@ -128,7 +129,7 @@ function parseFollowing(data) {
}
function normalizeScreenName(value) {
return String(value || '').trim().replace(/^\/+/, '').replace(/^@+/, '');
return normalizeTwitterScreenName(value);
}
cli({
@@ -139,7 +140,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{
name: 'user',
@@ -156,7 +156,11 @@ cli({
if (!Number.isInteger(limit) || limit <= 0) {
throw new ArgumentError('twitter following --limit must be a positive integer', 'Example: opencli twitter following @elonmusk --limit 200');
}
let targetUser = normalizeScreenName(kwargs.user);
const rawUser = String(kwargs.user ?? '').trim();
let targetUser = normalizeScreenName(rawUser);
if (rawUser && !targetUser) {
throw new ArgumentError('twitter following user must be a valid Twitter/X handle', 'Example: opencli twitter following @elonmusk --limit 200');
}
const cookies = await page.getCookies({ url: 'https://x.com' });
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
@@ -164,13 +168,25 @@ cli({
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
if (!targetUser) {
const href = await page.evaluate(() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
});
if (!href)
// Force a navigation to the home surface so the AppTabBar sidebar
// is rendered; the framework pre-nav lands on bare x.com which
// does not always expose AppTabBar_Profile_Link.
await page.goto('https://x.com/home');
await page.wait({ selector: '[data-testid="primaryColumn"]' });
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
// unwrap so the href string is usable downstream.
// NOTE: the function-literal form `() => ...` silently drops
// primitive return values through the bridge — only the template
// string form preserves the `data` field.
const href = unwrapBrowserResult(await page.evaluate(`() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
}`));
if (!href || typeof href !== 'string')
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
targetUser = normalizeScreenName(href);
if (!targetUser)
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
targetUser = normalizeScreenName(href.replace('/', ''));
}
if (!targetUser) {
throw new ArgumentError('twitter following user cannot be empty', 'Example: opencli twitter following @elonmusk --limit 200');
@@ -186,12 +202,12 @@ cli({
};
// Get userId from screen_name
const userLookup = await page.evaluate(async (url, headers) => {
const userLookup = unwrapBrowserResult(await page.evaluate(async (url, headers) => {
const resp = await fetch(url, { headers, credentials: 'include' });
if (!resp.ok) return { error: resp.status };
const d = await resp.json();
return { userId: d.data?.user?.result?.rest_id || null };
}, buildUserByScreenNameUrl(userByScreenNameQueryId, targetUser), headers);
}, buildUserByScreenNameUrl(userByScreenNameQueryId, targetUser), headers));
if (userLookup?.error === 401 || userLookup?.error === 403) {
throw new AuthRequiredError('x.com', `Twitter user lookup failed (HTTP ${userLookup.error})`);
}
@@ -206,14 +222,14 @@ cli({
const seen = new Set();
let cursor = null;
const maxPages = Math.ceil(limit / 50) + 2;
for (let i = 0; i < maxPages && allUsers.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && allUsers.length < limit; i++) {
const fetchCount = Math.min(50, limit - allUsers.length + 10);
const apiUrl = buildFollowingUrl(followingQueryId, userId, fetchCount, cursor);
const data = await page.evaluate(async (url, headers) => {
const data = unwrapBrowserResult(await page.evaluate(async (url, headers) => {
const r = await fetch(url, { headers, credentials: 'include' });
return r.ok ? await r.json() : { error: r.status };
}, apiUrl, headers);
}, apiUrl, headers));
if (data?.error) {
if (data.error === 401 || data.error === 403)
throw new AuthRequiredError('x.com', `Twitter following request failed (HTTP ${data.error})`);
+51 -7
View File
@@ -157,6 +157,8 @@ describe('twitter following helpers', () => {
expect(__test__.normalizeScreenName('@elonmusk')).toBe('elonmusk');
expect(__test__.normalizeScreenName('/elonmusk')).toBe('elonmusk');
expect(__test__.normalizeScreenName(' @@alice ')).toBe('alice');
expect(__test__.normalizeScreenName('/home')).toBe('');
expect(__test__.normalizeScreenName('/elonmusk/extra')).toBe('');
});
});
@@ -201,23 +203,28 @@ function followingPayload(users, cursor) {
};
}
function createFollowingPage(followingResponses, { ct0 = 'token', userLookup = { userId: '42' } } = {}) {
function bridgeEnvelope(data) {
return { session: 'site:twitter', data };
}
function createFollowingPage(followingResponses, { ct0 = 'token', userLookup = { userId: '42' }, envelope = false } = {}) {
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => (ct0 ? [{ name: 'ct0', value: ct0 }] : [])),
evaluate: vi.fn(async (script, ...args) => {
const wrap = (value) => envelope ? bridgeEnvelope(value) : value;
if (typeof script === 'function') {
const haystack = [script.toString(), ...args.map((arg) => String(arg))].join('\n');
if (haystack.includes('/UserByScreenName')) return userLookup;
if (haystack.includes('/Following')) return followingResponses.shift() || followingPayload([], null);
if (haystack.includes('AppTabBar_Profile_Link')) return '/viewer';
if (haystack.includes('/UserByScreenName')) return wrap(userLookup);
if (haystack.includes('/Following')) return wrap(followingResponses.shift() || followingPayload([], null));
if (haystack.includes('AppTabBar_Profile_Link')) return wrap('/viewer');
throw new Error(`Unexpected evaluate function: ${haystack.slice(0, 80)}`);
}
if (script.includes('operationName')) return null;
if (script.includes('/UserByScreenName')) return userLookup;
if (script.includes('/Following')) return followingResponses.shift() || followingPayload([], null);
if (script.includes('AppTabBar_Profile_Link')) return '/viewer';
if (script.includes('/UserByScreenName')) return wrap(userLookup);
if (script.includes('/Following')) return wrap(followingResponses.shift() || followingPayload([], null));
if (script.includes('AppTabBar_Profile_Link')) return wrap('/viewer');
throw new Error(`Unexpected evaluate script: ${script.slice(0, 80)}`);
}),
};
@@ -253,6 +260,29 @@ describe('twitter following command', () => {
expect(page.goto).not.toHaveBeenCalled();
});
it('rejects invalid explicit users before cookies or navigation', async () => {
const command = getRegistry().get('twitter/following');
const page = createFollowingPage([]);
await expect(command.func(page, { user: 'elonmusk/extra', limit: 10 })).rejects.toBeInstanceOf(ArgumentError);
expect(page.getCookies).not.toHaveBeenCalled();
expect(page.goto).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('rejects route-like AppTabBar hrefs as AuthRequiredError', async () => {
const command = getRegistry().get('twitter/following');
const page = createFollowingPage([]);
page.evaluate.mockImplementation(async (script, ...args) => {
const haystack = [typeof script === 'function' ? script.toString() : String(script), ...args.map((arg) => String(arg))].join('\n');
if (haystack.includes('AppTabBar_Profile_Link')) return '/home';
throw new Error(`Unexpected evaluate: ${haystack.slice(0, 80)}`);
});
await expect(command.func(page, { limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
});
it('maps first-page auth failures to AuthRequiredError', async () => {
const command = getRegistry().get('twitter/following');
const page = createFollowingPage([{ error: 401 }]);
@@ -277,6 +307,20 @@ describe('twitter following command', () => {
await expect(command.func(page, { user: 'elonmusk', limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
});
it('unwraps Browser Bridge envelopes for user lookup and following payloads', async () => {
const command = getRegistry().get('twitter/following');
const page = createFollowingPage([followingPayload(['alice'], null)], { envelope: true });
const rows = await command.func(page, { user: 'elonmusk', limit: 10 });
expect(rows.map((row) => row.screen_name)).toEqual(['alice']);
const callText = (call) => call.map((part) => typeof part === 'function' ? part.toString() : String(part)).join('\n');
const followingCall = page.evaluate.mock.calls.find((call) => callText(call).includes('/Following')) || [];
const followingUrl = String(followingCall[1] || '');
expect(decodeURIComponent(followingUrl)).toContain('"userId":"42"');
expect(decodeURIComponent(followingUrl)).not.toContain('[object Object]');
});
it('fails fast when the following timeline is empty', async () => {
const command = getRegistry().get('twitter/following');
const page = createFollowingPage([followingPayload([], null)]);
+28 -14
View File
@@ -1,9 +1,10 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { resolveTwitterQueryId, sanitizeQueryId, extractMedia } from './shared.js';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { normalizeTwitterScreenName, resolveTwitterQueryId, sanitizeQueryId, extractMedia, unwrapBrowserResult } from './shared.js';
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
const LIKES_QUERY_ID = 'RozQdCp4CilQzrcuU0NY5w';
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
const MAX_PAGINATION_PAGES = 100;
const FEATURES = {
rweb_video_screen_enabled: false,
profile_label_improvements_pcf_label_in_post_enabled: true,
@@ -142,7 +143,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (with or without @). Defaults to the logged-in user when omitted.' },
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of liked tweets to return (default 20).' },
@@ -151,20 +151,33 @@ cli({
columns: ['id', 'author', 'name', 'text', 'likes', 'retweets', 'created_at', 'url', 'has_media', 'media_urls'],
func: async (page, kwargs) => {
const limit = kwargs.limit || 20;
let username = (kwargs.username || '').replace(/^@/, '');
const rawUsername = String(kwargs.username ?? '').trim();
let username = normalizeTwitterScreenName(rawUsername);
if (rawUsername && !username) {
throw new ArgumentError('twitter likes username must be a valid Twitter/X handle', 'Example: opencli twitter likes @jack --limit 20');
}
const cookies = await page.getCookies({ url: 'https://x.com' });
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
if (!ct0)
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
// If no username provided, detect the logged-in user
// If no username provided, detect the logged-in user.
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
// unwrap so the href string is usable downstream.
if (!username) {
const href = await page.evaluate(`() => {
// Force a navigation to the home surface so the AppTabBar sidebar
// is rendered; the framework pre-nav lands on bare x.com which
// does not always expose AppTabBar_Profile_Link.
await page.goto('https://x.com/home');
await page.wait({ selector: '[data-testid="primaryColumn"]' });
const href = unwrapBrowserResult(await page.evaluate(`() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
}`);
if (!href)
}`));
if (!href || typeof href !== 'string')
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
username = normalizeTwitterScreenName(href);
if (!username)
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
username = href.replace('/', '');
}
const likesQueryId = await resolveTwitterQueryId(page, 'Likes', LIKES_QUERY_ID);
const userByScreenNameQueryId = await resolveTwitterQueryId(page, 'UserByScreenName', USER_BY_SCREEN_NAME_QUERY_ID);
@@ -175,27 +188,28 @@ cli({
'X-Twitter-Active-User': 'yes',
});
// Get userId from screen_name
const userId = await page.evaluate(`async () => {
const userId = unwrapBrowserResult(await page.evaluate(`async () => {
const screenName = ${JSON.stringify(username)};
const url = ${JSON.stringify(buildUserByScreenNameUrl(userByScreenNameQueryId, username))};
const resp = await fetch(url, { headers: ${headers}, credentials: 'include' });
if (!resp.ok) return null;
const d = await resp.json();
return d.data?.user?.result?.rest_id || null;
}`);
}`));
if (!userId) {
throw new CommandExecutionError(`Could not find user @${username}`);
}
const allTweets = [];
const seen = new Set();
let cursor = null;
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
const fetchCount = Math.min(100, limit - allTweets.length + 10);
const apiUrl = buildLikesUrl(likesQueryId, userId, fetchCount, cursor);
const data = await page.evaluate(`async () => {
const data = unwrapBrowserResult(await page.evaluate(`async () => {
const r = await fetch("${apiUrl}", { headers: ${headers}, credentials: 'include' });
return r.ok ? await r.json() : { error: r.status };
}`);
}`));
if (data?.error) {
if (allTweets.length === 0)
throw new CommandExecutionError(`HTTP ${data.error}: Failed to fetch likes. queryId may have expired.`);
+111 -1
View File
@@ -1,5 +1,50 @@
import { describe, expect, it } from 'vitest';
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
import { __test__ } from './likes.js';
function likesPayload() {
return {
data: {
user: {
result: {
timeline_v2: {
timeline: {
instructions: [{
entries: [{
entryId: 'tweet-1',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '1',
legacy: {
full_text: 'liked post',
favorite_count: 7,
retweet_count: 2,
created_at: 'now',
},
core: {
user_results: {
result: {
legacy: { screen_name: 'alice', name: 'Alice' },
},
},
},
},
},
},
},
}],
}],
},
},
},
},
},
};
}
describe('twitter likes helpers', () => {
it('falls back when queryId contains unsafe characters', () => {
expect(__test__.sanitizeQueryId('safe_Query-123', 'fallback')).toBe('safe_Query-123');
@@ -83,3 +128,68 @@ describe('twitter likes helpers', () => {
});
});
});
describe('twitter likes command', () => {
it('rejects invalid explicit username before cookies or navigation', async () => {
const command = getRegistry().get('twitter/likes');
const page = {
goto: vi.fn(),
wait: vi.fn(),
getCookies: vi.fn(),
evaluate: vi.fn(),
};
await expect(command.func(page, { username: 'viewer/extra', limit: 10 })).rejects.toBeInstanceOf(ArgumentError);
expect(page.getCookies).not.toHaveBeenCalled();
expect(page.goto).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('rejects route-like AppTabBar hrefs as AuthRequiredError', async () => {
const command = getRegistry().get('twitter/likes');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
evaluate: vi.fn(async (script) => {
if (String(script).includes('AppTabBar_Profile_Link')) return '/home';
throw new Error(`Unexpected evaluate: ${String(script).slice(0, 80)}`);
}),
};
await expect(command.func(page, { limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
expect(page.evaluate).toHaveBeenCalledTimes(1);
});
it('unwraps Browser Bridge envelopes for default-self user lookup and likes payload', async () => {
const command = getRegistry().get('twitter/likes');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
evaluate: vi.fn(async (script) => {
const text = String(script);
if (text.includes('AppTabBar_Profile_Link')) {
return { session: 'site:twitter', data: '/viewer' };
}
if (text.includes('operationName')) return null;
if (text.includes('/UserByScreenName')) {
return { session: 'site:twitter', data: '42' };
}
if (text.includes('/Likes')) {
return { session: 'site:twitter', data: likesPayload() };
}
throw new Error(`Unexpected evaluate: ${text.slice(0, 80)}`);
}),
};
const rows = await command.func(page, { limit: 1 });
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({ id: '1', author: 'alice', text: 'liked post' });
const likesCall = page.evaluate.mock.calls.find(([script]) => String(script).includes('/Likes')) || [];
expect(decodeURIComponent(String(likesCall[0]))).toContain('"userId":"42"');
expect(decodeURIComponent(String(likesCall[0]))).not.toContain('[object Object]');
});
});
+128 -204
View File
@@ -1,11 +1,14 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { resolveTwitterQueryId } from './shared.js';
import { parseListsManagement } from './lists.js';
import { TWITTER_BEARER_TOKEN } from './utils.js';
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
const LISTS_MANAGEMENT_QUERY_ID = '78UbkyXwXBD98IgUWXOy9g';
// 2026-05 fallback — X rotates queryIds; resolveTwitterQueryId() does live lookup,
// this constant is just the default if live lookup fails.
const LIST_ADD_MEMBER_QUERY_ID = 'vWPi0CTMoPFsjsL6W4IynQ';
const LISTS_MANAGEMENT_FEATURES = {
rweb_video_screen_enabled: false,
@@ -62,6 +65,65 @@ function buildUserByScreenNameUrl(queryId, screenName) {
+ `&features=${encodeURIComponent(feats)}`;
}
function fatalGraphqlErrors(errors) {
const list = Array.isArray(errors) ? errors : [];
return list.filter((e) =>
!(e?.path || []).join('.').includes('default_banner_media_results')
&& !/decode/i.test(e?.message || '')
);
}
export function buildListAddMemberRow({ addResult, memberCountBefore, listId, username, userId }) {
if (!addResult?.httpOk) {
throw new CommandExecutionError(
`Failed to add @${username} to list ${listId}: HTTP ${addResult?.status ?? 0}${addResult?.fetchError ? ' (' + addResult.fetchError + ')' : ''}${addResult?.raw ? ' — ' + addResult.raw : ''}`
);
}
// X often returns a partial GraphQL error on `default_banner_media_results`
// even on successful mutations. Treat only missing main data or non-decode
// GraphQL errors as command failures.
const hasMemberCount = addResult.mc !== null && addResult.mc !== undefined;
const fatalErrors = fatalGraphqlErrors(addResult.errors);
if (!hasMemberCount && fatalErrors.length) {
const msg = fatalErrors.map((e) => e.message || JSON.stringify(e)).join('; ');
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: ${msg.slice(0, 300)}`);
}
if (!hasMemberCount) {
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: no member_count in response`);
}
const memberCountAfter = Number(addResult.mc);
if (!Number.isFinite(memberCountAfter)) {
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: invalid member_count in response`);
}
if (memberCountAfter < memberCountBefore) {
throw new CommandExecutionError(
`Failed to add @${username} to list ${listId}: member_count decreased unexpectedly (${memberCountBefore}${memberCountAfter})`
);
}
const countIncreased = memberCountAfter > memberCountBefore;
if (!countIncreased && addResult.isMember !== true) {
throw new CommandExecutionError(
`Failed to add @${username} to list ${listId}: member_count unchanged (${memberCountBefore}${memberCountAfter}) and response did not confirm membership`
);
}
const noop = !countIncreased;
const verifiedBy = `member_count ${memberCountBefore}${memberCountAfter}`;
return {
listId,
username,
userId: String(userId),
status: noop ? 'noop' : 'success',
message: noop
? `@${username} is already a member of list ${listId}`
: `Added @${username} to list ${listId} (verified via ${verifiedBy})`,
};
}
cli({
site: 'twitter',
name: 'list-add',
@@ -79,10 +141,10 @@ cli({
const listId = String(kwargs.listId || '').trim();
const username = String(kwargs.username || '').replace(/^@/, '').trim();
if (!listId || !/^\d+$/.test(listId)) {
throw new CommandExecutionError(`Invalid listId: ${JSON.stringify(kwargs.listId)}. Expected numeric ID (see \`opencli twitter lists\`).`);
throw new ArgumentError(`Invalid listId: ${JSON.stringify(kwargs.listId)}. Expected numeric ID.`, 'Example: opencli twitter list-add 123456789 alice');
}
if (!username) {
throw new CommandExecutionError('Username is required');
throw new ArgumentError('twitter list-add username is required', 'Example: opencli twitter list-add 123456789 alice');
}
// Strategy.UI does not get a domain URL pre-nav from the framework.
// This page context is load-bearing for pre-target GraphQL calls below.
@@ -101,25 +163,33 @@ cli({
'X-Twitter-Active-User': 'yes',
});
// opencli >=1.7.x wraps page.evaluate return values as { session, data }.
// Unwrap before use so JSON.stringify of nested values doesn't become "[object Object]".
const unwrap = (v) => (v && typeof v === 'object' && 'session' in v && 'data' in v ? v.data : v);
const userLookupUrl = buildUserByScreenNameUrl(userByScreenNameQueryId, username);
const userId = await page.evaluate(`async () => {
const userIdRaw = await page.evaluate(`async () => {
const resp = await fetch(${JSON.stringify(userLookupUrl)}, { headers: ${headers}, credentials: 'include' });
if (!resp.ok) return null;
const d = await resp.json();
return d.data?.user?.result?.rest_id || null;
}`);
const userId = unwrap(userIdRaw);
if (!userId) {
throw new CommandExecutionError(`Could not resolve user @${username}`);
}
// ListsManagementPageTimeline — used both for id→name resolution and post-op verification.
// ListsManagementPageTimeline — used for list existence check + before/after member_count.
const listsQueryId = await resolveTwitterQueryId(page, 'ListsManagementPageTimeline', LISTS_MANAGEMENT_QUERY_ID);
const listsUrl = `/i/api/graphql/${listsQueryId}/ListsManagementPageTimeline?features=${encodeURIComponent(JSON.stringify(LISTS_MANAGEMENT_FEATURES))}`;
const listsData = await page.evaluate(`async () => {
const listsDataRaw = await page.evaluate(`async () => {
const r = await fetch(${JSON.stringify(listsUrl)}, { headers: ${headers}, credentials: 'include' });
if (!r.ok) return { __error: 'HTTP ' + r.status };
return await r.json();
}`);
// Don't unwrap listsData: opencli spreads GraphQL response to top-level + adds session;
// parseListsManagement reads `.data.viewer.*` from this shape directly.
const listsData = listsDataRaw;
const parsedLists = listsData && !listsData.__error
? parseListsManagement(listsData, new Set())
: [];
@@ -131,209 +201,63 @@ cli({
throw new CommandExecutionError(`List ${listId} not found among your lists (${parsedLists.length} lists fetched).`);
}
// Use UI strategy — programmatically open "Add/Remove from Lists" dialog and toggle the target list.
await page.goto(`https://x.com/${username}`);
await page.wait({ selector: '[data-testid="primaryColumn"]' });
const targetName = targetList.name;
const uiResult = await page.evaluate(`(async () => {
const sleep = (ms) => new Promise(r => setTimeout(r, ms));
const findOne = (sel, root = document) => root.querySelector(sel);
const waitFor = async (fn, { timeoutMs = 8000, intervalMs = 200 } = {}) => {
const t0 = Date.now();
while (Date.now() - t0 < timeoutMs) {
const v = fn();
if (v) return v;
await sleep(intervalMs);
}
return null;
};
// Direct GraphQL ListAddMember mutation.
//
// Previously this command opened the X profile, clicked "…" → "Add/remove from Lists",
// navigated the dialog and used nativeClick on the Save button. In 2026-05 X replaced
// the dialog with a full-page route (/i/lists/add_member), breaking that UI flow.
//
// The mutation is the same one the UI fires under the hood; calling it directly is
// both more reliable and ~10x faster (no goto-profile + scroll-dialog roundtrip).
const memberCountBefore = Number(targetList.members) || 0;
const listAddMemberQueryId = await resolveTwitterQueryId(page, 'ListAddMember', LIST_ADD_MEMBER_QUERY_ID);
const addUrl = `/i/api/graphql/${listAddMemberQueryId}/ListAddMember`;
const addBody = JSON.stringify({
variables: { listId, userId: String(userId) },
queryId: listAddMemberQueryId,
});
const addResultJsonRaw = await page.evaluate(`async () => {
try {
// Install fetch + XHR interceptors to observe list-membership mutations.
const MUTATION_RE = /ListAddMember|ListRemoveMember|lists\\/members\\/(create|destroy)|ListManagement.*Add|ListManagement.*Remove|\\/add_member|\\/remove_member|ListAddMembers|ListRemoveMembers|list.*member.*create|list.*member.*destroy/i;
if (!window.__opencliListMutations) {
window.__opencliListMutations = [];
window.__opencliAllRequests = [];
const origFetch = window.fetch.bind(window);
window.fetch = async function(...args) {
const url = typeof args[0] === 'string' ? args[0] : (args[0] && args[0].url) || '';
const method = (args[1] && args[1].method) || 'GET';
let resp;
try { resp = await origFetch(...args); }
catch (err) {
if (MUTATION_RE.test(url)) window.__opencliListMutations.push({ url, method, status: 0, error: String(err), ts: Date.now(), via: 'fetch' });
throw err;
}
if (method !== 'GET' && method !== 'HEAD') {
window.__opencliAllRequests.push({ url, method, status: resp.status, ts: Date.now(), via: 'fetch' });
}
if (MUTATION_RE.test(url)) {
window.__opencliListMutations.push({ url, method, status: resp.status, ts: Date.now(), via: 'fetch' });
}
return resp;
};
// Also hook XMLHttpRequest
const OrigXhrOpen = XMLHttpRequest.prototype.open;
const OrigXhrSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.open = function(method, url, ...rest) {
this.__opencliMethod = method;
this.__opencliUrl = url;
return OrigXhrOpen.call(this, method, url, ...rest);
};
XMLHttpRequest.prototype.send = function(...args) {
const xhr = this;
xhr.addEventListener('loadend', () => {
const url = xhr.__opencliUrl || '';
const method = xhr.__opencliMethod || 'GET';
if (method !== 'GET' && method !== 'HEAD') {
window.__opencliAllRequests.push({ url, method, status: xhr.status, ts: Date.now(), via: 'xhr' });
}
if (MUTATION_RE.test(url)) {
window.__opencliListMutations.push({ url, method, status: xhr.status, ts: Date.now(), via: 'xhr' });
}
});
return OrigXhrSend.apply(this, args);
};
}
window.__opencliListMutations.length = 0;
window.__opencliAllRequests.length = 0;
const caret = await waitFor(() => findOne('[data-testid="userActions"]'));
if (!caret) return { ok: false, message: 'Could not find user actions (…) button. Are you logged in?' };
caret.click();
await sleep(600);
const menuItems = Array.from(document.querySelectorAll('[role="menuitem"]'));
const addToListItem = menuItems.find(el => /add\\/remove|从列表|列表|add to list|add or remove/i.test(el.innerText));
if (!addToListItem) {
return { ok: false, message: 'Could not find "Add/remove from Lists" menu item' };
}
addToListItem.click();
await sleep(1200);
const dialog = await waitFor(() => findOne('[role="dialog"]'));
if (!dialog) return { ok: false, message: 'List selection dialog did not open' };
const targetName = ${JSON.stringify(targetName)};
// Find the real scroll container (virtualized list). Try a few candidates.
const scrollCandidates = [
dialog.querySelector('[data-viewportview="true"]'),
dialog.querySelector('[aria-label]')?.parentElement,
...Array.from(dialog.querySelectorAll('div')).filter(d => d.scrollHeight > d.clientHeight + 10),
].filter(Boolean);
let row = null;
let scrollEl = scrollCandidates[0] || dialog;
for (const se of scrollCandidates) {
if (se.scrollHeight > se.clientHeight + 10) { scrollEl = se; break; }
}
let lastScrollTop = -1;
for (let i = 0; i < 12; i++) {
const cells = Array.from(dialog.querySelectorAll('[data-testid="cellInnerDiv"]'));
row = cells.find(c => (c.innerText || '').split('\\n')[0].trim() === targetName);
if (row) break;
// Incremental scroll within the container
const prev = scrollEl.scrollTop;
scrollEl.scrollTop = prev + Math.max(200, scrollEl.clientHeight - 100);
if (scrollEl.scrollTop === prev) {
// Couldn't scroll further. Give up.
if (scrollEl.scrollTop === lastScrollTop) break;
}
lastScrollTop = scrollEl.scrollTop;
await sleep(500);
}
if (!row) {
const names = Array.from(dialog.querySelectorAll('[data-testid="cellInnerDiv"]'))
.map(c => (c.innerText || '').split('\\n')[0].trim()).filter(Boolean);
const dialogText = (dialog.innerText || '').slice(0, 500);
return { ok: false, message: 'List "' + targetName + '" not found. Cells: [' + names.join(' | ') + ']. DialogText: ' + dialogText };
}
const listCell = row.querySelector('[data-testid="listCell"]') || row.querySelector('[role="checkbox"]') || row;
const readChecked = () => {
const v = listCell.getAttribute('aria-checked');
return v === 'true' || v === 'false' ? v : null;
};
await sleep(600);
let ariaChecked = readChecked();
for (let i = 0; i < 8; i++) {
await sleep(500);
const next = readChecked();
if (next && next === ariaChecked) break;
ariaChecked = next || ariaChecked;
}
const isMember = ariaChecked === 'true';
if (isMember) {
const closeBtn = findOne('[data-testid="app-bar-close"]') || findOne('[aria-label="Close"]');
if (closeBtn) closeBtn.click();
return { ok: true, noop: true };
}
try { listCell.scrollIntoView({ block: 'center' }); } catch {}
await sleep(400);
const mutationsBefore = window.__opencliListMutations.length;
const rowRect = listCell.getBoundingClientRect();
// Find the Save button (top-right of dialog). Match by text "Save" / "Done" / CJK equivalents.
const saveButton = Array.from(dialog.querySelectorAll('[role="button"], button')).find(b => {
const txt = (b.innerText || '').trim();
return /^(Save|Done|保存|完成|儲存)$/i.test(txt);
const r = await fetch(${JSON.stringify(addUrl)}, {
method: 'POST',
headers: Object.assign({}, ${headers}, { 'Content-Type': 'application/json' }),
credentials: 'include',
body: ${JSON.stringify(addBody)},
});
const saveRect = saveButton ? saveButton.getBoundingClientRect() : null;
return {
ok: true,
needsNativeInteraction: true,
rowClickX: Math.round(rowRect.left + rowRect.width / 2),
rowClickY: Math.round(rowRect.top + rowRect.height / 2),
saveClickX: saveRect ? Math.round(saveRect.left + saveRect.width / 2) : null,
saveClickY: saveRect ? Math.round(saveRect.top + saveRect.height / 2) : null,
saveText: saveButton ? (saveButton.innerText || '').trim() : null,
mutationsBefore,
ariaBefore: ariaChecked,
};
const text = await r.text();
let body;
let raw = null;
try { body = JSON.parse(text); } catch { body = null; raw = text.slice(0, 300); }
const list = body && body.data && body.data.list ? body.data.list : null;
return JSON.stringify([
r.ok,
r.status,
list ? list.member_count : null,
list ? list.is_member : null,
body && body.errors ? body.errors : null,
raw,
null,
]);
} catch (e) {
return { ok: false, message: 'UI error: ' + (e?.message || String(e)) };
return JSON.stringify([false, 0, null, null, null, null, String(e)]);
}
})()`);
if (!uiResult.ok) {
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: ${uiResult.message}`);
}`);
const addResultJson = unwrap(addResultJsonRaw);
let addResultTuple;
try {
addResultTuple = JSON.parse(addResultJson);
} catch {
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: malformed mutation response envelope`);
}
const addResult = Object.create(null);
addResult.httpOk = Boolean(addResultTuple?.[0]);
addResult.status = Number(addResultTuple?.[1]) || 0;
addResult.mc = addResultTuple?.[2];
addResult.isMember = addResultTuple?.[3];
addResult.errors = addResultTuple?.[4];
addResult.raw = addResultTuple?.[5];
addResult.fetchError = addResultTuple?.[6];
let verifiedBy = null;
if (uiResult.needsNativeInteraction) {
if (typeof page.nativeClick !== 'function' || typeof page.nativeKeyPress !== 'function') {
throw new CommandExecutionError('Requires up-to-date Chrome extension (nativeClick + nativeKeyPress).');
}
if (!uiResult.saveClickX) {
throw new CommandExecutionError(`Save button not found in dialog (X expected text Save/Done). Dialog structure may have changed.`);
}
const memberCountBefore = Number(targetList.members) || 0;
// 1. Trusted click on row → aria flips false→true (optimistic UI)
await page.nativeClick(uiResult.rowClickX, uiResult.rowClickY);
await new Promise((r) => setTimeout(r, 800));
// 2. Trusted click on Save button → X commits to server
await page.nativeClick(uiResult.saveClickX, uiResult.saveClickY);
await new Promise((r) => setTimeout(r, 3500));
// Ground truth: re-fetch ListsManagementPageTimeline and compare member_count
const listsAfter = await page.evaluate(`async () => {
const r = await fetch(${JSON.stringify(listsUrl)}, { headers: ${headers}, credentials: 'include' });
if (!r.ok) return { __error: 'HTTP ' + r.status };
return await r.json();
}`);
const parsedAfter = listsAfter && !listsAfter.__error
? parseListsManagement(listsAfter, new Set())
: [];
const afterList = parsedAfter.find((l) => l.id === listId);
const memberCountAfter = afterList ? Number(afterList.members) || 0 : -1;
if (memberCountAfter > memberCountBefore) {
verifiedBy = `member_count ${memberCountBefore}${memberCountAfter}`;
} else {
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: member_count unchanged (${memberCountBefore}${memberCountAfter}). X's UI flipped but did not commit — try reloading page/extension.`);
}
}
return [{
listId,
username,
userId: String(userId),
status: uiResult.noop ? 'noop' : 'success',
message: uiResult.noop
? `@${username} is already a member of list ${listId}`
: `Added @${username} to list ${listId} (verified via ${verifiedBy})`,
}];
return [buildListAddMemberRow({ addResult, memberCountBefore, listId, username, userId })];
},
});
+97 -1
View File
@@ -1,6 +1,7 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import './list-add.js';
import { ArgumentError, CommandExecutionError } from '@jackwener/opencli/errors';
import { buildListAddMemberRow } from './list-add.js';
describe('twitter list-add registration', () => {
it('registers the list-add command with the expected shape', () => {
@@ -34,4 +35,99 @@ describe('twitter list-add registration', () => {
expect(page.wait).toHaveBeenCalledWith(3);
expect(page.getCookies).toHaveBeenCalledWith({ url: 'https://x.com' });
});
it('rejects invalid user input before navigation', async () => {
const cmd = getRegistry().get('twitter/list-add');
const page = {
goto: vi.fn(),
wait: vi.fn(),
getCookies: vi.fn(),
evaluate: vi.fn(),
};
await expect(cmd.func(page, { listId: 'abc', username: 'alice' })).rejects.toBeInstanceOf(ArgumentError);
await expect(cmd.func(page, { listId: '123', username: '' })).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
});
it('builds success rows when member_count increases despite non-fatal decode errors', () => {
const row = buildListAddMemberRow({
addResult: {
httpOk: true,
status: 200,
mc: 11,
isMember: true,
errors: [{ path: ['data', 'list', 'default_banner_media_results'], message: 'decode failed' }],
},
memberCountBefore: 10,
listId: '123',
username: 'alice',
userId: '42',
});
expect(row).toMatchObject({
listId: '123',
username: 'alice',
userId: '42',
status: 'success',
});
expect(row.message).toContain('member_count 10 → 11');
});
it('treats unchanged member_count as noop only when membership is confirmed', () => {
const row = buildListAddMemberRow({
addResult: { httpOk: true, status: 200, mc: 10, isMember: true, errors: null },
memberCountBefore: 10,
listId: '123',
username: 'alice',
userId: '42',
});
expect(row.status).toBe('noop');
expect(row.message).toBe('@alice is already a member of list 123');
});
it('fails typed when unchanged member_count does not confirm membership', () => {
expect(() => buildListAddMemberRow({
addResult: { httpOk: true, status: 200, mc: 10, isMember: false, errors: null },
memberCountBefore: 10,
listId: '123',
username: 'alice',
userId: '42',
})).toThrow(CommandExecutionError);
});
it('fails typed when member_count decreases unexpectedly', () => {
expect(() => buildListAddMemberRow({
addResult: { httpOk: true, status: 200, mc: 9, isMember: true, errors: null },
memberCountBefore: 10,
listId: '123',
username: 'alice',
userId: '42',
})).toThrow(/decreased unexpectedly/);
});
it('fails typed when GraphQL response has no usable member_count', () => {
expect(() => buildListAddMemberRow({
addResult: {
httpOk: true,
status: 200,
mc: undefined,
isMember: null,
errors: [{ message: 'List is unavailable', path: ['data', 'list'] }],
},
memberCountBefore: 10,
listId: '123',
username: 'alice',
userId: '42',
})).toThrow(/List is unavailable/);
expect(() => buildListAddMemberRow({
addResult: { httpOk: true, status: 200, mc: null, isMember: null, errors: { message: 'not an array' } },
memberCountBefore: 10,
listId: '123',
username: 'alice',
userId: '42',
})).toThrow(/no member_count/);
});
});
+13 -5
View File
@@ -1,9 +1,11 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { extractMedia } from './shared.js';
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
const LIST_TWEETS_QUERY_ID = 'RlZzktZY_9wJynoepm8ZsA';
const OPERATION_NAME = 'ListLatestTweetsTimeline';
const MAX_PAGINATION_PAGES = 100;
const FEATURES = {
rweb_video_screen_enabled: false,
@@ -70,6 +72,7 @@ export function extractTimelineTweet(result, seen) {
replies: legacy.reply_count || 0,
created_at: legacy.created_at || '',
url: `https://x.com/${screenName}/status/${tw.rest_id}`,
...extractMedia(legacy),
};
}
@@ -112,13 +115,12 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'listId', positional: true, type: 'string', required: true, help: 'Numeric ID of a Twitter/X list (e.g. from `opencli twitter lists`)' },
{ name: 'limit', type: 'int', default: 50 },
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the list timeline by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the list\'s native (recency) ordering.' },
],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'replies', 'created_at', 'url'],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'replies', 'created_at', 'url', 'has_media', 'media_urls'],
func: async (page, kwargs) => {
const listId = String(kwargs.listId || '').trim();
if (!listId || !/^\d+$/.test(listId)) {
@@ -129,7 +131,11 @@ cli({
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
if (!ct0)
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
const queryId = await page.evaluate(`async () => {
// opencli >=1.7.x wraps primitive page.evaluate returns as { session, data: <value> }.
// Without unwrap, the string queryId becomes "[object Object]" when interpolated into the URL,
// causing HTTP 400 "queryId may have expired".
const unwrap = (v) => (v && typeof v === 'object' && 'session' in v && 'data' in v ? v.data : v);
const queryIdRaw = await page.evaluate(`async () => {
try {
const ghResp = await fetch('https://raw.githubusercontent.com/fa0311/twitter-openapi/refs/heads/main/src/config/placeholder.json');
if (ghResp.ok) {
@@ -152,7 +158,8 @@ cli({
}
} catch {}
return null;
}`) || LIST_TWEETS_QUERY_ID;
}`);
const queryId = unwrap(queryIdRaw) || LIST_TWEETS_QUERY_ID;
const headers = JSON.stringify({
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
'X-Csrf-Token': ct0,
@@ -162,7 +169,8 @@ cli({
const allTweets = [];
const seen = new Set();
let cursor = null;
for (let i = 0; i < 10 && allTweets.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
const fetchCount = Math.min(100, limit - allTweets.length + 10);
const apiUrl = buildUrl(queryId, listId, fetchCount, cursor);
const data = await page.evaluate(`async () => {
+48
View File
@@ -30,9 +30,57 @@ describe('twitter list-tweets parser', () => {
replies: 2,
created_at: 'Wed Apr 16 10:00:00 +0000 2026',
url: 'https://x.com/bob/status/99',
has_media: false,
media_urls: [],
});
});
it('includes photo media URLs from extended_entities', () => {
const tweet = extractTimelineTweet({
rest_id: '101',
legacy: {
full_text: 'pic post',
extended_entities: {
media: [
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/abc.jpg' },
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/def.jpg' },
],
},
},
core: { user_results: { result: { legacy: { screen_name: 'dave' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls).toEqual([
'https://pbs.twimg.com/media/abc.jpg',
'https://pbs.twimg.com/media/def.jpg',
]);
});
it('extracts mp4 variant URL for video media', () => {
const tweet = extractTimelineTweet({
rest_id: '102',
legacy: {
full_text: 'video post',
extended_entities: {
media: [{
type: 'video',
media_url_https: 'https://pbs.twimg.com/amplify_video_thumb/thumb.jpg',
video_info: {
variants: [
{ content_type: 'application/x-mpegURL', url: 'https://video.twimg.com/playlist.m3u8' },
{ content_type: 'video/mp4', bitrate: 832000, url: 'https://video.twimg.com/low.mp4' },
{ content_type: 'video/mp4', bitrate: 2176000, url: 'https://video.twimg.com/high.mp4' },
],
},
}],
},
},
core: { user_results: { result: { legacy: { screen_name: 'erin' } } } },
}, new Set());
expect(tweet?.has_media).toBe(true);
expect(tweet?.media_urls?.[0]).toMatch(/\.mp4$/);
});
it('prefers long-form note_tweet text over truncated legacy full_text', () => {
const tweet = extractTimelineTweet({
rest_id: '100',
+5 -3
View File
@@ -92,7 +92,6 @@ export const command = cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 50, help: 'Maximum number of lists to return (default 50).' },
],
@@ -103,7 +102,9 @@ export const command = cli({
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
if (!ct0)
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
const queryId = await page.evaluate(`async () => {
// opencli >=1.7.x wraps primitive page.evaluate returns as { session, data: <value> }.
const unwrap = (v) => (v && typeof v === 'object' && 'session' in v && 'data' in v ? v.data : v);
const queryIdRaw = await page.evaluate(`async () => {
try {
const ghResp = await fetch('https://raw.githubusercontent.com/fa0311/twitter-openapi/refs/heads/main/src/config/placeholder.json');
if (ghResp.ok) {
@@ -126,7 +127,8 @@ export const command = cli({
}
} catch {}
return null;
}`) || LISTS_QUERY_ID;
}`);
const queryId = unwrap(queryIdRaw) || LISTS_QUERY_ID;
const headers = JSON.stringify({
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
'X-Csrf-Token': ct0,
-1
View File
@@ -8,7 +8,6 @@ cli({
domain: 'x.com',
strategy: Strategy.INTERCEPT,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of notifications to return (default 20).' },
],
+23 -4
View File
@@ -161,12 +161,25 @@ async function submitTweet(page, text) {
const normalize = s => String(s || '').replace(/\u00a0/g, ' ').replace(/\s+/g, ' ').trim();
const expectedText = normalize(expected);
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
const statusUrl = (root = document) => {
const links = Array.from(root.querySelectorAll('a[href*="/status/"]'));
for (const link of links) {
const href = link.href || link.getAttribute('href') || '';
if (!href) continue;
try {
const url = new URL(href, window.location.origin);
const match = url.pathname.match(/^\\/(?:[^/]+|i)\\/status\\/(\\d+)/);
if (match) return { url: url.href, id: match[1] };
} catch {}
}
return {};
};
for (let i = 0; i < ${JSON.stringify(iterations)}; i++) {
await new Promise(r => setTimeout(r, ${JSON.stringify(SUBMIT_POLL_MS)}));
const toasts = Array.from(document.querySelectorAll('[role="alert"], [data-testid="toast"]'))
.filter((el) => visible(el));
const successToast = toasts.find((el) => /sent|posted|your post was sent|your tweet was sent/i.test(el.textContent || ''));
if (successToast) return { ok: true, message: 'Tweet posted successfully.' };
if (successToast) return { ok: true, message: 'Tweet posted successfully.', ...statusUrl(successToast) };
const alert = toasts.find((el) => /failed|error|try again|not sent|could not/i.test(el.textContent || ''));
if (alert) return { ok: false, message: (alert.textContent || 'Tweet failed to post.').trim() };
@@ -175,7 +188,7 @@ async function submitTweet(page, text) {
const hasMedia = !!document.querySelector('[data-testid="attachments"], [data-testid="tweetPhoto"]')
|| document.querySelectorAll('img[src^="blob:"], video[src^="blob:"]').length > 0;
if (!composerStillHasText && !hasMedia) {
return { ok: true, message: 'Tweet posted successfully.' };
return { ok: true, message: 'Tweet posted successfully.', ...statusUrl() };
}
}
return { ok: false, message: 'Tweet submission did not complete before timeout.' };
@@ -194,7 +207,7 @@ cli({
{ name: 'text', type: 'string', required: true, positional: true, help: 'The text content of the tweet' },
{ name: 'images', type: 'string', required: false, help: 'Image paths, comma-separated, max 4 (jpg/png/gif/webp)' },
],
columns: ['status', 'message', 'text'],
columns: ['status', 'message', 'text', 'id', 'url'],
func: async (page, kwargs) => {
if (!page)
throw new CommandExecutionError('Browser session required for twitter post');
@@ -231,6 +244,12 @@ cli({
await page.wait(1);
const result = await submitTweet(page, text);
return [{ status: result?.ok ? 'success' : 'failed', message: result?.message ?? 'Tweet failed to post.', text }];
return [{
status: result?.ok ? 'success' : 'failed',
message: result?.message ?? 'Tweet failed to post.',
text,
...(result?.id ? { id: result.id } : {}),
...(result?.url ? { url: result.url } : {}),
}];
}
});
+30
View File
@@ -46,6 +46,11 @@ function makePage(evaluateResults = [], overrides = {}) {
describe('twitter post command', () => {
const getCommand = () => getRegistry().get('twitter/post');
it('registers created tweet id/url columns', () => {
const command = getCommand();
expect(command?.columns).toEqual(['status', 'message', 'text', 'id', 'url']);
});
it('posts text-only tweet successfully through the current compose route', async () => {
const command = getCommand();
const page = makePage([
@@ -63,6 +68,31 @@ describe('twitter post command', () => {
expect(page.insertText).toHaveBeenCalledWith('hello world');
});
it('returns the created tweet URL from the success toast when available', async () => {
const command = getCommand();
const page = makePage([
{ ok: true },
{ ok: true },
{ ok: true },
{
ok: true,
message: 'Tweet posted successfully.',
id: '2054239044884693381',
url: 'https://x.com/darthjajaj6z/status/2054239044884693381',
},
]);
const result = await command.func(page, { text: 'with url' });
expect(result).toEqual([{
status: 'success',
message: 'Tweet posted successfully.',
text: 'with url',
id: '2054239044884693381',
url: 'https://x.com/darthjajaj6z/status/2054239044884693381',
}]);
});
it('returns failed when text area not found', async () => {
const command = getCommand();
const page = makePage([
+16 -9
View File
@@ -1,6 +1,6 @@
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { resolveTwitterQueryId } from './shared.js';
import { normalizeTwitterScreenName, resolveTwitterQueryId, unwrapBrowserResult } from './shared.js';
import { TWITTER_BEARER_TOKEN } from './utils.js';
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
cli({
@@ -11,24 +11,31 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (with or without @). Defaults to the logged-in user when omitted.' },
],
columns: ['screen_name', 'name', 'bio', 'location', 'url', 'followers', 'following', 'tweets', 'likes', 'verified', 'created_at'],
func: async (page, kwargs) => {
let username = (kwargs.username || '').replace(/^@/, '');
// If no username, detect the logged-in user
const rawUsername = String(kwargs.username ?? '').trim();
let username = normalizeTwitterScreenName(rawUsername);
if (rawUsername && !username) {
throw new ArgumentError('twitter profile username must be a valid Twitter/X handle', 'Example: opencli twitter profile @jack');
}
// If no username, detect the logged-in user.
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
// unwrap so the href string is usable downstream.
if (!username) {
await page.goto('https://x.com/home');
await page.wait({ selector: '[data-testid="primaryColumn"]' });
const href = await page.evaluate(`() => {
const href = unwrapBrowserResult(await page.evaluate(`() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
}`);
if (!href)
}`));
if (!href || typeof href !== 'string')
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
username = normalizeTwitterScreenName(href);
if (!username)
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
username = href.replace('/', '');
}
// Navigate directly to the user's profile page (gives us cookie context)
await page.goto(`https://x.com/${username}`);
+39
View File
@@ -0,0 +1,39 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
import './profile.js';
describe('twitter profile command', () => {
it('rejects invalid explicit usernames before navigation', async () => {
const command = getRegistry().get('twitter/profile');
const page = {
goto: vi.fn(),
wait: vi.fn(),
getCookies: vi.fn(),
evaluate: vi.fn(),
};
await expect(command.func(page, { username: 'viewer/extra' })).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
expect(page.getCookies).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('rejects route-like AppTabBar hrefs instead of navigating to that route profile', async () => {
const command = getRegistry().get('twitter/profile');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(),
evaluate: vi.fn(async (script) => {
if (String(script).includes('AppTabBar_Profile_Link')) return '/home';
throw new Error(`Unexpected evaluate: ${String(script).slice(0, 80)}`);
}),
};
await expect(command.func(page, {})).rejects.toBeInstanceOf(AuthRequiredError);
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
expect(page.goto).toHaveBeenCalledTimes(1);
expect(page.getCookies).not.toHaveBeenCalled();
});
});
+133 -10
View File
@@ -10,6 +10,10 @@ import {
resolveImagePath,
} from './utils.js';
const COMPOSER_SELECTOR = '[data-testid="tweetTextarea_0"]';
const SUBMIT_POLL_MS = 500;
const SUBMIT_TIMEOUT_MS = 15_000;
function buildReplyComposerUrl(rawUrl) {
// Replaces the legacy local extractTweetId which used `/\/status\/(\d+)/`
// (silent: matched `/status/1234567` on substring `/status/123` and
@@ -19,7 +23,36 @@ function buildReplyComposerUrl(rawUrl) {
return `https://x.com/compose/post?in_reply_to=${target.id}`;
}
async function submitReply(page, text) {
function isPromiseCollectedError(err) {
const msg = err instanceof Error ? err.message : String(err);
return msg.includes('Promise was collected');
}
async function openReplyComposer(page, rawUrl) {
await page.goto(buildReplyComposerUrl(rawUrl), { waitUntil: 'load', settleMs: 2500 });
try {
await page.wait({ selector: COMPOSER_SELECTOR, timeout: 15 });
return { ok: true };
} catch {
// X sometimes leaves /compose/post?in_reply_to=<id> on the Home
// timeline behind a loading dialog. Fall back to the canonical tweet
// page and click the visible Reply action there.
await page.goto(rawUrl, { waitUntil: 'load', settleMs: 2500 });
const clicked = await page.evaluate(`(() => {
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
const buttons = Array.from(document.querySelectorAll('[data-testid="reply"]'));
const btn = buttons.find((el) => visible(el) && !el.disabled && el.getAttribute('aria-disabled') !== 'true');
if (!btn) return { ok: false, message: 'Could not find the reply button on the target tweet.' };
btn.click();
return { ok: true };
})()`);
if (!clicked?.ok) return clicked;
await page.wait({ selector: COMPOSER_SELECTOR, timeout: 15 });
return { ok: true };
}
}
async function insertReplyText(page, text) {
return page.evaluate(`(async () => {
try {
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
@@ -44,23 +77,109 @@ async function submitReply(page, text) {
}
await new Promise(r => setTimeout(r, 1000));
const normalize = s => String(s || '').replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
const actual = box.innerText || box.textContent || '';
if (!normalize(actual).includes(normalize(textToInsert))) {
return { ok: false, message: 'Could not verify reply text in the composer after typing.', actualText: actual };
}
return { ok: true };
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
}
async function clickReplyButton(page) {
return page.evaluate(`(() => {
try {
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
const buttons = Array.from(
document.querySelectorAll('[data-testid="tweetButton"], [data-testid="tweetButtonInline"]')
);
const btn = buttons.find((el) => visible(el) && !el.disabled);
const btn = buttons.find((el) => visible(el) && !el.disabled && el.getAttribute('aria-disabled') !== 'true');
if (!btn) {
return { ok: false, message: 'Reply button is disabled or not found.' };
}
btn.click();
return { ok: true, message: 'Reply posted successfully.' };
return { ok: true };
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
}
async function detectReplySent(page) {
return page.evaluate(`(() => {
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
const toasts = Array.from(document.querySelectorAll('[role="alert"], [data-testid="toast"]'))
.filter((el) => visible(el));
const successToast = toasts.find((el) => /sent|posted|your post was sent|your tweet was sent/i.test(el.textContent || ''));
if (!successToast) return { ok: false };
const link = successToast.querySelector('a[href*="/status/"]');
return {
ok: true,
message: 'Reply posted successfully.',
url: link?.href || link?.getAttribute('href') || undefined
};
})()`);
}
async function waitForReplySent(page, text) {
const iterations = Math.ceil(SUBMIT_TIMEOUT_MS / SUBMIT_POLL_MS);
try {
return await page.evaluate(`(async () => {
const expected = ${JSON.stringify(text)};
const normalize = s => String(s || '').replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
const expectedText = normalize(expected);
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
for (let i = 0; i < ${JSON.stringify(iterations)}; i++) {
await new Promise(r => setTimeout(r, ${JSON.stringify(SUBMIT_POLL_MS)}));
const toasts = Array.from(document.querySelectorAll('[role="alert"], [data-testid="toast"]'))
.filter((el) => visible(el));
const successToast = toasts.find((el) => /sent|posted|your post was sent|your tweet was sent/i.test(el.textContent || ''));
if (successToast) {
const link = successToast.querySelector('a[href*="/status/"]');
return {
ok: true,
message: 'Reply posted successfully.',
url: link?.href || link?.getAttribute('href') || undefined
};
}
const alert = toasts.find((el) => /failed|error|try again|not sent|could not/i.test(el.textContent || ''));
if (alert) return { ok: false, message: (alert.textContent || 'Reply failed to post.').trim() };
const boxes = Array.from(document.querySelectorAll('[data-testid="tweetTextarea_0"]')).filter(visible);
const composerStillHasText = boxes.some((box) => normalize(box.innerText || box.textContent || '').includes(expectedText));
if (!composerStillHasText) return { ok: true, message: 'Reply posted successfully.' };
}
return { ok: false, message: 'Reply submission did not complete before timeout.' };
})()`);
} catch (err) {
// X may route the SPA immediately after click, making CDP collect the
// polling promise even though the reply was submitted. If the page now
// shows the success toast, report success instead of a false negative.
if (!isPromiseCollectedError(err)) throw err;
await page.wait(2);
const recovered = await detectReplySent(page);
if (recovered?.ok) return recovered;
throw err;
}
}
async function submitReply(page, text) {
const typed = await insertReplyText(page, text);
if (!typed?.ok) return typed;
let clicked;
try {
clicked = await clickReplyButton(page);
} catch (err) {
if (!isPromiseCollectedError(err)) throw err;
}
if (clicked && !clicked.ok) return clicked;
return waitForReplySent(page, text);
}
cli({
site: 'twitter',
name: 'reply',
@@ -75,7 +194,7 @@ cli({
{ name: 'image', help: 'Optional local image path to attach to the reply' },
{ name: 'image-url', help: 'Optional remote image URL to download and attach to the reply' },
],
columns: ['status', 'message', 'text'],
columns: ['status', 'message', 'text', 'url'],
func: async (page, kwargs) => {
if (!page)
throw new CommandExecutionError('Browser session required for twitter reply');
@@ -92,21 +211,24 @@ cli({
localImagePath = downloaded.absPath;
cleanupDir = downloaded.cleanupDir;
}
// Dedicated composer is more reliable than the inline tweet page reply box.
await page.goto(buildReplyComposerUrl(kwargs.url), { waitUntil: 'load', settleMs: 2500 });
await page.wait({ selector: '[data-testid="tweetTextarea_0"]', timeout: 15 });
// Dedicated composer is normally more reliable than the inline
// tweet page reply box, but X occasionally leaves that route on the
// Home timeline behind a loading dialog. openReplyComposer falls
// back to the target tweet's visible Reply action.
const composer = await openReplyComposer(page, kwargs.url);
if (!composer?.ok) {
return [{ status: 'failed', message: composer?.message ?? 'Could not open the reply composer.', text: kwargs.text }];
}
if (localImagePath) {
await page.wait({ selector: COMPOSER_FILE_INPUT_SELECTOR, timeout: 20 });
await attachComposerImage(page, localImagePath);
}
const result = await submitReply(page, kwargs.text);
if (result.ok) {
await page.wait(3); // Wait for network submission to complete
}
return [{
status: result.ok ? 'success' : 'failed',
message: result.message,
text: kwargs.text,
...(result.url ? { url: result.url } : {}),
...(kwargs.image ? { image: kwargs.image } : {}),
...(kwargs['image-url'] ? { 'image-url': kwargs['image-url'] } : {}),
}];
@@ -119,4 +241,5 @@ cli({
});
export const __test__ = {
buildReplyComposerUrl,
isPromiseCollectedError,
};
+55
View File
@@ -13,6 +13,8 @@ describe('twitter reply command', () => {
const cmd = getRegistry().get('twitter/reply');
expect(cmd?.func).toBeTypeOf('function');
const page = createPageMock([
{ ok: true },
{ ok: true },
{ ok: true, message: 'Reply posted successfully.' },
]);
const result = await cmd.func(page, {
@@ -38,6 +40,8 @@ describe('twitter reply command', () => {
const setFileInput = vi.fn().mockResolvedValue(undefined);
const page = createPageMock([
{ ok: true, previewCount: 1 },
{ ok: true },
{ ok: true },
{ ok: true, message: 'Reply posted successfully.' },
], {
setFileInput,
@@ -74,6 +78,8 @@ describe('twitter reply command', () => {
const setFileInput = vi.fn().mockResolvedValue(undefined);
const page = createPageMock([
{ ok: true, previewCount: 1 },
{ ok: true },
{ ok: true },
{ ok: true, message: 'Reply posted successfully.' },
], {
setFileInput,
@@ -102,6 +108,55 @@ describe('twitter reply command', () => {
]);
vi.unstubAllGlobals();
});
it('falls back to the target tweet page when the dedicated composer route does not expose a textarea', async () => {
const cmd = getRegistry().get('twitter/reply');
expect(cmd?.func).toBeTypeOf('function');
const wait = vi.fn()
.mockRejectedValueOnce(new Error('Selector not found: [data-testid="tweetTextarea_0"]'))
.mockResolvedValue(undefined);
const page = createPageMock([
{ ok: true }, // click target tweet page Reply button
{ ok: true }, // insert reply text
{ ok: true }, // click composer Reply button
{ ok: true, message: 'Reply posted successfully.' }, // submit completed
], { wait });
const url = 'https://x.com/_kop6/status/2040254679301718161?s=20';
const result = await cmd.func(page, { url, text: 'fallback reply' });
expect(page.goto).toHaveBeenNthCalledWith(1, 'https://x.com/compose/post?in_reply_to=2040254679301718161', { waitUntil: 'load', settleMs: 2500 });
expect(page.goto).toHaveBeenNthCalledWith(2, url, { waitUntil: 'load', settleMs: 2500 });
expect(page.evaluate.mock.calls[0][0]).toContain('[data-testid="reply"]');
expect(wait).toHaveBeenLastCalledWith({ selector: '[data-testid="tweetTextarea_0"]', timeout: 15 });
expect(result).toEqual([{ status: 'success', message: 'Reply posted successfully.', text: 'fallback reply' }]);
});
it('treats an X success toast as success after a Promise was collected error', async () => {
const cmd = getRegistry().get('twitter/reply');
expect(cmd?.func).toBeTypeOf('function');
const evaluate = vi.fn()
.mockResolvedValueOnce({ ok: true }) // insert reply text
.mockResolvedValueOnce({ ok: true }) // click Reply
.mockRejectedValueOnce(new Error('{"code":-32000,"message":"Promise was collected"}'))
.mockResolvedValueOnce({
ok: true,
message: 'Reply posted successfully.',
url: 'https://x.com/me/status/123',
});
const page = createPageMock([], { evaluate });
const result = await cmd.func(page, {
url: 'https://x.com/_kop6/status/2040254679301718161?s=20',
text: 'toast recovery',
});
expect(page.wait).toHaveBeenCalledWith(2);
expect(result).toEqual([{
status: 'success',
message: 'Reply posted successfully.',
text: 'toast recovery',
url: 'https://x.com/me/status/123',
}]);
});
it('rejects using --image and --image-url together', async () => {
const cmd = getRegistry().get('twitter/reply');
expect(cmd?.func).toBeTypeOf('function');
+195 -178
View File
@@ -1,7 +1,7 @@
import { ArgumentError, CommandExecutionError } from '@jackwener/opencli/errors';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { extractMedia } from './shared.js';
import { applyTopByEngagement } from './utils.js';
import { extractMedia, normalizeTwitterGraphqlPayload, resolveTwitterOperationMetadata } from './shared.js';
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
// ── Public-search operator surface ─────────────────────────────────────
//
@@ -35,6 +35,68 @@ const PRODUCT_TO_F_PARAM = Object.freeze({
videos: 'video',
});
const PRODUCT_TO_GRAPHQL_PRODUCT = Object.freeze({
top: 'Top',
live: 'Latest',
photos: 'Photos',
videos: 'Videos',
});
const MAX_PAGINATION_PAGES = 100;
const SEARCH_TIMELINE_OPERATION = {
queryId: 'VhUd6vHVmLBcw0uX-6jMLA',
features: {
rweb_video_screen_enabled: true,
rweb_cashtags_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: true,
rweb_cashtags_composer_attachment_enabled: true,
responsive_web_jetfuel_frame: true,
responsive_web_grok_share_attachment_enabled: true,
responsive_web_grok_annotations_enabled: true,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
content_disclosure_indicator_enabled: true,
content_disclosure_ai_generated_indicator_enabled: true,
responsive_web_grok_show_grok_translated_post: false,
responsive_web_grok_analysis_button_from_backend: true,
post_ctas_fetch_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
responsive_web_enhance_cards_enabled: false,
},
fieldToggles: {
withPayments: true,
withAuxiliaryUserLabels: true,
withArticleRichContentState: true,
withArticlePlainText: true,
withArticleSummaryText: true,
withArticleVoiceOver: true,
withGrokAnalyze: true,
withDisallowedReplyControls: true,
},
};
const FROM_USER_PATTERN = /^[A-Za-z0-9_]{1,15}$/;
const EXCLUDE_TO_OPERATOR = Object.freeze({
@@ -99,125 +161,96 @@ function resolveSearchFParam(kwargs) {
return kwargs.filter === 'live' ? 'live' : 'top';
}
/**
* Trigger Twitter search SPA navigation with fallback strategies.
*
* Primary: pushState + popstate (works in most environments).
* Fallback: Type into the search input and press Enter when pushState fails
* intermittently (e.g. due to Twitter A/B tests or timing races — see #690).
*
* Both strategies preserve the JS context so the fetch interceptor stays alive.
*
* @param {object} page
* @param {string} query — final composed query (already merged with operators)
* @param {string} fParam — Twitter URL `f=` value (top|live|image|video)
*/
async function navigateToSearch(page, query, fParam) {
const searchUrl = JSON.stringify(`/search?q=${encodeURIComponent(query)}&f=${fParam}`);
let lastPath = '';
// Strategy 1 (primary): pushState + popstate with retry
for (let attempt = 1; attempt <= 2; attempt++) {
await page.evaluate(`
(() => {
window.history.pushState({}, '', ${searchUrl});
window.dispatchEvent(new PopStateEvent('popstate', { state: {} }));
})()
`);
try {
await page.wait({ selector: '[data-testid="primaryColumn"]' });
}
catch {
// selector timeout — fall through to path check or next attempt
}
lastPath = String(await page.evaluate('() => window.location.pathname') || '');
if (lastPath.startsWith('/search')) {
return;
}
if (attempt < 2) {
await page.wait(1);
}
}
// Strategy 2 (fallback): Use the search input on /explore.
// The nativeSetter + Enter approach triggers Twitter's own form handler,
// performing SPA navigation without a full page reload.
const queryStr = JSON.stringify(query);
const navResult = await page.evaluate(`(async () => {
try {
const input = document.querySelector('[data-testid="SearchBox_Search_Input"]');
if (!input) return { ok: false };
input.focus();
await new Promise(r => setTimeout(r, 300));
const nativeSetter = Object.getOwnPropertyDescriptor(
window.HTMLInputElement.prototype, 'value'
)?.set;
if (!nativeSetter) return { ok: false };
nativeSetter.call(input, ${queryStr});
input.dispatchEvent(new Event('input', { bubbles: true }));
input.dispatchEvent(new Event('change', { bubbles: true }));
await new Promise(r => setTimeout(r, 500));
input.dispatchEvent(new KeyboardEvent('keydown', {
key: 'Enter', code: 'Enter', keyCode: 13, bubbles: true
}));
return { ok: true };
} catch {
return { ok: false };
}
})()`);
if (navResult?.ok) {
try {
await page.wait({ selector: '[data-testid="primaryColumn"]' });
}
catch {
// fall through to path check
}
lastPath = String(await page.evaluate('() => window.location.pathname') || '');
if (lastPath.startsWith('/search')) {
// The fallback path doesn't carry the f= URL param, so click the
// matching tab to align with the requested product. Only `live`
// currently surfaces a distinct tab label — `image`/`video` tabs
// also need an explicit click, so try them all.
const tabClicked = await clickProductTabIfNeeded(page, fParam);
if (!tabClicked) {
throw new CommandExecutionError(`SPA fallback reached /search but could not select the requested product tab: ${fParam}`);
}
return;
}
}
throw new CommandExecutionError(`SPA navigation to /search failed. Final path: ${lastPath || '(empty)'}. Twitter may have changed its routing.`);
function resolveSearchProduct(kwargs) {
const product = kwargs.product || (kwargs.filter === 'live' ? 'live' : 'top');
return PRODUCT_TO_GRAPHQL_PRODUCT[product] || 'Top';
}
/**
* After the search-input fallback lands on /search, the f= param is missing
* from the URL. Click the matching tab in the result page header so the
* SearchTimeline call uses the right filter. No-op for fParam=top (default).
*/
async function clickProductTabIfNeeded(page, fParam) {
if (fParam === 'top') return true;
const tabLabels = JSON.stringify({
live: ['Latest', '最新'],
image: ['Photos', 'Images', '照片', '图片'],
video: ['Videos', '视频'],
}[fParam] || []);
if (tabLabels === '[]') return true;
const clicked = await page.evaluate(`(() => {
const labels = ${tabLabels};
const tabs = document.querySelectorAll('[role="tab"]');
for (const tab of tabs) {
const txt = (tab.textContent || '').trim();
if (labels.some(l => txt.includes(l))) {
tab.click();
return true;
function normalizeOperation(operation) {
if (typeof operation === 'string') {
return {
queryId: operation,
features: SEARCH_TIMELINE_OPERATION.features,
fieldToggles: SEARCH_TIMELINE_OPERATION.fieldToggles,
};
}
return {
queryId: operation?.queryId || SEARCH_TIMELINE_OPERATION.queryId,
features: operation?.features || SEARCH_TIMELINE_OPERATION.features,
fieldToggles: operation?.fieldToggles || SEARCH_TIMELINE_OPERATION.fieldToggles,
};
}
function buildSearchTimelineRequest(operation, rawQuery, product, count, cursor) {
const normalized = normalizeOperation(operation);
const vars = {
rawQuery,
count,
querySource: 'typed_query',
product,
};
if (cursor) vars.cursor = cursor;
return [
`/i/api/graphql/${normalized.queryId}/SearchTimeline`,
{
variables: vars,
features: normalized.features,
fieldToggles: normalized.fieldToggles,
},
];
}
function unwrapTweetResult(result) {
if (!result) return null;
if (result.__typename === 'TweetWithVisibilityResults' && result.tweet) return result.tweet;
if (result.tweet) return result.tweet;
return result;
}
function tweetToRow(result, seen) {
const tweet = unwrapTweetResult(result);
if (!tweet?.rest_id || seen.has(tweet.rest_id)) return null;
seen.add(tweet.rest_id);
const tweetUser = tweet.core?.user_results?.result;
return {
id: tweet.rest_id,
author: tweetUser?.core?.screen_name || tweetUser?.legacy?.screen_name || 'unknown',
text: tweet.note_tweet?.note_tweet_results?.result?.text || tweet.legacy?.full_text || '',
created_at: tweet.legacy?.created_at || '',
likes: tweet.legacy?.favorite_count || 0,
views: tweet.views?.count || '0',
url: `https://x.com/i/status/${tweet.rest_id}`,
...extractMedia(tweet.legacy),
};
}
function parseSearchTimeline(data, seen) {
const rows = [];
let nextCursor = null;
const instructions = data?.data?.search_by_raw_query?.search_timeline?.timeline?.instructions || [];
const visit = (value) => {
if (!value || typeof value !== 'object') return;
if (value.tweet_results?.result) {
const row = tweetToRow(value.tweet_results.result, seen);
if (row) rows.push(row);
}
}
return false;
})()`);
if (!clicked) return false;
await page.wait(2);
return true;
if (
(value.entryType === 'TimelineTimelineCursor' || value.__typename === 'TimelineTimelineCursor')
&& (value.cursorType === 'Bottom' || value.cursorType === 'ShowMore')
&& value.value
) {
nextCursor = value.value;
}
if (Array.isArray(value)) {
for (const item of value) visit(item);
return;
}
for (const child of Object.values(value)) {
if (child && typeof child === 'object') visit(child);
}
};
visit(instructions);
return { rows, nextCursor };
}
cli({
@@ -226,9 +259,8 @@ cli({
access: 'read',
description: 'Search Twitter/X for tweets, with optional --from / --has / --exclude / --product filters mapped to X\'s search operators',
domain: 'x.com',
strategy: Strategy.INTERCEPT, // Use intercept strategy
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'query', type: 'string', required: true, positional: true, help: 'Search query. Raw X operators (e.g. "exact phrase", #tag, OR, lang:en, since:YYYY-MM-DD, from:, since:) are passed through unchanged.' },
{ name: 'filter', type: 'string', default: 'top', choices: ['top', 'live'], help: 'Legacy alias for --product. Kept for backwards compatibility; if --product is set it wins.' },
@@ -248,65 +280,47 @@ cli({
if (!Number.isInteger(Number(kwargs.limit)) || Number(kwargs.limit) <= 0) {
throw new ArgumentError('twitter search --limit must be a positive integer', 'Example: opencli twitter search opencli --limit 15');
}
const fParam = resolveSearchFParam(kwargs);
// 1. Navigate to x.com/explore (has a search input at the top)
await page.goto('https://x.com/explore');
await page.wait(3);
// 2. Install interceptor BEFORE triggering search.
// SPA navigation preserves the JS context, so the monkey-patched
// fetch will capture the SearchTimeline API call.
await page.installInterceptor('SearchTimeline');
// 3. Trigger SPA navigation to search results via history API.
// pushState + popstate triggers React Router's listener without
// a full page reload, so the interceptor stays alive.
// Note: the previous approach (nativeSetter + Enter keydown on the
// search input) does not reliably trigger Twitter's form submission.
await navigateToSearch(page, finalQuery, fParam);
// 4. Scroll to trigger additional pagination
await page.autoScroll({ times: 3, delayMs: 2000 });
// 5. Retrieve captured data
const requests = await page.getInterceptedRequests();
if (!requests || requests.length === 0)
return [];
let results = [];
const cookies = await page.getCookies({ url: 'https://x.com' });
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
if (!ct0) throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
await page.goto('https://x.com/home', { waitUntil: 'load', settleMs: 1000 });
const operation = await resolveTwitterOperationMetadata(page, 'SearchTimeline', SEARCH_TIMELINE_OPERATION);
const headers = JSON.stringify({
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
'X-Csrf-Token': ct0,
'X-Twitter-Auth-Type': 'OAuth2Session',
'X-Twitter-Active-User': 'yes',
'Content-Type': 'application/json',
});
const product = resolveSearchProduct(kwargs);
const results = [];
const seen = new Set();
for (const req of requests) {
try {
const insts = req?.data?.search_by_raw_query?.search_timeline?.timeline?.instructions || [];
const addEntries = insts.find((i) => i.type === 'TimelineAddEntries')
|| insts.find((i) => i.entries && Array.isArray(i.entries));
if (!addEntries?.entries)
continue;
for (const entry of addEntries.entries) {
if (!entry.entryId.startsWith('tweet-'))
continue;
let tweet = entry.content?.itemContent?.tweet_results?.result;
if (!tweet)
continue;
// Handle retweet wrapping
if (tweet.__typename === 'TweetWithVisibilityResults' && tweet.tweet) {
tweet = tweet.tweet;
}
if (!tweet.rest_id || seen.has(tweet.rest_id))
continue;
seen.add(tweet.rest_id);
// Twitter moved screen_name from legacy to core
const tweetUser = tweet.core?.user_results?.result;
results.push({
id: tweet.rest_id,
author: tweetUser?.core?.screen_name || tweetUser?.legacy?.screen_name || 'unknown',
text: tweet.note_tweet?.note_tweet_results?.result?.text || tweet.legacy?.full_text || '',
created_at: tweet.legacy?.created_at || '',
likes: tweet.legacy?.favorite_count || 0,
views: tweet.views?.count || '0',
url: `https://x.com/i/status/${tweet.rest_id}`,
...extractMedia(tweet.legacy),
});
}
}
catch (e) {
// ignore parsing errors for individual payloads
let cursor = null;
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && results.length < kwargs.limit; i++) {
const fetchCount = Number(kwargs.limit) - results.length + 10;
const [requestUrl, requestPayload] = buildSearchTimelineRequest(operation, finalQuery, product, fetchCount, cursor);
const requestBody = JSON.stringify(requestPayload);
const data = normalizeTwitterGraphqlPayload(await page.evaluate(`async () => {
const options = {
method: 'POST',
headers: ${headers},
credentials: 'include',
};
options['body'] = ${JSON.stringify(requestBody)};
const r = await fetch(${JSON.stringify(requestUrl)}, {
...options,
});
return r.ok ? await r.json() : { error: r.status };
}`));
if (data?.error) {
if (results.length === 0) throw new CommandExecutionError(`HTTP ${data.error}: SearchTimeline fetch failed — queryId may have expired`);
break;
}
const { rows, nextCursor } = parseSearchTimeline(data, seen);
results.push(...rows);
if (!nextCursor || nextCursor === cursor) break;
cursor = nextCursor;
}
const trimmed = results.slice(0, kwargs.limit);
return applyTopByEngagement(trimmed, kwargs['top-by-engagement']);
@@ -316,6 +330,9 @@ cli({
export const __test__ = {
buildSearchQuery,
resolveSearchFParam,
resolveSearchProduct,
buildSearchTimelineRequest,
parseSearchTimeline,
HAS_CHOICES,
EXCLUDE_CHOICES,
PRODUCT_CHOICES,
+96 -258
View File
@@ -2,71 +2,67 @@ import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { __test__ } from './search.js';
const { buildSearchQuery, resolveSearchFParam, HAS_CHOICES, EXCLUDE_CHOICES, PRODUCT_CHOICES, EXCLUDE_TO_OPERATOR, PRODUCT_TO_F_PARAM, FROM_USER_PATTERN } = __test__;
const { buildSearchQuery, resolveSearchFParam, resolveSearchProduct, buildSearchTimelineRequest, parseSearchTimeline, HAS_CHOICES, EXCLUDE_CHOICES, PRODUCT_CHOICES, EXCLUDE_TO_OPERATOR, PRODUCT_TO_F_PARAM, FROM_USER_PATTERN } = __test__;
describe('twitter search command', () => {
it('retries transient SPA navigation failures before giving up', async () => {
function makeSearchPage(data) {
return {
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
goto: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn()
.mockResolvedValueOnce(null) // resolveTwitterQueryId fallback
.mockResolvedValueOnce(data),
};
}
it('fetches SearchTimeline directly instead of relying on SPA navigation', async () => {
const command = getRegistry().get('twitter/search');
expect(command?.func).toBeTypeOf('function');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/explore')
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/search');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([
{
data: {
search_by_raw_query: {
search_timeline: {
timeline: {
instructions: [
const page = makeSearchPage({
data: {
search_by_raw_query: {
search_timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'tweet-1',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '1',
legacy: {
full_text: 'hello world',
favorite_count: 7,
created_at: 'Thu Mar 26 10:30:00 +0000 2026',
},
core: {
user_results: {
result: {
core: {
screen_name: 'alice',
},
},
entryId: 'tweet-1',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '1',
legacy: {
full_text: 'hello world',
favorite_count: 7,
created_at: 'Thu Mar 26 10:30:00 +0000 2026',
},
core: {
user_results: {
result: {
core: {
screen_name: 'alice',
},
},
views: {
count: '12',
},
},
},
views: {
count: '12',
},
},
},
},
],
},
},
],
},
},
],
},
},
},
]),
};
},
});
const result = await command.func(page, { query: 'from:alice', filter: 'top', limit: 5 });
expect(result).toEqual([
{
@@ -81,112 +77,62 @@ describe('twitter search command', () => {
media_urls: [],
},
]);
expect(page.installInterceptor).toHaveBeenCalledWith('SearchTimeline');
expect(evaluate).toHaveBeenCalledTimes(4);
expect(page.getCookies).toHaveBeenCalledWith({ url: 'https://x.com' });
expect(page.goto).toHaveBeenCalledWith('https://x.com/home', { waitUntil: 'load', settleMs: 1000 });
const searchFetch = page.evaluate.mock.calls[1][0];
expect(searchFetch).toContain('/SearchTimeline');
expect(searchFetch).toContain("method: 'POST'");
expect(searchFetch).toContain('\\"rawQuery\\":\\"from:alice\\"');
});
it('uses f=live in search URL when filter is live', async () => {
it('uses the requested GraphQL product', async () => {
const command = getRegistry().get('twitter/search');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/search');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
};
await command.func(page, { query: 'breaking news', filter: 'live', limit: 5 });
const pushStateCall = evaluate.mock.calls[0][0];
expect(pushStateCall).toContain('f=live');
expect(pushStateCall).toContain(encodeURIComponent('breaking news'));
const page = makeSearchPage({ data: { search_by_raw_query: { search_timeline: { timeline: { instructions: [] } } } } });
await command.func(page, { query: 'cats', product: 'videos', limit: 5 });
expect(page.evaluate.mock.calls[1][0]).toContain('\\"product\\":\\"Videos\\"');
});
it('uses f=top in search URL when filter is top', async () => {
it('paginates past the old five-page cap until the requested limit is reached', async () => {
const command = getRegistry().get('twitter/search');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/search');
let pageIndex = 0;
const page = {
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
};
await command.func(page, { query: 'test', filter: 'top', limit: 5 });
const pushStateCall = evaluate.mock.calls[0][0];
expect(pushStateCall).toContain('f=top');
});
it('falls back to top when filter is omitted', async () => {
const command = getRegistry().get('twitter/search');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/search');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
};
await command.func(page, { query: 'test', limit: 5 });
const pushStateCall = evaluate.mock.calls[0][0];
expect(pushStateCall).toContain('f=top');
});
it('falls back to search input when pushState fails twice', async () => {
const command = getRegistry().get('twitter/search');
expect(command?.func).toBeTypeOf('function');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined) // pushState attempt 1
.mockResolvedValueOnce('/explore') // pathname check 1 — not /search
.mockResolvedValueOnce(undefined) // pushState attempt 2
.mockResolvedValueOnce('/explore') // pathname check 2 — still not /search
.mockResolvedValueOnce({ ok: true }) // search input fallback succeeds
.mockResolvedValueOnce('/search'); // pathname check after fallback
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([
{
evaluate: vi.fn().mockImplementation(async () => {
if (pageIndex === 0) {
pageIndex += 1;
return null;
}
const id = String(pageIndex);
pageIndex += 1;
return {
data: {
search_by_raw_query: {
search_timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'tweet-99',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '99',
legacy: {
full_text: 'fallback works',
favorite_count: 3,
created_at: 'Wed Apr 02 12:00:00 +0000 2026',
},
core: {
user_results: {
result: {
core: { screen_name: 'bob' },
},
},
},
views: { count: '5' },
rest_id: id,
legacy: { full_text: `tweet ${id}`, created_at: 'now' },
core: { user_results: { result: { core: { screen_name: 'alice' } } } },
},
},
},
},
},
{
content: {
entryType: 'TimelineTimelineCursor',
cursorType: 'Bottom',
value: `cursor-${id}`,
},
},
],
},
],
@@ -194,100 +140,13 @@ describe('twitter search command', () => {
},
},
},
},
]),
};
}),
};
const result = await command.func(page, { query: 'test fallback', filter: 'top', limit: 5 });
expect(result).toEqual([
{
id: '99',
author: 'bob',
text: 'fallback works',
created_at: 'Wed Apr 02 12:00:00 +0000 2026',
likes: 3,
views: '5',
url: 'https://x.com/i/status/99',
has_media: false,
media_urls: [],
},
]);
// 6 evaluate calls: 2x pushState + 2x pathname check + 1x fallback + 1x pathname check
expect(evaluate).toHaveBeenCalledTimes(6);
expect(page.autoScroll).toHaveBeenCalled();
});
it('clicks the requested product tab after fallback navigation when f= param is absent', async () => {
const command = getRegistry().get('twitter/search');
expect(command?.func).toBeTypeOf('function');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined) // pushState attempt 1
.mockResolvedValueOnce('/explore')
.mockResolvedValueOnce(undefined) // pushState attempt 2
.mockResolvedValueOnce('/explore')
.mockResolvedValueOnce({ ok: true }) // search input fallback
.mockResolvedValueOnce('/search')
.mockResolvedValueOnce(true); // product tab click
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
};
const result = await command.func(page, { query: 'cats', product: 'photos', limit: 5 });
expect(result).toEqual([]);
expect(evaluate).toHaveBeenCalledTimes(7);
expect(evaluate.mock.calls[6][0]).toContain('Photos');
expect(page.autoScroll).toHaveBeenCalled();
});
it('throws when fallback navigation cannot select the requested product tab', async () => {
const command = getRegistry().get('twitter/search');
expect(command?.func).toBeTypeOf('function');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined) // pushState attempt 1
.mockResolvedValueOnce('/explore')
.mockResolvedValueOnce(undefined) // pushState attempt 2
.mockResolvedValueOnce('/explore')
.mockResolvedValueOnce({ ok: true }) // search input fallback
.mockResolvedValueOnce('/search')
.mockResolvedValueOnce(false); // requested tab missing
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn(),
};
await expect(command.func(page, { query: 'cats', product: 'videos', limit: 5 }))
.rejects
.toThrow(/could not select the requested product tab: video/);
expect(page.autoScroll).not.toHaveBeenCalled();
expect(page.getInterceptedRequests).not.toHaveBeenCalled();
});
it('throws with the final path after both attempts fail', async () => {
const command = getRegistry().get('twitter/search');
expect(command?.func).toBeTypeOf('function');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined) // pushState attempt 1
.mockResolvedValueOnce('/explore') // pathname check 1
.mockResolvedValueOnce(undefined) // pushState attempt 2
.mockResolvedValueOnce('/login') // pathname check 2
.mockResolvedValueOnce({ ok: false }); // search input fallback
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn(),
};
await expect(command.func(page, { query: 'from:alice', filter: 'top', limit: 5 }))
.rejects
.toThrow('Final path: /login');
expect(page.autoScroll).not.toHaveBeenCalled();
expect(page.getInterceptedRequests).not.toHaveBeenCalled();
expect(evaluate).toHaveBeenCalledTimes(5);
const result = await command.func(page, { query: 'opencli', limit: 7 });
expect(result).toHaveLength(7);
expect(result.map((row) => row.id)).toEqual(['1', '2', '3', '4', '5', '6', '7']);
expect(page.evaluate).toHaveBeenCalledTimes(8);
});
});
@@ -411,18 +270,15 @@ describe('twitter search filter helpers', () => {
});
describe('twitter search end-to-end with new filters', () => {
it('encodes the composed query and product=live into the f= URL param', async () => {
it('encodes the composed query and product=live into the GraphQL request', async () => {
const command = getRegistry().get('twitter/search');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/search');
.mockResolvedValueOnce(null)
.mockResolvedValueOnce({ data: { search_by_raw_query: { search_timeline: { timeline: { instructions: [] } } } } });
const page = {
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
};
await command.func(page, {
query: 'breaking news',
@@ -432,37 +288,26 @@ describe('twitter search end-to-end with new filters', () => {
product: 'live',
limit: 5,
});
const pushStateCall = evaluate.mock.calls[0][0];
// f=live wins because --product=live trumps the default --filter
expect(pushStateCall).toContain('f=live');
// composed query should be percent-encoded inside the URL
const encoded = encodeURIComponent('breaking news from:alice filter:images -filter:nativeretweets');
expect(pushStateCall).toContain(encoded);
const searchFetch = evaluate.mock.calls[1][0];
expect(searchFetch).toContain('\\"product\\":\\"Latest\\"');
expect(searchFetch).toContain('\\"rawQuery\\":\\"breaking news from:alice filter:images -filter:nativeretweets\\"');
});
it('throws ArgumentError when query and all filters are empty', async () => {
const command = getRegistry().get('twitter/search');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn(),
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn(),
};
await expect(command.func(page, { query: ' ', limit: 5 }))
.rejects
.toThrow(/empty/i);
expect(page.installInterceptor).not.toHaveBeenCalled();
expect(page.goto).not.toHaveBeenCalled();
});
it('throws ArgumentError for invalid --from before navigation', async () => {
const command = getRegistry().get('twitter/search');
const page = {
goto: vi.fn(),
wait: vi.fn(),
installInterceptor: vi.fn(),
evaluate: vi.fn(),
autoScroll: vi.fn(),
getInterceptedRequests: vi.fn(),
};
await expect(command.func(page, { query: 'hi', from: 'alice filter:links', limit: 5 }))
.rejects
@@ -473,11 +318,7 @@ describe('twitter search end-to-end with new filters', () => {
const command = getRegistry().get('twitter/search');
const page = {
goto: vi.fn(),
wait: vi.fn(),
installInterceptor: vi.fn(),
evaluate: vi.fn(),
autoScroll: vi.fn(),
getInterceptedRequests: vi.fn(),
};
await expect(command.func(page, { query: 'hi', limit: 0 }))
.rejects
@@ -487,19 +328,16 @@ describe('twitter search end-to-end with new filters', () => {
it('runs with only filters set (empty <query>)', async () => {
const command = getRegistry().get('twitter/search');
const evaluate = vi.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce('/search');
.mockResolvedValueOnce(null)
.mockResolvedValueOnce({ data: { search_by_raw_query: { search_timeline: { timeline: { instructions: [] } } } } });
const page = {
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
installInterceptor: vi.fn().mockResolvedValue(undefined),
evaluate,
autoScroll: vi.fn().mockResolvedValue(undefined),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
};
const result = await command.func(page, { query: '', from: 'alice', limit: 5 });
expect(result).toEqual([]);
const pushStateCall = evaluate.mock.calls[0][0];
expect(pushStateCall).toContain(encodeURIComponent('from:alice'));
const searchFetch = evaluate.mock.calls[1][0];
expect(searchFetch).toContain('\\"rawQuery\\":\\"from:alice\\"');
});
});
+167 -16
View File
@@ -1,8 +1,28 @@
import { ArgumentError } from '@jackwener/opencli/errors';
const QUERY_ID_PATTERN = /^[A-Za-z0-9_-]+$/;
const SCREEN_NAME_PATTERN = /^[A-Za-z0-9_]{1,15}$/;
const TWEET_PATH_PATTERN = /^\/(?:[^/]+|i)\/status\/(\d+)\/?$/;
const TWEET_HOSTS = new Set(['x.com', 'twitter.com']);
const SCREEN_NAME_HOSTS = new Set(['x.com', 'twitter.com', 'mobile.twitter.com']);
const RESERVED_SCREEN_NAME_PATHS = new Set([
'compose',
'explore',
'help',
'home',
'i',
'intent',
'jobs',
'login',
'logout',
'messages',
'notifications',
'privacy',
'search',
'settings',
'signup',
'tos',
]);
function isTwitterHost(hostname) {
return TWEET_HOSTS.has(hostname)
@@ -81,9 +101,138 @@ export function buildTwitterArticleScopeSource(tweetId) {
export function sanitizeQueryId(resolved, fallbackId) {
return typeof resolved === 'string' && QUERY_ID_PATTERN.test(resolved) ? resolved : fallbackId;
}
export async function resolveTwitterQueryId(page, operationName, fallbackId) {
export function normalizeTwitterScreenName(value) {
const raw = String(value ?? '').trim();
if (!raw) return '';
let candidate = '';
try {
const url = raw.startsWith('/') ? new URL(raw, 'https://x.com') : new URL(raw);
if (
url.protocol !== 'https:' ||
url.username ||
url.password ||
url.port ||
!SCREEN_NAME_HOSTS.has(url.hostname)
) {
return '';
}
const segments = url.pathname.split('/').filter(Boolean);
if (segments.length !== 1) return '';
candidate = segments[0];
} catch {
if (raw.includes('/') || raw.includes('?') || raw.includes('#')) return '';
candidate = raw.replace(/^@+/, '');
}
if (!SCREEN_NAME_PATTERN.test(candidate)) return '';
if (RESERVED_SCREEN_NAME_PATHS.has(candidate.toLowerCase())) return '';
return candidate;
}
function keysToFlags(keys) {
if (!Array.isArray(keys)) return {};
return Object.fromEntries(keys.filter((key) => typeof key === 'string' && key).map((key) => [key, true]));
}
function normalizeOperationFallback(fallback) {
if (typeof fallback === 'string') return { queryId: fallback, features: {}, fieldToggles: {} };
return {
queryId: fallback?.queryId || null,
features: fallback?.features || {},
fieldToggles: fallback?.fieldToggles || {},
};
}
export function unwrapBrowserResult(value) {
if (
value
&& typeof value === 'object'
&& typeof value.session === 'string'
&& Object.prototype.hasOwnProperty.call(value, 'data')
) {
return value.data;
}
return value;
}
export function normalizeTwitterGraphqlPayload(value) {
const unwrapped = unwrapBrowserResult(value);
if (unwrapped?.data && typeof unwrapped.data === 'object') return unwrapped;
if (
unwrapped
&& typeof unwrapped === 'object'
&& (
Object.prototype.hasOwnProperty.call(unwrapped, 'user')
|| Object.prototype.hasOwnProperty.call(unwrapped, 'search_by_raw_query')
)
) {
return { data: unwrapped };
}
return unwrapped;
}
export function sanitizeTwitterOperationMetadata(resolved, fallback) {
const value = unwrapBrowserResult(resolved);
const normalizedFallback = normalizeOperationFallback(fallback);
// Empty resolved features / fieldToggles must defer to the baked fallback.
// The bundle parser can find a queryId but miss `featureSwitches:[...]` (e.g.
// a minification change, or the 2500-char snippet window truncating before
// the array). When that happens, keysToFlags(undefined) returns {}; if we
// kept it, Twitter would receive an empty `features` map and respond 400,
// surfacing a misleading "queryId expired" error.
return {
queryId: sanitizeQueryId(value?.queryId, normalizedFallback.queryId),
features: value?.features
&& typeof value.features === 'object'
&& Object.keys(value.features).length > 0
? value.features
: normalizedFallback.features,
fieldToggles: value?.fieldToggles
&& typeof value.fieldToggles === 'object'
&& Object.keys(value.fieldToggles).length > 0
? value.fieldToggles
: normalizedFallback.fieldToggles,
};
}
export async function resolveTwitterOperationMetadata(page, operationName, fallback) {
const resolved = await page.evaluate(`async () => {
const operationName = ${JSON.stringify(operationName)};
const keysToFlags = (keys) => Object.fromEntries((keys || []).map((key) => [key, true]));
const quotedKeys = (source) => source
? Array.from(source.matchAll(/"([^"]+)"/g)).map((match) => match[1])
: [];
const parseOperation = (text) => {
const marker = 'operationName:"' + operationName + '"';
const index = text.indexOf(marker);
if (index < 0) return null;
const start = Math.max(0, text.lastIndexOf('e.exports=', index));
const endMarker = text.indexOf('}}}', index);
const snippet = text.slice(start, endMarker > index ? endMarker + 3 : index + 2500);
const queryId = snippet.match(/queryId:"([A-Za-z0-9_-]+)"/)?.[1] || null;
if (!queryId) return null;
return {
queryId,
features: keysToFlags(quotedKeys(snippet.match(/featureSwitches:\\[([^\\]]*)\\]/)?.[1])),
fieldToggles: keysToFlags(quotedKeys(snippet.match(/fieldToggles:\\[([^\\]]*)\\]/)?.[1])),
};
};
try {
const scripts = Array.from(document.scripts)
.map(s => s.src)
.filter(Boolean)
.concat(performance.getEntriesByType('resource')
.map(r => r.name)
.filter(r => r.includes('client-web') && r.endsWith('.js')));
const uniqueScripts = Array.from(new Set(scripts));
for (const scriptUrl of uniqueScripts.slice(-30)) {
try {
const text = await (await fetch(scriptUrl)).text();
const operation = parseOperation(text);
if (operation) return operation;
} catch {}
}
} catch {}
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
try {
@@ -92,27 +241,25 @@ export async function resolveTwitterQueryId(page, operationName, fallbackId) {
if (ghResp.ok) {
const data = await ghResp.json();
const entry = data?.[operationName];
if (entry && entry.queryId) return entry.queryId;
if (entry && entry.queryId) {
return {
queryId: entry.queryId,
features: keysToFlags(entry.featureSwitches),
fieldToggles: keysToFlags(entry.fieldToggles),
};
}
}
} catch {
clearTimeout(timeout);
}
try {
const scripts = performance.getEntriesByType('resource')
.filter(r => r.name.includes('client-web') && r.name.endsWith('.js'))
.map(r => r.name);
for (const scriptUrl of scripts.slice(0, 15)) {
try {
const text = await (await fetch(scriptUrl)).text();
const re = new RegExp('queryId:"([A-Za-z0-9_-]+)"[^}]{0,200}operationName:"' + operationName + '"');
const match = text.match(re);
if (match) return match[1];
} catch {}
}
} catch {}
return null;
}`);
return sanitizeQueryId(resolved, fallbackId);
return sanitizeTwitterOperationMetadata(resolved, fallback);
}
export async function resolveTwitterQueryId(page, operationName, fallbackId) {
const operation = await resolveTwitterOperationMetadata(page, operationName, fallbackId);
return operation.queryId;
}
/**
* Extract media flags and URLs from a tweet's `legacy` object.
@@ -143,6 +290,10 @@ export function extractMedia(legacy) {
}
export const __test__ = {
sanitizeQueryId,
sanitizeTwitterOperationMetadata,
unwrapBrowserResult,
normalizeTwitterGraphqlPayload,
normalizeTwitterScreenName,
extractMedia,
parseTweetUrl,
buildTwitterArticleScopeSource,
+102 -1
View File
@@ -3,7 +3,108 @@ import { JSDOM } from 'jsdom';
import { __test__ } from './shared.js';
import { ArgumentError } from '@jackwener/opencli/errors';
const { extractMedia, parseTweetUrl, buildTwitterArticleScopeSource } = __test__;
const { extractMedia, parseTweetUrl, buildTwitterArticleScopeSource, unwrapBrowserResult, normalizeTwitterGraphqlPayload, normalizeTwitterScreenName, sanitizeTwitterOperationMetadata } = __test__;
describe('twitter browser result helpers', () => {
it('unwraps Browser Bridge exec envelopes', () => {
expect(unwrapBrowserResult({ session: 'site:twitter', data: '123' })).toBe('123');
expect(unwrapBrowserResult({ data: { user: true } })).toEqual({ data: { user: true } });
});
it('sanitizes operation metadata after unwrapping Browser Bridge envelopes', () => {
const result = sanitizeTwitterOperationMetadata({
session: 'site:twitter',
data: {
queryId: 'abc_123',
features: { feature: true },
fieldToggles: { field: true },
},
}, { queryId: 'fallback', features: {}, fieldToggles: {} });
expect(result).toEqual({
queryId: 'abc_123',
features: { feature: true },
fieldToggles: { field: true },
});
});
it('falls back to baked features / fieldToggles when the bundle parser returns empty maps', () => {
// Regression guard: resolveTwitterOperationMetadata's bundle parser can
// find a queryId but miss `featureSwitches:[...]` (e.g. minification
// change, or the 2500-char snippet window truncating before the array).
// In that case keysToFlags(undefined) returns {}; if sanitize kept the
// empty map, Twitter would receive a request with no features and reply
// 400, surfacing a misleading "queryId expired" error.
const result = sanitizeTwitterOperationMetadata({
queryId: 'newQueryId',
features: {},
fieldToggles: {},
}, {
queryId: 'fallback',
features: { fallback_feature: true },
fieldToggles: { fallback_field: true },
});
expect(result).toEqual({
queryId: 'newQueryId',
features: { fallback_feature: true },
fieldToggles: { fallback_field: true },
});
});
it('falls back when resolved features are non-object falsy values', () => {
const result = sanitizeTwitterOperationMetadata({
queryId: 'newQueryId',
features: null,
fieldToggles: undefined,
}, {
queryId: 'fallback',
features: { fallback_feature: true },
fieldToggles: { fallback_field: true },
});
expect(result.features).toEqual({ fallback_feature: true });
expect(result.fieldToggles).toEqual({ fallback_field: true });
});
it('normalizes GraphQL payloads when the bridge strips the top-level data key', () => {
expect(normalizeTwitterGraphqlPayload({ user: { result: {} } })).toEqual({
data: { user: { result: {} } },
});
expect(normalizeTwitterGraphqlPayload({ search_by_raw_query: { search_timeline: {} } })).toEqual({
data: { search_by_raw_query: { search_timeline: {} } },
});
expect(normalizeTwitterGraphqlPayload({ data: { user: {} } })).toEqual({ data: { user: {} } });
});
});
describe('twitter normalizeTwitterScreenName', () => {
it('accepts exact handles and exact Twitter/X profile URLs', () => {
expect(normalizeTwitterScreenName('@viewer')).toBe('viewer');
expect(normalizeTwitterScreenName('/viewer')).toBe('viewer');
expect(normalizeTwitterScreenName('https://x.com/viewer')).toBe('viewer');
expect(normalizeTwitterScreenName('https://twitter.com/viewer?lang=en')).toBe('viewer');
expect(normalizeTwitterScreenName('https://mobile.twitter.com/viewer')).toBe('viewer');
});
it('rejects route collisions, malformed handles, and non-exact profile URLs', () => {
const invalid = [
'/home',
'/viewer/extra',
'viewer/extra',
'viewer?tab=posts',
'https://x.com/home',
'https://x.com/viewer/status/1',
'http://x.com/viewer',
'https://evil.com/viewer',
'https://x.com.evil.com/viewer',
'https://x.com:444/viewer',
'https://user:pass@x.com/viewer',
'bad-handle',
'abcdefghijklmnop',
];
for (const value of invalid) {
expect(normalizeTwitterScreenName(value)).toBe('');
}
});
});
describe('twitter parseTweetUrl', () => {
it('accepts exact Twitter/X tweet URLs and preserves query parameters', () => {
-1
View File
@@ -100,7 +100,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'tweet-id', positional: true, type: 'string', required: true, help: 'Tweet numeric ID (e.g. 1234567890) or full status URL' },
{ name: 'limit', type: 'int', default: 50 },
+3 -2
View File
@@ -5,6 +5,7 @@ import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
// ── Twitter GraphQL constants ──────────────────────────────────────────
const HOME_TIMELINE_QUERY_ID = 'c-CzHF1LboFilMpsx4ZCrQ';
const HOME_LATEST_TIMELINE_QUERY_ID = 'BKB7oi212Fi7kQtCBGE4zA';
const MAX_PAGINATION_PAGES = 100;
// Endpoint config: for-you uses GET HomeTimeline, following uses POST HomeLatestTimeline
const TIMELINE_ENDPOINTS = {
'for-you': { endpoint: 'HomeTimeline', method: 'GET', fallbackQueryId: HOME_TIMELINE_QUERY_ID },
@@ -141,7 +142,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{
name: 'type',
@@ -176,7 +176,8 @@ cli({
const allTweets = [];
const seen = new Set();
let cursor = null;
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
const fetchCount = Math.min(40, limit - allTweets.length + 5); // over-fetch slightly for promoted filtering
const variables = buildTimelineVariables(timelineType, fetchCount, cursor);
const apiUrl = buildHomeTimelineUrl(queryId, endpoint, variables);
-1
View File
@@ -17,7 +17,6 @@ cli({
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'limit', type: 'int', default: 20, help: 'Number of trends to show' },
],
+147 -52
View File
@@ -1,15 +1,19 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { resolveTwitterQueryId, sanitizeQueryId, extractMedia } from './shared.js';
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { resolveTwitterOperationMetadata, sanitizeQueryId, extractMedia, normalizeTwitterGraphqlPayload, unwrapBrowserResult } from './shared.js';
import { normalizeTwitterScreenName } from './shared.js';
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
const USER_TWEETS_QUERY_ID = '6fWQaBPK51aGyC_VC7t9GQ';
const USER_TWEETS_QUERY_ID = 'lrMzG9qPQHpqJdP3AbM-bQ';
const USER_BY_SCREEN_NAME_QUERY_ID = 'IGgvgiOx4QZndDHuD3x9TQ';
const MAX_PAGINATION_PAGES = 100;
const USER_TWEETS_FEATURES = {
rweb_video_screen_enabled: false,
rweb_video_screen_enabled: true,
rweb_cashtags_enabled: true,
payments_enabled: false,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
@@ -20,6 +24,7 @@ const USER_TWEETS_FEATURES = {
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: true,
rweb_cashtags_composer_attachment_enabled: true,
responsive_web_jetfuel_frame: true,
responsive_web_grok_share_attachment_enabled: true,
responsive_web_grok_annotations_enabled: true,
@@ -46,8 +51,21 @@ const USER_TWEETS_FEATURES = {
responsive_web_enhance_cards_enabled: false,
};
const USER_TWEETS_FIELD_TOGGLES = {
withPayments: true,
withAuxiliaryUserLabels: true,
withArticleRichContentState: true,
withArticlePlainText: true,
withArticleSummaryText: true,
withArticleVoiceOver: true,
withGrokAnalyze: true,
withDisallowedReplyControls: true,
};
const USER_BY_SCREEN_NAME_FEATURES = {
hidden_profile_subscriptions_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
verified_phone_label_enabled: false,
@@ -61,7 +79,59 @@ const USER_BY_SCREEN_NAME_FEATURES = {
responsive_web_graphql_timeline_navigation_enabled: true,
};
function buildUserTweetsUrl(queryId, userId, count, cursor) {
const USER_BY_SCREEN_NAME_FIELD_TOGGLES = {
withPayments: true,
withAuxiliaryUserLabels: true,
};
const USER_TWEETS_OPERATION = {
queryId: USER_TWEETS_QUERY_ID,
features: USER_TWEETS_FEATURES,
fieldToggles: USER_TWEETS_FIELD_TOGGLES,
};
const USER_BY_SCREEN_NAME_OPERATION = {
queryId: USER_BY_SCREEN_NAME_QUERY_ID,
features: USER_BY_SCREEN_NAME_FEATURES,
fieldToggles: USER_BY_SCREEN_NAME_FIELD_TOGGLES,
};
function normalizeUserTweetsOperation(operation) {
if (typeof operation === 'string') {
return { queryId: operation, features: USER_TWEETS_FEATURES, fieldToggles: USER_TWEETS_FIELD_TOGGLES };
}
return {
queryId: operation?.queryId || USER_TWEETS_QUERY_ID,
features: operation?.features || USER_TWEETS_FEATURES,
fieldToggles: operation?.fieldToggles || USER_TWEETS_FIELD_TOGGLES,
};
}
function normalizeUserByScreenNameOperation(operation) {
if (typeof operation === 'string') {
return { queryId: operation, features: USER_BY_SCREEN_NAME_FEATURES, fieldToggles: USER_BY_SCREEN_NAME_FIELD_TOGGLES };
}
return {
queryId: operation?.queryId || USER_BY_SCREEN_NAME_QUERY_ID,
features: operation?.features || USER_BY_SCREEN_NAME_FEATURES,
fieldToggles: operation?.fieldToggles || USER_BY_SCREEN_NAME_FIELD_TOGGLES,
};
}
function appendGraphqlParams(path, variables, operation) {
const fieldToggles = operation.fieldToggles || {};
const params = [
`variables=${encodeURIComponent(JSON.stringify(variables))}`,
`features=${encodeURIComponent(JSON.stringify(operation.features || {}))}`,
];
if (Object.keys(fieldToggles).length > 0) {
params.push(`fieldToggles=${encodeURIComponent(JSON.stringify(fieldToggles))}`);
}
return `${path}?${params.join('&')}`;
}
function buildUserTweetsUrl(operation, userId, count, cursor) {
const normalized = normalizeUserTweetsOperation(operation);
const vars = {
userId,
count,
@@ -70,21 +140,20 @@ function buildUserTweetsUrl(queryId, userId, count, cursor) {
withVoice: true,
};
if (cursor) vars.cursor = cursor;
return `/i/api/graphql/${queryId}/UserTweets`
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
+ `&features=${encodeURIComponent(JSON.stringify(USER_TWEETS_FEATURES))}`;
return appendGraphqlParams(`/i/api/graphql/${normalized.queryId}/UserTweets`, vars, normalized);
}
function buildUserByScreenNameUrl(queryId, screenName) {
function buildUserByScreenNameUrl(operation, screenName) {
const normalized = normalizeUserByScreenNameOperation(operation);
const vars = { screen_name: screenName, withSafetyModeUserFields: true };
return `/i/api/graphql/${queryId}/UserByScreenName`
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
+ `&features=${encodeURIComponent(JSON.stringify(USER_BY_SCREEN_NAME_FEATURES))}`;
return appendGraphqlParams(`/i/api/graphql/${normalized.queryId}/UserByScreenName`, vars, normalized);
}
function extractTweet(result, seen) {
if (!result) return null;
const tw = result.tweet || result;
const tw = result.__typename === 'TweetWithVisibilityResults' && result.tweet
? result.tweet
: (result.tweet || result);
const legacy = tw.legacy || {};
if (!tw.rest_id || seen.has(tw.rest_id)) return null;
seen.add(tw.rest_id);
@@ -112,32 +181,35 @@ function extractTweet(result, seen) {
function parseUserTweets(data, seen) {
const tweets = [];
let nextCursor = null;
const instructions = data?.data?.user?.result?.timeline_v2?.timeline?.instructions
|| data?.data?.user?.result?.timeline?.timeline?.instructions
|| [];
for (const inst of instructions) {
if (inst.type === 'TimelinePinEntry') continue;
for (const entry of inst.entries || []) {
const content = entry.content;
if (content?.entryType === 'TimelineTimelineCursor' || content?.__typename === 'TimelineTimelineCursor') {
if (content.cursorType === 'Bottom' || content.cursorType === 'ShowMore') nextCursor = content.value;
continue;
}
if (entry.entryId?.startsWith('cursor-bottom-') || entry.entryId?.startsWith('cursor-showMore-')) {
nextCursor = content?.value || content?.itemContent?.value || nextCursor;
continue;
}
const direct = extractTweet(content?.itemContent?.tweet_results?.result, seen);
if (direct) {
tweets.push(direct);
continue;
}
for (const item of content?.items || []) {
const nested = extractTweet(item.item?.itemContent?.tweet_results?.result, seen);
if (nested) tweets.push(nested);
}
const result = data?.data?.user?.result || {};
const instructionSets = [
result.timeline_v2?.timeline?.instructions,
result.timeline?.timeline?.instructions,
].filter(Array.isArray);
const instructions = instructionSets.flat();
const visit = (value) => {
if (!value || typeof value !== 'object') return;
if (value.type === 'TimelinePinEntry') return;
if (value.tweet_results?.result) {
const tweet = extractTweet(value.tweet_results.result, seen);
if (tweet) tweets.push(tweet);
}
}
if (
(value.entryType === 'TimelineTimelineCursor' || value.__typename === 'TimelineTimelineCursor')
&& (value.cursorType === 'Bottom' || value.cursorType === 'ShowMore')
&& value.value
) {
nextCursor = value.value;
}
if (Array.isArray(value)) {
for (const item of value) visit(item);
return;
}
for (const child of Object.values(value)) {
if (child && typeof child === 'object') visit(child);
}
};
visit(instructions);
return { tweets, nextCursor };
}
@@ -145,28 +217,50 @@ cli({
site: 'twitter',
name: 'tweets',
access: 'read',
description: "Fetch a Twitter user's most recent tweets (chronological, excludes pinned)",
description: "Fetch a Twitter user's most recent tweets (chronological, excludes pinned; defaults to the logged-in user when no username is given)",
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
siteSession: 'persistent',
args: [
{ name: 'username', type: 'string', positional: true, required: true, help: 'Twitter screen name (with or without @)' },
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (with or without @). Defaults to the logged-in user when omitted.' },
{ name: 'limit', type: 'int', default: 20, help: 'Max tweets to return' },
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the tweets by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the chronological ordering.' },
],
columns: ['id', 'author', 'created_at', 'is_retweet', 'text', 'likes', 'retweets', 'replies', 'views', 'url', 'has_media', 'media_urls'],
func: async (page, kwargs) => {
const limit = Math.max(1, Math.min(200, kwargs.limit || 20));
const username = String(kwargs.username || '').replace(/^@/, '').trim();
if (!username) throw new CommandExecutionError('username is required');
const rawUsername = String(kwargs.username ?? '').trim();
let username = normalizeTwitterScreenName(rawUsername);
if (rawUsername && !username) {
throw new ArgumentError('twitter tweets username must be a valid Twitter/X handle', 'Example: opencli twitter tweets @jack --limit 20');
}
// When no username is given, detect the logged-in user (own tweets).
// Mirrors the self-detection pattern used by twitter/profile and
// twitter/likes so agents can pull own-account data without having
// to know their own screen name up front.
if (!username) {
await page.goto('https://x.com/home');
await page.wait({ selector: '[data-testid="primaryColumn"]' });
// Bridge wraps primitive page.evaluate returns as { session, data:<value> }.
// unwrapBrowserResult drops that envelope so the href string is usable.
const href = unwrapBrowserResult(await page.evaluate(`() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
}`));
if (!href || typeof href !== 'string')
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
username = normalizeTwitterScreenName(href);
if (!username) {
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
}
}
const cookies = await page.getCookies({ url: 'https://x.com' });
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
if (!ct0) throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
const userTweetsQueryId = await resolveTwitterQueryId(page, 'UserTweets', USER_TWEETS_QUERY_ID);
const userByScreenNameQueryId = await resolveTwitterQueryId(page, 'UserByScreenName', USER_BY_SCREEN_NAME_QUERY_ID);
const userTweetsOperation = await resolveTwitterOperationMetadata(page, 'UserTweets', USER_TWEETS_OPERATION);
const userByScreenNameOperation = await resolveTwitterOperationMetadata(page, 'UserByScreenName', USER_BY_SCREEN_NAME_OPERATION);
const headers = JSON.stringify({
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
@@ -175,25 +269,26 @@ cli({
'X-Twitter-Active-User': 'yes',
});
const ubsUrl = buildUserByScreenNameUrl(userByScreenNameQueryId, username);
const userId = await page.evaluate(`async () => {
const ubsUrl = buildUserByScreenNameUrl(userByScreenNameOperation, username);
const userId = unwrapBrowserResult(await page.evaluate(`async () => {
const resp = await fetch("${ubsUrl}", { headers: ${headers}, credentials: 'include' });
if (!resp.ok) return null;
const d = await resp.json();
return d?.data?.user?.result?.rest_id || null;
}`);
}`));
if (!userId) throw new CommandExecutionError(`Could not resolve @${username}`);
const seen = new Set();
const all = [];
let cursor = null;
for (let i = 0; i < 5 && all.length < limit; i++) {
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
for (let i = 0; i < MAX_PAGINATION_PAGES && all.length < limit; i++) {
const fetchCount = Math.min(100, limit - all.length + 10);
const url = buildUserTweetsUrl(userTweetsQueryId, userId, fetchCount, cursor);
const data = await page.evaluate(`async () => {
const url = buildUserTweetsUrl(userTweetsOperation, userId, fetchCount, cursor);
const data = normalizeTwitterGraphqlPayload(await page.evaluate(`async () => {
const r = await fetch("${url}", { headers: ${headers}, credentials: 'include' });
return r.ok ? await r.json() : { error: r.status };
}`);
}`));
if (data?.error) {
if (all.length === 0) throw new CommandExecutionError(`HTTP ${data.error}: UserTweets fetch failed — queryId may have expired`);
break;
+238 -1
View File
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest';
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
import { __test__ } from './tweets.js';
describe('twitter tweets helpers', () => {
@@ -8,6 +9,140 @@ describe('twitter tweets helpers', () => {
expect(cmd?.columns).toEqual(['id', 'author', 'created_at', 'is_retweet', 'text', 'likes', 'retweets', 'replies', 'views', 'url', 'has_media', 'media_urls']);
});
it('makes the username argument optional so it can default to the logged-in user', () => {
const cmd = getRegistry().get('twitter/tweets');
const usernameArg = cmd?.args?.find((arg) => arg.name === 'username');
expect(usernameArg).toBeDefined();
expect(usernameArg?.required).not.toBe(true);
expect(usernameArg?.help || '').toMatch(/default/i);
expect(cmd?.description || '').toMatch(/default/i);
});
it('detects the logged-in user via AppTabBar_Profile_Link when no username is given', async () => {
const cmd = getRegistry().get('twitter/tweets');
const evaluatedScripts = [];
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
evaluate: vi.fn(async (script) => {
const text = typeof script === 'function' ? script.toString() : String(script);
evaluatedScripts.push(text);
if (text.includes('AppTabBar_Profile_Link')) return '/viewer';
if (text.includes('operationName')) return null; // operation metadata resolver
if (text.includes('/UserByScreenName')) return '42';
if (text.includes('/UserTweets')) {
return {
data: {
user: {
result: {
timeline_v2: {
timeline: {
instructions: [
{
entries: [
{
entryId: 'tweet-1',
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '1',
legacy: {
full_text: 'own post',
favorite_count: 0,
retweet_count: 0,
reply_count: 0,
created_at: 'now',
},
core: {
user_results: {
result: {
legacy: { screen_name: 'viewer', name: 'Viewer' },
},
},
},
},
},
},
},
},
],
},
],
},
},
},
},
},
};
}
return null;
}),
};
const rows = await cmd.func(page, { limit: 1 });
// Navigated home to read the logged-in user
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
// AppTabBar_Profile_Link probe happened before any GraphQL fetch
const probeIdx = evaluatedScripts.findIndex((t) => t.includes('AppTabBar_Profile_Link'));
const graphqlIdx = evaluatedScripts.findIndex((t) => t.includes('/UserByScreenName'));
expect(probeIdx).toBeGreaterThanOrEqual(0);
expect(graphqlIdx).toBeGreaterThan(probeIdx);
// The detected handle ('viewer') was used for the UserByScreenName lookup
const lookup = evaluatedScripts.find((t) => t.includes('/UserByScreenName')) || '';
expect(decodeURIComponent(lookup)).toContain('"screen_name":"viewer"');
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({ id: '1', author: 'viewer', url: 'https://x.com/viewer/status/1' });
});
it('throws AuthRequiredError when no username is given and the logged-in user cannot be detected', async () => {
const cmd = getRegistry().get('twitter/tweets');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => []),
evaluate: vi.fn(async (script) => {
const text = typeof script === 'function' ? script.toString() : String(script);
if (text.includes('AppTabBar_Profile_Link')) return null;
return null;
}),
};
await expect(cmd.func(page, {})).rejects.toBeInstanceOf(AuthRequiredError);
});
it('rejects invalid explicit username before navigation', async () => {
const cmd = getRegistry().get('twitter/tweets');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
evaluate: vi.fn(),
};
await expect(cmd.func(page, { username: 'viewer/extra' })).rejects.toBeInstanceOf(ArgumentError);
expect(page.goto).not.toHaveBeenCalled();
expect(page.getCookies).not.toHaveBeenCalled();
expect(page.evaluate).not.toHaveBeenCalled();
});
it('rejects non-profile AppTabBar hrefs instead of querying route names as users', async () => {
const cmd = getRegistry().get('twitter/tweets');
const page = {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
evaluate: vi.fn(async (script) => {
const text = typeof script === 'function' ? script.toString() : String(script);
if (text.includes('AppTabBar_Profile_Link')) return '/home';
throw new Error(`Unexpected evaluate: ${text.slice(0, 80)}`);
}),
};
await expect(cmd.func(page, {})).rejects.toBeInstanceOf(AuthRequiredError);
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
expect(page.evaluate).toHaveBeenCalledTimes(1);
});
it('falls back when queryId contains unsafe characters', () => {
expect(__test__.sanitizeQueryId('safe_Query-123', 'fallback')).toBe('safe_Query-123');
expect(__test__.sanitizeQueryId('bad"id', 'fallback')).toBe('fallback');
@@ -60,6 +195,18 @@ describe('twitter tweets helpers', () => {
expect(b.is_retweet).toBe(true);
});
it('unwraps TweetWithVisibilityResults', () => {
const tweet = __test__.extractTweet({
__typename: 'TweetWithVisibilityResults',
tweet: {
rest_id: '42',
legacy: { full_text: 'visible post', favorite_count: 2, retweet_count: 0, reply_count: 0, created_at: 'now' },
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
},
}, new Set());
expect(tweet).toMatchObject({ id: '42', author: 'alice', text: 'visible post' });
});
it('parses chronological tweets and skips pinned instruction', () => {
const chronEntry = {
entryId: 'tweet-1',
@@ -122,4 +269,94 @@ describe('twitter tweets helpers', () => {
url: 'https://x.com/alice/status/1',
});
});
it('recursively parses tweets nested in timeline modules', () => {
const payload = {
data: {
user: {
result: {
timeline_v2: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'profile-conversation-1',
content: {
entryType: 'TimelineTimelineModule',
items: [
{
item: {
itemContent: {
tweet_results: {
result: {
rest_id: '2',
legacy: { full_text: 'nested post', favorite_count: 1, retweet_count: 0, reply_count: 0, created_at: 'now' },
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
},
},
},
},
},
],
},
},
{
entryId: 'cursor-bottom-2',
content: { entryType: 'TimelineTimelineCursor', cursorType: 'Bottom', value: 'next' },
},
],
},
],
},
},
},
},
},
};
const result = __test__.parseUserTweets(payload, new Set());
expect(result.nextCursor).toBe('next');
expect(result.tweets).toHaveLength(1);
expect(result.tweets[0]).toMatchObject({ id: '2', text: 'nested post' });
});
it('uses populated timeline instructions when timeline_v2 is present but empty', () => {
const payload = {
data: {
user: {
result: {
timeline_v2: { timeline: { instructions: [] } },
timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
content: {
itemContent: {
tweet_results: {
result: {
rest_id: '3',
legacy: { full_text: 'fallback timeline post', favorite_count: 0, retweet_count: 0, reply_count: 0, created_at: 'now' },
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
},
},
},
},
},
],
},
],
},
},
},
},
},
};
const result = __test__.parseUserTweets(payload, new Set());
expect(result.tweets).toHaveLength(1);
expect(result.tweets[0]).toMatchObject({ id: '3', text: 'fallback timeline post' });
});
});
+3 -4
View File
@@ -2,6 +2,7 @@
* Weibo comments — get comments on a post.
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
cli({
site: 'weibo',
name: 'comments',
@@ -19,7 +20,7 @@ cli({
await page.goto('https://weibo.com');
await page.wait(2);
const id = String(kwargs.id);
const data = await page.evaluate(`
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(async () => {
const id = ${JSON.stringify(id)};
const count = ${count};
@@ -46,9 +47,7 @@ cli({
return item;
});
})()
`);
if (!Array.isArray(data))
return [];
`)), 'weibo comments');
return data;
},
});
+85
View File
@@ -0,0 +1,85 @@
import { describe, expect, it, vi } from 'vitest';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { getRegistry } from '@jackwener/opencli/registry';
import './comments.js';
import './favorites.js';
import './feed.js';
import './hot.js';
import './me.js';
import './post.js';
import './search.js';
import './user.js';
function envelope(data) {
return { session: 'site:weibo:test', data };
}
function makePage(evaluateResults = []) {
const queue = [...evaluateResults];
return {
goto: vi.fn().mockResolvedValue(undefined),
wait: vi.fn().mockResolvedValue(undefined),
evaluate: vi.fn(async (script) => {
if (String(script).includes('window.scrollBy')) return undefined;
return queue.length ? queue.shift() : undefined;
}),
};
}
describe('weibo read adapters Browser Bridge envelopes', () => {
it('unwraps comments, feed, hot, search, and favorites array payloads', async () => {
await expect(getRegistry().get('weibo/comments').func(
makePage([envelope([{ rank: 1, author: 'a', text: 't', likes: 0, replies: 0, time: '' }])]),
{ id: '123', limit: 1 },
)).resolves.toHaveLength(1);
await expect(getRegistry().get('weibo/feed').func(
makePage([envelope('123456'), envelope([{ id: 'm1', author: 'a', text: 't', reposts: 0, comments: 0, likes: 0, time: '', url: 'https://weibo.com/1/m1' }])]),
{ type: 'for-you', limit: 1 },
)).resolves.toHaveLength(1);
await expect(getRegistry().get('weibo/hot').func(
makePage([envelope([{ rank: 1, word: 'opencli', hot_value: 1, category: '', label: '', url: 'https://s.weibo.com/weibo?q=opencli' }])]),
{ limit: 1 },
)).resolves.toHaveLength(1);
await expect(getRegistry().get('weibo/search').func(
makePage([envelope([{ id: 'm1', title: 'OpenCLI', author: 'a', time: '', url: 'https://weibo.com/1/m1' }])]),
{ keyword: 'opencli', limit: 1 },
)).resolves.toEqual([{ rank: 1, id: 'm1', title: 'OpenCLI', author: 'a', time: '', url: 'https://weibo.com/1/m1' }]);
await expect(getRegistry().get('weibo/favorites').func(
makePage([envelope('123456'), envelope([{ text: '作者A\n这是一条收藏微博', url: 'https://weibo.com/123/AbCd1' }])]),
{ limit: 1 },
)).resolves.toHaveLength(1);
});
it('unwraps me, post, and user object payloads', async () => {
await expect(getRegistry().get('weibo/me').func(
makePage([envelope('123456'), envelope({ screen_name: 'me', uid: '123456' })]),
{},
)).resolves.toMatchObject({ screen_name: 'me', uid: '123456' });
await expect(getRegistry().get('weibo/post').func(
makePage([envelope({ id: '1', text: 'post' })]),
{ id: '1' },
)).resolves.toContainEqual({ field: 'text', value: 'post' });
await expect(getRegistry().get('weibo/user').func(
makePage([envelope({ screen_name: 'alice', uid: '42' })]),
{ id: '42' },
)).resolves.toMatchObject({ screen_name: 'alice', uid: '42' });
});
it('fails typed instead of returning empty rows for malformed post-unwrap payloads', async () => {
await expect(getRegistry().get('weibo/hot').func(
makePage([envelope({ error: 'API error' })]),
{ limit: 1 },
)).rejects.toBeInstanceOf(CommandExecutionError);
await expect(getRegistry().get('weibo/user').func(
makePage([envelope([{ screen_name: 'wrong shape' }])]),
{ id: '42' },
)).rejects.toBeInstanceOf(CommandExecutionError);
});
});
+4 -4
View File
@@ -1,6 +1,6 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import { ArgumentError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { getSelfUid } from './utils.js';
import { getSelfUid, requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
const DEFAULT_LIMIT = 20;
const MAX_LIMIT = 50;
@@ -123,7 +123,7 @@ cli({
await page.wait(1);
}
const rawData = await page.evaluate(`
const rawData = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(() => {
const scrollers = document.querySelectorAll('.wbpro-scroller-item, .vue-recycle-scroller__item-view');
const out = [];
@@ -145,9 +145,9 @@ cli({
}
return out;
})()
`);
`)), 'weibo favorites');
if (!Array.isArray(rawData) || rawData.length === 0) {
if (rawData.length === 0) {
throw new EmptyResultError('weibo favorites', 'No favorites were visible on the favorites page');
}
+3 -5
View File
@@ -2,7 +2,7 @@
* Weibo feed — for-you or following timeline.
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { getSelfUid } from './utils.js';
import { getSelfUid, requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
const TIMELINE_ENDPOINTS = {
'for-you': 'unreadfriendstimeline',
following: 'friendstimeline',
@@ -31,7 +31,7 @@ cli({
await page.goto('https://weibo.com');
await page.wait(2);
const uid = await getSelfUid(page);
const data = await page.evaluate(`
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(async () => {
const uid = ${JSON.stringify(uid)};
const count = ${count};
@@ -63,9 +63,7 @@ cli({
return item;
});
})()
`);
if (!Array.isArray(data))
return [];
`)), 'weibo feed');
return data;
},
});
+3 -4
View File
@@ -2,6 +2,7 @@
* Weibo hot search — browser cookie API.
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
cli({
site: 'weibo',
name: 'hot',
@@ -16,7 +17,7 @@ cli({
func: async (page, kwargs) => {
const count = Math.min(kwargs.limit || 30, 50);
await page.goto('https://weibo.com');
const data = await page.evaluate(`
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(async () => {
const resp = await fetch('/ajax/statuses/hot_band', {credentials: 'include'});
if (!resp.ok) return {error: 'HTTP ' + resp.status};
@@ -32,9 +33,7 @@ cli({
url: 'https://s.weibo.com/weibo?q=' + encodeURIComponent('#' + item.word + '#')
}));
})()
`);
if (!Array.isArray(data))
return [];
`)), 'weibo hot');
return data.slice(0, count);
},
});
+3 -5
View File
@@ -3,7 +3,7 @@
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { getSelfUid } from './utils.js';
import { getSelfUid, requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
cli({
site: 'weibo',
name: 'me',
@@ -17,7 +17,7 @@ cli({
await page.goto('https://weibo.com');
await page.wait(2);
const uid = await getSelfUid(page);
const data = await page.evaluate(`
const data = requireObjectEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(async () => {
const uid = ${JSON.stringify(uid)};
@@ -67,9 +67,7 @@ cli({
profile_url: 'https://weibo.com' + (p.profile_url || '/u/' + p.id),
};
})()
`);
if (!data || typeof data !== 'object')
throw new CommandExecutionError('Failed to fetch profile');
`)), 'weibo me');
if (data.error)
throw new CommandExecutionError(String(data.error));
return data;
+3 -4
View File
@@ -3,6 +3,7 @@
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
cli({
site: 'weibo',
name: 'post',
@@ -18,7 +19,7 @@ cli({
await page.goto('https://weibo.com');
await page.wait(2);
const id = String(kwargs.id);
const data = await page.evaluate(`
const data = requireObjectEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(async () => {
const id = ${JSON.stringify(id)};
const strip = (html) => (html || '').replace(/<[^>]+>/g, '').replace(/&nbsp;/g, ' ').replace(/&lt;/g, '<').replace(/&gt;/g, '>').replace(/&amp;/g, '&').trim();
@@ -63,9 +64,7 @@ cli({
return result;
})()
`);
if (!data || typeof data !== 'object')
throw new CommandExecutionError('Failed to fetch post');
`)), 'weibo post');
if (data.error)
throw new CommandExecutionError(String(data.error));
return Object.entries(data).map(([field, value]) => ({
+4 -3
View File
@@ -3,6 +3,7 @@
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import { requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
cli({
site: 'weibo',
name: 'search',
@@ -21,7 +22,7 @@ cli({
const keyword = encodeURIComponent(String(kwargs.keyword ?? '').trim());
await page.goto(`https://s.weibo.com/weibo?q=${keyword}`);
await page.wait(2);
const data = await page.evaluate(`
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(() => {
const clean = (value) => (value || '').replace(/\\s+/g, ' ').trim();
const absoluteUrl = (href) => {
@@ -67,8 +68,8 @@ cli({
return rows;
})()
`);
if (!Array.isArray(data) || data.length === 0) {
`)), 'weibo search');
if (data.length === 0) {
throw new CliError('NOT_FOUND', 'No Weibo search results found', 'Try a different keyword or ensure you are logged into weibo.com');
}
return data.slice(0, limit).map((item, index) => ({
+3 -4
View File
@@ -3,6 +3,7 @@
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
cli({
site: 'weibo',
name: 'user',
@@ -18,7 +19,7 @@ cli({
await page.goto('https://weibo.com');
await page.wait(2);
const id = String(kwargs.id);
const data = await page.evaluate(`
const data = requireObjectEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
(async () => {
const id = ${JSON.stringify(id)};
const isUid = /^\\d+$/.test(id);
@@ -54,9 +55,7 @@ cli({
ip_location: d.ip_location || '',
};
})()
`);
if (!data || typeof data !== 'object')
throw new CommandExecutionError('Failed to fetch user profile');
`)), 'weibo user');
if (data.error)
throw new CommandExecutionError(String(data.error));
return data;
+34 -5
View File
@@ -1,10 +1,39 @@
/**
* Shared Weibo utilities — uid extraction.
*/
import { AuthRequiredError } from '@jackwener/opencli/errors';
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
/**
* `page.evaluate` may return either the raw IIFE value or a
* `{ session, data }` envelope depending on the browser-bridge version.
* Adapter code that inspected the payload directly (e.g. `Array.isArray`,
* truthiness checks on uid strings) silently received the envelope wrapper
* instead of the inner value. This helper normalizes both shapes so callers
* can keep their existing checks unchanged.
*/
export function unwrapEvaluateResult(payload) {
if (payload && !Array.isArray(payload) && typeof payload === 'object' && 'session' in payload && 'data' in payload) {
return payload.data;
}
return payload;
}
export function requireArrayEvaluateResult(payload, label) {
if (!Array.isArray(payload)) {
if (payload && typeof payload === 'object' && 'error' in payload) {
throw new CommandExecutionError(`${label}: ${String(payload.error)}`);
}
throw new CommandExecutionError(`${label} returned malformed extraction payload`);
}
return payload;
}
export function requireObjectEvaluateResult(payload, label) {
if (!payload || Array.isArray(payload) || typeof payload !== 'object') {
throw new CommandExecutionError(`${label} returned malformed extraction payload`);
}
return payload;
}
/** Get the currently logged-in user's uid from Vue store or config API. */
export async function getSelfUid(page) {
const uid = await page.evaluate(`
const uid = unwrapEvaluateResult(await page.evaluate(`
(() => {
const app = document.querySelector('#app')?.__vue_app__;
const store = app?.config?.globalProperties?.$store;
@@ -12,18 +41,18 @@ export async function getSelfUid(page) {
if (uid) return String(uid);
return null;
})()
`);
`));
if (uid)
return uid;
// Fallback: config API
const config = await page.evaluate(`
const config = unwrapEvaluateResult(await page.evaluate(`
(async () => {
const resp = await fetch('/ajax/config/get_config', {credentials: 'include'});
if (!resp.ok) return null;
const data = await resp.json();
return data.ok && data.data?.uid ? String(data.data.uid) : null;
})()
`);
`));
if (config)
return config;
throw new AuthRequiredError('weibo.com');
+36
View File
@@ -0,0 +1,36 @@
import { describe, expect, it } from 'vitest';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { requireArrayEvaluateResult, requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
describe('unwrapEvaluateResult (browser-bridge envelope normalization)', () => {
it('returns the raw array unchanged when payload is already an array', () => {
const arr = [{ id: '1' }, { id: '2' }];
expect(unwrapEvaluateResult(arr)).toBe(arr);
});
it('unwraps { session, data: [...] } envelope to the inner array', () => {
const arr = [{ id: '1' }];
const env = { session: 'site:weibo:abc', data: arr };
expect(unwrapEvaluateResult(env)).toBe(arr);
});
it('unwraps primitive data (e.g. uid string) from Browser Bridge envelopes', () => {
expect(unwrapEvaluateResult({ session: 'site:weibo:abc', data: '1234567890' })).toBe('1234567890');
});
it('unwraps null payload data so getSelfUid fallback can trigger', () => {
expect(unwrapEvaluateResult({ session: 'site:weibo:abc', data: null })).toBe(null);
});
it('passes non-envelope objects through unchanged (e.g. profile result)', () => {
const obj = { screen_name: 'alice', uid: '42' };
expect(unwrapEvaluateResult(obj)).toBe(obj);
});
it('handles null and undefined safely', () => {
expect(unwrapEvaluateResult(null)).toBe(null);
expect(unwrapEvaluateResult(undefined)).toBe(undefined);
});
it('keeps malformed array/object payloads as typed command failures after unwrap', () => {
expect(requireArrayEvaluateResult([{ id: '1' }], 'weibo feed')).toEqual([{ id: '1' }]);
expect(() => requireArrayEvaluateResult({ error: 'API error' }, 'weibo feed')).toThrow(CommandExecutionError);
expect(() => requireArrayEvaluateResult({ error: 'API error' }, 'weibo feed')).toThrow('weibo feed: API error');
expect(requireObjectEvaluateResult({ uid: '42' }, 'weibo me')).toEqual({ uid: '42' });
expect(() => requireObjectEvaluateResult([{ uid: '42' }], 'weibo me')).toThrow(CommandExecutionError);
});
});
+23 -2
View File
@@ -14,12 +14,33 @@ export function parseCommentLimit(raw, fallback = 20) {
return fallback;
return Math.max(1, Math.min(Math.floor(n), 50));
}
export function parseXhsLikeCountText(value) {
const integerRe = /^(?:\d+|\d{1,3}(?:[,]\d{3})+)\+?$/u;
const shortformRe = /^((?:\d+|\d{1,3}(?:[,]\d{3})+)(?:\.\d+)?)([wWkK万千])\+?$/u;
const raw = String(value ?? '').replace(/\s+/g, '');
if (!raw)
return 0;
if (integerRe.test(raw))
return Number(raw.replace(/[,+]/g, ''));
const short = raw.match(shortformRe);
if (!short)
return 0;
const numeric = Number(short[1].replace(/[,]/g, ''));
if (!Number.isFinite(numeric))
return 0;
const unit = short[2].toLowerCase();
const multiplier = unit === 'w' || unit === '万' ? 10000 : 1000;
return Math.round(numeric * multiplier);
}
/**
* Host-agnostic IIFE that scrolls a note's comment list and extracts
* top-level comments (and optionally nested 楼中楼 replies). Exported so
* the rednote adapter can reuse the exact same selector chain.
*/
export function buildCommentsExtractJs(withReplies) {
const parseLikeCountText = parseXhsLikeCountText.toString();
return `
(async () => {
const wait = (ms) => new Promise(r => setTimeout(r, ms))
@@ -44,9 +65,9 @@ export function buildCommentsExtractJs(withReplies) {
}
const clean = (el) => (el?.textContent || '').replace(/\\s+/g, ' ').trim()
const parseLikeCountText = ${parseLikeCountText}
const parseLikes = (el) => {
const raw = clean(el)
return /^\\d+$/.test(raw) ? Number(raw) : 0
return parseLikeCountText(clean(el))
}
const expandReplyThreads = async (root) => {
if (!withReplies || !root) return
+63 -1
View File
@@ -1,6 +1,7 @@
import { describe, expect, it, vi } from 'vitest';
import { JSDOM } from 'jsdom';
import { getRegistry } from '@jackwener/opencli/registry';
import './comments.js';
import { buildCommentsExtractJs, parseXhsLikeCountText } from './comments.js';
function createPageMock(evaluateResult) {
return {
goto: vi.fn().mockResolvedValue(undefined),
@@ -25,6 +26,41 @@ function createPageMock(evaluateResult) {
waitForCapture: vi.fn().mockResolvedValue(undefined),
};
}
async function runCommentsExtract(html) {
const dom = new JSDOM(html, { url: 'https://www.xiaohongshu.com/search_result/abc123?xsec_token=tok' });
const previousDocument = globalThis.document;
const previousLocation = globalThis.location;
globalThis.document = dom.window.document;
globalThis.location = dom.window.location;
try {
return await eval(buildCommentsExtractJs(false));
} finally {
globalThis.document = previousDocument;
globalThis.location = previousLocation;
}
}
describe('parseXhsLikeCountText', () => {
it('parses exact integer and shortform like counts', () => {
expect(parseXhsLikeCountText('0')).toBe(0);
expect(parseXhsLikeCountText('42')).toBe(42);
expect(parseXhsLikeCountText('1,234')).toBe(1234);
expect(parseXhsLikeCountText('1234+')).toBe(1234);
expect(parseXhsLikeCountText('2.1w')).toBe(21000);
expect(parseXhsLikeCountText('1.5万')).toBe(15000);
expect(parseXhsLikeCountText('1.2k')).toBe(1200);
expect(parseXhsLikeCountText('3千')).toBe(3000);
expect(parseXhsLikeCountText(' 2.1 w + ')).toBe(21000);
});
it('returns 0 for unknown shapes without overparsing arbitrary text', () => {
for (const raw of ['', null, undefined, '赞', 'likes 2.1w', '2w人', '1,23', '1.2.3k', '.', '1.5']) {
expect(parseXhsLikeCountText(raw)).toBe(0);
}
});
});
describe('xiaohongshu comments', () => {
const command = getRegistry().get('xiaohongshu/comments');
it('returns ranked comment rows for signed full URLs', async () => {
@@ -120,6 +156,32 @@ describe('xiaohongshu comments', () => {
expect(script).toContain("const afterCount = scroller.querySelectorAll('.parent-comment').length");
expect(script).toContain('if (afterCount <= beforeCount) break');
});
it('extracts shortform like counts from the shared xiaohongshu/rednote DOM script', async () => {
const data = await runCommentsExtract(`
<main>
<section class="parent-comment">
<div class="comment-item">
<div class="author-wrapper"><span class="name">Alice</span></div>
<div class="content">Great note</div>
<span class="count">2.1w</span>
<span class="date">today</span>
</div>
</section>
<section class="parent-comment">
<div class="comment-item">
<span class="user-name">Bob</span>
<div class="note-text">Malformed count</div>
<span class="count">likes 2.1w</span>
</div>
</section>
</main>
`);
expect(data.results).toEqual([
{ author: 'Alice', text: 'Great note', likes: 21000, time: 'today', is_reply: false, reply_to: '' },
{ author: 'Bob', text: 'Malformed count', likes: 0, time: '', is_reply: false, reply_to: '' },
]);
});
it('respects the limit for top-level comments', async () => {
const manyComments = Array.from({ length: 10 }, (_, i) => ({
author: `User${i}`,
+87 -25
View File
@@ -6,16 +6,24 @@
* Ref: https://github.com/jackwener/opencli/issues/10
*/
import { cli, Strategy } from '@jackwener/opencli/registry';
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
/**
* Wait for search results or login wall using MutationObserver (max 5s).
* Returns 'content' if note items appeared, 'login_wall' if login gate
* detected, or 'timeout' if neither appeared within the deadline.
*
* Note-item detection tries the legacy `section.note-item` class first
* (still observed in many sessions, including rednote) and falls back to
* a `<section>` element containing a `/search_result/` or `/explore/`
* link. Issue #1506 reports the class being dropped on some xhs renders.
*/
const WAIT_FOR_CONTENT_JS = `
new Promise((resolve) => {
const findNoteCard = () => document.querySelector(
'section.note-item, section:has(a[href*="/search_result/"]), section:has(a[href*="/explore/"])'
);
const detect = () => {
if (document.querySelector('section.note-item')) return 'content';
if (findNoteCard()) return 'content';
if (/登录后查看搜索结果/.test(document.body?.innerText || '')) return 'login_wall';
return null;
};
@@ -52,6 +60,26 @@ export function stripXhsAuthorDateSuffix(value) {
const stripped = text.replace(/\s*(?:\d{1,2}天前|\d+小时前|\d+分钟前|\d+秒前|刚刚|昨天|前天|\d+周前|\d+个月前|\d{1,2}-\d{1,2}|\d{4}-\d{1,2}-\d{1,2})$/u, '').trim();
return stripped || text;
}
/**
* `page.evaluate` may return either the raw IIFE value or a
* `{ session, data }` envelope depending on the browser-bridge version.
* Adapter code that called `Array.isArray(payload)` directly on the
* envelope silently received [] for every search. This helper normalizes
* both shapes so callers can keep their Array.isArray checks unchanged.
*/
export function unwrapEvaluateResult(payload) {
if (payload && !Array.isArray(payload) && typeof payload === 'object' && 'session' in payload && 'data' in payload) {
return payload.data;
}
return payload;
}
function requireSearchRows(payload, phase) {
const rows = unwrapEvaluateResult(payload);
if (!Array.isArray(rows)) {
throw new CommandExecutionError(`Unexpected Xiaohongshu search ${phase} payload shape; expected an array of rows.`);
}
return rows;
}
export function parseLimit(raw) {
const parsed = Number(raw ?? 20);
if (!Number.isFinite(parsed) || !Number.isInteger(parsed)) {
@@ -94,9 +122,22 @@ export function buildScrollUntilJs(targetCount, maxScrolls = 15) {
const style = getComputedStyle(el);
return style.display !== 'none' && style.visibility !== 'hidden';
};
// Note containers: legacy \`section.note-item\` first, fallback to
// any \`<section>\` that wraps a search-result/explore note link
// (#1506 reports the class being dropped on some xhs renders).
const collectNoteCards = () => {
const classMatches = document.querySelectorAll('section.note-item');
if (classMatches.length > 0) return classMatches;
const sections = new Set();
for (const a of document.querySelectorAll('a[href*="/search_result/"], a[href*="/explore/"]')) {
const section = a.closest('section');
if (section) sections.add(section);
}
return sections;
};
const countItems = () => {
let count = 0;
for (const el of document.querySelectorAll('section.note-item')) {
for (const el of collectNoteCards()) {
if (isVisibleNote(el)) count++;
}
return count;
@@ -161,10 +202,24 @@ export function buildSearchExtractJs(webHost) {
const results = [];
const seen = new Set();
document.querySelectorAll('section.note-item').forEach(el => {
// Note containers: legacy \`section.note-item\` first, fallback to any
// \`<section>\` wrapping a search-result/explore link (#1506 reports the
// class being dropped on some xhs renders).
const collectNoteCards = () => {
const classMatches = document.querySelectorAll('section.note-item');
if (classMatches.length > 0) return classMatches;
const sections = new Set();
for (const a of document.querySelectorAll('a[href*="/search_result/"], a[href*="/explore/"]')) {
const section = a.closest('section');
if (section) sections.add(section);
}
return sections;
};
for (const el of collectNoteCards()) {
// Skip "related searches" sections
if (el.classList.contains('query-note-item')) return;
if (!isVisibleNote(el)) return;
if (el.classList?.contains('query-note-item')) continue;
if (!isVisibleNote(el)) continue;
const titleEl = el.querySelector('.title, .note-title, a.title, .footer .title span');
const nameEl = el.querySelector('a.author .name, .author-name, .nick-name, .name');
@@ -184,20 +239,29 @@ export function buildSearchExtractJs(webHost) {
const authorLinkEl = el.querySelector('a.author, a[href*="/user/profile/"]');
const url = normalizeUrl(detailLinkEl?.getAttribute('href') || '');
if (!url) return;
if (!url) continue;
const key = url;
if (seen.has(key)) return;
if (seen.has(key)) continue;
seen.add(key);
// Fallback title: the new bare-section render keeps the note caption
// inside the search_result anchor's first span, not in a class-named
// .title element. Pull from there when the class-based pick is empty.
let title = cleanText(titleEl?.textContent || '');
if (!title) {
const captionSpan = detailLinkEl?.querySelector('span');
title = cleanText(captionSpan?.textContent || '');
}
results.push({
title: cleanText(titleEl?.textContent || ''),
title,
author,
likes: cleanText(likesEl?.textContent || '0'),
url,
author_url: normalizeUrl(authorLinkEl?.getAttribute('href') || ''),
});
});
}
return results;
})()
@@ -223,7 +287,7 @@ export const command = cli({
// Wait for search results to render (or login wall to appear).
// Uses MutationObserver to resolve as soon as content appears,
// instead of a fixed delay + blind retry.
const waitResult = await page.evaluate(WAIT_FOR_CONTENT_JS);
const waitResult = unwrapEvaluateResult(await page.evaluate(WAIT_FOR_CONTENT_JS));
if (waitResult === 'login_wall') {
throw new AuthRequiredError('www.xiaohongshu.com', 'Xiaohongshu search results are blocked behind a login wall');
}
@@ -231,25 +295,23 @@ export const command = cli({
// layout, so scrolling to the bottom can evict the initially visible
// note cards from the DOM and make extraction return [] even though the
// browser rendered results correctly.
const initialPayload = await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com'));
let payload = Array.isArray(initialPayload) ? initialPayload : [];
const initialPayload = requireSearchRows(await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com')), 'initial extraction');
const payload = [...initialPayload];
if (payload.length < limit) {
// Scroll until enough rows are rendered or the lazy-load plateaus.
// Replaces the previous fixed `autoScroll({ times: 2 })` which capped
// extraction at ~13 notes regardless of `--limit` (#1471).
await page.evaluate(buildScrollUntilJs(limit));
const scrolledPayload = await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com'));
if (Array.isArray(scrolledPayload)) {
const seen = new Set(payload.map((item) => item.url).filter(Boolean));
for (const item of scrolledPayload) {
if (item?.url && seen.has(item.url))
continue;
if (item?.url)
seen.add(item.url);
payload.push(item);
if (payload.length >= limit)
break;
}
const scrolledPayload = requireSearchRows(await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com')), 'post-scroll extraction');
const seen = new Set(payload.map((item) => item.url).filter(Boolean));
for (const item of scrolledPayload) {
if (item?.url && seen.has(item.url))
continue;
if (item?.url)
seen.add(item.url);
payload.push(item);
if (payload.length >= limit)
break;
}
}
const data = payload;
+66 -11
View File
@@ -1,7 +1,7 @@
import { describe, expect, it, vi } from 'vitest';
import { getRegistry } from '@jackwener/opencli/registry';
import { JSDOM } from 'jsdom';
import { __test__, buildScrollUntilJs, noteIdToDate } from './search.js';
import { __test__, buildScrollUntilJs, noteIdToDate, unwrapEvaluateResult } from './search.js';
function markVisible(el) {
el.getBoundingClientRect = () => ({ width: 100, height: 100 });
@@ -57,24 +57,37 @@ describe('xiaohongshu search', () => {
expect(page.evaluate).toHaveBeenCalledTimes(1);
expect(page.autoScroll).not.toHaveBeenCalled();
});
it('unwraps a browser-bridge envelope before handling login-wall wait result', async () => {
const cmd = getRegistry().get('xiaohongshu/search');
const page = createPageMock([
{ session: 'site:xiaohongshu', data: 'login_wall' },
]);
await expect(cmd.func(page, { query: '特斯拉', limit: 5 })).rejects.toMatchObject({
code: 'AUTH_REQUIRED',
message: expect.stringContaining('blocked behind a login wall'),
});
expect(page.evaluate).toHaveBeenCalledTimes(1);
});
it('returns ranked results with search_result url and author_url preserved', async () => {
const cmd = getRegistry().get('xiaohongshu/search');
expect(cmd?.func).toBeTypeOf('function');
const detailUrl = 'https://www.xiaohongshu.com/search_result/68e90be80000000004022e66?xsec_token=test-token&xsec_source=';
const authorUrl = 'https://www.xiaohongshu.com/user/profile/635a9c720000000018028b40?xsec_token=user-token&xsec_source=pc_search';
const rows = [
{
title: '某鱼买FSD被坑了4万',
author: '随风',
likes: '261',
url: detailUrl,
author_url: authorUrl,
},
];
const page = createPageMock([
// First evaluate: MutationObserver wait (content appeared)
'content',
// Second evaluate: initial DOM extraction (already enough results)
[
{
title: '某鱼买FSD被坑了4万',
author: '随风',
likes: '261',
url: detailUrl,
author_url: authorUrl,
},
],
// Second evaluate: initial DOM extraction (already enough results) through Browser Bridge envelope.
{ session: 'site:xiaohongshu', data: rows },
]);
const result = await cmd.func(page, { query: '特斯拉', limit: 1 });
// Should only do one goto (the search page itself), no per-note detail navigation
@@ -91,6 +104,18 @@ describe('xiaohongshu search', () => {
},
]);
});
it('fails typed instead of silently returning [] for malformed extraction payloads', async () => {
const cmd = getRegistry().get('xiaohongshu/search');
const page = createPageMock([
'content',
{ session: 'site:xiaohongshu', data: { rows: [] } },
]);
await expect(cmd.func(page, { query: '测试', limit: 1 })).rejects.toMatchObject({
code: 'COMMAND_EXEC',
message: expect.stringContaining('payload shape'),
});
});
it('filters out results with no title and respects the limit', async () => {
const cmd = getRegistry().get('xiaohongshu/search');
expect(cmd?.func).toBeTypeOf('function');
@@ -135,6 +160,10 @@ describe('xiaohongshu search', () => {
'content',
// Second evaluate: initial extraction (no rows rendered)
[],
// Third evaluate: scroll-until row count
0,
// Fourth evaluate: post-scroll extraction (still no rows)
[],
]);
const result = (await cmd.func(page, { query: '测试等待', limit: 5 }));
expect(result).toHaveLength(0);
@@ -268,3 +297,29 @@ describe('noteIdToDate (ObjectID timestamp parsing)', () => {
expect(noteIdToDate('https://www.xiaohongshu.com/search_result/000000000000000000000000')).toBe('');
});
});
describe('unwrapEvaluateResult (browser-bridge envelope normalization)', () => {
it('returns the raw array unchanged when payload is already an array', () => {
const arr = [{ title: 'a' }, { title: 'b' }];
expect(unwrapEvaluateResult(arr)).toBe(arr);
});
it('unwraps { session, data: [...] } envelope to the inner array', () => {
const arr = [{ title: 'a' }];
const env = { session: 'site:xiaohongshu:abc', data: arr };
expect(unwrapEvaluateResult(env)).toBe(arr);
});
it('unwraps primitive data from Browser Bridge envelopes', () => {
expect(unwrapEvaluateResult({ session: 'site:xiaohongshu:abc', data: 'login_wall' })).toBe('login_wall');
});
it('passes non-envelope objects through unchanged', () => {
const obj = { results: [], loginWall: true };
expect(unwrapEvaluateResult(obj)).toBe(obj);
});
it('handles null and undefined safely', () => {
expect(unwrapEvaluateResult(null)).toBe(null);
expect(unwrapEvaluateResult(undefined)).toBe(undefined);
});
it('unwraps non-array envelope data so callers can validate the payload shape', () => {
const env = { session: 'x', data: { not: 'an array' } };
expect(unwrapEvaluateResult(env)).toEqual({ not: 'an array' });
});
});

Some files were not shown because too many files have changed in this diff Show More