Compare commits
40 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| aae6e823b4 | |||
| 3f62cc45bf | |||
| b6f352b318 | |||
| dadf01b56f | |||
| 1239798d04 | |||
| 9ccc896585 | |||
| 1a69f40a80 | |||
| 300607f692 | |||
| 42b5a4e68d | |||
| bccd275d66 | |||
| edfa5f0da3 | |||
| 16b02bcc58 | |||
| 5af2ff1d6c | |||
| 9c25bc7009 | |||
| 8c88a3cbf3 | |||
| 9c4f4a3d30 | |||
| 29c135b656 | |||
| 7edf53783f | |||
| af7b94152f | |||
| 6b26aedd56 | |||
| 68b18cdbcd | |||
| cddc84776c | |||
| 4f5fcd9acb | |||
| 40b2f75098 | |||
| feab24f76c | |||
| 8ef7e903b8 | |||
| 7c5bafd49b | |||
| f66996a148 | |||
| 4fac911425 | |||
| b52da639a3 | |||
| b1dca04ddd | |||
| f481585ba1 | |||
| 723f2b9147 | |||
| 2babed84e9 | |||
| a6ca53c7cf | |||
| c3912d8e5c | |||
| 67599ea67c | |||
| f321a6096d | |||
| 59ebf551f0 | |||
| 04a57029b3 |
@@ -110,8 +110,12 @@ jobs:
|
||||
- name: Run unit tests under Bun
|
||||
run: bun vitest run --project unit --reporter=verbose
|
||||
|
||||
# Adapter tests are pure unit tests — OS doesn't affect results.
|
||||
# Adapter tests are pure unit tests — OS doesn't affect results. Gated off
|
||||
# `pull_request` to keep PR CI under ~2 minutes; adapter authors run focused
|
||||
# tests locally before pushing, and `push` to main / nightly cron / manual
|
||||
# dispatch still guard the merged state.
|
||||
adapter-test:
|
||||
if: github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
steps:
|
||||
|
||||
+94
-13
@@ -1,27 +1,108 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
## [1.7.22](https://github.com/jackwener/opencli/compare/v1.7.21...v1.7.22) (2026-05-15)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **browser** — `page.evaluate()` / `evaluateInFrame()` now return the user JavaScript value directly. Browser Bridge `exec` previously routed through a shared `pageScopedResult` helper that spread / wrapped the lease's `session` into the result `data`, contaminating arbitrary user returns: array / primitive returns came back as `{ session, data }` envelopes, and plain-object returns had an extra `session` key injected (overwriting any user `session` field). `google search` and `xiaohongshu search` were the visible repro — Chrome rendered results correctly but adapters extracted an empty array. Fixed in extension 1.0.14 by reverting `pageScopedResult` to its pre-1461 form (`{ id, ok, data, page }`); no client-side unwrap is needed.
|
||||
* **google/search** — wait for `#rso a h3` before extracting, falling back to the existing fixed wait. On Chrome 148 + Linux Wayland the DOM can settle before SERP anchors are populated, making extraction return empty even with the envelope bug fixed.
|
||||
* **xiaohongshu/search** — extract initially visible cards before scrolling, then merge post-scroll rows by URL. Xiaohongshu's virtualized masonry layout can evict the initial cards from the DOM after scroll, so the previous always-scroll-then-extract flow could lose the top results.
|
||||
External CLI ergonomics + two adapter envelope/auth fixes. New `longbridge` external CLI entry; `opencli list` / root help now render human-readable brand labels for executables whose bare name is ambiguous.
|
||||
|
||||
### Features
|
||||
|
||||
* **browser** — add `page.evaluate(fn, ...args)` for type-safe browser-context evaluation with JSON-serialized arguments. String evaluation remains supported, but new adapter code should use function form to avoid implicit `wrapForEval` auto-IIFE magic.
|
||||
* **external** — add the Longbridge CLI as a built-in external CLI passthrough (`opencli longbridge ...`) for Longbridge OpenAPI market data, account, and trading commands. ([#1584](https://github.com/jackwener/opencli/issues/1584))
|
||||
* **external-cli** — render brand alias `name(package)` in `opencli list` and root help when the bare executable name is ambiguous. Built-in entries `ntn` → `ntn(notion)`, `dws` → `dws(DingTalk Workspace)`, `wecom-cli` → `wecom-cli(企业微信)` now self-explain in help output. `package` field is repurposed to cover both upstream distribution names (e.g. `tg-cli`) and human-readable brand labels (e.g. `notion`, `企业微信`). ([#1585](https://github.com/jackwener/opencli/issues/1585))
|
||||
|
||||
### ⚠ BREAKING CHANGES
|
||||
### Bug Fixes
|
||||
|
||||
* **browser** — replace the `--session <name>` flag with a `<session>` positional argument that immediately follows `browser`. `opencli browser work click 12` instead of `opencli browser --session work click 12`; `opencli browser work bind` instead of `opencli browser bind --session work`. Required-flag semantics are now encoded structurally as a positional, matching the Docker/git convention for required operation-target identifiers. The internal `--session` flag is preserved for the daemon protocol and for direct `program.parseAsync` callers but is no longer part of the user-facing surface.
|
||||
* **env** — remove `OPENCLI_KEEP_TAB`. The flag was a debugging shortcut, not a config dimension: `--keep-tab true|false` on the command line is the single source of truth, and adapter `siteSession: 'persistent'` already pins persistent site tabs as a hard constraint. Removing the env eliminates a globally-leaking process state that overrode every browser command in the shell.
|
||||
* **extension** — remove the internal `surface\\0session` command-session backdoor. Browser Bridge commands now route only through structured `session` + `surface` fields; lease-key strings remain an extension-internal registry detail.
|
||||
* **boss** — map `code=24` (identity mismatch) to `AuthRequiredError` so re-login is signaled instead of surfacing as a generic API error. ([#1573](https://github.com/jackwener/opencli/issues/1573))
|
||||
* **weibo** — unwrap Browser Bridge `page.evaluate` envelopes in read adapters. ([#1568](https://github.com/jackwener/opencli/issues/1568))
|
||||
|
||||
## [1.7.21](https://github.com/jackwener/opencli/compare/v1.7.20...v1.7.21) (2026-05-14)
|
||||
|
||||
Adapter polish release: new web search adapters, better Browser Bridge tab group reuse, and social adapters returning to one-shot tab leases. Extension package version is bumped to 1.0.15 for the Browser Bridge fix.
|
||||
|
||||
### Features
|
||||
|
||||
* **search** — add DuckDuckGo, Brave, and Yahoo web search adapters. ([#1546](https://github.com/jackwener/opencli/issues/1546))
|
||||
* **boss** — support job-seeker `chatlist` and `chatmsg` adapters. ([#1539](https://github.com/jackwener/opencli/issues/1539))
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **extension** — reuse existing `OpenCLI Adapter` tab groups before creating new ones, including cross-window discovery, legacy `OpenCLI` title fallback, and deterministic candidate selection. ([#1541](https://github.com/jackwener/opencli/issues/1541))
|
||||
* **twitter, reddit** — default browser-backed social adapters back to ephemeral tab leases. Twitter/X and Reddit commands now release their site tab after each run while keeping the shared Adapter window available for reuse; persistent sessions remain reserved for AI/chat-style adapters that need long-lived conversation state. ([#1569](https://github.com/jackwener/opencli/issues/1569))
|
||||
* **xiaohongshu, rednote** — unwrap Browser Bridge `page.evaluate` envelopes in search adapters. ([#1561](https://github.com/jackwener/opencli/issues/1561))
|
||||
* **facebook/feed** — add fallback extraction for empty article nodes. ([#1538](https://github.com/jackwener/opencli/issues/1538))
|
||||
|
||||
### Internal
|
||||
|
||||
* **extension 1.0.14** — `pageScopedResult` no longer injects `session` into `data`. The field had no consumers and contaminated `exec` results with arbitrary user-JS shapes; routing-relevant identity is already exposed via `Result.page`.
|
||||
* **extension 1.0.13** — remove the internal command-session lease-key backdoor.
|
||||
* **ci** — add Windows native binding lockfile entries for Rolldown/Rollup optional packages. ([#1563](https://github.com/jackwener/opencli/issues/1563))
|
||||
* **extension** — add regression coverage for the adapter tab group `groupId` tiebreaker. ([#1566](https://github.com/jackwener/opencli/issues/1566))
|
||||
|
||||
## [1.7.20](https://github.com/jackwener/opencli/compare/v1.7.19...v1.7.20) (2026-05-14)
|
||||
|
||||
External CLI surface cleanup + Browser Bridge WebSocket lifecycle hardening. Two BREAKING changes around external CLIs: built-in `tg`/`discord`/`wx` (was `tg-cli`/`discord-cli`/`wx-cli`) now match their real binary names, and Notion's in-tree CDP adapter is replaced by the official `ntn` external CLI.
|
||||
|
||||
### ⚠ BREAKING CHANGES
|
||||
|
||||
* **notion** — remove the in-tree `clis/notion/` CDP-on-Desktop adapter (8 commands: `status` / `search` / `read` / `new` / `write` / `sidebar` / `favorites` / `export`). Notion has shipped an official CLI at <https://ntn.dev>, registered as a first-class external CLI in `external-clis.yaml`. Migration: install `ntn` from <https://ntn.dev> (`curl -fsSL https://ntn.dev | bash`), then use `opencli ntn <command>`. Auto-install is intentionally not configured because the official installer is a shell script while OpenCLI external installs run shell-free command strings. The official CLI uses the public Notion API rather than reverse-engineering the Desktop UI, so it survives Notion app updates and exposes a wider command surface (blocks / databases / properties / comments) than the reverse-engineered adapter could. ([#1559](https://github.com/jackwener/opencli/issues/1559))
|
||||
* **external** — drop the `-cli` suffix from built-in external CLI subcommand names. `opencli tg-cli`, `opencli discord-cli`, `opencli wx-cli` are now `opencli tg`, `opencli discord`, `opencli wx`, matching the real binary names that those tools install as. Root help still shows the package lineage as `tg(tg-cli)` / `discord(discord-cli)` / `wx(wx-cli)`. ([#1544](https://github.com/jackwener/opencli/issues/1544))
|
||||
|
||||
### Features
|
||||
|
||||
* **twitter** — `bookmarks` and `bookmark-folder` now include media via `extractMedia`, reaching parity with `timeline` / `search`. ([#1555](https://github.com/jackwener/opencli/issues/1555))
|
||||
* **twitter/list-tweets** — include media via `extractMedia` (parity with `timeline` / `search`). ([#1464](https://github.com/jackwener/opencli/issues/1464))
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **daemon** — report ambiguous browser command outcomes with a distinct `command_result_unknown` errorCode and `503` when the extension WebSocket drops between command dispatch and result delivery. `sendCommandRaw()` treats this code as hard non-retryable, so write-side commands (`navigate` / `click` / `type` / `eval`) won't be silently re-issued and double-executed. Daemon exposes a `commandResultUnknown` counter on `/status` for future observability. ([#1558](https://github.com/jackwener/opencli/issues/1558))
|
||||
* **extension** — keep active daemon WebSocket; stale sockets no longer clobber active connection (`onopen` / `onclose` / `onmessage` are all gated by `ws !== thisWs` short-circuit), and `safeSend` only fires when `readyState === OPEN`. ([#1540](https://github.com/jackwener/opencli/issues/1540))
|
||||
* **extension** — coalesce concurrent daemon WebSocket connects via an in-flight promise. Startup / keepalive / reconnect triggering `connect()` during the daemon-probe or context-lookup async gap no longer creates duplicate real WebSocket connections. ([#1554](https://github.com/jackwener/opencli/issues/1554))
|
||||
* **external** — distinguish external CLI executable names from distribution/project names in root help. Built-in aliases such as `tg`, `discord`, `wx` remain the callable `opencli <name> ...` entrypoints while help renders `tg(tg-cli)`, `discord(discord-cli)`, `wx(wx-cli)` to show their package lineage. ([#1560](https://github.com/jackwener/opencli/issues/1560))
|
||||
|
||||
### Docs
|
||||
|
||||
* **browser** — clarify named session lifecycle in the Browser Bridge guide. ([#1542](https://github.com/jackwener/opencli/issues/1542))
|
||||
|
||||
## [1.7.19](https://github.com/jackwener/opencli/compare/v1.7.18...v1.7.19) (2026-05-14)
|
||||
|
||||
Major hotfix + simplification batch. Extension bumped to 1.0.14. Node floor lowered to v20 so the long tail of Node v20–v21.6 users no longer crashes at module load. `opencli browser` user surface replaces required-flag `--session <name>` with a `<session>` positional. `page.evaluate(fn, ...args)` adds a type-safe alternative to the implicit auto-IIFE string form. Twitter cursor pagination no longer silently caps at ~500 items.
|
||||
|
||||
### ⚠ BREAKING CHANGES
|
||||
|
||||
* **browser** — replace the `--session <name>` flag with a `<session>` positional argument that immediately follows `browser`. `opencli browser work click 12` instead of `opencli browser --session work click 12`; `opencli browser work bind` instead of `opencli browser bind --session work`. Required-flag semantics are now encoded structurally as a positional, matching the Docker/git convention for required operation-target identifiers. The internal `--session` flag is preserved for the daemon protocol and for direct `program.parseAsync` callers but is no longer part of the user-facing surface. ([#1505](https://github.com/jackwener/opencli/issues/1505))
|
||||
* **env** — remove `OPENCLI_KEEP_TAB`. The flag was a debugging shortcut, not a config dimension: `--keep-tab true|false` on the command line is the single source of truth, and adapter `siteSession: 'persistent'` already pins persistent site tabs as a hard constraint. Removing the env eliminates a globally-leaking process state that overrode every browser command in the shell. ([#1509](https://github.com/jackwener/opencli/issues/1509))
|
||||
* **extension** — remove the internal `surface\\0session` command-session backdoor. Browser Bridge commands now route only through structured `session` + `surface` fields; lease-key strings remain an extension-internal registry detail. ([#1510](https://github.com/jackwener/opencli/issues/1510))
|
||||
|
||||
### Features
|
||||
|
||||
* **browser** — add `page.evaluate(fn, ...args)` for type-safe browser-context evaluation with JSON-serialized arguments. String evaluation remains supported, but new adapter code should use function form to avoid implicit `wrapForEval` auto-IIFE magic. ([#1508](https://github.com/jackwener/opencli/issues/1508))
|
||||
* **twitter** — default `tweets` command to the logged-in user when `user` is omitted, and fix the sibling envelope-unwrap silent bug. ([#1531](https://github.com/jackwener/opencli/issues/1531))
|
||||
* **zhihu** — add `answer-detail` to fetch a single answer's full content. ([#1528](https://github.com/jackwener/opencli/issues/1528))
|
||||
* **zhihu** — paginate question answers and recommendations. ([#1517](https://github.com/jackwener/opencli/issues/1517))
|
||||
* **reddit/read** — `--expand-more` via `/api/morechildren` + 7-kind typed errors. ([#1492](https://github.com/jackwener/opencli/issues/1492))
|
||||
* **reddit** — add `whoami`, `home`, `subreddit-info` read commands. ([#1491](https://github.com/jackwener/opencli/issues/1491))
|
||||
* **ctrip** — add `hotel-search` + flight browser-mode commands. ([#1489](https://github.com/jackwener/opencli/issues/1489))
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **browser** — `page.evaluate()` / `evaluateInFrame()` now return the user JavaScript value directly. Browser Bridge `exec` previously routed through a shared `pageScopedResult` helper that spread / wrapped the lease's `session` into the result `data`, contaminating arbitrary user returns: array / primitive returns came back as `{ session, data }` envelopes, and plain-object returns had an extra `session` key injected (overwriting any user `session` field). `google search` and `xiaohongshu search` were the visible repro — Chrome rendered results correctly but adapters extracted an empty array. Fixed in extension 1.0.14 by reverting `pageScopedResult` to its pre-1461 form (`{ id, ok, data, page }`); no client-side unwrap is needed. ([#1518](https://github.com/jackwener/opencli/issues/1518))
|
||||
* **twitter** — raise fixed cursor-pagination caps in `bookmarks` / `likes` / `tweets` / `timeline` / `bookmark-folder` / `list-tweets` / `search` / `following`. The old `i < 5` / `i < 10` literals and following's `Math.ceil(limit / 50) + 2` formula imposed hidden result ceilings below `--limit`; the loop now treats the page count as a high runaway guard while `--limit` and cursor exhaustion control normal pagination. ([#1532](https://github.com/jackwener/opencli/issues/1532))
|
||||
* **twitter** — repair `list-add` / `list-tweets` / `lists` / `following` after 2026-05 site changes. ([#1503](https://github.com/jackwener/opencli/issues/1503))
|
||||
* **twitter** — repair `search` and `tweets` readback. ([#1512](https://github.com/jackwener/opencli/issues/1512))
|
||||
* **twitter** — make reply submission robust. ([#1511](https://github.com/jackwener/opencli/issues/1511))
|
||||
* **google/search** — wait for `#rso a h3` before extracting, falling back to the existing fixed wait. On Chrome 148 + Linux Wayland the DOM can settle before SERP anchors are populated, making extraction return empty even with the envelope bug fixed. ([#1518](https://github.com/jackwener/opencli/issues/1518))
|
||||
* **google/search** — wrap evaluate return value in object to fix serialization. ([#1523](https://github.com/jackwener/opencli/issues/1523))
|
||||
* **google-scholar/search** — wrap evaluate return to fix serialization. ([#1525](https://github.com/jackwener/opencli/issues/1525))
|
||||
* **xiaohongshu/search** — extract initially visible cards before scrolling, then merge post-scroll rows by URL. Xiaohongshu's virtualized masonry layout can evict the initial cards from the DOM after scroll, so the previous always-scroll-then-extract flow could lose the top results. ([#1518](https://github.com/jackwener/opencli/issues/1518))
|
||||
* **xiaohongshu** — `parseLikes` handles `2.1w` / `1.5万` / `1.2k` shortforms. ([#1504](https://github.com/jackwener/opencli/issues/1504))
|
||||
* **xiaohongshu+rednote/search** — fall back to href-based note cards when `section.note-item` class is dropped. ([#1507](https://github.com/jackwener/opencli/issues/1507))
|
||||
* **xueqiu** — `kline` / `earnings-date` format dates in Asia/Shanghai instead of UTC. ([#1498](https://github.com/jackwener/opencli/issues/1498))
|
||||
* **download** — clamp progress percentages. ([#1520](https://github.com/jackwener/opencli/issues/1520))
|
||||
|
||||
### Internal
|
||||
|
||||
* **runtime** — lower the Node floor to `>=20.0.0`. Three coupled changes: drop all `util.styleText()` usage (added in Node v21.7.0 / v20.12.0; previously crashed v21.0–v21.6 at module load), downgrade `undici` from `^8.0.2` (engines `>=22.19.0`) to `^6.25.0` (engines `>=18.17`, retains `Agent` / `EnvHttpProxyAgent` / `fetch`), and lower `MIN_SUPPORTED_NODE_MAJOR` from 21 to 20 so the startup guard matches the declared `engines.node`. Smoke-tested on v20.0.0 / v21.2.0 / v22.22.2. The semantic markers (`[OK]` / `[WARN]` / `[FAIL]` / `ℹ` / `⚠` / `✖`) keep their meaning; ANSI colors were redundant for the primarily agent-facing CLI. ([#1524](https://github.com/jackwener/opencli/issues/1524))
|
||||
* **extension 1.0.14** — `pageScopedResult` no longer injects `session` into `data`. The field had no consumers and contaminated `exec` results with arbitrary user-JS shapes; routing-relevant identity is already exposed via `Result.page`. ([#1518](https://github.com/jackwener/opencli/issues/1518))
|
||||
* **extension 1.0.13** — remove the internal command-session lease-key backdoor. ([#1510](https://github.com/jackwener/opencli/issues/1510))
|
||||
* **ci** — drop `e2e-headed` and `adapter-test` from `pull_request` triggers (kept on `push` to main / nightly / `workflow_dispatch`). PR-time CI now targets ~2 min wall-time. ([#1521](https://github.com/jackwener/opencli/issues/1521), [#1522](https://github.com/jackwener/opencli/issues/1522))
|
||||
* **scripts** — auto-refresh `dist/` before `build-manifest`. ([#1490](https://github.com/jackwener/opencli/issues/1490))
|
||||
|
||||
## [1.7.18](https://github.com/jackwener/opencli/compare/v1.7.17...v1.7.18) (2026-05-12)
|
||||
|
||||
|
||||
@@ -14,17 +14,17 @@ OpenCLI gives you one surface for three different kinds of automation:
|
||||
- **Let AI Agents operate any website** — install the `opencli-adapter-author` skill in your AI agent (Claude Code, Cursor, etc.), and it can navigate, click, type/fill, extract, and inspect any page through your logged-in browser via `opencli browser` primitives.
|
||||
- **Write new adapters** end-to-end with `opencli browser` + the `opencli-adapter-author` skill, which guides from first recon through field decoding, code, and `opencli browser verify`.
|
||||
|
||||
It also works as a **CLI hub** for local tools such as `gh`, `docker`, `tg-cli`, `discord-cli`, `wx-cli`, and other binaries you register yourself, plus **desktop app adapters** for Electron apps like Cursor, Codex, Antigravity, ChatGPT, and Notion.
|
||||
It also works as a **CLI hub** for local tools such as `gh`, `docker`, `longbridge`, `tg`, `discord`, `wx`, `ntn` (Notion), and other binaries you register yourself, plus **desktop app adapters** for Electron apps like Cursor, Codex, Antigravity, and ChatGPT.
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Desktop App Control** — Drive Electron apps (Cursor, Codex, ChatGPT, Notion, etc.) directly from the terminal via CDP.
|
||||
- **Desktop App Control** — Drive Electron apps (Cursor, Codex, ChatGPT, etc.) directly from the terminal via CDP.
|
||||
- **Browser Automation for AI Agents** — Install the `opencli-adapter-author` skill, and your AI agent can operate any website: navigate, click, type/fill, extract, screenshot — all through your logged-in Chrome session.
|
||||
- **Multi-profile Browser Bridge** — Install the extension in each Chrome profile you want to use, then route commands with `--profile`, `OPENCLI_PROFILE`, or `opencli profile use`.
|
||||
- **Website → CLI** — Turn any website into a deterministic CLI: 100+ site surfaces are already registered, or write your own with the `opencli-adapter-author` skill + `opencli browser verify`.
|
||||
- **Account-safe** — Reuses Chrome/Chromium logged-in state; your credentials never leave the browser.
|
||||
- **AI Agent ready** — One skill takes you from site recon through API discovery, field decoding, adapter writing, and verification.
|
||||
- **CLI Hub** — Discover, auto-install, and passthrough commands to any external CLI (gh, docker, obsidian, tg-cli, discord-cli, wx-cli, etc).
|
||||
- **CLI Hub** — Discover, auto-install, and passthrough commands to any external CLI (gh, docker, obsidian, tg, discord, wx, etc).
|
||||
- **Zero LLM cost** — No tokens consumed at runtime. Run 10,000 times and pay nothing.
|
||||
- **Deterministic** — Same command, same output schema, every time. Pipeable, scriptable, CI-friendly.
|
||||
|
||||
@@ -181,7 +181,7 @@ When the site you need is not yet covered, use the `opencli-adapter-author` skil
|
||||
|
||||
OpenCLI is not only for websites. It can also:
|
||||
|
||||
- expose local binaries like `gh`, `docker`, `obsidian`, `tg-cli`, `discord-cli`, `wx-cli`, or custom tools through `opencli <tool> ...`
|
||||
- expose local binaries like `gh`, `docker`, `obsidian`, `tg`, `discord`, `wx`, or custom tools through `opencli <tool> ...`
|
||||
- control Electron desktop apps through dedicated adapters and CDP-backed integrations
|
||||
|
||||
## Prerequisites
|
||||
@@ -283,19 +283,21 @@ To load the source Browser Bridge extension:
|
||||
|
||||
## CLI Hub
|
||||
|
||||
OpenCLI acts as a universal hub for your existing command-line tools — unified discovery, pure passthrough execution, and auto-install (if a tool isn't installed, OpenCLI runs `brew install <tool>` automatically before re-running the command).
|
||||
OpenCLI acts as a universal hub for your existing command-line tools — unified discovery, pure passthrough execution, and auto-install when a safe package-manager command is configured.
|
||||
|
||||
| External CLI | Description | Example |
|
||||
|--------------|-------------|---------|
|
||||
| **gh** | GitHub CLI | `opencli gh pr list --limit 5` |
|
||||
| **obsidian** | Obsidian vault management | `opencli obsidian search query="AI"` |
|
||||
| **docker** | Docker | `opencli docker ps` |
|
||||
| **longbridge** | Longbridge CLI — market data, account management, and trading via Longbridge OpenAPI | `opencli longbridge quote TSLA.US --format json` |
|
||||
| **ntn** | Notion CLI — official Notion API CLI for pages, databases, blocks, search, comments | `opencli ntn pages list` |
|
||||
| **lark-cli** | Lark/Feishu — messages, docs, calendar, tasks, 200+ commands | `opencli lark-cli calendar +agenda` |
|
||||
| **dws** | DingTalk — cross-platform CLI for DingTalk's full suite, designed for humans and AI agents | `opencli dws msg send --to user "hello"` |
|
||||
| **wecom-cli** | WeCom/企业微信 — CLI for WeCom open platform, for humans and AI agents | `opencli wecom-cli msg send --to user "hello"` |
|
||||
| **tg-cli** | Telegram — local-first sync, search, and export via MTProto for AI agents | `opencli tg search "AI news" -f json` |
|
||||
| **discord-cli** | Discord — local-first sync, search, and export via SQLite for AI agents | `opencli discord recent --channel general` |
|
||||
| **wx-cli** | WeChat — query local WeChat data: sessions, messages, search, contacts, export | `opencli wx search "OpenCLI"` |
|
||||
| **tg(tg-cli)** | Telegram — local-first sync, search, and export via MTProto for AI agents | `opencli tg search "AI news" -f json` |
|
||||
| **discord(discord-cli)** | Discord — local-first sync, search, and export via SQLite for AI agents | `opencli discord recent --channel general` |
|
||||
| **wx(wx-cli)** | WeChat — query local WeChat data: sessions, messages, search, contacts, export | `opencli wx search "OpenCLI"` |
|
||||
| **vercel** | Vercel — deploy projects, manage domains, env vars, logs | `opencli vercel deploy --prod` |
|
||||
|
||||
**Register your own** — add any local CLI so AI agents can discover it via `opencli list`:
|
||||
@@ -304,6 +306,8 @@ OpenCLI acts as a universal hub for your existing command-line tools — unified
|
||||
opencli external register mycli
|
||||
```
|
||||
|
||||
**Manual install** — some external CLIs use official shell-script installers rather than shell-free package-manager commands. For `ntn`, install from <https://ntn.dev> first, then run `opencli ntn ...`.
|
||||
|
||||
### Desktop App Adapters
|
||||
|
||||
Control Electron desktop apps directly from the terminal. Each adapter has its own detailed documentation:
|
||||
@@ -315,7 +319,6 @@ Control Electron desktop apps directly from the terminal. Each adapter has its o
|
||||
| **Antigravity** | Control Antigravity Ultra from terminal | [Doc](./docs/adapters/desktop/antigravity.md) |
|
||||
| **ChatGPT App** | Automate ChatGPT macOS desktop app | [Doc](./docs/adapters/desktop/chatgpt-app.md) |
|
||||
| **ChatWise** | Multi-LLM client (GPT-4, Claude, Gemini) | [Doc](./docs/adapters/desktop/chatwise.md) |
|
||||
| **Notion** | Search, read, write Notion pages | [Doc](./docs/adapters/desktop/notion.md) |
|
||||
| **Discord** | Discord Desktop — messages, channels, servers | [Doc](./docs/adapters/desktop/discord.md) |
|
||||
| **Doubao** | Control Doubao AI desktop app via CDP | [Doc](./docs/adapters/desktop/doubao-app.md) |
|
||||
|
||||
|
||||
+10
-10
@@ -14,16 +14,16 @@ OpenCLI 可以用同一套 CLI 做三类事情:
|
||||
- **让 AI Agent 操作任意网站**:在你的 AI Agent(Claude Code、Cursor 等)中安装 `opencli-adapter-author` skill,Agent 就能用你的已登录浏览器导航、点击、输入/填充、提取任意网页内容。
|
||||
- **把新网站写成 CLI**:用 `opencli browser` 原语 + `opencli-adapter-author` skill,从站点侦察、API 发现、字段解码到 `opencli browser verify` 一条龙。
|
||||
|
||||
除了网站能力,OpenCLI 还是一个 **CLI 枢纽**:你可以把 `gh`、`docker`、`tg-cli`、`discord-cli`、`wx-cli` 等本地工具统一注册到 `opencli` 下,也可以通过桌面端适配器控制 Cursor、Codex、Antigravity、ChatGPT、Notion 等 Electron 应用。
|
||||
除了网站能力,OpenCLI 还是一个 **CLI 枢纽**:你可以把 `gh`、`docker`、`longbridge`、`tg`、`discord`、`wx`、`ntn`(Notion)等本地工具统一注册到 `opencli` 下,也可以通过桌面端适配器控制 Cursor、Codex、Antigravity、ChatGPT 等 Electron 应用。
|
||||
|
||||
## 亮点
|
||||
|
||||
- **桌面应用控制** — 通过 CDP 直接在终端驱动 Electron 应用(Cursor、Codex、ChatGPT、Notion 等)。
|
||||
- **桌面应用控制** — 通过 CDP 直接在终端驱动 Electron 应用(Cursor、Codex、ChatGPT 等)。
|
||||
- **AI Agent 浏览器自动化** — 安装 `opencli-adapter-author` skill,你的 AI Agent 就能操作任意网站:导航、点击、输入/填充、提取、截图——全部通过你的已登录 Chrome 会话完成。
|
||||
- **网站 → CLI** — 把任何网站变成确定性 CLI:100+ 站点能力已注册,或用 `opencli-adapter-author` skill + `opencli browser verify` 自己写。
|
||||
- **账号安全** — 复用 Chrome/Chromium 登录态,凭证永远不会离开浏览器。
|
||||
- **面向 AI Agent** — 一个 skill 带你走完站点侦察、API 发现、字段解码、适配器编写、验证的全流程。
|
||||
- **CLI 枢纽** — 统一发现、自动安装、纯透传任何外部 CLI(gh、docker、obsidian、tg-cli、discord-cli、wx-cli 等)。
|
||||
- **CLI 枢纽** — 统一发现、自动安装、纯透传任何外部 CLI(gh、docker、obsidian、tg、discord、wx 等)。
|
||||
- **零 LLM 成本** — 运行时不消耗模型 token,跑 10,000 次也不花一分钱。
|
||||
- **确定性输出** — 相同命令,相同输出结构,每次一致。可管道、可脚本、CI 友好。
|
||||
|
||||
@@ -165,7 +165,7 @@ Agent 在内部自动处理所有 `opencli browser` 命令——你只需用自
|
||||
|
||||
OpenCLI 不只是网站 CLI,还可以:
|
||||
|
||||
- 统一代理本地二进制工具,例如 `gh`、`docker`、`obsidian`、`tg-cli`、`discord-cli`、`wx-cli`
|
||||
- 统一代理本地二进制工具,例如 `gh`、`docker`、`obsidian`、`tg`、`discord`、`wx`
|
||||
- 通过专门适配器和 CDP 集成控制 Electron 桌面应用
|
||||
|
||||
## 前置要求
|
||||
@@ -241,7 +241,6 @@ npm link
|
||||
| **chatwise** | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` | 桌面端 |
|
||||
| **doubao** | `status` `new` `send` `read` `ask` `history` `detail` `meeting-summary` `meeting-transcript` | 浏览器 |
|
||||
| **doubao-app** | `status` `new` `send` `read` `ask` `screenshot` `dump` | 桌面端 |
|
||||
| **notion** | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` | 桌面端 |
|
||||
| **discord-app** | `status` `send` `read` `channels` `servers` `search` `members` | 桌面端 |
|
||||
| **v2ex** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | 公开 / 浏览器 |
|
||||
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `comments` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 浏览器 |
|
||||
@@ -333,17 +332,19 @@ OpenCLI 也可以作为你现有命令行工具的统一入口,负责发现、
|
||||
| **gh** | GitHub CLI | `opencli gh pr list --limit 5` |
|
||||
| **obsidian** | Obsidian 仓库管理 | `opencli obsidian search query="AI"` |
|
||||
| **docker** | Docker 命令行工具 | `opencli docker ps` |
|
||||
| **longbridge** | Longbridge CLI — 通过 Longbridge OpenAPI 获取行情、账户和交易能力 | `opencli longbridge quote TSLA.US --format json` |
|
||||
| **ntn** | Notion CLI — 基于官方 Notion API 的页面、数据库、块、搜索、评论命令 | `opencli ntn pages list` |
|
||||
| **lark-cli** | 飞书 CLI — 消息、文档、日历、任务,200+ 命令 | `opencli lark-cli calendar +agenda` |
|
||||
| **dws** | 钉钉 CLI — 钉钉全套产品能力的跨平台命令行工具,支持人类和 AI Agent 使用 | `opencli dws msg send --to user "hello"` |
|
||||
| **wecom-cli** | 企业微信 CLI — 企业微信开放平台命令行工具,支持人类和 AI Agent 使用 | `opencli wecom-cli msg send --to user "hello"` |
|
||||
| **tg-cli** | Telegram CLI — 基于 MTProto 的本地优先同步、搜索、导出,面向 AI Agent | `opencli tg search "AI news" -f json` |
|
||||
| **discord-cli** | Discord CLI — 基于 SQLite 的本地优先同步、搜索、导出,面向 AI Agent | `opencli discord recent --channel general` |
|
||||
| **wx-cli** | 微信本地数据 CLI — 会话、聊天记录、搜索、联系人、导出 | `opencli wx search "OpenCLI"` |
|
||||
| **tg(tg-cli)** | Telegram CLI — 基于 MTProto 的本地优先同步、搜索、导出,面向 AI Agent | `opencli tg search "AI news" -f json` |
|
||||
| **discord(discord-cli)** | Discord CLI — 基于 SQLite 的本地优先同步、搜索、导出,面向 AI Agent | `opencli discord recent --channel general` |
|
||||
| **wx(wx-cli)** | 微信本地数据 CLI — 会话、聊天记录、搜索、联系人、导出 | `opencli wx search "OpenCLI"` |
|
||||
| **vercel** | Vercel — 部署项目、管理域名、环境变量、日志 | `opencli vercel deploy --prod` |
|
||||
|
||||
**零配置透传**:OpenCLI 会把你的输入原样转发给底层二进制,保留原生 stdout / stderr 行为。
|
||||
|
||||
**自动安装**:如果你运行 `opencli gh ...` 时系统中还没有 `gh`,OpenCLI 会优先尝试通过系统包管理器安装,然后自动重试命令。
|
||||
**自动安装**:如果某个外部 CLI 配置了安全的包管理器安装命令,OpenCLI 会优先尝试安装后再执行;`ntn` 的官方安装方式是 shell 脚本,请先按 <https://ntn.dev> 手动安装。
|
||||
|
||||
**注册自定义本地 CLI**:
|
||||
|
||||
@@ -362,7 +363,6 @@ opencli register mycli
|
||||
| **Antigravity** | 在终端直接控制 Antigravity Ultra | [Doc](./docs/adapters/desktop/antigravity.md) |
|
||||
| **ChatGPT App** | 自动化操作 ChatGPT macOS 桌面客户端 | [Doc](./docs/adapters/desktop/chatgpt-app.md) |
|
||||
| **ChatWise** | 多 LLM 客户端(GPT-4、Claude、Gemini) | [Doc](./docs/adapters/desktop/chatwise.md) |
|
||||
| **Notion** | 搜索、读取、写入 Notion 页面 | [Doc](./docs/adapters/desktop/notion.md) |
|
||||
| **Discord** | Discord 桌面版 — 消息、频道、服务器 | [Doc](./docs/adapters/desktop/discord.md) |
|
||||
| **Doubao** | 通过 CDP 控制豆包桌面应用 | [Doc](./docs/adapters/desktop/doubao-app.md) |
|
||||
|
||||
|
||||
+348
-257
@@ -3484,7 +3484,7 @@
|
||||
{
|
||||
"site": "boss",
|
||||
"name": "chatlist",
|
||||
"description": "BOSS直聘查看聊天列表(招聘端)",
|
||||
"description": "BOSS直聘查看聊天列表(招聘端/求职端)",
|
||||
"access": "read",
|
||||
"domain": "www.zhipin.com",
|
||||
"strategy": "cookie",
|
||||
@@ -3509,12 +3509,26 @@
|
||||
"type": "str",
|
||||
"default": "0",
|
||||
"required": false,
|
||||
"help": "Filter by job ID (0=all)"
|
||||
"help": "Filter by job ID (0=all, boss side only)"
|
||||
},
|
||||
{
|
||||
"name": "side",
|
||||
"type": "str",
|
||||
"default": "auto",
|
||||
"required": false,
|
||||
"help": "Identity side: auto (default), boss (recruiter), or geek (job-seeker)",
|
||||
"choices": [
|
||||
"auto",
|
||||
"boss",
|
||||
"geek"
|
||||
]
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"name",
|
||||
"company",
|
||||
"job",
|
||||
"title",
|
||||
"last_msg",
|
||||
"last_time",
|
||||
"uid",
|
||||
@@ -3528,7 +3542,7 @@
|
||||
{
|
||||
"site": "boss",
|
||||
"name": "chatmsg",
|
||||
"description": "BOSS直聘查看与候选人的聊天消息",
|
||||
"description": "BOSS直聘查看聊天消息历史(招聘端/求职端)",
|
||||
"access": "read",
|
||||
"domain": "www.zhipin.com",
|
||||
"strategy": "cookie",
|
||||
@@ -3547,6 +3561,18 @@
|
||||
"default": 1,
|
||||
"required": false,
|
||||
"help": "Page number"
|
||||
},
|
||||
{
|
||||
"name": "side",
|
||||
"type": "str",
|
||||
"default": "auto",
|
||||
"required": false,
|
||||
"help": "Identity side: auto (default), boss (recruiter), or geek (job-seeker)",
|
||||
"choices": [
|
||||
"auto",
|
||||
"boss",
|
||||
"geek"
|
||||
]
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
@@ -4009,6 +4035,47 @@
|
||||
"sourceFile": "boss/stats.js",
|
||||
"navigateBefore": false
|
||||
},
|
||||
{
|
||||
"site": "brave",
|
||||
"name": "search",
|
||||
"description": "Search Brave Search",
|
||||
"access": "read",
|
||||
"domain": "search.brave.com",
|
||||
"strategy": "public",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "keyword",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Search query"
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
"type": "int",
|
||||
"default": 10,
|
||||
"required": false,
|
||||
"help": "Number of results per page (max 18)"
|
||||
},
|
||||
{
|
||||
"name": "offset",
|
||||
"type": "int",
|
||||
"default": 0,
|
||||
"required": false,
|
||||
"help": "Page offset (0, 1, 2...). Brave returns ~18 results per page"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"rank",
|
||||
"title",
|
||||
"url",
|
||||
"snippet"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "brave/search.js",
|
||||
"sourceFile": "brave/search.js"
|
||||
},
|
||||
{
|
||||
"site": "chaoxing",
|
||||
"name": "assignments",
|
||||
@@ -8704,6 +8771,93 @@
|
||||
"sourceFile": "douyin/videos.js",
|
||||
"navigateBefore": "https://creator.douyin.com"
|
||||
},
|
||||
{
|
||||
"site": "duckduckgo",
|
||||
"name": "search",
|
||||
"description": "Search DuckDuckGo",
|
||||
"access": "read",
|
||||
"domain": "html.duckduckgo.com",
|
||||
"strategy": "public",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "keyword",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Search query"
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
"type": "int",
|
||||
"default": 10,
|
||||
"required": false,
|
||||
"help": "Number of results per page (1-10). For multi-page, use --offset"
|
||||
},
|
||||
{
|
||||
"name": "offset",
|
||||
"type": "int",
|
||||
"default": 0,
|
||||
"required": false,
|
||||
"help": "Result offset for pagination (0, 10, 20...). Uses XHR POST internally"
|
||||
},
|
||||
{
|
||||
"name": "region",
|
||||
"type": "str",
|
||||
"required": false,
|
||||
"help": "Region code (e.g. jp-jp, us-en, cn-zh). Default: all regions"
|
||||
},
|
||||
{
|
||||
"name": "time",
|
||||
"type": "str",
|
||||
"required": false,
|
||||
"help": "Time range: d (day), w (week), m (month), y (year)"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"rank",
|
||||
"title",
|
||||
"url",
|
||||
"snippet",
|
||||
"displayUrl",
|
||||
"icon",
|
||||
"resultType"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "duckduckgo/search.js",
|
||||
"sourceFile": "duckduckgo/search.js"
|
||||
},
|
||||
{
|
||||
"site": "duckduckgo",
|
||||
"name": "suggest",
|
||||
"description": "DuckDuckGo search suggestions",
|
||||
"access": "read",
|
||||
"domain": "duckduckgo.com",
|
||||
"strategy": "public",
|
||||
"browser": false,
|
||||
"args": [
|
||||
{
|
||||
"name": "keyword",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Search query prefix"
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
"type": "int",
|
||||
"default": 8,
|
||||
"required": false,
|
||||
"help": "Max number of suggestions"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"phrase"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "duckduckgo/suggest.js",
|
||||
"sourceFile": "duckduckgo/suggest.js"
|
||||
},
|
||||
{
|
||||
"site": "eastmoney",
|
||||
"name": "announcement",
|
||||
@@ -9387,7 +9541,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "facebook/feed.js",
|
||||
"sourceFile": "facebook/feed.js",
|
||||
"navigateBefore": "https://www.facebook.com"
|
||||
"navigateBefore": false
|
||||
},
|
||||
{
|
||||
"site": "facebook",
|
||||
@@ -15753,181 +15907,6 @@
|
||||
"sourceFile": "notebooklm/summary.js",
|
||||
"navigateBefore": false
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "export",
|
||||
"description": "Export the current Notion page as Markdown",
|
||||
"access": "read",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "output",
|
||||
"type": "str",
|
||||
"required": false,
|
||||
"help": "Output file (default: /tmp/notion-export.md)"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"Status",
|
||||
"File"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/export.js",
|
||||
"sourceFile": "notion/export.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "favorites",
|
||||
"description": "List pages from the Notion Favorites section in the sidebar",
|
||||
"access": "read",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [],
|
||||
"columns": [
|
||||
"Index",
|
||||
"Title",
|
||||
"Icon"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/favorites.js",
|
||||
"sourceFile": "notion/favorites.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "new",
|
||||
"description": "Create a new page in Notion",
|
||||
"access": "write",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "title",
|
||||
"type": "str",
|
||||
"required": false,
|
||||
"positional": true,
|
||||
"help": "Page title (optional)"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"Status"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/new.js",
|
||||
"sourceFile": "notion/new.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "read",
|
||||
"description": "Read the content of the currently open Notion page",
|
||||
"access": "read",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [],
|
||||
"columns": [
|
||||
"Title",
|
||||
"Content"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/read.js",
|
||||
"sourceFile": "notion/read.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "search",
|
||||
"description": "Search pages and databases in Notion via Quick Find (Cmd+P)",
|
||||
"access": "read",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "query",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Search query"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"Index",
|
||||
"Title"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/search.js",
|
||||
"sourceFile": "notion/search.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "sidebar",
|
||||
"description": "List pages and databases from the Notion sidebar",
|
||||
"access": "read",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [],
|
||||
"columns": [
|
||||
"Index",
|
||||
"Title"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/sidebar.js",
|
||||
"sourceFile": "notion/sidebar.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "status",
|
||||
"description": "Check active CDP connection to Notion Desktop",
|
||||
"access": "read",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [],
|
||||
"columns": [
|
||||
"Status",
|
||||
"Url",
|
||||
"Title"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/status.js",
|
||||
"sourceFile": "notion/status.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "notion",
|
||||
"name": "write",
|
||||
"description": "Append text content to the currently open Notion page",
|
||||
"access": "write",
|
||||
"domain": "localhost",
|
||||
"strategy": "ui",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "text",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Text to append to the page"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"Status"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "notion/write.js",
|
||||
"sourceFile": "notion/write.js",
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "nowcoder",
|
||||
"name": "companies",
|
||||
@@ -19165,8 +19144,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/comment.js",
|
||||
"sourceFile": "reddit/comment.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19196,8 +19174,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/frontpage.js",
|
||||
"sourceFile": "reddit/frontpage.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19229,8 +19206,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/home.js",
|
||||
"sourceFile": "reddit/home.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19299,8 +19275,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/popular.js",
|
||||
"sourceFile": "reddit/popular.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19352,6 +19327,20 @@
|
||||
"default": 2000,
|
||||
"required": false,
|
||||
"help": "Max characters per comment body (min 100)"
|
||||
},
|
||||
{
|
||||
"name": "expand-more",
|
||||
"type": "bool",
|
||||
"default": false,
|
||||
"required": false,
|
||||
"help": "Follow Reddit \"more comments\" stubs by calling /api/morechildren.json"
|
||||
},
|
||||
{
|
||||
"name": "expand-rounds",
|
||||
"type": "int",
|
||||
"default": 2,
|
||||
"required": false,
|
||||
"help": "Max expansion passes when --expand-more is on (1–5; each round can fan out new \"more\" stubs)"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
@@ -19363,8 +19352,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/read.js",
|
||||
"sourceFile": "reddit/read.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19430,8 +19418,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/save.js",
|
||||
"sourceFile": "reddit/save.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19460,8 +19447,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/saved.js",
|
||||
"sourceFile": "reddit/saved.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19519,8 +19505,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/search.js",
|
||||
"sourceFile": "reddit/search.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19570,8 +19555,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/subreddit.js",
|
||||
"sourceFile": "reddit/subreddit.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19597,8 +19581,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/subreddit-info.js",
|
||||
"sourceFile": "reddit/subreddit-info.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19631,8 +19614,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/subscribe.js",
|
||||
"sourceFile": "reddit/subscribe.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19665,8 +19647,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/upvote.js",
|
||||
"sourceFile": "reddit/upvote.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19695,8 +19676,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/upvoted.js",
|
||||
"sourceFile": "reddit/upvoted.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19722,8 +19702,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/user.js",
|
||||
"sourceFile": "reddit/user.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19758,8 +19737,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/user-comments.js",
|
||||
"sourceFile": "reddit/user-comments.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19795,8 +19773,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/user-posts.js",
|
||||
"sourceFile": "reddit/user-posts.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "reddit",
|
||||
@@ -19814,8 +19791,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "reddit/whoami.js",
|
||||
"sourceFile": "reddit/whoami.js",
|
||||
"navigateBefore": "https://reddit.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://reddit.com"
|
||||
},
|
||||
{
|
||||
"site": "rednote",
|
||||
@@ -22603,8 +22579,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/article.js",
|
||||
"sourceFile": "twitter/article.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -22697,13 +22672,14 @@
|
||||
"retweets",
|
||||
"bookmarks",
|
||||
"created_at",
|
||||
"url"
|
||||
"url",
|
||||
"has_media",
|
||||
"media_urls"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "twitter/bookmark-folder.js",
|
||||
"sourceFile": "twitter/bookmark-folder.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -22723,8 +22699,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/bookmark-folders.js",
|
||||
"sourceFile": "twitter/bookmark-folders.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -22758,13 +22733,14 @@
|
||||
"retweets",
|
||||
"bookmarks",
|
||||
"created_at",
|
||||
"url"
|
||||
"url",
|
||||
"has_media",
|
||||
"media_urls"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "twitter/bookmarks.js",
|
||||
"sourceFile": "twitter/bookmarks.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -22838,8 +22814,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/download.js",
|
||||
"sourceFile": "twitter/download.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -22899,8 +22874,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/followers.js",
|
||||
"sourceFile": "twitter/followers.js",
|
||||
"navigateBefore": true,
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -22935,8 +22909,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/following.js",
|
||||
"sourceFile": "twitter/following.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23036,8 +23009,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/likes.js",
|
||||
"sourceFile": "twitter/likes.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23150,13 +23122,14 @@
|
||||
"retweets",
|
||||
"replies",
|
||||
"created_at",
|
||||
"url"
|
||||
"url",
|
||||
"has_media",
|
||||
"media_urls"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "twitter/list-tweets.js",
|
||||
"sourceFile": "twitter/list-tweets.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23185,8 +23158,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/lists.js",
|
||||
"sourceFile": "twitter/lists.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23215,8 +23187,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/notifications.js",
|
||||
"sourceFile": "twitter/notifications.js",
|
||||
"navigateBefore": true,
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": true
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23244,7 +23215,9 @@
|
||||
"columns": [
|
||||
"status",
|
||||
"message",
|
||||
"text"
|
||||
"text",
|
||||
"id",
|
||||
"url"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "twitter/post.js",
|
||||
@@ -23284,8 +23257,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/profile.js",
|
||||
"sourceFile": "twitter/profile.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23372,7 +23344,8 @@
|
||||
"columns": [
|
||||
"status",
|
||||
"message",
|
||||
"text"
|
||||
"text",
|
||||
"url"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "twitter/reply.js",
|
||||
@@ -23460,7 +23433,7 @@
|
||||
"description": "Search Twitter/X for tweets, with optional --from / --has / --exclude / --product filters mapped to X's search operators",
|
||||
"access": "read",
|
||||
"domain": "x.com",
|
||||
"strategy": "intercept",
|
||||
"strategy": "cookie",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
@@ -23553,8 +23526,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/search.js",
|
||||
"sourceFile": "twitter/search.js",
|
||||
"navigateBefore": true,
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23600,8 +23572,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/thread.js",
|
||||
"sourceFile": "twitter/thread.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23654,8 +23625,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/timeline.js",
|
||||
"sourceFile": "twitter/timeline.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -23682,13 +23652,12 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/trending.js",
|
||||
"sourceFile": "twitter/trending.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
"name": "tweets",
|
||||
"description": "Fetch a Twitter user's most recent tweets (chronological, excludes pinned)",
|
||||
"description": "Fetch a Twitter user's most recent tweets (chronological, excludes pinned; defaults to the logged-in user when no username is given)",
|
||||
"access": "read",
|
||||
"domain": "x.com",
|
||||
"strategy": "cookie",
|
||||
@@ -23697,9 +23666,9 @@
|
||||
{
|
||||
"name": "username",
|
||||
"type": "string",
|
||||
"required": true,
|
||||
"required": false,
|
||||
"positional": true,
|
||||
"help": "Twitter screen name (with or without @)"
|
||||
"help": "Twitter screen name (with or without @). Defaults to the logged-in user when omitted."
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
@@ -23733,8 +23702,7 @@
|
||||
"type": "js",
|
||||
"modulePath": "twitter/tweets.js",
|
||||
"sourceFile": "twitter/tweets.js",
|
||||
"navigateBefore": "https://x.com",
|
||||
"siteSession": "persistent"
|
||||
"navigateBefore": "https://x.com"
|
||||
},
|
||||
{
|
||||
"site": "twitter",
|
||||
@@ -26851,6 +26819,47 @@
|
||||
"sourceFile": "xueqiu/watchlist.js",
|
||||
"navigateBefore": "https://xueqiu.com"
|
||||
},
|
||||
{
|
||||
"site": "yahoo",
|
||||
"name": "search",
|
||||
"description": "Search Yahoo (powered by Bing)",
|
||||
"access": "read",
|
||||
"domain": "search.yahoo.com",
|
||||
"strategy": "public",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "keyword",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Search query"
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
"type": "int",
|
||||
"default": 7,
|
||||
"required": false,
|
||||
"help": "Number of results per page (max 7)"
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"type": "int",
|
||||
"default": 1,
|
||||
"required": false,
|
||||
"help": "Page number (1, 2, 3...). Yahoo returns ~7 results per page"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"rank",
|
||||
"title",
|
||||
"url",
|
||||
"snippet"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "yahoo/search.js",
|
||||
"sourceFile": "yahoo/search.js"
|
||||
},
|
||||
{
|
||||
"site": "yahoo-finance",
|
||||
"name": "quote",
|
||||
@@ -28195,6 +28204,47 @@
|
||||
"sourceFile": "zhihu/answer.js",
|
||||
"navigateBefore": "https://www.zhihu.com"
|
||||
},
|
||||
{
|
||||
"site": "zhihu",
|
||||
"name": "answer-detail",
|
||||
"description": "知乎单个回答完整内容(按 answer ID 获取)",
|
||||
"access": "read",
|
||||
"domain": "www.zhihu.com",
|
||||
"strategy": "cookie",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "id",
|
||||
"type": "str",
|
||||
"required": true,
|
||||
"positional": true,
|
||||
"help": "Answer ID, full Zhihu answer URL, or typed target (answer:<qid>:<aid>)"
|
||||
},
|
||||
{
|
||||
"name": "max-content",
|
||||
"type": "int",
|
||||
"default": 0,
|
||||
"required": false,
|
||||
"help": "Optional cap on stripped content length in characters (0 = no truncation, return the full answer)"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"id",
|
||||
"author",
|
||||
"votes",
|
||||
"comments",
|
||||
"question_id",
|
||||
"question_title",
|
||||
"url",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
"content"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "zhihu/answer-detail.js",
|
||||
"sourceFile": "zhihu/answer-detail.js",
|
||||
"navigateBefore": "https://www.zhihu.com"
|
||||
},
|
||||
{
|
||||
"site": "zhihu",
|
||||
"name": "collection",
|
||||
@@ -28529,7 +28579,18 @@
|
||||
"type": "int",
|
||||
"default": 5,
|
||||
"required": false,
|
||||
"help": "Number of answers"
|
||||
"help": "Number of answers (max 1000; use normal-sized requests)"
|
||||
},
|
||||
{
|
||||
"name": "sort",
|
||||
"type": "str",
|
||||
"default": "default",
|
||||
"required": false,
|
||||
"help": "Answer order: default or created",
|
||||
"choices": [
|
||||
"default",
|
||||
"created"
|
||||
]
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
@@ -28543,6 +28604,36 @@
|
||||
"sourceFile": "zhihu/question.js",
|
||||
"navigateBefore": "https://www.zhihu.com"
|
||||
},
|
||||
{
|
||||
"site": "zhihu",
|
||||
"name": "recommend",
|
||||
"description": "知乎首页推荐",
|
||||
"access": "read",
|
||||
"domain": "www.zhihu.com",
|
||||
"strategy": "cookie",
|
||||
"browser": true,
|
||||
"args": [
|
||||
{
|
||||
"name": "limit",
|
||||
"type": "int",
|
||||
"default": 20,
|
||||
"required": false,
|
||||
"help": "Number of items to return (max 1000; use normal-sized requests)"
|
||||
}
|
||||
],
|
||||
"columns": [
|
||||
"rank",
|
||||
"type",
|
||||
"title",
|
||||
"author",
|
||||
"votes",
|
||||
"url"
|
||||
],
|
||||
"type": "js",
|
||||
"modulePath": "zhihu/recommend.js",
|
||||
"sourceFile": "zhihu/recommend.js",
|
||||
"navigateBefore": "https://www.zhihu.com"
|
||||
},
|
||||
{
|
||||
"site": "zhihu",
|
||||
"name": "search",
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
import { ArgumentError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
|
||||
export function requireSearchQuery(value, label = 'keyword') {
|
||||
const query = String(value ?? '').trim();
|
||||
if (!query) {
|
||||
throw new ArgumentError(`${label} cannot be empty`);
|
||||
}
|
||||
return query;
|
||||
}
|
||||
|
||||
export function requireBoundedInteger(value, defaultValue, min, max, label) {
|
||||
const raw = value ?? defaultValue;
|
||||
const parsed = typeof raw === 'number' ? raw : Number(raw);
|
||||
if (!Number.isInteger(parsed)) {
|
||||
throw new ArgumentError(`${label} must be an integer between ${min} and ${max}, got ${JSON.stringify(value)}`);
|
||||
}
|
||||
if (parsed < min || parsed > max) {
|
||||
throw new ArgumentError(`${label} must be between ${min} and ${max}, got ${parsed}`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
export function requireNonNegativeInteger(value, defaultValue, label) {
|
||||
const raw = value ?? defaultValue;
|
||||
const parsed = typeof raw === 'number' ? raw : Number(raw);
|
||||
if (!Number.isInteger(parsed) || parsed < 0) {
|
||||
throw new ArgumentError(`${label} must be a non-negative integer, got ${JSON.stringify(value)}`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
export function unwrapBrowserResult(value) {
|
||||
if (value && typeof value === 'object' && !Array.isArray(value) && 'session' in value && 'data' in value) {
|
||||
return value.data;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function requireRows(value, label) {
|
||||
const rows = unwrapBrowserResult(value);
|
||||
if (!Array.isArray(rows)) {
|
||||
throw new CommandExecutionError(`${label} returned an unexpected payload shape; expected an array of result rows.`);
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
export function toHttpsUrl(value, baseUrl) {
|
||||
const raw = String(value ?? '').trim();
|
||||
if (!raw) return '';
|
||||
try {
|
||||
const url = new URL(raw, baseUrl);
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:') return '';
|
||||
return url.href;
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
export function emptySearchResults(site, query) {
|
||||
return new EmptyResultError(`${site} search`, `No ${site} results matched "${query}".`);
|
||||
}
|
||||
|
||||
export async function runBrowserStep(label, fn) {
|
||||
try {
|
||||
return await fn();
|
||||
} catch (error) {
|
||||
if (error?.code || error?.name === 'ArgumentError') throw error;
|
||||
throw new CommandExecutionError(`${label} failed: ${error?.message ?? error}`);
|
||||
}
|
||||
}
|
||||
+96
-14
@@ -1,10 +1,60 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { requirePage, navigateToChat, fetchFriendList } from './utils.js';
|
||||
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import {
|
||||
requirePage, navigateToChat, navigateToGeekChat,
|
||||
fetchFriendList, fetchGeekFriendLabelList, fetchGeekFriendInfoList,
|
||||
readEncryptSystemId, assertOk, IDENTITY_MISMATCH_CODE,
|
||||
readPositiveInteger,
|
||||
} from './utils.js';
|
||||
|
||||
function formatMsgTime(ms) {
|
||||
if (!ms) return '';
|
||||
return new Date(ms).toLocaleString('zh-CN');
|
||||
}
|
||||
|
||||
function mapBossRow(f) {
|
||||
return {
|
||||
name: f.name || '',
|
||||
company: '',
|
||||
job: f.jobName || '',
|
||||
title: '',
|
||||
last_msg: f.lastMessageInfo?.text || '',
|
||||
last_time: f.lastTime || '',
|
||||
uid: f.encryptUid || '',
|
||||
security_id: f.securityId || '',
|
||||
};
|
||||
}
|
||||
|
||||
async function buildGeekRows(page, limit) {
|
||||
const encryptSystemId = await readEncryptSystemId(page);
|
||||
const labelList = await fetchGeekFriendLabelList(page, { encryptSystemId });
|
||||
if (labelList.length === 0) {
|
||||
return [];
|
||||
}
|
||||
const slicedLabels = labelList.slice(0, limit);
|
||||
const friendIds = slicedLabels.map((f) => f.friendId).filter(Boolean);
|
||||
const enriched = await fetchGeekFriendInfoList(page, friendIds);
|
||||
const enrichMap = new Map(enriched.map((f) => [String(f.friendId ?? f.uid), f]));
|
||||
return slicedLabels.map((f) => {
|
||||
const e = enrichMap.get(String(f.friendId)) || {};
|
||||
return {
|
||||
name: e.name || f.name || '',
|
||||
company: e.brandName || f.brandName || '',
|
||||
job: e.jobName || f.jobName || '',
|
||||
title: e.bossTitle || f.bossTitle || '',
|
||||
last_msg: e.lastMessageInfo?.showText || e.lastMsg || f.lastMsg || '',
|
||||
last_time: e.lastTime || formatMsgTime(e.lastMessageInfo?.msgTime) || formatMsgTime(f.updateTime) || '',
|
||||
uid: e.encryptUid || f.encryptFriendId || String(e.uid ?? e.friendId ?? f.friendId ?? ''),
|
||||
security_id: e.securityId || '',
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'boss',
|
||||
name: 'chatlist',
|
||||
access: 'read',
|
||||
description: 'BOSS直聘查看聊天列表(招聘端)',
|
||||
description: 'BOSS直聘查看聊天列表(招聘端/求职端)',
|
||||
domain: 'www.zhipin.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
@@ -12,23 +62,55 @@ cli({
|
||||
args: [
|
||||
{ name: 'page', type: 'int', default: 1, help: 'Page number' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Number of results' },
|
||||
{ name: 'job-id', default: '0', help: 'Filter by job ID (0=all)' },
|
||||
{ name: 'job-id', default: '0', help: 'Filter by job ID (0=all, boss side only)' },
|
||||
{ name: 'side', default: 'auto', choices: ['auto', 'boss', 'geek'], help: 'Identity side: auto (default), boss (recruiter), or geek (job-seeker)' },
|
||||
],
|
||||
columns: ['name', 'job', 'last_msg', 'last_time', 'uid', 'security_id'],
|
||||
columns: ['name', 'company', 'job', 'title', 'last_msg', 'last_time', 'uid', 'security_id'],
|
||||
func: async (page, kwargs) => {
|
||||
requirePage(page);
|
||||
const limit = readPositiveInteger(kwargs.limit, 'chatlist --limit', 20, 100);
|
||||
const pageNum = readPositiveInteger(kwargs.page, 'chatlist --page', 1);
|
||||
const side = kwargs.side || 'auto';
|
||||
|
||||
if (side === 'boss') {
|
||||
await navigateToChat(page);
|
||||
const friends = await fetchFriendList(page, {
|
||||
pageNum,
|
||||
jobId: kwargs['job-id'] || '0',
|
||||
});
|
||||
if (friends.length === 0)
|
||||
throw new EmptyResultError('boss chatlist', 'No recruiter-side chat sessions were returned.');
|
||||
return friends.slice(0, limit).map(mapBossRow);
|
||||
}
|
||||
|
||||
if (side === 'geek') {
|
||||
await navigateToGeekChat(page);
|
||||
const rows = await buildGeekRows(page, limit);
|
||||
if (rows.length === 0)
|
||||
throw new EmptyResultError('boss chatlist', 'No job-seeker-side chat sessions were returned.');
|
||||
return rows;
|
||||
}
|
||||
|
||||
// auto: try recruiter first, fall back to geek on identity mismatch
|
||||
await navigateToChat(page);
|
||||
const friends = await fetchFriendList(page, {
|
||||
pageNum: kwargs.page || 1,
|
||||
const bossResult = await fetchFriendList(page, {
|
||||
pageNum,
|
||||
jobId: kwargs['job-id'] || '0',
|
||||
allowNonZero: true,
|
||||
});
|
||||
return friends.slice(0, kwargs.limit || 20).map((f) => ({
|
||||
name: f.name || '',
|
||||
job: f.jobName || '',
|
||||
last_msg: f.lastMessageInfo?.text || '',
|
||||
last_time: f.lastTime || '',
|
||||
uid: f.encryptUid || '',
|
||||
security_id: f.securityId || '',
|
||||
}));
|
||||
if (Array.isArray(bossResult)) {
|
||||
if (bossResult.length === 0)
|
||||
throw new EmptyResultError('boss chatlist', 'No recruiter-side chat sessions were returned.');
|
||||
return bossResult.slice(0, limit).map(mapBossRow);
|
||||
}
|
||||
if (bossResult.code === IDENTITY_MISMATCH_CODE) {
|
||||
await navigateToGeekChat(page);
|
||||
const rows = await buildGeekRows(page, limit);
|
||||
if (rows.length === 0)
|
||||
throw new EmptyResultError('boss chatlist', 'No job-seeker-side chat sessions were returned.');
|
||||
return rows;
|
||||
}
|
||||
assertOk(bossResult);
|
||||
throw new CommandExecutionError('Boss chatlist returned an unexpected response');
|
||||
},
|
||||
});
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import './chatlist.js';
|
||||
|
||||
const BOSS_FRIEND = {
|
||||
name: '张三',
|
||||
jobName: '后端工程师',
|
||||
lastMessageInfo: { text: '你好' },
|
||||
lastTime: '2024-01-01 10:00',
|
||||
encryptUid: 'enc-boss-uid',
|
||||
securityId: 'boss-sec-id',
|
||||
};
|
||||
|
||||
const GEEK_LABEL_FRIEND = {
|
||||
friendId: 12345,
|
||||
name: '李四',
|
||||
brandName: '字节跳动',
|
||||
jobName: '产品经理',
|
||||
bossTitle: 'HR',
|
||||
lastMsg: '感谢投递',
|
||||
updateTime: 1704067200000,
|
||||
encryptFriendId: 'enc-geek-uid',
|
||||
};
|
||||
|
||||
const GEEK_ENRICHED = {
|
||||
friendId: 12345,
|
||||
uid: 99999,
|
||||
name: '李四',
|
||||
brandName: '字节跳动',
|
||||
jobName: '产品经理',
|
||||
bossTitle: 'HR总监',
|
||||
encryptUid: 'enc-geek-uid',
|
||||
securityId: 'geek-sec-id',
|
||||
lastMessageInfo: { showText: '感谢投递', msgTime: 1704067200000 },
|
||||
lastTime: '2024-01-01',
|
||||
};
|
||||
|
||||
function createPageMock(evaluateImpl) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockImplementation(evaluateImpl),
|
||||
};
|
||||
}
|
||||
|
||||
describe('boss chatlist', () => {
|
||||
const command = getRegistry().get('boss/chatlist');
|
||||
|
||||
it('--side boss preserves existing behavior with 8-column output', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'boss' });
|
||||
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/chat/index'));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({
|
||||
name: '张三',
|
||||
company: '',
|
||||
job: '后端工程师',
|
||||
title: '',
|
||||
last_msg: '你好',
|
||||
uid: 'enc-boss-uid',
|
||||
security_id: 'boss-sec-id',
|
||||
});
|
||||
});
|
||||
|
||||
it('--side geek maps enriched getGeekFriendList data into 8 columns', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_ENRICHED] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' });
|
||||
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/geek/chat'));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({
|
||||
name: '李四',
|
||||
company: '字节跳动',
|
||||
job: '产品经理',
|
||||
title: 'HR总监',
|
||||
uid: 'enc-geek-uid',
|
||||
security_id: 'geek-sec-id',
|
||||
});
|
||||
});
|
||||
|
||||
it('--side geek falls back to label fields when enrichment has no match', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' });
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].name).toBe('李四');
|
||||
expect(rows[0].company).toBe('字节跳动');
|
||||
expect(rows[0].security_id).toBe('');
|
||||
});
|
||||
|
||||
it('rejects invalid --limit before navigating', async () => {
|
||||
const page = createPageMock(async () => ({}));
|
||||
await expect(
|
||||
command.func(page, { page: 1, limit: 0, 'job-id': '0', side: 'geek' })
|
||||
).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('--side geek reports a true empty chat list as EmptyResultError', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' })
|
||||
).rejects.toBeInstanceOf(EmptyResultError);
|
||||
});
|
||||
|
||||
it('treats malformed geek enrichment payload as CommandExecutionError', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: {} };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'geek' })
|
||||
).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('treats null Boss API payload as CommandExecutionError', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) return null;
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'boss' })
|
||||
).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('maps expired Boss cookies to AuthRequiredError', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 7, message: 'Cookie 已过期' };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'boss' })
|
||||
).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
});
|
||||
|
||||
it('--side auto falls back to geek when recruiter returns code 24', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 24, message: '请切换身份后再试' };
|
||||
}
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_LABEL_FRIEND] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_ENRICHED] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'auto' });
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].company).toBe('字节跳动');
|
||||
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/geek/chat'));
|
||||
});
|
||||
|
||||
it('--side auto uses recruiter results when code 0 and does not call geek API', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { page: 1, limit: 20, 'job-id': '0', side: 'auto' });
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].name).toBe('张三');
|
||||
const evaluateCalls = page.evaluate.mock.calls.map((c) => c[0]);
|
||||
expect(evaluateCalls.some((s) => s.includes('geekFilterByLabel'))).toBe(false);
|
||||
});
|
||||
|
||||
it('registers --side as a choices-constrained arg defaulting to auto', () => {
|
||||
const sideArg = command.args.find((a) => a.name === 'side');
|
||||
expect(sideArg?.choices).toEqual(['auto', 'boss', 'geek']);
|
||||
expect(sideArg?.default).toBe('auto');
|
||||
});
|
||||
});
|
||||
+98
-24
@@ -1,10 +1,72 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { requirePage, navigateToChat, bossFetch, findFriendByUid } from './utils.js';
|
||||
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import {
|
||||
requirePage, navigateToChat, navigateToGeekChat,
|
||||
bossFetch, findFriendByUid, findGeekFriendByUid,
|
||||
fetchGeekHistoryMsg, readEncryptSystemId,
|
||||
assertOk, IDENTITY_MISMATCH_CODE,
|
||||
readPositiveInteger, readRequiredString,
|
||||
} from './utils.js';
|
||||
|
||||
const TYPE_MAP = {
|
||||
1: '文本', 2: '图片', 3: '招呼', 4: '简历', 5: '系统',
|
||||
6: '名片', 7: '语音', 8: '视频', 9: '表情',
|
||||
};
|
||||
|
||||
function mapBossMsg(m, friend) {
|
||||
const fromObj = m.from || {};
|
||||
const isSelf = typeof fromObj === 'object' ? fromObj.uid !== friend.uid : false;
|
||||
return {
|
||||
from: isSelf ? '我' : (typeof fromObj === 'object' ? fromObj.name : friend.name),
|
||||
type: TYPE_MAP[m.type] || `其他(${m.type})`,
|
||||
text: m.text || m.body?.text || '',
|
||||
time: m.time ? new Date(m.time).toLocaleString('zh-CN') : '',
|
||||
};
|
||||
}
|
||||
|
||||
function mapGeekMsg(m, friend) {
|
||||
const fromUid = m.from && m.from.uid;
|
||||
const isFromBoss = fromUid != null && String(fromUid) === String(friend.uid);
|
||||
return {
|
||||
from: isFromBoss ? '对方' : '我',
|
||||
type: TYPE_MAP[m.type] || `其他(${m.type})`,
|
||||
text: m.text || m.body?.text || m.body?.content || m.body?.showText ||
|
||||
JSON.stringify(m.body || {}).slice(0, 120),
|
||||
time: m.time ? new Date(m.time).toLocaleString('zh-CN') : '',
|
||||
};
|
||||
}
|
||||
|
||||
async function bossChatMsg(page, kwargs, existingFriend) {
|
||||
const friend = existingFriend ?? await findFriendByUid(page, kwargs.uid);
|
||||
if (!friend) throw new EmptyResultError('boss chatmsg', '未找到该候选人');
|
||||
if (!friend.securityId) throw new CommandExecutionError('该聊天缺少 securityId,无法获取历史消息');
|
||||
const gid = friend.uid;
|
||||
const securityId = encodeURIComponent(friend.securityId);
|
||||
const msgUrl = `https://www.zhipin.com/wapi/zpchat/boss/historyMsg?gid=${gid}&securityId=${securityId}&page=${kwargs.page}&c=20&src=0`;
|
||||
const msgData = await bossFetch(page, msgUrl);
|
||||
const messages = msgData.zpData?.messages ?? msgData.zpData?.historyMsgList;
|
||||
if (!Array.isArray(messages)) {
|
||||
throw new CommandExecutionError('Boss recruiter history response did not include a message list');
|
||||
}
|
||||
if (messages.length === 0) {
|
||||
throw new EmptyResultError('boss chatmsg', 'Boss returned no messages for this chat.');
|
||||
}
|
||||
return messages.map((m) => mapBossMsg(m, friend));
|
||||
}
|
||||
|
||||
async function geekChatMsg(page, kwargs, encryptSystemId) {
|
||||
const friend = await findGeekFriendByUid(page, kwargs.uid, { encryptSystemId });
|
||||
if (!friend) throw new EmptyResultError('boss chatmsg', '未找到该聊天(geek 侧)');
|
||||
if (!friend.securityId) throw new CommandExecutionError('该聊天缺少 securityId,无法获取历史消息');
|
||||
const messages = await fetchGeekHistoryMsg(page, friend, { page: kwargs.page });
|
||||
return messages.map((m) => mapGeekMsg(m, friend));
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'boss',
|
||||
name: 'chatmsg',
|
||||
access: 'read',
|
||||
description: 'BOSS直聘查看与候选人的聊天消息',
|
||||
description: 'BOSS直聘查看聊天消息历史(招聘端/求职端)',
|
||||
domain: 'www.zhipin.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
@@ -12,32 +74,44 @@ cli({
|
||||
args: [
|
||||
{ name: 'uid', required: true, positional: true, help: 'Encrypted UID (from chatlist)' },
|
||||
{ name: 'page', type: 'int', default: 1, help: 'Page number' },
|
||||
{ name: 'side', default: 'auto', choices: ['auto', 'boss', 'geek'], help: 'Identity side: auto (default), boss (recruiter), or geek (job-seeker)' },
|
||||
],
|
||||
columns: ['from', 'type', 'text', 'time'],
|
||||
func: async (page, kwargs) => {
|
||||
requirePage(page);
|
||||
const uid = readRequiredString(kwargs.uid, 'chatmsg uid');
|
||||
const pageNum = readPositiveInteger(kwargs.page, 'chatmsg --page', 1);
|
||||
const normalizedKwargs = { ...kwargs, uid, page: pageNum };
|
||||
const side = kwargs.side || 'auto';
|
||||
|
||||
if (side === 'boss') {
|
||||
await navigateToChat(page);
|
||||
return await bossChatMsg(page, normalizedKwargs);
|
||||
}
|
||||
|
||||
if (side === 'geek') {
|
||||
await navigateToGeekChat(page);
|
||||
const encryptSystemId = await readEncryptSystemId(page);
|
||||
return await geekChatMsg(page, normalizedKwargs, encryptSystemId);
|
||||
}
|
||||
|
||||
// auto: try recruiter first, fall back to geek when not found or identity mismatch
|
||||
await navigateToChat(page);
|
||||
const friend = await findFriendByUid(page, kwargs.uid);
|
||||
if (!friend)
|
||||
throw new Error('未找到该候选人');
|
||||
const gid = friend.uid;
|
||||
const securityId = encodeURIComponent(friend.securityId);
|
||||
const msgUrl = `https://www.zhipin.com/wapi/zpchat/boss/historyMsg?gid=${gid}&securityId=${securityId}&page=${kwargs.page}&c=20&src=0`;
|
||||
const msgData = await bossFetch(page, msgUrl);
|
||||
const TYPE_MAP = {
|
||||
1: '文本', 2: '图片', 3: '招呼', 4: '简历', 5: '系统',
|
||||
6: '名片', 7: '语音', 8: '视频', 9: '表情',
|
||||
};
|
||||
const messages = msgData.zpData?.messages || msgData.zpData?.historyMsgList || [];
|
||||
return messages.map((m) => {
|
||||
const fromObj = m.from || {};
|
||||
const isSelf = typeof fromObj === 'object' ? fromObj.uid !== friend.uid : false;
|
||||
return {
|
||||
from: isSelf ? '我' : (typeof fromObj === 'object' ? fromObj.name : friend.name),
|
||||
type: TYPE_MAP[m.type] || '其他(' + m.type + ')',
|
||||
text: m.text || m.body?.text || '',
|
||||
time: m.time ? new Date(m.time).toLocaleString('zh-CN') : '',
|
||||
};
|
||||
});
|
||||
const bossResult = await findFriendByUid(page, uid, { allowNonZero: true });
|
||||
if (bossResult?.friend) {
|
||||
return await bossChatMsg(page, normalizedKwargs, bossResult.friend);
|
||||
}
|
||||
// Not found or identity mismatch — check for hard errors before falling back
|
||||
if (bossResult?.code && bossResult.code !== 0 && bossResult.code !== IDENTITY_MISMATCH_CODE) {
|
||||
assertOk(bossResult);
|
||||
}
|
||||
// Fall back to geek side
|
||||
await navigateToGeekChat(page);
|
||||
const encryptSystemId = await readEncryptSystemId(page);
|
||||
const geekFriend = await findGeekFriendByUid(page, uid, { encryptSystemId });
|
||||
if (!geekFriend) throw new EmptyResultError('boss chatmsg', 'uid 在招聘端与求职端聊天列表中均未找到');
|
||||
if (!geekFriend.securityId) throw new CommandExecutionError('该聊天缺少 securityId,无法获取历史消息');
|
||||
const messages = await fetchGeekHistoryMsg(page, geekFriend, { page: pageNum });
|
||||
return messages.map((m) => mapGeekMsg(m, geekFriend));
|
||||
},
|
||||
});
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import './chatmsg.js';
|
||||
|
||||
const BOSS_FRIEND = {
|
||||
uid: 12345,
|
||||
encryptUid: 'enc-boss-uid',
|
||||
securityId: 'boss-sec-id',
|
||||
name: '候选人甲',
|
||||
};
|
||||
const BOSS_MSGS = [
|
||||
{ type: 1, text: 'Hello', from: { uid: 99999, name: 'HR' }, time: 1704067200000 },
|
||||
{ type: 1, text: '感谢', from: { uid: 12345, name: '候选人甲' }, time: 1704067201000 },
|
||||
];
|
||||
|
||||
const GEEK_FRIEND_LABEL = {
|
||||
friendId: 11111,
|
||||
encryptFriendId: 'enc-geek-uid',
|
||||
name: 'Boss张',
|
||||
brandName: '公司A',
|
||||
};
|
||||
const GEEK_FRIEND_ENRICHED = {
|
||||
friendId: 11111,
|
||||
uid: 67890,
|
||||
encryptUid: 'enc-geek-uid',
|
||||
securityId: 'geek-sec-id',
|
||||
name: 'Boss张',
|
||||
};
|
||||
const GEEK_MSGS = [
|
||||
{ type: 1, text: '欢迎投递', received: true, time: 1704067200000, from: { uid: 67890, name: 'Boss张' } },
|
||||
{ type: 1, text: '谢谢', received: true, time: 1704067201000, from: { uid: 99999, name: '我' } },
|
||||
];
|
||||
|
||||
function createPageMock(evaluateImpl) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockImplementation(evaluateImpl),
|
||||
};
|
||||
}
|
||||
|
||||
describe('boss chatmsg', () => {
|
||||
const command = getRegistry().get('boss/chatmsg');
|
||||
|
||||
it('rejects empty uid before navigating', async () => {
|
||||
const page = createPageMock(async () => ({}));
|
||||
await expect(
|
||||
command.func(page, { uid: ' ', page: 1, side: 'geek' })
|
||||
).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects invalid --page before navigating', async () => {
|
||||
const page = createPageMock(async () => ({}));
|
||||
await expect(
|
||||
command.func(page, { uid: 'enc-geek-uid', page: 0, side: 'geek' })
|
||||
).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('--side boss preserves existing behavior', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
|
||||
}
|
||||
if (script.includes('boss/historyMsg')) {
|
||||
return { code: 0, zpData: { messages: BOSS_MSGS } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { uid: 'enc-boss-uid', page: 1, side: 'boss' });
|
||||
expect(page.goto).toHaveBeenCalledWith(expect.stringContaining('/web/chat/index'));
|
||||
expect(rows).toHaveLength(2);
|
||||
expect(rows[0].from).toBe('我');
|
||||
expect(rows[1].from).toBe('候选人甲');
|
||||
});
|
||||
|
||||
it('--side geek calls historyMsg with bossId, securityId, page, c=20, src=0', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
|
||||
}
|
||||
if (script.includes('geek/historyMsg')) {
|
||||
return { code: 0, zpData: { messages: GEEK_MSGS } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' });
|
||||
const historyScript = page.evaluate.mock.calls.find((c) => c[0].includes('geek/historyMsg'))?.[0];
|
||||
expect(historyScript).toBeDefined();
|
||||
expect(historyScript).toContain('bossId=67890');
|
||||
expect(historyScript).toContain('securityId=');
|
||||
expect(historyScript).toContain('page=1');
|
||||
expect(historyScript).toContain('c=20');
|
||||
expect(historyScript).toContain('src=0');
|
||||
});
|
||||
|
||||
it('--side geek uses from.uid to determine direction, not received flag', async () => {
|
||||
// Both messages have received:true (mirrors real geek historyMsg API behaviour)
|
||||
// Direction is determined by whether m.from.uid matches the boss's uid (67890)
|
||||
const msgsAllReceived = [
|
||||
{ type: 1, text: '欢迎投递', received: true, time: 1704067200000, from: { uid: 67890, name: 'Boss张' } },
|
||||
{ type: 1, text: '谢谢', received: true, time: 1704067201000, from: { uid: 99999, name: '我' } },
|
||||
];
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
|
||||
}
|
||||
if (script.includes('geek/historyMsg')) {
|
||||
return { code: 0, zpData: { messages: msgsAllReceived } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' });
|
||||
// from.uid=67890 matches friend.uid=67890 → boss sent it → '对方'
|
||||
expect(rows[0].from).toBe('对方');
|
||||
// from.uid=99999 does not match → geek sent it → '我'
|
||||
expect(rows[1].from).toBe('我');
|
||||
});
|
||||
|
||||
it('non-text message body does not crash and produces truncated JSON', async () => {
|
||||
const nonTextMsg = { type: 99, received: true, time: 1704067200000, body: { action: 'resume_request', detail: 'X' } };
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
|
||||
}
|
||||
if (script.includes('geek/historyMsg')) {
|
||||
return { code: 0, zpData: { messages: [nonTextMsg] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' });
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].text).toContain('resume_request');
|
||||
});
|
||||
|
||||
it('--side auto falls back to geek when recruiter returns code 24', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 24, message: '请切换身份后再试' };
|
||||
}
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
|
||||
}
|
||||
if (script.includes('geek/historyMsg')) {
|
||||
return { code: 0, zpData: { messages: GEEK_MSGS } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
const rows = await command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'auto' });
|
||||
expect(rows).toHaveLength(2);
|
||||
expect(rows[0].from).toBe('对方');
|
||||
});
|
||||
|
||||
it('--side geek throws when uid is not found in geek chat list', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { uid: 'unknown-uid', page: 1, side: 'geek' })
|
||||
).rejects.toBeInstanceOf(EmptyResultError);
|
||||
});
|
||||
|
||||
it('--side boss maps expired cookies to AuthRequiredError', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 7, message: 'Cookie 已过期' };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { uid: 'enc-boss-uid', page: 1, side: 'boss' })
|
||||
).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
});
|
||||
|
||||
it('--side boss treats missing history list as parser drift', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('getBossFriendListV2')) {
|
||||
return { code: 0, zpData: { friendList: [BOSS_FRIEND] } };
|
||||
}
|
||||
if (script.includes('boss/historyMsg')) {
|
||||
return { code: 0, zpData: {} };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { uid: 'enc-boss-uid', page: 1, side: 'boss' })
|
||||
).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('--side geek reports an empty history as EmptyResultError', async () => {
|
||||
const page = createPageMock(async (script) => {
|
||||
if (script.includes('document.cookie')) return 'test-enc-sys-id';
|
||||
if (script.includes('geekFilterByLabel')) {
|
||||
return { code: 0, zpData: { friendList: [GEEK_FRIEND_LABEL] } };
|
||||
}
|
||||
if (script.includes('getGeekFriendList.json')) {
|
||||
return { code: 0, zpData: { result: [GEEK_FRIEND_ENRICHED] } };
|
||||
}
|
||||
if (script.includes('geek/historyMsg')) {
|
||||
return { code: 0, zpData: { messages: [] } };
|
||||
}
|
||||
return {};
|
||||
});
|
||||
await expect(
|
||||
command.func(page, { uid: 'enc-geek-uid', page: 1, side: 'geek' })
|
||||
).rejects.toBeInstanceOf(EmptyResultError);
|
||||
});
|
||||
});
|
||||
+257
-12
@@ -1,8 +1,11 @@
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
|
||||
// ── Constants ───────────────────────────────────────────────────────────────
|
||||
const BOSS_DOMAIN = 'www.zhipin.com';
|
||||
const CHAT_URL = `https://${BOSS_DOMAIN}/web/chat/index`;
|
||||
const COOKIE_EXPIRED_CODES = new Set([7, 37]);
|
||||
const COOKIE_EXPIRED_MSG = 'Cookie 已过期!请在当前 Chrome 浏览器中重新登录 BOSS 直聘。';
|
||||
const RECRUITER_ONLY_MSG = '该命令仅支持招聘端(BOSS 端)账号,请使用招聘者账号登录后重试。';
|
||||
const DEFAULT_TIMEOUT = 15_000;
|
||||
// ── Core helpers ────────────────────────────────────────────────────────────
|
||||
/**
|
||||
@@ -10,7 +13,24 @@ const DEFAULT_TIMEOUT = 15_000;
|
||||
*/
|
||||
export function requirePage(page) {
|
||||
if (!page)
|
||||
throw new Error('Browser page required');
|
||||
throw new CommandExecutionError('Browser page required');
|
||||
}
|
||||
export function readPositiveInteger(raw, name, fallback, max) {
|
||||
const value = raw === undefined || raw === null || raw === '' ? fallback : Number(raw);
|
||||
if (!Number.isInteger(value) || value < 1) {
|
||||
throw new ArgumentError(`boss ${name} must be a positive integer`);
|
||||
}
|
||||
if (max !== undefined && value > max) {
|
||||
throw new ArgumentError(`boss ${name} must be <= ${max}`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
export function readRequiredString(raw, name) {
|
||||
const value = String(raw ?? '').trim();
|
||||
if (!value) {
|
||||
throw new ArgumentError(`boss ${name} cannot be empty`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
/**
|
||||
* Navigate to BOSS chat page and wait for it to settle.
|
||||
@@ -33,19 +53,37 @@ export async function navigateTo(page, url, waitSeconds = 1) {
|
||||
*/
|
||||
export function checkAuth(data) {
|
||||
if (COOKIE_EXPIRED_CODES.has(data.code)) {
|
||||
throw new Error(COOKIE_EXPIRED_MSG);
|
||||
throw new AuthRequiredError(BOSS_DOMAIN, COOKIE_EXPIRED_MSG);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Map BOSS code=24 ("请切换身份后再试") to a typed AuthRequiredError.
|
||||
* Recruiter-only commands (recommend, joblist, stats, resume, mark,
|
||||
* exchange, invite, greet, batchgreet) have no geek-side equivalent;
|
||||
* surfacing this as a generic COMMAND_EXEC hides what the user must do.
|
||||
* chatlist / chatmsg avoid this path by using `allowNonZero: true` and
|
||||
* branching to the geek-side fetch when they see code 24.
|
||||
*/
|
||||
function checkRecruiterSide(data) {
|
||||
if (data.code === IDENTITY_MISMATCH_CODE) {
|
||||
throw new AuthRequiredError(BOSS_DOMAIN, RECRUITER_ONLY_MSG);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Throw if the API response is not code 0.
|
||||
* Checks for cookie expiry first, then throws with the provided message.
|
||||
* Checks for cookie expiry first, then identity mismatch, then throws
|
||||
* with the provided message.
|
||||
*/
|
||||
export function assertOk(data, errorPrefix) {
|
||||
if (!data || typeof data !== 'object') {
|
||||
throw new CommandExecutionError(`${errorPrefix ? `${errorPrefix}: ` : ''}Boss API returned malformed response`);
|
||||
}
|
||||
if (data.code === 0)
|
||||
return;
|
||||
checkAuth(data);
|
||||
checkRecruiterSide(data);
|
||||
const prefix = errorPrefix ? `${errorPrefix}: ` : '';
|
||||
throw new Error(`${prefix}${data.message || 'Unknown error'} (code=${data.code})`);
|
||||
throw new CommandExecutionError(`${prefix}${data.message || 'Unknown error'} (code=${data.code})`);
|
||||
}
|
||||
/**
|
||||
* Make a credentialed XHR request via page.evaluate().
|
||||
@@ -80,7 +118,19 @@ export async function bossFetch(page, url, opts = {}) {
|
||||
});
|
||||
}
|
||||
`;
|
||||
const data = await page.evaluate(script);
|
||||
let data;
|
||||
try {
|
||||
data = await page.evaluate(script);
|
||||
} catch (error) {
|
||||
if (error instanceof AuthRequiredError || error instanceof CommandExecutionError) {
|
||||
throw error;
|
||||
}
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
throw new CommandExecutionError(`Boss API request failed: ${message}`);
|
||||
}
|
||||
if (!data || typeof data !== 'object') {
|
||||
throw new CommandExecutionError('Boss API returned malformed response');
|
||||
}
|
||||
// Auto-check auth unless caller opts out
|
||||
if (!opts.allowNonZero && data.code !== 0) {
|
||||
assertOk(data);
|
||||
@@ -95,8 +145,13 @@ export async function fetchFriendList(page, opts = {}) {
|
||||
const pageNum = opts.pageNum ?? 1;
|
||||
const jobId = opts.jobId ?? '0';
|
||||
const url = `https://${BOSS_DOMAIN}/wapi/zprelation/friend/getBossFriendListV2.json?page=${pageNum}&status=0&jobId=${jobId}`;
|
||||
const data = await bossFetch(page, url);
|
||||
return data.zpData?.friendList || [];
|
||||
const data = await bossFetch(page, url, { allowNonZero: opts.allowNonZero });
|
||||
if (opts.allowNonZero && data.code !== 0) return data;
|
||||
const list = data.zpData?.friendList;
|
||||
if (!Array.isArray(list)) {
|
||||
throw new CommandExecutionError('Boss friend list response did not include zpData.friendList');
|
||||
}
|
||||
return list;
|
||||
}
|
||||
/**
|
||||
* Fetch the recommended candidates (greetRecSortList).
|
||||
@@ -104,7 +159,11 @@ export async function fetchFriendList(page, opts = {}) {
|
||||
export async function fetchRecommendList(page) {
|
||||
const url = `https://${BOSS_DOMAIN}/wapi/zprelation/friend/greetRecSortList`;
|
||||
const data = await bossFetch(page, url);
|
||||
return data.zpData?.friendList || [];
|
||||
const list = data.zpData?.friendList;
|
||||
if (!Array.isArray(list)) {
|
||||
throw new CommandExecutionError('Boss recommend response did not include zpData.friendList');
|
||||
}
|
||||
return list;
|
||||
}
|
||||
/**
|
||||
* Find a friend by encryptUid, searching through friend list and optionally greet list.
|
||||
@@ -115,10 +174,14 @@ export async function findFriendByUid(page, encryptUid, opts = {}) {
|
||||
const checkGreetList = opts.checkGreetList ?? false;
|
||||
// Search friend list pages
|
||||
for (let p = 1; p <= maxPages; p++) {
|
||||
const friends = await fetchFriendList(page, { pageNum: p });
|
||||
const result = await fetchFriendList(page, { pageNum: p, allowNonZero: opts.allowNonZero });
|
||||
if (opts.allowNonZero && !Array.isArray(result)) {
|
||||
return { friend: null, code: result.code };
|
||||
}
|
||||
const friends = Array.isArray(result) ? result : [];
|
||||
const found = friends.find((f) => f.encryptUid === encryptUid);
|
||||
if (found)
|
||||
return found;
|
||||
return opts.allowNonZero ? { friend: found, code: 0 } : found;
|
||||
if (friends.length === 0)
|
||||
break;
|
||||
}
|
||||
@@ -127,9 +190,9 @@ export async function findFriendByUid(page, encryptUid, opts = {}) {
|
||||
const greetList = await fetchRecommendList(page);
|
||||
const found = greetList.find((f) => f.encryptUid === encryptUid);
|
||||
if (found)
|
||||
return found;
|
||||
return opts.allowNonZero ? { friend: found, code: 0 } : found;
|
||||
}
|
||||
return null;
|
||||
return opts.allowNonZero ? { friend: null, code: 0 } : null;
|
||||
}
|
||||
// ── UI automation helpers ───────────────────────────────────────────────────
|
||||
/**
|
||||
@@ -221,3 +284,185 @@ export function verbose(msg) {
|
||||
console.error(`[opencli:boss] ${msg}`);
|
||||
}
|
||||
}
|
||||
// ── Geek-side helpers ────────────────────────────────────────────────────────
|
||||
export const IDENTITY_MISMATCH_CODE = 24;
|
||||
const GEEK_CHAT_URL = `https://${BOSS_DOMAIN}/web/geek/chat`;
|
||||
/**
|
||||
* Navigate to the job-seeker chat page.
|
||||
* Establishes the cookie + JS-global context needed for geek-side API calls.
|
||||
*/
|
||||
export async function navigateToGeekChat(page, waitSeconds = 2) {
|
||||
await page.goto(GEEK_CHAT_URL);
|
||||
await page.wait({ time: waitSeconds });
|
||||
}
|
||||
/**
|
||||
* Read the encryptSystemId value required by the geek-side list API.
|
||||
* Strategy (in order):
|
||||
* 1. Vue app state / Pinia stores / $route.query (Option 1 — runtime source)
|
||||
* 2. performance.getEntriesByType('resource') — parse from geekFilterByLabel URL
|
||||
* that the page itself already issued (Option 2 — most deterministic)
|
||||
* 3. cookie, inline <script> SSR state, known window globals, localStorage (fallbacks)
|
||||
* Returns empty string if nothing is found; the API may still succeed without it.
|
||||
* Caller must have navigated to the geek chat page first.
|
||||
*/
|
||||
export async function readEncryptSystemId(page) {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
// 1. Vue app state / Pinia / $route.query
|
||||
// The chat component reads encryptSystemId from the app runtime to build
|
||||
// its own geekFilterByLabel request, so the value lives in the Vue tree.
|
||||
try {
|
||||
const appEl = document.querySelector('#app') || document.querySelector('[data-v-app]');
|
||||
const vueApp = appEl && (appEl.__vue_app__ || appEl._vei);
|
||||
if (vueApp) {
|
||||
// 1a. Pinia stores (Vue 3 standard state management on BOSS直聘)
|
||||
const pinia = vueApp.config && vueApp.config.globalProperties.$pinia;
|
||||
if (pinia && pinia.state && pinia.state.value) {
|
||||
for (const store of Object.values(pinia.state.value)) {
|
||||
try {
|
||||
const flat = JSON.stringify(store);
|
||||
if (flat.includes('encryptSystemId')) {
|
||||
const m = flat.match(/"encryptSystemId":"([^"]+)"/);
|
||||
if (m) return m[1];
|
||||
}
|
||||
} catch (_) {}
|
||||
}
|
||||
}
|
||||
// 1b. Vue Router current route query
|
||||
const router = vueApp.config && vueApp.config.globalProperties.$router;
|
||||
const query = router && router.currentRoute && router.currentRoute.value && router.currentRoute.value.query;
|
||||
if (query && query.encryptSystemId) return query.encryptSystemId;
|
||||
}
|
||||
} catch (_) {}
|
||||
// 2. Performance resource entries — the page already issued geekFilterByLabel
|
||||
// with encryptSystemId in the URL; read it back from the resource timing API.
|
||||
try {
|
||||
const entries = performance.getEntriesByType('resource');
|
||||
for (const entry of entries) {
|
||||
if (!entry.name.includes('geekFilterByLabel')) continue;
|
||||
const u = new URL(entry.name);
|
||||
const v = u.searchParams.get('encryptSystemId');
|
||||
if (v) return v;
|
||||
}
|
||||
} catch (_) {}
|
||||
// 3. cookie
|
||||
try {
|
||||
const m = document.cookie.match(/encryptSystemId=([^;]+)/i);
|
||||
if (m) return decodeURIComponent(m[1]);
|
||||
} catch (_) {}
|
||||
// 4. inline <script> SSR state (Nuxt embeds server state here)
|
||||
try {
|
||||
for (const s of document.querySelectorAll('script:not([src])')) {
|
||||
const t = s.textContent || '';
|
||||
if (!t.includes('encryptSystemId')) continue;
|
||||
const m = t.match(/"encryptSystemId":"([^"]+)"/);
|
||||
if (m) return m[1];
|
||||
}
|
||||
} catch (_) {}
|
||||
// 5. known BOSS / Nuxt window globals
|
||||
const KNOWN = [
|
||||
'__NUXT__', '__INITIAL_STATE__', '__ZP_INFO__', '__BOSS_ZP__',
|
||||
'pageGlobalVar', 'ZP_DATA', '__ZP_DATA__', '__PAGE_DATA__',
|
||||
];
|
||||
for (const k of KNOWN) {
|
||||
const obj = window[k];
|
||||
if (!obj || typeof obj !== 'object') continue;
|
||||
try {
|
||||
const flat = JSON.stringify(obj);
|
||||
if (!flat.includes('encryptSystemId')) continue;
|
||||
const m = flat.match(/"encryptSystemId":"([^"]+)"/);
|
||||
if (m) return m[1];
|
||||
} catch (_) {}
|
||||
}
|
||||
// 6. localStorage
|
||||
try {
|
||||
for (let i = 0; i < localStorage.length; i++) {
|
||||
const k = localStorage.key(i);
|
||||
if (!k) continue;
|
||||
if (k.toLowerCase().includes('encryptsystemid')) {
|
||||
const v = localStorage.getItem(k);
|
||||
if (v) return v;
|
||||
}
|
||||
const v = localStorage.getItem(k) || '';
|
||||
if (v.includes('encryptSystemId')) {
|
||||
const m = v.match(/"encryptSystemId":"([^"]+)"/);
|
||||
if (m) return m[1];
|
||||
}
|
||||
}
|
||||
} catch (_) {}
|
||||
return '';
|
||||
})()
|
||||
`);
|
||||
return result || '';
|
||||
}
|
||||
/**
|
||||
* Fetch the job-seeker chat list (brief info, no securityId).
|
||||
* Use fetchGeekFriendInfoList to enrich with securityId before calling chatmsg.
|
||||
*/
|
||||
export async function fetchGeekFriendLabelList(page, opts = {}) {
|
||||
const labelId = opts.labelId ?? 0;
|
||||
const encryptSystemId = opts.encryptSystemId ?? '';
|
||||
const url = `https://${BOSS_DOMAIN}/wapi/zprelation/friend/geekFilterByLabel?labelId=${labelId}&encryptSystemId=${encodeURIComponent(encryptSystemId)}`;
|
||||
const data = await bossFetch(page, url, { allowNonZero: opts.allowNonZero });
|
||||
if (opts.allowNonZero && data.code !== 0) return data;
|
||||
const list = data.zpData?.friendList;
|
||||
if (!Array.isArray(list)) {
|
||||
throw new CommandExecutionError('Boss geek chat list response did not include zpData.friendList');
|
||||
}
|
||||
return list;
|
||||
}
|
||||
/**
|
||||
* Enrich a batch of geek friends with full fields including securityId.
|
||||
* Processes in batches of 50 to avoid oversized request bodies.
|
||||
*/
|
||||
export async function fetchGeekFriendInfoList(page, friendIds = []) {
|
||||
if (!friendIds.length) return [];
|
||||
const BATCH_SIZE = 50;
|
||||
const results = [];
|
||||
for (let i = 0; i < friendIds.length; i += BATCH_SIZE) {
|
||||
const batch = friendIds.slice(i, i + BATCH_SIZE).map(String);
|
||||
const body = `friendIds=${batch.join(',')}`;
|
||||
const data = await bossFetch(page, `https://${BOSS_DOMAIN}/wapi/zprelation/friend/getGeekFriendList.json`, {
|
||||
method: 'POST',
|
||||
body,
|
||||
});
|
||||
const batchResult = data.zpData?.result;
|
||||
if (!Array.isArray(batchResult)) {
|
||||
throw new CommandExecutionError('Boss geek friend enrichment response did not include zpData.result');
|
||||
}
|
||||
results.push(...batchResult);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
/**
|
||||
* Find a geek-side friend by encrypted uid.
|
||||
* Merges label-list and enriched data; returns null if not found.
|
||||
*/
|
||||
export async function findGeekFriendByUid(page, encryptUid, opts = {}) {
|
||||
const labelList = await fetchGeekFriendLabelList(page, { encryptSystemId: opts.encryptSystemId });
|
||||
const candidate = labelList.find((f) => f.encryptFriendId === encryptUid ||
|
||||
String(f.uid) === String(encryptUid) ||
|
||||
String(f.friendId) === String(encryptUid));
|
||||
if (!candidate) return null;
|
||||
const enriched = await fetchGeekFriendInfoList(page, [candidate.friendId]);
|
||||
return { ...candidate, ...(enriched[0] || {}) };
|
||||
}
|
||||
/**
|
||||
* Fetch message history for a geek-side chat.
|
||||
* friend must have .uid (boss's numeric id) and .securityId.
|
||||
*/
|
||||
export async function fetchGeekHistoryMsg(page, friend, opts = {}) {
|
||||
const pageNum = opts.page ?? 1;
|
||||
const bossId = friend.uid;
|
||||
const securityId = encodeURIComponent(friend.securityId || '');
|
||||
const url = `https://${BOSS_DOMAIN}/wapi/zpchat/geek/historyMsg?bossId=${bossId}&securityId=${securityId}&page=${pageNum}&c=20&src=0`;
|
||||
const data = await bossFetch(page, url);
|
||||
const messages = data.zpData?.messages ?? data.zpData?.historyMsgList;
|
||||
if (!Array.isArray(messages)) {
|
||||
throw new CommandExecutionError('Boss geek history response did not include a message list');
|
||||
}
|
||||
if (messages.length === 0) {
|
||||
throw new EmptyResultError('boss chatmsg', 'Boss returned no messages for this chat.');
|
||||
}
|
||||
return messages;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { assertOk } from './utils.js';
|
||||
|
||||
describe('assertOk', () => {
|
||||
it('returns silently on code 0', () => {
|
||||
expect(() => assertOk({ code: 0 })).not.toThrow();
|
||||
});
|
||||
|
||||
it('maps expired cookie codes (7, 37) to AuthRequiredError', () => {
|
||||
expect(() => assertOk({ code: 7, message: 'expired' })).toThrow(AuthRequiredError);
|
||||
expect(() => assertOk({ code: 37, message: 'expired' })).toThrow(AuthRequiredError);
|
||||
});
|
||||
|
||||
it('maps code 24 (identity mismatch) to AuthRequiredError with recruiter-only hint', () => {
|
||||
try {
|
||||
assertOk({ code: 24, message: '请切换身份后再试' });
|
||||
throw new Error('assertOk should have thrown');
|
||||
} catch (err) {
|
||||
expect(err).toBeInstanceOf(AuthRequiredError);
|
||||
expect(String(err.message)).toContain('招聘端');
|
||||
}
|
||||
});
|
||||
|
||||
it('falls through to CommandExecutionError for other non-zero codes', () => {
|
||||
expect(() => assertOk({ code: 99, message: 'something else' }))
|
||||
.toThrow(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('throws CommandExecutionError on malformed (non-object) response', () => {
|
||||
expect(() => assertOk(null)).toThrow(CommandExecutionError);
|
||||
expect(() => assertOk('not-an-object')).toThrow(CommandExecutionError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,80 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import {
|
||||
emptySearchResults,
|
||||
requireBoundedInteger,
|
||||
requireNonNegativeInteger,
|
||||
requireRows,
|
||||
requireSearchQuery,
|
||||
runBrowserStep,
|
||||
toHttpsUrl,
|
||||
} from '../_shared/search-adapter.js';
|
||||
|
||||
function buildExtractorJs(limit) {
|
||||
return `
|
||||
(function() {
|
||||
var results = [];
|
||||
var seen = {};
|
||||
var items = document.querySelectorAll('.snippet');
|
||||
for (var i = 0; i < items.length; i++) {
|
||||
if (results.length >= ${limit}) break;
|
||||
var el = items[i];
|
||||
if (el.classList.contains('standalone') || el.classList.contains('ad')) continue;
|
||||
var titleEl = el.querySelector('.search-snippet-title');
|
||||
var snippetEl = el.querySelector('.generic-snippet .content');
|
||||
var linkEl = el.querySelector('.result-content a');
|
||||
if (!titleEl) continue;
|
||||
var title = titleEl.textContent.trim();
|
||||
var href = linkEl ? linkEl.getAttribute('href') || '' : '';
|
||||
var snippet = snippetEl ? snippetEl.textContent.trim() : '';
|
||||
if (!title || !href || seen[href]) continue;
|
||||
if (href.indexOf('/') === 0) continue;
|
||||
seen[href] = true;
|
||||
results.push([title, href, snippet]);
|
||||
}
|
||||
return results;
|
||||
})()`;
|
||||
}
|
||||
|
||||
const command = cli({
|
||||
site: 'brave',
|
||||
name: 'search',
|
||||
access: 'read',
|
||||
description: 'Search Brave Search',
|
||||
domain: 'search.brave.com',
|
||||
strategy: Strategy.PUBLIC,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'keyword', positional: true, required: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of results per page (max 18)' },
|
||||
{ name: 'offset', type: 'int', default: 0, help: 'Page offset (0, 1, 2...). Brave returns ~18 results per page' },
|
||||
],
|
||||
columns: ['rank', 'title', 'url', 'snippet'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = requireBoundedInteger(kwargs.limit, 10, 1, 18, '--limit');
|
||||
const query = requireSearchQuery(kwargs.keyword);
|
||||
const keyword = encodeURIComponent(query);
|
||||
const offset = requireNonNegativeInteger(kwargs.offset, 0, '--offset');
|
||||
let url = `https://search.brave.com/search?q=${keyword}`;
|
||||
if (offset > 0) url += `&offset=${offset}`;
|
||||
await runBrowserStep('brave search navigation', () => page.goto(url));
|
||||
try {
|
||||
await page.wait({ selector: '.snippet', timeout: 10 });
|
||||
} catch {
|
||||
await page.wait(3).catch(function() {});
|
||||
}
|
||||
const raw = await runBrowserStep('brave search extraction', () => page.evaluate(buildExtractorJs(limit)));
|
||||
const results = requireRows(raw, 'brave search');
|
||||
if (results.length === 0) {
|
||||
throw emptySearchResults('Brave', query);
|
||||
}
|
||||
const rows = results
|
||||
.map(function(r, index) {
|
||||
return { rank: index + 1 + offset * 18, title: r[0], url: toHttpsUrl(r[1], 'https://search.brave.com'), snippet: r[2] };
|
||||
})
|
||||
.filter((row) => row.url);
|
||||
if (rows.length === 0) throw emptySearchResults('Brave', query);
|
||||
return rows;
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = { command };
|
||||
@@ -0,0 +1,76 @@
|
||||
import { describe, it, expect, vi } from 'vitest';
|
||||
|
||||
const { __test__ } = await import('./search.js');
|
||||
const command = __test__.command;
|
||||
|
||||
function createPageMock(evaluateResult = []) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue(evaluateResult),
|
||||
};
|
||||
}
|
||||
|
||||
describe('brave search', () => {
|
||||
it('should register as a valid command', () => {
|
||||
expect(command).toBeDefined();
|
||||
expect(command.site).toBe('brave');
|
||||
expect(command.name).toBe('search');
|
||||
expect(command.access).toBe('read');
|
||||
expect(command.browser).toBe(true);
|
||||
expect(command.strategy).toBe('public');
|
||||
expect(command.domain).toBe('search.brave.com');
|
||||
});
|
||||
|
||||
it('should define keyword positional arg', () => {
|
||||
const kwArg = command.args.find(a => a.name === 'keyword');
|
||||
expect(kwArg).toBeDefined();
|
||||
expect(kwArg.positional).toBe(true);
|
||||
expect(kwArg.required).toBe(true);
|
||||
});
|
||||
|
||||
it('should define limit arg with default 10', () => {
|
||||
const limitArg = command.args.find(a => a.name === 'limit');
|
||||
expect(limitArg).toBeDefined();
|
||||
expect(limitArg.type).toBe('int');
|
||||
expect(limitArg.default).toBe(10);
|
||||
});
|
||||
|
||||
it('should define output columns', () => {
|
||||
expect(command.columns).toContain('rank');
|
||||
expect(command.columns).toContain('title');
|
||||
expect(command.columns).toContain('url');
|
||||
expect(command.columns).toContain('snippet');
|
||||
});
|
||||
|
||||
it('rejects empty query, invalid limit, and invalid offset before navigation', async () => {
|
||||
const page = createPageMock();
|
||||
await expect(command.func(page, { keyword: '', limit: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 19 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 5, offset: -1 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('unwraps browser envelopes and returns ranked HTTPS rows', async () => {
|
||||
const page = createPageMock({
|
||||
session: 'site:brave',
|
||||
data: [['OpenCLI', 'https://github.com/jackwener/OpenCLI', 'CLI browser tooling']],
|
||||
});
|
||||
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 1, offset: 1 })).resolves.toEqual([{
|
||||
rank: 19,
|
||||
title: 'OpenCLI',
|
||||
url: 'https://github.com/jackwener/OpenCLI',
|
||||
snippet: 'CLI browser tooling',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('fails typed instead of silently returning [] for malformed extraction payloads', async () => {
|
||||
const page = createPageMock({ rows: [] });
|
||||
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 1 })).rejects.toMatchObject({
|
||||
code: 'COMMAND_EXEC',
|
||||
message: expect.stringContaining('payload shape'),
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError } from '@jackwener/opencli/errors';
|
||||
import {
|
||||
emptySearchResults,
|
||||
requireBoundedInteger,
|
||||
requireNonNegativeInteger,
|
||||
requireRows,
|
||||
requireSearchQuery,
|
||||
runBrowserStep,
|
||||
toHttpsUrl,
|
||||
} from '../_shared/search-adapter.js';
|
||||
|
||||
function decodeDdgUrl(href) {
|
||||
if (!href) return '';
|
||||
try {
|
||||
const url = new URL(href, 'https://duckduckgo.com');
|
||||
const uddg = url.searchParams.get('uddg');
|
||||
return toHttpsUrl(uddg || href, 'https://duckduckgo.com');
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
function buildExtractFn(limit) {
|
||||
return 'function(doc){' +
|
||||
'var r=[];var seen={};var items=doc.querySelectorAll(".result");' +
|
||||
'for(var i=0;i<items.length;i++){' +
|
||||
'if(r.length>=' + limit + ')break;' +
|
||||
'var el=items[i];var te=el.querySelector(".result__a");' +
|
||||
'var se=el.querySelector(".result__snippet");' +
|
||||
'var ue=el.querySelector(".result__url");' +
|
||||
'var ie=el.querySelector(".result__icon__img");' +
|
||||
'var cls=el.className||"";var rt="web";' +
|
||||
'if(cls.indexOf("result--ad")!==-1||cls.indexOf("result--ads")!==-1||cls.indexOf("badge--ad")!==-1)continue;' +
|
||||
'if(!te)continue;' +
|
||||
'var t=(te.textContent||"").trim();' +
|
||||
'var h=te.getAttribute("href")||"";' +
|
||||
'var sn=se?(se.textContent||"").trim():"";' +
|
||||
'var du=ue?(ue.textContent||"").trim():"";' +
|
||||
'var ic=ie?(ie.getAttribute("src")||""):"";' +
|
||||
'if(cls.indexOf("news-result")!==-1)rt="news";' +
|
||||
'else if(cls.indexOf("video-result")!==-1)rt="video";' +
|
||||
'else if(cls.indexOf("image-result")!==-1)rt="image";' +
|
||||
'if(!t||!h||seen[h])continue;seen[h]=true;' +
|
||||
'r.push([t,h,sn,du,ic,rt]);' +
|
||||
'}return r;}';
|
||||
}
|
||||
|
||||
function buildExtractorJs(limit) {
|
||||
return '(' + buildExtractFn(limit) + '(document))';
|
||||
}
|
||||
|
||||
function buildPaginateJs(limit, keyword, offset, region) {
|
||||
var params = 'q=' + encodeURIComponent(keyword) + '&s=' + offset + '&v=l&o=json';
|
||||
if (region) params += '&kl=' + encodeURIComponent(region);
|
||||
return (
|
||||
'new Promise(function($r){' +
|
||||
'var x=new XMLHttpRequest();' +
|
||||
'x.open("POST","/html/",true);' +
|
||||
'x.setRequestHeader("Content-Type","application/x-www-form-urlencoded");' +
|
||||
'x.onload=function(){' +
|
||||
'try{var d=new DOMParser().parseFromString(x.responseText,"text/html");' +
|
||||
'$r(' + buildExtractFn(limit) + '(d));' +
|
||||
'}catch(e){$r({error:"parse",message:String(e&&e.message||e)})}' +
|
||||
'};' +
|
||||
'x.onerror=function(){$r({error:"network"})};' +
|
||||
'x.send("' + params + '");' +
|
||||
'})'
|
||||
);
|
||||
}
|
||||
|
||||
const command = cli({
|
||||
site: 'duckduckgo',
|
||||
name: 'search',
|
||||
access: 'read',
|
||||
description: 'Search DuckDuckGo',
|
||||
domain: 'html.duckduckgo.com',
|
||||
strategy: Strategy.PUBLIC,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'keyword', positional: true, required: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of results per page (1-10). For multi-page, use --offset' },
|
||||
{ name: 'offset', type: 'int', default: 0, help: 'Result offset for pagination (0, 10, 20...). Uses XHR POST internally' },
|
||||
{ name: 'region', help: 'Region code (e.g. jp-jp, us-en, cn-zh). Default: all regions' },
|
||||
{ name: 'time', help: 'Time range: d (day), w (week), m (month), y (year)' },
|
||||
],
|
||||
columns: ['rank', 'title', 'url', 'snippet', 'displayUrl', 'icon', 'resultType'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = requireBoundedInteger(kwargs.limit, 10, 1, 10, '--limit');
|
||||
const keyword = requireSearchQuery(kwargs.keyword);
|
||||
const offset = requireNonNegativeInteger(kwargs.offset, 0, '--offset');
|
||||
if (offset % 10 !== 0) {
|
||||
throw new ArgumentError('--offset must be a multiple of 10 for DuckDuckGo HTML pagination');
|
||||
}
|
||||
if (kwargs.time && !/^(d|w|m|y)$/.test(String(kwargs.time))) {
|
||||
throw new ArgumentError('--time must be one of d, w, m, or y');
|
||||
}
|
||||
let url = `https://html.duckduckgo.com/html/?q=${encodeURIComponent(keyword)}`;
|
||||
if (kwargs.region) url += `&kl=${encodeURIComponent(String(kwargs.region))}`;
|
||||
if (kwargs.time) url += `&df=${encodeURIComponent(String(kwargs.time))}`;
|
||||
await runBrowserStep('duckduckgo search navigation', () => page.goto(url));
|
||||
try {
|
||||
await page.wait({ selector: '.result', timeout: 8 });
|
||||
} catch {
|
||||
await page.wait(3).catch(function() {});
|
||||
}
|
||||
var raw;
|
||||
if (offset === 0) {
|
||||
raw = await runBrowserStep('duckduckgo search extraction', () => page.evaluate(buildExtractorJs(limit)));
|
||||
} else {
|
||||
raw = await runBrowserStep('duckduckgo search pagination extraction', () => page.evaluate(buildPaginateJs(limit, keyword, offset, kwargs.region)));
|
||||
}
|
||||
const rows = requireRows(raw, 'duckduckgo search');
|
||||
if (rows.length === 0) {
|
||||
throw emptySearchResults('DuckDuckGo', keyword);
|
||||
}
|
||||
return rows.map(function(r, index) {
|
||||
return {
|
||||
rank: index + 1 + offset,
|
||||
title: r[0],
|
||||
url: decodeDdgUrl(r[1]),
|
||||
snippet: r[2],
|
||||
displayUrl: r[3],
|
||||
icon: r[4],
|
||||
resultType: r[5],
|
||||
};
|
||||
}).filter((row) => row.url);
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = { command };
|
||||
@@ -0,0 +1,128 @@
|
||||
import { describe, it, expect, vi } from 'vitest';
|
||||
import { JSDOM } from 'jsdom';
|
||||
|
||||
const { __test__ } = await import('./search.js');
|
||||
const command = __test__.command;
|
||||
|
||||
function createPageMock(evaluateResult = []) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue(evaluateResult),
|
||||
};
|
||||
}
|
||||
|
||||
describe('duckduckgo search', () => {
|
||||
it('should register as a valid command', () => {
|
||||
expect(command).toBeDefined();
|
||||
expect(command.site).toBe('duckduckgo');
|
||||
expect(command.name).toBe('search');
|
||||
expect(command.access).toBe('read');
|
||||
expect(command.browser).toBe(true);
|
||||
expect(command.strategy).toBe('public');
|
||||
expect(command.domain).toBe('html.duckduckgo.com');
|
||||
});
|
||||
|
||||
it('should define keyword positional arg', () => {
|
||||
const kwArg = command.args.find(a => a.name === 'keyword');
|
||||
expect(kwArg).toBeDefined();
|
||||
expect(kwArg.positional).toBe(true);
|
||||
expect(kwArg.required).toBe(true);
|
||||
});
|
||||
|
||||
it('should define limit arg with default 10', () => {
|
||||
const limitArg = command.args.find(a => a.name === 'limit');
|
||||
expect(limitArg).toBeDefined();
|
||||
expect(limitArg.type).toBe('int');
|
||||
expect(limitArg.default).toBe(10);
|
||||
});
|
||||
|
||||
it('should define columns for output', () => {
|
||||
expect(command.columns).toContain('rank');
|
||||
expect(command.columns).toContain('title');
|
||||
expect(command.columns).toContain('url');
|
||||
expect(command.columns).toContain('snippet');
|
||||
expect(command.columns).toContain('displayUrl');
|
||||
expect(command.columns).toContain('icon');
|
||||
expect(command.columns).toContain('resultType');
|
||||
});
|
||||
|
||||
it('rejects empty query and out-of-range pagination before navigation', async () => {
|
||||
const page = createPageMock();
|
||||
await expect(command.func(page, { keyword: ' ', limit: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 11 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 5, offset: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('decodes DuckDuckGo redirect URLs and assigns listing rank', async () => {
|
||||
const page = createPageMock([
|
||||
[
|
||||
'OpenCLI',
|
||||
'/l/?uddg=https%3A%2F%2Fgithub.com%2Fjackwener%2FOpenCLI',
|
||||
'CLI browser tooling',
|
||||
'github.com/jackwener/OpenCLI',
|
||||
'',
|
||||
'web',
|
||||
],
|
||||
]);
|
||||
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 1 })).resolves.toEqual([{
|
||||
rank: 1,
|
||||
title: 'OpenCLI',
|
||||
url: 'https://github.com/jackwener/OpenCLI',
|
||||
snippet: 'CLI browser tooling',
|
||||
displayUrl: 'github.com/jackwener/OpenCLI',
|
||||
icon: '',
|
||||
resultType: 'web',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('executes the DOM extractor, filters ads, and returns canonical rows', async () => {
|
||||
const dom = new JSDOM(`
|
||||
<div class="result result--ad">
|
||||
<a class="result__a" href="https://ads.example/">Sponsored result</a>
|
||||
</div>
|
||||
<div class="result">
|
||||
<a class="result__a" href="/l/?uddg=https%3A%2F%2Fexample.com%2Farticle">Organic result</a>
|
||||
<a class="result__snippet">Organic snippet</a>
|
||||
<span class="result__url">example.com/article</span>
|
||||
<img class="result__icon__img" src="https://icons.duckduckgo.com/ip3/example.com.ico">
|
||||
</div>
|
||||
`);
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(async (source) => Function('document', `return ${source};`)(dom.window.document)),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 5 })).resolves.toEqual([{
|
||||
rank: 1,
|
||||
title: 'Organic result',
|
||||
url: 'https://example.com/article',
|
||||
snippet: 'Organic snippet',
|
||||
displayUrl: 'example.com/article',
|
||||
icon: 'https://icons.duckduckgo.com/ip3/example.com.ico',
|
||||
resultType: 'web',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('unwraps browser envelopes for paginated extraction', async () => {
|
||||
const page = createPageMock({ session: 'site:duckduckgo', data: [
|
||||
['Result', 'https://example.com/', 'snippet', 'example.com', '', 'web'],
|
||||
] });
|
||||
|
||||
const result = await command.func(page, { keyword: 'opencli', limit: 1, offset: 10 });
|
||||
|
||||
expect(result[0]).toMatchObject({ rank: 11, url: 'https://example.com/' });
|
||||
});
|
||||
|
||||
it('fails typed instead of returning [] for malformed extraction payloads', async () => {
|
||||
const page = createPageMock({ rows: [] });
|
||||
|
||||
await expect(command.func(page, { keyword: 'opencli', limit: 1 })).rejects.toMatchObject({
|
||||
code: 'COMMAND_EXEC',
|
||||
message: expect.stringContaining('payload shape'),
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,45 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { requireBoundedInteger, requireSearchQuery } from '../_shared/search-adapter.js';
|
||||
|
||||
const command = cli({
|
||||
site: 'duckduckgo',
|
||||
name: 'suggest',
|
||||
access: 'read',
|
||||
description: 'DuckDuckGo search suggestions',
|
||||
domain: 'duckduckgo.com',
|
||||
strategy: Strategy.PUBLIC,
|
||||
browser: false,
|
||||
args: [
|
||||
{ name: 'keyword', positional: true, required: true, help: 'Search query prefix' },
|
||||
{ name: 'limit', type: 'int', default: 8, help: 'Max number of suggestions' },
|
||||
],
|
||||
columns: ['phrase'],
|
||||
func: async (kwargs) => {
|
||||
const limit = requireBoundedInteger(kwargs.limit, 8, 1, 20, '--limit');
|
||||
const keyword = encodeURIComponent(requireSearchQuery(kwargs.keyword));
|
||||
const url = `https://duckduckgo.com/ac/?q=${keyword}&type=list`;
|
||||
let resp;
|
||||
try {
|
||||
resp = await fetch(url);
|
||||
} catch (err) {
|
||||
throw new CommandExecutionError(`DuckDuckGo suggest request failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
if (!resp.ok) {
|
||||
throw new CommandExecutionError(`DuckDuckGo suggest returned HTTP ${resp.status}`);
|
||||
}
|
||||
let data;
|
||||
try {
|
||||
data = await resp.json();
|
||||
} catch (err) {
|
||||
throw new CommandExecutionError(`DuckDuckGo suggest returned malformed JSON: ${err?.message ?? err}`);
|
||||
}
|
||||
const phrases = Array.isArray(data) && data.length > 1 && Array.isArray(data[1]) ? data[1] : [];
|
||||
return phrases
|
||||
.filter((phrase) => typeof phrase === 'string' && phrase.trim())
|
||||
.slice(0, limit)
|
||||
.map(function(p) { return { phrase: p }; });
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = { command };
|
||||
@@ -0,0 +1,66 @@
|
||||
import { afterEach, describe, it, expect, vi } from 'vitest';
|
||||
|
||||
const { __test__ } = await import('./suggest.js');
|
||||
const command = __test__.command;
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
describe('duckduckgo suggest', () => {
|
||||
it('should register as a valid command', () => {
|
||||
expect(command).toBeDefined();
|
||||
expect(command.site).toBe('duckduckgo');
|
||||
expect(command.name).toBe('suggest');
|
||||
expect(command.access).toBe('read');
|
||||
expect(command.browser).toBe(false);
|
||||
expect(command.strategy).toBe('public');
|
||||
});
|
||||
|
||||
it('should define keyword positional arg', () => {
|
||||
const kwArg = command.args.find(a => a.name === 'keyword');
|
||||
expect(kwArg).toBeDefined();
|
||||
expect(kwArg.positional).toBe(true);
|
||||
expect(kwArg.required).toBe(true);
|
||||
});
|
||||
|
||||
it('should define limit arg with default 8', () => {
|
||||
const limitArg = command.args.find(a => a.name === 'limit');
|
||||
expect(limitArg).toBeDefined();
|
||||
expect(limitArg.default).toBe(8);
|
||||
});
|
||||
|
||||
it('should define phrase column', () => {
|
||||
expect(command.columns).toEqual(['phrase']);
|
||||
});
|
||||
|
||||
it('rejects empty query and invalid limit before fetch', async () => {
|
||||
const fetchSpy = vi.spyOn(globalThis, 'fetch');
|
||||
await expect(command.func({ keyword: '', limit: 5 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
await expect(command.func({ keyword: 'opencli', limit: 21 })).rejects.toMatchObject({ code: 'ARGUMENT' });
|
||||
expect(fetchSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns filtered suggestion rows from the public API payload', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ['open', ['opencli', '', 'open source']],
|
||||
});
|
||||
|
||||
await expect(command.func({ keyword: 'open', limit: 3 })).resolves.toEqual([
|
||||
{ phrase: 'opencli' },
|
||||
{ phrase: 'open source' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('maps fetch and malformed JSON failures to typed command errors', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockRejectedValueOnce(new Error('offline'));
|
||||
await expect(command.func({ keyword: 'open', limit: 3 })).rejects.toMatchObject({ code: 'COMMAND_EXEC' });
|
||||
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => { throw new Error('bad json'); },
|
||||
});
|
||||
await expect(command.func({ keyword: 'open', limit: 3 })).rejects.toMatchObject({ code: 'COMMAND_EXEC' });
|
||||
});
|
||||
});
|
||||
+299
-54
@@ -1,60 +1,305 @@
|
||||
import { cli } from '@jackwener/opencli/registry';
|
||||
cli({
|
||||
site: 'facebook',
|
||||
name: 'feed',
|
||||
access: 'read',
|
||||
description: 'Get your Facebook news feed',
|
||||
domain: 'www.facebook.com',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of posts' },
|
||||
],
|
||||
columns: ['index', 'author', 'content', 'likes', 'comments', 'shares'],
|
||||
pipeline: [
|
||||
{ navigate: { url: 'https://www.facebook.com/', settleMs: 4000 } },
|
||||
{ evaluate: `(() => {
|
||||
const limit = \${{ args.limit }};
|
||||
const posts = document.querySelectorAll('[role="article"]');
|
||||
return Array.from(posts)
|
||||
.filter(el => {
|
||||
const text = el.textContent.trim();
|
||||
// Filter out "People you may know" suggestions (both CN and EN)
|
||||
return text.length > 30 &&
|
||||
!text.startsWith('可能认识') &&
|
||||
!text.startsWith('People you may know') &&
|
||||
!text.startsWith('People You May Know');
|
||||
})
|
||||
.slice(0, limit)
|
||||
.map((el, i) => {
|
||||
// Author from header link
|
||||
const headerLink = el.querySelector('h2 a, h3 a, h4 a, strong a');
|
||||
const author = headerLink ? headerLink.textContent.trim() : '';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
// Post text: grab visible spans, filter noise
|
||||
const spans = Array.from(el.querySelectorAll('div[dir="auto"]'))
|
||||
.map(s => s.textContent.trim())
|
||||
.filter(t => t.length > 10 && t.length < 500);
|
||||
const content = spans.length > 0 ? spans[0] : '';
|
||||
const FACEBOOK_HOME = 'https://www.facebook.com/';
|
||||
const MAX_LIMIT = 50;
|
||||
|
||||
// Engagement: find like/comment/share counts (CN + EN)
|
||||
const allText = el.textContent;
|
||||
const likesMatch = allText.match(/所有心情:([\\d,.\\s]*[\\d万亿KMk]+)/) ||
|
||||
allText.match(/All:\\s*([\\d,.KMk]+)/) ||
|
||||
allText.match(/([\\d,.KMk]+)\\s*(?:likes?|reactions?)/i);
|
||||
const commentsMatch = allText.match(/([\\d,.]+\\s*[万亿]?)\\s*条评论/) ||
|
||||
allText.match(/([\\d,.KMk]+)\\s*comments?/i);
|
||||
const sharesMatch = allText.match(/([\\d,.]+\\s*[万亿]?)\\s*次分享/) ||
|
||||
allText.match(/([\\d,.KMk]+)\\s*shares?/i);
|
||||
function requireLimit(value) {
|
||||
const n = Number(value);
|
||||
if (!Number.isInteger(n) || n < 1 || n > MAX_LIMIT) {
|
||||
throw new ArgumentError(`facebook feed --limit must be an integer between 1 and ${MAX_LIMIT}`);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
function unwrapBrowserResult(value) {
|
||||
if (value && typeof value === 'object' && 'data' in value) {
|
||||
return value.data;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function buildFeedExtractScript(limit) {
|
||||
return `(() => {
|
||||
const limit = ${limit};
|
||||
|
||||
function clean(value) {
|
||||
return String(value || '').replace(/\\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function textOf(el) {
|
||||
return clean(el && el.textContent);
|
||||
}
|
||||
|
||||
function labelOf(el) {
|
||||
return clean(el && el.getAttribute && el.getAttribute('aria-label'));
|
||||
}
|
||||
|
||||
function isAuthPage() {
|
||||
const path = window.location && window.location.pathname ? window.location.pathname : '';
|
||||
const body = textOf(document.body);
|
||||
return /^\\/(login|checkpoint)(\\/|$|\\.php)/.test(path)
|
||||
|| /^(Log in to Facebook|Facebook登录|登录 Facebook)/i.test(body)
|
||||
|| /You must log in to continue/i.test(body);
|
||||
}
|
||||
|
||||
function isExplicitEmptyFeed() {
|
||||
const body = textOf(document.body);
|
||||
return /No posts available|Nothing to show|暂无动态|没有更多动态|还没有帖子/i.test(body);
|
||||
}
|
||||
|
||||
function isSuggestionOrChrome(text) {
|
||||
return /^(People you may know|People You May Know|可能认识的人?|你可能认识的人?)/i.test(text)
|
||||
|| /^(Suggested for you|Suggested Groups|推荐小组|推荐内容)/i.test(text);
|
||||
}
|
||||
|
||||
function isSponsored(text) {
|
||||
return /(^|\\s)(Sponsored|赞助|广告)(\\s|$)/i.test(text);
|
||||
}
|
||||
|
||||
function isActionText(text) {
|
||||
return /^(Like|Comment|Share|Send|Follow|赞|评论|分享|发送|关注)$/i.test(text);
|
||||
}
|
||||
|
||||
function isMetricText(text) {
|
||||
return /^(All:|所有心情:)/i.test(text)
|
||||
|| /\\b(likes?|reactions?|comments?|shares?)\\b/i.test(text)
|
||||
|| /(条评论|次分享)$/.test(text);
|
||||
}
|
||||
|
||||
function isTimestampText(text) {
|
||||
return /^(\\d+\\s*(s|m|h|d|w|mo|yr|min|sec|second|minute|hour|day|week|month|year)s?|Just now|Yesterday|刚刚|昨天|\\d+小时|\\d+天)(\\s*[·•.])?$/i.test(text);
|
||||
}
|
||||
|
||||
function postUrlFrom(root) {
|
||||
const links = Array.from(root.querySelectorAll('a[href]'));
|
||||
for (const link of links) {
|
||||
const href = link.href || link.getAttribute('href') || '';
|
||||
if (/\\/posts\\/|\\/permalink\\.php|\\/story\\.php|\\/photo\\/\\?fbid=|\\/groups\\/[^/]+\\/posts\\//i.test(href)) {
|
||||
return href;
|
||||
}
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function actionKinds(root) {
|
||||
const kinds = new Set();
|
||||
for (const el of root.querySelectorAll('[aria-label]')) {
|
||||
const label = labelOf(el);
|
||||
if (/^(Like|赞)$/i.test(label)) kinds.add('like');
|
||||
if (/^(Comment|评论)$/i.test(label)) kinds.add('comment');
|
||||
if (/^(Share|分享)$/i.test(label)) kinds.add('share');
|
||||
}
|
||||
return kinds;
|
||||
}
|
||||
|
||||
function visibleBlocks(root) {
|
||||
const seen = new Set();
|
||||
return Array.from(root.querySelectorAll('[dir="auto"]'))
|
||||
.map(textOf)
|
||||
.filter((text) => {
|
||||
if (!text || text.length > 600 || seen.has(text)) return false;
|
||||
seen.add(text);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
function findAuthor(root) {
|
||||
const links = [
|
||||
root.querySelector('h2 a[href], h3 a[href], h4 a[href], strong a[href]'),
|
||||
...Array.from(root.querySelectorAll('a[role="link"][href]')),
|
||||
].filter(Boolean);
|
||||
for (const link of links) {
|
||||
const text = textOf(link);
|
||||
const href = link.href || link.getAttribute('href') || '';
|
||||
if (text.length > 1 && text.length <= 80
|
||||
&& !isActionText(text)
|
||||
&& !isMetricText(text)
|
||||
&& !isTimestampText(text)
|
||||
&& !/\\/groups\\/|\\/watch\\/|\\/reel\\/|\\/events\\/|\\/friends\\//i.test(href)) {
|
||||
return text;
|
||||
}
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function contentBlocks(root, author) {
|
||||
return visibleBlocks(root).filter((text) => {
|
||||
if (text === author) return false;
|
||||
if (text.length <= 10) return false;
|
||||
if (isSuggestionOrChrome(text) || isSponsored(text)) return false;
|
||||
if (isActionText(text) || isMetricText(text) || isTimestampText(text)) return false;
|
||||
if (/^(See more|查看更多|更多)$/i.test(text)) return false;
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
function extractPost(root, index) {
|
||||
const fullText = textOf(root);
|
||||
if (!fullText || isSuggestionOrChrome(fullText) || isSponsored(fullText)) return null;
|
||||
|
||||
const author = findAuthor(root);
|
||||
const blocks = contentBlocks(root, author);
|
||||
const content = clean(blocks.join(' '));
|
||||
const postUrl = postUrlFrom(root);
|
||||
const kinds = actionKinds(root);
|
||||
|
||||
if (!author && !content) return null;
|
||||
if (!content && !postUrl && kinds.size < 2) return null;
|
||||
|
||||
const likesMatch = fullText.match(/所有心情:\\s*(\\d[\\d,.\\s万亿KMk]*)/)
|
||||
|| fullText.match(/All:\\s*(\\d[\\d,.KMk]*)/)
|
||||
|| fullText.match(/(\\d[\\d,.KMk]*)\\s*(?:likes?|reactions?)/i);
|
||||
const commentsMatch = fullText.match(/([\\d,.]+\\s*[万亿]?)\\s*条评论/)
|
||||
|| fullText.match(/(\\d[\\d,.KMk]*)\\s*comments?/i);
|
||||
const sharesMatch = fullText.match(/([\\d,.]+\\s*[万亿]?)\\s*次分享/)
|
||||
|| fullText.match(/(\\d[\\d,.KMk]*)\\s*shares?/i);
|
||||
|
||||
return {
|
||||
index: i + 1,
|
||||
index,
|
||||
author: author.substring(0, 50),
|
||||
content: content.replace(/\\n/g, ' ').substring(0, 120),
|
||||
likes: likesMatch ? likesMatch[1] : '-',
|
||||
comments: commentsMatch ? commentsMatch[1] : '-',
|
||||
shares: sharesMatch ? sharesMatch[1] : '-',
|
||||
content: content.substring(0, 120),
|
||||
likes: likesMatch ? clean(likesMatch[1]) : '-',
|
||||
comments: commentsMatch ? clean(commentsMatch[1]) : '-',
|
||||
shares: sharesMatch ? clean(sharesMatch[1]) : '-',
|
||||
};
|
||||
});
|
||||
})()
|
||||
` },
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
function primaryContainers() {
|
||||
return Array.from(document.querySelectorAll('[role="article"]'))
|
||||
.filter((el) => textOf(el).length > 30);
|
||||
}
|
||||
|
||||
function fallbackContainers() {
|
||||
const main = document.querySelector('[role="main"]');
|
||||
if (!main) return [];
|
||||
const buttons = Array.from(main.querySelectorAll('[aria-label="Like"], [aria-label="赞"], [aria-label="Comment"], [aria-label="评论"], [aria-label="Share"], [aria-label="分享"]'));
|
||||
const seen = new WeakSet();
|
||||
const containers = [];
|
||||
for (const button of buttons) {
|
||||
let node = button.parentElement;
|
||||
for (let depth = 0; depth < 16 && node && node !== main && node !== document.body; depth += 1, node = node.parentElement) {
|
||||
const text = textOf(node);
|
||||
const kinds = actionKinds(node);
|
||||
const blocks = visibleBlocks(node);
|
||||
const hasPostEvidence = Boolean(postUrlFrom(node)) || blocks.some((block) => block.length > 20 && !isActionText(block) && !isMetricText(block));
|
||||
if (text.length >= 80 && kinds.has('like') && (kinds.has('comment') || kinds.has('share')) && hasPostEvidence) {
|
||||
if (!seen.has(node)) {
|
||||
seen.add(node);
|
||||
containers.push(node);
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return containers;
|
||||
}
|
||||
|
||||
function dedupe(containers) {
|
||||
const seen = new Set();
|
||||
const result = [];
|
||||
for (const node of containers) {
|
||||
const key = postUrlFrom(node) || contentBlocks(node, findAuthor(node)).join('|').substring(0, 200);
|
||||
if (!key || seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
result.push(node);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
if (isAuthPage()) return { status: 'auth', rows: [], diagnostics: {} };
|
||||
|
||||
const primary = primaryContainers();
|
||||
const combined = dedupe([...primary, ...fallbackContainers()]);
|
||||
const rows = [];
|
||||
for (const container of combined) {
|
||||
const row = extractPost(container, rows.length + 1);
|
||||
if (row) rows.push(row);
|
||||
if (rows.length >= limit) break;
|
||||
}
|
||||
|
||||
return {
|
||||
status: rows.length ? 'ok' : (isExplicitEmptyFeed() ? 'empty' : 'no_rows'),
|
||||
rows,
|
||||
diagnostics: {
|
||||
articleCount: document.querySelectorAll('[role="article"]').length,
|
||||
primaryCount: primary.length,
|
||||
fallbackActionCount: document.querySelectorAll('[role="main"] [aria-label="Like"], [role="main"] [aria-label="赞"], [role="main"] [aria-label="Comment"], [role="main"] [aria-label="评论"]').length,
|
||||
mainTextLength: textOf(document.querySelector('[role="main"]')).length,
|
||||
},
|
||||
};
|
||||
})()`;
|
||||
}
|
||||
|
||||
async function getFacebookFeed(page, kwargs) {
|
||||
const limit = requireLimit(kwargs.limit ?? 10);
|
||||
try {
|
||||
await page.goto(FACEBOOK_HOME, { settleMs: 4000 });
|
||||
} catch (err) {
|
||||
throw new CommandExecutionError(
|
||||
`Failed to navigate to facebook feed: ${err instanceof Error ? err.message : err}`,
|
||||
'Check that facebook.com is reachable and the browser extension is connected.',
|
||||
);
|
||||
}
|
||||
|
||||
let payload;
|
||||
try {
|
||||
payload = unwrapBrowserResult(await page.evaluate(buildFeedExtractScript(limit)));
|
||||
} catch (err) {
|
||||
throw new CommandExecutionError(
|
||||
`Failed to read facebook feed: ${err instanceof Error ? err.message : err}`,
|
||||
'Facebook may not have rendered or the feed markup may have changed.',
|
||||
);
|
||||
}
|
||||
|
||||
if (!payload || typeof payload !== 'object' || !Array.isArray(payload.rows)) {
|
||||
throw new CommandExecutionError('facebook feed returned malformed extraction payload');
|
||||
}
|
||||
|
||||
if (payload.status === 'auth') {
|
||||
throw new AuthRequiredError('www.facebook.com', 'Open Chrome and log in to Facebook before retrying.');
|
||||
}
|
||||
|
||||
if (payload.rows.length > 0) {
|
||||
return payload.rows;
|
||||
}
|
||||
|
||||
if (payload.status === 'empty') {
|
||||
throw new EmptyResultError('facebook feed', 'Facebook did not show any feed posts for this account.');
|
||||
}
|
||||
|
||||
const diagnostics = payload.diagnostics || {};
|
||||
if (diagnostics.articleCount || diagnostics.fallbackActionCount || diagnostics.mainTextLength > 200) {
|
||||
throw new CommandExecutionError(
|
||||
'facebook feed page rendered but no feed rows could be extracted',
|
||||
`Diagnostics: articles=${diagnostics.articleCount || 0}, actions=${diagnostics.fallbackActionCount || 0}, mainTextLength=${diagnostics.mainTextLength || 0}.`,
|
||||
);
|
||||
}
|
||||
|
||||
throw new EmptyResultError('facebook feed', 'No Facebook feed content was visible in the current browser session.');
|
||||
}
|
||||
|
||||
const command = {
|
||||
site: 'facebook',
|
||||
name: 'feed',
|
||||
access: 'read',
|
||||
description: 'Get your Facebook news feed',
|
||||
domain: 'www.facebook.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of posts' },
|
||||
],
|
||||
columns: ['index', 'author', 'content', 'likes', 'comments', 'shares'],
|
||||
func: getFacebookFeed,
|
||||
};
|
||||
|
||||
cli(command);
|
||||
|
||||
export const __test__ = {
|
||||
buildFeedExtractScript,
|
||||
command,
|
||||
getFacebookFeed,
|
||||
requireLimit,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { JSDOM } from 'jsdom';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { __test__ } from './feed.js';
|
||||
|
||||
function runExtract(html, limit = 10, url = 'https://www.facebook.com/') {
|
||||
const dom = new JSDOM(html, { url });
|
||||
return Function('window', 'document', `return ${__test__.buildFeedExtractScript(limit)};`)(dom.window, dom.window.document);
|
||||
}
|
||||
|
||||
function createPage(payload) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue(payload),
|
||||
};
|
||||
}
|
||||
|
||||
describe('facebook feed', () => {
|
||||
it('registers the feed command with the existing row contract', () => {
|
||||
const cmd = getRegistry().get('facebook/feed');
|
||||
expect(cmd).toBeDefined();
|
||||
expect(cmd.columns).toEqual(['index', 'author', 'content', 'likes', 'comments', 'shares']);
|
||||
});
|
||||
|
||||
it('extracts existing role=article feed rows', () => {
|
||||
const payload = runExtract(`
|
||||
<main role="main">
|
||||
<div role="article">
|
||||
<h2><a href="https://www.facebook.com/alice">Alice Example</a></h2>
|
||||
<div dir="auto">This is a normal Facebook feed post with enough text to extract.</div>
|
||||
<span>All: 12</span>
|
||||
<span>3 comments</span>
|
||||
<span>2 shares</span>
|
||||
<div aria-label="Like"></div><div aria-label="Comment"></div>
|
||||
</div>
|
||||
</main>
|
||||
`);
|
||||
|
||||
expect(payload.status).toBe('ok');
|
||||
expect(payload.rows).toEqual([{
|
||||
index: 1,
|
||||
author: 'Alice Example',
|
||||
content: 'This is a normal Facebook feed post with enough text to extract.',
|
||||
likes: '12',
|
||||
comments: '3',
|
||||
shares: '2',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('falls back from empty article nodes to action-bounded feed containers', () => {
|
||||
const payload = runExtract(`
|
||||
<main role="main">
|
||||
<div role="article"></div>
|
||||
<section>
|
||||
<div>
|
||||
<h2><a href="https://www.facebook.com/bob/posts/123">Bob Builder</a></h2>
|
||||
<div dir="auto">Fallback post body from a Facebook feed card with empty article text.</div>
|
||||
<a href="https://www.facebook.com/bob/posts/123">Permalink</a>
|
||||
<span>All: 1.2K</span>
|
||||
<span>4 comments</span>
|
||||
<span>1 shares</span>
|
||||
<div><button aria-label="Like">Like</button><button aria-label="Comment">Comment</button></div>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
`);
|
||||
|
||||
expect(payload.status).toBe('ok');
|
||||
expect(payload.rows).toEqual([{
|
||||
index: 1,
|
||||
author: 'Bob Builder',
|
||||
content: 'Fallback post body from a Facebook feed card with empty article text.',
|
||||
likes: '1.2K',
|
||||
comments: '4',
|
||||
shares: '1',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('does not turn suggestions or side chrome action buttons into feed rows', () => {
|
||||
const payload = runExtract(`
|
||||
<main role="main">
|
||||
<aside>
|
||||
<h2>People you may know</h2>
|
||||
<div dir="auto">Charlie Suggested</div>
|
||||
<div dir="auto">Add friend from suggested people card with plenty of text.</div>
|
||||
<button aria-label="Like">Like</button>
|
||||
<button aria-label="Comment">Comment</button>
|
||||
</aside>
|
||||
<nav>
|
||||
<div dir="auto">Navigation item with a Like button but not a feed post.</div>
|
||||
<button aria-label="Like">Like</button>
|
||||
<button aria-label="Comment">Comment</button>
|
||||
</nav>
|
||||
</main>
|
||||
`);
|
||||
|
||||
expect(payload.status).toBe('no_rows');
|
||||
expect(payload.rows).toEqual([]);
|
||||
});
|
||||
|
||||
it('still considers bounded fallback rows when article nodes are suggestion chrome', () => {
|
||||
const payload = runExtract(`
|
||||
<main role="main">
|
||||
<div role="article">
|
||||
<h2>People you may know</h2>
|
||||
<div dir="auto">Suggested profile card with enough text to look article-like.</div>
|
||||
<button aria-label="Like">Like</button>
|
||||
<button aria-label="Comment">Comment</button>
|
||||
</div>
|
||||
<section>
|
||||
<div>
|
||||
<h2><a href="https://www.facebook.com/dana/posts/456">Dana Poster</a></h2>
|
||||
<div dir="auto">Fallback feed post should still be extracted after suggestion articles are filtered.</div>
|
||||
<a href="https://www.facebook.com/dana/posts/456">Permalink</a>
|
||||
<button aria-label="Like">Like</button>
|
||||
<button aria-label="Comment">Comment</button>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
`, 1);
|
||||
|
||||
expect(payload.status).toBe('ok');
|
||||
expect(payload.rows).toEqual([{
|
||||
index: 1,
|
||||
author: 'Dana Poster',
|
||||
content: 'Fallback feed post should still be extracted after suggestion articles are filtered.',
|
||||
likes: '-',
|
||||
comments: '-',
|
||||
shares: '-',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('reports auth pages from the browser extractor', () => {
|
||||
const payload = runExtract('<main role="main">Log in to Facebook</main>', 10, 'https://www.facebook.com/login/');
|
||||
expect(payload.status).toBe('auth');
|
||||
expect(payload.rows).toEqual([]);
|
||||
});
|
||||
|
||||
it('validates limit before browser navigation', async () => {
|
||||
const page = createPage({ status: 'ok', rows: [] });
|
||||
await expect(__test__.command.func(page, { limit: 0 })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('maps browser envelopes and returns extracted rows', async () => {
|
||||
const page = createPage({ session: 'site:facebook', data: { status: 'ok', rows: [{ index: 1, author: 'A', content: 'Body', likes: '-', comments: '-', shares: '-' }] } });
|
||||
|
||||
await expect(__test__.command.func(page, { limit: 1 })).resolves.toEqual([{
|
||||
index: 1,
|
||||
author: 'A',
|
||||
content: 'Body',
|
||||
likes: '-',
|
||||
comments: '-',
|
||||
shares: '-',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('maps auth, real empty, parser drift, and malformed payloads to typed errors', async () => {
|
||||
await expect(__test__.command.func(createPage({ status: 'auth', rows: [] }), { limit: 1 }))
|
||||
.rejects.toBeInstanceOf(AuthRequiredError);
|
||||
await expect(__test__.command.func(createPage({ status: 'empty', rows: [] }), { limit: 1 }))
|
||||
.rejects.toBeInstanceOf(EmptyResultError);
|
||||
await expect(__test__.command.func(createPage({ status: 'no_rows', rows: [], diagnostics: { articleCount: 1, fallbackActionCount: 2, mainTextLength: 500 } }), { limit: 1 }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
await expect(__test__.command.func(createPage({ rows: null }), { limit: 1 }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,5 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { clampInt, requireNonEmptyQuery } from '../_shared/common.js';
|
||||
|
||||
cli({
|
||||
@@ -18,12 +19,17 @@ cli({
|
||||
const limit = clampInt(kwargs.limit, 10, 1, 20);
|
||||
const query = requireNonEmptyQuery(kwargs.query);
|
||||
await page.goto(`https://scholar.google.com/scholar?q=${encodeURIComponent(query)}&hl=zh-CN`);
|
||||
await page.wait(3);
|
||||
const data = await page.evaluate(`
|
||||
try {
|
||||
await page.wait({ selector: '.gs_r.gs_or.gs_scl', timeout: 5 });
|
||||
} catch {
|
||||
await page.wait(3);
|
||||
}
|
||||
const wrapper = await page.evaluate(`
|
||||
(() => {
|
||||
const normalize = v => (v || '').replace(/\\s+/g, ' ').trim();
|
||||
const results = [];
|
||||
for (const el of document.querySelectorAll('.gs_r.gs_or.gs_scl')) {
|
||||
const resultCards = Array.from(document.querySelectorAll('.gs_r.gs_or.gs_scl'));
|
||||
for (const el of resultCards) {
|
||||
const container = el.querySelector('.gs_ri') || el;
|
||||
const titleEl = container.querySelector('.gs_rt a, h3 a');
|
||||
const title = normalize(titleEl?.textContent);
|
||||
@@ -50,9 +56,18 @@ cli({
|
||||
});
|
||||
if (results.length >= ${limit}) break;
|
||||
}
|
||||
return results;
|
||||
return { items: results, resultCount: resultCards.length };
|
||||
})()
|
||||
`);
|
||||
return Array.isArray(data) ? data : [];
|
||||
if (!wrapper || typeof wrapper !== 'object' || !Array.isArray(wrapper.items)) {
|
||||
throw new CommandExecutionError('Google Scholar search returned an unexpected payload shape');
|
||||
}
|
||||
if (wrapper.items.length === 0) {
|
||||
if (Number(wrapper.resultCount) > 0) {
|
||||
throw new CommandExecutionError('Google Scholar result cards were present but no rows could be extracted');
|
||||
}
|
||||
throw new EmptyResultError('google-scholar/search', 'Try a different query or check whether Google Scholar returned a CAPTCHA.');
|
||||
}
|
||||
return wrapper.items;
|
||||
},
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import './search.js';
|
||||
|
||||
@@ -25,14 +26,46 @@ describe('google-scholar search command', () => {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue([]),
|
||||
evaluate: vi.fn().mockResolvedValue({ items: [{ rank: 1, title: 'Paper' }], resultCount: 1 }),
|
||||
};
|
||||
|
||||
await command.func(page, { query: 'transformer' });
|
||||
const rows = await command.func(page, { query: 'transformer' });
|
||||
|
||||
const script = page.evaluate.mock.calls[0][0];
|
||||
expect(script).toContain("document.querySelectorAll('.gs_r.gs_or.gs_scl')");
|
||||
expect(script).not.toContain(".gs_r.gs_or.gs_scl, .gs_ri");
|
||||
expect(script).toContain("const container = el.querySelector('.gs_ri') || el");
|
||||
expect(script).toContain('return { items: results, resultCount: resultCards.length }');
|
||||
expect(rows).toEqual([{ rank: 1, title: 'Paper' }]);
|
||||
});
|
||||
|
||||
it('throws typed empty when Scholar returns no result cards', async () => {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue({ items: [], resultCount: 0 }),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { query: 'no results expected' })).rejects.toThrow(EmptyResultError);
|
||||
});
|
||||
|
||||
it('throws command execution when result cards exist but parser extracts no rows', async () => {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue({ items: [], resultCount: 2 }),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { query: 'parser drift' })).rejects.toThrow(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('throws command execution for malformed evaluate payloads instead of treating them as empty', async () => {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue({ items: { rank: 1 }, resultCount: 1 }),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { query: 'bad payload' })).rejects.toThrow(CommandExecutionError);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -39,12 +39,12 @@ cli({
|
||||
catch {
|
||||
await page.wait(2);
|
||||
}
|
||||
const results = await page.evaluate(`
|
||||
const wrapper = await page.evaluate(`
|
||||
(function() {
|
||||
var results = [];
|
||||
var seenUrls = {};
|
||||
var rso = document.querySelector('#rso');
|
||||
if (!rso) return results;
|
||||
if (!rso) return {items: results};
|
||||
|
||||
// -- Featured snippet (scoped to #rso to avoid matching unrelated elements) --
|
||||
var featuredEl = rso.querySelector('.xpdopen .hgKElc')
|
||||
@@ -126,10 +126,11 @@ cli({
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
return {items: results};
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(results) || results.length === 0) {
|
||||
const results = (wrapper && wrapper.items) || [];
|
||||
if (results.length === 0) {
|
||||
throw new CliError('NOT_FOUND', 'No search results found', 'Try a different keyword or check for CAPTCHA');
|
||||
}
|
||||
return results;
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
import * as fs from 'node:fs';
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const exportCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'export',
|
||||
access: 'read',
|
||||
description: 'Export the current Notion page as Markdown',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'output', required: false, help: 'Output file (default: /tmp/notion-export.md)' },
|
||||
],
|
||||
columns: ['Status', 'File'],
|
||||
func: async (page, kwargs) => {
|
||||
const outputPath = kwargs.output || '/tmp/notion-export.md';
|
||||
const result = await page.evaluate(`
|
||||
(function() {
|
||||
const titleEl = document.querySelector('[data-block-id] [placeholder="Untitled"], h1.notion-title, [class*="title"]');
|
||||
const title = titleEl ? (titleEl.textContent || '').trim() : document.title;
|
||||
|
||||
const frame = document.querySelector('.notion-page-content, [class*="page-content"], main');
|
||||
const content = frame ? (frame.innerText || '').trim() : document.body.innerText;
|
||||
|
||||
return { title, content };
|
||||
})()
|
||||
`);
|
||||
const md = `# ${result.title}\n\n${result.content}`;
|
||||
fs.writeFileSync(outputPath, md);
|
||||
return [{ Status: 'Success', File: outputPath }];
|
||||
},
|
||||
});
|
||||
@@ -1,85 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const favoritesCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'favorites',
|
||||
access: 'read',
|
||||
description: 'List pages from the Notion Favorites section in the sidebar',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [],
|
||||
columns: ['Index', 'Title', 'Icon'],
|
||||
func: async (page) => {
|
||||
const items = await page.evaluate(`
|
||||
(function() {
|
||||
const results = [];
|
||||
|
||||
// Strategy 1: Use Notion's own class 'notion-outliner-bookmarks-header-container'
|
||||
const headerContainer = document.querySelector('.notion-outliner-bookmarks-header-container');
|
||||
if (headerContainer) {
|
||||
// Walk up to the section parent that wraps header + items
|
||||
let section = headerContainer.parentElement;
|
||||
if (section && section.children.length === 1) section = section.parentElement;
|
||||
|
||||
if (section) {
|
||||
const treeItems = section.querySelectorAll('[role="treeitem"]');
|
||||
treeItems.forEach((item) => {
|
||||
// Title text is in a div.notranslate sibling of the icon area
|
||||
const titleEl = item.querySelector('div.notranslate:not(.notion-record-icon)');
|
||||
const title = titleEl
|
||||
? titleEl.textContent.trim()
|
||||
: (item.textContent || '').trim().substring(0, 80);
|
||||
|
||||
// Icon/emoji is in the notion-record-icon element
|
||||
const iconEl = item.querySelector('.notion-record-icon');
|
||||
const icon = iconEl ? iconEl.textContent.trim().substring(0, 4) : '';
|
||||
|
||||
if (title && title.length > 0) {
|
||||
results.push({ Index: results.length + 1, Title: title, Icon: icon || '📄' });
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Strategy 2: Fallback — find "Favorites" text node and walk DOM
|
||||
if (results.length === 0) {
|
||||
const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT, null);
|
||||
let node;
|
||||
let favEl = null;
|
||||
while (node = walker.nextNode()) {
|
||||
const text = node.textContent.trim();
|
||||
if (text === 'Favorites' || text === '收藏' || text === '收藏夹') {
|
||||
favEl = node.parentElement;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (favEl) {
|
||||
let section = favEl;
|
||||
for (let i = 0; i < 6; i++) {
|
||||
const p = section.parentElement;
|
||||
if (!p || p === document.body) break;
|
||||
const treeItems = p.querySelectorAll(':scope > [role="treeitem"]');
|
||||
if (treeItems.length > 0) { section = p; break; }
|
||||
section = p;
|
||||
}
|
||||
|
||||
const treeItems = section.querySelectorAll('[role="treeitem"]');
|
||||
treeItems.forEach((item) => {
|
||||
const text = (item.textContent || '').trim().substring(0, 120);
|
||||
if (text && text.length > 1 && !text.match(/^(Favorites|收藏夹?)$/)) {
|
||||
results.push({ Index: results.length + 1, Title: text, Icon: '📄' });
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
})()
|
||||
`);
|
||||
if (items.length === 0) {
|
||||
return [{ Index: 0, Title: 'No favorites found. Make sure sidebar is visible and you have favorites.', Icon: '⚠️' }];
|
||||
}
|
||||
return items;
|
||||
},
|
||||
});
|
||||
@@ -1,35 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const newCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'new',
|
||||
access: 'write',
|
||||
description: 'Create a new page in Notion',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'title', required: false, positional: true, help: 'Page title (optional)' },
|
||||
],
|
||||
columns: ['Status'],
|
||||
func: async (page, kwargs) => {
|
||||
const title = kwargs.title;
|
||||
// Cmd+N creates a new page in Notion
|
||||
const isMac = process.platform === 'darwin';
|
||||
await page.pressKey(isMac ? 'Meta+N' : 'Control+N');
|
||||
await page.wait(1);
|
||||
// If title is provided, type it into the title field
|
||||
if (title) {
|
||||
await page.evaluate(`
|
||||
(function(t) {
|
||||
const titleEl = document.querySelector('[placeholder="Untitled"], [data-content-editable-leaf] [placeholder]');
|
||||
if (titleEl) {
|
||||
titleEl.focus();
|
||||
document.execCommand('insertText', false, t);
|
||||
}
|
||||
})(${JSON.stringify(title)})
|
||||
`);
|
||||
await page.wait(0.5);
|
||||
}
|
||||
return [{ Status: title ? `Created page: ${title}` : 'New blank page created' }];
|
||||
},
|
||||
});
|
||||
@@ -1,31 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const readCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'read',
|
||||
access: 'read',
|
||||
description: 'Read the content of the currently open Notion page',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [],
|
||||
columns: ['Title', 'Content'],
|
||||
func: async (page) => {
|
||||
const result = await page.evaluate(`
|
||||
(function() {
|
||||
// Get the page title
|
||||
const titleEl = document.querySelector('[data-block-id] [placeholder="Untitled"], .notion-page-block .notranslate, h1.notion-title, [class*="title"]');
|
||||
const title = titleEl ? (titleEl.textContent || '').trim() : document.title;
|
||||
|
||||
// Get the page content — Notion renders blocks in a frame
|
||||
const frame = document.querySelector('.notion-page-content, [class*="page-content"], .layout-content, main');
|
||||
const content = frame ? (frame.innerText || frame.textContent || '').trim() : '';
|
||||
|
||||
return { title, content };
|
||||
})()
|
||||
`);
|
||||
return [{
|
||||
Title: result.title || 'Untitled',
|
||||
Content: result.content || '(empty page)',
|
||||
}];
|
||||
},
|
||||
});
|
||||
@@ -1,47 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const searchCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'search',
|
||||
access: 'read',
|
||||
description: 'Search pages and databases in Notion via Quick Find (Cmd+P)',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [{ name: 'query', required: true, positional: true, help: 'Search query' }],
|
||||
columns: ['Index', 'Title'],
|
||||
func: async (page, kwargs) => {
|
||||
const query = kwargs.query;
|
||||
// Open Quick Find
|
||||
const isMac = process.platform === 'darwin';
|
||||
await page.pressKey(isMac ? 'Meta+P' : 'Control+P');
|
||||
await page.wait(0.5);
|
||||
// Type the search query
|
||||
await page.evaluate(`
|
||||
(function(q) {
|
||||
const input = document.querySelector('input[placeholder*="Search"], input[type="text"]');
|
||||
if (input) {
|
||||
const setter = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set;
|
||||
setter.call(input, q);
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
}
|
||||
})(${JSON.stringify(query)})
|
||||
`);
|
||||
await page.wait(1.5);
|
||||
// Scrape results
|
||||
const results = await page.evaluate(`
|
||||
(function() {
|
||||
const items = document.querySelectorAll('[role="option"], [class*="searchResult"], [class*="quick-find"] [role="button"]');
|
||||
return Array.from(items).slice(0, 20).map((item, i) => ({
|
||||
Index: i + 1,
|
||||
Title: (item.textContent || '').trim().substring(0, 120),
|
||||
}));
|
||||
})()
|
||||
`);
|
||||
// Close Quick Find
|
||||
await page.pressKey('Escape');
|
||||
if (results.length === 0) {
|
||||
return [{ Index: 0, Title: `No results for "${query}"` }];
|
||||
}
|
||||
return results;
|
||||
},
|
||||
});
|
||||
@@ -1,42 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const sidebarCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'sidebar',
|
||||
access: 'read',
|
||||
description: 'List pages and databases from the Notion sidebar',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [],
|
||||
columns: ['Index', 'Title'],
|
||||
func: async (page) => {
|
||||
const items = await page.evaluate(`
|
||||
(function() {
|
||||
const results = [];
|
||||
// Notion sidebar items
|
||||
const selectors = [
|
||||
'[class*="sidebar"] [role="treeitem"]',
|
||||
'[class*="sidebar"] a',
|
||||
'.notion-sidebar [role="button"]',
|
||||
'nav [role="treeitem"]',
|
||||
];
|
||||
|
||||
for (const sel of selectors) {
|
||||
const nodes = document.querySelectorAll(sel);
|
||||
if (nodes.length > 0) {
|
||||
nodes.forEach((n, i) => {
|
||||
const text = (n.textContent || '').trim().substring(0, 100);
|
||||
if (text && text.length > 1) results.push({ Index: i + 1, Title: text });
|
||||
});
|
||||
break;
|
||||
}
|
||||
}
|
||||
return results;
|
||||
})()
|
||||
`);
|
||||
if (items.length === 0) {
|
||||
return [{ Index: 0, Title: 'No sidebar items found. Toggle the sidebar first.' }];
|
||||
}
|
||||
return items;
|
||||
},
|
||||
});
|
||||
@@ -1,17 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const statusCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'status',
|
||||
access: 'read',
|
||||
description: 'Check active CDP connection to Notion Desktop',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [],
|
||||
columns: ['Status', 'Url', 'Title'],
|
||||
func: async (page) => {
|
||||
const url = await page.evaluate('window.location.href');
|
||||
const title = await page.evaluate('document.title');
|
||||
return [{ Status: 'Connected', Url: url, Title: title }];
|
||||
},
|
||||
});
|
||||
@@ -1,41 +0,0 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
export const writeCommand = cli({
|
||||
site: 'notion',
|
||||
name: 'write',
|
||||
access: 'write',
|
||||
description: 'Append text content to the currently open Notion page',
|
||||
domain: 'localhost',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
args: [{ name: 'text', required: true, positional: true, help: 'Text to append to the page' }],
|
||||
columns: ['Status'],
|
||||
func: async (page, kwargs) => {
|
||||
const text = kwargs.text;
|
||||
// Focus the page body and move to the end
|
||||
await page.evaluate(`
|
||||
(function(text) {
|
||||
// Find the editable area in Notion
|
||||
const editables = document.querySelectorAll('.notion-page-content [contenteditable="true"], [class*="page-content"] [contenteditable="true"]');
|
||||
let target = editables.length > 0 ? editables[editables.length - 1] : null;
|
||||
|
||||
if (!target) {
|
||||
// Fallback: just find any contenteditable
|
||||
const all = document.querySelectorAll('[contenteditable="true"]');
|
||||
target = all.length > 0 ? all[all.length - 1] : null;
|
||||
}
|
||||
|
||||
if (!target) throw new Error('Could not find editable area in Notion page');
|
||||
|
||||
target.focus();
|
||||
// Move to end
|
||||
const sel = window.getSelection();
|
||||
sel.selectAllChildren(target);
|
||||
sel.collapseToEnd();
|
||||
|
||||
document.execCommand('insertText', false, text);
|
||||
})(${JSON.stringify(text)})
|
||||
`);
|
||||
await page.wait(0.5);
|
||||
return [{ Status: 'Text appended successfully' }];
|
||||
},
|
||||
});
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'post-id', type: 'string', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
|
||||
{ name: 'text', type: 'string', required: true, positional: true, help: 'Comment text' },
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 15 },
|
||||
],
|
||||
|
||||
@@ -23,7 +23,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 25, help: `Number of posts (1–${REDDIT_HOME_MAX_LIMIT})` },
|
||||
],
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20 },
|
||||
],
|
||||
|
||||
+402
-57
@@ -5,9 +5,98 @@
|
||||
* - Top-K comments by score at each level
|
||||
* - Configurable depth and replies-per-level
|
||||
* - Indented output showing conversation threads
|
||||
* - Optional --expand-more to follow Reddit's "more comments" stubs via
|
||||
* /api/morechildren.json (rdt-cli parity, PR B of #1481 follow-up)
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
|
||||
const REDDIT_EXPAND_ROUNDS_MIN = 1;
|
||||
const REDDIT_EXPAND_ROUNDS_MAX = 5;
|
||||
const DEFAULT_EXPAND_ROUNDS = 2;
|
||||
const REDDIT_POST_ID_RE = /^[a-z0-9]+$/i;
|
||||
|
||||
function normalizeBareRedditPostId(value) {
|
||||
const postId = String(value || '').trim();
|
||||
if (!REDDIT_POST_ID_RE.test(postId)) {
|
||||
throw new ArgumentError(
|
||||
'Post ID must be a Reddit post id, t3_ fullname, or reddit.com post URL.',
|
||||
'Use a bare post id like 1abc123, a fullname like t3_1abc123, or a full Reddit post URL.',
|
||||
);
|
||||
}
|
||||
return postId.toLowerCase();
|
||||
}
|
||||
|
||||
export function normalizeRedditPostId(value) {
|
||||
const raw = String(value || '').trim();
|
||||
if (!raw) {
|
||||
throw new ArgumentError(
|
||||
'Post ID is required.',
|
||||
'Use a bare post id like 1abc123, a fullname like t3_1abc123, or a full Reddit post URL.',
|
||||
);
|
||||
}
|
||||
|
||||
const fullname = raw.match(/^t3_([a-z0-9]+)$/i);
|
||||
if (fullname) return normalizeBareRedditPostId(fullname[1]);
|
||||
|
||||
if (/^https?:\/\//i.test(raw)) {
|
||||
let parsed;
|
||||
try {
|
||||
parsed = new URL(raw);
|
||||
} catch {
|
||||
throw new ArgumentError(`Invalid Reddit post URL: ${raw}`);
|
||||
}
|
||||
const host = parsed.hostname.toLowerCase();
|
||||
if (parsed.protocol !== 'https:' || (host !== 'reddit.com' && !host.endsWith('.reddit.com'))) {
|
||||
throw new ArgumentError(
|
||||
'Post URL must be an https reddit.com URL.',
|
||||
'Use a URL like https://www.reddit.com/r/sub/comments/1abc123/title_slug/',
|
||||
);
|
||||
}
|
||||
const parts = parsed.pathname.split('/').filter(Boolean);
|
||||
const commentsIndex = parts.indexOf('comments');
|
||||
const postIndex = commentsIndex + 1;
|
||||
if (commentsIndex < 0 || parts.length <= postIndex) {
|
||||
throw new ArgumentError(
|
||||
'Post URL must include the target post id.',
|
||||
'Use a URL like https://www.reddit.com/r/sub/comments/1abc123/title_slug/',
|
||||
);
|
||||
}
|
||||
if (parts.length > postIndex + 3) {
|
||||
throw new ArgumentError(
|
||||
'Post URL must end at the post slug or comment permalink id.',
|
||||
'Remove extra path segments after the post slug or comment id.',
|
||||
);
|
||||
}
|
||||
if (parts.length === postIndex + 3) normalizeBareRedditPostId(parts[postIndex + 2]);
|
||||
return normalizeBareRedditPostId(parts[postIndex]);
|
||||
}
|
||||
|
||||
if (raw.includes('/') || raw.startsWith('t1_')) {
|
||||
throw new ArgumentError(
|
||||
'Post ID must be a Reddit post id, t3_ fullname, or reddit.com post URL.',
|
||||
'Use a bare post id like 1abc123, a fullname like t3_1abc123, or a full Reddit post URL.',
|
||||
);
|
||||
}
|
||||
|
||||
return normalizeBareRedditPostId(raw);
|
||||
}
|
||||
|
||||
export function parseExpandRounds(raw) {
|
||||
if (raw === undefined || raw === null || raw === '') return DEFAULT_EXPAND_ROUNDS;
|
||||
const n = Number(raw);
|
||||
if (
|
||||
!Number.isFinite(n) || !Number.isInteger(n)
|
||||
|| n < REDDIT_EXPAND_ROUNDS_MIN || n > REDDIT_EXPAND_ROUNDS_MAX
|
||||
) {
|
||||
throw new ArgumentError(
|
||||
`expand-rounds must be an integer in [${REDDIT_EXPAND_ROUNDS_MIN}, ${REDDIT_EXPAND_ROUNDS_MAX}].`,
|
||||
`Got: ${raw}`,
|
||||
);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'reddit',
|
||||
name: 'read',
|
||||
@@ -16,7 +105,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'post-id', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or full URL' },
|
||||
{ name: 'sort', default: 'best', help: 'Comment sort: best, top, new, controversial, old, qa' },
|
||||
@@ -24,80 +112,314 @@ cli({
|
||||
{ name: 'depth', type: 'int', default: 2, help: 'Max reply depth (1=no replies, 2=one level of replies, etc.)' },
|
||||
{ name: 'replies', type: 'int', default: 5, help: 'Max replies shown per comment at each level (sorted by score)' },
|
||||
{ name: 'max-length', type: 'int', default: 2000, help: 'Max characters per comment body (min 100)' },
|
||||
{
|
||||
name: 'expand-more',
|
||||
type: 'bool',
|
||||
default: false,
|
||||
help: 'Follow Reddit "more comments" stubs by calling /api/morechildren.json',
|
||||
},
|
||||
{
|
||||
name: 'expand-rounds',
|
||||
type: 'int',
|
||||
default: DEFAULT_EXPAND_ROUNDS,
|
||||
help: `Max expansion passes when --expand-more is on (${REDDIT_EXPAND_ROUNDS_MIN}–${REDDIT_EXPAND_ROUNDS_MAX}; each round can fan out new "more" stubs)`,
|
||||
},
|
||||
],
|
||||
columns: ['type', 'author', 'score', 'text'],
|
||||
func: async (page, kwargs) => {
|
||||
// Note: --limit / --depth / --replies / --max-length keep their original
|
||||
// Math.max-style behaviour for backward compatibility (grandfathered in
|
||||
// the typed-error-lint baseline). The new --expand-rounds argument is
|
||||
// strictly validated via parseExpandRounds — no silent clamp.
|
||||
const sort = kwargs.sort ?? 'best';
|
||||
const limit = Math.max(1, kwargs.limit ?? 25);
|
||||
const maxDepth = Math.max(1, kwargs.depth ?? 2);
|
||||
const maxReplies = Math.max(1, kwargs.replies ?? 5);
|
||||
const maxLength = Math.max(100, kwargs['max-length'] ?? 2000);
|
||||
const expandMore = Boolean(kwargs['expand-more']);
|
||||
const expandRounds = parseExpandRounds(kwargs['expand-rounds']);
|
||||
const postId = normalizeRedditPostId(kwargs['post-id']);
|
||||
|
||||
await page.goto('https://www.reddit.com');
|
||||
const data = await page.evaluate(`
|
||||
|
||||
// The in-browser script returns a discriminated union so we can map
|
||||
// each failure mode to its proper typed error on the Node side
|
||||
// (page.evaluate boundary can't carry typed error instances). Kinds:
|
||||
// - inaccessible: 401/403/404 on /comments/<id>.json (post-specific,
|
||||
// not session auth — same session works for other posts)
|
||||
// - auth: /api/morechildren.json 401/403 (session-level on
|
||||
// the write-like expand endpoint — see two-pronged auth detection
|
||||
// sediment from PR #1428)
|
||||
// - http: 5xx or other non-ok
|
||||
// - malformed: 200 but Reddit shape is unexpected (schema drift)
|
||||
// - parser-drift: tree non-empty but walk produced 0 rows
|
||||
// - expand-failed: morechildren returned errors
|
||||
// - ok: rows array
|
||||
//
|
||||
// Intermediate keys (`rows` / `detail` / `httpStatus` / `where`)
|
||||
// deliberately avoid the declared columns (`type`/`author`/`score`/
|
||||
// `text`) to sidestep the silent-column-drop audit (PR #1329).
|
||||
const result = await page.evaluate(`
|
||||
(async function() {
|
||||
var postId = ${JSON.stringify(kwargs['post-id'])};
|
||||
var urlMatch = postId.match(/comments\\/([a-z0-9]+)/);
|
||||
if (urlMatch) postId = urlMatch[1];
|
||||
var postId = ${JSON.stringify(postId)};
|
||||
var linkFullname = 't3_' + postId;
|
||||
|
||||
var sort = ${JSON.stringify(sort)};
|
||||
var limit = ${limit};
|
||||
var maxDepth = ${maxDepth};
|
||||
var maxReplies = ${maxReplies};
|
||||
var maxLength = ${maxLength};
|
||||
var expandMore = ${JSON.stringify(expandMore)};
|
||||
var expandRounds = ${expandRounds};
|
||||
|
||||
// Request more from API than top-level limit to get inline replies
|
||||
// depth param tells Reddit how deep to inline replies vs "more" stubs
|
||||
// ---------------------------------------------------------------
|
||||
// Step 1: fetch the post + initial comment tree
|
||||
// ---------------------------------------------------------------
|
||||
// Request more from API than top-level limit to get inline replies.
|
||||
// depth param tells Reddit how deep to inline replies vs "more" stubs.
|
||||
var apiLimit = Math.max(limit * 3, 100);
|
||||
var res = await fetch(
|
||||
'/comments/' + postId + '.json?sort=' + sort + '&limit=' + apiLimit + '&depth=' + (maxDepth + 1) + '&raw_json=1',
|
||||
{ credentials: 'include' }
|
||||
);
|
||||
if (!res.ok) return { error: 'Reddit API returned HTTP ' + res.status };
|
||||
|
||||
if (res.status === 401 || res.status === 403 || res.status === 404) {
|
||||
return { kind: 'inaccessible', detail: 'Reddit post ' + postId + ' is not accessible (HTTP ' + res.status + ').' };
|
||||
}
|
||||
if (!res.ok) {
|
||||
return { kind: 'http', httpStatus: res.status, where: '/comments/' + postId + '.json' };
|
||||
}
|
||||
var data;
|
||||
try { data = await res.json(); } catch(e) { return { error: 'Failed to parse response' }; }
|
||||
if (!Array.isArray(data) || data.length < 2) return { error: 'Unexpected response format' };
|
||||
|
||||
var results = [];
|
||||
|
||||
// Post
|
||||
var post = data[0] && data[0].data && data[0].data.children && data[0].data.children[0] && data[0].data.children[0].data;
|
||||
if (post) {
|
||||
var body = post.selftext || '';
|
||||
if (body.length > maxLength) body = body.slice(0, maxLength) + '\\n... [truncated]';
|
||||
results.push({
|
||||
type: 'POST',
|
||||
author: post.author || '[deleted]',
|
||||
score: post.score || 0,
|
||||
text: post.title + (body ? '\\n\\n' + body : '') + (post.url && !post.is_self ? '\\n' + post.url : ''),
|
||||
});
|
||||
try { data = await res.json(); } catch (e) {
|
||||
return { kind: 'malformed', detail: 'Failed to parse Reddit /comments/' + postId + '.json response: ' + (e && e.message || e) };
|
||||
}
|
||||
if (!Array.isArray(data) || data.length < 2) {
|
||||
return { kind: 'malformed', detail: 'Reddit /comments/' + postId + '.json had unexpected envelope shape (length ' + (Array.isArray(data) ? data.length : typeof data) + ').' };
|
||||
}
|
||||
|
||||
// Recursive comment walker
|
||||
// depth 0 = top-level comments; maxDepth is exclusive,
|
||||
// so --depth 1 means top-level only, --depth 2 means one reply level, etc.
|
||||
var post = data[0] && data[0].data && data[0].data.children && data[0].data.children[0] && data[0].data.children[0].data;
|
||||
if (!post) {
|
||||
return { kind: 'malformed', detail: 'Reddit /comments/' + postId + '.json had no post body.' };
|
||||
}
|
||||
var topListing = data[1] && data[1].data && Array.isArray(data[1].data.children) ? data[1].data.children : null;
|
||||
if (!topListing) {
|
||||
return { kind: 'malformed', detail: 'Reddit /comments/' + postId + '.json had no comment listing.' };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 2: optionally follow "more" stubs via /api/morechildren.json
|
||||
// ---------------------------------------------------------------
|
||||
// Each "more" thing has a .data.children array (t1 ids to fetch).
|
||||
// The morechildren API returns a FLAT list of things; we re-thread
|
||||
// them by parent_id (either t3_<postId> for top-level or t1_<id>
|
||||
// for nested). Each round may surface new "more" stubs (because
|
||||
// expansion is bounded by Reddit's depth param), so we iterate up
|
||||
// to expandRounds times.
|
||||
var expandMeta = { rounds: 0, fetched: 0, capped: false, errors: [] };
|
||||
|
||||
if (expandMore) {
|
||||
// Index every existing t1 node so we can splice replies onto it.
|
||||
var t1Index = {};
|
||||
function indexT1(arr) {
|
||||
if (!Array.isArray(arr)) return;
|
||||
for (var i = 0; i < arr.length; i++) {
|
||||
var node = arr[i];
|
||||
if (node && node.kind === 't1' && node.data && node.data.id) {
|
||||
t1Index[node.data.name || ('t1_' + node.data.id)] = node;
|
||||
if (node.data.replies && node.data.replies.data && node.data.replies.data.children) {
|
||||
indexT1(node.data.replies.data.children);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
indexT1(topListing);
|
||||
|
||||
// Collect "more" stubs (with non-empty children) from anywhere in
|
||||
// the tree. Each stub knows its host array via a closure-bound
|
||||
// reference we attach.
|
||||
function collectMoreStubs(parentArr, parentT1) {
|
||||
var out = [];
|
||||
if (!Array.isArray(parentArr)) return out;
|
||||
for (var i = 0; i < parentArr.length; i++) {
|
||||
var n = parentArr[i];
|
||||
if (!n || !n.data) continue;
|
||||
if (n.kind === 'more' && Array.isArray(n.data.children) && n.data.children.length > 0) {
|
||||
out.push({ stub: n, hostArr: parentArr, hostT1: parentT1 });
|
||||
} else if (n.kind === 't1' && n.data.replies && n.data.replies.data && n.data.replies.data.children) {
|
||||
var nested = collectMoreStubs(n.data.replies.data.children, n);
|
||||
for (var k = 0; k < nested.length; k++) out.push(nested[k]);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
for (var r = 0; r < expandRounds; r++) {
|
||||
var stubs = collectMoreStubs(topListing, null);
|
||||
if (stubs.length === 0) break;
|
||||
|
||||
// Build the union of t1 ids to request this round. Reddit's
|
||||
// morechildren API caps at ~100 ids per call; batch accordingly.
|
||||
var allIds = [];
|
||||
for (var s = 0; s < stubs.length; s++) {
|
||||
var st = stubs[s].stub;
|
||||
for (var c = 0; c < st.data.children.length; c++) allIds.push(st.data.children[c]);
|
||||
}
|
||||
if (allIds.length === 0) break;
|
||||
|
||||
// dedupe preserving order
|
||||
var seen = {};
|
||||
var uniqIds = [];
|
||||
for (var j = 0; j < allIds.length; j++) {
|
||||
if (!seen[allIds[j]]) { seen[allIds[j]] = 1; uniqIds.push(allIds[j]); }
|
||||
}
|
||||
|
||||
var fetchedThings = [];
|
||||
var batchSize = 100;
|
||||
var batchFailed = false;
|
||||
for (var b = 0; b < uniqIds.length; b += batchSize) {
|
||||
var batch = uniqIds.slice(b, b + batchSize);
|
||||
var body = 'api_type=json'
|
||||
+ '&link_id=' + encodeURIComponent(linkFullname)
|
||||
+ '&children=' + encodeURIComponent(batch.join(','))
|
||||
+ '&sort=' + encodeURIComponent(sort)
|
||||
+ '&raw_json=1';
|
||||
var mcRes;
|
||||
try {
|
||||
mcRes = await fetch('/api/morechildren', {
|
||||
method: 'POST',
|
||||
credentials: 'include',
|
||||
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
||||
body: body,
|
||||
});
|
||||
} catch (e) {
|
||||
return { kind: 'expand-failed', detail: 'morechildren request threw: ' + (e && e.message || e), expandMeta: expandMeta };
|
||||
}
|
||||
if (mcRes.status === 401 || mcRes.status === 403) {
|
||||
return { kind: 'auth', detail: '/api/morechildren returned HTTP ' + mcRes.status + ' (write/expand likely requires login)' };
|
||||
}
|
||||
if (!mcRes.ok) {
|
||||
return { kind: 'http', httpStatus: mcRes.status, where: '/api/morechildren (round ' + (r + 1) + ', batch ' + ((b / batchSize) + 1) + ')' };
|
||||
}
|
||||
var mcData;
|
||||
try { mcData = await mcRes.json(); } catch (e) {
|
||||
return { kind: 'malformed', detail: 'Failed to parse /api/morechildren response: ' + (e && e.message || e) };
|
||||
}
|
||||
var errs = mcData && mcData.json && mcData.json.errors;
|
||||
if (Array.isArray(errs) && errs.length > 0) {
|
||||
return { kind: 'expand-failed', detail: 'Reddit /api/morechildren rejected: ' + errs.map(function(e) { return e.join(': '); }).join('; '), expandMeta: expandMeta };
|
||||
}
|
||||
var things = mcData && mcData.json && mcData.json.data && mcData.json.data.things;
|
||||
if (!Array.isArray(things)) {
|
||||
return { kind: 'malformed', detail: '/api/morechildren returned no things array.' };
|
||||
}
|
||||
for (var t = 0; t < things.length; t++) fetchedThings.push(things[t]);
|
||||
}
|
||||
expandMeta.rounds = r + 1;
|
||||
expandMeta.fetched += fetchedThings.length;
|
||||
|
||||
var fetchedById = {};
|
||||
for (var t = 0; t < fetchedThings.length; t++) {
|
||||
var thing = fetchedThings[t];
|
||||
if (!thing || !thing.data) continue;
|
||||
if (thing.data.id) fetchedById[thing.data.id] = thing;
|
||||
if (thing.data.name) fetchedById[thing.data.name] = thing;
|
||||
}
|
||||
|
||||
var inserted = {};
|
||||
function thingKey(thing) {
|
||||
return thing && thing.data && (thing.data.name || (thing.kind + '_' + thing.data.id));
|
||||
}
|
||||
|
||||
// Replace each collected stub in-place so expansion preserves the
|
||||
// surrounding tree order instead of appending fetched comments at
|
||||
// the end of the parent array.
|
||||
for (var s = 0; s < stubs.length; s++) {
|
||||
var rec = stubs[s];
|
||||
var idx = rec.hostArr.indexOf(rec.stub);
|
||||
if (idx < 0) continue;
|
||||
var expectedParent = rec.hostT1
|
||||
? (rec.hostT1.data.name || ('t1_' + rec.hostT1.data.id))
|
||||
: linkFullname;
|
||||
var replacements = [];
|
||||
for (var c = 0; c < rec.stub.data.children.length; c++) {
|
||||
var childId = rec.stub.data.children[c];
|
||||
var replacement = fetchedById[childId] || fetchedById['t1_' + childId];
|
||||
if (!replacement || !replacement.data) {
|
||||
expandMeta.errors.push('missing: ' + childId + ' parent=' + expectedParent);
|
||||
continue;
|
||||
}
|
||||
var key = thingKey(replacement);
|
||||
if (key && inserted[key]) continue;
|
||||
if (replacement.data.parent_id !== expectedParent) {
|
||||
expandMeta.errors.push('orphan: ' + (replacement.data.id || '?') + ' parent=' + (replacement.data.parent_id || '?'));
|
||||
continue;
|
||||
}
|
||||
replacements.push(replacement);
|
||||
if (key) inserted[key] = 1;
|
||||
if (replacement.kind === 't1' && replacement.data && replacement.data.id) {
|
||||
t1Index[replacement.data.name || ('t1_' + replacement.data.id)] = replacement;
|
||||
}
|
||||
}
|
||||
rec.hostArr.splice(idx, 1, ...replacements);
|
||||
}
|
||||
|
||||
for (var t = 0; t < fetchedThings.length; t++) {
|
||||
var unplaced = fetchedThings[t];
|
||||
var unplacedKey = thingKey(unplaced);
|
||||
if (unplacedKey && inserted[unplacedKey]) continue;
|
||||
if (unplaced && unplaced.data) {
|
||||
expandMeta.errors.push('unplaced: ' + (unplaced.data.id || '?') + ' parent=' + (unplaced.data.parent_id || '?'));
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
if (r + 1 >= expandRounds) {
|
||||
// If after the last round there are still "more" stubs, mark capped.
|
||||
var remaining = collectMoreStubs(topListing, null);
|
||||
if (remaining.length > 0) expandMeta.capped = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (expandMeta.errors.length > 0) {
|
||||
return { kind: 'expand-failed', detail: 'Reddit /api/morechildren returned unplaceable comments: ' + expandMeta.errors.slice(0, 5).join('; '), expandMeta: expandMeta };
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 3: walk the (possibly augmented) tree into indented rows
|
||||
// ---------------------------------------------------------------
|
||||
var rows = [];
|
||||
|
||||
// Post header row.
|
||||
var body = post.selftext || '';
|
||||
if (body.length > maxLength) body = body.slice(0, maxLength) + '\\n... [truncated]';
|
||||
rows.push({
|
||||
type: 'POST',
|
||||
author: post.author || '[deleted]',
|
||||
score: post.score || 0,
|
||||
text: post.title + (body ? '\\n\\n' + body : '') + (post.url && !post.is_self ? '\\n' + post.url : ''),
|
||||
});
|
||||
|
||||
// Recursive comment walker.
|
||||
function walkComment(node, depth) {
|
||||
if (!node || node.kind !== 't1') return;
|
||||
var d = node.data;
|
||||
var body = d.body || '';
|
||||
if (body.length > maxLength) body = body.slice(0, maxLength) + '...';
|
||||
var cBody = d.body || '';
|
||||
if (cBody.length > maxLength) cBody = cBody.slice(0, maxLength) + '...';
|
||||
|
||||
// Indent prefix: apply to every line so multiline bodies stay aligned
|
||||
var indent = '';
|
||||
for (var i = 0; i < depth; i++) indent += ' ';
|
||||
var prefix = depth === 0 ? '' : indent + '> ';
|
||||
var indentedBody = depth === 0
|
||||
? body
|
||||
: body.split('\\n').map(function(line) { return prefix + line; }).join('\\n');
|
||||
? cBody
|
||||
: cBody.split('\\n').map(function(line) { return prefix + line; }).join('\\n');
|
||||
|
||||
results.push({
|
||||
rows.push({
|
||||
type: depth === 0 ? 'L0' : 'L' + depth,
|
||||
author: d.author || '[deleted]',
|
||||
score: d.score || 0,
|
||||
text: indentedBody,
|
||||
});
|
||||
|
||||
// Count all available replies (for accurate "more" count)
|
||||
var t1Children = [];
|
||||
var moreCount = 0;
|
||||
if (d.replies && d.replies.data && d.replies.data.children) {
|
||||
@@ -111,13 +433,12 @@ cli({
|
||||
}
|
||||
}
|
||||
|
||||
// At depth cutoff: don't recurse, but show all replies as hidden
|
||||
if (depth + 1 >= maxDepth) {
|
||||
var totalHidden = t1Children.length + moreCount;
|
||||
if (totalHidden > 0) {
|
||||
var cutoffIndent = '';
|
||||
for (var j = 0; j <= depth; j++) cutoffIndent += ' ';
|
||||
results.push({
|
||||
rows.push({
|
||||
type: 'L' + (depth + 1),
|
||||
author: '',
|
||||
score: '',
|
||||
@@ -127,19 +448,17 @@ cli({
|
||||
return;
|
||||
}
|
||||
|
||||
// Sort by score descending, take top N
|
||||
t1Children.sort(function(a, b) { return (b.data.score || 0) - (a.data.score || 0); });
|
||||
var toProcess = Math.min(t1Children.length, maxReplies);
|
||||
for (var i = 0; i < toProcess; i++) {
|
||||
walkComment(t1Children[i], depth + 1);
|
||||
}
|
||||
|
||||
// Show hidden count (skipped replies + "more" stubs)
|
||||
var hidden = t1Children.length - toProcess + moreCount;
|
||||
if (hidden > 0) {
|
||||
var moreIndent = '';
|
||||
for (var j = 0; j <= depth; j++) moreIndent += ' ';
|
||||
results.push({
|
||||
rows.push({
|
||||
type: 'L' + (depth + 1),
|
||||
author: '',
|
||||
score: '',
|
||||
@@ -148,24 +467,24 @@ cli({
|
||||
}
|
||||
}
|
||||
|
||||
// Walk top-level comments
|
||||
var topLevel = data[1].data.children || [];
|
||||
var t1TopLevel = [];
|
||||
for (var i = 0; i < topLevel.length; i++) {
|
||||
if (topLevel[i].kind === 't1') t1TopLevel.push(topLevel[i]);
|
||||
for (var i = 0; i < topListing.length; i++) {
|
||||
if (topListing[i].kind === 't1') t1TopLevel.push(topListing[i]);
|
||||
}
|
||||
|
||||
// Top-level are already sorted by Reddit (sort param), take top N
|
||||
// Detect parser drift: tree had content but the walker produced nothing.
|
||||
// We must check this AFTER the walk because top-level may be only "more"
|
||||
// stubs (legitimate empty case for a brand-new post).
|
||||
var preWalkSize = topListing.length;
|
||||
for (var i = 0; i < Math.min(t1TopLevel.length, limit); i++) {
|
||||
walkComment(t1TopLevel[i], 0);
|
||||
}
|
||||
|
||||
// Count remaining
|
||||
var moreTopLevel = topLevel.filter(function(c) { return c.kind === 'more'; })
|
||||
var moreTopLevel = topListing.filter(function(c) { return c.kind === 'more'; })
|
||||
.reduce(function(sum, c) { return sum + (c.data.count || 0); }, 0);
|
||||
var hiddenTopLevel = Math.max(0, t1TopLevel.length - limit) + moreTopLevel;
|
||||
if (hiddenTopLevel > 0) {
|
||||
results.push({
|
||||
rows.push({
|
||||
type: '',
|
||||
author: '',
|
||||
score: '',
|
||||
@@ -173,15 +492,41 @@ cli({
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
// If we produced nothing beyond the POST row but the comment listing
|
||||
// wasn't empty, that's parser drift (e.g. Reddit changed t1/more
|
||||
// schema). Surface as CommandExecutionError on the Node side.
|
||||
if (rows.length <= 1 && preWalkSize > 0 && t1TopLevel.length > 0) {
|
||||
return { kind: 'parser-drift', detail: 'Reddit comment listing for post ' + postId + ' had ' + t1TopLevel.length + ' t1 entries but walker produced no rows.' };
|
||||
}
|
||||
|
||||
return { kind: 'ok', rows: rows, expandMeta: expandMeta };
|
||||
})()
|
||||
`);
|
||||
if (!data || typeof data !== 'object')
|
||||
throw new CommandExecutionError('Failed to fetch post data');
|
||||
if (!Array.isArray(data) && data.error)
|
||||
throw new CommandExecutionError(data.error);
|
||||
if (!Array.isArray(data))
|
||||
throw new CommandExecutionError('Unexpected response');
|
||||
return data;
|
||||
|
||||
if (!result || typeof result !== 'object') {
|
||||
throw new CommandExecutionError('Reddit /comments fetch returned no result envelope.');
|
||||
}
|
||||
if (result.kind === 'inaccessible') {
|
||||
throw new EmptyResultError(result.detail);
|
||||
}
|
||||
if (result.kind === 'auth') {
|
||||
throw new AuthRequiredError('reddit.com', result.detail);
|
||||
}
|
||||
if (result.kind === 'http') {
|
||||
throw new CommandExecutionError(`HTTP ${result.httpStatus} from ${result.where}`);
|
||||
}
|
||||
if (result.kind === 'malformed') {
|
||||
throw new CommandExecutionError(result.detail);
|
||||
}
|
||||
if (result.kind === 'parser-drift') {
|
||||
throw new CommandExecutionError(result.detail);
|
||||
}
|
||||
if (result.kind === 'expand-failed') {
|
||||
throw new CommandExecutionError(result.detail);
|
||||
}
|
||||
if (result.kind !== 'ok' || !Array.isArray(result.rows)) {
|
||||
throw new CommandExecutionError(`Unexpected result from reddit read: ${JSON.stringify(result)}`);
|
||||
}
|
||||
return result.rows;
|
||||
},
|
||||
});
|
||||
|
||||
+317
-14
@@ -1,20 +1,163 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { normalizeRedditPostId, parseExpandRounds } from './read.js';
|
||||
import './read.js';
|
||||
|
||||
function makePage(result) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue(result),
|
||||
};
|
||||
}
|
||||
|
||||
function redditPostEnvelope(children) {
|
||||
return [
|
||||
{
|
||||
data: {
|
||||
children: [{
|
||||
data: {
|
||||
title: 'Post title',
|
||||
selftext: '',
|
||||
author: 'op',
|
||||
score: 10,
|
||||
is_self: true,
|
||||
},
|
||||
}],
|
||||
},
|
||||
},
|
||||
{ data: { children } },
|
||||
];
|
||||
}
|
||||
|
||||
function commentThing(id, body, parent = 't3_abc123', score = 1) {
|
||||
return {
|
||||
kind: 't1',
|
||||
data: {
|
||||
id,
|
||||
name: `t1_${id}`,
|
||||
parent_id: parent,
|
||||
author: id,
|
||||
score,
|
||||
body,
|
||||
replies: '',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function moreThing(id, children, parent = 't3_abc123', count = children.length) {
|
||||
return {
|
||||
kind: 'more',
|
||||
data: { id, parent_id: parent, children, count },
|
||||
};
|
||||
}
|
||||
|
||||
function jsonResponse(payload, status = 200) {
|
||||
return {
|
||||
ok: status >= 200 && status < 300,
|
||||
status,
|
||||
json: vi.fn().mockResolvedValue(payload),
|
||||
};
|
||||
}
|
||||
|
||||
function makeRuntimePage(fetchImpl) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
const previousFetch = globalThis.fetch;
|
||||
globalThis.fetch = fetchImpl;
|
||||
try {
|
||||
return await eval(script);
|
||||
} finally {
|
||||
globalThis.fetch = previousFetch;
|
||||
}
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
describe('reddit read adapter', () => {
|
||||
const command = getRegistry().get('reddit/read');
|
||||
it('opts into the Reddit persistent site session', () => {
|
||||
|
||||
it('uses an ephemeral Reddit site tab by default', () => {
|
||||
expect(command?.browser).toBe(true);
|
||||
expect(command?.siteSession).toBe('persistent');
|
||||
expect(command?.siteSession).toBeUndefined();
|
||||
expect(command?.columns).toEqual(['type', 'author', 'score', 'text']);
|
||||
});
|
||||
it('returns threaded rows from the browser-evaluated payload', async () => {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue([
|
||||
|
||||
it('exposes the new --expand-more / --expand-rounds args', () => {
|
||||
const argNames = command.args.map((a) => a.name);
|
||||
expect(argNames).toContain('expand-more');
|
||||
expect(argNames).toContain('expand-rounds');
|
||||
const expandMore = command.args.find((a) => a.name === 'expand-more');
|
||||
expect(expandMore.type).toBe('bool');
|
||||
expect(expandMore.default).toBe(false);
|
||||
const rounds = command.args.find((a) => a.name === 'expand-rounds');
|
||||
expect(rounds.type).toBe('int');
|
||||
expect(rounds.default).toBe(2);
|
||||
});
|
||||
|
||||
describe('normalizeRedditPostId', () => {
|
||||
it('accepts bare ids, t3 fullnames, and exact reddit post URLs', () => {
|
||||
expect(normalizeRedditPostId('1AbC23')).toBe('1abc23');
|
||||
expect(normalizeRedditPostId('t3_1AbC23')).toBe('1abc23');
|
||||
expect(normalizeRedditPostId('https://www.reddit.com/r/opencli/comments/1abc23/title_slug/?sort=top')).toBe('1abc23');
|
||||
expect(normalizeRedditPostId('https://www.reddit.com/r/opencli/comments/1abc23/title_slug/okf3s7u/?context=3')).toBe('1abc23');
|
||||
expect(normalizeRedditPostId('https://old.reddit.com/comments/1abc23/title_slug/')).toBe('1abc23');
|
||||
});
|
||||
|
||||
it('rejects invalid or structurally loose post identities before navigation', () => {
|
||||
for (const bad of [
|
||||
'',
|
||||
't1_okf3s7u',
|
||||
'https://reddit.com.evil.com/r/opencli/comments/1abc23/title_slug/',
|
||||
'http://www.reddit.com/r/opencli/comments/1abc23/title_slug/',
|
||||
'https://www.reddit.com/r/opencli/comments/',
|
||||
'https://www.reddit.com/r/opencli/comments/1abc23/title_slug/okf3s7u/evil',
|
||||
'not/a/post',
|
||||
]) {
|
||||
expect(() => normalizeRedditPostId(bad)).toThrow(ArgumentError);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseExpandRounds', () => {
|
||||
it('returns the default for absent input but throws on out-of-range / non-integer', () => {
|
||||
expect(parseExpandRounds(undefined)).toBe(2);
|
||||
expect(parseExpandRounds(null)).toBe(2);
|
||||
expect(parseExpandRounds('')).toBe(2);
|
||||
expect(parseExpandRounds(1)).toBe(1);
|
||||
expect(parseExpandRounds(5)).toBe(5);
|
||||
for (const bad of [0, -1, 6, 1.5, NaN, 'abc']) {
|
||||
expect(() => parseExpandRounds(bad)).toThrow(ArgumentError);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects a bad --expand-rounds BEFORE navigating', async () => {
|
||||
const page = makePage({ kind: 'ok', rows: [] });
|
||||
await expect(command.func(page, { 'post-id': 'abc123', 'expand-rounds': 99 }))
|
||||
.rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects a bad post identity BEFORE navigating', async () => {
|
||||
const page = makePage({ kind: 'ok', rows: [] });
|
||||
await expect(command.func(page, { 'post-id': 'https://evil.test/r/x/comments/abc/title/' }))
|
||||
.rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns rows when the evaluate script reports kind=ok', async () => {
|
||||
const page = makePage({
|
||||
kind: 'ok',
|
||||
rows: [
|
||||
{ type: 'POST', author: 'alice', score: 10, text: 'Title' },
|
||||
{ type: 'L0', author: 'bob', score: 5, text: 'Comment' },
|
||||
]),
|
||||
};
|
||||
],
|
||||
expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] },
|
||||
});
|
||||
const result = await command.func(page, { 'post-id': 'abc123', limit: 5 });
|
||||
expect(page.goto).toHaveBeenCalledWith('https://www.reddit.com');
|
||||
expect(result).toEqual([
|
||||
@@ -22,11 +165,171 @@ describe('reddit read adapter', () => {
|
||||
{ type: 'L0', author: 'bob', score: 5, text: 'Comment' },
|
||||
]);
|
||||
});
|
||||
it('surfaces adapter-level API errors clearly', async () => {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue({ error: 'Reddit API returned HTTP 403' }),
|
||||
};
|
||||
await expect(command.func(page, { 'post-id': 'abc123' })).rejects.toThrow('Reddit API returned HTTP 403');
|
||||
|
||||
it('maps the five failure kinds to the right typed errors', async () => {
|
||||
await expect(command.func(makePage({ kind: 'inaccessible', detail: 'post 403' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(EmptyResultError);
|
||||
|
||||
await expect(command.func(makePage({ kind: 'auth', detail: 'morechildren 401' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(AuthRequiredError);
|
||||
|
||||
await expect(command.func(makePage({ kind: 'http', httpStatus: 503, where: '/comments/abc.json' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
|
||||
await expect(command.func(makePage({ kind: 'malformed', detail: 'no comment listing' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
|
||||
await expect(command.func(makePage({ kind: 'parser-drift', detail: 'walker drift' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
|
||||
await expect(command.func(makePage({ kind: 'expand-failed', detail: 'morechildren errors' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('throws CommandExecutionError on an unknown envelope shape (no kind)', async () => {
|
||||
await expect(command.func(makePage({ random: 'stuff' }), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
await expect(command.func(makePage(null), { 'post-id': 'abc123' }))
|
||||
.rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('embeds expandMore=false by default and inlines flags into the evaluate script', async () => {
|
||||
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] } });
|
||||
await command.func(page, { 'post-id': 'xyz', sort: 'top', limit: 3 });
|
||||
const script = page.evaluate.mock.calls[0][0];
|
||||
expect(script).toContain('var expandMore = false');
|
||||
expect(script).toContain('var expandRounds = 2');
|
||||
expect(script).toContain('var sort = "top"');
|
||||
expect(script).toContain('var limit = 3');
|
||||
expect(script).toContain('var postId = "xyz"');
|
||||
});
|
||||
|
||||
it('embeds expandMore=true and the requested expandRounds when --expand-more is on', async () => {
|
||||
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 3, fetched: 12, capped: true, errors: [] } });
|
||||
await command.func(page, { 'post-id': 'xyz', 'expand-more': true, 'expand-rounds': 3 });
|
||||
const script = page.evaluate.mock.calls[0][0];
|
||||
expect(script).toContain('var expandMore = true');
|
||||
expect(script).toContain('var expandRounds = 3');
|
||||
// The /api/morechildren request body construction must be present in
|
||||
// the evaluate script (round-trips the link_id + children CSV).
|
||||
expect(script).toContain("'/api/morechildren'");
|
||||
expect(script).toContain("'api_type=json'");
|
||||
expect(script).toContain("encodeURIComponent(linkFullname)");
|
||||
expect(script).toContain("encodeURIComponent(batch.join(','))");
|
||||
});
|
||||
|
||||
it('normalizes a full reddit URL before building the browser script', async () => {
|
||||
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] } });
|
||||
await command.func(page, { 'post-id': 'https://www.reddit.com/r/python/comments/1abc23/title_slug/' });
|
||||
const script = page.evaluate.mock.calls[0][0];
|
||||
expect(script).toContain('var postId = "1abc23"');
|
||||
expect(script).not.toContain('postIdRaw.match');
|
||||
});
|
||||
|
||||
it('expands morechildren in the original tree position instead of appending to the parent', async () => {
|
||||
const fetchMock = vi.fn(async (url) => {
|
||||
if (String(url).startsWith('/comments/')) {
|
||||
return jsonResponse(redditPostEnvelope([
|
||||
commentThing('a', 'A'),
|
||||
moreThing('more_top', ['b', 'c']),
|
||||
commentThing('d', 'D'),
|
||||
]));
|
||||
}
|
||||
if (String(url) === '/api/morechildren') {
|
||||
return jsonResponse({
|
||||
json: {
|
||||
errors: [],
|
||||
data: { things: [commentThing('b', 'B'), commentThing('c', 'C')] },
|
||||
},
|
||||
});
|
||||
}
|
||||
throw new Error(`unexpected URL ${url}`);
|
||||
});
|
||||
const page = makeRuntimePage(fetchMock);
|
||||
|
||||
const result = await command.func(page, {
|
||||
'post-id': 'abc123',
|
||||
'expand-more': true,
|
||||
limit: 10,
|
||||
replies: 10,
|
||||
});
|
||||
|
||||
expect(result.map((row) => row.author)).toEqual(['op', 'a', 'b', 'c', 'd']);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
'/api/morechildren',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
body: expect.stringContaining('link_id=t3_abc123'),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('fails expand-more when Reddit returns a child that cannot be placed in the requested tree', async () => {
|
||||
const fetchMock = vi.fn(async (url) => {
|
||||
if (String(url).startsWith('/comments/')) {
|
||||
return jsonResponse(redditPostEnvelope([moreThing('more_top', ['b'])]));
|
||||
}
|
||||
if (String(url) === '/api/morechildren') {
|
||||
return jsonResponse({
|
||||
json: {
|
||||
errors: [],
|
||||
data: { things: [commentThing('b', 'B', 't3_other')] },
|
||||
},
|
||||
});
|
||||
}
|
||||
throw new Error(`unexpected URL ${url}`);
|
||||
});
|
||||
const page = makeRuntimePage(fetchMock);
|
||||
|
||||
await expect(command.func(page, {
|
||||
'post-id': 'abc123',
|
||||
'expand-more': true,
|
||||
})).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('fails expand-more when Reddit omits a requested child instead of silently dropping the stub', async () => {
|
||||
const fetchMock = vi.fn(async (url) => {
|
||||
if (String(url).startsWith('/comments/')) {
|
||||
return jsonResponse(redditPostEnvelope([moreThing('more_top', ['b', 'c'])]));
|
||||
}
|
||||
if (String(url) === '/api/morechildren') {
|
||||
return jsonResponse({
|
||||
json: {
|
||||
errors: [],
|
||||
data: { things: [commentThing('b', 'B')] },
|
||||
},
|
||||
});
|
||||
}
|
||||
throw new Error(`unexpected URL ${url}`);
|
||||
});
|
||||
const page = makeRuntimePage(fetchMock);
|
||||
|
||||
await expect(command.func(page, {
|
||||
'post-id': 'abc123',
|
||||
'expand-more': true,
|
||||
})).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('uses 5-kind discriminated union keys that DO NOT collide with declared columns', () => {
|
||||
// Read the evaluate template once to assert the intermediate keys we
|
||||
// return on the browser side never name any of `type` / `author` /
|
||||
// `score` / `text` (the declared columns) — that pattern would
|
||||
// trigger the silent-column-drop audit.
|
||||
const page = makePage({ kind: 'ok', rows: [], expandMeta: { rounds: 0, fetched: 0, capped: false, errors: [] } });
|
||||
return command.func(page, { 'post-id': 'xyz' }).then(() => {
|
||||
const script = page.evaluate.mock.calls[0][0];
|
||||
// Each return shape uses kind / detail / httpStatus / where /
|
||||
// rows / expandMeta. None overlap with the four declared
|
||||
// columns. The walker IS allowed to push column-shaped row
|
||||
// objects into `rows` — that's the final shape, not an
|
||||
// intermediate one.
|
||||
expect(script).toContain("kind: 'inaccessible'");
|
||||
expect(script).toContain("kind: 'auth'");
|
||||
expect(script).toContain("kind: 'http'");
|
||||
expect(script).toContain("kind: 'malformed'");
|
||||
expect(script).toContain("kind: 'parser-drift'");
|
||||
expect(script).toContain("kind: 'expand-failed'");
|
||||
expect(script).toContain("kind: 'ok'");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'post-id', type: 'string', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
|
||||
{ name: 'undo', type: 'boolean', default: false, help: 'Unsave instead of save' },
|
||||
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 15 },
|
||||
],
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'query', type: 'string', required: true, positional: true, help: 'Reddit search query' },
|
||||
{
|
||||
|
||||
@@ -32,7 +32,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'name', type: 'string', required: true, positional: true, help: 'Subreddit name (no `r/` prefix needed)' },
|
||||
],
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'name', type: 'string', required: true, positional: true, help: 'Subreddit name (no `r/` prefix; e.g. `python`)' },
|
||||
{
|
||||
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'subreddit', type: 'string', required: true, positional: true, help: 'Subreddit name (e.g. python)' },
|
||||
{ name: 'undo', type: 'boolean', default: false, help: 'Unsubscribe instead of subscribe' },
|
||||
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'post-id', type: 'string', required: true, positional: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
|
||||
{ name: 'direction', type: 'string', default: 'up', help: 'Vote direction: up, down, none' },
|
||||
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 15 },
|
||||
],
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', type: 'string', required: true, positional: true, help: 'Reddit username (no `u/` prefix needed)' },
|
||||
{ name: 'limit', type: 'int', default: 15 },
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', type: 'string', required: true, positional: true, help: 'Reddit username (no `u/` prefix needed)' },
|
||||
{ name: 'limit', type: 'int', default: 15 },
|
||||
|
||||
@@ -7,7 +7,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', type: 'string', required: true, positional: true, help: 'Reddit username (no `u/` prefix needed)' },
|
||||
],
|
||||
|
||||
@@ -9,7 +9,6 @@ cli({
|
||||
domain: 'reddit.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [],
|
||||
columns: ['field', 'value'],
|
||||
func: async (page) => {
|
||||
|
||||
@@ -31,6 +31,14 @@ function createPageMock(evaluateResult) {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'sid', value: 'secret', domain: 'www.rednote.com' }]),
|
||||
};
|
||||
}
|
||||
function createSearchPageMock(evaluateResults) {
|
||||
const page = createPageMock(undefined);
|
||||
page.evaluate = vi.fn();
|
||||
for (const result of evaluateResults) {
|
||||
page.evaluate.mockResolvedValueOnce(result);
|
||||
}
|
||||
return page;
|
||||
}
|
||||
|
||||
describe('rednote note URL identity', () => {
|
||||
const download = getRegistry().get('rednote/download');
|
||||
@@ -130,6 +138,63 @@ describe('rednote argument validation', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('rednote search browser-bridge envelopes', () => {
|
||||
const search = getRegistry().get('rednote/search');
|
||||
|
||||
it('unwraps login-wall wait result envelopes before auth handling', async () => {
|
||||
const page = createSearchPageMock([
|
||||
{ session: 'site:rednote', data: 'login_wall' },
|
||||
]);
|
||||
|
||||
await expect(search.func(page, { query: 'tesla', limit: 5 })).rejects.toMatchObject({
|
||||
code: 'AUTH_REQUIRED',
|
||||
message: expect.stringContaining('blocked behind a login wall'),
|
||||
});
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('unwraps search extraction envelopes and preserves rednote row shape', async () => {
|
||||
const url = 'https://www.rednote.com/search_result/68e90be80000000004022e66?xsec_token=test-token';
|
||||
const page = createSearchPageMock([
|
||||
'content',
|
||||
1,
|
||||
{
|
||||
session: 'site:rednote',
|
||||
data: [{
|
||||
title: 'rednote result',
|
||||
author: 'author',
|
||||
likes: '12',
|
||||
url,
|
||||
author_url: 'https://www.rednote.com/user/profile/u1',
|
||||
}],
|
||||
},
|
||||
]);
|
||||
|
||||
await expect(search.func(page, { query: 'tesla', limit: 1 })).resolves.toEqual([{
|
||||
rank: 1,
|
||||
title: 'rednote result',
|
||||
author: 'author',
|
||||
likes: '12',
|
||||
published_at: '2025-10-10',
|
||||
url,
|
||||
author_url: 'https://www.rednote.com/user/profile/u1',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('fails typed instead of silently returning [] for malformed extraction payloads', async () => {
|
||||
const page = createSearchPageMock([
|
||||
'content',
|
||||
1,
|
||||
{ session: 'site:rednote', data: { rows: [] } },
|
||||
]);
|
||||
|
||||
await expect(search.func(page, { query: 'tesla', limit: 1 })).rejects.toMatchObject({
|
||||
code: 'COMMAND_EXEC',
|
||||
message: expect.stringContaining('payload shape'),
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('rednote Pinia store failures', () => {
|
||||
it('maps feed store read failure to CommandExecutionError', async () => {
|
||||
const command = getRegistry().get('rednote/feed');
|
||||
|
||||
+11
-5
@@ -6,8 +6,8 @@
|
||||
* 1:1 comparison between the two frontends.
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import { buildScrollUntilJs, buildSearchExtractJs, noteIdToDate } from '../xiaohongshu/search.js';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { buildScrollUntilJs, buildSearchExtractJs, noteIdToDate, unwrapEvaluateResult } from '../xiaohongshu/search.js';
|
||||
|
||||
function parseLimit(raw) {
|
||||
const parsed = Number(raw);
|
||||
@@ -19,6 +19,13 @@ function parseLimit(raw) {
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
function requireSearchRows(payload) {
|
||||
const rows = unwrapEvaluateResult(payload);
|
||||
if (!Array.isArray(rows)) {
|
||||
throw new CommandExecutionError('Unexpected Rednote search extraction payload shape; expected an array of rows.');
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for search results or login wall using MutationObserver (max 5s).
|
||||
@@ -78,7 +85,7 @@ cli({
|
||||
const limit = parseLimit(kwargs.limit ?? 20);
|
||||
const keyword = encodeURIComponent(kwargs.query);
|
||||
await page.goto(`https://www.rednote.com/search_result?keyword=${keyword}&source=web_search_result_notes`);
|
||||
const waitResult = await page.evaluate(WAIT_FOR_CONTENT_JS);
|
||||
const waitResult = unwrapEvaluateResult(await page.evaluate(WAIT_FOR_CONTENT_JS));
|
||||
if (waitResult === 'login_wall') {
|
||||
throw new AuthRequiredError('www.rednote.com', 'Rednote search results are blocked behind a login wall');
|
||||
}
|
||||
@@ -87,8 +94,7 @@ cli({
|
||||
// `autoScroll({ times: 2 })` capped extraction at ~13 notes regardless
|
||||
// of `--limit`.
|
||||
await page.evaluate(buildScrollUntilJs(limit));
|
||||
const payload = await page.evaluate(buildSearchExtractJs('www.rednote.com'));
|
||||
const data = Array.isArray(payload) ? payload : [];
|
||||
const data = requireSearchRows(await page.evaluate(buildSearchExtractJs('www.rednote.com')));
|
||||
return data
|
||||
.filter((item) => item.title)
|
||||
.slice(0, limit)
|
||||
|
||||
@@ -11,7 +11,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'tweet-id', type: 'string', positional: true, required: true, help: 'Tweet ID or URL containing the article' },
|
||||
],
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
import { resolveTwitterQueryId } from './shared.js';
|
||||
import { extractMedia, resolveTwitterQueryId } from './shared.js';
|
||||
|
||||
// Companion to bookmark-folders.js: reads tweets inside a single folder.
|
||||
// X exposes folder contents through a separate timeline operation
|
||||
@@ -11,6 +11,7 @@ import { resolveTwitterQueryId } from './shared.js';
|
||||
const OPERATION_NAME = 'BookmarkFolderTimeline';
|
||||
const FALLBACK_QUERY_ID = '13H7EUATwethsj_jZ6QQAQ';
|
||||
const FOLDER_ID_PATTERN = /^[A-Za-z0-9_-]+$/;
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
|
||||
const FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
@@ -53,7 +54,7 @@ function buildFolderTimelineUrl(queryId, folderId, count, cursor) {
|
||||
+ `&features=${encodeURIComponent(JSON.stringify(FEATURES))}`;
|
||||
}
|
||||
|
||||
function extractFolderTweet(result, seen) {
|
||||
export function extractFolderTweet(result, seen) {
|
||||
if (!result) return null;
|
||||
const tw = result.tweet || result;
|
||||
const legacy = tw.legacy || {};
|
||||
@@ -71,6 +72,7 @@ function extractFolderTweet(result, seen) {
|
||||
bookmarks: legacy.bookmark_count || 0,
|
||||
created_at: legacy.created_at || '',
|
||||
url: screenName ? `https://x.com/${screenName}/status/${tw.rest_id}` : `https://x.com/i/status/${tw.rest_id}`,
|
||||
...extractMedia(legacy),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -122,13 +124,12 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'folder-id', positional: true, type: 'string', required: true, help: 'Folder id from `opencli twitter bookmark-folders`.' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of bookmarks to return (default 20).' },
|
||||
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the folder by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the API\'s native (saved-time) ordering.' },
|
||||
],
|
||||
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url'],
|
||||
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url', 'has_media', 'media_urls'],
|
||||
func: async (page, kwargs) => {
|
||||
const folderId = String(kwargs['folder-id'] || '').trim();
|
||||
if (!folderId || !FOLDER_ID_PATTERN.test(folderId)) {
|
||||
@@ -158,7 +159,8 @@ cli({
|
||||
const allTweets = [];
|
||||
const seen = new Set();
|
||||
let cursor = null;
|
||||
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
|
||||
const fetchCount = Math.min(100, limit - allTweets.length + 10);
|
||||
const apiUrl = buildFolderTimelineUrl(queryId, folderId, fetchCount, cursor);
|
||||
const data = await page.evaluate(`async () => {
|
||||
@@ -182,6 +184,7 @@ cli({
|
||||
|
||||
export const __test__ = {
|
||||
parseBookmarkFolderTimeline,
|
||||
extractFolderTweet,
|
||||
buildFolderTimelineUrl,
|
||||
FOLDER_ID_PATTERN,
|
||||
};
|
||||
|
||||
@@ -2,7 +2,7 @@ import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { __test__ } from './bookmark-folder.js';
|
||||
|
||||
const { parseBookmarkFolderTimeline, buildFolderTimelineUrl, FOLDER_ID_PATTERN } = __test__;
|
||||
const { parseBookmarkFolderTimeline, extractFolderTweet, buildFolderTimelineUrl, FOLDER_ID_PATTERN } = __test__;
|
||||
|
||||
describe('twitter bookmark-folder URL builder', () => {
|
||||
it('embeds the folder id and count in the variables payload', () => {
|
||||
@@ -97,6 +97,8 @@ describe('twitter bookmark-folder timeline parser', () => {
|
||||
bookmarks: 3,
|
||||
created_at: 'Tue Mar 17 09:00:00 +0000 2026',
|
||||
url: 'https://x.com/alice/status/1',
|
||||
has_media: false,
|
||||
media_urls: [],
|
||||
},
|
||||
]);
|
||||
expect(nextCursor).toBe('NEXT_CURSOR');
|
||||
@@ -247,6 +249,62 @@ describe('twitter bookmark-folder timeline parser', () => {
|
||||
it('returns empty array + null cursor for unknown envelope', () => {
|
||||
expect(parseBookmarkFolderTimeline({}, new Set())).toEqual({ tweets: [], nextCursor: null });
|
||||
});
|
||||
|
||||
it('includes photo media URLs from extended_entities', () => {
|
||||
const tweet = extractFolderTweet({
|
||||
rest_id: '101',
|
||||
legacy: {
|
||||
full_text: 'pic folder tweet',
|
||||
extended_entities: {
|
||||
media: [
|
||||
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/abc.jpg' },
|
||||
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/def.jpg' },
|
||||
],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'eve' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls).toEqual([
|
||||
'https://pbs.twimg.com/media/abc.jpg',
|
||||
'https://pbs.twimg.com/media/def.jpg',
|
||||
]);
|
||||
});
|
||||
|
||||
it('extracts mp4 variant URL for video media', () => {
|
||||
const tweet = extractFolderTweet({
|
||||
rest_id: '102',
|
||||
legacy: {
|
||||
full_text: 'video folder tweet',
|
||||
extended_entities: {
|
||||
media: [{
|
||||
type: 'video',
|
||||
media_url_https: 'https://pbs.twimg.com/amplify_video_thumb/thumb.jpg',
|
||||
video_info: {
|
||||
variants: [
|
||||
{ content_type: 'application/x-mpegURL', url: 'https://video.twimg.com/playlist.m3u8' },
|
||||
{ content_type: 'video/mp4', bitrate: 832000, url: 'https://video.twimg.com/low.mp4' },
|
||||
{ content_type: 'video/mp4', bitrate: 2176000, url: 'https://video.twimg.com/high.mp4' },
|
||||
],
|
||||
},
|
||||
}],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'frank' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls?.[0]).toMatch(/\.mp4$/);
|
||||
});
|
||||
|
||||
it('returns has_media false / media_urls empty when no media present', () => {
|
||||
const tweet = extractFolderTweet({
|
||||
rest_id: '103',
|
||||
legacy: { full_text: 'text only', favorite_count: 0, retweet_count: 0, bookmark_count: 0 },
|
||||
core: { user_results: { result: { legacy: { screen_name: 'gail' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(false);
|
||||
expect(tweet?.media_urls).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('twitter bookmark-folder id validation', () => {
|
||||
|
||||
@@ -77,7 +77,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [],
|
||||
columns: ['id', 'name', 'items', 'created_at'],
|
||||
func: async (page) => {
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { extractMedia } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
const BOOKMARKS_QUERY_ID = 'Fy0QMy4q_aZCpkO0PnyLYw';
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
const FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
profile_label_improvements_pcf_label_in_post_enabled: true,
|
||||
@@ -41,7 +43,7 @@ function buildBookmarksUrl(count, cursor) {
|
||||
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
|
||||
+ `&features=${encodeURIComponent(JSON.stringify(FEATURES))}`;
|
||||
}
|
||||
function extractBookmarkTweet(result, seen) {
|
||||
export function extractBookmarkTweet(result, seen) {
|
||||
if (!result)
|
||||
return null;
|
||||
const tw = result.tweet || result;
|
||||
@@ -63,9 +65,10 @@ function extractBookmarkTweet(result, seen) {
|
||||
bookmarks: legacy.bookmark_count || 0,
|
||||
created_at: legacy.created_at || '',
|
||||
url: `https://x.com/${screenName}/status/${tw.rest_id}`,
|
||||
...extractMedia(legacy),
|
||||
};
|
||||
}
|
||||
function parseBookmarks(data, seen) {
|
||||
export function parseBookmarks(data, seen) {
|
||||
const tweets = [];
|
||||
let nextCursor = null;
|
||||
const instructions = data?.data?.bookmark_timeline_v2?.timeline?.instructions
|
||||
@@ -105,12 +108,11 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of bookmarks to return (default 20).' },
|
||||
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the bookmarks by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the API\'s native (saved-time) ordering.' },
|
||||
],
|
||||
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url'],
|
||||
columns: ['id', 'author', 'text', 'likes', 'retweets', 'bookmarks', 'created_at', 'url', 'has_media', 'media_urls'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = kwargs.limit || 20;
|
||||
const cookies = await page.getCookies({ url: 'https://x.com' });
|
||||
@@ -150,7 +152,8 @@ cli({
|
||||
const allTweets = [];
|
||||
const seen = new Set();
|
||||
let cursor = null;
|
||||
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
|
||||
const fetchCount = Math.min(100, limit - allTweets.length + 10);
|
||||
const apiUrl = buildBookmarksUrl(fetchCount, cursor).replace(BOOKMARKS_QUERY_ID, queryId);
|
||||
const data = await page.evaluate(`async () => {
|
||||
@@ -172,3 +175,7 @@ cli({
|
||||
return applyTopByEngagement(trimmed, kwargs['top-by-engagement']);
|
||||
},
|
||||
});
|
||||
export const __test__ = {
|
||||
parseBookmarks,
|
||||
extractBookmarkTweet,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './bookmarks.js';
|
||||
|
||||
const { parseBookmarks, extractBookmarkTweet } = __test__;
|
||||
|
||||
describe('twitter bookmarks parser', () => {
|
||||
it('extracts a baseline tweet with no media (has_media false, media_urls empty)', () => {
|
||||
const tweet = extractBookmarkTweet({
|
||||
rest_id: '1',
|
||||
legacy: {
|
||||
full_text: 'plain bookmark',
|
||||
favorite_count: 5,
|
||||
retweet_count: 1,
|
||||
bookmark_count: 2,
|
||||
created_at: 'Wed Apr 16 10:00:00 +0000 2026',
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
|
||||
}, new Set());
|
||||
expect(tweet).toEqual({
|
||||
id: '1',
|
||||
author: 'alice',
|
||||
name: 'Alice',
|
||||
text: 'plain bookmark',
|
||||
likes: 5,
|
||||
retweets: 1,
|
||||
bookmarks: 2,
|
||||
created_at: 'Wed Apr 16 10:00:00 +0000 2026',
|
||||
url: 'https://x.com/alice/status/1',
|
||||
has_media: false,
|
||||
media_urls: [],
|
||||
});
|
||||
});
|
||||
|
||||
it('includes photo media URLs from extended_entities', () => {
|
||||
const tweet = extractBookmarkTweet({
|
||||
rest_id: '101',
|
||||
legacy: {
|
||||
full_text: 'pic bookmark',
|
||||
extended_entities: {
|
||||
media: [
|
||||
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/abc.jpg' },
|
||||
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/def.jpg' },
|
||||
],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'bob' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls).toEqual([
|
||||
'https://pbs.twimg.com/media/abc.jpg',
|
||||
'https://pbs.twimg.com/media/def.jpg',
|
||||
]);
|
||||
});
|
||||
|
||||
it('extracts mp4 variant URL for video media', () => {
|
||||
const tweet = extractBookmarkTweet({
|
||||
rest_id: '102',
|
||||
legacy: {
|
||||
full_text: 'video bookmark',
|
||||
extended_entities: {
|
||||
media: [{
|
||||
type: 'video',
|
||||
media_url_https: 'https://pbs.twimg.com/amplify_video_thumb/thumb.jpg',
|
||||
video_info: {
|
||||
variants: [
|
||||
{ content_type: 'application/x-mpegURL', url: 'https://video.twimg.com/playlist.m3u8' },
|
||||
{ content_type: 'video/mp4', bitrate: 832000, url: 'https://video.twimg.com/low.mp4' },
|
||||
{ content_type: 'video/mp4', bitrate: 2176000, url: 'https://video.twimg.com/high.mp4' },
|
||||
],
|
||||
},
|
||||
}],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'carol' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls?.[0]).toMatch(/\.mp4$/);
|
||||
});
|
||||
|
||||
it('falls back to entities.media when extended_entities is absent', () => {
|
||||
const tweet = extractBookmarkTweet({
|
||||
rest_id: '103',
|
||||
legacy: {
|
||||
full_text: 'entities-only media',
|
||||
entities: {
|
||||
media: [{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/legacy.jpg' }],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'dave' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls).toEqual(['https://pbs.twimg.com/media/legacy.jpg']);
|
||||
});
|
||||
|
||||
it('prefers note_tweet text over truncated full_text', () => {
|
||||
const tweet = extractBookmarkTweet({
|
||||
rest_id: '2',
|
||||
legacy: { full_text: 'short text…', favorite_count: 0, retweet_count: 0, bookmark_count: 0 },
|
||||
note_tweet: { note_tweet_results: { result: { text: 'full long-form text body' } } },
|
||||
core: { user_results: { result: { core: { screen_name: 'erin' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.text).toBe('full long-form text body');
|
||||
});
|
||||
|
||||
it('deduplicates tweets across the seen Set', () => {
|
||||
const data = {
|
||||
data: {
|
||||
bookmark_timeline_v2: {
|
||||
timeline: {
|
||||
instructions: [{
|
||||
entries: [
|
||||
{
|
||||
entryId: 'tweet-3',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '3',
|
||||
legacy: { full_text: 'first', favorite_count: 0, retweet_count: 0, bookmark_count: 0 },
|
||||
core: { user_results: { result: { legacy: { screen_name: 'frank' } } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
entryId: 'tweet-3-dup',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '3',
|
||||
legacy: { full_text: 'duplicate' },
|
||||
core: { user_results: { result: { legacy: { screen_name: 'frank' } } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
}],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const seen = new Set();
|
||||
const { tweets } = parseBookmarks(data, seen);
|
||||
expect(tweets).toHaveLength(1);
|
||||
expect(tweets[0].text).toBe('first');
|
||||
});
|
||||
|
||||
it('extracts cursor + tweets from the bookmark_timeline_v2 envelope', () => {
|
||||
const data = {
|
||||
data: {
|
||||
bookmark_timeline_v2: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
{
|
||||
type: 'TimelineAddEntries',
|
||||
entries: [
|
||||
{
|
||||
entryId: 'tweet-4',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '4',
|
||||
legacy: {
|
||||
full_text: 'envelope tweet',
|
||||
favorite_count: 1,
|
||||
retweet_count: 0,
|
||||
bookmark_count: 0,
|
||||
extended_entities: {
|
||||
media: [{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/x.jpg' }],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'gina' } } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
entryId: 'cursor-bottom-Y',
|
||||
content: { __typename: 'TimelineTimelineCursor', cursorType: 'Bottom', value: 'NEXT' },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const { tweets, nextCursor } = parseBookmarks(data, new Set());
|
||||
expect(tweets).toHaveLength(1);
|
||||
expect(tweets[0].id).toBe('4');
|
||||
expect(tweets[0].has_media).toBe(true);
|
||||
expect(tweets[0].media_urls).toEqual(['https://pbs.twimg.com/media/x.jpg']);
|
||||
expect(nextCursor).toBe('NEXT');
|
||||
});
|
||||
|
||||
it('returns empty tweets + null cursor for unknown envelope', () => {
|
||||
expect(parseBookmarks({}, new Set())).toEqual({ tweets: [], nextCursor: null });
|
||||
});
|
||||
});
|
||||
@@ -15,7 +15,6 @@ cli({
|
||||
description: 'Download Twitter/X media (images and videos). Provide either <username> to scan a profile\'s media tab, or --tweet-url to download a single tweet.',
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', positional: true, help: 'Twitter username (with or without @) to scan their /media tab. Either <username> or --tweet-url is required.' },
|
||||
{ name: 'tweet-url', help: 'Single tweet URL to download. Use this OR <username>, not both required at once.' },
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { ArgumentError, AuthRequiredError, selectorError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { normalizeTwitterScreenName, unwrapBrowserResult } from './shared.js';
|
||||
|
||||
/**
|
||||
* Extract follower rows from Twitter/X follower-list SPA cells.
|
||||
@@ -72,7 +73,7 @@ async function extractFollowersFromDOM(page) {
|
||||
}
|
||||
|
||||
function normalizeScreenName(value) {
|
||||
return String(value ?? '').trim().replace(/^\/+/, '').replace(/^@+/, '');
|
||||
return normalizeTwitterScreenName(value);
|
||||
}
|
||||
|
||||
cli({
|
||||
@@ -83,7 +84,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{
|
||||
name: 'user',
|
||||
@@ -103,18 +103,27 @@ cli({
|
||||
throw new ArgumentError('limit must be a positive integer');
|
||||
}
|
||||
|
||||
let targetUser = normalizeScreenName(kwargs.user);
|
||||
const rawUser = String(kwargs.user ?? '').trim();
|
||||
let targetUser = normalizeScreenName(rawUser);
|
||||
if (rawUser && !targetUser) {
|
||||
throw new ArgumentError('twitter followers user must be a valid Twitter/X handle', 'Example: opencli twitter followers @elonmusk --limit 100');
|
||||
}
|
||||
if (!targetUser) {
|
||||
await page.goto('https://x.com/home');
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
const href = await page.evaluate(`() => {
|
||||
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
|
||||
// unwrap so the href string is usable downstream.
|
||||
const href = unwrapBrowserResult(await page.evaluate(`() => {
|
||||
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
|
||||
return link ? link.getAttribute('href') : null;
|
||||
}`);
|
||||
if (!href) {
|
||||
}`));
|
||||
if (!href || typeof href !== 'string') {
|
||||
throw new AuthRequiredError('x.com', 'Could not find logged-in user profile link. Are you logged in?');
|
||||
}
|
||||
targetUser = normalizeScreenName(href);
|
||||
if (!targetUser) {
|
||||
throw new AuthRequiredError('x.com', 'Could not find logged-in user profile link. Are you logged in?');
|
||||
}
|
||||
}
|
||||
if (!targetUser) {
|
||||
throw new ArgumentError('twitter followers user cannot be empty', 'Example: opencli twitter followers @elonmusk --limit 100');
|
||||
@@ -173,3 +182,8 @@ cli({
|
||||
return allFollowers.slice(0, limit);
|
||||
}
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
extractFollowersFromDOM,
|
||||
normalizeScreenName,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import { __test__ } from './followers.js';
|
||||
|
||||
describe('twitter followers command', () => {
|
||||
it('normalizes exact profile handles and rejects route-like hrefs', () => {
|
||||
expect(__test__.normalizeScreenName('@viewer')).toBe('viewer');
|
||||
expect(__test__.normalizeScreenName('/viewer')).toBe('viewer');
|
||||
expect(__test__.normalizeScreenName('https://x.com/viewer')).toBe('viewer');
|
||||
expect(__test__.normalizeScreenName('/home')).toBe('');
|
||||
expect(__test__.normalizeScreenName('/viewer/extra')).toBe('');
|
||||
});
|
||||
|
||||
it('rejects invalid explicit users before navigation', async () => {
|
||||
const command = getRegistry().get('twitter/followers');
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
evaluate: vi.fn(),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { user: 'viewer/extra', limit: 10 })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.wait).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects non-profile AppTabBar hrefs instead of navigating to route followers', async () => {
|
||||
const command = getRegistry().get('twitter/followers');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
if (String(script).includes('AppTabBar_Profile_Link')) return '/home';
|
||||
throw new Error(`Unexpected evaluate: ${String(script).slice(0, 80)}`);
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
|
||||
expect(page.goto).not.toHaveBeenCalledWith('https://x.com/home/followers');
|
||||
});
|
||||
});
|
||||
+32
-16
@@ -1,10 +1,11 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { resolveTwitterQueryId, sanitizeQueryId } from './shared.js';
|
||||
import { normalizeTwitterScreenName, resolveTwitterQueryId, sanitizeQueryId, unwrapBrowserResult } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN } from './utils.js';
|
||||
|
||||
const FOLLOWING_QUERY_ID = 'zx6e-TLzRkeDO_a7p4b3JQ'; // Following fallback
|
||||
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
|
||||
const FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
@@ -128,7 +129,7 @@ function parseFollowing(data) {
|
||||
}
|
||||
|
||||
function normalizeScreenName(value) {
|
||||
return String(value || '').trim().replace(/^\/+/, '').replace(/^@+/, '');
|
||||
return normalizeTwitterScreenName(value);
|
||||
}
|
||||
|
||||
cli({
|
||||
@@ -139,7 +140,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{
|
||||
name: 'user',
|
||||
@@ -156,7 +156,11 @@ cli({
|
||||
if (!Number.isInteger(limit) || limit <= 0) {
|
||||
throw new ArgumentError('twitter following --limit must be a positive integer', 'Example: opencli twitter following @elonmusk --limit 200');
|
||||
}
|
||||
let targetUser = normalizeScreenName(kwargs.user);
|
||||
const rawUser = String(kwargs.user ?? '').trim();
|
||||
let targetUser = normalizeScreenName(rawUser);
|
||||
if (rawUser && !targetUser) {
|
||||
throw new ArgumentError('twitter following user must be a valid Twitter/X handle', 'Example: opencli twitter following @elonmusk --limit 200');
|
||||
}
|
||||
|
||||
const cookies = await page.getCookies({ url: 'https://x.com' });
|
||||
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
|
||||
@@ -164,13 +168,25 @@ cli({
|
||||
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
|
||||
|
||||
if (!targetUser) {
|
||||
const href = await page.evaluate(() => {
|
||||
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
|
||||
return link ? link.getAttribute('href') : null;
|
||||
});
|
||||
if (!href)
|
||||
// Force a navigation to the home surface so the AppTabBar sidebar
|
||||
// is rendered; the framework pre-nav lands on bare x.com which
|
||||
// does not always expose AppTabBar_Profile_Link.
|
||||
await page.goto('https://x.com/home');
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
|
||||
// unwrap so the href string is usable downstream.
|
||||
// NOTE: the function-literal form `() => ...` silently drops
|
||||
// primitive return values through the bridge — only the template
|
||||
// string form preserves the `data` field.
|
||||
const href = unwrapBrowserResult(await page.evaluate(`() => {
|
||||
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
|
||||
return link ? link.getAttribute('href') : null;
|
||||
}`));
|
||||
if (!href || typeof href !== 'string')
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
targetUser = normalizeScreenName(href);
|
||||
if (!targetUser)
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
targetUser = normalizeScreenName(href.replace('/', ''));
|
||||
}
|
||||
if (!targetUser) {
|
||||
throw new ArgumentError('twitter following user cannot be empty', 'Example: opencli twitter following @elonmusk --limit 200');
|
||||
@@ -186,12 +202,12 @@ cli({
|
||||
};
|
||||
|
||||
// Get userId from screen_name
|
||||
const userLookup = await page.evaluate(async (url, headers) => {
|
||||
const userLookup = unwrapBrowserResult(await page.evaluate(async (url, headers) => {
|
||||
const resp = await fetch(url, { headers, credentials: 'include' });
|
||||
if (!resp.ok) return { error: resp.status };
|
||||
const d = await resp.json();
|
||||
return { userId: d.data?.user?.result?.rest_id || null };
|
||||
}, buildUserByScreenNameUrl(userByScreenNameQueryId, targetUser), headers);
|
||||
}, buildUserByScreenNameUrl(userByScreenNameQueryId, targetUser), headers));
|
||||
if (userLookup?.error === 401 || userLookup?.error === 403) {
|
||||
throw new AuthRequiredError('x.com', `Twitter user lookup failed (HTTP ${userLookup.error})`);
|
||||
}
|
||||
@@ -206,14 +222,14 @@ cli({
|
||||
const seen = new Set();
|
||||
let cursor = null;
|
||||
|
||||
const maxPages = Math.ceil(limit / 50) + 2;
|
||||
for (let i = 0; i < maxPages && allUsers.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && allUsers.length < limit; i++) {
|
||||
const fetchCount = Math.min(50, limit - allUsers.length + 10);
|
||||
const apiUrl = buildFollowingUrl(followingQueryId, userId, fetchCount, cursor);
|
||||
const data = await page.evaluate(async (url, headers) => {
|
||||
const data = unwrapBrowserResult(await page.evaluate(async (url, headers) => {
|
||||
const r = await fetch(url, { headers, credentials: 'include' });
|
||||
return r.ok ? await r.json() : { error: r.status };
|
||||
}, apiUrl, headers);
|
||||
}, apiUrl, headers));
|
||||
if (data?.error) {
|
||||
if (data.error === 401 || data.error === 403)
|
||||
throw new AuthRequiredError('x.com', `Twitter following request failed (HTTP ${data.error})`);
|
||||
|
||||
@@ -157,6 +157,8 @@ describe('twitter following helpers', () => {
|
||||
expect(__test__.normalizeScreenName('@elonmusk')).toBe('elonmusk');
|
||||
expect(__test__.normalizeScreenName('/elonmusk')).toBe('elonmusk');
|
||||
expect(__test__.normalizeScreenName(' @@alice ')).toBe('alice');
|
||||
expect(__test__.normalizeScreenName('/home')).toBe('');
|
||||
expect(__test__.normalizeScreenName('/elonmusk/extra')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -201,23 +203,28 @@ function followingPayload(users, cursor) {
|
||||
};
|
||||
}
|
||||
|
||||
function createFollowingPage(followingResponses, { ct0 = 'token', userLookup = { userId: '42' } } = {}) {
|
||||
function bridgeEnvelope(data) {
|
||||
return { session: 'site:twitter', data };
|
||||
}
|
||||
|
||||
function createFollowingPage(followingResponses, { ct0 = 'token', userLookup = { userId: '42' }, envelope = false } = {}) {
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => (ct0 ? [{ name: 'ct0', value: ct0 }] : [])),
|
||||
evaluate: vi.fn(async (script, ...args) => {
|
||||
const wrap = (value) => envelope ? bridgeEnvelope(value) : value;
|
||||
if (typeof script === 'function') {
|
||||
const haystack = [script.toString(), ...args.map((arg) => String(arg))].join('\n');
|
||||
if (haystack.includes('/UserByScreenName')) return userLookup;
|
||||
if (haystack.includes('/Following')) return followingResponses.shift() || followingPayload([], null);
|
||||
if (haystack.includes('AppTabBar_Profile_Link')) return '/viewer';
|
||||
if (haystack.includes('/UserByScreenName')) return wrap(userLookup);
|
||||
if (haystack.includes('/Following')) return wrap(followingResponses.shift() || followingPayload([], null));
|
||||
if (haystack.includes('AppTabBar_Profile_Link')) return wrap('/viewer');
|
||||
throw new Error(`Unexpected evaluate function: ${haystack.slice(0, 80)}`);
|
||||
}
|
||||
if (script.includes('operationName')) return null;
|
||||
if (script.includes('/UserByScreenName')) return userLookup;
|
||||
if (script.includes('/Following')) return followingResponses.shift() || followingPayload([], null);
|
||||
if (script.includes('AppTabBar_Profile_Link')) return '/viewer';
|
||||
if (script.includes('/UserByScreenName')) return wrap(userLookup);
|
||||
if (script.includes('/Following')) return wrap(followingResponses.shift() || followingPayload([], null));
|
||||
if (script.includes('AppTabBar_Profile_Link')) return wrap('/viewer');
|
||||
throw new Error(`Unexpected evaluate script: ${script.slice(0, 80)}`);
|
||||
}),
|
||||
};
|
||||
@@ -253,6 +260,29 @@ describe('twitter following command', () => {
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects invalid explicit users before cookies or navigation', async () => {
|
||||
const command = getRegistry().get('twitter/following');
|
||||
const page = createFollowingPage([]);
|
||||
|
||||
await expect(command.func(page, { user: 'elonmusk/extra', limit: 10 })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.getCookies).not.toHaveBeenCalled();
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects route-like AppTabBar hrefs as AuthRequiredError', async () => {
|
||||
const command = getRegistry().get('twitter/following');
|
||||
const page = createFollowingPage([]);
|
||||
page.evaluate.mockImplementation(async (script, ...args) => {
|
||||
const haystack = [typeof script === 'function' ? script.toString() : String(script), ...args.map((arg) => String(arg))].join('\n');
|
||||
if (haystack.includes('AppTabBar_Profile_Link')) return '/home';
|
||||
throw new Error(`Unexpected evaluate: ${haystack.slice(0, 80)}`);
|
||||
});
|
||||
|
||||
await expect(command.func(page, { limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
|
||||
});
|
||||
|
||||
it('maps first-page auth failures to AuthRequiredError', async () => {
|
||||
const command = getRegistry().get('twitter/following');
|
||||
const page = createFollowingPage([{ error: 401 }]);
|
||||
@@ -277,6 +307,20 @@ describe('twitter following command', () => {
|
||||
await expect(command.func(page, { user: 'elonmusk', limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
});
|
||||
|
||||
it('unwraps Browser Bridge envelopes for user lookup and following payloads', async () => {
|
||||
const command = getRegistry().get('twitter/following');
|
||||
const page = createFollowingPage([followingPayload(['alice'], null)], { envelope: true });
|
||||
|
||||
const rows = await command.func(page, { user: 'elonmusk', limit: 10 });
|
||||
|
||||
expect(rows.map((row) => row.screen_name)).toEqual(['alice']);
|
||||
const callText = (call) => call.map((part) => typeof part === 'function' ? part.toString() : String(part)).join('\n');
|
||||
const followingCall = page.evaluate.mock.calls.find((call) => callText(call).includes('/Following')) || [];
|
||||
const followingUrl = String(followingCall[1] || '');
|
||||
expect(decodeURIComponent(followingUrl)).toContain('"userId":"42"');
|
||||
expect(decodeURIComponent(followingUrl)).not.toContain('[object Object]');
|
||||
});
|
||||
|
||||
it('fails fast when the following timeline is empty', async () => {
|
||||
const command = getRegistry().get('twitter/following');
|
||||
const page = createFollowingPage([followingPayload([], null)]);
|
||||
|
||||
+28
-14
@@ -1,9 +1,10 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { resolveTwitterQueryId, sanitizeQueryId, extractMedia } from './shared.js';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { normalizeTwitterScreenName, resolveTwitterQueryId, sanitizeQueryId, extractMedia, unwrapBrowserResult } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
const LIKES_QUERY_ID = 'RozQdCp4CilQzrcuU0NY5w';
|
||||
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
const FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
profile_label_improvements_pcf_label_in_post_enabled: true,
|
||||
@@ -142,7 +143,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (with or without @). Defaults to the logged-in user when omitted.' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of liked tweets to return (default 20).' },
|
||||
@@ -151,20 +151,33 @@ cli({
|
||||
columns: ['id', 'author', 'name', 'text', 'likes', 'retweets', 'created_at', 'url', 'has_media', 'media_urls'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = kwargs.limit || 20;
|
||||
let username = (kwargs.username || '').replace(/^@/, '');
|
||||
const rawUsername = String(kwargs.username ?? '').trim();
|
||||
let username = normalizeTwitterScreenName(rawUsername);
|
||||
if (rawUsername && !username) {
|
||||
throw new ArgumentError('twitter likes username must be a valid Twitter/X handle', 'Example: opencli twitter likes @jack --limit 20');
|
||||
}
|
||||
const cookies = await page.getCookies({ url: 'https://x.com' });
|
||||
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
|
||||
if (!ct0)
|
||||
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
|
||||
// If no username provided, detect the logged-in user
|
||||
// If no username provided, detect the logged-in user.
|
||||
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
|
||||
// unwrap so the href string is usable downstream.
|
||||
if (!username) {
|
||||
const href = await page.evaluate(`() => {
|
||||
// Force a navigation to the home surface so the AppTabBar sidebar
|
||||
// is rendered; the framework pre-nav lands on bare x.com which
|
||||
// does not always expose AppTabBar_Profile_Link.
|
||||
await page.goto('https://x.com/home');
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
const href = unwrapBrowserResult(await page.evaluate(`() => {
|
||||
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
|
||||
return link ? link.getAttribute('href') : null;
|
||||
}`);
|
||||
if (!href)
|
||||
}`));
|
||||
if (!href || typeof href !== 'string')
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
username = normalizeTwitterScreenName(href);
|
||||
if (!username)
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
username = href.replace('/', '');
|
||||
}
|
||||
const likesQueryId = await resolveTwitterQueryId(page, 'Likes', LIKES_QUERY_ID);
|
||||
const userByScreenNameQueryId = await resolveTwitterQueryId(page, 'UserByScreenName', USER_BY_SCREEN_NAME_QUERY_ID);
|
||||
@@ -175,27 +188,28 @@ cli({
|
||||
'X-Twitter-Active-User': 'yes',
|
||||
});
|
||||
// Get userId from screen_name
|
||||
const userId = await page.evaluate(`async () => {
|
||||
const userId = unwrapBrowserResult(await page.evaluate(`async () => {
|
||||
const screenName = ${JSON.stringify(username)};
|
||||
const url = ${JSON.stringify(buildUserByScreenNameUrl(userByScreenNameQueryId, username))};
|
||||
const resp = await fetch(url, { headers: ${headers}, credentials: 'include' });
|
||||
if (!resp.ok) return null;
|
||||
const d = await resp.json();
|
||||
return d.data?.user?.result?.rest_id || null;
|
||||
}`);
|
||||
}`));
|
||||
if (!userId) {
|
||||
throw new CommandExecutionError(`Could not find user @${username}`);
|
||||
}
|
||||
const allTweets = [];
|
||||
const seen = new Set();
|
||||
let cursor = null;
|
||||
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
|
||||
const fetchCount = Math.min(100, limit - allTweets.length + 10);
|
||||
const apiUrl = buildLikesUrl(likesQueryId, userId, fetchCount, cursor);
|
||||
const data = await page.evaluate(`async () => {
|
||||
const data = unwrapBrowserResult(await page.evaluate(`async () => {
|
||||
const r = await fetch("${apiUrl}", { headers: ${headers}, credentials: 'include' });
|
||||
return r.ok ? await r.json() : { error: r.status };
|
||||
}`);
|
||||
}`));
|
||||
if (data?.error) {
|
||||
if (allTweets.length === 0)
|
||||
throw new CommandExecutionError(`HTTP ${data.error}: Failed to fetch likes. queryId may have expired.`);
|
||||
|
||||
+111
-1
@@ -1,5 +1,50 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import { __test__ } from './likes.js';
|
||||
|
||||
function likesPayload() {
|
||||
return {
|
||||
data: {
|
||||
user: {
|
||||
result: {
|
||||
timeline_v2: {
|
||||
timeline: {
|
||||
instructions: [{
|
||||
entries: [{
|
||||
entryId: 'tweet-1',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '1',
|
||||
legacy: {
|
||||
full_text: 'liked post',
|
||||
favorite_count: 7,
|
||||
retweet_count: 2,
|
||||
created_at: 'now',
|
||||
},
|
||||
core: {
|
||||
user_results: {
|
||||
result: {
|
||||
legacy: { screen_name: 'alice', name: 'Alice' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}],
|
||||
}],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('twitter likes helpers', () => {
|
||||
it('falls back when queryId contains unsafe characters', () => {
|
||||
expect(__test__.sanitizeQueryId('safe_Query-123', 'fallback')).toBe('safe_Query-123');
|
||||
@@ -83,3 +128,68 @@ describe('twitter likes helpers', () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('twitter likes command', () => {
|
||||
it('rejects invalid explicit username before cookies or navigation', async () => {
|
||||
const command = getRegistry().get('twitter/likes');
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
getCookies: vi.fn(),
|
||||
evaluate: vi.fn(),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { username: 'viewer/extra', limit: 10 })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.getCookies).not.toHaveBeenCalled();
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects route-like AppTabBar hrefs as AuthRequiredError', async () => {
|
||||
const command = getRegistry().get('twitter/likes');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
if (String(script).includes('AppTabBar_Profile_Link')) return '/home';
|
||||
throw new Error(`Unexpected evaluate: ${String(script).slice(0, 80)}`);
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { limit: 10 })).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('unwraps Browser Bridge envelopes for default-self user lookup and likes payload', async () => {
|
||||
const command = getRegistry().get('twitter/likes');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
const text = String(script);
|
||||
if (text.includes('AppTabBar_Profile_Link')) {
|
||||
return { session: 'site:twitter', data: '/viewer' };
|
||||
}
|
||||
if (text.includes('operationName')) return null;
|
||||
if (text.includes('/UserByScreenName')) {
|
||||
return { session: 'site:twitter', data: '42' };
|
||||
}
|
||||
if (text.includes('/Likes')) {
|
||||
return { session: 'site:twitter', data: likesPayload() };
|
||||
}
|
||||
throw new Error(`Unexpected evaluate: ${text.slice(0, 80)}`);
|
||||
}),
|
||||
};
|
||||
|
||||
const rows = await command.func(page, { limit: 1 });
|
||||
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ id: '1', author: 'alice', text: 'liked post' });
|
||||
const likesCall = page.evaluate.mock.calls.find(([script]) => String(script).includes('/Likes')) || [];
|
||||
expect(decodeURIComponent(String(likesCall[0]))).toContain('"userId":"42"');
|
||||
expect(decodeURIComponent(String(likesCall[0]))).not.toContain('[object Object]');
|
||||
});
|
||||
});
|
||||
|
||||
+128
-204
@@ -1,11 +1,14 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { resolveTwitterQueryId } from './shared.js';
|
||||
import { parseListsManagement } from './lists.js';
|
||||
import { TWITTER_BEARER_TOKEN } from './utils.js';
|
||||
|
||||
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
|
||||
const LISTS_MANAGEMENT_QUERY_ID = '78UbkyXwXBD98IgUWXOy9g';
|
||||
// 2026-05 fallback — X rotates queryIds; resolveTwitterQueryId() does live lookup,
|
||||
// this constant is just the default if live lookup fails.
|
||||
const LIST_ADD_MEMBER_QUERY_ID = 'vWPi0CTMoPFsjsL6W4IynQ';
|
||||
|
||||
const LISTS_MANAGEMENT_FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
@@ -62,6 +65,65 @@ function buildUserByScreenNameUrl(queryId, screenName) {
|
||||
+ `&features=${encodeURIComponent(feats)}`;
|
||||
}
|
||||
|
||||
function fatalGraphqlErrors(errors) {
|
||||
const list = Array.isArray(errors) ? errors : [];
|
||||
return list.filter((e) =>
|
||||
!(e?.path || []).join('.').includes('default_banner_media_results')
|
||||
&& !/decode/i.test(e?.message || '')
|
||||
);
|
||||
}
|
||||
|
||||
export function buildListAddMemberRow({ addResult, memberCountBefore, listId, username, userId }) {
|
||||
if (!addResult?.httpOk) {
|
||||
throw new CommandExecutionError(
|
||||
`Failed to add @${username} to list ${listId}: HTTP ${addResult?.status ?? 0}${addResult?.fetchError ? ' (' + addResult.fetchError + ')' : ''}${addResult?.raw ? ' — ' + addResult.raw : ''}`
|
||||
);
|
||||
}
|
||||
|
||||
// X often returns a partial GraphQL error on `default_banner_media_results`
|
||||
// even on successful mutations. Treat only missing main data or non-decode
|
||||
// GraphQL errors as command failures.
|
||||
const hasMemberCount = addResult.mc !== null && addResult.mc !== undefined;
|
||||
const fatalErrors = fatalGraphqlErrors(addResult.errors);
|
||||
if (!hasMemberCount && fatalErrors.length) {
|
||||
const msg = fatalErrors.map((e) => e.message || JSON.stringify(e)).join('; ');
|
||||
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: ${msg.slice(0, 300)}`);
|
||||
}
|
||||
if (!hasMemberCount) {
|
||||
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: no member_count in response`);
|
||||
}
|
||||
|
||||
const memberCountAfter = Number(addResult.mc);
|
||||
if (!Number.isFinite(memberCountAfter)) {
|
||||
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: invalid member_count in response`);
|
||||
}
|
||||
|
||||
if (memberCountAfter < memberCountBefore) {
|
||||
throw new CommandExecutionError(
|
||||
`Failed to add @${username} to list ${listId}: member_count decreased unexpectedly (${memberCountBefore} → ${memberCountAfter})`
|
||||
);
|
||||
}
|
||||
|
||||
const countIncreased = memberCountAfter > memberCountBefore;
|
||||
if (!countIncreased && addResult.isMember !== true) {
|
||||
throw new CommandExecutionError(
|
||||
`Failed to add @${username} to list ${listId}: member_count unchanged (${memberCountBefore} → ${memberCountAfter}) and response did not confirm membership`
|
||||
);
|
||||
}
|
||||
|
||||
const noop = !countIncreased;
|
||||
const verifiedBy = `member_count ${memberCountBefore} → ${memberCountAfter}`;
|
||||
return {
|
||||
listId,
|
||||
username,
|
||||
userId: String(userId),
|
||||
status: noop ? 'noop' : 'success',
|
||||
message: noop
|
||||
? `@${username} is already a member of list ${listId}`
|
||||
: `Added @${username} to list ${listId} (verified via ${verifiedBy})`,
|
||||
};
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'twitter',
|
||||
name: 'list-add',
|
||||
@@ -79,10 +141,10 @@ cli({
|
||||
const listId = String(kwargs.listId || '').trim();
|
||||
const username = String(kwargs.username || '').replace(/^@/, '').trim();
|
||||
if (!listId || !/^\d+$/.test(listId)) {
|
||||
throw new CommandExecutionError(`Invalid listId: ${JSON.stringify(kwargs.listId)}. Expected numeric ID (see \`opencli twitter lists\`).`);
|
||||
throw new ArgumentError(`Invalid listId: ${JSON.stringify(kwargs.listId)}. Expected numeric ID.`, 'Example: opencli twitter list-add 123456789 alice');
|
||||
}
|
||||
if (!username) {
|
||||
throw new CommandExecutionError('Username is required');
|
||||
throw new ArgumentError('twitter list-add username is required', 'Example: opencli twitter list-add 123456789 alice');
|
||||
}
|
||||
// Strategy.UI does not get a domain URL pre-nav from the framework.
|
||||
// This page context is load-bearing for pre-target GraphQL calls below.
|
||||
@@ -101,25 +163,33 @@ cli({
|
||||
'X-Twitter-Active-User': 'yes',
|
||||
});
|
||||
|
||||
// opencli >=1.7.x wraps page.evaluate return values as { session, data }.
|
||||
// Unwrap before use so JSON.stringify of nested values doesn't become "[object Object]".
|
||||
const unwrap = (v) => (v && typeof v === 'object' && 'session' in v && 'data' in v ? v.data : v);
|
||||
|
||||
const userLookupUrl = buildUserByScreenNameUrl(userByScreenNameQueryId, username);
|
||||
const userId = await page.evaluate(`async () => {
|
||||
const userIdRaw = await page.evaluate(`async () => {
|
||||
const resp = await fetch(${JSON.stringify(userLookupUrl)}, { headers: ${headers}, credentials: 'include' });
|
||||
if (!resp.ok) return null;
|
||||
const d = await resp.json();
|
||||
return d.data?.user?.result?.rest_id || null;
|
||||
}`);
|
||||
const userId = unwrap(userIdRaw);
|
||||
if (!userId) {
|
||||
throw new CommandExecutionError(`Could not resolve user @${username}`);
|
||||
}
|
||||
|
||||
// ListsManagementPageTimeline — used both for id→name resolution and post-op verification.
|
||||
// ListsManagementPageTimeline — used for list existence check + before/after member_count.
|
||||
const listsQueryId = await resolveTwitterQueryId(page, 'ListsManagementPageTimeline', LISTS_MANAGEMENT_QUERY_ID);
|
||||
const listsUrl = `/i/api/graphql/${listsQueryId}/ListsManagementPageTimeline?features=${encodeURIComponent(JSON.stringify(LISTS_MANAGEMENT_FEATURES))}`;
|
||||
const listsData = await page.evaluate(`async () => {
|
||||
const listsDataRaw = await page.evaluate(`async () => {
|
||||
const r = await fetch(${JSON.stringify(listsUrl)}, { headers: ${headers}, credentials: 'include' });
|
||||
if (!r.ok) return { __error: 'HTTP ' + r.status };
|
||||
return await r.json();
|
||||
}`);
|
||||
// Don't unwrap listsData: opencli spreads GraphQL response to top-level + adds session;
|
||||
// parseListsManagement reads `.data.viewer.*` from this shape directly.
|
||||
const listsData = listsDataRaw;
|
||||
const parsedLists = listsData && !listsData.__error
|
||||
? parseListsManagement(listsData, new Set())
|
||||
: [];
|
||||
@@ -131,209 +201,63 @@ cli({
|
||||
throw new CommandExecutionError(`List ${listId} not found among your lists (${parsedLists.length} lists fetched).`);
|
||||
}
|
||||
|
||||
// Use UI strategy — programmatically open "Add/Remove from Lists" dialog and toggle the target list.
|
||||
await page.goto(`https://x.com/${username}`);
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
const targetName = targetList.name;
|
||||
const uiResult = await page.evaluate(`(async () => {
|
||||
const sleep = (ms) => new Promise(r => setTimeout(r, ms));
|
||||
const findOne = (sel, root = document) => root.querySelector(sel);
|
||||
const waitFor = async (fn, { timeoutMs = 8000, intervalMs = 200 } = {}) => {
|
||||
const t0 = Date.now();
|
||||
while (Date.now() - t0 < timeoutMs) {
|
||||
const v = fn();
|
||||
if (v) return v;
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
// Direct GraphQL ListAddMember mutation.
|
||||
//
|
||||
// Previously this command opened the X profile, clicked "…" → "Add/remove from Lists",
|
||||
// navigated the dialog and used nativeClick on the Save button. In 2026-05 X replaced
|
||||
// the dialog with a full-page route (/i/lists/add_member), breaking that UI flow.
|
||||
//
|
||||
// The mutation is the same one the UI fires under the hood; calling it directly is
|
||||
// both more reliable and ~10x faster (no goto-profile + scroll-dialog roundtrip).
|
||||
const memberCountBefore = Number(targetList.members) || 0;
|
||||
const listAddMemberQueryId = await resolveTwitterQueryId(page, 'ListAddMember', LIST_ADD_MEMBER_QUERY_ID);
|
||||
const addUrl = `/i/api/graphql/${listAddMemberQueryId}/ListAddMember`;
|
||||
const addBody = JSON.stringify({
|
||||
variables: { listId, userId: String(userId) },
|
||||
queryId: listAddMemberQueryId,
|
||||
});
|
||||
const addResultJsonRaw = await page.evaluate(`async () => {
|
||||
try {
|
||||
// Install fetch + XHR interceptors to observe list-membership mutations.
|
||||
const MUTATION_RE = /ListAddMember|ListRemoveMember|lists\\/members\\/(create|destroy)|ListManagement.*Add|ListManagement.*Remove|\\/add_member|\\/remove_member|ListAddMembers|ListRemoveMembers|list.*member.*create|list.*member.*destroy/i;
|
||||
if (!window.__opencliListMutations) {
|
||||
window.__opencliListMutations = [];
|
||||
window.__opencliAllRequests = [];
|
||||
const origFetch = window.fetch.bind(window);
|
||||
window.fetch = async function(...args) {
|
||||
const url = typeof args[0] === 'string' ? args[0] : (args[0] && args[0].url) || '';
|
||||
const method = (args[1] && args[1].method) || 'GET';
|
||||
let resp;
|
||||
try { resp = await origFetch(...args); }
|
||||
catch (err) {
|
||||
if (MUTATION_RE.test(url)) window.__opencliListMutations.push({ url, method, status: 0, error: String(err), ts: Date.now(), via: 'fetch' });
|
||||
throw err;
|
||||
}
|
||||
if (method !== 'GET' && method !== 'HEAD') {
|
||||
window.__opencliAllRequests.push({ url, method, status: resp.status, ts: Date.now(), via: 'fetch' });
|
||||
}
|
||||
if (MUTATION_RE.test(url)) {
|
||||
window.__opencliListMutations.push({ url, method, status: resp.status, ts: Date.now(), via: 'fetch' });
|
||||
}
|
||||
return resp;
|
||||
};
|
||||
// Also hook XMLHttpRequest
|
||||
const OrigXhrOpen = XMLHttpRequest.prototype.open;
|
||||
const OrigXhrSend = XMLHttpRequest.prototype.send;
|
||||
XMLHttpRequest.prototype.open = function(method, url, ...rest) {
|
||||
this.__opencliMethod = method;
|
||||
this.__opencliUrl = url;
|
||||
return OrigXhrOpen.call(this, method, url, ...rest);
|
||||
};
|
||||
XMLHttpRequest.prototype.send = function(...args) {
|
||||
const xhr = this;
|
||||
xhr.addEventListener('loadend', () => {
|
||||
const url = xhr.__opencliUrl || '';
|
||||
const method = xhr.__opencliMethod || 'GET';
|
||||
if (method !== 'GET' && method !== 'HEAD') {
|
||||
window.__opencliAllRequests.push({ url, method, status: xhr.status, ts: Date.now(), via: 'xhr' });
|
||||
}
|
||||
if (MUTATION_RE.test(url)) {
|
||||
window.__opencliListMutations.push({ url, method, status: xhr.status, ts: Date.now(), via: 'xhr' });
|
||||
}
|
||||
});
|
||||
return OrigXhrSend.apply(this, args);
|
||||
};
|
||||
}
|
||||
window.__opencliListMutations.length = 0;
|
||||
window.__opencliAllRequests.length = 0;
|
||||
|
||||
const caret = await waitFor(() => findOne('[data-testid="userActions"]'));
|
||||
if (!caret) return { ok: false, message: 'Could not find user actions (…) button. Are you logged in?' };
|
||||
caret.click();
|
||||
await sleep(600);
|
||||
const menuItems = Array.from(document.querySelectorAll('[role="menuitem"]'));
|
||||
const addToListItem = menuItems.find(el => /add\\/remove|从列表|列表|add to list|add or remove/i.test(el.innerText));
|
||||
if (!addToListItem) {
|
||||
return { ok: false, message: 'Could not find "Add/remove from Lists" menu item' };
|
||||
}
|
||||
addToListItem.click();
|
||||
await sleep(1200);
|
||||
const dialog = await waitFor(() => findOne('[role="dialog"]'));
|
||||
if (!dialog) return { ok: false, message: 'List selection dialog did not open' };
|
||||
|
||||
const targetName = ${JSON.stringify(targetName)};
|
||||
// Find the real scroll container (virtualized list). Try a few candidates.
|
||||
const scrollCandidates = [
|
||||
dialog.querySelector('[data-viewportview="true"]'),
|
||||
dialog.querySelector('[aria-label]')?.parentElement,
|
||||
...Array.from(dialog.querySelectorAll('div')).filter(d => d.scrollHeight > d.clientHeight + 10),
|
||||
].filter(Boolean);
|
||||
let row = null;
|
||||
let scrollEl = scrollCandidates[0] || dialog;
|
||||
for (const se of scrollCandidates) {
|
||||
if (se.scrollHeight > se.clientHeight + 10) { scrollEl = se; break; }
|
||||
}
|
||||
let lastScrollTop = -1;
|
||||
for (let i = 0; i < 12; i++) {
|
||||
const cells = Array.from(dialog.querySelectorAll('[data-testid="cellInnerDiv"]'));
|
||||
row = cells.find(c => (c.innerText || '').split('\\n')[0].trim() === targetName);
|
||||
if (row) break;
|
||||
// Incremental scroll within the container
|
||||
const prev = scrollEl.scrollTop;
|
||||
scrollEl.scrollTop = prev + Math.max(200, scrollEl.clientHeight - 100);
|
||||
if (scrollEl.scrollTop === prev) {
|
||||
// Couldn't scroll further. Give up.
|
||||
if (scrollEl.scrollTop === lastScrollTop) break;
|
||||
}
|
||||
lastScrollTop = scrollEl.scrollTop;
|
||||
await sleep(500);
|
||||
}
|
||||
if (!row) {
|
||||
const names = Array.from(dialog.querySelectorAll('[data-testid="cellInnerDiv"]'))
|
||||
.map(c => (c.innerText || '').split('\\n')[0].trim()).filter(Boolean);
|
||||
const dialogText = (dialog.innerText || '').slice(0, 500);
|
||||
return { ok: false, message: 'List "' + targetName + '" not found. Cells: [' + names.join(' | ') + ']. DialogText: ' + dialogText };
|
||||
}
|
||||
const listCell = row.querySelector('[data-testid="listCell"]') || row.querySelector('[role="checkbox"]') || row;
|
||||
const readChecked = () => {
|
||||
const v = listCell.getAttribute('aria-checked');
|
||||
return v === 'true' || v === 'false' ? v : null;
|
||||
};
|
||||
await sleep(600);
|
||||
let ariaChecked = readChecked();
|
||||
for (let i = 0; i < 8; i++) {
|
||||
await sleep(500);
|
||||
const next = readChecked();
|
||||
if (next && next === ariaChecked) break;
|
||||
ariaChecked = next || ariaChecked;
|
||||
}
|
||||
const isMember = ariaChecked === 'true';
|
||||
if (isMember) {
|
||||
const closeBtn = findOne('[data-testid="app-bar-close"]') || findOne('[aria-label="Close"]');
|
||||
if (closeBtn) closeBtn.click();
|
||||
return { ok: true, noop: true };
|
||||
}
|
||||
try { listCell.scrollIntoView({ block: 'center' }); } catch {}
|
||||
await sleep(400);
|
||||
const mutationsBefore = window.__opencliListMutations.length;
|
||||
const rowRect = listCell.getBoundingClientRect();
|
||||
// Find the Save button (top-right of dialog). Match by text "Save" / "Done" / CJK equivalents.
|
||||
const saveButton = Array.from(dialog.querySelectorAll('[role="button"], button')).find(b => {
|
||||
const txt = (b.innerText || '').trim();
|
||||
return /^(Save|Done|保存|完成|儲存)$/i.test(txt);
|
||||
const r = await fetch(${JSON.stringify(addUrl)}, {
|
||||
method: 'POST',
|
||||
headers: Object.assign({}, ${headers}, { 'Content-Type': 'application/json' }),
|
||||
credentials: 'include',
|
||||
body: ${JSON.stringify(addBody)},
|
||||
});
|
||||
const saveRect = saveButton ? saveButton.getBoundingClientRect() : null;
|
||||
return {
|
||||
ok: true,
|
||||
needsNativeInteraction: true,
|
||||
rowClickX: Math.round(rowRect.left + rowRect.width / 2),
|
||||
rowClickY: Math.round(rowRect.top + rowRect.height / 2),
|
||||
saveClickX: saveRect ? Math.round(saveRect.left + saveRect.width / 2) : null,
|
||||
saveClickY: saveRect ? Math.round(saveRect.top + saveRect.height / 2) : null,
|
||||
saveText: saveButton ? (saveButton.innerText || '').trim() : null,
|
||||
mutationsBefore,
|
||||
ariaBefore: ariaChecked,
|
||||
};
|
||||
const text = await r.text();
|
||||
let body;
|
||||
let raw = null;
|
||||
try { body = JSON.parse(text); } catch { body = null; raw = text.slice(0, 300); }
|
||||
const list = body && body.data && body.data.list ? body.data.list : null;
|
||||
return JSON.stringify([
|
||||
r.ok,
|
||||
r.status,
|
||||
list ? list.member_count : null,
|
||||
list ? list.is_member : null,
|
||||
body && body.errors ? body.errors : null,
|
||||
raw,
|
||||
null,
|
||||
]);
|
||||
} catch (e) {
|
||||
return { ok: false, message: 'UI error: ' + (e?.message || String(e)) };
|
||||
return JSON.stringify([false, 0, null, null, null, null, String(e)]);
|
||||
}
|
||||
})()`);
|
||||
|
||||
if (!uiResult.ok) {
|
||||
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: ${uiResult.message}`);
|
||||
}`);
|
||||
const addResultJson = unwrap(addResultJsonRaw);
|
||||
let addResultTuple;
|
||||
try {
|
||||
addResultTuple = JSON.parse(addResultJson);
|
||||
} catch {
|
||||
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: malformed mutation response envelope`);
|
||||
}
|
||||
const addResult = Object.create(null);
|
||||
addResult.httpOk = Boolean(addResultTuple?.[0]);
|
||||
addResult.status = Number(addResultTuple?.[1]) || 0;
|
||||
addResult.mc = addResultTuple?.[2];
|
||||
addResult.isMember = addResultTuple?.[3];
|
||||
addResult.errors = addResultTuple?.[4];
|
||||
addResult.raw = addResultTuple?.[5];
|
||||
addResult.fetchError = addResultTuple?.[6];
|
||||
|
||||
let verifiedBy = null;
|
||||
if (uiResult.needsNativeInteraction) {
|
||||
if (typeof page.nativeClick !== 'function' || typeof page.nativeKeyPress !== 'function') {
|
||||
throw new CommandExecutionError('Requires up-to-date Chrome extension (nativeClick + nativeKeyPress).');
|
||||
}
|
||||
if (!uiResult.saveClickX) {
|
||||
throw new CommandExecutionError(`Save button not found in dialog (X expected text Save/Done). Dialog structure may have changed.`);
|
||||
}
|
||||
const memberCountBefore = Number(targetList.members) || 0;
|
||||
// 1. Trusted click on row → aria flips false→true (optimistic UI)
|
||||
await page.nativeClick(uiResult.rowClickX, uiResult.rowClickY);
|
||||
await new Promise((r) => setTimeout(r, 800));
|
||||
// 2. Trusted click on Save button → X commits to server
|
||||
await page.nativeClick(uiResult.saveClickX, uiResult.saveClickY);
|
||||
await new Promise((r) => setTimeout(r, 3500));
|
||||
// Ground truth: re-fetch ListsManagementPageTimeline and compare member_count
|
||||
const listsAfter = await page.evaluate(`async () => {
|
||||
const r = await fetch(${JSON.stringify(listsUrl)}, { headers: ${headers}, credentials: 'include' });
|
||||
if (!r.ok) return { __error: 'HTTP ' + r.status };
|
||||
return await r.json();
|
||||
}`);
|
||||
const parsedAfter = listsAfter && !listsAfter.__error
|
||||
? parseListsManagement(listsAfter, new Set())
|
||||
: [];
|
||||
const afterList = parsedAfter.find((l) => l.id === listId);
|
||||
const memberCountAfter = afterList ? Number(afterList.members) || 0 : -1;
|
||||
if (memberCountAfter > memberCountBefore) {
|
||||
verifiedBy = `member_count ${memberCountBefore} → ${memberCountAfter}`;
|
||||
} else {
|
||||
throw new CommandExecutionError(`Failed to add @${username} to list ${listId}: member_count unchanged (${memberCountBefore} → ${memberCountAfter}). X's UI flipped but did not commit — try reloading page/extension.`);
|
||||
}
|
||||
}
|
||||
|
||||
return [{
|
||||
listId,
|
||||
username,
|
||||
userId: String(userId),
|
||||
status: uiResult.noop ? 'noop' : 'success',
|
||||
message: uiResult.noop
|
||||
? `@${username} is already a member of list ${listId}`
|
||||
: `Added @${username} to list ${listId} (verified via ${verifiedBy})`,
|
||||
}];
|
||||
return [buildListAddMemberRow({ addResult, memberCountBefore, listId, username, userId })];
|
||||
},
|
||||
});
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import './list-add.js';
|
||||
import { ArgumentError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { buildListAddMemberRow } from './list-add.js';
|
||||
|
||||
describe('twitter list-add registration', () => {
|
||||
it('registers the list-add command with the expected shape', () => {
|
||||
@@ -34,4 +35,99 @@ describe('twitter list-add registration', () => {
|
||||
expect(page.wait).toHaveBeenCalledWith(3);
|
||||
expect(page.getCookies).toHaveBeenCalledWith({ url: 'https://x.com' });
|
||||
});
|
||||
|
||||
it('rejects invalid user input before navigation', async () => {
|
||||
const cmd = getRegistry().get('twitter/list-add');
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
getCookies: vi.fn(),
|
||||
evaluate: vi.fn(),
|
||||
};
|
||||
|
||||
await expect(cmd.func(page, { listId: 'abc', username: 'alice' })).rejects.toBeInstanceOf(ArgumentError);
|
||||
await expect(cmd.func(page, { listId: '123', username: '' })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('builds success rows when member_count increases despite non-fatal decode errors', () => {
|
||||
const row = buildListAddMemberRow({
|
||||
addResult: {
|
||||
httpOk: true,
|
||||
status: 200,
|
||||
mc: 11,
|
||||
isMember: true,
|
||||
errors: [{ path: ['data', 'list', 'default_banner_media_results'], message: 'decode failed' }],
|
||||
},
|
||||
memberCountBefore: 10,
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
});
|
||||
|
||||
expect(row).toMatchObject({
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
status: 'success',
|
||||
});
|
||||
expect(row.message).toContain('member_count 10 → 11');
|
||||
});
|
||||
|
||||
it('treats unchanged member_count as noop only when membership is confirmed', () => {
|
||||
const row = buildListAddMemberRow({
|
||||
addResult: { httpOk: true, status: 200, mc: 10, isMember: true, errors: null },
|
||||
memberCountBefore: 10,
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
});
|
||||
|
||||
expect(row.status).toBe('noop');
|
||||
expect(row.message).toBe('@alice is already a member of list 123');
|
||||
});
|
||||
|
||||
it('fails typed when unchanged member_count does not confirm membership', () => {
|
||||
expect(() => buildListAddMemberRow({
|
||||
addResult: { httpOk: true, status: 200, mc: 10, isMember: false, errors: null },
|
||||
memberCountBefore: 10,
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
})).toThrow(CommandExecutionError);
|
||||
});
|
||||
|
||||
it('fails typed when member_count decreases unexpectedly', () => {
|
||||
expect(() => buildListAddMemberRow({
|
||||
addResult: { httpOk: true, status: 200, mc: 9, isMember: true, errors: null },
|
||||
memberCountBefore: 10,
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
})).toThrow(/decreased unexpectedly/);
|
||||
});
|
||||
|
||||
it('fails typed when GraphQL response has no usable member_count', () => {
|
||||
expect(() => buildListAddMemberRow({
|
||||
addResult: {
|
||||
httpOk: true,
|
||||
status: 200,
|
||||
mc: undefined,
|
||||
isMember: null,
|
||||
errors: [{ message: 'List is unavailable', path: ['data', 'list'] }],
|
||||
},
|
||||
memberCountBefore: 10,
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
})).toThrow(/List is unavailable/);
|
||||
|
||||
expect(() => buildListAddMemberRow({
|
||||
addResult: { httpOk: true, status: 200, mc: null, isMember: null, errors: { message: 'not an array' } },
|
||||
memberCountBefore: 10,
|
||||
listId: '123',
|
||||
username: 'alice',
|
||||
userId: '42',
|
||||
})).toThrow(/no member_count/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { extractMedia } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
|
||||
const LIST_TWEETS_QUERY_ID = 'RlZzktZY_9wJynoepm8ZsA';
|
||||
const OPERATION_NAME = 'ListLatestTweetsTimeline';
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
|
||||
const FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
@@ -70,6 +72,7 @@ export function extractTimelineTweet(result, seen) {
|
||||
replies: legacy.reply_count || 0,
|
||||
created_at: legacy.created_at || '',
|
||||
url: `https://x.com/${screenName}/status/${tw.rest_id}`,
|
||||
...extractMedia(legacy),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -112,13 +115,12 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'listId', positional: true, type: 'string', required: true, help: 'Numeric ID of a Twitter/X list (e.g. from `opencli twitter lists`)' },
|
||||
{ name: 'limit', type: 'int', default: 50 },
|
||||
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the list timeline by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the list\'s native (recency) ordering.' },
|
||||
],
|
||||
columns: ['id', 'author', 'text', 'likes', 'retweets', 'replies', 'created_at', 'url'],
|
||||
columns: ['id', 'author', 'text', 'likes', 'retweets', 'replies', 'created_at', 'url', 'has_media', 'media_urls'],
|
||||
func: async (page, kwargs) => {
|
||||
const listId = String(kwargs.listId || '').trim();
|
||||
if (!listId || !/^\d+$/.test(listId)) {
|
||||
@@ -129,7 +131,11 @@ cli({
|
||||
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
|
||||
if (!ct0)
|
||||
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
|
||||
const queryId = await page.evaluate(`async () => {
|
||||
// opencli >=1.7.x wraps primitive page.evaluate returns as { session, data: <value> }.
|
||||
// Without unwrap, the string queryId becomes "[object Object]" when interpolated into the URL,
|
||||
// causing HTTP 400 "queryId may have expired".
|
||||
const unwrap = (v) => (v && typeof v === 'object' && 'session' in v && 'data' in v ? v.data : v);
|
||||
const queryIdRaw = await page.evaluate(`async () => {
|
||||
try {
|
||||
const ghResp = await fetch('https://raw.githubusercontent.com/fa0311/twitter-openapi/refs/heads/main/src/config/placeholder.json');
|
||||
if (ghResp.ok) {
|
||||
@@ -152,7 +158,8 @@ cli({
|
||||
}
|
||||
} catch {}
|
||||
return null;
|
||||
}`) || LIST_TWEETS_QUERY_ID;
|
||||
}`);
|
||||
const queryId = unwrap(queryIdRaw) || LIST_TWEETS_QUERY_ID;
|
||||
const headers = JSON.stringify({
|
||||
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
|
||||
'X-Csrf-Token': ct0,
|
||||
@@ -162,7 +169,8 @@ cli({
|
||||
const allTweets = [];
|
||||
const seen = new Set();
|
||||
let cursor = null;
|
||||
for (let i = 0; i < 10 && allTweets.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
|
||||
const fetchCount = Math.min(100, limit - allTweets.length + 10);
|
||||
const apiUrl = buildUrl(queryId, listId, fetchCount, cursor);
|
||||
const data = await page.evaluate(`async () => {
|
||||
|
||||
@@ -30,9 +30,57 @@ describe('twitter list-tweets parser', () => {
|
||||
replies: 2,
|
||||
created_at: 'Wed Apr 16 10:00:00 +0000 2026',
|
||||
url: 'https://x.com/bob/status/99',
|
||||
has_media: false,
|
||||
media_urls: [],
|
||||
});
|
||||
});
|
||||
|
||||
it('includes photo media URLs from extended_entities', () => {
|
||||
const tweet = extractTimelineTweet({
|
||||
rest_id: '101',
|
||||
legacy: {
|
||||
full_text: 'pic post',
|
||||
extended_entities: {
|
||||
media: [
|
||||
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/abc.jpg' },
|
||||
{ type: 'photo', media_url_https: 'https://pbs.twimg.com/media/def.jpg' },
|
||||
],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'dave' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls).toEqual([
|
||||
'https://pbs.twimg.com/media/abc.jpg',
|
||||
'https://pbs.twimg.com/media/def.jpg',
|
||||
]);
|
||||
});
|
||||
|
||||
it('extracts mp4 variant URL for video media', () => {
|
||||
const tweet = extractTimelineTweet({
|
||||
rest_id: '102',
|
||||
legacy: {
|
||||
full_text: 'video post',
|
||||
extended_entities: {
|
||||
media: [{
|
||||
type: 'video',
|
||||
media_url_https: 'https://pbs.twimg.com/amplify_video_thumb/thumb.jpg',
|
||||
video_info: {
|
||||
variants: [
|
||||
{ content_type: 'application/x-mpegURL', url: 'https://video.twimg.com/playlist.m3u8' },
|
||||
{ content_type: 'video/mp4', bitrate: 832000, url: 'https://video.twimg.com/low.mp4' },
|
||||
{ content_type: 'video/mp4', bitrate: 2176000, url: 'https://video.twimg.com/high.mp4' },
|
||||
],
|
||||
},
|
||||
}],
|
||||
},
|
||||
},
|
||||
core: { user_results: { result: { legacy: { screen_name: 'erin' } } } },
|
||||
}, new Set());
|
||||
expect(tweet?.has_media).toBe(true);
|
||||
expect(tweet?.media_urls?.[0]).toMatch(/\.mp4$/);
|
||||
});
|
||||
|
||||
it('prefers long-form note_tweet text over truncated legacy full_text', () => {
|
||||
const tweet = extractTimelineTweet({
|
||||
rest_id: '100',
|
||||
|
||||
@@ -92,7 +92,6 @@ export const command = cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 50, help: 'Maximum number of lists to return (default 50).' },
|
||||
],
|
||||
@@ -103,7 +102,9 @@ export const command = cli({
|
||||
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
|
||||
if (!ct0)
|
||||
throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
|
||||
const queryId = await page.evaluate(`async () => {
|
||||
// opencli >=1.7.x wraps primitive page.evaluate returns as { session, data: <value> }.
|
||||
const unwrap = (v) => (v && typeof v === 'object' && 'session' in v && 'data' in v ? v.data : v);
|
||||
const queryIdRaw = await page.evaluate(`async () => {
|
||||
try {
|
||||
const ghResp = await fetch('https://raw.githubusercontent.com/fa0311/twitter-openapi/refs/heads/main/src/config/placeholder.json');
|
||||
if (ghResp.ok) {
|
||||
@@ -126,7 +127,8 @@ export const command = cli({
|
||||
}
|
||||
} catch {}
|
||||
return null;
|
||||
}`) || LISTS_QUERY_ID;
|
||||
}`);
|
||||
const queryId = unwrap(queryIdRaw) || LISTS_QUERY_ID;
|
||||
const headers = JSON.stringify({
|
||||
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
|
||||
'X-Csrf-Token': ct0,
|
||||
|
||||
@@ -8,7 +8,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.INTERCEPT,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Maximum number of notifications to return (default 20).' },
|
||||
],
|
||||
|
||||
+23
-4
@@ -161,12 +161,25 @@ async function submitTweet(page, text) {
|
||||
const normalize = s => String(s || '').replace(/\u00a0/g, ' ').replace(/\s+/g, ' ').trim();
|
||||
const expectedText = normalize(expected);
|
||||
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
|
||||
const statusUrl = (root = document) => {
|
||||
const links = Array.from(root.querySelectorAll('a[href*="/status/"]'));
|
||||
for (const link of links) {
|
||||
const href = link.href || link.getAttribute('href') || '';
|
||||
if (!href) continue;
|
||||
try {
|
||||
const url = new URL(href, window.location.origin);
|
||||
const match = url.pathname.match(/^\\/(?:[^/]+|i)\\/status\\/(\\d+)/);
|
||||
if (match) return { url: url.href, id: match[1] };
|
||||
} catch {}
|
||||
}
|
||||
return {};
|
||||
};
|
||||
for (let i = 0; i < ${JSON.stringify(iterations)}; i++) {
|
||||
await new Promise(r => setTimeout(r, ${JSON.stringify(SUBMIT_POLL_MS)}));
|
||||
const toasts = Array.from(document.querySelectorAll('[role="alert"], [data-testid="toast"]'))
|
||||
.filter((el) => visible(el));
|
||||
const successToast = toasts.find((el) => /sent|posted|your post was sent|your tweet was sent/i.test(el.textContent || ''));
|
||||
if (successToast) return { ok: true, message: 'Tweet posted successfully.' };
|
||||
if (successToast) return { ok: true, message: 'Tweet posted successfully.', ...statusUrl(successToast) };
|
||||
const alert = toasts.find((el) => /failed|error|try again|not sent|could not/i.test(el.textContent || ''));
|
||||
if (alert) return { ok: false, message: (alert.textContent || 'Tweet failed to post.').trim() };
|
||||
|
||||
@@ -175,7 +188,7 @@ async function submitTweet(page, text) {
|
||||
const hasMedia = !!document.querySelector('[data-testid="attachments"], [data-testid="tweetPhoto"]')
|
||||
|| document.querySelectorAll('img[src^="blob:"], video[src^="blob:"]').length > 0;
|
||||
if (!composerStillHasText && !hasMedia) {
|
||||
return { ok: true, message: 'Tweet posted successfully.' };
|
||||
return { ok: true, message: 'Tweet posted successfully.', ...statusUrl() };
|
||||
}
|
||||
}
|
||||
return { ok: false, message: 'Tweet submission did not complete before timeout.' };
|
||||
@@ -194,7 +207,7 @@ cli({
|
||||
{ name: 'text', type: 'string', required: true, positional: true, help: 'The text content of the tweet' },
|
||||
{ name: 'images', type: 'string', required: false, help: 'Image paths, comma-separated, max 4 (jpg/png/gif/webp)' },
|
||||
],
|
||||
columns: ['status', 'message', 'text'],
|
||||
columns: ['status', 'message', 'text', 'id', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
if (!page)
|
||||
throw new CommandExecutionError('Browser session required for twitter post');
|
||||
@@ -231,6 +244,12 @@ cli({
|
||||
|
||||
await page.wait(1);
|
||||
const result = await submitTweet(page, text);
|
||||
return [{ status: result?.ok ? 'success' : 'failed', message: result?.message ?? 'Tweet failed to post.', text }];
|
||||
return [{
|
||||
status: result?.ok ? 'success' : 'failed',
|
||||
message: result?.message ?? 'Tweet failed to post.',
|
||||
text,
|
||||
...(result?.id ? { id: result.id } : {}),
|
||||
...(result?.url ? { url: result.url } : {}),
|
||||
}];
|
||||
}
|
||||
});
|
||||
|
||||
@@ -46,6 +46,11 @@ function makePage(evaluateResults = [], overrides = {}) {
|
||||
describe('twitter post command', () => {
|
||||
const getCommand = () => getRegistry().get('twitter/post');
|
||||
|
||||
it('registers created tweet id/url columns', () => {
|
||||
const command = getCommand();
|
||||
expect(command?.columns).toEqual(['status', 'message', 'text', 'id', 'url']);
|
||||
});
|
||||
|
||||
it('posts text-only tweet successfully through the current compose route', async () => {
|
||||
const command = getCommand();
|
||||
const page = makePage([
|
||||
@@ -63,6 +68,31 @@ describe('twitter post command', () => {
|
||||
expect(page.insertText).toHaveBeenCalledWith('hello world');
|
||||
});
|
||||
|
||||
it('returns the created tweet URL from the success toast when available', async () => {
|
||||
const command = getCommand();
|
||||
const page = makePage([
|
||||
{ ok: true },
|
||||
{ ok: true },
|
||||
{ ok: true },
|
||||
{
|
||||
ok: true,
|
||||
message: 'Tweet posted successfully.',
|
||||
id: '2054239044884693381',
|
||||
url: 'https://x.com/darthjajaj6z/status/2054239044884693381',
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await command.func(page, { text: 'with url' });
|
||||
|
||||
expect(result).toEqual([{
|
||||
status: 'success',
|
||||
message: 'Tweet posted successfully.',
|
||||
text: 'with url',
|
||||
id: '2054239044884693381',
|
||||
url: 'https://x.com/darthjajaj6z/status/2054239044884693381',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('returns failed when text area not found', async () => {
|
||||
const command = getCommand();
|
||||
const page = makePage([
|
||||
|
||||
+16
-9
@@ -1,6 +1,6 @@
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { resolveTwitterQueryId } from './shared.js';
|
||||
import { normalizeTwitterScreenName, resolveTwitterQueryId, unwrapBrowserResult } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN } from './utils.js';
|
||||
const USER_BY_SCREEN_NAME_QUERY_ID = 'qRednkZG-rn1P6b48NINmQ';
|
||||
cli({
|
||||
@@ -11,24 +11,31 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (with or without @). Defaults to the logged-in user when omitted.' },
|
||||
],
|
||||
columns: ['screen_name', 'name', 'bio', 'location', 'url', 'followers', 'following', 'tweets', 'likes', 'verified', 'created_at'],
|
||||
func: async (page, kwargs) => {
|
||||
let username = (kwargs.username || '').replace(/^@/, '');
|
||||
// If no username, detect the logged-in user
|
||||
const rawUsername = String(kwargs.username ?? '').trim();
|
||||
let username = normalizeTwitterScreenName(rawUsername);
|
||||
if (rawUsername && !username) {
|
||||
throw new ArgumentError('twitter profile username must be a valid Twitter/X handle', 'Example: opencli twitter profile @jack');
|
||||
}
|
||||
// If no username, detect the logged-in user.
|
||||
// Bridge wraps primitive page.evaluate returns as { session, data:<value> };
|
||||
// unwrap so the href string is usable downstream.
|
||||
if (!username) {
|
||||
await page.goto('https://x.com/home');
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
const href = await page.evaluate(`() => {
|
||||
const href = unwrapBrowserResult(await page.evaluate(`() => {
|
||||
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
|
||||
return link ? link.getAttribute('href') : null;
|
||||
}`);
|
||||
if (!href)
|
||||
}`));
|
||||
if (!href || typeof href !== 'string')
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
username = normalizeTwitterScreenName(href);
|
||||
if (!username)
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
username = href.replace('/', '');
|
||||
}
|
||||
// Navigate directly to the user's profile page (gives us cookie context)
|
||||
await page.goto(`https://x.com/${username}`);
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import './profile.js';
|
||||
|
||||
describe('twitter profile command', () => {
|
||||
it('rejects invalid explicit usernames before navigation', async () => {
|
||||
const command = getRegistry().get('twitter/profile');
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
getCookies: vi.fn(),
|
||||
evaluate: vi.fn(),
|
||||
};
|
||||
|
||||
await expect(command.func(page, { username: 'viewer/extra' })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.getCookies).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects route-like AppTabBar hrefs instead of navigating to that route profile', async () => {
|
||||
const command = getRegistry().get('twitter/profile');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
if (String(script).includes('AppTabBar_Profile_Link')) return '/home';
|
||||
throw new Error(`Unexpected evaluate: ${String(script).slice(0, 80)}`);
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(command.func(page, {})).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
|
||||
expect(page.goto).toHaveBeenCalledTimes(1);
|
||||
expect(page.getCookies).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
+133
-10
@@ -10,6 +10,10 @@ import {
|
||||
resolveImagePath,
|
||||
} from './utils.js';
|
||||
|
||||
const COMPOSER_SELECTOR = '[data-testid="tweetTextarea_0"]';
|
||||
const SUBMIT_POLL_MS = 500;
|
||||
const SUBMIT_TIMEOUT_MS = 15_000;
|
||||
|
||||
function buildReplyComposerUrl(rawUrl) {
|
||||
// Replaces the legacy local extractTweetId which used `/\/status\/(\d+)/`
|
||||
// (silent: matched `/status/1234567` on substring `/status/123` and
|
||||
@@ -19,7 +23,36 @@ function buildReplyComposerUrl(rawUrl) {
|
||||
return `https://x.com/compose/post?in_reply_to=${target.id}`;
|
||||
}
|
||||
|
||||
async function submitReply(page, text) {
|
||||
function isPromiseCollectedError(err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
return msg.includes('Promise was collected');
|
||||
}
|
||||
|
||||
async function openReplyComposer(page, rawUrl) {
|
||||
await page.goto(buildReplyComposerUrl(rawUrl), { waitUntil: 'load', settleMs: 2500 });
|
||||
try {
|
||||
await page.wait({ selector: COMPOSER_SELECTOR, timeout: 15 });
|
||||
return { ok: true };
|
||||
} catch {
|
||||
// X sometimes leaves /compose/post?in_reply_to=<id> on the Home
|
||||
// timeline behind a loading dialog. Fall back to the canonical tweet
|
||||
// page and click the visible Reply action there.
|
||||
await page.goto(rawUrl, { waitUntil: 'load', settleMs: 2500 });
|
||||
const clicked = await page.evaluate(`(() => {
|
||||
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
|
||||
const buttons = Array.from(document.querySelectorAll('[data-testid="reply"]'));
|
||||
const btn = buttons.find((el) => visible(el) && !el.disabled && el.getAttribute('aria-disabled') !== 'true');
|
||||
if (!btn) return { ok: false, message: 'Could not find the reply button on the target tweet.' };
|
||||
btn.click();
|
||||
return { ok: true };
|
||||
})()`);
|
||||
if (!clicked?.ok) return clicked;
|
||||
await page.wait({ selector: COMPOSER_SELECTOR, timeout: 15 });
|
||||
return { ok: true };
|
||||
}
|
||||
}
|
||||
|
||||
async function insertReplyText(page, text) {
|
||||
return page.evaluate(`(async () => {
|
||||
try {
|
||||
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
|
||||
@@ -44,23 +77,109 @@ async function submitReply(page, text) {
|
||||
}
|
||||
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
const normalize = s => String(s || '').replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
|
||||
const actual = box.innerText || box.textContent || '';
|
||||
if (!normalize(actual).includes(normalize(textToInsert))) {
|
||||
return { ok: false, message: 'Could not verify reply text in the composer after typing.', actualText: actual };
|
||||
}
|
||||
return { ok: true };
|
||||
} catch (e) {
|
||||
return { ok: false, message: e.toString() };
|
||||
}
|
||||
})()`);
|
||||
}
|
||||
|
||||
async function clickReplyButton(page) {
|
||||
return page.evaluate(`(() => {
|
||||
try {
|
||||
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
|
||||
const buttons = Array.from(
|
||||
document.querySelectorAll('[data-testid="tweetButton"], [data-testid="tweetButtonInline"]')
|
||||
);
|
||||
const btn = buttons.find((el) => visible(el) && !el.disabled);
|
||||
const btn = buttons.find((el) => visible(el) && !el.disabled && el.getAttribute('aria-disabled') !== 'true');
|
||||
if (!btn) {
|
||||
return { ok: false, message: 'Reply button is disabled or not found.' };
|
||||
}
|
||||
|
||||
btn.click();
|
||||
return { ok: true, message: 'Reply posted successfully.' };
|
||||
return { ok: true };
|
||||
} catch (e) {
|
||||
return { ok: false, message: e.toString() };
|
||||
}
|
||||
})()`);
|
||||
}
|
||||
|
||||
async function detectReplySent(page) {
|
||||
return page.evaluate(`(() => {
|
||||
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
|
||||
const toasts = Array.from(document.querySelectorAll('[role="alert"], [data-testid="toast"]'))
|
||||
.filter((el) => visible(el));
|
||||
const successToast = toasts.find((el) => /sent|posted|your post was sent|your tweet was sent/i.test(el.textContent || ''));
|
||||
if (!successToast) return { ok: false };
|
||||
const link = successToast.querySelector('a[href*="/status/"]');
|
||||
return {
|
||||
ok: true,
|
||||
message: 'Reply posted successfully.',
|
||||
url: link?.href || link?.getAttribute('href') || undefined
|
||||
};
|
||||
})()`);
|
||||
}
|
||||
|
||||
async function waitForReplySent(page, text) {
|
||||
const iterations = Math.ceil(SUBMIT_TIMEOUT_MS / SUBMIT_POLL_MS);
|
||||
try {
|
||||
return await page.evaluate(`(async () => {
|
||||
const expected = ${JSON.stringify(text)};
|
||||
const normalize = s => String(s || '').replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
|
||||
const expectedText = normalize(expected);
|
||||
const visible = (el) => !!el && (el.offsetParent !== null || el.getClientRects().length > 0);
|
||||
for (let i = 0; i < ${JSON.stringify(iterations)}; i++) {
|
||||
await new Promise(r => setTimeout(r, ${JSON.stringify(SUBMIT_POLL_MS)}));
|
||||
const toasts = Array.from(document.querySelectorAll('[role="alert"], [data-testid="toast"]'))
|
||||
.filter((el) => visible(el));
|
||||
const successToast = toasts.find((el) => /sent|posted|your post was sent|your tweet was sent/i.test(el.textContent || ''));
|
||||
if (successToast) {
|
||||
const link = successToast.querySelector('a[href*="/status/"]');
|
||||
return {
|
||||
ok: true,
|
||||
message: 'Reply posted successfully.',
|
||||
url: link?.href || link?.getAttribute('href') || undefined
|
||||
};
|
||||
}
|
||||
const alert = toasts.find((el) => /failed|error|try again|not sent|could not/i.test(el.textContent || ''));
|
||||
if (alert) return { ok: false, message: (alert.textContent || 'Reply failed to post.').trim() };
|
||||
|
||||
const boxes = Array.from(document.querySelectorAll('[data-testid="tweetTextarea_0"]')).filter(visible);
|
||||
const composerStillHasText = boxes.some((box) => normalize(box.innerText || box.textContent || '').includes(expectedText));
|
||||
if (!composerStillHasText) return { ok: true, message: 'Reply posted successfully.' };
|
||||
}
|
||||
return { ok: false, message: 'Reply submission did not complete before timeout.' };
|
||||
})()`);
|
||||
} catch (err) {
|
||||
// X may route the SPA immediately after click, making CDP collect the
|
||||
// polling promise even though the reply was submitted. If the page now
|
||||
// shows the success toast, report success instead of a false negative.
|
||||
if (!isPromiseCollectedError(err)) throw err;
|
||||
await page.wait(2);
|
||||
const recovered = await detectReplySent(page);
|
||||
if (recovered?.ok) return recovered;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async function submitReply(page, text) {
|
||||
const typed = await insertReplyText(page, text);
|
||||
if (!typed?.ok) return typed;
|
||||
let clicked;
|
||||
try {
|
||||
clicked = await clickReplyButton(page);
|
||||
} catch (err) {
|
||||
if (!isPromiseCollectedError(err)) throw err;
|
||||
}
|
||||
if (clicked && !clicked.ok) return clicked;
|
||||
return waitForReplySent(page, text);
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'twitter',
|
||||
name: 'reply',
|
||||
@@ -75,7 +194,7 @@ cli({
|
||||
{ name: 'image', help: 'Optional local image path to attach to the reply' },
|
||||
{ name: 'image-url', help: 'Optional remote image URL to download and attach to the reply' },
|
||||
],
|
||||
columns: ['status', 'message', 'text'],
|
||||
columns: ['status', 'message', 'text', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
if (!page)
|
||||
throw new CommandExecutionError('Browser session required for twitter reply');
|
||||
@@ -92,21 +211,24 @@ cli({
|
||||
localImagePath = downloaded.absPath;
|
||||
cleanupDir = downloaded.cleanupDir;
|
||||
}
|
||||
// Dedicated composer is more reliable than the inline tweet page reply box.
|
||||
await page.goto(buildReplyComposerUrl(kwargs.url), { waitUntil: 'load', settleMs: 2500 });
|
||||
await page.wait({ selector: '[data-testid="tweetTextarea_0"]', timeout: 15 });
|
||||
// Dedicated composer is normally more reliable than the inline
|
||||
// tweet page reply box, but X occasionally leaves that route on the
|
||||
// Home timeline behind a loading dialog. openReplyComposer falls
|
||||
// back to the target tweet's visible Reply action.
|
||||
const composer = await openReplyComposer(page, kwargs.url);
|
||||
if (!composer?.ok) {
|
||||
return [{ status: 'failed', message: composer?.message ?? 'Could not open the reply composer.', text: kwargs.text }];
|
||||
}
|
||||
if (localImagePath) {
|
||||
await page.wait({ selector: COMPOSER_FILE_INPUT_SELECTOR, timeout: 20 });
|
||||
await attachComposerImage(page, localImagePath);
|
||||
}
|
||||
const result = await submitReply(page, kwargs.text);
|
||||
if (result.ok) {
|
||||
await page.wait(3); // Wait for network submission to complete
|
||||
}
|
||||
return [{
|
||||
status: result.ok ? 'success' : 'failed',
|
||||
message: result.message,
|
||||
text: kwargs.text,
|
||||
...(result.url ? { url: result.url } : {}),
|
||||
...(kwargs.image ? { image: kwargs.image } : {}),
|
||||
...(kwargs['image-url'] ? { 'image-url': kwargs['image-url'] } : {}),
|
||||
}];
|
||||
@@ -119,4 +241,5 @@ cli({
|
||||
});
|
||||
export const __test__ = {
|
||||
buildReplyComposerUrl,
|
||||
isPromiseCollectedError,
|
||||
};
|
||||
|
||||
@@ -13,6 +13,8 @@ describe('twitter reply command', () => {
|
||||
const cmd = getRegistry().get('twitter/reply');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
const page = createPageMock([
|
||||
{ ok: true },
|
||||
{ ok: true },
|
||||
{ ok: true, message: 'Reply posted successfully.' },
|
||||
]);
|
||||
const result = await cmd.func(page, {
|
||||
@@ -38,6 +40,8 @@ describe('twitter reply command', () => {
|
||||
const setFileInput = vi.fn().mockResolvedValue(undefined);
|
||||
const page = createPageMock([
|
||||
{ ok: true, previewCount: 1 },
|
||||
{ ok: true },
|
||||
{ ok: true },
|
||||
{ ok: true, message: 'Reply posted successfully.' },
|
||||
], {
|
||||
setFileInput,
|
||||
@@ -74,6 +78,8 @@ describe('twitter reply command', () => {
|
||||
const setFileInput = vi.fn().mockResolvedValue(undefined);
|
||||
const page = createPageMock([
|
||||
{ ok: true, previewCount: 1 },
|
||||
{ ok: true },
|
||||
{ ok: true },
|
||||
{ ok: true, message: 'Reply posted successfully.' },
|
||||
], {
|
||||
setFileInput,
|
||||
@@ -102,6 +108,55 @@ describe('twitter reply command', () => {
|
||||
]);
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
it('falls back to the target tweet page when the dedicated composer route does not expose a textarea', async () => {
|
||||
const cmd = getRegistry().get('twitter/reply');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
const wait = vi.fn()
|
||||
.mockRejectedValueOnce(new Error('Selector not found: [data-testid="tweetTextarea_0"]'))
|
||||
.mockResolvedValue(undefined);
|
||||
const page = createPageMock([
|
||||
{ ok: true }, // click target tweet page Reply button
|
||||
{ ok: true }, // insert reply text
|
||||
{ ok: true }, // click composer Reply button
|
||||
{ ok: true, message: 'Reply posted successfully.' }, // submit completed
|
||||
], { wait });
|
||||
|
||||
const url = 'https://x.com/_kop6/status/2040254679301718161?s=20';
|
||||
const result = await cmd.func(page, { url, text: 'fallback reply' });
|
||||
|
||||
expect(page.goto).toHaveBeenNthCalledWith(1, 'https://x.com/compose/post?in_reply_to=2040254679301718161', { waitUntil: 'load', settleMs: 2500 });
|
||||
expect(page.goto).toHaveBeenNthCalledWith(2, url, { waitUntil: 'load', settleMs: 2500 });
|
||||
expect(page.evaluate.mock.calls[0][0]).toContain('[data-testid="reply"]');
|
||||
expect(wait).toHaveBeenLastCalledWith({ selector: '[data-testid="tweetTextarea_0"]', timeout: 15 });
|
||||
expect(result).toEqual([{ status: 'success', message: 'Reply posted successfully.', text: 'fallback reply' }]);
|
||||
});
|
||||
it('treats an X success toast as success after a Promise was collected error', async () => {
|
||||
const cmd = getRegistry().get('twitter/reply');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce({ ok: true }) // insert reply text
|
||||
.mockResolvedValueOnce({ ok: true }) // click Reply
|
||||
.mockRejectedValueOnce(new Error('{"code":-32000,"message":"Promise was collected"}'))
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
message: 'Reply posted successfully.',
|
||||
url: 'https://x.com/me/status/123',
|
||||
});
|
||||
const page = createPageMock([], { evaluate });
|
||||
|
||||
const result = await cmd.func(page, {
|
||||
url: 'https://x.com/_kop6/status/2040254679301718161?s=20',
|
||||
text: 'toast recovery',
|
||||
});
|
||||
|
||||
expect(page.wait).toHaveBeenCalledWith(2);
|
||||
expect(result).toEqual([{
|
||||
status: 'success',
|
||||
message: 'Reply posted successfully.',
|
||||
text: 'toast recovery',
|
||||
url: 'https://x.com/me/status/123',
|
||||
}]);
|
||||
});
|
||||
it('rejects using --image and --image-url together', async () => {
|
||||
const cmd = getRegistry().get('twitter/reply');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
|
||||
+195
-178
@@ -1,7 +1,7 @@
|
||||
import { ArgumentError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { extractMedia } from './shared.js';
|
||||
import { applyTopByEngagement } from './utils.js';
|
||||
import { extractMedia, normalizeTwitterGraphqlPayload, resolveTwitterOperationMetadata } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
|
||||
// ── Public-search operator surface ─────────────────────────────────────
|
||||
//
|
||||
@@ -35,6 +35,68 @@ const PRODUCT_TO_F_PARAM = Object.freeze({
|
||||
videos: 'video',
|
||||
});
|
||||
|
||||
const PRODUCT_TO_GRAPHQL_PRODUCT = Object.freeze({
|
||||
top: 'Top',
|
||||
live: 'Latest',
|
||||
photos: 'Photos',
|
||||
videos: 'Videos',
|
||||
});
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
|
||||
const SEARCH_TIMELINE_OPERATION = {
|
||||
queryId: 'VhUd6vHVmLBcw0uX-6jMLA',
|
||||
features: {
|
||||
rweb_video_screen_enabled: true,
|
||||
rweb_cashtags_enabled: true,
|
||||
profile_label_improvements_pcf_label_in_post_enabled: true,
|
||||
responsive_web_profile_redirect_enabled: true,
|
||||
rweb_tipjar_consumption_enabled: true,
|
||||
verified_phone_label_enabled: false,
|
||||
creator_subscriptions_tweet_preview_api_enabled: true,
|
||||
responsive_web_graphql_timeline_navigation_enabled: true,
|
||||
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
|
||||
premium_content_api_read_enabled: false,
|
||||
communities_web_enable_tweet_community_results_fetch: true,
|
||||
c9s_tweet_anatomy_moderator_badge_enabled: true,
|
||||
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
|
||||
responsive_web_grok_analyze_post_followups_enabled: true,
|
||||
rweb_cashtags_composer_attachment_enabled: true,
|
||||
responsive_web_jetfuel_frame: true,
|
||||
responsive_web_grok_share_attachment_enabled: true,
|
||||
responsive_web_grok_annotations_enabled: true,
|
||||
articles_preview_enabled: true,
|
||||
responsive_web_edit_tweet_api_enabled: true,
|
||||
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
|
||||
view_counts_everywhere_api_enabled: true,
|
||||
longform_notetweets_consumption_enabled: true,
|
||||
responsive_web_twitter_article_tweet_consumption_enabled: true,
|
||||
content_disclosure_indicator_enabled: true,
|
||||
content_disclosure_ai_generated_indicator_enabled: true,
|
||||
responsive_web_grok_show_grok_translated_post: false,
|
||||
responsive_web_grok_analysis_button_from_backend: true,
|
||||
post_ctas_fetch_enabled: false,
|
||||
freedom_of_speech_not_reach_fetch_enabled: true,
|
||||
standardized_nudges_misinfo: true,
|
||||
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
|
||||
longform_notetweets_rich_text_read_enabled: true,
|
||||
longform_notetweets_inline_media_enabled: true,
|
||||
responsive_web_grok_image_annotation_enabled: true,
|
||||
responsive_web_grok_imagine_annotation_enabled: true,
|
||||
responsive_web_grok_community_note_auto_translation_is_enabled: false,
|
||||
responsive_web_enhance_cards_enabled: false,
|
||||
},
|
||||
fieldToggles: {
|
||||
withPayments: true,
|
||||
withAuxiliaryUserLabels: true,
|
||||
withArticleRichContentState: true,
|
||||
withArticlePlainText: true,
|
||||
withArticleSummaryText: true,
|
||||
withArticleVoiceOver: true,
|
||||
withGrokAnalyze: true,
|
||||
withDisallowedReplyControls: true,
|
||||
},
|
||||
};
|
||||
|
||||
const FROM_USER_PATTERN = /^[A-Za-z0-9_]{1,15}$/;
|
||||
|
||||
const EXCLUDE_TO_OPERATOR = Object.freeze({
|
||||
@@ -99,125 +161,96 @@ function resolveSearchFParam(kwargs) {
|
||||
return kwargs.filter === 'live' ? 'live' : 'top';
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger Twitter search SPA navigation with fallback strategies.
|
||||
*
|
||||
* Primary: pushState + popstate (works in most environments).
|
||||
* Fallback: Type into the search input and press Enter when pushState fails
|
||||
* intermittently (e.g. due to Twitter A/B tests or timing races — see #690).
|
||||
*
|
||||
* Both strategies preserve the JS context so the fetch interceptor stays alive.
|
||||
*
|
||||
* @param {object} page
|
||||
* @param {string} query — final composed query (already merged with operators)
|
||||
* @param {string} fParam — Twitter URL `f=` value (top|live|image|video)
|
||||
*/
|
||||
async function navigateToSearch(page, query, fParam) {
|
||||
const searchUrl = JSON.stringify(`/search?q=${encodeURIComponent(query)}&f=${fParam}`);
|
||||
let lastPath = '';
|
||||
// Strategy 1 (primary): pushState + popstate with retry
|
||||
for (let attempt = 1; attempt <= 2; attempt++) {
|
||||
await page.evaluate(`
|
||||
(() => {
|
||||
window.history.pushState({}, '', ${searchUrl});
|
||||
window.dispatchEvent(new PopStateEvent('popstate', { state: {} }));
|
||||
})()
|
||||
`);
|
||||
try {
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
}
|
||||
catch {
|
||||
// selector timeout — fall through to path check or next attempt
|
||||
}
|
||||
lastPath = String(await page.evaluate('() => window.location.pathname') || '');
|
||||
if (lastPath.startsWith('/search')) {
|
||||
return;
|
||||
}
|
||||
if (attempt < 2) {
|
||||
await page.wait(1);
|
||||
}
|
||||
}
|
||||
// Strategy 2 (fallback): Use the search input on /explore.
|
||||
// The nativeSetter + Enter approach triggers Twitter's own form handler,
|
||||
// performing SPA navigation without a full page reload.
|
||||
const queryStr = JSON.stringify(query);
|
||||
const navResult = await page.evaluate(`(async () => {
|
||||
try {
|
||||
const input = document.querySelector('[data-testid="SearchBox_Search_Input"]');
|
||||
if (!input) return { ok: false };
|
||||
|
||||
input.focus();
|
||||
await new Promise(r => setTimeout(r, 300));
|
||||
|
||||
const nativeSetter = Object.getOwnPropertyDescriptor(
|
||||
window.HTMLInputElement.prototype, 'value'
|
||||
)?.set;
|
||||
if (!nativeSetter) return { ok: false };
|
||||
nativeSetter.call(input, ${queryStr});
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
input.dispatchEvent(new Event('change', { bubbles: true }));
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
|
||||
input.dispatchEvent(new KeyboardEvent('keydown', {
|
||||
key: 'Enter', code: 'Enter', keyCode: 13, bubbles: true
|
||||
}));
|
||||
|
||||
return { ok: true };
|
||||
} catch {
|
||||
return { ok: false };
|
||||
}
|
||||
})()`);
|
||||
if (navResult?.ok) {
|
||||
try {
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
}
|
||||
catch {
|
||||
// fall through to path check
|
||||
}
|
||||
lastPath = String(await page.evaluate('() => window.location.pathname') || '');
|
||||
if (lastPath.startsWith('/search')) {
|
||||
// The fallback path doesn't carry the f= URL param, so click the
|
||||
// matching tab to align with the requested product. Only `live`
|
||||
// currently surfaces a distinct tab label — `image`/`video` tabs
|
||||
// also need an explicit click, so try them all.
|
||||
const tabClicked = await clickProductTabIfNeeded(page, fParam);
|
||||
if (!tabClicked) {
|
||||
throw new CommandExecutionError(`SPA fallback reached /search but could not select the requested product tab: ${fParam}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
}
|
||||
throw new CommandExecutionError(`SPA navigation to /search failed. Final path: ${lastPath || '(empty)'}. Twitter may have changed its routing.`);
|
||||
function resolveSearchProduct(kwargs) {
|
||||
const product = kwargs.product || (kwargs.filter === 'live' ? 'live' : 'top');
|
||||
return PRODUCT_TO_GRAPHQL_PRODUCT[product] || 'Top';
|
||||
}
|
||||
|
||||
/**
|
||||
* After the search-input fallback lands on /search, the f= param is missing
|
||||
* from the URL. Click the matching tab in the result page header so the
|
||||
* SearchTimeline call uses the right filter. No-op for fParam=top (default).
|
||||
*/
|
||||
async function clickProductTabIfNeeded(page, fParam) {
|
||||
if (fParam === 'top') return true;
|
||||
const tabLabels = JSON.stringify({
|
||||
live: ['Latest', '最新'],
|
||||
image: ['Photos', 'Images', '照片', '图片'],
|
||||
video: ['Videos', '视频'],
|
||||
}[fParam] || []);
|
||||
if (tabLabels === '[]') return true;
|
||||
const clicked = await page.evaluate(`(() => {
|
||||
const labels = ${tabLabels};
|
||||
const tabs = document.querySelectorAll('[role="tab"]');
|
||||
for (const tab of tabs) {
|
||||
const txt = (tab.textContent || '').trim();
|
||||
if (labels.some(l => txt.includes(l))) {
|
||||
tab.click();
|
||||
return true;
|
||||
function normalizeOperation(operation) {
|
||||
if (typeof operation === 'string') {
|
||||
return {
|
||||
queryId: operation,
|
||||
features: SEARCH_TIMELINE_OPERATION.features,
|
||||
fieldToggles: SEARCH_TIMELINE_OPERATION.fieldToggles,
|
||||
};
|
||||
}
|
||||
return {
|
||||
queryId: operation?.queryId || SEARCH_TIMELINE_OPERATION.queryId,
|
||||
features: operation?.features || SEARCH_TIMELINE_OPERATION.features,
|
||||
fieldToggles: operation?.fieldToggles || SEARCH_TIMELINE_OPERATION.fieldToggles,
|
||||
};
|
||||
}
|
||||
|
||||
function buildSearchTimelineRequest(operation, rawQuery, product, count, cursor) {
|
||||
const normalized = normalizeOperation(operation);
|
||||
const vars = {
|
||||
rawQuery,
|
||||
count,
|
||||
querySource: 'typed_query',
|
||||
product,
|
||||
};
|
||||
if (cursor) vars.cursor = cursor;
|
||||
return [
|
||||
`/i/api/graphql/${normalized.queryId}/SearchTimeline`,
|
||||
{
|
||||
variables: vars,
|
||||
features: normalized.features,
|
||||
fieldToggles: normalized.fieldToggles,
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
function unwrapTweetResult(result) {
|
||||
if (!result) return null;
|
||||
if (result.__typename === 'TweetWithVisibilityResults' && result.tweet) return result.tweet;
|
||||
if (result.tweet) return result.tweet;
|
||||
return result;
|
||||
}
|
||||
|
||||
function tweetToRow(result, seen) {
|
||||
const tweet = unwrapTweetResult(result);
|
||||
if (!tweet?.rest_id || seen.has(tweet.rest_id)) return null;
|
||||
seen.add(tweet.rest_id);
|
||||
const tweetUser = tweet.core?.user_results?.result;
|
||||
return {
|
||||
id: tweet.rest_id,
|
||||
author: tweetUser?.core?.screen_name || tweetUser?.legacy?.screen_name || 'unknown',
|
||||
text: tweet.note_tweet?.note_tweet_results?.result?.text || tweet.legacy?.full_text || '',
|
||||
created_at: tweet.legacy?.created_at || '',
|
||||
likes: tweet.legacy?.favorite_count || 0,
|
||||
views: tweet.views?.count || '0',
|
||||
url: `https://x.com/i/status/${tweet.rest_id}`,
|
||||
...extractMedia(tweet.legacy),
|
||||
};
|
||||
}
|
||||
|
||||
function parseSearchTimeline(data, seen) {
|
||||
const rows = [];
|
||||
let nextCursor = null;
|
||||
const instructions = data?.data?.search_by_raw_query?.search_timeline?.timeline?.instructions || [];
|
||||
const visit = (value) => {
|
||||
if (!value || typeof value !== 'object') return;
|
||||
if (value.tweet_results?.result) {
|
||||
const row = tweetToRow(value.tweet_results.result, seen);
|
||||
if (row) rows.push(row);
|
||||
}
|
||||
}
|
||||
return false;
|
||||
})()`);
|
||||
if (!clicked) return false;
|
||||
await page.wait(2);
|
||||
return true;
|
||||
if (
|
||||
(value.entryType === 'TimelineTimelineCursor' || value.__typename === 'TimelineTimelineCursor')
|
||||
&& (value.cursorType === 'Bottom' || value.cursorType === 'ShowMore')
|
||||
&& value.value
|
||||
) {
|
||||
nextCursor = value.value;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) visit(item);
|
||||
return;
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
if (child && typeof child === 'object') visit(child);
|
||||
}
|
||||
};
|
||||
visit(instructions);
|
||||
return { rows, nextCursor };
|
||||
}
|
||||
|
||||
cli({
|
||||
@@ -226,9 +259,8 @@ cli({
|
||||
access: 'read',
|
||||
description: 'Search Twitter/X for tweets, with optional --from / --has / --exclude / --product filters mapped to X\'s search operators',
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.INTERCEPT, // Use intercept strategy
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'query', type: 'string', required: true, positional: true, help: 'Search query. Raw X operators (e.g. "exact phrase", #tag, OR, lang:en, since:YYYY-MM-DD, from:, since:) are passed through unchanged.' },
|
||||
{ name: 'filter', type: 'string', default: 'top', choices: ['top', 'live'], help: 'Legacy alias for --product. Kept for backwards compatibility; if --product is set it wins.' },
|
||||
@@ -248,65 +280,47 @@ cli({
|
||||
if (!Number.isInteger(Number(kwargs.limit)) || Number(kwargs.limit) <= 0) {
|
||||
throw new ArgumentError('twitter search --limit must be a positive integer', 'Example: opencli twitter search opencli --limit 15');
|
||||
}
|
||||
const fParam = resolveSearchFParam(kwargs);
|
||||
// 1. Navigate to x.com/explore (has a search input at the top)
|
||||
await page.goto('https://x.com/explore');
|
||||
await page.wait(3);
|
||||
// 2. Install interceptor BEFORE triggering search.
|
||||
// SPA navigation preserves the JS context, so the monkey-patched
|
||||
// fetch will capture the SearchTimeline API call.
|
||||
await page.installInterceptor('SearchTimeline');
|
||||
// 3. Trigger SPA navigation to search results via history API.
|
||||
// pushState + popstate triggers React Router's listener without
|
||||
// a full page reload, so the interceptor stays alive.
|
||||
// Note: the previous approach (nativeSetter + Enter keydown on the
|
||||
// search input) does not reliably trigger Twitter's form submission.
|
||||
await navigateToSearch(page, finalQuery, fParam);
|
||||
// 4. Scroll to trigger additional pagination
|
||||
await page.autoScroll({ times: 3, delayMs: 2000 });
|
||||
// 5. Retrieve captured data
|
||||
const requests = await page.getInterceptedRequests();
|
||||
if (!requests || requests.length === 0)
|
||||
return [];
|
||||
let results = [];
|
||||
const cookies = await page.getCookies({ url: 'https://x.com' });
|
||||
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
|
||||
if (!ct0) throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
|
||||
await page.goto('https://x.com/home', { waitUntil: 'load', settleMs: 1000 });
|
||||
const operation = await resolveTwitterOperationMetadata(page, 'SearchTimeline', SEARCH_TIMELINE_OPERATION);
|
||||
const headers = JSON.stringify({
|
||||
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
|
||||
'X-Csrf-Token': ct0,
|
||||
'X-Twitter-Auth-Type': 'OAuth2Session',
|
||||
'X-Twitter-Active-User': 'yes',
|
||||
'Content-Type': 'application/json',
|
||||
});
|
||||
const product = resolveSearchProduct(kwargs);
|
||||
const results = [];
|
||||
const seen = new Set();
|
||||
for (const req of requests) {
|
||||
try {
|
||||
const insts = req?.data?.search_by_raw_query?.search_timeline?.timeline?.instructions || [];
|
||||
const addEntries = insts.find((i) => i.type === 'TimelineAddEntries')
|
||||
|| insts.find((i) => i.entries && Array.isArray(i.entries));
|
||||
if (!addEntries?.entries)
|
||||
continue;
|
||||
for (const entry of addEntries.entries) {
|
||||
if (!entry.entryId.startsWith('tweet-'))
|
||||
continue;
|
||||
let tweet = entry.content?.itemContent?.tweet_results?.result;
|
||||
if (!tweet)
|
||||
continue;
|
||||
// Handle retweet wrapping
|
||||
if (tweet.__typename === 'TweetWithVisibilityResults' && tweet.tweet) {
|
||||
tweet = tweet.tweet;
|
||||
}
|
||||
if (!tweet.rest_id || seen.has(tweet.rest_id))
|
||||
continue;
|
||||
seen.add(tweet.rest_id);
|
||||
// Twitter moved screen_name from legacy to core
|
||||
const tweetUser = tweet.core?.user_results?.result;
|
||||
results.push({
|
||||
id: tweet.rest_id,
|
||||
author: tweetUser?.core?.screen_name || tweetUser?.legacy?.screen_name || 'unknown',
|
||||
text: tweet.note_tweet?.note_tweet_results?.result?.text || tweet.legacy?.full_text || '',
|
||||
created_at: tweet.legacy?.created_at || '',
|
||||
likes: tweet.legacy?.favorite_count || 0,
|
||||
views: tweet.views?.count || '0',
|
||||
url: `https://x.com/i/status/${tweet.rest_id}`,
|
||||
...extractMedia(tweet.legacy),
|
||||
});
|
||||
}
|
||||
}
|
||||
catch (e) {
|
||||
// ignore parsing errors for individual payloads
|
||||
let cursor = null;
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && results.length < kwargs.limit; i++) {
|
||||
const fetchCount = Number(kwargs.limit) - results.length + 10;
|
||||
const [requestUrl, requestPayload] = buildSearchTimelineRequest(operation, finalQuery, product, fetchCount, cursor);
|
||||
const requestBody = JSON.stringify(requestPayload);
|
||||
const data = normalizeTwitterGraphqlPayload(await page.evaluate(`async () => {
|
||||
const options = {
|
||||
method: 'POST',
|
||||
headers: ${headers},
|
||||
credentials: 'include',
|
||||
};
|
||||
options['body'] = ${JSON.stringify(requestBody)};
|
||||
const r = await fetch(${JSON.stringify(requestUrl)}, {
|
||||
...options,
|
||||
});
|
||||
return r.ok ? await r.json() : { error: r.status };
|
||||
}`));
|
||||
if (data?.error) {
|
||||
if (results.length === 0) throw new CommandExecutionError(`HTTP ${data.error}: SearchTimeline fetch failed — queryId may have expired`);
|
||||
break;
|
||||
}
|
||||
const { rows, nextCursor } = parseSearchTimeline(data, seen);
|
||||
results.push(...rows);
|
||||
if (!nextCursor || nextCursor === cursor) break;
|
||||
cursor = nextCursor;
|
||||
}
|
||||
const trimmed = results.slice(0, kwargs.limit);
|
||||
return applyTopByEngagement(trimmed, kwargs['top-by-engagement']);
|
||||
@@ -316,6 +330,9 @@ cli({
|
||||
export const __test__ = {
|
||||
buildSearchQuery,
|
||||
resolveSearchFParam,
|
||||
resolveSearchProduct,
|
||||
buildSearchTimelineRequest,
|
||||
parseSearchTimeline,
|
||||
HAS_CHOICES,
|
||||
EXCLUDE_CHOICES,
|
||||
PRODUCT_CHOICES,
|
||||
|
||||
+96
-258
@@ -2,71 +2,67 @@ import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { __test__ } from './search.js';
|
||||
|
||||
const { buildSearchQuery, resolveSearchFParam, HAS_CHOICES, EXCLUDE_CHOICES, PRODUCT_CHOICES, EXCLUDE_TO_OPERATOR, PRODUCT_TO_F_PARAM, FROM_USER_PATTERN } = __test__;
|
||||
const { buildSearchQuery, resolveSearchFParam, resolveSearchProduct, buildSearchTimelineRequest, parseSearchTimeline, HAS_CHOICES, EXCLUDE_CHOICES, PRODUCT_CHOICES, EXCLUDE_TO_OPERATOR, PRODUCT_TO_F_PARAM, FROM_USER_PATTERN } = __test__;
|
||||
describe('twitter search command', () => {
|
||||
it('retries transient SPA navigation failures before giving up', async () => {
|
||||
function makeSearchPage(data) {
|
||||
return {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn()
|
||||
.mockResolvedValueOnce(null) // resolveTwitterQueryId fallback
|
||||
.mockResolvedValueOnce(data),
|
||||
};
|
||||
}
|
||||
|
||||
it('fetches SearchTimeline directly instead of relying on SPA navigation', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
expect(command?.func).toBeTypeOf('function');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/explore')
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/search');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([
|
||||
{
|
||||
data: {
|
||||
search_by_raw_query: {
|
||||
search_timeline: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
const page = makeSearchPage({
|
||||
data: {
|
||||
search_by_raw_query: {
|
||||
search_timeline: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
{
|
||||
type: 'TimelineAddEntries',
|
||||
entries: [
|
||||
{
|
||||
type: 'TimelineAddEntries',
|
||||
entries: [
|
||||
{
|
||||
entryId: 'tweet-1',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '1',
|
||||
legacy: {
|
||||
full_text: 'hello world',
|
||||
favorite_count: 7,
|
||||
created_at: 'Thu Mar 26 10:30:00 +0000 2026',
|
||||
},
|
||||
core: {
|
||||
user_results: {
|
||||
result: {
|
||||
core: {
|
||||
screen_name: 'alice',
|
||||
},
|
||||
},
|
||||
entryId: 'tweet-1',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '1',
|
||||
legacy: {
|
||||
full_text: 'hello world',
|
||||
favorite_count: 7,
|
||||
created_at: 'Thu Mar 26 10:30:00 +0000 2026',
|
||||
},
|
||||
core: {
|
||||
user_results: {
|
||||
result: {
|
||||
core: {
|
||||
screen_name: 'alice',
|
||||
},
|
||||
},
|
||||
views: {
|
||||
count: '12',
|
||||
},
|
||||
},
|
||||
},
|
||||
views: {
|
||||
count: '12',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
]),
|
||||
};
|
||||
},
|
||||
});
|
||||
const result = await command.func(page, { query: 'from:alice', filter: 'top', limit: 5 });
|
||||
expect(result).toEqual([
|
||||
{
|
||||
@@ -81,112 +77,62 @@ describe('twitter search command', () => {
|
||||
media_urls: [],
|
||||
},
|
||||
]);
|
||||
expect(page.installInterceptor).toHaveBeenCalledWith('SearchTimeline');
|
||||
expect(evaluate).toHaveBeenCalledTimes(4);
|
||||
expect(page.getCookies).toHaveBeenCalledWith({ url: 'https://x.com' });
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home', { waitUntil: 'load', settleMs: 1000 });
|
||||
const searchFetch = page.evaluate.mock.calls[1][0];
|
||||
expect(searchFetch).toContain('/SearchTimeline');
|
||||
expect(searchFetch).toContain("method: 'POST'");
|
||||
expect(searchFetch).toContain('\\"rawQuery\\":\\"from:alice\\"');
|
||||
});
|
||||
it('uses f=live in search URL when filter is live', async () => {
|
||||
|
||||
it('uses the requested GraphQL product', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/search');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
};
|
||||
await command.func(page, { query: 'breaking news', filter: 'live', limit: 5 });
|
||||
const pushStateCall = evaluate.mock.calls[0][0];
|
||||
expect(pushStateCall).toContain('f=live');
|
||||
expect(pushStateCall).toContain(encodeURIComponent('breaking news'));
|
||||
const page = makeSearchPage({ data: { search_by_raw_query: { search_timeline: { timeline: { instructions: [] } } } } });
|
||||
await command.func(page, { query: 'cats', product: 'videos', limit: 5 });
|
||||
expect(page.evaluate.mock.calls[1][0]).toContain('\\"product\\":\\"Videos\\"');
|
||||
});
|
||||
it('uses f=top in search URL when filter is top', async () => {
|
||||
|
||||
it('paginates past the old five-page cap until the requested limit is reached', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/search');
|
||||
let pageIndex = 0;
|
||||
const page = {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
};
|
||||
await command.func(page, { query: 'test', filter: 'top', limit: 5 });
|
||||
const pushStateCall = evaluate.mock.calls[0][0];
|
||||
expect(pushStateCall).toContain('f=top');
|
||||
});
|
||||
it('falls back to top when filter is omitted', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/search');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
};
|
||||
await command.func(page, { query: 'test', limit: 5 });
|
||||
const pushStateCall = evaluate.mock.calls[0][0];
|
||||
expect(pushStateCall).toContain('f=top');
|
||||
});
|
||||
it('falls back to search input when pushState fails twice', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
expect(command?.func).toBeTypeOf('function');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 1
|
||||
.mockResolvedValueOnce('/explore') // pathname check 1 — not /search
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 2
|
||||
.mockResolvedValueOnce('/explore') // pathname check 2 — still not /search
|
||||
.mockResolvedValueOnce({ ok: true }) // search input fallback succeeds
|
||||
.mockResolvedValueOnce('/search'); // pathname check after fallback
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([
|
||||
{
|
||||
evaluate: vi.fn().mockImplementation(async () => {
|
||||
if (pageIndex === 0) {
|
||||
pageIndex += 1;
|
||||
return null;
|
||||
}
|
||||
const id = String(pageIndex);
|
||||
pageIndex += 1;
|
||||
return {
|
||||
data: {
|
||||
search_by_raw_query: {
|
||||
search_timeline: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
{
|
||||
type: 'TimelineAddEntries',
|
||||
entries: [
|
||||
{
|
||||
entryId: 'tweet-99',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '99',
|
||||
legacy: {
|
||||
full_text: 'fallback works',
|
||||
favorite_count: 3,
|
||||
created_at: 'Wed Apr 02 12:00:00 +0000 2026',
|
||||
},
|
||||
core: {
|
||||
user_results: {
|
||||
result: {
|
||||
core: { screen_name: 'bob' },
|
||||
},
|
||||
},
|
||||
},
|
||||
views: { count: '5' },
|
||||
rest_id: id,
|
||||
legacy: { full_text: `tweet ${id}`, created_at: 'now' },
|
||||
core: { user_results: { result: { core: { screen_name: 'alice' } } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
content: {
|
||||
entryType: 'TimelineTimelineCursor',
|
||||
cursorType: 'Bottom',
|
||||
value: `cursor-${id}`,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
@@ -194,100 +140,13 @@ describe('twitter search command', () => {
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
]),
|
||||
};
|
||||
}),
|
||||
};
|
||||
const result = await command.func(page, { query: 'test fallback', filter: 'top', limit: 5 });
|
||||
expect(result).toEqual([
|
||||
{
|
||||
id: '99',
|
||||
author: 'bob',
|
||||
text: 'fallback works',
|
||||
created_at: 'Wed Apr 02 12:00:00 +0000 2026',
|
||||
likes: 3,
|
||||
views: '5',
|
||||
url: 'https://x.com/i/status/99',
|
||||
has_media: false,
|
||||
media_urls: [],
|
||||
},
|
||||
]);
|
||||
// 6 evaluate calls: 2x pushState + 2x pathname check + 1x fallback + 1x pathname check
|
||||
expect(evaluate).toHaveBeenCalledTimes(6);
|
||||
expect(page.autoScroll).toHaveBeenCalled();
|
||||
});
|
||||
it('clicks the requested product tab after fallback navigation when f= param is absent', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
expect(command?.func).toBeTypeOf('function');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 1
|
||||
.mockResolvedValueOnce('/explore')
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 2
|
||||
.mockResolvedValueOnce('/explore')
|
||||
.mockResolvedValueOnce({ ok: true }) // search input fallback
|
||||
.mockResolvedValueOnce('/search')
|
||||
.mockResolvedValueOnce(true); // product tab click
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
};
|
||||
const result = await command.func(page, { query: 'cats', product: 'photos', limit: 5 });
|
||||
expect(result).toEqual([]);
|
||||
expect(evaluate).toHaveBeenCalledTimes(7);
|
||||
expect(evaluate.mock.calls[6][0]).toContain('Photos');
|
||||
expect(page.autoScroll).toHaveBeenCalled();
|
||||
});
|
||||
it('throws when fallback navigation cannot select the requested product tab', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
expect(command?.func).toBeTypeOf('function');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 1
|
||||
.mockResolvedValueOnce('/explore')
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 2
|
||||
.mockResolvedValueOnce('/explore')
|
||||
.mockResolvedValueOnce({ ok: true }) // search input fallback
|
||||
.mockResolvedValueOnce('/search')
|
||||
.mockResolvedValueOnce(false); // requested tab missing
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn(),
|
||||
};
|
||||
await expect(command.func(page, { query: 'cats', product: 'videos', limit: 5 }))
|
||||
.rejects
|
||||
.toThrow(/could not select the requested product tab: video/);
|
||||
expect(page.autoScroll).not.toHaveBeenCalled();
|
||||
expect(page.getInterceptedRequests).not.toHaveBeenCalled();
|
||||
});
|
||||
it('throws with the final path after both attempts fail', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
expect(command?.func).toBeTypeOf('function');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 1
|
||||
.mockResolvedValueOnce('/explore') // pathname check 1
|
||||
.mockResolvedValueOnce(undefined) // pushState attempt 2
|
||||
.mockResolvedValueOnce('/login') // pathname check 2
|
||||
.mockResolvedValueOnce({ ok: false }); // search input fallback
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn(),
|
||||
};
|
||||
await expect(command.func(page, { query: 'from:alice', filter: 'top', limit: 5 }))
|
||||
.rejects
|
||||
.toThrow('Final path: /login');
|
||||
expect(page.autoScroll).not.toHaveBeenCalled();
|
||||
expect(page.getInterceptedRequests).not.toHaveBeenCalled();
|
||||
expect(evaluate).toHaveBeenCalledTimes(5);
|
||||
const result = await command.func(page, { query: 'opencli', limit: 7 });
|
||||
expect(result).toHaveLength(7);
|
||||
expect(result.map((row) => row.id)).toEqual(['1', '2', '3', '4', '5', '6', '7']);
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(8);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -411,18 +270,15 @@ describe('twitter search filter helpers', () => {
|
||||
});
|
||||
|
||||
describe('twitter search end-to-end with new filters', () => {
|
||||
it('encodes the composed query and product=live into the f= URL param', async () => {
|
||||
it('encodes the composed query and product=live into the GraphQL request', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/search');
|
||||
.mockResolvedValueOnce(null)
|
||||
.mockResolvedValueOnce({ data: { search_by_raw_query: { search_timeline: { timeline: { instructions: [] } } } } });
|
||||
const page = {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
};
|
||||
await command.func(page, {
|
||||
query: 'breaking news',
|
||||
@@ -432,37 +288,26 @@ describe('twitter search end-to-end with new filters', () => {
|
||||
product: 'live',
|
||||
limit: 5,
|
||||
});
|
||||
const pushStateCall = evaluate.mock.calls[0][0];
|
||||
// f=live wins because --product=live trumps the default --filter
|
||||
expect(pushStateCall).toContain('f=live');
|
||||
// composed query should be percent-encoded inside the URL
|
||||
const encoded = encodeURIComponent('breaking news from:alice filter:images -filter:nativeretweets');
|
||||
expect(pushStateCall).toContain(encoded);
|
||||
const searchFetch = evaluate.mock.calls[1][0];
|
||||
expect(searchFetch).toContain('\\"product\\":\\"Latest\\"');
|
||||
expect(searchFetch).toContain('\\"rawQuery\\":\\"breaking news from:alice filter:images -filter:nativeretweets\\"');
|
||||
});
|
||||
it('throws ArgumentError when query and all filters are empty', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn(),
|
||||
};
|
||||
await expect(command.func(page, { query: ' ', limit: 5 }))
|
||||
.rejects
|
||||
.toThrow(/empty/i);
|
||||
expect(page.installInterceptor).not.toHaveBeenCalled();
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
it('throws ArgumentError for invalid --from before navigation', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
installInterceptor: vi.fn(),
|
||||
evaluate: vi.fn(),
|
||||
autoScroll: vi.fn(),
|
||||
getInterceptedRequests: vi.fn(),
|
||||
};
|
||||
await expect(command.func(page, { query: 'hi', from: 'alice filter:links', limit: 5 }))
|
||||
.rejects
|
||||
@@ -473,11 +318,7 @@ describe('twitter search end-to-end with new filters', () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
installInterceptor: vi.fn(),
|
||||
evaluate: vi.fn(),
|
||||
autoScroll: vi.fn(),
|
||||
getInterceptedRequests: vi.fn(),
|
||||
};
|
||||
await expect(command.func(page, { query: 'hi', limit: 0 }))
|
||||
.rejects
|
||||
@@ -487,19 +328,16 @@ describe('twitter search end-to-end with new filters', () => {
|
||||
it('runs with only filters set (empty <query>)', async () => {
|
||||
const command = getRegistry().get('twitter/search');
|
||||
const evaluate = vi.fn()
|
||||
.mockResolvedValueOnce(undefined)
|
||||
.mockResolvedValueOnce('/search');
|
||||
.mockResolvedValueOnce(null)
|
||||
.mockResolvedValueOnce({ data: { search_by_raw_query: { search_timeline: { timeline: { instructions: [] } } } } });
|
||||
const page = {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'ct0', value: 'csrf' }]),
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
};
|
||||
const result = await command.func(page, { query: '', from: 'alice', limit: 5 });
|
||||
expect(result).toEqual([]);
|
||||
const pushStateCall = evaluate.mock.calls[0][0];
|
||||
expect(pushStateCall).toContain(encodeURIComponent('from:alice'));
|
||||
const searchFetch = evaluate.mock.calls[1][0];
|
||||
expect(searchFetch).toContain('\\"rawQuery\\":\\"from:alice\\"');
|
||||
});
|
||||
});
|
||||
|
||||
+167
-16
@@ -1,8 +1,28 @@
|
||||
import { ArgumentError } from '@jackwener/opencli/errors';
|
||||
|
||||
const QUERY_ID_PATTERN = /^[A-Za-z0-9_-]+$/;
|
||||
const SCREEN_NAME_PATTERN = /^[A-Za-z0-9_]{1,15}$/;
|
||||
const TWEET_PATH_PATTERN = /^\/(?:[^/]+|i)\/status\/(\d+)\/?$/;
|
||||
const TWEET_HOSTS = new Set(['x.com', 'twitter.com']);
|
||||
const SCREEN_NAME_HOSTS = new Set(['x.com', 'twitter.com', 'mobile.twitter.com']);
|
||||
const RESERVED_SCREEN_NAME_PATHS = new Set([
|
||||
'compose',
|
||||
'explore',
|
||||
'help',
|
||||
'home',
|
||||
'i',
|
||||
'intent',
|
||||
'jobs',
|
||||
'login',
|
||||
'logout',
|
||||
'messages',
|
||||
'notifications',
|
||||
'privacy',
|
||||
'search',
|
||||
'settings',
|
||||
'signup',
|
||||
'tos',
|
||||
]);
|
||||
|
||||
function isTwitterHost(hostname) {
|
||||
return TWEET_HOSTS.has(hostname)
|
||||
@@ -81,9 +101,138 @@ export function buildTwitterArticleScopeSource(tweetId) {
|
||||
export function sanitizeQueryId(resolved, fallbackId) {
|
||||
return typeof resolved === 'string' && QUERY_ID_PATTERN.test(resolved) ? resolved : fallbackId;
|
||||
}
|
||||
export async function resolveTwitterQueryId(page, operationName, fallbackId) {
|
||||
|
||||
export function normalizeTwitterScreenName(value) {
|
||||
const raw = String(value ?? '').trim();
|
||||
if (!raw) return '';
|
||||
let candidate = '';
|
||||
try {
|
||||
const url = raw.startsWith('/') ? new URL(raw, 'https://x.com') : new URL(raw);
|
||||
if (
|
||||
url.protocol !== 'https:' ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
url.port ||
|
||||
!SCREEN_NAME_HOSTS.has(url.hostname)
|
||||
) {
|
||||
return '';
|
||||
}
|
||||
const segments = url.pathname.split('/').filter(Boolean);
|
||||
if (segments.length !== 1) return '';
|
||||
candidate = segments[0];
|
||||
} catch {
|
||||
if (raw.includes('/') || raw.includes('?') || raw.includes('#')) return '';
|
||||
candidate = raw.replace(/^@+/, '');
|
||||
}
|
||||
if (!SCREEN_NAME_PATTERN.test(candidate)) return '';
|
||||
if (RESERVED_SCREEN_NAME_PATHS.has(candidate.toLowerCase())) return '';
|
||||
return candidate;
|
||||
}
|
||||
|
||||
function keysToFlags(keys) {
|
||||
if (!Array.isArray(keys)) return {};
|
||||
return Object.fromEntries(keys.filter((key) => typeof key === 'string' && key).map((key) => [key, true]));
|
||||
}
|
||||
|
||||
function normalizeOperationFallback(fallback) {
|
||||
if (typeof fallback === 'string') return { queryId: fallback, features: {}, fieldToggles: {} };
|
||||
return {
|
||||
queryId: fallback?.queryId || null,
|
||||
features: fallback?.features || {},
|
||||
fieldToggles: fallback?.fieldToggles || {},
|
||||
};
|
||||
}
|
||||
|
||||
export function unwrapBrowserResult(value) {
|
||||
if (
|
||||
value
|
||||
&& typeof value === 'object'
|
||||
&& typeof value.session === 'string'
|
||||
&& Object.prototype.hasOwnProperty.call(value, 'data')
|
||||
) {
|
||||
return value.data;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function normalizeTwitterGraphqlPayload(value) {
|
||||
const unwrapped = unwrapBrowserResult(value);
|
||||
if (unwrapped?.data && typeof unwrapped.data === 'object') return unwrapped;
|
||||
if (
|
||||
unwrapped
|
||||
&& typeof unwrapped === 'object'
|
||||
&& (
|
||||
Object.prototype.hasOwnProperty.call(unwrapped, 'user')
|
||||
|| Object.prototype.hasOwnProperty.call(unwrapped, 'search_by_raw_query')
|
||||
)
|
||||
) {
|
||||
return { data: unwrapped };
|
||||
}
|
||||
return unwrapped;
|
||||
}
|
||||
|
||||
export function sanitizeTwitterOperationMetadata(resolved, fallback) {
|
||||
const value = unwrapBrowserResult(resolved);
|
||||
const normalizedFallback = normalizeOperationFallback(fallback);
|
||||
// Empty resolved features / fieldToggles must defer to the baked fallback.
|
||||
// The bundle parser can find a queryId but miss `featureSwitches:[...]` (e.g.
|
||||
// a minification change, or the 2500-char snippet window truncating before
|
||||
// the array). When that happens, keysToFlags(undefined) returns {}; if we
|
||||
// kept it, Twitter would receive an empty `features` map and respond 400,
|
||||
// surfacing a misleading "queryId expired" error.
|
||||
return {
|
||||
queryId: sanitizeQueryId(value?.queryId, normalizedFallback.queryId),
|
||||
features: value?.features
|
||||
&& typeof value.features === 'object'
|
||||
&& Object.keys(value.features).length > 0
|
||||
? value.features
|
||||
: normalizedFallback.features,
|
||||
fieldToggles: value?.fieldToggles
|
||||
&& typeof value.fieldToggles === 'object'
|
||||
&& Object.keys(value.fieldToggles).length > 0
|
||||
? value.fieldToggles
|
||||
: normalizedFallback.fieldToggles,
|
||||
};
|
||||
}
|
||||
|
||||
export async function resolveTwitterOperationMetadata(page, operationName, fallback) {
|
||||
const resolved = await page.evaluate(`async () => {
|
||||
const operationName = ${JSON.stringify(operationName)};
|
||||
const keysToFlags = (keys) => Object.fromEntries((keys || []).map((key) => [key, true]));
|
||||
const quotedKeys = (source) => source
|
||||
? Array.from(source.matchAll(/"([^"]+)"/g)).map((match) => match[1])
|
||||
: [];
|
||||
const parseOperation = (text) => {
|
||||
const marker = 'operationName:"' + operationName + '"';
|
||||
const index = text.indexOf(marker);
|
||||
if (index < 0) return null;
|
||||
const start = Math.max(0, text.lastIndexOf('e.exports=', index));
|
||||
const endMarker = text.indexOf('}}}', index);
|
||||
const snippet = text.slice(start, endMarker > index ? endMarker + 3 : index + 2500);
|
||||
const queryId = snippet.match(/queryId:"([A-Za-z0-9_-]+)"/)?.[1] || null;
|
||||
if (!queryId) return null;
|
||||
return {
|
||||
queryId,
|
||||
features: keysToFlags(quotedKeys(snippet.match(/featureSwitches:\\[([^\\]]*)\\]/)?.[1])),
|
||||
fieldToggles: keysToFlags(quotedKeys(snippet.match(/fieldToggles:\\[([^\\]]*)\\]/)?.[1])),
|
||||
};
|
||||
};
|
||||
try {
|
||||
const scripts = Array.from(document.scripts)
|
||||
.map(s => s.src)
|
||||
.filter(Boolean)
|
||||
.concat(performance.getEntriesByType('resource')
|
||||
.map(r => r.name)
|
||||
.filter(r => r.includes('client-web') && r.endsWith('.js')));
|
||||
const uniqueScripts = Array.from(new Set(scripts));
|
||||
for (const scriptUrl of uniqueScripts.slice(-30)) {
|
||||
try {
|
||||
const text = await (await fetch(scriptUrl)).text();
|
||||
const operation = parseOperation(text);
|
||||
if (operation) return operation;
|
||||
} catch {}
|
||||
}
|
||||
} catch {}
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 5000);
|
||||
try {
|
||||
@@ -92,27 +241,25 @@ export async function resolveTwitterQueryId(page, operationName, fallbackId) {
|
||||
if (ghResp.ok) {
|
||||
const data = await ghResp.json();
|
||||
const entry = data?.[operationName];
|
||||
if (entry && entry.queryId) return entry.queryId;
|
||||
if (entry && entry.queryId) {
|
||||
return {
|
||||
queryId: entry.queryId,
|
||||
features: keysToFlags(entry.featureSwitches),
|
||||
fieldToggles: keysToFlags(entry.fieldToggles),
|
||||
};
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
try {
|
||||
const scripts = performance.getEntriesByType('resource')
|
||||
.filter(r => r.name.includes('client-web') && r.name.endsWith('.js'))
|
||||
.map(r => r.name);
|
||||
for (const scriptUrl of scripts.slice(0, 15)) {
|
||||
try {
|
||||
const text = await (await fetch(scriptUrl)).text();
|
||||
const re = new RegExp('queryId:"([A-Za-z0-9_-]+)"[^}]{0,200}operationName:"' + operationName + '"');
|
||||
const match = text.match(re);
|
||||
if (match) return match[1];
|
||||
} catch {}
|
||||
}
|
||||
} catch {}
|
||||
return null;
|
||||
}`);
|
||||
return sanitizeQueryId(resolved, fallbackId);
|
||||
return sanitizeTwitterOperationMetadata(resolved, fallback);
|
||||
}
|
||||
|
||||
export async function resolveTwitterQueryId(page, operationName, fallbackId) {
|
||||
const operation = await resolveTwitterOperationMetadata(page, operationName, fallbackId);
|
||||
return operation.queryId;
|
||||
}
|
||||
/**
|
||||
* Extract media flags and URLs from a tweet's `legacy` object.
|
||||
@@ -143,6 +290,10 @@ export function extractMedia(legacy) {
|
||||
}
|
||||
export const __test__ = {
|
||||
sanitizeQueryId,
|
||||
sanitizeTwitterOperationMetadata,
|
||||
unwrapBrowserResult,
|
||||
normalizeTwitterGraphqlPayload,
|
||||
normalizeTwitterScreenName,
|
||||
extractMedia,
|
||||
parseTweetUrl,
|
||||
buildTwitterArticleScopeSource,
|
||||
|
||||
+102
-1
@@ -3,7 +3,108 @@ import { JSDOM } from 'jsdom';
|
||||
import { __test__ } from './shared.js';
|
||||
import { ArgumentError } from '@jackwener/opencli/errors';
|
||||
|
||||
const { extractMedia, parseTweetUrl, buildTwitterArticleScopeSource } = __test__;
|
||||
const { extractMedia, parseTweetUrl, buildTwitterArticleScopeSource, unwrapBrowserResult, normalizeTwitterGraphqlPayload, normalizeTwitterScreenName, sanitizeTwitterOperationMetadata } = __test__;
|
||||
|
||||
describe('twitter browser result helpers', () => {
|
||||
it('unwraps Browser Bridge exec envelopes', () => {
|
||||
expect(unwrapBrowserResult({ session: 'site:twitter', data: '123' })).toBe('123');
|
||||
expect(unwrapBrowserResult({ data: { user: true } })).toEqual({ data: { user: true } });
|
||||
});
|
||||
|
||||
it('sanitizes operation metadata after unwrapping Browser Bridge envelopes', () => {
|
||||
const result = sanitizeTwitterOperationMetadata({
|
||||
session: 'site:twitter',
|
||||
data: {
|
||||
queryId: 'abc_123',
|
||||
features: { feature: true },
|
||||
fieldToggles: { field: true },
|
||||
},
|
||||
}, { queryId: 'fallback', features: {}, fieldToggles: {} });
|
||||
expect(result).toEqual({
|
||||
queryId: 'abc_123',
|
||||
features: { feature: true },
|
||||
fieldToggles: { field: true },
|
||||
});
|
||||
});
|
||||
|
||||
it('falls back to baked features / fieldToggles when the bundle parser returns empty maps', () => {
|
||||
// Regression guard: resolveTwitterOperationMetadata's bundle parser can
|
||||
// find a queryId but miss `featureSwitches:[...]` (e.g. minification
|
||||
// change, or the 2500-char snippet window truncating before the array).
|
||||
// In that case keysToFlags(undefined) returns {}; if sanitize kept the
|
||||
// empty map, Twitter would receive a request with no features and reply
|
||||
// 400, surfacing a misleading "queryId expired" error.
|
||||
const result = sanitizeTwitterOperationMetadata({
|
||||
queryId: 'newQueryId',
|
||||
features: {},
|
||||
fieldToggles: {},
|
||||
}, {
|
||||
queryId: 'fallback',
|
||||
features: { fallback_feature: true },
|
||||
fieldToggles: { fallback_field: true },
|
||||
});
|
||||
expect(result).toEqual({
|
||||
queryId: 'newQueryId',
|
||||
features: { fallback_feature: true },
|
||||
fieldToggles: { fallback_field: true },
|
||||
});
|
||||
});
|
||||
|
||||
it('falls back when resolved features are non-object falsy values', () => {
|
||||
const result = sanitizeTwitterOperationMetadata({
|
||||
queryId: 'newQueryId',
|
||||
features: null,
|
||||
fieldToggles: undefined,
|
||||
}, {
|
||||
queryId: 'fallback',
|
||||
features: { fallback_feature: true },
|
||||
fieldToggles: { fallback_field: true },
|
||||
});
|
||||
expect(result.features).toEqual({ fallback_feature: true });
|
||||
expect(result.fieldToggles).toEqual({ fallback_field: true });
|
||||
});
|
||||
|
||||
it('normalizes GraphQL payloads when the bridge strips the top-level data key', () => {
|
||||
expect(normalizeTwitterGraphqlPayload({ user: { result: {} } })).toEqual({
|
||||
data: { user: { result: {} } },
|
||||
});
|
||||
expect(normalizeTwitterGraphqlPayload({ search_by_raw_query: { search_timeline: {} } })).toEqual({
|
||||
data: { search_by_raw_query: { search_timeline: {} } },
|
||||
});
|
||||
expect(normalizeTwitterGraphqlPayload({ data: { user: {} } })).toEqual({ data: { user: {} } });
|
||||
});
|
||||
});
|
||||
|
||||
describe('twitter normalizeTwitterScreenName', () => {
|
||||
it('accepts exact handles and exact Twitter/X profile URLs', () => {
|
||||
expect(normalizeTwitterScreenName('@viewer')).toBe('viewer');
|
||||
expect(normalizeTwitterScreenName('/viewer')).toBe('viewer');
|
||||
expect(normalizeTwitterScreenName('https://x.com/viewer')).toBe('viewer');
|
||||
expect(normalizeTwitterScreenName('https://twitter.com/viewer?lang=en')).toBe('viewer');
|
||||
expect(normalizeTwitterScreenName('https://mobile.twitter.com/viewer')).toBe('viewer');
|
||||
});
|
||||
|
||||
it('rejects route collisions, malformed handles, and non-exact profile URLs', () => {
|
||||
const invalid = [
|
||||
'/home',
|
||||
'/viewer/extra',
|
||||
'viewer/extra',
|
||||
'viewer?tab=posts',
|
||||
'https://x.com/home',
|
||||
'https://x.com/viewer/status/1',
|
||||
'http://x.com/viewer',
|
||||
'https://evil.com/viewer',
|
||||
'https://x.com.evil.com/viewer',
|
||||
'https://x.com:444/viewer',
|
||||
'https://user:pass@x.com/viewer',
|
||||
'bad-handle',
|
||||
'abcdefghijklmnop',
|
||||
];
|
||||
for (const value of invalid) {
|
||||
expect(normalizeTwitterScreenName(value)).toBe('');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('twitter parseTweetUrl', () => {
|
||||
it('accepts exact Twitter/X tweet URLs and preserves query parameters', () => {
|
||||
|
||||
@@ -100,7 +100,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'tweet-id', positional: true, type: 'string', required: true, help: 'Tweet numeric ID (e.g. 1234567890) or full status URL' },
|
||||
{ name: 'limit', type: 'int', default: 50 },
|
||||
|
||||
@@ -5,6 +5,7 @@ import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
// ── Twitter GraphQL constants ──────────────────────────────────────────
|
||||
const HOME_TIMELINE_QUERY_ID = 'c-CzHF1LboFilMpsx4ZCrQ';
|
||||
const HOME_LATEST_TIMELINE_QUERY_ID = 'BKB7oi212Fi7kQtCBGE4zA';
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
// Endpoint config: for-you uses GET HomeTimeline, following uses POST HomeLatestTimeline
|
||||
const TIMELINE_ENDPOINTS = {
|
||||
'for-you': { endpoint: 'HomeTimeline', method: 'GET', fallbackQueryId: HOME_TIMELINE_QUERY_ID },
|
||||
@@ -141,7 +142,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{
|
||||
name: 'type',
|
||||
@@ -176,7 +176,8 @@ cli({
|
||||
const allTweets = [];
|
||||
const seen = new Set();
|
||||
let cursor = null;
|
||||
for (let i = 0; i < 5 && allTweets.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && allTweets.length < limit; i++) {
|
||||
const fetchCount = Math.min(40, limit - allTweets.length + 5); // over-fetch slightly for promoted filtering
|
||||
const variables = buildTimelineVariables(timelineType, fetchCount, cursor);
|
||||
const apiUrl = buildHomeTimelineUrl(queryId, endpoint, variables);
|
||||
|
||||
@@ -17,7 +17,6 @@ cli({
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Number of trends to show' },
|
||||
],
|
||||
|
||||
+147
-52
@@ -1,15 +1,19 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { resolveTwitterQueryId, sanitizeQueryId, extractMedia } from './shared.js';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { resolveTwitterOperationMetadata, sanitizeQueryId, extractMedia, normalizeTwitterGraphqlPayload, unwrapBrowserResult } from './shared.js';
|
||||
import { normalizeTwitterScreenName } from './shared.js';
|
||||
import { TWITTER_BEARER_TOKEN, applyTopByEngagement } from './utils.js';
|
||||
|
||||
const USER_TWEETS_QUERY_ID = '6fWQaBPK51aGyC_VC7t9GQ';
|
||||
const USER_TWEETS_QUERY_ID = 'lrMzG9qPQHpqJdP3AbM-bQ';
|
||||
const USER_BY_SCREEN_NAME_QUERY_ID = 'IGgvgiOx4QZndDHuD3x9TQ';
|
||||
const MAX_PAGINATION_PAGES = 100;
|
||||
|
||||
const USER_TWEETS_FEATURES = {
|
||||
rweb_video_screen_enabled: false,
|
||||
rweb_video_screen_enabled: true,
|
||||
rweb_cashtags_enabled: true,
|
||||
payments_enabled: false,
|
||||
profile_label_improvements_pcf_label_in_post_enabled: true,
|
||||
responsive_web_profile_redirect_enabled: true,
|
||||
rweb_tipjar_consumption_enabled: true,
|
||||
verified_phone_label_enabled: false,
|
||||
creator_subscriptions_tweet_preview_api_enabled: true,
|
||||
@@ -20,6 +24,7 @@ const USER_TWEETS_FEATURES = {
|
||||
c9s_tweet_anatomy_moderator_badge_enabled: true,
|
||||
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
|
||||
responsive_web_grok_analyze_post_followups_enabled: true,
|
||||
rweb_cashtags_composer_attachment_enabled: true,
|
||||
responsive_web_jetfuel_frame: true,
|
||||
responsive_web_grok_share_attachment_enabled: true,
|
||||
responsive_web_grok_annotations_enabled: true,
|
||||
@@ -46,8 +51,21 @@ const USER_TWEETS_FEATURES = {
|
||||
responsive_web_enhance_cards_enabled: false,
|
||||
};
|
||||
|
||||
const USER_TWEETS_FIELD_TOGGLES = {
|
||||
withPayments: true,
|
||||
withAuxiliaryUserLabels: true,
|
||||
withArticleRichContentState: true,
|
||||
withArticlePlainText: true,
|
||||
withArticleSummaryText: true,
|
||||
withArticleVoiceOver: true,
|
||||
withGrokAnalyze: true,
|
||||
withDisallowedReplyControls: true,
|
||||
};
|
||||
|
||||
const USER_BY_SCREEN_NAME_FEATURES = {
|
||||
hidden_profile_subscriptions_enabled: true,
|
||||
profile_label_improvements_pcf_label_in_post_enabled: true,
|
||||
responsive_web_profile_redirect_enabled: true,
|
||||
rweb_tipjar_consumption_enabled: true,
|
||||
responsive_web_graphql_exclude_directive_enabled: true,
|
||||
verified_phone_label_enabled: false,
|
||||
@@ -61,7 +79,59 @@ const USER_BY_SCREEN_NAME_FEATURES = {
|
||||
responsive_web_graphql_timeline_navigation_enabled: true,
|
||||
};
|
||||
|
||||
function buildUserTweetsUrl(queryId, userId, count, cursor) {
|
||||
const USER_BY_SCREEN_NAME_FIELD_TOGGLES = {
|
||||
withPayments: true,
|
||||
withAuxiliaryUserLabels: true,
|
||||
};
|
||||
|
||||
const USER_TWEETS_OPERATION = {
|
||||
queryId: USER_TWEETS_QUERY_ID,
|
||||
features: USER_TWEETS_FEATURES,
|
||||
fieldToggles: USER_TWEETS_FIELD_TOGGLES,
|
||||
};
|
||||
|
||||
const USER_BY_SCREEN_NAME_OPERATION = {
|
||||
queryId: USER_BY_SCREEN_NAME_QUERY_ID,
|
||||
features: USER_BY_SCREEN_NAME_FEATURES,
|
||||
fieldToggles: USER_BY_SCREEN_NAME_FIELD_TOGGLES,
|
||||
};
|
||||
|
||||
function normalizeUserTweetsOperation(operation) {
|
||||
if (typeof operation === 'string') {
|
||||
return { queryId: operation, features: USER_TWEETS_FEATURES, fieldToggles: USER_TWEETS_FIELD_TOGGLES };
|
||||
}
|
||||
return {
|
||||
queryId: operation?.queryId || USER_TWEETS_QUERY_ID,
|
||||
features: operation?.features || USER_TWEETS_FEATURES,
|
||||
fieldToggles: operation?.fieldToggles || USER_TWEETS_FIELD_TOGGLES,
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeUserByScreenNameOperation(operation) {
|
||||
if (typeof operation === 'string') {
|
||||
return { queryId: operation, features: USER_BY_SCREEN_NAME_FEATURES, fieldToggles: USER_BY_SCREEN_NAME_FIELD_TOGGLES };
|
||||
}
|
||||
return {
|
||||
queryId: operation?.queryId || USER_BY_SCREEN_NAME_QUERY_ID,
|
||||
features: operation?.features || USER_BY_SCREEN_NAME_FEATURES,
|
||||
fieldToggles: operation?.fieldToggles || USER_BY_SCREEN_NAME_FIELD_TOGGLES,
|
||||
};
|
||||
}
|
||||
|
||||
function appendGraphqlParams(path, variables, operation) {
|
||||
const fieldToggles = operation.fieldToggles || {};
|
||||
const params = [
|
||||
`variables=${encodeURIComponent(JSON.stringify(variables))}`,
|
||||
`features=${encodeURIComponent(JSON.stringify(operation.features || {}))}`,
|
||||
];
|
||||
if (Object.keys(fieldToggles).length > 0) {
|
||||
params.push(`fieldToggles=${encodeURIComponent(JSON.stringify(fieldToggles))}`);
|
||||
}
|
||||
return `${path}?${params.join('&')}`;
|
||||
}
|
||||
|
||||
function buildUserTweetsUrl(operation, userId, count, cursor) {
|
||||
const normalized = normalizeUserTweetsOperation(operation);
|
||||
const vars = {
|
||||
userId,
|
||||
count,
|
||||
@@ -70,21 +140,20 @@ function buildUserTweetsUrl(queryId, userId, count, cursor) {
|
||||
withVoice: true,
|
||||
};
|
||||
if (cursor) vars.cursor = cursor;
|
||||
return `/i/api/graphql/${queryId}/UserTweets`
|
||||
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
|
||||
+ `&features=${encodeURIComponent(JSON.stringify(USER_TWEETS_FEATURES))}`;
|
||||
return appendGraphqlParams(`/i/api/graphql/${normalized.queryId}/UserTweets`, vars, normalized);
|
||||
}
|
||||
|
||||
function buildUserByScreenNameUrl(queryId, screenName) {
|
||||
function buildUserByScreenNameUrl(operation, screenName) {
|
||||
const normalized = normalizeUserByScreenNameOperation(operation);
|
||||
const vars = { screen_name: screenName, withSafetyModeUserFields: true };
|
||||
return `/i/api/graphql/${queryId}/UserByScreenName`
|
||||
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
|
||||
+ `&features=${encodeURIComponent(JSON.stringify(USER_BY_SCREEN_NAME_FEATURES))}`;
|
||||
return appendGraphqlParams(`/i/api/graphql/${normalized.queryId}/UserByScreenName`, vars, normalized);
|
||||
}
|
||||
|
||||
function extractTweet(result, seen) {
|
||||
if (!result) return null;
|
||||
const tw = result.tweet || result;
|
||||
const tw = result.__typename === 'TweetWithVisibilityResults' && result.tweet
|
||||
? result.tweet
|
||||
: (result.tweet || result);
|
||||
const legacy = tw.legacy || {};
|
||||
if (!tw.rest_id || seen.has(tw.rest_id)) return null;
|
||||
seen.add(tw.rest_id);
|
||||
@@ -112,32 +181,35 @@ function extractTweet(result, seen) {
|
||||
function parseUserTweets(data, seen) {
|
||||
const tweets = [];
|
||||
let nextCursor = null;
|
||||
const instructions = data?.data?.user?.result?.timeline_v2?.timeline?.instructions
|
||||
|| data?.data?.user?.result?.timeline?.timeline?.instructions
|
||||
|| [];
|
||||
for (const inst of instructions) {
|
||||
if (inst.type === 'TimelinePinEntry') continue;
|
||||
for (const entry of inst.entries || []) {
|
||||
const content = entry.content;
|
||||
if (content?.entryType === 'TimelineTimelineCursor' || content?.__typename === 'TimelineTimelineCursor') {
|
||||
if (content.cursorType === 'Bottom' || content.cursorType === 'ShowMore') nextCursor = content.value;
|
||||
continue;
|
||||
}
|
||||
if (entry.entryId?.startsWith('cursor-bottom-') || entry.entryId?.startsWith('cursor-showMore-')) {
|
||||
nextCursor = content?.value || content?.itemContent?.value || nextCursor;
|
||||
continue;
|
||||
}
|
||||
const direct = extractTweet(content?.itemContent?.tweet_results?.result, seen);
|
||||
if (direct) {
|
||||
tweets.push(direct);
|
||||
continue;
|
||||
}
|
||||
for (const item of content?.items || []) {
|
||||
const nested = extractTweet(item.item?.itemContent?.tweet_results?.result, seen);
|
||||
if (nested) tweets.push(nested);
|
||||
}
|
||||
const result = data?.data?.user?.result || {};
|
||||
const instructionSets = [
|
||||
result.timeline_v2?.timeline?.instructions,
|
||||
result.timeline?.timeline?.instructions,
|
||||
].filter(Array.isArray);
|
||||
const instructions = instructionSets.flat();
|
||||
const visit = (value) => {
|
||||
if (!value || typeof value !== 'object') return;
|
||||
if (value.type === 'TimelinePinEntry') return;
|
||||
if (value.tweet_results?.result) {
|
||||
const tweet = extractTweet(value.tweet_results.result, seen);
|
||||
if (tweet) tweets.push(tweet);
|
||||
}
|
||||
}
|
||||
if (
|
||||
(value.entryType === 'TimelineTimelineCursor' || value.__typename === 'TimelineTimelineCursor')
|
||||
&& (value.cursorType === 'Bottom' || value.cursorType === 'ShowMore')
|
||||
&& value.value
|
||||
) {
|
||||
nextCursor = value.value;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) visit(item);
|
||||
return;
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
if (child && typeof child === 'object') visit(child);
|
||||
}
|
||||
};
|
||||
visit(instructions);
|
||||
return { tweets, nextCursor };
|
||||
}
|
||||
|
||||
@@ -145,28 +217,50 @@ cli({
|
||||
site: 'twitter',
|
||||
name: 'tweets',
|
||||
access: 'read',
|
||||
description: "Fetch a Twitter user's most recent tweets (chronological, excludes pinned)",
|
||||
description: "Fetch a Twitter user's most recent tweets (chronological, excludes pinned; defaults to the logged-in user when no username is given)",
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
siteSession: 'persistent',
|
||||
args: [
|
||||
{ name: 'username', type: 'string', positional: true, required: true, help: 'Twitter screen name (with or without @)' },
|
||||
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (with or without @). Defaults to the logged-in user when omitted.' },
|
||||
{ name: 'limit', type: 'int', default: 20, help: 'Max tweets to return' },
|
||||
{ name: 'top-by-engagement', type: 'int', default: 0, help: 'When set to N>0, re-rank the tweets by weighted engagement (likes×1 + retweets×3 + replies×2 + bookmarks×5 + log10(views+1)×0.5) and return the top N. Default 0 keeps the chronological ordering.' },
|
||||
],
|
||||
columns: ['id', 'author', 'created_at', 'is_retweet', 'text', 'likes', 'retweets', 'replies', 'views', 'url', 'has_media', 'media_urls'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = Math.max(1, Math.min(200, kwargs.limit || 20));
|
||||
const username = String(kwargs.username || '').replace(/^@/, '').trim();
|
||||
if (!username) throw new CommandExecutionError('username is required');
|
||||
const rawUsername = String(kwargs.username ?? '').trim();
|
||||
let username = normalizeTwitterScreenName(rawUsername);
|
||||
if (rawUsername && !username) {
|
||||
throw new ArgumentError('twitter tweets username must be a valid Twitter/X handle', 'Example: opencli twitter tweets @jack --limit 20');
|
||||
}
|
||||
// When no username is given, detect the logged-in user (own tweets).
|
||||
// Mirrors the self-detection pattern used by twitter/profile and
|
||||
// twitter/likes so agents can pull own-account data without having
|
||||
// to know their own screen name up front.
|
||||
if (!username) {
|
||||
await page.goto('https://x.com/home');
|
||||
await page.wait({ selector: '[data-testid="primaryColumn"]' });
|
||||
// Bridge wraps primitive page.evaluate returns as { session, data:<value> }.
|
||||
// unwrapBrowserResult drops that envelope so the href string is usable.
|
||||
const href = unwrapBrowserResult(await page.evaluate(`() => {
|
||||
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
|
||||
return link ? link.getAttribute('href') : null;
|
||||
}`));
|
||||
if (!href || typeof href !== 'string')
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
username = normalizeTwitterScreenName(href);
|
||||
if (!username) {
|
||||
throw new AuthRequiredError('x.com', 'Could not detect logged-in user. Are you logged in?');
|
||||
}
|
||||
}
|
||||
|
||||
const cookies = await page.getCookies({ url: 'https://x.com' });
|
||||
const ct0 = cookies.find((c) => c.name === 'ct0')?.value || null;
|
||||
if (!ct0) throw new AuthRequiredError('x.com', 'Not logged into x.com (no ct0 cookie)');
|
||||
|
||||
const userTweetsQueryId = await resolveTwitterQueryId(page, 'UserTweets', USER_TWEETS_QUERY_ID);
|
||||
const userByScreenNameQueryId = await resolveTwitterQueryId(page, 'UserByScreenName', USER_BY_SCREEN_NAME_QUERY_ID);
|
||||
const userTweetsOperation = await resolveTwitterOperationMetadata(page, 'UserTweets', USER_TWEETS_OPERATION);
|
||||
const userByScreenNameOperation = await resolveTwitterOperationMetadata(page, 'UserByScreenName', USER_BY_SCREEN_NAME_OPERATION);
|
||||
|
||||
const headers = JSON.stringify({
|
||||
'Authorization': `Bearer ${decodeURIComponent(TWITTER_BEARER_TOKEN)}`,
|
||||
@@ -175,25 +269,26 @@ cli({
|
||||
'X-Twitter-Active-User': 'yes',
|
||||
});
|
||||
|
||||
const ubsUrl = buildUserByScreenNameUrl(userByScreenNameQueryId, username);
|
||||
const userId = await page.evaluate(`async () => {
|
||||
const ubsUrl = buildUserByScreenNameUrl(userByScreenNameOperation, username);
|
||||
const userId = unwrapBrowserResult(await page.evaluate(`async () => {
|
||||
const resp = await fetch("${ubsUrl}", { headers: ${headers}, credentials: 'include' });
|
||||
if (!resp.ok) return null;
|
||||
const d = await resp.json();
|
||||
return d?.data?.user?.result?.rest_id || null;
|
||||
}`);
|
||||
}`));
|
||||
if (!userId) throw new CommandExecutionError(`Could not resolve @${username}`);
|
||||
|
||||
const seen = new Set();
|
||||
const all = [];
|
||||
let cursor = null;
|
||||
for (let i = 0; i < 5 && all.length < limit; i++) {
|
||||
// Runaway guard only; --limit and cursor exhaustion control normal pagination.
|
||||
for (let i = 0; i < MAX_PAGINATION_PAGES && all.length < limit; i++) {
|
||||
const fetchCount = Math.min(100, limit - all.length + 10);
|
||||
const url = buildUserTweetsUrl(userTweetsQueryId, userId, fetchCount, cursor);
|
||||
const data = await page.evaluate(`async () => {
|
||||
const url = buildUserTweetsUrl(userTweetsOperation, userId, fetchCount, cursor);
|
||||
const data = normalizeTwitterGraphqlPayload(await page.evaluate(`async () => {
|
||||
const r = await fetch("${url}", { headers: ${headers}, credentials: 'include' });
|
||||
return r.ok ? await r.json() : { error: r.status };
|
||||
}`);
|
||||
}`));
|
||||
if (data?.error) {
|
||||
if (all.length === 0) throw new CommandExecutionError(`HTTP ${data.error}: UserTweets fetch failed — queryId may have expired`);
|
||||
break;
|
||||
|
||||
+238
-1
@@ -1,5 +1,6 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import { __test__ } from './tweets.js';
|
||||
|
||||
describe('twitter tweets helpers', () => {
|
||||
@@ -8,6 +9,140 @@ describe('twitter tweets helpers', () => {
|
||||
expect(cmd?.columns).toEqual(['id', 'author', 'created_at', 'is_retweet', 'text', 'likes', 'retweets', 'replies', 'views', 'url', 'has_media', 'media_urls']);
|
||||
});
|
||||
|
||||
it('makes the username argument optional so it can default to the logged-in user', () => {
|
||||
const cmd = getRegistry().get('twitter/tweets');
|
||||
const usernameArg = cmd?.args?.find((arg) => arg.name === 'username');
|
||||
expect(usernameArg).toBeDefined();
|
||||
expect(usernameArg?.required).not.toBe(true);
|
||||
expect(usernameArg?.help || '').toMatch(/default/i);
|
||||
expect(cmd?.description || '').toMatch(/default/i);
|
||||
});
|
||||
|
||||
it('detects the logged-in user via AppTabBar_Profile_Link when no username is given', async () => {
|
||||
const cmd = getRegistry().get('twitter/tweets');
|
||||
const evaluatedScripts = [];
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
const text = typeof script === 'function' ? script.toString() : String(script);
|
||||
evaluatedScripts.push(text);
|
||||
if (text.includes('AppTabBar_Profile_Link')) return '/viewer';
|
||||
if (text.includes('operationName')) return null; // operation metadata resolver
|
||||
if (text.includes('/UserByScreenName')) return '42';
|
||||
if (text.includes('/UserTweets')) {
|
||||
return {
|
||||
data: {
|
||||
user: {
|
||||
result: {
|
||||
timeline_v2: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
{
|
||||
entries: [
|
||||
{
|
||||
entryId: 'tweet-1',
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '1',
|
||||
legacy: {
|
||||
full_text: 'own post',
|
||||
favorite_count: 0,
|
||||
retweet_count: 0,
|
||||
reply_count: 0,
|
||||
created_at: 'now',
|
||||
},
|
||||
core: {
|
||||
user_results: {
|
||||
result: {
|
||||
legacy: { screen_name: 'viewer', name: 'Viewer' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}),
|
||||
};
|
||||
const rows = await cmd.func(page, { limit: 1 });
|
||||
// Navigated home to read the logged-in user
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
|
||||
// AppTabBar_Profile_Link probe happened before any GraphQL fetch
|
||||
const probeIdx = evaluatedScripts.findIndex((t) => t.includes('AppTabBar_Profile_Link'));
|
||||
const graphqlIdx = evaluatedScripts.findIndex((t) => t.includes('/UserByScreenName'));
|
||||
expect(probeIdx).toBeGreaterThanOrEqual(0);
|
||||
expect(graphqlIdx).toBeGreaterThan(probeIdx);
|
||||
// The detected handle ('viewer') was used for the UserByScreenName lookup
|
||||
const lookup = evaluatedScripts.find((t) => t.includes('/UserByScreenName')) || '';
|
||||
expect(decodeURIComponent(lookup)).toContain('"screen_name":"viewer"');
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ id: '1', author: 'viewer', url: 'https://x.com/viewer/status/1' });
|
||||
});
|
||||
|
||||
it('throws AuthRequiredError when no username is given and the logged-in user cannot be detected', async () => {
|
||||
const cmd = getRegistry().get('twitter/tweets');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => []),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
const text = typeof script === 'function' ? script.toString() : String(script);
|
||||
if (text.includes('AppTabBar_Profile_Link')) return null;
|
||||
return null;
|
||||
}),
|
||||
};
|
||||
await expect(cmd.func(page, {})).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
});
|
||||
|
||||
it('rejects invalid explicit username before navigation', async () => {
|
||||
const cmd = getRegistry().get('twitter/tweets');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
|
||||
evaluate: vi.fn(),
|
||||
};
|
||||
|
||||
await expect(cmd.func(page, { username: 'viewer/extra' })).rejects.toBeInstanceOf(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
expect(page.getCookies).not.toHaveBeenCalled();
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects non-profile AppTabBar hrefs instead of querying route names as users', async () => {
|
||||
const cmd = getRegistry().get('twitter/tweets');
|
||||
const page = {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn(async () => [{ name: 'ct0', value: 'token' }]),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
const text = typeof script === 'function' ? script.toString() : String(script);
|
||||
if (text.includes('AppTabBar_Profile_Link')) return '/home';
|
||||
throw new Error(`Unexpected evaluate: ${text.slice(0, 80)}`);
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(cmd.func(page, {})).rejects.toBeInstanceOf(AuthRequiredError);
|
||||
expect(page.goto).toHaveBeenCalledWith('https://x.com/home');
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('falls back when queryId contains unsafe characters', () => {
|
||||
expect(__test__.sanitizeQueryId('safe_Query-123', 'fallback')).toBe('safe_Query-123');
|
||||
expect(__test__.sanitizeQueryId('bad"id', 'fallback')).toBe('fallback');
|
||||
@@ -60,6 +195,18 @@ describe('twitter tweets helpers', () => {
|
||||
expect(b.is_retweet).toBe(true);
|
||||
});
|
||||
|
||||
it('unwraps TweetWithVisibilityResults', () => {
|
||||
const tweet = __test__.extractTweet({
|
||||
__typename: 'TweetWithVisibilityResults',
|
||||
tweet: {
|
||||
rest_id: '42',
|
||||
legacy: { full_text: 'visible post', favorite_count: 2, retweet_count: 0, reply_count: 0, created_at: 'now' },
|
||||
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
|
||||
},
|
||||
}, new Set());
|
||||
expect(tweet).toMatchObject({ id: '42', author: 'alice', text: 'visible post' });
|
||||
});
|
||||
|
||||
it('parses chronological tweets and skips pinned instruction', () => {
|
||||
const chronEntry = {
|
||||
entryId: 'tweet-1',
|
||||
@@ -122,4 +269,94 @@ describe('twitter tweets helpers', () => {
|
||||
url: 'https://x.com/alice/status/1',
|
||||
});
|
||||
});
|
||||
|
||||
it('recursively parses tweets nested in timeline modules', () => {
|
||||
const payload = {
|
||||
data: {
|
||||
user: {
|
||||
result: {
|
||||
timeline_v2: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
{
|
||||
type: 'TimelineAddEntries',
|
||||
entries: [
|
||||
{
|
||||
entryId: 'profile-conversation-1',
|
||||
content: {
|
||||
entryType: 'TimelineTimelineModule',
|
||||
items: [
|
||||
{
|
||||
item: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '2',
|
||||
legacy: { full_text: 'nested post', favorite_count: 1, retweet_count: 0, reply_count: 0, created_at: 'now' },
|
||||
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
entryId: 'cursor-bottom-2',
|
||||
content: { entryType: 'TimelineTimelineCursor', cursorType: 'Bottom', value: 'next' },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const result = __test__.parseUserTweets(payload, new Set());
|
||||
expect(result.nextCursor).toBe('next');
|
||||
expect(result.tweets).toHaveLength(1);
|
||||
expect(result.tweets[0]).toMatchObject({ id: '2', text: 'nested post' });
|
||||
});
|
||||
|
||||
it('uses populated timeline instructions when timeline_v2 is present but empty', () => {
|
||||
const payload = {
|
||||
data: {
|
||||
user: {
|
||||
result: {
|
||||
timeline_v2: { timeline: { instructions: [] } },
|
||||
timeline: {
|
||||
timeline: {
|
||||
instructions: [
|
||||
{
|
||||
type: 'TimelineAddEntries',
|
||||
entries: [
|
||||
{
|
||||
content: {
|
||||
itemContent: {
|
||||
tweet_results: {
|
||||
result: {
|
||||
rest_id: '3',
|
||||
legacy: { full_text: 'fallback timeline post', favorite_count: 0, retweet_count: 0, reply_count: 0, created_at: 'now' },
|
||||
core: { user_results: { result: { legacy: { screen_name: 'alice', name: 'Alice' } } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const result = __test__.parseUserTweets(payload, new Set());
|
||||
expect(result.tweets).toHaveLength(1);
|
||||
expect(result.tweets[0]).toMatchObject({ id: '3', text: 'fallback timeline post' });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
* Weibo comments — get comments on a post.
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'comments',
|
||||
@@ -19,7 +20,7 @@ cli({
|
||||
await page.goto('https://weibo.com');
|
||||
await page.wait(2);
|
||||
const id = String(kwargs.id);
|
||||
const data = await page.evaluate(`
|
||||
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const id = ${JSON.stringify(id)};
|
||||
const count = ${count};
|
||||
@@ -46,9 +47,7 @@ cli({
|
||||
return item;
|
||||
});
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data))
|
||||
return [];
|
||||
`)), 'weibo comments');
|
||||
return data;
|
||||
},
|
||||
});
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import './comments.js';
|
||||
import './favorites.js';
|
||||
import './feed.js';
|
||||
import './hot.js';
|
||||
import './me.js';
|
||||
import './post.js';
|
||||
import './search.js';
|
||||
import './user.js';
|
||||
|
||||
function envelope(data) {
|
||||
return { session: 'site:weibo:test', data };
|
||||
}
|
||||
|
||||
function makePage(evaluateResults = []) {
|
||||
const queue = [...evaluateResults];
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(async (script) => {
|
||||
if (String(script).includes('window.scrollBy')) return undefined;
|
||||
return queue.length ? queue.shift() : undefined;
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
describe('weibo read adapters Browser Bridge envelopes', () => {
|
||||
it('unwraps comments, feed, hot, search, and favorites array payloads', async () => {
|
||||
await expect(getRegistry().get('weibo/comments').func(
|
||||
makePage([envelope([{ rank: 1, author: 'a', text: 't', likes: 0, replies: 0, time: '' }])]),
|
||||
{ id: '123', limit: 1 },
|
||||
)).resolves.toHaveLength(1);
|
||||
|
||||
await expect(getRegistry().get('weibo/feed').func(
|
||||
makePage([envelope('123456'), envelope([{ id: 'm1', author: 'a', text: 't', reposts: 0, comments: 0, likes: 0, time: '', url: 'https://weibo.com/1/m1' }])]),
|
||||
{ type: 'for-you', limit: 1 },
|
||||
)).resolves.toHaveLength(1);
|
||||
|
||||
await expect(getRegistry().get('weibo/hot').func(
|
||||
makePage([envelope([{ rank: 1, word: 'opencli', hot_value: 1, category: '', label: '', url: 'https://s.weibo.com/weibo?q=opencli' }])]),
|
||||
{ limit: 1 },
|
||||
)).resolves.toHaveLength(1);
|
||||
|
||||
await expect(getRegistry().get('weibo/search').func(
|
||||
makePage([envelope([{ id: 'm1', title: 'OpenCLI', author: 'a', time: '', url: 'https://weibo.com/1/m1' }])]),
|
||||
{ keyword: 'opencli', limit: 1 },
|
||||
)).resolves.toEqual([{ rank: 1, id: 'm1', title: 'OpenCLI', author: 'a', time: '', url: 'https://weibo.com/1/m1' }]);
|
||||
|
||||
await expect(getRegistry().get('weibo/favorites').func(
|
||||
makePage([envelope('123456'), envelope([{ text: '作者A\n这是一条收藏微博', url: 'https://weibo.com/123/AbCd1' }])]),
|
||||
{ limit: 1 },
|
||||
)).resolves.toHaveLength(1);
|
||||
});
|
||||
|
||||
it('unwraps me, post, and user object payloads', async () => {
|
||||
await expect(getRegistry().get('weibo/me').func(
|
||||
makePage([envelope('123456'), envelope({ screen_name: 'me', uid: '123456' })]),
|
||||
{},
|
||||
)).resolves.toMatchObject({ screen_name: 'me', uid: '123456' });
|
||||
|
||||
await expect(getRegistry().get('weibo/post').func(
|
||||
makePage([envelope({ id: '1', text: 'post' })]),
|
||||
{ id: '1' },
|
||||
)).resolves.toContainEqual({ field: 'text', value: 'post' });
|
||||
|
||||
await expect(getRegistry().get('weibo/user').func(
|
||||
makePage([envelope({ screen_name: 'alice', uid: '42' })]),
|
||||
{ id: '42' },
|
||||
)).resolves.toMatchObject({ screen_name: 'alice', uid: '42' });
|
||||
});
|
||||
|
||||
it('fails typed instead of returning empty rows for malformed post-unwrap payloads', async () => {
|
||||
await expect(getRegistry().get('weibo/hot').func(
|
||||
makePage([envelope({ error: 'API error' })]),
|
||||
{ limit: 1 },
|
||||
)).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
|
||||
await expect(getRegistry().get('weibo/user').func(
|
||||
makePage([envelope([{ screen_name: 'wrong shape' }])]),
|
||||
{ id: '42' },
|
||||
)).rejects.toBeInstanceOf(CommandExecutionError);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
|
||||
import { getSelfUid } from './utils.js';
|
||||
import { getSelfUid, requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
|
||||
const DEFAULT_LIMIT = 20;
|
||||
const MAX_LIMIT = 50;
|
||||
@@ -123,7 +123,7 @@ cli({
|
||||
await page.wait(1);
|
||||
}
|
||||
|
||||
const rawData = await page.evaluate(`
|
||||
const rawData = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(() => {
|
||||
const scrollers = document.querySelectorAll('.wbpro-scroller-item, .vue-recycle-scroller__item-view');
|
||||
const out = [];
|
||||
@@ -145,9 +145,9 @@ cli({
|
||||
}
|
||||
return out;
|
||||
})()
|
||||
`);
|
||||
`)), 'weibo favorites');
|
||||
|
||||
if (!Array.isArray(rawData) || rawData.length === 0) {
|
||||
if (rawData.length === 0) {
|
||||
throw new EmptyResultError('weibo favorites', 'No favorites were visible on the favorites page');
|
||||
}
|
||||
|
||||
|
||||
+3
-5
@@ -2,7 +2,7 @@
|
||||
* Weibo feed — for-you or following timeline.
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { getSelfUid } from './utils.js';
|
||||
import { getSelfUid, requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
const TIMELINE_ENDPOINTS = {
|
||||
'for-you': 'unreadfriendstimeline',
|
||||
following: 'friendstimeline',
|
||||
@@ -31,7 +31,7 @@ cli({
|
||||
await page.goto('https://weibo.com');
|
||||
await page.wait(2);
|
||||
const uid = await getSelfUid(page);
|
||||
const data = await page.evaluate(`
|
||||
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const uid = ${JSON.stringify(uid)};
|
||||
const count = ${count};
|
||||
@@ -63,9 +63,7 @@ cli({
|
||||
return item;
|
||||
});
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data))
|
||||
return [];
|
||||
`)), 'weibo feed');
|
||||
return data;
|
||||
},
|
||||
});
|
||||
|
||||
+3
-4
@@ -2,6 +2,7 @@
|
||||
* Weibo hot search — browser cookie API.
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'hot',
|
||||
@@ -16,7 +17,7 @@ cli({
|
||||
func: async (page, kwargs) => {
|
||||
const count = Math.min(kwargs.limit || 30, 50);
|
||||
await page.goto('https://weibo.com');
|
||||
const data = await page.evaluate(`
|
||||
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const resp = await fetch('/ajax/statuses/hot_band', {credentials: 'include'});
|
||||
if (!resp.ok) return {error: 'HTTP ' + resp.status};
|
||||
@@ -32,9 +33,7 @@ cli({
|
||||
url: 'https://s.weibo.com/weibo?q=' + encodeURIComponent('#' + item.word + '#')
|
||||
}));
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data))
|
||||
return [];
|
||||
`)), 'weibo hot');
|
||||
return data.slice(0, count);
|
||||
},
|
||||
});
|
||||
|
||||
+3
-5
@@ -3,7 +3,7 @@
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { getSelfUid } from './utils.js';
|
||||
import { getSelfUid, requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'me',
|
||||
@@ -17,7 +17,7 @@ cli({
|
||||
await page.goto('https://weibo.com');
|
||||
await page.wait(2);
|
||||
const uid = await getSelfUid(page);
|
||||
const data = await page.evaluate(`
|
||||
const data = requireObjectEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const uid = ${JSON.stringify(uid)};
|
||||
|
||||
@@ -67,9 +67,7 @@ cli({
|
||||
profile_url: 'https://weibo.com' + (p.profile_url || '/u/' + p.id),
|
||||
};
|
||||
})()
|
||||
`);
|
||||
if (!data || typeof data !== 'object')
|
||||
throw new CommandExecutionError('Failed to fetch profile');
|
||||
`)), 'weibo me');
|
||||
if (data.error)
|
||||
throw new CommandExecutionError(String(data.error));
|
||||
return data;
|
||||
|
||||
+3
-4
@@ -3,6 +3,7 @@
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'post',
|
||||
@@ -18,7 +19,7 @@ cli({
|
||||
await page.goto('https://weibo.com');
|
||||
await page.wait(2);
|
||||
const id = String(kwargs.id);
|
||||
const data = await page.evaluate(`
|
||||
const data = requireObjectEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const id = ${JSON.stringify(id)};
|
||||
const strip = (html) => (html || '').replace(/<[^>]+>/g, '').replace(/ /g, ' ').replace(/</g, '<').replace(/>/g, '>').replace(/&/g, '&').trim();
|
||||
@@ -63,9 +64,7 @@ cli({
|
||||
|
||||
return result;
|
||||
})()
|
||||
`);
|
||||
if (!data || typeof data !== 'object')
|
||||
throw new CommandExecutionError('Failed to fetch post');
|
||||
`)), 'weibo post');
|
||||
if (data.error)
|
||||
throw new CommandExecutionError(String(data.error));
|
||||
return Object.entries(data).map(([field, value]) => ({
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CliError } from '@jackwener/opencli/errors';
|
||||
import { requireArrayEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'search',
|
||||
@@ -21,7 +22,7 @@ cli({
|
||||
const keyword = encodeURIComponent(String(kwargs.keyword ?? '').trim());
|
||||
await page.goto(`https://s.weibo.com/weibo?q=${keyword}`);
|
||||
await page.wait(2);
|
||||
const data = await page.evaluate(`
|
||||
const data = requireArrayEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(() => {
|
||||
const clean = (value) => (value || '').replace(/\\s+/g, ' ').trim();
|
||||
const absoluteUrl = (href) => {
|
||||
@@ -67,8 +68,8 @@ cli({
|
||||
|
||||
return rows;
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(data) || data.length === 0) {
|
||||
`)), 'weibo search');
|
||||
if (data.length === 0) {
|
||||
throw new CliError('NOT_FOUND', 'No Weibo search results found', 'Try a different keyword or ensure you are logged into weibo.com');
|
||||
}
|
||||
return data.slice(0, limit).map((item, index) => ({
|
||||
|
||||
+3
-4
@@ -3,6 +3,7 @@
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
cli({
|
||||
site: 'weibo',
|
||||
name: 'user',
|
||||
@@ -18,7 +19,7 @@ cli({
|
||||
await page.goto('https://weibo.com');
|
||||
await page.wait(2);
|
||||
const id = String(kwargs.id);
|
||||
const data = await page.evaluate(`
|
||||
const data = requireObjectEvaluateResult(unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const id = ${JSON.stringify(id)};
|
||||
const isUid = /^\\d+$/.test(id);
|
||||
@@ -54,9 +55,7 @@ cli({
|
||||
ip_location: d.ip_location || '',
|
||||
};
|
||||
})()
|
||||
`);
|
||||
if (!data || typeof data !== 'object')
|
||||
throw new CommandExecutionError('Failed to fetch user profile');
|
||||
`)), 'weibo user');
|
||||
if (data.error)
|
||||
throw new CommandExecutionError(String(data.error));
|
||||
return data;
|
||||
|
||||
+34
-5
@@ -1,10 +1,39 @@
|
||||
/**
|
||||
* Shared Weibo utilities — uid extraction.
|
||||
*/
|
||||
import { AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import { AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
/**
|
||||
* `page.evaluate` may return either the raw IIFE value or a
|
||||
* `{ session, data }` envelope depending on the browser-bridge version.
|
||||
* Adapter code that inspected the payload directly (e.g. `Array.isArray`,
|
||||
* truthiness checks on uid strings) silently received the envelope wrapper
|
||||
* instead of the inner value. This helper normalizes both shapes so callers
|
||||
* can keep their existing checks unchanged.
|
||||
*/
|
||||
export function unwrapEvaluateResult(payload) {
|
||||
if (payload && !Array.isArray(payload) && typeof payload === 'object' && 'session' in payload && 'data' in payload) {
|
||||
return payload.data;
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
export function requireArrayEvaluateResult(payload, label) {
|
||||
if (!Array.isArray(payload)) {
|
||||
if (payload && typeof payload === 'object' && 'error' in payload) {
|
||||
throw new CommandExecutionError(`${label}: ${String(payload.error)}`);
|
||||
}
|
||||
throw new CommandExecutionError(`${label} returned malformed extraction payload`);
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
export function requireObjectEvaluateResult(payload, label) {
|
||||
if (!payload || Array.isArray(payload) || typeof payload !== 'object') {
|
||||
throw new CommandExecutionError(`${label} returned malformed extraction payload`);
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
/** Get the currently logged-in user's uid from Vue store or config API. */
|
||||
export async function getSelfUid(page) {
|
||||
const uid = await page.evaluate(`
|
||||
const uid = unwrapEvaluateResult(await page.evaluate(`
|
||||
(() => {
|
||||
const app = document.querySelector('#app')?.__vue_app__;
|
||||
const store = app?.config?.globalProperties?.$store;
|
||||
@@ -12,18 +41,18 @@ export async function getSelfUid(page) {
|
||||
if (uid) return String(uid);
|
||||
return null;
|
||||
})()
|
||||
`);
|
||||
`));
|
||||
if (uid)
|
||||
return uid;
|
||||
// Fallback: config API
|
||||
const config = await page.evaluate(`
|
||||
const config = unwrapEvaluateResult(await page.evaluate(`
|
||||
(async () => {
|
||||
const resp = await fetch('/ajax/config/get_config', {credentials: 'include'});
|
||||
if (!resp.ok) return null;
|
||||
const data = await resp.json();
|
||||
return data.ok && data.data?.uid ? String(data.data.uid) : null;
|
||||
})()
|
||||
`);
|
||||
`));
|
||||
if (config)
|
||||
return config;
|
||||
throw new AuthRequiredError('weibo.com');
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
import { requireArrayEvaluateResult, requireObjectEvaluateResult, unwrapEvaluateResult } from './utils.js';
|
||||
|
||||
describe('unwrapEvaluateResult (browser-bridge envelope normalization)', () => {
|
||||
it('returns the raw array unchanged when payload is already an array', () => {
|
||||
const arr = [{ id: '1' }, { id: '2' }];
|
||||
expect(unwrapEvaluateResult(arr)).toBe(arr);
|
||||
});
|
||||
it('unwraps { session, data: [...] } envelope to the inner array', () => {
|
||||
const arr = [{ id: '1' }];
|
||||
const env = { session: 'site:weibo:abc', data: arr };
|
||||
expect(unwrapEvaluateResult(env)).toBe(arr);
|
||||
});
|
||||
it('unwraps primitive data (e.g. uid string) from Browser Bridge envelopes', () => {
|
||||
expect(unwrapEvaluateResult({ session: 'site:weibo:abc', data: '1234567890' })).toBe('1234567890');
|
||||
});
|
||||
it('unwraps null payload data so getSelfUid fallback can trigger', () => {
|
||||
expect(unwrapEvaluateResult({ session: 'site:weibo:abc', data: null })).toBe(null);
|
||||
});
|
||||
it('passes non-envelope objects through unchanged (e.g. profile result)', () => {
|
||||
const obj = { screen_name: 'alice', uid: '42' };
|
||||
expect(unwrapEvaluateResult(obj)).toBe(obj);
|
||||
});
|
||||
it('handles null and undefined safely', () => {
|
||||
expect(unwrapEvaluateResult(null)).toBe(null);
|
||||
expect(unwrapEvaluateResult(undefined)).toBe(undefined);
|
||||
});
|
||||
it('keeps malformed array/object payloads as typed command failures after unwrap', () => {
|
||||
expect(requireArrayEvaluateResult([{ id: '1' }], 'weibo feed')).toEqual([{ id: '1' }]);
|
||||
expect(() => requireArrayEvaluateResult({ error: 'API error' }, 'weibo feed')).toThrow(CommandExecutionError);
|
||||
expect(() => requireArrayEvaluateResult({ error: 'API error' }, 'weibo feed')).toThrow('weibo feed: API error');
|
||||
expect(requireObjectEvaluateResult({ uid: '42' }, 'weibo me')).toEqual({ uid: '42' });
|
||||
expect(() => requireObjectEvaluateResult([{ uid: '42' }], 'weibo me')).toThrow(CommandExecutionError);
|
||||
});
|
||||
});
|
||||
@@ -14,12 +14,33 @@ export function parseCommentLimit(raw, fallback = 20) {
|
||||
return fallback;
|
||||
return Math.max(1, Math.min(Math.floor(n), 50));
|
||||
}
|
||||
|
||||
export function parseXhsLikeCountText(value) {
|
||||
const integerRe = /^(?:\d+|\d{1,3}(?:[,,]\d{3})+)\+?$/u;
|
||||
const shortformRe = /^((?:\d+|\d{1,3}(?:[,,]\d{3})+)(?:\.\d+)?)([wWkK万千])\+?$/u;
|
||||
const raw = String(value ?? '').replace(/\s+/g, '');
|
||||
if (!raw)
|
||||
return 0;
|
||||
if (integerRe.test(raw))
|
||||
return Number(raw.replace(/[,+,]/g, ''));
|
||||
const short = raw.match(shortformRe);
|
||||
if (!short)
|
||||
return 0;
|
||||
const numeric = Number(short[1].replace(/[,,]/g, ''));
|
||||
if (!Number.isFinite(numeric))
|
||||
return 0;
|
||||
const unit = short[2].toLowerCase();
|
||||
const multiplier = unit === 'w' || unit === '万' ? 10000 : 1000;
|
||||
return Math.round(numeric * multiplier);
|
||||
}
|
||||
|
||||
/**
|
||||
* Host-agnostic IIFE that scrolls a note's comment list and extracts
|
||||
* top-level comments (and optionally nested 楼中楼 replies). Exported so
|
||||
* the rednote adapter can reuse the exact same selector chain.
|
||||
*/
|
||||
export function buildCommentsExtractJs(withReplies) {
|
||||
const parseLikeCountText = parseXhsLikeCountText.toString();
|
||||
return `
|
||||
(async () => {
|
||||
const wait = (ms) => new Promise(r => setTimeout(r, ms))
|
||||
@@ -44,9 +65,9 @@ export function buildCommentsExtractJs(withReplies) {
|
||||
}
|
||||
|
||||
const clean = (el) => (el?.textContent || '').replace(/\\s+/g, ' ').trim()
|
||||
const parseLikeCountText = ${parseLikeCountText}
|
||||
const parseLikes = (el) => {
|
||||
const raw = clean(el)
|
||||
return /^\\d+$/.test(raw) ? Number(raw) : 0
|
||||
return parseLikeCountText(clean(el))
|
||||
}
|
||||
const expandReplyThreads = async (root) => {
|
||||
if (!withReplies || !root) return
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { JSDOM } from 'jsdom';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import './comments.js';
|
||||
import { buildCommentsExtractJs, parseXhsLikeCountText } from './comments.js';
|
||||
function createPageMock(evaluateResult) {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
@@ -25,6 +26,41 @@ function createPageMock(evaluateResult) {
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
};
|
||||
}
|
||||
|
||||
async function runCommentsExtract(html) {
|
||||
const dom = new JSDOM(html, { url: 'https://www.xiaohongshu.com/search_result/abc123?xsec_token=tok' });
|
||||
const previousDocument = globalThis.document;
|
||||
const previousLocation = globalThis.location;
|
||||
globalThis.document = dom.window.document;
|
||||
globalThis.location = dom.window.location;
|
||||
try {
|
||||
return await eval(buildCommentsExtractJs(false));
|
||||
} finally {
|
||||
globalThis.document = previousDocument;
|
||||
globalThis.location = previousLocation;
|
||||
}
|
||||
}
|
||||
|
||||
describe('parseXhsLikeCountText', () => {
|
||||
it('parses exact integer and shortform like counts', () => {
|
||||
expect(parseXhsLikeCountText('0')).toBe(0);
|
||||
expect(parseXhsLikeCountText('42')).toBe(42);
|
||||
expect(parseXhsLikeCountText('1,234')).toBe(1234);
|
||||
expect(parseXhsLikeCountText('1,234+')).toBe(1234);
|
||||
expect(parseXhsLikeCountText('2.1w')).toBe(21000);
|
||||
expect(parseXhsLikeCountText('1.5万')).toBe(15000);
|
||||
expect(parseXhsLikeCountText('1.2k')).toBe(1200);
|
||||
expect(parseXhsLikeCountText('3千')).toBe(3000);
|
||||
expect(parseXhsLikeCountText(' 2.1 w + ')).toBe(21000);
|
||||
});
|
||||
|
||||
it('returns 0 for unknown shapes without overparsing arbitrary text', () => {
|
||||
for (const raw of ['', null, undefined, '赞', 'likes 2.1w', '2w人', '1,23', '1.2.3k', '.', '1.5']) {
|
||||
expect(parseXhsLikeCountText(raw)).toBe(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('xiaohongshu comments', () => {
|
||||
const command = getRegistry().get('xiaohongshu/comments');
|
||||
it('returns ranked comment rows for signed full URLs', async () => {
|
||||
@@ -120,6 +156,32 @@ describe('xiaohongshu comments', () => {
|
||||
expect(script).toContain("const afterCount = scroller.querySelectorAll('.parent-comment').length");
|
||||
expect(script).toContain('if (afterCount <= beforeCount) break');
|
||||
});
|
||||
it('extracts shortform like counts from the shared xiaohongshu/rednote DOM script', async () => {
|
||||
const data = await runCommentsExtract(`
|
||||
<main>
|
||||
<section class="parent-comment">
|
||||
<div class="comment-item">
|
||||
<div class="author-wrapper"><span class="name">Alice</span></div>
|
||||
<div class="content">Great note</div>
|
||||
<span class="count">2.1w</span>
|
||||
<span class="date">today</span>
|
||||
</div>
|
||||
</section>
|
||||
<section class="parent-comment">
|
||||
<div class="comment-item">
|
||||
<span class="user-name">Bob</span>
|
||||
<div class="note-text">Malformed count</div>
|
||||
<span class="count">likes 2.1w</span>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
`);
|
||||
|
||||
expect(data.results).toEqual([
|
||||
{ author: 'Alice', text: 'Great note', likes: 21000, time: 'today', is_reply: false, reply_to: '' },
|
||||
{ author: 'Bob', text: 'Malformed count', likes: 0, time: '', is_reply: false, reply_to: '' },
|
||||
]);
|
||||
});
|
||||
it('respects the limit for top-level comments', async () => {
|
||||
const manyComments = Array.from({ length: 10 }, (_, i) => ({
|
||||
author: `User${i}`,
|
||||
|
||||
+87
-25
@@ -6,16 +6,24 @@
|
||||
* Ref: https://github.com/jackwener/opencli/issues/10
|
||||
*/
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
import { ArgumentError, AuthRequiredError } from '@jackwener/opencli/errors';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
|
||||
/**
|
||||
* Wait for search results or login wall using MutationObserver (max 5s).
|
||||
* Returns 'content' if note items appeared, 'login_wall' if login gate
|
||||
* detected, or 'timeout' if neither appeared within the deadline.
|
||||
*
|
||||
* Note-item detection tries the legacy `section.note-item` class first
|
||||
* (still observed in many sessions, including rednote) and falls back to
|
||||
* a `<section>` element containing a `/search_result/` or `/explore/`
|
||||
* link. Issue #1506 reports the class being dropped on some xhs renders.
|
||||
*/
|
||||
const WAIT_FOR_CONTENT_JS = `
|
||||
new Promise((resolve) => {
|
||||
const findNoteCard = () => document.querySelector(
|
||||
'section.note-item, section:has(a[href*="/search_result/"]), section:has(a[href*="/explore/"])'
|
||||
);
|
||||
const detect = () => {
|
||||
if (document.querySelector('section.note-item')) return 'content';
|
||||
if (findNoteCard()) return 'content';
|
||||
if (/登录后查看搜索结果/.test(document.body?.innerText || '')) return 'login_wall';
|
||||
return null;
|
||||
};
|
||||
@@ -52,6 +60,26 @@ export function stripXhsAuthorDateSuffix(value) {
|
||||
const stripped = text.replace(/\s*(?:\d{1,2}天前|\d+小时前|\d+分钟前|\d+秒前|刚刚|昨天|前天|\d+周前|\d+个月前|\d{1,2}-\d{1,2}|\d{4}-\d{1,2}-\d{1,2})$/u, '').trim();
|
||||
return stripped || text;
|
||||
}
|
||||
/**
|
||||
* `page.evaluate` may return either the raw IIFE value or a
|
||||
* `{ session, data }` envelope depending on the browser-bridge version.
|
||||
* Adapter code that called `Array.isArray(payload)` directly on the
|
||||
* envelope silently received [] for every search. This helper normalizes
|
||||
* both shapes so callers can keep their Array.isArray checks unchanged.
|
||||
*/
|
||||
export function unwrapEvaluateResult(payload) {
|
||||
if (payload && !Array.isArray(payload) && typeof payload === 'object' && 'session' in payload && 'data' in payload) {
|
||||
return payload.data;
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
function requireSearchRows(payload, phase) {
|
||||
const rows = unwrapEvaluateResult(payload);
|
||||
if (!Array.isArray(rows)) {
|
||||
throw new CommandExecutionError(`Unexpected Xiaohongshu search ${phase} payload shape; expected an array of rows.`);
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
export function parseLimit(raw) {
|
||||
const parsed = Number(raw ?? 20);
|
||||
if (!Number.isFinite(parsed) || !Number.isInteger(parsed)) {
|
||||
@@ -94,9 +122,22 @@ export function buildScrollUntilJs(targetCount, maxScrolls = 15) {
|
||||
const style = getComputedStyle(el);
|
||||
return style.display !== 'none' && style.visibility !== 'hidden';
|
||||
};
|
||||
// Note containers: legacy \`section.note-item\` first, fallback to
|
||||
// any \`<section>\` that wraps a search-result/explore note link
|
||||
// (#1506 reports the class being dropped on some xhs renders).
|
||||
const collectNoteCards = () => {
|
||||
const classMatches = document.querySelectorAll('section.note-item');
|
||||
if (classMatches.length > 0) return classMatches;
|
||||
const sections = new Set();
|
||||
for (const a of document.querySelectorAll('a[href*="/search_result/"], a[href*="/explore/"]')) {
|
||||
const section = a.closest('section');
|
||||
if (section) sections.add(section);
|
||||
}
|
||||
return sections;
|
||||
};
|
||||
const countItems = () => {
|
||||
let count = 0;
|
||||
for (const el of document.querySelectorAll('section.note-item')) {
|
||||
for (const el of collectNoteCards()) {
|
||||
if (isVisibleNote(el)) count++;
|
||||
}
|
||||
return count;
|
||||
@@ -161,10 +202,24 @@ export function buildSearchExtractJs(webHost) {
|
||||
const results = [];
|
||||
const seen = new Set();
|
||||
|
||||
document.querySelectorAll('section.note-item').forEach(el => {
|
||||
// Note containers: legacy \`section.note-item\` first, fallback to any
|
||||
// \`<section>\` wrapping a search-result/explore link (#1506 reports the
|
||||
// class being dropped on some xhs renders).
|
||||
const collectNoteCards = () => {
|
||||
const classMatches = document.querySelectorAll('section.note-item');
|
||||
if (classMatches.length > 0) return classMatches;
|
||||
const sections = new Set();
|
||||
for (const a of document.querySelectorAll('a[href*="/search_result/"], a[href*="/explore/"]')) {
|
||||
const section = a.closest('section');
|
||||
if (section) sections.add(section);
|
||||
}
|
||||
return sections;
|
||||
};
|
||||
|
||||
for (const el of collectNoteCards()) {
|
||||
// Skip "related searches" sections
|
||||
if (el.classList.contains('query-note-item')) return;
|
||||
if (!isVisibleNote(el)) return;
|
||||
if (el.classList?.contains('query-note-item')) continue;
|
||||
if (!isVisibleNote(el)) continue;
|
||||
|
||||
const titleEl = el.querySelector('.title, .note-title, a.title, .footer .title span');
|
||||
const nameEl = el.querySelector('a.author .name, .author-name, .nick-name, .name');
|
||||
@@ -184,20 +239,29 @@ export function buildSearchExtractJs(webHost) {
|
||||
const authorLinkEl = el.querySelector('a.author, a[href*="/user/profile/"]');
|
||||
|
||||
const url = normalizeUrl(detailLinkEl?.getAttribute('href') || '');
|
||||
if (!url) return;
|
||||
if (!url) continue;
|
||||
|
||||
const key = url;
|
||||
if (seen.has(key)) return;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
|
||||
// Fallback title: the new bare-section render keeps the note caption
|
||||
// inside the search_result anchor's first span, not in a class-named
|
||||
// .title element. Pull from there when the class-based pick is empty.
|
||||
let title = cleanText(titleEl?.textContent || '');
|
||||
if (!title) {
|
||||
const captionSpan = detailLinkEl?.querySelector('span');
|
||||
title = cleanText(captionSpan?.textContent || '');
|
||||
}
|
||||
|
||||
results.push({
|
||||
title: cleanText(titleEl?.textContent || ''),
|
||||
title,
|
||||
author,
|
||||
likes: cleanText(likesEl?.textContent || '0'),
|
||||
url,
|
||||
author_url: normalizeUrl(authorLinkEl?.getAttribute('href') || ''),
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
})()
|
||||
@@ -223,7 +287,7 @@ export const command = cli({
|
||||
// Wait for search results to render (or login wall to appear).
|
||||
// Uses MutationObserver to resolve as soon as content appears,
|
||||
// instead of a fixed delay + blind retry.
|
||||
const waitResult = await page.evaluate(WAIT_FOR_CONTENT_JS);
|
||||
const waitResult = unwrapEvaluateResult(await page.evaluate(WAIT_FOR_CONTENT_JS));
|
||||
if (waitResult === 'login_wall') {
|
||||
throw new AuthRequiredError('www.xiaohongshu.com', 'Xiaohongshu search results are blocked behind a login wall');
|
||||
}
|
||||
@@ -231,25 +295,23 @@ export const command = cli({
|
||||
// layout, so scrolling to the bottom can evict the initially visible
|
||||
// note cards from the DOM and make extraction return [] even though the
|
||||
// browser rendered results correctly.
|
||||
const initialPayload = await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com'));
|
||||
let payload = Array.isArray(initialPayload) ? initialPayload : [];
|
||||
const initialPayload = requireSearchRows(await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com')), 'initial extraction');
|
||||
const payload = [...initialPayload];
|
||||
if (payload.length < limit) {
|
||||
// Scroll until enough rows are rendered or the lazy-load plateaus.
|
||||
// Replaces the previous fixed `autoScroll({ times: 2 })` which capped
|
||||
// extraction at ~13 notes regardless of `--limit` (#1471).
|
||||
await page.evaluate(buildScrollUntilJs(limit));
|
||||
const scrolledPayload = await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com'));
|
||||
if (Array.isArray(scrolledPayload)) {
|
||||
const seen = new Set(payload.map((item) => item.url).filter(Boolean));
|
||||
for (const item of scrolledPayload) {
|
||||
if (item?.url && seen.has(item.url))
|
||||
continue;
|
||||
if (item?.url)
|
||||
seen.add(item.url);
|
||||
payload.push(item);
|
||||
if (payload.length >= limit)
|
||||
break;
|
||||
}
|
||||
const scrolledPayload = requireSearchRows(await page.evaluate(buildSearchExtractJs('www.xiaohongshu.com')), 'post-scroll extraction');
|
||||
const seen = new Set(payload.map((item) => item.url).filter(Boolean));
|
||||
for (const item of scrolledPayload) {
|
||||
if (item?.url && seen.has(item.url))
|
||||
continue;
|
||||
if (item?.url)
|
||||
seen.add(item.url);
|
||||
payload.push(item);
|
||||
if (payload.length >= limit)
|
||||
break;
|
||||
}
|
||||
}
|
||||
const data = payload;
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { getRegistry } from '@jackwener/opencli/registry';
|
||||
import { JSDOM } from 'jsdom';
|
||||
import { __test__, buildScrollUntilJs, noteIdToDate } from './search.js';
|
||||
import { __test__, buildScrollUntilJs, noteIdToDate, unwrapEvaluateResult } from './search.js';
|
||||
|
||||
function markVisible(el) {
|
||||
el.getBoundingClientRect = () => ({ width: 100, height: 100 });
|
||||
@@ -57,24 +57,37 @@ describe('xiaohongshu search', () => {
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(1);
|
||||
expect(page.autoScroll).not.toHaveBeenCalled();
|
||||
});
|
||||
it('unwraps a browser-bridge envelope before handling login-wall wait result', async () => {
|
||||
const cmd = getRegistry().get('xiaohongshu/search');
|
||||
const page = createPageMock([
|
||||
{ session: 'site:xiaohongshu', data: 'login_wall' },
|
||||
]);
|
||||
|
||||
await expect(cmd.func(page, { query: '特斯拉', limit: 5 })).rejects.toMatchObject({
|
||||
code: 'AUTH_REQUIRED',
|
||||
message: expect.stringContaining('blocked behind a login wall'),
|
||||
});
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
it('returns ranked results with search_result url and author_url preserved', async () => {
|
||||
const cmd = getRegistry().get('xiaohongshu/search');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
const detailUrl = 'https://www.xiaohongshu.com/search_result/68e90be80000000004022e66?xsec_token=test-token&xsec_source=';
|
||||
const authorUrl = 'https://www.xiaohongshu.com/user/profile/635a9c720000000018028b40?xsec_token=user-token&xsec_source=pc_search';
|
||||
const rows = [
|
||||
{
|
||||
title: '某鱼买FSD被坑了4万',
|
||||
author: '随风',
|
||||
likes: '261',
|
||||
url: detailUrl,
|
||||
author_url: authorUrl,
|
||||
},
|
||||
];
|
||||
const page = createPageMock([
|
||||
// First evaluate: MutationObserver wait (content appeared)
|
||||
'content',
|
||||
// Second evaluate: initial DOM extraction (already enough results)
|
||||
[
|
||||
{
|
||||
title: '某鱼买FSD被坑了4万',
|
||||
author: '随风',
|
||||
likes: '261',
|
||||
url: detailUrl,
|
||||
author_url: authorUrl,
|
||||
},
|
||||
],
|
||||
// Second evaluate: initial DOM extraction (already enough results) through Browser Bridge envelope.
|
||||
{ session: 'site:xiaohongshu', data: rows },
|
||||
]);
|
||||
const result = await cmd.func(page, { query: '特斯拉', limit: 1 });
|
||||
// Should only do one goto (the search page itself), no per-note detail navigation
|
||||
@@ -91,6 +104,18 @@ describe('xiaohongshu search', () => {
|
||||
},
|
||||
]);
|
||||
});
|
||||
it('fails typed instead of silently returning [] for malformed extraction payloads', async () => {
|
||||
const cmd = getRegistry().get('xiaohongshu/search');
|
||||
const page = createPageMock([
|
||||
'content',
|
||||
{ session: 'site:xiaohongshu', data: { rows: [] } },
|
||||
]);
|
||||
|
||||
await expect(cmd.func(page, { query: '测试', limit: 1 })).rejects.toMatchObject({
|
||||
code: 'COMMAND_EXEC',
|
||||
message: expect.stringContaining('payload shape'),
|
||||
});
|
||||
});
|
||||
it('filters out results with no title and respects the limit', async () => {
|
||||
const cmd = getRegistry().get('xiaohongshu/search');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
@@ -135,6 +160,10 @@ describe('xiaohongshu search', () => {
|
||||
'content',
|
||||
// Second evaluate: initial extraction (no rows rendered)
|
||||
[],
|
||||
// Third evaluate: scroll-until row count
|
||||
0,
|
||||
// Fourth evaluate: post-scroll extraction (still no rows)
|
||||
[],
|
||||
]);
|
||||
const result = (await cmd.func(page, { query: '测试等待', limit: 5 }));
|
||||
expect(result).toHaveLength(0);
|
||||
@@ -268,3 +297,29 @@ describe('noteIdToDate (ObjectID timestamp parsing)', () => {
|
||||
expect(noteIdToDate('https://www.xiaohongshu.com/search_result/000000000000000000000000')).toBe('');
|
||||
});
|
||||
});
|
||||
describe('unwrapEvaluateResult (browser-bridge envelope normalization)', () => {
|
||||
it('returns the raw array unchanged when payload is already an array', () => {
|
||||
const arr = [{ title: 'a' }, { title: 'b' }];
|
||||
expect(unwrapEvaluateResult(arr)).toBe(arr);
|
||||
});
|
||||
it('unwraps { session, data: [...] } envelope to the inner array', () => {
|
||||
const arr = [{ title: 'a' }];
|
||||
const env = { session: 'site:xiaohongshu:abc', data: arr };
|
||||
expect(unwrapEvaluateResult(env)).toBe(arr);
|
||||
});
|
||||
it('unwraps primitive data from Browser Bridge envelopes', () => {
|
||||
expect(unwrapEvaluateResult({ session: 'site:xiaohongshu:abc', data: 'login_wall' })).toBe('login_wall');
|
||||
});
|
||||
it('passes non-envelope objects through unchanged', () => {
|
||||
const obj = { results: [], loginWall: true };
|
||||
expect(unwrapEvaluateResult(obj)).toBe(obj);
|
||||
});
|
||||
it('handles null and undefined safely', () => {
|
||||
expect(unwrapEvaluateResult(null)).toBe(null);
|
||||
expect(unwrapEvaluateResult(undefined)).toBe(undefined);
|
||||
});
|
||||
it('unwraps non-array envelope data so callers can validate the payload shape', () => {
|
||||
const env = { session: 'x', data: { not: 'an array' } };
|
||||
expect(unwrapEvaluateResult(env)).toEqual({ not: 'an array' });
|
||||
});
|
||||
});
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user