Compare commits

...

49 Commits

Author SHA1 Message Date
jackwener 31fccffeb5 docs: add shell auto-completion instructions 2026-03-17 17:48:54 +08:00
jackwener 331809e8d4 docs: sync commands list in readmes and skill doc 2026-03-17 17:47:38 +08:00
jackwener 43e69fdabc test: make Vitest project order explicit 2026-03-17 17:34:05 +08:00
rbbtsn0w 97a6664ecc fix(test): add e2e sequence.groupOrder in vitest config 2026-03-17 16:52:38 +08:00
jackwener d1da293ef9 0.7.5
Release / release (push) Has been cancelled
2026-03-17 16:14:02 +08:00
jackwener 25bd872a24 fix: doctor/setup edge cases — format detection, dynamic profiles, fish shell
- upsertJsonConfigToken: detect format by file path (opencode → mcp format,
  others → mcpServers). Previously empty files always got OpenCode format.
- Dynamic Chrome profile enumeration: scan for Default/Profile N directories
  instead of hardcoding 4 profiles.
- Fish shell: use 'set -gx' syntax for config.fish, not 'export'.
- Pass filePath through all callers (setup.ts, applyBrowserDoctorFix).
- Reduce setup auto-verify timeout from 8s to 5s.
- Add 7 new tests (19 total): empty file format, opencode path detection,
  claude.json path detection, fish shell set/replace/append, zshrc fallback.
2026-03-17 16:13:05 +08:00
jackwener ff3e5c6887 feat: enhance setup with precise token scan errors and auto-verify
- When token scan fails, diagnose exact cause via checkExtensionInstalled()
  (extension not installed vs token not in LevelDB)
- Show actionable fix instructions instead of generic warning
- Auto-verify browser connectivity after writing configs (Step 7)
- Simplify README setup flow to 2 steps (install + setup)
2026-03-17 16:07:38 +08:00
jackwener 2e66e3183c docs: reorder setup flow — doctor → setup → doctor --live 2026-03-17 16:02:37 +08:00
jackwener a1bcb23239 docs: reorder setup flow — doctor first, then setup
Logical flow: install extension → doctor (verify token discoverable) →
setup (distribute token to tools). --fix moved to a Tip block for
post-setup maintenance.
2026-03-17 16:00:46 +08:00
jackwener 6024af3aa0 docs: split doctor --fix into interactive and non-interactive examples 2026-03-17 15:58:53 +08:00
jackwener 1393ce3327 docs: sync Chinese README with doctor --live, command table polish 2026-03-17 14:57:13 +08:00
jackwener 0fe3b9b921 0.7.4
Release / release (push) Has been cancelled
2026-03-17 14:55:54 +08:00
jackwener 375beaa744 docs: polish README and SKILL
- Sort command table by count (descending), add Count column
- Add xiaohongshu `me`, boss `detail` to command references
- Add Self-healing setup highlight for doctor/setup workflow
- Document `doctor --live` and `doctor --fix` options
- Bump SKILL version to 0.7.3, expand tags
- Fix site count to 19, update descriptions
2026-03-17 14:50:14 +08:00
SonicKang 341c42c62f fix(opencode): use 'environment' instead of 'env' for MCP config (#29)
OpenCode config schema uses 'environment' property for MCP server
environment variables, not 'env'.

Schema reference: https://opencode.ai/config.json
2026-03-17 14:46:53 +08:00
jackwener 50b71c0936 fix: use binary read for LevelDB token discovery on all platforms
The previous strings+grep pipeline failed because LevelDB's internal
encoding fragments ASCII strings like 'auth-token' and the extension ID
across byte boundaries. Replace extractTokenViaStrings with a unified
binary read approach that scans for the extension ID prefix and searches
a 500-byte window for base64url tokens.

Also removes unused execSync import.
2026-03-17 14:44:34 +08:00
jackwener 981c167a0b feat(doctor): add extension install check and token connectivity test
- checkExtensionInstalled(): scans Chrome/Edge/Chromium Extensions dirs
- checkTokenConnectivity(): actual MCP handshake via --live flag
- Updated DoctorReport type and report rendering
- Added unit tests for new rendering (12/12 pass)
2026-03-17 14:38:16 +08:00
jackwener 2463689105 0.7.3
Release / release (push) Has been cancelled
2026-03-17 13:34:34 +08:00
jackwener c714254d8f docs: add YouTube video and transcript commands to README and SKILL 2026-03-17 13:30:42 +08:00
Ji 8e7490407c feat(youtube): add video metadata and transcript commands (#25)
Add two new YouTube adapters:

- **youtube video**: fetch metadata (title, views, description, etc.) from ytInitialPlayerResponse and ytInitialData
- **youtube transcript**: fetch subtitles via Android InnerTube API to bypass PoToken requirement on Web client caption URLs
  - Two output modes: --mode grouped (sentence merging, speaker detection, chapter headings) and --mode raw (precise sub-second timestamps)
  - CJK support with 30s time-window fallback for unpunctuated captions
  - Language selection with --lang and stderr warning on fallback
  - URL normalization for watch, youtu.be, shorts, embed, live formats

Co-authored-by: Ji Zhang <jizhang.work@gmail.com>
2026-03-17 13:26:30 +08:00
Ji 14dcd2bc5f feat(reddit): add threaded comment tree to read command (#26)
Replace flat top-level-only read.yaml with recursive tree walker:
- Configurable depth and breadth (--depth, --replies)
- Replies sorted by score, top-K selected at each level
- Hidden replies surfaced as [+N more replies]
- Multiline bodies preserve indentation at all depths
- Configurable --max_length (was hard-coded 500 chars)
- Input validation: all numeric params clamped to safe minimums
2026-03-17 13:18:11 +08:00
SiweiMa e9b9beedfe feat(linkedin): add job search adapter (#28)
* feat: add linkedin job search adapter

* fix(linkedin): fix parseCsvArg undefined bug, regex escapes in page.evaluate, replace hardcoded wait, add IPage type

* refactor(linkedin): extract evaluate logic, add progress logging, improve code structure

- Extract Voyager query/URL building into typed standalone functions
- Split fetchJobCards into its own function with per-batch evaluate
- Add SearchInput interface for type safety
- Add progress logging to enrichJobDetails (stderr)
- Add section comments for code organization
- Deduplicate normalize helpers in evaluate strings

---------

Co-authored-by: Siwei Ma <siweima@Siweis-MacBook-Pro.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-17 13:07:16 +08:00
jackwener 59de5fb3f5 0.7.2
Release / release (push) Has been cancelled
2026-03-17 01:38:29 +08:00
jackwener 7555f14369 refactor: deep code review improvements
- Add *.log to .gitignore, remove debug.log from tracking
- Fix dev-mode FS scan to discover .ts adapter files (not just .js)
- Deduplicate CONNECT_TIMEOUT: browser.ts now uses runtime.ts constant
- Fix CSV output: escape newlines in field values per RFC 4180
- Add proper type interfaces for validate/verify (remove any types)
- Remove unused hadOuterQuotes variable in snapshotFormatter
- Derive CliOptions from CliCommand via Omit+Partial to reduce duplication
- Expand dense one-liner action callbacks in main.ts for readability
2026-03-17 01:38:23 +08:00
jackwener 2652fa40e5 chore: change license from BSD-3-Clause to Apache-2.0 2026-03-17 01:34:51 +08:00
jackwener a7c367a61b docs: update README and SKILL for new Reddit adapters
- Reddit: 4 → 15 commands (popular, read, user, user-posts,
  user-comments, upvote, save, comment, subscribe, saved, upvoted)
- Twitter: add thread command
- Xiaohongshu: remove non-existent me command
- SKILL.md: expand Reddit examples with full 15-command reference
2026-03-17 01:33:43 +08:00
jackwener fbec2f6f5d feat(snapshot): filter contentinfo subtrees, bilibili ad URLs, boilerplate buttons
- Add contentinfo to subtree-level noise filtering (biggest single win)
  - Reuters: 51% → 62%, Google: 57% → 70%, Netflix: 48% → 60%
- Add cm.bilibili.com/cm/api/fees/ ad URL pattern
- Add 广告 keyword to ad detection
- Add back-to-top / 回到顶部 boilerplate button filtering
- Unify ad/boilerplate/contentinfo into single subtree-skip mechanism
- Add vitest config and comprehensive test suite (33 tests)
- Fixture tests skip gracefully when snapshot files are absent

Bump to v0.7.1
2026-03-17 01:26:49 +08:00
jackwener c2a5cbe90e chore(release): 0.7.0
Release / release (push) Has been cancelled
2026-03-16 20:29:27 +08:00
jackwener 34e20d33f2 docs: bump version in SKILL.md to 0.7.0 2026-03-16 20:29:27 +08:00
jackwener 1c496bb85f docs: add new twitter commands (article, follow, unfollow, bookmark, unbookmark)
Also update profile example to use positional argument.
2026-03-16 20:18:50 +08:00
jackwener 0c845d58c8 feat(twitter): implement article, profile, follow, unfollow, bookmark, & unbookmark adapters
This commit introduces the long-form Article adapter, a rewritten Profile adapter, and 4 new UI-based Write commands for managing relationships and bookmarks. Also adds support for positional arguments across the dynamic CLI engine.
2026-03-16 20:17:38 +08:00
jackwener 7f55950fed feat(reddit): add 11 new adapters borrowed from rdt-cli
Phase 1 - YAML adapters (read-only):
- popular: /r/popular feed
- read: read post + comments by ID
- user: view user profile (karma, account age)
- user-posts: user's submitted posts
- user-comments: user's comment history
- search: enhanced with sort/time/subreddit params
- subreddit: enhanced with time filter for top/controversial

Phase 2 - TypeScript adapters (write operations):
- upvote: upvote/downvote posts via /api/vote
- save: save/unsave posts via /api/save
- comment: post comments via /api/comment
- subscribe: subscribe/unsubscribe subreddits
- saved: browse saved posts (auto-resolves username)
- upvoted: browse upvoted posts (auto-resolves username)

Reddit adapters: 4 → 15
2026-03-16 19:56:55 +08:00
jackwener 1576396a21 0.6.3
Release / release (push) Has been cancelled
2026-03-16 19:33:25 +08:00
jackwener 77193a0003 Merge PR #20: feat(boss): add detail adapter + security_id in search
Closes #20

Added boss detail command with fixes:
- district/address field dedup
- template string injection safety
- empty jobInfo guard
- IPage-compatible wait
2026-03-16 19:33:17 +08:00
jackwener 05b7f1bccf fix(boss): improve detail adapter quality
- Fix district/address field duplication (district now uses areaDistrict·businessDistrict)
- Fix template string injection risk in evaluate script (use JSON.stringify)
- Add jobInfo empty guard with user-friendly error message
- Replace raw setTimeout with page.wait for IPage compatibility
- Update README docs to include boss detail command
2026-03-16 19:33:00 +08:00
jackwener 9889a6db11 v0.6.2
Release / release (push) Has been cancelled
2026-03-16 18:14:34 +08:00
jackwener 61ea05bff7 fix: URL injection, strictNullChecks, cross-platform build, +34 tests
Security:
- Fix URL injection in fetch.ts and bilibili.ts (JSON.stringify instead of string interpolation)
- Fix unused scroll() amount parameter

TypeScript:
- Enable strictNullChecks in tsconfig
- Change CliCommand.func signature to IPage (non-null) for browser adapters
- Fix 93 compile errors across all adapters

Build:
- Remove || true from build-manifest (report failures instead of silencing)
- Replace Unix shell commands with Node.js scripts for cross-platform builds

Code quality:
- Remove error-object special detection from pipeline executor
- Unify error handling to throw pattern

Tests:
- New interceptor.test.ts (11 tests)
- New executor.test.ts (13 tests)
- Rewrite output.test.ts with comprehensive coverage (10 tests)
2026-03-16 18:14:27 +08:00
xuelin e781d40408 feat(boss): add security_id to search output
Expose securityId in search results so users can pipe it to
`boss detail` for full job information.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 18:12:46 +08:00
xuelin c230f3e5ad feat(boss): add job detail adapter
Add `boss detail` command to fetch full job posting details using
securityId from search results.

Fields returned: job description, skills, welfare, boss info (name,
title, active time), company info (industry, scale, stage), address.

Tested with real API calls against multiple job postings.

Usage:
  opencli boss detail --security_id <id_from_search>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 18:09:49 +08:00
jackwener 3b2f88b2cf chore: sync package-lock.json version to 0.6.1 2026-03-16 17:38:13 +08:00
jackwener 7eec7ce89f fix: restore tests/ in vitest include for CI compatibility
vitest run tests/e2e/ intersects the CLI path with include patterns,
so tests/ must be in the include glob for CI to find test files.
2026-03-16 17:37:48 +08:00
AlexYue 788b069c02 feat: add E2E testing infrastructure with real Chrome in CI
## Changes

### E2E Test Suite (~52 test cases)
- public-commands.test.ts — Public API commands (hackernews, v2ex)
- browser-public.test.ts — Browser commands for public data across all sites
- browser-auth.test.ts — Graceful failure verification for login-required commands
- management.test.ts — Full coverage of management commands
- output-formats.test.ts — Output format validation (json/yaml/csv/md)
- smoke/api-health.test.ts — Scheduled API health checks

### Auto-detect Browser Mode
- buildMcpArgs uses CI env var to select mode:
  - Local (no CI) → --extension (connect to user's Chrome)
  - CI → standalone (launches its own browser)

### CI Pipeline
- e2e-headed.yml — Real Chrome via setup-chrome + xvfb in headed mode
- ci.yml — build + unit-test (2 shards) + smoke-test (scheduled/manual)
- Composite action for shared Chrome + xvfb setup

### Documentation
- New TESTING.md — Architecture, coverage, local setup, how to add tests

Co-authored-by: AlexYue <yj976240184@qq.com>
2026-03-16 17:35:16 +08:00
jackwener 433ad3a56a 0.6.1
Release / release (push) Has been cancelled
2026-03-16 14:20:22 +08:00
jackwener 50508b954e docs: expand bilibili commands, add setup hint after install, add doctor to troubleshooting 2026-03-16 14:16:54 +08:00
jackwener 35a843b8bd docs: use explicit PLAYWRIGHT_MCP_EXTENSION_TOKEN in auto-discover description 2026-03-16 14:16:12 +08:00
jackwener 486e513d07 fix(setup): only pre-select shell RC, let user choose other configs 2026-03-16 14:15:43 +08:00
jackwener 6c64f617c6 docs: add opencli setup to READMEs, remove hardcoded counts, update SKILL.md to v0.6.0 2026-03-16 14:13:38 +08:00
jackwener cd186bddd3 0.6.0
Release / release (push) Has been cancelled
2026-03-16 14:05:58 +08:00
jackwener 6486a42def fix(setup): clear screen before TUI to prevent page jumping 2026-03-16 14:04:38 +08:00
jackwener b308d5594a refactor(doctor/setup): polish UX and dedup code
- Doctor: chalk-colored output ([OK] green, [MISSING] red, etc.)
- Doctor: paths shortened with ~ and tool labels ([Codex], etc.)
- Doctor --fix: skip already-configured files
- TUI: hide cursor during interaction, proper Ctrl+C exit
- Setup: source hint after shell write, dedup shared helpers
- Tests: strip ANSI for assertions
2026-03-16 14:02:02 +08:00
70 changed files with 6189 additions and 460 deletions
+26
View File
@@ -0,0 +1,26 @@
name: Setup Chrome + xvfb
description: Install real Chrome and xvfb virtual display for headed browser testing
outputs:
chrome-path:
description: Path to the installed Chrome binary
value: ${{ steps.setup-chrome.outputs.chrome-path }}
runs:
using: composite
steps:
- name: Install real Chrome (stable)
uses: browser-actions/setup-chrome@v1
id: setup-chrome
with:
chrome-version: stable
- name: Verify Chrome installation
shell: bash
run: |
echo "Chrome path: ${{ steps.setup-chrome.outputs.chrome-path }}"
${{ steps.setup-chrome.outputs.chrome-path }} --version
- name: Install xvfb for headed mode
shell: bash
run: sudo apt-get install -y xvfb
+59 -3
View File
@@ -2,12 +2,16 @@ name: CI
on:
push:
branches: [main]
branches: [main, dev]
pull_request:
branches: [main]
branches: [main, dev]
schedule:
- cron: '0 8 * * 1' # Weekly Monday 08:00 UTC — smoke tests
workflow_dispatch:
jobs:
check:
# ── Fast gate: typecheck + build ──
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -15,6 +19,7 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
@@ -24,3 +29,54 @@ jobs:
- name: Build
run: npm run build
# ── Unit tests (vitest shard) ──
unit-test:
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run unit tests (shard ${{ matrix.shard }}/2)
run: npx vitest run src/ --reporter=verbose --shard=${{ matrix.shard }}/2
# ── Smoke tests (scheduled / manual only) ──
smoke-test:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Setup Chrome + xvfb
uses: ./.github/actions/setup-chrome
id: setup-chrome
- name: Build
run: npm run build
- name: Run smoke tests
run: |
xvfb-run --auto-servernum --server-args="-screen 0 1280x720x24" \
npx vitest run tests/smoke/ --reporter=verbose
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
timeout-minutes: 15
+37
View File
@@ -0,0 +1,37 @@
name: E2E Headed Chrome
on:
push:
branches: [main, dev]
pull_request:
branches: [main, dev]
workflow_dispatch:
jobs:
e2e-headed:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Setup Chrome + xvfb
uses: ./.github/actions/setup-chrome
id: setup-chrome
- name: Build
run: npm run build
- name: Run E2E tests (headed Chrome + xvfb)
run: |
xvfb-run --auto-servernum --server-args="-screen 0 1280x720x24" \
npx vitest run tests/e2e/ --reporter=verbose
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
+2
View File
@@ -2,3 +2,5 @@ node_modules/
dist/
*.tsbuildinfo
.opencli/
.mcp.json
*.log
+184 -22
View File
@@ -1,28 +1,190 @@
BSD 3-Clause License
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
Copyright (c) 2025, jackwener
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Definitions.
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to the Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by the Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding any notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
Copyright 2025 jackwener
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+93 -33
View File
@@ -1,7 +1,7 @@
# OpenCLI
> **Make any website your CLI.**
> Zero risk · Reuse Chrome login · AI-powered discovery
> Zero risk · Reuse Chrome login · AI-powered discovery · 83 commands · 19 sites
[中文文档](./README.zh-CN.md)
@@ -9,7 +9,7 @@
[![Node.js Version](https://img.shields.io/node/v/@jackwener/opencli?style=flat-square)](https://nodejs.org)
[![License](https://img.shields.io/npm/l/@jackwener/opencli?style=flat-square)](./LICENSE)
A CLI tool that turns **any website** into a command-line interface. **59 commands** across **18 sites**bilibili, zhihu, xiaohongshu, twitter, reddit, xueqiu, github, v2ex, hackernews, bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube, coupang — powered by browser session reuse and AI-native discovery.
A CLI tool that turns **any website** into a command-line interface — Bilibili, Zhihu, 小红书, Twitter/X, Reddit, YouTube, and [many more](#built-in-commands) — powered by browser session reuse and AI-native discovery.
---
@@ -21,6 +21,7 @@ A CLI tool that turns **any website** into a command-line interface. **59 comman
- [Built-in Commands](#built-in-commands)
- [Output Formats](#output-formats)
- [For AI Agents (Developer Guide)](#for-ai-agents-developer-guide)
- [Testing](#testing)
- [Troubleshooting](#troubleshooting)
- [Releasing New Versions](#releasing-new-versions)
- [License](#license)
@@ -31,8 +32,9 @@ A CLI tool that turns **any website** into a command-line interface. **59 comman
- **Account-safe** — Reuses Chrome's logged-in state; your credentials never leave the browser.
- **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies.
- **Self-healing setup** — `opencli setup` auto-discovers tokens; `opencli doctor` diagnoses config across 10+ tools; `--fix` repairs them all.
- **Dynamic Loader** — Simply drop `.ts` or `.yaml` adapters into the `clis/` folder for auto-registration.
- **Dual-Engine Architecture** — Supports both YAML declarative data pipelines and robust browser runtime typescript injections.
- **Dual-Engine Architecture** — Supports both YAML declarative data pipelines and robust browser runtime TypeScript injections.
## Prerequisites
@@ -46,11 +48,30 @@ OpenCLI connects to your browser through the Playwright MCP Bridge extension.
### Playwright MCP Bridge Extension Setup
1. Install **[Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm)** extension in Chrome.
2. Obtain your token by clicking the extension icon in the browser toolbar or from the extension settings page.
2. Run `opencli setup` — discovers the token, distributes it to your tools, and verifies connectivity:
**You must configure this token in BOTH your MCP configuration AND system environment variables.**
```bash
opencli setup
```
First, add it to your MCP client config (e.g. Claude/Cursor):
The interactive TUI will:
- 🔍 Auto-discover `PLAYWRIGHT_MCP_EXTENSION_TOKEN` from Chrome (no manual copy needed)
- ☑️ Show all detected tools (Codex, Cursor, Claude Code, Gemini CLI, etc.)
- ✏️ Update only the files you select (Space to toggle, Enter to confirm)
- 🔌 Auto-verify browser connectivity after writing configs
> **Tip**: Use `opencli doctor` for ongoing diagnosis and maintenance:
> ```bash
> opencli doctor # Read-only token & config diagnosis
> opencli doctor --live # Also test live browser connectivity
> opencli doctor --fix # Fix mismatched configs (interactive)
> opencli doctor --fix -y # Fix all configs non-interactively
> ```
<details>
<summary>Manual setup (alternative)</summary>
Add token to your MCP client config (e.g. Claude/Cursor):
```json
{
@@ -66,17 +87,13 @@ First, add it to your MCP client config (e.g. Claude/Cursor):
}
```
And, so that `opencli` commands can use it directly in the terminal, export it in your shell environment (e.g. `~/.zshrc`):
Export in shell (e.g. `~/.zshrc`):
```bash
export PLAYWRIGHT_MCP_EXTENSION_TOKEN="<your-token-here>"
```
After configuring, run `opencli doctor` to verify your token is correctly set up across all locations:
```bash
opencli doctor
```
</details>
## Quick Start
@@ -84,6 +101,7 @@ opencli doctor
```bash
npm install -g @jackwener/opencli
opencli setup # One-time: configure Playwright MCP token
```
Then use directly:
@@ -114,28 +132,50 @@ opencli list # Now you can use it anywhere!
npm install -g @jackwener/opencli@latest
```
### Auto-Completion
OpenCLI supports command auto-completion for `zsh`, `bash`, and `fish`.
For `zsh` (add to your `~/.zshrc`):
```bash
eval "$(opencli completion zsh)"
```
For `bash` (add to your `~/.bashrc`):
```bash
eval "$(opencli completion bash)"
```
For `fish` (add to your `~/.config/fish/config.fish`):
```fish
opencli completion fish | source
```
## Built-in Commands
| Site | Commands | Mode |
|------|----------|------|
| **bilibili** | `hot` `search` `me` `favorite` ... (11 commands) | 🔐 Browser |
| **zhihu** | `hot` `search` `question` | 🔐 Browser |
| **xiaohongshu** | `search` `notifications` `feed` `me` `user` | 🔐 Browser |
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `watchlist` | 🔐 Browser |
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `following` `followers` `notifications` `post` `reply` `delete` `like` | 🔐 Browser |
| **reddit** | `hot` `frontpage` `search` `subreddit` | 🔐 Browser |
| **weibo** | `hot` | 🔐 Browser |
| **boss** | `search` | 🔐 Browser |
| **coupang** | `search` `add-to-cart` | 🔐 Browser |
| **youtube** | `search` | 🔐 Browser |
| **yahoo-finance** | `quote` | 🔐 Browser |
| **reuters** | `search` | 🔐 Browser |
| **smzdm** | `search` | 🔐 Browser |
| **ctrip** | `search` | 🔐 Browser |
| **github** | `search` | 🌐 Public |
| **v2ex** | `hot` `latest` `topic` `daily` `me` `notifications` | 🌐 Public / 🔐 Browser |
| **hackernews** | `top` | 🌐 Public |
| **bbc** | `news` | 🌐 Public |
**19 sites · 80+ commands** — run `opencli list` for the live registry.
| Site | Commands | Count | Mode |
|------|----------|:-----:|------|
| **twitter** | `article` `bookmark` `bookmarks` `delete` `follow` `followers` `following` `like` `notifications` `post` `profile` `reply` `search` `thread` `timeline` `trending` `unbookmark` `unfollow` | 18 | 🔐 Browser |
| **reddit** | `comment` `frontpage` `hot` `popular` `read` `save` `saved` `search` `subreddit` `subscribe` `upvote` `upvoted` `user` `user-comments` `user-posts` | 15 | 🔐 Browser |
| **bilibili** | `dynamic` `favorite` `feed` `following` `history` `hot` `me` `ranking` `search` `subtitle` `user-videos` | 11 | 🔐 Browser |
| **v2ex** | `daily` `hot` `latest` `me` `notifications` `topic` | 6 | 🌐 / 🔐 |
| **xiaohongshu** | `feed` `me` `notifications` `search` `user` | 5 | 🔐 Browser |
| **xueqiu** | `feed` `hot` `hot-stock` `search` `stock` `watchlist` | 6 | 🔐 Browser |
| **youtube** | `search` `transcript` `video` | 3 | 🔐 Browser |
| **zhihu** | `hot` `question` `search` | 3 | 🔐 Browser |
| **boss** | `detail` `search` | 2 | 🔐 Browser |
| **coupang** | `add-to-cart` `search` | 2 | 🔐 Browser |
| **bbc** | `news` | 1 | 🌐 Public |
| **ctrip** | `search` | 1 | 🔐 Browser |
| **github** | `search` | 1 | 🌐 Public |
| **hackernews** | `top` | 1 | 🌐 Public |
| **linkedin** | `search` | 1 | 🔐 Browser |
| **reuters** | `search` | 1 | 🔐 Browser |
| **smzdm** | `search` | 1 | 🔐 Browser |
| **weibo** | `hot` | 1 | 🔐 Browser |
| **yahoo-finance** | `quote` | 1 | 🔐 Browser |
## Output Formats
@@ -176,6 +216,24 @@ opencli cascade https://api.example.com/data
Explore outputs to `.opencli/explore/<site>/` (manifest.json, endpoints.json, capabilities.json, auth.json).
## Testing
See **[TESTING.md](./TESTING.md)** for the full testing guide, including:
- Current test coverage (unit + E2E tests across 19 sites)
- How to run tests locally
- How to add tests when creating new adapters
- CI/CD pipeline with sharding
- Headless browser mode (`OPENCLI_HEADLESS=1`)
```bash
# Quick start
npm run build
npx vitest run # All tests
npx vitest run src/ # Unit tests only
npx vitest run tests/e2e/ # E2E tests
```
## Troubleshooting
- **"Failed to connect to Playwright MCP Bridge"**
@@ -185,6 +243,8 @@ Explore outputs to `.opencli/explore/<site>/` (manifest.json, endpoints.json, ca
- Your login session in Chrome might have expired. Open a normal Chrome tab, navigate to the target site, and log in or refresh the page to prove you are human.
- **Node API errors**
- Make sure you are using Node.js >= 18. Some dependencies require modern Node APIs.
- **Token issues**
- Run `opencli doctor` to diagnose token configuration across all tools.
## Releasing New Versions
@@ -198,4 +258,4 @@ The CI will automatically build, create a GitHub release, and publish to npm.
## License
[BSD-3-Clause](./LICENSE)
[Apache-2.0](./LICENSE)
+75 -33
View File
@@ -1,7 +1,7 @@
# OpenCLI
> **把任何网站变成你的命令行工具。**
> 零风控 · 复用 Chrome 登录 · AI 自动发现接口
> 零风控 · 复用 Chrome 登录 · AI 自动发现接口 · 83 命令 · 19 站点
[English](./README.md)
@@ -9,7 +9,7 @@
[![Node.js Version](https://img.shields.io/node/v/@jackwener/opencli?style=flat-square)](https://nodejs.org)
[![License](https://img.shields.io/npm/l/@jackwener/opencli?style=flat-square)](./LICENSE)
OpenCLI 将任何网站变成命令行工具。**59 个命令**覆盖 **18 个站点** — B站、知乎、小红书、Twitter、Reddit、雪球、GitHub、V2EX、Hacker News、BBC、微博、BOSS直聘、Yahoo Finance、路透社、什么值得买、携程、YouTube、Coupang — 复用浏览器登录态,AI 驱动探索。
OpenCLI 将任何网站变成命令行工具 — B站、知乎、小红书、Twitter/X、Reddit、YouTube 等 [19 个站点](#内置命令) — 复用浏览器登录态,AI 驱动探索。
---
@@ -29,8 +29,9 @@ OpenCLI 将任何网站变成命令行工具。**59 个命令**覆盖 **18 个
## 亮点
- **59 个命令,18 个站点** — B站、知乎、小红书、Twitter、Reddit、雪球(xueqiu)、GitHub、V2EX、Hacker News、BBC、微博、BOSS直聘、Yahoo Finance、路透社、什么值得买、携程、YouTube、Coupang
- **多站点覆盖** — B站、知乎、小红书、Twitter、Reddit 等 19 个站点,83 个命令
- **零风控** — 复用 Chrome 登录态,无需存储任何凭证
- **自修复配置** — `opencli setup` 自动发现 Token`opencli doctor` 诊断 10+ 工具配置;`--fix` 一键修复
- **AI 原生** — `explore` 自动发现 API`synthesize` 生成适配器,`cascade` 探测认证策略
- **动态加载引擎** — 声明式的 `.yaml` 或者底层定制的 `.ts` 适配器,放入 `clis/` 文件夹即可自动注册生效
@@ -46,11 +47,30 @@ OpenCLI 通过 Playwright MCP Bridge 扩展与你的浏览器通信。
### Playwright MCP Bridge 扩展配置
1. 安装 **[Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm)** 扩展
2. 在浏览器插件栏点击该插件,或者在插件设置页获取你的 Extension Token。
2. 运行 `opencli setup` — 自动发现 Token、分发到各工具、验证连通性:
**你必须将这个 Token 同时配置到你的 MCP 配置文件 AND 环境变量中。**
```bash
opencli setup
```
首先,配置你的 MCP 客户端(如 Claude/Cursor 等)
交互式 TUI 会
- 🔍 从 Chrome 自动发现 `PLAYWRIGHT_MCP_EXTENSION_TOKEN`(无需手动复制)
- ☑️ 显示所有支持的工具(Codex、Cursor、Claude Code、Gemini CLI 等)
- ✏️ 只更新你选中的文件(空格切换,回车确认)
- 🔌 完成后自动验证浏览器连通性
> **Tip**:后续诊断和维护用 `opencli doctor`
> ```bash
> opencli doctor # 只读 Token 与配置诊断
> opencli doctor --live # 额外测试浏览器连通性
> opencli doctor --fix # 修复不一致的配置(交互确认)
> opencli doctor --fix -y # 无交互直接修复所有配置
> ```
<details>
<summary>手动配置(备选方案)</summary>
配置你的 MCP 客户端(如 Claude/Cursor 等):
```json
{
@@ -66,17 +86,13 @@ OpenCLI 通过 Playwright MCP Bridge 扩展与你的浏览器通信。
}
```
并且,为了让 `opencli` 命令行也能直接使用它,你必须在你的终端系统环境变量中导出(建议写进 `~/.zshrc``~/.bashrc`):
在终端环境变量中导出(建议写进 `~/.zshrc`):
```bash
export PLAYWRIGHT_MCP_EXTENSION_TOKEN="<你的-token>"
```
配置完成后,运行 `opencli doctor` 检测你的 Token 是否在所有位置都正确配置:
```bash
opencli doctor
```
</details>
## 快速开始
@@ -84,6 +100,7 @@ opencli doctor
```bash
npm install -g @jackwener/opencli
opencli setup # 首次使用:配置 Playwright MCP token
```
直接使用:
@@ -114,28 +131,50 @@ opencli list # 可以在任何地方使用了!
npm install -g @jackwener/opencli@latest
```
### 命令自动补全
OpenCLI 支持 `zsh``bash``fish` 的命令自动补全参数与提示。
对于 `zsh` (添加至 `~/.zshrc`):
```bash
eval "$(opencli completion zsh)"
```
对于 `bash` (添加至 `~/.bashrc`):
```bash
eval "$(opencli completion bash)"
```
对于 `fish` (添加至 `~/.config/fish/config.fish`):
```fish
opencli completion fish | source
```
## 内置命令
| 站点 | 命令 | 模式 |
|------|------|------|
| **bilibili** | `hot` `search` `me` `favorite` ...(共11个) | 🔐 浏览器 |
| **zhihu** | `hot` `search` `question` | 🔐 浏览器 |
| **xiaohongshu** | `search` `notifications` `feed` `me` `user` | 🔐 浏览器 |
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `watchlist` | 🔐 浏览器 |
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `following` `followers` `notifications` `post` `reply` `delete` `like` | 🔐 浏览器 |
| **reddit** | `hot` `frontpage` `search` `subreddit` | 🔐 浏览器 |
| **weibo** | `hot` | 🔐 浏览器 |
| **boss** | `search` | 🔐 浏览器 |
| **coupang** | `search` `add-to-cart` | 🔐 浏览器 |
| **youtube** | `search` | 🔐 浏览器 |
| **yahoo-finance** | `quote` | 🔐 浏览器 |
| **reuters** | `search` | 🔐 浏览器 |
| **smzdm** | `search` | 🔐 浏览器 |
| **ctrip** | `search` | 🔐 浏览器 |
| **github** | `search` | 🌐 公共 API |
| **v2ex** | `hot` `latest` `topic` `daily` `me` `notifications` | 🌐 公共 API / 🔐 浏览器 |
| **hackernews** | `top` | 🌐 公共 API |
| **bbc** | `news` | 🌐 公共 API |
**19 个站点 · 80+ 命令** — 运行 `opencli list` 查看完整注册表。
| 站点 | 命令 | 数量 | 模式 |
|------|------|:----:|------|
| **twitter** | `article` `bookmark` `bookmarks` `delete` `follow` `followers` `following` `like` `notifications` `post` `profile` `reply` `search` `thread` `timeline` `trending` `unbookmark` `unfollow` | 18 | 🔐 浏览器 |
| **reddit** | `comment` `frontpage` `hot` `popular` `read` `save` `saved` `search` `subreddit` `subscribe` `upvote` `upvoted` `user` `user-comments` `user-posts` | 15 | 🔐 浏览器 |
| **bilibili** | `dynamic` `favorite` `feed` `following` `history` `hot` `me` `ranking` `search` `subtitle` `user-videos` | 11 | 🔐 浏览器 |
| **v2ex** | `daily` `hot` `latest` `me` `notifications` `topic` | 6 | 🌐 / 🔐 |
| **xiaohongshu** | `feed` `me` `notifications` `search` `user` | 5 | 🔐 浏览器 |
| **xueqiu** | `feed` `hot` `hot-stock` `search` `stock` `watchlist` | 6 | 🔐 浏览器 |
| **youtube** | `search` `transcript` `video` | 3 | 🔐 浏览器 |
| **zhihu** | `hot` `question` `search` | 3 | 🔐 浏览器 |
| **boss** | `detail` `search` | 2 | 🔐 浏览器 |
| **coupang** | `add-to-cart` `search` | 2 | 🔐 浏览器 |
| **bbc** | `news` | 1 | 🌐 公共 API |
| **ctrip** | `search` | 1 | 🔐 浏览器 |
| **github** | `search` | 1 | 🌐 公共 API |
| **hackernews** | `top` | 1 | 🌐 公共 API |
| **linkedin** | `search` | 1 | 🔐 浏览器 |
| **reuters** | `search` | 1 | 🔐 浏览器 |
| **smzdm** | `search` | 1 | 🔐 浏览器 |
| **weibo** | `hot` | 1 | 🔐 浏览器 |
| **yahoo-finance** | `quote` | 1 | 🔐 浏览器 |
## 输出格式
@@ -185,6 +224,9 @@ opencli cascade https://api.example.com/data
- Chrome 里的登录态可能已经过期(甚至被要求过滑动验证码)。请打开当前 Chrome 页面,在新标签页重新手工登录或刷新该页面。
- **Node API 错误 (如 parseArgs, fs 等)**
- 确保 Node.js 版本 `>= 18`。旧版不支持我们使用的现代核心库 API。
- **Token 问题**
- 运行 `opencli doctor` 诊断所有工具的 Token 配置状态。
- 使用 `opencli doctor --live` 测试浏览器连通性。
## 版本发布
@@ -198,4 +240,4 @@ git push --follow-tags
## License
[BSD-3-Clause](./LICENSE)
[Apache-2.0](./LICENSE)
+97 -58
View File
@@ -1,9 +1,9 @@
---
name: opencli
description: "OpenCLI — Make any website your CLI. Zero risk, AI-powered, reuse Chrome login."
version: 0.5.1
description: "OpenCLI — Make any website your CLI. Zero risk, AI-powered, reuse Chrome login. 83 commands across 19 sites."
version: 0.7.3
author: jackwener
tags: [cli, browser, web, mcp, playwright, bilibili, zhihu, twitter, github, v2ex, hackernews, reddit, xiaohongshu, xueqiu, AI, agent]
tags: [cli, browser, web, mcp, playwright, bilibili, zhihu, twitter, github, v2ex, hackernews, reddit, xiaohongshu, xueqiu, youtube, boss, coupang, ctrip, reuters, smzdm, weibo, yahoo-finance, bbc, linkedin, AI, agent]
---
# OpenCLI
@@ -34,7 +34,8 @@ npm update -g @jackwener/opencli
Browser commands require:
1. Chrome browser running **(logged into target sites)**
2. [Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm) extension installed and configured
2. [Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm) extension installed
3. Run `opencli setup` to auto-discover token and configure all tools
> **Note**: You must be logged into the target website in Chrome before running commands. Tabs opened during command execution are auto-closed afterwards.
@@ -46,89 +47,122 @@ Public API commands (`hackernews`, `github search`, `v2ex`) need no browser.
```bash
# Bilibili (browser)
opencli bilibili hot --limit 10 # B站热门视频
opencli bilibili search --keyword "rust" # 搜索视频
opencli bilibili me # 我的信息
opencli bilibili favorite # 我的收藏
opencli bilibili history --limit 20 # 观看历史
opencli bilibili feed --limit 10 # 动态时间线
opencli bilibili user-videos --uid 12345 # 用户投稿
opencli bilibili subtitle --bvid BV1xxx # 获取视频字幕 (支持 --lang zh-CN)
opencli bilibili dynamic --limit 10 # 动态
opencli bilibili ranking --limit 10 # 排行榜
opencli bilibili following --limit 20 # 我的关注列表 (支持 --uid 查看他人)
opencli bilibili dynamic # Get Bilibili user dynamic feed
opencli bilibili favorite # 我的默认收藏夹
opencli bilibili feed # 关注的人的动态时间线
opencli bilibili following # 获取 Bilibili 用户的关注列表
opencli bilibili history # 我的观看历史
opencli bilibili hot # B站热门视频
opencli bilibili me # My Bilibili profile info
opencli bilibili ranking # Get Bilibili video ranking board
opencli bilibili search # Search Bilibili videos or users
opencli bilibili subtitle # 获取 Bilibili 视频的字幕
opencli bilibili user-videos # 查看指定用户的投稿视频
# 知乎 (browser)
opencli zhihu hot --limit 10 # 知乎热榜
opencli zhihu search --keyword "AI" # 搜索
opencli zhihu question --id 34816524 # 问题详情和回答
opencli zhihu hot # 知乎热榜
opencli zhihu question # 知乎问题详情和回答
opencli zhihu search # 知乎搜索
# 小红书 (browser)
opencli xiaohongshu search --keyword "美食" # 搜索笔记
opencli xiaohongshu notifications # 通知(mentions/likes/connections
opencli xiaohongshu feed --limit 10 # 推荐 Feed
opencli xiaohongshu me # 我的信息
opencli xiaohongshu user --uid xxx # 用户主页
opencli xiaohongshu feed # 小红书首页推荐 Feed (via Pinia Store Action)
opencli xiaohongshu me # 我的小红书个人信息
opencli xiaohongshu notifications # 小红书通知 (mentions/likes/connections)
opencli xiaohongshu search # 搜索小红书笔记
opencli xiaohongshu user # Get user notes from Xiaohongshu
# 雪球 Xueqiu (browser)
opencli xueqiu hot-stock --limit 10 # 雪球热门股票榜
opencli xueqiu stock --symbol SH600519 # 查看股票实时行情
opencli xueqiu watchlist # 获取自选股/持仓列表
opencli xueqiu feed # 我的关注 timeline
opencli xueqiu hot --limit 10 # 雪球热榜
opencli xueqiu search --keyword "特斯拉" # 搜索
opencli xueqiu feed # 获取雪球首页时间线(关注用户的动态)
opencli xueqiu hot # 获取雪球热门动态
opencli xueqiu hot-stock # 获取雪球热门股票榜
opencli xueqiu search # 搜索雪球股票(代码或名称)
opencli xueqiu stock # 获取雪球股票实时行情
opencli xueqiu watchlist # 获取雪球自选股列表
# GitHub (public)
opencli github search --keyword "cli" # 搜索仓库
opencli github search # Search GitHub repositories
# Twitter/X (browser)
opencli twitter trending --limit 10 # 热门话题
opencli twitter bookmarks --limit 20 # 获取收藏的书签推文
opencli twitter search --keyword "AI" # 搜索推文
opencli twitter profile --username elonmusk # 用户资料
opencli twitter timeline --limit 20 # 时间线
# Twitter/X (browser & ui)
opencli twitter article # Fetch a Twitter Article (long-form content) and export as Markdown
opencli twitter bookmark # Bookmark a tweet [UI]
opencli twitter bookmarks # 获取 Twitter 书签列表
opencli twitter delete # Delete a specific tweet by URL [UI]
opencli twitter follow # Follow a Twitter user [UI]
opencli twitter followers # Get accounts following a Twitter/X user
opencli twitter following # Get accounts a Twitter/X user is following
opencli twitter like # Like a specific tweet [UI]
opencli twitter notifications # Get Twitter/X notifications
opencli twitter post # Post a new tweet/thread [UI]
opencli twitter profile # Fetch a Twitter user profile (bio, stats, etc.)
opencli twitter reply # Reply to a specific tweet [UI]
opencli twitter search # Search Twitter/X for tweets
opencli twitter thread # Get a tweet thread (original + all replies)
opencli twitter timeline # Twitter Home Timeline
opencli twitter trending # Twitter/X trending topics
opencli twitter unbookmark # Remove a tweet from bookmarks [UI]
opencli twitter unfollow # Unfollow a Twitter user [UI]
# Reddit (browser)
opencli reddit hot --limit 10 # 热门帖子
opencli reddit hot --subreddit programming # 指定子版块
opencli reddit frontpage --limit 10 # 首页
opencli reddit search --keyword "AI" # 搜索
opencli reddit subreddit --name rust # 子版块浏览
# Reddit (browser & cookie)
opencli reddit comment # Post a comment on a Reddit post
opencli reddit frontpage # Reddit Frontpage / r/all
opencli reddit hot # Reddit 热门帖子
opencli reddit popular # Reddit Popular posts (/r/popular)
opencli reddit read # Read a Reddit post and its comments
opencli reddit save # Save or unsave a Reddit post
opencli reddit saved # Browse your saved Reddit posts
opencli reddit search # Search Reddit Posts
opencli reddit subreddit # Get posts from a specific Subreddit
opencli reddit subscribe # Subscribe or unsubscribe to a subreddit
opencli reddit upvote # Upvote or downvote a Reddit post
opencli reddit upvoted # Browse your upvoted Reddit posts
opencli reddit user # View a Reddit user profile
opencli reddit user-comments # View a Reddit user's comment history
opencli reddit user-posts # View a Reddit user's submitted posts
# V2EX (public + browser)
opencli v2ex hot --limit 10 # 热门话题
opencli v2ex latest --limit 10 # 最新话题
opencli v2ex topic --id 1024 # 主题详情
opencli v2ex daily # 每日签到 (browser)
opencli v2ex me # 我的信息 (browser)
opencli v2ex notifications --limit 10 # 通知 (browser)
# V2EX (public & cookie)
opencli v2ex daily # V2EX 每日签到并领取铜币
opencli v2ex hot # V2EX 热门话题
opencli v2ex latest # V2EX 最新话题
opencli v2ex me # V2EX 获取个人资料 (余额/未读提醒)
opencli v2ex notifications # V2EX 获取提醒 (回复/由于)
opencli v2ex topic # V2EX 主题详情和回复
# Hacker News (public)
opencli hackernews top --limit 10 # Top stories
opencli hackernews top # Hacker News top stories
# BBC (public)
opencli bbc news --limit 10 # BBC News RSS headlines
opencli bbc news # BBC News headlines (RSS)
# 微博 (browser)
opencli weibo hot --limit 10 # 微博热搜
opencli weibo hot # 微博热搜
# BOSS直聘 (browser)
opencli boss search --query "AI agent" # 搜索职位
opencli boss detail # BOSS直聘查看职位详情
opencli boss search # BOSS直聘搜索职位
# YouTube (browser)
opencli youtube search --query "rust" # 搜索视频
opencli youtube search # Search YouTube videos
opencli youtube transcript # Get YouTube video transcript/subtitles
opencli youtube video # Get YouTube video metadata (title, views, description, etc.)
# Yahoo Finance (browser)
opencli yahoo-finance quote --symbol AAPL # 股票行情
opencli yahoo-finance quote # Yahoo Finance 股票行情
# Reuters (browser)
opencli reuters search --query "AI" # 路透社搜索
opencli reuters search # Reuters 路透社新闻搜索
# 什么值得买 (browser)
opencli smzdm search --keyword "耳机" # 搜索好价
opencli smzdm search # 什么值得买搜索好价
# 携程 (browser)
opencli ctrip search --query "三亚" # 搜索目的地
opencli ctrip search # 携程旅行搜索
# Coupang (browser)
opencli coupang search # Search Coupang products with logged-in browser session
opencli coupang add-to-cart # Add a Coupang product to cart using logged-in browser session
# LinkedIn (header)
opencli linkedin search # Search LinkedIn
```
### Management Commands
@@ -139,6 +173,11 @@ opencli list --json # JSON output
opencli list -f yaml # YAML output
opencli validate # Validate all CLI definitions
opencli validate bilibili # Validate specific site
opencli setup # Interactive token setup (auto-discover + TUI checkbox)
opencli doctor # Diagnose token & extension config across all tools
opencli doctor --live # Also test live browser connectivity
opencli doctor --fix # Fix mismatched configs (interactive confirmation)
opencli doctor --fix -y # Fix all configs non-interactively
```
### AI Agent Workflow
+233
View File
@@ -0,0 +1,233 @@
# Testing Guide
> 面向开发者和 AI Agent 的测试参考手册。
## 目录
- [测试架构](#测试架构)
- [当前覆盖范围](#当前覆盖范围)
- [本地运行测试](#本地运行测试)
- [如何添加新测试](#如何添加新测试)
- [CI/CD 流水线](#cicd-流水线)
- [浏览器模式](#浏览器模式)
- [站点兼容性](#站点兼容性)
---
## 测试架构
测试分为三层,全部使用 **vitest** 运行:
```
tests/
├── e2e/ # E2E 集成测试(子进程运行真实 CLI)
│ ├── helpers.ts # runCli() 共享工具
│ ├── public-commands.test.ts # 公开 API 命令(无需浏览器)
│ ├── browser-public.test.ts # 浏览器命令(公开数据)
│ ├── browser-auth.test.ts # 需登录命令(graceful failure 测试)
│ ├── management.test.ts # 管理命令(list, validate, verify, help
│ └── output-formats.test.ts # 输出格式(json/yaml/csv/md
├── smoke/ # 烟雾测试(仅定时 / 手动触发)
│ └── api-health.test.ts # 外部 API 可用性检测
src/
├── *.test.ts # 单元测试(已有 8 个)
```
| 层 | 位置 | 运行方式 | 用途 |
|---|---|---|---|
| 单元测试 | `src/**/*.test.ts` | `npx vitest run src/` | 内部模块逻辑 |
| E2E 测试 | `tests/e2e/*.test.ts` | `npx vitest run tests/e2e/` | 真实 CLI 命令执行 |
| 烟雾测试 | `tests/smoke/*.test.ts` | `npx vitest run tests/smoke/` | 外部 API 健康 |
---
## 当前覆盖范围
### 单元测试(8 个文件)
| 文件 | 覆盖内容 |
|---|---|
| `browser.test.ts` | JSON-RPC、tab 管理、extension/standalone 模式切换 |
| `engine.test.ts` | 命令发现与执行 |
| `registry.test.ts` | 命令注册与策略分配 |
| `output.test.ts` | 输出格式渲染 |
| `doctor.test.ts` | Token 诊断 |
| `coupang.test.ts` | 数据归一化 |
| `pipeline/template.test.ts` | 模板表达式求值 |
| `pipeline/transform.test.ts` | 数据变换步骤 |
### E2E 测试(~52 个用例)
| 文件 | 覆盖站点/功能 | 测试数 |
|---|---|---|
| `public-commands.test.ts` | hackernews/top, v2ex/hot, v2ex/latest, v2ex/topic | 5 |
| `browser-public.test.ts` | bbc, bilibili×3, weibo, zhihu×2, reddit×2, twitter, xueqiu×2, reuters, youtube, smzdm, boss, ctrip, coupang, xiaohongshu, yahoo-finance, v2ex/daily | 21 |
| `browser-auth.test.ts` | bilibili/me,dynamic,favorite,history,following + twitter/bookmarks,timeline,notifications + v2ex/me,notifications + xueqiu/feed,watchlist + xiaohongshu/feed,notifications | 14 |
| `management.test.ts` | list×5 格式, validate×3 级别, verify, --version, --help, unknown cmd | 12 |
| `output-formats.test.ts` | json, yaml, csv, md 格式验证 | 5 |
### 烟雾测试
公开 API 可用性(hackernews, v2ex×2, v2ex/topic+ 全站点注册完整性检查。
---
## 本地运行测试
### 前置条件
```bash
npm ci # 安装依赖
npm run build # 编译(E2E 测试需要 dist/main.js
```
### 运行命令
```bash
# 全部单元测试
npx vitest run src/
# 全部 E2E 测试(会真实调用外部 API)
npx vitest run tests/e2e/
# 单个测试文件
npx vitest run tests/e2e/management.test.ts
# 全部测试(单元 + E2E
npx vitest run
# 烟雾测试
npx vitest run tests/smoke/
# watch 模式(开发时推荐)
npx vitest src/
```
### 浏览器命令本地测试须知
-`PLAYWRIGHT_MCP_EXTENSION_TOKEN` 时,opencli 自动启动一个独立浏览器实例
- `browser-public.test.ts` 使用 `tryBrowserCommand()`,站点反爬导致空数据时 warn + pass
- `browser-auth.test.ts` 验证 **graceful failure**(不 crash 不 hang 即通过)
- 如需测试完整登录态,保持 Chrome 登录态 + 设置 `PLAYWRIGHT_MCP_EXTENSION_TOKEN`,手动跑对应测试
---
## 如何添加新测试
### 新增 YAML Adapter(如 `src/clis/producthunt/trending.yaml`
1. **无需额外操作**`validate` 测试会自动覆盖 YAML 结构验证
2. 根据 adapter 类型,在对应文件加一个 `it()` block
```typescript
// 如果 browser: false(公开 API)→ tests/e2e/public-commands.test.ts
it('producthunt trending returns data', async () => {
const { stdout, code } = await runCli(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
```
```typescript
// 如果 browser: true 但可公开访问 → tests/e2e/browser-public.test.ts
it('producthunt trending returns data', async () => {
const data = await tryBrowserCommand(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'producthunt trending');
}, 60_000);
```
```typescript
// 如果 browser: true 且需登录 → tests/e2e/browser-auth.test.ts
it('producthunt me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['producthunt', 'me', '-f', 'json'], 'producthunt me');
}, 60_000);
```
### 新增管理命令(如 `opencli export`
`tests/e2e/management.test.ts` 添加测试。
### 新增内部模块
`src/` 下对应位置创建 `*.test.ts`
### 决策流程图
```
新增功能 → 是内部模块? → 是 → src/ 下加 *.test.ts
↓ 否
是 CLI 命令? → browser: false? → tests/e2e/public-commands.test.ts
↓ true
公开数据? → tests/e2e/browser-public.test.ts
↓ 需登录
tests/e2e/browser-auth.test.ts
```
---
## CI/CD 流水线
### ci.yml(主流水线)
| Job | 触发条件 | 内容 |
|---|---|---|
| **build** | push/PR to main,dev | typecheck + build |
| **unit-test** | push/PR to main,dev | 单元测试,2 shard 并行 |
| **smoke-test** | 每周一 08:00 UTC / 手动 | xvfb + real Chrome,外部 API 健康检查 |
### e2e-headed.ymlE2E 测试)
| Job | 触发条件 | 内容 |
|---|---|---|
| **e2e-headed** | push/PR to main,dev | xvfb + real Chrome,全部 E2E 测试 |
E2E 使用 `browser-actions/setup-chrome` 安装真实 Chrome,配合 `xvfb-run` 提供虚拟显示器,以 headed 模式运行浏览器。
### Sharding
单元测试使用 vitest 内置 shard
```yaml
strategy:
matrix:
shard: [1, 2]
steps:
- run: npx vitest run src/ --shard=${{ matrix.shard }}/2
```
---
## 浏览器模式
opencli 根据 `PLAYWRIGHT_MCP_EXTENSION_TOKEN` 环境变量自动选择模式:
| 条件 | 模式 | MCP 参数 | 使用场景 |
|---|---|---|---|
| Token 已设置 | Extension 模式 | `--extension` | 本地用户,连接已登录的 Chrome |
| Token 未设置 | Standalone 模式 | (无特殊 flag) | CI 或无扩展环境,自启浏览器 |
CI 中使用 `OPENCLI_BROWSER_EXECUTABLE_PATH` 指定真实 Chrome 路径:
```yaml
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
```
---
## 站点兼容性
在 GitHub Actions 美国 runner 上,部分站点因地域限制或登录要求返回空数据。E2E 测试对这些站点使用 warn + pass 策略,不影响 CI 绿灯。
| 站点 | CI 状态 | 限制原因 |
|---|---|---|
| hackernews, bbc, v2ex | ✅ 返回数据 | 无限制 |
| yahoo-finance | ✅ 返回数据 | 无限制 |
| bilibili, zhihu, weibo, xiaohongshu | ⚠️ 空数据 | 地域限制(中国站点) |
| reddit, twitter, youtube | ⚠️ 空数据 | 需登录或 cookie |
| smzdm, boss, ctrip, coupang, xueqiu | ⚠️ 空数据 | 地域限制 / 需登录 |
> 使用 self-hosted runner(国内服务器)可解决地域限制问题。
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@jackwener/opencli",
"version": "0.5.2",
"version": "0.7.5",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@jackwener/opencli",
"version": "0.5.2",
"version": "0.7.5",
"license": "BSD-3-Clause",
"dependencies": {
"chalk": "^5.3.0",
+5 -5
View File
@@ -1,6 +1,6 @@
{
"name": "@jackwener/opencli",
"version": "0.5.2",
"version": "0.7.5",
"publishConfig": {
"access": "public"
},
@@ -16,9 +16,9 @@
"scripts": {
"dev": "tsx src/main.ts",
"build": "tsc && npm run clean-yaml && npm run copy-yaml && npm run build-manifest",
"build-manifest": "node dist/build-manifest.js || true",
"clean-yaml": "find dist/clis -name '*.yaml' -o -name '*.yml' 2>/dev/null | xargs rm -f",
"copy-yaml": "find src/clis -name '*.yaml' -o -name '*.yml' | while read f; do d=\"dist/${f#src/}\"; mkdir -p \"$(dirname \"$d\")\"; cp \"$f\" \"$d\"; done",
"build-manifest": "node dist/build-manifest.js",
"clean-yaml": "node -e \"const{readdirSync:r,rmSync:d,existsSync:e,statSync:s}=require('fs'),p=require('path');function w(dir){if(!e(dir))return;for(const f of r(dir)){const fp=p.join(dir,f);s(fp).isDirectory()?w(fp):/\\.ya?ml$/.test(f)&&d(fp)}}w('dist/clis')\"",
"copy-yaml": "node -e \"const{readdirSync:r,copyFileSync:c,mkdirSync:m,existsSync:e,statSync:s}=require('fs'),p=require('path');function w(src,dst){if(!e(src))return;for(const f of r(src)){const sp=p.join(src,f),dp=p.join(dst,f);s(sp).isDirectory()?w(sp,dp):/\\.ya?ml$/.test(f)&&(m(p.dirname(dp),{recursive:!0}),c(sp,dp))}}w('src/clis','dist/clis')\"",
"start": "node dist/main.js",
"typecheck": "tsc --noEmit",
"lint": "tsc --noEmit",
@@ -34,7 +34,7 @@
"playwright"
],
"author": "jackwener",
"license": "BSD-3-Clause",
"license": "Apache-2.0",
"repository": {
"type": "git",
"url": "git+https://github.com/jackwener/opencli.git"
+2 -2
View File
@@ -84,10 +84,10 @@ export async function apiGet(
}
export async function fetchJson(page: IPage, url: string): Promise<any> {
const escapedUrl = url.replace(/"/g, '\\"');
const urlJs = JSON.stringify(url);
return page.evaluate(`
async () => {
const res = await fetch("${escapedUrl}", { credentials: "include" });
const res = await fetch(${urlJs}, { credentials: "include" });
return await res.json();
}
`);
+54 -16
View File
@@ -49,23 +49,61 @@ describe('browser helpers', () => {
expect(__test__.appendLimited('12345', '67890', 8)).toBe('34567890');
});
it('builds Playwright MCP args with kebab-case executable path', () => {
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
executablePath: '/mnt/c/Program Files/Google/Chrome/Application/chrome.exe',
})).toEqual([
'/tmp/cli.js',
'--extension',
'--executable-path',
'/mnt/c/Program Files/Google/Chrome/Application/chrome.exe',
]);
it('builds extension MCP args in local mode (no CI)', () => {
const savedCI = process.env.CI;
delete process.env.CI;
try {
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
executablePath: '/mnt/c/Program Files/Google/Chrome/Application/chrome.exe',
})).toEqual([
'/tmp/cli.js',
'--extension',
'--executable-path',
'/mnt/c/Program Files/Google/Chrome/Application/chrome.exe',
]);
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
})).toEqual([
'/tmp/cli.js',
'--extension',
]);
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
})).toEqual([
'/tmp/cli.js',
'--extension',
]);
} finally {
if (savedCI !== undefined) {
process.env.CI = savedCI;
} else {
delete process.env.CI;
}
}
});
it('builds standalone MCP args in CI mode', () => {
const savedCI = process.env.CI;
process.env.CI = 'true';
try {
// CI mode: no --extension — browser launches in standalone headed mode
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
})).toEqual([
'/tmp/cli.js',
]);
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
executablePath: '/usr/bin/chromium',
})).toEqual([
'/tmp/cli.js',
'--executable-path',
'/usr/bin/chromium',
]);
} finally {
if (savedCI !== undefined) {
process.env.CI = savedCI;
} else {
delete process.env.CI;
}
}
});
it('times out slow promises', async () => {
+13 -7
View File
@@ -13,9 +13,7 @@ import { formatSnapshot } from './snapshotFormatter.js';
import { PKG_VERSION } from './version.js';
import { normalizeEvaluateSource } from './pipeline/template.js';
import { generateInterceptorJs, generateReadInterceptedJs } from './interceptor.js';
import { withTimeoutMs } from './runtime.js';
const CONNECT_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_CONNECT_TIMEOUT ?? '30', 10);
import { withTimeoutMs, DEFAULT_BROWSER_CONNECT_TIMEOUT } from './runtime.js';
const STDERR_BUFFER_LIMIT = 16 * 1024;
const INITIAL_TABS_TIMEOUT_MS = 1500;
const TAB_CLEANUP_TIMEOUT_MS = 2000;
@@ -214,7 +212,7 @@ export class Page implements IPage {
return this.call('tools/call', { name: 'browser_console_messages', arguments: { level } });
}
async scroll(direction: string = 'down', amount: number = 500): Promise<void> {
async scroll(direction: string = 'down', _amount: number = 500): Promise<void> {
await this.call('tools/call', { name: 'browser_press_key', arguments: { key: direction === 'down' ? 'PageDown' : 'PageUp' } });
}
@@ -344,11 +342,12 @@ export class PlaywrightMCP {
PlaywrightMCP._registerGlobalCleanup();
PlaywrightMCP._activeInsts.add(this);
this._state = 'connecting';
const timeout = opts.timeout ?? CONNECT_TIMEOUT;
const timeout = opts.timeout ?? DEFAULT_BROWSER_CONNECT_TIMEOUT;
return new Promise<Page>((resolve, reject) => {
const isDebug = process.env.DEBUG?.includes('opencli:mcp');
const debugLog = (msg: string) => isDebug && console.error(`[opencli:mcp] ${msg}`);
const useExtension = !!process.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN;
const extensionToken = process.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN;
const tokenFingerprint = getTokenFingerprint(extensionToken);
let stderrBuffer = '';
@@ -392,7 +391,8 @@ export class PlaywrightMCP {
executablePath: process.env.OPENCLI_BROWSER_EXECUTABLE_PATH,
});
if (process.env.OPENCLI_VERBOSE) {
console.error(`[opencli] Extension token: ${extensionToken ? `configured (fingerprint ${tokenFingerprint})` : 'missing'}`);
console.error(`[opencli] Mode: ${useExtension ? 'extension' : 'standalone'}`);
if (useExtension) console.error(`[opencli] Extension token: fingerprint ${tokenFingerprint}`);
}
debugLog(`Spawning node ${mcpArgs.join(' ')}`);
@@ -610,7 +610,13 @@ function appendLimited(current: string, chunk: string, limit: number): string {
}
function buildMcpArgs(input: { mcpPath: string; executablePath?: string | null }): string[] {
const args = [input.mcpPath, '--extension'];
const args = [input.mcpPath];
if (!process.env.CI) {
// Local: always connect to user's running Chrome via MCP Bridge extension
args.push('--extension');
}
// CI: standalone mode — @playwright/mcp launches its own browser (headed by default).
// xvfb provides a virtual display for headed mode in GitHub Actions.
if (input.executablePath) {
args.push('--executable-path', input.executablePath);
}
+3
View File
@@ -30,6 +30,7 @@ interface ManifestEntry {
type?: string;
default?: any;
required?: boolean;
positional?: boolean;
help?: string;
choices?: string[];
}>;
@@ -140,6 +141,7 @@ function scanTs(filePath: string, site: string): ManifestEntry {
const defaultMatch = body.match(/default\s*:\s*([^,}]+)/);
const requiredMatch = body.match(/required\s*:\s*(true|false)/);
const helpMatch = body.match(/help\s*:\s*['"`]([^'"`]*)['"`]/);
const positionalMatch = body.match(/positional\s*:\s*(true|false)/);
let defaultVal: any = undefined;
if (defaultMatch) {
@@ -156,6 +158,7 @@ function scanTs(filePath: string, site: string): ManifestEntry {
type: typeMatch?.[1] ?? 'str',
default: defaultVal,
required: requiredMatch?.[1] === 'true',
positional: positionalMatch?.[1] === 'true' || undefined,
help: helpMatch?.[1] ?? '',
});
}
+115
View File
@@ -0,0 +1,115 @@
/**
* BOSS直聘 job detail — fetch full job posting details via browser cookie API.
*
* Uses securityId from search results to call the detail API.
* Returns: job description, skills, welfare, boss info, company info, address.
*/
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
cli({
site: 'boss',
name: 'detail',
description: 'BOSS直聘查看职位详情',
domain: 'www.zhipin.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'security_id', required: true, help: 'Security ID from search results (securityId field)' },
],
columns: [
'name', 'salary', 'experience', 'degree', 'city', 'district',
'description', 'skills', 'welfare',
'boss_name', 'boss_title', 'active_time',
'company', 'industry', 'scale', 'stage',
'address', 'url',
],
func: async (page: IPage | null, kwargs) => {
if (!page) throw new Error('Browser page required');
const securityId = kwargs.security_id;
// Navigate to zhipin.com first to establish cookie context (referrer + cookies)
await page.goto('https://www.zhipin.com/web/geek/job');
await page.wait({ time: 1 });
const targetUrl = `https://www.zhipin.com/wapi/zpgeek/job/detail.json?securityId=${encodeURIComponent(securityId)}`;
if (process.env.OPENCLI_VERBOSE || process.env.DEBUG?.includes('opencli')) {
console.error(`[opencli:boss] Fetching job detail...`);
}
const evaluateScript = `
async () => {
return new Promise((resolve, reject) => {
const xhr = new window.XMLHttpRequest();
xhr.open('GET', ${JSON.stringify(targetUrl)}, true);
xhr.withCredentials = true;
xhr.timeout = 15000;
xhr.setRequestHeader('Accept', 'application/json, text/plain, */*');
xhr.onload = () => {
if (xhr.status >= 200 && xhr.status < 300) {
try {
resolve(JSON.parse(xhr.responseText));
} catch (e) {
reject(new Error('Failed to parse JSON. Raw (200 chars): ' + xhr.responseText.substring(0, 200)));
}
} else {
reject(new Error('XHR HTTP Status: ' + xhr.status));
}
};
xhr.onerror = () => reject(new Error('XHR Network Error'));
xhr.ontimeout = () => reject(new Error('XHR Timeout'));
xhr.send();
});
}
`;
let data: any;
try {
data = await page.evaluate(evaluateScript);
} catch (e: any) {
throw new Error('API evaluate failed: ' + e.message);
}
if (data.code !== 0) {
if (data.code === 37) {
throw new Error('Cookie 已过期!请在当前 Chrome 浏览器中重新登录 BOSS 直聘。');
}
throw new Error(`BOSS API error: ${data.message || 'Unknown'} (code=${data.code})`);
}
const zpData = data.zpData || {};
const jobInfo = zpData.jobInfo || {};
const bossInfo = zpData.bossInfo || {};
const brandComInfo = zpData.brandComInfo || {};
if (!jobInfo.jobName) {
throw new Error('该职位信息不存在或已下架');
}
return [{
name: jobInfo.jobName || '',
salary: jobInfo.salaryDesc || '',
experience: jobInfo.experienceName || '',
degree: jobInfo.degreeName || '',
city: jobInfo.locationName || '',
district: [jobInfo.areaDistrict, jobInfo.businessDistrict].filter(Boolean).join('·'),
description: jobInfo.postDescription || '',
skills: (jobInfo.showSkills || []).join(', '),
welfare: (brandComInfo.labels || []).join(', '),
boss_name: bossInfo.name || '',
boss_title: bossInfo.title || '',
active_time: bossInfo.activeTimeDesc || '',
company: brandComInfo.brandName || bossInfo.brandName || '',
industry: brandComInfo.industryName || '',
scale: brandComInfo.scaleName || '',
stage: brandComInfo.stageName || '',
address: jobInfo.address || '',
url: jobInfo.encryptId
? 'https://www.zhipin.com/job_detail/' + jobInfo.encryptId + '.html'
: '',
}];
},
});
+2 -1
View File
@@ -81,7 +81,7 @@ cli({
{ name: 'page', type: 'int', default: 1, help: 'Page number' },
{ name: 'limit', type: 'int', default: 15, help: 'Number of results' },
],
columns: ['name', 'salary', 'company', 'area', 'experience', 'degree', 'skills', 'boss', 'url'],
columns: ['name', 'salary', 'company', 'area', 'experience', 'degree', 'skills', 'boss', 'security_id', 'url'],
func: async (page: IPage | null, kwargs) => {
if (!page) throw new Error('Browser page required');
@@ -191,6 +191,7 @@ cli({
degree: j.jobDegree,
skills: (j.skills || []).join(','),
boss: j.bossName + ' · ' + j.bossTitle,
security_id: j.securityId || '',
url: 'https://www.zhipin.com/job_detail/' + j.encryptJobId + '.html',
});
addedInBatch++;
+416
View File
@@ -0,0 +1,416 @@
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
// ── Filter value mappings ──────────────────────────────────────────────
const EXPERIENCE_LEVELS: Record<string, string> = {
internship: '1',
entry: '2',
'entry-level': '2',
associate: '3',
mid: '4',
senior: '4',
'mid-senior': '4',
'mid-senior-level': '4',
director: '5',
executive: '6',
};
const JOB_TYPES: Record<string, string> = {
'full-time': 'F',
fulltime: 'F',
full: 'F',
'part-time': 'P',
parttime: 'P',
part: 'P',
contract: 'C',
temporary: 'T',
temp: 'T',
volunteer: 'V',
internship: 'I',
other: 'O',
};
const DATE_POSTED: Record<string, string> = {
any: 'on',
month: 'r2592000',
'past-month': 'r2592000',
week: 'r604800',
'past-week': 'r604800',
day: 'r86400',
'24h': 'r86400',
'past-24h': 'r86400',
};
const REMOTE_TYPES: Record<string, string> = {
onsite: '1',
'on-site': '1',
hybrid: '3',
remote: '2',
};
// ── Helpers ────────────────────────────────────────────────────────────
function parseCsvArg(value: unknown): string[] {
if (value === undefined || value === null || value === '') return [];
return String(value)
.split(',')
.map(item => item.trim())
.filter(Boolean);
}
function mapFilterValues(input: unknown, mapping: Record<string, string>, label: string): string[] {
const values = parseCsvArg(input);
const resolved = values.map(value => {
const key = value.toLowerCase();
const mapped = mapping[key];
if (!mapped) throw new Error(`Unsupported ${label}: ${value}`);
return mapped;
});
return [...new Set(resolved)];
}
function normalizeWhitespace(value: unknown): string {
return String(value ?? '').replace(/\s+/g, ' ').trim();
}
function decodeLinkedinRedirect(url: string): string {
if (!url) return '';
try {
const parsed = new URL(url);
if (parsed.pathname === '/redir/redirect/') {
return parsed.searchParams.get('url') || url;
}
} catch {}
return url;
}
// ── Voyager query builder (runs in Node, NOT inside page.evaluate) ────
interface SearchInput {
keywords: string;
location: string;
limit: number;
start: number;
companyIds: string[];
experienceLevels: string[];
jobTypes: string[];
datePostedValues: string[];
remoteTypes: string[];
}
function buildVoyagerSearchQuery(input: SearchInput): string {
const hasFilters =
input.companyIds.length ||
input.experienceLevels.length ||
input.jobTypes.length ||
input.datePostedValues.length ||
input.remoteTypes.length;
const parts = [
'origin:' + (hasFilters ? 'JOB_SEARCH_PAGE_JOB_FILTER' : 'JOB_SEARCH_PAGE_OTHER_ENTRY'),
'keywords:' + input.keywords,
];
if (input.location) {
parts.push('locationUnion:(seoLocation:(location:' + input.location + '))');
}
const filters: string[] = [];
if (input.companyIds.length) filters.push('company:List(' + input.companyIds.join(',') + ')');
if (input.experienceLevels.length) filters.push('experience:List(' + input.experienceLevels.join(',') + ')');
if (input.jobTypes.length) filters.push('jobType:List(' + input.jobTypes.join(',') + ')');
if (input.datePostedValues.length) filters.push('timePostedRange:List(' + input.datePostedValues.join(',') + ')');
if (input.remoteTypes.length) filters.push('workplaceType:List(' + input.remoteTypes.join(',') + ')');
if (filters.length) parts.push('selectedFilters:(' + filters.join(',') + ')');
parts.push('spellCorrectionEnabled:true');
return '(' + parts.join(',') + ')';
}
function buildVoyagerUrl(input: SearchInput, offset: number, count: number): string {
const params = new URLSearchParams({
decorationId: 'com.linkedin.voyager.dash.deco.jobs.search.JobSearchCardsCollection-220',
count: String(count),
q: 'jobSearch',
});
const query = encodeURIComponent(buildVoyagerSearchQuery(input))
.replace(/%3A/gi, ':')
.replace(/%2C/gi, ',')
.replace(/%28/gi, '(')
.replace(/%29/gi, ')');
return '/voyager/api/voyagerJobsDashJobCards?' + params.toString() + '&query=' + query + '&start=' + offset;
}
// ── Company ID resolution (requires DOM interaction) ──────────────────
async function resolveCompanyIds(page: IPage, input: unknown): Promise<string[]> {
const rawValues = parseCsvArg(input);
const ids = new Set<string>();
const names: string[] = [];
for (const value of rawValues) {
if (/^\d+$/.test(value)) ids.add(value);
else names.push(value);
}
if (!names.length) return [...ids];
const resolved = await page.evaluate(`(async () => {
const targets = ${JSON.stringify(names)};
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
const normalize = (v) => (v || '').toLowerCase().replace(/\\s+/g, ' ').trim();
// Open "All filters" panel to expose company filter inputs
const allBtn = [...document.querySelectorAll('button')]
.find(b => ((b.innerText || '').trim().replace(/\\s+/g, ' ')) === 'All filters');
if (allBtn) { allBtn.click(); await sleep(300); }
const getCompanyMap = () => {
const map = {};
for (const el of document.querySelectorAll('input[name="company-filter-value"]')) {
const text = (el.parentElement?.innerText || el.closest('label')?.innerText || '')
.replace(/\\s+/g, ' ').trim().replace(/\\s*Filter by.*$/i, '').trim();
if (text) map[normalize(text)] = el.value;
}
return map;
};
const match = (map, name) => {
const n = normalize(name);
if (map[n]) return map[n];
const k = Object.keys(map).find(e => e === n || e.includes(n) || n.includes(e));
return k ? map[k] : null;
};
const results = {};
let map = getCompanyMap();
for (const name of targets) {
let found = match(map, name);
if (!found) {
const inp = [...document.querySelectorAll('input')]
.find(el => el.getAttribute('aria-label') === 'Add a company');
if (inp) {
inp.focus();
inp.value = name;
inp.dispatchEvent(new Event('input', { bubbles: true }));
inp.dispatchEvent(new KeyboardEvent('keyup', { key: 'Enter', bubbles: true }));
await sleep(1200);
map = getCompanyMap();
found = match(map, name);
inp.value = '';
inp.dispatchEvent(new Event('input', { bubbles: true }));
await sleep(100);
}
}
results[name] = found || null;
}
return results;
})()`);
const unresolved: string[] = [];
for (const name of names) {
const id = resolved?.[name];
if (id) ids.add(id);
else unresolved.push(name);
}
if (unresolved.length) {
throw new Error(`Could not resolve LinkedIn company filter: ${unresolved.join(', ')}`);
}
return [...ids];
}
// ── Voyager API fetch (runs inside page context for cookie access) ────
async function fetchJobCards(
page: IPage,
input: SearchInput,
): Promise<Array<Record<string, any>>> {
const MAX_BATCH = 25;
const allJobs: Array<Record<string, any>> = [];
let offset = input.start;
while (allJobs.length < input.limit) {
const count = Math.min(MAX_BATCH, input.limit - allJobs.length);
const apiPath = buildVoyagerUrl(input, offset, count);
const batch = await page.evaluate(`(async () => {
const jsession = document.cookie.split(';').map(p => p.trim())
.find(p => p.startsWith('JSESSIONID='))?.slice('JSESSIONID='.length);
if (!jsession) return { error: 'LinkedIn JSESSIONID cookie not found. Please sign in to LinkedIn in the browser.' };
const csrf = jsession.replace(/^"|"$/g, '');
const res = await fetch(${JSON.stringify(apiPath)}, {
credentials: 'include',
headers: { 'csrf-token': csrf, 'x-restli-protocol-version': '2.0.0' },
});
if (!res.ok) {
const text = await res.text();
return { error: 'LinkedIn API error: HTTP ' + res.status + ' ' + text.slice(0, 200) };
}
return res.json();
})()`);
if (!batch || batch.error) {
throw new Error(batch?.error || 'LinkedIn search returned an unexpected response');
}
const elements: any[] = Array.isArray(batch?.elements) ? batch.elements : [];
if (elements.length === 0) break;
for (const element of elements) {
const card = element?.jobCardUnion?.jobPostingCard;
if (!card) continue;
// Extract job ID from URN fields
const jobId = [card.jobPostingUrn, card.jobPosting?.entityUrn, card.entityUrn]
.filter(Boolean)
.map(s => String(s).match(/(\d+)/)?.[1])
.find(Boolean) ?? '';
// Extract listed date
const listedItem = (card.footerItems || []).find((i: any) => i?.type === 'LISTED_DATE' && i?.timeAt);
const listed = listedItem?.timeAt ? new Date(listedItem.timeAt).toISOString().slice(0, 10) : '';
allJobs.push({
title: card.jobPostingTitle || card.title?.text || '',
company: card.primaryDescription?.text || '',
location: card.secondaryDescription?.text || '',
listed,
salary: card.tertiaryDescription?.text || '',
url: jobId ? 'https://www.linkedin.com/jobs/view/' + jobId : '',
});
}
if (elements.length < count) break;
offset += elements.length;
}
return allJobs.slice(0, input.limit).map((item, index) => ({
rank: input.start + index + 1,
...item,
}));
}
// ── Job detail enrichment (--details flag) ────────────────────────────
async function enrichJobDetails(
page: IPage,
jobs: Array<Record<string, any>>,
): Promise<Array<Record<string, any>>> {
const enriched: Array<Record<string, any>> = [];
for (let i = 0; i < jobs.length; i++) {
const job = jobs[i];
console.error(`[opencli:linkedin] Fetching details ${i + 1}/${jobs.length}: ${job.title}`);
if (!job.url) {
enriched.push({ ...job, description: '', apply_url: '' });
continue;
}
try {
await page.goto(job.url);
await page.wait({ text: 'About the job', timeout: 8 });
// Expand "Show more" button if present
await page.evaluate(`(() => {
const norm = (v) => (v || '').replace(/\\s+/g, ' ').trim().toLowerCase();
const section = [...document.querySelectorAll('div, section, article')]
.find(el => norm(el.querySelector('h1,h2,h3,h4')?.textContent || '') === 'about the job');
const btn = [...(section?.querySelectorAll('button, a[role="button"]') || [])]
.find(el => /more/.test(norm(el.textContent || '')) || /more/.test(norm(el.getAttribute('aria-label') || '')));
if (btn) btn.click();
})()`);
await page.wait(1);
// Extract description and apply URL
const detail = await page.evaluate(`(() => {
const norm = (v) => (v || '').replace(/\\s+/g, ' ').trim();
// Find the most specific (shortest) container with "About the job" heading
// Shortest = most specific DOM node, avoiding outer wrappers that include unrelated text
const candidates = [...document.querySelectorAll('div, section, article')]
.map(el => ({
heading: norm(el.querySelector('h1,h2,h3,h4')?.textContent || ''),
text: norm(el.innerText || ''),
}))
.filter(c => c.text && c.heading.toLowerCase() === 'about the job' && c.text.length > 'About the job'.length)
.sort((a, b) => a.text.length - b.text.length);
const description = candidates[0]?.text.replace(/^About the job\\s*/i, '') || '';
const applyLink = [...document.querySelectorAll('a[href]')]
.map(a => ({ href: a.href || '', text: norm(a.textContent || ''), aria: norm(a.getAttribute('aria-label') || '') }))
.find(a => /apply/i.test(a.text) || /apply/i.test(a.aria));
return { description, applyUrl: applyLink?.href || '' };
})()`);
enriched.push({
...job,
description: normalizeWhitespace(detail?.description),
apply_url: decodeLinkedinRedirect(String(detail?.applyUrl ?? '')),
});
} catch {
enriched.push({ ...job, description: '', apply_url: '' });
}
}
return enriched;
}
// ── CLI registration ──────────────────────────────────────────────────
cli({
site: 'linkedin',
name: 'search',
description: 'Search LinkedIn jobs',
domain: 'www.linkedin.com',
strategy: Strategy.HEADER,
browser: true,
args: [
{ name: 'query', type: 'string', required: true, help: 'Job search keywords' },
{ name: 'location', type: 'string', required: false, help: 'Location text such as San Francisco Bay Area' },
{ name: 'limit', type: 'int', default: 10, help: 'Number of jobs to return (max 100)' },
{ name: 'start', type: 'int', default: 0, help: 'Result offset for pagination' },
{ name: 'details', type: 'bool', default: false, help: 'Include full job description and apply URL (slower)' },
{ name: 'company', type: 'string', required: false, help: 'Comma-separated company names or LinkedIn company IDs' },
{ name: 'experience_level', type: 'string', required: false, help: 'Comma-separated: internship, entry, associate, mid-senior, director, executive' },
{ name: 'job_type', type: 'string', required: false, help: 'Comma-separated: full-time, part-time, contract, temporary, volunteer, internship, other' },
{ name: 'date_posted', type: 'string', required: false, help: 'One of: any, month, week, 24h' },
{ name: 'remote', type: 'string', required: false, help: 'Comma-separated: on-site, hybrid, remote' },
],
columns: ['rank', 'title', 'company', 'location', 'listed', 'salary', 'url'],
func: async (page, kwargs) => {
const limit = Math.max(1, Math.min(kwargs.limit ?? 10, 100));
const start = Math.max(0, kwargs.start ?? 0);
const includeDetails = Boolean(kwargs.details);
const location = (kwargs.location ?? '').trim();
const keywords = String(kwargs.query ?? '').trim();
if (!keywords) throw new Error('query is required');
const searchParams = new URLSearchParams({ keywords });
if (location) searchParams.set('location', location);
await page.goto(`https://www.linkedin.com/jobs/search/?${searchParams.toString()}`);
await page.wait({ text: 'Jobs', timeout: 10 });
const companyIds = await resolveCompanyIds(page, kwargs.company);
const input: SearchInput = {
keywords,
location,
limit,
start,
companyIds,
experienceLevels: mapFilterValues(kwargs.experience_level, EXPERIENCE_LEVELS, 'experience_level'),
jobTypes: mapFilterValues(kwargs.job_type, JOB_TYPES, 'job_type'),
datePostedValues: mapFilterValues(kwargs.date_posted, DATE_POSTED, 'date_posted'),
remoteTypes: mapFilterValues(kwargs.remote, REMOTE_TYPES, 'remote'),
};
const data = await fetchJobCards(page, input);
if (!includeDetails) return data;
return enrichJobDetails(page, data);
},
});
+60
View File
@@ -0,0 +1,60 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'comment',
description: 'Post a comment on a Reddit post',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'post_id', type: 'string', required: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
{ name: 'text', type: 'string', required: true, help: 'Comment text' },
],
columns: ['status', 'message'],
func: async (page, kwargs) => {
if (!page) throw new Error('Requires browser');
await page.goto('https://www.reddit.com');
await page.wait(3);
const result = await page.evaluate(`(async () => {
try {
let postId = ${JSON.stringify(kwargs.post_id)};
const urlMatch = postId.match(/comments\\/([a-z0-9]+)/);
if (urlMatch) postId = urlMatch[1];
const fullname = postId.startsWith('t3_') || postId.startsWith('t1_')
? postId : 't3_' + postId;
const text = ${JSON.stringify(kwargs.text)};
// Get modhash
const meRes = await fetch('/api/me.json', { credentials: 'include' });
const me = await meRes.json();
const modhash = me?.data?.modhash || '';
const res = await fetch('/api/comment', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: 'parent=' + encodeURIComponent(fullname)
+ '&text=' + encodeURIComponent(text)
+ '&api_type=json'
+ (modhash ? '&uh=' + encodeURIComponent(modhash) : ''),
});
if (!res.ok) return { ok: false, message: 'HTTP ' + res.status };
const data = await res.json();
const errors = data?.json?.errors;
if (errors && errors.length > 0) {
return { ok: false, message: errors.map(e => e.join(': ')).join('; ') };
}
return { ok: true, message: 'Comment posted on ' + fullname };
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
return [{ status: result.ok ? 'success' : 'failed', message: result.message }];
}
});
+40
View File
@@ -0,0 +1,40 @@
site: reddit
name: popular
description: Reddit Popular posts (/r/popular)
domain: reddit.com
strategy: cookie
browser: true
args:
limit:
type: int
default: 20
columns: [rank, title, subreddit, score, comments, url]
pipeline:
- navigate: https://www.reddit.com
- evaluate: |
(async () => {
const limit = ${{ args.limit }};
const res = await fetch('/r/popular.json?limit=' + limit + '&raw_json=1', {
credentials: 'include'
});
const d = await res.json();
return (d?.data?.children || []).map(c => ({
title: c.data.title,
subreddit: c.data.subreddit_name_prefixed,
score: c.data.score,
comments: c.data.num_comments,
author: c.data.author,
url: 'https://www.reddit.com' + c.data.permalink,
}));
})()
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
subreddit: ${{ item.subreddit }}
score: ${{ item.score }}
comments: ${{ item.comments }}
url: ${{ item.url }}
- limit: ${{ args.limit }}
+186
View File
@@ -0,0 +1,186 @@
/**
* Reddit post reader with threaded comment tree.
*
* Replaces the original flat read.yaml with recursive comment traversal:
* - Top-K comments by score at each level
* - Configurable depth and replies-per-level
* - Indented output showing conversation threads
*/
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'read',
description: 'Read a Reddit post and its comments',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
args: [
{ name: 'post_id', required: true, help: 'Post ID (e.g. 1abc123) or full URL' },
{ name: 'sort', default: 'best', help: 'Comment sort: best, top, new, controversial, old, qa' },
{ name: 'limit', type: 'int', default: 25, help: 'Number of top-level comments' },
{ name: 'depth', type: 'int', default: 2, help: 'Max reply depth (1=no replies, 2=one level of replies, etc.)' },
{ name: 'replies', type: 'int', default: 5, help: 'Max replies shown per comment at each level (sorted by score)' },
{ name: 'max_length', type: 'int', default: 2000, help: 'Max characters per comment body (min 100)' },
],
columns: ['type', 'author', 'score', 'text'],
func: async (page, kwargs) => {
const sort = kwargs.sort ?? 'best';
const limit = Math.max(1, kwargs.limit ?? 25);
const maxDepth = Math.max(1, kwargs.depth ?? 2);
const maxReplies = Math.max(1, kwargs.replies ?? 5);
const maxLength = Math.max(100, kwargs.max_length ?? 2000);
await page.goto('https://www.reddit.com');
await page.wait(2);
const data = await page.evaluate(`
(async function() {
var postId = ${JSON.stringify(kwargs.post_id)};
var urlMatch = postId.match(/comments\\/([a-z0-9]+)/);
if (urlMatch) postId = urlMatch[1];
var sort = ${JSON.stringify(sort)};
var limit = ${limit};
var maxDepth = ${maxDepth};
var maxReplies = ${maxReplies};
var maxLength = ${maxLength};
// Request more from API than top-level limit to get inline replies
// depth param tells Reddit how deep to inline replies vs "more" stubs
var apiLimit = Math.max(limit * 3, 100);
var res = await fetch(
'/comments/' + postId + '.json?sort=' + sort + '&limit=' + apiLimit + '&depth=' + (maxDepth + 1) + '&raw_json=1',
{ credentials: 'include' }
);
if (!res.ok) return { error: 'Reddit API returned HTTP ' + res.status };
var data;
try { data = await res.json(); } catch(e) { return { error: 'Failed to parse response' }; }
if (!Array.isArray(data) || data.length < 2) return { error: 'Unexpected response format' };
var results = [];
// Post
var post = data[0] && data[0].data && data[0].data.children && data[0].data.children[0] && data[0].data.children[0].data;
if (post) {
var body = post.selftext || '';
if (body.length > maxLength) body = body.slice(0, maxLength) + '\\n... [truncated]';
results.push({
type: 'POST',
author: post.author || '[deleted]',
score: post.score || 0,
text: post.title + (body ? '\\n\\n' + body : '') + (post.url && !post.is_self ? '\\n' + post.url : ''),
});
}
// Recursive comment walker
// depth 0 = top-level comments; maxDepth is exclusive,
// so --depth 1 means top-level only, --depth 2 means one reply level, etc.
function walkComment(node, depth) {
if (!node || node.kind !== 't1') return;
var d = node.data;
var body = d.body || '';
if (body.length > maxLength) body = body.slice(0, maxLength) + '...';
// Indent prefix: apply to every line so multiline bodies stay aligned
var indent = '';
for (var i = 0; i < depth; i++) indent += ' ';
var prefix = depth === 0 ? '' : indent + '> ';
var indentedBody = depth === 0
? body
: body.split('\\n').map(function(line) { return prefix + line; }).join('\\n');
results.push({
type: depth === 0 ? 'L0' : 'L' + depth,
author: d.author || '[deleted]',
score: d.score || 0,
text: indentedBody,
});
// Count all available replies (for accurate "more" count)
var t1Children = [];
var moreCount = 0;
if (d.replies && d.replies.data && d.replies.data.children) {
var children = d.replies.data.children;
for (var i = 0; i < children.length; i++) {
if (children[i].kind === 't1') {
t1Children.push(children[i]);
} else if (children[i].kind === 'more') {
moreCount += children[i].data.count || 0;
}
}
}
// At depth cutoff: don't recurse, but show all replies as hidden
if (depth + 1 >= maxDepth) {
var totalHidden = t1Children.length + moreCount;
if (totalHidden > 0) {
var cutoffIndent = '';
for (var j = 0; j <= depth; j++) cutoffIndent += ' ';
results.push({
type: 'L' + (depth + 1),
author: '',
score: '',
text: cutoffIndent + '[+' + totalHidden + ' more replies]',
});
}
return;
}
// Sort by score descending, take top N
t1Children.sort(function(a, b) { return (b.data.score || 0) - (a.data.score || 0); });
var toProcess = Math.min(t1Children.length, maxReplies);
for (var i = 0; i < toProcess; i++) {
walkComment(t1Children[i], depth + 1);
}
// Show hidden count (skipped replies + "more" stubs)
var hidden = t1Children.length - toProcess + moreCount;
if (hidden > 0) {
var moreIndent = '';
for (var j = 0; j <= depth; j++) moreIndent += ' ';
results.push({
type: 'L' + (depth + 1),
author: '',
score: '',
text: moreIndent + '[+' + hidden + ' more replies]',
});
}
}
// Walk top-level comments
var topLevel = data[1].data.children || [];
var t1TopLevel = [];
for (var i = 0; i < topLevel.length; i++) {
if (topLevel[i].kind === 't1') t1TopLevel.push(topLevel[i]);
}
// Top-level are already sorted by Reddit (sort param), take top N
for (var i = 0; i < Math.min(t1TopLevel.length, limit); i++) {
walkComment(t1TopLevel[i], 0);
}
// Count remaining
var moreTopLevel = topLevel.filter(function(c) { return c.kind === 'more'; })
.reduce(function(sum, c) { return sum + (c.data.count || 0); }, 0);
var hiddenTopLevel = Math.max(0, t1TopLevel.length - limit) + moreTopLevel;
if (hiddenTopLevel > 0) {
results.push({
type: '',
author: '',
score: '',
text: '[+' + hiddenTopLevel + ' more top-level comments]',
});
}
return results;
})()
`);
if (!data || typeof data !== 'object') throw new Error('Failed to fetch post data');
if (!Array.isArray(data) && data.error) throw new Error(data.error);
if (!Array.isArray(data)) throw new Error('Unexpected response');
return data;
},
});
+54
View File
@@ -0,0 +1,54 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'save',
description: 'Save or unsave a Reddit post',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'post_id', type: 'string', required: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
{ name: 'undo', type: 'boolean', default: false, help: 'Unsave instead of save' },
],
columns: ['status', 'message'],
func: async (page, kwargs) => {
if (!page) throw new Error('Requires browser');
await page.goto('https://www.reddit.com');
await page.wait(3);
const result = await page.evaluate(`(async () => {
try {
let postId = ${JSON.stringify(kwargs.post_id)};
const urlMatch = postId.match(/comments\\/([a-z0-9]+)/);
if (urlMatch) postId = urlMatch[1];
const fullname = postId.startsWith('t3_') || postId.startsWith('t1_')
? postId : 't3_' + postId;
const undo = ${kwargs.undo ? 'true' : 'false'};
const endpoint = undo ? '/api/unsave' : '/api/save';
// Get modhash
const meRes = await fetch('/api/me.json', { credentials: 'include' });
const me = await meRes.json();
const modhash = me?.data?.modhash || '';
const res = await fetch(endpoint, {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: 'id=' + encodeURIComponent(fullname)
+ (modhash ? '&uh=' + encodeURIComponent(modhash) : ''),
});
if (!res.ok) return { ok: false, message: 'HTTP ' + res.status };
return { ok: true, message: (undo ? 'Unsaved' : 'Saved') + ' ' + fullname };
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
return [{ status: result.ok ? 'success' : 'failed', message: result.message }];
}
});
+48
View File
@@ -0,0 +1,48 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'saved',
description: 'Browse your saved Reddit posts',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'limit', type: 'int', default: 15 },
],
columns: ['title', 'subreddit', 'score', 'comments', 'url'],
func: async (page, kwargs) => {
if (!page) throw new Error('Requires browser');
await page.goto('https://www.reddit.com');
await page.wait(3);
const result = await page.evaluate(`(async () => {
try {
// Get current username
const meRes = await fetch('/api/me.json?raw_json=1', { credentials: 'include' });
const me = await meRes.json();
const username = me?.name || me?.data?.name;
if (!username) return { error: 'Not logged in — cannot determine username' };
const limit = ${kwargs.limit};
const res = await fetch('/user/' + username + '/saved.json?limit=' + limit + '&raw_json=1', {
credentials: 'include'
});
const d = await res.json();
return (d?.data?.children || []).map(c => ({
title: c.data.title || c.data.body?.slice(0, 100) || '-',
subreddit: c.data.subreddit_name_prefixed || 'r/' + (c.data.subreddit || '?'),
score: c.data.score || 0,
comments: c.data.num_comments || 0,
url: 'https://www.reddit.com' + (c.data.permalink || ''),
}));
} catch (e) {
return { error: e.toString() };
}
})()`);
if (result?.error) throw new Error(result.error);
return (result || []).slice(0, kwargs.limit);
}
});
+37 -11
View File
@@ -9,26 +9,52 @@ args:
query:
type: string
required: true
subreddit:
type: string
default: ""
description: "Search within a specific subreddit"
sort:
type: string
default: relevance
description: "Sort order: relevance, hot, top, new, comments"
time:
type: string
default: all
description: "Time filter: hour, day, week, month, year, all"
limit:
type: int
default: 15
columns: [title, subreddit, author, upvotes, comments, url]
columns: [title, subreddit, author, score, comments, url]
pipeline:
- navigate: https://www.reddit.com
- evaluate: |
(async () => {
const q = encodeURIComponent('${{ args.query }}');
const res = await fetch('/search.json?q=' + q + '&limit=${{ args.limit }}', { credentials: 'include' });
const j = await res.json();
return j?.data?.children || [];
const q = encodeURIComponent(${{ args.query | json }});
const sub = ${{ args.subreddit | json }};
const sort = ${{ args.sort | json }};
const time = ${{ args.time | json }};
const limit = ${{ args.limit }};
const basePath = sub ? '/r/' + sub + '/search.json' : '/search.json';
const params = 'q=' + q + '&sort=' + sort + '&t=' + time + '&limit=' + limit
+ '&restrict_sr=' + (sub ? 'on' : 'off') + '&raw_json=1';
const res = await fetch(basePath + '?' + params, { credentials: 'include' });
const d = await res.json();
return (d?.data?.children || []).map(c => ({
title: c.data.title,
subreddit: c.data.subreddit_name_prefixed,
author: c.data.author,
score: c.data.score,
comments: c.data.num_comments,
url: 'https://www.reddit.com' + c.data.permalink,
}));
})()
- map:
title: ${{ item.data.title }}
subreddit: ${{ item.data.subreddit_name_prefixed }}
author: ${{ item.data.author }}
upvotes: ${{ item.data.score }}
comments: ${{ item.data.num_comments }}
url: https://www.reddit.com${{ item.data.permalink }}
title: ${{ item.title }}
subreddit: ${{ item.subreddit }}
author: ${{ item.author }}
score: ${{ item.score }}
comments: ${{ item.comments }}
url: ${{ item.url }}
- limit: ${{ args.limit }}
+14 -4
View File
@@ -12,7 +12,11 @@ args:
sort:
type: string
default: hot
description: "Sorting method: hot, new, top, rising"
description: "Sorting method: hot, new, top, rising, controversial"
time:
type: string
default: all
description: "Time filter for top/controversial: hour, day, week, month, year, all"
limit:
type: int
default: 15
@@ -23,10 +27,16 @@ pipeline:
- navigate: https://www.reddit.com
- evaluate: |
(async () => {
let sub = '${{ args.name }}';
let sub = ${{ args.name | json }};
if (sub.startsWith('r/')) sub = sub.slice(2);
const sort = '${{ args.sort }}';
const res = await fetch('/r/' + sub + '/' + sort + '.json?limit=${{ args.limit }}', { credentials: 'include' });
const sort = ${{ args.sort | json }};
const time = ${{ args.time | json }};
const limit = ${{ args.limit }};
let url = '/r/' + sub + '/' + sort + '.json?limit=' + limit + '&raw_json=1';
if ((sort === 'top' || sort === 'controversial') && time) {
url += '&t=' + time;
}
const res = await fetch(url, { credentials: 'include' });
const j = await res.json();
return j?.data?.children || [];
})()
+53
View File
@@ -0,0 +1,53 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'subscribe',
description: 'Subscribe or unsubscribe to a subreddit',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'subreddit', type: 'string', required: true, help: 'Subreddit name (e.g. python)' },
{ name: 'undo', type: 'boolean', default: false, help: 'Unsubscribe instead of subscribe' },
],
columns: ['status', 'message'],
func: async (page, kwargs) => {
if (!page) throw new Error('Requires browser');
await page.goto('https://www.reddit.com');
await page.wait(3);
const result = await page.evaluate(`(async () => {
try {
let sub = ${JSON.stringify(kwargs.subreddit)};
if (sub.startsWith('r/')) sub = sub.slice(2);
const undo = ${kwargs.undo ? 'true' : 'false'};
const action = undo ? 'unsub' : 'sub';
// Get modhash
const meRes = await fetch('/api/me.json', { credentials: 'include' });
const me = await meRes.json();
const modhash = me?.data?.modhash || '';
const res = await fetch('/api/subscribe', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: 'sr_name=' + encodeURIComponent(sub)
+ '&action=' + action
+ (modhash ? '&uh=' + encodeURIComponent(modhash) : ''),
});
if (!res.ok) return { ok: false, message: 'HTTP ' + res.status };
const label = undo ? 'Unsubscribed from' : 'Subscribed to';
return { ok: true, message: label + ' r/' + sub };
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
return [{ status: result.ok ? 'success' : 'failed', message: result.message }];
}
});
+67
View File
@@ -0,0 +1,67 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'upvote',
description: 'Upvote or downvote a Reddit post',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'post_id', type: 'string', required: true, help: 'Post ID (e.g. 1abc123) or fullname (t3_xxx)' },
{ name: 'direction', type: 'string', default: 'up', help: 'Vote direction: up, down, none' },
],
columns: ['status', 'message'],
func: async (page, kwargs) => {
if (!page) throw new Error('Requires browser');
await page.goto('https://www.reddit.com');
await page.wait(3);
const result = await page.evaluate(`(async () => {
try {
let postId = ${JSON.stringify(kwargs.post_id)};
// Extract ID from URL if needed
const urlMatch = postId.match(/comments\\/([a-z0-9]+)/);
if (urlMatch) postId = urlMatch[1];
// Build fullname
const fullname = postId.startsWith('t3_') || postId.startsWith('t1_')
? postId : 't3_' + postId;
const dir = ${JSON.stringify(kwargs.direction)};
const direction = dir === 'down' ? -1 : dir === 'none' ? 0 : 1;
// Get modhash from Reddit config
const configEl = document.getElementById('config');
let modhash = '';
if (configEl) {
modhash = configEl.querySelector('[name="uh"]')?.getAttribute('content') || '';
}
if (!modhash) {
// Try fetching from /api/me.json
const meRes = await fetch('/api/me.json', { credentials: 'include' });
const me = await meRes.json();
modhash = me?.data?.modhash || '';
}
const res = await fetch('/api/vote', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: 'id=' + encodeURIComponent(fullname)
+ '&dir=' + direction
+ (modhash ? '&uh=' + encodeURIComponent(modhash) : ''),
});
if (!res.ok) return { ok: false, message: 'HTTP ' + res.status };
const labels = { '1': 'Upvoted', '-1': 'Downvoted', '0': 'Vote removed' };
return { ok: true, message: (labels[String(direction)] || 'Voted') + ' ' + fullname };
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
return [{ status: result.ok ? 'success' : 'failed', message: result.message }];
}
});
+48
View File
@@ -0,0 +1,48 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'reddit',
name: 'upvoted',
description: 'Browse your upvoted Reddit posts',
domain: 'reddit.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'limit', type: 'int', default: 15 },
],
columns: ['title', 'subreddit', 'score', 'comments', 'url'],
func: async (page, kwargs) => {
if (!page) throw new Error('Requires browser');
await page.goto('https://www.reddit.com');
await page.wait(3);
const result = await page.evaluate(`(async () => {
try {
// Get current username
const meRes = await fetch('/api/me.json?raw_json=1', { credentials: 'include' });
const me = await meRes.json();
const username = me?.name || me?.data?.name;
if (!username) return { error: 'Not logged in — cannot determine username' };
const limit = ${kwargs.limit};
const res = await fetch('/user/' + username + '/upvoted.json?limit=' + limit + '&raw_json=1', {
credentials: 'include'
});
const d = await res.json();
return (d?.data?.children || []).map(c => ({
title: c.data.title || '-',
subreddit: c.data.subreddit_name_prefixed || 'r/' + (c.data.subreddit || '?'),
score: c.data.score || 0,
comments: c.data.num_comments || 0,
url: 'https://www.reddit.com' + (c.data.permalink || ''),
}));
} catch (e) {
return { error: e.toString() };
}
})()`);
if (result?.error) throw new Error(result.error);
return (result || []).slice(0, kwargs.limit);
}
});
+45
View File
@@ -0,0 +1,45 @@
site: reddit
name: user-comments
description: View a Reddit user's comment history
domain: reddit.com
strategy: cookie
browser: true
args:
username:
type: string
required: true
limit:
type: int
default: 15
columns: [subreddit, score, body, url]
pipeline:
- navigate: https://www.reddit.com
- evaluate: |
(async () => {
const username = ${{ args.username | json }};
const name = username.startsWith('u/') ? username.slice(2) : username;
const limit = ${{ args.limit }};
const res = await fetch('/user/' + name + '/comments.json?limit=' + limit + '&raw_json=1', {
credentials: 'include'
});
const d = await res.json();
return (d?.data?.children || []).map(c => {
let body = c.data.body || '';
if (body.length > 300) body = body.slice(0, 300) + '...';
return {
subreddit: c.data.subreddit_name_prefixed,
score: c.data.score,
body: body,
url: 'https://www.reddit.com' + c.data.permalink,
};
});
})()
- map:
subreddit: ${{ item.subreddit }}
score: ${{ item.score }}
body: ${{ item.body }}
url: ${{ item.url }}
- limit: ${{ args.limit }}
+43
View File
@@ -0,0 +1,43 @@
site: reddit
name: user-posts
description: View a Reddit user's submitted posts
domain: reddit.com
strategy: cookie
browser: true
args:
username:
type: string
required: true
limit:
type: int
default: 15
columns: [title, subreddit, score, comments, url]
pipeline:
- navigate: https://www.reddit.com
- evaluate: |
(async () => {
const username = ${{ args.username | json }};
const name = username.startsWith('u/') ? username.slice(2) : username;
const limit = ${{ args.limit }};
const res = await fetch('/user/' + name + '/submitted.json?limit=' + limit + '&raw_json=1', {
credentials: 'include'
});
const d = await res.json();
return (d?.data?.children || []).map(c => ({
title: c.data.title,
subreddit: c.data.subreddit_name_prefixed,
score: c.data.score,
comments: c.data.num_comments,
url: 'https://www.reddit.com' + c.data.permalink,
}));
})()
- map:
title: ${{ item.title }}
subreddit: ${{ item.subreddit }}
score: ${{ item.score }}
comments: ${{ item.comments }}
url: ${{ item.url }}
- limit: ${{ args.limit }}
+39
View File
@@ -0,0 +1,39 @@
site: reddit
name: user
description: View a Reddit user profile
domain: reddit.com
strategy: cookie
browser: true
args:
username:
type: string
required: true
columns: [field, value]
pipeline:
- navigate: https://www.reddit.com
- evaluate: |
(async () => {
const username = ${{ args.username | json }};
const name = username.startsWith('u/') ? username.slice(2) : username;
const res = await fetch('/user/' + name + '/about.json?raw_json=1', {
credentials: 'include'
});
const d = await res.json();
const u = d?.data || d || {};
const created = u.created_utc ? new Date(u.created_utc * 1000).toISOString().split('T')[0] : '-';
return [
{ field: 'Username', value: 'u/' + (u.name || name) },
{ field: 'Post Karma', value: String(u.link_karma || 0) },
{ field: 'Comment Karma', value: String(u.comment_karma || 0) },
{ field: 'Total Karma', value: String(u.total_karma || (u.link_karma||0) + (u.comment_karma||0)) },
{ field: 'Account Created', value: created },
{ field: 'Gold', value: u.is_gold ? '⭐ Yes' : 'No' },
{ field: 'Verified', value: u.verified ? '✅ Yes' : 'No' },
];
})()
- map:
field: ${{ item.field }}
value: ${{ item.value }}
+161
View File
@@ -0,0 +1,161 @@
import { cli, Strategy } from '../../registry.js';
cli({
site: 'twitter',
name: 'article',
description: 'Fetch a Twitter Article (long-form content) and export as Markdown',
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'tweet_id', type: 'string', positional: true, required: true, help: 'Tweet ID or URL containing the article' },
],
columns: ['title', 'author', 'content', 'url'],
func: async (page, kwargs) => {
// Extract tweet ID from URL if needed
let tweetId = kwargs.tweet_id;
const urlMatch = tweetId.match(/\/(?:status|article)\/(\d+)/);
if (urlMatch) tweetId = urlMatch[1];
// Navigate to the tweet page for cookie context
await page.goto(`https://x.com/i/status/${tweetId}`);
await page.wait(3);
const result = await page.evaluate(`
async () => {
const tweetId = "${tweetId}";
const ct0 = document.cookie.split(';').map(c=>c.trim()).find(c=>c.startsWith('ct0='))?.split('=')[1];
if (!ct0) return {error: 'No ct0 cookie — not logged into x.com'};
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA';
const headers = {
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
'X-Csrf-Token': ct0,
'X-Twitter-Auth-Type': 'OAuth2Session',
'X-Twitter-Active-User': 'yes'
};
const variables = JSON.stringify({
tweetId: tweetId,
withCommunity: false,
includePromotedContent: false,
withVoice: false,
});
const features = JSON.stringify({
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
articles_preview_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
verified_phone_label_enabled: false,
});
const fieldToggles = JSON.stringify({
withArticleRichContentState: true,
withArticlePlainText: true,
});
// Dynamically resolve queryId: GitHub community source → JS bundle scan → hardcoded fallback
async function resolveQueryId(operationName, fallbackId) {
try {
const ghResp = await fetch('https://raw.githubusercontent.com/fa0311/twitter-openapi/refs/heads/main/src/config/placeholder.json');
if (ghResp.ok) {
const data = await ghResp.json();
const entry = data[operationName];
if (entry && entry.queryId) return entry.queryId;
}
} catch {}
try {
const scripts = performance.getEntriesByType('resource')
.filter(r => r.name.includes('client-web') && r.name.endsWith('.js'))
.map(r => r.name);
for (const scriptUrl of scripts.slice(0, 15)) {
try {
const text = await (await fetch(scriptUrl)).text();
const re = new RegExp('queryId:"([A-Za-z0-9_-]+)"[^}]{0,200}operationName:"' + operationName + '"');
const m = text.match(re);
if (m) return m[1];
} catch {}
}
} catch {}
return fallbackId;
}
const queryId = await resolveQueryId('TweetResultByRestId', '7xflPyRiUxGVbJd4uWmbfg');
const url = '/i/api/graphql/' + queryId + '/TweetResultByRestId?variables='
+ encodeURIComponent(variables)
+ '&features=' + encodeURIComponent(features)
+ '&fieldToggles=' + encodeURIComponent(fieldToggles);
const resp = await fetch(url, {headers, credentials: 'include'});
if (!resp.ok) return {error: 'HTTP ' + resp.status, hint: 'Tweet may not exist or queryId expired'};
const d = await resp.json();
const result = d.data?.tweetResult?.result;
if (!result) return {error: 'Article not found'};
// Unwrap TweetWithVisibilityResults
const tw = result.tweet || result;
const legacy = tw.legacy || {};
const user = tw.core?.user_results?.result;
const screenName = user?.legacy?.screen_name || user?.core?.screen_name || 'unknown';
// Extract article content
const articleResults = tw.article?.article_results?.result;
if (!articleResults) {
// Fallback: return note_tweet text if present
const noteText = tw.note_tweet?.note_tweet_results?.result?.text;
if (noteText) {
return [{
title: '(Note Tweet)',
author: screenName,
content: noteText,
url: 'https://x.com/' + screenName + '/status/' + tweetId,
}];
}
return {error: 'Tweet ' + tweetId + ' has no article content'};
}
const title = articleResults.title || '(Untitled)';
const contentState = articleResults.content_state || {};
const blocks = contentState.blocks || [];
// Convert draft.js blocks to Markdown
const parts = [];
let orderedCounter = 0;
for (const block of blocks) {
const blockType = block.type || 'unstyled';
if (blockType === 'atomic') continue;
const text = block.text || '';
if (!text) continue;
if (blockType !== 'ordered-list-item') orderedCounter = 0;
if (blockType === 'header-one') parts.push('# ' + text);
else if (blockType === 'header-two') parts.push('## ' + text);
else if (blockType === 'header-three') parts.push('### ' + text);
else if (blockType === 'blockquote') parts.push('> ' + text);
else if (blockType === 'unordered-list-item') parts.push('- ' + text);
else if (blockType === 'ordered-list-item') {
orderedCounter++;
parts.push(orderedCounter + '. ' + text);
}
else if (blockType === 'code-block') parts.push('\`\`\`\\n' + text + '\\n\`\`\`');
else parts.push(text);
}
return [{
title,
author: screenName,
content: parts.join('\\n\\n') || legacy.full_text || '',
url: 'https://x.com/' + screenName + '/status/' + tweetId,
}];
}
`);
if (result?.error) {
throw new Error(result.error + (result.hint ? ` (${result.hint})` : ''));
}
return result || [];
}
});
+67
View File
@@ -0,0 +1,67 @@
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
cli({
site: 'twitter',
name: 'bookmark',
description: 'Bookmark a tweet',
domain: 'x.com',
strategy: Strategy.UI,
browser: true,
args: [
{ name: 'url', type: 'string', positional: true, required: true, help: 'Tweet URL to bookmark' },
],
columns: ['status', 'message'],
func: async (page: IPage | null, kwargs: any) => {
if (!page) throw new Error('Requires browser');
await page.goto(kwargs.url);
await page.wait(5);
const result = await page.evaluate(`(async () => {
try {
let attempts = 0;
let bookmarkBtn = null;
let removeBtn = null;
while (attempts < 20) {
// Check if already bookmarked
removeBtn = document.querySelector('[data-testid="removeBookmark"]');
if (removeBtn) {
return { ok: true, message: 'Tweet is already bookmarked.' };
}
bookmarkBtn = document.querySelector('[data-testid="bookmark"]');
if (bookmarkBtn) break;
await new Promise(r => setTimeout(r, 500));
attempts++;
}
if (!bookmarkBtn) {
return { ok: false, message: 'Could not find Bookmark button. Are you logged in?' };
}
bookmarkBtn.click();
await new Promise(r => setTimeout(r, 1000));
// Verify
const verify = document.querySelector('[data-testid="removeBookmark"]');
if (verify) {
return { ok: true, message: 'Tweet successfully bookmarked.' };
} else {
return { ok: false, message: 'Bookmark action initiated but UI did not update.' };
}
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
if (result.ok) await page.wait(2);
return [{
status: result.ok ? 'success' : 'failed',
message: result.message
}];
}
});
+69
View File
@@ -0,0 +1,69 @@
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
cli({
site: 'twitter',
name: 'follow',
description: 'Follow a Twitter user',
domain: 'x.com',
strategy: Strategy.UI,
browser: true,
args: [
{ name: 'username', type: 'string', positional: true, required: true, help: 'Twitter screen name (without @)' },
],
columns: ['status', 'message'],
func: async (page: IPage | null, kwargs: any) => {
if (!page) throw new Error('Requires browser');
const username = kwargs.username.replace(/^@/, '');
await page.goto(`https://x.com/${username}`);
await page.wait(5);
const result = await page.evaluate(`(async () => {
try {
let attempts = 0;
let followBtn = null;
let unfollowTestId = null;
while (attempts < 20) {
// Check if already following (button shows screen_name-unfollow)
unfollowTestId = document.querySelector('[data-testid$="-unfollow"]');
if (unfollowTestId) {
return { ok: true, message: 'Already following @${username}.' };
}
// Look for the Follow button
followBtn = document.querySelector('[data-testid$="-follow"]');
if (followBtn) break;
await new Promise(r => setTimeout(r, 500));
attempts++;
}
if (!followBtn) {
return { ok: false, message: 'Could not find Follow button. Are you logged in?' };
}
followBtn.click();
await new Promise(r => setTimeout(r, 1500));
// Verify
const verify = document.querySelector('[data-testid$="-unfollow"]');
if (verify) {
return { ok: true, message: 'Successfully followed @${username}.' };
} else {
return { ok: false, message: 'Follow action initiated but UI did not update.' };
}
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
if (result.ok) await page.wait(2);
return [{
status: result.ok ? 'success' : 'failed',
message: result.message
}];
}
});
+1 -1
View File
@@ -30,7 +30,7 @@ cli({
let results: any[] = [];
for (const req of requests) {
try {
let instructions = [];
let instructions: any[] = [];
if (req.data?.data?.viewer?.timeline_response?.timeline?.instructions) {
instructions = req.data.data.viewer.timeline_response.timeline.instructions;
} else if (req.data?.data?.viewer_v2?.user_results?.result?.notification_timeline?.timeline?.instructions) {
+114 -46
View File
@@ -3,59 +3,127 @@ import { cli, Strategy } from '../../registry.js';
cli({
site: 'twitter',
name: 'profile',
description: 'Fetch tweets from a user profile',
description: 'Fetch a Twitter user profile (bio, stats, etc.)',
domain: 'x.com',
strategy: Strategy.INTERCEPT,
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'username', type: 'string', required: true },
{ name: 'limit', type: 'int', default: 15 },
{ name: 'username', type: 'string', positional: true, help: 'Twitter screen name (without @). Defaults to logged-in user.' },
],
columns: ['id', 'text', 'likes', 'views', 'url'],
columns: ['screen_name', 'name', 'bio', 'location', 'url', 'followers', 'following', 'tweets', 'likes', 'verified', 'created_at'],
func: async (page, kwargs) => {
// Navigate to user profile via search for reliability
await page.goto(`https://x.com/search?q=from:${kwargs.username}&f=live`);
await page.wait(5);
let username = (kwargs.username || '').replace(/^@/, '');
// Inject XHR interceptor
await page.installInterceptor('SearchTimeline');
// Trigger API by scrolling
await page.autoScroll({ times: 3, delayMs: 2000 });
// Retrieve data
const requests = await page.getInterceptedRequests();
if (!requests || requests.length === 0) return [];
let results: any[] = [];
for (const req of requests) {
try {
const insts = req.data.data.search_by_raw_query.search_timeline.timeline.instructions;
const addEntries = insts.find((i: any) => i.type === 'TimelineAddEntries');
if (!addEntries) continue;
for (const entry of addEntries.entries) {
if (!entry.entryId.startsWith('tweet-')) continue;
let tweet = entry.content?.itemContent?.tweet_results?.result;
if (!tweet) continue;
if (tweet.__typename === 'TweetWithVisibilityResults' && tweet.tweet) {
tweet = tweet.tweet;
}
results.push({
id: tweet.rest_id,
text: tweet.legacy?.full_text || '',
likes: tweet.legacy?.favorite_count || 0,
views: tweet.views?.count || '0',
url: `https://x.com/i/status/${tweet.rest_id}`
});
}
} catch (e) {
}
// If no username, detect the logged-in user
if (!username) {
await page.goto('https://x.com/home');
await page.wait(5);
const href = await page.evaluate(`() => {
const link = document.querySelector('a[data-testid="AppTabBar_Profile_Link"]');
return link ? link.getAttribute('href') : null;
}`);
if (!href) throw new Error('Could not detect logged-in user. Are you logged in?');
username = href.replace('/', '');
}
return results.slice(0, kwargs.limit);
// Navigate directly to the user's profile page (gives us cookie context)
await page.goto(`https://x.com/${username}`);
await page.wait(3);
const result = await page.evaluate(`
async () => {
const screenName = "${username}";
const ct0 = document.cookie.split(';').map(c=>c.trim()).find(c=>c.startsWith('ct0='))?.split('=')[1];
if (!ct0) return {error: 'No ct0 cookie — not logged into x.com'};
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA';
const headers = {
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
'X-Csrf-Token': ct0,
'X-Twitter-Auth-Type': 'OAuth2Session',
'X-Twitter-Active-User': 'yes'
};
const variables = JSON.stringify({
screen_name: screenName,
withSafetyModeUserFields: true,
});
const features = JSON.stringify({
hidden_profile_subscriptions_enabled: true,
rweb_tipjar_consumption_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
verified_phone_label_enabled: false,
subscriptions_verification_info_is_identity_verified_enabled: true,
subscriptions_verification_info_verified_since_enabled: true,
highlights_tweets_tab_ui_enabled: true,
responsive_web_twitter_article_notes_tab_enabled: true,
subscriptions_feature_can_gift_premium: true,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
responsive_web_graphql_timeline_navigation_enabled: true,
});
// Dynamically resolve queryId: GitHub community source → JS bundle scan → hardcoded fallback
async function resolveQueryId(operationName, fallbackId) {
try {
const ghResp = await fetch('https://raw.githubusercontent.com/fa0311/twitter-openapi/refs/heads/main/src/config/placeholder.json');
if (ghResp.ok) {
const data = await ghResp.json();
const entry = data[operationName];
if (entry && entry.queryId) return entry.queryId;
}
} catch {}
try {
const scripts = performance.getEntriesByType('resource')
.filter(r => r.name.includes('client-web') && r.name.endsWith('.js'))
.map(r => r.name);
for (const scriptUrl of scripts.slice(0, 15)) {
try {
const text = await (await fetch(scriptUrl)).text();
const re = new RegExp('queryId:"([A-Za-z0-9_-]+)"[^}]{0,200}operationName:"' + operationName + '"');
const m = text.match(re);
if (m) return m[1];
} catch {}
}
} catch {}
return fallbackId;
}
const queryId = await resolveQueryId('UserByScreenName', 'qRednkZG-rn1P6b48NINmQ');
const url = '/i/api/graphql/' + queryId + '/UserByScreenName?variables='
+ encodeURIComponent(variables)
+ '&features=' + encodeURIComponent(features);
const resp = await fetch(url, {headers, credentials: 'include'});
if (!resp.ok) return {error: 'HTTP ' + resp.status, hint: 'User may not exist or queryId expired'};
const d = await resp.json();
const result = d.data?.user?.result;
if (!result) return {error: 'User @' + screenName + ' not found'};
const legacy = result.legacy || {};
const expandedUrl = legacy.entities?.url?.urls?.[0]?.expanded_url || '';
return [{
screen_name: legacy.screen_name || screenName,
name: legacy.name || '',
bio: legacy.description || '',
location: legacy.location || '',
url: expandedUrl,
followers: legacy.followers_count || 0,
following: legacy.friends_count || 0,
tweets: legacy.statuses_count || 0,
likes: legacy.favourites_count || 0,
verified: result.is_blue_verified || legacy.verified || false,
created_at: legacy.created_at || '',
}];
}
`);
if (result?.error) {
throw new Error(result.error + (result.hint ? ` (${result.hint})` : ''));
}
return result || [];
}
});
+181
View File
@@ -0,0 +1,181 @@
import { cli, Strategy } from '../../registry.js';
// ── Twitter GraphQL constants ──────────────────────────────────────────
const BEARER_TOKEN = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA';
const TWEET_DETAIL_QUERY_ID = 'nBS-WpgA6ZG0CyNHD517JQ';
const FEATURES = {
responsive_web_graphql_exclude_directive_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
longform_notetweets_consumption_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
freedom_of_speech_not_reach_fetch_enabled: true,
};
const FIELD_TOGGLES = { withArticleRichContentState: true, withArticlePlainText: false };
// ── Pure functions (type-safe, testable) ───────────────────────────────
interface ThreadTweet {
id: string;
author: string;
text: string;
likes: number;
retweets: number;
in_reply_to?: string;
created_at?: string;
url: string;
}
function buildTweetDetailUrl(tweetId: string, cursor?: string | null): string {
const vars: Record<string, any> = {
focalTweetId: tweetId,
referrer: 'tweet',
with_rux_injections: false,
includePromotedContent: false,
rankingMode: 'Recency',
withCommunity: true,
withQuickPromoteEligibilityTweetFields: true,
withBirdwatchNotes: true,
withVoice: true,
};
if (cursor) vars.cursor = cursor;
return `/i/api/graphql/${TWEET_DETAIL_QUERY_ID}/TweetDetail`
+ `?variables=${encodeURIComponent(JSON.stringify(vars))}`
+ `&features=${encodeURIComponent(JSON.stringify(FEATURES))}`
+ `&fieldToggles=${encodeURIComponent(JSON.stringify(FIELD_TOGGLES))}`;
}
function extractTweet(r: any, seen: Set<string>): ThreadTweet | null {
if (!r) return null;
const tw = r.tweet || r;
const l = tw.legacy || {};
if (!tw.rest_id || seen.has(tw.rest_id)) return null;
seen.add(tw.rest_id);
const u = tw.core?.user_results?.result;
const noteText = tw.note_tweet?.note_tweet_results?.result?.text;
const screenName = u?.legacy?.screen_name || u?.core?.screen_name || 'unknown';
return {
id: tw.rest_id,
author: screenName,
text: noteText || l.full_text || '',
likes: l.favorite_count || 0,
retweets: l.retweet_count || 0,
in_reply_to: l.in_reply_to_status_id_str || undefined,
created_at: l.created_at,
url: `https://x.com/${screenName}/status/${tw.rest_id}`,
};
}
function parseTweetDetail(data: any, seen: Set<string>): { tweets: ThreadTweet[]; nextCursor: string | null } {
const tweets: ThreadTweet[] = [];
let nextCursor: string | null = null;
const instructions =
data?.data?.threaded_conversation_with_injections_v2?.instructions
|| data?.data?.tweetResult?.result?.timeline?.instructions
|| [];
for (const inst of instructions) {
for (const entry of inst.entries || []) {
// Cursor entries
const c = entry.content;
if (c?.entryType === 'TimelineTimelineCursor' || c?.__typename === 'TimelineTimelineCursor') {
if (c.cursorType === 'Bottom' || c.cursorType === 'ShowMore') nextCursor = c.value;
continue;
}
if (entry.entryId?.startsWith('cursor-bottom-') || entry.entryId?.startsWith('cursor-showMore-')) {
nextCursor = c?.itemContent?.value || c?.value || nextCursor;
continue;
}
// Direct tweet entry
const tw = extractTweet(c?.itemContent?.tweet_results?.result, seen);
if (tw) tweets.push(tw);
// Conversation module (nested replies)
for (const item of c?.items || []) {
const nested = extractTweet(item.item?.itemContent?.tweet_results?.result, seen);
if (nested) tweets.push(nested);
}
}
}
return { tweets, nextCursor };
}
// ── CLI definition ────────────────────────────────────────────────────
cli({
site: 'twitter',
name: 'thread',
description: 'Get a tweet thread (original + all replies)',
domain: 'x.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'tweet_id', type: 'string', required: true },
{ name: 'limit', type: 'int', default: 50 },
],
columns: ['id', 'author', 'text', 'likes', 'retweets', 'url'],
func: async (page, kwargs) => {
let tweetId = kwargs.tweet_id;
const urlMatch = tweetId.match(/\/status\/(\d+)/);
if (urlMatch) tweetId = urlMatch[1];
// Navigate to x.com for cookie context
await page.goto('https://x.com');
await page.wait(3);
// Extract CSRF token — the only thing we need from the browser
const ct0 = await page.evaluate(`() => {
return document.cookie.split(';').map(c=>c.trim()).find(c=>c.startsWith('ct0='))?.split('=')[1] || null;
}`);
if (!ct0) throw new Error('Not logged into x.com (no ct0 cookie)');
// Build auth headers in TypeScript
const headers = JSON.stringify({
'Authorization': `Bearer ${decodeURIComponent(BEARER_TOKEN)}`,
'X-Csrf-Token': ct0,
'X-Twitter-Auth-Type': 'OAuth2Session',
'X-Twitter-Active-User': 'yes',
});
// Paginate — fetch in browser, parse in TypeScript
const allTweets: ThreadTweet[] = [];
const seen = new Set<string>();
let cursor: string | null = null;
for (let i = 0; i < 5; i++) {
const apiUrl = buildTweetDetailUrl(tweetId, cursor);
// Browser-side: just fetch + return JSON (3 lines)
const data = await page.evaluate(`async () => {
const r = await fetch("${apiUrl}", { headers: ${headers}, credentials: 'include' });
return r.ok ? await r.json() : { error: r.status };
}`);
if (data?.error) {
if (allTweets.length === 0) throw new Error(`HTTP ${data.error}: Tweet not found or queryId expired`);
break;
}
// TypeScript-side: type-safe parsing + cursor extraction
const { tweets, nextCursor } = parseTweetDetail(data, seen);
allTweets.push(...tweets);
if (!nextCursor || nextCursor === cursor) break;
cursor = nextCursor;
}
return allTweets.slice(0, kwargs.limit);
},
});
+66
View File
@@ -0,0 +1,66 @@
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
cli({
site: 'twitter',
name: 'unbookmark',
description: 'Remove a tweet from bookmarks',
domain: 'x.com',
strategy: Strategy.UI,
browser: true,
args: [
{ name: 'url', type: 'string', positional: true, required: true, help: 'Tweet URL to unbookmark' },
],
columns: ['status', 'message'],
func: async (page: IPage | null, kwargs: any) => {
if (!page) throw new Error('Requires browser');
await page.goto(kwargs.url);
await page.wait(5);
const result = await page.evaluate(`(async () => {
try {
let attempts = 0;
let removeBtn = null;
while (attempts < 20) {
// Check if not bookmarked
const bookmarkBtn = document.querySelector('[data-testid="bookmark"]');
if (bookmarkBtn) {
return { ok: true, message: 'Tweet is not bookmarked (already removed).' };
}
removeBtn = document.querySelector('[data-testid="removeBookmark"]');
if (removeBtn) break;
await new Promise(r => setTimeout(r, 500));
attempts++;
}
if (!removeBtn) {
return { ok: false, message: 'Could not find Remove Bookmark button. Are you logged in?' };
}
removeBtn.click();
await new Promise(r => setTimeout(r, 1000));
// Verify
const verify = document.querySelector('[data-testid="bookmark"]');
if (verify) {
return { ok: true, message: 'Tweet successfully removed from bookmarks.' };
} else {
return { ok: false, message: 'Unbookmark action initiated but UI did not update.' };
}
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
if (result.ok) await page.wait(2);
return [{
status: result.ok ? 'success' : 'failed',
message: result.message
}];
}
});
+75
View File
@@ -0,0 +1,75 @@
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
cli({
site: 'twitter',
name: 'unfollow',
description: 'Unfollow a Twitter user',
domain: 'x.com',
strategy: Strategy.UI,
browser: true,
args: [
{ name: 'username', type: 'string', positional: true, required: true, help: 'Twitter screen name (without @)' },
],
columns: ['status', 'message'],
func: async (page: IPage | null, kwargs: any) => {
if (!page) throw new Error('Requires browser');
const username = kwargs.username.replace(/^@/, '');
await page.goto(`https://x.com/${username}`);
await page.wait(5);
const result = await page.evaluate(`(async () => {
try {
let attempts = 0;
let unfollowBtn = null;
while (attempts < 20) {
// Check if already not following
const followBtn = document.querySelector('[data-testid$="-follow"]');
if (followBtn) {
return { ok: true, message: 'Not following @${username} (already unfollowed).' };
}
unfollowBtn = document.querySelector('[data-testid$="-unfollow"]');
if (unfollowBtn) break;
await new Promise(r => setTimeout(r, 500));
attempts++;
}
if (!unfollowBtn) {
return { ok: false, message: 'Could not find Unfollow button. Are you logged in?' };
}
// Click the unfollow button — this opens a confirmation dialog
unfollowBtn.click();
await new Promise(r => setTimeout(r, 1000));
// Confirm the unfollow in the dialog
const confirmBtn = document.querySelector('[data-testid="confirmationSheetConfirm"]');
if (confirmBtn) {
confirmBtn.click();
await new Promise(r => setTimeout(r, 1000));
}
// Verify
const verify = document.querySelector('[data-testid$="-follow"]');
if (verify) {
return { ok: true, message: 'Successfully unfollowed @${username}.' };
} else {
return { ok: false, message: 'Unfollow action initiated but UI did not update.' };
}
} catch (e) {
return { ok: false, message: e.toString() };
}
})()`);
if (result.ok) await page.wait(2);
return [{
status: result.ok ? 'success' : 'failed',
message: result.message
}];
}
});
+108
View File
@@ -0,0 +1,108 @@
import { describe, it, expect } from 'vitest';
import { groupTranscriptSegments, formatGroupedTranscript } from './transcript-group.js';
describe('groupTranscriptSegments', () => {
it('groups segments by sentence boundaries', () => {
const segments = [
{ start: 0, text: 'Hello there.' },
{ start: 2, text: 'How are you doing today?' },
{ start: 5, text: 'I am' },
{ start: 6, text: 'doing well.' },
];
const result = groupTranscriptSegments(segments);
expect(result).toHaveLength(3);
expect(result[0].text).toBe('Hello there.');
expect(result[1].text).toBe('How are you doing today?');
expect(result[2].text).toBe('I am doing well.');
});
it('flushes on large time gaps', () => {
const segments = [
{ start: 0, text: 'First part' },
{ start: 2, text: 'still first' },
{ start: 25, text: 'second part after gap' },
];
const result = groupTranscriptSegments(segments);
expect(result).toHaveLength(2);
expect(result[0].text).toBe('First part still first');
expect(result[1].text).toBe('second part after gap');
});
it('respects 30s max group span for unpunctuated text', () => {
// Simulate CJK captions without punctuation
const segments = Array.from({ length: 20 }, (_, i) => ({
start: i * 2,
text: `segment${i}`,
}));
const result = groupTranscriptSegments(segments);
// 20 segments * 2s = 40s total, should be split into at least 2 groups
expect(result.length).toBeGreaterThanOrEqual(2);
// No single group should span more than ~30s
for (const g of result) {
const words = g.text.split(' ');
// With 2s per segment and 30s max, each group should have at most ~16 segments
expect(words.length).toBeLessThanOrEqual(16);
}
});
it('detects speaker changes via >> markers', () => {
const segments = [
{ start: 0, text: '>> How are you?' },
{ start: 3, text: '>> I am fine.' },
];
const result = groupTranscriptSegments(segments);
expect(result.some(g => g.speakerChange)).toBe(true);
expect(result.some(g => g.speaker !== undefined)).toBe(true);
});
it('recognizes CJK sentence-ending punctuation', () => {
const segments = [
{ start: 0, text: '你好世界。' },
{ start: 2, text: '这是测试' },
{ start: 4, text: '内容。' },
];
const result = groupTranscriptSegments(segments);
expect(result).toHaveLength(2);
expect(result[0].text).toBe('你好世界。');
expect(result[1].text).toBe('这是测试 内容。');
});
it('returns empty array for empty input', () => {
expect(groupTranscriptSegments([])).toEqual([]);
});
});
describe('formatGroupedTranscript', () => {
it('formats timestamps correctly', () => {
const segments = [
{ start: 65, text: 'One minute five.', speakerChange: false },
{ start: 3661, text: 'One hour one minute.', speakerChange: false },
];
const { rows } = formatGroupedTranscript(segments);
expect(rows[0].timestamp).toBe('1:05');
expect(rows[1].timestamp).toBe('1:01:01');
});
it('inserts chapter headings at correct positions', () => {
const segments = [
{ start: 0, text: 'Intro text.', speakerChange: false },
{ start: 60, text: 'Chapter content.', speakerChange: false },
];
const chapters = [{ title: 'Introduction', start: 0 }, { title: 'Main', start: 50 }];
const { rows } = formatGroupedTranscript(segments, chapters);
expect(rows[0].text).toBe('[Chapter] Introduction');
expect(rows[1].text).toBe('Intro text.');
expect(rows[2].text).toBe('[Chapter] Main');
expect(rows[3].text).toBe('Chapter content.');
});
it('labels speakers', () => {
const segments = [
{ start: 0, text: 'Hello.', speakerChange: true, speaker: 0 },
{ start: 5, text: 'Hi there.', speakerChange: true, speaker: 1 },
];
const { rows } = formatGroupedTranscript(segments);
expect(rows[0].speaker).toBe('Speaker 1');
expect(rows[1].speaker).toBe('Speaker 2');
});
});
+287
View File
@@ -0,0 +1,287 @@
/**
* Transcript grouping: sentence merging, speaker detection, and chapter support.
* Ported and simplified from Defuddle's YouTube extractor.
*
* Raw segments (2-3 second fragments) are grouped into readable paragraphs:
* - Sentence boundaries: merge until sentence-ending punctuation (.!?)
* - Speaker turns: detect ">>" markers from YouTube auto-captions
* - Chapters: optional chapter headings inserted at appropriate timestamps
*/
// Include CJK sentence-ending punctuation: 。!? (fullwidth: .!?)
const SENTENCE_END = /[.!?\u3002\uFF01\uFF1F\uFF0E]["'\u2019\u201D)]*\s*$/;
const QUESTION_END = /[?\uFF1F]["'\u2019\u201D)]*\s*$/;
const TRANSCRIPT_GROUP_GAP_SECONDS = 20;
const TURN_MERGE_MAX_WORDS = 80;
const TURN_MERGE_MAX_SPAN_SECONDS = 45;
const SHORT_UTTERANCE_MAX_WORDS = 3;
const FIRST_GROUP_MERGE_MIN_WORDS = 8;
export interface RawSegment {
start: number;
end: number;
text: string;
}
export interface GroupedSegment {
start: number;
text: string;
speakerChange: boolean;
speaker?: number;
}
export interface Chapter {
title: string;
start: number;
}
function countWords(text: string): number {
return text.split(/\s+/).filter(Boolean).length;
}
/**
* Group raw transcript segments into readable blocks.
* If speaker markers (>>) are present, groups by speaker turn.
* Otherwise, groups by sentence boundaries.
*/
export function groupTranscriptSegments(
segments: { start: number; text: string }[],
): GroupedSegment[] {
if (segments.length === 0) return [];
const hasSpeakerMarkers = segments.some(s => /^>>/.test(s.text));
return hasSpeakerMarkers ? groupBySpeaker(segments) : groupBySentence(segments);
}
/**
* Format grouped segments + chapters into a final text output.
*/
export function formatGroupedTranscript(
segments: GroupedSegment[],
chapters: Chapter[] = [],
): { rows: Array<{ timestamp: string; speaker: string; text: string }>; plainText: string } {
const sortedChapters = [...chapters].sort((a, b) => a.start - b.start);
let chapterIdx = 0;
const rows: Array<{ timestamp: string; speaker: string; text: string }> = [];
const textParts: string[] = [];
for (const segment of segments) {
// Insert chapter headings
while (chapterIdx < sortedChapters.length && sortedChapters[chapterIdx].start <= segment.start) {
const title = sortedChapters[chapterIdx].title;
rows.push({ timestamp: fmtTime(sortedChapters[chapterIdx].start), speaker: '', text: `[Chapter] ${title}` });
if (textParts.length > 0) textParts.push('');
textParts.push(`### ${title}`);
textParts.push('');
chapterIdx++;
}
const timestamp = fmtTime(segment.start);
const speaker = segment.speaker !== undefined ? `Speaker ${segment.speaker + 1}` : '';
rows.push({ timestamp, speaker, text: segment.text });
if (segment.speakerChange && textParts.length > 0) {
textParts.push('');
}
textParts.push(`${timestamp} ${segment.text}`);
}
return { rows, plainText: textParts.join('\n') };
}
function fmtTime(sec: number): string {
const h = Math.floor(sec / 3600);
const m = Math.floor((sec % 3600) / 60);
const s = Math.floor(sec % 60);
if (h > 0) {
return `${h}:${String(m).padStart(2, '0')}:${String(s).padStart(2, '0')}`;
}
return `${m}:${String(s).padStart(2, '0')}`;
}
// ── Sentence grouping ─────────────────────────────────────────────────────
// Max time span (seconds) for a single group when no sentence boundaries are found.
// Prevents unbounded merging for languages without punctuation (Chinese, etc.).
const MAX_GROUP_SPAN_SECONDS = 30;
function groupBySentence(
segments: { start: number; text: string }[],
): GroupedSegment[] {
const groups: GroupedSegment[] = [];
let buffer = '';
let bufferStart = 0;
let lastStart = 0;
const flush = () => {
if (buffer.trim()) {
groups.push({ start: bufferStart, text: buffer.trim(), speakerChange: false });
buffer = '';
}
};
for (const seg of segments) {
// Large gap between segments — always flush
if (buffer && seg.start - lastStart > TRANSCRIPT_GROUP_GAP_SECONDS) {
flush();
}
// Time-based flush: prevent unbounded groups for unpunctuated languages
if (buffer && seg.start - bufferStart > MAX_GROUP_SPAN_SECONDS) {
flush();
}
if (!buffer) bufferStart = seg.start;
buffer += (buffer ? ' ' : '') + seg.text;
lastStart = seg.start;
if (SENTENCE_END.test(seg.text)) flush();
}
flush();
return groups;
}
// ── Speaker grouping ──────────────────────────────────────────────────────
function groupBySpeaker(
segments: { start: number; text: string }[],
): GroupedSegment[] {
type Turn = {
start: number;
segments: { start: number; text: string }[];
speakerChange: boolean;
speaker?: number;
};
const turns: Turn[] = [];
let currentTurn: Turn | null = null;
let speakerIndex = -1;
let prevSegText = '';
for (const seg of segments) {
const isSpeakerChange = /^>>/.test(seg.text);
const cleanText = seg.text.replace(/^>>\s*/, '').replace(/^-\s+/, '');
const prevEndsWithComma = /,\s*$/.test(prevSegText);
const prevEndedSentence = (SENTENCE_END.test(prevSegText) || !prevSegText) && !prevEndsWithComma;
const isRealSpeakerChange = isSpeakerChange && prevEndedSentence;
if (isRealSpeakerChange) {
if (currentTurn) turns.push(currentTurn);
speakerIndex = (speakerIndex + 1) % 2;
currentTurn = {
start: seg.start,
segments: [{ start: seg.start, text: cleanText }],
speakerChange: true,
speaker: speakerIndex,
};
} else {
if (!currentTurn) {
currentTurn = { start: seg.start, segments: [], speakerChange: false };
}
currentTurn.segments.push({ start: seg.start, text: cleanText });
}
prevSegText = cleanText;
}
if (currentTurn) turns.push(currentTurn);
splitAffirmativeTurns(turns);
const groups: GroupedSegment[] = [];
for (const turn of turns) {
const sentenceGroups = turn.speaker === undefined
? groupBySentence(turn.segments)
: mergeSentenceGroupsWithinTurn(groupBySentence(turn.segments));
for (let i = 0; i < sentenceGroups.length; i++) {
groups.push({
...sentenceGroups[i],
speakerChange: i === 0 && turn.speakerChange,
speaker: turn.speaker,
});
}
}
return groups;
}
function splitAffirmativeTurns(turns: Array<{
start: number;
segments: { start: number; text: string }[];
speakerChange: boolean;
speaker?: number;
}>): void {
const affirmativePattern = /^(mhm|yeah|yes|yep|right|okay|ok|absolutely|sure|exactly|uh-huh|mm-hmm)[.!,]?\s+/i;
for (let i = 0; i < turns.length; i++) {
const turn = turns[i];
if (turn.speaker === undefined || turn.segments.length === 0) continue;
const firstSeg = turn.segments[0];
const match = affirmativePattern.exec(firstSeg.text);
if (!match) continue;
if (/,\s*$/.test(match[0])) continue;
const remainder = firstSeg.text.slice(match[0].length).trim();
const restSegments = turn.segments.slice(1);
const restWords = countWords(remainder) + restSegments.reduce((sum, s) => sum + countWords(s.text), 0);
if (restWords < 30) continue;
const affirmativeText = match[0].trimEnd();
const newRestSegments = remainder
? [{ start: firstSeg.start, text: remainder }, ...restSegments]
: restSegments;
turns.splice(i, 1, {
start: turn.start,
segments: [{ start: firstSeg.start, text: affirmativeText }],
speakerChange: turn.speakerChange,
speaker: turn.speaker,
}, {
start: newRestSegments[0].start,
segments: newRestSegments,
speakerChange: true,
speaker: turn.speaker === 0 ? 1 : 0,
});
i++;
}
}
function mergeSentenceGroupsWithinTurn(groups: GroupedSegment[]): GroupedSegment[] {
if (groups.length <= 1) return groups;
const merged: GroupedSegment[] = [];
let current = { ...groups[0] };
let currentIsFirstInTurn = true;
for (let i = 1; i < groups.length; i++) {
const next = groups[i];
if (shouldMergeSentenceGroups(current, next, currentIsFirstInTurn)) {
current.text = `${current.text} ${next.text}`;
continue;
}
merged.push(current);
current = { ...next };
currentIsFirstInTurn = false;
}
merged.push(current);
return merged;
}
function shouldMergeSentenceGroups(
current: { start: number; text: string },
next: { start: number; text: string },
currentIsFirstInTurn: boolean,
): boolean {
const currentWords = countWords(current.text);
const nextWords = countWords(next.text);
if (isShortStandaloneUtterance(current.text, currentWords)
|| isShortStandaloneUtterance(next.text, nextWords)) return false;
if (currentIsFirstInTurn && currentWords < FIRST_GROUP_MERGE_MIN_WORDS) return false;
if (QUESTION_END.test(current.text) || QUESTION_END.test(next.text)) return false;
if (currentWords + nextWords > TURN_MERGE_MAX_WORDS) return false;
if (next.start - current.start > TURN_MERGE_MAX_SPAN_SECONDS) return false;
return true;
}
function isShortStandaloneUtterance(text: string, words?: number): boolean {
const w = words ?? countWords(text);
return w > 0 && w <= SHORT_UTTERANCE_MAX_WORDS && SENTENCE_END.test(text);
}
+280
View File
@@ -0,0 +1,280 @@
/**
* YouTube transcript — uses InnerTube player API with Android client context.
*
* The Web client's caption URLs require a PoToken (proof of origin) generated
* by BotGuard at runtime. The Android client returns caption URLs that work
* without PoToken — same approach used by youtube-transcript-api (Python).
*
* Modes:
* --mode grouped (default): sentences merged, speaker detection, chapters
* --mode raw: every caption segment as-is with precise timestamps
*/
import { cli, Strategy } from '../../registry.js';
import { parseVideoId } from './utils.js';
import {
groupTranscriptSegments,
formatGroupedTranscript,
type RawSegment,
type Chapter,
} from './transcript-group.js';
cli({
site: 'youtube',
name: 'transcript',
description: 'Get YouTube video transcript/subtitles',
domain: 'www.youtube.com',
strategy: Strategy.COOKIE,
args: [
{ name: 'url', required: true, help: 'YouTube video URL or video ID' },
{ name: 'lang', required: false, help: 'Language code (e.g. en, zh-Hans). Omit to auto-select' },
{ name: 'mode', required: false, default: 'grouped', help: 'Output mode: grouped (readable paragraphs) or raw (every segment)' },
],
// columns intentionally omitted — raw and grouped modes return different schemas,
// so we let the renderer auto-detect columns from the data keys.
func: async (page, kwargs) => {
const videoId = parseVideoId(kwargs.url);
const videoUrl = `https://www.youtube.com/watch?v=${videoId}`;
await page.goto(videoUrl);
await page.wait(3);
const lang = kwargs.lang || '';
const mode = kwargs.mode || 'grouped';
// Step 1: Get caption track URL via Android InnerTube API
const captionData = await page.evaluate(`
(async () => {
const cfg = window.ytcfg?.data_ || {};
const apiKey = cfg.INNERTUBE_API_KEY;
if (!apiKey) return { error: 'INNERTUBE_API_KEY not found on page' };
const resp = await fetch('/youtubei/v1/player?key=' + apiKey + '&prettyPrint=false', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
context: { client: { clientName: 'ANDROID', clientVersion: '20.10.38' } },
videoId: ${JSON.stringify(videoId)}
})
});
if (!resp.ok) return { error: 'InnerTube player API returned HTTP ' + resp.status };
const data = await resp.json();
const renderer = data.captions?.playerCaptionsTracklistRenderer;
if (!renderer?.captionTracks?.length) {
return { error: 'No captions available for this video' };
}
const tracks = renderer.captionTracks;
const available = tracks.map(t => t.languageCode + (t.kind === 'asr' ? ' (auto)' : ''));
const langPref = ${JSON.stringify(lang)};
let track = null;
if (langPref) {
track = tracks.find(t => t.languageCode === langPref)
|| tracks.find(t => t.languageCode.startsWith(langPref));
}
if (!track) {
track = tracks.find(t => t.kind !== 'asr') || tracks[0];
}
return {
captionUrl: track.baseUrl,
language: track.languageCode,
kind: track.kind || 'manual',
available,
requestedLang: langPref || null,
langMatched: !!(langPref && track.languageCode === langPref),
langPrefixMatched: !!(langPref && track.languageCode !== langPref && track.languageCode.startsWith(langPref))
};
})()
`);
if (!captionData || typeof captionData === 'string') {
throw new Error(`Failed to get caption info: ${typeof captionData === 'string' ? captionData : 'null response'}`);
}
if (captionData.error) {
throw new Error(`${captionData.error}${captionData.available ? ' (available: ' + captionData.available.join(', ') + ')' : ''}`);
}
// Warn if --lang was specified but not matched
if (captionData.requestedLang && !captionData.langMatched && !captionData.langPrefixMatched) {
console.error(`Warning: --lang "${captionData.requestedLang}" not found. Using "${captionData.language}" instead. Available: ${captionData.available.join(', ')}`);
}
// Step 2: Fetch caption XML and parse segments
const segments: RawSegment[] = await page.evaluate(`
(async () => {
const resp = await fetch(${JSON.stringify(captionData.captionUrl)});
const xml = await resp.text();
if (!xml?.length) {
return { error: 'Caption URL returned empty response' };
}
function getAttr(tag, name) {
const needle = name + '="';
const idx = tag.indexOf(needle);
if (idx === -1) return '';
const valStart = idx + needle.length;
const valEnd = tag.indexOf('"', valStart);
if (valEnd === -1) return '';
return tag.substring(valStart, valEnd);
}
function decodeEntities(s) {
return s
.replaceAll('&amp;', '&')
.replaceAll('&lt;', '<')
.replaceAll('&gt;', '>')
.replaceAll('&quot;', '"')
.replaceAll('&#39;', "'");
}
const isFormat3 = xml.includes('<p t="');
const marker = isFormat3 ? '<p ' : '<text ';
const endMarker = isFormat3 ? '</p>' : '</text>';
const results = [];
let pos = 0;
while (true) {
const tagStart = xml.indexOf(marker, pos);
if (tagStart === -1) break;
let contentStart = xml.indexOf('>', tagStart);
if (contentStart === -1) break;
contentStart += 1;
const tagEnd = xml.indexOf(endMarker, contentStart);
if (tagEnd === -1) break;
const attrStr = xml.substring(tagStart + marker.length, contentStart - 1);
const content = xml.substring(contentStart, tagEnd);
let startSec, durSec;
if (isFormat3) {
startSec = (parseFloat(getAttr(attrStr, 't')) || 0) / 1000;
durSec = (parseFloat(getAttr(attrStr, 'd')) || 0) / 1000;
} else {
startSec = parseFloat(getAttr(attrStr, 'start')) || 0;
durSec = parseFloat(getAttr(attrStr, 'dur')) || 0;
}
// Strip inner tags (e.g. <s> in srv3 format) and decode entities
const text = decodeEntities(content.replace(/<[^>]+>/g, '')).split('\\\\n').join(' ').trim();
if (text) {
results.push({ start: startSec, end: startSec + durSec, text });
}
pos = tagEnd + endMarker.length;
}
if (results.length === 0) {
return { error: 'Parsed 0 segments from caption XML' };
}
return results;
})()
`);
if (!Array.isArray(segments)) {
throw new Error((segments as any)?.error || 'Failed to parse caption segments');
}
if (segments.length === 0) {
throw new Error('No caption segments found');
}
// Step 3: Fetch chapters (for grouped mode)
let chapters: Chapter[] = [];
if (mode === 'grouped') {
try {
const chapterData = await page.evaluate(`
(async () => {
const cfg = window.ytcfg?.data_ || {};
const apiKey = cfg.INNERTUBE_API_KEY;
if (!apiKey) return [];
const resp = await fetch('/youtubei/v1/next?key=' + apiKey + '&prettyPrint=false', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
context: { client: { clientName: 'WEB', clientVersion: '2.20240101.00.00' } },
videoId: ${JSON.stringify(videoId)}
})
});
if (!resp.ok) return [];
const data = await resp.json();
const chapters = [];
// Try chapterRenderer from player bar
const panels = data.playerOverlays?.playerOverlayRenderer
?.decoratedPlayerBarRenderer?.decoratedPlayerBarRenderer
?.playerBar?.multiMarkersPlayerBarRenderer?.markersMap;
if (Array.isArray(panels)) {
for (const panel of panels) {
const markers = panel.value?.chapters;
if (!Array.isArray(markers)) continue;
for (const marker of markers) {
const ch = marker.chapterRenderer;
if (!ch) continue;
const title = ch.title?.simpleText || '';
const startMs = ch.timeRangeStartMillis;
if (title && typeof startMs === 'number') {
chapters.push({ title, start: startMs / 1000 });
}
}
}
}
if (chapters.length > 0) return chapters;
// Fallback: macroMarkersListItemRenderer from engagement panels
const engPanels = data.engagementPanels;
if (!Array.isArray(engPanels)) return [];
for (const ep of engPanels) {
const content = ep.engagementPanelSectionListRenderer?.content;
const items = content?.macroMarkersListRenderer?.contents;
if (!Array.isArray(items)) continue;
for (const item of items) {
const renderer = item.macroMarkersListItemRenderer;
if (!renderer) continue;
const t = renderer.title?.simpleText || '';
const ts = renderer.timeDescription?.simpleText || '';
if (!t || !ts) continue;
const parts = ts.split(':').map(Number);
let secs = null;
if (parts.length === 3 && parts.every(n => !isNaN(n))) secs = parts[0]*3600 + parts[1]*60 + parts[2];
else if (parts.length === 2 && parts.every(n => !isNaN(n))) secs = parts[0]*60 + parts[1];
if (secs !== null) chapters.push({ title: t, start: secs });
}
}
return chapters;
})()
`);
if (Array.isArray(chapterData)) {
chapters = chapterData;
}
} catch {
// Chapters are optional — proceed without them
}
}
// Step 4: Format output based on mode
if (mode === 'raw') {
// Precise timestamps in seconds with decimals, matching bilibili/subtitle format
return segments.map((seg, i) => ({
index: i + 1,
start: Number(seg.start).toFixed(2) + 's',
end: Number(seg.end).toFixed(2) + 's',
text: seg.text,
}));
}
// Grouped mode: merge sentences, detect speakers, insert chapters
const grouped = groupTranscriptSegments(
segments.map(s => ({ start: s.start, text: s.text })),
);
const { rows } = formatGroupedTranscript(grouped, chapters);
return rows;
},
});
+28
View File
@@ -0,0 +1,28 @@
/**
* Shared YouTube utilities — URL parsing, video ID extraction, etc.
*/
/**
* Extract a YouTube video ID from a URL or bare video ID string.
* Supports: watch?v=, youtu.be/, /shorts/, /embed/, /live/, /v/
*/
export function parseVideoId(input: string): string {
if (!input.startsWith('http')) return input;
try {
const parsed = new URL(input);
if (parsed.searchParams.has('v')) {
return parsed.searchParams.get('v')!;
}
if (parsed.hostname === 'youtu.be') {
return parsed.pathname.slice(1).split('/')[0];
}
// Handle /shorts/xxx, /embed/xxx, /live/xxx, /v/xxx
const pathMatch = parsed.pathname.match(/^\/(shorts|embed|live|v)\/([^/?]+)/);
if (pathMatch) return pathMatch[2];
} catch {
// Not a valid URL — treat entire input as video ID
}
return input;
}
+116
View File
@@ -0,0 +1,116 @@
/**
* YouTube video metadata — read ytInitialPlayerResponse + ytInitialData from video page.
*/
import { cli, Strategy } from '../../registry.js';
import { parseVideoId } from './utils.js';
cli({
site: 'youtube',
name: 'video',
description: 'Get YouTube video metadata (title, views, description, etc.)',
domain: 'www.youtube.com',
strategy: Strategy.COOKIE,
args: [
{ name: 'url', required: true, help: 'YouTube video URL or video ID' },
],
columns: ['field', 'value'],
func: async (page, kwargs) => {
const videoId = parseVideoId(kwargs.url);
const videoUrl = `https://www.youtube.com/watch?v=${videoId}`;
await page.goto(videoUrl);
await page.wait(3);
const data = await page.evaluate(`
(async () => {
const player = window.ytInitialPlayerResponse;
const yt = window.ytInitialData;
if (!player) return { error: 'ytInitialPlayerResponse not found' };
const details = player.videoDetails || {};
const microformat = player.microformat?.playerMicroformatRenderer || {};
// Try to get full description from ytInitialData
let fullDescription = details.shortDescription || '';
try {
const contents = yt?.contents?.twoColumnWatchNextResults
?.results?.results?.contents;
if (contents) {
for (const c of contents) {
const desc = c.videoSecondaryInfoRenderer?.attributedDescription?.content;
if (desc) { fullDescription = desc; break; }
}
}
} catch {}
// Get like count if available
let likes = '';
try {
const contents = yt?.contents?.twoColumnWatchNextResults
?.results?.results?.contents;
if (contents) {
for (const c of contents) {
const buttons = c.videoPrimaryInfoRenderer?.videoActions
?.menuRenderer?.topLevelButtons;
if (buttons) {
for (const b of buttons) {
const toggle = b.segmentedLikeDislikeButtonViewModel
?.likeButtonViewModel?.likeButtonViewModel?.toggleButtonViewModel
?.toggleButtonViewModel?.defaultButtonViewModel?.buttonViewModel;
if (toggle?.title) { likes = toggle.title; break; }
}
}
}
}
} catch {}
// Get publish date
const publishDate = microformat.publishDate
|| microformat.uploadDate
|| details.publishDate || '';
// Get category
const category = microformat.category || '';
// Get channel subscriber count if available
let subscribers = '';
try {
const contents = yt?.contents?.twoColumnWatchNextResults
?.results?.results?.contents;
if (contents) {
for (const c of contents) {
const owner = c.videoSecondaryInfoRenderer?.owner
?.videoOwnerRenderer?.subscriberCountText?.simpleText;
if (owner) { subscribers = owner; break; }
}
}
} catch {}
return {
title: details.title || '',
channel: details.author || '',
channelId: details.channelId || '',
videoId: details.videoId || '',
views: details.viewCount || '',
likes,
subscribers,
duration: details.lengthSeconds ? details.lengthSeconds + 's' : '',
publishDate,
category,
description: fullDescription,
keywords: (details.keywords || []).join(', '),
isLive: details.isLiveContent || false,
thumbnail: details.thumbnail?.thumbnails?.slice(-1)?.[0]?.url || '',
};
})()
`);
if (!data || typeof data !== 'object') throw new Error('Failed to extract video metadata from page');
if (data.error) throw new Error(data.error);
// Return as field/value pairs for table display
return Object.entries(data).map(([field, value]) => ({
field,
value: String(value),
}));
},
});
+105 -7
View File
@@ -81,45 +81,143 @@ describe('json token helpers', () => {
},
}), 'abc123');
const parsed = JSON.parse(next);
expect(parsed.mcp.playwright.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN).toBe('abc123');
expect(parsed.mcp.playwright.environment.PLAYWRIGHT_MCP_EXTENSION_TOKEN).toBe('abc123');
});
it('creates standard mcpServers format for empty file (not OpenCode)', () => {
const next = upsertJsonConfigToken('', 'abc123');
const parsed = JSON.parse(next);
expect(parsed.mcpServers.playwright.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN).toBe('abc123');
expect(parsed.mcp).toBeUndefined();
});
it('creates OpenCode format when filePath contains opencode', () => {
const next = upsertJsonConfigToken('', 'abc123', '/home/user/.config/opencode/opencode.json');
const parsed = JSON.parse(next);
expect(parsed.mcp.playwright.environment.PLAYWRIGHT_MCP_EXTENSION_TOKEN).toBe('abc123');
expect(parsed.mcpServers).toBeUndefined();
});
it('creates standard format when filePath is claude.json', () => {
const next = upsertJsonConfigToken('', 'abc123', '/home/user/.claude.json');
const parsed = JSON.parse(next);
expect(parsed.mcpServers.playwright.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN).toBe('abc123');
});
});
describe('fish shell support', () => {
it('generates fish set -gx syntax for fish config path', () => {
const next = upsertShellToken('', 'abc123', '/home/user/.config/fish/config.fish');
expect(next).toContain('set -gx PLAYWRIGHT_MCP_EXTENSION_TOKEN "abc123"');
expect(next).not.toContain('export');
});
it('replaces existing fish set line', () => {
const content = 'set -gx PLAYWRIGHT_MCP_EXTENSION_TOKEN "old"\n';
const next = upsertShellToken(content, 'new', '/home/user/.config/fish/config.fish');
expect(next).toContain('set -gx PLAYWRIGHT_MCP_EXTENSION_TOKEN "new"');
expect(next).not.toContain('"old"');
});
it('appends fish syntax to existing fish config', () => {
const content = 'set -gx PATH /usr/bin\n';
const next = upsertShellToken(content, 'abc123', '/home/user/.config/fish/config.fish');
expect(next).toContain('set -gx PLAYWRIGHT_MCP_EXTENSION_TOKEN "abc123"');
expect(next).toContain('set -gx PATH /usr/bin');
});
it('uses export syntax for zshrc even with filePath', () => {
const next = upsertShellToken('', 'abc123', '/home/user/.zshrc');
expect(next).toContain('export PLAYWRIGHT_MCP_EXTENSION_TOKEN="abc123"');
expect(next).not.toContain('set -gx');
});
});
describe('doctor report rendering', () => {
const strip = (s: string) => s.replace(/\x1b\[[0-9;]*m/g, '');
it('renders OK-style report when tokens match', () => {
const text = renderBrowserDoctorReport({
const text = strip(renderBrowserDoctorReport({
envToken: 'abc123',
envFingerprint: 'fp1',
extensionToken: 'abc123',
extensionFingerprint: 'fp1',
extensionInstalled: true,
extensionBrowsers: ['Chrome'],
shellFiles: [{ path: '/tmp/.zshrc', exists: true, token: 'abc123', fingerprint: 'fp1' }],
configs: [{ path: '/tmp/mcp.json', exists: true, format: 'json', token: 'abc123', fingerprint: 'fp1', writable: true }],
recommendedToken: 'abc123',
recommendedFingerprint: 'fp1',
warnings: [],
issues: [],
});
}));
expect(text).toContain('[OK] Extension installed (Chrome)');
expect(text).toContain('[OK] Environment token: configured (fp1)');
expect(text).toContain('[OK] MCP config /tmp/mcp.json: configured (fp1)');
expect(text).toContain('[OK] /tmp/mcp.json');
expect(text).toContain('configured (fp1)');
});
it('renders MISMATCH-style report when fingerprints differ', () => {
const text = renderBrowserDoctorReport({
const text = strip(renderBrowserDoctorReport({
envToken: 'abc123',
envFingerprint: 'fp1',
extensionToken: null,
extensionFingerprint: null,
extensionInstalled: false,
extensionBrowsers: [],
shellFiles: [{ path: '/tmp/.zshrc', exists: true, token: 'def456', fingerprint: 'fp2' }],
configs: [{ path: '/tmp/mcp.json', exists: true, format: 'json', token: 'abc123', fingerprint: 'fp1', writable: true }],
recommendedToken: 'abc123',
recommendedFingerprint: 'fp1',
warnings: [],
issues: ['Detected inconsistent Playwright MCP tokens across env/config files.'],
});
}));
expect(text).toContain('[MISSING] Extension not installed in any browser');
expect(text).toContain('[MISMATCH] Environment token: configured (fp1)');
expect(text).toContain('[MISMATCH] Shell file /tmp/.zshrc: configured (fp2)');
expect(text).toContain('[MISMATCH] /tmp/.zshrc');
expect(text).toContain('configured (fp2)');
expect(text).toContain('[MISMATCH] Recommended token fingerprint: fp1');
});
it('renders connectivity OK when live test succeeds', () => {
const text = strip(renderBrowserDoctorReport({
envToken: 'abc123',
envFingerprint: 'fp1',
extensionToken: 'abc123',
extensionFingerprint: 'fp1',
extensionInstalled: true,
extensionBrowsers: ['Chrome'],
shellFiles: [],
configs: [],
recommendedToken: 'abc123',
recommendedFingerprint: 'fp1',
connectivity: { ok: true, durationMs: 1234 },
warnings: [],
issues: [],
}));
expect(text).toContain('[OK] Browser connectivity: connected in 1.2s');
});
it('renders connectivity WARN when not tested', () => {
const text = strip(renderBrowserDoctorReport({
envToken: 'abc123',
envFingerprint: 'fp1',
extensionToken: 'abc123',
extensionFingerprint: 'fp1',
extensionInstalled: true,
extensionBrowsers: ['Chrome'],
shellFiles: [],
configs: [],
recommendedToken: 'abc123',
recommendedFingerprint: 'fp1',
warnings: [],
issues: [],
}));
expect(text).toContain('[WARN] Browser connectivity: not tested (use --live)');
});
});
+272 -89
View File
@@ -1,20 +1,22 @@
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { execSync } from 'node:child_process';
import { createInterface } from 'node:readline/promises';
import { stdin as input, stdout as output } from 'node:process';
import chalk from 'chalk';
import type { IPage } from './types.js';
import { PlaywrightMCP, getTokenFingerprint } from './browser.js';
import { browserSession } from './runtime.js';
const PLAYWRIGHT_SERVER_NAME = 'playwright';
const PLAYWRIGHT_TOKEN_ENV = 'PLAYWRIGHT_MCP_EXTENSION_TOKEN';
export const PLAYWRIGHT_TOKEN_ENV = 'PLAYWRIGHT_MCP_EXTENSION_TOKEN';
const PLAYWRIGHT_EXTENSION_ID = 'mmlmfjhmonkocbjadbfplnigmagldckm';
const TOKEN_LINE_RE = /^(\s*export\s+PLAYWRIGHT_MCP_EXTENSION_TOKEN=)(['"]?)([^'"\\\n]+)\2\s*$/m;
export type DoctorOptions = {
fix?: boolean;
yes?: boolean;
live?: boolean;
shellRc?: string;
configPaths?: string[];
token?: string;
@@ -40,33 +42,66 @@ export type McpConfigStatus = {
parseError?: string;
};
export type ConnectivityResult = {
ok: boolean;
error?: string;
durationMs: number;
};
export type DoctorReport = {
cliVersion?: string;
envToken: string | null;
envFingerprint: string | null;
extensionToken: string | null;
extensionFingerprint: string | null;
extensionInstalled: boolean;
extensionBrowsers: string[];
shellFiles: ShellFileStatus[];
configs: McpConfigStatus[];
recommendedToken: string | null;
recommendedFingerprint: string | null;
connectivity?: ConnectivityResult;
warnings: string[];
issues: string[];
};
type ReportStatus = 'OK' | 'MISSING' | 'MISMATCH' | 'WARN';
function label(status: ReportStatus): string {
return `[${status}]`;
function colorLabel(status: ReportStatus): string {
switch (status) {
case 'OK': return chalk.green('[OK]');
case 'MISSING': return chalk.red('[MISSING]');
case 'MISMATCH': return chalk.yellow('[MISMATCH]');
case 'WARN': return chalk.yellow('[WARN]');
}
}
function statusLine(status: ReportStatus, text: string): string {
return `${label(status)} ${text}`;
return `${colorLabel(status)} ${text}`;
}
function tokenSummary(token: string | null, fingerprint: string | null): string {
if (!token) return 'missing';
return `configured (${fingerprint})`;
if (!token) return chalk.dim('missing');
return `configured ${chalk.dim(`(${fingerprint})`)}`;
}
export function shortenPath(p: string): string {
const home = os.homedir();
return home && p.startsWith(home) ? '~' + p.slice(home.length) : p;
}
export function toolName(p: string): string {
if (p.includes('.codex/')) return 'Codex';
if (p.includes('.cursor/')) return 'Cursor';
if (p.includes('.claude.json')) return 'Claude Code';
if (p.includes('antigravity')) return 'Antigravity';
if (p.includes('.gemini/settings')) return 'Gemini CLI';
if (p.includes('opencode')) return 'OpenCode';
if (p.includes('Claude/claude_desktop')) return 'Claude Desktop';
if (p.includes('.vscode/')) return 'VS Code';
if (p.includes('.mcp.json')) return 'Project MCP';
if (p.includes('.zshrc') || p.includes('.bashrc') || p.includes('.profile')) return 'Shell';
return '';
}
export function getDefaultShellRcPath(): string {
@@ -76,6 +111,15 @@ export function getDefaultShellRcPath(): string {
return path.join(os.homedir(), '.zshrc');
}
function isFishConfig(filePath: string): boolean {
return filePath.endsWith('config.fish') || filePath.includes('/fish/');
}
/** Detect if a JSON config file uses OpenCode's `mcp` format vs standard `mcpServers` */
function isOpenCodeConfig(filePath: string): boolean {
return filePath.includes('opencode');
}
export function getDefaultMcpConfigPaths(cwd: string = process.cwd()): string[] {
const home = os.homedir();
const candidates = [
@@ -101,7 +145,15 @@ export function readTokenFromShellContent(content: string): string | null {
return m?.[3] ?? null;
}
export function upsertShellToken(content: string, token: string): string {
export function upsertShellToken(content: string, token: string, filePath?: string): string {
if (filePath && isFishConfig(filePath)) {
// Fish shell uses `set -gx` instead of `export`
const fishLine = `set -gx ${PLAYWRIGHT_TOKEN_ENV} "${token}"`;
const fishRe = /^\s*set\s+(-gx\s+)?PLAYWRIGHT_MCP_EXTENSION_TOKEN\s+.*/m;
if (!content.trim()) return `${fishLine}\n`;
if (fishRe.test(content)) return content.replace(fishRe, fishLine);
return `${content.replace(/\s*$/, '')}\n${fishLine}\n`;
}
const nextLine = `export ${PLAYWRIGHT_TOKEN_ENV}="${token}"`;
if (!content.trim()) return `${nextLine}\n`;
if (TOKEN_LINE_RE.test(content)) return content.replace(TOKEN_LINE_RE, `$1"${
@@ -122,29 +174,37 @@ function readJsonConfigToken(content: string): string | null {
function readTokenFromJsonObject(parsed: any): string | null {
const direct = parsed?.mcpServers?.[PLAYWRIGHT_SERVER_NAME]?.env?.[PLAYWRIGHT_TOKEN_ENV];
if (typeof direct === 'string' && direct) return direct;
const opencode = parsed?.mcp?.[PLAYWRIGHT_SERVER_NAME]?.env?.[PLAYWRIGHT_TOKEN_ENV];
const opencode = parsed?.mcp?.[PLAYWRIGHT_SERVER_NAME]?.environment?.[PLAYWRIGHT_TOKEN_ENV];
if (typeof opencode === 'string' && opencode) return opencode;
return null;
}
export function upsertJsonConfigToken(content: string, token: string): string {
export function upsertJsonConfigToken(content: string, token: string, filePath?: string): string {
const parsed = content.trim() ? JSON.parse(content) : {};
if (parsed?.mcpServers) {
parsed.mcpServers[PLAYWRIGHT_SERVER_NAME] = parsed.mcpServers[PLAYWRIGHT_SERVER_NAME] ?? {
command: 'npx',
args: ['-y', '@playwright/mcp@latest', '--extension'],
};
parsed.mcpServers[PLAYWRIGHT_SERVER_NAME].env = parsed.mcpServers[PLAYWRIGHT_SERVER_NAME].env ?? {};
parsed.mcpServers[PLAYWRIGHT_SERVER_NAME].env[PLAYWRIGHT_TOKEN_ENV] = token;
} else {
// Determine format: use OpenCode format only if explicitly an opencode config,
// or if the existing content already uses `mcp` key (not `mcpServers`)
const useOpenCodeFormat = filePath
? isOpenCodeConfig(filePath)
: (!parsed.mcpServers && parsed.mcp);
if (useOpenCodeFormat) {
parsed.mcp = parsed.mcp ?? {};
parsed.mcp[PLAYWRIGHT_SERVER_NAME] = parsed.mcp[PLAYWRIGHT_SERVER_NAME] ?? {
command: ['npx', '-y', '@playwright/mcp@latest', '--extension'],
enabled: true,
type: 'local',
};
parsed.mcp[PLAYWRIGHT_SERVER_NAME].env = parsed.mcp[PLAYWRIGHT_SERVER_NAME].env ?? {};
parsed.mcp[PLAYWRIGHT_SERVER_NAME].env[PLAYWRIGHT_TOKEN_ENV] = token;
parsed.mcp[PLAYWRIGHT_SERVER_NAME].environment = parsed.mcp[PLAYWRIGHT_SERVER_NAME].environment ?? {};
parsed.mcp[PLAYWRIGHT_SERVER_NAME].environment[PLAYWRIGHT_TOKEN_ENV] = token;
} else {
parsed.mcpServers = parsed.mcpServers ?? {};
parsed.mcpServers[PLAYWRIGHT_SERVER_NAME] = parsed.mcpServers[PLAYWRIGHT_SERVER_NAME] ?? {
command: 'npx',
args: ['-y', '@playwright/mcp@latest', '--extension'],
};
parsed.mcpServers[PLAYWRIGHT_SERVER_NAME].env = parsed.mcpServers[PLAYWRIGHT_SERVER_NAME].env ?? {};
parsed.mcpServers[PLAYWRIGHT_SERVER_NAME].env[PLAYWRIGHT_TOKEN_ENV] = token;
}
return `${JSON.stringify(parsed, null, 2)}\n`;
}
@@ -177,7 +237,7 @@ export function upsertTomlConfigToken(content: string, token: string): string {
return `${prefix}[mcp_servers.playwright]\ntype = "stdio"\ncommand = "npx"\nargs = ["-y", "@playwright/mcp@latest", "--extension"]\n\n[mcp_servers.playwright.env]\n${tokenLine}\n`;
}
function fileExists(filePath: string): boolean {
export function fileExists(filePath: string): boolean {
try {
return fs.existsSync(filePath);
} catch {
@@ -227,12 +287,35 @@ function readConfigStatus(filePath: string): McpConfigStatus {
}
}
/**
* Dynamically enumerate Chrome profiles by scanning for 'Default' and 'Profile *'
* directories across all browser base paths. Falls back to ['Default'] if none found.
*/
function enumerateProfiles(baseDirs: string[]): string[] {
const profiles = new Set<string>();
for (const base of baseDirs) {
if (!fileExists(base)) continue;
try {
for (const entry of fs.readdirSync(base, { withFileTypes: true })) {
if (!entry.isDirectory()) continue;
if (entry.name === 'Default' || /^Profile \d+$/.test(entry.name)) {
profiles.add(entry.name);
}
}
} catch { /* permission denied, etc. */ }
}
return profiles.size > 0 ? [...profiles].sort() : ['Default'];
}
/**
* Discover the auth token stored by the Playwright MCP Bridge extension
* by scanning Chrome's LevelDB localStorage files directly.
*
* Uses `strings` + `grep` for fast binary scanning on macOS/Linux,
* with a pure-Node fallback on Windows.
* Reads LevelDB .ldb/.log files as raw binary and searches for the
* extension ID near base64url token values. This works reliably across
* platforms because LevelDB's internal encoding can split ASCII strings
* like "auth-token" and the extension ID across byte boundaries, making
* text-based tools like `strings` + `grep` unreliable.
*/
export function discoverExtensionToken(): string | null {
const home = os.homedir();
@@ -260,8 +343,7 @@ export function discoverExtensionToken(): string | null {
);
}
const profiles = ['Default', 'Profile 1', 'Profile 2', 'Profile 3'];
// Token is 43 chars of base64url (from 32 random bytes)
const profiles = enumerateProfiles(bases);
const tokenRe = /([A-Za-z0-9_-]{40,50})/;
for (const base of bases) {
@@ -269,14 +351,6 @@ export function discoverExtensionToken(): string | null {
const dir = path.join(base, profile, 'Local Storage', 'leveldb');
if (!fileExists(dir)) continue;
// Fast path: use strings + grep to find candidate files and extract token
if (platform !== 'win32') {
const token = extractTokenViaStrings(dir, tokenRe);
if (token) return token;
continue;
}
// Slow path (Windows): read binary files directly
const token = extractTokenViaBinaryRead(dir, tokenRe);
if (token) return token;
}
@@ -285,39 +359,20 @@ export function discoverExtensionToken(): string | null {
return null;
}
function extractTokenViaStrings(dir: string, tokenRe: RegExp): string | null {
try {
// Single shell pipeline: for each LevelDB file, extract strings, find lines
// after the extension ID, and filter for base64url token pattern.
//
// LevelDB `strings` output for the extension's auth-token entry:
// auth-token ← key name
// 4,mmlmfjhmonkocbjadbfplnigmagldckm.7 ← LevelDB internal key
// hqI86ncsD1QpcVcj-k9CyzTF-ieCQd_4KreZ_wy1WHA ← token value
//
// We get the line immediately after any EXTENSION_ID mention and check
// if it looks like a base64url token (40-50 chars, [A-Za-z0-9_-]).
const shellDir = dir.replace(/'/g, "'\\''");
const cmd = `for f in '${shellDir}'/*.ldb '${shellDir}'/*.log; do ` +
`[ -f "$f" ] && strings "$f" 2>/dev/null | ` +
`grep -A1 '${PLAYWRIGHT_EXTENSION_ID}' | ` +
`grep -v '${PLAYWRIGHT_EXTENSION_ID}' | ` +
`grep -E '^[A-Za-z0-9_-]{40,50}$' | head -1; ` +
`done 2>/dev/null`;
const result = execSync(cmd, { encoding: 'utf-8', timeout: 10000 }).trim();
// Take the first non-empty line
for (const line of result.split('\n')) {
const token = line.trim();
if (token && validateBase64urlToken(token)) return token;
}
} catch {}
return null;
}
function extractTokenViaBinaryRead(dir: string, tokenRe: RegExp): string | null {
// LevelDB fragments strings across byte boundaries, so we can't search
// for the full extension ID or "auth-token" as contiguous ASCII. Instead,
// search for a short prefix of the extension ID that reliably appears as
// contiguous bytes, then scan a window around each match for a base64url
// token value.
//
// Observed LevelDB layout near the auth-token entry:
// ... auth-t<binary> ... 4,mmlmfjh<binary>Pocbjadbfplnigmagldckm.7 ...
// <binary> hqI86ncsD1QpcVcj-k9CyzTF-ieCQd_4KreZ_wy1WHA <binary> ...
//
// The extension ID prefix "mmlmfjh" appears ~44 bytes before the token.
const extIdBuf = Buffer.from(PLAYWRIGHT_EXTENSION_ID);
const keyBuf = Buffer.from('auth-token');
const extIdPrefix = Buffer.from(PLAYWRIGHT_EXTENSION_ID.slice(0, 7)); // "mmlmfjh"
let files: string[];
try {
@@ -326,7 +381,7 @@ function extractTokenViaBinaryRead(dir: string, tokenRe: RegExp): string | null
.map(f => path.join(dir, f));
} catch { return null; }
// Sort by mtime descending
// Sort by mtime descending so we find the freshest token first
files.sort((a, b) => {
try { return fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs; } catch { return 0; }
});
@@ -335,14 +390,30 @@ function extractTokenViaBinaryRead(dir: string, tokenRe: RegExp): string | null
let data: Buffer;
try { data = fs.readFileSync(file); } catch { continue; }
// Quick check: does file contain both the extension ID and auth-token key?
const extPos = data.indexOf(extIdBuf);
if (extPos === -1) continue;
const keyPos = data.indexOf(keyBuf, Math.max(0, extPos - 500));
if (keyPos === -1) continue;
// Quick check: file must contain at least the prefix
if (data.indexOf(extIdPrefix) === -1) continue;
// Scan for token value after auth-token key
// Strategy 1: scan after each occurrence of the extension ID prefix
// for base64url tokens within a 500-byte window
let idx = 0;
while (true) {
const pos = data.indexOf(extIdPrefix, idx);
if (pos === -1) break;
const scanStart = pos;
const scanEnd = Math.min(data.length, pos + 500);
const window = data.subarray(scanStart, scanEnd).toString('latin1');
const m = window.match(tokenRe);
if (m && validateBase64urlToken(m[1])) {
// Make sure this isn't another extension ID that happens to match
if (m[1] !== PLAYWRIGHT_EXTENSION_ID) return m[1];
}
idx = pos + 1;
}
// Strategy 2 (fallback): original approach using full extension ID + auth-token key
const keyBuf = Buffer.from('auth-token');
idx = 0;
while (true) {
const kp = data.indexOf(keyBuf, idx);
if (kp === -1) break;
@@ -368,6 +439,69 @@ function validateBase64urlToken(token: string): boolean {
}
/**
* Check whether the Playwright MCP Bridge extension is installed in any browser.
* Scans Chrome/Chromium/Edge Extensions directories for the known extension ID.
*/
export function checkExtensionInstalled(): { installed: boolean; browsers: string[] } {
const home = os.homedir();
const platform = os.platform();
const browserDirs: Array<{ name: string; base: string }> = [];
if (platform === 'darwin') {
browserDirs.push(
{ name: 'Chrome', base: path.join(home, 'Library', 'Application Support', 'Google', 'Chrome') },
{ name: 'Chrome Canary', base: path.join(home, 'Library', 'Application Support', 'Google', 'Chrome Canary') },
{ name: 'Chromium', base: path.join(home, 'Library', 'Application Support', 'Chromium') },
{ name: 'Edge', base: path.join(home, 'Library', 'Application Support', 'Microsoft Edge') },
);
} else if (platform === 'linux') {
browserDirs.push(
{ name: 'Chrome', base: path.join(home, '.config', 'google-chrome') },
{ name: 'Chromium', base: path.join(home, '.config', 'chromium') },
{ name: 'Edge', base: path.join(home, '.config', 'microsoft-edge') },
);
} else if (platform === 'win32') {
const appData = process.env.LOCALAPPDATA || path.join(home, 'AppData', 'Local');
browserDirs.push(
{ name: 'Chrome', base: path.join(appData, 'Google', 'Chrome', 'User Data') },
{ name: 'Edge', base: path.join(appData, 'Microsoft', 'Edge', 'User Data') },
);
}
const profiles = enumerateProfiles(browserDirs.map(d => d.base));
const foundBrowsers: string[] = [];
for (const { name, base } of browserDirs) {
for (const profile of profiles) {
const extDir = path.join(base, profile, 'Extensions', PLAYWRIGHT_EXTENSION_ID);
if (fileExists(extDir)) {
foundBrowsers.push(name);
break; // one match per browser is enough
}
}
}
return { installed: foundBrowsers.length > 0, browsers: [...new Set(foundBrowsers)] };
}
/**
* Test token connectivity by attempting a real MCP connection.
* Connects, does the JSON-RPC handshake, and immediately closes.
*/
export async function checkTokenConnectivity(opts?: { timeout?: number }): Promise<ConnectivityResult> {
const timeout = opts?.timeout ?? 8;
const start = Date.now();
try {
const mcp = new PlaywrightMCP();
await mcp.connect({ timeout });
await mcp.close();
return { ok: true, durationMs: Date.now() - start };
} catch (err: any) {
return { ok: false, error: err?.message ?? String(err), durationMs: Date.now() - start };
}
}
export async function runBrowserDoctor(opts: DoctorOptions = {}): Promise<DoctorReport> {
const envToken = process.env[PLAYWRIGHT_TOKEN_ENV] ?? null;
const shellPath = opts.shellRc ?? getDefaultShellRcPath();
@@ -393,24 +527,38 @@ export async function runBrowserDoctor(opts: DoctorOptions = {}): Promise<Doctor
const uniqueTokens = [...new Set(allTokens)];
const recommendedToken = opts.token ?? extensionToken ?? envToken ?? (uniqueTokens.length === 1 ? uniqueTokens[0] : null) ?? null;
// Check extension installation
const extInstall = checkExtensionInstalled();
// Connectivity test (only when --live)
let connectivity: ConnectivityResult | undefined;
if (opts.live) {
connectivity = await checkTokenConnectivity();
}
const report: DoctorReport = {
cliVersion: opts.cliVersion,
envToken,
envFingerprint: getTokenFingerprint(envToken ?? undefined),
extensionToken,
extensionFingerprint: getTokenFingerprint(extensionToken ?? undefined),
extensionInstalled: extInstall.installed,
extensionBrowsers: extInstall.browsers,
shellFiles,
configs,
recommendedToken,
recommendedFingerprint: getTokenFingerprint(recommendedToken ?? undefined),
connectivity,
warnings: [],
issues: [],
};
if (!extInstall.installed) report.issues.push('Playwright MCP Bridge extension is not installed in any browser.');
if (!envToken) report.issues.push(`Current environment is missing ${PLAYWRIGHT_TOKEN_ENV}.`);
if (!shellFiles.some(s => s.token)) report.issues.push('Shell startup file does not export PLAYWRIGHT_MCP_EXTENSION_TOKEN.');
if (!configs.some(c => c.token)) report.issues.push('No scanned MCP config currently contains a Playwright extension token.');
if (uniqueTokens.length > 1) report.issues.push('Detected inconsistent Playwright MCP tokens across env/config files.');
if (connectivity && !connectivity.ok) report.issues.push(`Browser connectivity test failed: ${connectivity.error ?? 'unknown'}`);
for (const config of configs) {
if (config.parseError) report.warnings.push(`Could not parse ${config.path}: ${config.parseError}`);
}
@@ -429,7 +577,13 @@ export function renderBrowserDoctorReport(report: DoctorReport): string {
].filter((value): value is string => !!value);
const uniqueFingerprints = [...new Set(tokenFingerprints)];
const hasMismatch = uniqueFingerprints.length > 1;
const lines = [`opencli v${report.cliVersion ?? 'unknown'} doctor`, ''];
const lines = [chalk.bold(`opencli v${report.cliVersion ?? 'unknown'} doctor`), ''];
const installStatus: ReportStatus = report.extensionInstalled ? 'OK' : 'MISSING';
const installDetail = report.extensionInstalled
? `Extension installed (${report.extensionBrowsers.join(', ')})`
: 'Extension not installed in any browser';
lines.push(statusLine(installStatus, installDetail));
const extStatus: ReportStatus = !report.extensionToken ? 'MISSING' : hasMismatch ? 'MISMATCH' : 'OK';
lines.push(statusLine(extStatus, `Extension token (Chrome LevelDB): ${tokenSummary(report.extensionToken, report.extensionFingerprint)}`));
@@ -439,13 +593,15 @@ export function renderBrowserDoctorReport(report: DoctorReport): string {
for (const shell of report.shellFiles) {
const shellStatus: ReportStatus = !shell.token ? 'MISSING' : hasMismatch ? 'MISMATCH' : 'OK';
lines.push(statusLine(shellStatus, `Shell file ${shell.path}: ${tokenSummary(shell.token, shell.fingerprint)}`));
const tool = toolName(shell.path);
const suffix = tool ? chalk.dim(` [${tool}]`) : '';
lines.push(statusLine(shellStatus, `${shortenPath(shell.path)}${suffix}: ${tokenSummary(shell.token, shell.fingerprint)}`));
}
const existingConfigs = report.configs.filter(config => config.exists);
const missingConfigCount = report.configs.length - existingConfigs.length;
if (existingConfigs.length > 0) {
for (const config of existingConfigs) {
const parseSuffix = config.parseError ? ` (parse error: ${config.parseError})` : '';
const parseSuffix = config.parseError ? chalk.red(` (parse error)`) : '';
const configStatus: ReportStatus = config.parseError
? 'WARN'
: !config.token
@@ -453,24 +609,38 @@ export function renderBrowserDoctorReport(report: DoctorReport): string {
: hasMismatch
? 'MISMATCH'
: 'OK';
lines.push(statusLine(configStatus, `MCP config ${config.path}: ${tokenSummary(config.token, config.fingerprint)}${parseSuffix}`));
const tool = toolName(config.path);
const suffix = tool ? chalk.dim(` [${tool}]`) : '';
lines.push(statusLine(configStatus, `${shortenPath(config.path)}${suffix}: ${tokenSummary(config.token, config.fingerprint)}${parseSuffix}`));
}
} else {
lines.push(statusLine('MISSING', 'MCP config: no existing config files found in scanned locations'));
lines.push(statusLine('MISSING', 'MCP config: no existing config files found'));
}
if (missingConfigCount > 0) lines.push(` Other scanned config locations not present: ${missingConfigCount}`);
if (missingConfigCount > 0) lines.push(chalk.dim(` Other scanned config locations not present: ${missingConfigCount}`));
lines.push('');
// Connectivity result
if (report.connectivity) {
const connStatus: ReportStatus = report.connectivity.ok ? 'OK' : 'WARN';
const connDetail = report.connectivity.ok
? `Browser connectivity: connected in ${(report.connectivity.durationMs / 1000).toFixed(1)}s`
: `Browser connectivity: failed (${report.connectivity.error ?? 'unknown'})`;
lines.push(statusLine(connStatus, connDetail));
} else {
lines.push(statusLine('WARN', 'Browser connectivity: not tested (use --live)'));
}
lines.push(statusLine(
hasMismatch ? 'MISMATCH' : report.recommendedToken ? 'OK' : 'WARN',
`Recommended token fingerprint: ${report.recommendedFingerprint ?? 'unavailable'}`,
));
if (report.issues.length) {
lines.push('', 'Issues:');
for (const issue of report.issues) lines.push(`- ${issue}`);
lines.push('', chalk.yellow('Issues:'));
for (const issue of report.issues) lines.push(chalk.dim(` ${issue}`));
}
if (report.warnings.length) {
lines.push('', 'Warnings:');
for (const warning of report.warnings) lines.push(`- ${warning}`);
lines.push('', chalk.yellow('Warnings:'));
for (const warning of report.warnings) lines.push(chalk.dim(` ${warning}`));
}
return lines.join('\n');
}
@@ -485,7 +655,7 @@ async function confirmPrompt(question: string): Promise<boolean> {
}
}
function writeFileWithMkdir(filePath: string, content: string): void {
export function writeFileWithMkdir(filePath: string, content: string): void {
fs.mkdirSync(path.dirname(filePath), { recursive: true });
fs.writeFileSync(filePath, content, 'utf-8');
}
@@ -493,29 +663,42 @@ function writeFileWithMkdir(filePath: string, content: string): void {
export async function applyBrowserDoctorFix(report: DoctorReport, opts: DoctorOptions = {}): Promise<string[]> {
const token = opts.token ?? report.recommendedToken;
if (!token) throw new Error('No Playwright MCP token is available to write. Provide --token first.');
const fp = getTokenFingerprint(token);
const plannedWrites: string[] = [];
const shellPath = opts.shellRc ?? report.shellFiles[0]?.path ?? getDefaultShellRcPath();
plannedWrites.push(shellPath);
const shellStatus = report.shellFiles.find(s => s.path === shellPath);
if (shellStatus?.fingerprint !== fp) plannedWrites.push(shellPath);
for (const config of report.configs) {
if (!config.writable) continue;
if (config.fingerprint === fp) continue; // already correct
plannedWrites.push(config.path);
}
if (plannedWrites.length === 0) {
console.log(chalk.green('All config files are already up to date.'));
return [];
}
if (!opts.yes) {
const ok = await confirmPrompt(`Update ${plannedWrites.length} file(s) with Playwright MCP token fingerprint ${getTokenFingerprint(token)}?`);
const ok = await confirmPrompt(`Update ${plannedWrites.length} file(s) with Playwright MCP token fingerprint ${fp}?`);
if (!ok) return [];
}
const written: string[] = [];
const shellBefore = fileExists(shellPath) ? fs.readFileSync(shellPath, 'utf-8') : '';
writeFileWithMkdir(shellPath, upsertShellToken(shellBefore, token));
written.push(shellPath);
if (plannedWrites.includes(shellPath)) {
const shellBefore = fileExists(shellPath) ? fs.readFileSync(shellPath, 'utf-8') : '';
writeFileWithMkdir(shellPath, upsertShellToken(shellBefore, token, shellPath));
written.push(shellPath);
}
for (const config of report.configs) {
if (!config.writable || config.parseError) continue;
if (!plannedWrites.includes(config.path)) continue;
if (config.parseError) continue;
const before = fileExists(config.path) ? fs.readFileSync(config.path, 'utf-8') : '';
const next = config.format === 'toml' ? upsertTomlConfigToken(before, token) : upsertJsonConfigToken(before, token);
const next = config.format === 'toml'
? upsertTomlConfigToken(before, token)
: upsertJsonConfigToken(before, token, config.path);
writeFileWithMkdir(config.path, next);
written.push(config.path);
}
+6 -3
View File
@@ -101,7 +101,10 @@ async function discoverClisFromFs(dir: string): Promise<void> {
const filePath = path.join(siteDir, file);
if (file.endsWith('.yaml') || file.endsWith('.yml')) {
registerYamlCli(filePath, site);
} else if (file.endsWith('.js') && !file.endsWith('.d.js')) {
} else if (
(file.endsWith('.js') && !file.endsWith('.d.js')) ||
(file.endsWith('.ts') && !file.endsWith('.d.ts') && !file.endsWith('.test.ts'))
) {
promises.push(
import(`file://${filePath}`).catch((err: any) => {
process.stderr.write(`Warning: failed to load module ${filePath}: ${err.message}\n`);
@@ -185,7 +188,7 @@ export async function executeCommand(
const { getRegistry, fullName } = await import('./registry.js');
const updated = getRegistry().get(fullName(cmd));
if (updated && updated.func) {
return updated.func(page, kwargs, debug);
return updated.func(page!, kwargs, debug);
}
if (updated && updated.pipeline) {
return executePipeline(page, updated.pipeline, { args: kwargs, debug });
@@ -193,7 +196,7 @@ export async function executeCommand(
}
if (cmd.func) {
return cmd.func(page, kwargs, debug);
return cmd.func(page!, kwargs, debug);
}
if (cmd.pipeline) {
return executePipeline(page, cmd.pipeline, { args: kwargs, debug });
+94
View File
@@ -0,0 +1,94 @@
/**
* Tests for interceptor.ts: JavaScript code generators for XHR/Fetch interception.
*/
import { describe, it, expect } from 'vitest';
import { generateInterceptorJs, generateReadInterceptedJs, generateTapInterceptorJs } from './interceptor.js';
describe('generateInterceptorJs', () => {
it('generates valid JavaScript function source', () => {
const js = generateInterceptorJs('"api/search"');
expect(js).toContain('window.fetch');
expect(js).toContain('XMLHttpRequest');
expect(js).toContain('"api/search"');
// Should be a function expression wrapping
expect(js.trim()).toMatch(/^\(\)\s*=>/);
});
it('uses default array name and patch guard', () => {
const js = generateInterceptorJs('"test"');
expect(js).toContain('__opencli_intercepted');
expect(js).toContain('__opencli_interceptor_patched');
});
it('uses custom array name and patch guard', () => {
const js = generateInterceptorJs('"test"', {
arrayName: '__my_data',
patchGuard: '__my_guard',
});
expect(js).toContain('__my_data');
expect(js).toContain('__my_guard');
expect(js).not.toContain('__opencli_intercepted');
});
it('includes fetch clone and json parsing', () => {
const js = generateInterceptorJs('"api"');
expect(js).toContain('response.clone()');
expect(js).toContain('clone.json()');
});
it('includes XHR open and send patching', () => {
const js = generateInterceptorJs('"api"');
expect(js).toContain('XMLHttpRequest.prototype');
expect(js).toContain('__origOpen');
expect(js).toContain('__origSend');
});
});
describe('generateReadInterceptedJs', () => {
it('generates valid JavaScript to read and clear data', () => {
const js = generateReadInterceptedJs();
expect(js).toContain('__opencli_intercepted');
// Should clear the array after reading
expect(js).toContain('= []');
});
it('uses custom array name', () => {
const js = generateReadInterceptedJs('__custom_arr');
expect(js).toContain('__custom_arr');
expect(js).not.toContain('__opencli_intercepted');
});
});
describe('generateTapInterceptorJs', () => {
it('returns all required fields', () => {
const tap = generateTapInterceptorJs('"api/data"');
expect(tap.setupVar).toBeDefined();
expect(tap.capturedVar).toBe('captured');
expect(tap.promiseVar).toBe('capturePromise');
expect(tap.resolveVar).toBe('captureResolve');
expect(tap.fetchPatch).toBeDefined();
expect(tap.xhrPatch).toBeDefined();
expect(tap.restorePatch).toBeDefined();
});
it('contains the capture pattern in setup', () => {
const tap = generateTapInterceptorJs('"my-pattern"');
expect(tap.setupVar).toContain('"my-pattern"');
});
it('restores original fetch and XHR in restorePatch', () => {
const tap = generateTapInterceptorJs('"test"');
expect(tap.restorePatch).toContain('origFetch');
expect(tap.restorePatch).toContain('origXhrOpen');
expect(tap.restorePatch).toContain('origXhrSend');
});
it('uses first-match capture (only first response)', () => {
const tap = generateTapInterceptorJs('"test"');
// Both fetch and xhr patches should check !captured before storing
expect(tap.fetchPatch).toContain('!captured');
expect(tap.xhrPatch).toContain('!captured');
});
});
+36 -8
View File
@@ -62,10 +62,18 @@ program.command('list').description('List all available CLI commands').option('-
});
program.command('validate').description('Validate CLI definitions').argument('[target]', 'site or site/name')
.action(async (target) => { const { validateClisWithTarget, renderValidationReport } = await import('./validate.js'); console.log(renderValidationReport(validateClisWithTarget([BUILTIN_CLIS, USER_CLIS], target))); });
.action(async (target) => {
const { validateClisWithTarget, renderValidationReport } = await import('./validate.js');
console.log(renderValidationReport(validateClisWithTarget([BUILTIN_CLIS, USER_CLIS], target)));
});
program.command('verify').description('Validate + smoke test').argument('[target]').option('--smoke', 'Run smoke tests', false)
.action(async (target, opts) => { const { verifyClis, renderVerifyReport } = await import('./verify.js'); const r = await verifyClis({ builtinClis: BUILTIN_CLIS, userClis: USER_CLIS, target, smoke: opts.smoke }); console.log(renderVerifyReport(r)); process.exitCode = r.ok ? 0 : 1; });
.action(async (target, opts) => {
const { verifyClis, renderVerifyReport } = await import('./verify.js');
const r = await verifyClis({ builtinClis: BUILTIN_CLIS, userClis: USER_CLIS, target, smoke: opts.smoke });
console.log(renderVerifyReport(r));
process.exitCode = r.ok ? 0 : 1;
});
program.command('explore').alias('probe').description('Explore a website: discover APIs, stores, and recommend strategies').argument('<url>').option('--site <name>').option('--goal <text>').option('--wait <s>', '', '3').option('--auto', 'Enable interactive fuzzing (simulate clicks to trigger lazy APIs)').option('--click <labels>', 'Comma-separated labels to click before fuzzing (e.g. "字幕,CC,评论")')
.action(async (url, opts) => { const { exploreUrl, renderExploreSummary } = await import('./explore.js'); const clickLabels = opts.click ? opts.click.split(',').map((s: string) => s.trim()) : undefined; console.log(renderExploreSummary(await exploreUrl(url, { BrowserFactory: PlaywrightMCP, site: opts.site, goal: opts.goal, waitSeconds: parseFloat(opts.wait), auto: opts.auto, clickLabels }))); });
@@ -92,12 +100,13 @@ program.command('doctor')
.option('--fix', 'Apply suggested fixes to shell rc and detected MCP configs', false)
.option('-y, --yes', 'Skip confirmation prompts when applying fixes', false)
.option('--token <token>', 'Override token to write instead of auto-detecting')
.option('--live', 'Test browser connectivity (requires Chrome running)', false)
.option('--shell-rc <path>', 'Shell startup file to update')
.option('--mcp-config <paths>', 'Comma-separated MCP config paths to scan/update')
.action(async (opts) => {
const { runBrowserDoctor, renderBrowserDoctorReport, applyBrowserDoctorFix } = await import('./doctor.js');
const configPaths = opts.mcpConfig ? String(opts.mcpConfig).split(',').map((s: string) => s.trim()).filter(Boolean) : undefined;
const report = await runBrowserDoctor({ token: opts.token, shellRc: opts.shellRc, configPaths, cliVersion: PKG_VERSION });
const report = await runBrowserDoctor({ token: opts.token, live: opts.live, shellRc: opts.shellRc, configPaths, cliVersion: PKG_VERSION });
console.log(renderBrowserDoctorReport(report));
if (opts.fix) {
const written = await applyBrowserDoctorFix(report, { fix: true, yes: opts.yes, token: opts.token, shellRc: opts.shellRc, configPaths });
@@ -129,18 +138,37 @@ for (const [, cmd] of registry) {
if (!siteCmd) { siteCmd = program.command(cmd.site).description(`${cmd.site} commands`); siteGroups.set(cmd.site, siteCmd); }
const subCmd = siteCmd.command(cmd.name).description(cmd.description);
// Register positional args first, then named options
const positionalArgs: typeof cmd.args = [];
for (const arg of cmd.args) {
const flag = arg.required ? `--${arg.name} <value>` : `--${arg.name} [value]`;
if (arg.required) subCmd.requiredOption(flag, arg.help ?? '');
else if (arg.default != null) subCmd.option(flag, arg.help ?? '', String(arg.default));
else subCmd.option(flag, arg.help ?? '');
if (arg.positional) {
const bracket = arg.required ? `<${arg.name}>` : `[${arg.name}]`;
subCmd.argument(bracket, arg.help ?? '');
positionalArgs.push(arg);
} else {
const flag = arg.required ? `--${arg.name} <value>` : `--${arg.name} [value]`;
if (arg.required) subCmd.requiredOption(flag, arg.help ?? '');
else if (arg.default != null) subCmd.option(flag, arg.help ?? '', String(arg.default));
else subCmd.option(flag, arg.help ?? '');
}
}
subCmd.option('-f, --format <fmt>', 'Output format: table, json, yaml, md, csv', 'table').option('-v, --verbose', 'Debug output', false);
subCmd.action(async (actionOpts) => {
subCmd.action(async (...actionArgs: any[]) => {
// Commander passes positional args first, then options object, then the Command
const actionOpts = actionArgs[positionalArgs.length] ?? {};
const startTime = Date.now();
const kwargs: Record<string, any> = {};
// Collect positional args
for (let i = 0; i < positionalArgs.length; i++) {
const arg = positionalArgs[i];
const v = actionArgs[i];
if (v !== undefined) kwargs[arg.name] = coerce(v, arg.type ?? 'str');
else if (arg.default != null) kwargs[arg.name] = arg.default;
}
// Collect named options
for (const arg of cmd.args) {
if (arg.positional) continue;
const v = actionOpts[arg.name]; if (v !== undefined) kwargs[arg.name] = coerce(v, arg.type ?? 'str');
else if (arg.default != null) kwargs[arg.name] = arg.default;
}
+69 -4
View File
@@ -1,3 +1,7 @@
/**
* Tests for output.ts: render function format coverage.
*/
import { describe, it, expect, vi, afterEach } from 'vitest';
import { render } from './output.js';
@@ -6,11 +10,67 @@ afterEach(() => {
});
describe('render', () => {
it('renders JSON output', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([{ title: 'Hello', rank: 1 }], { fmt: 'json' });
expect(log).toHaveBeenCalledOnce();
const output = log.mock.calls[0]?.[0];
const parsed = JSON.parse(output);
expect(parsed).toEqual([{ title: 'Hello', rank: 1 }]);
});
it('renders Markdown table output', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([{ name: 'Alice', score: 100 }], { fmt: 'md', columns: ['name', 'score'] });
const calls = log.mock.calls.map(c => c[0]);
expect(calls[0]).toContain('| name | score |');
expect(calls[1]).toContain('| --- | --- |');
expect(calls[2]).toContain('| Alice | 100 |');
});
it('renders CSV output with proper quoting', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([{ name: 'Alice, Bob', value: 'say "hi"' }], { fmt: 'csv' });
const calls = log.mock.calls.map(c => c[0]);
// Header
expect(calls[0]).toBe('name,value');
// Values with commas/quotes are quoted
expect(calls[1]).toContain('"Alice, Bob"');
expect(calls[1]).toContain('"say ""hi"""');
});
it('handles null and undefined data', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render(null, { fmt: 'json' });
expect(log).toHaveBeenCalledWith(null);
});
it('renders single object as single-row table', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render({ title: 'Test' }, { fmt: 'json' });
const output = log.mock.calls[0]?.[0];
const parsed = JSON.parse(output);
expect(parsed).toEqual({ title: 'Test' });
});
it('handles empty array gracefully', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([], { fmt: 'table' });
// Should show "(no data)" for empty arrays
expect(log).toHaveBeenCalled();
});
it('uses custom columns for CSV', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([{ a: 1, b: 2, c: 3 }], { fmt: 'csv', columns: ['a', 'c'] });
const calls = log.mock.calls.map(c => c[0]);
expect(calls[0]).toBe('a,c');
expect(calls[1]).toBe('1,3');
});
it('renders YAML output', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([{ title: 'Hello', rank: 1 }], { fmt: 'yaml' });
expect(log).toHaveBeenCalledOnce();
expect(log.mock.calls[0]?.[0]).toContain('- title: Hello');
expect(log.mock.calls[0]?.[0]).toContain('rank: 1');
@@ -18,10 +78,15 @@ describe('render', () => {
it('renders yml alias as YAML output', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render({ title: 'Hello' }, { fmt: 'yml' });
expect(log).toHaveBeenCalledOnce();
expect(log.mock.calls[0]?.[0]).toContain('title: Hello');
});
it('handles null values in CSV cells', () => {
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
render([{ name: 'test', value: null }], { fmt: 'csv' });
const calls = log.mock.calls.map(c => c[0]);
expect(calls[1]).toBe('test,');
});
});
+2 -1
View File
@@ -82,7 +82,8 @@ function renderCsv(data: any, opts: RenderOptions): void {
for (const row of rows) {
console.log(columns.map(c => {
const v = String(row[c] ?? '');
return v.includes(',') || v.includes('"') ? `"${v.replace(/"/g, '""')}"` : v;
return v.includes(',') || v.includes('"') || v.includes('\n')
? `"${v.replace(/"/g, '""')}"` : v;
}).join(','));
}
}
+161
View File
@@ -0,0 +1,161 @@
/**
* Tests for pipeline/executor.ts: pipeline execution with mock page.
*/
import { describe, it, expect, vi } from 'vitest';
import { executePipeline } from './index.js';
import type { IPage } from '../types.js';
/** Create a minimal mock page for testing */
function createMockPage(overrides: Partial<IPage> = {}): IPage {
return {
goto: vi.fn(),
evaluate: vi.fn().mockResolvedValue(null),
snapshot: vi.fn().mockResolvedValue(''),
click: vi.fn(),
typeText: vi.fn(),
pressKey: vi.fn(),
wait: vi.fn(),
tabs: vi.fn().mockResolvedValue([]),
closeTab: vi.fn(),
newTab: vi.fn(),
selectTab: vi.fn(),
networkRequests: vi.fn().mockResolvedValue([]),
consoleMessages: vi.fn().mockResolvedValue(''),
scroll: vi.fn(),
autoScroll: vi.fn(),
installInterceptor: vi.fn(),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
...overrides,
};
}
describe('executePipeline', () => {
it('returns null for empty pipeline', async () => {
const result = await executePipeline(null, []);
expect(result).toBeNull();
});
it('skips null/invalid steps', async () => {
const result = await executePipeline(null, [null, undefined, 42] as any);
expect(result).toBeNull();
});
it('executes navigate step', async () => {
const page = createMockPage();
await executePipeline(page, [
{ navigate: 'https://example.com' },
]);
expect(page.goto).toHaveBeenCalledWith('https://example.com');
});
it('executes evaluate + select pipeline', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue({ data: { list: [{ name: 'a' }, { name: 'b' }] } }),
});
const result = await executePipeline(page, [
{ evaluate: '() => ({ data: { list: [{name: "a"}, {name: "b"}] } })' },
{ select: 'data.list' },
]);
expect(result).toEqual([{ name: 'a' }, { name: 'b' }]);
});
it('executes map step to transform items', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue([
{ title: 'Hello', count: 10 },
{ title: 'World', count: 20 },
]),
});
const result = await executePipeline(page, [
{ evaluate: 'test' },
{ map: { name: '${{ item.title }}', score: '${{ item.count }}' } },
]);
expect(result).toEqual([
{ name: 'Hello', score: 10 },
{ name: 'World', score: 20 },
]);
});
it('executes limit step', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue([1, 2, 3, 4, 5]),
});
const result = await executePipeline(page, [
{ evaluate: 'test' },
{ limit: '3' },
]);
expect(result).toEqual([1, 2, 3]);
});
it('executes sort step', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue([{ n: 3 }, { n: 1 }, { n: 2 }]),
});
const result = await executePipeline(page, [
{ evaluate: 'test' },
{ sort: { by: 'n', order: 'asc' } },
]);
expect(result).toEqual([{ n: 1 }, { n: 2 }, { n: 3 }]);
});
it('executes sort step with desc order', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue([{ n: 1 }, { n: 3 }, { n: 2 }]),
});
const result = await executePipeline(page, [
{ evaluate: 'test' },
{ sort: { by: 'n', order: 'desc' } },
]);
expect(result).toEqual([{ n: 3 }, { n: 2 }, { n: 1 }]);
});
it('executes wait step with number', async () => {
const page = createMockPage();
await executePipeline(page, [
{ wait: 2 },
]);
expect(page.wait).toHaveBeenCalledWith(2);
});
it('handles unknown steps gracefully in debug mode', async () => {
const stderr = vi.spyOn(process.stderr, 'write').mockImplementation(() => true);
await executePipeline(null, [
{ unknownStep: 'test' },
], { debug: true });
expect(stderr).toHaveBeenCalledWith(expect.stringContaining('Unknown step'));
stderr.mockRestore();
});
it('passes args through template rendering', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue([1, 2, 3, 4, 5]),
});
const result = await executePipeline(page, [
{ evaluate: 'test' },
{ limit: '${{ args.count }}' },
], { args: { count: 2 } });
expect(result).toEqual([1, 2]);
});
it('click step calls page.click', async () => {
const page = createMockPage();
await executePipeline(page, [
{ click: '@5' },
]);
expect(page.click).toHaveBeenCalledWith('5');
});
it('navigate preserves existing data through pipeline', async () => {
const page = createMockPage({
evaluate: vi.fn().mockResolvedValue([{ a: 1 }]),
});
const result = await executePipeline(page, [
{ evaluate: 'test' },
{ navigate: 'https://example.com' },
]);
// navigate should preserve existing data
expect(result).toEqual([{ a: 1 }]);
expect(page.goto).toHaveBeenCalledWith('https://example.com');
});
});
-5
View File
@@ -60,11 +60,6 @@ export async function executePipeline(
if (debug) process.stderr.write(` ${chalk.yellow('⚠')} Unknown step: ${op}\n`);
}
// Detect error objects returned by steps (e.g. tap store not found)
if (data && typeof data === 'object' && !Array.isArray(data) && data.error) {
process.stderr.write(` ${chalk.yellow('⚠')} ${chalk.yellow(op)}: ${data.error}\n`);
if (data.hint) process.stderr.write(` ${chalk.dim('💡')} ${chalk.dim(data.hint)}\n`);
}
if (debug) debugStepResult(op, data);
}
}
+4 -3
View File
@@ -45,11 +45,12 @@ async function fetchSingle(
}
const headersJs = JSON.stringify(renderedHeaders);
const escapedUrl = finalUrl.replace(/"/g, '\\"');
const urlJs = JSON.stringify(finalUrl);
const methodJs = JSON.stringify(method.toUpperCase());
return page.evaluate(`
async () => {
const resp = await fetch("${escapedUrl}", {
method: "${method}", headers: ${headersJs}, credentials: "include"
const resp = await fetch(${urlJs}, {
method: ${methodJs}, headers: ${headersJs}, credentials: "include"
});
return await resp.json();
}
+3 -9
View File
@@ -17,6 +17,7 @@ export interface Arg {
type?: string;
default?: any;
required?: boolean;
positional?: boolean;
help?: string;
choices?: string[];
}
@@ -30,7 +31,7 @@ export interface CliCommand {
browser?: boolean;
args: Arg[];
columns?: string[];
func?: (page: IPage | null, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
func?: (page: IPage, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
pipeline?: any[];
timeoutSeconds?: number;
source?: string;
@@ -41,18 +42,11 @@ export interface InternalCliCommand extends CliCommand {
_lazy?: boolean;
_modulePath?: string;
}
export interface CliOptions {
export interface CliOptions extends Partial<Omit<CliCommand, 'args' | 'description'>> {
site: string;
name: string;
description?: string;
domain?: string;
strategy?: Strategy;
browser?: boolean;
args?: Arg[];
columns?: string[];
func?: (page: IPage | null, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
pipeline?: any[];
timeoutSeconds?: number;
}
const _registry = new Map<string, CliCommand>();
+70 -51
View File
@@ -8,49 +8,26 @@ import * as fs from 'node:fs';
import chalk from 'chalk';
import { createInterface } from 'node:readline/promises';
import { stdin as input, stdout as output } from 'node:process';
import { exec } from 'node:child_process';
import {
type DoctorReport,
PLAYWRIGHT_TOKEN_ENV,
checkExtensionInstalled,
checkTokenConnectivity,
discoverExtensionToken,
fileExists,
getDefaultShellRcPath,
runBrowserDoctor,
shortenPath,
toolName,
upsertJsonConfigToken,
upsertShellToken,
upsertTomlConfigToken,
writeFileWithMkdir,
} from './doctor.js';
import { getTokenFingerprint } from './browser.js';
import { type CheckboxItem, checkboxPrompt } from './tui.js';
const PLAYWRIGHT_TOKEN_ENV = 'PLAYWRIGHT_MCP_EXTENSION_TOKEN';
function fileExists(p: string): boolean {
try { return fs.statSync(p).isFile() || fs.statSync(p).isDirectory(); } catch { return false; }
}
function writeFileWithMkdir(filePath: string, content: string) {
const dir = filePath.substring(0, filePath.lastIndexOf('/'));
if (dir && !fileExists(dir)) fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(filePath, content, 'utf-8');
}
function shortenPath(p: string): string {
const home = process.env.HOME || process.env.USERPROFILE || '';
return home && p.startsWith(home) ? '~' + p.slice(home.length) : p;
}
function toolName(p: string): string {
if (p.includes('.codex/')) return 'Codex';
if (p.includes('.cursor/')) return 'Cursor';
if (p.includes('.claude.json')) return 'Claude Code';
if (p.includes('antigravity')) return 'Antigravity';
if (p.includes('.gemini/settings')) return 'Gemini CLI';
if (p.includes('opencode')) return 'OpenCode';
if (p.includes('Claude/claude_desktop')) return 'Claude Desktop';
if (p.includes('.vscode/')) return 'VS Code';
if (p.includes('.mcp.json')) return 'Project MCP';
if (p.includes('.zshrc') || p.includes('.bashrc') || p.includes('.profile')) return 'Shell';
return '';
}
export async function runSetup(opts: { cliVersion?: string; token?: string } = {}) {
console.log();
console.log(chalk.bold(' opencli setup') + chalk.dim(' — Playwright MCP token configuration'));
@@ -86,11 +63,30 @@ export async function runSetup(opts: { cliVersion?: string; token?: string } = {
}
if (!token) {
console.log(` ${chalk.yellow('!')} No token found. Please enter it manually.`);
console.log(chalk.dim(' (Find it in the Playwright MCP Bridge extension → Status page)'));
// Give precise diagnosis of why token scan failed
const extInstall = checkExtensionInstalled();
console.log(` ${chalk.red('✗')} Browser token scan failed\n`);
if (!extInstall.installed) {
const extensionUrl = 'https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm';
console.log(chalk.dim(' Cause: Playwright MCP Bridge extension is not installed'));
console.log(chalk.dim(' Fix: Opening browser to extension store...'));
console.log(chalk.dim(` (If it doesn't open automatically, visit: ${extensionUrl})`));
const command = process.platform === 'darwin' ? `open "${extensionUrl}"` :
process.platform === 'win32' ? `start "" "${extensionUrl}"` :
`xdg-open "${extensionUrl}"`;
exec(command).on('error', () => {});
} else {
console.log(chalk.dim(` Cause: Extension is installed (${extInstall.browsers.join(', ')}) but token not found in LevelDB`));
console.log(chalk.dim(' Fix: 1) Open the extension popup and verify the token is generated'));
console.log(chalk.dim(' 2) Close Chrome completely, then re-run setup'));
}
console.log();
console.log(` You can enter the token manually, or fix the above and re-run ${chalk.bold('opencli setup')}.`);
console.log();
const rl = createInterface({ input, output });
const answer = await rl.question(' Token: ');
const answer = await rl.question(' Token (press Enter to abort): ');
rl.close();
token = answer.trim();
if (!token) {
@@ -113,8 +109,9 @@ export async function runSetup(opts: { cliVersion?: string; token?: string } = {
const shellStatus = report.shellFiles[0];
const shellFp = shellStatus?.fingerprint;
const shellOk = shellFp === fingerprint;
const shellTool = toolName(shellPath) || 'Shell';
items.push({
label: padRight(`${shortenPath(shellPath)}`, 50) + chalk.dim(` [${toolName(shellPath) || 'Shell'}]`),
label: padRight(shortenPath(shellPath), 50) + chalk.dim(` [${shellTool}]`),
value: `shell:${shellPath}`,
checked: !shellOk,
status: shellOk ? `configured (${shellFp})` : shellFp ? `mismatch (${shellFp})` : 'missing',
@@ -127,17 +124,18 @@ export async function runSetup(opts: { cliVersion?: string; token?: string } = {
const ok = fp === fingerprint;
const tool = toolName(config.path);
items.push({
label: padRight(`${shortenPath(config.path)}`, 50) + chalk.dim(tool ? ` [${tool}]` : ''),
label: padRight(shortenPath(config.path), 50) + chalk.dim(tool ? ` [${tool}]` : ''),
value: `config:${config.path}`,
checked: !ok,
checked: false, // let user explicitly select which tools to configure
status: ok ? `configured (${fp})` : !config.exists ? 'will create' : fp ? `mismatch (${fp})` : 'missing',
statusColor: ok ? 'green' : 'yellow',
});
}
// Step 4: Show interactive checkbox
console.clear();
const selected = await checkboxPrompt(items, {
title: ` Select files to update with token ${chalk.cyan(fingerprint)}:`,
title: ` ${chalk.bold('opencli setup')} token ${chalk.cyan(fingerprint)}`,
});
if (selected.length === 0) {
@@ -147,22 +145,24 @@ export async function runSetup(opts: { cliVersion?: string; token?: string } = {
// Step 5: Apply changes
const written: string[] = [];
let wroteShell = false;
for (const sel of selected) {
if (sel.startsWith('shell:')) {
const path = sel.slice('shell:'.length);
const before = fileExists(path) ? fs.readFileSync(path, 'utf-8') : '';
writeFileWithMkdir(path, upsertShellToken(before, token));
written.push(path);
const p = sel.slice('shell:'.length);
const before = fileExists(p) ? fs.readFileSync(p, 'utf-8') : '';
writeFileWithMkdir(p, upsertShellToken(before, token, p));
written.push(p);
wroteShell = true;
} else if (sel.startsWith('config:')) {
const path = sel.slice('config:'.length);
const config = report.configs.find(c => c.path === path);
const p = sel.slice('config:'.length);
const config = report.configs.find(c => c.path === p);
if (config && config.parseError) continue;
const before = fileExists(path) ? fs.readFileSync(path, 'utf-8') : '';
const format = config?.format ?? (path.endsWith('.toml') ? 'toml' : 'json');
const next = format === 'toml' ? upsertTomlConfigToken(before, token) : upsertJsonConfigToken(before, token);
writeFileWithMkdir(path, next);
written.push(path);
const before = fileExists(p) ? fs.readFileSync(p, 'utf-8') : '';
const format = config?.format ?? (p.endsWith('.toml') ? 'toml' : 'json');
const next = format === 'toml' ? upsertTomlConfigToken(before, token) : upsertJsonConfigToken(before, token, p);
writeFileWithMkdir(p, next);
written.push(p);
}
}
@@ -172,16 +172,35 @@ export async function runSetup(opts: { cliVersion?: string; token?: string } = {
if (written.length > 0) {
console.log(chalk.green.bold(` ✓ Updated ${written.length} file(s):`));
for (const p of written) {
console.log(` ${chalk.dim('•')} ${shortenPath(p)}`);
const tool = toolName(p);
console.log(` ${chalk.dim('•')} ${shortenPath(p)}${tool ? chalk.dim(` [${tool}]`) : ''}`);
}
if (wroteShell) {
console.log();
console.log(chalk.cyan(` 💡 Run ${chalk.bold(`source ${shortenPath(shellPath)}`)} to apply token to current shell.`));
}
} else {
console.log(chalk.yellow(' No files were changed.'));
}
console.log();
// Step 7: Auto-verify browser connectivity
console.log(chalk.dim(' Verifying browser connectivity...'));
try {
const result = await checkTokenConnectivity({ timeout: 5 });
if (result.ok) {
console.log(` ${chalk.green('✓')} Browser connected in ${(result.durationMs / 1000).toFixed(1)}s`);
} else {
console.log(` ${chalk.yellow('!')} Browser connectivity test failed: ${result.error ?? 'unknown'}`);
console.log(chalk.dim(' Make sure Chrome is running with the extension enabled.'));
}
} catch {
console.log(` ${chalk.yellow('!')} Could not verify connectivity (Chrome may not be running)`);
}
console.log();
}
function padRight(s: string, n: number): string {
// Account for ANSI escape codes in length calculation
const visible = s.replace(/\x1b\[[0-9;]*m/g, '');
return visible.length >= n ? s : s + ' '.repeat(n - visible.length);
}
+579
View File
@@ -0,0 +1,579 @@
/**
* Tests for snapshotFormatter.ts: Playwright MCP snapshot tree filtering.
*
* Uses sanitized excerpts from real websites (GitHub, Bilibili, Twitter)
* to validate noise filtering, annotation stripping, and output quality.
*/
import { describe, it, expect } from 'vitest';
import { formatSnapshot } from './snapshotFormatter.js';
// ---------------------------------------------------------------------------
// Fixtures: sanitized excerpts from real Playwright MCP snapshots
// ---------------------------------------------------------------------------
/** GitHub dashboard navigation bar (generic-heavy, refs, /url: lines) */
const GITHUB_NAV = `\
- generic [ref=e2]:
- region
- generic [ref=e3]:
- link "Skip to content" [ref=e4] [cursor=pointer]:
- /url: "#start-of-content"
- banner "Global Navigation Menu" [ref=e8]:
- generic [ref=e9]:
- generic [ref=e10]:
- button "Open menu" [ref=e12] [cursor=pointer]:
- img [ref=e13]
- link "Homepage" [ref=e15] [cursor=pointer]:
- /url: /
- img [ref=e16]
- generic [ref=e18]:
- navigation "Breadcrumbs" [ref=e19]:
- list [ref=e20]:
- listitem [ref=e21]:
- link "Dashboard" [ref=e22] [cursor=pointer]:
- /url: https://github.com/
- generic [ref=e23]: Dashboard
- button "Search or jump to…" [ref=e26] [cursor=pointer]:
- generic [ref=e27]:
- generic:
- img
- generic [ref=e28]:
- generic:
- text: Type
- generic: /
- text: to search`;
/** GitHub repo list sidebar (repetitive structure) */
const GITHUB_REPOS = `\
- navigation "Repositories" [ref=e79]:
- generic [ref=e80]:
- generic [ref=e81]:
- heading "Top repositories" [level=2] [ref=e82]
- link "New" [ref=e83] [cursor=pointer]:
- /url: /new
- generic [ref=e84]:
- generic:
- img
- generic [ref=e85]: New
- search "Top repositories" [ref=e86]:
- textbox "Find a repository…" [ref=e87]
- list [ref=e88]:
- listitem [ref=e89]:
- generic [ref=e90]:
- link "Repository" [ref=e91] [cursor=pointer]:
- /url: /jackwener/twitter-cli
- img "Repository" [ref=e92]
- link "jackwener/twitter-cli" [ref=e94] [cursor=pointer]:
- /url: /jackwener/twitter-cli
- listitem [ref=e95]:
- generic [ref=e96]:
- link "Repository" [ref=e97] [cursor=pointer]:
- /url: /jackwener/opencli
- img "Repository" [ref=e98]
- link "jackwener/opencli" [ref=e100] [cursor=pointer]:
- /url: /jackwener/opencli`;
/** Bilibili nav bar (Chinese text, multiple link categories) */
const BILIBILI_NAV = `\
- generic [ref=e3]:
- generic [ref=e4]:
- generic [ref=e5]:
- list [ref=e6]:
- listitem [ref=e7]:
- link "首页" [ref=e8] [cursor=pointer]:
- /url: //www.bilibili.com
- img [ref=e9]
- generic [ref=e11]: 首页
- listitem [ref=e12]:
- link "番剧" [ref=e13] [cursor=pointer]:
- /url: //www.bilibili.com/anime/
- listitem [ref=e14]:
- link "直播" [ref=e15] [cursor=pointer]:
- /url: //live.bilibili.com
- generic [ref=e32]:
- textbox "冷知识 金廷26年胜率100%" [ref=e34]
- img [ref=e36] [cursor=pointer]`;
/** Bilibili video card (deeply nested generic wrappers, view counts) */
const BILIBILI_VIDEO = `\
- generic [ref=e363]:
- link "超酷时刻 即将到来 3.3万 40 16:24" [ref=e364] [cursor=pointer]:
- /url: https://www.bilibili.com/video/BV1zVw5zoEFt
- generic [ref=e365]:
- img "超酷时刻 即将到来" [ref=e368]
- generic:
- generic:
- generic:
- generic:
- img
- generic: 3.3万
- generic:
- img
- generic: "40"
- generic: 16:24
- generic [ref=e370]:
- heading "超酷时刻 即将到来" [level=3] [ref=e371]:
- link "超酷时刻 即将到来" [ref=e372] [cursor=pointer]:
- /url: https://www.bilibili.com/video/BV1zVw5zoEFt
- link "Tesla特斯拉中国 · 13小时前" [ref=e374] [cursor=pointer]:
- /url: //space.bilibili.com/491190876
- img [ref=e375]
- generic "Tesla特斯拉中国" [ref=e379]
- generic [ref=e380]: · 13小时前`;
/** Empty paragraph blocks (Bilibili bottom section) */
const BILIBILI_EMPTY = `\
- generic [ref=e576]:
- generic:
- generic:
- generic:
- paragraph
- paragraph
- paragraph
- generic [ref=e577]:
- generic:
- generic:
- generic:
- paragraph
- paragraph
- paragraph`;
/** Twitter-style feed item (simulated based on common patterns) */
const TWITTER_TWEET = `\
- main [ref=e100]:
- region "Timeline" [ref=e101]:
- article [ref=e200]:
- generic [ref=e201]:
- generic [ref=e202]:
- link "@elonmusk" [ref=e203] [cursor=pointer]:
- /url: /elonmusk
- img "@elonmusk" [ref=e204]
- generic [ref=e205]:
- generic [ref=e206]: Elon Musk
- generic [ref=e207]: @elonmusk
- generic [ref=e208]:
- generic [ref=e209]: This is a very long tweet that goes on and on about various things including technology, space, and other random topics that make this text exceed any reasonable length limit we might want to set for display purposes in a CLI interface.
- generic [ref=e210]:
- button "Reply" [ref=e211] [cursor=pointer]:
- img [ref=e212]
- generic [ref=e213]: "42"
- button "Retweet" [ref=e214] [cursor=pointer]:
- img [ref=e215]
- generic [ref=e216]: "1.2K"
- button "Like" [ref=e217] [cursor=pointer]:
- img [ref=e218]
- generic [ref=e219]: "5.3K"
- separator [ref=e300]`;
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
describe('formatSnapshot', () => {
describe('basic behavior', () => {
it('returns empty string for empty/null input', () => {
expect(formatSnapshot('')).toBe('');
expect(formatSnapshot(null as any)).toBe('');
expect(formatSnapshot(undefined as any)).toBe('');
});
it('strips [ref=...] and [cursor=...] annotations', () => {
const input = '- button "Click me" [ref=e42] [cursor=pointer]';
const result = formatSnapshot(input);
expect(result).not.toContain('[ref=');
expect(result).not.toContain('[cursor=');
expect(result).toContain('button "Click me"');
});
it('removes /url: metadata lines', () => {
const input = `\
- link "Home" [ref=e1] [cursor=pointer]:
- /url: https://example.com
- generic [ref=e2]: Home`;
const result = formatSnapshot(input);
expect(result).not.toContain('/url:');
expect(result).not.toContain('https://example.com');
});
it('assigns sequential [@N] refs to interactive elements', () => {
const input = `\
- button "Save" [ref=e1]
- link "Cancel" [ref=e2]
- textbox "Name" [ref=e3]`;
const result = formatSnapshot(input);
expect(result).toContain('[@1] button "Save"');
expect(result).toContain('[@2] link "Cancel"');
expect(result).toContain('[@3] textbox "Name"');
});
});
describe('noise filtering', () => {
it('removes generic nodes without text', () => {
const input = `\
- generic [ref=e1]:
- generic [ref=e2]:
- button "Click" [ref=e3]`;
const result = formatSnapshot(input);
expect(result).not.toMatch(/^generic/m);
expect(result).toContain('button "Click"');
});
it('keeps generic nodes WITH text content', () => {
const input = '- generic [ref=e23]: Dashboard';
const result = formatSnapshot(input);
expect(result).toContain('generic: Dashboard');
});
it('removes img nodes without alt text', () => {
const input = `\
- img [ref=e13]
- img "Profile photo" [ref=e14]`;
const result = formatSnapshot(input);
expect(result).not.toContain('img\n');
expect(result).toContain('img "Profile photo"');
});
it('removes separator nodes', () => {
const input = '- separator [ref=e304]';
const result = formatSnapshot(input);
expect(result).toBe('');
});
it('removes presentation/none roles', () => {
const input = `\
- presentation [ref=e1]
- none [ref=e2]
- button "OK" [ref=e3]`;
const result = formatSnapshot(input);
expect(result).not.toContain('presentation');
expect(result).not.toContain('none');
expect(result).toContain('button "OK"');
});
});
describe('empty container pruning', () => {
it('prunes containers with no visible children', () => {
const input = `\
- list [ref=e88]:
- listitem [ref=e89]:
- generic [ref=e90]:
- img [ref=e91]`;
// After filtering: generic (no text) → removed, img (no alt) → removed
// listitem becomes empty → pruned, list becomes empty → pruned
const result = formatSnapshot(input);
expect(result).toBe('');
});
it('keeps containers with visible children', () => {
const input = `\
- list [ref=e1]:
- listitem [ref=e2]:
- link "Home" [ref=e3]`;
const result = formatSnapshot(input);
expect(result).toContain('list');
expect(result).toContain('listitem');
expect(result).toContain('link "Home"');
});
});
describe('maxDepth option', () => {
it('limits output to specified depth', () => {
const input = `\
- main [ref=e1]:
- heading "Dashboard" [ref=e2]
- navigation [ref=e3]:
- list [ref=e4]:
- link "Deep link" [ref=e5]`;
const result = formatSnapshot(input, { maxDepth: 2 });
expect(result).toContain('main');
expect(result).toContain('heading "Dashboard"');
// navigation is pruned: its only child list is empty after link is excluded by maxDepth
expect(result).not.toContain('navigation');
expect(result).not.toContain('Deep link');
});
it('handles maxDepth=0 correctly (was a bug)', () => {
const input = `\
- heading "Title" [ref=e1]
- link "Sub" [ref=e2]`;
const result = formatSnapshot(input, { maxDepth: 0 });
expect(result).toContain('heading "Title"');
expect(result).not.toContain('Sub');
});
});
describe('interactive mode', () => {
it('keeps interactive elements and landmarks', () => {
const result = formatSnapshot(GITHUB_NAV, { interactive: true });
// Interactive elements should be present
expect(result).toContain('button');
expect(result).toContain('link');
// Landmarks preserved
expect(result).toContain('banner');
expect(result).toContain('navigation');
});
it('filters non-interactive, non-landmark, textless nodes', () => {
const input = `\
- main [ref=e1]:
- generic [ref=e2]:
- generic [ref=e3]:
- button "Save" [ref=e4]
- generic [ref=e5]: some text content`;
const result = formatSnapshot(input, { interactive: true });
expect(result).toContain('main');
expect(result).toContain('button "Save"');
// generic with text is kept
expect(result).toContain('generic: some text content');
});
});
describe('compact mode', () => {
it('strips bracket annotations and collapses whitespace', () => {
const input = '- button "Save" [ref=e1] [cursor=pointer] [level=2]';
const result = formatSnapshot(input, { compact: true });
// ref/cursor already stripped, but [level=...] should also go in compact
expect(result).not.toContain('[level=');
expect(result).toContain('button');
});
});
describe('maxTextLength option', () => {
it('truncates long content lines', () => {
const input = '- heading "This is a very long heading that should be truncated at some point" [ref=e1]';
const result = formatSnapshot(input, { maxTextLength: 30 });
expect(result.length).toBeLessThanOrEqual(35); // some tolerance for ellipsis
expect(result).toContain('…');
});
});
// ---------------------------------------------------------------------------
// Real-world snapshot integration tests
// ---------------------------------------------------------------------------
describe('GitHub snapshot', () => {
it('drastically reduces nav bar output', () => {
const raw = GITHUB_NAV;
const rawLineCount = raw.split('\n').length;
const result = formatSnapshot(raw);
const resultLineCount = result.split('\n').length;
// Should significantly reduce line count
expect(resultLineCount).toBeLessThan(rawLineCount);
// Key content preserved
expect(result).toContain('link "Skip to content"');
expect(result).toContain('banner "Global Navigation Menu"');
expect(result).toContain('link "Dashboard"');
expect(result).toContain('button "Search or jump to…"');
// Noise removed
expect(result).not.toContain('[ref=');
expect(result).not.toContain('/url:');
});
it('preserves repo list structure', () => {
const result = formatSnapshot(GITHUB_REPOS);
expect(result).toContain('navigation "Repositories"');
expect(result).toContain('heading "Top repositories"');
expect(result).toContain('textbox "Find a repository…"');
expect(result).toContain('link "jackwener/twitter-cli"');
expect(result).toContain('link "jackwener/opencli"');
expect(result).toContain('img "Repository"');
// No refs or urls
expect(result).not.toContain('[ref=');
expect(result).not.toContain('/url:');
});
});
describe('Bilibili snapshot', () => {
it('cleans nav bar with Chinese text', () => {
const result = formatSnapshot(BILIBILI_NAV);
expect(result).toContain('link "首页"');
expect(result).toContain('link "番剧"');
expect(result).toContain('link "直播"');
expect(result).toContain('textbox "冷知识 金廷26年胜率100%"');
expect(result).not.toContain('[ref=');
});
it('handles video card with deeply nested wrappers', () => {
const result = formatSnapshot(BILIBILI_VIDEO);
expect(result).toContain('link "超酷时刻 即将到来 3.3万 40 16:24"');
expect(result).toContain('heading "超酷时刻 即将到来"');
expect(result).toContain('generic "Tesla特斯拉中国"');
// Deeply nested view count generics with text are kept
expect(result).toContain('3.3万');
});
it('prunes empty paragraph blocks', () => {
const result = formatSnapshot(BILIBILI_EMPTY);
// All content is generic (no text) and empty paragraphs
// After noise filtering, everything should be pruned
expect(result.trim()).toBe('');
});
});
describe('Twitter snapshot', () => {
it('preserves tweet structure', () => {
const result = formatSnapshot(TWITTER_TWEET);
expect(result).toContain('main');
expect(result).toContain('region "Timeline"');
expect(result).toContain('link "@elonmusk"');
expect(result).toContain('button "Reply"');
expect(result).toContain('button "Like"');
expect(result).not.toContain('separator');
});
it('truncates long tweet text with maxTextLength', () => {
const result = formatSnapshot(TWITTER_TWEET, { maxTextLength: 60 });
// The long tweet text should be truncated
expect(result).toContain('…');
// But short elements are unaffected
expect(result).toContain('button "Reply"');
});
it('interactive mode keeps only buttons and links', () => {
const result = formatSnapshot(TWITTER_TWEET, { interactive: true });
expect(result).toContain('link "@elonmusk"');
expect(result).toContain('button "Reply"');
expect(result).toContain('button "Retweet"');
expect(result).toContain('button "Like"');
// Structural landmarks kept
expect(result).toContain('main');
expect(result).toContain('region "Timeline"');
expect(result).toContain('article');
});
it('combined options: interactive + maxDepth', () => {
// With maxDepth: 2 and interactive, depth > 2 is filtered.
// article at depth 2 has only generic children (noise-filtered),
// so article gets pruned by container pruning, which cascades up.
const result = formatSnapshot(TWITTER_TWEET, { interactive: true, maxDepth: 2 });
expect(result).toContain('main');
expect(result).not.toContain('button "Reply"');
expect(result).not.toContain('link "@elonmusk"');
});
});
describe('reduction ratios on real data', () => {
it('achieves significant reduction on GitHub nav', () => {
const rawLines = GITHUB_NAV.split('\n').length;
const formatted = formatSnapshot(GITHUB_NAV);
const formattedLines = formatted.split('\n').filter(l => l.trim()).length;
// Expect at least 40% reduction
expect(formattedLines).toBeLessThan(rawLines * 0.6);
});
it('achieves significant reduction on Bilibili video card', () => {
const rawLines = BILIBILI_VIDEO.split('\n').length;
const formatted = formatSnapshot(BILIBILI_VIDEO);
const formattedLines = formatted.split('\n').filter(l => l.trim()).length;
// Expect at least 30% reduction
expect(formattedLines).toBeLessThan(rawLines * 0.7);
});
});
// ---------------------------------------------------------------------------
// Full-page snapshot fixture tests (loaded from __fixtures__/)
// ---------------------------------------------------------------------------
describe('full-page snapshots from fixtures', () => {
const fs = require('node:fs');
const path = require('node:path');
const fixturesDir = path.join(__dirname, '__fixtures__');
function loadFixture(name: string): string | null {
const p = path.join(fixturesDir, name);
if (!fs.existsSync(p)) return null;
return fs.readFileSync(p, 'utf-8');
}
it('GitHub: significant reduction and clean output', () => {
const raw = loadFixture('snapshot_github.txt');
if (!raw) return;
const rawLines = raw.split('\n').length;
const result = formatSnapshot(raw);
const resultLines = result.split('\n').filter((l: string) => l.trim()).length;
// Should achieve > 50% reduction on GitHub dashboard (heavy generic noise)
expect(resultLines).toBeLessThan(rawLines * 0.5);
// No annotations remain
expect(result).not.toContain('[ref=');
expect(result).not.toContain('[cursor=');
expect(result).not.toContain('/url:');
// Key content preserved
expect(result).toContain('link "Skip to content"');
expect(result).toContain('banner "Global Navigation Menu"');
expect(result).toContain('heading "Dashboard"');
});
it('Bilibili: significant reduction and Chinese text preserved', () => {
const raw = loadFixture('snapshot_bilibili.txt');
if (!raw) return;
const rawLines = raw.split('\n').length;
const result = formatSnapshot(raw);
const resultLines = result.split('\n').filter((l: string) => l.trim()).length;
// Should achieve > 40% reduction on Bilibili (lots of imgs and generics)
expect(resultLines).toBeLessThan(rawLines * 0.6);
// No annotations remain
expect(result).not.toContain('[ref=');
expect(result).not.toContain('[cursor=');
// Chinese text preserved
expect(result).toContain('link "首页"');
expect(result).toContain('link "番剧"');
});
it('Twitter/X: significant reduction and tweet structure preserved', () => {
const raw = loadFixture('snapshot_twitter.txt');
if (!raw) return;
const rawLines = raw.split('\n').length;
const result = formatSnapshot(raw);
const resultLines = result.split('\n').filter((l: string) => l.trim()).length;
// Should achieve > 40% reduction on Twitter/X
expect(resultLines).toBeLessThan(rawLines * 0.6);
// No annotations remain
expect(result).not.toContain('[ref=');
expect(result).not.toContain('[cursor=');
expect(result).not.toContain('/url:');
// Key structure preserved
expect(result).toContain('main');
});
it('GitHub interactive mode: drastic reduction', () => {
const raw = loadFixture('snapshot_github.txt');
if (!raw) return;
const result = formatSnapshot(raw, { interactive: true });
const resultLines = result.split('\n').filter((l: string) => l.trim()).length;
// Interactive mode should be much more aggressive
expect(resultLines).toBeLessThan(200);
// Interactive elements still present
expect(result).toContain('button');
expect(result).toContain('link');
expect(result).toContain('textbox');
});
it('Bilibili maxDepth=3: shallow view', () => {
const raw = loadFixture('snapshot_bilibili.txt');
if (!raw) return;
const result = formatSnapshot(raw, { maxDepth: 3 });
const resultLines = result.split('\n').filter((l: string) => l.trim()).length;
// Depth-limited should be very compact
expect(resultLines).toBeLessThan(50);
});
});
});
+401 -15
View File
@@ -1,35 +1,268 @@
/**
* Aria snapshot formatter: parses Playwright MCP snapshot text into clean format.
*
* Multi-pass pipeline:
* 1. Parse & filter: strip annotations, metadata, noise roles, ads, decorators
* 2. Deduplicate: generic/text child matching parent label
* 3. Deduplicate: heading + link with identical labels
* 4. Deduplicate: nested identical links
* 5. Prune: empty containers (iterative bottom-up)
* 6. Collapse: single-child containers
*/
export interface FormatOptions {
interactive?: boolean;
compact?: boolean;
maxDepth?: number;
maxTextLength?: number;
}
const DEFAULT_MAX_TEXT_LENGTH = 200;
// Roles that are pure noise and should always be filtered
const NOISE_ROLES = new Set([
'none', 'presentation', 'separator', 'paragraph', 'tooltip', 'status',
]);
// Roles whose entire subtree should be removed (footer boilerplate, etc.)
const SUBTREE_NOISE_ROLES = new Set([
'contentinfo',
]);
// Roles considered interactive (clickable/typeable)
const INTERACTIVE_ROLES = new Set([
'button', 'link', 'textbox', 'checkbox', 'radio',
'combobox', 'tab', 'menuitem', 'option', 'switch',
'slider', 'spinbutton', 'searchbox',
]);
// Structural landmark roles kept even in interactive mode
const LANDMARK_ROLES = new Set([
'main', 'navigation', 'banner', 'heading', 'search',
'region', 'list', 'listitem', 'article', 'complementary',
'group', 'toolbar', 'tablist',
]);
// Container roles eligible for pruning and collapse
const CONTAINER_ROLES = new Set([
'list', 'listitem', 'group', 'toolbar', 'tablist',
'navigation', 'region', 'complementary',
'search', 'article', 'paragraph', 'figure',
]);
// Decorator / separator text that adds no semantic value
const DECORATOR_TEXT = new Set(['•', '·', '|', '—', '-', '/', '\\']);
// Ad-related URL patterns
const AD_URL_PATTERNS = [
'googleadservices.com/pagead/',
'alb.reddit.com/cr?',
'doubleclick.net/',
'cm.bilibili.com/cm/api/fees/',
];
// Boilerplate button labels to filter (back-to-top, etc.)
const BOILERPLATE_LABELS = [
'回到顶部', 'back to top', 'scroll to top', 'go to top',
];
/**
* Parse role and text from a trimmed snapshot line.
* Handles quoted labels and trailing text after colon correctly,
* including lines wrapped in single quotes by Playwright.
*/
function parseLine(trimmed: string): { role: string; text: string; hasText: boolean; trailingText: string } {
// Unwrap outer single quotes if present (Playwright wraps lines with special chars)
let line = trimmed;
if (line.startsWith("'") && line.endsWith("':")) {
line = line.slice(1, -2) + ':';
} else if (line.startsWith("'") && line.endsWith("'")) {
line = line.slice(1, -1);
}
// Role is the first word
const roleMatch = line.match(/^([a-zA-Z]+)\b/);
const role = roleMatch ? roleMatch[1].toLowerCase() : '';
// Extract quoted text content (the semantic label)
const textMatch = line.match(/"([^"]*)"/);
const text = textMatch ? textMatch[1] : '';
// For trailing text: strip annotations and quoted strings first, then check after last colon
// This avoids matching colons inside quoted labels like "Account: user@email.com"
let stripped = line;
// Remove all quoted strings
stripped = stripped.replace(/"[^"]*"/g, '""');
// Remove all bracket annotations
stripped = stripped.replace(/\[[^\]]*\]/g, '');
const colonIdx = stripped.lastIndexOf(':');
let trailingText = '';
if (colonIdx !== -1) {
const afterColon = stripped.slice(colonIdx + 1).trim();
if (afterColon.length > 0) {
// Get the actual trailing text from original line at same position
const origColonIdx = line.lastIndexOf(':');
if (origColonIdx !== -1) {
trailingText = line.slice(origColonIdx + 1).trim();
}
}
}
return { role, text, hasText: text.length > 0 || trailingText.length > 0, trailingText };
}
/**
* Strip ALL bracket annotations from a content line, preserving quoted strings.
* Handles both double-quoted and outer single-quoted lines from Playwright.
*/
function stripAnnotations(content: string): string {
// Unwrap outer single quotes first
let line = content;
if (line.startsWith("'") && (line.endsWith("':") || line.endsWith("'"))) {
if (line.endsWith("':")) {
line = line.slice(1, -2) + ':';
} else {
line = line.slice(1, -1);
}
}
// Split by double quotes to protect quoted content
const parts = line.split('"');
for (let i = 0; i < parts.length; i += 2) {
// Only strip annotations from non-quoted parts (even indices)
parts[i] = parts[i].replace(/\s*\[[^\]]*\]/g, '');
}
let result = parts.join('"').replace(/\s{2,}/g, ' ').trim();
return result;
}
/**
* Check if a line is a metadata-only line (like /url: ...).
*/
function isMetadataLine(trimmed: string): boolean {
return /^\/[a-zA-Z]+:/.test(trimmed);
}
/**
* Check if text content is purely decorative (separators, dots, etc.)
*/
function isDecoratorText(text: string): boolean {
return DECORATOR_TEXT.has(text.trim());
}
/**
* Check if a node is ad-related based on its text content.
*/
function isAdNode(text: string, trailingText: string): boolean {
const t = (text + ' ' + trailingText).toLowerCase();
if (t.includes('sponsored') || t.includes('advertisement')) return true;
if (t.includes('广告')) return true;
// Check for ad tracking URLs in the label
for (const pattern of AD_URL_PATTERNS) {
if (text.includes(pattern) || trailingText.includes(pattern)) return true;
}
return false;
}
/**
* Check if a node is boilerplate UI (back-to-top, etc.)
*/
function isBoilerplateNode(text: string): boolean {
const t = text.toLowerCase();
return BOILERPLATE_LABELS.some(label => t.includes(label));
}
/**
* Check if a role is noise that should be filtered.
*/
function isNoiseNode(role: string, hasText: boolean, text: string, trailingText: string): boolean {
if (NOISE_ROLES.has(role)) return true;
// generic without text is a wrapper
if (role === 'generic' && !hasText) return true;
// img without alt text is noise
if (role === 'img' && !hasText) return true;
// Decorator-only text nodes
if ((role === 'generic' || role === 'text') && hasText) {
const content = trailingText || text;
if (isDecoratorText(content)) return true;
}
return false;
}
interface Entry {
depth: number;
content: string;
role: string;
text: string;
trailingText: string;
isInteractive: boolean;
isLandmark: boolean;
isSubtreeSkip: boolean; // ad nodes or boilerplate — skip entire subtree
}
export function formatSnapshot(raw: string, opts: FormatOptions = {}): string {
if (!raw || typeof raw !== 'string') return '';
const lines = raw.split('\n');
const result: string[] = [];
let refCounter = 0;
for (const line of lines) {
const maxTextLen = opts.maxTextLength ?? DEFAULT_MAX_TEXT_LENGTH;
const lines = raw.split('\n');
// === Pass 1: Parse, filter, and collect entries ===
const entries: Entry[] = [];
let refCounter = 0;
let skipUntilDepth = -1; // When >= 0, skip all nodes at depth > this value
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
if (!line.trim()) continue;
const indent = line.length - line.trimStart().length;
const depth = Math.floor(indent / 2);
if (opts.maxDepth && depth > opts.maxDepth) continue;
// If we're in a subtree skip zone, check depth
if (skipUntilDepth >= 0) {
if (depth > skipUntilDepth) continue; // still inside subtree
skipUntilDepth = -1; // exited subtree
}
let content = line.trimStart();
// Skip non-interactive elements in interactive mode
if (opts.interactive) {
const interactiveRoles = ['button', 'link', 'textbox', 'checkbox', 'radio', 'combobox', 'tab', 'menuitem', 'option'];
const role = content.split(/[\s[]/)[0]?.toLowerCase() ?? '';
if (!interactiveRoles.some(r => role.includes(r)) && depth > 1) continue;
// Strip leading "- "
if (content.startsWith('- ')) {
content = content.slice(2);
}
// Compact: strip verbose role descriptions
// Skip metadata lines
if (isMetadataLine(content)) continue;
// Apply maxDepth filter
if (opts.maxDepth !== undefined && depth > opts.maxDepth) continue;
const { role, text, hasText, trailingText } = parseLine(content);
// Skip noise nodes
if (isNoiseNode(role, hasText, text, trailingText)) continue;
// Skip subtree noise roles (contentinfo footer, etc.) — skip entire subtree
if (SUBTREE_NOISE_ROLES.has(role)) {
skipUntilDepth = depth;
continue;
}
// Strip annotations
content = stripAnnotations(content);
// Check if node should trigger subtree skip (ads, boilerplate)
const isSubtreeSkip = isAdNode(text, trailingText) || isBoilerplateNode(text);
// Interactive mode filter
const isInteractive = INTERACTIVE_ROLES.has(role);
const isLandmark = LANDMARK_ROLES.has(role);
if (opts.interactive && !isInteractive && !isLandmark && !hasText) continue;
// Compact mode
if (opts.compact) {
content = content
.replace(/\s*\[.*?\]\s*/g, ' ')
@@ -37,15 +270,168 @@ export function formatSnapshot(raw: string, opts: FormatOptions = {}): string {
.trim();
}
// Text truncation
if (maxTextLen > 0 && content.length > maxTextLen) {
content = content.slice(0, maxTextLen) + '…';
}
// Assign refs to interactive elements
const interactivePattern = /^(button|link|textbox|checkbox|radio|combobox|tab|menuitem|option)\b/i;
if (interactivePattern.test(content)) {
if (isInteractive) {
refCounter++;
content = `[@${refCounter}] ${content}`;
}
result.push(' '.repeat(depth) + content);
entries.push({ depth, content, role, text, trailingText, isInteractive, isLandmark, isSubtreeSkip });
}
return result.join('\n');
// === Pass 2: Remove subtree-skip nodes (ads, boilerplate, contentinfo) ===
let noAds: Entry[] = [];
for (let i = 0; i < entries.length; i++) {
const entry = entries[i];
if (entry.isSubtreeSkip) {
const skipDepth = entry.depth;
i++;
while (i < entries.length && entries[i].depth > skipDepth) {
i++;
}
i--;
continue;
}
noAds.push(entry);
}
// === Pass 3: Deduplicate child generic/text matching parent label ===
let deduped: Entry[] = [];
for (let i = 0; i < noAds.length; i++) {
const entry = noAds[i];
if (entry.role === 'generic' || entry.role === 'text') {
let parent: Entry | undefined;
for (let j = deduped.length - 1; j >= 0; j--) {
if (deduped[j].depth < entry.depth) {
parent = deduped[j];
break;
}
if (deduped[j].depth === entry.depth) break;
}
if (parent) {
const childText = entry.trailingText || entry.text;
if (childText && parent.text && childText === parent.text) {
continue;
}
}
}
deduped.push(entry);
}
// === Pass 4: Deduplicate heading + child link with identical label ===
// Pattern: heading "Title": → link "Title": (same text) → skip the link
const deduped2: Entry[] = [];
for (let i = 0; i < deduped.length; i++) {
const entry = deduped[i];
if (entry.role === 'heading' && entry.text) {
const next = deduped[i + 1];
if (next && next.role === 'link' && next.text === entry.text && next.depth === entry.depth + 1) {
// Keep the heading, skip the link. But preserve link's children re-parented.
deduped2.push(entry);
i++; // skip the link
continue;
}
}
deduped2.push(entry);
}
// === Pass 5: Deduplicate nested identical links ===
const deduped3: Entry[] = [];
for (let i = 0; i < deduped2.length; i++) {
const entry = deduped2[i];
if (entry.role === 'link' && entry.text) {
const next = deduped2[i + 1];
if (next && next.role === 'link' && next.text === entry.text && next.depth === entry.depth + 1) {
continue; // Skip parent, keep child
}
}
deduped3.push(entry);
}
// === Pass 6: Iteratively prune empty containers (bottom-up) ===
let current = deduped3;
let changed = true;
while (changed) {
changed = false;
const next: Entry[] = [];
for (let i = 0; i < current.length; i++) {
const entry = current[i];
if (CONTAINER_ROLES.has(entry.role) && !entry.text && !entry.trailingText) {
let hasChildren = false;
for (let j = i + 1; j < current.length; j++) {
if (current[j].depth <= entry.depth) break;
if (current[j].depth > entry.depth) {
hasChildren = true;
break;
}
}
if (!hasChildren) {
changed = true;
continue;
}
}
next.push(entry);
}
current = next;
}
// === Pass 7: Collapse single-child containers ===
const collapsed: Entry[] = [];
for (let i = 0; i < current.length; i++) {
const entry = current[i];
if (CONTAINER_ROLES.has(entry.role) && !entry.text && !entry.trailingText) {
let childCount = 0;
let childIdx = -1;
for (let j = i + 1; j < current.length; j++) {
if (current[j].depth <= entry.depth) break;
if (current[j].depth === entry.depth + 1) {
childCount++;
if (childCount === 1) childIdx = j;
}
}
if (childCount === 1 && childIdx !== -1) {
const child = current[childIdx];
let hasGrandchildren = false;
for (let j = childIdx + 1; j < current.length; j++) {
if (current[j].depth <= child.depth) break;
if (current[j].depth > child.depth) {
hasGrandchildren = true;
break;
}
}
if (!hasGrandchildren) {
const mergedContent = entry.content.replace(/:$/, '') + ' > ' + child.content;
collapsed.push({
...entry,
content: mergedContent,
role: child.role,
text: child.text,
trailingText: child.trailingText,
isInteractive: child.isInteractive,
});
i++;
continue;
}
}
}
collapsed.push(entry);
}
return collapsed.map(e => ' '.repeat(e.depth) + e.content).join('\n');
}
+19 -13
View File
@@ -76,28 +76,28 @@ export async function checkboxPrompt(
const wasRaw = stdin.isRaw;
stdin.setRawMode(true);
stdin.resume();
stdout.write('\x1b[?25l'); // Hide cursor
let rendered = '';
let firstDraw = true;
function draw() {
// Clear previous render
if (rendered) {
const lines = rendered.split('\n').length;
// Clear previous render (skip on first draw)
if (!firstDraw) {
const lines = render().split('\n').length;
stdout.write(`\x1b[${lines}A\x1b[J`);
}
rendered = render();
stdout.write(rendered);
firstDraw = false;
stdout.write(render());
}
function cleanup() {
stdin.setRawMode(wasRaw ?? false);
stdin.pause();
stdin.removeListener('data', onData);
// Clear the TUI
if (rendered) {
const lines = rendered.split('\n').length;
stdout.write(`\x1b[${lines}A\x1b[J`);
}
// Clear the TUI and restore cursor
const lines = render().split('\n').length;
stdout.write(`\x1b[${lines}A\x1b[J`);
stdout.write('\x1b[?25h'); // Show cursor
}
function onData(data: Buffer) {
@@ -150,13 +150,19 @@ export async function checkboxPrompt(
return;
}
// q / Esc / Ctrl+C — cancel
if (key === 'q' || key === '\x1b' || key === '\x03') {
// q / Esc — cancel
if (key === 'q' || key === '\x1b') {
cleanup();
stdout.write(` ${chalk.yellow('✗')} ${chalk.dim('Cancelled')}\n\n`);
resolve([]);
return;
}
// Ctrl+C — exit process
if (key === '\x03') {
cleanup();
process.exit(130);
}
}
stdin.on('data', onData);
+19 -4
View File
@@ -11,8 +11,22 @@ const KNOWN_STEP_NAMES = new Set([
'intercept', 'tap',
]);
export function validateClisWithTarget(dirs: string[], target?: string): any {
const results: any[] = [];
export interface FileValidationResult {
path: string;
errors: string[];
warnings: string[];
}
export interface ValidationReport {
ok: boolean;
results: FileValidationResult[];
errors: number;
warnings: number;
files: number;
}
export function validateClisWithTarget(dirs: string[], target?: string): ValidationReport {
const results: FileValidationResult[] = [];
let errors = 0; let warnings = 0; let files = 0;
for (const dir of dirs) {
if (!fs.existsSync(dir)) continue;
@@ -35,7 +49,7 @@ export function validateClisWithTarget(dirs: string[], target?: string): any {
return { ok: errors === 0, results, errors, warnings, files };
}
function validateYamlFile(filePath: string): any {
function validateYamlFile(filePath: string): FileValidationResult {
const errors: string[] = []; const warnings: string[] = [];
try {
const raw = fs.readFileSync(filePath, 'utf-8');
@@ -64,7 +78,7 @@ function validateYamlFile(filePath: string): any {
return { path: filePath, errors, warnings };
}
export function renderValidationReport(report: any): string {
export function renderValidationReport(report: ValidationReport): string {
const lines = [`opencli validate: ${report.ok ? 'PASS' : 'FAIL'}`, `Checked ${report.results.length} CLI(s) in ${report.files} file(s)`, `Errors: ${report.errors} Warnings: ${report.warnings}`];
for (const r of report.results) {
if (r.errors.length > 0 || r.warnings.length > 0) {
@@ -75,3 +89,4 @@ export function renderValidationReport(report: any): string {
}
return lines.join('\n');
}
+17 -3
View File
@@ -6,13 +6,27 @@
* to the `opencli test` command or CI pipelines.
*/
import { validateClisWithTarget, renderValidationReport } from './validate.js';
import { validateClisWithTarget, renderValidationReport, type ValidationReport } from './validate.js';
export async function verifyClis(opts: any): Promise<any> {
export interface VerifyOptions {
builtinClis: string;
userClis: string;
target?: string;
smoke?: boolean;
}
export interface VerifyReport {
ok: boolean;
validation: ValidationReport;
smoke: null;
}
export async function verifyClis(opts: VerifyOptions): Promise<VerifyReport> {
const report = validateClisWithTarget([opts.builtinClis, opts.userClis], opts.target);
return { ok: report.ok, validation: report, smoke: null };
}
export function renderVerifyReport(report: any): string {
export function renderVerifyReport(report: VerifyReport): string {
return renderValidationReport(report.validation);
}
+90
View File
@@ -0,0 +1,90 @@
/**
* E2E tests for login-required browser commands.
* These commands REQUIRE authentication (cookie/session).
* In CI (headless, no login), they should fail gracefully — NOT crash.
*
* These tests verify the error handling path, not the data extraction.
*/
import { describe, it, expect } from 'vitest';
import { runCli } from './helpers.js';
/**
* Verify a login-required command fails gracefully (no crash, no hang).
* Acceptable outcomes: exit code 1 with error message, OR timeout handled.
*/
async function expectGracefulAuthFailure(args: string[], label: string) {
const { stdout, stderr, code } = await runCli(args, { timeout: 60_000 });
// Should either fail with exit code 1 (error message) or succeed with empty data
// The key assertion: it should NOT hang forever or crash with unhandled exception
if (code !== 0) {
// Verify stderr has a meaningful error, not an unhandled crash
const output = stderr + stdout;
expect(output.length).toBeGreaterThan(0);
}
// If it somehow succeeds (e.g., partial public data), that's fine too
}
describe('login-required commands — graceful failure', () => {
// ── bilibili (requires cookie session) ──
it('bilibili me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'me', '-f', 'json'], 'bilibili me');
}, 60_000);
it('bilibili dynamic fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'dynamic', '--limit', '3', '-f', 'json'], 'bilibili dynamic');
}, 60_000);
it('bilibili favorite fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'favorite', '--limit', '3', '-f', 'json'], 'bilibili favorite');
}, 60_000);
it('bilibili history fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'history', '--limit', '3', '-f', 'json'], 'bilibili history');
}, 60_000);
it('bilibili following fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'following', '--limit', '3', '-f', 'json'], 'bilibili following');
}, 60_000);
// ── twitter (requires login) ──
it('twitter bookmarks fails gracefully without login', async () => {
await expectGracefulAuthFailure(['twitter', 'bookmarks', '--limit', '3', '-f', 'json'], 'twitter bookmarks');
}, 60_000);
it('twitter timeline fails gracefully without login', async () => {
await expectGracefulAuthFailure(['twitter', 'timeline', '--limit', '3', '-f', 'json'], 'twitter timeline');
}, 60_000);
it('twitter notifications fails gracefully without login', async () => {
await expectGracefulAuthFailure(['twitter', 'notifications', '--limit', '3', '-f', 'json'], 'twitter notifications');
}, 60_000);
// ── v2ex (requires login) ──
it('v2ex me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['v2ex', 'me', '-f', 'json'], 'v2ex me');
}, 60_000);
it('v2ex notifications fails gracefully without login', async () => {
await expectGracefulAuthFailure(['v2ex', 'notifications', '--limit', '3', '-f', 'json'], 'v2ex notifications');
}, 60_000);
// ── xueqiu (requires login) ──
it('xueqiu feed fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xueqiu', 'feed', '--limit', '3', '-f', 'json'], 'xueqiu feed');
}, 60_000);
it('xueqiu watchlist fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xueqiu', 'watchlist', '-f', 'json'], 'xueqiu watchlist');
}, 60_000);
// ── xiaohongshu (requires login) ──
it('xiaohongshu feed fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xiaohongshu', 'feed', '--limit', '3', '-f', 'json'], 'xiaohongshu feed');
}, 60_000);
it('xiaohongshu notifications fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xiaohongshu', 'notifications', '--limit', '3', '-f', 'json'], 'xiaohongshu notifications');
}, 60_000);
});
+169
View File
@@ -0,0 +1,169 @@
/**
* E2E tests for browser commands that access PUBLIC data (no login required).
* These use OPENCLI_HEADLESS=1 to launch a headless Chromium.
*
* NOTE: Some sites may block headless browsers with bot detection.
* Tests are wrapped with tryBrowserCommand() which allows graceful failure.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
/**
* Run a browser command — returns parsed data or null on failure.
*/
async function tryBrowserCommand(args: string[]): Promise<any[] | null> {
const { stdout, code } = await runCli(args, { timeout: 60_000 });
if (code !== 0) return null;
try {
const data = parseJsonOutput(stdout);
return Array.isArray(data) ? data : null;
} catch {
return null;
}
}
/**
* Assert browser command returns data OR log a warning if blocked.
* Empty results (bot detection, geo-blocking) are treated as a warning, not a failure.
*/
function expectDataOrSkip(data: any[] | null, label: string) {
if (data === null || data.length === 0) {
console.warn(`${label}: skipped — no data returned (likely bot detection or geo-blocking)`);
return;
}
expect(data.length).toBeGreaterThanOrEqual(1);
}
describe('browser public-data commands E2E', () => {
// ── bbc (browser: true, strategy: public) ──
it('bbc news returns headlines', async () => {
const data = await tryBrowserCommand(['bbc', 'news', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'bbc news');
if (data) {
expect(data[0]).toHaveProperty('title');
}
}, 60_000);
// ── v2ex daily (browser: true) ──
it('v2ex daily returns topics', async () => {
const data = await tryBrowserCommand(['v2ex', 'daily', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'v2ex daily');
}, 60_000);
// ── bilibili (browser: true, cookie strategy) ──
it('bilibili hot returns trending videos', async () => {
const data = await tryBrowserCommand(['bilibili', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'bilibili hot');
if (data) {
expect(data[0]).toHaveProperty('title');
}
}, 60_000);
it('bilibili ranking returns ranked videos', async () => {
const data = await tryBrowserCommand(['bilibili', 'ranking', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'bilibili ranking');
}, 60_000);
it('bilibili search returns results', async () => {
const data = await tryBrowserCommand(['bilibili', 'search', '--keyword', 'typescript', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'bilibili search');
}, 60_000);
// ── weibo (browser: true, cookie strategy) ──
it('weibo hot returns trending topics', async () => {
const data = await tryBrowserCommand(['weibo', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'weibo hot');
}, 60_000);
// ── zhihu (browser: true, cookie strategy) ──
it('zhihu hot returns trending questions', async () => {
const data = await tryBrowserCommand(['zhihu', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'zhihu hot');
if (data) {
expect(data[0]).toHaveProperty('title');
}
}, 60_000);
it('zhihu search returns results', async () => {
const data = await tryBrowserCommand(['zhihu', 'search', '--keyword', 'playwright', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'zhihu search');
}, 60_000);
// ── reddit (browser: true, cookie strategy) ──
it('reddit hot returns posts', async () => {
const data = await tryBrowserCommand(['reddit', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'reddit hot');
}, 60_000);
it('reddit frontpage returns posts', async () => {
const data = await tryBrowserCommand(['reddit', 'frontpage', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'reddit frontpage');
}, 60_000);
// ── twitter (browser: true) ──
it('twitter trending returns trends', async () => {
const data = await tryBrowserCommand(['twitter', 'trending', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'twitter trending');
}, 60_000);
// ── xueqiu (browser: true, cookie strategy) ──
it('xueqiu hot returns hot posts', async () => {
const data = await tryBrowserCommand(['xueqiu', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'xueqiu hot');
}, 60_000);
it('xueqiu hot-stock returns stocks', async () => {
const data = await tryBrowserCommand(['xueqiu', 'hot-stock', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'xueqiu hot-stock');
}, 60_000);
// ── reuters (browser: true) ──
it('reuters search returns articles', async () => {
const data = await tryBrowserCommand(['reuters', 'search', '--keyword', 'technology', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'reuters search');
}, 60_000);
// ── youtube (browser: true) ──
it('youtube search returns videos', async () => {
const data = await tryBrowserCommand(['youtube', 'search', '--keyword', 'typescript tutorial', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'youtube search');
}, 60_000);
// ── smzdm (browser: true) ──
it('smzdm search returns deals', async () => {
const data = await tryBrowserCommand(['smzdm', 'search', '--keyword', '键盘', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'smzdm search');
}, 60_000);
// ── boss (browser: true) ──
it('boss search returns jobs', async () => {
const data = await tryBrowserCommand(['boss', 'search', '--keyword', 'golang', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'boss search');
}, 60_000);
// ── ctrip (browser: true) ──
it('ctrip search returns flights', async () => {
const data = await tryBrowserCommand(['ctrip', 'search', '-f', 'json']);
expectDataOrSkip(data, 'ctrip search');
}, 60_000);
// ── coupang (browser: true) ──
it('coupang search returns products', async () => {
const data = await tryBrowserCommand(['coupang', 'search', '--keyword', 'laptop', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'coupang search');
}, 60_000);
// ── xiaohongshu (browser: true) ──
it('xiaohongshu search returns notes', async () => {
const data = await tryBrowserCommand(['xiaohongshu', 'search', '--keyword', '美食', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'xiaohongshu search');
}, 60_000);
// ── yahoo-finance (browser: true) ──
it('yahoo-finance quote returns stock data', async () => {
const data = await tryBrowserCommand(['yahoo-finance', 'quote', '--symbol', 'AAPL', '-f', 'json']);
expectDataOrSkip(data, 'yahoo-finance quote');
}, 60_000);
});
+63
View File
@@ -0,0 +1,63 @@
/**
* Shared helpers for E2E tests.
* Runs the built opencli binary as a subprocess.
*/
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import * as path from 'node:path';
import { fileURLToPath } from 'node:url';
const exec = promisify(execFile);
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const ROOT = path.resolve(__dirname, '../..');
const MAIN = path.join(ROOT, 'dist', 'main.js');
export interface CliResult {
stdout: string;
stderr: string;
code: number;
}
/**
* Run `opencli` as a child process with the given arguments.
* Without PLAYWRIGHT_MCP_EXTENSION_TOKEN, opencli auto-launches its own browser.
*/
export async function runCli(
args: string[],
opts: { timeout?: number; env?: Record<string, string> } = {},
): Promise<CliResult> {
const timeout = opts.timeout ?? 30_000;
try {
const { stdout, stderr } = await exec('node', [MAIN, ...args], {
cwd: ROOT,
timeout,
env: {
...process.env,
// Prevent chalk colors from polluting test assertions
FORCE_COLOR: '0',
NO_COLOR: '1',
...opts.env,
},
});
return { stdout, stderr, code: 0 };
} catch (err: any) {
return {
stdout: err.stdout ?? '',
stderr: err.stderr ?? '',
code: err.code ?? 1,
};
}
}
/**
* Parse JSON output from a CLI command.
* Throws a descriptive error if parsing fails.
*/
export function parseJsonOutput(stdout: string): any {
try {
return JSON.parse(stdout.trim());
} catch {
throw new Error(`Failed to parse CLI JSON output:\n${stdout.slice(0, 500)}`);
}
}
+106
View File
@@ -0,0 +1,106 @@
/**
* E2E tests for management/built-in commands.
* These commands require no external network access (except verify --smoke).
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
describe('management commands E2E', () => {
// ── list ──
it('list shows all registered commands', async () => {
const { stdout, code } = await runCli(['list', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
// Should have 50+ commands across 18 sites
expect(data.length).toBeGreaterThan(50);
// Each entry should have the standard fields
expect(data[0]).toHaveProperty('command');
expect(data[0]).toHaveProperty('site');
expect(data[0]).toHaveProperty('name');
expect(data[0]).toHaveProperty('strategy');
expect(data[0]).toHaveProperty('browser');
});
it('list default table format renders sites', async () => {
const { stdout, code } = await runCli(['list']);
expect(code).toBe(0);
// Should contain site names
expect(stdout).toContain('hackernews');
expect(stdout).toContain('bilibili');
expect(stdout).toContain('twitter');
expect(stdout).toContain('commands across');
});
it('list -f yaml produces valid yaml', async () => {
const { stdout, code } = await runCli(['list', '-f', 'yaml']);
expect(code).toBe(0);
expect(stdout).toContain('command:');
expect(stdout).toContain('site:');
});
it('list -f csv produces valid csv', async () => {
const { stdout, code } = await runCli(['list', '-f', 'csv']);
expect(code).toBe(0);
const lines = stdout.trim().split('\n');
expect(lines.length).toBeGreaterThan(50);
});
it('list -f md produces markdown table', async () => {
const { stdout, code } = await runCli(['list', '-f', 'md']);
expect(code).toBe(0);
expect(stdout).toContain('|');
expect(stdout).toContain('command');
});
// ── validate ──
it('validate passes for all built-in adapters', async () => {
const { stdout, code } = await runCli(['validate']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
expect(stdout).not.toContain('❌');
});
it('validate works for specific site', async () => {
const { stdout, code } = await runCli(['validate', 'hackernews']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
it('validate works for specific command', async () => {
const { stdout, code } = await runCli(['validate', 'hackernews/top']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
// ── verify ──
it('verify runs validation without smoke tests', async () => {
const { stdout, code } = await runCli(['verify']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
// ── version ──
it('--version shows version number', async () => {
const { stdout, code } = await runCli(['--version']);
expect(code).toBe(0);
expect(stdout.trim()).toMatch(/^\d+\.\d+\.\d+$/);
});
// ── help ──
it('--help shows usage', async () => {
const { stdout, code } = await runCli(['--help']);
expect(code).toBe(0);
expect(stdout).toContain('opencli');
expect(stdout).toContain('list');
expect(stdout).toContain('validate');
});
// ── unknown command ──
it('unknown command shows error', async () => {
const { stderr, code } = await runCli(['nonexistent-command-xyz']);
expect(code).toBe(1);
});
});
+48
View File
@@ -0,0 +1,48 @@
/**
* E2E tests for output format rendering.
* Uses hackernews (public, fast) as a stable data source.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
const FORMATS = ['json', 'yaml', 'csv', 'md'] as const;
describe('output formats E2E', () => {
for (const fmt of FORMATS) {
it(`hackernews top -f ${fmt} produces valid output`, async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '2', '-f', fmt]);
expect(code).toBe(0);
expect(stdout.trim().length).toBeGreaterThan(0);
if (fmt === 'json') {
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBe(2);
}
if (fmt === 'yaml') {
expect(stdout).toContain('title:');
}
if (fmt === 'csv') {
// CSV should have a header row + data rows
const lines = stdout.trim().split('\n');
expect(lines.length).toBeGreaterThanOrEqual(2);
}
if (fmt === 'md') {
// Markdown table should have pipe characters
expect(stdout).toContain('|');
}
}, 30_000);
}
it('list -f csv produces valid csv', async () => {
const { stdout, code } = await runCli(['list', '-f', 'csv']);
expect(code).toBe(0);
const lines = stdout.trim().split('\n');
// Header + many data lines
expect(lines.length).toBeGreaterThan(50);
});
});
+56
View File
@@ -0,0 +1,56 @@
/**
* E2E tests for public API commands (browser: false).
* These commands use Node.js fetch directly — no browser needed.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
describe('public commands E2E', () => {
// ── hackernews ──
it('hackernews top returns structured data', async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBe(3);
expect(data[0]).toHaveProperty('title');
expect(data[0]).toHaveProperty('score');
expect(data[0]).toHaveProperty('rank');
}, 30_000);
it('hackernews top respects --limit', async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '1', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBe(1);
}, 30_000);
// ── v2ex (public API, browser: false) ──
it('v2ex hot returns topics', async () => {
const { stdout, code } = await runCli(['v2ex', 'hot', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
it('v2ex latest returns topics', async () => {
const { stdout, code } = await runCli(['v2ex', 'latest', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
}, 30_000);
it('v2ex topic returns topic detail', async () => {
// Topic 1000001 is a well-known V2EX topic
const { stdout, code } = await runCli(['v2ex', 'topic', '--id', '1000001', '-f', 'json']);
// May fail if V2EX rate-limits, but should return structured data
if (code === 0) {
const data = parseJsonOutput(stdout);
expect(data).toBeDefined();
}
}, 30_000);
});
+72
View File
@@ -0,0 +1,72 @@
/**
* Smoke tests for external API health.
* Only run on schedule or manual dispatch — NOT on every push/PR.
* These verify that external APIs haven't changed their structure.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from '../e2e/helpers.js';
describe('API health smoke tests', () => {
// ── Public API commands (should always work) ──
it('hackernews API is responsive and returns expected structure', async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '5', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBe(5);
for (const item of data) {
expect(item).toHaveProperty('title');
expect(item).toHaveProperty('score');
expect(item).toHaveProperty('author');
expect(item).toHaveProperty('rank');
}
}, 30_000);
it('v2ex hot API is responsive', async () => {
const { stdout, code } = await runCli(['v2ex', 'hot', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
it('v2ex latest API is responsive', async () => {
const { stdout, code } = await runCli(['v2ex', 'latest', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBeGreaterThanOrEqual(1);
}, 30_000);
it('v2ex topic API is responsive', async () => {
const { stdout, code } = await runCli(['v2ex', 'topic', '--id', '1000001', '-f', 'json']);
if (code === 0) {
const data = parseJsonOutput(stdout);
expect(data).toBeDefined();
}
}, 30_000);
// ── Validate all adapters ──
it('all adapter definitions are valid', async () => {
const { stdout, code } = await runCli(['validate']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
// ── Command registry integrity ──
it('all expected sites are registered', async () => {
const { stdout, code } = await runCli(['list', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
const sites = new Set(data.map((d: any) => d.site));
// Verify all 17 sites are present
for (const expected of [
'hackernews', 'bbc', 'bilibili', 'v2ex', 'weibo', 'zhihu',
'twitter', 'reddit', 'xueqiu', 'reuters', 'youtube',
'smzdm', 'boss', 'ctrip', 'coupang', 'xiaohongshu',
'yahoo-finance',
]) {
expect(sites.has(expected)).toBe(true);
}
});
});
+1
View File
@@ -6,6 +6,7 @@
"outDir": "dist",
"rootDir": "src",
"strict": false,
"strictNullChecks": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
+22 -1
View File
@@ -2,6 +2,27 @@ import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
include: ['src/**/*.test.ts'],
projects: [
{
test: {
name: 'unit',
include: ['src/**/*.test.ts'],
// Run unit tests before e2e tests to avoid project-level contention in CI.
sequence: {
groupOrder: 0,
},
},
},
{
test: {
name: 'e2e',
include: ['tests/**/*.test.ts'],
maxWorkers: 2,
sequence: {
groupOrder: 1,
},
},
},
],
},
});