Compare commits

...

18 Commits

Author SHA1 Message Date
jackwener ae4929e73d test(e2e): remove redundant --keyword flags from browser-public tests
CI / build (push) Has been cancelled
CI / unit-test (1) (push) Has been cancelled
CI / unit-test (2) (push) Has been cancelled
CI / smoke-test (push) Has been cancelled
E2E Headed Chrome / e2e-headed (push) Has been cancelled
Positional args don't need --name prefix. Replace all instances of
['cmd', 'search', '--keyword', 'VALUE'] with ['cmd', 'search', 'VALUE'].
2026-03-23 17:50:41 +08:00
jackwener 92408e7ee9 fix: improve E2E testing infrastructure
- buildMcpArgs: use CI env var instead of token detection (preserves local behavior)
- browser.test.ts: align tests with CI-based mode detection
- api-health.test.ts: remove duplicate xueqiu entry
- browser-public.test.ts: unify BBC test with tryBrowserCommand
- vitest.config.ts: default to unit tests only (E2E via explicit path)
- ci.yml: remove unnecessary needs:build for unit-test, use composite action
- e2e-headed.yml: use composite action, increase timeout to 20min
- Add .github/actions/setup-chrome composite action for shared CI steps
2026-03-16 17:33:25 +08:00
AlexYue 257e9ec4f8 feat: add E2E testing infrastructure with real Chrome in CI (#1)
# feat: add E2E testing infrastructure with real Chrome in CI

## What

Establish a comprehensive E2E testing framework for opencli using **real Chrome + xvfb virtual display** in GitHub Actions CI.

## Changes

### E2E Test Suite (~52 test cases)
- `public-commands.test.ts` — Public API commands (hackernews, v2ex)
- `browser-public.test.ts` — Browser commands for public data across all 18 sites (21 tests)
- `browser-auth.test.ts` — Graceful failure verification for login-required commands (14 tests)
- `management.test.ts` — Full coverage of management commands (list/validate/verify/version/help)
- `output-formats.test.ts` — Output format validation (json/yaml/csv/md)
- `smoke/api-health.test.ts` — Scheduled API health checks

### Auto-detect Browser Mode
- `buildMcpArgs` automatically selects mode based on `PLAYWRIGHT_MCP_EXTENSION_TOKEN`:
  - Token present → `--extension` (local user, connects to logged-in Chrome)
  - Token absent → standalone (CI launches its own browser)
- No extra environment variables needed

### CI Pipeline
- `e2e-headed.yml` — Real Chrome via `browser-actions/setup-chrome` + `xvfb-run` in headed mode
- `ci.yml` — build + unit-test (2 shards) + smoke-test (scheduled/manual)
- Browser commands that return empty data due to geo-blocking or bot detection gracefully warn + pass without blocking CI

### Documentation
- New `TESTING.md` — Architecture, coverage, local setup, how to add tests, CI explanation
- Updated `README.md` — Added Testing section with quick-start commands
2026-03-16 16:22:00 +08:00
jackwener 6f629260e7 feat: add interactive 'opencli setup' command with TUI checkbox
New zero-dependency TUI checkbox component (src/tui.ts) with:
  - ↑↓/jk navigation, Space toggle, Tab toggle+move
  - 'a' toggle all, Enter confirm, q/Esc cancel

New 'opencli setup' command (src/setup.ts) that:
  - Auto-discovers token from Chrome LevelDB extension
  - Shows interactive multi-select for config files
  - Displays tool names (Codex, Cursor, Claude Code, etc.)
  - Color-coded status (green=ok, yellow=mismatch, red=missing)
  - Applies changes only to selected files
2026-03-16 13:53:03 +08:00
jackwener 1e0b83bb84 feat(doctor): add Antigravity and Gemini CLI config paths
Add ~/.gemini/settings.json (Gemini CLI) and
~/.gemini/antigravity/mcp_config.json (Antigravity) to the
default MCP config scan list.
2026-03-16 13:45:02 +08:00
jackwener 1ff184e24d feat(doctor): add Claude Code and project .mcp.json config paths
Add ~/.claude.json (Claude Code user-scoped MCP config) and
.mcp.json (Claude Code project-scoped MCP config) to the
default config scan list.
2026-03-16 13:43:03 +08:00
jackwener 8e84145ccc feat(doctor): auto-discover extension token from Chrome LevelDB
Scan Chrome/Edge/Chromium localStorage LevelDB files to extract the
Playwright MCP Bridge auth-token directly from the extension's storage.
Uses a fast 'strings | grep' shell pipeline (~200ms) on macOS/Linux
with a pure-Node fallback for Windows.

The discovered token is now shown in 'opencli doctor' output and takes
priority as the recommended token when using '--fix'.
2026-03-16 13:38:48 +08:00
jackwener c77c8a8e3a feat: add Coupang search and add-to-cart adapters
Add browser-backed Coupang adapters for search and add-to-cart workflows.
- coupang search: multi-strategy data collection (API/JSON-LD/bootstrap/DOM), structured fields (price, rating, rocket, delivery), pagination and rocket filter support
- coupang add-to-cart: logged-in browser session reuse, stops before checkout
- coupang.ts: comprehensive data normalization layer with badge/rocket/delivery mapping
- browser-tab.ts: withTemporaryTab utility for isolated tab operations
- Unit tests for core normalization functions

Co-authored-by: CodeBBakGoSu <127713112+CodeBBakGoSu@users.noreply.github.com>
2026-03-16 13:30:42 +08:00
jackwener 9e024d4e46 0.5.2
Release / release (push) Has been cancelled
2026-03-16 13:22:18 +08:00
jackwener 34a9bff2b3 refactor: eliminate code duplication, improve type safety, add tests
- NEW: src/interceptor.ts — unified XHR/Fetch interceptor (was duplicated 3x)
- NEW: src/version.ts — centralized PKG_VERSION (was duplicated 2x)
- NEW: src/constants.ts — shared VOLATILE_PARAMS, FIELD_ROLES etc.
- NEW: src/engine.test.ts, src/registry.test.ts — 14 new unit tests

- browser.ts: use shared normalizeEval, interceptor, withTimeoutMs, PKG_VERSION
- intercept.ts, tap.ts: use shared interceptor generators
- cascade.ts: extract shared buildFetchProbeJs (90% dedup)
- engine.ts: use InternalCliCommand (no more 'as any' casts)
- executor.ts: remove all 15 'as StepHandler' type casts
- runtime.ts: add withTimeoutMs, IBrowserFactory interface
- registry.ts: add InternalCliCommand type for internal fields
- validate.ts: add pipeline step name validation
- output.ts: remove 'null as any' and '.filter(() => true)' hacks
- explore.ts, synthesize.ts: use shared constants
- docs: fix V2EX commands (3→6), SKILL.md version, verify example

Tests: 88 passed (was 74), tsc --noEmit: 0 errors
2026-03-16 13:22:12 +08:00
jackwener b0c51ddf19 fix: use Playwright MCP --executable-path flag (kebab-case)
Extract buildMcpArgs() helper and fix --executablePath → --executable-path
to match the Playwright MCP CLI's expected flag format.

Closes #16

Co-authored-by: KasumiChen <KasumiChen@users.noreply.github.com>
2026-03-16 12:58:18 +08:00
jackwener 141c2cf7d5 0.5.1
Release / release (push) Has been cancelled
2026-03-16 12:54:45 +08:00
jackwener 7ee0c9c96b fix: tab cleanup regex to match actual Playwright MCP tab format
The tab list from Playwright MCP uses '- N: (current) [title](url)' format,
but extractTabEntries only matched 'Tab N ...' format. This caused
_initialTabIdentities to always be empty, so tabs were never cleaned up.

Now supports both formats.
2026-03-16 12:54:44 +08:00
jackwener 76eefed83d docs: add opencli doctor hint after token setup 2026-03-16 12:47:59 +08:00
jackwener 08ae9c3fed chore: sync SKILL.md version to 0.5.0 2026-03-16 12:46:30 +08:00
jackwener 5a1581e67a 0.5.0
Release / release (push) Has been cancelled
2026-03-16 12:46:16 +08:00
jackwener 7a24e8d74b fix: add engines field for Node badge, sync SKILL.md version, fix zh-CN intro 2026-03-16 12:46:15 +08:00
jackwener 95fb841fc7 refactor: remove all CDP code, keep extension-only mode
- Remove CDP auto-discovery (discoverChromeEndpoint, isCdpApiAvailable, isPortReachable)
- Remove CDP readiness probe and autoAllowCdpDialog
- Remove forceExtension plumbing from registry, runtime, main, and adapters
- Remove CDP diagnostics from doctor (extractTokenViaCdp, remoteDebugging fields)
- Simplify connect() to always use --extension mode
- Remove CDP env vars (OPENCLI_USE_CDP, OPENCLI_CDP_ENDPOINT, OPENCLI_FORCE_EXTENSION)
- Update README.md, README.zh-CN.md, SKILL.md to remove CDP sections
- Update tests to remove CDP-related assertions

Current state preserved in extension-and-cdp branch.
2026-03-16 12:44:09 +08:00
49 changed files with 3231 additions and 675 deletions
+26
View File
@@ -0,0 +1,26 @@
name: Setup Chrome + xvfb
description: Install real Chrome and xvfb virtual display for headed browser testing
outputs:
chrome-path:
description: Path to the installed Chrome binary
value: ${{ steps.setup-chrome.outputs.chrome-path }}
runs:
using: composite
steps:
- name: Install real Chrome (stable)
uses: browser-actions/setup-chrome@v1
id: setup-chrome
with:
chrome-version: stable
- name: Verify Chrome installation
shell: bash
run: |
echo "Chrome path: ${{ steps.setup-chrome.outputs.chrome-path }}"
${{ steps.setup-chrome.outputs.chrome-path }} --version
- name: Install xvfb for headed mode
shell: bash
run: sudo apt-get install -y xvfb
+59 -3
View File
@@ -2,12 +2,16 @@ name: CI
on:
push:
branches: [main]
branches: [main, dev]
pull_request:
branches: [main]
branches: [main, dev]
schedule:
- cron: '0 8 * * 1' # Weekly Monday 08:00 UTC — smoke tests
workflow_dispatch:
jobs:
check:
# ── Fast gate: typecheck + build ──
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -15,6 +19,7 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
@@ -24,3 +29,54 @@ jobs:
- name: Build
run: npm run build
# ── Unit tests (vitest shard) ──
unit-test:
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run unit tests (shard ${{ matrix.shard }}/2)
run: npx vitest run src/ --reporter=verbose --shard=${{ matrix.shard }}/2
# ── Smoke tests (scheduled / manual only) ──
smoke-test:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Setup Chrome + xvfb
uses: ./.github/actions/setup-chrome
id: setup-chrome
- name: Build
run: npm run build
- name: Run smoke tests
run: |
xvfb-run --auto-servernum --server-args="-screen 0 1280x720x24" \
npx vitest run tests/smoke/ --reporter=verbose
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
timeout-minutes: 15
+37
View File
@@ -0,0 +1,37 @@
name: E2E Headed Chrome
on:
push:
branches: [main, dev]
pull_request:
branches: [main, dev]
workflow_dispatch:
jobs:
e2e-headed:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Setup Chrome + xvfb
uses: ./.github/actions/setup-chrome
id: setup-chrome
- name: Build
run: npm run build
- name: Run E2E tests (headed Chrome + xvfb)
run: |
xvfb-run --auto-servernum --server-args="-screen 0 1280x720x24" \
npx vitest run tests/e2e/ --reporter=verbose
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
+1
View File
@@ -2,3 +2,4 @@ node_modules/
dist/
*.tsbuildinfo
.opencli/
.mcp.json
+28 -14
View File
@@ -9,7 +9,7 @@
[![Node.js Version](https://img.shields.io/node/v/@jackwener/opencli?style=flat-square)](https://nodejs.org)
[![License](https://img.shields.io/npm/l/@jackwener/opencli?style=flat-square)](./LICENSE)
A CLI tool that turns **any website** into a command-line interface. **57 commands** across **17 sites** — bilibili, zhihu, xiaohongshu, twitter, reddit, xueqiu, github, v2ex, hackernews, bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube — powered by browser session reuse and AI-native discovery.
A CLI tool that turns **any website** into a command-line interface. **59 commands** across **18 sites** — bilibili, zhihu, xiaohongshu, twitter, reddit, xueqiu, github, v2ex, hackernews, bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube, coupang — powered by browser session reuse and AI-native discovery.
---
@@ -21,6 +21,7 @@ A CLI tool that turns **any website** into a command-line interface. **57 comman
- [Built-in Commands](#built-in-commands)
- [Output Formats](#output-formats)
- [For AI Agents (Developer Guide)](#for-ai-agents-developer-guide)
- [Testing](#testing)
- [Troubleshooting](#troubleshooting)
- [Releasing New Versions](#releasing-new-versions)
- [License](#license)
@@ -41,9 +42,9 @@ A CLI tool that turns **any website** into a command-line interface. **57 comman
> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands. If you get empty data or errors, check your login status first.
OpenCLI needs a way to communicate with your browser. We highly recommend configuring **both** of the following methods for maximum reliability.
OpenCLI connects to your browser through the Playwright MCP Bridge extension.
### Connection Method A: Playwright MCP Bridge Extension (Primary)
### Playwright MCP Bridge Extension Setup
1. Install **[Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm)** extension in Chrome.
2. Obtain your token by clicking the extension icon in the browser toolbar or from the extension settings page.
@@ -72,15 +73,11 @@ And, so that `opencli` commands can use it directly in the terminal, export it i
export PLAYWRIGHT_MCP_EXTENSION_TOKEN="<your-token-here>"
```
### Connection Method B: Chrome 144+ Auto-Discovery (Fallback)
After configuring, run `opencli doctor` to verify your token is correctly set up across all locations:
No extensions needed. Just enable Chrome's built-in remote debugging:
1. Open `chrome://inspect#remote-debugging` in Chrome
2. Check **"Allow remote debugging for this browser instance"**
3. Set `OPENCLI_USE_CDP=1` before running opencli
*You can also manually specify an endpoint via `OPENCLI_CDP_ENDPOINT` env var.*
```bash
opencli doctor
```
## Quick Start
@@ -130,13 +127,14 @@ npm install -g @jackwener/opencli@latest
| **reddit** | `hot` `frontpage` `search` `subreddit` | 🔐 Browser |
| **weibo** | `hot` | 🔐 Browser |
| **boss** | `search` | 🔐 Browser |
| **coupang** | `search` `add-to-cart` | 🔐 Browser |
| **youtube** | `search` | 🔐 Browser |
| **yahoo-finance** | `quote` | 🔐 Browser |
| **reuters** | `search` | 🔐 Browser |
| **smzdm** | `search` | 🔐 Browser |
| **ctrip** | `search` | 🔐 Browser |
| **github** | `search` | 🌐 Public |
| **v2ex** | `hot` `latest` `topic` | 🌐 Public |
| **v2ex** | `hot` `latest` `topic` `daily` `me` `notifications` | 🌐 Public / 🔐 Browser |
| **hackernews** | `top` | 🌐 Public |
| **bbc** | `news` | 🌐 Public |
@@ -179,13 +177,29 @@ opencli cascade https://api.example.com/data
Explore outputs to `.opencli/explore/<site>/` (manifest.json, endpoints.json, capabilities.json, auth.json).
## Testing
See **[TESTING.md](./TESTING.md)** for the full testing guide, including:
- Current test coverage (unit + ~52 E2E tests across all 18 sites)
- How to run tests locally
- How to add tests when creating new adapters
- CI/CD pipeline with sharding
- Headless browser mode (`OPENCLI_HEADLESS=1`)
```bash
# Quick start
npm run build
npx vitest run # All tests
npx vitest run src/ # Unit tests only
npx vitest run tests/e2e/ # E2E tests
```
## Troubleshooting
- **"Failed to connect to Playwright MCP Bridge"**
- Ensure the Playwright MCP extension is installed and **enabled** in your running Chrome.
- Restart the Chrome browser if you just installed the extension.
- **"CDP command failed" or "boss search blocked"**
- Some sites (like BOSS Zhipin) actively block Chrome DevTools Protocol connections. OpenCLI falls back to cookie extraction, but ensure you didn't force `--chrome-mode` unnecessarily.
- **Empty data returns or 'Unauthorized' error**
- Your login session in Chrome might have expired. Open a normal Chrome tab, navigate to the target site, and log in or refresh the page to prove you are human.
- **Node API errors**
+10 -15
View File
@@ -9,7 +9,7 @@
[![Node.js Version](https://img.shields.io/node/v/@jackwener/opencli?style=flat-square)](https://nodejs.org)
[![License](https://img.shields.io/npm/l/@jackwener/opencli?style=flat-square)](./LICENSE)
OpenCLI 通过 Chrome 浏览器 + [Playwright MCP Bridge](https://github.com/nichochar/playwright-mcp) 扩展,将任何网站变成命令行工具。57个内置命令。不存密码、不泄 token,直接复用浏览器登录态。
OpenCLI 将任何网站变成命令行工具。**59 个命令**覆盖 **18 个站点** — B站、知乎、小红书、Twitter、Reddit、雪球、GitHub、V2EX、Hacker News、BBC、微博、BOSS直聘、Yahoo Finance、路透社、什么值得买、携程、YouTube、Coupang — 复用浏览器登录态AI 驱动探索
---
@@ -29,7 +29,7 @@ OpenCLI 通过 Chrome 浏览器 + [Playwright MCP Bridge](https://github.com/nic
## 亮点
- **57 个命令,17 个站点** — B站、知乎、小红书、Twitter、Reddit、雪球(xueqiu)、GitHub、V2EX、Hacker News、BBC、微博、BOSS直聘、Yahoo Finance、路透社、什么值得买、携程、YouTube
- **59 个命令,18 个站点** — B站、知乎、小红书、Twitter、Reddit、雪球(xueqiu)、GitHub、V2EX、Hacker News、BBC、微博、BOSS直聘、Yahoo Finance、路透社、什么值得买、携程、YouTube、Coupang
- **零风控** — 复用 Chrome 登录态,无需存储任何凭证
- **AI 原生** — `explore` 自动发现 API`synthesize` 生成适配器,`cascade` 探测认证策略
- **动态加载引擎** — 声明式的 `.yaml` 或者底层定制的 `.ts` 适配器,放入 `clis/` 文件夹即可自动注册生效
@@ -41,9 +41,9 @@ OpenCLI 通过 Chrome 浏览器 + [Playwright MCP Bridge](https://github.com/nic
> **⚠️ 重要**:大多数命令复用你的 Chrome 登录状态。运行命令前,你必须已在 Chrome 中打开目标网站并完成登录。如果获取到空数据或报错,请先检查你的浏览器登录状态。
为了让 OpenCLI 能够联通你的浏览器,你需要配置连接方式。**强烈建议以下两种方式都配置上**,互为后备:
OpenCLI 通过 Playwright MCP Bridge 扩展与你的浏览器通信。
### 连接方式 APlaywright MCP Bridge 扩展(首选)
### Playwright MCP Bridge 扩展配置
1. 安装 **[Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm)** 扩展
2. 在浏览器插件栏点击该插件,或者在插件设置页获取你的 Extension Token。
@@ -72,15 +72,11 @@ OpenCLI 通过 Chrome 浏览器 + [Playwright MCP Bridge](https://github.com/nic
export PLAYWRIGHT_MCP_EXTENSION_TOKEN="<你的-token>"
```
### 连接方式 BChrome 144+ CDP 自动发现(备选)
配置完成后,运行 `opencli doctor` 检测你的 Token 是否在所有位置都正确配置:
无需安装任何扩展。只需开启 Chrome 内置的远程调试:
1. 在 Chrome 中打开 `chrome://inspect#remote-debugging`
2. 勾选 **"允许对此浏览器实例进行远程调试" (Allow remote debugging for this browser instance)**
3. 运行时设置环境变量 `OPENCLI_USE_CDP=1`
*也可通过 `OPENCLI_CDP_ENDPOINT` 环境变量手动指定 CDP endpoint 地址。*
```bash
opencli doctor
```
## 快速开始
@@ -130,13 +126,14 @@ npm install -g @jackwener/opencli@latest
| **reddit** | `hot` `frontpage` `search` `subreddit` | 🔐 浏览器 |
| **weibo** | `hot` | 🔐 浏览器 |
| **boss** | `search` | 🔐 浏览器 |
| **coupang** | `search` `add-to-cart` | 🔐 浏览器 |
| **youtube** | `search` | 🔐 浏览器 |
| **yahoo-finance** | `quote` | 🔐 浏览器 |
| **reuters** | `search` | 🔐 浏览器 |
| **smzdm** | `search` | 🔐 浏览器 |
| **ctrip** | `search` | 🔐 浏览器 |
| **github** | `search` | 🌐 公共 API |
| **v2ex** | `hot` `latest` `topic` | 🌐 公共 API |
| **v2ex** | `hot` `latest` `topic` `daily` `me` `notifications` | 🌐 公共 API / 🔐 浏览器 |
| **hackernews** | `top` | 🌐 公共 API |
| **bbc** | `news` | 🌐 公共 API |
@@ -184,8 +181,6 @@ opencli cascade https://api.example.com/data
- **"Failed to connect to Playwright MCP Bridge"** 报错
- 确保你当前的 Chrome 已安装且**开启了** Playwright MCP Bridge 浏览器插件。
- 如果是刚装完插件,需要重启 Chrome 浏览器。
- **"CDP command failed" / "被风控拦截"**
- 有些网站(例如 BOSS 直聘)会因为开了 DevTools 或者 CDP 端口拦截验证。OpenCLI 有 cookie 降级机制,通常不需要干预,不用去强行加上 CDP 标识参数即可。
- **返回空数据,或者报错 "Unauthorized"**
- Chrome 里的登录态可能已经过期(甚至被要求过滑动验证码)。请打开当前 Chrome 页面,在新标签页重新手工登录或刷新该页面。
- **Node API 错误 (如 parseArgs, fs 等)**
+9 -10
View File
@@ -1,7 +1,7 @@
---
name: opencli
description: "OpenCLI — Make any website your CLI. Zero risk, AI-powered, reuse Chrome login."
version: 0.4.0
version: 0.5.1
author: jackwener
tags: [cli, browser, web, mcp, playwright, bilibili, zhihu, twitter, github, v2ex, hackernews, reddit, xiaohongshu, xueqiu, AI, agent]
---
@@ -34,8 +34,7 @@ npm update -g @jackwener/opencli
Browser commands require:
1. Chrome browser running **(logged into target sites)**
2. [Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm) extension (default connection mode)
3. **Alternative**: Chrome 144+ CDP auto-discovery — set `OPENCLI_USE_CDP=1` (no extension needed)
2. [Playwright MCP Bridge](https://chromewebstore.google.com/detail/playwright-mcp-bridge/mmlmfjhmonkocbjadbfplnigmagldckm) extension installed and configured
> **Note**: You must be logged into the target website in Chrome before running commands. Tabs opened during command execution are auto-closed afterwards.
@@ -96,10 +95,13 @@ opencli reddit frontpage --limit 10 # 首页
opencli reddit search --keyword "AI" # 搜索
opencli reddit subreddit --name rust # 子版块浏览
# V2EX (public)
# V2EX (public + browser)
opencli v2ex hot --limit 10 # 热门话题
opencli v2ex latest --limit 10 # 最新话题
opencli v2ex topic --id 1024 # 主题详情
opencli v2ex daily # 每日签到 (browser)
opencli v2ex me # 我的信息 (browser)
opencli v2ex notifications --limit 10 # 通知 (browser)
# Hacker News (public)
opencli hackernews top --limit 10 # Top stories
@@ -157,8 +159,8 @@ opencli cascade <api-url>
# Explore with interactive fuzzing (click buttons to trigger lazy APIs)
opencli explore <url> --auto --click "字幕,CC,评论"
# Verify: smoke-test a generated adapter
opencli verify <site/name> --smoke
# Verify: validate adapter definitions
opencli verify
```
## Output Formats
@@ -339,9 +341,6 @@ ${{ index + 1 }}
| `OPENCLI_BROWSER_CONNECT_TIMEOUT` | 30 | Browser connection timeout (sec) |
| `OPENCLI_BROWSER_COMMAND_TIMEOUT` | 45 | Command execution timeout (sec) |
| `OPENCLI_BROWSER_EXPLORE_TIMEOUT` | 120 | Explore timeout (sec) |
| `OPENCLI_CDP_ENDPOINT` | — | Manual CDP WebSocket endpoint (overrides auto-discovery) |
| `OPENCLI_USE_CDP` | — | Set to `1` to use Chrome 144+ CDP auto-discovery instead of extension |
| `OPENCLI_FORCE_EXTENSION` | — | Set to `1` to skip CDP and force extension mode |
| `PLAYWRIGHT_MCP_EXTENSION_TOKEN` | — | Auto-approve extension connection |
## Troubleshooting
@@ -349,6 +348,6 @@ ${{ index + 1 }}
| Issue | Solution |
|-------|----------|
| `npx not found` | Install Node.js: `brew install node` |
| `Timed out connecting to browser` | 1) Chrome must be open 2) Enable remote debugging at `chrome://inspect#remote-debugging` or install MCP Bridge extension |
| `Timed out connecting to browser` | 1) Chrome must be open 2) Install MCP Bridge extension and configure token |
| `Target page context` error | Add `navigate:` step before `evaluate:` in YAML |
| Empty table data | Check if evaluate returns JSON string (MCP parsing) or data path is wrong |
+233
View File
@@ -0,0 +1,233 @@
# Testing Guide
> 面向开发者和 AI Agent 的测试参考手册。
## 目录
- [测试架构](#测试架构)
- [当前覆盖范围](#当前覆盖范围)
- [本地运行测试](#本地运行测试)
- [如何添加新测试](#如何添加新测试)
- [CI/CD 流水线](#cicd-流水线)
- [浏览器模式](#浏览器模式)
- [站点兼容性](#站点兼容性)
---
## 测试架构
测试分为三层,全部使用 **vitest** 运行:
```
tests/
├── e2e/ # E2E 集成测试(子进程运行真实 CLI)
│ ├── helpers.ts # runCli() 共享工具
│ ├── public-commands.test.ts # 公开 API 命令(无需浏览器)
│ ├── browser-public.test.ts # 浏览器命令(公开数据)
│ ├── browser-auth.test.ts # 需登录命令(graceful failure 测试)
│ ├── management.test.ts # 管理命令(list, validate, verify, help
│ └── output-formats.test.ts # 输出格式(json/yaml/csv/md
├── smoke/ # 烟雾测试(仅定时 / 手动触发)
│ └── api-health.test.ts # 外部 API 可用性检测
src/
├── *.test.ts # 单元测试(已有 8 个)
```
| 层 | 位置 | 运行方式 | 用途 |
|---|---|---|---|
| 单元测试 | `src/**/*.test.ts` | `npx vitest run src/` | 内部模块逻辑 |
| E2E 测试 | `tests/e2e/*.test.ts` | `npx vitest run tests/e2e/` | 真实 CLI 命令执行 |
| 烟雾测试 | `tests/smoke/*.test.ts` | `npx vitest run tests/smoke/` | 外部 API 健康 |
---
## 当前覆盖范围
### 单元测试(8 个文件)
| 文件 | 覆盖内容 |
|---|---|
| `browser.test.ts` | JSON-RPC、tab 管理、extension/standalone 模式切换 |
| `engine.test.ts` | 命令发现与执行 |
| `registry.test.ts` | 命令注册与策略分配 |
| `output.test.ts` | 输出格式渲染 |
| `doctor.test.ts` | Token 诊断 |
| `coupang.test.ts` | 数据归一化 |
| `pipeline/template.test.ts` | 模板表达式求值 |
| `pipeline/transform.test.ts` | 数据变换步骤 |
### E2E 测试(~52 个用例)
| 文件 | 覆盖站点/功能 | 测试数 |
|---|---|---|
| `public-commands.test.ts` | hackernews/top, v2ex/hot, v2ex/latest, v2ex/topic | 5 |
| `browser-public.test.ts` | bbc, bilibili×3, weibo, zhihu×2, reddit×2, twitter, xueqiu×2, reuters, youtube, smzdm, boss, ctrip, coupang, xiaohongshu, yahoo-finance, v2ex/daily | 21 |
| `browser-auth.test.ts` | bilibili/me,dynamic,favorite,history,following + twitter/bookmarks,timeline,notifications + v2ex/me,notifications + xueqiu/feed,watchlist + xiaohongshu/feed,notifications | 14 |
| `management.test.ts` | list×5 格式, validate×3 级别, verify, --version, --help, unknown cmd | 12 |
| `output-formats.test.ts` | json, yaml, csv, md 格式验证 | 5 |
### 烟雾测试
公开 API 可用性(hackernews, v2ex×2, v2ex/topic+ 全站点注册完整性检查。
---
## 本地运行测试
### 前置条件
```bash
npm ci # 安装依赖
npm run build # 编译(E2E 测试需要 dist/main.js
```
### 运行命令
```bash
# 全部单元测试
npx vitest run src/
# 全部 E2E 测试(会真实调用外部 API)
npx vitest run tests/e2e/
# 单个测试文件
npx vitest run tests/e2e/management.test.ts
# 全部测试(单元 + E2E
npx vitest run
# 烟雾测试
npx vitest run tests/smoke/
# watch 模式(开发时推荐)
npx vitest src/
```
### 浏览器命令本地测试须知
-`PLAYWRIGHT_MCP_EXTENSION_TOKEN` 时,opencli 自动启动一个独立浏览器实例
- `browser-public.test.ts` 使用 `tryBrowserCommand()`,站点反爬导致空数据时 warn + pass
- `browser-auth.test.ts` 验证 **graceful failure**(不 crash 不 hang 即通过)
- 如需测试完整登录态,保持 Chrome 登录态 + 设置 `PLAYWRIGHT_MCP_EXTENSION_TOKEN`,手动跑对应测试
---
## 如何添加新测试
### 新增 YAML Adapter(如 `src/clis/producthunt/trending.yaml`
1. **无需额外操作**`validate` 测试会自动覆盖 YAML 结构验证
2. 根据 adapter 类型,在对应文件加一个 `it()` block
```typescript
// 如果 browser: false(公开 API)→ tests/e2e/public-commands.test.ts
it('producthunt trending returns data', async () => {
const { stdout, code } = await runCli(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
```
```typescript
// 如果 browser: true 但可公开访问 → tests/e2e/browser-public.test.ts
it('producthunt trending returns data', async () => {
const data = await tryBrowserCommand(['producthunt', 'trending', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'producthunt trending');
}, 60_000);
```
```typescript
// 如果 browser: true 且需登录 → tests/e2e/browser-auth.test.ts
it('producthunt me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['producthunt', 'me', '-f', 'json'], 'producthunt me');
}, 60_000);
```
### 新增管理命令(如 `opencli export`
`tests/e2e/management.test.ts` 添加测试。
### 新增内部模块
`src/` 下对应位置创建 `*.test.ts`
### 决策流程图
```
新增功能 → 是内部模块? → 是 → src/ 下加 *.test.ts
↓ 否
是 CLI 命令? → browser: false? → tests/e2e/public-commands.test.ts
↓ true
公开数据? → tests/e2e/browser-public.test.ts
↓ 需登录
tests/e2e/browser-auth.test.ts
```
---
## CI/CD 流水线
### ci.yml(主流水线)
| Job | 触发条件 | 内容 |
|---|---|---|
| **build** | push/PR to main,dev | typecheck + build |
| **unit-test** | push/PR to main,dev | 单元测试,2 shard 并行 |
| **smoke-test** | 每周一 08:00 UTC / 手动 | xvfb + real Chrome,外部 API 健康检查 |
### e2e-headed.ymlE2E 测试)
| Job | 触发条件 | 内容 |
|---|---|---|
| **e2e-headed** | push/PR to main,dev | xvfb + real Chrome,全部 E2E 测试 |
E2E 使用 `browser-actions/setup-chrome` 安装真实 Chrome,配合 `xvfb-run` 提供虚拟显示器,以 headed 模式运行浏览器。
### Sharding
单元测试使用 vitest 内置 shard
```yaml
strategy:
matrix:
shard: [1, 2]
steps:
- run: npx vitest run src/ --shard=${{ matrix.shard }}/2
```
---
## 浏览器模式
opencli 根据 `PLAYWRIGHT_MCP_EXTENSION_TOKEN` 环境变量自动选择模式:
| 条件 | 模式 | MCP 参数 | 使用场景 |
|---|---|---|---|
| Token 已设置 | Extension 模式 | `--extension` | 本地用户,连接已登录的 Chrome |
| Token 未设置 | Standalone 模式 | (无特殊 flag) | CI 或无扩展环境,自启浏览器 |
CI 中使用 `OPENCLI_BROWSER_EXECUTABLE_PATH` 指定真实 Chrome 路径:
```yaml
env:
OPENCLI_BROWSER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
```
---
## 站点兼容性
在 GitHub Actions 美国 runner 上,部分站点因地域限制或登录要求返回空数据。E2E 测试对这些站点使用 warn + pass 策略,不影响 CI 绿灯。
| 站点 | CI 状态 | 限制原因 |
|---|---|---|
| hackernews, bbc, v2ex | ✅ 返回数据 | 无限制 |
| yahoo-finance | ✅ 返回数据 | 无限制 |
| bilibili, zhihu, weibo, xiaohongshu | ⚠️ 空数据 | 地域限制(中国站点) |
| reddit, twitter, youtube | ⚠️ 空数据 | 需登录或 cookie |
| smzdm, boss, ctrip, coupang, xueqiu | ⚠️ 空数据 | 地域限制 / 需登录 |
> 使用 self-hosted runner(国内服务器)可解决地域限制问题。
+5 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@jackwener/opencli",
"version": "0.4.6",
"version": "0.5.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@jackwener/opencli",
"version": "0.4.6",
"version": "0.5.2",
"license": "BSD-3-Clause",
"dependencies": {
"chalk": "^5.3.0",
@@ -24,6 +24,9 @@
"tsx": "^4.19.3",
"typescript": "^5.8.2",
"vitest": "^4.1.0"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@colors/colors": {
+4 -1
View File
@@ -1,10 +1,13 @@
{
"name": "@jackwener/opencli",
"version": "0.4.6",
"version": "0.5.2",
"publishConfig": {
"access": "public"
},
"description": "Make any website your CLI. AI-powered.",
"engines": {
"node": ">=18.0.0"
},
"type": "module",
"main": "dist/main.js",
"bin": {
+69 -9
View File
@@ -20,6 +20,17 @@ describe('browser helpers', () => {
]);
});
it('extracts tab entries from MCP markdown format', () => {
const entries = __test__.extractTabEntries(
'- 0: (current) [Playwright MCP extension](chrome-extension://abc/connect.html)\n- 1: [知乎 - 首页](https://www.zhihu.com/)'
);
expect(entries).toEqual([
{ index: 0, identity: '(current) [Playwright MCP extension](chrome-extension://abc/connect.html)' },
{ index: 1, identity: '[知乎 - 首页](https://www.zhihu.com/)' },
]);
});
it('closes only tabs that were opened during the session', () => {
const tabsToClose = __test__.diffTabIndexes(
['https://example.com', 'Chrome Extension'],
@@ -38,8 +49,65 @@ describe('browser helpers', () => {
expect(__test__.appendLimited('12345', '67890', 8)).toBe('34567890');
});
it('builds extension MCP args in local mode (no CI)', () => {
const savedCI = process.env.CI;
delete process.env.CI;
try {
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
executablePath: '/mnt/c/Program Files/Google/Chrome/Application/chrome.exe',
})).toEqual([
'/tmp/cli.js',
'--extension',
'--executable-path',
'/mnt/c/Program Files/Google/Chrome/Application/chrome.exe',
]);
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
})).toEqual([
'/tmp/cli.js',
'--extension',
]);
} finally {
if (savedCI !== undefined) {
process.env.CI = savedCI;
} else {
delete process.env.CI;
}
}
});
it('builds standalone MCP args in CI mode', () => {
const savedCI = process.env.CI;
process.env.CI = 'true';
try {
// CI mode: no --extension — browser launches in standalone headed mode
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
})).toEqual([
'/tmp/cli.js',
]);
expect(__test__.buildMcpArgs({
mcpPath: '/tmp/cli.js',
executablePath: '/usr/bin/chromium',
})).toEqual([
'/tmp/cli.js',
'--executable-path',
'/usr/bin/chromium',
]);
} finally {
if (savedCI !== undefined) {
process.env.CI = savedCI;
} else {
delete process.env.CI;
}
}
});
it('times out slow promises', async () => {
await expect(__test__.withTimeout(new Promise(() => {}), 10, 'timeout')).rejects.toThrow('timeout');
await expect(__test__.withTimeoutMs(new Promise(() => {}), 10, 'timeout')).rejects.toThrow('timeout');
});
});
@@ -75,13 +143,5 @@ describe('PlaywrightMCP state', () => {
await expect(mcp.connect()).rejects.toThrow('Playwright MCP is closing');
});
it('tracks backend mode for lifecycle policy', async () => {
const mcp = new PlaywrightMCP();
expect((mcp as any)._mode).toBeNull();
await mcp.close();
expect((mcp as any)._mode).toBeNull();
});
});
+82 -295
View File
@@ -1,132 +1,31 @@
/**
* Browser interaction via Chrome DevTools Protocol.
* Connects to an existing Chrome browser through CDP auto-discovery or extension bridge.
* Browser interaction via Playwright MCP Bridge extension.
* Connects to an existing Chrome browser through the extension.
*/
import { spawn, execSync, type ChildProcess } from 'node:child_process';
import { createHash } from 'node:crypto';
import * as http from 'node:http';
import * as net from 'node:net';
import { fileURLToPath } from 'node:url';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { formatSnapshot } from './snapshotFormatter.js';
/**
* Chrome 144+ auto-discovery: read DevToolsActivePort file to get CDP endpoint.
*
* Starting with Chrome 144, users can enable remote debugging from
* chrome://inspect#remote-debugging without any command-line flags.
* Chrome writes the active port and browser GUID to a DevToolsActivePort file
* in the user data directory, which we read to construct the WebSocket endpoint.
*
* Priority: OPENCLI_CDP_ENDPOINT env > DevToolsActivePort auto-discovery > --extension fallback
*/
/** Quick TCP port probe to verify Chrome is actually listening */
function isPortReachable(port: number, host = '127.0.0.1', timeoutMs = 800): Promise<boolean> {
return new Promise(resolve => {
const sock = net.createConnection({ port, host });
sock.setTimeout(timeoutMs);
sock.on('connect', () => { sock.destroy(); resolve(true); });
sock.on('error', () => resolve(false));
sock.on('timeout', () => { sock.destroy(); resolve(false); });
});
}
/**
* Verify the CDP HTTP JSON API is functional.
* Chrome's chrome://inspect#remote-debugging mode writes DevToolsActivePort
* but doesn't expose the full CDP HTTP API (/json/version), which means
* Playwright's connectOverCDP won't work properly (init succeeds but
* all tool calls hang silently).
*/
export function isCdpApiAvailable(port: number, host = '127.0.0.1', timeoutMs = 2000): Promise<boolean> {
return new Promise(resolve => {
const req = http.get(`http://${host}:${port}/json/version`, { timeout: timeoutMs }, (res) => {
let body = '';
res.on('data', (chunk: Buffer) => { body += chunk.toString(); });
res.on('end', () => {
try {
const data = JSON.parse(body);
// A valid CDP endpoint returns { Browser, ... } with a webSocketDebuggerUrl
resolve(!!data && typeof data === 'object' && !!data.Browser);
} catch {
resolve(false);
}
});
});
req.on('error', () => resolve(false));
req.on('timeout', () => { req.destroy(); resolve(false); });
});
}
export async function discoverChromeEndpoint(): Promise<string | null> {
const candidates: string[] = [];
// User-specified Chrome data dir takes highest priority
if (process.env.CHROME_USER_DATA_DIR) {
candidates.push(path.join(process.env.CHROME_USER_DATA_DIR, 'DevToolsActivePort'));
}
// Standard Chrome/Edge user data dirs per platform
if (process.platform === 'win32') {
const localAppData = process.env.LOCALAPPDATA ?? path.join(os.homedir(), 'AppData', 'Local');
candidates.push(path.join(localAppData, 'Google', 'Chrome', 'User Data', 'DevToolsActivePort'));
candidates.push(path.join(localAppData, 'Microsoft', 'Edge', 'User Data', 'DevToolsActivePort'));
} else if (process.platform === 'darwin') {
candidates.push(path.join(os.homedir(), 'Library', 'Application Support', 'Google', 'Chrome', 'DevToolsActivePort'));
candidates.push(path.join(os.homedir(), 'Library', 'Application Support', 'Microsoft Edge', 'DevToolsActivePort'));
} else {
candidates.push(path.join(os.homedir(), '.config', 'google-chrome', 'DevToolsActivePort'));
candidates.push(path.join(os.homedir(), '.config', 'chromium', 'DevToolsActivePort'));
candidates.push(path.join(os.homedir(), '.config', 'microsoft-edge', 'DevToolsActivePort'));
}
for (const filePath of candidates) {
try {
const content = fs.readFileSync(filePath, 'utf-8').trim();
const lines = content.split('\n');
if (lines.length >= 2) {
const port = parseInt(lines[0], 10);
const browserPath = lines[1]; // e.g. /devtools/browser/<GUID>
if (port > 0 && browserPath.startsWith('/devtools/browser/')) {
const endpoint = `ws://127.0.0.1:${port}${browserPath}`;
// Verify the port is actually reachable (Chrome may have closed, leaving a stale file)
if (await isPortReachable(port)) {
// Verify CDP HTTP API is functional — chrome://inspect#remote-debugging
// writes DevToolsActivePort but doesn't expose the full CDP API,
// causing Playwright connectOverCDP to hang on all tool calls.
if (await isCdpApiAvailable(port)) {
return endpoint;
}
}
}
}
} catch {}
}
return null;
}
// Read version from package.json (single source of truth)
const __browser_dirname = path.dirname(fileURLToPath(import.meta.url));
const PKG_VERSION = (() => { try { return JSON.parse(fs.readFileSync(path.resolve(__browser_dirname, '..', 'package.json'), 'utf-8')).version; } catch { return '0.0.0'; } })();
import { PKG_VERSION } from './version.js';
import { normalizeEvaluateSource } from './pipeline/template.js';
import { generateInterceptorJs, generateReadInterceptedJs } from './interceptor.js';
import { withTimeoutMs } from './runtime.js';
const CONNECT_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_CONNECT_TIMEOUT ?? '30', 10);
const STDERR_BUFFER_LIMIT = 16 * 1024;
const INITIAL_TABS_TIMEOUT_MS = 1500;
const CDP_READINESS_PROBE_TIMEOUT_MS = 5000;
const TAB_CLEANUP_TIMEOUT_MS = 2000;
let _cachedMcpServerPath: string | null | undefined;
type ConnectFailureKind = 'missing-token' | 'extension-timeout' | 'extension-not-installed' | 'cdp-timeout' | 'mcp-init' | 'process-exit' | 'unknown';
type ConnectFailureKind = 'missing-token' | 'extension-timeout' | 'extension-not-installed' | 'mcp-init' | 'process-exit' | 'unknown';
type PlaywrightMCPState = 'idle' | 'connecting' | 'connected' | 'closing' | 'closed';
type PlaywrightMCPMode = 'extension' | 'cdp' | null;
type ConnectFailureInput = {
kind: ConnectFailureKind;
mode: 'extension' | 'cdp';
timeout: number;
hasExtensionToken: boolean;
tokenFingerprint?: string | null;
@@ -145,41 +44,31 @@ export function formatBrowserConnectError(input: ConnectFailureInput): Error {
const suffix = stderr ? `\n\nMCP stderr:\n${stderr}` : '';
const tokenHint = input.tokenFingerprint ? ` Token fingerprint: ${input.tokenFingerprint}.` : '';
if (input.mode === 'extension') {
if (input.kind === 'missing-token') {
return new Error(
'Failed to connect to Playwright MCP Bridge: PLAYWRIGHT_MCP_EXTENSION_TOKEN is not set.\n\n' +
'Without this token, Chrome will show a manual approval dialog for every new MCP connection. ' +
'Copy the token from the Playwright MCP Bridge extension and set it in BOTH your shell environment and MCP client config.' +
suffix,
);
}
if (input.kind === 'extension-not-installed') {
return new Error(
'Failed to connect to Playwright MCP Bridge: the browser extension did not attach.\n\n' +
'Make sure Chrome is running and the "Playwright MCP Bridge" extension is installed and enabled. ' +
'If Chrome shows an approval dialog, click Allow.' +
suffix,
);
}
if (input.kind === 'extension-timeout') {
const likelyCause = input.hasExtensionToken
? `The most likely cause is that PLAYWRIGHT_MCP_EXTENSION_TOKEN does not match the token currently shown by the browser extension.${tokenHint} Re-copy the token from the extension and update BOTH your shell environment and MCP client config.`
: 'PLAYWRIGHT_MCP_EXTENSION_TOKEN is not configured, so the extension may be waiting for manual approval.';
return new Error(
`Timed out connecting to Playwright MCP Bridge (${input.timeout}s).\n\n` +
`${likelyCause} If a browser prompt is visible, click Allow. You can also switch to Chrome remote debugging mode with OPENCLI_USE_CDP=1 as a fallback.` +
suffix,
);
}
if (input.kind === 'missing-token') {
return new Error(
'Failed to connect to Playwright MCP Bridge: PLAYWRIGHT_MCP_EXTENSION_TOKEN is not set.\n\n' +
'Without this token, Chrome will show a manual approval dialog for every new MCP connection. ' +
'Copy the token from the Playwright MCP Bridge extension and set it in BOTH your shell environment and MCP client config.' +
suffix,
);
}
if (input.mode === 'cdp' && input.kind === 'cdp-timeout') {
if (input.kind === 'extension-not-installed') {
return new Error(
`Timed out connecting to browser via CDP (${input.timeout}s).\n\n` +
'Make sure Chrome is running and remote debugging is enabled at chrome://inspect#remote-debugging, or set OPENCLI_CDP_ENDPOINT explicitly.' +
'Failed to connect to Playwright MCP Bridge: the browser extension did not attach.\n\n' +
'Make sure Chrome is running and the "Playwright MCP Bridge" extension is installed and enabled. ' +
'If Chrome shows an approval dialog, click Allow.' +
suffix,
);
}
if (input.kind === 'extension-timeout') {
const likelyCause = input.hasExtensionToken
? `The most likely cause is that PLAYWRIGHT_MCP_EXTENSION_TOKEN does not match the token currently shown by the browser extension.${tokenHint} Re-copy the token from the extension and update BOTH your shell environment and MCP client config.`
: 'PLAYWRIGHT_MCP_EXTENSION_TOKEN is not configured, so the extension may be waiting for manual approval.';
return new Error(
`Timed out connecting to Playwright MCP Bridge (${input.timeout}s).\n\n` +
`${likelyCause} If a browser prompt is visible, click Allow.` +
suffix,
);
}
@@ -199,7 +88,6 @@ export function formatBrowserConnectError(input: ConnectFailureInput): Error {
}
function inferConnectFailureKind(args: {
mode: 'extension' | 'cdp';
hasExtensionToken: boolean;
stderr: string;
rawMessage?: string;
@@ -207,7 +95,7 @@ function inferConnectFailureKind(args: {
}): ConnectFailureKind {
const haystack = `${args.rawMessage ?? ''}\n${args.stderr}`.toLowerCase();
if (args.mode === 'extension' && !args.hasExtensionToken)
if (!args.hasExtensionToken)
return 'missing-token';
if (haystack.includes('extension connection timeout') || haystack.includes('playwright mcp bridge'))
return 'extension-not-installed';
@@ -215,11 +103,7 @@ function inferConnectFailureKind(args: {
return 'mcp-init';
if (args.exited)
return 'process-exit';
if (args.mode === 'extension')
return 'extension-timeout';
if (args.mode === 'cdp')
return 'cdp-timeout';
return 'unknown';
return 'extension-timeout';
}
// JSON-RPC helpers
@@ -274,23 +158,10 @@ export class Page implements IPage {
async evaluate(js: string): Promise<any> {
// Normalize IIFE format to function format expected by MCP browser_evaluate
const normalized = this.normalizeEval(js);
const normalized = normalizeEvaluateSource(js);
return this.call('tools/call', { name: 'browser_evaluate', arguments: { function: normalized } });
}
private normalizeEval(source: string): string {
const s = source.trim();
if (!s) return '() => undefined';
// IIFE: (async () => {...})() → wrap as () => (...)
if (s.startsWith('(') && s.endsWith(')()')) return `() => (${s})`;
// Already a function/arrow
if (/^(async\s+)?\([^)]*\)\s*=>/.test(s)) return s;
if (/^(async\s+)?[A-Za-z_][A-Za-z0-9_]*\s*=>/.test(s)) return s;
if (s.startsWith('function ') || s.startsWith('async function ')) return s;
// Raw expression → wrap
return `() => (${s})`;
}
async snapshot(opts: { interactive?: boolean; compact?: boolean; maxDepth?: number; raw?: boolean } = {}): Promise<any> {
const raw = await this.call('tools/call', { name: 'browser_snapshot', arguments: {} });
if (opts.raw) return raw;
@@ -379,57 +250,15 @@ export class Page implements IPage {
}
async installInterceptor(pattern: string): Promise<void> {
const js = `
() => {
window.__opencli_xhr = window.__opencli_xhr || [];
window.__opencli_patterns = window.__opencli_patterns || [];
if (!window.__opencli_patterns.includes('${pattern}')) {
window.__opencli_patterns.push('${pattern}');
}
if (!window.__patched_xhr) {
const checkMatch = (url) => window.__opencli_patterns.some(p => url.includes(p));
const XHR = XMLHttpRequest.prototype;
const open = XHR.open;
const send = XHR.send;
XHR.open = function(method, url) {
this._url = url;
return open.call(this, method, url, ...Array.prototype.slice.call(arguments, 2));
};
XHR.send = function() {
this.addEventListener('load', function() {
if (checkMatch(this._url)) {
try { window.__opencli_xhr.push({url: this._url, data: JSON.parse(this.responseText)}); } catch(e){}
}
});
return send.apply(this, arguments);
};
const origFetch = window.fetch;
window.fetch = async function(...args) {
let u = typeof args[0] === 'string' ? args[0] : (args[0] && args[0].url) || '';
const res = await origFetch.apply(this, args);
setTimeout(async () => {
try {
if (checkMatch(u)) {
const clone = res.clone();
const j = await clone.json();
window.__opencli_xhr.push({url: u, data: j});
}
} catch(e) {}
}, 0);
return res;
};
window.__patched_xhr = true;
}
}
`;
await this.evaluate(js);
await this.evaluate(generateInterceptorJs(JSON.stringify(pattern), {
arrayName: '__opencli_xhr',
patchGuard: '__opencli_interceptor_patched',
}));
}
async getInterceptedRequests(): Promise<any[]> {
return (await this.evaluate('() => window.__opencli_xhr')) || [];
const result = await this.evaluate(generateReadInterceptedJs('__opencli_xhr'));
return result || [];
}
}
@@ -461,7 +290,6 @@ export class PlaywrightMCP {
private _initialTabIdentities: string[] = [];
private _closingPromise: Promise<void> | null = null;
private _state: PlaywrightMCPState = 'idle';
private _mode: PlaywrightMCPMode = null;
private _page: Page | null = null;
@@ -496,7 +324,6 @@ export class PlaywrightMCP {
this._page = null;
this._proc = null;
this._buffer = '';
this._mode = null;
this._initialTabIdentities = [];
this._rejectPendingRequests(new Error('Playwright MCP connect failed'));
PlaywrightMCP._activeInsts.delete(this);
@@ -505,7 +332,7 @@ export class PlaywrightMCP {
}
}
async connect(opts: { timeout?: number; forceExtension?: boolean } = {}): Promise<Page> {
async connect(opts: { timeout?: number } = {}): Promise<Page> {
if (this._state === 'connected' && this._page) return this._page;
if (this._state === 'connecting') throw new Error('Playwright MCP is already connecting');
if (this._state === 'closing') throw new Error('Playwright MCP is closing');
@@ -519,26 +346,10 @@ export class PlaywrightMCP {
this._state = 'connecting';
const timeout = opts.timeout ?? CONNECT_TIMEOUT;
// Connection priority:
// 1. OPENCLI_CDP_ENDPOINT env var → explicit CDP endpoint
// 2. OPENCLI_USE_CDP=1 → auto-discover via DevToolsActivePort
// 3. Default → --extension mode (Playwright MCP Bridge)
// Some anti-bot sites (e.g. BOSS Zhipin) detect CDP — use forceExtension to bypass.
const forceExt = opts.forceExtension || process.env.OPENCLI_FORCE_EXTENSION === '1';
let cdpEndpoint: string | null = null;
if (!forceExt) {
if (process.env.OPENCLI_CDP_ENDPOINT) {
cdpEndpoint = process.env.OPENCLI_CDP_ENDPOINT;
} else if (process.env.OPENCLI_USE_CDP === '1') {
cdpEndpoint = await discoverChromeEndpoint();
}
}
return new Promise<Page>((resolve, reject) => {
const isDebug = process.env.DEBUG?.includes('opencli:mcp');
const debugLog = (msg: string) => isDebug && console.error(`[opencli:mcp] ${msg}`);
const mode: 'extension' | 'cdp' = cdpEndpoint ? 'cdp' : 'extension';
this._mode = mode;
const useExtension = !!process.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN;
const extensionToken = process.env.PLAYWRIGHT_MCP_EXTENSION_TOKEN;
const tokenFingerprint = getTokenFingerprint(extensionToken);
let stderrBuffer = '';
@@ -552,7 +363,6 @@ export class PlaywrightMCP {
this._resetAfterFailedConnect();
reject(formatBrowserConnectError({
kind,
mode,
timeout,
hasExtensionToken: !!extensionToken,
tokenFingerprint,
@@ -573,26 +383,18 @@ export class PlaywrightMCP {
const timer = setTimeout(() => {
debugLog('Connection timed out');
settleError(inferConnectFailureKind({
mode,
hasExtensionToken: !!extensionToken,
stderr: stderrBuffer,
}));
}, timeout * 1000);
const mcpArgs: string[] = [mcpPath];
if (cdpEndpoint) {
mcpArgs.push('--cdp-endpoint', cdpEndpoint);
} else {
mcpArgs.push('--extension');
}
const mcpArgs = buildMcpArgs({
mcpPath,
executablePath: process.env.OPENCLI_BROWSER_EXECUTABLE_PATH,
});
if (process.env.OPENCLI_VERBOSE) {
console.error(`[opencli] CDP mode: ${cdpEndpoint ? `auto-discovered ${cdpEndpoint}` : 'fallback to --extension'}`);
if (mode === 'extension') {
console.error(`[opencli] Extension token: ${extensionToken ? `configured (fingerprint ${tokenFingerprint})` : 'missing'}`);
}
}
if (process.env.OPENCLI_BROWSER_EXECUTABLE_PATH) {
mcpArgs.push('--executablePath', process.env.OPENCLI_BROWSER_EXECUTABLE_PATH);
console.error(`[opencli] Mode: ${useExtension ? 'extension' : 'standalone'}`);
if (useExtension) console.error(`[opencli] Extension token: fingerprint ${tokenFingerprint}`);
}
debugLog(`Spawning node ${mcpArgs.join(' ')}`);
@@ -645,7 +447,6 @@ export class PlaywrightMCP {
this._rejectPendingRequests(new Error(`Playwright MCP process exited before response${code == null ? '' : ` (code ${code})`}`));
if (!settled) {
settleError(inferConnectFailureKind({
mode,
hasExtensionToken: !!extensionToken,
stderr: stderrBuffer,
exited: true,
@@ -663,7 +464,6 @@ export class PlaywrightMCP {
debugLog('Got initialize response');
if (resp.error) {
settleError(inferConnectFailureKind({
mode,
hasExtensionToken: !!extensionToken,
stderr: stderrBuffer,
rawMessage: `MCP init failed: ${resp.error.message}`,
@@ -675,31 +475,9 @@ export class PlaywrightMCP {
debugLog(`SEND: ${initializedMsg.trim()}`);
this._proc?.stdin?.write(initializedMsg);
if (mode === 'cdp') {
// CDP readiness probe: verify tool calls actually work.
// Some CDP endpoints (e.g. chrome://inspect mode) accept WebSocket
// connections and respond to MCP init but silently drop tool calls.
debugLog('CDP readiness probe (tabs)...');
withTimeout(page.tabs(), CDP_READINESS_PROBE_TIMEOUT_MS, 'CDP readiness probe timed out')
.then(() => {
debugLog('CDP readiness probe succeeded');
settleSuccess(page);
})
.catch((err) => {
debugLog(`CDP readiness probe failed: ${err.message}`);
settleError('cdp-timeout', {
rawMessage: 'CDP endpoint connected but tool calls are unresponsive. ' +
'This usually means Chrome was opened with chrome://inspect#remote-debugging ' +
'which is not fully compatible. Launch Chrome with --remote-debugging-port=9222 instead, ' +
'or use the Playwright MCP Bridge extension (default mode).',
});
});
return;
}
// Extension mode uses tabs as a readiness probe and for tab cleanup bookkeeping.
// Use tabs as a readiness probe and for tab cleanup bookkeeping.
debugLog('Fetching initial tabs count...');
withTimeout(page.tabs(), INITIAL_TABS_TIMEOUT_MS, 'Timed out fetching initial tabs').then((tabs: any) => {
withTimeoutMs(page.tabs(), INITIAL_TABS_TIMEOUT_MS, 'Timed out fetching initial tabs').then((tabs: any) => {
debugLog(`Tabs response: ${typeof tabs === 'string' ? tabs : JSON.stringify(tabs)}`);
this._initialTabIdentities = extractTabIdentities(tabs);
settleSuccess(page);
@@ -714,6 +492,7 @@ export class PlaywrightMCP {
});
}
async close(): Promise<void> {
if (this._closingPromise) return this._closingPromise;
if (this._state === 'closed') return;
@@ -721,9 +500,9 @@ export class PlaywrightMCP {
this._closingPromise = (async () => {
try {
// Extension mode opens bridge/session tabs that we can clean up best-effort.
if (this._mode === 'extension' && this._page && this._proc && !this._proc.killed) {
if (this._page && this._proc && !this._proc.killed) {
try {
const tabs = await withTimeout(this._page.tabs(), TAB_CLEANUP_TIMEOUT_MS, 'Timed out fetching tabs during cleanup');
const tabs = await withTimeoutMs(this._page.tabs(), TAB_CLEANUP_TIMEOUT_MS, 'Timed out fetching tabs during cleanup');
const tabEntries = extractTabEntries(tabs);
const tabsToClose = diffTabIndexes(this._initialTabIdentities, tabEntries);
for (const index of tabsToClose) {
@@ -751,7 +530,6 @@ export class PlaywrightMCP {
this._rejectPendingRequests(new Error('Playwright MCP session closed'));
this._page = null;
this._proc = null;
this._mode = null;
this._state = 'closed';
PlaywrightMCP._activeInsts.delete(this);
}
@@ -779,12 +557,23 @@ function extractTabEntries(raw: any): Array<{ index: number; identity: string }>
.map(line => line.trim())
.filter(Boolean)
.map(line => {
const match = line.match(/Tab\s+(\d+)\s*(.*)$/);
if (!match) return null;
return {
index: parseInt(match[1], 10),
identity: match[2].trim() || `tab-${match[1]}`,
};
// Match actual Playwright MCP format: "- 0: (current) [title](url)" or "- 1: [title](url)"
const mcpMatch = line.match(/^-\s+(\d+):\s*(.*)$/);
if (mcpMatch) {
return {
index: parseInt(mcpMatch[1], 10),
identity: mcpMatch[2].trim() || `tab-${mcpMatch[1]}`,
};
}
// Legacy format: "Tab 0 ..."
const legacyMatch = line.match(/Tab\s+(\d+)\s*(.*)$/);
if (legacyMatch) {
return {
index: parseInt(legacyMatch[1], 10),
identity: legacyMatch[2].trim() || `tab-${legacyMatch[1]}`,
};
}
return null;
})
.filter((entry): entry is { index: number; identity: string } => entry !== null);
}
@@ -822,20 +611,18 @@ function appendLimited(current: string, chunk: string, limit: number): string {
return next.slice(-limit);
}
function withTimeout<T>(promise: Promise<T>, timeoutMs: number, message: string): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error(message)), timeoutMs);
promise.then(
(value) => {
clearTimeout(timer);
resolve(value);
},
(error) => {
clearTimeout(timer);
reject(error);
},
);
});
function buildMcpArgs(input: { mcpPath: string; executablePath?: string | null }): string[] {
const args = [input.mcpPath];
if (!process.env.CI) {
// Local: always connect to user's running Chrome via MCP Bridge extension
args.push('--extension');
}
// CI: standalone mode — @playwright/mcp launches its own browser (headed by default).
// xvfb provides a virtual display for headed mode in GitHub Actions.
if (input.executablePath) {
args.push('--executable-path', input.executablePath);
}
return args;
}
export const __test__ = {
@@ -843,8 +630,8 @@ export const __test__ = {
extractTabEntries,
diffTabIndexes,
appendLimited,
withTimeout,
isCdpApiAvailable,
buildMcpArgs,
withTimeoutMs,
};
function findMcpServerPath(): string | null {
+47 -75
View File
@@ -37,6 +37,49 @@ interface CascadeResult {
confidence: number;
}
/**
* Build the JavaScript source for a fetch probe.
* Shared logic for PUBLIC, COOKIE, and HEADER strategies.
*/
function buildFetchProbeJs(url: string, opts: {
credentials?: boolean;
extractCsrf?: boolean;
}): string {
const credentialsLine = opts.credentials ? `credentials: 'include',` : '';
const headerSetup = opts.extractCsrf
? `
const cookies = document.cookie.split(';').map(c => c.trim());
const csrf = cookies.find(c => c.startsWith('ct0=') || c.startsWith('csrf_token=') || c.startsWith('_csrf='))?.split('=').slice(1).join('=');
const headers = {};
if (csrf) { headers['X-Csrf-Token'] = csrf; headers['X-XSRF-Token'] = csrf; }
`
: 'const headers = {};';
return `
async () => {
try {
${headerSetup}
const resp = await fetch(${JSON.stringify(url)}, {
${credentialsLine}
headers
});
const status = resp.status;
if (!resp.ok) return { status, ok: false };
const text = await resp.text();
let hasData = false;
try {
const json = JSON.parse(text);
hasData = !!json && (Array.isArray(json) ? json.length > 0 :
typeof json === 'object' && Object.keys(json).length > 0);
// Check for API-level error codes (common in Chinese sites)
if (json.code !== undefined && json.code !== 0) hasData = false;
} catch {}
return { status, ok: true, hasData, preview: text.slice(0, 200) };
} catch (e) { return { ok: false, error: e.message }; }
}
`;
}
/**
* Probe an endpoint with a specific strategy.
* Returns whether the probe succeeded and basic response info.
@@ -45,32 +88,14 @@ export async function probeEndpoint(
page: IPage,
url: string,
strategy: Strategy,
opts: { timeout?: number } = {},
_opts: { timeout?: number } = {},
): Promise<ProbeResult> {
const result: ProbeResult = { strategy, success: false };
try {
switch (strategy) {
case Strategy.PUBLIC: {
// Try direct fetch without browser (no credentials)
const js = `
async () => {
try {
const resp = await fetch(${JSON.stringify(url)});
const status = resp.status;
if (!resp.ok) return { status, ok: false };
const text = await resp.text();
let hasData = false;
try {
const json = JSON.parse(text);
hasData = !!json && (Array.isArray(json) ? json.length > 0 :
typeof json === 'object' && Object.keys(json).length > 0);
} catch {}
return { status, ok: true, hasData, preview: text.slice(0, 200) };
} catch (e) { return { ok: false, error: e.message }; }
}
`;
const resp = await page.evaluate(js);
const resp = await page.evaluate(buildFetchProbeJs(url, {}));
result.statusCode = resp?.status;
result.success = resp?.ok && resp?.hasData;
result.hasData = resp?.hasData;
@@ -79,27 +104,7 @@ export async function probeEndpoint(
}
case Strategy.COOKIE: {
// Fetch with credentials: 'include' (uses browser cookies)
const js = `
async () => {
try {
const resp = await fetch(${JSON.stringify(url)}, { credentials: 'include' });
const status = resp.status;
if (!resp.ok) return { status, ok: false };
const text = await resp.text();
let hasData = false;
try {
const json = JSON.parse(text);
hasData = !!json && (Array.isArray(json) ? json.length > 0 :
typeof json === 'object' && Object.keys(json).length > 0);
// Check for API-level error codes (common in Chinese sites)
if (json.code !== undefined && json.code !== 0) hasData = false;
} catch {}
return { status, ok: true, hasData, preview: text.slice(0, 200) };
} catch (e) { return { ok: false, error: e.message }; }
}
`;
const resp = await page.evaluate(js);
const resp = await page.evaluate(buildFetchProbeJs(url, { credentials: true }));
result.statusCode = resp?.status;
result.success = resp?.ok && resp?.hasData;
result.hasData = resp?.hasData;
@@ -108,39 +113,7 @@ export async function probeEndpoint(
}
case Strategy.HEADER: {
// Fetch with credentials + try to extract common auth headers
const js = `
async () => {
try {
// Try to extract CSRF tokens from cookies
const cookies = document.cookie.split(';').map(c => c.trim());
const csrf = cookies.find(c => c.startsWith('ct0=') || c.startsWith('csrf_token=') || c.startsWith('_csrf='))?.split('=').slice(1).join('=');
const headers = {};
if (csrf) {
headers['X-Csrf-Token'] = csrf;
headers['X-XSRF-Token'] = csrf;
}
const resp = await fetch(${JSON.stringify(url)}, {
credentials: 'include',
headers
});
const status = resp.status;
if (!resp.ok) return { status, ok: false };
const text = await resp.text();
let hasData = false;
try {
const json = JSON.parse(text);
hasData = !!json && (Array.isArray(json) ? json.length > 0 :
typeof json === 'object' && Object.keys(json).length > 0);
if (json.code !== undefined && json.code !== 0) hasData = false;
} catch {}
return { status, ok: true, hasData, preview: text.slice(0, 200) };
} catch (e) { return { ok: false, error: e.message }; }
}
`;
const resp = await page.evaluate(js);
const resp = await page.evaluate(buildFetchProbeJs(url, { credentials: true, extractCsrf: true }));
result.statusCode = resp?.status;
result.success = resp?.ok && resp?.hasData;
result.hasData = resp?.hasData;
@@ -151,7 +124,6 @@ export async function probeEndpoint(
case Strategy.INTERCEPT:
case Strategy.UI:
// These require specific implementation per-site
// Mark as needing manual implementation
result.success = false;
result.error = `Strategy ${strategy} requires site-specific implementation`;
break;
+1 -1
View File
@@ -69,7 +69,7 @@ cli({
description: 'BOSS直聘搜索职位',
domain: 'www.zhipin.com',
strategy: Strategy.COOKIE,
forceExtension: true, // BOSS Zhipin detects CDP mode — must use extension bridge
browser: true,
args: [
{ name: 'query', required: true, help: 'Search keyword (e.g. AI agent, 前端)' },
+149
View File
@@ -0,0 +1,149 @@
import { cli, Strategy } from '../../registry.js';
import { canonicalizeProductUrl, normalizeProductId } from '../../coupang.js';
function escapeJsString(value: string): string {
return JSON.stringify(value);
}
function buildAddToCartEvaluate(expectedProductId: string): string {
return `
(async () => {
const expectedProductId = ${escapeJsString(expectedProductId)};
const text = document.body.innerText || '';
const loginHints = {
hasLoginLink: Boolean(document.querySelector('a[href*="login"], a[title*="로그인"]')),
hasMyCoupang: /마이쿠팡/.test(text),
};
const pathMatch = location.pathname.match(/\\/vp\\/products\\/(\\d+)/);
const currentProductId = pathMatch?.[1] || '';
if (expectedProductId && currentProductId && expectedProductId !== currentProductId) {
return { ok: false, reason: 'PRODUCT_MISMATCH', currentProductId, loginHints };
}
const optionSelectors = [
'select',
'[role="listbox"]',
'.prod-option, .product-option, .option-select, .option-dropdown',
];
const hasRequiredOption = optionSelectors.some((selector) => {
try {
const nodes = Array.from(document.querySelectorAll(selector));
return nodes.some((node) => {
const label = (node.textContent || '') + ' ' + (node.getAttribute?.('aria-label') || '');
return /옵션|색상|사이즈|용량|선택/i.test(label);
});
} catch {
return false;
}
});
if (hasRequiredOption) {
return { ok: false, reason: 'OPTION_REQUIRED', currentProductId, loginHints };
}
const clickCandidate = (elements) => {
for (const element of elements) {
if (!(element instanceof HTMLElement)) continue;
const label = ((element.innerText || '') + ' ' + (element.getAttribute('aria-label') || '')).trim();
if (/장바구니|카트|cart/i.test(label) && !/sold out|품절/i.test(label)) {
element.click();
return true;
}
}
return false;
};
const beforeCount = (() => {
const node = document.querySelector('[class*="cart"] .count, #headerCartCount, .cart-count');
const text = node?.textContent || '';
const num = Number(text.replace(/[^\\d]/g, ''));
return Number.isFinite(num) ? num : null;
})();
const buttons = Array.from(document.querySelectorAll('button, a[role="button"], input[type="button"]'));
const clicked = clickCandidate(buttons);
if (!clicked) {
return { ok: false, reason: 'ADD_TO_CART_BUTTON_NOT_FOUND', currentProductId, loginHints };
}
await new Promise((resolve) => setTimeout(resolve, 2500));
const afterText = document.body.innerText || '';
const successMessage = /장바구니에 담|장바구니 담기 완료|added to cart/i.test(afterText);
const afterCount = (() => {
const node = document.querySelector('[class*="cart"] .count, #headerCartCount, .cart-count');
const text = node?.textContent || '';
const num = Number(text.replace(/[^\\d]/g, ''));
return Number.isFinite(num) ? num : null;
})();
const countIncreased =
beforeCount != null &&
afterCount != null &&
afterCount >= beforeCount &&
(afterCount > beforeCount || beforeCount === 0);
return {
ok: successMessage || countIncreased,
reason: successMessage || countIncreased ? 'SUCCESS' : 'UNKNOWN',
currentProductId,
beforeCount,
afterCount,
loginHints,
};
})()
`;
}
cli({
site: 'coupang',
name: 'add-to-cart',
description: 'Add a Coupang product to cart using logged-in browser session',
domain: 'www.coupang.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'productId', required: false, help: 'Coupang product ID' },
{ name: 'url', required: false, help: 'Canonical product URL' },
],
columns: ['ok', 'product_id', 'url', 'message'],
func: async (page, kwargs) => {
const rawProductId = kwargs.productId ?? kwargs.product_id;
const productId = normalizeProductId(rawProductId);
const targetUrl = canonicalizeProductUrl(kwargs.url, productId);
if (!productId && !targetUrl) {
throw new Error('Either --product-id or --url is required');
}
const finalUrl = targetUrl || canonicalizeProductUrl('', productId);
await page.goto(finalUrl);
await page.wait(3);
const result = await page.evaluate(buildAddToCartEvaluate(productId));
const loginHints = result?.loginHints ?? {};
if (loginHints.hasLoginLink && !loginHints.hasMyCoupang) {
throw new Error('Coupang login required. Please log into Coupang in Chrome and retry.');
}
const actualProductId = normalizeProductId(result?.currentProductId || productId);
if (result?.reason === 'PRODUCT_MISMATCH') {
throw new Error(`Product mismatch: expected ${productId}, got ${actualProductId || 'unknown'}`);
}
if (result?.reason === 'OPTION_REQUIRED') {
throw new Error('This product requires option selection and is not supported in v1.');
}
if (result?.reason === 'ADD_TO_CART_BUTTON_NOT_FOUND') {
throw new Error('Could not find an add-to-cart button on the product page.');
}
if (!result?.ok) {
throw new Error('Failed to confirm add-to-cart success.');
}
return [{
ok: true,
product_id: actualProductId || productId,
url: finalUrl,
message: 'Added to cart',
}];
},
});
+466
View File
@@ -0,0 +1,466 @@
import { cli, Strategy } from '../../registry.js';
import { mergeSearchItems, normalizeSearchItem, sanitizeSearchItems } from '../../coupang.js';
function escapeJsString(value: string): string {
return JSON.stringify(value);
}
function buildApplyFilterEvaluate(filter: string): string {
return `
() => {
const filter = ${escapeJsString(filter)};
const labels = Array.from(document.querySelectorAll('label'));
const normalize = (value) => (value == null ? '' : String(value).trim().toLowerCase());
const target = labels.find((label) => {
const component = normalize(label.getAttribute('data-component-name'));
const imgAlt = normalize(label.querySelector('img')?.getAttribute('alt'));
const text = normalize(label.textContent);
if (filter === 'rocket') {
return (
component.includes('deliveryfilteroption-rocket_luxury,rocket_wow,coupang_global') ||
imgAlt.includes('rocket_luxury,rocket_wow,coupang_global') ||
imgAlt.includes('rocket-all') ||
text.includes('로켓')
);
}
return component.includes(filter) || imgAlt.includes(filter) || text.includes(filter);
});
if (!target) {
return { ok: false, reason: 'FILTER_NOT_FOUND' };
}
target.click();
return {
ok: true,
reason: 'FILTER_CLICKED',
component: target.getAttribute('data-component-name') || '',
text: (target.textContent || '').trim(),
alt: target.querySelector('img')?.getAttribute('alt') || '',
};
}
`;
}
function buildCurrentLocationEvaluate(): string {
return `
() => ({
href: location.href
})
`;
}
function buildSearchEvaluate(query: string, limit: number, pageNumber: number): string {
return `
(async () => {
const query = ${escapeJsString(query)};
const limit = ${limit};
const pageNumber = ${pageNumber};
const normalizeText = (value) => (value == null ? '' : String(value).trim());
const parseNum = (value) => {
const text = normalizeText(value).replace(/[^\\d.]/g, '');
if (!text) return null;
const num = Number(text);
return Number.isFinite(num) ? num : null;
};
const extractPriceFromText = (text) => {
const matches = normalizeText(text).match(/\\d{1,3}(?:,\\d{3})*원/g) || [];
if (!matches.length) return '';
if (matches.length >= 2) return matches[matches.length - 2];
return matches[0];
};
const extractPriceInfo = (root) => {
const priceArea =
root.querySelector('.PriceArea_priceArea__NntJz, [class*="PriceArea_priceArea"], [class*="priceArea"]') ||
root;
const priceAreaText = normalizeText(priceArea.textContent || '');
const originalPrice = normalizeText(
priceArea.querySelector(
'del, .base-price, .origin-price, .original-price, .strike-price, [class*="base-price"], [class*="origin-price"], [class*="line-through"]'
)?.textContent || ''
);
const originalPriceNum = parseNum(originalPrice);
const unitPrice =
normalizeText(
priceArea.querySelector('.unit-price, [class*="unit-price"], [class*="unitPrice"]')?.textContent || ''
) ||
priceAreaText.match(/\\([^)]*당\\s*[^)]*원[^)]*\\)/)?.[0] ||
'';
const candidates = Array.from(priceArea.querySelectorAll('span, strong, div'))
.map((node) => {
const text = normalizeText(node.textContent || '');
if (!text || !/\\d/.test(text)) return null;
if (/\\d{1,2}:\\d{2}:\\d{2}/.test(text)) return null;
if (/당\\s*\\d/.test(text)) return null;
if (/^\\d+%$/.test(text)) return null;
const num = parseNum(text);
if (num == null) return null;
const className = normalizeText(node.getAttribute('class') || '').toLowerCase();
let score = 0;
if (/price|sale|selling|final/.test(className)) score += 6;
if (/red/.test(className)) score += 5;
if (/font-bold|bold/.test(className)) score += 3;
if (/line-through/.test(className)) score -= 12;
if (text.includes('원')) score += 2;
if (originalPriceNum != null && num === originalPriceNum) score -= 10;
if (num < 100) score -= 10;
return { text, num, score };
})
.filter(Boolean)
.sort((a, b) => {
if (b.score !== a.score) return b.score - a.score;
if (originalPriceNum != null) {
const aPrefer = a.num !== originalPriceNum ? 1 : 0;
const bPrefer = b.num !== originalPriceNum ? 1 : 0;
if (bPrefer !== aPrefer) return bPrefer - aPrefer;
}
return b.num - a.num;
});
const currentPrice =
normalizeText(candidates.find((candidate) => candidate.num !== originalPriceNum)?.text || '') ||
normalizeText(candidates[0]?.text || '') ||
extractPriceFromText(priceAreaText) ||
'';
return {
price: currentPrice,
originalPrice,
unitPrice,
};
};
const canonicalUrl = (url, productId) => {
if (url) {
try {
const parsed = new URL(url, 'https://www.coupang.com');
const match = parsed.pathname.match(/\\/vp\\/products\\/(\\d+)/);
return 'https://www.coupang.com/vp/products/' + (match?.[1] || productId || '');
} catch {}
}
return productId ? 'https://www.coupang.com/vp/products/' + productId : '';
};
const normalize = (raw) => {
const rawText = normalizeText(raw.text || raw.badgeText || raw.deliveryText || raw.summary);
const productId = normalizeText(
raw.productId || raw.product_id || raw.id || raw.productNo ||
raw?.product?.productId || raw?.item?.id
).match(/(\\d{6,})/)?.[1] || '';
const title = normalizeText(
raw.title || raw.name || raw.productName || raw.productTitle || raw.itemName
);
const price = parseNum(raw.price || raw.salePrice || raw.finalPrice || raw.sellingPrice);
const originalPrice = parseNum(raw.originalPrice || raw.basePrice || raw.listPrice || raw.originPrice);
const unitPrice = normalizeText(raw.unitPrice || raw.unit_price || raw.unitPriceText);
const rating = parseNum(raw.rating || raw.star || raw.reviewRating);
const reviewCount = parseNum(raw.reviewCount || raw.ratingCount || raw.reviewCnt || raw.reviews);
const badge = Array.isArray(raw.badges) ? raw.badges.map(normalizeText).filter(Boolean).join(', ') : normalizeText(raw.badge || raw.labels);
const seller = normalizeText(raw.seller || raw.sellerName || raw.vendorName || raw.merchantName);
const category = normalizeText(raw.category || raw.categoryName || raw.categoryPath);
const discountRate = parseNum(raw.discountRate || raw.discount || raw.discountPercent);
const url = canonicalUrl(raw.url || raw.productUrl || raw.link, productId);
return {
productId,
title,
price,
originalPrice,
unitPrice,
discountRate,
rating,
reviewCount,
rocket: normalizeText(raw.rocket || raw.rocketType),
deliveryType: normalizeText(raw.deliveryType || raw.deliveryBadge || raw.shippingType || raw.shippingBadge),
deliveryPromise: normalizeText(raw.deliveryPromise || raw.promise || raw.arrivalText || raw.arrivalBadge),
seller,
badge,
category,
url,
};
};
const byApi = async () => {
const candidates = [
'/np/search?q=' + encodeURIComponent(query) + '&component=&channel=user&page=' + pageNumber,
'/np/search?component=&q=' + encodeURIComponent(query) + '&channel=user&page=' + pageNumber,
];
for (const path of candidates) {
try {
const resp = await fetch(path, { credentials: 'include' });
if (!resp.ok) continue;
const text = await resp.text();
const data = text.trim().startsWith('<') ? null : JSON.parse(text);
const maybeItems =
data?.data?.products ||
data?.data?.productList ||
data?.products ||
data?.productList ||
data?.items;
if (Array.isArray(maybeItems) && maybeItems.length) {
return maybeItems.slice(0, limit).map(normalize);
}
} catch {}
}
return [];
};
const byBootstrap = () => {
const isProductLike = (item) => {
if (!item || typeof item !== 'object') return false;
const values = [item.productId, item.product_id, item.id, item.productNo, item.url, item.productUrl, item.link, item.title, item.productName];
return values.some((value) => /\\/vp\\/products\\/|\\d{6,}/.test(normalizeText(value)));
};
const collectProducts = (node) => {
const queue = [node];
while (queue.length) {
const current = queue.shift();
if (!current || typeof current !== 'object') continue;
if (Array.isArray(current)) {
const productish = current.filter(isProductLike);
if (productish.length >= 3) return productish.slice(0, limit).map(normalize);
queue.push(...current.slice(0, 50));
continue;
}
for (const value of Object.values(current)) queue.push(value);
}
return [];
};
const scriptNodes = Array.from(document.scripts);
for (const script of scriptNodes) {
const text = script.textContent || '';
if (!text || !/product|search/i.test(text)) continue;
const arrayMatches = [
...text.matchAll(/"products?"\\s*:\\s*(\\[[\\s\\S]{100,}?\\])/g),
...text.matchAll(/"itemList"\\s*:\\s*(\\[[\\s\\S]{100,}?\\])/g),
];
for (const match of arrayMatches) {
try {
const products = JSON.parse(match[1]);
if (Array.isArray(products) && products.length) {
return products.slice(0, limit).map(normalize);
}
} catch {}
}
}
const globals = [
window.__NEXT_DATA__,
window.__APOLLO_STATE__,
window.__INITIAL_STATE__,
window.__STATE__,
window.__PRELOADED_STATE__,
];
for (const candidate of globals) {
if (!candidate || typeof candidate !== 'object') continue;
const found = collectProducts(candidate);
if (found.length) return found;
}
return [];
};
const byJsonLd = () => {
const scripts = Array.from(document.querySelectorAll('script[type="application/ld+json"]'));
for (const script of scripts) {
const text = script.textContent || '';
if (!text) continue;
try {
const payload = JSON.parse(text);
const docs = Array.isArray(payload) ? payload : [payload];
for (const doc of docs) {
const items =
doc?.itemListElement ||
doc?.about?.itemListElement ||
doc?.mainEntity?.itemListElement ||
[];
if (!Array.isArray(items) || !items.length) continue;
const mapped = items.map((entry) => {
const item = entry?.item || entry;
return normalize({
productId: item?.url || item?.sku || item?.productID,
title: item?.name,
price: item?.offers?.price,
originalPrice: item?.offers?.highPrice,
rating: item?.aggregateRating?.ratingValue,
reviewCount: item?.aggregateRating?.reviewCount,
seller: item?.offers?.seller?.name,
badge: item?.offers?.availability,
category: item?.category,
url: item?.url,
});
}).filter((item) => item.productId || item.url || item.title);
if (mapped.length) return mapped.slice(0, limit);
}
} catch {}
}
return [];
};
const byDom = () => {
const domScanLimit = Math.max(limit * 6, 60);
const cards = Array.from(new Set([
...document.querySelectorAll('li.search-product'),
...document.querySelectorAll('li[class*="search-product"], div[class*="search-product"], article[class*="search-product"]'),
...document.querySelectorAll('li[class*="ProductUnit_productUnit"], [class*="ProductUnit_productUnit"]'),
...document.querySelectorAll('.impression-logged, [class*="promotion-item"], [class*="product-item"]'),
...document.querySelectorAll('[data-product-id]'),
...document.querySelectorAll('[data-id]'),
...document.querySelectorAll('a[href*="/vp/products/"]'),
])).slice(0, domScanLimit);
const items = [];
for (const el of cards) {
const root = el.closest('li, div, article, section') || el;
const html = root.innerHTML || '';
const priceInfo = extractPriceInfo(root);
const badgeImages = Array.from(root.querySelectorAll('img[data-badge-id]'));
const badgeIds = badgeImages
.map((node) => node.getAttribute('data-badge-id') || '')
.filter(Boolean);
const badgeSrcText = badgeImages
.map((node) => (node.getAttribute('data-badge-id') || '') + ' ' + (node.getAttribute('src') || ''))
.join(' ');
const productId =
root.getAttribute('data-product-id') ||
el.getAttribute('data-product-id') ||
root.querySelector('a[href*="/vp/products/"]')?.getAttribute('data-product-id') ||
root.querySelector('a[href*="/vp/products/"]')?.getAttribute('href')?.match(/\\/vp\\/products\\/(\\d+)/)?.[1] ||
html.match(/\\/vp\\/products\\/(\\d+)/)?.[1] ||
(el.getAttribute('href') || '').match(/\\/vp\\/products\\/(\\d+)/)?.[1] ||
'';
const title =
root.querySelector('.name, .title, .product-name, .search-product-title, .item-title, .ProductUnit_productNameV2__cV9cw, [class*="ProductUnit_productName"], [class*="productName"], [class*="product-name"], [class*="title"]')?.textContent ||
root.querySelector('img[alt]')?.getAttribute('alt') ||
html.match(/alt="([^"]+)"/)?.[1] ||
(root.textContent || '').replace(/\\s+/g, ' ').trim().match(/^(.+?)(\\d{1,3},\\d{3}원|무료배송|내일\\(|오늘\\(|새벽)/)?.[1] ||
el.getAttribute('title') ||
'';
const price = priceInfo.price || '';
const originalPrice = priceInfo.originalPrice || '';
const unitPrice = priceInfo.unitPrice || '';
const rating =
root.querySelector('.rating, .star em, [class*="rating"], [class*="star"], [class*="ProductRating"] [aria-label], [aria-label][class*="ProductRating"]')?.getAttribute?.('aria-label') ||
root.querySelector('.rating, .star em, [class*="rating"], [class*="star"], [class*="ProductRating"]')?.textContent ||
'';
const reviewCount =
root.querySelector('.rating-total-count, .count, .review-count, .promotion-item-review-count, [class*="review"], [class*="count"], [class*="ProductRating"] span, [class*="ProductRating"] [class*="fw-text"]')?.textContent ||
'';
const seller =
root.querySelector('.seller, .vendor, .search-product-wrap .vendor-name, [class*="vendor"], [class*="seller"]')?.textContent ||
'';
const category =
root.getAttribute('data-category') ||
root.querySelector('[class*="category"]')?.textContent ||
'';
const text = (root.textContent || '').replace(/\\s+/g, ' ').trim();
const badgeNodes = Array.from(root.querySelectorAll('.badge, .delivery, .tag, .icon-service, .pdd-text, .delivery-text, [class*="badge"], [class*="delivery"]'));
const hrefNode = root.querySelector('a[href*="/vp/products/"]');
items.push(normalize({
productId,
title,
price,
originalPrice,
unitPrice,
rating,
reviewCount,
seller,
badges: [...badgeIds, ...badgeNodes.map((node) => node.textContent || '').filter(Boolean)],
rocket: badgeSrcText + ' ' + badgeNodes.map((node) => node.textContent || '').join(' '),
deliveryType: badgeNodes.map((node) => node.textContent || '').join(' ') + ' ' + text,
deliveryPromise: badgeNodes.map((node) => node.textContent || '').join(' ') + ' ' + text,
category,
text,
url: hrefNode?.getAttribute('href') || '',
}));
}
return items.slice(0, domScanLimit);
};
let items = await byApi();
if (!items.length) items = byJsonLd();
if (!items.length) items = byBootstrap();
const domItems = byDom();
if (!items.length) items = domItems;
return {
loginHints: {
hasLoginLink: Boolean(document.querySelector('a[href*="login"], a[title*="로그인"]')),
hasMyCoupang: /마이쿠팡/.test(document.body.innerText),
},
items,
domItems,
};
})()
`;
}
cli({
site: 'coupang',
name: 'search',
description: 'Search Coupang products with logged-in browser session',
domain: 'www.coupang.com',
strategy: Strategy.COOKIE,
browser: true,
args: [
{ name: 'query', required: true, help: 'Search keyword' },
{ name: 'page', type: 'int', default: 1, help: 'Search result page number' },
{ name: 'limit', type: 'int', default: 20, help: 'Max results (max 50)' },
{ name: 'filter', required: false, help: 'Optional search filter (currently supports: rocket)' },
],
columns: ['rank', 'title', 'price', 'unit_price', 'rating', 'review_count', 'rocket', 'delivery_type', 'delivery_promise', 'url'],
func: async (page, kwargs) => {
const query = String(kwargs.query || '').trim();
const pageNumber = Math.max(Number(kwargs.page || 1), 1);
const limit = Math.min(Math.max(Number(kwargs.limit || 20), 1), 50);
const filter = String(kwargs.filter || '').trim().toLowerCase();
if (!query) throw new Error('Query is required');
const initialPage = filter ? 1 : pageNumber;
const url = `https://www.coupang.com/np/search?q=${encodeURIComponent(query)}&channel=user&page=${initialPage}`;
await page.goto(url);
await page.wait(3);
if (filter) {
const filterResult = await page.evaluate(buildApplyFilterEvaluate(filter));
if (!filterResult?.ok) {
throw new Error(`Unsupported or unavailable filter: ${filter}`);
}
await page.wait(3);
if (pageNumber > 1) {
const locationInfo = await page.evaluate(buildCurrentLocationEvaluate());
const filteredUrl = new URL(locationInfo?.href || url);
filteredUrl.searchParams.set('page', String(pageNumber));
await page.goto(filteredUrl.toString());
await page.wait(3);
}
}
await page.autoScroll({ times: filter ? 3 : 2, delayMs: 1500 });
const raw = await page.evaluate(buildSearchEvaluate(query, limit, pageNumber));
const loginHints = raw?.loginHints ?? {};
const items = Array.isArray(raw?.items) ? raw.items : [];
const domItems = Array.isArray(raw?.domItems) ? raw.domItems : [];
const normalizedBase = sanitizeSearchItems(
items.map((item: Record<string, unknown>, index: number) => normalizeSearchItem(item, index)),
limit
);
const normalizedDom = sanitizeSearchItems(
domItems.map((item: Record<string, unknown>, index: number) => normalizeSearchItem(item, index)),
Math.max(limit * 6, 60)
);
const normalized = filter
? sanitizeSearchItems(normalizedDom, limit)
: mergeSearchItems(normalizedBase, normalizedDom, limit);
if (!normalized.length && loginHints.hasLoginLink && !loginHints.hasMyCoupang) {
throw new Error('Coupang login required. Please log into Coupang in Chrome and retry.');
}
return normalized;
},
});
+1 -1
View File
@@ -11,7 +11,7 @@ cli({
domain: 'www.v2ex.com',
strategy: Strategy.COOKIE,
browser: true,
forceExtension: true,
args: [],
columns: ['status', 'message'],
func: async (page: IPage | null) => {
+1 -1
View File
@@ -11,7 +11,7 @@ cli({
domain: 'www.v2ex.com',
strategy: Strategy.COOKIE,
browser: true,
forceExtension: true,
args: [],
columns: ['username', 'balance', 'unread_notifications', 'daily_reward_ready'],
func: async (page: IPage | null) => {
+1 -1
View File
@@ -11,7 +11,7 @@ cli({
domain: 'www.v2ex.com',
strategy: Strategy.COOKIE,
browser: true,
forceExtension: true,
args: [
{ name: 'limit', type: 'int', default: 20, help: 'Number of notifications' }
],
+35
View File
@@ -0,0 +1,35 @@
/**
* Shared constants used across explore, synthesize, and pipeline modules.
*/
/** URL query params that are volatile/ephemeral and should be stripped from patterns */
export const VOLATILE_PARAMS = new Set([
'w_rid', 'wts', '_', 'callback', 'timestamp', 't', 'nonce', 'sign',
]);
/** Search-related query parameter names */
export const SEARCH_PARAMS = new Set([
'q', 'query', 'keyword', 'search', 'wd', 'kw', 'search_query', 'w',
]);
/** Pagination-related query parameter names */
export const PAGINATION_PARAMS = new Set([
'page', 'pn', 'offset', 'cursor', 'next', 'page_num',
]);
/** Limit/page-size query parameter names */
export const LIMIT_PARAMS = new Set([
'limit', 'count', 'size', 'per_page', 'page_size', 'ps', 'num',
]);
/** Field role → common API field names mapping */
export const FIELD_ROLES: Record<string, string[]> = {
title: ['title', 'name', 'text', 'content', 'desc', 'description', 'headline', 'subject'],
url: ['url', 'uri', 'link', 'href', 'permalink', 'jump_url', 'web_url', 'share_url'],
author: ['author', 'username', 'user_name', 'nickname', 'nick', 'owner', 'creator', 'up_name', 'uname'],
score: ['score', 'hot', 'heat', 'likes', 'like_count', 'view_count', 'views', 'play', 'favorite_count', 'reply_count'],
time: ['time', 'created_at', 'publish_time', 'pub_time', 'date', 'ctime', 'mtime', 'pubdate', 'created'],
id: ['id', 'aid', 'bvid', 'mid', 'uid', 'oid', 'note_id', 'item_id'],
cover: ['cover', 'pic', 'image', 'thumbnail', 'poster', 'avatar'],
category: ['category', 'tag', 'type', 'tname', 'channel', 'section'],
};
+78
View File
@@ -0,0 +1,78 @@
import { describe, expect, it } from 'vitest';
import {
canonicalizeProductUrl,
dedupeSearchItems,
normalizeProductId,
normalizeSearchItem,
sanitizeSearchItems,
} from './coupang.js';
describe('normalizeProductId', () => {
it('extracts product id from canonical path', () => {
expect(normalizeProductId('https://www.coupang.com/vp/products/123456789')).toBe('123456789');
});
it('preserves numeric ids', () => {
expect(normalizeProductId('987654321')).toBe('987654321');
});
});
describe('canonicalizeProductUrl', () => {
it('normalizes relative Coupang paths', () => {
expect(canonicalizeProductUrl('/vp/products/123456789?itemId=1', '')).toBe(
'https://www.coupang.com/vp/products/123456789'
);
});
it('builds url from product id', () => {
expect(canonicalizeProductUrl('', '123456789')).toBe('https://www.coupang.com/vp/products/123456789');
});
});
describe('normalizeSearchItem', () => {
it('maps raw fields into compare-ready shape', () => {
const item = normalizeSearchItem({
productId: '123456789',
productName: '무선 마우스',
salePrice: '29,900원',
originalPrice: '39,900원',
rating: '4.8',
reviewCount: '1,234',
sellerName: '쿠팡',
badge: ['ROCKET', 'TOMORROW', '무료배송'],
categoryName: 'PC',
url: '/vp/products/123456789?itemId=1',
}, 0);
expect(item).toMatchObject({
rank: 1,
product_id: '123456789',
title: '무선 마우스',
price: 29900,
original_price: 39900,
rating: 4.8,
review_count: 1234,
rocket: '로켓배송',
delivery_type: '무료배송',
delivery_promise: '내일도착',
seller: '쿠팡',
category: 'PC',
url: 'https://www.coupang.com/vp/products/123456789',
});
});
});
describe('sanitizeSearchItems', () => {
it('drops duplicates and invalid rows', () => {
const rows = [
normalizeSearchItem({ productId: '1', productName: 'A', price: '1000', url: '/vp/products/1' }, 0),
normalizeSearchItem({ productId: '1', productName: 'A', price: '1000', url: '/vp/products/1' }, 1),
normalizeSearchItem({ productId: '', productName: '', price: '1000' }, 2),
normalizeSearchItem({ productId: '2', productName: 'B', price: '2000', url: '/vp/products/2' }, 3),
];
expect(dedupeSearchItems(rows)).toHaveLength(3);
expect(sanitizeSearchItems(rows, 10)).toHaveLength(2);
expect(sanitizeSearchItems(rows, 10).map(item => item.rank)).toEqual([1, 2]);
});
});
+302
View File
@@ -0,0 +1,302 @@
export interface CoupangSearchItem {
rank: number;
product_id: string;
title: string;
price: number | null;
original_price: number | null;
unit_price: string;
discount_rate: number | null;
rating: number | null;
review_count: number | null;
rocket: string;
delivery_type: string;
delivery_promise: string;
seller: string;
badge: string;
category: string;
url: string;
}
function itemKey(item: CoupangSearchItem): string {
return item.url || item.product_id || `${item.title}:${item.price ?? ''}`;
}
const ROCKET_PATTERNS = ['판매자로켓', '로켓프레시', '로켓와우', '로켓배송', '로켓직구'] as const;
const DELIVERY_TYPE_PATTERNS = ['무료배송', '일반배송'] as const;
const DELIVERY_PROMISE_PATTERNS = ['오늘도착', '내일도착', '새벽도착', '오늘출발'] as const;
const BADGE_ID_TO_ROCKET: Record<string, string> = {
ROCKET: '로켓배송',
ROCKET_MERCHANT: '판매자로켓',
ROCKET_WOW: '로켓와우',
WOW: '로켓와우',
ROCKET_FRESH: '로켓프레시',
FRESH: '로켓프레시',
SELLER_ROCKET: '판매자로켓',
ROCKET_JIKGU: '로켓직구',
JIKGU: '로켓직구',
COUPANG_GLOBAL: '로켓직구',
};
const BADGE_ID_TO_PROMISE: Record<string, string> = {
DAWN: '새벽도착',
EARLY_DAWN: '새벽도착',
TOMORROW: '내일도착',
TODAY: '오늘도착',
SAME_DAY: '오늘도착',
TODAY_SHIP: '오늘출발',
TODAY_DISPATCH: '오늘출발',
};
function asString(value: unknown): string {
if (value == null) return '';
return String(value).trim();
}
function toNumber(value: unknown): number | null {
if (typeof value === 'number' && Number.isFinite(value)) return value;
const text = asString(value).replace(/[^\d.]/g, '');
if (!text) return null;
const num = Number(text);
return Number.isFinite(num) ? num : null;
}
function pickFirst(obj: Record<string, unknown>, paths: string[]): unknown {
for (const path of paths) {
const parts = path.split('.');
let current: unknown = obj;
let ok = true;
for (const part of parts) {
if (!current || typeof current !== 'object' || !(part in (current as Record<string, unknown>))) {
ok = false;
break;
}
current = (current as Record<string, unknown>)[part];
}
if (ok && current != null && asString(current) !== '') return current;
}
return null;
}
export function normalizeProductId(raw: unknown): string {
const text = asString(raw);
if (!text) return '';
const match = text.match(/\/vp\/products\/(\d+)/) || text.match(/\b(\d{6,})\b/);
return match?.[1] ?? text;
}
export function canonicalizeProductUrl(rawUrl: unknown, productId?: unknown): string {
const raw = asString(rawUrl);
if (raw) {
try {
const url = new URL(raw.startsWith('http') ? raw : `https://www.coupang.com${raw}`);
if (!url.hostname.includes('coupang.com')) return '';
const id = normalizeProductId(url.pathname) || normalizeProductId(productId);
if (!id) return url.toString();
return `https://www.coupang.com/vp/products/${id}`;
} catch {
return '';
}
}
const id = normalizeProductId(productId);
return id ? `https://www.coupang.com/vp/products/${id}` : '';
}
function extractTokens(values: unknown[]): string[] {
return values
.flatMap((value) => {
const text = asString(value);
if (!text) return [];
return text.split(/[,\s|]+/);
})
.map((token) => token.trim().toUpperCase())
.filter(Boolean);
}
function normalizeJoinedText(...values: unknown[]): string {
return values
.map(asString)
.filter(Boolean)
.join(' ')
.replace(/schema\.org\/[A-Za-z]+/gi, ' ')
.replace(/\s+/g, ' ')
.trim();
}
function normalizeRocket(...values: unknown[]): string {
const tokens = extractTokens(values);
for (const token of tokens) {
if (BADGE_ID_TO_ROCKET[token]) return BADGE_ID_TO_ROCKET[token];
}
const text = normalizeJoinedText(...values);
if (!text) return '';
if (/판매자\s*로켓/.test(text)) return '판매자로켓';
if (/로켓\s*프레시|새벽\s*도착\s*보장/.test(text)) return '로켓프레시';
if (/로켓\s*와우/.test(text)) return '로켓와우';
if (/로켓\s*직구|직구/.test(text)) return '로켓직구';
if (/로켓\s*배송/.test(text)) return '로켓배송';
return ROCKET_PATTERNS.find(pattern => text.includes(pattern)) ?? '';
}
function normalizeDeliveryType(...values: unknown[]): string {
const text = normalizeJoinedText(...values);
if (!text) return '';
if (/무료\s*배송/.test(text)) return '무료배송';
if (/일반\s*배송/.test(text)) return '일반배송';
return DELIVERY_TYPE_PATTERNS.find(pattern => text.includes(pattern)) ?? '';
}
function normalizeDeliveryPromise(...values: unknown[]): string {
const tokens = extractTokens(values);
for (const token of tokens) {
if (BADGE_ID_TO_PROMISE[token]) return BADGE_ID_TO_PROMISE[token];
}
const text = normalizeJoinedText(...values);
if (!text) return '';
if (/오늘\s*출발/.test(text)) return '오늘출발';
if (/오늘.*도착/.test(text)) return '오늘도착';
if (/새벽.*도착/.test(text)) return '새벽도착';
if (/내일.*도착/.test(text)) return '내일도착';
return DELIVERY_PROMISE_PATTERNS.find(pattern => text.includes(pattern)) ?? '';
}
function normalizeBadge(value: unknown): string {
const normalizeOne = (entry: unknown): string => {
const text = asString(entry);
if (!text) return '';
if (/schema\.org\//i.test(text)) {
return text.split('/').pop() ?? '';
}
return text;
};
if (Array.isArray(value)) {
return value.map(normalizeOne).filter(Boolean).join(', ');
}
return normalizeOne(value);
}
export function normalizeSearchItem(raw: Record<string, unknown>, index: number): CoupangSearchItem {
const productId = normalizeProductId(
pickFirst(raw, ['productId', 'product_id', 'id', 'productNo', 'item.id', 'product.productId', 'url'])
);
const title = asString(
pickFirst(raw, ['title', 'name', 'productName', 'productTitle', 'itemName', 'item.title'])
);
const price = toNumber(
pickFirst(raw, ['price', 'salePrice', 'finalPrice', 'sellingPrice', 'discountPrice', 'item.price'])
);
const originalPrice = toNumber(
pickFirst(raw, ['originalPrice', 'basePrice', 'listPrice', 'originPrice', 'strikePrice'])
);
const unitPrice = asString(
pickFirst(raw, ['unitPrice', 'unit_price', 'unitPriceText'])
);
const rating = toNumber(
pickFirst(raw, ['rating', 'star', 'reviewRating', 'review.rating', 'item.rating'])
);
const reviewCount = toNumber(
pickFirst(raw, ['reviewCount', 'ratingCount', 'reviews', 'reviewCnt', 'item.reviewCount'])
);
const deliveryHintValues = [
pickFirst(raw, ['deliveryType', 'deliveryBadge', 'badgeLabel', 'shippingType', 'shippingBadge']),
pickFirst(raw, ['badge', 'badges', 'labels', 'benefitBadge', 'promotionBadge']),
pickFirst(raw, ['text', 'summary']),
pickFirst(raw, ['deliveryPromise', 'promise', 'arrivalText', 'arrivalBadge']),
pickFirst(raw, ['rocket', 'rocketType']),
];
const deliveryType = normalizeDeliveryType(...deliveryHintValues);
const deliveryPromise = normalizeDeliveryPromise(...deliveryHintValues);
const rocket = normalizeRocket(...deliveryHintValues);
const badge = normalizeBadge(
pickFirst(raw, ['badge', 'badges', 'labels', 'benefitBadge', 'promotionBadge'])
);
const category = asString(
pickFirst(raw, ['category', 'categoryName', 'categoryPath', 'item.category'])
);
const seller = asString(
pickFirst(raw, ['seller', 'sellerName', 'vendorName', 'merchantName', 'item.seller'])
);
const url = canonicalizeProductUrl(
pickFirst(raw, ['url', 'productUrl', 'link', 'item.url']),
productId
);
const discountRate = toNumber(
pickFirst(raw, ['discountRate', 'discount', 'discountPercent', 'discount_rate'])
);
return {
rank: index + 1,
product_id: productId,
title,
price,
original_price: originalPrice,
unit_price: unitPrice,
discount_rate: discountRate,
rating,
review_count: reviewCount,
rocket,
delivery_type: deliveryType,
delivery_promise: deliveryPromise,
seller,
badge,
category,
url,
};
}
export function dedupeSearchItems(items: CoupangSearchItem[]): CoupangSearchItem[] {
const seen = new Set<string>();
const out: CoupangSearchItem[] = [];
for (const item of items) {
const key = itemKey(item);
if (!key || seen.has(key)) continue;
seen.add(key);
out.push({ ...item, rank: out.length + 1 });
}
return out;
}
export function sanitizeSearchItems(items: CoupangSearchItem[], limit: number): CoupangSearchItem[] {
return dedupeSearchItems(
items.filter(item => Boolean(item.title && (item.product_id || item.url)))
).slice(0, limit);
}
export function mergeSearchItems(base: CoupangSearchItem[], extra: CoupangSearchItem[], limit: number): CoupangSearchItem[] {
const extraMap = new Map<string, CoupangSearchItem>();
for (const item of extra) {
const key = itemKey(item);
if (key) extraMap.set(key, item);
}
const merged = base.map((item) => {
const key = itemKey(item);
const patch = key ? extraMap.get(key) : null;
if (!patch) return item;
return {
...item,
price: patch.price ?? item.price,
original_price: patch.original_price ?? item.original_price,
unit_price: patch.unit_price || item.unit_price,
discount_rate: patch.discount_rate ?? item.discount_rate,
rating: patch.rating ?? item.rating,
review_count: patch.review_count ?? item.review_count,
rocket: patch.rocket || item.rocket,
delivery_type: patch.delivery_type || item.delivery_type,
delivery_promise: patch.delivery_promise || item.delivery_promise,
seller: patch.seller || item.seller,
badge: patch.badge || item.badge,
category: patch.category || item.category,
url: patch.url || item.url,
};
});
const mergedKeys = new Set(merged.map(item => itemKey(item)).filter(Boolean));
const appended = extra.filter(item => {
const key = itemKey(item);
return key && !mergedKeys.has(key);
});
return sanitizeSearchItems([...merged, ...appended], limit);
}
+5 -13
View File
@@ -90,20 +90,16 @@ describe('doctor report rendering', () => {
const text = renderBrowserDoctorReport({
envToken: 'abc123',
envFingerprint: 'fp1',
extensionToken: 'abc123',
extensionFingerprint: 'fp1',
shellFiles: [{ path: '/tmp/.zshrc', exists: true, token: 'abc123', fingerprint: 'fp1' }],
configs: [{ path: '/tmp/mcp.json', exists: true, format: 'json', token: 'abc123', fingerprint: 'fp1', writable: true }],
remoteDebuggingEnabled: true,
remoteDebuggingEndpoint: 'ws://127.0.0.1:9222/devtools/browser/test',
cdpEnabled: false,
cdpToken: null,
cdpFingerprint: null,
recommendedToken: 'abc123',
recommendedFingerprint: 'fp1',
warnings: [],
issues: [],
});
expect(text).toContain('[OK] Chrome remote debugging: enabled');
expect(text).toContain('[OK] Environment token: configured (fp1)');
expect(text).toContain('[OK] MCP config /tmp/mcp.json: configured (fp1)');
});
@@ -112,20 +108,16 @@ describe('doctor report rendering', () => {
const text = renderBrowserDoctorReport({
envToken: 'abc123',
envFingerprint: 'fp1',
extensionToken: null,
extensionFingerprint: null,
shellFiles: [{ path: '/tmp/.zshrc', exists: true, token: 'def456', fingerprint: 'fp2' }],
configs: [{ path: '/tmp/mcp.json', exists: true, format: 'json', token: 'abc123', fingerprint: 'fp1', writable: true }],
remoteDebuggingEnabled: false,
remoteDebuggingEndpoint: null,
cdpEnabled: false,
cdpToken: null,
cdpFingerprint: null,
recommendedToken: 'abc123',
recommendedFingerprint: 'fp1',
warnings: ['Chrome remote debugging appears to be disabled or Chrome is not currently exposing a DevTools endpoint.'],
warnings: [],
issues: ['Detected inconsistent Playwright MCP tokens across env/config files.'],
});
expect(text).toContain('[WARN] Chrome remote debugging: disabled');
expect(text).toContain('[MISMATCH] Environment token: configured (fp1)');
expect(text).toContain('[MISMATCH] Shell file /tmp/.zshrc: configured (fp2)');
expect(text).toContain('[MISMATCH] Recommended token fingerprint: fp1');
+159 -58
View File
@@ -1,10 +1,11 @@
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { execSync } from 'node:child_process';
import { createInterface } from 'node:readline/promises';
import { stdin as input, stdout as output } from 'node:process';
import type { IPage } from './types.js';
import { PlaywrightMCP, discoverChromeEndpoint, getTokenFingerprint } from './browser.js';
import { PlaywrightMCP, getTokenFingerprint } from './browser.js';
import { browserSession } from './runtime.js';
const PLAYWRIGHT_SERVER_NAME = 'playwright';
@@ -43,13 +44,10 @@ export type DoctorReport = {
cliVersion?: string;
envToken: string | null;
envFingerprint: string | null;
extensionToken: string | null;
extensionFingerprint: string | null;
shellFiles: ShellFileStatus[];
configs: McpConfigStatus[];
remoteDebuggingEnabled: boolean;
remoteDebuggingEndpoint: string | null;
cdpEnabled: boolean;
cdpToken: string | null;
cdpFingerprint: string | null;
recommendedToken: string | null;
recommendedFingerprint: string | null;
warnings: string[];
@@ -84,12 +82,16 @@ export function getDefaultMcpConfigPaths(cwd: string = process.cwd()): string[]
path.join(home, '.codex', 'config.toml'),
path.join(home, '.codex', 'mcp.json'),
path.join(home, '.cursor', 'mcp.json'),
path.join(home, '.claude.json'),
path.join(home, '.gemini', 'settings.json'),
path.join(home, '.gemini', 'antigravity', 'mcp_config.json'),
path.join(home, '.config', 'opencode', 'opencode.json'),
path.join(home, 'Library', 'Application Support', 'Claude', 'claude_desktop_config.json'),
path.join(home, '.config', 'Claude', 'claude_desktop_config.json'),
path.join(cwd, '.cursor', 'mcp.json'),
path.join(cwd, '.vscode', 'mcp.json'),
path.join(cwd, '.opencode', 'opencode.json'),
path.join(cwd, '.mcp.json'),
];
return [...new Set(candidates)];
}
@@ -225,45 +227,149 @@ function readConfigStatus(filePath: string): McpConfigStatus {
}
}
async function extractTokenViaCdp(): Promise<string | null> {
if (!(process.env.OPENCLI_USE_CDP === '1' || process.env.OPENCLI_CDP_ENDPOINT))
return null;
const candidates = [
`chrome-extension://${PLAYWRIGHT_EXTENSION_ID}/options.html`,
`chrome-extension://${PLAYWRIGHT_EXTENSION_ID}/popup.html`,
`chrome-extension://${PLAYWRIGHT_EXTENSION_ID}/connect.html`,
`chrome-extension://${PLAYWRIGHT_EXTENSION_ID}/index.html`,
];
const result = await browserSession(PlaywrightMCP, async (page: IPage) => {
for (const url of candidates) {
try {
await page.goto(url);
await page.wait(1);
const token = await page.evaluate(`() => {
const values = new Set();
const push = (value) => {
if (!value || typeof value !== 'string') return;
for (const match of value.matchAll(/[A-Za-z0-9_-]{24,}/g)) values.add(match[0]);
};
document.querySelectorAll('input, textarea, code, pre, span, div').forEach((el) => {
push(el.value);
push(el.textContent || '');
push(el.getAttribute && el.getAttribute('value'));
});
return Array.from(values);
}`);
const matches = Array.isArray(token) ? token.filter((v: string) => v.length >= 24) : [];
if (matches.length > 0) return matches.sort((a: string, b: string) => b.length - a.length)[0];
} catch {}
/**
* Discover the auth token stored by the Playwright MCP Bridge extension
* by scanning Chrome's LevelDB localStorage files directly.
*
* Uses `strings` + `grep` for fast binary scanning on macOS/Linux,
* with a pure-Node fallback on Windows.
*/
export function discoverExtensionToken(): string | null {
const home = os.homedir();
const platform = os.platform();
const bases: string[] = [];
if (platform === 'darwin') {
bases.push(
path.join(home, 'Library', 'Application Support', 'Google', 'Chrome'),
path.join(home, 'Library', 'Application Support', 'Google', 'Chrome Canary'),
path.join(home, 'Library', 'Application Support', 'Chromium'),
path.join(home, 'Library', 'Application Support', 'Microsoft Edge'),
);
} else if (platform === 'linux') {
bases.push(
path.join(home, '.config', 'google-chrome'),
path.join(home, '.config', 'chromium'),
path.join(home, '.config', 'microsoft-edge'),
);
} else if (platform === 'win32') {
const appData = process.env.LOCALAPPDATA || path.join(home, 'AppData', 'Local');
bases.push(
path.join(appData, 'Google', 'Chrome', 'User Data'),
path.join(appData, 'Microsoft', 'Edge', 'User Data'),
);
}
const profiles = ['Default', 'Profile 1', 'Profile 2', 'Profile 3'];
// Token is 43 chars of base64url (from 32 random bytes)
const tokenRe = /([A-Za-z0-9_-]{40,50})/;
for (const base of bases) {
for (const profile of profiles) {
const dir = path.join(base, profile, 'Local Storage', 'leveldb');
if (!fileExists(dir)) continue;
// Fast path: use strings + grep to find candidate files and extract token
if (platform !== 'win32') {
const token = extractTokenViaStrings(dir, tokenRe);
if (token) return token;
continue;
}
// Slow path (Windows): read binary files directly
const token = extractTokenViaBinaryRead(dir, tokenRe);
if (token) return token;
}
return null;
});
return typeof result === 'string' && result ? result : null;
}
return null;
}
function extractTokenViaStrings(dir: string, tokenRe: RegExp): string | null {
try {
// Single shell pipeline: for each LevelDB file, extract strings, find lines
// after the extension ID, and filter for base64url token pattern.
//
// LevelDB `strings` output for the extension's auth-token entry:
// auth-token ← key name
// 4,mmlmfjhmonkocbjadbfplnigmagldckm.7 ← LevelDB internal key
// hqI86ncsD1QpcVcj-k9CyzTF-ieCQd_4KreZ_wy1WHA ← token value
//
// We get the line immediately after any EXTENSION_ID mention and check
// if it looks like a base64url token (40-50 chars, [A-Za-z0-9_-]).
const shellDir = dir.replace(/'/g, "'\\''");
const cmd = `for f in '${shellDir}'/*.ldb '${shellDir}'/*.log; do ` +
`[ -f "$f" ] && strings "$f" 2>/dev/null | ` +
`grep -A1 '${PLAYWRIGHT_EXTENSION_ID}' | ` +
`grep -v '${PLAYWRIGHT_EXTENSION_ID}' | ` +
`grep -E '^[A-Za-z0-9_-]{40,50}$' | head -1; ` +
`done 2>/dev/null`;
const result = execSync(cmd, { encoding: 'utf-8', timeout: 10000 }).trim();
// Take the first non-empty line
for (const line of result.split('\n')) {
const token = line.trim();
if (token && validateBase64urlToken(token)) return token;
}
} catch {}
return null;
}
function extractTokenViaBinaryRead(dir: string, tokenRe: RegExp): string | null {
const extIdBuf = Buffer.from(PLAYWRIGHT_EXTENSION_ID);
const keyBuf = Buffer.from('auth-token');
let files: string[];
try {
files = fs.readdirSync(dir)
.filter(f => f.endsWith('.ldb') || f.endsWith('.log'))
.map(f => path.join(dir, f));
} catch { return null; }
// Sort by mtime descending
files.sort((a, b) => {
try { return fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs; } catch { return 0; }
});
for (const file of files) {
let data: Buffer;
try { data = fs.readFileSync(file); } catch { continue; }
// Quick check: does file contain both the extension ID and auth-token key?
const extPos = data.indexOf(extIdBuf);
if (extPos === -1) continue;
const keyPos = data.indexOf(keyBuf, Math.max(0, extPos - 500));
if (keyPos === -1) continue;
// Scan for token value after auth-token key
let idx = 0;
while (true) {
const kp = data.indexOf(keyBuf, idx);
if (kp === -1) break;
const contextStart = Math.max(0, kp - 500);
if (data.indexOf(extIdBuf, contextStart) !== -1 && data.indexOf(extIdBuf, contextStart) < kp) {
const after = data.subarray(kp + keyBuf.length, kp + keyBuf.length + 200).toString('latin1');
const m = after.match(tokenRe);
if (m && validateBase64urlToken(m[1])) return m[1];
}
idx = kp + 1;
}
}
return null;
}
function validateBase64urlToken(token: string): boolean {
try {
const b64 = token.replace(/-/g, '+').replace(/_/g, '/');
const decoded = Buffer.from(b64, 'base64');
return decoded.length >= 28 && decoded.length <= 36;
} catch { return false; }
}
export async function runBrowserDoctor(opts: DoctorOptions = {}): Promise<DoctorReport> {
const envToken = process.env[PLAYWRIGHT_TOKEN_ENV] ?? null;
const remoteDebuggingEndpoint = await discoverChromeEndpoint().catch(() => null);
const shellPath = opts.shellRc ?? getDefaultShellRcPath();
const shellFiles: ShellFileStatus[] = [shellPath].map((filePath) => {
if (!fileExists(filePath)) return { path: filePath, exists: false, token: null, fingerprint: null };
@@ -273,29 +379,28 @@ export async function runBrowserDoctor(opts: DoctorOptions = {}): Promise<Doctor
});
const configPaths = opts.configPaths?.length ? opts.configPaths : getDefaultMcpConfigPaths();
const configs = configPaths.map(readConfigStatus);
const cdpToken = !opts.token && !envToken ? await extractTokenViaCdp().catch(() => null) : null;
// Try to discover the token directly from the Chrome extension's localStorage
const extensionToken = discoverExtensionToken();
const allTokens = [
opts.token ?? null,
extensionToken,
envToken,
...shellFiles.map(s => s.token),
...configs.map(c => c.token),
cdpToken,
].filter((v): v is string => !!v);
const uniqueTokens = [...new Set(allTokens)];
const recommendedToken = opts.token ?? envToken ?? (uniqueTokens.length === 1 ? uniqueTokens[0] : cdpToken) ?? null;
const recommendedToken = opts.token ?? extensionToken ?? envToken ?? (uniqueTokens.length === 1 ? uniqueTokens[0] : null) ?? null;
const report: DoctorReport = {
cliVersion: opts.cliVersion,
envToken,
envFingerprint: getTokenFingerprint(envToken ?? undefined),
extensionToken,
extensionFingerprint: getTokenFingerprint(extensionToken ?? undefined),
shellFiles,
configs,
remoteDebuggingEnabled: !!remoteDebuggingEndpoint,
remoteDebuggingEndpoint,
cdpEnabled: process.env.OPENCLI_USE_CDP === '1' || !!process.env.OPENCLI_CDP_ENDPOINT,
cdpToken,
cdpFingerprint: getTokenFingerprint(cdpToken ?? undefined),
recommendedToken,
recommendedFingerprint: getTokenFingerprint(recommendedToken ?? undefined),
warnings: [],
@@ -306,19 +411,18 @@ export async function runBrowserDoctor(opts: DoctorOptions = {}): Promise<Doctor
if (!shellFiles.some(s => s.token)) report.issues.push('Shell startup file does not export PLAYWRIGHT_MCP_EXTENSION_TOKEN.');
if (!configs.some(c => c.token)) report.issues.push('No scanned MCP config currently contains a Playwright extension token.');
if (uniqueTokens.length > 1) report.issues.push('Detected inconsistent Playwright MCP tokens across env/config files.');
if (!report.remoteDebuggingEnabled) report.warnings.push('Chrome remote debugging appears to be disabled or Chrome is not currently exposing a DevTools endpoint.');
for (const config of configs) {
if (config.parseError) report.warnings.push(`Could not parse ${config.path}: ${config.parseError}`);
}
if (!recommendedToken) {
if (report.cdpEnabled) report.warnings.push('CDP is enabled, but no token could be extracted automatically from the extension UI.');
else report.warnings.push('No token source found. Enable OPENCLI_USE_CDP=1 to allow a best-effort token read from the extension page.');
report.warnings.push('No token source found.');
}
return report;
}
export function renderBrowserDoctorReport(report: DoctorReport): string {
const tokenFingerprints = [
report.extensionFingerprint,
report.envFingerprint,
...report.shellFiles.map(shell => shell.fingerprint),
...report.configs.filter(config => config.exists).map(config => config.fingerprint),
@@ -326,8 +430,9 @@ export function renderBrowserDoctorReport(report: DoctorReport): string {
const uniqueFingerprints = [...new Set(tokenFingerprints)];
const hasMismatch = uniqueFingerprints.length > 1;
const lines = [`opencli v${report.cliVersion ?? 'unknown'} doctor`, ''];
lines.push(statusLine(report.remoteDebuggingEnabled ? 'OK' : 'WARN', `Chrome remote debugging: ${report.remoteDebuggingEnabled ? 'enabled' : 'disabled'}`));
if (report.remoteDebuggingEndpoint) lines.push(` ${report.remoteDebuggingEndpoint}`);
const extStatus: ReportStatus = !report.extensionToken ? 'MISSING' : hasMismatch ? 'MISMATCH' : 'OK';
lines.push(statusLine(extStatus, `Extension token (Chrome LevelDB): ${tokenSummary(report.extensionToken, report.extensionFingerprint)}`));
const envStatus: ReportStatus = !report.envToken ? 'MISSING' : hasMismatch ? 'MISMATCH' : 'OK';
lines.push(statusLine(envStatus, `Environment token: ${tokenSummary(report.envToken, report.envFingerprint)}`));
@@ -354,10 +459,6 @@ export function renderBrowserDoctorReport(report: DoctorReport): string {
lines.push(statusLine('MISSING', 'MCP config: no existing config files found in scanned locations'));
}
if (missingConfigCount > 0) lines.push(` Other scanned config locations not present: ${missingConfigCount}`);
if (report.cdpEnabled) {
const cdpStatus: ReportStatus = report.cdpToken ? 'OK' : 'WARN';
lines.push(statusLine(cdpStatus, `CDP token probe: ${tokenSummary(report.cdpToken, report.cdpFingerprint)}`));
}
lines.push('');
lines.push(statusLine(
hasMismatch ? 'MISMATCH' : report.recommendedToken ? 'OK' : 'WARN',
@@ -391,7 +492,7 @@ function writeFileWithMkdir(filePath: string, content: string): void {
export async function applyBrowserDoctorFix(report: DoctorReport, opts: DoctorOptions = {}): Promise<string[]> {
const token = opts.token ?? report.recommendedToken;
if (!token) throw new Error('No Playwright MCP token is available to write. Provide --token or enable CDP token probing first.');
if (!token) throw new Error('No Playwright MCP token is available to write. Provide --token first.');
const plannedWrites: string[] = [];
const shellPath = opts.shellRc ?? report.shellFiles[0]?.path ?? getDefaultShellRcPath();
+77
View File
@@ -0,0 +1,77 @@
/**
* Tests for engine.ts: CLI discovery and command execution.
*/
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { discoverClis, executeCommand } from './engine.js';
import { getRegistry, cli, Strategy } from './registry.js';
describe('discoverClis', () => {
it('handles non-existent directories gracefully', async () => {
// Should not throw for missing directories
await expect(discoverClis('/tmp/nonexistent-opencli-test-dir')).resolves.not.toThrow();
});
});
describe('executeCommand', () => {
it('executes a command with func', async () => {
const cmd = cli({
site: 'test-engine',
name: 'func-test',
description: 'test command with func',
browser: false,
strategy: Strategy.PUBLIC,
func: async (_page, kwargs) => {
return [{ title: kwargs.query ?? 'default' }];
},
});
const result = await executeCommand(cmd, null, { query: 'hello' });
expect(result).toEqual([{ title: 'hello' }]);
});
it('executes a command with pipeline', async () => {
const cmd = cli({
site: 'test-engine',
name: 'pipe-test',
description: 'test command with pipeline',
browser: false,
strategy: Strategy.PUBLIC,
pipeline: [
{ evaluate: '() => [{ n: 1 }, { n: 2 }, { n: 3 }]' },
{ limit: '2' },
],
});
// Pipeline commands require page for evaluate step, so we'll test the error path
await expect(executeCommand(cmd, null, {})).rejects.toThrow();
});
it('throws for command with no func or pipeline', async () => {
const cmd = cli({
site: 'test-engine',
name: 'empty-test',
description: 'empty command',
browser: false,
});
await expect(executeCommand(cmd, null, {})).rejects.toThrow('has no func or pipeline');
});
it('passes debug flag to func', async () => {
let receivedDebug = false;
const cmd = cli({
site: 'test-engine',
name: 'debug-test',
description: 'debug test',
browser: false,
func: async (_page, _kwargs, debug) => {
receivedDebug = debug ?? false;
return [];
},
});
await executeCommand(cmd, null, {}, true);
expect(receivedDebug).toBe(true);
});
});
+5 -5
View File
@@ -11,7 +11,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import yaml from 'js-yaml';
import { type CliCommand, type Arg, Strategy, registerCommand } from './registry.js';
import { type CliCommand, type InternalCliCommand, type Arg, Strategy, registerCommand } from './registry.js';
import type { IPage } from './types.js';
import { executePipeline } from './pipeline.js';
@@ -66,7 +66,7 @@ function loadFromManifest(manifestPath: string, clisDir: string): void {
// The actual module is loaded lazily on first executeCommand().
const strategy = (Strategy as any)[(entry.strategy ?? 'cookie').toUpperCase()] ?? Strategy.COOKIE;
const modulePath = path.resolve(clisDir, entry.modulePath);
const cmd: CliCommand = {
const cmd: InternalCliCommand = {
site: entry.site,
name: entry.name,
description: entry.description ?? '',
@@ -77,7 +77,6 @@ function loadFromManifest(manifestPath: string, clisDir: string): void {
columns: entry.columns,
timeoutSeconds: entry.timeout,
source: modulePath,
// Mark as lazy — executeCommand will load the module before running
_lazy: true,
_modulePath: modulePath,
};
@@ -170,8 +169,9 @@ export async function executeCommand(
debug: boolean = false,
): Promise<any> {
// Lazy-load TS module on first execution
if ((cmd as any)._lazy && (cmd as any)._modulePath) {
const modulePath = (cmd as any)._modulePath;
const internal = cmd as InternalCliCommand;
if (internal._lazy && internal._modulePath) {
const modulePath = internal._modulePath;
if (!_loadedModules.has(modulePath)) {
try {
await import(`file://${modulePath}`);
+2 -15
View File
@@ -9,6 +9,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import { DEFAULT_BROWSER_EXPLORE_TIMEOUT, browserSession, runWithTimeout } from './runtime.js';
import { VOLATILE_PARAMS, SEARCH_PARAMS, PAGINATION_PARAMS, LIMIT_PARAMS, FIELD_ROLES } from './constants.js';
// ── Site name detection ────────────────────────────────────────────────────
@@ -43,21 +44,7 @@ export function slugify(value: string): string {
// ── Field & capability inference ───────────────────────────────────────────
const FIELD_ROLES: Record<string, string[]> = {
title: ['title', 'name', 'text', 'content', 'desc', 'description', 'headline', 'subject'],
url: ['url', 'uri', 'link', 'href', 'permalink', 'jump_url', 'web_url', 'share_url'],
author: ['author', 'username', 'user_name', 'nickname', 'nick', 'owner', 'creator', 'up_name', 'uname'],
score: ['score', 'hot', 'heat', 'likes', 'like_count', 'view_count', 'views', 'play', 'favorite_count', 'reply_count'],
time: ['time', 'created_at', 'publish_time', 'pub_time', 'date', 'ctime', 'mtime', 'pubdate', 'created'],
id: ['id', 'aid', 'bvid', 'mid', 'uid', 'oid', 'note_id', 'item_id'],
cover: ['cover', 'pic', 'image', 'thumbnail', 'poster', 'avatar'],
category: ['category', 'tag', 'type', 'tname', 'channel', 'section'],
};
const SEARCH_PARAMS = new Set(['q', 'query', 'keyword', 'search', 'wd', 'kw', 'search_query', 'w']);
const PAGINATION_PARAMS = new Set(['page', 'pn', 'offset', 'cursor', 'next', 'page_num']);
const LIMIT_PARAMS = new Set(['limit', 'count', 'size', 'per_page', 'page_size', 'ps', 'num']);
const VOLATILE_PARAMS = new Set(['w_rid', 'wts', '_', 'callback', 'timestamp', 't', 'nonce', 'sign']);
// (constants now imported from constants.ts)
// ── Network analysis ───────────────────────────────────────────────────────
+153
View File
@@ -0,0 +1,153 @@
/**
* Shared XHR/Fetch interceptor JavaScript generators.
*
* Provides a single source of truth for monkey-patching browser
* fetch() and XMLHttpRequest to capture API responses matching
* a URL pattern. Used by:
* - Page.installInterceptor() (browser.ts)
* - stepIntercept (pipeline/steps/intercept.ts)
* - stepTap (pipeline/steps/tap.ts)
*/
/**
* Generate JavaScript source that installs a fetch/XHR interceptor.
* Captured responses are pushed to `window.__opencli_intercepted`.
*
* @param patternExpr - JS expression resolving to a URL substring to match (e.g. a JSON.stringify'd string)
* @param opts.arrayName - Global array name for captured data (default: '__opencli_intercepted')
* @param opts.patchGuard - Global boolean name to prevent double-patching (default: '__opencli_interceptor_patched')
*/
export function generateInterceptorJs(
patternExpr: string,
opts: { arrayName?: string; patchGuard?: string } = {},
): string {
const arr = opts.arrayName ?? '__opencli_intercepted';
const guard = opts.patchGuard ?? '__opencli_interceptor_patched';
return `
() => {
window.${arr} = window.${arr} || [];
const __pattern = ${patternExpr};
if (!window.${guard}) {
const __checkMatch = (url) => __pattern && url.includes(__pattern);
// ── Patch fetch ──
const __origFetch = window.fetch;
window.fetch = async function(...args) {
const reqUrl = typeof args[0] === 'string' ? args[0]
: (args[0] && args[0].url) || '';
const response = await __origFetch.apply(this, args);
if (__checkMatch(reqUrl)) {
try {
const clone = response.clone();
const json = await clone.json();
window.${arr}.push(json);
} catch(e) {}
}
return response;
};
// ── Patch XMLHttpRequest ──
const __XHR = XMLHttpRequest.prototype;
const __origOpen = __XHR.open;
const __origSend = __XHR.send;
__XHR.open = function(method, url) {
this.__opencli_url = String(url);
return __origOpen.apply(this, arguments);
};
__XHR.send = function() {
if (__checkMatch(this.__opencli_url)) {
this.addEventListener('load', function() {
try {
window.${arr}.push(JSON.parse(this.responseText));
} catch(e) {}
});
}
return __origSend.apply(this, arguments);
};
window.${guard} = true;
}
}
`;
}
/**
* Generate JavaScript source to read and clear intercepted data.
*/
export function generateReadInterceptedJs(arrayName: string = '__opencli_intercepted'): string {
return `
() => {
const data = window.${arrayName} || [];
window.${arrayName} = [];
return data;
}
`;
}
/**
* Generate a self-contained tap interceptor for store-action bridge.
* Unlike the global interceptor, this one:
* - Installs temporarily, restores originals in finally block
* - Resolves a promise on first capture (for immediate await)
* - Returns captured data directly
*/
export function generateTapInterceptorJs(patternExpr: string): {
setupVar: string;
capturedVar: string;
promiseVar: string;
resolveVar: string;
fetchPatch: string;
xhrPatch: string;
restorePatch: string;
} {
return {
setupVar: `
let captured = null;
let captureResolve;
const capturePromise = new Promise(r => { captureResolve = r; });
const capturePattern = ${patternExpr};
`,
capturedVar: 'captured',
promiseVar: 'capturePromise',
resolveVar: 'captureResolve',
fetchPatch: `
const origFetch = window.fetch;
window.fetch = async function(...fetchArgs) {
const resp = await origFetch.apply(this, fetchArgs);
try {
const url = typeof fetchArgs[0] === 'string' ? fetchArgs[0]
: fetchArgs[0] instanceof Request ? fetchArgs[0].url : String(fetchArgs[0]);
if (capturePattern && url.includes(capturePattern) && !captured) {
try { captured = await resp.clone().json(); captureResolve(); } catch {}
}
} catch {}
return resp;
};
`,
xhrPatch: `
const origXhrOpen = XMLHttpRequest.prototype.open;
const origXhrSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.open = function(method, url) {
this.__tapUrl = String(url);
return origXhrOpen.apply(this, arguments);
};
XMLHttpRequest.prototype.send = function(body) {
if (capturePattern && this.__tapUrl?.includes(capturePattern)) {
this.addEventListener('load', function() {
if (!captured) {
try { captured = JSON.parse(this.responseText); captureResolve(); } catch {}
}
});
}
return origXhrSend.apply(this, arguments);
};
`,
restorePatch: `
window.fetch = origFetch;
XMLHttpRequest.prototype.open = origXhrOpen;
XMLHttpRequest.prototype.send = origXhrSend;
`,
};
}
+10 -6
View File
@@ -3,7 +3,6 @@
* opencli — Make any website your CLI. AI-powered.
*/
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { fileURLToPath } from 'node:url';
@@ -14,16 +13,13 @@ import { type CliCommand, fullName, getRegistry, strategyLabel } from './registr
import { render as renderOutput } from './output.js';
import { PlaywrightMCP } from './browser.js';
import { browserSession, DEFAULT_BROWSER_COMMAND_TIMEOUT, runWithTimeout } from './runtime.js';
import { PKG_VERSION } from './version.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const BUILTIN_CLIS = path.resolve(__dirname, 'clis');
const USER_CLIS = path.join(os.homedir(), '.opencli', 'clis');
// Read version from package.json (single source of truth)
const pkgJsonPath = path.resolve(__dirname, '..', 'package.json');
const PKG_VERSION = JSON.parse(fs.readFileSync(pkgJsonPath, 'utf-8')).version ?? '0.0.0';
await discoverClis(BUILTIN_CLIS, USER_CLIS);
const program = new Command();
@@ -115,6 +111,14 @@ program.command('doctor')
}
});
program.command('setup')
.description('Interactive setup: configure Playwright MCP token across all detected tools')
.option('--token <token>', 'Provide token directly instead of auto-detecting')
.action(async (opts) => {
const { runSetup } = await import('./setup.js');
await runSetup({ cliVersion: PKG_VERSION, token: opts.token });
});
// ── Dynamic site commands ──────────────────────────────────────────────────
const registry = getRegistry();
@@ -144,7 +148,7 @@ for (const [, cmd] of registry) {
if (actionOpts.verbose) process.env.OPENCLI_VERBOSE = '1';
let result: any;
if (cmd.browser) {
result = await browserSession(PlaywrightMCP, async (page) => runWithTimeout(executeCommand(cmd, page, kwargs, actionOpts.verbose), { timeout: cmd.timeoutSeconds ?? DEFAULT_BROWSER_COMMAND_TIMEOUT, label: fullName(cmd) }), { forceExtension: cmd.forceExtension });
result = await browserSession(PlaywrightMCP, async (page) => runWithTimeout(executeCommand(cmd, page, kwargs, actionOpts.verbose), { timeout: cmd.timeoutSeconds ?? DEFAULT_BROWSER_COMMAND_TIMEOUT, label: fullName(cmd) }));
} else { result = await executeCommand(cmd, null, kwargs, actionOpts.verbose); }
if (actionOpts.verbose && (!result || (Array.isArray(result) && result.length === 0))) {
console.error(chalk.yellow(`[Verbose] Warning: Command returned an empty result. If the website structural API changed or requires authentication, check the network or update the adapter.`));
-4
View File
@@ -40,10 +40,6 @@ function renderTable(data: any, opts: RenderOptions): void {
style: { head: [], border: [] },
wordWrap: true,
wrapOnWordBoundary: true,
colWidths: columns.map((_c, i) => {
if (i === 0) return 6;
return null as any;
}).filter(() => true),
});
for (const row of rows) {
+15 -15
View File
@@ -15,26 +15,26 @@ export interface PipelineContext {
debug?: boolean;
}
/** Step handler signature */
/** Step handler: all steps conform to (page, params, data, args) => Promise<any> */
type StepHandler = (page: IPage | null, params: any, data: any, args: Record<string, any>) => Promise<any>;
/** Registry of all available step handlers */
const STEP_HANDLERS: Record<string, StepHandler> = {
navigate: stepNavigate as StepHandler,
navigate: stepNavigate,
fetch: stepFetch,
select: stepSelect as StepHandler,
evaluate: stepEvaluate as StepHandler,
snapshot: stepSnapshot as StepHandler,
click: stepClick as StepHandler,
type: stepType as StepHandler,
wait: stepWait as StepHandler,
press: stepPress as StepHandler,
map: stepMap as StepHandler,
filter: stepFilter as StepHandler,
sort: stepSort as StepHandler,
limit: stepLimit as StepHandler,
intercept: stepIntercept as StepHandler,
tap: stepTap as StepHandler,
select: stepSelect,
evaluate: stepEvaluate,
snapshot: stepSnapshot,
click: stepClick,
type: stepType,
wait: stepWait,
press: stepPress,
map: stepMap,
filter: stepFilter,
sort: stepSort,
limit: stepLimit,
intercept: stepIntercept,
tap: stepTap,
};
export async function executePipeline(
+4 -55
View File
@@ -4,6 +4,7 @@
import type { IPage } from '../../types.js';
import { render } from '../template.js';
import { generateInterceptorJs, generateReadInterceptedJs } from '../../interceptor.js';
export async function stepIntercept(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
const cfg = typeof params === 'object' ? params : {};
@@ -15,52 +16,7 @@ export async function stepIntercept(page: IPage, params: any, data: any, args: R
if (!capturePattern) return data;
// Step 1: Inject fetch/XHR interceptor BEFORE trigger
await page.evaluate(`
() => {
window.__opencli_intercepted = window.__opencli_intercepted || [];
const pattern = ${JSON.stringify(capturePattern)};
if (!window.__opencli_fetch_patched) {
const origFetch = window.fetch;
window.fetch = async function(...args) {
const reqUrl = typeof args[0] === 'string' ? args[0] : (args[0] && args[0].url) || '';
const response = await origFetch.apply(this, args);
setTimeout(async () => {
try {
if (reqUrl.includes(pattern)) {
const clone = response.clone();
const json = await clone.json();
window.__opencli_intercepted.push(json);
}
} catch(e) {}
}, 0);
return response;
};
window.__opencli_fetch_patched = true;
}
if (!window.__opencli_xhr_patched) {
const XHR = XMLHttpRequest.prototype;
const open = XHR.open;
const send = XHR.send;
XHR.open = function(method, url, ...args) {
this._reqUrl = url;
return open.call(this, method, url, ...args);
};
XHR.send = function(...args) {
this.addEventListener('load', function() {
try {
if (this._reqUrl && this._reqUrl.includes(pattern)) {
window.__opencli_intercepted.push(JSON.parse(this.responseText));
}
} catch(e) {}
});
return send.apply(this, args);
};
window.__opencli_xhr_patched = true;
}
}
`);
await page.evaluate(generateInterceptorJs(JSON.stringify(capturePattern)));
// Step 2: Execute the trigger action
if (trigger.startsWith('navigate:')) {
@@ -81,16 +37,9 @@ export async function stepIntercept(page: IPage, params: any, data: any, args: R
await page.wait(Math.min(timeout, 3));
// Step 4: Retrieve captured data
const matchingResponses = await page.evaluate(`
() => {
const data = window.__opencli_intercepted || [];
window.__opencli_intercepted = []; // clear after reading
return data;
}
`);
const matchingResponses = await page.evaluate(generateReadInterceptedJs());
// Step 4: Select from response if specified
// Step 5: Select from response if specified
let result = matchingResponses.length === 1 ? matchingResponses[0] :
matchingResponses.length > 1 ? matchingResponses : data;
+12 -51
View File
@@ -11,6 +11,7 @@
import type { IPage } from '../../types.js';
import { render } from '../template.js';
import { generateTapInterceptorJs } from '../../interceptor.js';
export async function stepTap(page: IPage, params: any, data: any, args: Record<string, any>): Promise<any> {
const cfg = typeof params === 'object' ? params : {};
@@ -38,53 +39,15 @@ export async function stepTap(page: IPage, params: any, data: any, args: Record<
? `store[${JSON.stringify(actionName)}](${actionArgsRendered.join(', ')})`
: `store[${JSON.stringify(actionName)}]()`;
// Use shared interceptor generator for fetch/XHR patching
const tap = generateTapInterceptorJs(JSON.stringify(capturePattern));
const js = `
async () => {
// ── 1. Setup capture proxy (fetch + XHR dual interception) ──
let captured = null;
let captureResolve;
const capturePromise = new Promise(r => { captureResolve = r; });
const capturePattern = ${JSON.stringify(capturePattern)};
// Intercept fetch API
const origFetch = window.fetch;
window.fetch = async function(...fetchArgs) {
const resp = await origFetch.apply(this, fetchArgs);
try {
const url = typeof fetchArgs[0] === 'string' ? fetchArgs[0]
: fetchArgs[0] instanceof Request ? fetchArgs[0].url : String(fetchArgs[0]);
if (capturePattern && url.includes(capturePattern) && !captured) {
try { captured = await resp.clone().json(); captureResolve(); } catch {}
}
} catch {}
return resp;
};
// Intercept XMLHttpRequest
const origXhrOpen = XMLHttpRequest.prototype.open;
const origXhrSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.open = function(method, url) {
this.__tapUrl = String(url);
return origXhrOpen.apply(this, arguments);
};
XMLHttpRequest.prototype.send = function(body) {
if (capturePattern && this.__tapUrl?.includes(capturePattern)) {
const xhr = this;
const origHandler = xhr.onreadystatechange;
xhr.onreadystatechange = function() {
if (xhr.readyState === 4 && !captured) {
try { captured = JSON.parse(xhr.responseText); captureResolve(); } catch {}
}
if (origHandler) origHandler.apply(this, arguments);
};
const origOnload = xhr.onload;
xhr.onload = function() {
if (!captured) { try { captured = JSON.parse(xhr.responseText); captureResolve(); } catch {} }
if (origOnload) origOnload.apply(this, arguments);
};
}
return origXhrSend.apply(this, arguments);
};
${tap.setupVar}
${tap.fetchPatch}
${tap.xhrPatch}
try {
// ── 2. Find store ──
@@ -119,19 +82,17 @@ export async function stepTap(page: IPage, params: any, data: any, args: Record<
await ${actionCall};
// ── 4. Wait for network response ──
if (!captured) {
if (!${tap.capturedVar}) {
const timeoutPromise = new Promise(r => setTimeout(r, ${timeout} * 1000));
await Promise.race([capturePromise, timeoutPromise]);
await Promise.race([${tap.promiseVar}, timeoutPromise]);
}
} finally {
// ── 5. Always restore originals ──
window.fetch = origFetch;
XMLHttpRequest.prototype.open = origXhrOpen;
XMLHttpRequest.prototype.send = origXhrSend;
${tap.restorePatch}
}
if (!captured) return { error: 'No matching response captured for pattern: ' + capturePattern };
return captured${selectChain} ?? captured;
if (!${tap.capturedVar}) return { error: 'No matching response captured for pattern: ' + capturePattern };
return ${tap.capturedVar}${selectChain} ?? ${tap.capturedVar};
}
`;
+106
View File
@@ -0,0 +1,106 @@
/**
* Tests for registry.ts: Strategy enum, cli() registration, helpers.
*/
import { describe, it, expect, beforeEach } from 'vitest';
import { cli, getRegistry, fullName, strategyLabel, registerCommand, Strategy, type CliCommand } from './registry.js';
describe('cli() registration', () => {
it('registers a command and returns it', () => {
const cmd = cli({
site: 'test-registry',
name: 'hello',
description: 'A test command',
strategy: Strategy.PUBLIC,
browser: false,
});
expect(cmd.site).toBe('test-registry');
expect(cmd.name).toBe('hello');
expect(cmd.strategy).toBe(Strategy.PUBLIC);
expect(cmd.browser).toBe(false);
expect(cmd.args).toEqual([]);
});
it('puts registered command in the registry', () => {
cli({
site: 'test-registry',
name: 'registered',
description: 'test',
});
const registry = getRegistry();
expect(registry.has('test-registry/registered')).toBe(true);
});
it('defaults strategy to COOKIE when browser is true', () => {
const cmd = cli({
site: 'test-registry',
name: 'default-strategy',
});
expect(cmd.strategy).toBe(Strategy.COOKIE);
expect(cmd.browser).toBe(true);
});
it('defaults strategy to PUBLIC when browser is false', () => {
const cmd = cli({
site: 'test-registry',
name: 'no-browser',
browser: false,
});
expect(cmd.strategy).toBe(Strategy.PUBLIC);
});
it('overwrites existing command on re-registration', () => {
cli({ site: 'test-registry', name: 'overwrite', description: 'v1' });
cli({ site: 'test-registry', name: 'overwrite', description: 'v2' });
const reg = getRegistry();
expect(reg.get('test-registry/overwrite')?.description).toBe('v2');
});
});
describe('fullName', () => {
it('returns site/name', () => {
const cmd: CliCommand = {
site: 'bilibili', name: 'hot', description: '', args: [],
};
expect(fullName(cmd)).toBe('bilibili/hot');
});
});
describe('strategyLabel', () => {
it('returns strategy string', () => {
const cmd: CliCommand = {
site: 'test', name: 'test', description: '', args: [],
strategy: Strategy.INTERCEPT,
};
expect(strategyLabel(cmd)).toBe('intercept');
});
it('returns public when no strategy set', () => {
const cmd: CliCommand = {
site: 'test', name: 'test', description: '', args: [],
};
expect(strategyLabel(cmd)).toBe('public');
});
});
describe('registerCommand', () => {
it('registers a pre-built command', () => {
const cmd: CliCommand = {
site: 'test-registry',
name: 'direct-reg',
description: 'directly registered',
args: [],
strategy: Strategy.HEADER,
browser: true,
};
registerCommand(cmd);
const reg = getRegistry();
expect(reg.get('test-registry/direct-reg')?.strategy).toBe(Strategy.HEADER);
});
});
+5 -9
View File
@@ -34,13 +34,13 @@ export interface CliCommand {
pipeline?: any[];
timeoutSeconds?: number;
source?: string;
/** Internal: lazy-loaded TS module support */
_lazy?: boolean;
_modulePath?: string;
/** Force extension bridge mode (bypass CDP), for anti-bot sites */
forceExtension?: boolean;
}
/** Internal extension for lazy-loaded TS modules (not exposed in public API) */
export interface InternalCliCommand extends CliCommand {
_lazy?: boolean;
_modulePath?: string;
}
export interface CliOptions {
site: string;
name: string;
@@ -53,10 +53,7 @@ export interface CliOptions {
func?: (page: IPage | null, kwargs: Record<string, any>, debug?: boolean) => Promise<any>;
pipeline?: any[];
timeoutSeconds?: number;
/** Force extension bridge mode (bypass CDP), for anti-bot sites */
forceExtension?: boolean;
}
const _registry = new Map<string, CliCommand>();
export function cli(opts: CliOptions): CliCommand {
@@ -72,7 +69,6 @@ export function cli(opts: CliOptions): CliCommand {
func: opts.func,
pipeline: opts.pipeline,
timeoutSeconds: opts.timeoutSeconds,
forceExtension: opts.forceExtension,
};
const key = fullName(cmd);
+23 -10
View File
@@ -9,29 +9,42 @@ export const DEFAULT_BROWSER_COMMAND_TIMEOUT = parseInt(process.env.OPENCLI_BROW
export const DEFAULT_BROWSER_EXPLORE_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_EXPLORE_TIMEOUT ?? '120', 10);
export const DEFAULT_BROWSER_SMOKE_TIMEOUT = parseInt(process.env.OPENCLI_BROWSER_SMOKE_TIMEOUT ?? '60', 10);
/**
* Timeout with seconds unit. Used for high-level command timeouts.
*/
export async function runWithTimeout<T>(
promise: Promise<T>,
opts: { timeout: number; label?: string },
): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timer = setTimeout(() => {
reject(new Error(`${opts.label ?? 'Operation'} timed out after ${opts.timeout}s`));
}, opts.timeout * 1000);
return withTimeoutMs(promise, opts.timeout * 1000, `${opts.label ?? 'Operation'} timed out after ${opts.timeout}s`);
}
promise
.then((result) => { clearTimeout(timer); resolve(result); })
.catch((err) => { clearTimeout(timer); reject(err); });
/**
* Timeout with milliseconds unit. Used for low-level internal timeouts.
*/
export function withTimeoutMs<T>(promise: Promise<T>, timeoutMs: number, message: string): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error(message)), timeoutMs);
promise.then(
(value) => { clearTimeout(timer); resolve(value); },
(error) => { clearTimeout(timer); reject(error); },
);
});
}
/** Interface for browser factory (PlaywrightMCP or test mocks) */
export interface IBrowserFactory {
connect(opts?: { timeout?: number }): Promise<IPage>;
close(): Promise<void>;
}
export async function browserSession<T>(
BrowserFactory: new () => any,
BrowserFactory: new () => IBrowserFactory,
fn: (page: IPage) => Promise<T>,
opts?: { forceExtension?: boolean },
): Promise<T> {
const mcp = new BrowserFactory();
try {
const page = await mcp.connect({ timeout: DEFAULT_BROWSER_CONNECT_TIMEOUT, forceExtension: opts?.forceExtension });
const page = await mcp.connect({ timeout: DEFAULT_BROWSER_CONNECT_TIMEOUT });
return await fn(page);
} finally {
await mcp.close().catch(() => {});
+187
View File
@@ -0,0 +1,187 @@
/**
* setup.ts — Interactive Playwright MCP token setup
*
* Discovers the extension token, shows an interactive checkbox
* for selecting which config files to update, and applies changes.
*/
import * as fs from 'node:fs';
import chalk from 'chalk';
import { createInterface } from 'node:readline/promises';
import { stdin as input, stdout as output } from 'node:process';
import {
type DoctorReport,
discoverExtensionToken,
getDefaultShellRcPath,
runBrowserDoctor,
upsertJsonConfigToken,
upsertShellToken,
upsertTomlConfigToken,
} from './doctor.js';
import { getTokenFingerprint } from './browser.js';
import { type CheckboxItem, checkboxPrompt } from './tui.js';
const PLAYWRIGHT_TOKEN_ENV = 'PLAYWRIGHT_MCP_EXTENSION_TOKEN';
function fileExists(p: string): boolean {
try { return fs.statSync(p).isFile() || fs.statSync(p).isDirectory(); } catch { return false; }
}
function writeFileWithMkdir(filePath: string, content: string) {
const dir = filePath.substring(0, filePath.lastIndexOf('/'));
if (dir && !fileExists(dir)) fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(filePath, content, 'utf-8');
}
function shortenPath(p: string): string {
const home = process.env.HOME || process.env.USERPROFILE || '';
return home && p.startsWith(home) ? '~' + p.slice(home.length) : p;
}
function toolName(p: string): string {
if (p.includes('.codex/')) return 'Codex';
if (p.includes('.cursor/')) return 'Cursor';
if (p.includes('.claude.json')) return 'Claude Code';
if (p.includes('antigravity')) return 'Antigravity';
if (p.includes('.gemini/settings')) return 'Gemini CLI';
if (p.includes('opencode')) return 'OpenCode';
if (p.includes('Claude/claude_desktop')) return 'Claude Desktop';
if (p.includes('.vscode/')) return 'VS Code';
if (p.includes('.mcp.json')) return 'Project MCP';
if (p.includes('.zshrc') || p.includes('.bashrc') || p.includes('.profile')) return 'Shell';
return '';
}
export async function runSetup(opts: { cliVersion?: string; token?: string } = {}) {
console.log();
console.log(chalk.bold(' opencli setup') + chalk.dim(' — Playwright MCP token configuration'));
console.log();
// Step 1: Discover token
let token = opts.token ?? null;
if (!token) {
const extensionToken = discoverExtensionToken();
const envToken = process.env[PLAYWRIGHT_TOKEN_ENV] ?? null;
if (extensionToken && envToken && extensionToken === envToken) {
token = extensionToken;
console.log(` ${chalk.green('✓')} Token auto-discovered from Chrome extension`);
console.log(` Fingerprint: ${chalk.bold(getTokenFingerprint(token) ?? 'unknown')}`);
} else if (extensionToken) {
token = extensionToken;
console.log(` ${chalk.green('✓')} Token discovered from Chrome extension ` +
chalk.dim(`(${getTokenFingerprint(token)})`));
if (envToken && envToken !== extensionToken) {
console.log(` ${chalk.yellow('!')} Environment has different token ` +
chalk.dim(`(${getTokenFingerprint(envToken)})`));
}
} else if (envToken) {
token = envToken;
console.log(` ${chalk.green('✓')} Token from environment variable ` +
chalk.dim(`(${getTokenFingerprint(token)})`));
}
} else {
console.log(` ${chalk.green('✓')} Using provided token ` +
chalk.dim(`(${getTokenFingerprint(token)})`));
}
if (!token) {
console.log(` ${chalk.yellow('!')} No token found. Please enter it manually.`);
console.log(chalk.dim(' (Find it in the Playwright MCP Bridge extension → Status page)'));
console.log();
const rl = createInterface({ input, output });
const answer = await rl.question(' Token: ');
rl.close();
token = answer.trim();
if (!token) {
console.log(chalk.red('\n No token provided. Aborting.\n'));
return;
}
}
const fingerprint = getTokenFingerprint(token) ?? 'unknown';
console.log();
// Step 2: Scan all config locations
const report = await runBrowserDoctor({ token, cliVersion: opts.cliVersion });
// Step 3: Build checkbox items
const items: CheckboxItem[] = [];
// Shell file
const shellPath = report.shellFiles[0]?.path ?? getDefaultShellRcPath();
const shellStatus = report.shellFiles[0];
const shellFp = shellStatus?.fingerprint;
const shellOk = shellFp === fingerprint;
items.push({
label: padRight(`${shortenPath(shellPath)}`, 50) + chalk.dim(` [${toolName(shellPath) || 'Shell'}]`),
value: `shell:${shellPath}`,
checked: !shellOk,
status: shellOk ? `configured (${shellFp})` : shellFp ? `mismatch (${shellFp})` : 'missing',
statusColor: shellOk ? 'green' : shellFp ? 'yellow' : 'red',
});
// Config files
for (const config of report.configs) {
const fp = config.fingerprint;
const ok = fp === fingerprint;
const tool = toolName(config.path);
items.push({
label: padRight(`${shortenPath(config.path)}`, 50) + chalk.dim(tool ? ` [${tool}]` : ''),
value: `config:${config.path}`,
checked: !ok,
status: ok ? `configured (${fp})` : !config.exists ? 'will create' : fp ? `mismatch (${fp})` : 'missing',
statusColor: ok ? 'green' : 'yellow',
});
}
// Step 4: Show interactive checkbox
const selected = await checkboxPrompt(items, {
title: ` Select files to update with token ${chalk.cyan(fingerprint)}:`,
});
if (selected.length === 0) {
console.log(chalk.dim(' No changes made.\n'));
return;
}
// Step 5: Apply changes
const written: string[] = [];
for (const sel of selected) {
if (sel.startsWith('shell:')) {
const path = sel.slice('shell:'.length);
const before = fileExists(path) ? fs.readFileSync(path, 'utf-8') : '';
writeFileWithMkdir(path, upsertShellToken(before, token));
written.push(path);
} else if (sel.startsWith('config:')) {
const path = sel.slice('config:'.length);
const config = report.configs.find(c => c.path === path);
if (config && config.parseError) continue;
const before = fileExists(path) ? fs.readFileSync(path, 'utf-8') : '';
const format = config?.format ?? (path.endsWith('.toml') ? 'toml' : 'json');
const next = format === 'toml' ? upsertTomlConfigToken(before, token) : upsertJsonConfigToken(before, token);
writeFileWithMkdir(path, next);
written.push(path);
}
}
process.env[PLAYWRIGHT_TOKEN_ENV] = token;
// Step 6: Summary
if (written.length > 0) {
console.log(chalk.green.bold(` ✓ Updated ${written.length} file(s):`));
for (const p of written) {
console.log(` ${chalk.dim('•')} ${shortenPath(p)}`);
}
} else {
console.log(chalk.yellow(' No files were changed.'));
}
console.log();
}
function padRight(s: string, n: number): string {
// Account for ANSI escape codes in length calculation
const visible = s.replace(/\x1b\[[0-9;]*m/g, '');
return visible.length >= n ? s : s + ' '.repeat(n - visible.length);
}
+5 -5
View File
@@ -6,12 +6,12 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import yaml from 'js-yaml';
import { VOLATILE_PARAMS, SEARCH_PARAMS, LIMIT_PARAMS, PAGINATION_PARAMS } from './constants.js';
/** Volatile params to strip from generated URLs */
const VOLATILE_PARAMS = new Set(['w_rid', 'wts', 'callback', '_', 'timestamp', 't', 'nonce', 'sign']);
const SEARCH_PARAM_NAMES = new Set(['q', 'query', 'keyword', 'search', 'wd', 'kw', 'w', 'search_query']);
const LIMIT_PARAM_NAMES = new Set(['ps', 'page_size', 'limit', 'count', 'per_page', 'size', 'num']);
const PAGE_PARAM_NAMES = new Set(['pn', 'page', 'page_num', 'offset', 'cursor']);
/** Renamed aliases for backward compatibility with local references */
const SEARCH_PARAM_NAMES = SEARCH_PARAMS;
const LIMIT_PARAM_NAMES = LIMIT_PARAMS;
const PAGE_PARAM_NAMES = PAGINATION_PARAMS;
export function synthesizeFromExplore(
target: string,
+165
View File
@@ -0,0 +1,165 @@
/**
* tui.ts — Zero-dependency interactive TUI components
*
* Uses raw stdin mode + ANSI escape codes for interactive prompts.
*/
import chalk from 'chalk';
export interface CheckboxItem {
label: string;
value: string;
checked: boolean;
/** Optional status to display after the label */
status?: string;
statusColor?: 'green' | 'yellow' | 'red' | 'dim';
}
/**
* Interactive multi-select checkbox prompt.
*
* Controls:
* ↑/↓ or j/k — navigate
* Space — toggle selection
* a — toggle all
* Enter — confirm
* q/Esc — cancel (returns empty)
*/
export async function checkboxPrompt(
items: CheckboxItem[],
opts: { title?: string; hint?: string } = {},
): Promise<string[]> {
if (items.length === 0) return [];
const { stdin, stdout } = process;
if (!stdin.isTTY) {
// Non-interactive: return all checked items
return items.filter(i => i.checked).map(i => i.value);
}
let cursor = 0;
const state = items.map(i => ({ ...i }));
function colorStatus(status: string | undefined, color: CheckboxItem['statusColor']): string {
if (!status) return '';
switch (color) {
case 'green': return chalk.green(status);
case 'yellow': return chalk.yellow(status);
case 'red': return chalk.red(status);
case 'dim': return chalk.dim(status);
default: return chalk.dim(status);
}
}
function render() {
// Move cursor to start and clear
let out = '';
if (opts.title) {
out += `\n${chalk.bold(opts.title)}\n\n`;
}
for (let i = 0; i < state.length; i++) {
const item = state[i];
const pointer = i === cursor ? chalk.cyan('') : ' ';
const checkbox = item.checked ? chalk.green('◉') : chalk.dim('○');
const label = i === cursor ? chalk.bold(item.label) : item.label;
const status = colorStatus(item.status, item.statusColor);
out += ` ${pointer} ${checkbox} ${label}${status ? ` ${status}` : ''}\n`;
}
out += `\n ${chalk.dim('↑↓ navigate · Space toggle · a all · Enter confirm · q cancel')}\n`;
return out;
}
return new Promise<string[]>((resolve) => {
const wasRaw = stdin.isRaw;
stdin.setRawMode(true);
stdin.resume();
let rendered = '';
function draw() {
// Clear previous render
if (rendered) {
const lines = rendered.split('\n').length;
stdout.write(`\x1b[${lines}A\x1b[J`);
}
rendered = render();
stdout.write(rendered);
}
function cleanup() {
stdin.setRawMode(wasRaw ?? false);
stdin.pause();
stdin.removeListener('data', onData);
// Clear the TUI
if (rendered) {
const lines = rendered.split('\n').length;
stdout.write(`\x1b[${lines}A\x1b[J`);
}
}
function onData(data: Buffer) {
const key = data.toString();
// Arrow up / k
if (key === '\x1b[A' || key === 'k') {
cursor = (cursor - 1 + state.length) % state.length;
draw();
return;
}
// Arrow down / j
if (key === '\x1b[B' || key === 'j') {
cursor = (cursor + 1) % state.length;
draw();
return;
}
// Space — toggle
if (key === ' ') {
state[cursor].checked = !state[cursor].checked;
draw();
return;
}
// Tab — toggle and move down
if (key === '\t') {
state[cursor].checked = !state[cursor].checked;
cursor = (cursor + 1) % state.length;
draw();
return;
}
// 'a' — toggle all
if (key === 'a') {
const allChecked = state.every(i => i.checked);
for (const item of state) item.checked = !allChecked;
draw();
return;
}
// Enter — confirm
if (key === '\r' || key === '\n') {
cleanup();
const selected = state.filter(i => i.checked).map(i => i.value);
// Show summary
stdout.write(` ${chalk.green('✓')} ${chalk.bold(`${selected.length} file(s) selected`)}\n\n`);
resolve(selected);
return;
}
// q / Esc / Ctrl+C — cancel
if (key === 'q' || key === '\x1b' || key === '\x03') {
cleanup();
stdout.write(` ${chalk.yellow('✗')} ${chalk.dim('Cancelled')}\n\n`);
resolve([]);
return;
}
}
stdin.on('data', onData);
draw();
});
}
+22
View File
@@ -3,6 +3,14 @@ import * as fs from 'node:fs';
import * as path from 'node:path';
import yaml from 'js-yaml';
/** All recognized pipeline step names */
const KNOWN_STEP_NAMES = new Set([
'navigate', 'click', 'type', 'wait', 'press', 'snapshot', 'scroll',
'fetch', 'evaluate',
'select', 'map', 'filter', 'sort', 'limit',
'intercept', 'tap',
]);
export function validateClisWithTarget(dirs: string[], target?: string): any {
const results: any[] = [];
let errors = 0; let warnings = 0; let files = 0;
@@ -38,6 +46,20 @@ function validateYamlFile(filePath: string): any {
if (def.pipeline && !Array.isArray(def.pipeline)) errors.push('"pipeline" must be an array');
if (def.columns && !Array.isArray(def.columns)) errors.push('"columns" must be an array');
if (def.args && typeof def.args !== 'object') errors.push('"args" must be an object');
// Validate pipeline step names (catch typos like 'navaigate')
if (Array.isArray(def.pipeline)) {
for (let i = 0; i < def.pipeline.length; i++) {
const step = def.pipeline[i];
if (step && typeof step === 'object') {
const stepKeys = Object.keys(step);
for (const key of stepKeys) {
if (!KNOWN_STEP_NAMES.has(key)) {
warnings.push(`Pipeline step ${i}: unknown step name "${key}" (did you mean one of: ${[...KNOWN_STEP_NAMES].join(', ')}?)`);
}
}
}
}
}
} catch (e: any) { errors.push(`YAML parse error: ${e.message}`); }
return { path: filePath, errors, warnings };
}
+10 -1
View File
@@ -1,9 +1,18 @@
/** Verification: validate + smoke. */
/**
* Verification: runs validation and optional smoke test.
*
* The smoke test is intentionally kept as a stub — full browser-based
* smoke testing requires a running browser session and is better suited
* to the `opencli test` command or CI pipelines.
*/
import { validateClisWithTarget, renderValidationReport } from './validate.js';
export async function verifyClis(opts: any): Promise<any> {
const report = validateClisWithTarget([opts.builtinClis, opts.userClis], opts.target);
return { ok: report.ok, validation: report, smoke: null };
}
export function renderVerifyReport(report: any): string {
return renderValidationReport(report.validation);
}
+18
View File
@@ -0,0 +1,18 @@
/**
* Single source of truth for package version.
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { fileURLToPath } from 'node:url';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const pkgJsonPath = path.resolve(__dirname, '..', 'package.json');
export const PKG_VERSION: string = (() => {
try {
return JSON.parse(fs.readFileSync(pkgJsonPath, 'utf-8')).version;
} catch {
return '0.0.0';
}
})();
+90
View File
@@ -0,0 +1,90 @@
/**
* E2E tests for login-required browser commands.
* These commands REQUIRE authentication (cookie/session).
* In CI (headless, no login), they should fail gracefully — NOT crash.
*
* These tests verify the error handling path, not the data extraction.
*/
import { describe, it, expect } from 'vitest';
import { runCli } from './helpers.js';
/**
* Verify a login-required command fails gracefully (no crash, no hang).
* Acceptable outcomes: exit code 1 with error message, OR timeout handled.
*/
async function expectGracefulAuthFailure(args: string[], label: string) {
const { stdout, stderr, code } = await runCli(args, { timeout: 60_000 });
// Should either fail with exit code 1 (error message) or succeed with empty data
// The key assertion: it should NOT hang forever or crash with unhandled exception
if (code !== 0) {
// Verify stderr has a meaningful error, not an unhandled crash
const output = stderr + stdout;
expect(output.length).toBeGreaterThan(0);
}
// If it somehow succeeds (e.g., partial public data), that's fine too
}
describe('login-required commands — graceful failure', () => {
// ── bilibili (requires cookie session) ──
it('bilibili me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'me', '-f', 'json'], 'bilibili me');
}, 60_000);
it('bilibili dynamic fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'dynamic', '--limit', '3', '-f', 'json'], 'bilibili dynamic');
}, 60_000);
it('bilibili favorite fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'favorite', '--limit', '3', '-f', 'json'], 'bilibili favorite');
}, 60_000);
it('bilibili history fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'history', '--limit', '3', '-f', 'json'], 'bilibili history');
}, 60_000);
it('bilibili following fails gracefully without login', async () => {
await expectGracefulAuthFailure(['bilibili', 'following', '--limit', '3', '-f', 'json'], 'bilibili following');
}, 60_000);
// ── twitter (requires login) ──
it('twitter bookmarks fails gracefully without login', async () => {
await expectGracefulAuthFailure(['twitter', 'bookmarks', '--limit', '3', '-f', 'json'], 'twitter bookmarks');
}, 60_000);
it('twitter timeline fails gracefully without login', async () => {
await expectGracefulAuthFailure(['twitter', 'timeline', '--limit', '3', '-f', 'json'], 'twitter timeline');
}, 60_000);
it('twitter notifications fails gracefully without login', async () => {
await expectGracefulAuthFailure(['twitter', 'notifications', '--limit', '3', '-f', 'json'], 'twitter notifications');
}, 60_000);
// ── v2ex (requires login) ──
it('v2ex me fails gracefully without login', async () => {
await expectGracefulAuthFailure(['v2ex', 'me', '-f', 'json'], 'v2ex me');
}, 60_000);
it('v2ex notifications fails gracefully without login', async () => {
await expectGracefulAuthFailure(['v2ex', 'notifications', '--limit', '3', '-f', 'json'], 'v2ex notifications');
}, 60_000);
// ── xueqiu (requires login) ──
it('xueqiu feed fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xueqiu', 'feed', '--limit', '3', '-f', 'json'], 'xueqiu feed');
}, 60_000);
it('xueqiu watchlist fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xueqiu', 'watchlist', '-f', 'json'], 'xueqiu watchlist');
}, 60_000);
// ── xiaohongshu (requires login) ──
it('xiaohongshu feed fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xiaohongshu', 'feed', '--limit', '3', '-f', 'json'], 'xiaohongshu feed');
}, 60_000);
it('xiaohongshu notifications fails gracefully without login', async () => {
await expectGracefulAuthFailure(['xiaohongshu', 'notifications', '--limit', '3', '-f', 'json'], 'xiaohongshu notifications');
}, 60_000);
});
+169
View File
@@ -0,0 +1,169 @@
/**
* E2E tests for browser commands that access PUBLIC data (no login required).
* These use OPENCLI_HEADLESS=1 to launch a headless Chromium.
*
* NOTE: Some sites may block headless browsers with bot detection.
* Tests are wrapped with tryBrowserCommand() which allows graceful failure.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
/**
* Run a browser command — returns parsed data or null on failure.
*/
async function tryBrowserCommand(args: string[]): Promise<any[] | null> {
const { stdout, code } = await runCli(args, { timeout: 60_000 });
if (code !== 0) return null;
try {
const data = parseJsonOutput(stdout);
return Array.isArray(data) ? data : null;
} catch {
return null;
}
}
/**
* Assert browser command returns data OR log a warning if blocked.
* Empty results (bot detection, geo-blocking) are treated as a warning, not a failure.
*/
function expectDataOrSkip(data: any[] | null, label: string) {
if (data === null || data.length === 0) {
console.warn(`${label}: skipped — no data returned (likely bot detection or geo-blocking)`);
return;
}
expect(data.length).toBeGreaterThanOrEqual(1);
}
describe('browser public-data commands E2E', () => {
// ── bbc (browser: true, strategy: public) ──
it('bbc news returns headlines', async () => {
const data = await tryBrowserCommand(['bbc', 'news', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'bbc news');
if (data) {
expect(data[0]).toHaveProperty('title');
}
}, 60_000);
// ── v2ex daily (browser: true) ──
it('v2ex daily returns topics', async () => {
const data = await tryBrowserCommand(['v2ex', 'daily', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'v2ex daily');
}, 60_000);
// ── bilibili (browser: true, cookie strategy) ──
it('bilibili hot returns trending videos', async () => {
const data = await tryBrowserCommand(['bilibili', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'bilibili hot');
if (data) {
expect(data[0]).toHaveProperty('title');
}
}, 60_000);
it('bilibili ranking returns ranked videos', async () => {
const data = await tryBrowserCommand(['bilibili', 'ranking', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'bilibili ranking');
}, 60_000);
it('bilibili search returns results', async () => {
const data = await tryBrowserCommand(['bilibili', 'search', 'typescript', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'bilibili search');
}, 60_000);
// ── weibo (browser: true, cookie strategy) ──
it('weibo hot returns trending topics', async () => {
const data = await tryBrowserCommand(['weibo', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'weibo hot');
}, 60_000);
// ── zhihu (browser: true, cookie strategy) ──
it('zhihu hot returns trending questions', async () => {
const data = await tryBrowserCommand(['zhihu', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'zhihu hot');
if (data) {
expect(data[0]).toHaveProperty('title');
}
}, 60_000);
it('zhihu search returns results', async () => {
const data = await tryBrowserCommand(['zhihu', 'search', 'playwright', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'zhihu search');
}, 60_000);
// ── reddit (browser: true, cookie strategy) ──
it('reddit hot returns posts', async () => {
const data = await tryBrowserCommand(['reddit', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'reddit hot');
}, 60_000);
it('reddit frontpage returns posts', async () => {
const data = await tryBrowserCommand(['reddit', 'frontpage', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'reddit frontpage');
}, 60_000);
// ── twitter (browser: true) ──
it('twitter trending returns trends', async () => {
const data = await tryBrowserCommand(['twitter', 'trending', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'twitter trending');
}, 60_000);
// ── xueqiu (browser: true, cookie strategy) ──
it('xueqiu hot returns hot posts', async () => {
const data = await tryBrowserCommand(['xueqiu', 'hot', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'xueqiu hot');
}, 60_000);
it('xueqiu hot-stock returns stocks', async () => {
const data = await tryBrowserCommand(['xueqiu', 'hot-stock', '--limit', '5', '-f', 'json']);
expectDataOrSkip(data, 'xueqiu hot-stock');
}, 60_000);
// ── reuters (browser: true) ──
it('reuters search returns articles', async () => {
const data = await tryBrowserCommand(['reuters', 'search', 'technology', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'reuters search');
}, 60_000);
// ── youtube (browser: true) ──
it('youtube search returns videos', async () => {
const data = await tryBrowserCommand(['youtube', 'search', 'typescript tutorial', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'youtube search');
}, 60_000);
// ── smzdm (browser: true) ──
it('smzdm search returns deals', async () => {
const data = await tryBrowserCommand(['smzdm', 'search', '键盘', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'smzdm search');
}, 60_000);
// ── boss (browser: true) ──
it('boss search returns jobs', async () => {
const data = await tryBrowserCommand(['boss', 'search', 'golang', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'boss search');
}, 60_000);
// ── ctrip (browser: true) ──
it('ctrip search returns flights', async () => {
const data = await tryBrowserCommand(['ctrip', 'search', '-f', 'json']);
expectDataOrSkip(data, 'ctrip search');
}, 60_000);
// ── coupang (browser: true) ──
it('coupang search returns products', async () => {
const data = await tryBrowserCommand(['coupang', 'search', 'laptop', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'coupang search');
}, 60_000);
// ── xiaohongshu (browser: true) ──
it('xiaohongshu search returns notes', async () => {
const data = await tryBrowserCommand(['xiaohongshu', 'search', '美食', '--limit', '3', '-f', 'json']);
expectDataOrSkip(data, 'xiaohongshu search');
}, 60_000);
// ── yahoo-finance (browser: true) ──
it('yahoo-finance quote returns stock data', async () => {
const data = await tryBrowserCommand(['yahoo-finance', 'quote', '--symbol', 'AAPL', '-f', 'json']);
expectDataOrSkip(data, 'yahoo-finance quote');
}, 60_000);
});
+63
View File
@@ -0,0 +1,63 @@
/**
* Shared helpers for E2E tests.
* Runs the built opencli binary as a subprocess.
*/
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import * as path from 'node:path';
import { fileURLToPath } from 'node:url';
const exec = promisify(execFile);
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const ROOT = path.resolve(__dirname, '../..');
const MAIN = path.join(ROOT, 'dist', 'main.js');
export interface CliResult {
stdout: string;
stderr: string;
code: number;
}
/**
* Run `opencli` as a child process with the given arguments.
* Without PLAYWRIGHT_MCP_EXTENSION_TOKEN, opencli auto-launches its own browser.
*/
export async function runCli(
args: string[],
opts: { timeout?: number; env?: Record<string, string> } = {},
): Promise<CliResult> {
const timeout = opts.timeout ?? 30_000;
try {
const { stdout, stderr } = await exec('node', [MAIN, ...args], {
cwd: ROOT,
timeout,
env: {
...process.env,
// Prevent chalk colors from polluting test assertions
FORCE_COLOR: '0',
NO_COLOR: '1',
...opts.env,
},
});
return { stdout, stderr, code: 0 };
} catch (err: any) {
return {
stdout: err.stdout ?? '',
stderr: err.stderr ?? '',
code: err.code ?? 1,
};
}
}
/**
* Parse JSON output from a CLI command.
* Throws a descriptive error if parsing fails.
*/
export function parseJsonOutput(stdout: string): any {
try {
return JSON.parse(stdout.trim());
} catch {
throw new Error(`Failed to parse CLI JSON output:\n${stdout.slice(0, 500)}`);
}
}
+106
View File
@@ -0,0 +1,106 @@
/**
* E2E tests for management/built-in commands.
* These commands require no external network access (except verify --smoke).
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
describe('management commands E2E', () => {
// ── list ──
it('list shows all registered commands', async () => {
const { stdout, code } = await runCli(['list', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
// Should have 50+ commands across 18 sites
expect(data.length).toBeGreaterThan(50);
// Each entry should have the standard fields
expect(data[0]).toHaveProperty('command');
expect(data[0]).toHaveProperty('site');
expect(data[0]).toHaveProperty('name');
expect(data[0]).toHaveProperty('strategy');
expect(data[0]).toHaveProperty('browser');
});
it('list default table format renders sites', async () => {
const { stdout, code } = await runCli(['list']);
expect(code).toBe(0);
// Should contain site names
expect(stdout).toContain('hackernews');
expect(stdout).toContain('bilibili');
expect(stdout).toContain('twitter');
expect(stdout).toContain('commands across');
});
it('list -f yaml produces valid yaml', async () => {
const { stdout, code } = await runCli(['list', '-f', 'yaml']);
expect(code).toBe(0);
expect(stdout).toContain('command:');
expect(stdout).toContain('site:');
});
it('list -f csv produces valid csv', async () => {
const { stdout, code } = await runCli(['list', '-f', 'csv']);
expect(code).toBe(0);
const lines = stdout.trim().split('\n');
expect(lines.length).toBeGreaterThan(50);
});
it('list -f md produces markdown table', async () => {
const { stdout, code } = await runCli(['list', '-f', 'md']);
expect(code).toBe(0);
expect(stdout).toContain('|');
expect(stdout).toContain('command');
});
// ── validate ──
it('validate passes for all built-in adapters', async () => {
const { stdout, code } = await runCli(['validate']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
expect(stdout).not.toContain('❌');
});
it('validate works for specific site', async () => {
const { stdout, code } = await runCli(['validate', 'hackernews']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
it('validate works for specific command', async () => {
const { stdout, code } = await runCli(['validate', 'hackernews/top']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
// ── verify ──
it('verify runs validation without smoke tests', async () => {
const { stdout, code } = await runCli(['verify']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
// ── version ──
it('--version shows version number', async () => {
const { stdout, code } = await runCli(['--version']);
expect(code).toBe(0);
expect(stdout.trim()).toMatch(/^\d+\.\d+\.\d+$/);
});
// ── help ──
it('--help shows usage', async () => {
const { stdout, code } = await runCli(['--help']);
expect(code).toBe(0);
expect(stdout).toContain('opencli');
expect(stdout).toContain('list');
expect(stdout).toContain('validate');
});
// ── unknown command ──
it('unknown command shows error', async () => {
const { stderr, code } = await runCli(['nonexistent-command-xyz']);
expect(code).toBe(1);
});
});
+48
View File
@@ -0,0 +1,48 @@
/**
* E2E tests for output format rendering.
* Uses hackernews (public, fast) as a stable data source.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
const FORMATS = ['json', 'yaml', 'csv', 'md'] as const;
describe('output formats E2E', () => {
for (const fmt of FORMATS) {
it(`hackernews top -f ${fmt} produces valid output`, async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '2', '-f', fmt]);
expect(code).toBe(0);
expect(stdout.trim().length).toBeGreaterThan(0);
if (fmt === 'json') {
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBe(2);
}
if (fmt === 'yaml') {
expect(stdout).toContain('title:');
}
if (fmt === 'csv') {
// CSV should have a header row + data rows
const lines = stdout.trim().split('\n');
expect(lines.length).toBeGreaterThanOrEqual(2);
}
if (fmt === 'md') {
// Markdown table should have pipe characters
expect(stdout).toContain('|');
}
}, 30_000);
}
it('list -f csv produces valid csv', async () => {
const { stdout, code } = await runCli(['list', '-f', 'csv']);
expect(code).toBe(0);
const lines = stdout.trim().split('\n');
// Header + many data lines
expect(lines.length).toBeGreaterThan(50);
});
});
+56
View File
@@ -0,0 +1,56 @@
/**
* E2E tests for public API commands (browser: false).
* These commands use Node.js fetch directly — no browser needed.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from './helpers.js';
describe('public commands E2E', () => {
// ── hackernews ──
it('hackernews top returns structured data', async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBe(3);
expect(data[0]).toHaveProperty('title');
expect(data[0]).toHaveProperty('score');
expect(data[0]).toHaveProperty('rank');
}, 30_000);
it('hackernews top respects --limit', async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '1', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBe(1);
}, 30_000);
// ── v2ex (public API, browser: false) ──
it('v2ex hot returns topics', async () => {
const { stdout, code } = await runCli(['v2ex', 'hot', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
it('v2ex latest returns topics', async () => {
const { stdout, code } = await runCli(['v2ex', 'latest', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(Array.isArray(data)).toBe(true);
expect(data.length).toBeGreaterThanOrEqual(1);
}, 30_000);
it('v2ex topic returns topic detail', async () => {
// Topic 1000001 is a well-known V2EX topic
const { stdout, code } = await runCli(['v2ex', 'topic', '--id', '1000001', '-f', 'json']);
// May fail if V2EX rate-limits, but should return structured data
if (code === 0) {
const data = parseJsonOutput(stdout);
expect(data).toBeDefined();
}
}, 30_000);
});
+72
View File
@@ -0,0 +1,72 @@
/**
* Smoke tests for external API health.
* Only run on schedule or manual dispatch — NOT on every push/PR.
* These verify that external APIs haven't changed their structure.
*/
import { describe, it, expect } from 'vitest';
import { runCli, parseJsonOutput } from '../e2e/helpers.js';
describe('API health smoke tests', () => {
// ── Public API commands (should always work) ──
it('hackernews API is responsive and returns expected structure', async () => {
const { stdout, code } = await runCli(['hackernews', 'top', '--limit', '5', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBe(5);
for (const item of data) {
expect(item).toHaveProperty('title');
expect(item).toHaveProperty('score');
expect(item).toHaveProperty('author');
expect(item).toHaveProperty('rank');
}
}, 30_000);
it('v2ex hot API is responsive', async () => {
const { stdout, code } = await runCli(['v2ex', 'hot', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBeGreaterThanOrEqual(1);
expect(data[0]).toHaveProperty('title');
}, 30_000);
it('v2ex latest API is responsive', async () => {
const { stdout, code } = await runCli(['v2ex', 'latest', '--limit', '3', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
expect(data.length).toBeGreaterThanOrEqual(1);
}, 30_000);
it('v2ex topic API is responsive', async () => {
const { stdout, code } = await runCli(['v2ex', 'topic', '--id', '1000001', '-f', 'json']);
if (code === 0) {
const data = parseJsonOutput(stdout);
expect(data).toBeDefined();
}
}, 30_000);
// ── Validate all adapters ──
it('all adapter definitions are valid', async () => {
const { stdout, code } = await runCli(['validate']);
expect(code).toBe(0);
expect(stdout).toContain('PASS');
});
// ── Command registry integrity ──
it('all expected sites are registered', async () => {
const { stdout, code } = await runCli(['list', '-f', 'json']);
expect(code).toBe(0);
const data = parseJsonOutput(stdout);
const sites = new Set(data.map((d: any) => d.site));
// Verify all 17 sites are present
for (const expected of [
'hackernews', 'bbc', 'bilibili', 'v2ex', 'weibo', 'zhihu',
'twitter', 'reddit', 'xueqiu', 'reuters', 'youtube',
'smzdm', 'boss', 'ctrip', 'coupang', 'xiaohongshu',
'yahoo-finance',
]) {
expect(sites.has(expected)).toBe(true);
}
});
});