Compare commits
96 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ba67a3e086 | |||
| ed1a61a445 | |||
| fe82b3882f | |||
| 4d036a5364 | |||
| a23de8fe7d | |||
| b8f1abc3a1 | |||
| aa3edfefc0 | |||
| b657c946a2 | |||
| bb137ce901 | |||
| 7d7203891f | |||
| 081efe37f7 | |||
| 777b882040 | |||
| eead9e0aa5 | |||
| a21cc5e9f0 | |||
| abd46bccba | |||
| 098c7f4f92 | |||
| d721eb6c6c | |||
| 341bb87e09 | |||
| 127dc3edea | |||
| f88e7569e9 | |||
| 9c2a777d11 | |||
| 773178345d | |||
| 1ddca55b4a | |||
| 4818871309 | |||
| c908d9cd47 | |||
| 9b425c7550 | |||
| 12443f049e | |||
| ee0c2b65ea | |||
| 01057527f5 | |||
| 67fb022f16 | |||
| 7ea3f6d3fb | |||
| 818ae61889 | |||
| 811e4f5c3e | |||
| fea47abcec | |||
| 519b3cfe85 | |||
| 57534cf8b3 | |||
| 62fde40aab | |||
| 0204dbb018 | |||
| 1858eace6e | |||
| 0fbeb2d77f | |||
| 33ca81b785 | |||
| 4c3cd3878e | |||
| 5f541ea42b | |||
| ca2165cbc7 | |||
| d0803857f1 | |||
| 701d9e859f | |||
| 59d41f28e1 | |||
| fcdc15d385 | |||
| 030adc9341 | |||
| 62e3c55993 | |||
| 8e37f66e53 | |||
| 770c28301a | |||
| cf79ec5c23 | |||
| 8c00ad9f02 | |||
| 3eb2e88c85 | |||
| b280f19321 | |||
| a32b65be4a | |||
| ab7eca35e4 | |||
| 107ed28449 | |||
| 79b4e069f0 | |||
| d2b563e55b | |||
| f8e9b08223 | |||
| 1ae1c82c4a | |||
| 440c001a20 | |||
| 5925849414 | |||
| d8d9643e89 | |||
| bb5c2b1fc6 | |||
| f44fcd512b | |||
| bcaf6121b8 | |||
| 812f27a05e | |||
| ab0af2de5c | |||
| 5c655ee3c8 | |||
| 0b15561025 | |||
| f9857f8c7b | |||
| c9e29d9f22 | |||
| 75ddb6319b | |||
| 5ec34ebc53 | |||
| 959ec5fe1c | |||
| dbcfbc3bb6 | |||
| 5ae9658a21 | |||
| f415e829a9 | |||
| 210e6fabb7 | |||
| 73a8508972 | |||
| 0fc3bc2d85 | |||
| bc42c8b258 | |||
| c4e7a94bc4 | |||
| 3b0dcfddf0 | |||
| 4f13484aa3 | |||
| 5ab3f5d7d5 | |||
| 55c3259f28 | |||
| 70bd87b98c | |||
| ea0cf4d0b0 | |||
| 9919f03aed | |||
| 3834292871 | |||
| 53b06db53d | |||
| f79d5ab838 |
@@ -1,249 +0,0 @@
|
||||
---
|
||||
name: cross-project-adapter-migration
|
||||
description: "Cross-project CLI command migration workflow for opencli. Use when importing commands from external CLI projects (python/node) like rdt-cli, twitter-cli, etc. Covers: source analysis → gap matrix → batch migration → README/SKILL.md update."
|
||||
---
|
||||
|
||||
# Cross-Project Adapter Migration
|
||||
|
||||
> 从外部 CLI 项目(Python/Node/Go 等)批量迁移命令到 opencli 的标准化流程。
|
||||
|
||||
## When to Use
|
||||
|
||||
- 用户说"把 xxx-cli 的命令迁移过来"
|
||||
- 用户说"看看 xxx 项目有什么可以借鉴的"
|
||||
- 用户说"对齐 xxx-cli 的功能"
|
||||
- 在为新平台扩展 opencli 时,发现已有第三方 CLI 工具
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- 熟悉 [CLI-EXPLORER.md](file:///Users/jakevin/code/opencli/CLI-EXPLORER.md)(adapter 开发决策树)
|
||||
- 熟悉 [SKILL.md](file:///Users/jakevin/code/opencli/SKILL.md)(命令参考 & 模板)
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: 源项目分析
|
||||
|
||||
### 1.1 克隆 & 理解源项目
|
||||
|
||||
```bash
|
||||
# 克隆源项目到 /tmp 做分析
|
||||
git clone <source_repo_url> /tmp/<source-cli>
|
||||
```
|
||||
|
||||
分析重点:
|
||||
- **命令列表**:找到所有可用命令(查看 CLI 入口文件、help 输出或 README)
|
||||
- **认证方式**:Cookie?API Key?OAuth?浏览器自动化?
|
||||
- **数据源**:公开 API?GraphQL?页面抓取?
|
||||
- **输出字段**:每个命令返回哪些数据字段
|
||||
|
||||
### 1.2 生成命令清单
|
||||
|
||||
列出源项目所有命令,包括:
|
||||
|
||||
| 命令 | 类型 | API/方法 | 输出字段 |
|
||||
|------|------|---------|---------|
|
||||
| `xxx feed` | Read | `GET /api/feed` | title, author, time |
|
||||
| `xxx post` | Write | `POST /api/tweet` | status, id |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: 功能对比矩阵
|
||||
|
||||
### 2.1 查看 opencli 现有命令
|
||||
|
||||
```bash
|
||||
ls src/clis/<site>/ # 查看已有适配器
|
||||
opencli list | grep <site> # 确认已注册命令
|
||||
```
|
||||
|
||||
### 2.2 生成对比矩阵
|
||||
|
||||
对每个源项目命令,标注三种状态:
|
||||
|
||||
| 功能 | 源项目 | opencli 现有 | 行动 |
|
||||
|------|--------|-------------|------|
|
||||
| feed | ✅ `xxx feed` | ❌ 无 | ✅ **新增** |
|
||||
| search | ✅ `xxx search` | ✅ `search.ts` | ❌ 已有,跳过 |
|
||||
| hot | ✅ `xxx hot` | ⚠️ `hot.yaml`(不完整) | ✅ **增强** |
|
||||
| like | ✅ `xxx like` | ✅ `like.ts` | ❌ 已有,跳过 |
|
||||
|
||||
### 2.3 筛选迁移目标
|
||||
|
||||
去掉已有的、低价值的,保留高价值缺失命令,按 Read/Write 分类:
|
||||
|
||||
**筛选原则**:
|
||||
- ✅ 高使用频率的命令优先
|
||||
- ✅ 已有但不完整的命令标记为"增强"
|
||||
- ❌ 源项目特有但 opencli 架构不支持的功能(如需要持久化存储的)跳过
|
||||
- ❌ 与现有功能完全重复的跳过
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: 批量实现
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 实现前必须查阅 [CLI-EXPLORER.md](file:///Users/jakevin/code/opencli/CLI-EXPLORER.md) 确认策略选择。
|
||||
|
||||
### 3.1 选择实现方式
|
||||
|
||||
基于决策树分类:
|
||||
|
||||
| 类别 | 方式 | 适用条件 |
|
||||
|------|------|---------|
|
||||
| **Read + 简单 API** | YAML pipeline | 纯 fetch/select/map,无复杂 JS |
|
||||
| **Read + GraphQL/分页/签名** | TypeScript adapter | 需要 JS 逻辑 |
|
||||
| **Write 操作** | TypeScript + `Strategy.UI` | 点击/输入等 DOM 操作 |
|
||||
| **Write + API** | TypeScript + `Strategy.COOKIE/HEADER` | 直接 POST API |
|
||||
|
||||
### 3.2 实现顺序
|
||||
|
||||
**先 Read 后 Write,先 YAML 后 TS**:
|
||||
|
||||
1. **Phase A**: YAML Read 适配器(最快,通常每个 10-20 行)
|
||||
2. **Phase B**: TS Read 适配器(需要 evaluate/intercept 的)
|
||||
3. **Phase C**: TS Write 适配器(需 UI 自动化或 POST API)
|
||||
|
||||
### 3.3 实现模板
|
||||
|
||||
#### YAML Read 适配器模板(Cookie 策略)
|
||||
|
||||
```yaml
|
||||
site: <site>
|
||||
name: <command>
|
||||
description: <描述>
|
||||
domain: www.<site>.com
|
||||
strategy: cookie
|
||||
browser: true
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.<site>.com
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const res = await fetch('<api_endpoint>', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return (d.data?.items || []).map(item => ({
|
||||
title: item.title,
|
||||
// ... map source fields
|
||||
}));
|
||||
})()
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title]
|
||||
```
|
||||
|
||||
#### TS Write 适配器模板(UI 策略)
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: '<site>',
|
||||
name: '<command>',
|
||||
description: '<描述>',
|
||||
strategy: Strategy.UI,
|
||||
args: [{ name: 'target', required: true, help: '<参数说明>' }],
|
||||
columns: ['status', 'message'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto(`https://www.<site>.com/${kwargs.target}`);
|
||||
await page.wait({ text: '<expected_text>', timeout: 10 });
|
||||
|
||||
// 获取 snapshot 找到目标按钮
|
||||
const snapshot = await page.accessibility.snapshot();
|
||||
// 点击按钮 ...
|
||||
|
||||
return [{ status: 'success', message: '<action> completed' }];
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### 3.4 公共模式复用
|
||||
|
||||
迁移过程中如果发现多个适配器共享逻辑,考虑提取到 `src/clis/<site>/utils.ts` 工具文件:
|
||||
|
||||
```typescript
|
||||
// src/clis/<site>/utils.ts
|
||||
export async function fetchWithAuth(page, url) { ... }
|
||||
export function parseItem(raw) { ... }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: 验证 & 发布
|
||||
|
||||
### 4.1 构建验证
|
||||
|
||||
```bash
|
||||
npx tsc --noEmit # TypeScript 编译检查
|
||||
opencli list | grep <site> # 确认所有命令已注册
|
||||
```
|
||||
|
||||
### 4.2 运行验证(关键!)
|
||||
|
||||
每个新命令必须实际运行:
|
||||
|
||||
```bash
|
||||
# Read 命令
|
||||
opencli <site> <command> --limit 3 -f json
|
||||
opencli <site> <command> --limit 3 -v # verbose 查看 pipeline
|
||||
|
||||
# Write 命令(谨慎!会实际操作)
|
||||
opencli <site> <command> <test_target>
|
||||
```
|
||||
|
||||
### 4.3 更新文档
|
||||
|
||||
迁移完成后必须更新以下文件:
|
||||
|
||||
1. **README.md** — 在对应平台区域添加新命令示例
|
||||
2. **SKILL.md** — 在 Commands Reference 中添加新命令
|
||||
|
||||
### 4.4 提交 & 推送
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "feat(<site>): migrate <N> commands from <source-cli>
|
||||
|
||||
- Phase A: <N> YAML adapters (read operations)
|
||||
- Phase B: <N> TS adapters (write operations)
|
||||
- Source: <source_repo_url>"
|
||||
git push
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] 源项目命令清单已生成
|
||||
- [ ] 对比矩阵已确认,高价值缺失命令已筛选
|
||||
- [ ] 用户确认迁移范围
|
||||
- [ ] Phase A: YAML Read 适配器已完成
|
||||
- [ ] Phase B: TS Read 适配器已完成
|
||||
- [ ] Phase C: TS Write 适配器已完成
|
||||
- [ ] `npx tsc --noEmit` 编译通过
|
||||
- [ ] 所有新命令已实际运行验证
|
||||
- [ ] README.md 已更新
|
||||
- [ ] SKILL.md 已更新
|
||||
- [ ] 已 commit + push
|
||||
|
||||
## 实战案例参考
|
||||
|
||||
### rdt-cli → opencli Reddit(2026-03-16)
|
||||
|
||||
- **源项目**: `rdt-cli`(25 个 Python 命令)
|
||||
- **筛选结果**: 13 个高价值命令
|
||||
- **实现**: 7 个 YAML(read) + 6 个 TS(write)
|
||||
- **产出**: +11 文件,+767 行代码,Reddit 适配器从 4 → 15(+275%)
|
||||
|
||||
### twitter-cli → opencli Twitter(2026-03-16)
|
||||
|
||||
- **源项目**: `twitter-cli`(20+ Python 命令)
|
||||
- **筛选结果**: 11 个待实现
|
||||
- **策略**: Read 用 `Strategy.COOKIE` + GraphQL fetch,Write 用 `Strategy.UI`
|
||||
@@ -1,54 +0,0 @@
|
||||
---
|
||||
description: Migrate commands from an external CLI project into opencli adapters
|
||||
---
|
||||
|
||||
// turbo-all
|
||||
|
||||
## Steps
|
||||
|
||||
1. Clone the source CLI project for analysis:
|
||||
```bash
|
||||
git clone <source_repo_url> /tmp/<source-cli>
|
||||
```
|
||||
|
||||
2. Analyze source project: list all commands, auth method, API endpoints, and output fields.
|
||||
|
||||
3. Check existing opencli adapters for the target site:
|
||||
```bash
|
||||
ls src/clis/<site>/
|
||||
opencli list | grep <site>
|
||||
```
|
||||
|
||||
4. Generate a comparison matrix table (source commands vs opencli existing). Mark each as: ✅ **New** / ✅ **Enhance** / ❌ **Skip**. Ask user to confirm which commands to migrate.
|
||||
|
||||
5. Implement YAML Read adapters first (highest ROI, 10-20 lines each). Place files in `src/clis/<site>/<name>.yaml`.
|
||||
|
||||
6. Implement TS Read adapters for complex cases (GraphQL, pagination, signing). Place files in `src/clis/<site>/<name>.ts`.
|
||||
|
||||
7. Implement TS Write adapters using `Strategy.UI` or `Strategy.COOKIE`. Place files in `src/clis/<site>/<name>.ts`.
|
||||
|
||||
8. Verify build:
|
||||
```bash
|
||||
npx tsc --noEmit
|
||||
```
|
||||
|
||||
9. Verify all commands are registered:
|
||||
```bash
|
||||
opencli list | grep <site>
|
||||
```
|
||||
|
||||
10. Run each new command to verify it works:
|
||||
```bash
|
||||
opencli <site> <command> --limit 3 -f json
|
||||
```
|
||||
|
||||
11. Update README.md with new command examples in the appropriate platform section.
|
||||
|
||||
12. Update SKILL.md Commands Reference with new commands.
|
||||
|
||||
13. Commit and push:
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "feat(<site>): migrate <N> commands from <source-cli>"
|
||||
git push
|
||||
```
|
||||
@@ -39,13 +39,15 @@ jobs:
|
||||
run: npm run build
|
||||
|
||||
# ── Unit tests (vitest shard) ──
|
||||
# PR: ubuntu + Node 22 only (fast feedback, 2 jobs).
|
||||
# Push to main/dev: full matrix for cross-platform/cross-version coverage (12 jobs).
|
||||
unit-test:
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, macos-latest, windows-latest]
|
||||
node-version: ['20', '22']
|
||||
os: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && fromJSON('["ubuntu-latest","macos-latest","windows-latest"]') || fromJSON('["ubuntu-latest"]') }}
|
||||
node-version: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && fromJSON('["20","22"]') || fromJSON('["22"]') }}
|
||||
shard: [1, 2]
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
@@ -82,12 +84,9 @@ jobs:
|
||||
- name: Run unit tests under Bun
|
||||
run: bun vitest run --project unit --reporter=verbose
|
||||
|
||||
# Adapter tests are pure unit tests — OS doesn't affect results.
|
||||
adapter-test:
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, macos-latest, windows-latest]
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
@@ -22,3 +22,5 @@ docs/.vitepress/cache
|
||||
|
||||
# Database files
|
||||
*.db
|
||||
autoresearch/results/
|
||||
extension/dist/
|
||||
|
||||
@@ -1,5 +1,89 @@
|
||||
# Changelog
|
||||
|
||||
## [1.6.1](https://github.com/jackwener/opencli/compare/v1.6.0...v1.6.1) (2026-04-02)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* sync package-lock.json version with package.json ([#698](https://github.com/jackwener/opencli/issues/698))
|
||||
|
||||
|
||||
## [1.6.0](https://github.com/jackwener/opencli/compare/v1.5.9...v1.6.0) (2026-04-02)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **opencli-operate:** add browser control commands for Claude Code skill ([#614](https://github.com/jackwener/opencli/issues/614))
|
||||
* **docs:** add tab completion to getting started guides ([#658](https://github.com/jackwener/opencli/issues/658))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **twitter:** resolve article ID to tweet ID before GraphQL query ([#688](https://github.com/jackwener/opencli/issues/688))
|
||||
* **xiaohongshu:** clarify empty note shell hint ([#686](https://github.com/jackwener/opencli/issues/686))
|
||||
* **skills:** add YAML frontmatter for discovery and improve descriptions ([#694](https://github.com/jackwener/opencli/issues/694))
|
||||
|
||||
|
||||
### Refactoring
|
||||
|
||||
* centralize daemon transport client ([#692](https://github.com/jackwener/opencli/issues/692))
|
||||
|
||||
|
||||
## [1.5.9](https://github.com/jackwener/opencli/compare/v1.5.8...v1.5.9) (2026-04-02)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **amazon:** add browser adapter — bestsellers, search, product, offer, discussion ([#659](https://github.com/jackwener/opencli/issues/659))
|
||||
* **skills:** create skills/ directory structure with opencli-usage, opencli-explorer, opencli-oneshot ([#670](https://github.com/jackwener/opencli/issues/670))
|
||||
* **record:** add minimal record write candidates ([#665](https://github.com/jackwener/opencli/issues/665))
|
||||
|
||||
|
||||
### Refactoring
|
||||
|
||||
* src cleanup — deduplicate errors, cache VM, extract BasePage, remove Playwright MCP legacy ([#667](https://github.com/jackwener/opencli/issues/667))
|
||||
* remove bind-current, restore owned-only browser automation model ([#664](https://github.com/jackwener/opencli/issues/664))
|
||||
|
||||
|
||||
### Chores
|
||||
|
||||
* remove .agents directory ([#668](https://github.com/jackwener/opencli/issues/668))
|
||||
|
||||
|
||||
## [1.5.8](https://github.com/jackwener/opencli/compare/v1.5.7...v1.5.8) (2026-04-01)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **extension:** avoid mutating healthy tabs before debugger attach and add regression coverage ([#662](https://github.com/jackwener/opencli/issues/662))
|
||||
|
||||
|
||||
## [1.5.7](https://github.com/jackwener/opencli/compare/v1.5.6...v1.5.7) (2026-04-01)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **daemon:** replace 5min idle timeout with long-lived daemon model (4h default, dual-condition exit) ([#641](https://github.com/jackwener/opencli/issues/641))
|
||||
* **daemon:** add `opencli daemon status/stop/restart` CLI commands ([#641](https://github.com/jackwener/opencli/issues/641))
|
||||
* **youtube:** add search filters — `--type` shorts/video/channel, `--upload`, `--sort` ([#616](https://github.com/jackwener/opencli/issues/616))
|
||||
* **notebooklm:** add read commands and compatibility layer ([#622](https://github.com/jackwener/opencli/issues/622))
|
||||
* **instagram:** add media download command ([#623](https://github.com/jackwener/opencli/issues/623))
|
||||
* **stealth:** harden CDP debugger detection countermeasures ([#644](https://github.com/jackwener/opencli/issues/644))
|
||||
* **v2ex:** add id, node, url, content, member fields to topic output ([#646](https://github.com/jackwener/opencli/issues/646), [#648](https://github.com/jackwener/opencli/issues/648))
|
||||
* **electron:** auto-launcher — zero-config CDP connection ([#653](https://github.com/jackwener/opencli/issues/653))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **douyin:** repair creator draft flow — switch from broken API pipeline to UI-driven approach ([#640](https://github.com/jackwener/opencli/issues/640))
|
||||
* **douyin:** support current creator API response shapes for activities, profile, collections, hashtag, videos ([#618](https://github.com/jackwener/opencli/issues/618))
|
||||
* **bilibili:** distinguish login-gated subtitles from empty results ([#645](https://github.com/jackwener/opencli/issues/645))
|
||||
* **facebook:** avoid in-page redirect in search — use navigate step instead of window.location.href ([#642](https://github.com/jackwener/opencli/issues/642))
|
||||
* **substack:** update selectors for DOM redesign ([#624](https://github.com/jackwener/opencli/issues/624))
|
||||
* **weread:** recover book details from cached shelf fallback ([#628](https://github.com/jackwener/opencli/issues/628))
|
||||
* **docs:** use relative links in adapter index ([#629](https://github.com/jackwener/opencli/issues/629))
|
||||
|
||||
|
||||
## [1.4.1](https://github.com/jackwener/opencli/compare/v1.4.0...v1.4.1) (2026-03-25)
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# OpenCLI
|
||||
|
||||
> **Make any website, Electron App, or Local Tool your CLI.**
|
||||
> **Make any website, Electron App, or Local Tool your CLI.**
|
||||
> Zero risk · Reuse Chrome login · AI-powered discovery · Universal CLI Hub
|
||||
|
||||
[](./README.zh-CN.md)
|
||||
@@ -10,17 +10,19 @@
|
||||
|
||||
A CLI tool that turns **any website**, **Electron app**, or **local CLI tool** into a command-line interface — Bilibili, Zhihu, 小红书, Twitter/X, Reddit, YouTube, Antigravity, `gh`, `docker`, and [many more](#built-in-commands) — powered by browser session reuse and AI-native discovery.
|
||||
|
||||
**Built for AI Agents**: Simply configure an instruction in your global `AGENT.md` or `.cursorrules` guiding the AI to execute `opencli list` via Bash to discover available tools. Register your favorite local CLIs (`opencli register mycli`), and the AI will automatically learn how to invoke all your tools perfectly!
|
||||
**Built for AI Agents** — Configure an instruction in your `AGENT.md` or `.cursorrules` to run `opencli list` via Bash. The AI will automatically discover and invoke all available tools.
|
||||
|
||||
**CLI All Electron Apps! The Most Powerful Update Has Arrived!**
|
||||
Turn ANY Electron application into a CLI tool! Recombine, script, and extend applications like Antigravity Ultra seamlessly. Now AI can control itself natively. Unlimited possibilities await!
|
||||
**CLI Hub** — Register any local CLI (`opencli register mycli`) so AI agents can discover and call it alongside built-in commands. Auto-installs missing tools via your package manager (e.g. if `gh` isn't installed, `opencli gh ...` runs `brew install gh` first then re-executes seamlessly).
|
||||
|
||||
**CLI for Electron Apps** — Turn any Electron application into a CLI tool. Recombine, script, and extend apps like Antigravity Ultra from the terminal. AI agents can now control other AI apps natively.
|
||||
|
||||
---
|
||||
|
||||
## Highlights
|
||||
|
||||
- **CLI All Electron** — CLI-ify apps like Antigravity Ultra! Now AI can control itself natively using cc/openclaw!
|
||||
- **CLI All Electron** — CLI-ify apps like Antigravity Ultra! Now AI can control itself natively.
|
||||
- **Account-safe** — Reuses Chrome's logged-in state; your credentials never leave the browser.
|
||||
- **Anti-detection built-in** — Patches `navigator.webdriver`, stubs `window.chrome`, fakes plugin lists, cleans ChromeDriver/Playwright globals, and strips CDP frames from Error stack traces. Extensive anti-fingerprinting and risk-control evasion measures baked in at every layer.
|
||||
- **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies.
|
||||
- **External CLI Hub** — Discover, auto-install, and passthrough commands to any external CLI (gh, obsidian, docker, etc). Zero setup.
|
||||
- **Self-healing setup** — `opencli doctor` diagnoses and auto-starts the daemon, extension, and live browser connectivity.
|
||||
@@ -43,193 +45,147 @@ There are many great browser automation tools. Here's when opencli is the right
|
||||
|
||||
- **Zero LLM cost** — No tokens consumed at runtime. Run 10,000 times and pay nothing.
|
||||
- **Deterministic** — Same command, same output schema, every time. Pipeable, scriptable, CI-friendly.
|
||||
- **Broad coverage** — 50+ sites across global and Chinese platforms (Bilibili, Zhihu, Xiaohongshu, Reddit, HackerNews, and more), plus desktop Electron apps via CDP.
|
||||
- **Broad coverage** — 70+ sites across global and Chinese platforms (Bilibili, Zhihu, Xiaohongshu, Reddit, HackerNews, and more), plus desktop Electron apps via CDP.
|
||||
|
||||
> For a detailed comparison with Browser-Use, Crawl4AI, Firecrawl, and others, see the [Comparison Guide](./docs/comparison.md).
|
||||
|
||||
## Prerequisites
|
||||
---
|
||||
|
||||
- **Node.js**: >= 20.0.0 (or **Bun** >= 1.0 — see [Runtime Support](#runtime-support) below)
|
||||
- **Chrome** running **and logged into the target site** (e.g. bilibili.com, zhihu.com, xiaohongshu.com).
|
||||
## Quick Start
|
||||
|
||||
> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands. If you get empty data or errors, check your login status first.
|
||||
### 1. Install Browser Bridge Extension
|
||||
|
||||
### Runtime Support
|
||||
> OpenCLI connects to your browser through a lightweight **Browser Bridge** Chrome Extension + micro-daemon (zero config, auto-start).
|
||||
|
||||
OpenCLI works with both **Node.js** (≥ 20) and **Bun** (≥ 1.0). All commands and adapters are runtime-agnostic.
|
||||
|
||||
```bash
|
||||
# Development with Bun (faster startup)
|
||||
npm run dev:bun
|
||||
|
||||
# Run the built CLI with Bun
|
||||
npm run start:bun
|
||||
|
||||
# Run unit tests under Bun
|
||||
npm run test:bun
|
||||
|
||||
# Run E2E tests with Bun as the runtime
|
||||
OPENCLI_TEST_RUNTIME=bun npm run test:e2e
|
||||
```
|
||||
|
||||
Use `opencli doctor` to check your current runtime — it displays the active engine (e.g. `node v22.13.0` or `bun 1.1.42`).
|
||||
|
||||
OpenCLI connects to your browser through a lightweight **Browser Bridge** Chrome Extension + micro-daemon (zero config, auto-start).
|
||||
|
||||
### Browser Bridge Extension Setup
|
||||
|
||||
You can install the extension via either method:
|
||||
|
||||
**Method 1: Download Pre-built Release (Recommended)**
|
||||
1. Go to the GitHub [Releases page](https://github.com/jackwener/opencli/releases) and download the latest `opencli-extension.zip`.
|
||||
2. Unzip the file and open `chrome://extensions`, enable **Developer mode** (top-right toggle).
|
||||
3. Click **Load unpacked** and select the unzipped folder.
|
||||
|
||||
**Method 2: Load Source (For Developers)**
|
||||
1. Open `chrome://extensions` and enable **Developer mode**.
|
||||
2. Click **Load unpacked** and select the `extension/` directory from this repository.
|
||||
### 2. Install OpenCLI
|
||||
|
||||
That's it! The daemon auto-starts when you run any browser command. No tokens, no manual configuration.
|
||||
|
||||
> **Tip**: Use `opencli doctor` for ongoing diagnosis:
|
||||
> ```bash
|
||||
> opencli doctor # Check extension + daemon connectivity
|
||||
> ```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Install via npm (recommended)
|
||||
**Install via npm (recommended)**
|
||||
|
||||
```bash
|
||||
npm install -g @jackwener/opencli
|
||||
```
|
||||
|
||||
Then use directly:
|
||||
### 3. Verify & Try
|
||||
|
||||
```bash
|
||||
opencli list # See all commands
|
||||
opencli list -f yaml # List commands as YAML
|
||||
opencli hackernews top --limit 5 # Public API, no browser
|
||||
opencli bilibili hot --limit 5 # Browser command
|
||||
opencli zhihu hot -f json # JSON output
|
||||
opencli zhihu hot -f yaml # YAML output
|
||||
opencli doctor # Check extension + daemon connectivity
|
||||
opencli daemon status # Check daemon state (PID, uptime, memory)
|
||||
```
|
||||
|
||||
### Install from source (for developers)
|
||||
**Try it out:**
|
||||
|
||||
```bash
|
||||
git clone git@github.com:jackwener/opencli.git
|
||||
cd opencli
|
||||
npm install
|
||||
npm run build
|
||||
npm link # Link binary globally
|
||||
opencli list # Now you can use it anywhere!
|
||||
opencli list # See all commands
|
||||
opencli hackernews top --limit 5 # Public API, no browser needed
|
||||
opencli bilibili hot --limit 5 # Browser command (requires Extension)
|
||||
```
|
||||
|
||||
### 4. Browser Automation — Make Websites Accessible for AI Agents
|
||||
|
||||
#### AI Agent Quickstart (1 step)
|
||||
|
||||
Point your AI agent (Claude Code, Cursor) to [`skills/opencli-operate/SKILL.md`](./skills/opencli-operate/SKILL.md). It has everything needed.
|
||||
|
||||
#### Human Quickstart (3 steps)
|
||||
|
||||
```bash
|
||||
opencli operate open https://news.ycombinator.com # 1. Open a page
|
||||
opencli operate state # 2. See interactive elements
|
||||
opencli operate eval "document.title" # 3. Extract data
|
||||
```
|
||||
|
||||
More commands: `click`, `type`, `select`, `keys`, `wait`, `get`, `screenshot`, `scroll`, `back`, `eval`, `network`, `init`, `verify`, `close`.
|
||||
|
||||
See [`skills/opencli-operate/SKILL.md`](./skills/opencli-operate/SKILL.md) for full documentation.
|
||||
|
||||
### Update
|
||||
|
||||
```bash
|
||||
npm install -g @jackwener/opencli@latest
|
||||
```
|
||||
|
||||
### Install AI Skills
|
||||
|
||||
OpenCLI provides [skills](./skills/) for AI agents (Claude Code, etc.):
|
||||
|
||||
```bash
|
||||
# Install all OpenCLI skills
|
||||
npx skills add jackwener/opencli
|
||||
|
||||
# Or install specific skills
|
||||
npx skills add jackwener/opencli --skill opencli-usage # Command reference
|
||||
npx skills add jackwener/opencli --skill opencli-operate # Browser automation for AI agents
|
||||
npx skills add jackwener/opencli --skill opencli-explorer # Adapter development guide
|
||||
npx skills add jackwener/opencli --skill opencli-oneshot # Quick command reference
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### For Developers
|
||||
|
||||
**Install from source**
|
||||
|
||||
```bash
|
||||
git clone git@github.com:jackwener/opencli.git && cd opencli && npm install && npm run build && npm link
|
||||
```
|
||||
|
||||
**Load Source Browser Bridge Extension**
|
||||
|
||||
1. Open `chrome://extensions` and enable **Developer mode** (top-right toggle).
|
||||
2. Click **Load unpacked** and select the `extension/` directory from this repository.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js**: >= 20.0.0 (or **Bun** >= 1.0)
|
||||
- **Chrome** running **and logged into the target site** (e.g. bilibili.com, zhihu.com, xiaohongshu.com).
|
||||
|
||||
> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands. If you get empty data or errors, check your login status first.
|
||||
|
||||
## Built-in Commands
|
||||
|
||||
Run `opencli list` for the live registry.
|
||||
| Site | Commands |
|
||||
|------|----------|
|
||||
| **xiaohongshu** | `search` `note` `comments` `feed` `user` `download` `publish` `notifications` `creator-notes` `creator-notes-summary` `creator-note-detail` `creator-profile` `creator-stats` |
|
||||
| **bilibili** | `hot` `search` `history` `feed` `ranking` `download` `comments` `dynamic` `favorite` `following` `me` `subtitle` `user-videos` |
|
||||
| **tieba** | `hot` `posts` `search` `read` |
|
||||
| **twitter** | `trending` `search` `timeline` `bookmarks` `post` `download` `profile` `article` `like` `likes` `notifications` `reply` `reply-dm` `thread` `follow` `unfollow` `followers` `following` `block` `unblock` `bookmark` `unbookmark` `delete` `hide-reply` `accept` |
|
||||
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `user` `user-posts` `user-comments` `read` `save` `saved` `subscribe` `upvote` `upvoted` `comment` |
|
||||
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` |
|
||||
| **gemini** | `new` `ask` `image` |
|
||||
| **notebooklm** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` |
|
||||
| **spotify** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` |
|
||||
|
||||
| Site | Commands | Mode |
|
||||
|------|----------|------|
|
||||
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | Browser |
|
||||
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | Browser |
|
||||
| **cursor** | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` | Desktop |
|
||||
| **bilibili** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | Browser |
|
||||
| **codex** | `status` `send` `read` `new` `dump` `extract-diff` `model` `ask` `screenshot` `history` `export` | Desktop |
|
||||
| **chatwise** | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` | Desktop |
|
||||
| **doubao** | `status` `new` `send` `read` `ask` | Browser |
|
||||
| **doubao-app** | `status` `new` `send` `read` `ask` `screenshot` `dump` | Desktop |
|
||||
| **notion** | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` | Desktop |
|
||||
| **discord-app** | `status` `send` `read` `channels` `servers` `search` `members` | Desktop |
|
||||
| **v2ex** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | Public / Browser |
|
||||
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | Browser |
|
||||
| **antigravity** | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` | Desktop |
|
||||
| **chatgpt** | `status` `new` `send` `read` `ask` `model` | Desktop |
|
||||
| **xiaohongshu** | `search` `notifications` `feed` `user` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | Browser |
|
||||
| **apple-podcasts** | `search` `episodes` `top` | Public |
|
||||
| **xiaoyuzhou** | `podcast` `podcast-episodes` `episode` | Public |
|
||||
| **zhihu** | `hot` `search` `question` `download` | Browser |
|
||||
| **weixin** | `download` | Browser |
|
||||
| **youtube** | `search` `video` `transcript` | Browser |
|
||||
| **boss** | `search` `detail` `recommend` `joblist` `greet` `batchgreet` `send` `chatlist` `chatmsg` `invite` `mark` `exchange` `resume` `stats` | Browser |
|
||||
| **coupang** | `search` `add-to-cart` | Browser |
|
||||
| **bbc** | `news` | Public |
|
||||
| **bloomberg** | `main` `markets` `economics` `industries` `tech` `politics` `businessweek` `opinions` `feeds` `news` | Public / Browser |
|
||||
| **ctrip** | `search` | Browser |
|
||||
| **devto** | `top` `tag` `user` | Public |
|
||||
| **dictionary** | `search` `synonyms` `examples` | Public |
|
||||
| **arxiv** | `search` `paper` | Public |
|
||||
| **paperreview** | `submit` `review` `feedback` | Public |
|
||||
| **wikipedia** | `search` `summary` `random` `trending` | Public |
|
||||
| **hackernews** | `top` `new` `best` `ask` `show` `jobs` `search` `user` | Public |
|
||||
| **jd** | `item` | Browser |
|
||||
| **linkedin** | `search` `timeline` | Browser |
|
||||
| **reuters** | `search` | Browser |
|
||||
| **smzdm** | `search` | Browser |
|
||||
| **web** | `read` | Browser |
|
||||
| **weibo** | `hot` `search` | Browser |
|
||||
| **yahoo-finance** | `quote` | Browser |
|
||||
| **sinafinance** | `news` | 🌐 Public |
|
||||
| **barchart** | `quote` `options` `greeks` `flow` | Browser |
|
||||
| **chaoxing** | `assignments` `exams` | Browser |
|
||||
| **grok** | `ask` | Browser |
|
||||
| **hf** | `top` | Public |
|
||||
| **jike** | `feed` `search` `create` `like` `comment` `repost` `notifications` `post` `topic` `user` | Browser |
|
||||
| **jimeng** | `generate` `history` | Browser |
|
||||
| **yollomi** | `generate` `video` `edit` `upload` `models` `remove-bg` `upscale` `face-swap` `restore` `try-on` `background` `object-remover` | Browser |
|
||||
| **linux-do** | `feed` `categories` `tags` `search` `topic` `user-topics` `user-posts` | Browser |
|
||||
| **stackoverflow** | `hot` `search` `bounties` `unanswered` | Public |
|
||||
| **steam** | `top-sellers` | Public |
|
||||
| **weread** | `shelf` `search` `book` `highlights` `notes` `notebooks` `ranking` | Browser |
|
||||
| **douban** | `search` `top250` `subject` `photos` `download` `marks` `reviews` `movie-hot` `book-hot` | Browser |
|
||||
| **facebook** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` | Browser |
|
||||
| **google** | `news` `search` `suggest` `trends` | Public |
|
||||
| **36kr** | `news` `hot` `search` `article` | Public / Browser |
|
||||
| **imdb** | `search` `title` `top` `trending` `person` `reviews` | Public |
|
||||
| **producthunt** | `posts` `today` `hot` `browse` | Public / Browser |
|
||||
| **instagram** | `explore` `profile` `search` `user` `followers` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `saved` | Browser |
|
||||
| **lobsters** | `hot` `newest` `active` `tag` | Public |
|
||||
| **medium** | `feed` `search` `user` | Browser |
|
||||
| **sinablog** | `hot` `search` `article` `user` | Browser |
|
||||
| **substack** | `feed` `search` `publication` | Browser |
|
||||
| **pixiv** | `ranking` `search` `user` `illusts` `detail` `download` | Browser |
|
||||
| **tiktok** | `explore` `search` `profile` `user` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `live` `notifications` `friends` | Browser |
|
||||
70+ adapters in total — **[→ see all supported sites & commands](./docs/adapters/index.md)**
|
||||
|
||||
## CLI Hub
|
||||
|
||||
### External CLI Hub
|
||||
OpenCLI acts as a universal hub for your existing command-line tools — unified discovery, pure passthrough execution, and auto-install (if a tool isn't installed, OpenCLI runs `brew install <tool>` automatically before re-running the command).
|
||||
|
||||
OpenCLI acts as a universal hub for your existing command-line tools. It provides unified discovery, automatic installation, and pure passthrough execution.
|
||||
|
||||
| External CLI | Description | Commands Example |
|
||||
|--------------|-------------|------------------|
|
||||
| External CLI | Description | Example |
|
||||
|--------------|-------------|---------|
|
||||
| **gh** | GitHub CLI | `opencli gh pr list --limit 5` |
|
||||
| **obsidian** | Obsidian vault management | `opencli obsidian search query="AI"` |
|
||||
| **docker** | Docker command-line interface | `opencli docker ps` |
|
||||
| **readwise** | Readwise & Reader CLI | `opencli readwise login` |
|
||||
| **gws** | Google Workspace CLI — Docs, Sheets, Drive, Gmail, Calendar | `opencli gws docs list` |
|
||||
| **docker** | Docker | `opencli docker ps` |
|
||||
| **lark-cli** | Lark/Feishu — messages, docs, calendar, tasks, 200+ commands | `opencli lark-cli calendar +agenda` |
|
||||
| **dingtalk** | DingTalk — cross-platform CLI for DingTalk's full suite, designed for humans and AI agents | `opencli dingtalk msg send --to user "hello"` |
|
||||
| **wecom** | WeCom/企业微信 — CLI for WeCom open platform, for humans and AI agents | `opencli wecom msg send --to user "hello"` |
|
||||
| **vercel** | Vercel — deploy projects, manage domains, env vars, logs | `opencli vercel deploy --prod` |
|
||||
|
||||
**Zero Configuration**: OpenCLI purely passes your inputs to the underlying binary via standard I/O streams. The external CLI works exactly as it naturally would, maintaining its standard output formats.
|
||||
**Register your own** — add any local CLI so AI agents can discover it via `opencli list`:
|
||||
|
||||
**Auto-Installation**: If you run `opencli gh ...` and `gh` is not installed on your system, OpenCLI will automatically try to install it using your system's package manager (e.g., `brew install gh`) before seamlessly re-running the command.
|
||||
|
||||
**Register Your Own**:
|
||||
Add any local CLI to your OpenCLI registry so AI agents can automatically discover it via the `opencli list` command.
|
||||
```bash
|
||||
opencli register mycli
|
||||
```
|
||||
|
||||
### Desktop App Adapters
|
||||
|
||||
Each desktop adapter has its own detailed documentation with commands reference, setup guide, and examples:
|
||||
|
||||
If you want to add support for a new Electron desktop app, start with [docs/guide/electron-app-cli.md](./docs/guide/electron-app-cli.md) and the deeper [Electron guide](./docs/advanced/electron.md).
|
||||
Control Electron desktop apps directly from the terminal. Each adapter has its own detailed documentation:
|
||||
|
||||
| App | Description | Doc |
|
||||
|-----|-------------|-----|
|
||||
@@ -242,93 +198,73 @@ If you want to add support for a new Electron desktop app, start with [docs/guid
|
||||
| **Discord** | Discord Desktop — messages, channels, servers | [Doc](./docs/adapters/desktop/discord.md) |
|
||||
| **Doubao** | Control Doubao AI desktop app via CDP | [Doc](./docs/adapters/desktop/doubao-app.md) |
|
||||
|
||||
To add a new Electron app, start with [docs/guide/electron-app-cli.md](./docs/guide/electron-app-cli.md).
|
||||
|
||||
## Download Support
|
||||
|
||||
OpenCLI supports downloading images, videos, and articles from supported platforms.
|
||||
|
||||
### Supported Platforms
|
||||
|
||||
| Platform | Content Types | Notes |
|
||||
|----------|---------------|-------|
|
||||
| **xiaohongshu** | Images, Videos | Downloads all media from a note |
|
||||
| **bilibili** | Videos | Requires `yt-dlp` installed |
|
||||
| **twitter** | Images, Videos | Downloads from user media tab or single tweet |
|
||||
| **douban** | Images | Downloads poster / still image lists from movie subjects |
|
||||
| **pixiv** | Images | Downloads original-quality illustrations, supports multi-page works |
|
||||
| **zhihu** | Articles (Markdown) | Exports articles with optional image download |
|
||||
| **weixin** | Articles (Markdown) | Exports WeChat Official Account articles |
|
||||
| **twitter** | Images, Videos | From user media tab or single tweet |
|
||||
| **douban** | Images | Poster / still image lists |
|
||||
| **pixiv** | Images | Original-quality illustrations, multi-page |
|
||||
| **zhihu** | Articles (Markdown) | Exports with optional image download |
|
||||
| **weixin** | Articles (Markdown) | WeChat Official Account articles |
|
||||
|
||||
### Prerequisites
|
||||
|
||||
For video downloads from streaming platforms, you need to install `yt-dlp`:
|
||||
For video downloads, install `yt-dlp` first: `brew install yt-dlp`
|
||||
|
||||
```bash
|
||||
# Install yt-dlp
|
||||
pip install yt-dlp
|
||||
# or
|
||||
brew install yt-dlp
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
|
||||
```bash
|
||||
# Download images/videos from Xiaohongshu note
|
||||
opencli xiaohongshu download abc123 --output ./xhs
|
||||
|
||||
# Download Bilibili video (requires yt-dlp)
|
||||
opencli bilibili download BV1xxx --output ./bilibili
|
||||
opencli bilibili download BV1xxx --quality 1080p # Specify quality
|
||||
|
||||
# Download Twitter media from user
|
||||
opencli twitter download elonmusk --limit 20 --output ./twitter
|
||||
|
||||
# Download single tweet media
|
||||
opencli twitter download --tweet-url "https://x.com/user/status/123" --output ./twitter
|
||||
|
||||
# Download Douban posters / stills
|
||||
opencli douban download 30382501 --output ./douban
|
||||
|
||||
# Export Zhihu article to Markdown
|
||||
opencli zhihu download "https://zhuanlan.zhihu.com/p/xxx" --output ./zhihu
|
||||
|
||||
# Export with local images
|
||||
opencli zhihu download "https://zhuanlan.zhihu.com/p/xxx" --download-images
|
||||
|
||||
# Export WeChat article to Markdown
|
||||
opencli weixin download --url "https://mp.weixin.qq.com/s/xxx" --output ./weixin
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Output Formats
|
||||
|
||||
All built-in commands support `--format` / `-f` with `table`, `json`, `yaml`, `md`, and `csv`.
|
||||
The `list` command supports the same format options, and keeps `--json` for backward compatibility.
|
||||
All built-in commands support `--format` / `-f` with `table` (default), `json`, `yaml`, `md`, and `csv`.
|
||||
|
||||
```bash
|
||||
opencli list -f yaml # Command registry as YAML
|
||||
opencli bilibili hot -f table # Default: rich terminal table
|
||||
opencli bilibili hot -f json # JSON (pipe to jq or LLMs)
|
||||
opencli bilibili hot -f yaml # YAML (human-readable structured output)
|
||||
opencli bilibili hot -f md # Markdown
|
||||
opencli bilibili hot -f csv # CSV
|
||||
opencli bilibili hot -f json # Pipe to jq or LLMs
|
||||
opencli bilibili hot -f csv # Spreadsheet-friendly
|
||||
opencli bilibili hot -v # Verbose: show pipeline debug steps
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
opencli follows Unix `sysexits.h` conventions so it integrates naturally with shell pipelines and CI scripts:
|
||||
|
||||
| Code | Meaning | When |
|
||||
|------|---------|------|
|
||||
| `0` | Success | Command completed normally |
|
||||
| `1` | Generic error | Unexpected / unclassified failure |
|
||||
| `2` | Usage error | Bad arguments or unknown command |
|
||||
| `66` | Empty result | No data returned (`EX_NOINPUT`) |
|
||||
| `69` | Service unavailable | Browser Bridge not connected (`EX_UNAVAILABLE`) |
|
||||
| `75` | Temporary failure | Command timed out — retry (`EX_TEMPFAIL`) |
|
||||
| `77` | Auth required | Not logged in to target site (`EX_NOPERM`) |
|
||||
| `78` | Config error | Missing credentials or bad config (`EX_CONFIG`) |
|
||||
| `130` | Interrupted | Ctrl-C / SIGINT |
|
||||
|
||||
```bash
|
||||
opencli spotify status || echo "exit $?" # 69 if browser not running
|
||||
opencli github issues 2>/dev/null
|
||||
[ $? -eq 77 ] && opencli github auth # auto-auth if not logged in
|
||||
```
|
||||
|
||||
## Plugins
|
||||
|
||||
Extend OpenCLI with community-contributed adapters. Plugins use the same YAML/TS format as built-in commands and are automatically discovered at startup.
|
||||
Extend OpenCLI with community-contributed adapters:
|
||||
|
||||
```bash
|
||||
opencli plugin install github:user/opencli-plugin-my-tool # Install
|
||||
opencli plugin list # List installed
|
||||
opencli plugin update my-tool # Update to latest
|
||||
opencli plugin update --all # Update all installed plugins
|
||||
opencli plugin uninstall my-tool # Remove
|
||||
opencli plugin install github:user/opencli-plugin-my-tool
|
||||
opencli plugin list
|
||||
opencli plugin update --all
|
||||
opencli plugin uninstall my-tool
|
||||
```
|
||||
|
||||
`opencli plugin list` also shows the tracked short commit hash when a plugin version is recorded in `~/.opencli/plugins.lock.json`.
|
||||
|
||||
| Plugin | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| [opencli-plugin-github-trending](https://github.com/ByteYue/opencli-plugin-github-trending) | YAML | GitHub Trending repositories |
|
||||
@@ -339,53 +275,33 @@ See [Plugins Guide](./docs/guide/plugins.md) for creating your own plugin.
|
||||
|
||||
## For AI Agents (Developer Guide)
|
||||
|
||||
If you are an AI assistant tasked with creating a new command adapter for `opencli`, please follow the AI Agent workflow below:
|
||||
|
||||
> **Quick mode**: To generate a single command for a specific page URL, see [CLI-ONESHOT.md](./CLI-ONESHOT.md) — just a URL + one-line goal, 4 steps done.
|
||||
|
||||
> **Full mode**: Before writing any adapter code, read [CLI-EXPLORER.md](./CLI-EXPLORER.md). It contains the complete browser exploration workflow, the 5-tier authentication strategy decision tree, and debugging guide.
|
||||
|
||||
```bash
|
||||
# 1. Deep Explore — discover APIs, infer capabilities, detect framework
|
||||
opencli explore https://example.com --site mysite
|
||||
|
||||
# 2. Synthesize — generate YAML adapters from explore artifacts
|
||||
opencli synthesize mysite
|
||||
|
||||
# 3. Generate — one-shot: explore → synthesize → register
|
||||
opencli generate https://example.com --goal "hot"
|
||||
|
||||
# 4. Strategy Cascade — auto-probe: PUBLIC → COOKIE → HEADER
|
||||
opencli cascade https://api.example.com/data
|
||||
opencli explore https://example.com --site mysite # Discover APIs + capabilities
|
||||
opencli synthesize mysite # Generate YAML adapters
|
||||
opencli generate https://example.com --goal "hot" # One-shot: explore → synthesize → register
|
||||
opencli cascade https://api.example.com/data # Auto-probe: PUBLIC → COOKIE → HEADER
|
||||
```
|
||||
|
||||
Explore outputs to `.opencli/explore/<site>/` (manifest.json, endpoints.json, capabilities.json, auth.json).
|
||||
|
||||
## Testing
|
||||
|
||||
See **[TESTING.md](./TESTING.md)** for how to run and write tests.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **"Extension not connected"**
|
||||
- Ensure the opencli Browser Bridge extension is installed and **enabled** in `chrome://extensions`.
|
||||
- **"attach failed: Cannot access a chrome-extension:// URL"**
|
||||
- Another Chrome extension (e.g. youmind, New Tab Override, or AI assistant extensions) may be interfering. Try **disabling other extensions** temporarily, then retry.
|
||||
- **Empty data returns or 'Unauthorized' error**
|
||||
- Your login session in Chrome might have expired. Open a normal Chrome tab, navigate to the target site, and log in or refresh the page.
|
||||
- **Node API errors**
|
||||
- Make sure you are using Node.js >= 20. Some dependencies require modern Node APIs.
|
||||
- **Daemon issues**
|
||||
- Check daemon status: `curl localhost:19825/status`
|
||||
- View extension logs: `curl localhost:19825/logs`
|
||||
|
||||
- **"Extension not connected"** — Ensure the Browser Bridge extension is installed and **enabled** in `chrome://extensions`.
|
||||
- **"attach failed: Cannot access a chrome-extension:// URL"** — Another extension may be interfering. Try disabling other extensions temporarily.
|
||||
- **Empty data or 'Unauthorized' error** — Your Chrome login session may have expired. Navigate to the target site and log in again.
|
||||
- **Node API errors** — Ensure Node.js >= 20. Some dependencies require modern Node APIs.
|
||||
- **Daemon issues** — Check status: `curl localhost:19825/status` · View logs: `curl localhost:19825/logs`
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#jackwener/opencli&Date)
|
||||
|
||||
|
||||
|
||||
## License
|
||||
|
||||
[Apache-2.0](./LICENSE)
|
||||
|
||||
+58
-6
@@ -45,7 +45,7 @@ CLI all electron!现在支持把所有 electron 应用 CLI 化,从而组合
|
||||
|
||||
- **零 LLM 成本** — 运行时不消耗任何 token,跑一万次不花一分钱
|
||||
- **确定性** — 同一命令永远返回同一结构,可管道化、可脚本化、CI 友好
|
||||
- **覆盖广泛** — 50+ 站点,横跨全球与中国平台(B站、知乎、小红书、Reddit、HackerNews 等),并支持通过 CDP 控制桌面 Electron 应用
|
||||
- **覆盖广泛** — 70+ 站点,横跨全球与中国平台(B站、知乎、小红书、Reddit、HackerNews 等),并支持通过 CDP 控制桌面 Electron 应用
|
||||
|
||||
> 与 Browser-Use、Crawl4AI、Firecrawl 等工具的详细对比,请查看 [Comparison Guide](./docs/comparison.md)。
|
||||
|
||||
@@ -73,9 +73,11 @@ OpenCLI 通过轻量化的 **Browser Bridge** Chrome 扩展 + 微型 daemon 与
|
||||
|
||||
完成!运行任何 opencli 浏览器命令时,后台微型 daemon 会自动启动与浏览器通信。无需配 API Token,零代码配置。
|
||||
|
||||
> **Tip**:后续诊断用 `opencli doctor`:
|
||||
> **Tip**:后续诊断和 daemon 管理:
|
||||
> ```bash
|
||||
> opencli doctor # 检查扩展和 daemon 连通性
|
||||
> opencli daemon status # 查看 daemon 状态
|
||||
> opencli daemon stop # 停止 daemon
|
||||
> ```
|
||||
|
||||
## 快速开始
|
||||
@@ -114,6 +116,21 @@ opencli list # 可以在任何地方使用了!
|
||||
npm install -g @jackwener/opencli@latest
|
||||
```
|
||||
|
||||
### 安装 AI Skills
|
||||
|
||||
OpenCLI 提供 [skills](./skills/) 供 AI Agent(Claude Code 等)使用:
|
||||
|
||||
```bash
|
||||
# 安装所有 OpenCLI skills
|
||||
npx skills add jackwener/opencli
|
||||
|
||||
# 或安装特定 skill
|
||||
npx skills add jackwener/opencli --skill opencli-usage # 命令参考
|
||||
npx skills add jackwener/opencli --skill opencli-operate # 浏览器自动化(AI Agent 专用)
|
||||
npx skills add jackwener/opencli --skill opencli-explorer # 适配器开发指南
|
||||
npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参考
|
||||
```
|
||||
|
||||
## 内置命令
|
||||
|
||||
运行 `opencli list` 查看完整注册表。
|
||||
@@ -122,16 +139,17 @@ npm install -g @jackwener/opencli@latest
|
||||
|------|------|------|
|
||||
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | 浏览器 |
|
||||
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | 浏览器 |
|
||||
| **tieba** | `hot` `posts` `search` `read` | 浏览器 |
|
||||
| **cursor** | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` | 桌面端 |
|
||||
| **bilibili** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | 浏览器 |
|
||||
| **codex** | `status` `send` `read` `new` `dump` `extract-diff` `model` `ask` `screenshot` `history` `export` | 桌面端 |
|
||||
| **chatwise** | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` | 桌面端 |
|
||||
| **doubao** | `status` `new` `send` `read` `ask` | 浏览器 |
|
||||
| **doubao** | `status` `new` `send` `read` `ask` `history` `detail` `meeting-summary` `meeting-transcript` | 浏览器 |
|
||||
| **doubao-app** | `status` `new` `send` `read` `ask` `screenshot` `dump` | 桌面端 |
|
||||
| **notion** | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` | 桌面端 |
|
||||
| **discord-app** | `status` `send` `read` `channels` `servers` `search` `members` | 桌面端 |
|
||||
| **v2ex** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | 公开 / 浏览器 |
|
||||
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 浏览器 |
|
||||
| **xueqiu** | `feed` `hot-stock` `hot` `search` `stock` `comments` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 浏览器 |
|
||||
| **antigravity** | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` | 桌面端 |
|
||||
| **chatgpt** | `status` `new` `send` `read` `ask` `model` | 桌面端 |
|
||||
| **xiaohongshu** | `search` `notifications` `feed` `user` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | 浏览器 |
|
||||
@@ -173,6 +191,10 @@ npm install -g @jackwener/opencli@latest
|
||||
| **douban** | `search` `top250` `subject` `photos` `download` `marks` `reviews` `movie-hot` `book-hot` | 浏览器 |
|
||||
| **facebook** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` | 浏览器 |
|
||||
| **google** | `news` `search` `suggest` `trends` | 公开 |
|
||||
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` | 浏览器 |
|
||||
| **gemini** | `new` `ask` `image` | 浏览器 |
|
||||
| **spotify** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` | OAuth API |
|
||||
| **notebooklm** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` | 浏览器 |
|
||||
| **36kr** | `news` `hot` `search` `article` | 公开 / 浏览器 |
|
||||
| **imdb** | `search` `title` `top` `trending` `person` `reviews` | 公开 |
|
||||
| **producthunt** | `posts` `today` `hot` `browse` | 公开 / 浏览器 |
|
||||
@@ -183,7 +205,10 @@ npm install -g @jackwener/opencli@latest
|
||||
| **substack** | `feed` `search` `publication` | 浏览器 |
|
||||
| **pixiv** | `ranking` `search` `user` `illusts` `detail` `download` | 浏览器 |
|
||||
| **tiktok** | `explore` `search` `profile` `user` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `live` `notifications` `friends` | 浏览器 |
|
||||
| **bluesky** | `search` `trending` `user` `profile` `thread` `feeds` `followers` `following` `starter-packs` | 公开 |
|
||||
| **douyin** | `videos` `publish` `drafts` `draft` `delete` `stats` `profile` `update` `hashtag` `location` `activities` `collections` | 浏览器 |
|
||||
|
||||
70+ 适配器 — **[→ 查看完整命令列表](./docs/adapters/index.md)**
|
||||
|
||||
### 外部 CLI 枢纽
|
||||
|
||||
@@ -194,8 +219,10 @@ OpenCLI 也可以作为你现有命令行工具的统一入口,负责发现、
|
||||
| **gh** | GitHub CLI | `opencli gh pr list --limit 5` |
|
||||
| **obsidian** | Obsidian 仓库管理 | `opencli obsidian search query="AI"` |
|
||||
| **docker** | Docker 命令行工具 | `opencli docker ps` |
|
||||
| **readwise** | Readwise / Reader CLI | `opencli readwise login` |
|
||||
| **gws** | Google Workspace CLI — Docs, Sheets, Drive, Gmail, Calendar | `opencli gws docs list` |
|
||||
| **lark-cli** | 飞书 CLI — 消息、文档、日历、任务,200+ 命令 | `opencli lark-cli calendar +agenda` |
|
||||
| **dingtalk** | 钉钉 CLI — 钉钉全套产品能力的跨平台命令行工具,支持人类和 AI Agent 使用 | `opencli dingtalk msg send --to user "hello"` |
|
||||
| **wecom** | 企业微信 CLI — 企业微信开放平台命令行工具,支持人类和 AI Agent 使用 | `opencli wecom msg send --to user "hello"` |
|
||||
| **vercel** | Vercel — 部署项目、管理域名、环境变量、日志 | `opencli vercel deploy --prod` |
|
||||
|
||||
**零配置透传**:OpenCLI 会把你的输入原样转发给底层二进制,保留原生 stdout / stderr 行为。
|
||||
|
||||
@@ -295,6 +322,31 @@ opencli bilibili hot -f csv # CSV
|
||||
opencli bilibili hot -v # 详细模式:展示管线执行步骤调试信息
|
||||
```
|
||||
|
||||
## 退出码
|
||||
|
||||
opencli 遵循 Unix `sysexits.h` 惯例,可无缝接入 shell 管道和 CI 脚本:
|
||||
|
||||
| 退出码 | 含义 | 触发场景 |
|
||||
|--------|------|----------|
|
||||
| `0` | 成功 | 命令正常完成 |
|
||||
| `1` | 通用错误 | 未分类的意外错误 |
|
||||
| `2` | 用法错误 | 参数错误或未知命令 |
|
||||
| `66` | 无数据 | 命令返回空结果(`EX_NOINPUT`) |
|
||||
| `69` | 服务不可用 | Browser Bridge 未连接(`EX_UNAVAILABLE`) |
|
||||
| `75` | 临时失败 | 命令超时,可重试(`EX_TEMPFAIL`) |
|
||||
| `77` | 需要认证 | 未登录目标网站(`EX_NOPERM`) |
|
||||
| `78` | 配置错误 | 凭证缺失或配置有误(`EX_CONFIG`) |
|
||||
| `130` | 中断 | Ctrl-C / SIGINT |
|
||||
|
||||
```bash
|
||||
opencli bilibili hot 2>/dev/null
|
||||
case $? in
|
||||
0) echo "ok" ;;
|
||||
69) echo "请先启动 Browser Bridge" ;;
|
||||
77) echo "请先登录 bilibili.com" ;;
|
||||
esac
|
||||
```
|
||||
|
||||
## 插件
|
||||
|
||||
通过社区贡献的插件扩展 OpenCLI。插件使用与内置命令相同的 YAML/TS 格式,启动时自动发现。
|
||||
|
||||
@@ -1,879 +0,0 @@
|
||||
---
|
||||
name: opencli
|
||||
description: "OpenCLI — Make any website or Electron App your CLI. Zero risk, AI-powered, reuse Chrome login."
|
||||
version: 1.4.1
|
||||
author: jackwener
|
||||
tags: [cli, browser, web, chrome-extension, cdp, bilibili, zhihu, twitter, github, v2ex, hackernews, reddit, xiaohongshu, xueqiu, youtube, boss, coupang, yollomi, AI, agent]
|
||||
---
|
||||
|
||||
# OpenCLI
|
||||
|
||||
> Make any website or Electron App your CLI. Reuse Chrome login, zero risk, AI-powered discovery.
|
||||
|
||||
> [!CAUTION]
|
||||
> **AI Agent 必读:创建或修改任何适配器之前,你必须先阅读 [CLI-EXPLORER.md](./CLI-EXPLORER.md)!**
|
||||
> 该文档包含完整的 API 发现工作流(必须使用浏览器探索)、5 级认证策略决策树、平台 SDK 速查表、`tap` 步骤调试流程、分页 API 模板、级联请求模式、以及常见陷阱。
|
||||
> **本文件(SKILL.md)仅提供命令参考和简化模板,不足以正确开发适配器。**
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 创建或修改 adapter 时,再额外遵守 3 条收口规则:
|
||||
> 1. 主参数优先用 positional arg,不要把 `query` / `id` / `url` 默认做成 `--query` / `--id` / `--url`
|
||||
> 2. 预期中的 adapter 失败优先抛 `CliError` 子类,不要直接 throw 原始 `Error`
|
||||
> 3. 新增 adapter 或新增用户可发现命令时,同步更新 adapter docs、`docs/adapters/index.md`、sidebar,以及 README/README.zh-CN 中受影响的入口
|
||||
|
||||
## Install & Run
|
||||
|
||||
```bash
|
||||
# npm global install (recommended)
|
||||
npm install -g @jackwener/opencli
|
||||
opencli <command>
|
||||
|
||||
# Or from source
|
||||
cd ~/code/opencli && npm install
|
||||
npx tsx src/main.ts <command>
|
||||
|
||||
# Update to latest
|
||||
npm update -g @jackwener/opencli
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Browser commands require:
|
||||
1. Chrome browser running **(logged into target sites)**
|
||||
2. **opencli Browser Bridge** Chrome extension installed (load `extension/` as unpacked in `chrome://extensions`)
|
||||
3. No further setup needed — the daemon auto-starts on first browser command
|
||||
|
||||
> **Note**: You must be logged into the target website in Chrome before running commands. Tabs opened during command execution are auto-closed afterwards.
|
||||
|
||||
Public API commands (`hackernews`, `v2ex`) need no browser.
|
||||
|
||||
## Commands Reference
|
||||
|
||||
### Data Commands
|
||||
|
||||
```bash
|
||||
# Bilibili (browser)
|
||||
opencli bilibili hot --limit 10 # B站热门视频
|
||||
opencli bilibili search "rust" # 搜索视频 (query positional)
|
||||
opencli bilibili me # 我的信息
|
||||
opencli bilibili favorite # 我的收藏
|
||||
opencli bilibili history --limit 20 # 观看历史
|
||||
opencli bilibili feed --limit 10 # 动态时间线
|
||||
opencli bilibili user-videos --uid 12345 # 用户投稿
|
||||
opencli bilibili subtitle --bvid BV1xxx # 获取视频字幕 (支持 --lang zh-CN)
|
||||
opencli bilibili dynamic --limit 10 # 动态
|
||||
opencli bilibili ranking --limit 10 # 排行榜
|
||||
opencli bilibili following --limit 20 # 我的关注列表 (支持 --uid 查看他人)
|
||||
|
||||
# 知乎 (browser)
|
||||
opencli zhihu hot --limit 10 # 知乎热榜
|
||||
opencli zhihu search "AI" # 搜索 (query positional)
|
||||
opencli zhihu question 34816524 # 问题详情和回答 (id positional)
|
||||
|
||||
# 小红书 (browser)
|
||||
opencli xiaohongshu search "美食" # 搜索笔记 (query positional)
|
||||
opencli xiaohongshu notifications # 通知(mentions/likes/connections)
|
||||
opencli xiaohongshu feed --limit 10 # 推荐 Feed
|
||||
opencli xiaohongshu user xxx # 用户主页 (id positional)
|
||||
opencli xiaohongshu creator-notes --limit 10 # 创作者笔记列表
|
||||
opencli xiaohongshu creator-note-detail --note-id xxx # 笔记详情
|
||||
opencli xiaohongshu creator-notes-summary # 笔记数据概览
|
||||
opencli xiaohongshu creator-profile # 创作者资料
|
||||
opencli xiaohongshu creator-stats # 创作者数据统计
|
||||
|
||||
# 雪球 Xueqiu (browser)
|
||||
opencli xueqiu hot-stock --limit 10 # 雪球热门股票榜
|
||||
opencli xueqiu stock --symbol SH600519 # 查看股票实时行情
|
||||
opencli xueqiu watchlist # 获取自选股/持仓列表
|
||||
opencli xueqiu feed # 我的关注 timeline
|
||||
opencli xueqiu hot --limit 10 # 雪球热榜
|
||||
opencli xueqiu search "特斯拉" # 搜索 (query positional)
|
||||
opencli xueqiu earnings-date SH600519 # 股票财报发布日期 (symbol positional)
|
||||
opencli xueqiu fund-holdings # 蛋卷基金持仓明细 (支持 --account 过滤)
|
||||
opencli xueqiu fund-snapshot # 蛋卷基金快照(总资产、子账户、持仓)
|
||||
|
||||
# GitHub (via gh External CLI)
|
||||
opencli gh repo list # 列出仓库 (passthrough to gh)
|
||||
opencli gh pr list --limit 5 # PR 列表
|
||||
opencli gh issue list # Issue 列表
|
||||
|
||||
# Twitter/X (browser)
|
||||
opencli twitter trending --limit 10 # 热门话题
|
||||
opencli twitter bookmarks --limit 20 # 获取收藏的书签推文
|
||||
opencli twitter search "AI" # 搜索推文 (query positional)
|
||||
opencli twitter profile elonmusk # 用户资料
|
||||
opencli twitter timeline --limit 20 # 时间线
|
||||
opencli twitter thread 1234567890 # 推文 thread(原文 + 回复)
|
||||
opencli twitter article 1891511252174299446 # 推文长文内容
|
||||
opencli twitter follow elonmusk # 关注用户
|
||||
opencli twitter unfollow elonmusk # 取消关注
|
||||
opencli twitter bookmark https://x.com/... # 收藏推文
|
||||
opencli twitter unbookmark https://x.com/... # 取消收藏
|
||||
opencli twitter post "Hello world" # 发布推文 (text positional)
|
||||
opencli twitter like https://x.com/... # 点赞推文 (url positional)
|
||||
opencli twitter reply https://x.com/... "Nice!" # 回复推文 (url + text positional)
|
||||
opencli twitter delete https://x.com/... # 删除推文 (url positional)
|
||||
opencli twitter block elonmusk # 屏蔽用户 (username positional)
|
||||
opencli twitter unblock elonmusk # 取消屏蔽 (username positional)
|
||||
opencli twitter followers elonmusk # 用户的粉丝列表 (user positional)
|
||||
opencli twitter following elonmusk # 用户的关注列表 (user positional)
|
||||
opencli twitter notifications --limit 20 # 通知列表
|
||||
opencli twitter hide-reply https://x.com/... # 隐藏回复 (url positional)
|
||||
opencli twitter download elonmusk # 下载用户媒体 (username positional, 支持 --tweet-url)
|
||||
opencli twitter accept "群,微信" # 自动接受含关键词的 DM 请求 (query positional)
|
||||
opencli twitter reply-dm "消息内容" # 批量回复 DM (text positional)
|
||||
|
||||
# Reddit (browser)
|
||||
opencli reddit hot --limit 10 # 热门帖子
|
||||
opencli reddit hot --subreddit programming # 指定子版块
|
||||
opencli reddit frontpage --limit 10 # 首页 /r/all
|
||||
opencli reddit popular --limit 10 # /r/popular 热门
|
||||
opencli reddit search "AI" --sort top --time week # 搜索(支持排序+时间过滤)
|
||||
opencli reddit subreddit rust --sort top --time month # 子版块浏览(支持时间过滤)
|
||||
opencli reddit read --post-id 1abc123 # 阅读帖子 + 评论
|
||||
opencli reddit user spez # 用户资料(karma、注册时间)
|
||||
opencli reddit user-posts spez # 用户发帖历史
|
||||
opencli reddit user-comments spez # 用户评论历史
|
||||
opencli reddit upvote --post-id xxx --direction up # 投票(up/down/none)
|
||||
opencli reddit save --post-id xxx # 收藏帖子
|
||||
opencli reddit comment --post-id xxx "Great!" # 发表评论 (text positional)
|
||||
opencli reddit subscribe --subreddit python # 订阅子版块
|
||||
opencli reddit saved --limit 10 # 我的收藏
|
||||
opencli reddit upvoted --limit 10 # 我的赞
|
||||
|
||||
# V2EX (public + browser)
|
||||
opencli v2ex hot --limit 10 # 热门话题
|
||||
opencli v2ex latest --limit 10 # 最新话题
|
||||
opencli v2ex topic 1024 # 主题详情 (id positional)
|
||||
opencli v2ex daily # 每日签到 (browser)
|
||||
opencli v2ex me # 我的信息 (browser)
|
||||
opencli v2ex notifications --limit 10 # 通知 (browser)
|
||||
opencli v2ex node python # 节点话题列表 (name positional)
|
||||
opencli v2ex nodes --limit 30 # 所有节点列表
|
||||
opencli v2ex member username # 用户资料 (username positional)
|
||||
opencli v2ex user username # 用户发帖列表 (username positional)
|
||||
opencli v2ex replies 1024 # 主题回复列表 (id positional)
|
||||
|
||||
# Hacker News (public)
|
||||
opencli hackernews top --limit 10 # Top stories
|
||||
opencli hackernews new --limit 10 # Newest stories
|
||||
opencli hackernews best --limit 10 # Best stories
|
||||
opencli hackernews ask --limit 10 # Ask HN posts
|
||||
opencli hackernews show --limit 10 # Show HN posts
|
||||
opencli hackernews jobs --limit 10 # Job postings
|
||||
opencli hackernews search "rust" # 搜索 (query positional)
|
||||
opencli hackernews user dang # 用户资料 (username positional)
|
||||
|
||||
# BBC (public)
|
||||
opencli bbc news --limit 10 # BBC News RSS headlines
|
||||
|
||||
# 微博 (browser)
|
||||
opencli weibo hot --limit 10 # 微博热搜
|
||||
|
||||
# BOSS直聘 (browser)
|
||||
opencli boss search "AI agent" # 搜索职位 (query positional)
|
||||
opencli boss detail --security-id xxx # 职位详情
|
||||
opencli boss recommend --limit 10 # 推荐职位
|
||||
opencli boss joblist --limit 10 # 职位列表
|
||||
opencli boss greet --security-id xxx # 打招呼
|
||||
opencli boss batchgreet --job-id xxx # 批量打招呼
|
||||
opencli boss send --uid xxx "消息内容" # 发消息 (text positional)
|
||||
opencli boss chatlist --limit 10 # 聊天列表
|
||||
opencli boss chatmsg --security-id xxx # 聊天记录
|
||||
opencli boss invite --security-id xxx # 邀请沟通
|
||||
opencli boss mark --security-id xxx # 标记管理
|
||||
opencli boss exchange --security-id xxx # 交换联系方式
|
||||
opencli boss resume # 简历管理
|
||||
opencli boss stats # 数据统计
|
||||
|
||||
# YouTube (browser)
|
||||
opencli youtube search "rust" # 搜索视频 (query positional)
|
||||
opencli youtube video "https://www.youtube.com/watch?v=xxx" # 视频元数据
|
||||
opencli youtube transcript "https://www.youtube.com/watch?v=xxx" # 获取视频字幕/转录
|
||||
opencli youtube transcript "xxx" --lang zh-Hans --mode raw # 指定语言 + 原始时间戳模式
|
||||
|
||||
# Yahoo Finance (browser)
|
||||
opencli yahoo-finance quote --symbol AAPL # 股票行情
|
||||
|
||||
# Sina Finance
|
||||
opencli sinafinance news --limit 10 --type 1 # 7x24实时快讯 (0=全部 1=A股 2=宏观 3=公司 4=数据 5=市场 6=国际 7=观点 8=央行 9=其它)
|
||||
|
||||
# Reuters (browser)
|
||||
opencli reuters search "AI" # 路透社搜索 (query positional)
|
||||
|
||||
# 什么值得买 (browser)
|
||||
opencli smzdm search "耳机" # 搜索好价 (query positional)
|
||||
|
||||
# 携程 (browser)
|
||||
opencli ctrip search "三亚" # 搜索目的地 (query positional)
|
||||
|
||||
# Antigravity (Electron/CDP)
|
||||
opencli antigravity status # 检查 CDP 连接
|
||||
opencli antigravity send "hello" # 发送文本到当前 agent 聊天框
|
||||
opencli antigravity read # 读取整个聊天记录面板
|
||||
opencli antigravity new # 清空聊天、开启新对话
|
||||
opencli antigravity dump # 导出 DOM 和快照调试信息
|
||||
opencli antigravity extract-code # 自动抽取 AI 回复中的代码块
|
||||
opencli antigravity model claude # 切换底层模型
|
||||
opencli antigravity watch # 流式监听增量消息
|
||||
|
||||
# Barchart (browser)
|
||||
opencli barchart quote --symbol AAPL # 股票行情
|
||||
opencli barchart options --symbol AAPL # 期权链
|
||||
opencli barchart greeks --symbol AAPL # 期权 Greeks
|
||||
opencli barchart flow --limit 20 # 异常期权活动
|
||||
|
||||
# Jike 即刻 (browser)
|
||||
opencli jike feed --limit 10 # 动态流
|
||||
opencli jike search "AI" # 搜索 (query positional)
|
||||
opencli jike create "内容" # 发布动态 (text positional)
|
||||
opencli jike like xxx # 点赞 (id positional)
|
||||
opencli jike comment xxx "评论" # 评论 (id + text positional)
|
||||
opencli jike repost xxx # 转发 (id positional)
|
||||
opencli jike notifications # 通知
|
||||
|
||||
# Linux.do (public + browser)
|
||||
opencli linux-do hot --limit 10 # 热门话题
|
||||
opencli linux-do latest --limit 10 # 最新话题
|
||||
opencli linux-do search "rust" # 搜索 (query positional)
|
||||
opencli linux-do topic 1024 # 主题详情 (id positional)
|
||||
opencli linux-do categories --limit 20 # 分类列表 (browser)
|
||||
opencli linux-do category dev 7 # 分类内话题 (slug + id positional, browser)
|
||||
|
||||
# StackOverflow (public)
|
||||
opencli stackoverflow hot --limit 10 # 热门问题
|
||||
opencli stackoverflow search "typescript" # 搜索 (query positional)
|
||||
opencli stackoverflow bounties --limit 10 # 悬赏问题
|
||||
|
||||
# WeRead 微信读书 (browser)
|
||||
opencli weread shelf --limit 10 # 书架
|
||||
opencli weread search "AI" # 搜索图书 (query positional)
|
||||
opencli weread book xxx # 图书详情 (book-id positional)
|
||||
opencli weread highlights xxx # 划线笔记 (book-id positional)
|
||||
opencli weread notes xxx # 想法笔记 (book-id positional)
|
||||
opencli weread ranking --limit 10 # 排行榜
|
||||
|
||||
# Jimeng 即梦 AI (browser)
|
||||
opencli jimeng generate --prompt "描述" # AI 生图
|
||||
opencli jimeng history --limit 10 # 生成历史
|
||||
|
||||
# Yollomi yollomi.com (browser — 需在 Chrome 登录 yollomi.com,复用站点 session)
|
||||
opencli yollomi models --type image # 列出图像模型与积分
|
||||
opencli yollomi generate "提示词" --model z-image-turbo # 文生图
|
||||
opencli yollomi video "提示词" --model kling-2-1 # 视频
|
||||
opencli yollomi upload ./photo.jpg # 上传得 URL,供 img2img / 工具链使用
|
||||
opencli yollomi remove-bg <image-url> # 去背景(免费)
|
||||
opencli yollomi edit <image-url> "改成油画风格" # Qwen 图像编辑
|
||||
opencli yollomi background <image-url> # AI 背景生成 (5 credits)
|
||||
opencli yollomi face-swap --source <url> --target <url> # 换脸 (3 credits)
|
||||
opencli yollomi object-remover <image-url> <mask-url> # AI 去除物体 (3 credits)
|
||||
opencli yollomi restore <image-url> # AI 修复老照片 (4 credits)
|
||||
opencli yollomi try-on --person <url> --cloth <url> # 虚拟试衣 (3 credits)
|
||||
opencli yollomi upscale <image-url> # AI 超分辨率 (1 credit, 支持 --scale 2/4)
|
||||
|
||||
# Grok (default + explicit web)
|
||||
opencli grok ask --prompt "问题" # 提问 Grok(兼容默认路径)
|
||||
opencli grok ask --prompt "问题" --web # 显式 grok.com consumer web UI 路径
|
||||
|
||||
# HuggingFace (public)
|
||||
opencli hf top --limit 10 # 热门模型
|
||||
|
||||
# 超星学习通 (browser)
|
||||
opencli chaoxing assignments # 作业列表
|
||||
opencli chaoxing exams # 考试列表
|
||||
|
||||
# Douban 豆瓣 (browser)
|
||||
opencli douban search "三体" # 搜索 (query positional)
|
||||
opencli douban top250 # 豆瓣 Top 250
|
||||
opencli douban subject 1234567 # 条目详情 (id positional)
|
||||
opencli douban photos 30382501 # 图片列表 / 直链(默认海报)
|
||||
opencli douban download 30382501 # 下载海报 / 剧照
|
||||
opencli douban marks --limit 10 # 我的标记
|
||||
opencli douban reviews --limit 10 # 短评
|
||||
|
||||
# Facebook (browser)
|
||||
opencli facebook feed --limit 10 # 动态流
|
||||
opencli facebook profile username # 用户资料 (id positional)
|
||||
opencli facebook search "AI" # 搜索 (query positional)
|
||||
opencli facebook friends # 好友列表
|
||||
opencli facebook groups # 群组
|
||||
opencli facebook events # 活动
|
||||
opencli facebook notifications # 通知
|
||||
opencli facebook memories # 回忆
|
||||
opencli facebook add-friend username # 添加好友 (id positional)
|
||||
opencli facebook join-group groupid # 加入群组 (id positional)
|
||||
|
||||
# Instagram (browser)
|
||||
opencli instagram explore # 探索
|
||||
opencli instagram profile username # 用户资料 (id positional)
|
||||
opencli instagram search "AI" # 搜索 (query positional)
|
||||
opencli instagram user username # 用户详情 (id positional)
|
||||
opencli instagram followers username # 粉丝 (id positional)
|
||||
opencli instagram following username # 关注 (id positional)
|
||||
opencli instagram follow username # 关注用户 (id positional)
|
||||
opencli instagram unfollow username # 取消关注 (id positional)
|
||||
opencli instagram like postid # 点赞 (id positional)
|
||||
opencli instagram unlike postid # 取消点赞 (id positional)
|
||||
opencli instagram comment postid "评论" # 评论 (id + text positional)
|
||||
opencli instagram save postid # 收藏 (id positional)
|
||||
opencli instagram unsave postid # 取消收藏 (id positional)
|
||||
opencli instagram saved # 已收藏列表
|
||||
|
||||
# TikTok (browser)
|
||||
opencli tiktok explore # 探索
|
||||
opencli tiktok search "AI" # 搜索 (query positional)
|
||||
opencli tiktok profile username # 用户资料 (id positional)
|
||||
opencli tiktok user username # 用户详情 (id positional)
|
||||
opencli tiktok following username # 关注列表 (id positional)
|
||||
opencli tiktok follow username # 关注 (id positional)
|
||||
opencli tiktok unfollow username # 取消关注 (id positional)
|
||||
opencli tiktok like videoid # 点赞 (id positional)
|
||||
opencli tiktok unlike videoid # 取消点赞 (id positional)
|
||||
opencli tiktok comment videoid "评论" # 评论 (id + text positional)
|
||||
opencli tiktok save videoid # 收藏 (id positional)
|
||||
opencli tiktok unsave videoid # 取消收藏 (id positional)
|
||||
opencli tiktok live # 直播
|
||||
opencli tiktok notifications # 通知
|
||||
opencli tiktok friends # 朋友
|
||||
|
||||
# Medium (browser)
|
||||
opencli medium feed --limit 10 # 动态流
|
||||
opencli medium search "AI" # 搜索 (query positional)
|
||||
opencli medium user username # 用户主页 (id positional)
|
||||
|
||||
# Substack (browser)
|
||||
opencli substack feed --limit 10 # 订阅动态
|
||||
opencli substack search "AI" # 搜索 (query positional)
|
||||
opencli substack publication name # 出版物详情 (id positional)
|
||||
|
||||
# Sinablog 新浪博客 (browser)
|
||||
opencli sinablog hot --limit 10 # 热门
|
||||
opencli sinablog search "AI" # 搜索 (query positional)
|
||||
opencli sinablog article url # 文章详情
|
||||
opencli sinablog user username # 用户主页 (id positional)
|
||||
|
||||
# Lobsters (public)
|
||||
opencli lobsters hot --limit 10 # 热门
|
||||
opencli lobsters newest --limit 10 # 最新
|
||||
opencli lobsters active --limit 10 # 活跃
|
||||
opencli lobsters tag rust # 按标签筛选 (tag positional)
|
||||
|
||||
# Google (public)
|
||||
opencli google news --limit 10 # 新闻
|
||||
opencli google search "AI" # 搜索 (query positional)
|
||||
opencli google suggest "AI" # 搜索建议 (query positional)
|
||||
opencli google trends # 趋势
|
||||
|
||||
# DEV.to (public)
|
||||
opencli devto top --limit 10 # 热门文章
|
||||
opencli devto tag javascript --limit 10 # 按标签 (tag positional)
|
||||
opencli devto user username # 用户文章 (username positional)
|
||||
|
||||
# Steam (public)
|
||||
opencli steam top-sellers --limit 10 # 热销游戏
|
||||
|
||||
# Apple Podcasts (public)
|
||||
opencli apple-podcasts top --limit 10 # 热门播客排行榜 (支持 --country us/cn/gb/jp)
|
||||
opencli apple-podcasts search "科技" # 搜索播客 (query positional)
|
||||
opencli apple-podcasts episodes 12345 # 播客剧集列表 (id positional, 用 search 获取 ID)
|
||||
|
||||
# arXiv (public)
|
||||
opencli arxiv search "attention" # 搜索论文 (query positional)
|
||||
opencli arxiv paper 1706.03762 # 论文详情 (id positional)
|
||||
|
||||
# Bloomberg (public RSS + browser)
|
||||
opencli bloomberg main --limit 10 # Bloomberg 首页头条 (RSS)
|
||||
opencli bloomberg markets --limit 10 # 市场新闻 (RSS)
|
||||
opencli bloomberg tech --limit 10 # 科技新闻 (RSS)
|
||||
opencli bloomberg politics --limit 10 # 政治新闻 (RSS)
|
||||
opencli bloomberg economics --limit 10 # 经济新闻 (RSS)
|
||||
opencli bloomberg opinions --limit 10 # 观点 (RSS)
|
||||
opencli bloomberg industries --limit 10 # 行业新闻 (RSS)
|
||||
opencli bloomberg businessweek --limit 10 # Businessweek (RSS)
|
||||
opencli bloomberg feeds # 列出所有 RSS feed 别名
|
||||
opencli bloomberg news "https://..." # 阅读 Bloomberg 文章全文 (link positional, browser)
|
||||
|
||||
# Coupang 쿠팡 (browser)
|
||||
opencli coupang search "耳机" # 搜索商品 (query positional, 支持 --filter rocket)
|
||||
opencli coupang add-to-cart 12345 # 加入购物车 (product-id positional, 或 --url)
|
||||
|
||||
# Dictionary (public)
|
||||
opencli dictionary search "serendipity" # 单词释义 (word positional)
|
||||
opencli dictionary synonyms "happy" # 近义词 (word positional)
|
||||
opencli dictionary examples "ubiquitous" # 例句 (word positional)
|
||||
|
||||
# 豆包 Doubao Web (browser)
|
||||
opencli doubao status # 检查豆包页面状态
|
||||
opencli doubao new # 新建对话
|
||||
opencli doubao send "你好" # 发送消息 (text positional)
|
||||
opencli doubao read # 读取对话记录
|
||||
opencli doubao ask "问题" # 一键提问并等回复 (text positional)
|
||||
|
||||
# 京东 JD (browser)
|
||||
opencli jd item 100291143898 # 商品详情 (sku positional, 含价格/主图/规格)
|
||||
|
||||
# LinkedIn (browser)
|
||||
opencli linkedin search "AI engineer" # 搜索职位 (query positional, 支持 --location/--company/--remote)
|
||||
opencli linkedin timeline --limit 20 # 首页动态流
|
||||
|
||||
# Pixiv (browser)
|
||||
opencli pixiv ranking --limit 20 # 插画排行榜 (支持 --mode daily/weekly/monthly)
|
||||
opencli pixiv search "風景" # 搜索插画 (query positional)
|
||||
opencli pixiv user 12345 # 画师资料 (uid positional)
|
||||
opencli pixiv illusts 12345 # 画师作品列表 (user-id positional)
|
||||
opencli pixiv detail 12345 # 插画详情 (id positional)
|
||||
opencli pixiv download 12345 # 下载插画 (illust-id positional)
|
||||
|
||||
# Web (browser)
|
||||
opencli web read --url "https://..." # 抓取任意网页并导出为 Markdown
|
||||
|
||||
# 微信公众号 Weixin (browser)
|
||||
opencli weixin download --url "https://mp.weixin.qq.com/s/xxx" # 下载公众号文章为 Markdown
|
||||
|
||||
# 小宇宙 Xiaoyuzhou (public)
|
||||
opencli xiaoyuzhou podcast 12345 # 播客资料 (id positional)
|
||||
opencli xiaoyuzhou podcast-episodes 12345 # 播客剧集列表 (id positional)
|
||||
opencli xiaoyuzhou episode 12345 # 单集详情 (id positional)
|
||||
|
||||
# Wikipedia (public)
|
||||
opencli wikipedia search "AI" # 搜索 (query positional)
|
||||
opencli wikipedia summary "Python" # 摘要 (title positional)
|
||||
```
|
||||
|
||||
### Desktop Adapter Commands
|
||||
|
||||
```bash
|
||||
# Cursor (desktop — CDP via Electron)
|
||||
opencli cursor status # 检查连接
|
||||
opencli cursor send "message" # 发送消息
|
||||
opencli cursor read # 读取回复
|
||||
opencli cursor new # 新建对话
|
||||
opencli cursor dump # 导出 DOM 调试信息
|
||||
opencli cursor composer # Composer 模式
|
||||
opencli cursor model claude # 切换模型
|
||||
opencli cursor extract-code # 提取代码块
|
||||
opencli cursor ask "question" # 一键提问并等回复
|
||||
opencli cursor screenshot # 截图
|
||||
opencli cursor history # 对话历史
|
||||
opencli cursor export # 导出对话
|
||||
|
||||
# Codex (desktop — headless CLI agent)
|
||||
opencli codex status # 检查连接
|
||||
opencli codex send "message" # 发送消息
|
||||
opencli codex read # 读取回复
|
||||
opencli codex new # 新建对话
|
||||
opencli codex dump # 导出调试信息
|
||||
opencli codex extract-diff # 提取 diff
|
||||
opencli codex model gpt-4 # 切换模型
|
||||
opencli codex ask "question" # 一键提问并等回复
|
||||
opencli codex screenshot # 截图
|
||||
opencli codex history # 对话历史
|
||||
opencli codex export # 导出对话
|
||||
|
||||
# ChatGPT (desktop — macOS AppleScript/CDP)
|
||||
opencli chatgpt status # 检查应用状态
|
||||
opencli chatgpt new # 新建对话
|
||||
opencli chatgpt send "message" # 发送消息
|
||||
opencli chatgpt read # 读取回复
|
||||
opencli chatgpt ask "question" # 一键提问并等回复
|
||||
|
||||
# ChatWise (desktop — multi-LLM client)
|
||||
opencli chatwise status # 检查连接
|
||||
opencli chatwise new # 新建对话
|
||||
opencli chatwise send "message" # 发送消息
|
||||
opencli chatwise read # 读取回复
|
||||
opencli chatwise ask "question" # 一键提问并等回复
|
||||
opencli chatwise model claude # 切换模型
|
||||
opencli chatwise history # 对话历史
|
||||
opencli chatwise export # 导出对话
|
||||
opencli chatwise screenshot # 截图
|
||||
|
||||
# Notion (desktop — CDP via Electron)
|
||||
opencli notion status # 检查连接
|
||||
opencli notion search "keyword" # 搜索页面
|
||||
opencli notion read # 读取当前页面
|
||||
opencli notion new # 新建页面
|
||||
opencli notion write "content" # 写入内容
|
||||
opencli notion sidebar # 侧边栏导航
|
||||
opencli notion favorites # 收藏列表
|
||||
opencli notion export # 导出
|
||||
|
||||
# Discord App (desktop — CDP via Electron)
|
||||
opencli discord-app status # 检查连接
|
||||
opencli discord-app send "message" # 发送消息
|
||||
opencli discord-app read # 读取消息
|
||||
opencli discord-app channels # 频道列表
|
||||
opencli discord-app servers # 服务器列表
|
||||
opencli discord-app search "keyword" # 搜索
|
||||
opencli discord-app members # 成员列表
|
||||
|
||||
# Doubao App 豆包桌面版 (desktop — CDP via Electron)
|
||||
opencli doubao-app status # 检查连接
|
||||
opencli doubao-app new # 新建对话
|
||||
opencli doubao-app send "message" # 发送消息
|
||||
opencli doubao-app read # 读取回复
|
||||
opencli doubao-app ask "question" # 一键提问并等回复
|
||||
opencli doubao-app screenshot # 截图
|
||||
opencli doubao-app dump # 导出 DOM 调试信息
|
||||
```
|
||||
|
||||
### Management Commands
|
||||
|
||||
```bash
|
||||
opencli list # List all commands (including External CLIs)
|
||||
opencli list --json # JSON output
|
||||
opencli list -f yaml # YAML output
|
||||
opencli install <name> # Auto-install an external CLI (e.g., gh, obsidian)
|
||||
opencli register <name> # Register a local custom CLI for unified discovery
|
||||
opencli validate # Validate all CLI definitions
|
||||
opencli validate bilibili # Validate specific site
|
||||
opencli doctor # Diagnose browser bridge (auto-starts daemon, includes live test)
|
||||
```
|
||||
|
||||
### AI Agent Workflow
|
||||
|
||||
```bash
|
||||
# Deep Explore: network intercept → response analysis → capability inference
|
||||
opencli explore <url> --site <name>
|
||||
|
||||
# Synthesize: generate evaluate-based YAML pipelines from explore artifacts
|
||||
opencli synthesize <site>
|
||||
|
||||
# Generate: one-shot explore → synthesize → register
|
||||
opencli generate <url> --goal "hot"
|
||||
|
||||
# Record: YOU operate the page, opencli captures every API call → YAML candidates
|
||||
# Opens the URL in automation window, injects fetch/XHR interceptor into ALL tabs,
|
||||
# polls every 2s, auto-stops after 60s (or press Enter to stop early).
|
||||
opencli record <url> # 录制,site name 从域名推断
|
||||
opencli record <url> --site mysite # 指定 site name
|
||||
opencli record <url> --timeout 120000 # 自定义超时(毫秒,默认 60000)
|
||||
opencli record <url> --poll 1000 # 缩短轮询间隔(毫秒,默认 2000)
|
||||
opencli record <url> --out .opencli/record/x # 自定义输出目录
|
||||
# Output:
|
||||
# .opencli/record/<site>/captured.json ← 原始捕获数据(带 url/method/body)
|
||||
# .opencli/record/<site>/candidates/*.yaml ← 高置信度候选适配器(score ≥ 8,有 array 结果)
|
||||
|
||||
# Strategy Cascade: auto-probe PUBLIC → COOKIE → HEADER
|
||||
opencli cascade <api-url>
|
||||
|
||||
# Explore with interactive fuzzing (click buttons to trigger lazy APIs)
|
||||
opencli explore <url> --auto --click "字幕,CC,评论"
|
||||
|
||||
# Validate: validate adapter definitions
|
||||
opencli validate
|
||||
```
|
||||
|
||||
## Output Formats
|
||||
|
||||
All built-in commands support `--format` / `-f` with `table`, `json`, `yaml`, `md`, and `csv`.
|
||||
The `list` command supports the same formats and also keeps `--json` as a compatibility alias.
|
||||
|
||||
```bash
|
||||
opencli list -f yaml # YAML command registry
|
||||
opencli bilibili hot -f table # Default: rich table
|
||||
opencli bilibili hot -f json # JSON (pipe to jq, feed to AI agent)
|
||||
opencli bilibili hot -f yaml # YAML (readable structured output)
|
||||
opencli bilibili hot -f md # Markdown
|
||||
opencli bilibili hot -f csv # CSV
|
||||
```
|
||||
|
||||
## Verbose Mode
|
||||
|
||||
```bash
|
||||
opencli bilibili hot -v # Show each pipeline step and data flow
|
||||
```
|
||||
|
||||
## Record Workflow
|
||||
|
||||
`record` 是为「无法用 `explore` 自动发现」的页面(需要登录操作、复杂交互、SPA 内路由)准备的手动录制方案。
|
||||
|
||||
### 工作原理
|
||||
|
||||
```
|
||||
opencli record <url>
|
||||
→ 打开 automation window 并导航到目标 URL
|
||||
→ 向所有 tab 注入 fetch/XHR 拦截器(幂等,可重复注入)
|
||||
→ 每 2s 轮询一次:发现新 tab 自动注入,drain 所有 tab 的捕获缓冲区
|
||||
→ 超时(默认 60s)或按 Enter 停止
|
||||
→ 分析捕获到的 JSON 请求:去重 → 评分 → 生成候选 YAML
|
||||
```
|
||||
|
||||
**拦截器特性**:
|
||||
- 同时 patch `window.fetch` 和 `XMLHttpRequest`
|
||||
- 只捕获 `Content-Type: application/json` 的响应
|
||||
- 过滤纯对象少于 2 个 key 的响应(避免 tracking/ping)
|
||||
- 跨 tab 隔离:每个 tab 独立缓冲区,轮询时分别 drain
|
||||
- 幂等注入:同一 tab 二次注入时先 restore 原始函数再重新 patch,不丢失已捕获数据
|
||||
|
||||
### 使用步骤
|
||||
|
||||
```bash
|
||||
# 1. 启动录制(建议 --timeout 给足操作时间)
|
||||
opencli record "https://example.com/page" --timeout 120000
|
||||
|
||||
# 2. 在弹出的 automation window 里正常操作页面:
|
||||
# - 打开列表、搜索、点击条目、切换 Tab
|
||||
# - 凡是触发网络请求的操作都会被捕获
|
||||
|
||||
# 3. 完成操作后按 Enter 停止(或等超时自动停止)
|
||||
|
||||
# 4. 查看结果
|
||||
cat .opencli/record/<site>/captured.json # 原始捕获
|
||||
ls .opencli/record/<site>/candidates/ # 候选 YAML
|
||||
```
|
||||
|
||||
### 页面类型与捕获预期
|
||||
|
||||
| 页面类型 | 预期捕获量 | 说明 |
|
||||
|---------|-----------|------|
|
||||
| 列表/搜索页 | 多(5~20+) | 每次搜索/翻页都会触发新请求 |
|
||||
| 详情页(只读) | 少(1~5) | 首屏数据一次性返回,后续操作走 form/redirect |
|
||||
| SPA 内路由跳转 | 中等 | 路由切换会触发新接口,但首屏请求在注入前已发出 |
|
||||
| 需要登录的页面 | 视操作而定 | 确保 Chrome 已登录目标网站 |
|
||||
|
||||
> **注意**:如果页面在导航完成前就发出了大部分请求(服务端渲染 / SSR 注水),拦截器会错过这些请求。
|
||||
> 解决方案:在页面加载完成后,手动触发能产生新请求的操作(搜索、翻页、切 Tab、展开折叠项等)。
|
||||
|
||||
### 候选 YAML → TS CLI 转换
|
||||
|
||||
生成的候选 YAML 是起点,通常需要转换为 TypeScript(尤其是 tae 等内部系统):
|
||||
|
||||
**候选 YAML 结构**(自动生成):
|
||||
```yaml
|
||||
site: tae
|
||||
name: getList # 从 URL path 推断的名称
|
||||
strategy: cookie
|
||||
browser: true
|
||||
pipeline:
|
||||
- navigate: https://...
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const res = await fetch('/approval/getList.json?procInsId=...', { credentials: 'include' });
|
||||
const data = await res.json();
|
||||
return (data?.content?.operatorRecords || []).map(item => ({ ... }));
|
||||
})()
|
||||
```
|
||||
|
||||
**转换为 TS CLI**(参考 `src/clis/tae/add-expense.ts` 风格):
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'tae',
|
||||
name: 'get-approval',
|
||||
description: '查看报销单审批流程和操作记录',
|
||||
domain: 'tae.alibaba-inc.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'proc_ins_id', type: 'string', required: true, positional: true, help: '流程实例 ID(procInsId)' },
|
||||
],
|
||||
columns: ['step', 'operator', 'action', 'time'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://tae.alibaba-inc.com/expense/pc.html?_authType=SAML');
|
||||
await page.wait(2);
|
||||
const result = await page.evaluate(`(async () => {
|
||||
const res = await fetch('/approval/getList.json?taskId=&procInsId=${kwargs.proc_ins_id}', {
|
||||
credentials: 'include'
|
||||
});
|
||||
const data = await res.json();
|
||||
return data?.content?.operatorRecords || [];
|
||||
})()`);
|
||||
return (result as any[]).map((r, i) => ({
|
||||
step: i + 1,
|
||||
operator: r.operatorName || r.userId,
|
||||
action: r.operationType,
|
||||
time: r.operateTime,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
**转换要点**:
|
||||
1. URL 中的动态 ID(`procInsId`、`taskId` 等)提取为 `args`
|
||||
2. `captured.json` 里的真实 body 结构用于确定正确的数据路径(如 `content.operatorRecords`)
|
||||
3. tae 系统统一用 `{ success, content, errorCode, errorMsg }` 外层包裹,取数据要走 `content.*`
|
||||
4. 认证方式:cookie(`credentials: 'include'`),不需要额外 header
|
||||
5. 文件放入 `src/clis/<site>/`,无需手动注册,`npm run build` 后自动发现
|
||||
|
||||
### 故障排查
|
||||
|
||||
| 现象 | 原因 | 解法 |
|
||||
|------|------|------|
|
||||
| 捕获 0 条请求 | 拦截器注入失败,或页面无 JSON API | 检查 daemon 是否运行:`curl localhost:19825/status` |
|
||||
| 捕获量少(1~3 条) | 页面是只读详情页,首屏数据已在注入前发出 | 手动操作触发更多请求(搜索/翻页),或换用列表页 |
|
||||
| 候选 YAML 为 0 | 捕获到的 JSON 都没有 array 结构 | 直接看 `captured.json` 手写 TS CLI |
|
||||
| 新开的 tab 没有被拦截 | 轮询间隔内 tab 已关闭 | 缩短 `--poll 500` |
|
||||
| 二次运行 record 时数据不连续 | 正常,每次 `record` 启动都是新的 automation window | 无需处理 |
|
||||
|
||||
## Creating Adapters
|
||||
|
||||
> [!TIP]
|
||||
> **快速模式**:如果你只想为一个具体页面生成一个命令,直接看 [CLI-ONESHOT.md](./CLI-ONESHOT.md)。
|
||||
> 只需要一个 URL + 一句话描述,4 步搞定。
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **完整模式 — 在写任何代码之前,先阅读 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。**
|
||||
> 它包含:① AI Agent 浏览器探索工作流 ② 认证策略决策树 ③ 平台 SDK(如 Bilibili 的 `apiGet`/`fetchJson`)④ YAML vs TS 选择指南 ⑤ `tap` 步骤调试方法 ⑥ 级联请求模板 ⑦ 常见陷阱表。
|
||||
> **下方仅为简化模板参考,直接使用极易踩坑。**
|
||||
|
||||
### YAML Pipeline (declarative, recommended)
|
||||
|
||||
Create `src/clis/<site>/<name>.yaml`:
|
||||
|
||||
```yaml
|
||||
site: mysite
|
||||
name: hot
|
||||
description: Hot topics
|
||||
domain: www.mysite.com
|
||||
strategy: cookie # public | cookie | header | intercept | ui
|
||||
browser: true
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
description: Number of items
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.mysite.com
|
||||
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const res = await fetch('/api/hot', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return d.data.items.map(item => ({
|
||||
title: item.title,
|
||||
score: item.score,
|
||||
}));
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
score: ${{ item.score }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, score]
|
||||
```
|
||||
|
||||
For public APIs (no browser):
|
||||
|
||||
```yaml
|
||||
strategy: public
|
||||
browser: false
|
||||
|
||||
pipeline:
|
||||
- fetch:
|
||||
url: https://api.example.com/hot.json
|
||||
- select: data.items
|
||||
- map:
|
||||
title: ${{ item.title }}
|
||||
- limit: ${{ args.limit }}
|
||||
```
|
||||
|
||||
### TypeScript Adapter (programmatic)
|
||||
|
||||
Create `src/clis/<site>/<name>.ts`. It will be automatically dynamically loaded (DO NOT manually import it in `index.ts`):
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'mysite',
|
||||
name: 'search',
|
||||
strategy: Strategy.INTERCEPT, // Or COOKIE
|
||||
args: [{ name: 'query', required: true, positional: true }],
|
||||
columns: ['rank', 'title', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://www.mysite.com/search');
|
||||
|
||||
// Inject native XHR/Fetch interceptor hook
|
||||
await page.installInterceptor('/api/search');
|
||||
|
||||
// Auto scroll down to trigger lazy loading
|
||||
await page.autoScroll({ times: 3, delayMs: 2000 });
|
||||
|
||||
// Retrieve intercepted JSON payloads
|
||||
const requests = await page.getInterceptedRequests();
|
||||
|
||||
let results = [];
|
||||
for (const req of requests) {
|
||||
results.push(...req.data.items);
|
||||
}
|
||||
return results.map((item, i) => ({
|
||||
rank: i + 1, title: item.title, url: item.url,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
**When to use TS**: XHR interception (`page.installInterceptor`), infinite scrolling (`page.autoScroll`), cookie extraction, complex data transforms (like GraphQL unwrapping).
|
||||
|
||||
## Pipeline Steps
|
||||
|
||||
| Step | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| `navigate` | Go to URL | `navigate: https://example.com` |
|
||||
| `fetch` | HTTP request (browser cookies) | `fetch: { url: "...", params: { q: "..." } }` |
|
||||
| `evaluate` | Run JavaScript in page | `evaluate: \| (async () => { ... })()` |
|
||||
| `select` | Extract JSON path | `select: data.items` |
|
||||
| `map` | Map fields | `map: { title: "${{ item.title }}" }` |
|
||||
| `filter` | Filter items | `filter: item.score > 100` |
|
||||
| `sort` | Sort items | `sort: { by: score, order: desc }` |
|
||||
| `limit` | Cap result count | `limit: ${{ args.limit }}` |
|
||||
| `intercept` | Declarative XHR capture | `intercept: { trigger: "navigate:...", capture: "api/hot" }` |
|
||||
| `tap` | Store action + XHR capture | `tap: { store: "feed", action: "fetchFeeds", capture: "homefeed" }` |
|
||||
| `snapshot` | Page accessibility tree | `snapshot: { interactive: true }` |
|
||||
| `click` | Click element | `click: ${{ ref }}` |
|
||||
| `type` | Type text | `type: { ref: "@1", text: "hello" }` |
|
||||
| `wait` | Wait for time/text | `wait: 2` or `wait: { text: "loaded" }` |
|
||||
| `press` | Press key | `press: Enter` |
|
||||
|
||||
## Template Syntax
|
||||
|
||||
```yaml
|
||||
# Arguments with defaults
|
||||
${{ args.query }}
|
||||
${{ args.limit | default(20) }}
|
||||
|
||||
# Current item (in map/filter)
|
||||
${{ item.title }}
|
||||
${{ item.data.nested.field }}
|
||||
|
||||
# Index (0-based)
|
||||
${{ index }}
|
||||
${{ index + 1 }}
|
||||
```
|
||||
|
||||
## 5-Tier Authentication Strategy
|
||||
|
||||
| Tier | Name | Method | Example |
|
||||
|------|------|--------|---------|
|
||||
| 1 | `public` | No auth, Node.js fetch | Hacker News, V2EX |
|
||||
| 2 | `cookie` | Browser fetch with `credentials: include` | Bilibili, Zhihu |
|
||||
| 3 | `header` | Custom headers (ct0, Bearer) | Twitter GraphQL |
|
||||
| 4 | `intercept` | XHR interception + store mutation | 小红书 Pinia |
|
||||
| 5 | `ui` | Full UI automation (click/type/scroll) | Last resort |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENCLI_DAEMON_PORT` | 19825 | Daemon listen port |
|
||||
| `OPENCLI_BROWSER_CONNECT_TIMEOUT` | 30 | Browser connection timeout (sec) |
|
||||
| `OPENCLI_BROWSER_COMMAND_TIMEOUT` | 45 | Command execution timeout (sec) |
|
||||
| `OPENCLI_BROWSER_EXPLORE_TIMEOUT` | 120 | Explore timeout (sec) |
|
||||
| `OPENCLI_VERBOSE` | — | Show daemon/extension logs |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| `npx not found` | Install Node.js: `brew install node` |
|
||||
| `Extension not connected` | 1) Chrome must be open 2) Install opencli Browser Bridge extension |
|
||||
| `Target page context` error | Add `navigate:` step before `evaluate:` in YAML |
|
||||
| Empty table data | Check if evaluate returns correct data path |
|
||||
| Daemon issues | `curl localhost:19825/status` to check, `curl localhost:19825/logs` for extension logs |
|
||||
@@ -0,0 +1 @@
|
||||
56/59
|
||||
@@ -0,0 +1 @@
|
||||
31/31
|
||||
@@ -0,0 +1,688 @@
|
||||
[
|
||||
{
|
||||
"name": "extract-title-example",
|
||||
"steps": [
|
||||
"opencli operate open https://example.com",
|
||||
"opencli operate eval \"document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Example Domain"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-title-iana",
|
||||
"steps": [
|
||||
"opencli operate open https://www.iana.org",
|
||||
"opencli operate eval \"document.querySelector('h1')?.textContent\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-paragraph-wiki-js",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org/wiki/JavaScript",
|
||||
"opencli operate eval \"(() => { const ps = document.querySelectorAll('#mw-content-text .mw-parser-output > p'); for (const p of ps) { const t = p.textContent?.trim(); if (t && t.length > 50 && !t.startsWith('{')) return t.slice(0,300); } return ''; })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "programming language"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-paragraph-wiki-python",
|
||||
"steps": [
|
||||
"opencli operate open \"https://en.wikipedia.org/wiki/Python_(programming_language)\"",
|
||||
"opencli operate eval \"(() => { const ps = document.querySelectorAll('#mw-content-text .mw-parser-output > p'); for (const p of ps) { const t = p.textContent?.trim(); if (t && t.length > 50 && !t.startsWith('{')) return t.slice(0,300); } return ''; })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "programming language"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-github-stars",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/browser-use/browser-use",
|
||||
"opencli operate eval \"document.querySelector('#repo-stars-counter-star')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "\\d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-github-description",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/anthropics/claude-code",
|
||||
"opencli operate eval \"document.querySelector('p.f4, [data-testid=about-description], .f4.my-3, .BorderGrid-cell p')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-github-readme-heading",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/vercel/next.js",
|
||||
"opencli operate eval \"document.querySelector('article h1, article h2')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-npm-downloads",
|
||||
"steps": [
|
||||
"opencli operate open https://www.npmjs.com/package/zod",
|
||||
"opencli operate eval \"document.querySelector('[data-nosnippet]')?.textContent?.trim() || document.querySelector('p.f2874b88')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "\\d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "extract-npm-description",
|
||||
"steps": [
|
||||
"opencli operate open https://www.npmjs.com/package/express",
|
||||
"opencli operate eval \"document.querySelector('p[class*=description], [data-testid=package-description], #readme p')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-hn-top5",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.titleline > a')].slice(0,5).map(a=>({title:a.textContent,url:a.href})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-hn-top10",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.athing')].slice(0,10).map(tr=>{const a=tr.querySelector('.titleline>a');const s=tr.nextElementSibling?.querySelector('.score');return{title:a?.textContent,score:parseInt(s?.textContent)||0}}))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 10
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-books-5",
|
||||
"steps": [
|
||||
"opencli operate open https://books.toscrape.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.product_pod')].slice(0,5).map(el=>({title:el.querySelector('h3 a')?.getAttribute('title'),price:el.querySelector('.price_color')?.textContent})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-books-10",
|
||||
"steps": [
|
||||
"opencli operate open https://books.toscrape.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.product_pod')].slice(0,10).map(el=>({title:el.querySelector('h3 a')?.getAttribute('title'),price:el.querySelector('.price_color')?.textContent})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 10
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-quotes-3",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote')].slice(0,3).map(el=>({text:el.querySelector('.text')?.textContent,author:el.querySelector('.author')?.textContent})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-quotes-tags",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote')].slice(0,5).map(el=>({text:el.querySelector('.text')?.textContent,author:el.querySelector('.author')?.textContent,tags:[...el.querySelectorAll('.tag')].map(t=>t.textContent)})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-github-trending",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/trending",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row')].slice(0,3).map(el=>({name:el.querySelector('h2 a')?.textContent?.trim().replace(/\\s+/g,' '),desc:el.querySelector('p')?.textContent?.trim()})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-github-trending-lang",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/trending/python",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row')].slice(0,5).map(el=>({name:el.querySelector('h2 a')?.textContent?.trim().replace(/\\s+/g,' ')})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-jsonplaceholder-posts",
|
||||
"steps": [
|
||||
"opencli operate open https://jsonplaceholder.typicode.com/posts",
|
||||
"opencli operate eval \"JSON.stringify(JSON.parse(document.body.innerText).slice(0,5).map(p=>({id:p.id,title:p.title})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list-jsonplaceholder-users",
|
||||
"steps": [
|
||||
"opencli operate open https://jsonplaceholder.typicode.com/users",
|
||||
"opencli operate eval \"JSON.stringify(JSON.parse(document.body.innerText).map(u=>({name:u.name,email:u.email})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "search-google",
|
||||
"steps": [
|
||||
"opencli operate open https://www.google.com",
|
||||
"opencli operate eval \"document.querySelector('textarea[name=q], input[name=q]').value='opencli github';document.querySelector('form').submit();'submitted'\"",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('h3')].slice(0,3).map(h=>h.textContent))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "index 5 may vary"
|
||||
},
|
||||
{
|
||||
"name": "search-ddg",
|
||||
"steps": [
|
||||
"opencli operate open https://duckduckgo.com",
|
||||
"opencli operate state",
|
||||
"opencli operate type 1 \"weather beijing\"",
|
||||
"opencli operate keys Enter",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=result-title-a]')].slice(0,3).map(a=>a.textContent))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "search-ddg-tech",
|
||||
"steps": [
|
||||
"opencli operate open https://duckduckgo.com",
|
||||
"opencli operate eval \"document.querySelector('input[name=q]').value='TypeScript tutorial';document.querySelector('form').submit();'submitted'\"",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=result-title-a], .result__a')].slice(0,3).map(a=>({title:a.textContent,url:a.href})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "search-wiki",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org",
|
||||
"opencli operate eval \"document.querySelector('input[name=search]').value='Rust programming language';document.querySelector('form#searchform, form[role=search]').submit();'submitted'\"",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const ps = document.querySelectorAll('#mw-content-text .mw-parser-output > p'); for (const p of ps) { const t = p.textContent?.trim(); if (t && t.length > 50) return t.slice(0,300); } return ''; })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "programming language"
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "search-npm",
|
||||
"steps": [
|
||||
"opencli operate open https://www.npmjs.com",
|
||||
"opencli operate state",
|
||||
"opencli operate type 1 \"react\"",
|
||||
"opencli operate keys Enter",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=pkg-list-item] h3, section h3')].slice(0,3).map(h=>h.textContent?.trim()))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "search-github",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/search?q=browser+automation&type=repositories",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.search-title a, [data-testid=results-list] a.Link--primary')].slice(0,3).map(a=>a.textContent?.trim().replace(/\\\\s+/g,' ')))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "nav-click-link-example",
|
||||
"steps": [
|
||||
"opencli operate open https://example.com",
|
||||
"opencli operate eval \"document.querySelector('a')?.click();'clicked'\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "IANA"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nav-click-hn-first",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate eval \"document.querySelector('.titleline a')?.click();'clicked'\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nav-click-hn-comments",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate eval \"document.querySelector('.subtext a:last-child')?.click(); 'clicked'\"",
|
||||
"opencli operate eval \"document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nav-click-wiki-link",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org/wiki/JavaScript",
|
||||
"opencli operate eval \"document.querySelector('#toc a, .toc a, [href=\\\"#History\\\"]')?.click(); 'clicked'\"",
|
||||
"opencli operate eval \"document.querySelector('#History, #History ~ p')?.textContent?.slice(0,100)\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nav-click-github-tab",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/vercel/next.js",
|
||||
"opencli operate eval \"document.querySelector('[data-tab-item=i1issues-tab] a, #issues-tab')?.click(); 'clicked'\"",
|
||||
"opencli operate eval \"document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nav-go-back",
|
||||
"steps": [
|
||||
"opencli operate open https://example.com",
|
||||
"opencli operate eval \"document.querySelector('a')?.click();'clicked'\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate back",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Example Domain"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "nav-multi-step",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"document.querySelector('.next a')?.click(); 'clicked'\"",
|
||||
"opencli operate eval \"document.querySelector('.quote .text')?.textContent\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "scroll-footer-quotes",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate scroll down",
|
||||
"opencli operate scroll down",
|
||||
"opencli operate eval \"document.querySelector('footer, .footer, .tags-box')?.textContent?.trim().slice(0,100)\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "scroll-footer-books",
|
||||
"steps": [
|
||||
"opencli operate open https://books.toscrape.com",
|
||||
"opencli operate scroll down",
|
||||
"opencli operate scroll down",
|
||||
"opencli operate eval \"document.querySelector('.pager .current')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "\\d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "scroll-long-page",
|
||||
"steps": [
|
||||
"opencli operate open https://jsonplaceholder.typicode.com/posts",
|
||||
"opencli operate eval \"JSON.parse(document.body.innerText).length\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "\\d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "scroll-find-element",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"document.querySelector('.next a')?.href\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "scroll-lazy-load",
|
||||
"steps": [
|
||||
"opencli operate open https://books.toscrape.com",
|
||||
"opencli operate eval \"document.querySelectorAll('article.product_pod').length\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "\\d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "form-simple-name",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/forms/post",
|
||||
"opencli operate eval \"var el=document.querySelector('[name=custname]');el.value='OpenCLI Test';el.dispatchEvent(new Event('input',{bubbles:true}));el.value\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "OpenCLI"
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "form-text-inputs",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/forms/post",
|
||||
"opencli operate eval \"var n=document.querySelector('[name=custname]');n.value='Alice';n.dispatchEvent(new Event('input',{bubbles:true}));var t=document.querySelector('[name=custtel]');t.value='555-1234';t.dispatchEvent(new Event('input',{bubbles:true}));n.value+'|'+t.value\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Alice"
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "form-radio-select",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/forms/post",
|
||||
"opencli operate eval \"document.querySelector('[value=medium]').checked=true;document.querySelector('[value=medium]').dispatchEvent(new Event('change',{bubbles:true}));document.querySelector('[value=medium]').checked\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "true"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "form-checkbox",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/forms/post",
|
||||
"opencli operate eval \"var cb=document.querySelector('[value=cheese]');cb.checked=true;cb.dispatchEvent(new Event('change',{bubbles:true}));cb.checked\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "true"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "form-textarea",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/forms/post",
|
||||
"opencli operate eval \"var ta=document.querySelector('textarea[name=comments]');ta.value='AutoResearch test';ta.dispatchEvent(new Event('input',{bubbles:true}));ta.value\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoResearch"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "form-login-fake",
|
||||
"steps": [
|
||||
"opencli operate open https://the-internet.herokuapp.com/login",
|
||||
"opencli operate eval \"var u=document.querySelector('#username');u.value='testuser';u.dispatchEvent(new Event('input',{bubbles:true}));var p=document.querySelector('#password');p.value='testpass';p.dispatchEvent(new Event('input',{bubbles:true}));u.value+'|'+p.value\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "testuser"
|
||||
},
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "complex-wiki-toc",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org/wiki/JavaScript",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.toc li a, #toc li a, .vector-toc-contents a')].slice(0,8).map(a=>a.textContent?.trim()))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "complex-books-detail",
|
||||
"steps": [
|
||||
"opencli operate open https://books.toscrape.com",
|
||||
"opencli operate eval \"document.querySelector('article.product_pod h3 a')?.click();'clicked'\"",
|
||||
"opencli operate eval \"JSON.stringify({title:document.querySelector('h1')?.textContent,price:document.querySelector('.price_color')?.textContent})\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "complex-quotes-page2",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"document.querySelector('.next a')?.click();'clicked'\"",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote')].slice(0,3).map(el=>({text:el.querySelector('.text')?.textContent,author:el.querySelector('.author')?.textContent})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "complex-github-repo-info",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/expressjs/express",
|
||||
"opencli operate eval \"JSON.stringify({lang:document.querySelector('[itemprop=programmingLanguage]')?.textContent?.trim(),license:document.querySelector('[data-analytics-event*=license] span, .Layout-sidebar [href*=LICENSE]')?.textContent?.trim()})\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "complex-hn-story-comments",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate eval \"document.querySelector('.subtext a:last-child')?.click();'clicked'\"",
|
||||
"opencli operate eval \"document.querySelector('.fatitem .titleline a')?.textContent\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "complex-multi-extract",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org/wiki/TypeScript",
|
||||
"opencli operate eval \"JSON.stringify({title:document.title,firstParagraph:document.querySelector('#mw-content-text p')?.textContent?.slice(0,150)})\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "TypeScript"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "bench-reddit-top5",
|
||||
"steps": [
|
||||
"opencli operate open https://old.reddit.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('#siteTable .thing .title a.title')].slice(0,5).map(a=>a.textContent))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-imdb-matrix",
|
||||
"steps": [
|
||||
"opencli operate open https://www.imdb.com/title/tt0133093/",
|
||||
"opencli operate eval \"JSON.stringify({title:document.querySelector('h1 span, [data-testid=hero__pageTitle] span')?.textContent,year:document.querySelector('a[href*=releaseinfo], [data-testid=hero-title-block__metadata] a')?.textContent})\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "1999"
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-npm-zod",
|
||||
"steps": [
|
||||
"opencli operate open https://www.npmjs.com/package/zod",
|
||||
"opencli operate eval \"JSON.stringify({name:document.querySelector('h1 span, #top h2')?.textContent?.trim(),description:document.querySelector('[data-testid=package-description], p.package-description-redundant')?.textContent?.trim()})\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-wiki-search",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org/wiki/Machine_learning",
|
||||
"opencli operate eval \"(() => { const ps = document.querySelectorAll('#mw-content-text .mw-parser-output > p'); for (const p of ps) { const t = p.textContent?.trim(); if (t && t.length > 50) return t.slice(0,300); } return ''; })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "learning"
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-github-profile",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/torvalds",
|
||||
"opencli operate eval \"JSON.stringify({name:document.querySelector('[itemprop=name]')?.textContent?.trim(),bio:document.querySelector('[data-bio-text]')?.textContent?.trim()||document.querySelector('.p-note')?.textContent?.trim()})\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-books-category",
|
||||
"steps": [
|
||||
"opencli operate open https://books.toscrape.com",
|
||||
"opencli operate eval \"document.querySelector('a[href*=science]')?.click();'clicked'\"",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.product_pod h3 a')].slice(0,3).map(a=>a.getAttribute('title')))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-quotes-author",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"document.querySelector('.author + a, a[href*=author]')?.click();'clicked'\"",
|
||||
"opencli operate eval \"document.querySelector('.author-description, .author-details p')?.textContent?.slice(0,100)\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-ddg-images",
|
||||
"steps": [
|
||||
"opencli operate open https://duckduckgo.com",
|
||||
"opencli operate eval \"document.querySelector('input[name=q]').value='sunset';document.querySelector('form').submit();'submitted'\"",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=result-title-a], .result__a')].slice(0,3).map(a=>a.textContent))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"set": "test",
|
||||
"note": "index may vary"
|
||||
},
|
||||
{
|
||||
"name": "bench-httpbin-headers",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/headers",
|
||||
"opencli operate eval \"JSON.parse(document.body.innerText).headers['User-Agent']\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
},
|
||||
"set": "test"
|
||||
},
|
||||
{
|
||||
"name": "bench-jsonapi-todo",
|
||||
"steps": [
|
||||
"opencli operate open https://jsonplaceholder.typicode.com/todos",
|
||||
"opencli operate eval \"JSON.stringify(JSON.parse(document.body.innerText).slice(0,5).map(t=>({id:t.id,title:t.title,completed:t.completed})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 5
|
||||
},
|
||||
"set": "test"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,185 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
/**
|
||||
* Layer 1: Deterministic Browse Command Testing
|
||||
*
|
||||
* Runs predefined opencli operate command sequences against real websites.
|
||||
* No LLM involved — tests command reliability only.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx autoresearch/eval-browse.ts # Run all tasks
|
||||
* npx tsx autoresearch/eval-browse.ts --task hn-top5 # Run single task
|
||||
*/
|
||||
|
||||
import { execSync } from 'node:child_process';
|
||||
import { readFileSync, writeFileSync, mkdirSync, readdirSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const TASKS_FILE = join(__dirname, 'browse-tasks.json');
|
||||
const RESULTS_DIR = join(__dirname, 'results');
|
||||
const BASELINE_FILE = join(__dirname, 'baseline-browse.txt');
|
||||
|
||||
interface BrowseTask {
|
||||
name: string;
|
||||
steps: string[];
|
||||
judge: JudgeCriteria;
|
||||
set?: 'test';
|
||||
note?: string;
|
||||
}
|
||||
|
||||
type JudgeCriteria =
|
||||
| { type: 'contains'; value: string }
|
||||
| { type: 'arrayMinLength'; minLength: number }
|
||||
| { type: 'nonEmpty' }
|
||||
| { type: 'matchesPattern'; pattern: string };
|
||||
|
||||
interface TaskResult {
|
||||
name: string;
|
||||
passed: boolean;
|
||||
duration: number;
|
||||
error?: string;
|
||||
set: 'train' | 'test';
|
||||
}
|
||||
|
||||
function judge(criteria: JudgeCriteria, output: string): boolean {
|
||||
try {
|
||||
switch (criteria.type) {
|
||||
case 'contains':
|
||||
return output.toLowerCase().includes(criteria.value.toLowerCase());
|
||||
case 'arrayMinLength': {
|
||||
try {
|
||||
const arr = JSON.parse(output);
|
||||
if (Array.isArray(arr)) return arr.length >= criteria.minLength;
|
||||
} catch { /* not JSON array */ }
|
||||
return false;
|
||||
}
|
||||
case 'nonEmpty':
|
||||
return output.trim().length > 0 && output.trim() !== 'null' && output.trim() !== 'undefined';
|
||||
case 'matchesPattern':
|
||||
return new RegExp(criteria.pattern).test(output);
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function runCommand(cmd: string): string {
|
||||
try {
|
||||
return execSync(cmd, {
|
||||
cwd: join(__dirname, '..'),
|
||||
timeout: 30000,
|
||||
encoding: 'utf-8',
|
||||
env: process.env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
}).trim();
|
||||
} catch (err: any) {
|
||||
return err.stdout?.trim() ?? '';
|
||||
}
|
||||
}
|
||||
|
||||
function runTask(task: BrowseTask): TaskResult {
|
||||
const start = Date.now();
|
||||
let lastOutput = '';
|
||||
|
||||
try {
|
||||
for (const step of task.steps) {
|
||||
lastOutput = runCommand(step);
|
||||
}
|
||||
|
||||
const passed = judge(task.judge, lastOutput);
|
||||
|
||||
return {
|
||||
name: task.name,
|
||||
passed,
|
||||
duration: Date.now() - start,
|
||||
error: passed ? undefined : `Output: ${lastOutput.slice(0, 100)}`,
|
||||
set: task.set === 'test' ? 'test' : 'train',
|
||||
};
|
||||
} catch (err: any) {
|
||||
return {
|
||||
name: task.name,
|
||||
passed: false,
|
||||
duration: Date.now() - start,
|
||||
error: err.message?.slice(0, 100),
|
||||
set: task.set === 'test' ? 'test' : 'train',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const singleTask = args.includes('--task') ? args[args.indexOf('--task') + 1] : null;
|
||||
|
||||
const allTasks: BrowseTask[] = JSON.parse(readFileSync(TASKS_FILE, 'utf-8'));
|
||||
const tasks = singleTask ? allTasks.filter(t => t.name === singleTask) : allTasks;
|
||||
|
||||
if (tasks.length === 0) {
|
||||
console.error(`Task "${singleTask}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`\n🔬 Layer 1: Browse Commands — ${tasks.length} tasks\n`);
|
||||
|
||||
const results: TaskResult[] = [];
|
||||
|
||||
for (let i = 0; i < tasks.length; i++) {
|
||||
const task = tasks[i];
|
||||
process.stdout.write(` [${i + 1}/${tasks.length}] ${task.name}...`);
|
||||
|
||||
const result = runTask(task);
|
||||
results.push(result);
|
||||
|
||||
const icon = result.passed ? '✓' : '✗';
|
||||
console.log(` ${icon} (${(result.duration / 1000).toFixed(1)}s)`);
|
||||
|
||||
// Close browser between tasks for clean state
|
||||
if (i < tasks.length - 1) {
|
||||
try { runCommand('opencli operate close'); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
|
||||
// Final close
|
||||
try { runCommand('opencli operate close'); } catch { /* ignore */ }
|
||||
|
||||
// Summary
|
||||
const trainResults = results.filter(r => r.set === 'train');
|
||||
const testResults = results.filter(r => r.set === 'test');
|
||||
const totalPassed = results.filter(r => r.passed).length;
|
||||
const trainPassed = trainResults.filter(r => r.passed).length;
|
||||
const testPassed = testResults.filter(r => r.passed).length;
|
||||
const totalDuration = results.reduce((s, r) => s + r.duration, 0);
|
||||
|
||||
console.log(`\n${'─'.repeat(50)}`);
|
||||
console.log(` Score: ${totalPassed}/${results.length} (train: ${trainPassed}/${trainResults.length}, test: ${testPassed}/${testResults.length})`);
|
||||
console.log(` Time: ${Math.round(totalDuration / 60000)}min`);
|
||||
|
||||
const failures = results.filter(r => !r.passed);
|
||||
if (failures.length > 0) {
|
||||
console.log(`\n Failures:`);
|
||||
for (const f of failures) {
|
||||
console.log(` ✗ ${f.name}: ${f.error ?? 'unknown'}`);
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Save result
|
||||
mkdirSync(RESULTS_DIR, { recursive: true });
|
||||
const existing = readdirSync(RESULTS_DIR).filter(f => f.startsWith('browse-')).length;
|
||||
const roundNum = String(existing + 1).padStart(3, '0');
|
||||
const resultPath = join(RESULTS_DIR, `browse-${roundNum}.json`);
|
||||
writeFileSync(resultPath, JSON.stringify({
|
||||
timestamp: new Date().toISOString(),
|
||||
score: `${totalPassed}/${results.length}`,
|
||||
trainScore: `${trainPassed}/${trainResults.length}`,
|
||||
testScore: `${testPassed}/${testResults.length}`,
|
||||
duration: `${Math.round(totalDuration / 60000)}min`,
|
||||
tasks: results,
|
||||
}, null, 2), 'utf-8');
|
||||
console.log(` Results saved to: ${resultPath}`);
|
||||
console.log(`\nSCORE=${totalPassed}/${results.length}`);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,248 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
/**
|
||||
* Layer 2: Claude Code Skill E2E Testing (LLM Judge)
|
||||
*
|
||||
* Spawns Claude Code with the opencli-operate skill. Claude Code
|
||||
* completes the task using browse commands AND judges its own result.
|
||||
*
|
||||
* Task format: YAML with judge_context (multi-criteria, like Browser Use)
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx autoresearch/eval-skill.ts # Run all
|
||||
* npx tsx autoresearch/eval-skill.ts --task hn-top5 # Run single
|
||||
*/
|
||||
|
||||
import { execSync } from 'node:child_process';
|
||||
import { readFileSync, writeFileSync, mkdirSync, readdirSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const TASKS_FILE = join(__dirname, 'skill-tasks.yaml');
|
||||
const RESULTS_DIR = join(__dirname, 'results');
|
||||
const SKILL_PATH = join(__dirname, '..', 'skills', 'opencli-operate', 'SKILL.md');
|
||||
|
||||
// ── Types ──────────────────────────────────────────────────────────
|
||||
|
||||
interface SkillTask {
|
||||
name: string;
|
||||
task: string;
|
||||
url?: string;
|
||||
judge_context: string[];
|
||||
max_steps?: number;
|
||||
}
|
||||
|
||||
interface TaskResult {
|
||||
name: string;
|
||||
passed: boolean;
|
||||
duration: number;
|
||||
cost: number;
|
||||
explanation: string;
|
||||
}
|
||||
|
||||
// ── Task Definitions (inline, to avoid YAML dependency) ────────────
|
||||
|
||||
const TASKS: SkillTask[] = [
|
||||
// Extract
|
||||
{ name: "extract-title-example", task: "Extract the main heading text from this page", url: "https://example.com", judge_context: ["Output must contain 'Example Domain'"] },
|
||||
{ name: "extract-paragraph-wiki", task: "Extract the first paragraph of the JavaScript article", url: "https://en.wikipedia.org/wiki/JavaScript", judge_context: ["Output must mention 'programming language'", "Output must contain actual paragraph text, not just the title"] },
|
||||
{ name: "extract-github-stars", task: "Find the number of stars on this repository", url: "https://github.com/browser-use/browser-use", judge_context: ["Output must contain a number (the star count)"] },
|
||||
{ name: "extract-npm-downloads", task: "Find the weekly download count for this package", url: "https://www.npmjs.com/package/zod", judge_context: ["Output must contain a number (weekly downloads)"] },
|
||||
|
||||
// List extraction
|
||||
{ name: "list-hn-top5", task: "Extract the top 5 stories with their titles", url: "https://news.ycombinator.com", judge_context: ["Output must contain 5 story titles", "Each title must be an actual HN story, not made up"] },
|
||||
{ name: "list-books-5", task: "Extract the first 5 books with their title and price", url: "https://books.toscrape.com", judge_context: ["Output must contain 5 books", "Each book must have a title and a price"] },
|
||||
{ name: "list-quotes-3", task: "Extract the first 3 quotes with their text and author", url: "https://quotes.toscrape.com", judge_context: ["Output must contain 3 quotes", "Each quote must have text and an author name"] },
|
||||
{ name: "list-github-trending", task: "Extract the top 3 trending repositories with name and description", url: "https://github.com/trending", judge_context: ["Output must contain 3 repositories", "Each must have a repo name"] },
|
||||
{ name: "list-jsonplaceholder", task: "Extract the first 5 posts with their title", url: "https://jsonplaceholder.typicode.com/posts", judge_context: ["Output must contain 5 posts", "Each post must have a title"] },
|
||||
|
||||
// Search
|
||||
{ name: "search-ddg", task: "Search for 'TypeScript tutorial' and extract the first 3 result titles", url: "https://duckduckgo.com", judge_context: ["The agent must type a search query", "Output must contain at least 3 search result titles"] },
|
||||
{ name: "search-npm", task: "Search for 'react' and extract the top 3 package names", url: "https://www.npmjs.com", judge_context: ["The agent must search for 'react'", "Output must contain at least 3 package names"] },
|
||||
{ name: "search-wiki", task: "Search for 'Rust programming language' and extract the first sentence of the article", url: "https://en.wikipedia.org", judge_context: ["The agent must search and navigate to the article", "Output must mention 'programming language'"] },
|
||||
|
||||
// Navigation
|
||||
{ name: "nav-click-link", task: "Click the 'More information...' link and extract the heading of the new page", url: "https://example.com", judge_context: ["The agent must click a link", "Output must contain 'IANA' or reference the new page"] },
|
||||
{ name: "nav-click-hn", task: "Click on the first story link and tell me the title of the page you land on", url: "https://news.ycombinator.com", judge_context: ["The agent must click a story link", "Output must contain the title of the destination page"] },
|
||||
{ name: "nav-go-back", task: "Click the 'More information...' link, then go back, and tell me the heading of the original page", url: "https://example.com", judge_context: ["The agent must click a link then go back", "Output must contain 'Example Domain'"] },
|
||||
{ name: "nav-multi-step", task: "Click the Next page link at the bottom, then extract the first quote from page 2", url: "https://quotes.toscrape.com", judge_context: ["The agent must navigate to page 2", "Output must contain a quote from page 2"] },
|
||||
|
||||
// Scroll
|
||||
{ name: "scroll-footer", task: "Scroll to the bottom and extract the footer text", url: "https://quotes.toscrape.com", judge_context: ["The agent must scroll down", "Output must contain footer or bottom-of-page content"] },
|
||||
{ name: "scroll-pagination", task: "Find the pagination info at the bottom of the page", url: "https://books.toscrape.com", judge_context: ["Output must contain page number or pagination info"] },
|
||||
|
||||
// Form
|
||||
{ name: "form-fill-basic", task: "Fill the Customer Name with 'OpenCLI' and Telephone with '555-0100'. Do not submit.", url: "https://httpbin.org/forms/post", judge_context: ["The agent must type 'OpenCLI' into a name field", "The agent must type '555-0100' into a phone field", "The form must NOT be submitted"] },
|
||||
{ name: "form-radio", task: "Select the 'Medium' pizza size option. Do not submit.", url: "https://httpbin.org/forms/post", judge_context: ["The agent must select a radio button for Medium size"] },
|
||||
{ name: "form-login", task: "Fill the username with 'testuser' and password with 'testpass'. Do not submit.", url: "https://the-internet.herokuapp.com/login", judge_context: ["The agent must fill the username field", "The agent must fill the password field", "The form must NOT be submitted"] },
|
||||
|
||||
// Complex
|
||||
{ name: "complex-wiki-toc", task: "Extract the table of contents headings", url: "https://en.wikipedia.org/wiki/JavaScript", judge_context: ["Output must contain at least 5 section headings from the table of contents"] },
|
||||
{ name: "complex-books-detail", task: "Click on the first book and extract its title and price from the detail page", url: "https://books.toscrape.com", judge_context: ["The agent must click on a book", "Output must contain the book title", "Output must contain a price"] },
|
||||
{ name: "complex-quotes-page2", task: "Navigate to page 2 and extract the first 3 quotes with their authors", url: "https://quotes.toscrape.com", judge_context: ["The agent must navigate to page 2", "Output must contain 3 quotes with authors"] },
|
||||
{ name: "complex-multi-extract", task: "Extract both the page title and the first paragraph text", url: "https://en.wikipedia.org/wiki/TypeScript", judge_context: ["Output must contain 'TypeScript'", "Output must contain actual paragraph text"] },
|
||||
|
||||
// Bench (harder, real-world)
|
||||
{ name: "bench-reddit", task: "Extract the titles of the top 5 posts", url: "https://old.reddit.com", judge_context: ["Output must contain 5 post titles", "Titles must be actual Reddit posts"] },
|
||||
{ name: "bench-imdb", task: "Find the year and rating of The Matrix", url: "https://www.imdb.com/title/tt0133093/", judge_context: ["Output must contain '1999'", "Output must contain a rating number"] },
|
||||
{ name: "bench-github-profile", task: "Extract the bio and number of public repositories", url: "https://github.com/torvalds", judge_context: ["Output must contain bio text or 'Linux'", "Output must contain a number for repos"] },
|
||||
{ name: "bench-httpbin", task: "Extract the User-Agent header shown on this page", url: "https://httpbin.org/headers", judge_context: ["Output must contain a User-Agent string"] },
|
||||
{ name: "bench-jsonapi-todo", task: "Extract the first 5 todo items with their title and completion status", url: "https://jsonplaceholder.typicode.com/todos", judge_context: ["Output must contain 5 todo items", "Each must have a title and completed status"] },
|
||||
|
||||
// Codex form (the real test)
|
||||
{ name: "codex-form-fill", task: "Fill the basic information using 'opencli' as the identity (first name=open, last name=cli, email=opencli@example.com, GitHub username=opencli). Do NOT submit the form.", url: "https://openai.com/form/codex-for-oss/", judge_context: ["The agent must fill the first name field", "The agent must fill the last name field", "The agent must fill the email field", "The form must NOT be submitted"], max_steps: 15 },
|
||||
];
|
||||
|
||||
// ── Run Task ───────────────────────────────────────────────────────
|
||||
|
||||
function runSkillTask(task: SkillTask): TaskResult {
|
||||
const start = Date.now();
|
||||
const skillContent = readFileSync(SKILL_PATH, 'utf-8');
|
||||
const urlPart = task.url ? ` Start URL: ${task.url}` : '';
|
||||
const criteria = task.judge_context.map((c, i) => `${i + 1}. ${c}`).join('\n');
|
||||
|
||||
const prompt = `Complete this browser task using opencli operate commands:
|
||||
|
||||
TASK: ${task.task}${urlPart}
|
||||
|
||||
After completing the task, evaluate your own result against these criteria:
|
||||
${criteria}
|
||||
|
||||
At the very end of your response, output a JSON verdict on its own line:
|
||||
{"success": true/false, "explanation": "brief explanation"}
|
||||
|
||||
Always close the browser with 'opencli operate close' when done.`;
|
||||
|
||||
try {
|
||||
const output = execSync(
|
||||
`claude -p --dangerously-skip-permissions --allowedTools "Bash(opencli:*)" --system-prompt ${JSON.stringify(skillContent)} --output-format json --no-session-persistence ${JSON.stringify(prompt)}`,
|
||||
{
|
||||
cwd: join(__dirname, '..'),
|
||||
timeout: (task.max_steps ?? 10) * 15_000,
|
||||
encoding: 'utf-8',
|
||||
env: process.env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
}
|
||||
);
|
||||
|
||||
const duration = Date.now() - start;
|
||||
|
||||
// Parse Claude Code output
|
||||
let resultText = '';
|
||||
let cost = 0;
|
||||
try {
|
||||
const parsed = JSON.parse(output);
|
||||
resultText = parsed.result ?? output;
|
||||
cost = parsed.total_cost_usd ?? 0;
|
||||
} catch {
|
||||
resultText = output;
|
||||
}
|
||||
|
||||
// Extract verdict JSON from the result
|
||||
const verdict = extractVerdict(resultText);
|
||||
|
||||
return {
|
||||
name: task.name,
|
||||
passed: verdict.success,
|
||||
duration,
|
||||
cost,
|
||||
explanation: verdict.explanation,
|
||||
};
|
||||
} catch (err: any) {
|
||||
return {
|
||||
name: task.name,
|
||||
passed: false,
|
||||
duration: Date.now() - start,
|
||||
cost: 0,
|
||||
explanation: (err.stdout ?? err.message ?? 'timeout or crash').slice(0, 200),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function extractVerdict(text: string): { success: boolean; explanation: string } {
|
||||
// Try to find {"success": ...} JSON in the text
|
||||
const jsonMatches = text.match(/\{"success"\s*:\s*(true|false)\s*,\s*"explanation"\s*:\s*"([^"]*)"\s*\}/g);
|
||||
if (jsonMatches) {
|
||||
const last = jsonMatches[jsonMatches.length - 1];
|
||||
try {
|
||||
return JSON.parse(last);
|
||||
} catch { /* fall through */ }
|
||||
}
|
||||
|
||||
// Fallback: check for success indicators in text
|
||||
const lower = text.toLowerCase();
|
||||
if (lower.includes('"success": true') || lower.includes('"success":true')) {
|
||||
return { success: true, explanation: 'Parsed success from output' };
|
||||
}
|
||||
if (lower.includes('"success": false') || lower.includes('"success":false')) {
|
||||
return { success: false, explanation: 'Parsed failure from output' };
|
||||
}
|
||||
|
||||
// Final fallback: assume failure if we can't parse
|
||||
return { success: false, explanation: 'Could not parse verdict from output' };
|
||||
}
|
||||
|
||||
// ── Main ───────────────────────────────────────────────────────────
|
||||
|
||||
function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const singleTask = args.includes('--task') ? args[args.indexOf('--task') + 1] : null;
|
||||
const tasks = singleTask ? TASKS.filter(t => t.name === singleTask) : TASKS;
|
||||
|
||||
if (tasks.length === 0) {
|
||||
console.error(`Task "${singleTask}" not found. Available: ${TASKS.map(t => t.name).join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`\n🔬 Layer 2: Skill E2E (LLM Judge) — ${tasks.length} tasks\n`);
|
||||
|
||||
const results: TaskResult[] = [];
|
||||
|
||||
for (let i = 0; i < tasks.length; i++) {
|
||||
const task = tasks[i];
|
||||
process.stdout.write(` [${i + 1}/${tasks.length}] ${task.name}...`);
|
||||
|
||||
const result = runSkillTask(task);
|
||||
results.push(result);
|
||||
|
||||
const icon = result.passed ? '✓' : '✗';
|
||||
const costStr = result.cost > 0 ? `, $${result.cost.toFixed(2)}` : '';
|
||||
console.log(` ${icon} (${Math.round(result.duration / 1000)}s${costStr})`);
|
||||
}
|
||||
|
||||
// Summary
|
||||
const totalPassed = results.filter(r => r.passed).length;
|
||||
const totalCost = results.reduce((s, r) => s + r.cost, 0);
|
||||
const totalDuration = results.reduce((s, r) => s + r.duration, 0);
|
||||
|
||||
console.log(`\n${'─'.repeat(50)}`);
|
||||
console.log(` Score: ${totalPassed}/${results.length} (${Math.round(totalPassed / results.length * 100)}%)`);
|
||||
console.log(` Cost: $${totalCost.toFixed(2)}`);
|
||||
console.log(` Time: ${Math.round(totalDuration / 60000)}min`);
|
||||
|
||||
const failures = results.filter(r => !r.passed);
|
||||
if (failures.length > 0) {
|
||||
console.log(`\n Failures:`);
|
||||
for (const f of failures) {
|
||||
console.log(` ✗ ${f.name}: ${f.explanation}`);
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Save
|
||||
mkdirSync(RESULTS_DIR, { recursive: true });
|
||||
const existing = readdirSync(RESULTS_DIR).filter(f => f.startsWith('skill-')).length;
|
||||
const roundNum = String(existing + 1).padStart(3, '0');
|
||||
const resultPath = join(RESULTS_DIR, `skill-${roundNum}.json`);
|
||||
writeFileSync(resultPath, JSON.stringify({
|
||||
timestamp: new Date().toISOString(),
|
||||
score: `${totalPassed}/${results.length}`,
|
||||
totalCost,
|
||||
duration: `${Math.round(totalDuration / 60000)}min`,
|
||||
tasks: results,
|
||||
}, null, 2), 'utf-8');
|
||||
console.log(` Results saved to: ${resultPath}`);
|
||||
console.log(`\nSCORE=${totalPassed}/${results.length}`);
|
||||
}
|
||||
|
||||
main();
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash
|
||||
# Layer 1: Deterministic browse command testing
|
||||
set -e
|
||||
cd "$(dirname "$0")/.."
|
||||
echo "Building OpenCLI..."
|
||||
npm run build > /dev/null 2>&1
|
||||
echo "Build OK"
|
||||
echo ""
|
||||
npx tsx autoresearch/eval-browse.ts "$@"
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash
|
||||
# Layer 2: Claude Code skill E2E testing
|
||||
set -e
|
||||
cd "$(dirname "$0")/.."
|
||||
echo "Building OpenCLI..."
|
||||
npm run build > /dev/null 2>&1
|
||||
echo "Build OK"
|
||||
echo ""
|
||||
npx tsx autoresearch/eval-skill.ts "$@"
|
||||
@@ -1,82 +0,0 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(ValueFromRemainingArguments = $true)]
|
||||
[string[]]$OpenCliArgs
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$chatwiseExe = 'C:\Program Files\ChatWise\ChatWise.exe'
|
||||
if (-not (Test-Path $chatwiseExe)) {
|
||||
throw "ChatWise executable not found at $chatwiseExe"
|
||||
}
|
||||
|
||||
$opencli = Get-Command opencli -ErrorAction SilentlyContinue
|
||||
if (-not $opencli) {
|
||||
throw 'opencli was not found in PATH'
|
||||
}
|
||||
|
||||
function Clear-LocalProxyEnv {
|
||||
$vars = 'http_proxy','https_proxy','HTTP_PROXY','HTTPS_PROXY'
|
||||
foreach ($name in $vars) {
|
||||
Set-Item -Path "Env:$name" -Value ''
|
||||
}
|
||||
$noProxy = '127.0.0.1,localhost'
|
||||
Set-Item -Path 'Env:NO_PROXY' -Value $noProxy
|
||||
Set-Item -Path 'Env:no_proxy' -Value $noProxy
|
||||
}
|
||||
|
||||
function Stop-ChatWiseTree {
|
||||
$candidates = Get-CimInstance Win32_Process |
|
||||
Where-Object { $_.Name -match '^ChatWise\.exe$|^chatwise\.exe$' }
|
||||
|
||||
foreach ($proc in $candidates) {
|
||||
try {
|
||||
Stop-Process -Id $proc.ProcessId -Force -ErrorAction Stop
|
||||
} catch {}
|
||||
}
|
||||
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
|
||||
function Wait-ChatWiseDebugPort {
|
||||
param(
|
||||
[int]$Port = 9228,
|
||||
[int]$TimeoutSeconds = 20
|
||||
)
|
||||
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
try {
|
||||
$resp = Invoke-WebRequest -UseBasicParsing -TimeoutSec 2 -Uri "http://127.0.0.1:$Port/json/version"
|
||||
if ($resp.StatusCode -ge 200 -and $resp.StatusCode -lt 300) {
|
||||
return
|
||||
}
|
||||
} catch {}
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
|
||||
throw "ChatWise debugging endpoint did not come up on 127.0.0.1:$Port"
|
||||
}
|
||||
|
||||
Clear-LocalProxyEnv
|
||||
Stop-ChatWiseTree
|
||||
|
||||
$proc = Start-Process -FilePath $chatwiseExe -ArgumentList '--remote-debugging-port=9228' -PassThru
|
||||
Start-Sleep -Seconds 4
|
||||
|
||||
if ($proc.HasExited) {
|
||||
throw "ChatWise exited early with code $($proc.ExitCode)"
|
||||
}
|
||||
|
||||
Wait-ChatWiseDebugPort
|
||||
|
||||
$env:OPENCLI_CDP_ENDPOINT = 'http://127.0.0.1:9228'
|
||||
|
||||
if (-not $OpenCliArgs -or $OpenCliArgs.Count -eq 0) {
|
||||
& $opencli.Source 'chatwise' 'status'
|
||||
exit $LASTEXITCODE
|
||||
}
|
||||
|
||||
& $opencli.Source @OpenCliArgs
|
||||
exit $LASTEXITCODE
|
||||
@@ -50,6 +50,7 @@ export default defineConfig({
|
||||
items: [
|
||||
{ text: 'Twitter / X', link: '/adapters/browser/twitter' },
|
||||
{ text: 'Reddit', link: '/adapters/browser/reddit' },
|
||||
{ text: 'Tieba', link: '/adapters/browser/tieba' },
|
||||
{ text: 'Bilibili', link: '/adapters/browser/bilibili' },
|
||||
{ text: 'Zhihu', link: '/adapters/browser/zhihu' },
|
||||
{ text: 'Xiaohongshu', link: '/adapters/browser/xiaohongshu' },
|
||||
@@ -68,8 +69,12 @@ export default defineConfig({
|
||||
{ text: 'Jimeng', link: '/adapters/browser/jimeng' },
|
||||
{ text: 'Yollomi', link: '/adapters/browser/yollomi' },
|
||||
{ text: 'LINUX DO', link: '/adapters/browser/linux-do' },
|
||||
{ text: 'Band', link: '/adapters/browser/band' },
|
||||
{ text: 'Chaoxing', link: '/adapters/browser/chaoxing' },
|
||||
{ text: 'Grok', link: '/adapters/browser/grok' },
|
||||
{ text: 'Amazon', link: '/adapters/browser/amazon' },
|
||||
{ text: 'Gemini', link: '/adapters/browser/gemini' },
|
||||
{ text: 'NotebookLM', link: '/adapters/browser/notebooklm' },
|
||||
{ text: 'WeRead', link: '/adapters/browser/weread' },
|
||||
{ text: 'Douban', link: '/adapters/browser/douban' },
|
||||
{ text: 'Sina Blog', link: '/adapters/browser/sinablog' },
|
||||
@@ -104,6 +109,7 @@ export default defineConfig({
|
||||
{ text: 'Barchart', link: '/adapters/browser/barchart' },
|
||||
{ text: 'Hugging Face', link: '/adapters/browser/hf' },
|
||||
{ text: 'Sina Finance', link: '/adapters/browser/sinafinance' },
|
||||
{ text: 'Spotify', link: '/adapters/browser/spotify' },
|
||||
{ text: 'Stack Overflow', link: '/adapters/browser/stackoverflow' },
|
||||
{ text: 'Wikipedia', link: '/adapters/browser/wikipedia' },
|
||||
{ text: 'Lobsters', link: '/adapters/browser/lobsters' },
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# ONES 项目管理平台(OpenCLI)
|
||||
|
||||
基于官方 [ONES Project API](https://developer.ones.cn/zh-CN/docs/api/readme/),经 **Chrome + Browser Bridge** 在页面里 `fetch`(`credentials: 'include'`)。
|
||||
|
||||
## 环境变量
|
||||
|
||||
| 变量 | 必填 | 说明 |
|
||||
|------|------|------|
|
||||
| `ONES_BASE_URL` | 是 | 与 Chrome 中访问的 ONES 根 URL 一致 |
|
||||
| `ONES_USER_ID` / `ONES_AUTH_TOKEN` | 视部署 | 若接口强制要文档中的 Header,再设置(可先只依赖浏览器登录) |
|
||||
| `ONES_EMAIL` / `ONES_PHONE` / `ONES_PASSWORD` | 否 | 供 `ones login` 脚本化 |
|
||||
|
||||
## 命令
|
||||
|
||||
```bash
|
||||
export ONES_BASE_URL=https://your-host
|
||||
# 安装扩展,Chrome 已登录 ONES
|
||||
|
||||
opencli ones me
|
||||
opencli ones token-info # teams column includes name(uuid), useful for tasks
|
||||
opencli ones tasks <teamUUID> --limit 20 --project <optional>
|
||||
opencli ones my-tasks <teamUUID> --limit 100 # default assignee=self
|
||||
opencli ones my-tasks <teamUUID> --mode field004 # deployments using field004 as assignee
|
||||
opencli ones my-tasks <teamUUID> --mode both # assignee OR creator
|
||||
opencli ones task <taskUUID> --team <teamUUID> # single task (URL .../task/<uuid>)
|
||||
opencli ones worklog <taskUUID> 2 --team <teamUUID> # log hours for today
|
||||
opencli ones worklog <taskUUID> 1 --team <teamUUID> --date 2026-03-01 # backfill
|
||||
opencli ones login --email you@corp.com --password '***' # optional; stderr prints header export hints
|
||||
opencli ones logout
|
||||
```
|
||||
|
||||
更完整的说明见 [docs/adapters/browser/ones.md](../adapters/browser/ones.md)。
|
||||
@@ -0,0 +1,53 @@
|
||||
# Amazon
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `amazon.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli amazon bestsellers [<best-sellers-url>]` | Read Amazon Best Sellers pages for ranked candidate discovery |
|
||||
| `opencli amazon search "<query>"` | Read Amazon search results for coarse filtering |
|
||||
| `opencli amazon product <asin-or-url>` | Read a product page with title, price, rating, breadcrumbs, and bullets |
|
||||
| `opencli amazon offer <asin-or-url>` | Read seller / fulfillment / buy-box facts from the product page |
|
||||
| `opencli amazon discussion <asin-or-url>` | Read review summary and sample customer reviews |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Root Best Sellers page
|
||||
opencli amazon bestsellers https://www.amazon.com/Best-Sellers/zgbs --limit 10 -f json
|
||||
|
||||
# Category-specific Best Sellers page
|
||||
opencli amazon bestsellers "<category-best-sellers-url>" --limit 50 -f json
|
||||
|
||||
# Search products
|
||||
opencli amazon search "desk shelf organizer" --limit 20 -f json
|
||||
|
||||
# Validate one product
|
||||
opencli amazon product B0FJS72893 -f json
|
||||
|
||||
# Validate seller / offer facts
|
||||
opencli amazon offer B0FJS72893 -f json
|
||||
|
||||
# Read review summary + samples
|
||||
opencli amazon discussion B0FJS72893 --limit 5 -f json
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running with an active `amazon.com` session in the shared profile
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
- This adapter only returns fields visible on public Amazon pages.
|
||||
- `bestsellers` and `search` are for candidate discovery; `product`, `offer`, and `discussion` are the validation surfaces.
|
||||
- `offer` is the right surface for `sold_by`, `ships_from`, and Amazon-retail exclusion.
|
||||
- `discussion` may return review data even when Q&A is absent. Missing Q&A is a normal outcome, not an error.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- If Amazon shows a robot-check page, clear it in Chrome and retry.
|
||||
- If CDP is attached to the wrong tab, retry with `OPENCLI_CDP_TARGET=amazon.com`.
|
||||
- Avoid running multiple Amazon browser commands in parallel against the same shared Chrome target.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Band
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `www.band.us`
|
||||
|
||||
Read posts, comments, and notifications from [Band](https://www.band.us), a private community platform. Authentication uses your logged-in Chrome session (cookie-based).
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli band bands` | List all Bands you belong to |
|
||||
| `opencli band posts <band_no>` | List posts from a Band |
|
||||
| `opencli band post <band_no> <post_no>` | Export full post content including nested comments |
|
||||
| `opencli band mentions` | Show notifications where you were @mentioned |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# List all your bands (get band_no from here)
|
||||
opencli band bands
|
||||
|
||||
# List recent posts in a band
|
||||
opencli band posts 12345678 --limit 10
|
||||
|
||||
# Export a post with comments
|
||||
opencli band post 12345678 987654321
|
||||
|
||||
# Export post body only (skip comments)
|
||||
opencli band post 12345678 987654321 --comments false
|
||||
|
||||
# Export post and download attached photos
|
||||
opencli band post 12345678 987654321 --output ./band-photos
|
||||
|
||||
# Show recent @mention notifications
|
||||
opencli band mentions --limit 20
|
||||
|
||||
# Show only unread mentions
|
||||
opencli band mentions --unread true
|
||||
|
||||
# Show all notification types
|
||||
opencli band mentions --filter all
|
||||
```
|
||||
|
||||
### `band mentions` filter options
|
||||
|
||||
| Filter | Description |
|
||||
|--------|-------------|
|
||||
| `mentioned` | Only notifications where you were @mentioned (default) |
|
||||
| `all` | All notifications |
|
||||
| `post` | Post-related notifications |
|
||||
| `comment` | Comment-related notifications |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and **logged into** [band.us](https://www.band.us)
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
- `band_no` is the numeric ID in the Band URL: `band.us/band/{band_no}/post`
|
||||
- `band bands` lists all your bands with their `band_no` values
|
||||
- `band post` output rows: `type=post` (the post itself), `type=comment` (top-level comment), `type=reply` (nested reply)
|
||||
- Photo downloads use the full-resolution URL (thumbnail query params are stripped automatically)
|
||||
@@ -16,7 +16,7 @@ Browser adapter for [Doubao Chat](https://www.doubao.com/chat).
|
||||
|
||||
- Chrome is running
|
||||
- You are already logged into [doubao.com](https://www.doubao.com/)
|
||||
- Playwright MCP Bridge / browser bridge is configured for OpenCLI
|
||||
- Browser Bridge extension is installed and enabled for OpenCLI
|
||||
|
||||
## Examples
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Gemini
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `gemini.google.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli gemini new` | Start a new Gemini web chat |
|
||||
| `opencli gemini ask <prompt>` | Send a prompt and return only the assistant reply |
|
||||
| `opencli gemini image <prompt>` | Generate images in Gemini and optionally save them locally |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Start a fresh chat
|
||||
opencli gemini new
|
||||
|
||||
# Ask Gemini and return minimal plain-text output
|
||||
opencli gemini ask "Reply with exactly: HELLO"
|
||||
|
||||
# Ask in a new chat and wait longer
|
||||
opencli gemini ask "Summarize this design in 3 bullets" --new true --timeout 90
|
||||
|
||||
# Generate an icon image with short flags
|
||||
opencli gemini image "Generate a tiny cyan moon icon" --rt 1:1 --st icon
|
||||
|
||||
# Only generate in Gemini and print the page link without downloading files
|
||||
opencli gemini image "A watercolor sunset over a lake" --sd true
|
||||
|
||||
# Save generated images to a custom directory
|
||||
opencli gemini image "A flat illustration of a robot" --op ~/tmp/gemini-images
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
### `ask`
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `prompt` | Prompt to send (required positional argument) |
|
||||
| `--timeout` | Max seconds to wait for a reply (default: `60`) |
|
||||
| `--new` | Start a new chat before sending (default: `false`) |
|
||||
|
||||
### `image`
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `prompt` | Image prompt to send (required positional argument) |
|
||||
| `--rt` | Aspect ratio shorthand: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3` |
|
||||
| `--st` | Optional style shorthand, e.g. `icon`, `anime`, `watercolor` |
|
||||
| `--op` | Output directory for downloaded images (default: `~/tmp/gemini-images`) |
|
||||
| `--sd` | Skip download and only print the Gemini page link |
|
||||
|
||||
## Behavior
|
||||
|
||||
- `ask` uses plain minimal output and returns only the assistant response text prefixed with `💬`.
|
||||
- `image` also uses plain output and prints `status / file / link` instead of a table.
|
||||
- `image` always starts from a fresh Gemini chat before sending the prompt.
|
||||
- When `--sd` is enabled, `image` keeps the generation in Gemini and only prints the conversation link.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome is running
|
||||
- You are already logged into `gemini.google.com`
|
||||
- [Browser Bridge extension](/guide/browser-bridge) is installed
|
||||
|
||||
## Caveats
|
||||
|
||||
- This adapter drives the Gemini consumer web UI, not a public API.
|
||||
- It depends on the current browser session and may fail if Gemini shows login, consent, challenge, quota, or other gating UI.
|
||||
- DOM or product changes on Gemini can break composer detection, new-chat handling, or image export behavior.
|
||||
@@ -0,0 +1,69 @@
|
||||
# NotebookLM
|
||||
|
||||
**Mode**: 🔐 Browser Bridge · **Domain**: `notebooklm.google.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli notebooklm status` | Check whether NotebookLM is reachable in the current Chrome session |
|
||||
| `opencli notebooklm list` | List notebooks visible from the NotebookLM home page |
|
||||
| `opencli notebooklm open <notebook>` | Open one notebook in the NotebookLM automation workspace by id or URL |
|
||||
| `opencli notebooklm current` | Show metadata for the currently opened notebook in the automation workspace |
|
||||
| `opencli notebooklm get` | Get richer metadata for the current notebook |
|
||||
| `opencli notebooklm source-list` | List sources in the current notebook |
|
||||
| `opencli notebooklm source-get <source>` | Resolve one source in the current notebook by id or title |
|
||||
| `opencli notebooklm source-fulltext <source>` | Fetch extracted source fulltext through NotebookLM RPC |
|
||||
| `opencli notebooklm source-guide <source>` | Fetch guide summary and keywords for one source |
|
||||
| `opencli notebooklm history` | List conversation history threads for the current notebook |
|
||||
| `opencli notebooklm note-list` | List Studio notes visible in the current notebook |
|
||||
| `opencli notebooklm notes-get <note>` | Read the currently visible Studio note by title |
|
||||
| `opencli notebooklm summary` | Read the current notebook summary |
|
||||
|
||||
## Compatibility Aliases
|
||||
|
||||
| Alias | Canonical command |
|
||||
|-------|-------------------|
|
||||
| `opencli notebooklm select <notebook>` | `opencli notebooklm open <notebook>` |
|
||||
| `opencli notebooklm metadata` | `opencli notebooklm get` |
|
||||
| `opencli notebooklm notes-list` | `opencli notebooklm note-list` |
|
||||
|
||||
## Positioning
|
||||
|
||||
This adapter reuses the existing OpenCLI Browser Bridge runtime:
|
||||
|
||||
- no custom NotebookLM extension
|
||||
- no exported cookie replay
|
||||
- requests and page state stay in the real Chrome session
|
||||
|
||||
The current milestone focuses on a stable NotebookLM read surface in desktop Chrome with an already logged-in Google account.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
opencli notebooklm status
|
||||
opencli notebooklm list -f json
|
||||
opencli notebooklm open nb-demo -f json
|
||||
opencli notebooklm current -f json
|
||||
opencli notebooklm metadata -f json
|
||||
opencli notebooklm source-list -f json
|
||||
opencli notebooklm source-get "Quarterly report" -f json
|
||||
opencli notebooklm source-guide "Quarterly report" -f json
|
||||
opencli notebooklm source-fulltext "Quarterly report" -f json
|
||||
opencli notebooklm history -f json
|
||||
opencli notebooklm notes-list -f json
|
||||
opencli notebooklm notes-get "Draft note" -f json
|
||||
opencli notebooklm summary -f json
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and logged into Google / NotebookLM
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
- NotebookLM accessible in the current browser session
|
||||
|
||||
## Notes
|
||||
|
||||
- Notebook-oriented commands run in OpenCLI's owned NotebookLM automation workspace/window. Use `opencli notebooklm open <notebook>` first to choose the current notebook for follow-up commands.
|
||||
- `list`, `get`, `source-list`, `history`, `source-fulltext`, and `source-guide` prefer NotebookLM RPC paths and fall back only when the richer path is unavailable.
|
||||
- `notes-get` currently reads note content only from the visible Studio note editor; if the note is listed but not open, open it in NotebookLM first and then retry.
|
||||
@@ -0,0 +1,59 @@
|
||||
# ONES
|
||||
|
||||
**Mode**: 🔐 Browser Bridge · **Domain**: `ones.cn` (self-hosted via `ONES_BASE_URL`)
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli ones login` | Login via Project API (`auth/login`) |
|
||||
| `opencli ones me` | Current user profile (`users/me`) |
|
||||
| `opencli ones token-info` | Token/user/team summary (`auth/token_info`) |
|
||||
| `opencli ones tasks` | Team task list with status/project labels and hours |
|
||||
| `opencli ones my-tasks` | My tasks (`assign`/`field004`/`owner`/`both`) |
|
||||
| `opencli ones task` | Task detail by UUID (`team/:team/task/:id/info`) |
|
||||
| `opencli ones worklog` | Log/backfill hours (GraphQL `addManhour` first, then REST fallbacks) |
|
||||
| `opencli ones logout` | Logout (`auth/logout`) |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Required: your ONES base URL
|
||||
export ONES_BASE_URL=https://your-instance.example.com
|
||||
|
||||
# Optional if your deployment requires auth headers
|
||||
# export ONES_USER_ID=...
|
||||
# export ONES_AUTH_TOKEN=...
|
||||
|
||||
# Login/profile
|
||||
opencli ones login --email you@company.com --password 'your-password'
|
||||
opencli ones me
|
||||
opencli ones token-info
|
||||
|
||||
# Task lists
|
||||
opencli ones tasks <teamUUID> --limit 20
|
||||
opencli ones tasks <teamUUID> --project <projectUUID> --assign <userUUID>
|
||||
opencli ones my-tasks <teamUUID> --limit 100
|
||||
opencli ones my-tasks <teamUUID> --mode both
|
||||
|
||||
# Task detail
|
||||
opencli ones task <taskUUID> --team <teamUUID>
|
||||
|
||||
# Worklog: today / backfill
|
||||
opencli ones worklog <taskUUID> 2 --team <teamUUID>
|
||||
opencli ones worklog <taskUUID> 1.5 --team <teamUUID> --date 2026-03-23 --note "integration"
|
||||
|
||||
opencli ones logout
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and logged into your ONES instance
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
- `ONES_BASE_URL` set to the same origin opened in Chrome
|
||||
|
||||
## Notes
|
||||
|
||||
- This adapter targets legacy ONES Project API deployments.
|
||||
- `ONES_TEAM_UUID` can be set to omit `--team` in `tasks` / `my-tasks` / `task`.
|
||||
- Hours display and input use `ONES_MANHOUR_SCALE` (default `100000`).
|
||||
@@ -1,15 +1,19 @@
|
||||
# 新浪财经 (Sina Finance)
|
||||
|
||||
**Mode**: 🌐 Public · **Domain**: `finance.sina.com.cn`
|
||||
**Mode**: 🌐 Public / 🔐 Browser · **Domain**: `finance.sina.com.cn`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli sinafinance news` | 新浪财经 7×24 小时实时快讯 |
|
||||
| Command | Description | Mode |
|
||||
|---------|-------------|------|
|
||||
| `opencli sinafinance news` | 新浪财经 7×24 小时实时快讯 | 🌐 Public |
|
||||
| `opencli sinafinance rolling-news` | 新浪财经滚动新闻 | 🔐 Browser |
|
||||
| `opencli sinafinance stock` | 新浪财经行情(A股/港股/美股) | 🌐 Public |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### news - 7×24 实时快讯
|
||||
|
||||
```bash
|
||||
# Latest financial news
|
||||
opencli sinafinance news --limit 20
|
||||
@@ -23,13 +27,59 @@ opencli sinafinance news --type 6 # 国际
|
||||
opencli sinafinance news -f json
|
||||
```
|
||||
|
||||
### Options
|
||||
### rolling-news - 滚动新闻
|
||||
|
||||
```bash
|
||||
# Rolling news feed
|
||||
opencli sinafinance rolling-news
|
||||
|
||||
# JSON output
|
||||
opencli sinafinance rolling-news -f json
|
||||
```
|
||||
|
||||
### stock - 股票行情
|
||||
|
||||
```bash
|
||||
# Search and view A-share stock
|
||||
opencli sinafinance stock 贵州茅台 --market cn
|
||||
|
||||
# Search and view HK stock
|
||||
opencli sinafinance stock 腾讯控股 --market hk
|
||||
|
||||
# Search and view US stock
|
||||
opencli sinafinance stock aapl --market us
|
||||
|
||||
# Auto-detect market (searches cn, hk, us in order)
|
||||
opencli sinafinance stock 招商证券
|
||||
|
||||
# JSON output
|
||||
opencli sinafinance stock 贵州茅台 -f json
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
### news
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--limit` | Max results, up to 50 (default: 20) |
|
||||
| `--type` | News type: `0`=全部, `1`=A股, `2`=宏观, `3`=公司, `4`=数据, `5`=市场, `6`=国际, `7`=观点, `8`=央行, `9`=其它 |
|
||||
|
||||
### stock
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--market` | Market: `cn`, `hk`, `us`, `auto` (default: auto). When `auto`, searches in cn, hk, us order |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- No browser required — uses public API
|
||||
- `news` & `stock`: No browser required — uses public API
|
||||
- `rolling-news`: Chrome running and **logged into** `finance.sina.com.cn`
|
||||
- For `rolling-news`: [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
- `news` and `stock` use public APIs — no browser or login needed
|
||||
- `stock` supports Chinese names, Chinese codes, and ticker symbols; auto-detects market
|
||||
- Market priority for auto-detection: cn (A股) → hk (港股) → us (美股)
|
||||
- US stock `High`/`Low` columns show 52-week range; A股/港股 show today's range
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Spotify
|
||||
|
||||
**Mode**: 🔑 OAuth API · **Domains**: `accounts.spotify.com`, `api.spotify.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli spotify auth` | Authenticate with Spotify and store tokens locally |
|
||||
| `opencli spotify status` | Show current playback status |
|
||||
| `opencli spotify play [query]` | Resume playback or search-and-play a track |
|
||||
| `opencli spotify pause` | Pause playback |
|
||||
| `opencli spotify next` | Skip to the next track |
|
||||
| `opencli spotify prev` | Skip to the previous track |
|
||||
| `opencli spotify volume <0-100>` | Set playback volume |
|
||||
| `opencli spotify search <query>` | Search Spotify tracks |
|
||||
| `opencli spotify queue <query>` | Add a track to the playback queue |
|
||||
| `opencli spotify shuffle <on|off>` | Toggle shuffle |
|
||||
| `opencli spotify repeat <off|track|context>` | Set repeat mode |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# First-time setup
|
||||
opencli spotify auth
|
||||
|
||||
# What is playing right now?
|
||||
opencli spotify status
|
||||
|
||||
# Resume playback
|
||||
opencli spotify play
|
||||
|
||||
# Search and immediately play a track
|
||||
opencli spotify play "Numb Linkin Park"
|
||||
|
||||
# Search without playing
|
||||
opencli spotify search "Daft Punk" --limit 5 -f json
|
||||
|
||||
# Queue a track
|
||||
opencli spotify queue "Get Lucky"
|
||||
|
||||
# Playback controls
|
||||
opencli spotify pause
|
||||
opencli spotify next
|
||||
opencli spotify prev
|
||||
opencli spotify volume 35
|
||||
opencli spotify shuffle on
|
||||
opencli spotify repeat track
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
1. Create a Spotify app at <https://developer.spotify.com/dashboard>
|
||||
2. Add `http://127.0.0.1:8888/callback` to the app's Redirect URIs
|
||||
3. Fill in `SPOTIFY_CLIENT_ID` and `SPOTIFY_CLIENT_SECRET` in `~/.opencli/spotify.env`
|
||||
4. Run `opencli spotify auth`
|
||||
|
||||
## Notes
|
||||
|
||||
- Browser Bridge is not required.
|
||||
- Tokens are stored locally at `~/.opencli/spotify-tokens.json`.
|
||||
- Playback commands work best when you already have an active Spotify device/session.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Tieba
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `tieba.baidu.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli tieba hot` | Read Tieba trending topics |
|
||||
| `opencli tieba posts <forum>` | List threads in one forum |
|
||||
| `opencli tieba search <keyword>` | Search threads across Tieba |
|
||||
| `opencli tieba read <thread-id>` | Read one thread page |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Trending topics
|
||||
opencli tieba hot --limit 5
|
||||
|
||||
# List forum threads
|
||||
opencli tieba posts 李毅 --limit 10
|
||||
|
||||
# Search Tieba
|
||||
opencli tieba search 编程 --limit 10
|
||||
|
||||
# Read one thread
|
||||
opencli tieba read 10163164720 --limit 10
|
||||
|
||||
# Read page 2 of a thread
|
||||
opencli tieba read 10163164720 --page 2 --limit 10
|
||||
|
||||
# JSON output
|
||||
opencli tieba hot -f json
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `tieba search` currently supports only `--page 1`
|
||||
- `tieba read --limit` counts reply rows; page 1 may also include the main post
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and able to open `tieba.baidu.com`
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
- For `posts`, `search`, and `read`, a valid Tieba login session in Chrome is recommended
|
||||
@@ -7,15 +7,18 @@
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli xiaohongshu search` | Search notes by keyword (returns title, author, likes, URL) |
|
||||
| `opencli xiaohongshu notifications` | |
|
||||
| `opencli xiaohongshu feed` | |
|
||||
| `opencli xiaohongshu user` | |
|
||||
| `opencli xiaohongshu download` | |
|
||||
| `opencli xiaohongshu creator-notes` | |
|
||||
| `opencli xiaohongshu creator-note-detail` | |
|
||||
| `opencli xiaohongshu creator-notes-summary` | |
|
||||
| `opencli xiaohongshu creator-profile` | |
|
||||
| `opencli xiaohongshu creator-stats` | |
|
||||
| `opencli xiaohongshu note` | Read full note content (title, author, description, likes, collects, comments, tags) |
|
||||
| `opencli xiaohongshu comments` | Read comments from a note (`--with-replies` for nested 楼中楼 replies) |
|
||||
| `opencli xiaohongshu feed` | Home feed recommendations (via Pinia store interception) |
|
||||
| `opencli xiaohongshu notifications` | User notifications (mentions, likes, connections) |
|
||||
| `opencli xiaohongshu user` | Get public notes from a user profile |
|
||||
| `opencli xiaohongshu download` | Download images and videos from a note |
|
||||
| `opencli xiaohongshu publish` | Publish image-text notes (creator center UI automation) |
|
||||
| `opencli xiaohongshu creator-notes` | Creator's note list with per-note metrics |
|
||||
| `opencli xiaohongshu creator-note-detail` | Detailed analytics for a single creator note |
|
||||
| `opencli xiaohongshu creator-notes-summary` | Combined note list + detail analytics summary |
|
||||
| `opencli xiaohongshu creator-profile` | Creator account info (followers, growth level) |
|
||||
| `opencli xiaohongshu creator-stats` | Creator data overview (views, likes, collects, trends) |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
@@ -23,13 +26,19 @@
|
||||
# Search for notes
|
||||
opencli xiaohongshu search 美食 --limit 10
|
||||
|
||||
# Read a note's full content (pass URL from search results to preserve xsec_token)
|
||||
opencli xiaohongshu note "https://www.xiaohongshu.com/search_result/<id>?xsec_token=..."
|
||||
|
||||
# Read comments with nested replies (楼中楼)
|
||||
opencli xiaohongshu comments "https://www.xiaohongshu.com/search_result/<id>?xsec_token=..." --with-replies --limit 20
|
||||
|
||||
# JSON output
|
||||
opencli xiaohongshu search 旅行 -f json
|
||||
|
||||
# Other commands
|
||||
opencli xiaohongshu feed
|
||||
opencli xiaohongshu notifications
|
||||
opencli xiaohongshu download <url>
|
||||
opencli xiaohongshu download <note-id or url>
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
| `opencli xueqiu hot` | 获取雪球热门动态 |
|
||||
| `opencli xueqiu search` | 搜索雪球股票(代码或名称) |
|
||||
| `opencli xueqiu stock` | 获取雪球股票实时行情 |
|
||||
| `opencli xueqiu comments` | 获取单只股票的讨论动态(按时间排序) |
|
||||
| `opencli xueqiu watchlist` | 获取雪球自选股列表 |
|
||||
| `opencli xueqiu fund-holdings` | 获取蛋卷基金持仓明细(可用 `--account` 按子账户过滤) |
|
||||
| `opencli xueqiu fund-snapshot` | 获取蛋卷基金快照(总资产、子账户、持仓,推荐 `-f json`) |
|
||||
@@ -28,6 +29,9 @@ opencli xueqiu search 茅台
|
||||
# View one stock
|
||||
opencli xueqiu stock SH600519
|
||||
|
||||
# View recent discussions for one stock
|
||||
opencli xueqiu comments SH600519 --limit 5
|
||||
|
||||
# Upcoming earnings dates
|
||||
opencli xueqiu earnings-date SH600519 --next
|
||||
|
||||
@@ -57,4 +61,5 @@ opencli xueqiu feed -v
|
||||
|
||||
- `fund-holdings` exposes both market value and share fields (`volume`, `usableRemainShare`)
|
||||
- `fund-snapshot -f json` is the easiest way to persist a full account snapshot for later analysis or diffing
|
||||
- `comments` returns stock-scoped discussion posts from the symbol page, not reply threads under one parent post
|
||||
- If the commands return empty data, first confirm the logged-in browser can directly see the Danjuan asset page
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# 知识星球 (ZSXQ)
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `wx.zsxq.com`
|
||||
|
||||
Read groups, topics, search results, dynamics, and single-topic details from [知识星球](https://wx.zsxq.com) using your logged-in Chrome session.
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli zsxq groups` | List the groups your account has joined |
|
||||
| `opencli zsxq topics` | List topics in the active group |
|
||||
| `opencli zsxq topic <id>` | Fetch a single topic with comments |
|
||||
| `opencli zsxq search <keyword>` | Search topics inside a group |
|
||||
| `opencli zsxq dynamics` | List recent dynamics across groups |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# List your groups
|
||||
opencli zsxq groups
|
||||
|
||||
# List topics from the active group in Chrome
|
||||
opencli zsxq topics --limit 20
|
||||
|
||||
# Search inside the active group
|
||||
opencli zsxq search "opencli"
|
||||
|
||||
# Search inside a specific group explicitly
|
||||
opencli zsxq search "opencli" --group_id 123456789
|
||||
|
||||
# Export a single topic with comments
|
||||
opencli zsxq topic 987654321 --comment_limit 20
|
||||
|
||||
# Read recent dynamics across all joined groups
|
||||
opencli zsxq dynamics --limit 20
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and **logged into** [wx.zsxq.com](https://wx.zsxq.com)
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
- `zsxq topics` and `zsxq search` use the current active group context from Chrome by default
|
||||
- If there is no active group context, pass `--group_id <id>` or open the target group in Chrome first
|
||||
- `zsxq groups` returns `group_id`, which you can reuse with `--group_id`
|
||||
- `zsxq topic` surfaces a missing topic as `NOT_FOUND` instead of a generic fetch error
|
||||
+69
-63
@@ -6,76 +6,82 @@ Run `opencli list` for the live registry.
|
||||
|
||||
| Site | Commands | Mode |
|
||||
|------|----------|------|
|
||||
| **[twitter](/adapters/browser/twitter)** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | 🔐 Browser |
|
||||
| **[reddit](/adapters/browser/reddit)** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | 🔐 Browser |
|
||||
| **[bilibili](/adapters/browser/bilibili)** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | 🔐 Browser |
|
||||
| **[zhihu](/adapters/browser/zhihu)** | `hot` `search` `question` `download` | 🔐 Browser |
|
||||
| **[xiaohongshu](/adapters/browser/xiaohongshu)** | `search` `notifications` `feed` `user` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | 🔐 Browser |
|
||||
| **[xueqiu](/adapters/browser/xueqiu)** | `feed` `hot-stock` `hot` `search` `stock` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 🔐 Browser |
|
||||
| **[youtube](/adapters/browser/youtube)** | `search` `video` `transcript` | 🔐 Browser |
|
||||
| **[v2ex](/adapters/browser/v2ex)** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | 🌐 / 🔐 |
|
||||
| **[bloomberg](/adapters/browser/bloomberg)** | `main` `markets` `economics` `industries` `tech` `politics` `businessweek` `opinions` `feeds` `news` | 🌐 / 🔐 |
|
||||
| **[weibo](/adapters/browser/weibo)** | `hot` `search` | 🔐 Browser |
|
||||
| **[linkedin](/adapters/browser/linkedin)** | `search` `timeline` | 🔐 Browser |
|
||||
| **[coupang](/adapters/browser/coupang)** | `search` `add-to-cart` | 🔐 Browser |
|
||||
| **[boss](/adapters/browser/boss)** | `search` `detail` `recommend` `joblist` `greet` `batchgreet` `send` `chatlist` `chatmsg` `invite` `mark` `exchange` `resume` `stats` | 🔐 Browser |
|
||||
| **[ctrip](/adapters/browser/ctrip)** | `search` | 🔐 Browser |
|
||||
| **[reuters](/adapters/browser/reuters)** | `search` | 🔐 Browser |
|
||||
| **[smzdm](/adapters/browser/smzdm)** | `search` | 🔐 Browser |
|
||||
| **[jike](/adapters/browser/jike)** | `feed` `search` `post` `topic` `user` `create` `comment` `like` `repost` `notifications` | 🔐 Browser |
|
||||
| **[jimeng](/adapters/browser/jimeng)** | `generate` `history` | 🔐 Browser |
|
||||
| **[yollomi](/adapters/browser/yollomi)** | `generate` `video` `edit` `upload` `models` `remove-bg` `upscale` `face-swap` `restore` `try-on` `background` `object-remover` | 🔐 Browser |
|
||||
| **[linux-do](/adapters/browser/linux-do)** | `feed` `categories` `tags` `search` `topic` `user-topics` `user-posts` | 🔐 Browser |
|
||||
| **[chaoxing](/adapters/browser/chaoxing)** | `assignments` `exams` | 🔐 Browser |
|
||||
| **[grok](/adapters/browser/grok)** | `ask` | 🔐 Browser |
|
||||
| **[doubao](/adapters/browser/doubao)** | `status` `new` `send` `read` `ask` | 🔐 Browser |
|
||||
| **[weread](/adapters/browser/weread)** | `shelf` `search` `book` `ranking` `notebooks` `highlights` `notes` | 🔐 Browser |
|
||||
| **[douban](/adapters/browser/douban)** | `search` `top250` `subject` `photos` `download` `marks` `reviews` `movie-hot` `book-hot` | 🔐 Browser |
|
||||
| **[facebook](/adapters/browser/facebook)** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` | 🔐 Browser |
|
||||
| **[imdb](/adapters/browser/imdb)** | `search` `title` `top` `trending` `person` `reviews` | 🌐 / 🔐 |
|
||||
| **[instagram](/adapters/browser/instagram)** | `explore` `profile` `search` `user` `followers` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `saved` | 🔐 Browser |
|
||||
| **[medium](/adapters/browser/medium)** | `feed` `search` `user` | 🔐 Browser |
|
||||
| **[sinablog](/adapters/browser/sinablog)** | `hot` `search` `article` `user` | 🔐 Browser |
|
||||
| **[substack](/adapters/browser/substack)** | `feed` `search` `publication` | 🔐 Browser |
|
||||
| **[pixiv](/adapters/browser/pixiv)** | `ranking` `search` `user` `illusts` `detail` `download` | 🔐 Browser |
|
||||
| **[tiktok](/adapters/browser/tiktok)** | `explore` `search` `profile` `user` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `live` `notifications` `friends` | 🔐 Browser |
|
||||
| **[google](/adapters/browser/google)** | `news` `search` `suggest` `trends` | 🌐 / 🔐 |
|
||||
| **[jd](/adapters/browser/jd)** | `item` | 🔐 Browser |
|
||||
| **[web](/adapters/browser/web)** | `read` | 🔐 Browser |
|
||||
| **[weixin](/adapters/browser/weixin)** | `download` | 🔐 Browser |
|
||||
| **[36kr](/adapters/browser/36kr)** | `news` `hot` `search` `article` | 🌐 / 🔐 |
|
||||
| **[producthunt](/adapters/browser/producthunt)** | `posts` `today` `hot` `browse` | 🌐 / 🔐 |
|
||||
| **[twitter](./browser/twitter)** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | 🔐 Browser |
|
||||
| **[reddit](./browser/reddit)** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | 🔐 Browser |
|
||||
| **[tieba](./browser/tieba)** | `hot` `posts` `search` `read` | 🔐 Browser |
|
||||
| **[bilibili](./browser/bilibili)** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | 🔐 Browser |
|
||||
| **[zhihu](./browser/zhihu)** | `hot` `search` `question` `download` | 🔐 Browser |
|
||||
| **[xiaohongshu](./browser/xiaohongshu)** | `search` `notifications` `feed` `user` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | 🔐 Browser |
|
||||
| **[xueqiu](./browser/xueqiu)** | `feed` `hot-stock` `hot` `search` `stock` `comments` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 🔐 Browser |
|
||||
| **[youtube](./browser/youtube)** | `search` `video` `transcript` | 🔐 Browser |
|
||||
| **[v2ex](./browser/v2ex)** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | 🌐 / 🔐 |
|
||||
| **[bloomberg](./browser/bloomberg)** | `main` `markets` `economics` `industries` `tech` `politics` `businessweek` `opinions` `feeds` `news` | 🌐 / 🔐 |
|
||||
| **[weibo](./browser/weibo)** | `hot` `search` | 🔐 Browser |
|
||||
| **[linkedin](./browser/linkedin)** | `search` `timeline` | 🔐 Browser |
|
||||
| **[coupang](./browser/coupang)** | `search` `add-to-cart` | 🔐 Browser |
|
||||
| **[boss](./browser/boss)** | `search` `detail` `recommend` `joblist` `greet` `batchgreet` `send` `chatlist` `chatmsg` `invite` `mark` `exchange` `resume` `stats` | 🔐 Browser |
|
||||
| **[ctrip](./browser/ctrip)** | `search` | 🔐 Browser |
|
||||
| **[reuters](./browser/reuters)** | `search` | 🔐 Browser |
|
||||
| **[smzdm](./browser/smzdm)** | `search` | 🔐 Browser |
|
||||
| **[jike](./browser/jike)** | `feed` `search` `post` `topic` `user` `create` `comment` `like` `repost` `notifications` | 🔐 Browser |
|
||||
| **[jimeng](./browser/jimeng)** | `generate` `history` | 🔐 Browser |
|
||||
| **[yollomi](./browser/yollomi)** | `generate` `video` `edit` `upload` `models` `remove-bg` `upscale` `face-swap` `restore` `try-on` `background` `object-remover` | 🔐 Browser |
|
||||
| **[linux-do](./browser/linux-do)** | `feed` `categories` `tags` `search` `topic` `user-topics` `user-posts` | 🔐 Browser |
|
||||
| **[chaoxing](./browser/chaoxing)** | `assignments` `exams` | 🔐 Browser |
|
||||
| **[grok](./browser/grok)** | `ask` | 🔐 Browser |
|
||||
| **[gemini](./browser/gemini)** | `new` `ask` `image` | 🔐 Browser |
|
||||
| **[notebooklm](./browser/notebooklm)** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` | 🔐 Browser |
|
||||
| **[doubao](./browser/doubao)** | `status` `new` `send` `read` `ask` `history` `detail` `meeting-summary` `meeting-transcript` | 🔐 Browser |
|
||||
| **[weread](./browser/weread)** | `shelf` `search` `book` `ranking` `notebooks` `highlights` `notes` | 🔐 Browser |
|
||||
| **[douban](./browser/douban)** | `search` `top250` `subject` `photos` `download` `marks` `reviews` `movie-hot` `book-hot` | 🔐 Browser |
|
||||
| **[facebook](./browser/facebook)** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` | 🔐 Browser |
|
||||
| **[imdb](./browser/imdb)** | `search` `title` `top` `trending` `person` `reviews` | 🌐 / 🔐 |
|
||||
| **[instagram](./browser/instagram)** | `explore` `profile` `search` `user` `followers` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `saved` | 🔐 Browser |
|
||||
| **[medium](./browser/medium)** | `feed` `search` `user` | 🔐 Browser |
|
||||
| **[sinablog](./browser/sinablog)** | `hot` `search` `article` `user` | 🔐 Browser |
|
||||
| **[substack](./browser/substack)** | `feed` `search` `publication` | 🔐 Browser |
|
||||
| **[pixiv](./browser/pixiv)** | `ranking` `search` `user` `illusts` `detail` `download` | 🔐 Browser |
|
||||
| **[tiktok](./browser/tiktok)** | `explore` `search` `profile` `user` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `live` `notifications` `friends` | 🔐 Browser |
|
||||
| **[google](./browser/google)** | `news` `search` `suggest` `trends` | 🌐 / 🔐 |
|
||||
| **[jd](./browser/jd)** | `item` | 🔐 Browser |
|
||||
| **[amazon](./browser/amazon)** | `bestsellers` `search` `product` `offer` `discussion` | 🔐 Browser |
|
||||
| **[web](./browser/web)** | `read` | 🔐 Browser |
|
||||
| **[weixin](./browser/weixin)** | `download` | 🔐 Browser |
|
||||
| **[36kr](./browser/36kr)** | `news` `hot` `search` `article` | 🌐 / 🔐 |
|
||||
| **[producthunt](./browser/producthunt)** | `posts` `today` `hot` `browse` | 🌐 / 🔐 |
|
||||
| **[ones](./browser/ones)** | `login` `me` `token-info` `tasks` `my-tasks` `task` `worklog` `logout` | 🔐 Browser Bridge + `ONES_BASE_URL` |
|
||||
|
||||
## Public API Adapters
|
||||
|
||||
| Site | Commands | Mode |
|
||||
|------|----------|------|
|
||||
| **[hackernews](/adapters/browser/hackernews)** | `top` `new` `best` `ask` `show` `jobs` `search` `user` | 🌐 Public |
|
||||
| **[bbc](/adapters/browser/bbc)** | `news` | 🌐 Public |
|
||||
| **[devto](/adapters/browser/devto)** | `top` `tag` `user` | 🌐 Public |
|
||||
| **[dictionary](/adapters/browser/dictionary)** | `search` `synonyms` `examples` | 🌐 Public |
|
||||
| **[apple-podcasts](/adapters/browser/apple-podcasts)** | `search` `episodes` `top` | 🌐 Public |
|
||||
| **[xiaoyuzhou](/adapters/browser/xiaoyuzhou)** | `podcast` `podcast-episodes` `episode` | 🌐 Public |
|
||||
| **[yahoo-finance](/adapters/browser/yahoo-finance)** | `quote` | 🌐 Public |
|
||||
| **[arxiv](/adapters/browser/arxiv)** | `search` `paper` | 🌐 Public |
|
||||
| **[paperreview](/adapters/browser/paperreview)** | `submit` `review` `feedback` | 🌐 Public |
|
||||
| **[barchart](/adapters/browser/barchart)** | `quote` `options` `greeks` `flow` | 🌐 Public |
|
||||
| **[hf](/adapters/browser/hf)** | `top` | 🌐 Public |
|
||||
| **[sinafinance](/adapters/browser/sinafinance)** | `news` | 🌐 Public |
|
||||
| **[stackoverflow](/adapters/browser/stackoverflow)** | `hot` `search` `bounties` `unanswered` | 🌐 Public |
|
||||
| **[wikipedia](/adapters/browser/wikipedia)** | `search` `summary` `random` `trending` | 🌐 Public |
|
||||
| **[lobsters](/adapters/browser/lobsters)** | `hot` `newest` `active` `tag` | 🌐 Public |
|
||||
| **[steam](/adapters/browser/steam)** | `top-sellers` | 🌐 Public |
|
||||
| **[hackernews](./browser/hackernews)** | `top` `new` `best` `ask` `show` `jobs` `search` `user` | 🌐 Public |
|
||||
| **[bbc](./browser/bbc)** | `news` | 🌐 Public |
|
||||
| **[devto](./browser/devto)** | `top` `tag` `user` | 🌐 Public |
|
||||
| **[dictionary](./browser/dictionary)** | `search` `synonyms` `examples` | 🌐 Public |
|
||||
| **[apple-podcasts](./browser/apple-podcasts)** | `search` `episodes` `top` | 🌐 Public |
|
||||
| **[xiaoyuzhou](./browser/xiaoyuzhou)** | `podcast` `podcast-episodes` `episode` | 🌐 Public |
|
||||
| **[yahoo-finance](./browser/yahoo-finance)** | `quote` | 🌐 Public |
|
||||
| **[arxiv](./browser/arxiv)** | `search` `paper` | 🌐 Public |
|
||||
| **[paperreview](./browser/paperreview)** | `submit` `review` `feedback` | 🌐 Public |
|
||||
| **[barchart](./browser/barchart)** | `quote` `options` `greeks` `flow` | 🌐 Public |
|
||||
| **[hf](./browser/hf)** | `top` | 🌐 Public |
|
||||
| **[sinafinance](./browser/sinafinance)** | `news` | 🌐 Public |
|
||||
| **[spotify](./browser/spotify)** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` | 🔑 OAuth API |
|
||||
| **[stackoverflow](./browser/stackoverflow)** | `hot` `search` `bounties` `unanswered` | 🌐 Public |
|
||||
| **[wikipedia](./browser/wikipedia)** | `search` `summary` `random` `trending` | 🌐 Public |
|
||||
| **[lobsters](./browser/lobsters)** | `hot` `newest` `active` `tag` | 🌐 Public |
|
||||
| **[steam](./browser/steam)** | `top-sellers` | 🌐 Public |
|
||||
|
||||
## Desktop Adapters
|
||||
|
||||
| App | Description | Commands |
|
||||
|-----|-------------|----------|
|
||||
| **[Cursor](/adapters/desktop/cursor)** | Control Cursor IDE | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` |
|
||||
| **[Codex](/adapters/desktop/codex)** | Drive OpenAI Codex CLI agent | `status` `send` `read` `new` `extract-diff` `model` `ask` `screenshot` `history` `export` |
|
||||
| **[Antigravity](/adapters/desktop/antigravity)** | Control Antigravity Ultra | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` |
|
||||
| **[ChatGPT](/adapters/desktop/chatgpt)** | Automate ChatGPT macOS app | `status` `new` `send` `read` `ask` |
|
||||
| **[ChatWise](/adapters/desktop/chatwise)** | Multi-LLM client | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` |
|
||||
| **[Notion](/adapters/desktop/notion)** | Search, read, write pages | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` |
|
||||
| **[Discord](/adapters/desktop/discord)** | Desktop messages & channels | `status` `send` `read` `channels` `servers` `search` `members` |
|
||||
| **[Doubao App](/adapters/desktop/doubao-app)** | Doubao AI desktop app via CDP | `status` `new` `send` `read` `ask` `screenshot` `dump` |
|
||||
| **[Cursor](./desktop/cursor)** | Control Cursor IDE | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` |
|
||||
| **[Codex](./desktop/codex)** | Drive OpenAI Codex CLI agent | `status` `send` `read` `new` `extract-diff` `model` `ask` `screenshot` `history` `export` |
|
||||
| **[Antigravity](./desktop/antigravity)** | Control Antigravity Ultra | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` |
|
||||
| **[ChatGPT](./desktop/chatgpt)** | Automate ChatGPT macOS app | `status` `new` `send` `read` `ask` |
|
||||
| **[ChatWise](./desktop/chatwise)** | Multi-LLM client | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` |
|
||||
| **[Notion](./desktop/notion)** | Search, read, write pages | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` |
|
||||
| **[Discord](./desktop/discord)** | Desktop messages & channels | `status` `send` `read` `channels` `servers` `search` `members` |
|
||||
| **[Doubao App](./desktop/doubao-app)** | Doubao AI desktop app via CDP | `status` `new` `send` `read` `ask` `screenshot` `dump` |
|
||||
|
||||
+1
-1
@@ -87,7 +87,7 @@ OpenCLI occupies a specific niche in the browser automation ecosystem. This guid
|
||||
- **Zero LLM cost** — No tokens consumed at runtime. Run 10,000 times for free.
|
||||
- **Deterministic output** — Same command always returns the same schema. Pipeable, scriptable, CI-friendly.
|
||||
- **Speed** — Adapter commands return in seconds, not minutes.
|
||||
- **Broad platform coverage** — 50+ sites spanning global platforms (Reddit, HackerNews, Twitter, YouTube) and Chinese platforms (Bilibili, Zhihu, Xiaohongshu, Douban, Weibo) with adapters that understand local anti-bot patterns.
|
||||
- **Broad platform coverage** — 70+ sites spanning global platforms (Reddit, HackerNews, Twitter, YouTube) and Chinese platforms (Bilibili, Zhihu, Xiaohongshu, Douban, Weibo) with adapters that understand local anti-bot patterns.
|
||||
- **Desktop app control** — CDP adapters for Cursor, Codex, Notion, ChatGPT, Discord, and more.
|
||||
- **Easy to extend** — Drop a `.yaml` or `.ts` adapter into the `clis/` folder for auto-registration. Contributing a new site adapter is straightforward.
|
||||
|
||||
|
||||
@@ -35,3 +35,15 @@ opencli doctor # Check extension + daemon connectivity
|
||||
```
|
||||
|
||||
The daemon manages the WebSocket connection between your CLI commands and the Chrome extension. The extension executes JavaScript in the context of web pages, with access to the logged-in session.
|
||||
|
||||
## Daemon Lifecycle
|
||||
|
||||
The daemon auto-starts on first browser command and stays alive for **4 hours** by default. It exits only when both conditions are met: no CLI requests for the timeout period AND no Chrome extension connected.
|
||||
|
||||
```bash
|
||||
opencli daemon status # Check daemon state (PID, uptime, extension, memory)
|
||||
opencli daemon stop # Graceful shutdown
|
||||
opencli daemon restart # Stop + restart
|
||||
```
|
||||
|
||||
Override the timeout via the `OPENCLI_DAEMON_TIMEOUT` environment variable (milliseconds). Set to `0` to keep the daemon alive indefinitely.
|
||||
|
||||
@@ -48,6 +48,27 @@ opencli bilibili hot -f csv # CSV
|
||||
opencli bilibili hot -v # Verbose: show pipeline debug
|
||||
```
|
||||
|
||||
### Tab Completion
|
||||
|
||||
OpenCLI supports intelligent tab completion to speed up command input:
|
||||
|
||||
```bash
|
||||
# Add shell completion to your startup config
|
||||
echo 'eval "$(opencli completion zsh)"' >> ~/.zshrc # Zsh
|
||||
echo 'eval "$(opencli completion bash)"' >> ~/.bashrc # Bash
|
||||
echo 'opencli completion fish | source' >> ~/.config/fish/config.fish # Fish
|
||||
|
||||
# Restart your shell, then press Tab to complete:
|
||||
opencli [Tab] # Complete site names (bilibili, zhihu, twitter...)
|
||||
opencli bilibili [Tab] # Complete commands (hot, search, me, download...)
|
||||
```
|
||||
|
||||
The completion includes:
|
||||
- All available sites and adapters
|
||||
- Built-in commands (list, explore, validate...)
|
||||
- Command aliases
|
||||
- Real-time updates as you add new adapters
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Installation details](/guide/installation)
|
||||
|
||||
@@ -20,17 +20,22 @@
|
||||
### Daemon issues
|
||||
|
||||
```bash
|
||||
# Check daemon status
|
||||
curl localhost:19825/status
|
||||
# Check daemon status (PID, uptime, extension connection, memory)
|
||||
opencli daemon status
|
||||
|
||||
# View extension logs
|
||||
curl localhost:19825/logs
|
||||
|
||||
# Kill and restart daemon
|
||||
pkill -f opencli-daemon
|
||||
# Stop or restart the daemon
|
||||
opencli daemon stop
|
||||
opencli daemon restart
|
||||
|
||||
# Full diagnostics
|
||||
opencli doctor
|
||||
```
|
||||
|
||||
> The daemon auto-exits after 4 hours of inactivity (no CLI requests and no extension connection). Override with `OPENCLI_DAEMON_TIMEOUT` (milliseconds, `0` = never timeout).
|
||||
|
||||
### Desktop adapter connection issues
|
||||
|
||||
For Electron/CDP-based adapters (Cursor, Codex, etc.):
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,857 @@
|
||||
# Daemon Lifecycle Redesign Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the daemon's aggressive 5-minute idle timeout with a long-lived model (4h default) that requires both CLI inactivity AND Extension disconnection before exiting, plus add `daemon status/stop/restart` CLI commands.
|
||||
|
||||
**Architecture:** The daemon keeps its existing HTTP + WebSocket bridge architecture. We change the idle timeout logic to track two independent activity signals (CLI requests and Extension connection), add `/status` and `/shutdown` HTTP endpoints, reduce the Extension reconnect backoff cap, and register new CLI commands via Commander.js.
|
||||
|
||||
**Tech Stack:** Node.js, TypeScript, Commander.js, ws, Vitest
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `src/constants.ts` | Modify | Add `DEFAULT_DAEMON_IDLE_TIMEOUT` constant |
|
||||
| `src/daemon.ts` | Modify | Dual-condition idle timer, `/status` endpoint, `/shutdown` endpoint |
|
||||
| `src/daemon.test.ts` | Create | Unit tests for idle timer logic, `/status`, `/shutdown` |
|
||||
| `extension/src/protocol.ts` | Modify | Change `WS_RECONNECT_MAX_DELAY` from 60000 to 5000 |
|
||||
| `src/cli.ts` | Modify | Register `daemon` subcommand group |
|
||||
| `src/commands/daemon.ts` | Create | `status`, `stop`, `restart` subcommand implementations |
|
||||
| `src/commands/daemon.test.ts` | Create | Unit tests for daemon commands |
|
||||
| `src/browser/mcp.ts` | Modify | Better connection-waiting UX messages, 200ms poll interval |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add `DEFAULT_DAEMON_IDLE_TIMEOUT` constant
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/constants.ts`
|
||||
|
||||
- [ ] **Step 1: Add the constant**
|
||||
|
||||
In `src/constants.ts`, add after the `DEFAULT_DAEMON_PORT` line:
|
||||
|
||||
```typescript
|
||||
/** Default idle timeout before daemon auto-exits (ms). Override via OPENCLI_DAEMON_TIMEOUT env var. */
|
||||
export const DEFAULT_DAEMON_IDLE_TIMEOUT = 4 * 60 * 60 * 1000; // 4 hours
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add src/constants.ts
|
||||
git commit -m "feat(daemon): add DEFAULT_DAEMON_IDLE_TIMEOUT constant (4 hours)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Implement dual-condition idle timer in daemon
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/daemon.ts:27,29-57,116-123,196-198,245-262,265-269`
|
||||
- Test: `src/daemon.test.ts` (create)
|
||||
|
||||
- [ ] **Step 1: Write failing tests for the new idle timer logic**
|
||||
|
||||
Create `src/daemon.test.ts`:
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
// We test the idle timer logic by extracting it into testable functions.
|
||||
// The daemon module has side effects (starts server), so we test the logic unit directly.
|
||||
|
||||
describe('IdleManager', () => {
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it('does not start timer when extension is connected', async () => {
|
||||
const { IdleManager } = await import('./daemon.js');
|
||||
const exit = vi.fn();
|
||||
const mgr = new IdleManager(300_000, exit); // 5 min for fast test
|
||||
|
||||
mgr.setExtensionConnected(true);
|
||||
mgr.onCliRequest();
|
||||
|
||||
vi.advanceTimersByTime(300_000 + 1000);
|
||||
expect(exit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('starts timer when extension disconnects and CLI is idle', async () => {
|
||||
const { IdleManager } = await import('./daemon.js');
|
||||
const exit = vi.fn();
|
||||
const mgr = new IdleManager(300_000, exit);
|
||||
|
||||
mgr.onCliRequest(); // CLI was active
|
||||
mgr.setExtensionConnected(true);
|
||||
mgr.setExtensionConnected(false); // Extension disconnects
|
||||
|
||||
// Should not exit immediately — CLI was just active
|
||||
expect(exit).not.toHaveBeenCalled();
|
||||
|
||||
// Advance past timeout
|
||||
vi.advanceTimersByTime(300_000 + 1000);
|
||||
expect(exit).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('exits immediately on extension disconnect if CLI has been idle past timeout', async () => {
|
||||
const { IdleManager } = await import('./daemon.js');
|
||||
const exit = vi.fn();
|
||||
const mgr = new IdleManager(300_000, exit);
|
||||
|
||||
mgr.onCliRequest(); // Last CLI activity
|
||||
vi.advanceTimersByTime(400_000); // 400s elapsed — past 300s timeout
|
||||
|
||||
mgr.setExtensionConnected(true);
|
||||
mgr.setExtensionConnected(false);
|
||||
|
||||
expect(exit).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('resets timer on new CLI request', async () => {
|
||||
const { IdleManager } = await import('./daemon.js');
|
||||
const exit = vi.fn();
|
||||
const mgr = new IdleManager(300_000, exit);
|
||||
|
||||
mgr.onCliRequest();
|
||||
vi.advanceTimersByTime(200_000); // 200s elapsed
|
||||
mgr.onCliRequest(); // Reset
|
||||
|
||||
vi.advanceTimersByTime(200_000); // 200s more — only 200s since last request
|
||||
expect(exit).not.toHaveBeenCalled();
|
||||
|
||||
vi.advanceTimersByTime(100_001); // Now 300s+ since last request
|
||||
expect(exit).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('does not exit when timeout is 0 (disabled)', async () => {
|
||||
const { IdleManager } = await import('./daemon.js');
|
||||
const exit = vi.fn();
|
||||
const mgr = new IdleManager(0, exit);
|
||||
|
||||
mgr.onCliRequest();
|
||||
vi.advanceTimersByTime(24 * 60 * 60 * 1000); // 24 hours
|
||||
expect(exit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('clears timer when extension connects', async () => {
|
||||
const { IdleManager } = await import('./daemon.js');
|
||||
const exit = vi.fn();
|
||||
const mgr = new IdleManager(300_000, exit);
|
||||
|
||||
mgr.onCliRequest();
|
||||
vi.advanceTimersByTime(200_000); // Timer running
|
||||
|
||||
mgr.setExtensionConnected(true); // Should clear timer
|
||||
vi.advanceTimersByTime(200_000); // Would have fired
|
||||
expect(exit).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
```bash
|
||||
npx vitest run src/daemon.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL — `IdleManager` is not exported from `./daemon.js`
|
||||
|
||||
- [ ] **Step 3: Extract IdleManager class and refactor daemon.ts**
|
||||
|
||||
In `src/daemon.ts`, replace the idle timeout section (lines 27, 29-57) with:
|
||||
|
||||
Replace the `IDLE_TIMEOUT` constant (line 27):
|
||||
```typescript
|
||||
import { DEFAULT_DAEMON_PORT, DEFAULT_DAEMON_IDLE_TIMEOUT } from './constants.js';
|
||||
|
||||
const PORT = parseInt(process.env.OPENCLI_DAEMON_PORT ?? String(DEFAULT_DAEMON_PORT), 10);
|
||||
const IDLE_TIMEOUT = Number(process.env.OPENCLI_DAEMON_TIMEOUT ?? DEFAULT_DAEMON_IDLE_TIMEOUT);
|
||||
```
|
||||
|
||||
Replace the idle timer state and `resetIdleTimer` function (lines 37, 49-57) with the `IdleManager` class:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Manages daemon idle timeout with dual-condition logic:
|
||||
* exits only when BOTH CLI is idle AND Extension is disconnected.
|
||||
*/
|
||||
export class IdleManager {
|
||||
private _timer: ReturnType<typeof setTimeout> | null = null;
|
||||
private _lastCliRequestTime = Date.now();
|
||||
private _extensionConnected = false;
|
||||
private _timeoutMs: number;
|
||||
private _onExit: () => void;
|
||||
|
||||
constructor(timeoutMs: number, onExit: () => void) {
|
||||
this._timeoutMs = timeoutMs;
|
||||
this._onExit = onExit;
|
||||
}
|
||||
|
||||
/** Call when an HTTP request arrives from CLI */
|
||||
onCliRequest(): void {
|
||||
this._lastCliRequestTime = Date.now();
|
||||
this._resetTimer();
|
||||
}
|
||||
|
||||
/** Call when Extension WebSocket connects or disconnects */
|
||||
setExtensionConnected(connected: boolean): void {
|
||||
this._extensionConnected = connected;
|
||||
if (connected) {
|
||||
// Extension is alive — clear any pending exit timer
|
||||
this._clearTimer();
|
||||
} else {
|
||||
// Extension gone — check if CLI has also been idle long enough
|
||||
this._resetTimer();
|
||||
}
|
||||
}
|
||||
|
||||
private _clearTimer(): void {
|
||||
if (this._timer) {
|
||||
clearTimeout(this._timer);
|
||||
this._timer = null;
|
||||
}
|
||||
}
|
||||
|
||||
private _resetTimer(): void {
|
||||
this._clearTimer();
|
||||
|
||||
// Timeout disabled
|
||||
if (this._timeoutMs <= 0) return;
|
||||
|
||||
// Extension connected — don't start timer
|
||||
if (this._extensionConnected) return;
|
||||
|
||||
const elapsed = Date.now() - this._lastCliRequestTime;
|
||||
if (elapsed >= this._timeoutMs) {
|
||||
// CLI has been idle past the timeout already
|
||||
this._onExit();
|
||||
return;
|
||||
}
|
||||
|
||||
// Start timer for remaining duration
|
||||
this._timer = setTimeout(() => {
|
||||
this._onExit();
|
||||
}, this._timeoutMs - elapsed);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then create the global `idleManager` instance after the class definition:
|
||||
|
||||
```typescript
|
||||
const idleManager = new IdleManager(IDLE_TIMEOUT, () => {
|
||||
console.error('[daemon] Idle timeout (no CLI requests + no Extension), shutting down');
|
||||
process.exit(0);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Wire IdleManager into existing daemon code**
|
||||
|
||||
In the `handleRequest` function, replace `resetIdleTimer()` (line 142) with:
|
||||
```typescript
|
||||
idleManager.onCliRequest();
|
||||
```
|
||||
|
||||
In the `wss.on('connection')` handler (around line 196-198), add after `extensionWs = ws;`:
|
||||
```typescript
|
||||
idleManager.setExtensionConnected(true);
|
||||
```
|
||||
|
||||
In the `ws.on('close')` handler (around line 245-249), add after `extensionWs = null;`:
|
||||
```typescript
|
||||
idleManager.setExtensionConnected(false);
|
||||
```
|
||||
|
||||
In the `ws.on('error')` handler (around line 259-261), add after `extensionWs = null;`:
|
||||
```typescript
|
||||
idleManager.setExtensionConnected(false);
|
||||
```
|
||||
|
||||
In the `httpServer.listen` callback (line 268-269), replace `resetIdleTimer()` with:
|
||||
```typescript
|
||||
idleManager.onCliRequest(); // Start initial idle countdown
|
||||
```
|
||||
|
||||
Remove the old `resetIdleTimer` function and `idleTimer` variable entirely.
|
||||
|
||||
- [ ] **Step 5: Run tests to verify they pass**
|
||||
|
||||
```bash
|
||||
npx vitest run src/daemon.test.ts
|
||||
```
|
||||
|
||||
Expected: All 6 tests PASS
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/daemon.ts src/daemon.test.ts
|
||||
git commit -m "feat(daemon): replace fixed 5min timeout with dual-condition idle manager (4h default)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Add `/status` and `/shutdown` endpoints to daemon
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/daemon.ts:116-123`
|
||||
|
||||
- [ ] **Step 1: Add tests for /status and /shutdown endpoints**
|
||||
|
||||
Append to `src/daemon.test.ts`:
|
||||
|
||||
```typescript
|
||||
describe('/status endpoint', () => {
|
||||
it('returns daemon status with correct fields', async () => {
|
||||
// This is an integration test — tested via the daemon command tests.
|
||||
// Here we just verify the shape of the status response type.
|
||||
expect(true).toBe(true); // Placeholder — real coverage in Task 6
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Note: The `/status` and `/shutdown` endpoints run inside the daemon process, which makes them hard to unit test in isolation. They are integration-tested via the `opencli daemon status/stop` commands in Task 6.
|
||||
|
||||
- [ ] **Step 2: Enhance the existing `/status` endpoint**
|
||||
|
||||
In `src/daemon.ts`, replace the existing `/status` handler (lines 116-123) with:
|
||||
|
||||
```typescript
|
||||
if (req.method === 'GET' && pathname === '/status') {
|
||||
const uptime = process.uptime();
|
||||
const mem = process.memoryUsage();
|
||||
jsonResponse(res, 200, {
|
||||
ok: true,
|
||||
pid: process.pid,
|
||||
uptime,
|
||||
extensionConnected: extensionWs?.readyState === WebSocket.OPEN,
|
||||
pending: pending.size,
|
||||
lastCliRequestTime: idleManager.lastCliRequestTime,
|
||||
memoryMB: Math.round(mem.rss / 1024 / 1024 * 10) / 10,
|
||||
port: PORT,
|
||||
});
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
Also add a public getter to `IdleManager`:
|
||||
|
||||
```typescript
|
||||
get lastCliRequestTime(): number {
|
||||
return this._lastCliRequestTime;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add the `/shutdown` endpoint**
|
||||
|
||||
In `src/daemon.ts`, add before the `POST /command` handler:
|
||||
|
||||
```typescript
|
||||
if (req.method === 'POST' && pathname === '/shutdown') {
|
||||
jsonResponse(res, 200, { ok: true, message: 'Shutting down' });
|
||||
// Graceful shutdown after response is sent
|
||||
setTimeout(() => shutdown(), 100);
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run all tests**
|
||||
|
||||
```bash
|
||||
npx vitest run src/daemon.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/daemon.ts src/daemon.test.ts
|
||||
git commit -m "feat(daemon): enhance /status endpoint, add /shutdown endpoint"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Reduce Extension WebSocket reconnect backoff cap
|
||||
|
||||
**Files:**
|
||||
- Modify: `extension/src/protocol.ts:57`
|
||||
|
||||
- [ ] **Step 1: Change the constant**
|
||||
|
||||
In `extension/src/protocol.ts`, change line 57:
|
||||
|
||||
```typescript
|
||||
/** Max reconnect delay (ms) — kept short since daemon is long-lived */
|
||||
export const WS_RECONNECT_MAX_DELAY = 5000;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add extension/src/protocol.ts
|
||||
git commit -m "feat(extension): reduce WS reconnect backoff cap from 60s to 5s"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Implement `daemon status/stop/restart` CLI commands
|
||||
|
||||
**Files:**
|
||||
- Create: `src/commands/daemon.ts`
|
||||
- Modify: `src/cli.ts`
|
||||
|
||||
- [ ] **Step 1: Create daemon command module**
|
||||
|
||||
Create `src/commands/daemon.ts`:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* CLI commands for daemon lifecycle management:
|
||||
* opencli daemon status — show daemon state
|
||||
* opencli daemon stop — graceful shutdown
|
||||
* opencli daemon restart — stop + respawn
|
||||
*/
|
||||
|
||||
import chalk from 'chalk';
|
||||
import { DEFAULT_DAEMON_PORT } from '../constants.js';
|
||||
|
||||
const DAEMON_PORT = parseInt(process.env.OPENCLI_DAEMON_PORT ?? String(DEFAULT_DAEMON_PORT), 10);
|
||||
const DAEMON_URL = `http://127.0.0.1:${DAEMON_PORT}`;
|
||||
|
||||
interface DaemonStatus {
|
||||
ok: boolean;
|
||||
pid: number;
|
||||
uptime: number;
|
||||
extensionConnected: boolean;
|
||||
pending: number;
|
||||
lastCliRequestTime: number;
|
||||
memoryMB: number;
|
||||
port: number;
|
||||
}
|
||||
|
||||
async function fetchStatus(): Promise<DaemonStatus | null> {
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), 2000);
|
||||
const res = await fetch(`${DAEMON_URL}/status`, {
|
||||
headers: { 'X-OpenCLI': '1' },
|
||||
signal: controller.signal,
|
||||
});
|
||||
clearTimeout(timer);
|
||||
if (!res.ok) return null;
|
||||
return await res.json() as DaemonStatus;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function requestShutdown(): Promise<boolean> {
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), 5000);
|
||||
const res = await fetch(`${DAEMON_URL}/shutdown`, {
|
||||
method: 'POST',
|
||||
headers: { 'X-OpenCLI': '1' },
|
||||
signal: controller.signal,
|
||||
});
|
||||
clearTimeout(timer);
|
||||
return res.ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function formatUptime(seconds: number): string {
|
||||
const h = Math.floor(seconds / 3600);
|
||||
const m = Math.floor((seconds % 3600) / 60);
|
||||
if (h > 0) return `${h}h ${m}m`;
|
||||
if (m > 0) return `${m}m`;
|
||||
return `${Math.floor(seconds)}s`;
|
||||
}
|
||||
|
||||
function formatTimeSince(timestampMs: number): string {
|
||||
const seconds = (Date.now() - timestampMs) / 1000;
|
||||
if (seconds < 60) return `${Math.floor(seconds)}s ago`;
|
||||
const m = Math.floor(seconds / 60);
|
||||
if (m < 60) return `${m} min ago`;
|
||||
const h = Math.floor(m / 60);
|
||||
return `${h}h ${m % 60}m ago`;
|
||||
}
|
||||
|
||||
export async function daemonStatus(): Promise<void> {
|
||||
const status = await fetchStatus();
|
||||
if (!status) {
|
||||
console.log(`Daemon: ${chalk.dim('not running')}`);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Daemon: ${chalk.green('running')} (PID ${status.pid})`);
|
||||
console.log(`Uptime: ${formatUptime(status.uptime)}`);
|
||||
console.log(`Extension: ${status.extensionConnected ? chalk.green('connected') : chalk.yellow('disconnected')}`);
|
||||
console.log(`Last CLI request: ${formatTimeSince(status.lastCliRequestTime)}`);
|
||||
console.log(`Memory: ${status.memoryMB} MB`);
|
||||
console.log(`Port: ${status.port}`);
|
||||
}
|
||||
|
||||
export async function daemonStop(): Promise<void> {
|
||||
const status = await fetchStatus();
|
||||
if (!status) {
|
||||
console.log(chalk.dim('Daemon is not running.'));
|
||||
return;
|
||||
}
|
||||
|
||||
const ok = await requestShutdown();
|
||||
if (ok) {
|
||||
console.log(chalk.green('Daemon stopped.'));
|
||||
} else {
|
||||
console.error(chalk.red('Failed to stop daemon.'));
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
export async function daemonRestart(): Promise<void> {
|
||||
const status = await fetchStatus();
|
||||
if (status) {
|
||||
const ok = await requestShutdown();
|
||||
if (!ok) {
|
||||
console.error(chalk.red('Failed to stop daemon.'));
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
// Wait for daemon to exit
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
|
||||
// Import BrowserBridge to spawn a new daemon
|
||||
const { BrowserBridge } = await import('../browser/mcp.js');
|
||||
const bridge = new BrowserBridge();
|
||||
try {
|
||||
console.log('Starting daemon...');
|
||||
await bridge.connect({ timeout: 10 });
|
||||
console.log(chalk.green('Daemon restarted.'));
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Failed to restart daemon: ${err instanceof Error ? err.message : err}`));
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Register daemon commands in cli.ts**
|
||||
|
||||
In `src/cli.ts`, add the import at the top:
|
||||
|
||||
```typescript
|
||||
import { daemonStatus, daemonStop, daemonRestart } from './commands/daemon.js';
|
||||
```
|
||||
|
||||
Add the daemon subcommand group before the `// ── External CLIs` section (around line 380):
|
||||
|
||||
```typescript
|
||||
// ── Built-in: daemon ──────────────────────────────────────────────────────
|
||||
const daemonCmd = program.command('daemon').description('Manage the opencli daemon');
|
||||
daemonCmd
|
||||
.command('status')
|
||||
.description('Show daemon status')
|
||||
.action(async () => { await daemonStatus(); });
|
||||
daemonCmd
|
||||
.command('stop')
|
||||
.description('Stop the daemon')
|
||||
.action(async () => { await daemonStop(); });
|
||||
daemonCmd
|
||||
.command('restart')
|
||||
.description('Restart the daemon')
|
||||
.action(async () => { await daemonRestart(); });
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run linter/type check**
|
||||
|
||||
```bash
|
||||
npx tsc --noEmit
|
||||
```
|
||||
|
||||
Expected: No errors
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/commands/daemon.ts src/cli.ts
|
||||
git commit -m "feat(daemon): add opencli daemon status/stop/restart commands"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Write tests for daemon commands
|
||||
|
||||
**Files:**
|
||||
- Create: `src/commands/daemon.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write tests**
|
||||
|
||||
Create `src/commands/daemon.test.ts`:
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
// Mock fetch globally for all tests
|
||||
const mockFetch = vi.fn();
|
||||
vi.stubGlobal('fetch', mockFetch);
|
||||
|
||||
// Mock chalk to avoid ANSI in assertions
|
||||
vi.mock('chalk', () => ({
|
||||
default: {
|
||||
green: (s: string) => s,
|
||||
yellow: (s: string) => s,
|
||||
red: (s: string) => s,
|
||||
dim: (s: string) => s,
|
||||
},
|
||||
}));
|
||||
|
||||
describe('daemonStatus', () => {
|
||||
let consoleSpy: ReturnType<typeof vi.spyOn>;
|
||||
|
||||
beforeEach(() => {
|
||||
consoleSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
consoleSpy.mockRestore();
|
||||
mockFetch.mockReset();
|
||||
});
|
||||
|
||||
it('shows "not running" when daemon is unreachable', async () => {
|
||||
mockFetch.mockRejectedValue(new TypeError('fetch failed'));
|
||||
|
||||
const { daemonStatus } = await import('./daemon.js');
|
||||
await daemonStatus();
|
||||
|
||||
expect(consoleSpy).toHaveBeenCalledWith(expect.stringContaining('not running'));
|
||||
});
|
||||
|
||||
it('shows daemon info when running', async () => {
|
||||
mockFetch.mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
ok: true,
|
||||
pid: 12345,
|
||||
uptime: 7200,
|
||||
extensionConnected: true,
|
||||
pending: 0,
|
||||
lastCliRequestTime: Date.now() - 60_000,
|
||||
memoryMB: 12.3,
|
||||
port: 19825,
|
||||
}),
|
||||
});
|
||||
|
||||
const { daemonStatus } = await import('./daemon.js');
|
||||
await daemonStatus();
|
||||
|
||||
expect(consoleSpy).toHaveBeenCalledWith(expect.stringContaining('running'));
|
||||
expect(consoleSpy).toHaveBeenCalledWith(expect.stringContaining('12345'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('daemonStop', () => {
|
||||
let consoleSpy: ReturnType<typeof vi.spyOn>;
|
||||
let consoleErrSpy: ReturnType<typeof vi.spyOn>;
|
||||
|
||||
beforeEach(() => {
|
||||
consoleSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
consoleErrSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
consoleSpy.mockRestore();
|
||||
consoleErrSpy.mockRestore();
|
||||
mockFetch.mockReset();
|
||||
});
|
||||
|
||||
it('reports when daemon is not running', async () => {
|
||||
mockFetch.mockRejectedValue(new TypeError('fetch failed'));
|
||||
|
||||
const { daemonStop } = await import('./daemon.js');
|
||||
await daemonStop();
|
||||
|
||||
expect(consoleSpy).toHaveBeenCalledWith(expect.stringContaining('not running'));
|
||||
});
|
||||
|
||||
it('sends shutdown and reports success', async () => {
|
||||
// First call: fetchStatus
|
||||
// Second call: requestShutdown
|
||||
mockFetch
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({ ok: true, pid: 123, uptime: 100, extensionConnected: false, pending: 0, lastCliRequestTime: Date.now(), memoryMB: 10, port: 19825 }),
|
||||
})
|
||||
.mockResolvedValueOnce({ ok: true });
|
||||
|
||||
const { daemonStop } = await import('./daemon.js');
|
||||
await daemonStop();
|
||||
|
||||
expect(consoleSpy).toHaveBeenCalledWith(expect.stringContaining('stopped'));
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests**
|
||||
|
||||
```bash
|
||||
npx vitest run src/commands/daemon.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add src/commands/daemon.test.ts
|
||||
git commit -m "test(daemon): add tests for daemon status/stop commands"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Improve CLI connection-waiting UX
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/browser/mcp.ts:58-118`
|
||||
|
||||
- [ ] **Step 1: Improve error messages and poll interval**
|
||||
|
||||
In `src/browser/mcp.ts`, replace the `_ensureDaemon` method (lines 58-118) with:
|
||||
|
||||
```typescript
|
||||
private async _ensureDaemon(timeoutSeconds?: number): Promise<void> {
|
||||
const effectiveSeconds = (timeoutSeconds && timeoutSeconds > 0) ? timeoutSeconds : Math.ceil(DAEMON_SPAWN_TIMEOUT / 1000);
|
||||
const timeoutMs = effectiveSeconds * 1000;
|
||||
|
||||
// Fast path: extension already connected
|
||||
if (await isExtensionConnected()) return;
|
||||
|
||||
// Daemon running but no extension — wait for extension with progress
|
||||
if (await isDaemonRunning()) {
|
||||
if (process.env.OPENCLI_VERBOSE || process.stderr.isTTY) {
|
||||
process.stderr.write('⏳ Waiting for Chrome extension to connect...\n');
|
||||
process.stderr.write(' Make sure Chrome is open and the OpenCLI extension is enabled.\n');
|
||||
}
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
if (await isExtensionConnected()) return;
|
||||
}
|
||||
throw new Error(
|
||||
'Daemon is running but the Browser Extension is not connected.\n' +
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome.',
|
||||
);
|
||||
}
|
||||
|
||||
// No daemon — spawn one
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const parentDir = path.resolve(__dirname, '..');
|
||||
const daemonTs = path.join(parentDir, 'daemon.ts');
|
||||
const daemonJs = path.join(parentDir, 'daemon.js');
|
||||
const isTs = fs.existsSync(daemonTs);
|
||||
const daemonPath = isTs ? daemonTs : daemonJs;
|
||||
|
||||
if (process.env.OPENCLI_VERBOSE || process.stderr.isTTY) {
|
||||
process.stderr.write('⏳ Starting daemon...\n');
|
||||
}
|
||||
|
||||
const spawnArgs = isTs
|
||||
? [process.execPath, '--import', 'tsx/esm', daemonPath]
|
||||
: [process.execPath, daemonPath];
|
||||
|
||||
this._daemonProc = spawn(spawnArgs[0], spawnArgs.slice(1), {
|
||||
detached: true,
|
||||
stdio: 'ignore',
|
||||
env: { ...process.env },
|
||||
});
|
||||
this._daemonProc.unref();
|
||||
|
||||
// Wait for daemon + extension with faster polling
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
if (await isExtensionConnected()) return;
|
||||
}
|
||||
|
||||
if (await isDaemonRunning()) {
|
||||
throw new Error(
|
||||
'Daemon is running but the Browser Extension is not connected.\n' +
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome.',
|
||||
);
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
'Failed to start opencli daemon. Try running manually:\n' +
|
||||
` node ${daemonPath}\n` +
|
||||
`Make sure port ${DEFAULT_DAEMON_PORT} is available.`,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run existing browser tests to check for regressions**
|
||||
|
||||
```bash
|
||||
npx vitest run src/browser.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add src/browser/mcp.ts
|
||||
git commit -m "feat(daemon): improve CLI connection-waiting UX with progress messages and 200ms polling"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Run full test suite and verify
|
||||
|
||||
- [ ] **Step 1: Run type check**
|
||||
|
||||
```bash
|
||||
npx tsc --noEmit
|
||||
```
|
||||
|
||||
Expected: No errors
|
||||
|
||||
- [ ] **Step 2: Run all tests**
|
||||
|
||||
```bash
|
||||
npx vitest run
|
||||
```
|
||||
|
||||
Expected: All tests pass, no regressions
|
||||
|
||||
- [ ] **Step 3: Manual smoke test**
|
||||
|
||||
```bash
|
||||
# Check daemon status (should be "not running" if daemon isn't started)
|
||||
npx tsx src/main.ts daemon status
|
||||
|
||||
# Start daemon by running any browser command, then check status
|
||||
npx tsx src/main.ts daemon status
|
||||
|
||||
# Stop daemon
|
||||
npx tsx src/main.ts daemon stop
|
||||
|
||||
# Verify stopped
|
||||
npx tsx src/main.ts daemon status
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Final commit if any fixes needed**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: address issues found during smoke testing"
|
||||
```
|
||||
@@ -0,0 +1,170 @@
|
||||
# Performance: Smart Wait & INTERCEPT Fix
|
||||
|
||||
**Date**: 2026-03-28
|
||||
**Status**: Approved
|
||||
|
||||
## Problem
|
||||
|
||||
Three distinct performance/correctness issues:
|
||||
|
||||
1. **INTERCEPT strategy semantic bug**: After `installInterceptor()` + `goto()`, adapters call `wait(N)` — which now uses `waitForDomStableJs` and returns early when the DOM settles. But DOM-settle != network capture. The API response may arrive *after* DOM is stable, causing `getInterceptedRequests()` to return an empty array.
|
||||
|
||||
2. **Blind `wait(N)` in adapters**: ~30 high-traffic adapters (Twitter family, Medium, Substack, etc.) call `wait(5)` waiting for React/Vue to hydrate. These should wait for a specific DOM element to appear, not a fixed cap.
|
||||
|
||||
3. **Daemon cold-start polling**: Fixed 300ms poll loop means ~600ms before first successful `isExtensionConnected()` check, even though the daemon is typically ready in 500–800ms.
|
||||
|
||||
## Design
|
||||
|
||||
### Layer 1 — `waitForCapture()` (correctness fix + perf)
|
||||
|
||||
Add `waitForCapture(timeout?: number): Promise<void>` to `IPage`.
|
||||
|
||||
Polls `window.__opencli_xhr.length > 0` every 100ms inside the browser tab. Resolves as soon as ≥1 capture arrives; rejects after `timeout` seconds.
|
||||
|
||||
```typescript
|
||||
// dom-helpers.ts
|
||||
export function waitForCaptureJs(maxMs: number): string {
|
||||
return `
|
||||
new Promise((resolve, reject) => {
|
||||
const deadline = Date.now() + ${maxMs};
|
||||
const check = () => {
|
||||
if ((window.__opencli_xhr || []).length > 0) return resolve('captured');
|
||||
if (Date.now() > deadline) return reject(new Error('No capture within ${maxMs / 1000}s'));
|
||||
setTimeout(check, 100);
|
||||
};
|
||||
check();
|
||||
})
|
||||
`;
|
||||
}
|
||||
```
|
||||
|
||||
`page.ts` and `cdp.ts` implement `waitForCapture()` by calling `waitForCaptureJs`.
|
||||
|
||||
**All INTERCEPT adapters** replace `wait(N)` → `waitForCapture(N+2)` (slightly longer timeout as safety margin).
|
||||
|
||||
`stepIntercept` in `pipeline/steps/intercept.ts` replaces its internal `wait(timeout)` with `waitForCapture(timeout)`.
|
||||
|
||||
**Expected gain**: 36kr hot/search: 6s → ~1–2s. Twitter search/followers: 5–8s → ~1–3s.
|
||||
|
||||
### Layer 2 — `wait({ selector })` (semantic precision)
|
||||
|
||||
Extend `WaitOptions` with `selector?: string`.
|
||||
|
||||
Add `waitForSelectorJs(selector, timeoutMs)` to `dom-helpers.ts` — polls `document.querySelector(selector)` every 100ms, resolves on first match, rejects on timeout.
|
||||
|
||||
```typescript
|
||||
// types.ts
|
||||
export interface WaitOptions {
|
||||
text?: string;
|
||||
selector?: string; // NEW
|
||||
time?: number;
|
||||
timeout?: number;
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
// dom-helpers.ts
|
||||
export function waitForSelectorJs(selector: string, timeoutMs: number): string {
|
||||
return `
|
||||
new Promise((resolve, reject) => {
|
||||
const deadline = Date.now() + ${timeoutMs};
|
||||
const check = () => {
|
||||
if (document.querySelector(${JSON.stringify(selector)})) return resolve('found');
|
||||
if (Date.now() > deadline) return reject(new Error('Selector not found: ' + ${JSON.stringify(selector)}));
|
||||
setTimeout(check, 100);
|
||||
};
|
||||
check();
|
||||
})
|
||||
`;
|
||||
}
|
||||
```
|
||||
|
||||
`page.ts` and `cdp.ts` handle `selector` branch in `wait()`.
|
||||
|
||||
**High-impact adapter changes**:
|
||||
|
||||
| Adapter | Old | New |
|
||||
|---------|-----|-----|
|
||||
| `twitter/*` (15 adapters) | `wait(5)` | `wait({ selector: '[data-testid="primaryColumn"]', timeout: 6 })` |
|
||||
| `twitter/reply.ts` | `wait(5)` | `wait({ selector: '[data-testid="tweetTextarea_0"]', timeout: 8 })` |
|
||||
| `medium/utils.ts` | `wait(5)` + inline 3s setTimeout | `wait({ selector: 'article', timeout: 8 })` + remove inline sleep |
|
||||
| `substack/utils.ts` | `wait(5)` × 2 | `wait({ selector: 'article', timeout: 8 })` |
|
||||
| `bloomberg/news.ts` | `wait(5)` | `wait({ selector: 'article', timeout: 6 })` |
|
||||
| `sinablog/utils.ts` | `wait(5)` | `wait({ selector: 'article, .article', timeout: 6 })` |
|
||||
| `producthunt` (already covered by layer 1) | — | — |
|
||||
|
||||
**Expected gain**: Twitter commands: 5s → ~0.5–2s. Medium: 8s → ~1–3s.
|
||||
|
||||
### Layer 3 — Daemon exponential backoff (cold-start)
|
||||
|
||||
Replace fixed 300ms poll in `_ensureDaemon()` (`browser/mcp.ts`) with exponential backoff:
|
||||
|
||||
```typescript
|
||||
// before
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 300));
|
||||
if (await isExtensionConnected()) return;
|
||||
}
|
||||
|
||||
// after
|
||||
const backoffs = [50, 100, 200, 400, 800, 1500, 3000];
|
||||
let i = 0;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, backoffs[Math.min(i++, backoffs.length - 1)]));
|
||||
if (await isExtensionConnected()) return;
|
||||
}
|
||||
```
|
||||
|
||||
**Expected gain**: First cold-start check succeeds at ~150ms instead of ~600ms.
|
||||
|
||||
## Files Changed
|
||||
|
||||
### New / Modified (framework)
|
||||
- `src/types.ts` — `WaitOptions.selector`, `IPage.waitForCapture()`
|
||||
- `src/browser/dom-helpers.ts` — `waitForCaptureJs()`, `waitForSelectorJs()`
|
||||
- `src/browser/page.ts` — `waitForCapture()`, `wait()` selector branch
|
||||
- `src/browser/cdp.ts` — `waitForCapture()`, `wait()` selector branch
|
||||
- `src/browser/mcp.ts` — exponential backoff in `_ensureDaemon()`
|
||||
- `src/pipeline/steps/intercept.ts` — use `waitForCapture()`
|
||||
|
||||
### Modified (adapters — Layer 1, INTERCEPT)
|
||||
- `src/clis/36kr/hot.ts`
|
||||
- `src/clis/36kr/search.ts`
|
||||
- `src/clis/twitter/search.ts`
|
||||
- `src/clis/twitter/followers.ts`
|
||||
- `src/clis/twitter/following.ts`
|
||||
- `src/clis/producthunt/hot.ts`
|
||||
- `src/clis/producthunt/browse.ts`
|
||||
|
||||
### Modified (adapters — Layer 2, selector)
|
||||
- `src/clis/twitter/reply.ts`
|
||||
- `src/clis/twitter/follow.ts`
|
||||
- `src/clis/twitter/unfollow.ts`
|
||||
- `src/clis/twitter/like.ts`
|
||||
- `src/clis/twitter/bookmark.ts`
|
||||
- `src/clis/twitter/unbookmark.ts`
|
||||
- `src/clis/twitter/block.ts`
|
||||
- `src/clis/twitter/unblock.ts`
|
||||
- `src/clis/twitter/hide-reply.ts`
|
||||
- `src/clis/twitter/notifications.ts`
|
||||
- `src/clis/twitter/profile.ts`
|
||||
- `src/clis/twitter/thread.ts`
|
||||
- `src/clis/twitter/timeline.ts`
|
||||
- `src/clis/twitter/delete.ts`
|
||||
- `src/clis/twitter/reply-dm.ts`
|
||||
- `src/clis/medium/utils.ts`
|
||||
- `src/clis/substack/utils.ts`
|
||||
- `src/clis/bloomberg/news.ts`
|
||||
- `src/clis/sinablog/utils.ts`
|
||||
|
||||
## Delivery Order
|
||||
|
||||
1. Layer 1 (`waitForCapture`) — correctness fix, highest ROI
|
||||
2. Layer 3 (backoff) — 3-line change, zero risk
|
||||
3. Layer 2 (`wait({ selector })`) — largest adapter surface, can be done per-site
|
||||
|
||||
## Testing
|
||||
|
||||
- Unit tests: `waitForCaptureJs`, `waitForSelectorJs` exported and tested in `dom-helpers.test.ts` (if exists) or new test file
|
||||
- Adapter tests: existing tests must continue to pass (mock `page.wait` / `page.waitForCapture`)
|
||||
- Run: `npx vitest run --project unit --project adapter`
|
||||
@@ -0,0 +1,208 @@
|
||||
# Daemon Lifecycle Redesign
|
||||
|
||||
## Problem
|
||||
|
||||
OpenCLI's daemon auto-exits after 5 minutes of idle time. During typical development
|
||||
cycles (write code → test → modify → test again), coding intervals frequently exceed
|
||||
5 minutes. Each restart incurs 2-4 seconds of overhead (process spawn + Extension
|
||||
WebSocket reconnection), creating a noticeable and frustrating delay.
|
||||
|
||||
The current design treats the daemon as a disposable process, but the actual cost
|
||||
profile doesn't justify this:
|
||||
|
||||
| Cost of staying alive | Cost of restarting |
|
||||
|-----------------------|--------------------|
|
||||
| ~12 MB memory, 0% CPU | 2-4 seconds delay per restart |
|
||||
|
||||
The restart cost far outweighs the idle cost.
|
||||
|
||||
## Solution
|
||||
|
||||
Replace the aggressive 5-minute fixed timeout with a long-lived daemon model. The
|
||||
daemon stays running for hours, exits only when truly abandoned, and reconnects to
|
||||
the Chrome Extension faster when needed.
|
||||
|
||||
Four changes:
|
||||
|
||||
1. Extend idle timeout from 5 minutes to 4 hours (configurable)
|
||||
2. Require dual idle condition: both no CLI requests AND no Extension connection
|
||||
3. Reduce Extension WebSocket reconnect backoff cap from 60s to 5s
|
||||
4. Add `opencli daemon status/stop/restart` commands
|
||||
|
||||
## Design
|
||||
|
||||
### Timeout Strategy
|
||||
|
||||
**Current behavior:** A single idle timer resets on each HTTP request. After 5
|
||||
minutes without a request, the daemon calls `process.exit(0)`.
|
||||
|
||||
**New behavior:** The daemon tracks two activity signals independently:
|
||||
|
||||
- **CLI activity:** timestamp of the last HTTP request from any CLI invocation
|
||||
- **Extension activity:** whether a WebSocket connection from the Chrome Extension
|
||||
is currently open
|
||||
|
||||
The exit countdown starts only when BOTH conditions are met simultaneously:
|
||||
|
||||
- No CLI request for `IDLE_TIMEOUT` duration
|
||||
- No Extension WebSocket connection
|
||||
|
||||
If either signal is active, the daemon stays alive. This means:
|
||||
|
||||
- A connected Extension keeps the daemon alive indefinitely (user has Chrome open,
|
||||
likely still working)
|
||||
- Recent CLI activity keeps the daemon alive even if Extension temporarily
|
||||
disconnects (Chrome restarting, Extension updating)
|
||||
|
||||
**Timeout value:** 4 hours by default, configurable via `OPENCLI_DAEMON_TIMEOUT`
|
||||
environment variable. Value in milliseconds. Set to `0` to disable timeout entirely.
|
||||
|
||||
```typescript
|
||||
const DEFAULT_IDLE_TIMEOUT = 4 * 60 * 60 * 1000; // 4 hours
|
||||
const IDLE_TIMEOUT = Number(process.env.OPENCLI_DAEMON_TIMEOUT ?? DEFAULT_IDLE_TIMEOUT);
|
||||
```
|
||||
|
||||
**Timer implementation:**
|
||||
|
||||
```
|
||||
resetIdleTimer():
|
||||
clear existing timer
|
||||
if Extension is connected:
|
||||
do not start timer (Extension connection keeps daemon alive)
|
||||
return
|
||||
start timer with IDLE_TIMEOUT duration
|
||||
on timeout: process.exit(0)
|
||||
|
||||
On CLI HTTP request:
|
||||
update lastRequestTime
|
||||
resetIdleTimer()
|
||||
|
||||
On Extension WebSocket connect:
|
||||
clear timer (Extension keeps daemon alive)
|
||||
|
||||
On Extension WebSocket disconnect:
|
||||
elapsed = now - lastRequestTime
|
||||
if elapsed >= IDLE_TIMEOUT:
|
||||
process.exit(0) // CLI has been idle long enough already
|
||||
else:
|
||||
start timer with (IDLE_TIMEOUT - elapsed) // count remaining time
|
||||
```
|
||||
|
||||
### Extension Fast Reconnect
|
||||
|
||||
**Current behavior:** When the Extension loses its WebSocket connection to the
|
||||
daemon, it reconnects with exponential backoff: 2s → 4s → 8s → 16s → 32s → 60s
|
||||
(capped). In the worst case, the Extension waits up to 60 seconds before attempting
|
||||
reconnection.
|
||||
|
||||
**New behavior:** Cap the backoff at 5 seconds instead of 60 seconds.
|
||||
|
||||
```typescript
|
||||
// extension/src/background.ts
|
||||
const WS_RECONNECT_MAX_DELAY = 5000; // was 60000
|
||||
```
|
||||
|
||||
Rationale: with a 4-hour daemon timeout, the daemon is almost always running. Long
|
||||
backoff intervals are unnecessary and only increase reconnection latency. A 5-second
|
||||
cap means the Extension reconnects within 5 seconds of the daemon becoming available.
|
||||
|
||||
### Daemon Management Commands
|
||||
|
||||
Add three new CLI commands for daemon lifecycle management:
|
||||
|
||||
**`opencli daemon status`**
|
||||
|
||||
Queries the daemon's `/status` endpoint (new) and displays:
|
||||
|
||||
```
|
||||
Daemon: running (PID 12345)
|
||||
Uptime: 2h 15m
|
||||
Extension: connected
|
||||
Last CLI request: 8 min ago
|
||||
Memory: 12.3 MB
|
||||
Port: 19825
|
||||
```
|
||||
|
||||
If daemon is not running:
|
||||
|
||||
```
|
||||
Daemon: not running
|
||||
```
|
||||
|
||||
**`opencli daemon stop`**
|
||||
|
||||
Sends a `POST /shutdown` request to the daemon, which triggers a graceful shutdown:
|
||||
reject pending requests with a shutdown message, close WebSocket connections, close
|
||||
HTTP server, then exit.
|
||||
|
||||
**`opencli daemon restart`**
|
||||
|
||||
Equivalent to `stop` followed by spawning a new daemon. Useful when the daemon gets
|
||||
into a bad state.
|
||||
|
||||
**Daemon-side endpoints:**
|
||||
|
||||
- `GET /status` — returns JSON with PID, uptime, extension connection state, last
|
||||
request time, memory usage
|
||||
- `POST /shutdown` — initiates graceful shutdown
|
||||
|
||||
Both endpoints require the same `X-OpenCLI` header as existing endpoints for CSRF
|
||||
protection.
|
||||
|
||||
### CLI Connection Experience
|
||||
|
||||
**Current behavior:** When daemon is running but Extension is not connected, the CLI
|
||||
silently polls every 300ms and eventually times out with a generic error.
|
||||
|
||||
**New behavior:** Show a progress indicator and actionable message:
|
||||
|
||||
```
|
||||
⏳ Waiting for Chrome extension to connect...
|
||||
Make sure Chrome is open and the OpenCLI extension is enabled.
|
||||
```
|
||||
|
||||
Poll interval reduced from 300ms to 200ms for slightly faster detection.
|
||||
|
||||
If the daemon is not running at all (connection refused), the CLI spawns it as before
|
||||
and shows:
|
||||
|
||||
```
|
||||
⏳ Starting daemon...
|
||||
```
|
||||
|
||||
## Files Changed
|
||||
|
||||
| File | Change | Estimated LOC |
|
||||
|------|--------|---------------|
|
||||
| `src/daemon.ts` | Dual-condition idle timeout, `/status` endpoint, `/shutdown` endpoint | ~40 |
|
||||
| `extension/src/background.ts` | `WS_RECONNECT_MAX_DELAY` 60000 → 5000 | 1 |
|
||||
| `src/browser/daemon-client.ts` | Better connection-waiting UX, 200ms poll interval | ~20 |
|
||||
| `src/commands/daemon.ts` (new) | `status`, `stop`, `restart` subcommands | ~80 |
|
||||
| `src/constants.ts` | `DEFAULT_IDLE_TIMEOUT` constant | 2 |
|
||||
|
||||
**Total: ~143 lines of new/changed code.**
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- No breaking changes to CLI commands or Extension protocol
|
||||
- Existing `OPENCLI_DAEMON_PORT` environment variable continues to work
|
||||
- The only observable behavior change: daemon stays alive longer
|
||||
- New `daemon` subcommands are additive
|
||||
|
||||
## Testing
|
||||
|
||||
- Unit test: idle timer starts only when both CLI and Extension are idle
|
||||
- Unit test: idle timer is cleared when Extension connects
|
||||
- Unit test: `/status` returns correct state
|
||||
- Unit test: `/shutdown` triggers graceful exit
|
||||
- Integration test: daemon survives 10+ minutes without CLI requests while Extension
|
||||
is connected
|
||||
- Integration test: daemon exits after configured timeout when fully idle
|
||||
- Integration test: `opencli daemon status/stop/restart` work correctly
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- OS-level daemon management (launchd/systemd) — can be added later if needed
|
||||
- Daemon auto-update mechanism
|
||||
- Multi-daemon coordination
|
||||
- Persistent daemon state across restarts
|
||||
@@ -0,0 +1,144 @@
|
||||
# Browse Skill Testing Design
|
||||
|
||||
Two-layer testing framework for `opencli browse` commands and the
|
||||
Claude Code skill integration.
|
||||
|
||||
## Goal
|
||||
|
||||
Verify that `opencli browse` works reliably on real websites and that
|
||||
Claude Code can use the skill to complete browser tasks end-to-end.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
autoresearch/
|
||||
├── browse-tasks.json ← 59 task definitions with browse command sequences
|
||||
├── eval-browse.ts ← Layer 1: deterministic browse command testing
|
||||
├── eval-skill.ts ← Layer 2: Claude Code skill E2E testing
|
||||
├── run-browse.sh ← Launch Layer 1
|
||||
├── run-skill.sh ← Launch Layer 2
|
||||
├── baseline-browse.txt ← Layer 1 best score
|
||||
├── baseline-skill.txt ← Layer 2 best score
|
||||
└── results/ ← Per-run results (gitignored)
|
||||
```
|
||||
|
||||
## Layer 1: Deterministic Browse Command Testing
|
||||
|
||||
Tests `opencli browse` commands directly on real websites. No LLM
|
||||
involved — pure command reliability testing.
|
||||
|
||||
### How It Works
|
||||
|
||||
Each task defines a sequence of browse commands and a judge for the
|
||||
last command's output:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "hn-top-stories",
|
||||
"steps": [
|
||||
"opencli browse open https://news.ycombinator.com",
|
||||
"opencli browse eval \"JSON.stringify([...document.querySelectorAll('.titleline a')].slice(0,5).map(a=>({title:a.textContent,url:a.href})))\""
|
||||
],
|
||||
"judge": { "type": "arrayMinLength", "minLength": 5 }
|
||||
}
|
||||
```
|
||||
|
||||
### Execution
|
||||
|
||||
```bash
|
||||
./autoresearch/run-browse.sh
|
||||
```
|
||||
|
||||
- Runs all 59 tasks serially
|
||||
- Each task: execute steps → judge last step output → pass/fail
|
||||
- `opencli browse close` between tasks for clean state
|
||||
- Expected: ~2 minutes, $0 cost
|
||||
|
||||
### Task Categories
|
||||
|
||||
| Category | Count | Example |
|
||||
|----------|-------|---------|
|
||||
| extract | 9 | Open page, eval JS to extract data |
|
||||
| list | 10 | Open page, eval JS to extract array |
|
||||
| search | 6 | Open, type query, keys Enter, eval results |
|
||||
| nav | 7 | Open, click link, eval new page title |
|
||||
| scroll | 5 | Open, scroll, eval footer/hidden content |
|
||||
| form | 6 | Open, type into fields, eval field values |
|
||||
| complex | 6 | Multi-step: open → click → navigate → extract |
|
||||
| bench | 10 | Test set (various) |
|
||||
|
||||
## Layer 2: Claude Code Skill E2E Testing
|
||||
|
||||
Spawns Claude Code with the opencli-operate skill to complete tasks
|
||||
autonomously using browse commands.
|
||||
|
||||
### How It Works
|
||||
|
||||
```bash
|
||||
claude -p \
|
||||
--system-prompt "$(cat skills/opencli-operate/SKILL.md)" \
|
||||
--dangerously-skip-permissions \
|
||||
--allowedTools "Bash(opencli:*)" \
|
||||
--output-format json \
|
||||
"用 opencli browse 完成任务:Extract the top 5 stories from Hacker News with title and score. Start URL: https://news.ycombinator.com"
|
||||
```
|
||||
|
||||
### Execution
|
||||
|
||||
```bash
|
||||
./autoresearch/run-skill.sh
|
||||
```
|
||||
|
||||
- Runs all 59 tasks serially
|
||||
- Each task: spawn Claude Code → it uses browse commands autonomously → judge output
|
||||
- Expected: ~20 minutes, ~$5-10
|
||||
|
||||
### Judge
|
||||
|
||||
Both layers use the same judge types:
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `contains` | Output contains a substring |
|
||||
| `arrayMinLength` | Output is an array with ≥ N items |
|
||||
| `arrayFieldsPresent` | Array items have required fields |
|
||||
| `nonEmpty` | Output is non-empty |
|
||||
| `matchesPattern` | Output matches a regex |
|
||||
|
||||
## Output Format
|
||||
|
||||
```
|
||||
🔬 Layer 1: Browse Commands — 59 tasks
|
||||
|
||||
[1/59] extract-title-example... ✓ (0.5s)
|
||||
[2/59] hn-top-stories... ✓ (1.2s)
|
||||
...
|
||||
|
||||
Score: 55/59 (93%)
|
||||
Time: 2min
|
||||
Cost: $0
|
||||
|
||||
🔬 Layer 2: Skill E2E — 59 tasks
|
||||
|
||||
[1/59] extract-title-example... ✓ (8s, $0.01)
|
||||
[2/59] hn-top-stories... ✓ (15s, $0.08)
|
||||
...
|
||||
|
||||
Score: 52/59 (88%)
|
||||
Time: 20min
|
||||
Cost: $6.50
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- All 59 tasks run on real websites (no mocks)
|
||||
- Layer 1: zero LLM cost, ~2 min
|
||||
- Layer 2: ~$5-10 LLM cost, ~20 min
|
||||
- Results saved to `autoresearch/results/` (gitignored)
|
||||
- Baselines tracked in `baseline-browse.txt` and `baseline-skill.txt`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Layer 1 ≥ 90% (browse commands work on real sites)
|
||||
- Layer 2 ≥ 85% (Claude Code can use skill effectively)
|
||||
- Both layers cover all 8 task categories
|
||||
@@ -22,3 +22,15 @@ OpenCLI 通过轻量级 **Browser Bridge** Chrome 扩展 + 微守护进程连接
|
||||
```bash
|
||||
opencli doctor # 检查扩展 + 守护进程连接
|
||||
```
|
||||
|
||||
## Daemon 生命周期
|
||||
|
||||
Daemon 在首次运行浏览器命令时自动启动,默认保持 **4 小时**。仅当 CLI 空闲超时**且** Chrome 扩展未连接时才会退出。
|
||||
|
||||
```bash
|
||||
opencli daemon status # 查看 daemon 状态(PID、运行时长、扩展连接、内存)
|
||||
opencli daemon stop # 优雅关停
|
||||
opencli daemon restart # 重启
|
||||
```
|
||||
|
||||
通过 `OPENCLI_DAEMON_TIMEOUT` 环境变量覆盖超时时间(毫秒)。设为 `0` 则永不超时。
|
||||
|
||||
@@ -32,6 +32,27 @@ opencli bilibili hot -f md # Markdown
|
||||
opencli bilibili hot -f csv # CSV
|
||||
```
|
||||
|
||||
## 终端自动补全
|
||||
|
||||
OpenCLI 支持智能的 Tab 自动补全,加快命令输入:
|
||||
|
||||
```bash
|
||||
# 把自动补全加入 shell 启动配置
|
||||
echo 'eval "$(opencli completion zsh)"' >> ~/.zshrc # Zsh
|
||||
echo 'eval "$(opencli completion bash)"' >> ~/.bashrc # Bash
|
||||
echo 'opencli completion fish | source' >> ~/.config/fish/config.fish # Fish
|
||||
|
||||
# 重启 shell 后,按 Tab 键补全:
|
||||
opencli [Tab] # 补全站点名称(bilibili、zhihu、twitter...)
|
||||
opencli bilibili [Tab] # 补全命令(hot、search、me、download...)
|
||||
```
|
||||
|
||||
补全功能包含:
|
||||
- 所有可用的站点和适配器
|
||||
- 内置命令(list、explore、validate...)
|
||||
- 命令别名
|
||||
- 新增适配器时的实时更新
|
||||
|
||||
## 下一步
|
||||
|
||||
- [安装详情](/zh/guide/installation)
|
||||
|
||||
Vendored
-574
@@ -1,574 +0,0 @@
|
||||
const DAEMON_PORT = 19825;
|
||||
const DAEMON_HOST = "localhost";
|
||||
const DAEMON_WS_URL = `ws://${DAEMON_HOST}:${DAEMON_PORT}/ext`;
|
||||
const WS_RECONNECT_BASE_DELAY = 2e3;
|
||||
const WS_RECONNECT_MAX_DELAY = 6e4;
|
||||
|
||||
const attached = /* @__PURE__ */ new Set();
|
||||
const BLANK_PAGE$1 = "data:text/html,<html></html>";
|
||||
function isDebuggableUrl$1(url) {
|
||||
if (!url) return true;
|
||||
return url.startsWith("http://") || url.startsWith("https://") || url === BLANK_PAGE$1;
|
||||
}
|
||||
async function ensureAttached(tabId) {
|
||||
try {
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
if (!isDebuggableUrl$1(tab.url)) {
|
||||
attached.delete(tabId);
|
||||
throw new Error(`Cannot debug tab ${tabId}: URL is ${tab.url ?? "unknown"}`);
|
||||
}
|
||||
} catch (e) {
|
||||
if (e instanceof Error && e.message.startsWith("Cannot debug tab")) throw e;
|
||||
attached.delete(tabId);
|
||||
throw new Error(`Tab ${tabId} no longer exists`);
|
||||
}
|
||||
if (attached.has(tabId)) {
|
||||
try {
|
||||
await chrome.debugger.sendCommand({ tabId }, "Runtime.evaluate", {
|
||||
expression: "1",
|
||||
returnByValue: true
|
||||
});
|
||||
return;
|
||||
} catch {
|
||||
attached.delete(tabId);
|
||||
}
|
||||
}
|
||||
try {
|
||||
await chrome.debugger.attach({ tabId }, "1.3");
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
const hint = msg.includes("chrome-extension://") ? ". Tip: another Chrome extension may be interfering — try disabling other extensions" : "";
|
||||
if (msg.includes("Another debugger is already attached")) {
|
||||
try {
|
||||
await chrome.debugger.detach({ tabId });
|
||||
} catch {
|
||||
}
|
||||
try {
|
||||
await chrome.debugger.attach({ tabId }, "1.3");
|
||||
} catch {
|
||||
throw new Error(`attach failed: ${msg}${hint}`);
|
||||
}
|
||||
} else {
|
||||
throw new Error(`attach failed: ${msg}${hint}`);
|
||||
}
|
||||
}
|
||||
attached.add(tabId);
|
||||
try {
|
||||
await chrome.debugger.sendCommand({ tabId }, "Runtime.enable");
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
async function evaluate(tabId, expression) {
|
||||
await ensureAttached(tabId);
|
||||
const result = await chrome.debugger.sendCommand({ tabId }, "Runtime.evaluate", {
|
||||
expression,
|
||||
returnByValue: true,
|
||||
awaitPromise: true
|
||||
});
|
||||
if (result.exceptionDetails) {
|
||||
const errMsg = result.exceptionDetails.exception?.description || result.exceptionDetails.text || "Eval error";
|
||||
throw new Error(errMsg);
|
||||
}
|
||||
return result.result?.value;
|
||||
}
|
||||
const evaluateAsync = evaluate;
|
||||
async function screenshot(tabId, options = {}) {
|
||||
await ensureAttached(tabId);
|
||||
const format = options.format ?? "png";
|
||||
if (options.fullPage) {
|
||||
const metrics = await chrome.debugger.sendCommand({ tabId }, "Page.getLayoutMetrics");
|
||||
const size = metrics.cssContentSize || metrics.contentSize;
|
||||
if (size) {
|
||||
await chrome.debugger.sendCommand({ tabId }, "Emulation.setDeviceMetricsOverride", {
|
||||
mobile: false,
|
||||
width: Math.ceil(size.width),
|
||||
height: Math.ceil(size.height),
|
||||
deviceScaleFactor: 1
|
||||
});
|
||||
}
|
||||
}
|
||||
try {
|
||||
const params = { format };
|
||||
if (format === "jpeg" && options.quality !== void 0) {
|
||||
params.quality = Math.max(0, Math.min(100, options.quality));
|
||||
}
|
||||
const result = await chrome.debugger.sendCommand({ tabId }, "Page.captureScreenshot", params);
|
||||
return result.data;
|
||||
} finally {
|
||||
if (options.fullPage) {
|
||||
await chrome.debugger.sendCommand({ tabId }, "Emulation.clearDeviceMetricsOverride").catch(() => {
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
async function detach(tabId) {
|
||||
if (!attached.has(tabId)) return;
|
||||
attached.delete(tabId);
|
||||
try {
|
||||
await chrome.debugger.detach({ tabId });
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
function registerListeners() {
|
||||
chrome.tabs.onRemoved.addListener((tabId) => {
|
||||
attached.delete(tabId);
|
||||
});
|
||||
chrome.debugger.onDetach.addListener((source) => {
|
||||
if (source.tabId) attached.delete(source.tabId);
|
||||
});
|
||||
chrome.tabs.onUpdated.addListener(async (tabId, info) => {
|
||||
if (info.url && !isDebuggableUrl$1(info.url)) {
|
||||
await detach(tabId);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
let ws = null;
|
||||
let reconnectTimer = null;
|
||||
let reconnectAttempts = 0;
|
||||
const _origLog = console.log.bind(console);
|
||||
const _origWarn = console.warn.bind(console);
|
||||
const _origError = console.error.bind(console);
|
||||
function forwardLog(level, args) {
|
||||
if (!ws || ws.readyState !== WebSocket.OPEN) return;
|
||||
try {
|
||||
const msg = args.map((a) => typeof a === "string" ? a : JSON.stringify(a)).join(" ");
|
||||
ws.send(JSON.stringify({ type: "log", level, msg, ts: Date.now() }));
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
console.log = (...args) => {
|
||||
_origLog(...args);
|
||||
forwardLog("info", args);
|
||||
};
|
||||
console.warn = (...args) => {
|
||||
_origWarn(...args);
|
||||
forwardLog("warn", args);
|
||||
};
|
||||
console.error = (...args) => {
|
||||
_origError(...args);
|
||||
forwardLog("error", args);
|
||||
};
|
||||
function connect() {
|
||||
if (ws?.readyState === WebSocket.OPEN || ws?.readyState === WebSocket.CONNECTING) return;
|
||||
try {
|
||||
ws = new WebSocket(DAEMON_WS_URL);
|
||||
} catch {
|
||||
scheduleReconnect();
|
||||
return;
|
||||
}
|
||||
ws.onopen = () => {
|
||||
console.log("[opencli] Connected to daemon");
|
||||
reconnectAttempts = 0;
|
||||
if (reconnectTimer) {
|
||||
clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
ws?.send(JSON.stringify({ type: "hello", version: chrome.runtime.getManifest().version }));
|
||||
};
|
||||
ws.onmessage = async (event) => {
|
||||
try {
|
||||
const command = JSON.parse(event.data);
|
||||
const result = await handleCommand(command);
|
||||
ws?.send(JSON.stringify(result));
|
||||
} catch (err) {
|
||||
console.error("[opencli] Message handling error:", err);
|
||||
}
|
||||
};
|
||||
ws.onclose = () => {
|
||||
console.log("[opencli] Disconnected from daemon");
|
||||
ws = null;
|
||||
scheduleReconnect();
|
||||
};
|
||||
ws.onerror = () => {
|
||||
ws?.close();
|
||||
};
|
||||
}
|
||||
const MAX_EAGER_ATTEMPTS = 6;
|
||||
function scheduleReconnect() {
|
||||
if (reconnectTimer) return;
|
||||
reconnectAttempts++;
|
||||
if (reconnectAttempts > MAX_EAGER_ATTEMPTS) return;
|
||||
const delay = Math.min(WS_RECONNECT_BASE_DELAY * Math.pow(2, reconnectAttempts - 1), WS_RECONNECT_MAX_DELAY);
|
||||
reconnectTimer = setTimeout(() => {
|
||||
reconnectTimer = null;
|
||||
connect();
|
||||
}, delay);
|
||||
}
|
||||
const automationSessions = /* @__PURE__ */ new Map();
|
||||
const WINDOW_IDLE_TIMEOUT = 3e4;
|
||||
function getWorkspaceKey(workspace) {
|
||||
return workspace?.trim() || "default";
|
||||
}
|
||||
function resetWindowIdleTimer(workspace) {
|
||||
const session = automationSessions.get(workspace);
|
||||
if (!session) return;
|
||||
if (session.idleTimer) clearTimeout(session.idleTimer);
|
||||
session.idleDeadlineAt = Date.now() + WINDOW_IDLE_TIMEOUT;
|
||||
session.idleTimer = setTimeout(async () => {
|
||||
const current = automationSessions.get(workspace);
|
||||
if (!current) return;
|
||||
try {
|
||||
await chrome.windows.remove(current.windowId);
|
||||
console.log(`[opencli] Automation window ${current.windowId} (${workspace}) closed (idle timeout)`);
|
||||
} catch {
|
||||
}
|
||||
automationSessions.delete(workspace);
|
||||
}, WINDOW_IDLE_TIMEOUT);
|
||||
}
|
||||
async function getAutomationWindow(workspace) {
|
||||
const existing = automationSessions.get(workspace);
|
||||
if (existing) {
|
||||
try {
|
||||
await chrome.windows.get(existing.windowId);
|
||||
return existing.windowId;
|
||||
} catch {
|
||||
automationSessions.delete(workspace);
|
||||
}
|
||||
}
|
||||
const win = await chrome.windows.create({
|
||||
url: BLANK_PAGE,
|
||||
focused: false,
|
||||
width: 1280,
|
||||
height: 900,
|
||||
type: "normal",
|
||||
state: "minimized"
|
||||
});
|
||||
const session = {
|
||||
windowId: win.id,
|
||||
idleTimer: null,
|
||||
idleDeadlineAt: Date.now() + WINDOW_IDLE_TIMEOUT
|
||||
};
|
||||
automationSessions.set(workspace, session);
|
||||
console.log(`[opencli] Created automation window ${session.windowId} (${workspace})`);
|
||||
resetWindowIdleTimer(workspace);
|
||||
await new Promise((resolve) => setTimeout(resolve, 200));
|
||||
return session.windowId;
|
||||
}
|
||||
chrome.windows.onRemoved.addListener((windowId) => {
|
||||
for (const [workspace, session] of automationSessions.entries()) {
|
||||
if (session.windowId === windowId) {
|
||||
console.log(`[opencli] Automation window closed (${workspace})`);
|
||||
if (session.idleTimer) clearTimeout(session.idleTimer);
|
||||
automationSessions.delete(workspace);
|
||||
}
|
||||
}
|
||||
});
|
||||
let initialized = false;
|
||||
function initialize() {
|
||||
if (initialized) return;
|
||||
initialized = true;
|
||||
chrome.alarms.create("keepalive", { periodInMinutes: 0.4 });
|
||||
registerListeners();
|
||||
connect();
|
||||
console.log("[opencli] OpenCLI extension initialized");
|
||||
}
|
||||
chrome.runtime.onInstalled.addListener(() => {
|
||||
initialize();
|
||||
});
|
||||
chrome.runtime.onStartup.addListener(() => {
|
||||
initialize();
|
||||
});
|
||||
chrome.alarms.onAlarm.addListener((alarm) => {
|
||||
if (alarm.name === "keepalive") connect();
|
||||
});
|
||||
chrome.runtime.onMessage.addListener((msg, _sender, sendResponse) => {
|
||||
if (msg?.type === "getStatus") {
|
||||
sendResponse({
|
||||
connected: ws?.readyState === WebSocket.OPEN,
|
||||
reconnecting: reconnectTimer !== null
|
||||
});
|
||||
}
|
||||
return false;
|
||||
});
|
||||
async function handleCommand(cmd) {
|
||||
const workspace = getWorkspaceKey(cmd.workspace);
|
||||
resetWindowIdleTimer(workspace);
|
||||
try {
|
||||
switch (cmd.action) {
|
||||
case "exec":
|
||||
return await handleExec(cmd, workspace);
|
||||
case "navigate":
|
||||
return await handleNavigate(cmd, workspace);
|
||||
case "tabs":
|
||||
return await handleTabs(cmd, workspace);
|
||||
case "cookies":
|
||||
return await handleCookies(cmd);
|
||||
case "screenshot":
|
||||
return await handleScreenshot(cmd, workspace);
|
||||
case "close-window":
|
||||
return await handleCloseWindow(cmd, workspace);
|
||||
case "sessions":
|
||||
return await handleSessions(cmd);
|
||||
default:
|
||||
return { id: cmd.id, ok: false, error: `Unknown action: ${cmd.action}` };
|
||||
}
|
||||
} catch (err) {
|
||||
return {
|
||||
id: cmd.id,
|
||||
ok: false,
|
||||
error: err instanceof Error ? err.message : String(err)
|
||||
};
|
||||
}
|
||||
}
|
||||
const BLANK_PAGE = "data:text/html,<html></html>";
|
||||
function isDebuggableUrl(url) {
|
||||
if (!url) return true;
|
||||
return url.startsWith("http://") || url.startsWith("https://") || url === BLANK_PAGE;
|
||||
}
|
||||
function isSafeNavigationUrl(url) {
|
||||
return url.startsWith("http://") || url.startsWith("https://");
|
||||
}
|
||||
function normalizeUrlForComparison(url) {
|
||||
if (!url) return "";
|
||||
try {
|
||||
const parsed = new URL(url);
|
||||
if (parsed.protocol === "https:" && parsed.port === "443" || parsed.protocol === "http:" && parsed.port === "80") {
|
||||
parsed.port = "";
|
||||
}
|
||||
const pathname = parsed.pathname === "/" ? "" : parsed.pathname;
|
||||
return `${parsed.protocol}//${parsed.host}${pathname}${parsed.search}${parsed.hash}`;
|
||||
} catch {
|
||||
return url;
|
||||
}
|
||||
}
|
||||
function isTargetUrl(currentUrl, targetUrl) {
|
||||
return normalizeUrlForComparison(currentUrl) === normalizeUrlForComparison(targetUrl);
|
||||
}
|
||||
async function resolveTabId(tabId, workspace) {
|
||||
if (tabId !== void 0) {
|
||||
try {
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
const session = automationSessions.get(workspace);
|
||||
if (isDebuggableUrl(tab.url) && session && tab.windowId === session.windowId) return tabId;
|
||||
if (session && tab.windowId !== session.windowId) {
|
||||
console.warn(`[opencli] Tab ${tabId} belongs to window ${tab.windowId}, not automation window ${session.windowId}, re-resolving`);
|
||||
} else if (!isDebuggableUrl(tab.url)) {
|
||||
console.warn(`[opencli] Tab ${tabId} URL is not debuggable (${tab.url}), re-resolving`);
|
||||
}
|
||||
} catch {
|
||||
console.warn(`[opencli] Tab ${tabId} no longer exists, re-resolving`);
|
||||
}
|
||||
}
|
||||
const windowId = await getAutomationWindow(workspace);
|
||||
const tabs = await chrome.tabs.query({ windowId });
|
||||
const debuggableTab = tabs.find((t) => t.id && isDebuggableUrl(t.url));
|
||||
if (debuggableTab?.id) return debuggableTab.id;
|
||||
const reuseTab = tabs.find((t) => t.id);
|
||||
if (reuseTab?.id) {
|
||||
await chrome.tabs.update(reuseTab.id, { url: BLANK_PAGE });
|
||||
await new Promise((resolve) => setTimeout(resolve, 300));
|
||||
try {
|
||||
const updated = await chrome.tabs.get(reuseTab.id);
|
||||
if (isDebuggableUrl(updated.url)) return reuseTab.id;
|
||||
console.warn(`[opencli] data: URI was intercepted (${updated.url}), creating fresh tab`);
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
const newTab = await chrome.tabs.create({ windowId, url: BLANK_PAGE, active: true });
|
||||
if (!newTab.id) throw new Error("Failed to create tab in automation window");
|
||||
return newTab.id;
|
||||
}
|
||||
async function listAutomationTabs(workspace) {
|
||||
const session = automationSessions.get(workspace);
|
||||
if (!session) return [];
|
||||
try {
|
||||
return await chrome.tabs.query({ windowId: session.windowId });
|
||||
} catch {
|
||||
automationSessions.delete(workspace);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
async function listAutomationWebTabs(workspace) {
|
||||
const tabs = await listAutomationTabs(workspace);
|
||||
return tabs.filter((tab) => isDebuggableUrl(tab.url));
|
||||
}
|
||||
async function handleExec(cmd, workspace) {
|
||||
if (!cmd.code) return { id: cmd.id, ok: false, error: "Missing code" };
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
const data = await evaluateAsync(tabId, cmd.code);
|
||||
return { id: cmd.id, ok: true, data };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
async function handleNavigate(cmd, workspace) {
|
||||
if (!cmd.url) return { id: cmd.id, ok: false, error: "Missing url" };
|
||||
if (!isSafeNavigationUrl(cmd.url)) {
|
||||
return { id: cmd.id, ok: false, error: "Blocked URL scheme -- only http:// and https:// are allowed" };
|
||||
}
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
const beforeTab = await chrome.tabs.get(tabId);
|
||||
const beforeNormalized = normalizeUrlForComparison(beforeTab.url);
|
||||
const targetUrl = cmd.url;
|
||||
if (beforeTab.status === "complete" && isTargetUrl(beforeTab.url, targetUrl)) {
|
||||
return {
|
||||
id: cmd.id,
|
||||
ok: true,
|
||||
data: { title: beforeTab.title, url: beforeTab.url, tabId, timedOut: false }
|
||||
};
|
||||
}
|
||||
await detach(tabId);
|
||||
await chrome.tabs.update(tabId, { url: targetUrl });
|
||||
let timedOut = false;
|
||||
await new Promise((resolve) => {
|
||||
let settled = false;
|
||||
let checkTimer = null;
|
||||
let timeoutTimer = null;
|
||||
const finish = () => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
chrome.tabs.onUpdated.removeListener(listener);
|
||||
if (checkTimer) clearTimeout(checkTimer);
|
||||
if (timeoutTimer) clearTimeout(timeoutTimer);
|
||||
resolve();
|
||||
};
|
||||
const isNavigationDone = (url) => {
|
||||
return isTargetUrl(url, targetUrl) || normalizeUrlForComparison(url) !== beforeNormalized;
|
||||
};
|
||||
const listener = (id, info, tab2) => {
|
||||
if (id !== tabId) return;
|
||||
if (info.status === "complete" && isNavigationDone(tab2.url ?? info.url)) {
|
||||
finish();
|
||||
}
|
||||
};
|
||||
chrome.tabs.onUpdated.addListener(listener);
|
||||
checkTimer = setTimeout(async () => {
|
||||
try {
|
||||
const currentTab = await chrome.tabs.get(tabId);
|
||||
if (currentTab.status === "complete" && isNavigationDone(currentTab.url)) {
|
||||
finish();
|
||||
}
|
||||
} catch {
|
||||
}
|
||||
}, 100);
|
||||
timeoutTimer = setTimeout(() => {
|
||||
timedOut = true;
|
||||
console.warn(`[opencli] Navigate to ${targetUrl} timed out after 15s`);
|
||||
finish();
|
||||
}, 15e3);
|
||||
});
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
return {
|
||||
id: cmd.id,
|
||||
ok: true,
|
||||
data: { title: tab.title, url: tab.url, tabId, timedOut }
|
||||
};
|
||||
}
|
||||
async function handleTabs(cmd, workspace) {
|
||||
switch (cmd.op) {
|
||||
case "list": {
|
||||
const tabs = await listAutomationWebTabs(workspace);
|
||||
const data = tabs.map((t, i) => ({
|
||||
index: i,
|
||||
tabId: t.id,
|
||||
url: t.url,
|
||||
title: t.title,
|
||||
active: t.active
|
||||
}));
|
||||
return { id: cmd.id, ok: true, data };
|
||||
}
|
||||
case "new": {
|
||||
if (cmd.url && !isSafeNavigationUrl(cmd.url)) {
|
||||
return { id: cmd.id, ok: false, error: "Blocked URL scheme -- only http:// and https:// are allowed" };
|
||||
}
|
||||
const windowId = await getAutomationWindow(workspace);
|
||||
const tab = await chrome.tabs.create({ windowId, url: cmd.url ?? BLANK_PAGE, active: true });
|
||||
return { id: cmd.id, ok: true, data: { tabId: tab.id, url: tab.url } };
|
||||
}
|
||||
case "close": {
|
||||
if (cmd.index !== void 0) {
|
||||
const tabs = await listAutomationWebTabs(workspace);
|
||||
const target = tabs[cmd.index];
|
||||
if (!target?.id) return { id: cmd.id, ok: false, error: `Tab index ${cmd.index} not found` };
|
||||
await chrome.tabs.remove(target.id);
|
||||
await detach(target.id);
|
||||
return { id: cmd.id, ok: true, data: { closed: target.id } };
|
||||
}
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
await chrome.tabs.remove(tabId);
|
||||
await detach(tabId);
|
||||
return { id: cmd.id, ok: true, data: { closed: tabId } };
|
||||
}
|
||||
case "select": {
|
||||
if (cmd.index === void 0 && cmd.tabId === void 0)
|
||||
return { id: cmd.id, ok: false, error: "Missing index or tabId" };
|
||||
if (cmd.tabId !== void 0) {
|
||||
const session = automationSessions.get(workspace);
|
||||
let tab;
|
||||
try {
|
||||
tab = await chrome.tabs.get(cmd.tabId);
|
||||
} catch {
|
||||
return { id: cmd.id, ok: false, error: `Tab ${cmd.tabId} no longer exists` };
|
||||
}
|
||||
if (!session || tab.windowId !== session.windowId) {
|
||||
return { id: cmd.id, ok: false, error: `Tab ${cmd.tabId} is not in the automation window` };
|
||||
}
|
||||
await chrome.tabs.update(cmd.tabId, { active: true });
|
||||
return { id: cmd.id, ok: true, data: { selected: cmd.tabId } };
|
||||
}
|
||||
const tabs = await listAutomationWebTabs(workspace);
|
||||
const target = tabs[cmd.index];
|
||||
if (!target?.id) return { id: cmd.id, ok: false, error: `Tab index ${cmd.index} not found` };
|
||||
await chrome.tabs.update(target.id, { active: true });
|
||||
return { id: cmd.id, ok: true, data: { selected: target.id } };
|
||||
}
|
||||
default:
|
||||
return { id: cmd.id, ok: false, error: `Unknown tabs op: ${cmd.op}` };
|
||||
}
|
||||
}
|
||||
async function handleCookies(cmd) {
|
||||
if (!cmd.domain && !cmd.url) {
|
||||
return { id: cmd.id, ok: false, error: "Cookie scope required: provide domain or url to avoid dumping all cookies" };
|
||||
}
|
||||
const details = {};
|
||||
if (cmd.domain) details.domain = cmd.domain;
|
||||
if (cmd.url) details.url = cmd.url;
|
||||
const cookies = await chrome.cookies.getAll(details);
|
||||
const data = cookies.map((c) => ({
|
||||
name: c.name,
|
||||
value: c.value,
|
||||
domain: c.domain,
|
||||
path: c.path,
|
||||
secure: c.secure,
|
||||
httpOnly: c.httpOnly,
|
||||
expirationDate: c.expirationDate
|
||||
}));
|
||||
return { id: cmd.id, ok: true, data };
|
||||
}
|
||||
async function handleScreenshot(cmd, workspace) {
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
const data = await screenshot(tabId, {
|
||||
format: cmd.format,
|
||||
quality: cmd.quality,
|
||||
fullPage: cmd.fullPage
|
||||
});
|
||||
return { id: cmd.id, ok: true, data };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
async function handleCloseWindow(cmd, workspace) {
|
||||
const session = automationSessions.get(workspace);
|
||||
if (session) {
|
||||
try {
|
||||
await chrome.windows.remove(session.windowId);
|
||||
} catch {
|
||||
}
|
||||
if (session.idleTimer) clearTimeout(session.idleTimer);
|
||||
automationSessions.delete(workspace);
|
||||
}
|
||||
return { id: cmd.id, ok: true, data: { closed: true } };
|
||||
}
|
||||
async function handleSessions(cmd) {
|
||||
const now = Date.now();
|
||||
const data = await Promise.all([...automationSessions.entries()].map(async ([workspace, session]) => ({
|
||||
workspace,
|
||||
windowId: session.windowId,
|
||||
tabCount: (await chrome.tabs.query({ windowId: session.windowId })).filter((tab) => isDebuggableUrl(tab.url)).length,
|
||||
idleMsRemaining: Math.max(0, session.idleDeadlineAt - now)
|
||||
})));
|
||||
return { id: cmd.id, ok: true, data };
|
||||
}
|
||||
@@ -1,10 +1,11 @@
|
||||
{
|
||||
"manifest_version": 3,
|
||||
"name": "OpenCLI",
|
||||
"version": "1.5.1",
|
||||
"version": "1.5.5",
|
||||
"description": "Browser automation bridge for the OpenCLI CLI tool. Executes commands in isolated Chrome windows via a local daemon.",
|
||||
"permissions": [
|
||||
"debugger",
|
||||
"scripting",
|
||||
"tabs",
|
||||
"cookies",
|
||||
"activeTab",
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "opencli-extension",
|
||||
"version": "1.5.1",
|
||||
"version": "1.5.5",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "opencli-extension",
|
||||
"version": "1.5.1",
|
||||
"version": "1.5.5",
|
||||
"devDependencies": {
|
||||
"@types/chrome": "^0.0.287",
|
||||
"typescript": "^5.7.0",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "opencli-extension",
|
||||
"version": "1.5.1",
|
||||
"version": "1.5.5",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
||||
@@ -35,8 +35,12 @@ function createChromeMock() {
|
||||
{ id: 3, windowId: 1, url: 'chrome://extensions', title: 'chrome', active: false, status: 'complete' },
|
||||
];
|
||||
|
||||
const query = vi.fn(async (queryInfo: { windowId?: number } = {}) => {
|
||||
return tabs.filter((tab) => queryInfo.windowId === undefined || tab.windowId === queryInfo.windowId);
|
||||
const query = vi.fn(async (queryInfo: { windowId?: number; active?: boolean } = {}) => {
|
||||
return tabs.filter((tab) => {
|
||||
if (queryInfo.windowId !== undefined && tab.windowId !== queryInfo.windowId) return false;
|
||||
if (queryInfo.active !== undefined && !!tab.active !== queryInfo.active) return false;
|
||||
return true;
|
||||
});
|
||||
});
|
||||
const create = vi.fn(async ({ windowId, url, active }: { windowId?: number; url?: string; active?: boolean }) => {
|
||||
const tab: MockTab = {
|
||||
@@ -84,6 +88,8 @@ function createChromeMock() {
|
||||
runtime: {
|
||||
onInstalled: { addListener: vi.fn() } as Listener<() => void>,
|
||||
onStartup: { addListener: vi.fn() } as Listener<() => void>,
|
||||
onMessage: { addListener: vi.fn() } as Listener<(msg: unknown, sender: unknown, sendResponse: (value: unknown) => void) => void>,
|
||||
getManifest: vi.fn(() => ({ version: 'test-version' })),
|
||||
},
|
||||
cookies: {
|
||||
getAll: vi.fn(async () => []),
|
||||
@@ -193,4 +199,42 @@ describe('background tab isolation', () => {
|
||||
expect.objectContaining({ workspace: 'site:zhihu', windowId: 2 }),
|
||||
]));
|
||||
});
|
||||
|
||||
it('keeps site:notebooklm inside its owned automation window instead of rebinding to a user tab', async () => {
|
||||
const { chrome, tabs } = createChromeMock();
|
||||
tabs[0].url = 'https://notebooklm.google.com/';
|
||||
tabs[0].title = 'NotebookLM Home';
|
||||
tabs[1].url = 'https://notebooklm.google.com/notebook/nb-live';
|
||||
tabs[1].title = 'Live Notebook';
|
||||
vi.stubGlobal('chrome', chrome);
|
||||
|
||||
const mod = await import('./background');
|
||||
mod.__test__.setAutomationWindowId('site:notebooklm', 1);
|
||||
|
||||
const tabId = await mod.__test__.resolveTabId(undefined, 'site:notebooklm');
|
||||
|
||||
expect(tabId).toBe(1);
|
||||
expect(mod.__test__.getSession('site:notebooklm')).toEqual(expect.objectContaining({
|
||||
windowId: 1,
|
||||
}));
|
||||
});
|
||||
|
||||
it('idle timeout closes the automation window for site:notebooklm', async () => {
|
||||
const { chrome, tabs } = createChromeMock();
|
||||
tabs[0].url = 'https://notebooklm.google.com/';
|
||||
tabs[0].title = 'NotebookLM Home';
|
||||
tabs[0].active = true;
|
||||
|
||||
vi.useFakeTimers();
|
||||
vi.stubGlobal('chrome', chrome);
|
||||
|
||||
const mod = await import('./background');
|
||||
mod.__test__.setAutomationWindowId('site:notebooklm', 1);
|
||||
|
||||
mod.__test__.resetWindowIdleTimer('site:notebooklm');
|
||||
await vi.advanceTimersByTimeAsync(30001);
|
||||
|
||||
expect(chrome.windows.remove).toHaveBeenCalledWith(1);
|
||||
expect(mod.__test__.getSession('site:notebooklm')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
+108
-16
@@ -6,7 +6,7 @@
|
||||
*/
|
||||
|
||||
import type { Command, Result } from './protocol';
|
||||
import { DAEMON_WS_URL, WS_RECONNECT_BASE_DELAY, WS_RECONNECT_MAX_DELAY } from './protocol';
|
||||
import { DAEMON_WS_URL, DAEMON_PING_URL, WS_RECONNECT_BASE_DELAY, WS_RECONNECT_MAX_DELAY } from './protocol';
|
||||
import * as executor from './cdp';
|
||||
|
||||
let ws: WebSocket | null = null;
|
||||
@@ -34,9 +34,23 @@ console.error = (...args: unknown[]) => { _origError(...args); forwardLog('error
|
||||
|
||||
// ─── WebSocket connection ────────────────────────────────────────────
|
||||
|
||||
function connect(): void {
|
||||
/**
|
||||
* Probe the daemon via its /ping HTTP endpoint before attempting a WebSocket
|
||||
* connection. fetch() failures are silently catchable; new WebSocket() is not
|
||||
* — Chrome logs ERR_CONNECTION_REFUSED to the extension error page before any
|
||||
* JS handler can intercept it. By keeping the probe inside connect() every
|
||||
* call site remains unchanged and the guard can never be accidentally skipped.
|
||||
*/
|
||||
async function connect(): Promise<void> {
|
||||
if (ws?.readyState === WebSocket.OPEN || ws?.readyState === WebSocket.CONNECTING) return;
|
||||
|
||||
try {
|
||||
const res = await fetch(DAEMON_PING_URL, { signal: AbortSignal.timeout(1000) });
|
||||
if (!res.ok) return; // unexpected response — not our daemon
|
||||
} catch {
|
||||
return; // daemon not running — skip WebSocket to avoid console noise
|
||||
}
|
||||
|
||||
try {
|
||||
ws = new WebSocket(DAEMON_WS_URL);
|
||||
} catch {
|
||||
@@ -90,7 +104,7 @@ function scheduleReconnect(): void {
|
||||
const delay = Math.min(WS_RECONNECT_BASE_DELAY * Math.pow(2, reconnectAttempts - 1), WS_RECONNECT_MAX_DELAY);
|
||||
reconnectTimer = setTimeout(() => {
|
||||
reconnectTimer = null;
|
||||
connect();
|
||||
void connect();
|
||||
}, delay);
|
||||
}
|
||||
|
||||
@@ -146,13 +160,14 @@ async function getAutomationWindow(workspace: string): Promise<number> {
|
||||
|
||||
// Create a new window with a data: URI that New Tab Override extensions cannot intercept.
|
||||
// Using about:blank would be hijacked by extensions like "New Tab Override".
|
||||
// Note: Do NOT set `state` parameter here. Chrome 146+ rejects 'normal' as an invalid
|
||||
// state value for windows.create(). The window defaults to 'normal' state anyway.
|
||||
const win = await chrome.windows.create({
|
||||
url: BLANK_PAGE,
|
||||
focused: false,
|
||||
width: 1280,
|
||||
height: 900,
|
||||
type: 'normal',
|
||||
state: 'minimized',
|
||||
});
|
||||
const session: AutomationSession = {
|
||||
windowId: win.id!,
|
||||
@@ -187,7 +202,7 @@ function initialize(): void {
|
||||
initialized = true;
|
||||
chrome.alarms.create('keepalive', { periodInMinutes: 0.4 }); // ~24 seconds
|
||||
executor.registerListeners();
|
||||
connect();
|
||||
void connect();
|
||||
console.log('[opencli] OpenCLI extension initialized');
|
||||
}
|
||||
|
||||
@@ -200,7 +215,7 @@ chrome.runtime.onStartup.addListener(() => {
|
||||
});
|
||||
|
||||
chrome.alarms.onAlarm.addListener((alarm) => {
|
||||
if (alarm.name === 'keepalive') connect();
|
||||
if (alarm.name === 'keepalive') void connect();
|
||||
});
|
||||
|
||||
// ─── Popup status API ───────────────────────────────────────────────
|
||||
@@ -235,8 +250,12 @@ async function handleCommand(cmd: Command): Promise<Result> {
|
||||
return await handleScreenshot(cmd, workspace);
|
||||
case 'close-window':
|
||||
return await handleCloseWindow(cmd, workspace);
|
||||
case 'cdp':
|
||||
return await handleCdp(cmd, workspace);
|
||||
case 'sessions':
|
||||
return await handleSessions(cmd);
|
||||
case 'set-file-input':
|
||||
return await handleSetFileInput(cmd, workspace);
|
||||
default:
|
||||
return { id: cmd.id, ok: false, error: `Unknown action: ${cmd.action}` };
|
||||
}
|
||||
@@ -252,12 +271,12 @@ async function handleCommand(cmd: Command): Promise<Result> {
|
||||
// ─── Action handlers ─────────────────────────────────────────────────
|
||||
|
||||
/** Internal blank page used when no user URL is provided. */
|
||||
const BLANK_PAGE = 'data:text/html,<html></html>';
|
||||
const BLANK_PAGE = 'about:blank';
|
||||
|
||||
/** Check if a URL can be attached via CDP — only allow http(s) and our internal blank page. */
|
||||
/** Check if a URL can be attached via CDP — only allow http(s) and blank pages. */
|
||||
function isDebuggableUrl(url?: string): boolean {
|
||||
if (!url) return true; // empty/undefined = tab still loading, allow it
|
||||
return url.startsWith('http://') || url.startsWith('https://') || url === BLANK_PAGE;
|
||||
return url.startsWith('http://') || url.startsWith('https://') || url === 'about:blank' || url.startsWith('data:');
|
||||
}
|
||||
|
||||
/** Check if a URL is safe for user-facing navigation (http/https only). */
|
||||
@@ -284,6 +303,16 @@ function isTargetUrl(currentUrl: string | undefined, targetUrl: string): boolean
|
||||
return normalizeUrlForComparison(currentUrl) === normalizeUrlForComparison(targetUrl);
|
||||
}
|
||||
|
||||
function setWorkspaceSession(workspace: string, session: Pick<AutomationSession, 'windowId'>): void {
|
||||
const existing = automationSessions.get(workspace);
|
||||
if (existing?.idleTimer) clearTimeout(existing.idleTimer);
|
||||
automationSessions.set(workspace, {
|
||||
...session,
|
||||
idleTimer: null,
|
||||
idleDeadlineAt: Date.now() + WINDOW_IDLE_TIMEOUT,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve target tab in the automation window.
|
||||
* If explicit tabId is given, use that directly.
|
||||
@@ -297,9 +326,10 @@ async function resolveTabId(tabId: number | undefined, workspace: string): Promi
|
||||
try {
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
const session = automationSessions.get(workspace);
|
||||
if (isDebuggableUrl(tab.url) && session && tab.windowId === session.windowId) return tabId;
|
||||
if (session && tab.windowId !== session.windowId) {
|
||||
console.warn(`[opencli] Tab ${tabId} belongs to window ${tab.windowId}, not automation window ${session.windowId}, re-resolving`);
|
||||
const matchesSession = session ? tab.windowId === session.windowId : false;
|
||||
if (isDebuggableUrl(tab.url) && matchesSession) return tabId;
|
||||
if (session && !matchesSession) {
|
||||
console.warn(`[opencli] Tab ${tabId} is not bound to workspace ${workspace}, re-resolving`);
|
||||
} else if (!isDebuggableUrl(tab.url)) {
|
||||
// Tab exists but URL is not debuggable — fall through to auto-resolve
|
||||
console.warn(`[opencli] Tab ${tabId} URL is not debuggable (${tab.url}), re-resolving`);
|
||||
@@ -359,7 +389,8 @@ async function handleExec(cmd: Command, workspace: string): Promise<Result> {
|
||||
if (!cmd.code) return { id: cmd.id, ok: false, error: 'Missing code' };
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
const data = await executor.evaluateAsync(tabId, cmd.code);
|
||||
const aggressive = workspace.startsWith('operate:');
|
||||
const data = await executor.evaluateAsync(tabId, cmd.code, aggressive);
|
||||
return { id: cmd.id, ok: true, data };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
@@ -550,6 +581,50 @@ async function handleScreenshot(cmd: Command, workspace: string): Promise<Result
|
||||
}
|
||||
}
|
||||
|
||||
/** CDP methods permitted via the 'cdp' passthrough action. */
|
||||
const CDP_ALLOWLIST = new Set([
|
||||
// Agent DOM context
|
||||
'Accessibility.getFullAXTree',
|
||||
'DOM.getDocument',
|
||||
'DOM.getBoxModel',
|
||||
'DOM.getContentQuads',
|
||||
'DOM.querySelectorAll',
|
||||
'DOM.scrollIntoViewIfNeeded',
|
||||
'DOMSnapshot.captureSnapshot',
|
||||
// Native input events
|
||||
'Input.dispatchMouseEvent',
|
||||
'Input.dispatchKeyEvent',
|
||||
'Input.insertText',
|
||||
// Page metrics & screenshots
|
||||
'Page.getLayoutMetrics',
|
||||
'Page.captureScreenshot',
|
||||
// Runtime.enable needed for CDP attach setup (Runtime.evaluate goes through 'exec' action)
|
||||
'Runtime.enable',
|
||||
// Emulation (used by screenshot full-page)
|
||||
'Emulation.setDeviceMetricsOverride',
|
||||
'Emulation.clearDeviceMetricsOverride',
|
||||
]);
|
||||
|
||||
async function handleCdp(cmd: Command, workspace: string): Promise<Result> {
|
||||
if (!cmd.cdpMethod) return { id: cmd.id, ok: false, error: 'Missing cdpMethod' };
|
||||
if (!CDP_ALLOWLIST.has(cmd.cdpMethod)) {
|
||||
return { id: cmd.id, ok: false, error: `CDP method not permitted: ${cmd.cdpMethod}` };
|
||||
}
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
const aggressive = workspace.startsWith('operate:');
|
||||
await executor.ensureAttached(tabId, aggressive);
|
||||
const data = await chrome.debugger.sendCommand(
|
||||
{ tabId },
|
||||
cmd.cdpMethod,
|
||||
cmd.cdpParams ?? {},
|
||||
);
|
||||
return { id: cmd.id, ok: true, data };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
async function handleCloseWindow(cmd: Command, workspace: string): Promise<Result> {
|
||||
const session = automationSessions.get(workspace);
|
||||
if (session) {
|
||||
@@ -564,6 +639,19 @@ async function handleCloseWindow(cmd: Command, workspace: string): Promise<Resul
|
||||
return { id: cmd.id, ok: true, data: { closed: true } };
|
||||
}
|
||||
|
||||
async function handleSetFileInput(cmd: Command, workspace: string): Promise<Result> {
|
||||
if (!cmd.files || !Array.isArray(cmd.files) || cmd.files.length === 0) {
|
||||
return { id: cmd.id, ok: false, error: 'Missing or empty files array' };
|
||||
}
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
await executor.setFileInputFiles(tabId, cmd.files, cmd.selector);
|
||||
return { id: cmd.id, ok: true, data: { count: cmd.files.length } };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
async function handleSessions(cmd: Command): Promise<Result> {
|
||||
const now = Date.now();
|
||||
const data = await Promise.all([...automationSessions.entries()].map(async ([workspace, session]) => ({
|
||||
@@ -580,6 +668,9 @@ export const __test__ = {
|
||||
isTargetUrl,
|
||||
handleTabs,
|
||||
handleSessions,
|
||||
resolveTabId,
|
||||
resetWindowIdleTimer,
|
||||
getSession: (workspace: string = 'default') => automationSessions.get(workspace) ?? null,
|
||||
getAutomationWindowId: (workspace: string = 'default') => automationSessions.get(workspace)?.windowId ?? null,
|
||||
setAutomationWindowId: (workspace: string, windowId: number | null) => {
|
||||
if (windowId === null) {
|
||||
@@ -588,10 +679,11 @@ export const __test__ = {
|
||||
automationSessions.delete(workspace);
|
||||
return;
|
||||
}
|
||||
automationSessions.set(workspace, {
|
||||
setWorkspaceSession(workspace, {
|
||||
windowId,
|
||||
idleTimer: null,
|
||||
idleDeadlineAt: Date.now() + WINDOW_IDLE_TIMEOUT,
|
||||
});
|
||||
},
|
||||
setSession: (workspace: string, session: { windowId: number }) => {
|
||||
setWorkspaceSession(workspace, session);
|
||||
},
|
||||
};
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
function createChromeMock() {
|
||||
const tabs = {
|
||||
get: vi.fn(async (_tabId: number) => ({
|
||||
id: 1,
|
||||
windowId: 1,
|
||||
url: 'https://x.com/home',
|
||||
})),
|
||||
onRemoved: { addListener: vi.fn() },
|
||||
onUpdated: { addListener: vi.fn() },
|
||||
};
|
||||
|
||||
const debuggerApi = {
|
||||
attach: vi.fn(async () => {}),
|
||||
detach: vi.fn(async () => {}),
|
||||
sendCommand: vi.fn(async (_target: unknown, method: string) => {
|
||||
if (method === 'Runtime.evaluate') return { result: { value: 'ok' } };
|
||||
return {};
|
||||
}),
|
||||
onDetach: { addListener: vi.fn() },
|
||||
};
|
||||
|
||||
const scripting = {
|
||||
executeScript: vi.fn(async () => [{ result: { removed: 1 } }]),
|
||||
};
|
||||
|
||||
return {
|
||||
chrome: {
|
||||
tabs,
|
||||
debugger: debuggerApi,
|
||||
scripting,
|
||||
runtime: { id: 'opencli-test' },
|
||||
},
|
||||
debuggerApi,
|
||||
scripting,
|
||||
};
|
||||
}
|
||||
|
||||
describe('cdp attach recovery', () => {
|
||||
beforeEach(() => {
|
||||
vi.resetModules();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
it('does not mutate the DOM before a successful attach', async () => {
|
||||
const { chrome, debuggerApi, scripting } = createChromeMock();
|
||||
vi.stubGlobal('chrome', chrome);
|
||||
|
||||
const mod = await import('./cdp');
|
||||
const result = await mod.evaluate(1, '1');
|
||||
|
||||
expect(result).toBe('ok');
|
||||
expect(debuggerApi.attach).toHaveBeenCalledTimes(1);
|
||||
expect(scripting.executeScript).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('retries after cleanup when attach fails with a foreign extension error', async () => {
|
||||
const { chrome, debuggerApi, scripting } = createChromeMock();
|
||||
debuggerApi.attach
|
||||
.mockRejectedValueOnce(new Error('Cannot access a chrome-extension:// URL of different extension'))
|
||||
.mockResolvedValueOnce(undefined);
|
||||
vi.stubGlobal('chrome', chrome);
|
||||
|
||||
const mod = await import('./cdp');
|
||||
const result = await mod.evaluate(1, '1');
|
||||
|
||||
expect(result).toBe('ok');
|
||||
expect(scripting.executeScript).toHaveBeenCalledTimes(1);
|
||||
expect(debuggerApi.attach).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
+119
-38
@@ -8,16 +8,13 @@
|
||||
|
||||
const attached = new Set<number>();
|
||||
|
||||
/** Internal blank page used when no user URL is provided. */
|
||||
const BLANK_PAGE = 'data:text/html,<html></html>';
|
||||
|
||||
/** Check if a URL can be attached via CDP — only allow http(s) and our internal blank page. */
|
||||
/** Check if a URL can be attached via CDP — only allow http(s) and blank pages. */
|
||||
function isDebuggableUrl(url?: string): boolean {
|
||||
if (!url) return true; // empty/undefined = tab still loading, allow it
|
||||
return url.startsWith('http://') || url.startsWith('https://') || url === BLANK_PAGE;
|
||||
return url.startsWith('http://') || url.startsWith('https://') || url === 'about:blank' || url.startsWith('data:');
|
||||
}
|
||||
|
||||
async function ensureAttached(tabId: number): Promise<void> {
|
||||
export async function ensureAttached(tabId: number, aggressiveRetry: boolean = false): Promise<void> {
|
||||
// Verify the tab URL is debuggable before attempting attach
|
||||
try {
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
@@ -46,23 +43,46 @@ async function ensureAttached(tabId: number): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
await chrome.debugger.attach({ tabId }, '1.3');
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
const hint = msg.includes('chrome-extension://')
|
||||
// Retry attach up to 3 times — other extensions (1Password, Playwright MCP Bridge)
|
||||
// can temporarily interfere with chrome.debugger. A short delay usually resolves it.
|
||||
// Normal commands: 2 retries, 500ms delay (fast fail for non-operate use)
|
||||
// Operate commands: 5 retries, 1500ms delay (aggressive, tolerates extension interference)
|
||||
const MAX_ATTACH_RETRIES = aggressiveRetry ? 5 : 2;
|
||||
const RETRY_DELAY_MS = aggressiveRetry ? 1500 : 500;
|
||||
let lastError = '';
|
||||
|
||||
for (let attempt = 1; attempt <= MAX_ATTACH_RETRIES; attempt++) {
|
||||
try {
|
||||
// Force detach first to clear any stale state from other extensions
|
||||
try { await chrome.debugger.detach({ tabId }); } catch { /* ignore */ }
|
||||
await chrome.debugger.attach({ tabId }, '1.3');
|
||||
lastError = '';
|
||||
break; // Success
|
||||
} catch (e: unknown) {
|
||||
lastError = e instanceof Error ? e.message : String(e);
|
||||
if (attempt < MAX_ATTACH_RETRIES) {
|
||||
console.warn(`[opencli] attach attempt ${attempt}/${MAX_ATTACH_RETRIES} failed: ${lastError}, retrying in ${RETRY_DELAY_MS}ms...`);
|
||||
await new Promise(resolve => setTimeout(resolve, RETRY_DELAY_MS));
|
||||
// Re-verify tab URL before retrying (it may have changed)
|
||||
try {
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
if (!isDebuggableUrl(tab.url)) {
|
||||
lastError = `Tab URL changed to ${tab.url} during retry`;
|
||||
break; // Don't retry if URL became un-debuggable
|
||||
}
|
||||
} catch {
|
||||
lastError = `Tab ${tabId} no longer exists`;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (lastError) {
|
||||
const hint = lastError.includes('chrome-extension://')
|
||||
? '. Tip: another Chrome extension may be interfering — try disabling other extensions'
|
||||
: '';
|
||||
if (msg.includes('Another debugger is already attached')) {
|
||||
try { await chrome.debugger.detach({ tabId }); } catch { /* ignore */ }
|
||||
try {
|
||||
await chrome.debugger.attach({ tabId }, '1.3');
|
||||
} catch {
|
||||
throw new Error(`attach failed: ${msg}${hint}`);
|
||||
}
|
||||
} else {
|
||||
throw new Error(`attach failed: ${msg}${hint}`);
|
||||
}
|
||||
throw new Error(`attach failed: ${lastError}${hint}`);
|
||||
}
|
||||
attached.add(tabId);
|
||||
|
||||
@@ -73,26 +93,45 @@ async function ensureAttached(tabId: number): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
export async function evaluate(tabId: number, expression: string): Promise<unknown> {
|
||||
await ensureAttached(tabId);
|
||||
export async function evaluate(tabId: number, expression: string, aggressiveRetry: boolean = false): Promise<unknown> {
|
||||
// Retry the entire evaluate (attach + command).
|
||||
// Normal: 2 retries. Operate: 3 retries (tolerates extension interference).
|
||||
const MAX_EVAL_RETRIES = aggressiveRetry ? 3 : 2;
|
||||
for (let attempt = 1; attempt <= MAX_EVAL_RETRIES; attempt++) {
|
||||
try {
|
||||
await ensureAttached(tabId, aggressiveRetry);
|
||||
|
||||
const result = await chrome.debugger.sendCommand({ tabId }, 'Runtime.evaluate', {
|
||||
expression,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
}) as {
|
||||
result?: { type: string; value?: unknown; description?: string; subtype?: string };
|
||||
exceptionDetails?: { exception?: { description?: string }; text?: string };
|
||||
};
|
||||
const result = await chrome.debugger.sendCommand({ tabId }, 'Runtime.evaluate', {
|
||||
expression,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
}) as {
|
||||
result?: { type: string; value?: unknown; description?: string; subtype?: string };
|
||||
exceptionDetails?: { exception?: { description?: string }; text?: string };
|
||||
};
|
||||
|
||||
if (result.exceptionDetails) {
|
||||
const errMsg = result.exceptionDetails.exception?.description
|
||||
|| result.exceptionDetails.text
|
||||
|| 'Eval error';
|
||||
throw new Error(errMsg);
|
||||
if (result.exceptionDetails) {
|
||||
const errMsg = result.exceptionDetails.exception?.description
|
||||
|| result.exceptionDetails.text
|
||||
|| 'Eval error';
|
||||
throw new Error(errMsg);
|
||||
}
|
||||
|
||||
return result.result?.value;
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
// Only retry on attach/debugger errors, not on JS eval errors
|
||||
const isAttachError = msg.includes('attach failed') || msg.includes('Debugger is not attached')
|
||||
|| msg.includes('chrome-extension://') || msg.includes('Target closed');
|
||||
if (isAttachError && attempt < MAX_EVAL_RETRIES) {
|
||||
attached.delete(tabId); // Force re-attach on next attempt
|
||||
await new Promise(resolve => setTimeout(resolve, 1000));
|
||||
continue;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
return result.result?.value;
|
||||
throw new Error('evaluate: max retries exhausted');
|
||||
}
|
||||
|
||||
export const evaluateAsync = evaluate;
|
||||
@@ -147,6 +186,48 @@ export async function screenshot(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Set local file paths on a file input element via CDP DOM.setFileInputFiles.
|
||||
* This bypasses the need to send large base64 payloads through the message channel —
|
||||
* Chrome reads the files directly from the local filesystem.
|
||||
*
|
||||
* @param tabId - Target tab ID
|
||||
* @param files - Array of absolute local file paths
|
||||
* @param selector - CSS selector to find the file input (optional, defaults to first file input)
|
||||
*/
|
||||
export async function setFileInputFiles(
|
||||
tabId: number,
|
||||
files: string[],
|
||||
selector?: string,
|
||||
): Promise<void> {
|
||||
await ensureAttached(tabId);
|
||||
|
||||
// Enable DOM domain (required for DOM.querySelector and DOM.setFileInputFiles)
|
||||
await chrome.debugger.sendCommand({ tabId }, 'DOM.enable');
|
||||
|
||||
// Get the document root
|
||||
const doc = await chrome.debugger.sendCommand({ tabId }, 'DOM.getDocument') as {
|
||||
root: { nodeId: number };
|
||||
};
|
||||
|
||||
// Find the file input element
|
||||
const query = selector || 'input[type="file"]';
|
||||
const result = await chrome.debugger.sendCommand({ tabId }, 'DOM.querySelector', {
|
||||
nodeId: doc.root.nodeId,
|
||||
selector: query,
|
||||
}) as { nodeId: number };
|
||||
|
||||
if (!result.nodeId) {
|
||||
throw new Error(`No element found matching selector: ${query}`);
|
||||
}
|
||||
|
||||
// Set files directly via CDP — Chrome reads from local filesystem
|
||||
await chrome.debugger.sendCommand({ tabId }, 'DOM.setFileInputFiles', {
|
||||
files,
|
||||
nodeId: result.nodeId,
|
||||
});
|
||||
}
|
||||
|
||||
export async function detach(tabId: number): Promise<void> {
|
||||
if (!attached.has(tabId)) return;
|
||||
attached.delete(tabId);
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
* Everything else is just JS code sent via 'exec'.
|
||||
*/
|
||||
|
||||
export type Action = 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions';
|
||||
export type Action = 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions' | 'set-file-input' | 'cdp';
|
||||
|
||||
export interface Command {
|
||||
/** Unique request ID */
|
||||
@@ -32,6 +32,14 @@ export interface Command {
|
||||
quality?: number;
|
||||
/** Whether to capture full page (not just viewport) */
|
||||
fullPage?: boolean;
|
||||
/** Local file paths for set-file-input action */
|
||||
files?: string[];
|
||||
/** CSS selector for file input element (set-file-input action) */
|
||||
selector?: string;
|
||||
/** CDP method name for 'cdp' action (e.g. 'Accessibility.getFullAXTree') */
|
||||
cdpMethod?: string;
|
||||
/** CDP method params for 'cdp' action */
|
||||
cdpParams?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface Result {
|
||||
@@ -49,11 +57,10 @@ export interface Result {
|
||||
export const DAEMON_PORT = 19825;
|
||||
export const DAEMON_HOST = 'localhost';
|
||||
export const DAEMON_WS_URL = `ws://${DAEMON_HOST}:${DAEMON_PORT}/ext`;
|
||||
export const DAEMON_HTTP_URL = `http://${DAEMON_HOST}:${DAEMON_PORT}`;
|
||||
/** Lightweight health-check endpoint — probed before each WebSocket attempt. */
|
||||
export const DAEMON_PING_URL = `http://${DAEMON_HOST}:${DAEMON_PORT}/ping`;
|
||||
|
||||
/** Base reconnect delay for extension WebSocket (ms) */
|
||||
export const WS_RECONNECT_BASE_DELAY = 2000;
|
||||
/** Max reconnect delay (ms) */
|
||||
export const WS_RECONNECT_MAX_DELAY = 60000;
|
||||
/** Idle timeout before daemon auto-exits (ms) */
|
||||
export const DAEMON_IDLE_TIMEOUT = 5 * 60 * 1000;
|
||||
/** Max reconnect delay (ms) — kept short since daemon is long-lived */
|
||||
export const WS_RECONNECT_MAX_DELAY = 5000;
|
||||
|
||||
Generated
+22
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@jackwener/opencli",
|
||||
"version": "1.5.1",
|
||||
"version": "1.6.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@jackwener/opencli",
|
||||
"version": "1.5.1",
|
||||
"version": "1.6.1",
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
@@ -15,6 +15,7 @@
|
||||
"commander": "^14.0.3",
|
||||
"js-yaml": "^4.1.0",
|
||||
"turndown": "^7.2.2",
|
||||
"undici": "^7.24.6",
|
||||
"ws": "^8.18.0"
|
||||
},
|
||||
"bin": {
|
||||
@@ -195,6 +196,7 @@
|
||||
"integrity": "sha512-y1IOpG6OSmTpGg/CT0YBb/EAhR2nsC18QWp9Jy8HO9iGySpcwaTvs5kHa17daP3BMTwWyaX9/1tDTDQshZzXdg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@algolia/client-common": "5.49.2",
|
||||
"@algolia/requester-browser-xhr": "5.49.2",
|
||||
@@ -2162,6 +2164,7 @@
|
||||
"integrity": "sha512-1K0wtDaRONwfhL4h8bbJ9qTjmY6rhGgRvvagXkMBsAOMNr+3Q2SffHECh9DIuNVrMA1JwA0zCwhyepgBZVakng==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@algolia/abtesting": "1.15.2",
|
||||
"@algolia/client-abtesting": "5.49.2",
|
||||
@@ -2490,6 +2493,7 @@
|
||||
"integrity": "sha512-/yNdlIkpWbM0ptxno3ONTuf+2g318kh2ez3KSeZN5dZ8YC6AAmgeWz+GasYYiBJPFaYcSAPeu4GfhUaChzIJXA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"tabbable": "^6.4.0"
|
||||
}
|
||||
@@ -2618,6 +2622,7 @@
|
||||
"integrity": "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ==",
|
||||
"dev": true,
|
||||
"license": "MPL-2.0",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"detect-libc": "^2.0.3"
|
||||
},
|
||||
@@ -3089,6 +3094,7 @@
|
||||
"integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
@@ -3477,6 +3483,7 @@
|
||||
"integrity": "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"esbuild": "~0.27.0",
|
||||
"get-tsconfig": "^4.7.5"
|
||||
@@ -3506,6 +3513,7 @@
|
||||
"integrity": "sha512-bGdAIrZ0wiGDo5l8c++HWtbaNCWTS4UTv7RaTH/ThVIgjkveJt83m74bBHMJkuCbslY8ixgLBVZJIOiQlQTjfQ==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
@@ -3514,6 +3522,15 @@
|
||||
"node": ">=14.17"
|
||||
}
|
||||
},
|
||||
"node_modules/undici": {
|
||||
"version": "7.24.6",
|
||||
"resolved": "https://registry.npmjs.org/undici/-/undici-7.24.6.tgz",
|
||||
"integrity": "sha512-Xi4agocCbRzt0yYMZGMA6ApD7gvtUFaxm4ZmeacWI4cZxaF6C+8I8QfofC20NAePiB/IcvZmzkJ7XPa471AEtA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=20.18.1"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "6.21.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
|
||||
@@ -3630,6 +3647,7 @@
|
||||
"integrity": "sha512-1gFhNi+bHhRE/qKZOJXACm6tX4bA3Isy9KuKF15AgSRuRazNBOJfdDemPBU16/mpMxApDPrWvZ08DcLPEoRnuA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.3",
|
||||
@@ -4194,6 +4212,7 @@
|
||||
"integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"esbuild": "^0.21.3",
|
||||
"postcss": "^8.4.43",
|
||||
@@ -4336,6 +4355,7 @@
|
||||
"integrity": "sha512-hTHLc6VNZyzzEH/l7PFGjpcTvUgiaPK5mdLkbjrTeWSRcEfxFrv56g/XckIYlE9ckuobsdwqd5mk2g1sBkMewg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@vue/compiler-dom": "3.5.30",
|
||||
"@vue/compiler-sfc": "3.5.30",
|
||||
|
||||
+2
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@jackwener/opencli",
|
||||
"version": "1.5.1",
|
||||
"version": "1.6.1",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
@@ -58,6 +58,7 @@
|
||||
"commander": "^14.0.3",
|
||||
"js-yaml": "^4.1.0",
|
||||
"turndown": "^7.2.2",
|
||||
"undici": "^7.24.6",
|
||||
"ws": "^8.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
@@ -196,6 +196,22 @@ function main() {
|
||||
}
|
||||
}
|
||||
|
||||
// ── Spotify credentials template ────────────────────────────────────
|
||||
const opencliDir = join(home, '.opencli');
|
||||
const spotifyEnvFile = join(opencliDir, 'spotify.env');
|
||||
ensureDir(opencliDir);
|
||||
if (!existsSync(spotifyEnvFile)) {
|
||||
writeFileSync(spotifyEnvFile,
|
||||
`# Spotify credentials — get them at https://developer.spotify.com/dashboard\n` +
|
||||
`# Add http://127.0.0.1:8888/callback as a Redirect URI in your Spotify app\n` +
|
||||
`SPOTIFY_CLIENT_ID=your_spotify_client_id_here\n` +
|
||||
`SPOTIFY_CLIENT_SECRET=your_spotify_client_secret_here\n`,
|
||||
'utf8'
|
||||
);
|
||||
console.log(`✓ Spotify credentials template created at ${spotifyEnvFile}`);
|
||||
console.log(` Edit the file and add your Client ID and Secret, then run: opencli spotify auth`);
|
||||
}
|
||||
|
||||
// ── Browser Bridge setup hint ───────────────────────────────────────
|
||||
console.log('');
|
||||
console.log(' \x1b[1mNext step — Browser Bridge setup\x1b[0m');
|
||||
|
||||
@@ -0,0 +1,853 @@
|
||||
---
|
||||
name: opencli-explorer
|
||||
description: Use when creating a new OpenCLI adapter from scratch, adding support for a new website or platform, or exploring a site's API endpoints via browser DevTools. Covers API discovery workflow, authentication strategy selection, YAML/TS adapter writing, and testing.
|
||||
tags: [opencli, adapter, browser, api-discovery, cli, web-scraping, automation]
|
||||
---
|
||||
|
||||
# CLI-EXPLORER — 适配器探索式开发完全指南
|
||||
|
||||
> 本文档教你(或 AI Agent)如何为 OpenCLI 添加一个新网站的命令。
|
||||
> 从零到发布,覆盖 API 发现、方案选择、适配器编写、测试验证全流程。
|
||||
|
||||
> [!TIP]
|
||||
> **只想为一个具体页面快速生成一个命令?** 看 [CLI-ONESHOT.md](./CLI-ONESHOT.md)(~150 行,4 步搞定)。
|
||||
> 本文档适合从零探索一个新站点的完整流程。
|
||||
|
||||
---
|
||||
|
||||
## AI Agent 开发者必读:用浏览器探索
|
||||
|
||||
> [!CAUTION]
|
||||
> **你(AI Agent)必须通过浏览器打开目标网站去探索!**
|
||||
> 不要只靠 `opencli explore` 命令或静态分析来发现 API。
|
||||
> 你拥有浏览器工具,必须主动用它们浏览网页、观察网络请求、模拟用户交互。
|
||||
|
||||
### 为什么?
|
||||
|
||||
很多 API 是**懒加载**的(用户必须点击某个按钮/标签才会触发网络请求)。字幕、评论、关注列表等深层数据不会在页面首次加载时出现在 Network 面板中。**如果你不主动去浏览和交互页面,你永远发现不了这些 API。**
|
||||
|
||||
### AI Agent 探索工作流(必须遵循)
|
||||
|
||||
| 步骤 | 工具 | 做什么 |
|
||||
|------|------|--------|
|
||||
| 0. 打开浏览器 | `browser_navigate` | 导航到目标页面 |
|
||||
| 1. 观察页面 | `browser_snapshot` | 观察可交互元素(按钮/标签/链接) |
|
||||
| 2. 首次抓包 | `browser_network_requests` | 筛选 JSON API 端点,记录 URL pattern |
|
||||
| 3. 模拟交互 | `browser_click` + `browser_wait_for` | 点击"字幕""评论""关注"等按钮 |
|
||||
| 4. 二次抓包 | `browser_network_requests` | 对比步骤 2,找出新触发的 API |
|
||||
| 5. 验证 API | `browser_evaluate` | `fetch(url, {credentials:'include'})` 测试返回结构 |
|
||||
| 6. 写代码 | — | 基于确认的 API 写适配器 |
|
||||
|
||||
### 常犯错误
|
||||
|
||||
| ❌ 错误做法 | ✅ 正确做法 |
|
||||
|------------|------------|
|
||||
| 只用 `opencli explore` 命令,等结果自动出来 | 用浏览器工具打开页面,主动浏览 |
|
||||
| 直接在代码里 `fetch(url)`,不看浏览器实际请求 | 先在浏览器中确认 API 可用,再写代码 |
|
||||
| 页面打开后直接抓包,期望所有 API 都出现 | 模拟点击交互(展开评论/切换标签/加载更多) |
|
||||
| 遇到 HTTP 200 但空数据就放弃 | 检查是否需要 Wbi 签名或 Cookie 鉴权 |
|
||||
| 完全依赖 `__INITIAL_STATE__` 拿所有数据 | `__INITIAL_STATE__` 只有首屏数据,深层数据要调 API |
|
||||
|
||||
### 实战成功案例:5 分钟实现「关注列表」适配器
|
||||
|
||||
以下是用上述工作流实际发现 Bilibili 关注列表 API 的完整过程:
|
||||
|
||||
```
|
||||
1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
|
||||
2. browser_network_requests → 发现:
|
||||
GET /x/relation/followings?vmid={uid}&pn=1&ps=24 → [200]
|
||||
GET /x/relation/stat?vmid={uid} → [200]
|
||||
3. browser_evaluate → 验证 API:
|
||||
fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
|
||||
→ { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
|
||||
4. 结论:标准 Cookie API,无需 Wbi 签名
|
||||
5. 写 following.ts → 一次构建通过
|
||||
```
|
||||
|
||||
**关键决策点**:
|
||||
- 直接访问 `fans/follow` 页面(不是首页),页面加载就会触发 following API
|
||||
- 看到 URL 里没有 `/wbi/` → 不需要签名 → 直接用 `fetchJson` 而非 `apiGet`
|
||||
- API 返回 `code: 0` + 非空 `list` → Tier 2 Cookie 策略确认
|
||||
|
||||
---
|
||||
|
||||
## 核心流程
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌────────┐
|
||||
│ 1. 发现 API │ ──▶ │ 2. 选择策略 │ ──▶ │ 3. 写适配器 │ ──▶ │ 4. 测试 │
|
||||
└─────────────┘ └─────────────┘ └──────────────┘ └────────┘
|
||||
explore cascade YAML / TS run + verify
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1: 发现 API
|
||||
|
||||
### 1a. 自动化发现(推荐)
|
||||
|
||||
OpenCLI 内置 Deep Explore,自动分析网站网络请求:
|
||||
|
||||
```bash
|
||||
opencli explore https://www.example.com --site mysite
|
||||
```
|
||||
|
||||
输出到 `.opencli/explore/mysite/`:
|
||||
|
||||
| 文件 | 内容 |
|
||||
|------|------|
|
||||
| `manifest.json` | 站点元数据、框架检测(Vue2/3、React、Next.js、Pinia、Vuex) |
|
||||
| `endpoints.json` | 已发现的 API 端点,按评分排序,含 URL pattern、方法、响应类型 |
|
||||
| `capabilities.json` | 推理出的功能(`hot`、`search`、`feed`…),含置信度和推荐参数 |
|
||||
| `auth.json` | 认证方式检测(Cookie/Header/无认证),策略候选列表 |
|
||||
|
||||
### 1b. 手动抓包验证
|
||||
|
||||
Explore 的自动分析可能不完美,用 verbose 模式手动确认:
|
||||
|
||||
```bash
|
||||
# 在浏览器中打开目标页面,观察网络请求
|
||||
opencli explore https://www.example.com --site mysite -v
|
||||
|
||||
# 或直接用 evaluate 测试 API
|
||||
opencli bilibili hot -v # 查看已有命令的 pipeline 每步数据流
|
||||
```
|
||||
|
||||
关注抓包结果中的关键信息:
|
||||
- **URL pattern**: `/api/v2/hot?limit=20` → 这就是你要调用的端点
|
||||
- **Method**: `GET` / `POST`
|
||||
- **Request Headers**: Cookie? Bearer? 自定义签名头(X-s、X-t)?
|
||||
- **Response Body**: JSON 结构,特别是数据在哪个路径(`data.items`、`data.list`)
|
||||
|
||||
### 1c. 高阶 API 发现捷径法则 (Heuristics)
|
||||
|
||||
在开始死磕复杂的抓包拦截之前,按照以下优先级进行尝试:
|
||||
|
||||
1. **后缀爆破法 (`.json`)**: 像 Reddit 这样复杂的网站,只要在其 URL 后加上 `.json`(例如 `/r/all.json`),就能在带 Cookie 的情况下直接利用 `fetch` 拿到极其干净的 REST 数据(Tier 2 Cookie 策略极速秒杀)。另外如功能完备的**雪球 (xueqiu)** 也可以走这种纯 API 的方式极简获取,成为你构建简单 YAML 的黄金标杆。
|
||||
2. **全局状态查找法 (`__INITIAL_STATE__`)**: 许多服务端渲染 (SSR) 的网站(如小红书、Bilibili)会将首页或详情页的完整数据挂载到全局 window 对象上。与其去拦截网络请求,不如直接 `page.evaluate('() => window.__INITIAL_STATE__')` 获取整个数据树。
|
||||
3. **主动交互触发法 (Active Interaction)**: 很多深层 API(如视频字幕、评论下的回复)是懒加载的。在静态抓包找不到数据时,尝试在 `evaluate` 步骤或手动打断点时,主动去**点击(Click)页面上的对应按钮**(如"CC"、"展开全部"),从而诱发隐藏的 Network Fetch。
|
||||
4. **框架探测与 Store Action 截断**: 如果站点使用 Vue + Pinia,可以使用 `tap` 步骤调用 action,让前端框架代替你完成复杂的鉴权签名封装。
|
||||
5. **底层 XHR/Fetch 拦截**: 最后手段,当上述都不行时,使用 TypeScript 适配器进行无侵入式的请求抓取。
|
||||
|
||||
### 1d. 框架检测
|
||||
|
||||
Explore 自动检测前端框架。如果需要手动确认:
|
||||
|
||||
```bash
|
||||
# 在已打开目标网站的情况下
|
||||
opencli evaluate "(()=>{
|
||||
const vue3 = !!document.querySelector('#app')?.__vue_app__;
|
||||
const vue2 = !!document.querySelector('#app')?.__vue__;
|
||||
const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
|
||||
const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
|
||||
return JSON.stringify({vue3, vue2, react, pinia});
|
||||
})()"
|
||||
```
|
||||
|
||||
Vue + Pinia 的站点(如小红书)可以直接通过 Store Action 绕过签名。
|
||||
|
||||
---
|
||||
|
||||
## Step 2: 选择认证策略
|
||||
|
||||
OpenCLI 提供 5 级认证策略。使用 `cascade` 命令自动探测:
|
||||
|
||||
```bash
|
||||
opencli cascade https://api.example.com/hot
|
||||
```
|
||||
|
||||
### 策略决策树
|
||||
|
||||
```
|
||||
直接 fetch(url) 能拿到数据?
|
||||
→ ✅ Tier 1: public(公开 API,不需要浏览器)
|
||||
→ ❌ fetch(url, {credentials:'include'}) 带 Cookie 能拿到?
|
||||
→ ✅ Tier 2: cookie(最常见,evaluate 步骤内 fetch)
|
||||
→ ❌ → 加上 Bearer / CSRF header 后能拿到?
|
||||
→ ✅ Tier 3: header(如 Twitter ct0 + Bearer)
|
||||
→ ❌ → 网站有 Pinia/Vuex Store?
|
||||
→ ✅ Tier 4: intercept(Store Action + XHR 拦截)
|
||||
→ ❌ Tier 5: ui(UI 自动化,最后手段)
|
||||
```
|
||||
|
||||
### 各策略对比
|
||||
|
||||
| Tier | 策略 | 速度 | 复杂度 | 适用场景 | 实例 |
|
||||
|------|------|------|--------|---------|------|
|
||||
| 1 | `public` | ⚡ ~1s | 最简 | 公开 API,无需登录 | Hacker News, V2EX |
|
||||
| 2 | `cookie` | 🔄 ~7s | 简单 | Cookie 认证即可 | Bilibili, Zhihu, Reddit |
|
||||
| 3 | `header` | 🔄 ~7s | 中等 | 需要 CSRF token 或 Bearer | Twitter GraphQL |
|
||||
| 4 | `intercept` | 🔄 ~10s | 较高 | 请求有复杂签名 | 小红书 (Pinia + XHR) |
|
||||
| 5 | `ui` | 🐌 ~15s+ | 最高 | 无 API,纯 DOM 解析 | 遗留网站 |
|
||||
|
||||
---
|
||||
|
||||
## Step 2.5: 准备工作(写代码之前)
|
||||
|
||||
### 先找模板:从最相似的现有适配器开始
|
||||
|
||||
**不要从零开始写**。先看看同站点已有哪些适配器:
|
||||
|
||||
```bash
|
||||
ls src/clis/<site>/ # 看看已有什么
|
||||
cat src/clis/<site>/feed.ts # 读最相似的那个
|
||||
```
|
||||
|
||||
最高效的方式是 **复制最相似的适配器,然后改 3 个地方**:
|
||||
1. `name` → 新命令名
|
||||
2. API URL → 你在 Step 1 发现的端点
|
||||
3. 字段映射 → 对应新 API 的字段
|
||||
|
||||
### 平台 SDK 速查表
|
||||
|
||||
写 TS 适配器之前,先看看你的目标站点有没有**现成的 helper 函数**可以复用:
|
||||
|
||||
#### Bilibili (`src/clis/bilibili/utils.ts`)
|
||||
|
||||
| 函数 | 用途 | 何时使用 |
|
||||
|------|------|----------|
|
||||
| `fetchJson(page, url)` | 带 Cookie 的 fetch + JSON 解析 | 普通 Cookie-tier API |
|
||||
| `apiGet(page, path, {signed, params})` | 带 Wbi 签名的 API 调用 | URL 含 `/wbi/` 的接口 |
|
||||
| `getSelfUid(page)` | 获取当前登录用户的 UID | "我的xxx" 类命令 |
|
||||
| `resolveUid(page, input)` | 解析用户输入的 UID(支持数字/URL) | `--uid` 参数处理 |
|
||||
| `wbiSign(page, params)` | 底层 Wbi 签名生成 | 通常不直接用,`apiGet` 已封装 |
|
||||
| `stripHtml(s)` | 去除 HTML 标签 | 清理富文本字段 |
|
||||
|
||||
**如何判断需不需要 `apiGet`**?看 Network 请求 URL:
|
||||
- 含 `/wbi/` 或 `w_rid=` → 必须用 `apiGet(..., { signed: true })`
|
||||
- 不含 → 直接用 `fetchJson`
|
||||
|
||||
> 其他站点(Twitter、小红书等)暂无专用 SDK,直接用 `page.evaluate` + `fetch` 即可。
|
||||
|
||||
---
|
||||
|
||||
## Step 3: 编写适配器
|
||||
|
||||
### YAML vs TS?先看决策树
|
||||
|
||||
```
|
||||
你的 pipeline 里有 evaluate 步骤(内嵌 JS 代码)?
|
||||
→ ✅ 用 TypeScript (src/clis/<site>/<name>.ts),保存即自动动态注册
|
||||
→ ❌ 纯声明式(navigate + tap + map + limit)?
|
||||
→ ✅ 用 YAML (src/clis/<site>/<name>.yaml),保存即自动注册
|
||||
```
|
||||
|
||||
| 场景 | 选择 | 示例 |
|
||||
|------|------|------|
|
||||
| 纯 fetch/select/map/limit | YAML | `v2ex/hot.yaml`, `hackernews/top.yaml` |
|
||||
| navigate + evaluate(fetch) + map | YAML(评估复杂度) | `zhihu/hot.yaml` |
|
||||
| navigate + tap + map | YAML ✅ | `xiaohongshu/feed.yaml`, `xiaohongshu/notifications.yaml` |
|
||||
| 有复杂 JS 逻辑(Pinia state 读取、条件分支) | TS | `xiaohongshu/me.ts`, `bilibili/me.ts` |
|
||||
| XHR 拦截 + 签名 | TS | `xiaohongshu/search.ts` |
|
||||
| GraphQL / 分页 / Wbi 签名 | TS | `bilibili/search.ts`, `twitter/search.ts` |
|
||||
|
||||
> **经验法则**:如果你发现 YAML 里嵌了超过 10 行 JS,改用 TS 更可维护。
|
||||
|
||||
### 通用模式:分页 API
|
||||
|
||||
很多 API 使用 `pn`(页码)+ `ps`(每页数量)分页。标准处理模式:
|
||||
|
||||
```typescript
|
||||
args: [
|
||||
{ name: 'page', type: 'int', required: false, default: 1, help: '页码' },
|
||||
{ name: 'limit', type: 'int', required: false, default: 50, help: '每页数量 (最大 50)' },
|
||||
],
|
||||
func: async (page, kwargs) => {
|
||||
const pn = kwargs.page ?? 1;
|
||||
const ps = Math.min(kwargs.limit ?? 50, 50); // 尊重 API 的 ps 上限
|
||||
const payload = await fetchJson(page,
|
||||
`https://api.example.com/list?pn=${pn}&ps=${ps}`
|
||||
);
|
||||
return payload.data?.list || [];
|
||||
},
|
||||
```
|
||||
|
||||
> 大多数站点的 `ps` 上限是 20~50。超过会被静默截断或返回错误。
|
||||
|
||||
### 方式 A: YAML Pipeline(声明式,推荐)
|
||||
|
||||
文件路径: `src/clis/<site>/<name>.yaml`,放入即自动注册。
|
||||
|
||||
#### Tier 1 — 公开 API 模板
|
||||
|
||||
```yaml
|
||||
# src/clis/v2ex/hot.yaml
|
||||
site: v2ex
|
||||
name: hot
|
||||
description: V2EX 热门话题
|
||||
domain: www.v2ex.com
|
||||
strategy: public
|
||||
browser: false
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
pipeline:
|
||||
- fetch:
|
||||
url: https://www.v2ex.com/api/topics/hot.json
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
replies: ${{ item.replies }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, replies]
|
||||
```
|
||||
|
||||
#### Tier 2 — Cookie 认证模板(最常用)
|
||||
|
||||
```yaml
|
||||
# src/clis/zhihu/hot.yaml
|
||||
site: zhihu
|
||||
name: hot
|
||||
description: 知乎热榜
|
||||
domain: www.zhihu.com
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.zhihu.com # 先加载页面建立 session
|
||||
|
||||
- evaluate: | # 在浏览器内发请求,自动带 Cookie
|
||||
(async () => {
|
||||
const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
|
||||
credentials: 'include'
|
||||
});
|
||||
const d = await res.json();
|
||||
return (d?.data || []).map(item => {
|
||||
const t = item.target || {};
|
||||
return {
|
||||
title: t.title,
|
||||
heat: item.detail_text || '',
|
||||
answers: t.answer_count,
|
||||
};
|
||||
});
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
heat: ${{ item.heat }}
|
||||
answers: ${{ item.answers }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, heat, answers]
|
||||
```
|
||||
|
||||
> **关键**: `evaluate` 步骤内的 `fetch` 运行在浏览器页面内,自动携带 `credentials: 'include'`,无需手动处理 Cookie。
|
||||
|
||||
#### 进阶 — 带搜索参数
|
||||
|
||||
```yaml
|
||||
# src/clis/zhihu/search.yaml
|
||||
site: zhihu
|
||||
name: search
|
||||
description: 知乎搜索
|
||||
|
||||
args:
|
||||
query:
|
||||
type: str
|
||||
required: true
|
||||
positional: true
|
||||
description: Search query
|
||||
limit:
|
||||
type: int
|
||||
default: 10
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.zhihu.com
|
||||
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const q = encodeURIComponent('${{ args.query }}');
|
||||
const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${{ args.limit }}', {
|
||||
credentials: 'include'
|
||||
});
|
||||
const d = await res.json();
|
||||
return (d?.data || [])
|
||||
.filter(item => item.type === 'search_result')
|
||||
.map(item => ({
|
||||
title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
|
||||
type: item.object?.type || '',
|
||||
author: item.object?.author?.name || '',
|
||||
votes: item.object?.voteup_count || 0,
|
||||
}));
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
type: ${{ item.type }}
|
||||
author: ${{ item.author }}
|
||||
votes: ${{ item.votes }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, type, author, votes]
|
||||
```
|
||||
|
||||
#### Tier 4 — Store Action Bridge(`tap` 步骤,intercept 策略推荐)
|
||||
|
||||
适用于 Vue + Pinia/Vuex 的网站(如小红书),无须手动写 XHR 拦截代码:
|
||||
|
||||
```yaml
|
||||
# src/clis/xiaohongshu/notifications.yaml
|
||||
site: xiaohongshu
|
||||
name: notifications
|
||||
description: "小红书通知"
|
||||
domain: www.xiaohongshu.com
|
||||
strategy: intercept
|
||||
browser: true
|
||||
|
||||
args:
|
||||
type:
|
||||
type: str
|
||||
default: mentions
|
||||
description: "Notification type: mentions, likes, or connections"
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
columns: [rank, user, action, content, note, time]
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.xiaohongshu.com/notification
|
||||
- wait: 3
|
||||
- tap:
|
||||
store: notification # Pinia store name
|
||||
action: getNotification # Store action to call
|
||||
args: # Action arguments
|
||||
- ${{ args.type | default('mentions') }}
|
||||
capture: /you/ # URL pattern to capture response
|
||||
select: data.message_list # Extract sub-path from response
|
||||
timeout: 8
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
user: ${{ item.user_info.nickname }}
|
||||
action: ${{ item.title }}
|
||||
content: ${{ item.comment_info.content }}
|
||||
- limit: ${{ args.limit | default(20) }}
|
||||
```
|
||||
|
||||
> **`tap` 步骤自动完成**:注入 fetch+XHR 双拦截 → 查找 Pinia/Vuex store → 调用 action → 捕获匹配 URL 的响应 → 清理拦截。
|
||||
> 如果 store 或 action 找不到,会返回 `hint` 列出所有可用的 store actions,方便调试。
|
||||
|
||||
| tap 参数 | 必填 | 说明 |
|
||||
|---------|------|------|
|
||||
| `store` | ✅ | Pinia store 名称(如 `feed`, `search`, `notification`) |
|
||||
| `action` | ✅ | Store action 方法名 |
|
||||
| `capture` | ✅ | URL 子串匹配(匹配网络请求 URL) |
|
||||
| `args` | ❌ | 传给 action 的参数数组 |
|
||||
| `select` | ❌ | 从 captured JSON 中提取的路径(如 `data.items`) |
|
||||
| `timeout` | ❌ | 等待网络响应的超时秒数(默认 5s) |
|
||||
| `framework` | ❌ | `pinia` 或 `vuex`(默认自动检测) |
|
||||
|
||||
### 方式 B: TypeScript 适配器(编程式)
|
||||
|
||||
适用于需要嵌入 JS 代码读取 Pinia state、XHR 拦截、GraphQL、分页、复杂数据转换等场景。
|
||||
|
||||
文件路径: `src/clis/<site>/<name>.ts`。文件将会在运行时被动态扫描并注册(切勿在 `index.ts` 中手动 `import`)。
|
||||
|
||||
#### Tier 3 — Header 认证(Twitter)
|
||||
|
||||
```typescript
|
||||
// src/clis/twitter/search.ts
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'twitter',
|
||||
name: 'search',
|
||||
description: 'Search tweets',
|
||||
strategy: Strategy.HEADER,
|
||||
args: [{ name: 'query', required: true, positional: true }],
|
||||
columns: ['rank', 'author', 'text', 'likes'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://x.com');
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
// 从 Cookie 提取 CSRF token
|
||||
const ct0 = document.cookie.split(';')
|
||||
.map(c => c.trim())
|
||||
.find(c => c.startsWith('ct0='))?.split('=')[1];
|
||||
if (!ct0) return { error: 'Not logged in' };
|
||||
|
||||
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
|
||||
const headers = {
|
||||
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
|
||||
'X-Csrf-Token': ct0,
|
||||
'X-Twitter-Auth-Type': 'OAuth2Session',
|
||||
};
|
||||
|
||||
const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
|
||||
const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
|
||||
const res = await fetch(url, { headers, credentials: 'include' });
|
||||
return await res.json();
|
||||
})()
|
||||
`);
|
||||
// ... 解析 data
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
#### Tier 4 — XHR/Fetch 双重拦截 (Twitter/小红书 通用模式)
|
||||
|
||||
```typescript
|
||||
// src/clis/xiaohongshu/user.ts
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'xiaohongshu',
|
||||
name: 'user',
|
||||
description: '获取用户笔记',
|
||||
strategy: Strategy.INTERCEPT,
|
||||
args: [{ name: 'id', required: true }],
|
||||
columns: ['rank', 'title', 'likes', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
|
||||
await page.wait(5);
|
||||
|
||||
// XHR/Fetch 底层拦截:捕获所有包含 'v1/user/posted' 的请求
|
||||
await page.installInterceptor('v1/user/posted');
|
||||
|
||||
// 触发后端 API:模拟人类用户向底部滚动2次
|
||||
await page.autoScroll({ times: 2, delayMs: 2000 });
|
||||
|
||||
// 提取所有被拦截捕获的 JSON 响应体
|
||||
const requests = await page.getInterceptedRequests();
|
||||
if (!requests || requests.length === 0) return [];
|
||||
|
||||
let results = [];
|
||||
for (const req of requests) {
|
||||
if (req.data?.data?.notes) {
|
||||
for (const note of req.data.data.notes) {
|
||||
results.push({
|
||||
title: note.display_title || '',
|
||||
likes: note.interact_info?.liked_count || '0',
|
||||
url: `https://explore/${note.note_id || note.id}`
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results.slice(0, 20).map((item, i) => ({
|
||||
rank: i + 1, ...item,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
> **拦截核心思路**:不自己构造签名,而是利用 `installInterceptor` 劫持网站自己的 `XMLHttpRequest` 和 `fetch`,让网站发请求,我们直接在底层取出解析好的 `response.json()`。
|
||||
|
||||
> **级联请求**(如 BVID→CID→字幕)的完整模板和要点见下方[进阶模式: 级联请求](#进阶模式-级联请求-cascading-requests)章节。
|
||||
|
||||
---
|
||||
|
||||
## Step 4: 测试
|
||||
|
||||
> **构建通过 ≠ 功能正常**。`npm run build` 只验证 TypeScript / YAML 语法,不验证运行时行为。
|
||||
> 每个新命令 **必须实际运行** 并确认输出正确后才算完成。
|
||||
|
||||
### 必做清单
|
||||
|
||||
```bash
|
||||
# 1. 构建(确认语法无误)
|
||||
npm run build
|
||||
|
||||
# 2. 确认命令已注册
|
||||
opencli list | grep mysite
|
||||
|
||||
# 3. 实际运行命令(最关键!)
|
||||
opencli mysite hot --limit 3 -v # verbose 查看每步数据流
|
||||
opencli mysite hot --limit 3 -f json # JSON 输出确认字段完整
|
||||
```
|
||||
|
||||
### tap 步骤调试(intercept 策略专用)
|
||||
|
||||
> **不要猜 store name / action name**。先用 evaluate 探索,再写 YAML。
|
||||
|
||||
#### Step 1: 列出所有 Pinia store
|
||||
|
||||
在浏览器中打开目标网站后:
|
||||
|
||||
```bash
|
||||
opencli evaluate "(() => {
|
||||
const app = document.querySelector('#app')?.__vue_app__;
|
||||
const pinia = app?.config?.globalProperties?.\$pinia;
|
||||
return [...pinia._s.keys()];
|
||||
})()"
|
||||
# 输出: ["user", "feed", "search", "notification", ...]
|
||||
```
|
||||
|
||||
#### Step 2: 查看 store 的 action 名称
|
||||
|
||||
故意写一个错误 action 名,tap 会返回所有可用 actions:
|
||||
|
||||
```
|
||||
⚠ tap: Action not found: wrongName on store notification
|
||||
💡 Available: getNotification, replyComment, getNotificationCount, reset
|
||||
```
|
||||
|
||||
#### Step 3: 用 network requests 确认 capture 模式
|
||||
|
||||
```bash
|
||||
# 在浏览器打开目标页面,查看网络请求
|
||||
# 找到目标 API 的 URL 特征(如 "/you/mentions"、"homefeed")
|
||||
```
|
||||
|
||||
#### 完整流程
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────┐
|
||||
│ 1. navigate │ ──▶ │ 2. 探索 store │ ──▶ │ 3. 写 YAML │ ──▶ │ 4. 测试 │
|
||||
│ 到目标页面 │ │ name/action │ │ tap 步骤 │ │ 运行验证 │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘ └────────┘
|
||||
```
|
||||
|
||||
### Verbose 模式 & 输出验证
|
||||
|
||||
```bash
|
||||
opencli bilibili hot --limit 1 -v # 查看 pipeline 每步数据流
|
||||
opencli mysite hot -f json | jq '.[0]' # 确认 JSON 可被解析
|
||||
opencli mysite hot -f csv > data.csv # 确认 CSV 可导入
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: 提交发布
|
||||
|
||||
文件放入 `src/clis/<site>/` 即自动注册(YAML 或 TS 无需手动 import),然后:
|
||||
|
||||
```bash
|
||||
opencli list | grep mysite # 确认注册
|
||||
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push
|
||||
```
|
||||
|
||||
> **架构理念**:OpenCLI 内建 **Zero-Dependency jq** 数据流 — 所有解析在 `evaluate` 的原生 JS 内完成,外层 YAML 用 `select`/`map` 提取,无需依赖系统 `jq` 二进制。
|
||||
|
||||
---
|
||||
|
||||
## 进阶模式: 级联请求 (Cascading Requests)
|
||||
|
||||
当目标数据需要多步 API 链式获取时(如 `BVID → CID → 字幕列表 → 字幕内容`),必须使用 **TS 适配器**。YAML 无法处理这种多步逻辑。
|
||||
|
||||
### 模板代码
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { apiGet } from './utils.js'; // 复用平台 SDK
|
||||
|
||||
cli({
|
||||
site: 'bilibili',
|
||||
name: 'subtitle',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [{ name: 'bvid', required: true }],
|
||||
columns: ['index', 'from', 'to', 'content'],
|
||||
func: async (page: IPage | null, kwargs: any) => {
|
||||
if (!page) throw new Error('Requires browser');
|
||||
|
||||
// Step 1: 建立 Session
|
||||
await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);
|
||||
|
||||
// Step 2: 从页面提取中间 ID (__INITIAL_STATE__)
|
||||
const cid = await page.evaluate(`(async () => {
|
||||
return window.__INITIAL_STATE__?.videoData?.cid;
|
||||
})()`);
|
||||
if (!cid) throw new Error('无法提取 CID');
|
||||
|
||||
// Step 3: 用中间 ID 调用下一级 API (自动 Wbi 签名)
|
||||
const payload = await apiGet(page, '/x/player/wbi/v2', {
|
||||
params: { bvid: kwargs.bvid, cid },
|
||||
signed: true, // ← 自动生成 w_rid
|
||||
});
|
||||
|
||||
// Step 4: 检测风控降级 (空值断言)
|
||||
const subtitles = payload.data?.subtitle?.subtitles || [];
|
||||
const url = subtitles[0]?.subtitle_url;
|
||||
if (!url) throw new Error('subtitle_url 为空,疑似风控降级');
|
||||
|
||||
// Step 5: 拉取最终数据 (CDN JSON)
|
||||
const items = await page.evaluate(`(async () => {
|
||||
const res = await fetch(${JSON.stringify('https:' + url)});
|
||||
const json = await res.json();
|
||||
return { data: json.body || json };
|
||||
})()`);
|
||||
|
||||
return items.data.map((item, idx) => ({ ... }));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### 关键要点
|
||||
|
||||
| 步骤 | 注意事项 |
|
||||
|------|----------|
|
||||
| 提取中间 ID | 优先从 `__INITIAL_STATE__` 拿,避免额外 API 调用 |
|
||||
| Wbi 签名 | B 站 `/wbi/` 接口**强制校验** `w_rid`,纯 `fetch` 会被 403 |
|
||||
| 空值断言 | 即使 HTTP 200,核心字段可能为空串(风控降级) |
|
||||
| CDN URL | 常以 `//` 开头,记得补 `https:` |
|
||||
| `JSON.stringify` | 拼接 URL 到 evaluate 时必须用它转义,避免注入 |
|
||||
|
||||
---
|
||||
|
||||
## 常见陷阱
|
||||
|
||||
| 陷阱 | 表现 | 解决方案 |
|
||||
|------|------|---------|
|
||||
| 缺少 `navigate` | evaluate 报 `Target page context` 错误 | 在 evaluate 前加 `navigate:` 步骤 |
|
||||
| 嵌套字段访问 | `${{ item.node?.title }}` 不工作 | 在 evaluate 中 flatten 数据,不在模板中用 optional chaining |
|
||||
| 缺少 `strategy: public` | 公开 API 也启动浏览器,7s → 1s | 公开 API 加上 `strategy: public` + `browser: false` |
|
||||
| evaluate 返回字符串 | map 步骤收到 `""` 而非数组 | pipeline 有 auto-parse,但建议在 evaluate 内 `.map()` 整形 |
|
||||
| 搜索参数被 URL 编码 | `${{ args.query }}` 被浏览器二次编码 | 在 evaluate 内用 `encodeURIComponent()` 手动编码 |
|
||||
| Cookie 过期 | 返回 401 / 空数据 | 在浏览器里重新登录目标站点 |
|
||||
| Extension tab 残留 | Chrome 多出 `chrome-extension://` tab | 已自动清理;若残留,手动关闭即可 |
|
||||
| TS evaluate 格式 | `() => {}` 报 `result is not a function` | TS 中 `page.evaluate()` 必须用 IIFE:`(async () => { ... })()` |
|
||||
| 页面异步加载 | evaluate 拿到空数据(store state 还没更新) | 在 evaluate 内用 polling 等待数据出现,或增加 `wait` 时间 |
|
||||
| YAML 内嵌大段 JS | 调试困难,字符串转义问题 | 超过 10 行 JS 的命令改用 TS adapter |
|
||||
| **风控被拦截(伪200)** | 获取到的 JSON 里核心数据是 `""` (空串) | 极易被误判。必须添加断言!无核心数据立刻要求升级鉴权 Tier 并重新配置 Cookie |
|
||||
| **API 没找见** | `explore` 工具打分出来的都拿不到深层数据 | 点击页面按钮诱发懒加载数据,再结合 `getInterceptedRequests` 获取 |
|
||||
|
||||
---
|
||||
|
||||
## 用 AI Agent 自动生成适配器
|
||||
|
||||
最快的方式是让 AI Agent 完成全流程:
|
||||
|
||||
```bash
|
||||
# 一键:探索 → 分析 → 合成 → 注册
|
||||
opencli generate https://www.example.com --goal "hot"
|
||||
|
||||
# 或分步执行:
|
||||
opencli explore https://www.example.com --site mysite # 发现 API
|
||||
opencli explore https://www.example.com --auto --click "字幕,CC" # 模拟点击触发懒加载 API
|
||||
opencli synthesize mysite # 生成候选 YAML
|
||||
opencli verify mysite/hot --smoke # 冒烟测试
|
||||
```
|
||||
|
||||
生成的候选 YAML 保存在 `.opencli/explore/mysite/candidates/`,可直接复制到 `src/clis/mysite/` 并微调。
|
||||
|
||||
## Record Workflow
|
||||
|
||||
`record` 是为「无法用 `explore` 自动发现」的页面(需要登录操作、复杂交互、SPA 内路由)准备的手动录制方案。
|
||||
|
||||
### 工作原理
|
||||
|
||||
```
|
||||
opencli record <url>
|
||||
→ 打开 automation window 并导航到目标 URL
|
||||
→ 向所有 tab 注入 fetch/XHR 拦截器(幂等,可重复注入)
|
||||
→ 每 2s 轮询一次:发现新 tab 自动注入,drain 所有 tab 的捕获缓冲区
|
||||
→ 超时(默认 60s)或按 Enter 停止
|
||||
→ 分析捕获到的 JSON 请求:去重 → 评分 → 生成候选 YAML
|
||||
```
|
||||
|
||||
**拦截器特性**:
|
||||
- 同时 patch `window.fetch` 和 `XMLHttpRequest`
|
||||
- 只捕获 `Content-Type: application/json` 的响应
|
||||
- 过滤纯对象少于 2 个 key 的响应(避免 tracking/ping)
|
||||
- 跨 tab 隔离:每个 tab 独立缓冲区,轮询时分别 drain
|
||||
- 幂等注入:同一 tab 二次注入时先 restore 原始函数再重新 patch,不丢失已捕获数据
|
||||
|
||||
### 使用步骤
|
||||
|
||||
```bash
|
||||
# 1. 启动录制(建议 --timeout 给足操作时间)
|
||||
opencli record "https://example.com/page" --timeout 120000
|
||||
|
||||
# 2. 在弹出的 automation window 里正常操作页面:
|
||||
# - 打开列表、搜索、点击条目、切换 Tab
|
||||
# - 凡是触发网络请求的操作都会被捕获
|
||||
|
||||
# 3. 完成操作后按 Enter 停止(或等超时自动停止)
|
||||
|
||||
# 4. 查看结果
|
||||
cat .opencli/record/<site>/captured.json # 原始捕获
|
||||
ls .opencli/record/<site>/candidates/ # 候选 YAML
|
||||
```
|
||||
|
||||
### 页面类型与捕获预期
|
||||
|
||||
| 页面类型 | 预期捕获量 | 说明 |
|
||||
|---------|-----------|------|
|
||||
| 列表/搜索页 | 多(5~20+) | 每次搜索/翻页都会触发新请求 |
|
||||
| 详情页(只读) | 少(1~5) | 首屏数据一次性返回,后续操作走 form/redirect |
|
||||
| SPA 内路由跳转 | 中等 | 路由切换会触发新接口,但首屏请求在注入前已发出 |
|
||||
| 需要登录的页面 | 视操作而定 | 确保 Chrome 已登录目标网站 |
|
||||
|
||||
> **注意**:如果页面在导航完成前就发出了大部分请求(服务端渲染 / SSR 注水),拦截器会错过这些请求。
|
||||
> 解决方案:在页面加载完成后,手动触发能产生新请求的操作(搜索、翻页、切 Tab、展开折叠项等)。
|
||||
|
||||
### 候选 YAML → TS CLI 转换
|
||||
|
||||
生成的候选 YAML 是起点,通常需要转换为 TypeScript(尤其是 tae 等内部系统):
|
||||
|
||||
**候选 YAML 结构**(自动生成):
|
||||
```yaml
|
||||
site: tae
|
||||
name: getList # 从 URL path 推断的名称
|
||||
strategy: cookie
|
||||
browser: true
|
||||
pipeline:
|
||||
- navigate: https://...
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const res = await fetch('/approval/getList.json?procInsId=...', { credentials: 'include' });
|
||||
const data = await res.json();
|
||||
return (data?.content?.operatorRecords || []).map(item => ({ ... }));
|
||||
})()
|
||||
```
|
||||
|
||||
**转换为 TS CLI**(参考 `src/clis/tae/add-expense.ts` 风格):
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'tae',
|
||||
name: 'get-approval',
|
||||
description: '查看报销单审批流程和操作记录',
|
||||
domain: 'tae.alibaba-inc.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'proc_ins_id', type: 'string', required: true, positional: true, help: '流程实例 ID(procInsId)' },
|
||||
],
|
||||
columns: ['step', 'operator', 'action', 'time'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://tae.alibaba-inc.com/expense/pc.html?_authType=SAML');
|
||||
await page.wait(2);
|
||||
const result = await page.evaluate(`(async () => {
|
||||
const res = await fetch('/approval/getList.json?taskId=&procInsId=${kwargs.proc_ins_id}', {
|
||||
credentials: 'include'
|
||||
});
|
||||
const data = await res.json();
|
||||
return data?.content?.operatorRecords || [];
|
||||
})()`);
|
||||
return (result as any[]).map((r, i) => ({
|
||||
step: i + 1,
|
||||
operator: r.operatorName || r.userId,
|
||||
action: r.operationType,
|
||||
time: r.operateTime,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
**转换要点**:
|
||||
1. URL 中的动态 ID(`procInsId`、`taskId` 等)提取为 `args`
|
||||
2. `captured.json` 里的真实 body 结构用于确定正确的数据路径(如 `content.operatorRecords`)
|
||||
3. tae 系统统一用 `{ success, content, errorCode, errorMsg }` 外层包裹,取数据要走 `content.*`
|
||||
4. 认证方式:cookie(`credentials: 'include'`),不需要额外 header
|
||||
5. 文件放入 `src/clis/<site>/`,无需手动注册,`npm run build` 后自动发现
|
||||
|
||||
### 故障排查
|
||||
|
||||
| 现象 | 原因 | 解法 |
|
||||
|------|------|------|
|
||||
| 捕获 0 条请求 | 拦截器注入失败,或页面无 JSON API | 检查 daemon 是否运行:`curl localhost:19825/status` |
|
||||
| 捕获量少(1~3 条) | 页面是只读详情页,首屏数据已在注入前发出 | 手动操作触发更多请求(搜索/翻页),或换用列表页 |
|
||||
| 候选 YAML 为 0 | 捕获到的 JSON 都没有 array 结构 | 直接看 `captured.json` 手写 TS CLI |
|
||||
| 新开的 tab 没有被拦截 | 轮询间隔内 tab 已关闭 | 缩短 `--poll 500` |
|
||||
| 二次运行 record 时数据不连续 | 正常,每次 `record` 启动都是新的 automation window | 无需处理 |
|
||||
@@ -0,0 +1,222 @@
|
||||
---
|
||||
name: opencli-oneshot
|
||||
description: Use when quickly generating a single OpenCLI command from a specific URL and goal description. 4-step process — open page, capture API, write YAML adapter, test. For full site exploration, use opencli-explorer instead.
|
||||
tags: [opencli, adapter, quick-start, yaml, cli, one-shot, automation]
|
||||
---
|
||||
|
||||
# CLI-ONESHOT — 单点快速 CLI 生成
|
||||
|
||||
> 给一个 URL + 一句话描述,4 步生成一个 CLI 命令。
|
||||
> 完整探索式开发请看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
|
||||
|
||||
---
|
||||
|
||||
## 输入
|
||||
|
||||
| 项目 | 示例 |
|
||||
|------|------|
|
||||
| **URL** | `https://x.com/jakevin7/lists` |
|
||||
| **Goal** | 获取我的 Twitter Lists |
|
||||
|
||||
---
|
||||
|
||||
## 流程
|
||||
|
||||
### Step 1: 打开页面 + 抓包
|
||||
|
||||
```
|
||||
1. browser_navigate → 打开目标 URL
|
||||
2. 等待 3-5 秒(让页面加载完、API 请求触发)
|
||||
3. browser_network_requests → 筛选 JSON API
|
||||
```
|
||||
|
||||
**关键**:只关注返回 `application/json` 的请求,忽略静态资源。
|
||||
如果没有自动触发 API,手动点击目标按钮/标签再抓一次。
|
||||
|
||||
### Step 2: 锁定一个接口
|
||||
|
||||
从抓包结果中找到**那个**目标 API。看这几个字段:
|
||||
|
||||
| 字段 | 关注什么 |
|
||||
|------|----------|
|
||||
| URL | API 路径 pattern(如 `/i/api/graphql/xxx/ListsManagePinTimeline`) |
|
||||
| Method | GET / POST |
|
||||
| Headers | 有 Cookie? Bearer? CSRF? 自定义签名? |
|
||||
| Response | 数据在哪个路径(如 `data.list.lists`) |
|
||||
|
||||
### Step 3: 验证接口能复现
|
||||
|
||||
在 `browser_evaluate` 中用 `fetch` 复现请求:
|
||||
|
||||
```javascript
|
||||
// Tier 2 (Cookie): 大多数情况
|
||||
fetch('/api/endpoint', { credentials: 'include' }).then(r => r.json())
|
||||
|
||||
// Tier 3 (Header): 如 Twitter 需要额外 header
|
||||
const ct0 = document.cookie.match(/ct0=([^;]+)/)?.[1];
|
||||
fetch('/api/endpoint', {
|
||||
headers: { 'Authorization': 'Bearer ...', 'X-Csrf-Token': ct0 },
|
||||
credentials: 'include'
|
||||
}).then(r => r.json())
|
||||
```
|
||||
|
||||
如果 fetch 能拿到数据 → 用 YAML 或简单 TS adapter。
|
||||
如果 fetch 拿不到(签名/风控)→ 用 intercept 策略。
|
||||
|
||||
### Step 4: 套模板,生成 adapter
|
||||
|
||||
根据 Step 3 判定的策略,选一个模板生成文件。
|
||||
|
||||
---
|
||||
|
||||
## 认证速查
|
||||
|
||||
```
|
||||
fetch(url) 直接能拿到? → Tier 1: public (YAML, browser: false)
|
||||
fetch(url, {credentials:'include'})? → Tier 2: cookie (YAML)
|
||||
加 Bearer/CSRF header 后拿到? → Tier 3: header (TS)
|
||||
都不行,但页面自己能请求成功? → Tier 4: intercept (TS, installInterceptor)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 模板
|
||||
|
||||
### YAML — Cookie/Public(最简)
|
||||
|
||||
```yaml
|
||||
# src/clis/<site>/<name>.yaml
|
||||
site: mysite
|
||||
name: mycommand
|
||||
description: "一句话描述"
|
||||
domain: www.example.com
|
||||
strategy: cookie # 或 public (加 browser: false)
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.example.com/target-page
|
||||
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const res = await fetch('/api/target', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return (d.data?.items || []).map(item => ({
|
||||
title: item.title,
|
||||
value: item.value,
|
||||
}));
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
value: ${{ item.value }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, value]
|
||||
```
|
||||
|
||||
### TS — Intercept(抓包模式)
|
||||
|
||||
```typescript
|
||||
// src/clis/<site>/<name>.ts
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'mysite',
|
||||
name: 'mycommand',
|
||||
description: '一句话描述',
|
||||
domain: 'www.example.com',
|
||||
strategy: Strategy.INTERCEPT,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20 },
|
||||
],
|
||||
columns: ['rank', 'title', 'value'],
|
||||
func: async (page, kwargs) => {
|
||||
// 1. 导航
|
||||
await page.goto('https://www.example.com/target-page');
|
||||
await page.wait(3);
|
||||
|
||||
// 2. 注入拦截器(URL 子串匹配)
|
||||
await page.installInterceptor('target-api-keyword');
|
||||
|
||||
// 3. 触发 API(滚动/点击)
|
||||
await page.autoScroll({ times: 2, delayMs: 2000 });
|
||||
|
||||
// 4. 读取拦截的响应
|
||||
const requests = await page.getInterceptedRequests();
|
||||
if (!requests?.length) return [];
|
||||
|
||||
let results: any[] = [];
|
||||
for (const req of requests) {
|
||||
const items = req.data?.data?.items || [];
|
||||
results.push(...items);
|
||||
}
|
||||
|
||||
return results.slice(0, kwargs.limit).map((item, i) => ({
|
||||
rank: i + 1,
|
||||
title: item.title || '',
|
||||
value: item.value || '',
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### TS — Header(如 Twitter GraphQL)
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'twitter',
|
||||
name: 'mycommand',
|
||||
description: '一句话描述',
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.HEADER,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20 },
|
||||
],
|
||||
columns: ['rank', 'name', 'value'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://x.com');
|
||||
const data = await page.evaluate(`(async () => {
|
||||
const ct0 = document.cookie.match(/ct0=([^;]+)/)?.[1];
|
||||
if (!ct0) return { error: 'Not logged in' };
|
||||
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
|
||||
const res = await fetch('/i/api/graphql/QUERY_ID/Endpoint', {
|
||||
headers: {
|
||||
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
|
||||
'X-Csrf-Token': ct0,
|
||||
'X-Twitter-Auth-Type': 'OAuth2Session',
|
||||
},
|
||||
credentials: 'include',
|
||||
});
|
||||
return res.json();
|
||||
})()`);
|
||||
// 解析 data...
|
||||
return [];
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 测试(必做)
|
||||
|
||||
```bash
|
||||
npm run build # 语法检查
|
||||
opencli list | grep mysite # 确认注册
|
||||
opencli mysite mycommand --limit 3 -v # 实际运行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 就这样,没了
|
||||
|
||||
写完文件 → build → run → 提交。有问题再看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
|
||||
@@ -0,0 +1,213 @@
|
||||
---
|
||||
name: opencli-operate
|
||||
description: Make websites accessible for AI agents. Navigate, click, type, extract, wait — using Chrome with existing login sessions. No LLM API key needed.
|
||||
allowed-tools: Bash(opencli:*), Read, Edit, Write
|
||||
---
|
||||
|
||||
# OpenCLI — Make Websites Accessible for AI Agents
|
||||
|
||||
Control Chrome step-by-step via CLI. Reuses existing login sessions — no passwords needed.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
opencli doctor # Verify extension + daemon connectivity
|
||||
```
|
||||
|
||||
Requires: Chrome running + OpenCLI Browser Bridge extension installed.
|
||||
|
||||
## Quickstart for AI Agents (1 step)
|
||||
|
||||
Point your AI agent to this file. It contains everything needed to operate browsers.
|
||||
|
||||
## Quickstart for Humans (3 steps)
|
||||
|
||||
```bash
|
||||
npm install -g @jackwener/opencli # 1. Install
|
||||
# Install extension from chrome://extensions # 2. Load extension
|
||||
opencli operate open https://example.com # 3. Go!
|
||||
```
|
||||
|
||||
## Core Workflow
|
||||
|
||||
1. **Navigate**: `opencli operate open <url>`
|
||||
2. **Inspect**: `opencli operate state` → see elements with `[N]` indices
|
||||
3. **Interact**: use indices — `click`, `type`, `select`, `keys`
|
||||
4. **Wait**: `opencli operate wait selector ".loaded"` or `wait text "Success"`
|
||||
5. **Verify**: `opencli operate get title` or `opencli operate screenshot`
|
||||
6. **Repeat**: browser stays open between commands
|
||||
7. **Save**: write a TS adapter to `~/.opencli/clis/<site>/<command>.ts`
|
||||
|
||||
## Commands
|
||||
|
||||
### Navigation
|
||||
|
||||
```bash
|
||||
opencli operate open <url> # Open URL
|
||||
opencli operate back # Go back
|
||||
opencli operate scroll down # Scroll (up/down, --amount N)
|
||||
opencli operate scroll up --amount 1000
|
||||
```
|
||||
|
||||
### Inspect
|
||||
|
||||
```bash
|
||||
opencli operate state # Elements with [N] indices
|
||||
opencli operate screenshot [path.png] # Screenshot
|
||||
```
|
||||
|
||||
### Get (structured data)
|
||||
|
||||
```bash
|
||||
opencli operate get title # Page title
|
||||
opencli operate get url # Current URL
|
||||
opencli operate get text <index> # Element text content
|
||||
opencli operate get value <index> # Input/textarea value
|
||||
opencli operate get html # Full page HTML
|
||||
opencli operate get html --selector "h1" # Scoped HTML
|
||||
opencli operate get attributes <index> # Element attributes
|
||||
```
|
||||
|
||||
### Interact
|
||||
|
||||
```bash
|
||||
opencli operate click <index> # Click element [N]
|
||||
opencli operate type <index> "text" # Type into element [N]
|
||||
opencli operate select <index> "option" # Select dropdown
|
||||
opencli operate keys "Enter" # Press key (Enter, Escape, Tab, Control+a)
|
||||
```
|
||||
|
||||
### Wait
|
||||
|
||||
```bash
|
||||
opencli operate wait selector ".loaded" # Wait for element
|
||||
opencli operate wait selector ".spinner" --timeout 5000 # With timeout
|
||||
opencli operate wait text "Success" # Wait for text
|
||||
opencli operate wait time 3 # Wait N seconds
|
||||
```
|
||||
|
||||
### Extract
|
||||
|
||||
```bash
|
||||
opencli operate eval "document.title"
|
||||
opencli operate eval "JSON.stringify([...document.querySelectorAll('h2')].map(e => e.textContent))"
|
||||
```
|
||||
|
||||
### Network (API Discovery)
|
||||
|
||||
```bash
|
||||
opencli operate network # Show captured API requests (auto-captured since open)
|
||||
opencli operate network --detail 3 # Show full response body of request #3
|
||||
opencli operate network --all # Include static resources
|
||||
```
|
||||
|
||||
### Sedimentation (Save as CLI)
|
||||
|
||||
```bash
|
||||
opencli operate init hn/top # Generate adapter scaffold
|
||||
opencli operate verify hn/top # Test the adapter
|
||||
```
|
||||
|
||||
### Session
|
||||
|
||||
```bash
|
||||
opencli operate close # Close automation window
|
||||
```
|
||||
|
||||
## Example: Extract HN Stories
|
||||
|
||||
```bash
|
||||
opencli operate open https://news.ycombinator.com
|
||||
opencli operate state # See [1] a "Story 1", [2] a "Story 2"...
|
||||
opencli operate eval "JSON.stringify([...document.querySelectorAll('.titleline a')].slice(0,5).map(a => ({title: a.textContent, url: a.href})))"
|
||||
opencli operate close
|
||||
```
|
||||
|
||||
## Example: Fill a Form
|
||||
|
||||
```bash
|
||||
opencli operate open https://httpbin.org/forms/post
|
||||
opencli operate state # See [3] input "Customer Name", [4] input "Telephone"
|
||||
opencli operate type 3 "OpenCLI"
|
||||
opencli operate type 4 "555-0100"
|
||||
opencli operate get value 3 # Verify: "OpenCLI"
|
||||
opencli operate close
|
||||
```
|
||||
|
||||
## Saving as Reusable CLI — Complete Workflow
|
||||
|
||||
### Step-by-step sedimentation flow:
|
||||
|
||||
```bash
|
||||
# 1. Explore the website
|
||||
opencli operate open https://news.ycombinator.com
|
||||
opencli operate state # Understand DOM structure
|
||||
|
||||
# 2. Discover APIs (crucial for high-quality adapters)
|
||||
opencli operate eval "fetch('/api/...').then(r=>r.json())" # Trigger API calls
|
||||
opencli operate network # See captured API requests
|
||||
opencli operate network --detail 0 # Inspect response body
|
||||
|
||||
# 3. Generate scaffold
|
||||
opencli operate init hn/top # Creates ~/.opencli/clis/hn/top.ts
|
||||
|
||||
# 4. Edit the adapter (fill in func logic)
|
||||
# - If API found: use fetch() directly (Strategy.PUBLIC or COOKIE)
|
||||
# - If no API: use page.evaluate() for DOM extraction (Strategy.UI)
|
||||
|
||||
# 5. Verify
|
||||
opencli operate verify hn/top # Runs the adapter and shows output
|
||||
|
||||
# 6. If verify fails, edit and retry
|
||||
# 7. Close when done
|
||||
opencli operate close
|
||||
```
|
||||
|
||||
### Example adapter:
|
||||
|
||||
```typescript
|
||||
// ~/.opencli/clis/hn/top.ts
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'hn',
|
||||
name: 'top',
|
||||
description: 'Top Hacker News stories',
|
||||
domain: 'news.ycombinator.com',
|
||||
strategy: Strategy.PUBLIC,
|
||||
browser: false,
|
||||
args: [{ name: 'limit', type: 'int', default: 5 }],
|
||||
columns: ['rank', 'title', 'score', 'url'],
|
||||
func: async (_page, kwargs) => {
|
||||
const limit = Math.min(Math.max(1, kwargs.limit ?? 5), 50);
|
||||
const resp = await fetch('https://hacker-news.firebaseio.com/v0/topstories.json');
|
||||
const ids = await resp.json();
|
||||
return Promise.all(
|
||||
ids.slice(0, limit).map(async (id: number, i: number) => {
|
||||
const item = await (await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`)).json();
|
||||
return { rank: i + 1, title: item.title, score: item.score, url: item.url ?? '' };
|
||||
})
|
||||
);
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Save to `~/.opencli/clis/<site>/<command>.ts` → immediately available as `opencli <site> <command>`.
|
||||
|
||||
### Strategy Guide
|
||||
|
||||
| Strategy | When | browser: |
|
||||
|----------|------|----------|
|
||||
| `Strategy.PUBLIC` | Public API, no auth | `false` |
|
||||
| `Strategy.COOKIE` | Needs login cookies | `true` |
|
||||
| `Strategy.UI` | Direct DOM interaction | `true` |
|
||||
|
||||
**Always prefer API over UI** — if you discovered an API during browsing, use `fetch()` directly.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Error | Fix |
|
||||
|-------|-----|
|
||||
| "Browser not connected" | Run `opencli doctor` |
|
||||
| "attach failed: chrome-extension://" | Disable 1Password temporarily |
|
||||
| Element not found | `opencli operate scroll down` then `opencli operate state` |
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: opencli-usage
|
||||
description: "Use when running OpenCLI commands to interact with websites (Bilibili, Twitter, Reddit, Xiaohongshu, etc.), desktop apps (Cursor, Notion), or public APIs (HackerNews, arXiv). Covers installation, command reference, and output formats for 70+ adapters."
|
||||
version: 1.6.0
|
||||
author: jackwener
|
||||
tags: [opencli, cli, browser, web, chrome-extension, cdp, bilibili, twitter, reddit, xiaohongshu, github, youtube, AI, agent, automation]
|
||||
---
|
||||
|
||||
# OpenCLI Usage Guide
|
||||
|
||||
> Make any website or Electron App your CLI. Reuse Chrome login, zero risk, AI-powered discovery.
|
||||
|
||||
## Install & Run
|
||||
|
||||
```bash
|
||||
# npm global install (recommended)
|
||||
npm install -g @jackwener/opencli
|
||||
opencli <command>
|
||||
|
||||
# Or from source
|
||||
cd ~/code/opencli && npm install
|
||||
npx tsx src/main.ts <command>
|
||||
|
||||
# Update to latest
|
||||
npm update -g @jackwener/opencli
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Browser commands require:
|
||||
1. Chrome browser running **(logged into target sites)**
|
||||
2. **opencli Browser Bridge** Chrome extension installed (load `extension/` as unpacked in `chrome://extensions`)
|
||||
3. No further setup needed — the daemon auto-starts on first browser command
|
||||
|
||||
> **Note**: You must be logged into the target website in Chrome before running commands. Tabs opened during command execution are auto-closed afterwards.
|
||||
|
||||
Public API commands (`hackernews`, `v2ex`) need no browser.
|
||||
|
||||
## Quick Lookup by Capability
|
||||
|
||||
| Capability | Platforms (partial list) | File |
|
||||
|-----------|--------------------------|------|
|
||||
| **search** | Bilibili, Twitter, Reddit, Xiaohongshu, Zhihu, YouTube, Google, arXiv, LinkedIn, Pixiv, etc. | browser.md / public-api.md |
|
||||
| **hot/trending** | Bilibili, Twitter, Weibo, HackerNews, Reddit, V2EX, Xueqiu, Lobsters, Douban | browser.md / public-api.md |
|
||||
| **feed/timeline** | Twitter, Reddit, Xiaohongshu, Xueqiu, Jike, Facebook, Instagram, Medium | browser.md |
|
||||
| **user/profile** | Twitter, Reddit, Instagram, TikTok, Facebook, Bilibili, Pixiv | browser.md |
|
||||
| **post/create** | Twitter, Jike | browser.md |
|
||||
| **AI chat** | Grok, Doubao, Kimi, DeepSeek, Qwen, ChatGPT, Cursor, Codex | browser.md / desktop.md |
|
||||
| **finance/stock** | Xueqiu, Yahoo Finance, Barchart, Sina Finance, Bloomberg | browser.md / public-api.md |
|
||||
| **web scraping** | `opencli web read --url <url>` — any URL to Markdown | browser.md |
|
||||
|
||||
## Command Quick Reference
|
||||
|
||||
Usage: `opencli <site> <command> [args] [--limit N] [-f json|yaml|md|csv|table]`
|
||||
|
||||
### Browser-based (login required)
|
||||
|
||||
| Site | Commands |
|
||||
|------|----------|
|
||||
| **bilibili** | `hot` `search` `me` `favorite` `history` `feed` `user-videos` `subtitle` `dynamic` `ranking` `following` |
|
||||
| **zhihu** | `hot` `search` `question` |
|
||||
| **xiaohongshu** | `search` `notifications` `feed` `user` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` |
|
||||
| **xueqiu** | `hot-stock` `stock` `watchlist` `feed` `hot` `search` `earnings-date` `fund-holdings` `fund-snapshot` |
|
||||
| **twitter** | `trending` `bookmarks` `search` `profile` `timeline` `thread` `article` `follow` `unfollow` `bookmark` `unbookmark` `post` `like` `reply` `delete` `block` `unblock` `followers` `following` `notifications` `hide-reply` `download` `accept` `reply-dm` |
|
||||
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` |
|
||||
| **youtube** | `search` `video` `transcript` |
|
||||
| **facebook** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` |
|
||||
| **instagram** | `explore` `profile` `search` `user` `followers` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `saved` |
|
||||
| **tiktok** | `explore` `search` `profile` `user` `following` `follow` `unfollow` `like` `unlike` `comment` `save` `unsave` `live` `notifications` `friends` |
|
||||
| **linkedin** | `search` `timeline` |
|
||||
| **medium** | `feed` `search` `user` |
|
||||
| **substack** | `feed` `search` `publication` |
|
||||
| **sinablog** | `hot` `search` `article` `user` |
|
||||
| **weibo** | `hot` |
|
||||
| **boss** | `search` `detail` `recommend` `joblist` `greet` `batchgreet` `send` `chatlist` `chatmsg` `invite` `mark` `exchange` `resume` `stats` |
|
||||
| **douban** | `search` `top250` `subject` `photos` `download` `marks` `reviews` |
|
||||
| **pixiv** | `ranking` `search` `user` `illusts` `detail` `download` |
|
||||
| **jike** | `feed` `search` `create` `like` `comment` `repost` `notifications` |
|
||||
| **yahoo-finance** | `quote` |
|
||||
| **barchart** | `quote` `options` `greeks` `flow` |
|
||||
| **sinafinance** | `news` |
|
||||
| **reuters** | `search` |
|
||||
| **coupang** | `search` `add-to-cart` |
|
||||
| **jd** | `item` |
|
||||
| **smzdm** | `search` |
|
||||
| **ctrip** | `search` |
|
||||
| **weread** | `shelf` `search` `book` `highlights` `notes` `ranking` |
|
||||
| **chaoxing** | `assignments` `exams` |
|
||||
| **jimeng** | `generate` `history` |
|
||||
| **yollomi** | `models` `generate` `video` `upload` `remove-bg` `edit` `background` `face-swap` `object-remover` `restore` `try-on` `upscale` |
|
||||
| **web** | `read` — any URL to Markdown |
|
||||
| **weixin** | `download` — 公众号 article to Markdown |
|
||||
| **v2ex** (browser) | `daily` `me` `notifications` |
|
||||
| **linux-do** (browser) | `categories` `category` |
|
||||
| **bloomberg** (browser) | `news` — full article reader |
|
||||
| **grok** | `ask` |
|
||||
| **doubao** | `status` `new` `send` `read` `ask` |
|
||||
| **kimi** | `status` `new` `ask` |
|
||||
| **deepseek** | `status` `new` `ask` |
|
||||
| **qwen** | `status` `new` `ask` |
|
||||
|
||||
### Desktop (CDP/Electron)
|
||||
|
||||
| Site | Commands |
|
||||
|------|----------|
|
||||
| **gh** | `repo` `pr` `issue` — passthrough to gh CLI |
|
||||
| **cursor** | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` |
|
||||
| **codex** | `status` `send` `read` `new` `dump` `extract-diff` `model` `ask` `screenshot` `history` `export` |
|
||||
| **chatgpt** | `status` `new` `send` `read` `ask` |
|
||||
| **chatwise** | `status` `new` `send` `read` `ask` `model` `history` `export` `screenshot` |
|
||||
| **notion** | `status` `search` `read` `new` `write` `sidebar` `favorites` `export` |
|
||||
| **discord-app** | `status` `send` `read` `channels` `servers` `search` `members` |
|
||||
| **doubao-app** | `status` `new` `send` `read` `ask` `screenshot` `dump` |
|
||||
| **antigravity** | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` |
|
||||
|
||||
### Public API (no browser)
|
||||
|
||||
| Site | Commands |
|
||||
|------|----------|
|
||||
| **hackernews** | `top` `new` `best` `ask` `show` `jobs` `search` `user` |
|
||||
| **v2ex** (public) | `hot` `latest` `topic` `node` `nodes` `member` `user` `replies` |
|
||||
| **bbc** | `news` |
|
||||
| **lobsters** | `hot` `newest` `active` `tag` |
|
||||
| **google** | `news` `search` `suggest` `trends` |
|
||||
| **devto** | `top` `tag` `user` |
|
||||
| **steam** | `top-sellers` |
|
||||
| **apple-podcasts** | `top` `search` `episodes` |
|
||||
| **arxiv** | `search` `paper` |
|
||||
| **bloomberg** (RSS) | `main` `markets` `tech` `politics` `economics` `opinions` `industries` `businessweek` `feeds` |
|
||||
| **dictionary** | `search` `synonyms` `examples` |
|
||||
| **hf** | `top` |
|
||||
| **stackoverflow** | `hot` `search` `bounties` |
|
||||
| **xiaoyuzhou** | `podcast` `podcast-episodes` `episode` |
|
||||
| **wikipedia** | `search` `summary` |
|
||||
| **producthunt** | `today` `week` `month` `search` |
|
||||
|
||||
### Management
|
||||
|
||||
```bash
|
||||
opencli list [-f json|yaml] # List all commands
|
||||
opencli validate [site] # Validate adapter definitions
|
||||
opencli doctor # Diagnose browser bridge
|
||||
opencli explore <url> # AI-powered API discovery
|
||||
opencli record <url> # Record API calls manually
|
||||
```
|
||||
|
||||
All commands support: `--format` / `-f` with `table` `json` `yaml` `md` `csv`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **opencli-explorer** — Full guide for creating new adapters (API discovery, auth strategy, YAML/TS writing)
|
||||
- **opencli-oneshot** — Quick 4-step template for adding a single command from a URL
|
||||
@@ -0,0 +1,429 @@
|
||||
# Browser-based Commands
|
||||
|
||||
Commands that require Chrome browser with login state.
|
||||
|
||||
## Bilibili (哔哩哔哩)
|
||||
|
||||
```bash
|
||||
opencli bilibili hot --limit 10 # B站热门视频
|
||||
opencli bilibili search "rust" # 搜索视频 (query positional)
|
||||
opencli bilibili me # 我的信息
|
||||
opencli bilibili favorite # 我的收藏
|
||||
opencli bilibili history --limit 20 # 观看历史
|
||||
opencli bilibili feed --limit 10 # 动态时间线
|
||||
opencli bilibili user-videos --uid 12345 # 用户投稿
|
||||
opencli bilibili subtitle --bvid BV1xxx # 获取视频字幕 (支持 --lang zh-CN)
|
||||
opencli bilibili dynamic --limit 10 # 动态
|
||||
opencli bilibili ranking --limit 10 # 排行榜
|
||||
opencli bilibili following --limit 20 # 我的关注列表 (支持 --uid 查看他人)
|
||||
```
|
||||
|
||||
## Zhihu (知乎)
|
||||
|
||||
```bash
|
||||
opencli zhihu hot --limit 10 # 知乎热榜
|
||||
opencli zhihu search "AI" # 搜索 (query positional)
|
||||
opencli zhihu question 34816524 # 问题详情和回答 (id positional)
|
||||
```
|
||||
|
||||
## Xiaohongshu (小红书)
|
||||
|
||||
```bash
|
||||
opencli xiaohongshu search "美食" # 搜索笔记 (query positional)
|
||||
opencli xiaohongshu notifications # 通知(mentions/likes/connections)
|
||||
opencli xiaohongshu feed --limit 10 # 推荐 Feed
|
||||
opencli xiaohongshu user xxx # 用户主页 (id positional)
|
||||
opencli xiaohongshu creator-notes --limit 10 # 创作者笔记列表
|
||||
opencli xiaohongshu creator-note-detail --note-id xxx # 笔记详情
|
||||
opencli xiaohongshu creator-notes-summary # 笔记数据概览
|
||||
opencli xiaohongshu creator-profile # 创作者资料
|
||||
opencli xiaohongshu creator-stats # 创作者数据统计
|
||||
```
|
||||
|
||||
## Xueqiu (雪球)
|
||||
|
||||
```bash
|
||||
opencli xueqiu hot-stock --limit 10 # 雪球热门股票榜
|
||||
opencli xueqiu stock --symbol SH600519 # 查看股票实时行情
|
||||
opencli xueqiu watchlist # 获取自选股/持仓列表
|
||||
opencli xueqiu feed # 我的关注 timeline
|
||||
opencli xueqiu hot --limit 10 # 雪球热榜
|
||||
opencli xueqiu search "特斯拉" # 搜索 (query positional)
|
||||
opencli xueqiu earnings-date SH600519 # 股票财报发布日期 (symbol positional)
|
||||
opencli xueqiu fund-holdings # 蛋卷基金持仓明细 (支持 --account 过滤)
|
||||
opencli xueqiu fund-snapshot # 蛋卷基金快照(总资产、子账户、持仓)
|
||||
```
|
||||
|
||||
## Twitter/X
|
||||
|
||||
```bash
|
||||
opencli twitter trending --limit 10 # 热门话题
|
||||
opencli twitter bookmarks --limit 20 # 获取收藏的书签推文
|
||||
opencli twitter search "AI" # 搜索推文 (query positional)
|
||||
opencli twitter profile elonmusk # 用户资料
|
||||
opencli twitter timeline --limit 20 # 时间线
|
||||
opencli twitter thread 1234567890 # 推文 thread(原文 + 回复)
|
||||
opencli twitter article 1891511252174299446 # 推文长文内容
|
||||
opencli twitter follow elonmusk # 关注用户
|
||||
opencli twitter unfollow elonmusk # 取消关注
|
||||
opencli twitter bookmark https://x.com/... # 收藏推文
|
||||
opencli twitter unbookmark https://x.com/... # 取消收藏
|
||||
opencli twitter post "Hello world" # 发布推文 (text positional)
|
||||
opencli twitter like https://x.com/... # 点赞推文 (url positional)
|
||||
opencli twitter reply https://x.com/... "Nice!" # 回复推文 (url + text positional)
|
||||
opencli twitter delete https://x.com/... # 删除推文 (url positional)
|
||||
opencli twitter block elonmusk # 屏蔽用户 (username positional)
|
||||
opencli twitter unblock elonmusk # 取消屏蔽 (username positional)
|
||||
opencli twitter followers elonmusk # 用户的粉丝列表 (user positional)
|
||||
opencli twitter following elonmusk # 用户的关注列表 (user positional)
|
||||
opencli twitter notifications --limit 20 # 通知列表
|
||||
opencli twitter hide-reply https://x.com/... # 隐藏回复 (url positional)
|
||||
opencli twitter download elonmusk # 下载用户媒体 (username positional, 支持 --tweet-url)
|
||||
opencli twitter accept "群,微信" # 自动接受含关键词的 DM 请求 (query positional)
|
||||
opencli twitter reply-dm "消息内容" # 批量回复 DM (text positional)
|
||||
```
|
||||
|
||||
## Reddit
|
||||
|
||||
```bash
|
||||
opencli reddit hot --limit 10 # 热门帖子
|
||||
opencli reddit hot --subreddit programming # 指定子版块
|
||||
opencli reddit frontpage --limit 10 # 首页 /r/all
|
||||
opencli reddit popular --limit 10 # /r/popular 热门
|
||||
opencli reddit search "AI" --sort top --time week # 搜索(支持排序+时间过滤)
|
||||
opencli reddit subreddit rust --sort top --time month # 子版块浏览(支持时间过滤)
|
||||
opencli reddit read --post-id 1abc123 # 阅读帖子 + 评论
|
||||
opencli reddit user spez # 用户资料(karma、注册时间)
|
||||
opencli reddit user-posts spez # 用户发帖历史
|
||||
opencli reddit user-comments spez # 用户评论历史
|
||||
opencli reddit upvote --post-id xxx --direction up # 投票(up/down/none)
|
||||
opencli reddit save --post-id xxx # 收藏帖子
|
||||
opencli reddit comment --post-id xxx "Great!" # 发表评论 (text positional)
|
||||
opencli reddit subscribe --subreddit python # 订阅子版块
|
||||
opencli reddit saved --limit 10 # 我的收藏
|
||||
opencli reddit upvoted --limit 10 # 我的赞
|
||||
```
|
||||
|
||||
## V2EX (Browser Features)
|
||||
|
||||
```bash
|
||||
opencli v2ex daily # 每日签到
|
||||
opencli v2ex me # 我的信息
|
||||
opencli v2ex notifications --limit 10 # 通知
|
||||
```
|
||||
|
||||
## Weibo (微博)
|
||||
|
||||
```bash
|
||||
opencli weibo hot --limit 10 # 微博热搜
|
||||
```
|
||||
|
||||
## BOSS直聘
|
||||
|
||||
```bash
|
||||
opencli boss search "AI agent" # 搜索职位 (query positional)
|
||||
opencli boss detail --security-id xxx # 职位详情
|
||||
opencli boss recommend --limit 10 # 推荐职位
|
||||
opencli boss joblist --limit 10 # 职位列表
|
||||
opencli boss greet --security-id xxx # 打招呼
|
||||
opencli boss batchgreet --job-id xxx # 批量打招呼
|
||||
opencli boss send --uid xxx "消息内容" # 发消息 (text positional)
|
||||
opencli boss chatlist --limit 10 # 聊天列表
|
||||
opencli boss chatmsg --security-id xxx # 聊天记录
|
||||
opencli boss invite --security-id xxx # 邀请沟通
|
||||
opencli boss mark --security-id xxx # 标记管理
|
||||
opencli boss exchange --security-id xxx # 交换联系方式
|
||||
opencli boss resume # 简历管理
|
||||
opencli boss stats # 数据统计
|
||||
```
|
||||
|
||||
## YouTube
|
||||
|
||||
```bash
|
||||
opencli youtube search "rust" # 搜索视频 (query positional)
|
||||
opencli youtube video "https://www.youtube.com/watch?v=xxx" # 视频元数据
|
||||
opencli youtube transcript "https://www.youtube.com/watch?v=xxx" # 获取视频字幕/转录
|
||||
opencli youtube transcript "xxx" --lang zh-Hans --mode raw # 指定语言 + 原始时间戳模式
|
||||
```
|
||||
|
||||
## Yahoo Finance
|
||||
|
||||
```bash
|
||||
opencli yahoo-finance quote --symbol AAPL # 股票行情
|
||||
```
|
||||
|
||||
## Sina Finance
|
||||
|
||||
```bash
|
||||
opencli sinafinance news --limit 10 --type 1 # 7x24实时快讯
|
||||
# Types: 0=全部 1=A股 2=宏观 3=公司 4=数据 5=市场 6=国际 7=观点 8=央行 9=其它
|
||||
```
|
||||
|
||||
## Reuters (路透社)
|
||||
|
||||
```bash
|
||||
opencli reuters search "AI" # 路透社搜索 (query positional)
|
||||
```
|
||||
|
||||
## SMZDM (什么值得买)
|
||||
|
||||
```bash
|
||||
opencli smzdm search "耳机" # 搜索好价 (query positional)
|
||||
```
|
||||
|
||||
## Ctrip (携程)
|
||||
|
||||
```bash
|
||||
opencli ctrip search "三亚" # 搜索目的地 (query positional)
|
||||
```
|
||||
|
||||
## Barchart
|
||||
|
||||
```bash
|
||||
opencli barchart quote --symbol AAPL # 股票行情
|
||||
opencli barchart options --symbol AAPL # 期权链
|
||||
opencli barchart greeks --symbol AAPL # 期权 Greeks
|
||||
opencli barchart flow --limit 20 # 异常期权活动
|
||||
```
|
||||
|
||||
## Jike (即刻)
|
||||
|
||||
```bash
|
||||
opencli jike feed --limit 10 # 动态流
|
||||
opencli jike search "AI" # 搜索 (query positional)
|
||||
opencli jike create "内容" # 发布动态 (text positional)
|
||||
opencli jike like xxx # 点赞 (id positional)
|
||||
opencli jike comment xxx "评论" # 评论 (id + text positional)
|
||||
opencli jike repost xxx # 转发 (id positional)
|
||||
opencli jike notifications # 通知
|
||||
```
|
||||
|
||||
## Linux.do (Browser Features)
|
||||
|
||||
```bash
|
||||
opencli linux-do categories --limit 20 # 分类列表
|
||||
opencli linux-do category dev 7 # 分类内话题 (slug + id positional)
|
||||
```
|
||||
|
||||
## WeRead (微信读书)
|
||||
|
||||
```bash
|
||||
opencli weread shelf --limit 10 # 书架
|
||||
opencli weread search "AI" # 搜索图书 (query positional)
|
||||
opencli weread book xxx # 图书详情 (book-id positional)
|
||||
opencli weread highlights xxx # 划线笔记 (book-id positional)
|
||||
opencli weread notes xxx # 想法笔记 (book-id positional)
|
||||
opencli weread ranking --limit 10 # 排行榜
|
||||
```
|
||||
|
||||
## Jimeng (即梦 AI)
|
||||
|
||||
```bash
|
||||
opencli jimeng generate --prompt "描述" # AI 生图
|
||||
opencli jimeng history --limit 10 # 生成历史
|
||||
```
|
||||
|
||||
## Chaoxing (超星学习通)
|
||||
|
||||
```bash
|
||||
opencli chaoxing assignments # 作业列表
|
||||
opencli chaoxing exams # 考试列表
|
||||
```
|
||||
|
||||
## Douban (豆瓣)
|
||||
|
||||
```bash
|
||||
opencli douban search "三体" # 搜索 (query positional)
|
||||
opencli douban top250 # 豆瓣 Top 250
|
||||
opencli douban subject 1234567 # 条目详情 (id positional)
|
||||
opencli douban photos 30382501 # 图片列表 / 直链(默认海报)
|
||||
opencli douban download 30382501 # 下载海报 / 剧照
|
||||
opencli douban marks --limit 10 # 我的标记
|
||||
opencli douban reviews --limit 10 # 短评
|
||||
```
|
||||
|
||||
## Facebook
|
||||
|
||||
```bash
|
||||
opencli facebook feed --limit 10 # 动态流
|
||||
opencli facebook profile username # 用户资料 (id positional)
|
||||
opencli facebook search "AI" # 搜索 (query positional)
|
||||
opencli facebook friends # 好友列表
|
||||
opencli facebook groups # 群组
|
||||
opencli facebook events # 活动
|
||||
opencli facebook notifications # 通知
|
||||
opencli facebook memories # 回忆
|
||||
opencli facebook add-friend username # 添加好友 (id positional)
|
||||
opencli facebook join-group groupid # 加入群组 (id positional)
|
||||
```
|
||||
|
||||
## Instagram
|
||||
|
||||
```bash
|
||||
opencli instagram explore # 探索
|
||||
opencli instagram profile username # 用户资料 (id positional)
|
||||
opencli instagram search "AI" # 搜索 (query positional)
|
||||
opencli instagram user username # 用户详情 (id positional)
|
||||
opencli instagram followers username # 粉丝 (id positional)
|
||||
opencli instagram following username # 关注 (id positional)
|
||||
opencli instagram follow username # 关注用户 (id positional)
|
||||
opencli instagram unfollow username # 取消关注 (id positional)
|
||||
opencli instagram like postid # 点赞 (id positional)
|
||||
opencli instagram unlike postid # 取消点赞 (id positional)
|
||||
opencli instagram comment postid "评论" # 评论 (id + text positional)
|
||||
opencli instagram save postid # 收藏 (id positional)
|
||||
opencli instagram unsave postid # 取消收藏 (id positional)
|
||||
opencli instagram saved # 已收藏列表
|
||||
```
|
||||
|
||||
## TikTok
|
||||
|
||||
```bash
|
||||
opencli tiktok explore # 探索
|
||||
opencli tiktok search "AI" # 搜索 (query positional)
|
||||
opencli tiktok profile username # 用户资料 (id positional)
|
||||
opencli tiktok user username # 用户详情 (id positional)
|
||||
opencli tiktok following username # 关注列表 (id positional)
|
||||
opencli tiktok follow username # 关注 (id positional)
|
||||
opencli tiktok unfollow username # 取消关注 (id positional)
|
||||
opencli tiktok like videoid # 点赞 (id positional)
|
||||
opencli tiktok unlike videoid # 取消点赞 (id positional)
|
||||
opencli tiktok comment videoid "评论" # 评论 (id + text positional)
|
||||
opencli tiktok save videoid # 收藏 (id positional)
|
||||
opencli tiktok unsave videoid # 取消收藏 (id positional)
|
||||
opencli tiktok live # 直播
|
||||
opencli tiktok notifications # 通知
|
||||
opencli tiktok friends # 朋友
|
||||
```
|
||||
|
||||
## Medium
|
||||
|
||||
```bash
|
||||
opencli medium feed --limit 10 # 动态流
|
||||
opencli medium search "AI" # 搜索 (query positional)
|
||||
opencli medium user username # 用户主页 (id positional)
|
||||
```
|
||||
|
||||
## Substack
|
||||
|
||||
```bash
|
||||
opencli substack feed --limit 10 # 订阅动态
|
||||
opencli substack search "AI" # 搜索 (query positional)
|
||||
opencli substack publication name # 出版物详情 (id positional)
|
||||
```
|
||||
|
||||
## Sinablog (新浪博客)
|
||||
|
||||
```bash
|
||||
opencli sinablog hot --limit 10 # 热门
|
||||
opencli sinablog search "AI" # 搜索 (query positional)
|
||||
opencli sinablog article url # 文章详情
|
||||
opencli sinablog user username # 用户主页 (id positional)
|
||||
```
|
||||
|
||||
## Coupang (쿠팡)
|
||||
|
||||
```bash
|
||||
opencli coupang search "耳机" # 搜索商品 (query positional, 支持 --filter rocket)
|
||||
opencli coupang add-to-cart 12345 # 加入购物车 (product-id positional, 或 --url)
|
||||
```
|
||||
|
||||
## Yollomi (browser — 需在 Chrome 登录 yollomi.com)
|
||||
|
||||
```bash
|
||||
opencli yollomi models --type image # 列出图像模型与积分
|
||||
opencli yollomi generate "提示词" --model z-image-turbo # 文生图
|
||||
opencli yollomi video "提示词" --model kling-2-1 # 视频
|
||||
opencli yollomi upload ./photo.jpg # 上传得 URL,供 img2img / 工具链使用
|
||||
opencli yollomi remove-bg <image-url> # 去背景(免费)
|
||||
opencli yollomi edit <image-url> "改成油画风格" # Qwen 图像编辑
|
||||
opencli yollomi background <image-url> # AI 背景生成 (5 credits)
|
||||
opencli yollomi face-swap --source <url> --target <url> # 换脸 (3 credits)
|
||||
opencli yollomi object-remover <image-url> <mask-url> # AI 去除物体 (3 credits)
|
||||
opencli yollomi restore <image-url> # AI 修复老照片 (4 credits)
|
||||
opencli yollomi try-on --person <url> --cloth <url> # 虚拟试衣 (3 credits)
|
||||
opencli yollomi upscale <image-url> # AI 超分辨率 (1 credit, 支持 --scale 2/4)
|
||||
```
|
||||
|
||||
## Doubao Web (豆包)
|
||||
|
||||
```bash
|
||||
opencli doubao status # 检查豆包页面状态
|
||||
opencli doubao new # 新建对话
|
||||
opencli doubao send "你好" # 发送消息 (text positional)
|
||||
opencli doubao read # 读取对话记录
|
||||
opencli doubao ask "问题" # 一键提问并等回复 (text positional)
|
||||
```
|
||||
|
||||
## Grok
|
||||
|
||||
```bash
|
||||
opencli grok ask --prompt "问题" # 提问 Grok(兼容默认路径)
|
||||
opencli grok ask --prompt "问题" --web # 显式 grok.com consumer web UI 路径
|
||||
```
|
||||
|
||||
## Pixiv
|
||||
|
||||
```bash
|
||||
opencli pixiv ranking --limit 20 # 插画排行榜 (支持 --mode daily/weekly/monthly)
|
||||
opencli pixiv search "風景" # 搜索插画 (query positional)
|
||||
opencli pixiv user 12345 # 画师资料 (uid positional)
|
||||
opencli pixiv illusts 12345 # 画师作品列表 (user-id positional)
|
||||
opencli pixiv detail 12345 # 插画详情 (id positional)
|
||||
opencli pixiv download 12345 # 下载插画 (illust-id positional)
|
||||
```
|
||||
|
||||
## Web
|
||||
|
||||
```bash
|
||||
opencli web read --url "https://..." # 抓取任意网页并导出为 Markdown
|
||||
```
|
||||
|
||||
## Weixin (微信公众号)
|
||||
|
||||
```bash
|
||||
opencli weixin download --url "https://mp.weixin.qq.com/s/xxx" # 下载公众号文章为 Markdown
|
||||
```
|
||||
|
||||
## JD (京东)
|
||||
|
||||
```bash
|
||||
opencli jd item 100291143898 # 商品详情 (sku positional, 含价格/主图/规格)
|
||||
```
|
||||
|
||||
## LinkedIn
|
||||
|
||||
```bash
|
||||
opencli linkedin search "AI engineer" # 搜索职位 (query positional, 支持 --location/--company/--remote)
|
||||
opencli linkedin timeline --limit 20 # 首页动态流
|
||||
```
|
||||
|
||||
## Bloomberg (Browser - Full Article)
|
||||
|
||||
```bash
|
||||
opencli bloomberg news "https://..." # 阅读 Bloomberg 文章全文 (link positional, browser)
|
||||
```
|
||||
|
||||
## Kimi
|
||||
|
||||
```bash
|
||||
opencli kimi status # 检查 Kimi 页面状态
|
||||
opencli kimi new # 新建对话
|
||||
opencli kimi ask "问题" # 提问 (prompt positional)
|
||||
```
|
||||
|
||||
## DeepSeek
|
||||
|
||||
```bash
|
||||
opencli deepseek status # 检查 DeepSeek 页面状态
|
||||
opencli deepseek new # 新建对话
|
||||
opencli deepseek ask "问题" # 提问 (prompt positional)
|
||||
```
|
||||
|
||||
## Qwen (通义千问)
|
||||
|
||||
```bash
|
||||
opencli qwen status # 检查 Qwen 页面状态
|
||||
opencli qwen new # 新建对话
|
||||
opencli qwen ask "问题" # 提问 (prompt positional)
|
||||
```
|
||||
@@ -0,0 +1,118 @@
|
||||
# Desktop Adapter Commands
|
||||
|
||||
Commands that interact with desktop applications via CDP (Chrome DevTools Protocol) on Electron apps, or external CLI tools.
|
||||
|
||||
## GitHub (via gh CLI)
|
||||
|
||||
```bash
|
||||
opencli gh repo list # 列出仓库 (passthrough to gh)
|
||||
opencli gh pr list --limit 5 # PR 列表
|
||||
opencli gh issue list # Issue 列表
|
||||
```
|
||||
|
||||
## Cursor (desktop — CDP via Electron)
|
||||
|
||||
```bash
|
||||
opencli cursor status # 检查连接
|
||||
opencli cursor send "message" # 发送消息
|
||||
opencli cursor read # 读取回复
|
||||
opencli cursor new # 新建对话
|
||||
opencli cursor dump # 导出 DOM 调试信息
|
||||
opencli cursor composer # Composer 模式
|
||||
opencli cursor model claude # 切换模型
|
||||
opencli cursor extract-code # 提取代码块
|
||||
opencli cursor ask "question" # 一键提问并等回复
|
||||
opencli cursor screenshot # 截图
|
||||
opencli cursor history # 对话历史
|
||||
opencli cursor export # 导出对话
|
||||
```
|
||||
|
||||
## Codex (desktop — headless CLI agent)
|
||||
|
||||
```bash
|
||||
opencli codex status # 检查连接
|
||||
opencli codex send "message" # 发送消息
|
||||
opencli codex read # 读取回复
|
||||
opencli codex new # 新建对话
|
||||
opencli codex dump # 导出调试信息
|
||||
opencli codex extract-diff # 提取 diff
|
||||
opencli codex model gpt-4 # 切换模型
|
||||
opencli codex ask "question" # 一键提问并等回复
|
||||
opencli codex screenshot # 截图
|
||||
opencli codex history # 对话历史
|
||||
opencli codex export # 导出对话
|
||||
```
|
||||
|
||||
## ChatGPT (desktop — macOS AppleScript/CDP)
|
||||
|
||||
```bash
|
||||
opencli chatgpt status # 检查应用状态
|
||||
opencli chatgpt new # 新建对话
|
||||
opencli chatgpt send "message" # 发送消息
|
||||
opencli chatgpt read # 读取回复
|
||||
opencli chatgpt ask "question" # 一键提问并等回复
|
||||
```
|
||||
|
||||
## ChatWise (desktop — multi-LLM client)
|
||||
|
||||
```bash
|
||||
opencli chatwise status # 检查连接
|
||||
opencli chatwise new # 新建对话
|
||||
opencli chatwise send "message" # 发送消息
|
||||
opencli chatwise read # 读取回复
|
||||
opencli chatwise ask "question" # 一键提问并等回复
|
||||
opencli chatwise model claude # 切换模型
|
||||
opencli chatwise history # 对话历史
|
||||
opencli chatwise export # 导出对话
|
||||
opencli chatwise screenshot # 截图
|
||||
```
|
||||
|
||||
## Notion (desktop — CDP via Electron)
|
||||
|
||||
```bash
|
||||
opencli notion status # 检查连接
|
||||
opencli notion search "keyword" # 搜索页面
|
||||
opencli notion read # 读取当前页面
|
||||
opencli notion new # 新建页面
|
||||
opencli notion write "content" # 写入内容
|
||||
opencli notion sidebar # 侧边栏导航
|
||||
opencli notion favorites # 收藏列表
|
||||
opencli notion export # 导出
|
||||
```
|
||||
|
||||
## Discord App (desktop — CDP via Electron)
|
||||
|
||||
```bash
|
||||
opencli discord-app status # 检查连接
|
||||
opencli discord-app send "message" # 发送消息
|
||||
opencli discord-app read # 读取消息
|
||||
opencli discord-app channels # 频道列表
|
||||
opencli discord-app servers # 服务器列表
|
||||
opencli discord-app search "keyword" # 搜索
|
||||
opencli discord-app members # 成员列表
|
||||
```
|
||||
|
||||
## Doubao App 豆包桌面版 (desktop — CDP via Electron)
|
||||
|
||||
```bash
|
||||
opencli doubao-app status # 检查连接
|
||||
opencli doubao-app new # 新建对话
|
||||
opencli doubao-app send "message" # 发送消息
|
||||
opencli doubao-app read # 读取回复
|
||||
opencli doubao-app ask "question" # 一键提问并等回复
|
||||
opencli doubao-app screenshot # 截图
|
||||
opencli doubao-app dump # 导出 DOM 调试信息
|
||||
```
|
||||
|
||||
## Antigravity (Electron/CDP)
|
||||
|
||||
```bash
|
||||
opencli antigravity status # 检查 CDP 连接
|
||||
opencli antigravity send "hello" # 发送文本到当前 agent 聊天框
|
||||
opencli antigravity read # 读取整个聊天记录面板
|
||||
opencli antigravity new # 清空聊天、开启新对话
|
||||
opencli antigravity dump # 导出 DOM 和快照调试信息
|
||||
opencli antigravity extract-code # 自动抽取 AI 回复中的代码块
|
||||
opencli antigravity model claude # 切换底层模型
|
||||
opencli antigravity watch # 流式监听增量消息
|
||||
```
|
||||
@@ -0,0 +1,82 @@
|
||||
# Plugin System & Advanced Features
|
||||
|
||||
## Management Commands
|
||||
|
||||
```bash
|
||||
opencli list # List all commands (including External CLIs)
|
||||
opencli list --json # JSON output
|
||||
opencli list -f yaml # YAML output
|
||||
opencli install <name> # Auto-install an external CLI (e.g., gh, obsidian)
|
||||
opencli register <name> # Register a local custom CLI for unified discovery
|
||||
opencli validate # Validate all CLI definitions
|
||||
opencli validate bilibili # Validate specific site
|
||||
opencli doctor # Diagnose browser bridge (auto-starts daemon, includes live test)
|
||||
```
|
||||
|
||||
## AI Agent Workflow
|
||||
|
||||
```bash
|
||||
# Deep Explore: network intercept → response analysis → capability inference
|
||||
opencli explore <url> --site <name>
|
||||
|
||||
# Synthesize: generate evaluate-based YAML pipelines from explore artifacts
|
||||
opencli synthesize <site>
|
||||
|
||||
# Generate: one-shot explore → synthesize → register
|
||||
opencli generate <url> --goal "hot"
|
||||
|
||||
# Record: YOU operate the page, opencli captures every API call → YAML candidates
|
||||
opencli record <url> # 录制,site name 从域名推断
|
||||
opencli record <url> --site mysite # 指定 site name
|
||||
opencli record <url> --timeout 120000 # 自定义超时(毫秒,默认 60000)
|
||||
opencli record <url> --poll 1000 # 缩短轮询间隔(毫秒,默认 2000)
|
||||
opencli record <url> --out .opencli/record/x # 自定义输出目录
|
||||
|
||||
# Strategy Cascade: auto-probe PUBLIC → COOKIE → HEADER
|
||||
opencli cascade <api-url>
|
||||
|
||||
# Explore with interactive fuzzing (click buttons to trigger lazy APIs)
|
||||
opencli explore <url> --auto --click "字幕,CC,评论"
|
||||
|
||||
# Validate: validate adapter definitions
|
||||
opencli validate
|
||||
```
|
||||
|
||||
## Output Formats
|
||||
|
||||
All built-in commands support `--format` / `-f` with `table`, `json`, `yaml`, `md`, and `csv`.
|
||||
|
||||
```bash
|
||||
opencli list -f yaml # YAML command registry
|
||||
opencli bilibili hot -f table # Default: rich table
|
||||
opencli bilibili hot -f json # JSON (pipe to jq, feed to AI agent)
|
||||
opencli bilibili hot -f yaml # YAML (readable structured output)
|
||||
opencli bilibili hot -f md # Markdown
|
||||
opencli bilibili hot -f csv # CSV
|
||||
```
|
||||
|
||||
## Verbose Mode
|
||||
|
||||
```bash
|
||||
opencli bilibili hot -v # Show each pipeline step and data flow
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENCLI_DAEMON_PORT` | 19825 | Daemon listen port |
|
||||
| `OPENCLI_BROWSER_CONNECT_TIMEOUT` | 30 | Browser connection timeout (sec) |
|
||||
| `OPENCLI_BROWSER_COMMAND_TIMEOUT` | 45 | Command execution timeout (sec) |
|
||||
| `OPENCLI_BROWSER_EXPLORE_TIMEOUT` | 120 | Explore timeout (sec) |
|
||||
| `OPENCLI_VERBOSE` | — | Show daemon/extension logs |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| `npx not found` | Install Node.js: `brew install node` |
|
||||
| `Extension not connected` | 1) Chrome must be open 2) Install opencli Browser Bridge extension |
|
||||
| `Target page context` error | Add `navigate:` step before `evaluate:` in YAML |
|
||||
| Empty table data | Check if evaluate returns correct data path |
|
||||
| Daemon issues | `curl localhost:19825/status` to check, `curl localhost:19825/logs` for extension logs |
|
||||
@@ -0,0 +1,149 @@
|
||||
# Public API Commands
|
||||
|
||||
Commands that work without browser or authentication.
|
||||
|
||||
## Hacker News
|
||||
|
||||
```bash
|
||||
opencli hackernews top --limit 10 # Top stories
|
||||
opencli hackernews new --limit 10 # Newest stories
|
||||
opencli hackernews best --limit 10 # Best stories
|
||||
opencli hackernews ask --limit 10 # Ask HN posts
|
||||
opencli hackernews show --limit 10 # Show HN posts
|
||||
opencli hackernews jobs --limit 10 # Job postings
|
||||
opencli hackernews search "rust" # 搜索 (query positional)
|
||||
opencli hackernews user dang # 用户资料 (username positional)
|
||||
```
|
||||
|
||||
## V2EX (Public Features)
|
||||
|
||||
```bash
|
||||
opencli v2ex hot --limit 10 # 热门话题
|
||||
opencli v2ex latest --limit 10 # 最新话题
|
||||
opencli v2ex topic 1024 # 主题详情 (id positional)
|
||||
opencli v2ex node python # 节点话题列表 (name positional)
|
||||
opencli v2ex nodes --limit 30 # 所有节点列表
|
||||
opencli v2ex member username # 用户资料 (username positional)
|
||||
opencli v2ex user username # 用户发帖列表 (username positional)
|
||||
opencli v2ex replies 1024 # 主题回复列表 (id positional)
|
||||
```
|
||||
|
||||
## BBC News
|
||||
|
||||
```bash
|
||||
opencli bbc news --limit 10 # BBC News RSS headlines
|
||||
```
|
||||
|
||||
## Sina Finance
|
||||
|
||||
```bash
|
||||
opencli sinafinance news --limit 10 --type 1 # 7x24实时快讯
|
||||
# Types: 0=全部 1=A股 2=宏观 3=公司 4=数据 5=市场 6=国际 7=观点 8=央行 9=其它
|
||||
```
|
||||
|
||||
## Lobsters
|
||||
|
||||
```bash
|
||||
opencli lobsters hot --limit 10 # 热门
|
||||
opencli lobsters newest --limit 10 # 最新
|
||||
opencli lobsters active --limit 10 # 活跃
|
||||
opencli lobsters tag rust # 按标签筛选 (tag positional)
|
||||
```
|
||||
|
||||
## Google
|
||||
|
||||
```bash
|
||||
opencli google news --limit 10 # 新闻
|
||||
opencli google search "AI" # 搜索 (query positional)
|
||||
opencli google suggest "AI" # 搜索建议 (query positional)
|
||||
opencli google trends # 趋势
|
||||
```
|
||||
|
||||
## DEV.to
|
||||
|
||||
```bash
|
||||
opencli devto top --limit 10 # 热门文章
|
||||
opencli devto tag javascript --limit 10 # 按标签 (tag positional)
|
||||
opencli devto user username # 用户文章 (username positional)
|
||||
```
|
||||
|
||||
## Steam
|
||||
|
||||
```bash
|
||||
opencli steam top-sellers --limit 10 # 热销游戏
|
||||
```
|
||||
|
||||
## Apple Podcasts
|
||||
|
||||
```bash
|
||||
opencli apple-podcasts top --limit 10 # 热门播客排行榜 (支持 --country us/cn/gb/jp)
|
||||
opencli apple-podcasts search "科技" # 搜索播客 (query positional)
|
||||
opencli apple-podcasts episodes 12345 # 播客剧集列表 (id positional)
|
||||
```
|
||||
|
||||
## arXiv
|
||||
|
||||
```bash
|
||||
opencli arxiv search "attention" # 搜索论文 (query positional)
|
||||
opencli arxiv paper 1706.03762 # 论文详情 (id positional)
|
||||
```
|
||||
|
||||
## StackOverflow
|
||||
|
||||
```bash
|
||||
opencli stackoverflow hot --limit 10 # 热门问题
|
||||
opencli stackoverflow search "typescript" # 搜索 (query positional)
|
||||
opencli stackoverflow bounties --limit 10 # 悬赏问题
|
||||
```
|
||||
|
||||
## Xiaoyuzhou (小宇宙)
|
||||
|
||||
```bash
|
||||
opencli xiaoyuzhou podcast 12345 # 播客资料 (id positional)
|
||||
opencli xiaoyuzhou podcast-episodes 12345 # 播客剧集列表 (id positional)
|
||||
opencli xiaoyuzhou episode 12345 # 单集详情 (id positional)
|
||||
```
|
||||
|
||||
## Wikipedia
|
||||
|
||||
```bash
|
||||
opencli wikipedia search "AI" # 搜索 (query positional)
|
||||
opencli wikipedia summary "Python" # 摘要 (title positional)
|
||||
```
|
||||
|
||||
## Bloomberg (RSS)
|
||||
|
||||
```bash
|
||||
opencli bloomberg main --limit 10 # Bloomberg 首页头条
|
||||
opencli bloomberg markets --limit 10 # 市场新闻
|
||||
opencli bloomberg tech --limit 10 # 科技新闻
|
||||
opencli bloomberg politics --limit 10 # 政治新闻
|
||||
opencli bloomberg economics --limit 10 # 经济新闻
|
||||
opencli bloomberg opinions --limit 10 # 观点
|
||||
opencli bloomberg industries --limit 10 # 行业新闻
|
||||
opencli bloomberg businessweek --limit 10 # Businessweek
|
||||
opencli bloomberg feeds # 列出所有 RSS feed 别名
|
||||
```
|
||||
|
||||
## Dictionary
|
||||
|
||||
```bash
|
||||
opencli dictionary search "serendipity" # 单词释义 (word positional)
|
||||
opencli dictionary synonyms "happy" # 近义词 (word positional)
|
||||
opencli dictionary examples "ubiquitous" # 例句 (word positional)
|
||||
```
|
||||
|
||||
## HuggingFace
|
||||
|
||||
```bash
|
||||
opencli hf top --limit 10 # 热门模型
|
||||
```
|
||||
|
||||
## Product Hunt
|
||||
|
||||
```bash
|
||||
opencli producthunt today --limit 10 # 今日产品
|
||||
opencli producthunt week --limit 10 # 本周产品
|
||||
opencli producthunt month --limit 10 # 本月产品
|
||||
opencli producthunt search "AI" # 搜索产品 (query positional)
|
||||
```
|
||||
+34
-24
@@ -1,10 +1,14 @@
|
||||
import { afterEach, describe, it, expect, vi } from 'vitest';
|
||||
import { BrowserBridge, __test__, generateStealthJs } from './browser/index.js';
|
||||
import { describe, it, expect, vi } from 'vitest';
|
||||
import { BrowserBridge, generateStealthJs } from './browser/index.js';
|
||||
import { extractTabEntries, diffTabIndexes, appendLimited } from './browser/tabs.js';
|
||||
import { withTimeoutMs } from './runtime.js';
|
||||
import { __test__ as cdpTest } from './browser/cdp.js';
|
||||
import { isRetryableSettleError } from './browser/page.js';
|
||||
import * as daemonClient from './browser/daemon-client.js';
|
||||
|
||||
describe('browser helpers', () => {
|
||||
it('extracts tab entries from string snapshots', () => {
|
||||
const entries = __test__.extractTabEntries('Tab 0 https://example.com\nTab 1 Chrome Extension');
|
||||
const entries = extractTabEntries('Tab 0 https://example.com\nTab 1 Chrome Extension');
|
||||
|
||||
expect(entries).toEqual([
|
||||
{ index: 0, identity: 'https://example.com' },
|
||||
@@ -13,7 +17,7 @@ describe('browser helpers', () => {
|
||||
});
|
||||
|
||||
it('extracts tab entries from MCP markdown format', () => {
|
||||
const entries = __test__.extractTabEntries(
|
||||
const entries = extractTabEntries(
|
||||
'- 0: (current) [Playwright MCP extension](chrome-extension://abc/connect.html)\n- 1: [知乎 - 首页](https://www.zhihu.com/)'
|
||||
);
|
||||
|
||||
@@ -24,7 +28,7 @@ describe('browser helpers', () => {
|
||||
});
|
||||
|
||||
it('closes only tabs that were opened during the session', () => {
|
||||
const tabsToClose = __test__.diffTabIndexes(
|
||||
const tabsToClose = diffTabIndexes(
|
||||
['https://example.com', 'Chrome Extension'],
|
||||
[
|
||||
{ index: 0, identity: 'https://example.com' },
|
||||
@@ -38,15 +42,21 @@ describe('browser helpers', () => {
|
||||
});
|
||||
|
||||
it('keeps only the tail of stderr buffers', () => {
|
||||
expect(__test__.appendLimited('12345', '67890', 8)).toBe('34567890');
|
||||
expect(appendLimited('12345', '67890', 8)).toBe('34567890');
|
||||
});
|
||||
|
||||
it('times out slow promises', async () => {
|
||||
await expect(__test__.withTimeoutMs(new Promise(() => {}), 10, 'timeout')).rejects.toThrow('timeout');
|
||||
await expect(withTimeoutMs(new Promise(() => {}), 10, 'timeout')).rejects.toThrow('timeout');
|
||||
});
|
||||
|
||||
it('retries settle only for target-invalidated errors', () => {
|
||||
expect(isRetryableSettleError(new Error('{"code":-32000,"message":"Inspected target navigated or closed"}'))).toBe(true);
|
||||
expect(isRetryableSettleError(new Error('attach failed: target no longer exists'))).toBe(false);
|
||||
expect(isRetryableSettleError(new Error('malformed exec payload'))).toBe(false);
|
||||
});
|
||||
|
||||
it('prefers the real Electron app target over DevTools and blank pages', () => {
|
||||
const target = __test__.selectCDPTarget([
|
||||
const target = cdpTest.selectCDPTarget([
|
||||
{
|
||||
type: 'page',
|
||||
title: 'DevTools - localhost:9224',
|
||||
@@ -73,7 +83,7 @@ describe('browser helpers', () => {
|
||||
it('honors OPENCLI_CDP_TARGET when multiple inspectable targets exist', () => {
|
||||
vi.stubEnv('OPENCLI_CDP_TARGET', 'codex');
|
||||
|
||||
const target = __test__.selectCDPTarget([
|
||||
const target = cdpTest.selectCDPTarget([
|
||||
{
|
||||
type: 'app',
|
||||
title: 'Cursor',
|
||||
@@ -94,43 +104,43 @@ describe('browser helpers', () => {
|
||||
|
||||
describe('BrowserBridge state', () => {
|
||||
it('transitions to closed after close()', async () => {
|
||||
const mcp = new BrowserBridge();
|
||||
const bridge = new BrowserBridge();
|
||||
|
||||
expect(mcp.state).toBe('idle');
|
||||
expect(bridge.state).toBe('idle');
|
||||
|
||||
await mcp.close();
|
||||
await bridge.close();
|
||||
|
||||
expect(mcp.state).toBe('closed');
|
||||
expect(bridge.state).toBe('closed');
|
||||
});
|
||||
|
||||
it('rejects connect() after the session has been closed', async () => {
|
||||
const mcp = new BrowserBridge();
|
||||
await mcp.close();
|
||||
const bridge = new BrowserBridge();
|
||||
await bridge.close();
|
||||
|
||||
await expect(mcp.connect()).rejects.toThrow('Session is closed');
|
||||
await expect(bridge.connect()).rejects.toThrow('Session is closed');
|
||||
});
|
||||
|
||||
it('rejects connect() while already connecting', async () => {
|
||||
const mcp = new BrowserBridge();
|
||||
(mcp as any)._state = 'connecting';
|
||||
const bridge = new BrowserBridge();
|
||||
(bridge as any)._state = 'connecting';
|
||||
|
||||
await expect(mcp.connect()).rejects.toThrow('Already connecting');
|
||||
await expect(bridge.connect()).rejects.toThrow('Already connecting');
|
||||
});
|
||||
|
||||
it('rejects connect() while closing', async () => {
|
||||
const mcp = new BrowserBridge();
|
||||
(mcp as any)._state = 'closing';
|
||||
const bridge = new BrowserBridge();
|
||||
(bridge as any)._state = 'closing';
|
||||
|
||||
await expect(mcp.connect()).rejects.toThrow('Session is closing');
|
||||
await expect(bridge.connect()).rejects.toThrow('Session is closing');
|
||||
});
|
||||
|
||||
it('fails fast when daemon is running but extension is disconnected', async () => {
|
||||
vi.spyOn(daemonClient, 'isExtensionConnected').mockResolvedValue(false);
|
||||
vi.spyOn(daemonClient, 'isDaemonRunning').mockResolvedValue(true);
|
||||
|
||||
const mcp = new BrowserBridge();
|
||||
const bridge = new BrowserBridge();
|
||||
|
||||
await expect(mcp.connect()).rejects.toThrow('Browser Extension is not connected');
|
||||
await expect(bridge.connect({ timeout: 0.1 })).rejects.toThrow('Browser Extension is not connected');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
/**
|
||||
* BasePage — shared IPage method implementations for DOM helpers.
|
||||
*
|
||||
* Both Page (daemon-backed) and CDPPage (direct CDP) execute JS the same way
|
||||
* for DOM operations. This base class deduplicates ~200 lines of identical
|
||||
* click/type/scroll/wait/snapshot/interceptor methods.
|
||||
*
|
||||
* Subclasses implement the transport-specific methods: goto, evaluate,
|
||||
* getCookies, screenshot, tabs, etc.
|
||||
*/
|
||||
|
||||
import type { BrowserCookie, IPage, ScreenshotOptions, SnapshotOptions, WaitOptions } from '../types.js';
|
||||
import { generateSnapshotJs, scrollToRefJs, getFormStateJs } from './dom-snapshot.js';
|
||||
import {
|
||||
clickJs,
|
||||
typeTextJs,
|
||||
pressKeyJs,
|
||||
waitForTextJs,
|
||||
waitForCaptureJs,
|
||||
waitForSelectorJs,
|
||||
scrollJs,
|
||||
autoScrollJs,
|
||||
networkRequestsJs,
|
||||
waitForDomStableJs,
|
||||
} from './dom-helpers.js';
|
||||
import { formatSnapshot } from '../snapshotFormatter.js';
|
||||
|
||||
export abstract class BasePage implements IPage {
|
||||
protected _lastUrl: string | null = null;
|
||||
|
||||
// ── Transport-specific methods (must be implemented by subclasses) ──
|
||||
|
||||
abstract goto(url: string, options?: { waitUntil?: 'load' | 'none'; settleMs?: number }): Promise<void>;
|
||||
abstract evaluate(js: string): Promise<unknown>;
|
||||
abstract getCookies(opts?: { domain?: string; url?: string }): Promise<BrowserCookie[]>;
|
||||
abstract screenshot(options?: ScreenshotOptions): Promise<string>;
|
||||
abstract tabs(): Promise<unknown[]>;
|
||||
abstract closeTab(index?: number): Promise<void>;
|
||||
abstract newTab(): Promise<void>;
|
||||
abstract selectTab(index: number): Promise<void>;
|
||||
|
||||
// ── Shared DOM helper implementations ──
|
||||
|
||||
async click(ref: string): Promise<void> {
|
||||
await this.evaluate(clickJs(ref));
|
||||
}
|
||||
|
||||
async typeText(ref: string, text: string): Promise<void> {
|
||||
await this.evaluate(typeTextJs(ref, text));
|
||||
}
|
||||
|
||||
async pressKey(key: string): Promise<void> {
|
||||
await this.evaluate(pressKeyJs(key));
|
||||
}
|
||||
|
||||
async scrollTo(ref: string): Promise<unknown> {
|
||||
return this.evaluate(scrollToRefJs(ref));
|
||||
}
|
||||
|
||||
async getFormState(): Promise<Record<string, unknown>> {
|
||||
return (await this.evaluate(getFormStateJs())) as Record<string, unknown>;
|
||||
}
|
||||
|
||||
async scroll(direction: string = 'down', amount: number = 500): Promise<void> {
|
||||
await this.evaluate(scrollJs(direction, amount));
|
||||
}
|
||||
|
||||
async autoScroll(options?: { times?: number; delayMs?: number }): Promise<void> {
|
||||
const times = options?.times ?? 3;
|
||||
const delayMs = options?.delayMs ?? 2000;
|
||||
await this.evaluate(autoScrollJs(times, delayMs));
|
||||
}
|
||||
|
||||
async networkRequests(includeStatic: boolean = false): Promise<unknown[]> {
|
||||
const result = await this.evaluate(networkRequestsJs(includeStatic));
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
async consoleMessages(_level: string = 'info'): Promise<unknown[]> {
|
||||
return [];
|
||||
}
|
||||
|
||||
async wait(options: number | WaitOptions): Promise<void> {
|
||||
if (typeof options === 'number') {
|
||||
if (options >= 1) {
|
||||
try {
|
||||
const maxMs = options * 1000;
|
||||
await this.evaluate(waitForDomStableJs(maxMs, Math.min(500, maxMs)));
|
||||
return;
|
||||
} catch {
|
||||
// Fallback: fixed sleep
|
||||
}
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, options * 1000));
|
||||
return;
|
||||
}
|
||||
if (typeof options.time === 'number') {
|
||||
await new Promise(resolve => setTimeout(resolve, options.time! * 1000));
|
||||
return;
|
||||
}
|
||||
if (options.selector) {
|
||||
const timeout = (options.timeout ?? 10) * 1000;
|
||||
await this.evaluate(waitForSelectorJs(options.selector, timeout));
|
||||
return;
|
||||
}
|
||||
if (options.text) {
|
||||
const timeout = (options.timeout ?? 30) * 1000;
|
||||
await this.evaluate(waitForTextJs(options.text, timeout));
|
||||
}
|
||||
}
|
||||
|
||||
async snapshot(opts: SnapshotOptions = {}): Promise<unknown> {
|
||||
const snapshotJs = generateSnapshotJs({
|
||||
viewportExpand: opts.viewportExpand ?? 800,
|
||||
maxDepth: Math.max(1, Math.min(Number(opts.maxDepth) || 50, 200)),
|
||||
interactiveOnly: opts.interactive ?? false,
|
||||
maxTextLength: opts.maxTextLength ?? 120,
|
||||
includeScrollInfo: true,
|
||||
bboxDedup: true,
|
||||
});
|
||||
|
||||
try {
|
||||
return await this.evaluate(snapshotJs);
|
||||
} catch {
|
||||
return this._basicSnapshot(opts);
|
||||
}
|
||||
}
|
||||
|
||||
async getCurrentUrl(): Promise<string | null> {
|
||||
if (this._lastUrl) return this._lastUrl;
|
||||
try {
|
||||
const current = await this.evaluate('window.location.href');
|
||||
if (typeof current === 'string' && current) {
|
||||
this._lastUrl = current;
|
||||
return current;
|
||||
}
|
||||
} catch {
|
||||
// Best-effort
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async installInterceptor(pattern: string): Promise<void> {
|
||||
const { generateInterceptorJs } = await import('../interceptor.js');
|
||||
await this.evaluate(generateInterceptorJs(JSON.stringify(pattern), {
|
||||
arrayName: '__opencli_xhr',
|
||||
patchGuard: '__opencli_interceptor_patched',
|
||||
}));
|
||||
}
|
||||
|
||||
async getInterceptedRequests(): Promise<unknown[]> {
|
||||
const { generateReadInterceptedJs } = await import('../interceptor.js');
|
||||
const result = await this.evaluate(generateReadInterceptedJs('__opencli_xhr'));
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
async waitForCapture(timeout: number = 10): Promise<void> {
|
||||
const maxMs = timeout * 1000;
|
||||
await this.evaluate(waitForCaptureJs(maxMs));
|
||||
}
|
||||
|
||||
/** Fallback basic snapshot */
|
||||
protected async _basicSnapshot(opts: Pick<SnapshotOptions, 'interactive' | 'compact' | 'maxDepth' | 'raw'> = {}): Promise<unknown> {
|
||||
const maxDepth = Math.max(1, Math.min(Number(opts.maxDepth) || 50, 200));
|
||||
const code = `
|
||||
(async () => {
|
||||
function buildTree(node, depth) {
|
||||
if (depth > ${maxDepth}) return '';
|
||||
const role = node.getAttribute?.('role') || node.tagName?.toLowerCase() || 'generic';
|
||||
const name = node.getAttribute?.('aria-label') || node.getAttribute?.('alt') || node.textContent?.trim().slice(0, 80) || '';
|
||||
const isInteractive = ['a', 'button', 'input', 'select', 'textarea'].includes(node.tagName?.toLowerCase()) || node.getAttribute?.('tabindex') != null;
|
||||
|
||||
${opts.interactive ? 'if (!isInteractive && !node.children?.length) return "";' : ''}
|
||||
|
||||
let indent = ' '.repeat(depth);
|
||||
let line = indent + role;
|
||||
if (name) line += ' "' + name.replace(/"/g, '\\\\\\"') + '"';
|
||||
if (node.tagName?.toLowerCase() === 'a' && node.href) line += ' [' + node.href + ']';
|
||||
if (node.tagName?.toLowerCase() === 'input') line += ' [' + (node.type || 'text') + ']';
|
||||
|
||||
let result = line + '\\n';
|
||||
if (node.children) {
|
||||
for (const child of node.children) {
|
||||
result += buildTree(child, depth + 1);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
return buildTree(document.body, 0);
|
||||
})()
|
||||
`;
|
||||
const raw = await this.evaluate(code);
|
||||
if (opts.raw) return raw;
|
||||
if (typeof raw === 'string') return formatSnapshot(raw, opts);
|
||||
return raw;
|
||||
}
|
||||
}
|
||||
@@ -57,21 +57,30 @@ export class BrowserBridge implements IBrowserFactory {
|
||||
}
|
||||
|
||||
private async _ensureDaemon(timeoutSeconds?: number): Promise<void> {
|
||||
// Use default if not provided, zero, or negative
|
||||
const effectiveSeconds = (timeoutSeconds && timeoutSeconds > 0) ? timeoutSeconds : Math.ceil(DAEMON_SPAWN_TIMEOUT / 1000);
|
||||
const timeoutMs = effectiveSeconds * 1000;
|
||||
|
||||
// Fast path: extension already connected
|
||||
if (await isExtensionConnected()) return;
|
||||
|
||||
// Daemon running but no extension — wait for extension with progress
|
||||
if (await isDaemonRunning()) {
|
||||
if (process.env.OPENCLI_VERBOSE || process.stderr.isTTY) {
|
||||
process.stderr.write('⏳ Waiting for Chrome extension to connect...\n');
|
||||
process.stderr.write(' Make sure Chrome is open and the OpenCLI extension is enabled.\n');
|
||||
}
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
if (await isExtensionConnected()) return;
|
||||
}
|
||||
throw new Error(
|
||||
'Daemon is running but the Browser Extension is not connected.\n' +
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome.',
|
||||
);
|
||||
}
|
||||
|
||||
// Find daemon relative to this file — works for both:
|
||||
// npx tsx src/main.ts → src/browser/mcp.ts → src/daemon.ts
|
||||
// node dist/main.js → dist/browser/mcp.js → dist/daemon.js
|
||||
// No daemon — spawn one
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const parentDir = path.resolve(__dirname, '..');
|
||||
const daemonTs = path.join(parentDir, 'daemon.ts');
|
||||
@@ -79,12 +88,10 @@ export class BrowserBridge implements IBrowserFactory {
|
||||
const isTs = fs.existsSync(daemonTs);
|
||||
const daemonPath = isTs ? daemonTs : daemonJs;
|
||||
|
||||
if (process.env.OPENCLI_VERBOSE) {
|
||||
console.error(`[opencli] Starting daemon (${isTs ? 'ts' : 'js'})...`);
|
||||
if (process.env.OPENCLI_VERBOSE || process.stderr.isTTY) {
|
||||
process.stderr.write('⏳ Starting daemon...\n');
|
||||
}
|
||||
|
||||
// For compiled .js, use the current node binary directly (fast).
|
||||
// For .ts dev mode, node can't run .ts files — use tsx via --import.
|
||||
const spawnArgs = isTs
|
||||
? [process.execPath, '--import', 'tsx/esm', daemonPath]
|
||||
: [process.execPath, daemonPath];
|
||||
@@ -96,14 +103,13 @@ export class BrowserBridge implements IBrowserFactory {
|
||||
});
|
||||
this._daemonProc.unref();
|
||||
|
||||
// Wait for daemon to be ready AND extension to connect
|
||||
// Wait for daemon + extension with faster polling
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 300));
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
if (await isExtensionConnected()) return;
|
||||
}
|
||||
|
||||
// Daemon might be up but extension not connected — give a useful error
|
||||
if (await isDaemonRunning()) {
|
||||
throw new Error(
|
||||
'Daemon is running but the Browser Extension is not connected.\n' +
|
||||
+19
-115
@@ -11,22 +11,14 @@
|
||||
import { WebSocket, type RawData } from 'ws';
|
||||
import { request as httpRequest } from 'node:http';
|
||||
import { request as httpsRequest } from 'node:https';
|
||||
import type { BrowserCookie, IPage, ScreenshotOptions, SnapshotOptions, WaitOptions } from '../types.js';
|
||||
import type { BrowserCookie, IPage, ScreenshotOptions } from '../types.js';
|
||||
import type { IBrowserFactory } from '../runtime.js';
|
||||
import { wrapForEval } from './utils.js';
|
||||
import { generateSnapshotJs, scrollToRefJs, getFormStateJs } from './dom-snapshot.js';
|
||||
import { generateStealthJs } from './stealth.js';
|
||||
import {
|
||||
clickJs,
|
||||
typeTextJs,
|
||||
pressKeyJs,
|
||||
waitForTextJs,
|
||||
scrollJs,
|
||||
autoScrollJs,
|
||||
networkRequestsJs,
|
||||
waitForDomStableJs,
|
||||
} from './dom-helpers.js';
|
||||
import { waitForDomStableJs } from './dom-helpers.js';
|
||||
import { isRecord, saveBase64ToFile } from '../utils.js';
|
||||
import { getAllElectronApps } from '../electron-apps.js';
|
||||
import { BasePage } from './base-page.js';
|
||||
|
||||
export interface CDPTarget {
|
||||
type?: string;
|
||||
@@ -54,11 +46,11 @@ export class CDPBridge implements IBrowserFactory {
|
||||
private _pending = new Map<number, { resolve: (val: unknown) => void; reject: (err: Error) => void; timer: ReturnType<typeof setTimeout> }>();
|
||||
private _eventListeners = new Map<string, Set<(params: unknown) => void>>();
|
||||
|
||||
async connect(opts?: { timeout?: number; workspace?: string }): Promise<IPage> {
|
||||
async connect(opts?: { timeout?: number; workspace?: string; cdpEndpoint?: string }): Promise<IPage> {
|
||||
if (this._ws) throw new Error('CDPBridge is already connected. Call close() before reconnecting.');
|
||||
|
||||
const endpoint = process.env.OPENCLI_CDP_ENDPOINT;
|
||||
if (!endpoint) throw new Error('OPENCLI_CDP_ENDPOINT is not set');
|
||||
const endpoint = opts?.cdpEndpoint ?? process.env.OPENCLI_CDP_ENDPOINT;
|
||||
if (!endpoint) throw new Error('CDP endpoint not provided (pass cdpEndpoint or set OPENCLI_CDP_ENDPOINT)');
|
||||
|
||||
let wsUrl = endpoint;
|
||||
if (endpoint.startsWith('http')) {
|
||||
@@ -171,10 +163,11 @@ export class CDPBridge implements IBrowserFactory {
|
||||
}
|
||||
}
|
||||
|
||||
class CDPPage implements IPage {
|
||||
class CDPPage extends BasePage {
|
||||
private _pageEnabled = false;
|
||||
private _lastUrl: string | null = null;
|
||||
constructor(private bridge: CDPBridge) {}
|
||||
constructor(private bridge: CDPBridge) {
|
||||
super();
|
||||
}
|
||||
|
||||
async goto(url: string, options?: { waitUntil?: 'load' | 'none'; settleMs?: number }): Promise<void> {
|
||||
if (!this._pageEnabled) {
|
||||
@@ -213,64 +206,6 @@ class CDPPage implements IPage {
|
||||
: cookies;
|
||||
}
|
||||
|
||||
async snapshot(opts: SnapshotOptions = {}): Promise<unknown> {
|
||||
const snapshotJs = generateSnapshotJs({
|
||||
viewportExpand: opts.viewportExpand ?? 800,
|
||||
maxDepth: Math.max(1, Math.min(Number(opts.maxDepth) || 50, 200)),
|
||||
interactiveOnly: opts.interactive ?? false,
|
||||
maxTextLength: opts.maxTextLength ?? 120,
|
||||
includeScrollInfo: true,
|
||||
bboxDedup: true,
|
||||
});
|
||||
return this.evaluate(snapshotJs);
|
||||
}
|
||||
|
||||
async click(ref: string): Promise<void> {
|
||||
await this.evaluate(clickJs(ref));
|
||||
}
|
||||
|
||||
async typeText(ref: string, text: string): Promise<void> {
|
||||
await this.evaluate(typeTextJs(ref, text));
|
||||
}
|
||||
|
||||
async pressKey(key: string): Promise<void> {
|
||||
await this.evaluate(pressKeyJs(key));
|
||||
}
|
||||
|
||||
async scrollTo(ref: string): Promise<unknown> {
|
||||
return this.evaluate(scrollToRefJs(ref));
|
||||
}
|
||||
|
||||
async getFormState(): Promise<Record<string, unknown>> {
|
||||
return (await this.evaluate(getFormStateJs())) as Record<string, unknown>;
|
||||
}
|
||||
|
||||
async wait(options: number | WaitOptions): Promise<void> {
|
||||
if (typeof options === 'number') {
|
||||
await new Promise((resolve) => setTimeout(resolve, options * 1000));
|
||||
return;
|
||||
}
|
||||
if (typeof options.time === 'number') {
|
||||
const waitTime = options.time;
|
||||
await new Promise((resolve) => setTimeout(resolve, waitTime * 1000));
|
||||
return;
|
||||
}
|
||||
if (options.text) {
|
||||
const timeout = (options.timeout ?? 30) * 1000;
|
||||
await this.evaluate(waitForTextJs(options.text, timeout));
|
||||
}
|
||||
}
|
||||
|
||||
async scroll(direction: string = 'down', amount: number = 500): Promise<void> {
|
||||
await this.evaluate(scrollJs(direction, amount));
|
||||
}
|
||||
|
||||
async autoScroll(options?: { times?: number; delayMs?: number }): Promise<void> {
|
||||
const times = options?.times ?? 3;
|
||||
const delayMs = options?.delayMs ?? 2000;
|
||||
await this.evaluate(autoScrollJs(times, delayMs));
|
||||
}
|
||||
|
||||
async screenshot(options: ScreenshotOptions = {}): Promise<string> {
|
||||
const result = await this.bridge.send('Page.captureScreenshot', {
|
||||
format: options.format ?? 'png',
|
||||
@@ -284,11 +219,6 @@ class CDPPage implements IPage {
|
||||
return base64;
|
||||
}
|
||||
|
||||
async networkRequests(includeStatic: boolean = false): Promise<unknown[]> {
|
||||
const result = await this.evaluate(networkRequestsJs(includeStatic));
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
async tabs(): Promise<unknown[]> {
|
||||
return [];
|
||||
}
|
||||
@@ -304,28 +234,6 @@ class CDPPage implements IPage {
|
||||
async selectTab(_index: number): Promise<void> {
|
||||
// Not supported in direct CDP mode
|
||||
}
|
||||
|
||||
async consoleMessages(_level?: string): Promise<unknown[]> {
|
||||
return [];
|
||||
}
|
||||
|
||||
async getCurrentUrl(): Promise<string | null> {
|
||||
return this._lastUrl;
|
||||
}
|
||||
|
||||
async installInterceptor(pattern: string): Promise<void> {
|
||||
const { generateInterceptorJs } = await import('../interceptor.js');
|
||||
await this.evaluate(generateInterceptorJs(JSON.stringify(pattern), {
|
||||
arrayName: '__opencli_xhr',
|
||||
patchGuard: '__opencli_interceptor_patched',
|
||||
}));
|
||||
}
|
||||
|
||||
async getInterceptedRequests(): Promise<unknown[]> {
|
||||
const { generateReadInterceptedJs } = await import('../interceptor.js');
|
||||
const result = await this.evaluate(generateReadInterceptedJs('__opencli_xhr'));
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
}
|
||||
|
||||
function isCookie(value: unknown): value is BrowserCookie {
|
||||
@@ -383,19 +291,15 @@ function scoreCDPTarget(target: CDPTarget, preferredPattern?: RegExp): number {
|
||||
if (url === '' || url === 'about:blank') score -= 40;
|
||||
|
||||
if (title && title !== 'devtools') score += 25;
|
||||
if (title.includes('antigravity')) score += 120;
|
||||
if (title.includes('codex')) score += 120;
|
||||
if (title.includes('cursor')) score += 120;
|
||||
if (title.includes('chatwise')) score += 120;
|
||||
if (title.includes('notion')) score += 120;
|
||||
if (title.includes('discord')) score += 120;
|
||||
|
||||
if (url.includes('antigravity')) score += 100;
|
||||
if (url.includes('codex')) score += 100;
|
||||
if (url.includes('cursor')) score += 100;
|
||||
if (url.includes('chatwise')) score += 100;
|
||||
if (url.includes('notion')) score += 100;
|
||||
if (url.includes('discord')) score += 100;
|
||||
// Boost score for known Electron app names from the registry (builtin + user-defined)
|
||||
const appNames = Object.values(getAllElectronApps()).map(a => (a.displayName ?? a.processName).toLowerCase());
|
||||
for (const name of appNames) {
|
||||
if (title.includes(name)) { score += 120; break; }
|
||||
}
|
||||
for (const name of appNames) {
|
||||
if (url.includes(name)) { score += 100; break; }
|
||||
}
|
||||
|
||||
return score;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import {
|
||||
fetchDaemonStatus,
|
||||
isDaemonRunning,
|
||||
isExtensionConnected,
|
||||
requestDaemonShutdown,
|
||||
} from './daemon-client.js';
|
||||
|
||||
describe('daemon-client', () => {
|
||||
beforeEach(() => {
|
||||
vi.stubGlobal('fetch', vi.fn());
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('fetchDaemonStatus sends the shared status request and returns parsed data', async () => {
|
||||
const status = {
|
||||
ok: true,
|
||||
pid: 123,
|
||||
uptime: 10,
|
||||
extensionConnected: true,
|
||||
extensionVersion: '1.2.3',
|
||||
pending: 0,
|
||||
lastCliRequestTime: Date.now(),
|
||||
memoryMB: 32,
|
||||
port: 19825,
|
||||
};
|
||||
const fetchMock = vi.mocked(fetch);
|
||||
fetchMock.mockResolvedValue({
|
||||
ok: true,
|
||||
json: () => Promise.resolve(status),
|
||||
} as Response);
|
||||
|
||||
await expect(fetchDaemonStatus()).resolves.toEqual(status);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
expect.stringMatching(/\/status$/),
|
||||
expect.objectContaining({
|
||||
headers: expect.objectContaining({ 'X-OpenCLI': '1' }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('fetchDaemonStatus returns null on network failure', async () => {
|
||||
vi.mocked(fetch).mockRejectedValue(new Error('ECONNREFUSED'));
|
||||
|
||||
await expect(fetchDaemonStatus()).resolves.toBeNull();
|
||||
});
|
||||
|
||||
it('requestDaemonShutdown POSTs to the shared shutdown endpoint', async () => {
|
||||
const fetchMock = vi.mocked(fetch);
|
||||
fetchMock.mockResolvedValue({ ok: true } as Response);
|
||||
|
||||
await expect(requestDaemonShutdown()).resolves.toBe(true);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
expect.stringMatching(/\/shutdown$/),
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
headers: expect.objectContaining({ 'X-OpenCLI': '1' }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('isDaemonRunning reflects shared status availability', async () => {
|
||||
vi.mocked(fetch).mockResolvedValue({
|
||||
ok: true,
|
||||
json: () =>
|
||||
Promise.resolve({
|
||||
ok: true,
|
||||
pid: 123,
|
||||
uptime: 10,
|
||||
extensionConnected: false,
|
||||
pending: 0,
|
||||
lastCliRequestTime: Date.now(),
|
||||
memoryMB: 16,
|
||||
port: 19825,
|
||||
}),
|
||||
} as Response);
|
||||
|
||||
await expect(isDaemonRunning()).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it('isExtensionConnected reflects shared status payload', async () => {
|
||||
vi.mocked(fetch).mockResolvedValue({
|
||||
ok: true,
|
||||
json: () =>
|
||||
Promise.resolve({
|
||||
ok: true,
|
||||
pid: 123,
|
||||
uptime: 10,
|
||||
extensionConnected: false,
|
||||
pending: 0,
|
||||
lastCliRequestTime: Date.now(),
|
||||
memoryMB: 16,
|
||||
port: 19825,
|
||||
}),
|
||||
} as Response);
|
||||
|
||||
await expect(isExtensionConnected()).resolves.toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -6,9 +6,12 @@
|
||||
|
||||
import { DEFAULT_DAEMON_PORT } from '../constants.js';
|
||||
import type { BrowserSessionInfo } from '../types.js';
|
||||
import { sleep } from '../utils.js';
|
||||
import { isTransientBrowserError } from './errors.js';
|
||||
|
||||
const DAEMON_PORT = parseInt(process.env.OPENCLI_DAEMON_PORT ?? String(DEFAULT_DAEMON_PORT), 10);
|
||||
const DAEMON_URL = `http://127.0.0.1:${DAEMON_PORT}`;
|
||||
const OPENCLI_HEADERS = { 'X-OpenCLI': '1' };
|
||||
|
||||
let _idCounter = 0;
|
||||
|
||||
@@ -18,7 +21,7 @@ function generateId(): string {
|
||||
|
||||
export interface DaemonCommand {
|
||||
id: string;
|
||||
action: 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions';
|
||||
action: 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions' | 'set-file-input' | 'cdp';
|
||||
tabId?: number;
|
||||
code?: string;
|
||||
workspace?: string;
|
||||
@@ -29,6 +32,13 @@ export interface DaemonCommand {
|
||||
format?: 'png' | 'jpeg';
|
||||
quality?: number;
|
||||
fullPage?: boolean;
|
||||
|
||||
/** Local file paths for set-file-input action */
|
||||
files?: string[];
|
||||
/** CSS selector for file input element (set-file-input action) */
|
||||
selector?: string;
|
||||
cdpMethod?: string;
|
||||
cdpParams?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface DaemonResult {
|
||||
@@ -38,42 +48,65 @@ export interface DaemonResult {
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if daemon is running.
|
||||
*/
|
||||
export async function isDaemonRunning(): Promise<boolean> {
|
||||
export interface DaemonStatus {
|
||||
ok: boolean;
|
||||
pid: number;
|
||||
uptime: number;
|
||||
extensionConnected: boolean;
|
||||
extensionVersion?: string;
|
||||
pending: number;
|
||||
lastCliRequestTime: number;
|
||||
memoryMB: number;
|
||||
port: number;
|
||||
}
|
||||
|
||||
async function requestDaemon(pathname: string, init?: RequestInit & { timeout?: number }): Promise<Response> {
|
||||
const { timeout = 2000, headers, ...rest } = init ?? {};
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeout);
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), 2000);
|
||||
const res = await fetch(`${DAEMON_URL}/status`, {
|
||||
headers: { 'X-OpenCLI': '1' },
|
||||
return await fetch(`${DAEMON_URL}${pathname}`, {
|
||||
...rest,
|
||||
headers: { ...OPENCLI_HEADERS, ...headers },
|
||||
signal: controller.signal,
|
||||
});
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchDaemonStatus(opts?: { timeout?: number }): Promise<DaemonStatus | null> {
|
||||
try {
|
||||
const res = await requestDaemon('/status', { timeout: opts?.timeout ?? 2000 });
|
||||
if (!res.ok) return null;
|
||||
return await res.json() as DaemonStatus;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function requestDaemonShutdown(opts?: { timeout?: number }): Promise<boolean> {
|
||||
try {
|
||||
const res = await requestDaemon('/shutdown', { method: 'POST', timeout: opts?.timeout ?? 5000 });
|
||||
return res.ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if daemon is running.
|
||||
*/
|
||||
export async function isDaemonRunning(): Promise<boolean> {
|
||||
return (await fetchDaemonStatus()) !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if daemon is running AND the extension is connected.
|
||||
*/
|
||||
export async function isExtensionConnected(): Promise<boolean> {
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), 2000);
|
||||
const res = await fetch(`${DAEMON_URL}/status`, {
|
||||
headers: { 'X-OpenCLI': '1' },
|
||||
signal: controller.signal,
|
||||
});
|
||||
clearTimeout(timer);
|
||||
if (!res.ok) return false;
|
||||
const data = await res.json() as { extensionConnected?: boolean };
|
||||
return !!data.extensionConnected;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
const status = await fetchDaemonStatus();
|
||||
return !!status?.extensionConnected;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -92,29 +125,20 @@ export async function sendCommand(
|
||||
const id = generateId();
|
||||
const command: DaemonCommand = { id, action, ...params };
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), 30000);
|
||||
|
||||
const res = await fetch(`${DAEMON_URL}/command`, {
|
||||
const res = await requestDaemon('/command', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', 'X-OpenCLI': '1' },
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(command),
|
||||
signal: controller.signal,
|
||||
timeout: 30000,
|
||||
});
|
||||
clearTimeout(timer);
|
||||
|
||||
const result = (await res.json()) as DaemonResult;
|
||||
|
||||
if (!result.ok) {
|
||||
// Check if error is a transient extension issue worth retrying
|
||||
const errMsg = result.error ?? '';
|
||||
const isTransient = errMsg.includes('Extension disconnected')
|
||||
|| errMsg.includes('Extension not connected')
|
||||
|| errMsg.includes('attach failed')
|
||||
|| errMsg.includes('no longer exists');
|
||||
if (isTransient && attempt < maxRetries) {
|
||||
if (isTransientBrowserError(new Error(result.error ?? '')) && attempt < maxRetries) {
|
||||
// Longer delay for extension recovery (service worker restart)
|
||||
await new Promise(r => setTimeout(r, 1500));
|
||||
await sleep(1500);
|
||||
continue;
|
||||
}
|
||||
throw new Error(result.error ?? 'Daemon command failed');
|
||||
@@ -125,7 +149,7 @@ export async function sendCommand(
|
||||
const isRetryable = err instanceof TypeError // fetch network error
|
||||
|| (err instanceof Error && err.name === 'AbortError');
|
||||
if (isRetryable && attempt < maxRetries) {
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
await sleep(500);
|
||||
continue;
|
||||
}
|
||||
throw err;
|
||||
@@ -139,4 +163,3 @@ export async function listSessions(): Promise<BrowserSessionInfo[]> {
|
||||
const result = await sendCommand('sessions');
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
|
||||
+9
-21
@@ -1,12 +1,8 @@
|
||||
/**
|
||||
* Daemon discovery — simplified from MCP server path discovery.
|
||||
*
|
||||
* Only needs to check if the daemon is running. No more file system
|
||||
* scanning for @playwright/mcp locations.
|
||||
* Daemon discovery — checks if the daemon is running.
|
||||
*/
|
||||
|
||||
import { DEFAULT_DAEMON_PORT } from '../constants.js';
|
||||
import { isDaemonRunning } from './daemon-client.js';
|
||||
import { fetchDaemonStatus, isDaemonRunning } from './daemon-client.js';
|
||||
|
||||
export { isDaemonRunning };
|
||||
|
||||
@@ -18,21 +14,13 @@ export async function checkDaemonStatus(opts?: { timeout?: number }): Promise<{
|
||||
extensionConnected: boolean;
|
||||
extensionVersion?: string;
|
||||
}> {
|
||||
try {
|
||||
const port = parseInt(process.env.OPENCLI_DAEMON_PORT ?? String(DEFAULT_DAEMON_PORT), 10);
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), opts?.timeout ?? 2000);
|
||||
try {
|
||||
const res = await fetch(`http://127.0.0.1:${port}/status`, {
|
||||
headers: { 'X-OpenCLI': '1' },
|
||||
signal: controller.signal,
|
||||
});
|
||||
const data = await res.json() as { ok: boolean; extensionConnected: boolean; extensionVersion?: string };
|
||||
return { running: true, extensionConnected: data.extensionConnected, extensionVersion: data.extensionVersion };
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
} catch {
|
||||
const status = await fetchDaemonStatus({ timeout: opts?.timeout ?? 2000 });
|
||||
if (!status) {
|
||||
return { running: false, extensionConnected: false };
|
||||
}
|
||||
return {
|
||||
running: true,
|
||||
extensionConnected: status.extensionConnected,
|
||||
extensionVersion: status.extensionVersion,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { autoScrollJs, waitForCaptureJs, waitForSelectorJs } from './dom-helpers.js';
|
||||
|
||||
describe('autoScrollJs', () => {
|
||||
it('returns early without error when document.body is null', async () => {
|
||||
const g = globalThis as any;
|
||||
const origDoc = g.document;
|
||||
g.document = { body: null, documentElement: {} };
|
||||
g.window = g;
|
||||
const code = autoScrollJs(3, 500);
|
||||
// Should resolve without throwing
|
||||
await expect(eval(code)).resolves.not.toThrow();
|
||||
g.document = origDoc;
|
||||
delete g.window;
|
||||
});
|
||||
});
|
||||
|
||||
describe('waitForCaptureJs', () => {
|
||||
it('returns a non-empty string', () => {
|
||||
const code = waitForCaptureJs(1000);
|
||||
expect(typeof code).toBe('string');
|
||||
expect(code.length).toBeGreaterThan(0);
|
||||
expect(code).toContain('__opencli_xhr');
|
||||
expect(code).toContain('resolve');
|
||||
expect(code).toContain('reject');
|
||||
});
|
||||
|
||||
it('resolves "captured" when __opencli_xhr is populated before deadline', async () => {
|
||||
const g = globalThis as any;
|
||||
g.__opencli_xhr = [];
|
||||
g.window = g; // stub window for Node eval
|
||||
const code = waitForCaptureJs(1000);
|
||||
const promise = eval(code) as Promise<string>;
|
||||
g.__opencli_xhr.push({ data: 'test' });
|
||||
await expect(promise).resolves.toBe('captured');
|
||||
delete g.__opencli_xhr;
|
||||
delete g.window;
|
||||
});
|
||||
|
||||
it('rejects when __opencli_xhr stays empty past deadline', async () => {
|
||||
const g = globalThis as any;
|
||||
g.__opencli_xhr = [];
|
||||
g.window = g;
|
||||
const code = waitForCaptureJs(50); // 50ms timeout
|
||||
const promise = eval(code) as Promise<string>;
|
||||
await expect(promise).rejects.toThrow('No network capture within 0.05s');
|
||||
delete g.__opencli_xhr;
|
||||
delete g.window;
|
||||
});
|
||||
|
||||
it('resolves immediately when __opencli_xhr already has data', async () => {
|
||||
const g = globalThis as any;
|
||||
g.__opencli_xhr = [{ data: 'already here' }];
|
||||
g.window = g;
|
||||
const code = waitForCaptureJs(1000);
|
||||
await expect(eval(code) as Promise<string>).resolves.toBe('captured');
|
||||
delete g.__opencli_xhr;
|
||||
delete g.window;
|
||||
});
|
||||
});
|
||||
|
||||
describe('waitForSelectorJs', () => {
|
||||
it('returns a non-empty string', () => {
|
||||
const code = waitForSelectorJs('#app', 1000);
|
||||
expect(typeof code).toBe('string');
|
||||
expect(code).toContain('#app');
|
||||
expect(code).toContain('querySelector');
|
||||
expect(code).toContain('MutationObserver');
|
||||
});
|
||||
|
||||
it('resolves "found" immediately when selector already present', async () => {
|
||||
const g = globalThis as any;
|
||||
const fakeEl = { tagName: 'DIV' };
|
||||
g.document = { querySelector: (_: string) => fakeEl };
|
||||
const code = waitForSelectorJs('[data-testid="primaryColumn"]', 1000);
|
||||
await expect(eval(code) as Promise<string>).resolves.toBe('found');
|
||||
delete g.document;
|
||||
});
|
||||
|
||||
it('resolves "found" when selector appears after DOM mutation', async () => {
|
||||
const g = globalThis as any;
|
||||
let mutationCallback!: () => void;
|
||||
g.MutationObserver = class {
|
||||
constructor(cb: () => void) { mutationCallback = cb; }
|
||||
observe() {}
|
||||
disconnect() {}
|
||||
};
|
||||
let calls = 0;
|
||||
g.document = {
|
||||
querySelector: (_: string) => (calls++ > 0 ? { tagName: 'DIV' } : null),
|
||||
body: {},
|
||||
};
|
||||
const code = waitForSelectorJs('#app', 1000);
|
||||
const promise = eval(code) as Promise<string>;
|
||||
mutationCallback(); // simulate DOM mutation
|
||||
await expect(promise).resolves.toBe('found');
|
||||
delete g.document;
|
||||
delete g.MutationObserver;
|
||||
});
|
||||
|
||||
it('rejects when selector never appears within timeout', async () => {
|
||||
const g = globalThis as any;
|
||||
g.MutationObserver = class {
|
||||
constructor(_cb: () => void) {}
|
||||
observe() {}
|
||||
disconnect() {}
|
||||
};
|
||||
g.document = { querySelector: (_: string) => null, body: {} };
|
||||
const code = waitForSelectorJs('#missing', 50);
|
||||
await expect(eval(code) as Promise<string>).rejects.toThrow('Selector not found: #missing');
|
||||
delete g.document;
|
||||
delete g.MutationObserver;
|
||||
});
|
||||
});
|
||||
@@ -109,6 +109,7 @@ export function scrollJs(direction: string, amount: number): string {
|
||||
export function autoScrollJs(times: number, delayMs: number): string {
|
||||
return `
|
||||
(async () => {
|
||||
if (!document.body) return;
|
||||
for (let i = 0; i < ${times}; i++) {
|
||||
const lastHeight = document.body.scrollHeight;
|
||||
window.scrollTo(0, lastHeight);
|
||||
@@ -179,3 +180,47 @@ export function waitForDomStableJs(maxMs: number, quietMs: number): string {
|
||||
})
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate JS to wait until window.__opencli_xhr has ≥1 captured response.
|
||||
* Polls every 100ms. Resolves 'captured' on success; rejects after maxMs.
|
||||
* Used after installInterceptor() + goto() instead of a fixed sleep.
|
||||
*/
|
||||
export function waitForCaptureJs(maxMs: number): string {
|
||||
return `
|
||||
new Promise((resolve, reject) => {
|
||||
const deadline = Date.now() + ${maxMs};
|
||||
const check = () => {
|
||||
if ((window.__opencli_xhr || []).length > 0) return resolve('captured');
|
||||
if (Date.now() > deadline) return reject(new Error('No network capture within ${maxMs / 1000}s'));
|
||||
setTimeout(check, 100);
|
||||
};
|
||||
check();
|
||||
})
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate JS to wait until document.querySelector(selector) returns a match.
|
||||
* Uses MutationObserver for near-instant resolution; falls back to reject after timeoutMs.
|
||||
*/
|
||||
export function waitForSelectorJs(selector: string, timeoutMs: number): string {
|
||||
return `
|
||||
new Promise((resolve, reject) => {
|
||||
const sel = ${JSON.stringify(selector)};
|
||||
if (document.querySelector(sel)) return resolve('found');
|
||||
const cap = setTimeout(() => {
|
||||
obs.disconnect();
|
||||
reject(new Error('Selector not found: ' + sel));
|
||||
}, ${timeoutMs});
|
||||
const obs = new MutationObserver(() => {
|
||||
if (document.querySelector(sel)) {
|
||||
clearTimeout(cap);
|
||||
obs.disconnect();
|
||||
resolve('found');
|
||||
}
|
||||
});
|
||||
obs.observe(document.body || document.documentElement, { childList: true, subtree: true });
|
||||
})
|
||||
`;
|
||||
}
|
||||
|
||||
@@ -8,6 +8,28 @@
|
||||
import { BrowserConnectError, type BrowserConnectKind } from '../errors.js';
|
||||
import { DEFAULT_DAEMON_PORT } from '../constants.js';
|
||||
|
||||
/**
|
||||
* Transient browser error patterns — shared across daemon-client, pipeline executor,
|
||||
* and page retry logic. These errors indicate temporary conditions (extension restart,
|
||||
* service worker cycle, tab navigation) that are worth retrying.
|
||||
*/
|
||||
const TRANSIENT_ERROR_PATTERNS = [
|
||||
'Extension disconnected',
|
||||
'Extension not connected',
|
||||
'attach failed',
|
||||
'no longer exists',
|
||||
'CDP connection',
|
||||
'Daemon command failed',
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Check if an error message indicates a transient browser error worth retrying.
|
||||
*/
|
||||
export function isTransientBrowserError(err: unknown): boolean {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
return TRANSIENT_ERROR_PATTERNS.some(pattern => msg.includes(pattern));
|
||||
}
|
||||
|
||||
// Re-export so callers don't need to import from two places
|
||||
export type ConnectFailureKind = BrowserConnectKind;
|
||||
|
||||
|
||||
+1
-14
@@ -6,22 +6,9 @@
|
||||
*/
|
||||
|
||||
export { Page } from './page.js';
|
||||
export { BrowserBridge } from './mcp.js';
|
||||
export { BrowserBridge } from './bridge.js';
|
||||
export { CDPBridge } from './cdp.js';
|
||||
export { isDaemonRunning } from './daemon-client.js';
|
||||
export { generateSnapshotJs, scrollToRefJs, getFormStateJs } from './dom-snapshot.js';
|
||||
export { generateStealthJs } from './stealth.js';
|
||||
export type { DomSnapshotOptions } from './dom-snapshot.js';
|
||||
|
||||
import { extractTabEntries, diffTabIndexes, appendLimited } from './tabs.js';
|
||||
import { __test__ as cdpTest } from './cdp.js';
|
||||
import { withTimeoutMs } from '../runtime.js';
|
||||
|
||||
export const __test__ = {
|
||||
extractTabEntries,
|
||||
diffTabIndexes,
|
||||
appendLimited,
|
||||
withTimeoutMs,
|
||||
selectCDPTarget: cdpTest.selectCDPTarget,
|
||||
scoreCDPTarget: cdpTest.scoreCDPTarget,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const { sendCommandMock } = vi.hoisted(() => ({
|
||||
sendCommandMock: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('./daemon-client.js', () => ({
|
||||
sendCommand: sendCommandMock,
|
||||
}));
|
||||
|
||||
import { Page } from './page.js';
|
||||
|
||||
describe('Page.getCurrentUrl', () => {
|
||||
beforeEach(() => {
|
||||
sendCommandMock.mockReset();
|
||||
});
|
||||
|
||||
it('reads the real browser URL when no local navigation cache exists', async () => {
|
||||
sendCommandMock.mockResolvedValueOnce('https://notebooklm.google.com/notebook/nb-live');
|
||||
|
||||
const page = new Page('site:notebooklm');
|
||||
const url = await page.getCurrentUrl();
|
||||
|
||||
expect(url).toBe('https://notebooklm.google.com/notebook/nb-live');
|
||||
expect(sendCommandMock).toHaveBeenCalledTimes(1);
|
||||
expect(sendCommandMock).toHaveBeenCalledWith('exec', expect.objectContaining({
|
||||
workspace: 'site:notebooklm',
|
||||
}));
|
||||
});
|
||||
|
||||
it('caches the discovered browser URL for later reads', async () => {
|
||||
sendCommandMock.mockResolvedValueOnce('https://notebooklm.google.com/notebook/nb-live');
|
||||
|
||||
const page = new Page('site:notebooklm');
|
||||
expect(await page.getCurrentUrl()).toBe('https://notebooklm.google.com/notebook/nb-live');
|
||||
expect(await page.getCurrentUrl()).toBe('https://notebooklm.google.com/notebook/nb-live');
|
||||
|
||||
expect(sendCommandMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Page.evaluate', () => {
|
||||
beforeEach(() => {
|
||||
sendCommandMock.mockReset();
|
||||
});
|
||||
|
||||
it('retries once when the inspected target navigated during exec', async () => {
|
||||
sendCommandMock
|
||||
.mockRejectedValueOnce(new Error('{"code":-32000,"message":"Inspected target navigated or closed"}'))
|
||||
.mockResolvedValueOnce(42);
|
||||
|
||||
const page = new Page('site:notebooklm');
|
||||
const value = await page.evaluate('21 + 21');
|
||||
|
||||
expect(value).toBe(42);
|
||||
expect(sendCommandMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
+106
-187
@@ -10,34 +10,30 @@
|
||||
* chrome-extension:// tab that can't be debugged.
|
||||
*/
|
||||
|
||||
import { formatSnapshot } from '../snapshotFormatter.js';
|
||||
import type { BrowserCookie, IPage, ScreenshotOptions, SnapshotOptions, WaitOptions } from '../types.js';
|
||||
import type { BrowserCookie, ScreenshotOptions } from '../types.js';
|
||||
import { sendCommand } from './daemon-client.js';
|
||||
import { wrapForEval } from './utils.js';
|
||||
import { saveBase64ToFile } from '../utils.js';
|
||||
import { generateSnapshotJs, scrollToRefJs, getFormStateJs } from './dom-snapshot.js';
|
||||
import { generateStealthJs } from './stealth.js';
|
||||
import {
|
||||
clickJs,
|
||||
typeTextJs,
|
||||
pressKeyJs,
|
||||
waitForTextJs,
|
||||
scrollJs,
|
||||
autoScrollJs,
|
||||
networkRequestsJs,
|
||||
waitForDomStableJs,
|
||||
} from './dom-helpers.js';
|
||||
import { waitForDomStableJs } from './dom-helpers.js';
|
||||
import { BasePage } from './base-page.js';
|
||||
|
||||
export function isRetryableSettleError(err: unknown): boolean {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return message.includes('Inspected target navigated or closed')
|
||||
|| (message.includes('-32000') && message.toLowerCase().includes('target'));
|
||||
}
|
||||
|
||||
/**
|
||||
* Page — implements IPage by talking to the daemon via HTTP.
|
||||
*/
|
||||
export class Page implements IPage {
|
||||
constructor(private readonly workspace: string = 'default') {}
|
||||
export class Page extends BasePage {
|
||||
constructor(private readonly workspace: string = 'default') {
|
||||
super();
|
||||
}
|
||||
|
||||
/** Active tab ID, set after navigate and used in all subsequent commands */
|
||||
private _tabId: number | undefined;
|
||||
/** Last navigated URL, tracked in-memory to avoid extra round-trips */
|
||||
private _lastUrl: string | null = null;
|
||||
|
||||
/** Helper: spread workspace into command params */
|
||||
private _wsOpt(): { workspace: string } {
|
||||
@@ -75,15 +71,48 @@ export class Page implements IPage {
|
||||
// settleMs is now a timeout cap (default 1000ms), not a fixed wait.
|
||||
if (options?.waitUntil !== 'none') {
|
||||
const maxMs = options?.settleMs ?? 1000;
|
||||
await sendCommand('exec', {
|
||||
const settleOpts = {
|
||||
code: waitForDomStableJs(maxMs, Math.min(500, maxMs)),
|
||||
...this._cmdOpts(),
|
||||
});
|
||||
};
|
||||
try {
|
||||
await sendCommand('exec', settleOpts);
|
||||
} catch (err) {
|
||||
if (!isRetryableSettleError(err)) throw err;
|
||||
// SPA client-side redirects can invalidate the CDP target after
|
||||
// chrome.tabs reports 'complete'. Wait briefly for the new document
|
||||
// to load, then retry the settle probe once.
|
||||
try {
|
||||
await new Promise((r) => setTimeout(r, 200));
|
||||
await sendCommand('exec', settleOpts);
|
||||
} catch (retryErr) {
|
||||
if (!isRetryableSettleError(retryErr)) throw retryErr;
|
||||
// Retry also failed — give up silently. Settle is best-effort
|
||||
// after successful navigation; the next real command will surface
|
||||
// any persistent target error immediately.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async getCurrentUrl(): Promise<string | null> {
|
||||
return this._lastUrl;
|
||||
getActiveTabId(): number | undefined {
|
||||
return this._tabId;
|
||||
}
|
||||
|
||||
async evaluate(js: string): Promise<unknown> {
|
||||
const code = wrapForEval(js);
|
||||
try {
|
||||
return await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
} catch (err) {
|
||||
if (!isRetryableSettleError(err)) throw err;
|
||||
await new Promise((resolve) => setTimeout(resolve, 200));
|
||||
return sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
}
|
||||
|
||||
async getCookies(opts: { domain?: string; url?: string } = {}): Promise<BrowserCookie[]> {
|
||||
const result = await sendCommand('cookies', { ...this._wsOpt(), ...opts });
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
/** Close the automation window in the extension */
|
||||
@@ -95,131 +124,6 @@ export class Page implements IPage {
|
||||
}
|
||||
}
|
||||
|
||||
async evaluate(js: string): Promise<unknown> {
|
||||
const code = wrapForEval(js);
|
||||
return sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
|
||||
async getCookies(opts: { domain?: string; url?: string } = {}): Promise<BrowserCookie[]> {
|
||||
const result = await sendCommand('cookies', { ...this._wsOpt(), ...opts });
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
async snapshot(opts: SnapshotOptions = {}): Promise<unknown> {
|
||||
// Primary: use the advanced DOM snapshot engine with multi-layer pruning
|
||||
const snapshotJs = generateSnapshotJs({
|
||||
viewportExpand: opts.viewportExpand ?? 800,
|
||||
maxDepth: Math.max(1, Math.min(Number(opts.maxDepth) || 50, 200)),
|
||||
interactiveOnly: opts.interactive ?? false,
|
||||
maxTextLength: opts.maxTextLength ?? 120,
|
||||
includeScrollInfo: true,
|
||||
bboxDedup: true,
|
||||
});
|
||||
|
||||
try {
|
||||
const result = await sendCommand('exec', { code: snapshotJs, ...this._cmdOpts() });
|
||||
// The advanced engine already produces a clean, pruned, LLM-friendly output.
|
||||
// Do NOT pass through formatSnapshot — its format is incompatible.
|
||||
return result;
|
||||
} catch {
|
||||
// Fallback: basic DOM snapshot (original implementation)
|
||||
return this._basicSnapshot(opts);
|
||||
}
|
||||
}
|
||||
|
||||
/** Fallback basic snapshot — original buildTree approach */
|
||||
private async _basicSnapshot(opts: Pick<SnapshotOptions, 'interactive' | 'compact' | 'maxDepth' | 'raw'> = {}): Promise<unknown> {
|
||||
const maxDepth = Math.max(1, Math.min(Number(opts.maxDepth) || 50, 200));
|
||||
const code = `
|
||||
(async () => {
|
||||
function buildTree(node, depth) {
|
||||
if (depth > ${maxDepth}) return '';
|
||||
const role = node.getAttribute?.('role') || node.tagName?.toLowerCase() || 'generic';
|
||||
const name = node.getAttribute?.('aria-label') || node.getAttribute?.('alt') || node.textContent?.trim().slice(0, 80) || '';
|
||||
const isInteractive = ['a', 'button', 'input', 'select', 'textarea'].includes(node.tagName?.toLowerCase()) || node.getAttribute?.('tabindex') != null;
|
||||
|
||||
${opts.interactive ? 'if (!isInteractive && !node.children?.length) return "";' : ''}
|
||||
|
||||
let indent = ' '.repeat(depth);
|
||||
let line = indent + role;
|
||||
if (name) line += ' "' + name.replace(/"/g, '\\\\\\"') + '"';
|
||||
if (node.tagName?.toLowerCase() === 'a' && node.href) line += ' [' + node.href + ']';
|
||||
if (node.tagName?.toLowerCase() === 'input') line += ' [' + (node.type || 'text') + ']';
|
||||
|
||||
let result = line + '\\n';
|
||||
if (node.children) {
|
||||
for (const child of node.children) {
|
||||
result += buildTree(child, depth + 1);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
return buildTree(document.body, 0);
|
||||
})()
|
||||
`;
|
||||
const raw = await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
if (opts.raw) return raw;
|
||||
if (typeof raw === 'string') return formatSnapshot(raw, opts);
|
||||
return raw;
|
||||
}
|
||||
|
||||
async click(ref: string): Promise<void> {
|
||||
const code = clickJs(ref);
|
||||
await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
|
||||
async typeText(ref: string, text: string): Promise<void> {
|
||||
const code = typeTextJs(ref, text);
|
||||
await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
|
||||
async pressKey(key: string): Promise<void> {
|
||||
const code = pressKeyJs(key);
|
||||
await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
|
||||
async scrollTo(ref: string): Promise<unknown> {
|
||||
const code = scrollToRefJs(ref);
|
||||
return sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
|
||||
async getFormState(): Promise<Record<string, unknown>> {
|
||||
const code = getFormStateJs();
|
||||
return (await sendCommand('exec', { code, ...this._cmdOpts() })) as Record<string, unknown>;
|
||||
}
|
||||
|
||||
async wait(options: number | WaitOptions): Promise<void> {
|
||||
if (typeof options === 'number') {
|
||||
if (options >= 1) {
|
||||
// For waits >= 1s, use DOM-stable check: return early when the page
|
||||
// stops mutating, with the original wait time as the hard cap.
|
||||
// This turns e.g. `page.wait(5)` from a fixed 5s sleep into
|
||||
// "wait until DOM is stable, max 5s" — often completing in <1s.
|
||||
try {
|
||||
const maxMs = options * 1000;
|
||||
await sendCommand('exec', {
|
||||
code: waitForDomStableJs(maxMs, Math.min(500, maxMs)),
|
||||
...this._cmdOpts(),
|
||||
});
|
||||
return;
|
||||
} catch {
|
||||
// Fallback: fixed sleep (e.g. if page has no DOM yet)
|
||||
}
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, options * 1000));
|
||||
return;
|
||||
}
|
||||
if (typeof options.time === 'number') {
|
||||
await new Promise(resolve => setTimeout(resolve, options.time! * 1000));
|
||||
return;
|
||||
}
|
||||
if (options.text) {
|
||||
const timeout = (options.timeout ?? 30) * 1000;
|
||||
const code = waitForTextJs(options.text, timeout);
|
||||
await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
}
|
||||
}
|
||||
|
||||
async tabs(): Promise<unknown[]> {
|
||||
const result = await sendCommand('tabs', { op: 'list', ...this._wsOpt() });
|
||||
return Array.isArray(result) ? result : [];
|
||||
@@ -242,27 +146,8 @@ export class Page implements IPage {
|
||||
if (result?.selected) this._tabId = result.selected;
|
||||
}
|
||||
|
||||
async networkRequests(includeStatic: boolean = false): Promise<unknown[]> {
|
||||
const code = networkRequestsJs(includeStatic);
|
||||
const result = await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Console messages are not available in lightweight daemon mode.
|
||||
* Would require CDP Runtime.consoleAPICalled event listener.
|
||||
* @returns Always returns empty array.
|
||||
*/
|
||||
async consoleMessages(_level: string = 'info'): Promise<unknown[]> {
|
||||
return [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture a screenshot via CDP Page.captureScreenshot.
|
||||
* @param options.format - 'png' (default) or 'jpeg'
|
||||
* @param options.quality - JPEG quality 0-100
|
||||
* @param options.fullPage - capture full scrollable page
|
||||
* @param options.path - save to file path (returns base64 if omitted)
|
||||
*/
|
||||
async screenshot(options: ScreenshotOptions = {}): Promise<string> {
|
||||
const base64 = await sendCommand('screenshot', {
|
||||
@@ -279,34 +164,68 @@ export class Page implements IPage {
|
||||
return base64;
|
||||
}
|
||||
|
||||
async scroll(direction: string = 'down', amount: number = 500): Promise<void> {
|
||||
const code = scrollJs(direction, amount);
|
||||
await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
/**
|
||||
* Set local file paths on a file input element via CDP DOM.setFileInputFiles.
|
||||
* Chrome reads the files directly from the local filesystem, avoiding the
|
||||
* payload size limits of base64-in-evaluate.
|
||||
*/
|
||||
async setFileInput(files: string[], selector?: string): Promise<void> {
|
||||
const result = await sendCommand('set-file-input', {
|
||||
files,
|
||||
selector,
|
||||
...this._cmdOpts(),
|
||||
}) as { count?: number };
|
||||
if (!result?.count) {
|
||||
throw new Error('setFileInput returned no count — command may not be supported by the extension');
|
||||
}
|
||||
}
|
||||
|
||||
async autoScroll(options: { times?: number; delayMs?: number } = {}): Promise<void> {
|
||||
const times = options.times ?? 3;
|
||||
const delayMs = options.delayMs ?? 2000;
|
||||
const code = autoScrollJs(times, delayMs);
|
||||
await sendCommand('exec', { code, ...this._cmdOpts() });
|
||||
async cdp(method: string, params: Record<string, unknown> = {}): Promise<unknown> {
|
||||
return sendCommand('cdp', {
|
||||
cdpMethod: method,
|
||||
cdpParams: params,
|
||||
...this._cmdOpts(),
|
||||
});
|
||||
}
|
||||
|
||||
async installInterceptor(pattern: string): Promise<void> {
|
||||
const { generateInterceptorJs } = await import('../interceptor.js');
|
||||
// Must use evaluate() so wrapForEval() converts the arrow function into an IIFE;
|
||||
// sendCommand('exec') sends the code as-is, and CDP never executes a bare arrow.
|
||||
await this.evaluate(generateInterceptorJs(JSON.stringify(pattern), {
|
||||
arrayName: '__opencli_xhr',
|
||||
patchGuard: '__opencli_interceptor_patched',
|
||||
}));
|
||||
async nativeClick(x: number, y: number): Promise<void> {
|
||||
await this.cdp('Input.dispatchMouseEvent', {
|
||||
type: 'mousePressed',
|
||||
x, y,
|
||||
button: 'left',
|
||||
clickCount: 1,
|
||||
});
|
||||
await this.cdp('Input.dispatchMouseEvent', {
|
||||
type: 'mouseReleased',
|
||||
x, y,
|
||||
button: 'left',
|
||||
clickCount: 1,
|
||||
});
|
||||
}
|
||||
|
||||
async getInterceptedRequests(): Promise<unknown[]> {
|
||||
const { generateReadInterceptedJs } = await import('../interceptor.js');
|
||||
// Same as installInterceptor: must go through evaluate() for IIFE wrapping
|
||||
const result = await this.evaluate(generateReadInterceptedJs('__opencli_xhr'));
|
||||
return Array.isArray(result) ? result : [];
|
||||
async nativeType(text: string): Promise<void> {
|
||||
// Use Input.insertText for reliable Unicode/CJK text insertion
|
||||
await this.cdp('Input.insertText', { text });
|
||||
}
|
||||
|
||||
async nativeKeyPress(key: string, modifiers: string[] = []): Promise<void> {
|
||||
let modifierFlags = 0;
|
||||
for (const mod of modifiers) {
|
||||
if (mod === 'Alt') modifierFlags |= 1;
|
||||
if (mod === 'Ctrl') modifierFlags |= 2;
|
||||
if (mod === 'Meta') modifierFlags |= 4;
|
||||
if (mod === 'Shift') modifierFlags |= 8;
|
||||
}
|
||||
await this.cdp('Input.dispatchKeyEvent', {
|
||||
type: 'keyDown',
|
||||
key,
|
||||
modifiers: modifierFlags,
|
||||
});
|
||||
await this.cdp('Input.dispatchKeyEvent', {
|
||||
type: 'keyUp',
|
||||
key,
|
||||
modifiers: modifierFlags,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// (End of file)
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { generateStealthJs } from './stealth.js';
|
||||
|
||||
/**
|
||||
* Tests for the stealth anti-detection module.
|
||||
*
|
||||
* We test the generated JS string for expected content and structure.
|
||||
* Evaluating in Node is fragile because stealth patches target browser
|
||||
* globals (navigator, Performance, HTMLIFrameElement) that don't exist
|
||||
* or behave differently in Node. Instead we verify the code string
|
||||
* contains the right patches and is syntactically valid.
|
||||
*/
|
||||
|
||||
describe('generateStealthJs', () => {
|
||||
it('returns a non-empty string', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(typeof code).toBe('string');
|
||||
expect(code.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('is a valid self-contained IIFE', () => {
|
||||
const code = generateStealthJs();
|
||||
// Should start/end as an IIFE
|
||||
expect(code.trim()).toMatch(/^\(\(\) => \{/);
|
||||
expect(code.trim()).toMatch(/\}\)\(\)$/);
|
||||
});
|
||||
|
||||
it('patches navigator.webdriver', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain("navigator, 'webdriver'");
|
||||
expect(code).toContain('() => false');
|
||||
});
|
||||
|
||||
it('stubs window.chrome', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('window.chrome');
|
||||
expect(code).toContain('runtime');
|
||||
expect(code).toContain('loadTimes');
|
||||
expect(code).toContain('csi');
|
||||
});
|
||||
|
||||
it('fakes navigator.plugins if empty', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('navigator.plugins');
|
||||
expect(code).toContain('PDF Viewer');
|
||||
expect(code).toContain('Chrome PDF Viewer');
|
||||
});
|
||||
|
||||
it('ensures navigator.languages is non-empty', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('navigator.languages');
|
||||
expect(code).toContain("'en-US'");
|
||||
});
|
||||
|
||||
it('normalizes Permissions.query for notifications', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('Permissions');
|
||||
expect(code).toContain('notifications');
|
||||
});
|
||||
|
||||
it('cleans automation artifacts', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('__playwright');
|
||||
expect(code).toContain('__puppeteer');
|
||||
expect(code).toContain("'cdc_'");
|
||||
expect(code).toContain("'__cdc_'");
|
||||
});
|
||||
|
||||
it('filters CDP patterns from Error.stack', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('puppeteer_evaluation_script');
|
||||
expect(code).toContain("'pptr:'");
|
||||
expect(code).toContain("'debugger://'");
|
||||
});
|
||||
|
||||
it('neutralizes debugger statement traps', () => {
|
||||
const code = generateStealthJs();
|
||||
// Should patch Function constructor with new.target / Reflect.construct
|
||||
expect(code).toContain('_OrigFunction');
|
||||
expect(code).toContain('_PatchedFunction');
|
||||
expect(code).toContain('new.target');
|
||||
expect(code).toContain('Reflect.construct');
|
||||
// Should patch eval
|
||||
expect(code).toContain('_origEval');
|
||||
expect(code).toContain('_patchedEval');
|
||||
// Regex to strip debugger (lookbehind for statement boundaries)
|
||||
expect(code).toContain('_debuggerRe');
|
||||
});
|
||||
|
||||
it('uses shared toString disguise via WeakMap', () => {
|
||||
const code = generateStealthJs();
|
||||
// Shared infrastructure at the top of the IIFE
|
||||
expect(code).toContain('_origToString');
|
||||
expect(code).toContain('WeakMap');
|
||||
expect(code).toContain('_disguised');
|
||||
expect(code).toContain('_disguise');
|
||||
// Should NOT have per-instance toString overrides on Function/eval
|
||||
// (they go through _disguise instead)
|
||||
});
|
||||
|
||||
it('defends console method fingerprinting', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('_consoleMethods');
|
||||
expect(code).toContain("'log'");
|
||||
expect(code).toContain("'warn'");
|
||||
expect(code).toContain("'error'");
|
||||
expect(code).toContain('[native code]');
|
||||
// Uses saved _origToString reference
|
||||
expect(code).toContain('_origToString.call');
|
||||
});
|
||||
|
||||
it('defends window dimension detection', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('outerWidth');
|
||||
expect(code).toContain('outerHeight');
|
||||
expect(code).toContain('innerWidth');
|
||||
expect(code).toContain('innerHeight');
|
||||
});
|
||||
|
||||
it('filters Performance API entries', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('getEntries');
|
||||
expect(code).toContain('getEntriesByType');
|
||||
expect(code).toContain('getEntriesByName');
|
||||
expect(code).toContain('_suspiciousPatterns');
|
||||
});
|
||||
|
||||
it('cleans document $cdc_ properties', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain("'$cdc_'");
|
||||
expect(code).toContain("'$chrome_'");
|
||||
});
|
||||
|
||||
it('patches iframe contentWindow.chrome consistency', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('contentWindow');
|
||||
expect(code).toContain('HTMLIFrameElement');
|
||||
});
|
||||
|
||||
it('uses non-enumerable guard flag on EventTarget.prototype', () => {
|
||||
const code = generateStealthJs();
|
||||
expect(code).toContain('EventTarget.prototype');
|
||||
expect(code).toContain("'__lsn'");
|
||||
expect(code).toContain('enumerable: false');
|
||||
});
|
||||
|
||||
it('generates syntactically valid JavaScript', () => {
|
||||
const code = generateStealthJs();
|
||||
// new Function() parses the code without executing it in a real
|
||||
// browser context, catching syntax errors from template literal issues.
|
||||
expect(() => new Function(code)).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -150,6 +150,204 @@ export function generateStealthJs(): string {
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// ── Shared toString disguise infrastructure ──
|
||||
// Save the pristine Function.prototype.toString BEFORE any patches,
|
||||
// so all subsequent disguises use the real native reference.
|
||||
// Anti-bot scripts detect per-instance toString overrides via:
|
||||
// Function.hasOwnProperty('toString') → true if patched
|
||||
// Function.prototype.toString.call(fn) !== fn.toString()
|
||||
// Instead we patch Function.prototype.toString once with a WeakMap
|
||||
// lookup, making disguised functions indistinguishable from native.
|
||||
const _origToString = Function.prototype.toString;
|
||||
const _disguised = new WeakMap();
|
||||
try {
|
||||
Object.defineProperty(Function.prototype, 'toString', {
|
||||
value: function() {
|
||||
const override = _disguised.get(this);
|
||||
return override !== undefined ? override : _origToString.call(this);
|
||||
},
|
||||
writable: true, configurable: true,
|
||||
});
|
||||
} catch {}
|
||||
const _disguise = (fn, name) => {
|
||||
_disguised.set(fn, 'function ' + name + '() { [native code] }');
|
||||
try { Object.defineProperty(fn, 'name', { value: name, configurable: true }); } catch {}
|
||||
return fn;
|
||||
};
|
||||
|
||||
// 8. Anti-debugger statement trap
|
||||
// Sites inject debugger statements to detect DevTools/CDP.
|
||||
// When a CDP debugger is attached, the statement pauses execution
|
||||
// and the site measures the time gap to confirm automation.
|
||||
// We neutralize this by overriding the Function constructor and
|
||||
// eval to strip debugger statements from dynamically created code.
|
||||
// Note: this does NOT affect static debugger statements in parsed
|
||||
// scripts — those require CDP Debugger.setBreakpointsActive(false)
|
||||
// which we handle at the extension level.
|
||||
// Caveat: the regex targets standalone debugger statements (preceded
|
||||
// by a statement boundary) to minimise false positives inside string
|
||||
// literals, but cannot perfectly distinguish all cases without a
|
||||
// full parser. This is an acceptable trade-off for stealth code.
|
||||
try {
|
||||
const _OrigFunction = Function;
|
||||
// Match standalone debugger statements preceded by a statement
|
||||
// boundary (start of string, semicolon, brace, or newline).
|
||||
// This avoids most false positives inside string literals like
|
||||
// "use debugger mode" while still catching the anti-bot patterns.
|
||||
const _debuggerRe = /(?:^|(?<=[;{}\\n\\r]))\\s*debugger\\s*;?/g;
|
||||
const _cleanDebugger = (src) => typeof src === 'string' ? src.replace(_debuggerRe, '') : src;
|
||||
// Patch Function constructor to strip debugger from dynamic code.
|
||||
// Support both Function('code') and new Function('code') via
|
||||
// new.target / Reflect.construct.
|
||||
const _PatchedFunction = function(...args) {
|
||||
if (args.length > 0) {
|
||||
args[args.length - 1] = _cleanDebugger(args[args.length - 1]);
|
||||
}
|
||||
if (new.target) {
|
||||
return Reflect.construct(_OrigFunction, args, new.target);
|
||||
}
|
||||
return _OrigFunction.apply(this, args);
|
||||
};
|
||||
_PatchedFunction.prototype = _OrigFunction.prototype;
|
||||
Object.setPrototypeOf(_PatchedFunction, _OrigFunction);
|
||||
_disguise(_PatchedFunction, 'Function');
|
||||
try { window.Function = _PatchedFunction; } catch {}
|
||||
|
||||
// Patch eval to strip debugger
|
||||
const _origEval = window.eval;
|
||||
const _patchedEval = function(code) {
|
||||
return _origEval.call(this, _cleanDebugger(code));
|
||||
};
|
||||
_disguise(_patchedEval, 'eval');
|
||||
try { window.eval = _patchedEval; } catch {}
|
||||
} catch {}
|
||||
|
||||
// 9. Console method fingerprinting defense
|
||||
// When CDP Runtime.enable is called, Chrome replaces console.log etc.
|
||||
// with CDP-bound versions. These bound functions have a different
|
||||
// toString() output: "function log() { [native code] }" becomes
|
||||
// something like "function () { [native code] }" (no name) or the
|
||||
// bound function signature leaks. Anti-bot scripts check:
|
||||
// console.log.toString().includes('[native code]')
|
||||
// console.log.name === 'log'
|
||||
// We re-wrap console methods and register them via the shared
|
||||
// _disguise infrastructure so Function.prototype.toString.call()
|
||||
// also returns the correct native string.
|
||||
try {
|
||||
const _consoleMethods = ['log', 'warn', 'error', 'info', 'debug', 'table', 'trace', 'dir', 'group', 'groupEnd', 'groupCollapsed', 'clear', 'count', 'assert', 'profile', 'profileEnd', 'time', 'timeEnd', 'timeStamp'];
|
||||
for (const _m of _consoleMethods) {
|
||||
if (typeof console[_m] !== 'function') continue;
|
||||
const _origMethod = console[_m];
|
||||
const _nativeStr = 'function ' + _m + '() { [native code] }';
|
||||
// Only patch if toString is wrong (i.e. CDP has replaced it)
|
||||
try {
|
||||
const _currentStr = _origToString.call(_origMethod);
|
||||
if (_currentStr === _nativeStr) continue; // already looks native
|
||||
} catch {}
|
||||
const _wrapper = function() { return _origMethod.apply(console, arguments); };
|
||||
Object.defineProperty(_wrapper, 'length', { value: _origMethod.length || 0, configurable: true });
|
||||
_disguise(_wrapper, _m);
|
||||
try { console[_m] = _wrapper; } catch {}
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// 10. window.outerWidth/outerHeight defense
|
||||
// When DevTools or CDP debugger is attached, Chrome may alter the
|
||||
// window dimensions. Anti-bot scripts compare outerWidth/innerWidth
|
||||
// and outerHeight/innerHeight — a significant difference indicates
|
||||
// DevTools is open. We freeze the relationship so the delta stays
|
||||
// consistent with a normal browser window.
|
||||
// Thresholds: width delta > 100px or height delta > 200px indicates
|
||||
// a docked DevTools panel. When triggered, we report outerWidth
|
||||
// equal to innerWidth (normal for maximised windows) and
|
||||
// outerHeight as innerHeight + the captured "normal" delta (capped
|
||||
// to a reasonable range), so the result is plausible across OSes.
|
||||
try {
|
||||
const _normalWidthDelta = window.outerWidth - window.innerWidth;
|
||||
const _normalHeightDelta = window.outerHeight - window.innerHeight;
|
||||
// Only patch if the delta looks suspicious (e.g. DevTools docked)
|
||||
if (_normalWidthDelta > 100 || _normalHeightDelta > 200) {
|
||||
Object.defineProperty(window, 'outerWidth', {
|
||||
get: () => window.innerWidth,
|
||||
configurable: true,
|
||||
});
|
||||
// Use a clamped height offset (40-120px covers macOS ~78px,
|
||||
// Windows ~40px, and Linux ~37-50px title bar heights).
|
||||
const _heightOffset = Math.max(40, Math.min(120, _normalHeightDelta));
|
||||
Object.defineProperty(window, 'outerHeight', {
|
||||
get: () => window.innerHeight + _heightOffset,
|
||||
configurable: true,
|
||||
});
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// 11. Performance API cleanup
|
||||
// CDP injects internal resources and timing entries that don't exist
|
||||
// in normal browsing. Filter entries with debugger/devtools URLs.
|
||||
try {
|
||||
const _origGetEntries = Performance.prototype.getEntries;
|
||||
const _origGetByType = Performance.prototype.getEntriesByType;
|
||||
const _origGetByName = Performance.prototype.getEntriesByName;
|
||||
const _suspiciousPatterns = ['debugger', 'devtools', '__puppeteer', '__playwright', 'pptr:'];
|
||||
const _filterEntries = (entries) => {
|
||||
if (!Array.isArray(entries)) return entries;
|
||||
return entries.filter(e => {
|
||||
const name = e.name || '';
|
||||
return !_suspiciousPatterns.some(p => name.includes(p));
|
||||
});
|
||||
};
|
||||
Performance.prototype.getEntries = function() {
|
||||
return _filterEntries(_origGetEntries.call(this));
|
||||
};
|
||||
Performance.prototype.getEntriesByType = function(type) {
|
||||
return _filterEntries(_origGetByType.call(this, type));
|
||||
};
|
||||
Performance.prototype.getEntriesByName = function(name, type) {
|
||||
return _filterEntries(_origGetByName.call(this, name, type));
|
||||
};
|
||||
} catch {}
|
||||
|
||||
// 12. WebDriver-related property defense
|
||||
// Some anti-bot systems check additional navigator properties
|
||||
// and document properties that may indicate automation.
|
||||
try {
|
||||
// document.$cdc_ properties (ChromeDriver specific, backup for #6)
|
||||
for (const _prop of Object.getOwnPropertyNames(document)) {
|
||||
if (_prop.startsWith('$cdc_') || _prop.startsWith('$chrome_')) {
|
||||
try { delete document[_prop]; } catch {}
|
||||
}
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// 13. Iframe contentWindow.chrome consistency
|
||||
// Anti-bot scripts create iframes and check if
|
||||
// iframe.contentWindow.chrome exists and matches the parent.
|
||||
// CDP-controlled pages may have inconsistent iframe contexts.
|
||||
try {
|
||||
const _origHTMLIFrame = HTMLIFrameElement.prototype;
|
||||
const _origContentWindow = Object.getOwnPropertyDescriptor(_origHTMLIFrame, 'contentWindow');
|
||||
if (_origContentWindow && _origContentWindow.get) {
|
||||
Object.defineProperty(_origHTMLIFrame, 'contentWindow', {
|
||||
get: function() {
|
||||
const _w = _origContentWindow.get.call(this);
|
||||
if (_w) {
|
||||
try {
|
||||
if (!_w.chrome) {
|
||||
Object.defineProperty(_w, 'chrome', {
|
||||
value: window.chrome,
|
||||
writable: true,
|
||||
configurable: true,
|
||||
});
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
return _w;
|
||||
},
|
||||
configurable: true,
|
||||
});
|
||||
}
|
||||
} catch {}
|
||||
|
||||
return 'applied';
|
||||
})()
|
||||
`;
|
||||
|
||||
+5
-5
@@ -21,12 +21,12 @@ export function extractTabEntries(raw: unknown): Array<{ index: number; identity
|
||||
.map(line => line.trim())
|
||||
.filter(Boolean)
|
||||
.map(line => {
|
||||
// Match actual Playwright MCP format: "- 0: (current) [title](url)" or "- 1: [title](url)"
|
||||
const mcpMatch = line.match(/^-\s+(\d+):\s*(.*)$/);
|
||||
if (mcpMatch) {
|
||||
// Match tab list format: "- 0: (current) [title](url)" or "- 1: [title](url)"
|
||||
const tabMatch = line.match(/^-\s+(\d+):\s*(.*)$/);
|
||||
if (tabMatch) {
|
||||
return {
|
||||
index: parseInt(mcpMatch[1], 10),
|
||||
identity: mcpMatch[2].trim() || `tab-${mcpMatch[1]}`,
|
||||
index: parseInt(tabMatch[1], 10),
|
||||
identity: tabMatch[2].trim() || `tab-${tabMatch[1]}`,
|
||||
};
|
||||
}
|
||||
// Legacy format: "Tab 0 ..."
|
||||
|
||||
+121
-90
@@ -2,69 +2,8 @@ import { afterEach, describe, expect, it } from 'vitest';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { parseTsArgsBlock, scanTs, shouldReplaceManifestEntry } from './build-manifest.js';
|
||||
|
||||
describe('parseTsArgsBlock', () => {
|
||||
it('keeps args with nested choices arrays', () => {
|
||||
const args = parseTsArgsBlock(`
|
||||
{
|
||||
name: 'period',
|
||||
type: 'string',
|
||||
default: 'seven',
|
||||
help: 'Stats period: seven or thirty',
|
||||
choices: ['seven', 'thirty'],
|
||||
},
|
||||
`);
|
||||
|
||||
expect(args).toEqual([
|
||||
{
|
||||
name: 'period',
|
||||
type: 'string',
|
||||
default: 'seven',
|
||||
required: false,
|
||||
positional: undefined,
|
||||
help: 'Stats period: seven or thirty',
|
||||
choices: ['seven', 'thirty'],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('keeps hyphenated arg names from TS adapters', () => {
|
||||
const args = parseTsArgsBlock(`
|
||||
{
|
||||
name: 'tweet-url',
|
||||
help: 'Single tweet URL to download',
|
||||
},
|
||||
{
|
||||
name: 'download-images',
|
||||
type: 'boolean',
|
||||
default: false,
|
||||
help: 'Download images locally',
|
||||
},
|
||||
`);
|
||||
|
||||
expect(args).toEqual([
|
||||
{
|
||||
name: 'tweet-url',
|
||||
type: 'str',
|
||||
default: undefined,
|
||||
required: false,
|
||||
positional: undefined,
|
||||
help: 'Single tweet URL to download',
|
||||
choices: undefined,
|
||||
},
|
||||
{
|
||||
name: 'download-images',
|
||||
type: 'boolean',
|
||||
default: false,
|
||||
required: false,
|
||||
positional: undefined,
|
||||
help: 'Download images locally',
|
||||
choices: undefined,
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
import { cli, getRegistry, Strategy } from './registry.js';
|
||||
import { loadTsManifestEntries, shouldReplaceManifestEntry } from './build-manifest.js';
|
||||
|
||||
describe('manifest helper rules', () => {
|
||||
const tempDirs: string[] = [];
|
||||
@@ -127,43 +66,135 @@ describe('manifest helper rules', () => {
|
||||
const file = path.join(dir, 'utils.ts');
|
||||
fs.writeFileSync(file, `export function helper() { return 'noop'; }`);
|
||||
|
||||
expect(scanTs(file, 'demo')).toBeNull();
|
||||
return expect(loadTsManifestEntries(file, 'demo', async () => ({}))).resolves.toEqual([]);
|
||||
});
|
||||
|
||||
it('keeps literal domain and navigateBefore for TS adapters', () => {
|
||||
const file = path.join(process.cwd(), 'src', 'clis', 'xueqiu', 'fund-holdings.ts');
|
||||
const entry = scanTs(file, 'xueqiu');
|
||||
|
||||
expect(entry).toMatchObject({
|
||||
site: 'xueqiu',
|
||||
name: 'fund-holdings',
|
||||
domain: 'danjuanfunds.com',
|
||||
navigateBefore: 'https://danjuanfunds.com/my-money',
|
||||
type: 'ts',
|
||||
modulePath: 'xueqiu/fund-holdings.js',
|
||||
});
|
||||
});
|
||||
|
||||
it('captures deprecated metadata for TS adapters', () => {
|
||||
it('builds TS manifest entries from exported runtime commands', async () => {
|
||||
const site = `manifest-hydrate-${Date.now()}`;
|
||||
const key = `${site}/dynamic`;
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-manifest-'));
|
||||
tempDirs.push(dir);
|
||||
const file = path.join(dir, 'legacy.ts');
|
||||
fs.writeFileSync(file, `
|
||||
import { cli } from '../../registry.js';
|
||||
const file = path.join(dir, `${site}.ts`);
|
||||
fs.writeFileSync(file, `export const command = cli({ site: '${site}', name: 'dynamic' });`);
|
||||
|
||||
const entries = await loadTsManifestEntries(file, site, async () => ({
|
||||
command: cli({
|
||||
site,
|
||||
name: 'dynamic',
|
||||
description: 'dynamic command',
|
||||
strategy: Strategy.PUBLIC,
|
||||
browser: false,
|
||||
aliases: ['metadata'],
|
||||
args: [
|
||||
{
|
||||
name: 'model',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: 'Choose a model',
|
||||
choices: ['auto', 'thinking'],
|
||||
default: '30',
|
||||
},
|
||||
],
|
||||
domain: 'localhost',
|
||||
navigateBefore: 'https://example.com/session',
|
||||
deprecated: 'legacy command',
|
||||
replacedBy: 'opencli demo new',
|
||||
}),
|
||||
}));
|
||||
|
||||
expect(entries).toEqual([
|
||||
{
|
||||
site,
|
||||
name: 'dynamic',
|
||||
description: 'dynamic command',
|
||||
domain: 'localhost',
|
||||
strategy: 'public',
|
||||
browser: false,
|
||||
aliases: ['metadata'],
|
||||
args: [
|
||||
{
|
||||
name: 'model',
|
||||
type: 'str',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: 'Choose a model',
|
||||
choices: ['auto', 'thinking'],
|
||||
default: '30',
|
||||
},
|
||||
],
|
||||
type: 'ts',
|
||||
modulePath: `${site}/${site}.js`,
|
||||
navigateBefore: 'https://example.com/session',
|
||||
deprecated: 'legacy command',
|
||||
replacedBy: 'opencli demo new',
|
||||
},
|
||||
]);
|
||||
|
||||
getRegistry().delete(key);
|
||||
});
|
||||
|
||||
it('falls back to registry delta for side-effect-only cli modules', async () => {
|
||||
const site = `manifest-side-effect-${Date.now()}`;
|
||||
const key = `${site}/legacy`;
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-manifest-'));
|
||||
tempDirs.push(dir);
|
||||
const file = path.join(dir, `${site}.ts`);
|
||||
fs.writeFileSync(file, `cli({ site: '${site}', name: 'legacy' });`);
|
||||
|
||||
const entries = await loadTsManifestEntries(file, site, async () => {
|
||||
cli({
|
||||
site: 'demo',
|
||||
site,
|
||||
name: 'legacy',
|
||||
description: 'legacy command',
|
||||
deprecated: 'legacy is deprecated',
|
||||
replacedBy: 'opencli demo new',
|
||||
});
|
||||
`);
|
||||
|
||||
expect(scanTs(file, 'demo')).toMatchObject({
|
||||
site: 'demo',
|
||||
name: 'legacy',
|
||||
deprecated: 'legacy is deprecated',
|
||||
replacedBy: 'opencli demo new',
|
||||
return {};
|
||||
});
|
||||
|
||||
expect(entries).toEqual([
|
||||
{
|
||||
site,
|
||||
name: 'legacy',
|
||||
description: 'legacy command',
|
||||
strategy: 'cookie',
|
||||
browser: true,
|
||||
args: [],
|
||||
type: 'ts',
|
||||
modulePath: `${site}/${site}.js`,
|
||||
deprecated: 'legacy is deprecated',
|
||||
replacedBy: 'opencli demo new',
|
||||
},
|
||||
]);
|
||||
|
||||
getRegistry().delete(key);
|
||||
});
|
||||
|
||||
it('keeps every command a module exports instead of guessing by site', async () => {
|
||||
const site = `manifest-multi-${Date.now()}`;
|
||||
const screenKey = `${site}/screen`;
|
||||
const statusKey = `${site}/status`;
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-manifest-'));
|
||||
tempDirs.push(dir);
|
||||
const file = path.join(dir, `${site}.ts`);
|
||||
fs.writeFileSync(file, `export const screen = cli({ site: '${site}', name: 'screen' });`);
|
||||
|
||||
const entries = await loadTsManifestEntries(file, site, async () => ({
|
||||
screen: cli({
|
||||
site,
|
||||
name: 'screen',
|
||||
description: 'capture screen',
|
||||
}),
|
||||
status: cli({
|
||||
site,
|
||||
name: 'status',
|
||||
description: 'show status',
|
||||
}),
|
||||
}));
|
||||
|
||||
expect(entries.map(entry => entry.name)).toEqual(['screen', 'status']);
|
||||
|
||||
getRegistry().delete(screenKey);
|
||||
getRegistry().delete(statusKey);
|
||||
});
|
||||
});
|
||||
|
||||
+82
-176
@@ -14,6 +14,7 @@ import * as path from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import yaml from 'js-yaml';
|
||||
import { getErrorMessage } from './errors.js';
|
||||
import { fullName, getRegistry, type CliCommand } from './registry.js';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const CLIS_DIR = path.resolve(__dirname, 'clis');
|
||||
@@ -22,6 +23,7 @@ const OUTPUT = path.resolve(__dirname, '..', 'dist', 'cli-manifest.json');
|
||||
export interface ManifestEntry {
|
||||
site: string;
|
||||
name: string;
|
||||
aliases?: string[];
|
||||
description: string;
|
||||
domain?: string;
|
||||
strategy: string;
|
||||
@@ -52,116 +54,51 @@ import { type YamlCliDefinition, parseYamlArgs } from './yaml-schema.js';
|
||||
|
||||
import { isRecord } from './utils.js';
|
||||
|
||||
const CLI_MODULE_PATTERN = /\bcli\s*\(/;
|
||||
|
||||
function extractBalancedBlock(
|
||||
source: string,
|
||||
startIndex: number,
|
||||
openChar: string,
|
||||
closeChar: string,
|
||||
): string | null {
|
||||
let depth = 0;
|
||||
let quote: string | null = null;
|
||||
let escaped = false;
|
||||
|
||||
for (let i = startIndex; i < source.length; i++) {
|
||||
const ch = source[i];
|
||||
|
||||
if (quote) {
|
||||
if (escaped) {
|
||||
escaped = false;
|
||||
continue;
|
||||
}
|
||||
if (ch === '\\') {
|
||||
escaped = true;
|
||||
continue;
|
||||
}
|
||||
if (ch === quote) quote = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (ch === '"' || ch === '\'' || ch === '`') {
|
||||
quote = ch;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (ch === openChar) {
|
||||
depth++;
|
||||
} else if (ch === closeChar) {
|
||||
depth--;
|
||||
if (depth === 0) {
|
||||
return source.slice(startIndex + 1, i);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
function toManifestArgs(args: CliCommand['args']): ManifestEntry['args'] {
|
||||
return args.map(arg => ({
|
||||
name: arg.name,
|
||||
type: arg.type ?? 'str',
|
||||
default: arg.default,
|
||||
required: !!arg.required,
|
||||
positional: arg.positional || undefined,
|
||||
help: arg.help ?? '',
|
||||
choices: arg.choices,
|
||||
}));
|
||||
}
|
||||
|
||||
function extractTsArgsBlock(source: string): string | null {
|
||||
const argsMatch = source.match(/args\s*:/);
|
||||
if (!argsMatch || argsMatch.index === undefined) return null;
|
||||
|
||||
const bracketIndex = source.indexOf('[', argsMatch.index);
|
||||
if (bracketIndex === -1) return null;
|
||||
|
||||
return extractBalancedBlock(source, bracketIndex, '[', ']');
|
||||
function toTsModulePath(filePath: string, site: string): string {
|
||||
const baseName = path.basename(filePath, path.extname(filePath));
|
||||
return `${site}/${baseName}.js`;
|
||||
}
|
||||
|
||||
function parseInlineChoices(body: string): string[] | undefined {
|
||||
const choicesMatch = body.match(/choices\s*:\s*\[([^\]]*)\]/);
|
||||
if (!choicesMatch) return undefined;
|
||||
|
||||
const values = choicesMatch[1]
|
||||
.split(',')
|
||||
.map(s => s.trim().replace(/^['"`]|['"`]$/g, ''))
|
||||
.filter(Boolean);
|
||||
|
||||
return values.length > 0 ? values : undefined;
|
||||
function isCliCommandValue(value: unknown, site: string): value is CliCommand {
|
||||
return isRecord(value)
|
||||
&& typeof value.site === 'string'
|
||||
&& value.site === site
|
||||
&& typeof value.name === 'string'
|
||||
&& Array.isArray(value.args);
|
||||
}
|
||||
|
||||
export function parseTsArgsBlock(argsBlock: string): ManifestEntry['args'] {
|
||||
const args: ManifestEntry['args'] = [];
|
||||
let cursor = 0;
|
||||
|
||||
while (cursor < argsBlock.length) {
|
||||
const nameMatch = argsBlock.slice(cursor).match(/\{\s*name\s*:\s*['"`]([^'"`]+)['"`]/);
|
||||
if (!nameMatch || nameMatch.index === undefined) break;
|
||||
|
||||
const objectStart = cursor + nameMatch.index;
|
||||
const body = extractBalancedBlock(argsBlock, objectStart, '{', '}');
|
||||
if (body == null) break;
|
||||
|
||||
const typeMatch = body.match(/type\s*:\s*['"`](\w+)['"`]/);
|
||||
const defaultMatch = body.match(/default\s*:\s*([^,}]+)/);
|
||||
const requiredMatch = body.match(/required\s*:\s*(true|false)/);
|
||||
const helpMatch = body.match(/help\s*:\s*['"`]([^'"`]*)['"`]/);
|
||||
const positionalMatch = body.match(/positional\s*:\s*(true|false)/);
|
||||
|
||||
let defaultVal: unknown = undefined;
|
||||
if (defaultMatch) {
|
||||
const raw = defaultMatch[1].trim();
|
||||
if (raw === 'true') defaultVal = true;
|
||||
else if (raw === 'false') defaultVal = false;
|
||||
else if (/^\d+$/.test(raw)) defaultVal = parseInt(raw, 10);
|
||||
else if (/^\d+\.\d+$/.test(raw)) defaultVal = parseFloat(raw);
|
||||
else defaultVal = raw.replace(/^['"`]|['"`]$/g, '');
|
||||
}
|
||||
|
||||
args.push({
|
||||
name: nameMatch[1],
|
||||
type: typeMatch?.[1] ?? 'str',
|
||||
default: defaultVal,
|
||||
required: requiredMatch?.[1] === 'true',
|
||||
positional: positionalMatch?.[1] === 'true' || undefined,
|
||||
help: helpMatch?.[1] ?? '',
|
||||
choices: parseInlineChoices(body),
|
||||
});
|
||||
|
||||
cursor = objectStart + body.length;
|
||||
if (cursor <= objectStart) break; // safety: prevent infinite loop
|
||||
}
|
||||
|
||||
return args;
|
||||
function toManifestEntry(cmd: CliCommand, modulePath: string): ManifestEntry {
|
||||
return {
|
||||
site: cmd.site,
|
||||
name: cmd.name,
|
||||
aliases: cmd.aliases,
|
||||
description: cmd.description ?? '',
|
||||
domain: cmd.domain,
|
||||
strategy: (cmd.strategy ?? 'public').toString().toLowerCase(),
|
||||
browser: cmd.browser ?? true,
|
||||
args: toManifestArgs(cmd.args),
|
||||
columns: cmd.columns,
|
||||
timeout: cmd.timeoutSeconds,
|
||||
deprecated: cmd.deprecated,
|
||||
replacedBy: cmd.replacedBy,
|
||||
type: 'ts',
|
||||
modulePath,
|
||||
navigateBefore: cmd.navigateBefore,
|
||||
};
|
||||
}
|
||||
|
||||
function scanYaml(filePath: string, site: string): ManifestEntry | null {
|
||||
@@ -184,6 +121,9 @@ function scanYaml(filePath: string, site: string): ManifestEntry | null {
|
||||
domain: cliDef.domain,
|
||||
strategy: strategy.toLowerCase(),
|
||||
browser,
|
||||
aliases: isRecord(cliDef) && Array.isArray((cliDef as Record<string, unknown>).aliases)
|
||||
? ((cliDef as Record<string, unknown>).aliases as unknown[]).filter((value): value is string => typeof value === 'string')
|
||||
: undefined,
|
||||
args,
|
||||
columns: cliDef.columns,
|
||||
pipeline: cliDef.pipeline,
|
||||
@@ -199,83 +139,49 @@ function scanYaml(filePath: string, site: string): ManifestEntry | null {
|
||||
}
|
||||
}
|
||||
|
||||
export function scanTs(filePath: string, site: string): ManifestEntry | null {
|
||||
// TS adapters self-register via cli() at import time.
|
||||
// We statically parse the source to extract metadata for the manifest stub.
|
||||
const baseName = path.basename(filePath, path.extname(filePath));
|
||||
const relativePath = `${site}/${baseName}.js`;
|
||||
|
||||
export async function loadTsManifestEntries(
|
||||
filePath: string,
|
||||
site: string,
|
||||
importer: (moduleHref: string) => Promise<unknown> = moduleHref => import(moduleHref),
|
||||
): Promise<ManifestEntry[]> {
|
||||
try {
|
||||
const src = fs.readFileSync(filePath, 'utf-8');
|
||||
|
||||
// Helper/test modules should not appear as CLI commands in the manifest.
|
||||
if (!/\bcli\s*\(/.test(src)) return null;
|
||||
if (!CLI_MODULE_PATTERN.test(src)) return [];
|
||||
|
||||
const entry: ManifestEntry = {
|
||||
site,
|
||||
name: baseName,
|
||||
description: '',
|
||||
strategy: 'cookie',
|
||||
browser: true,
|
||||
args: [],
|
||||
type: 'ts',
|
||||
modulePath: relativePath,
|
||||
};
|
||||
const modulePath = toTsModulePath(filePath, site);
|
||||
const registry = getRegistry();
|
||||
const before = new Map(registry.entries());
|
||||
const mod = await importer(pathToFileURL(filePath).href);
|
||||
|
||||
// Extract description
|
||||
const descMatch = src.match(/description\s*:\s*['"`]([^'"`]*)['"`]/);
|
||||
if (descMatch) entry.description = descMatch[1];
|
||||
const exportedCommands = Object.values(isRecord(mod) ? mod : {})
|
||||
.filter(value => isCliCommandValue(value, site));
|
||||
|
||||
// Extract domain
|
||||
const domainMatch = src.match(/domain\s*:\s*['"`]([^'"`]*)['"`]/);
|
||||
if (domainMatch) entry.domain = domainMatch[1];
|
||||
const runtimeCommands = exportedCommands.length > 0
|
||||
? exportedCommands
|
||||
: [...registry.entries()]
|
||||
.filter(([key, cmd]) => {
|
||||
if (cmd.site !== site) return false;
|
||||
const previous = before.get(key);
|
||||
return !previous || previous !== cmd;
|
||||
})
|
||||
.map(([, cmd]) => cmd);
|
||||
|
||||
// Extract strategy
|
||||
const stratMatch = src.match(/strategy\s*:\s*Strategy\.(\w+)/);
|
||||
if (stratMatch) entry.strategy = stratMatch[1].toLowerCase();
|
||||
|
||||
// Extract browser: false (some adapters bypass browser entirely)
|
||||
const browserMatch = src.match(/browser\s*:\s*(true|false)/);
|
||||
if (browserMatch) entry.browser = browserMatch[1] === 'true';
|
||||
else entry.browser = entry.strategy !== 'public';
|
||||
|
||||
// Extract columns
|
||||
const colMatch = src.match(/columns\s*:\s*\[([^\]]*)\]/);
|
||||
if (colMatch) {
|
||||
entry.columns = colMatch[1].split(',').map(s => s.trim().replace(/^['"`]|['"`]$/g, '')).filter(Boolean);
|
||||
}
|
||||
|
||||
// Extract args array items: { name: '...', ... }
|
||||
const argsBlock = extractTsArgsBlock(src);
|
||||
if (argsBlock) {
|
||||
entry.args = parseTsArgsBlock(argsBlock);
|
||||
}
|
||||
|
||||
// Extract navigateBefore: false / true / 'https://...'
|
||||
const navBoolMatch = src.match(/navigateBefore\s*:\s*(true|false)/);
|
||||
if (navBoolMatch) {
|
||||
entry.navigateBefore = navBoolMatch[1] === 'true';
|
||||
} else {
|
||||
const navStringMatch = src.match(/navigateBefore\s*:\s*['"`]([^'"`]+)['"`]/);
|
||||
if (navStringMatch) entry.navigateBefore = navStringMatch[1];
|
||||
}
|
||||
|
||||
const deprecatedBoolMatch = src.match(/deprecated\s*:\s*(true|false)/);
|
||||
if (deprecatedBoolMatch) {
|
||||
entry.deprecated = deprecatedBoolMatch[1] === 'true';
|
||||
} else {
|
||||
const deprecatedStringMatch = src.match(/deprecated\s*:\s*['"`]([^'"`]+)['"`]/);
|
||||
if (deprecatedStringMatch) entry.deprecated = deprecatedStringMatch[1];
|
||||
}
|
||||
|
||||
const replacedByMatch = src.match(/replacedBy\s*:\s*['"`]([^'"`]+)['"`]/);
|
||||
if (replacedByMatch) entry.replacedBy = replacedByMatch[1];
|
||||
|
||||
return entry;
|
||||
const seen = new Set<string>();
|
||||
return runtimeCommands
|
||||
.filter((cmd) => {
|
||||
const key = fullName(cmd);
|
||||
if (seen.has(key)) return false;
|
||||
seen.add(key);
|
||||
return true;
|
||||
})
|
||||
.sort((a, b) => a.name.localeCompare(b.name))
|
||||
.map(cmd => toManifestEntry(cmd, modulePath));
|
||||
} catch (err) {
|
||||
// If parsing fails, log a warning (matching scanYaml behaviour) and skip the entry.
|
||||
process.stderr.write(`Warning: failed to scan ${filePath}: ${getErrorMessage(err)}\n`);
|
||||
return null;
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -288,7 +194,7 @@ export function shouldReplaceManifestEntry(current: ManifestEntry, next: Manifes
|
||||
return current.type === 'yaml' && next.type === 'ts';
|
||||
}
|
||||
|
||||
export function buildManifest(): ManifestEntry[] {
|
||||
export async function buildManifest(): Promise<ManifestEntry[]> {
|
||||
const manifest = new Map<string, ManifestEntry>();
|
||||
|
||||
if (fs.existsSync(CLIS_DIR)) {
|
||||
@@ -313,8 +219,8 @@ export function buildManifest(): ManifestEntry[] {
|
||||
(file.endsWith('.ts') && !file.endsWith('.d.ts') && !file.endsWith('.test.ts') && file !== 'index.ts') ||
|
||||
(file.endsWith('.js') && !file.endsWith('.d.js') && !file.endsWith('.test.js') && file !== 'index.js')
|
||||
) {
|
||||
const entry = scanTs(filePath, site);
|
||||
if (entry) {
|
||||
const entries = await loadTsManifestEntries(filePath, site);
|
||||
for (const entry of entries) {
|
||||
const key = `${entry.site}/${entry.name}`;
|
||||
const existing = manifest.get(key);
|
||||
if (!existing || shouldReplaceManifestEntry(existing, entry)) {
|
||||
@@ -329,11 +235,11 @@ export function buildManifest(): ManifestEntry[] {
|
||||
}
|
||||
}
|
||||
|
||||
return [...manifest.values()];
|
||||
return [...manifest.values()].sort((a, b) => a.site.localeCompare(b.site) || a.name.localeCompare(b.name));
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const manifest = buildManifest();
|
||||
async function main(): Promise<void> {
|
||||
const manifest = await buildManifest();
|
||||
fs.mkdirSync(path.dirname(OUTPUT), { recursive: true });
|
||||
fs.writeFileSync(OUTPUT, JSON.stringify(manifest, null, 2));
|
||||
|
||||
@@ -367,5 +273,5 @@ function main(): void {
|
||||
|
||||
const entrypoint = process.argv[1] ? pathToFileURL(path.resolve(process.argv[1])).href : null;
|
||||
if (entrypoint === import.meta.url) {
|
||||
main();
|
||||
void main();
|
||||
}
|
||||
|
||||
+427
-17
@@ -15,7 +15,15 @@ import { PKG_VERSION } from './version.js';
|
||||
import { printCompletionScript } from './completion.js';
|
||||
import { loadExternalClis, executeExternalCli, installExternalCli, registerExternalCli, isBinaryInstalled } from './external.js';
|
||||
import { registerAllCommands } from './commanderAdapter.js';
|
||||
import { getErrorMessage } from './errors.js';
|
||||
import { EXIT_CODES, getErrorMessage } from './errors.js';
|
||||
import { daemonStatus, daemonStop, daemonRestart } from './commands/daemon.js';
|
||||
|
||||
/** Create a browser page for operate commands. Uses 'operate' workspace for session persistence. */
|
||||
async function getOperatePage(): Promise<import('./types.js').IPage> {
|
||||
const { BrowserBridge } = await import('./browser/index.js');
|
||||
const bridge = new BrowserBridge();
|
||||
return bridge.connect({ timeout: 30, workspace: 'operate:default' });
|
||||
}
|
||||
|
||||
export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
const program = new Command();
|
||||
@@ -36,7 +44,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.option('--json', 'JSON output (deprecated)')
|
||||
.action((opts) => {
|
||||
const registry = getRegistry();
|
||||
const commands = [...registry.values()].sort((a, b) => fullName(a).localeCompare(fullName(b)));
|
||||
const commands = [...new Set(registry.values())].sort((a, b) => fullName(a).localeCompare(fullName(b)));
|
||||
const fmt = opts.json && opts.format === 'table' ? 'json' : opts.format;
|
||||
const isStructured = fmt === 'json' || fmt === 'yaml';
|
||||
|
||||
@@ -47,6 +55,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
command: fullName(c),
|
||||
site: c.site,
|
||||
name: c.name,
|
||||
aliases: c.aliases?.join(', ') ?? '',
|
||||
description: c.description,
|
||||
strategy: strategyLabel(c),
|
||||
browser: !!c.browser,
|
||||
@@ -54,7 +63,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
}));
|
||||
renderOutput(rows, {
|
||||
fmt,
|
||||
columns: ['command', 'site', 'name', 'description', 'strategy', 'browser', 'args',
|
||||
columns: ['command', 'site', 'name', 'aliases', 'description', 'strategy', 'browser', 'args',
|
||||
...(isStructured ? ['columns', 'domain'] : [])],
|
||||
title: 'opencli/list',
|
||||
source: 'opencli list',
|
||||
@@ -80,7 +89,8 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
const tag = label === 'public'
|
||||
? chalk.green('[public]')
|
||||
: chalk.yellow(`[${label}]`);
|
||||
console.log(` ${cmd.name} ${tag}${cmd.description ? chalk.dim(` — ${cmd.description}`) : ''}`);
|
||||
const aliases = cmd.aliases?.length ? chalk.dim(` (aliases: ${cmd.aliases.join(', ')})`) : '';
|
||||
console.log(` ${cmd.name} ${tag}${aliases}${cmd.description ? chalk.dim(` — ${cmd.description}`) : ''}`);
|
||||
}
|
||||
console.log();
|
||||
}
|
||||
@@ -120,7 +130,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
const { verifyClis, renderVerifyReport } = await import('./verify.js');
|
||||
const r = await verifyClis({ builtinClis: BUILTIN_CLIS, userClis: USER_CLIS, target, smoke: opts.smoke });
|
||||
console.log(renderVerifyReport(r));
|
||||
process.exitCode = r.ok ? 0 : 1;
|
||||
process.exitCode = r.ok ? EXIT_CODES.SUCCESS : EXIT_CODES.GENERIC_ERROR;
|
||||
});
|
||||
|
||||
// ── Built-in: explore / synthesize / generate / cascade ───────────────────
|
||||
@@ -180,7 +190,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
workspace,
|
||||
});
|
||||
console.log(renderGenerateSummary(r));
|
||||
process.exitCode = r.ok ? 0 : 1;
|
||||
process.exitCode = r.ok ? EXIT_CODES.SUCCESS : EXIT_CODES.GENERIC_ERROR;
|
||||
});
|
||||
|
||||
// ── Built-in: record ─────────────────────────────────────────────────────
|
||||
@@ -204,7 +214,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
timeoutMs: parseInt(opts.timeout, 10),
|
||||
});
|
||||
console.log(renderRecordSummary(result));
|
||||
process.exitCode = result.candidateCount > 0 ? 0 : 1;
|
||||
process.exitCode = result.candidateCount > 0 ? EXIT_CODES.SUCCESS : EXIT_CODES.EMPTY_RESULT;
|
||||
});
|
||||
|
||||
program
|
||||
@@ -226,6 +236,391 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
console.log(renderCascadeResult(result));
|
||||
});
|
||||
|
||||
// ── Built-in: operate (browser control for Claude Code skill) ───────────────
|
||||
//
|
||||
// Make websites accessible for AI agents.
|
||||
// All commands wrapped in operateAction() for consistent error handling.
|
||||
|
||||
const operate = program
|
||||
.command('operate')
|
||||
.description('Browser control — navigate, click, type, extract, wait (no LLM needed)');
|
||||
|
||||
/** Wrap operate actions with error handling and optional --json output */
|
||||
function operateAction(fn: (page: Awaited<ReturnType<typeof getOperatePage>>, ...args: any[]) => Promise<unknown>) {
|
||||
return async (...args: any[]) => {
|
||||
try {
|
||||
const page = await getOperatePage();
|
||||
await fn(page, ...args);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (msg.includes('Extension not connected') || msg.includes('Daemon')) {
|
||||
console.error(`Browser not connected. Run 'opencli doctor' to diagnose.`);
|
||||
} else if (msg.includes('attach failed') || msg.includes('chrome-extension://')) {
|
||||
console.error(`Browser attach failed — another extension may be interfering. Try disabling 1Password.`);
|
||||
} else {
|
||||
console.error(`Error: ${msg}`);
|
||||
}
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// ── Navigation ──
|
||||
|
||||
/** Network interceptor JS — injected on every open/navigate to capture fetch/XHR */
|
||||
const NETWORK_INTERCEPTOR_JS = `(function(){if(window.__opencli_net)return;window.__opencli_net=[];var M=200,B=50000,F=window.fetch;window.fetch=async function(){var r=await F.apply(this,arguments);try{var ct=r.headers.get('content-type')||'';if(ct.includes('json')||ct.includes('text')){var c=r.clone(),t=await c.text();if(window.__opencli_net.length<M){var b=null;if(t.length<=B)try{b=JSON.parse(t)}catch(e){b=t}window.__opencli_net.push({url:r.url||(arguments[0]&&arguments[0].url)||String(arguments[0]),method:(arguments[1]&&arguments[1].method)||'GET',status:r.status,size:t.length,ct:ct,body:b})}}}catch(e){}return r};var X=XMLHttpRequest.prototype,O=X.open,S=X.send;X.open=function(m,u){this._om=m;this._ou=u;return O.apply(this,arguments)};X.send=function(){var x=this;x.addEventListener('load',function(){try{var ct=x.getResponseHeader('content-type')||'';if((ct.includes('json')||ct.includes('text'))&&window.__opencli_net.length<M){var t=x.responseText,b=null;if(t&&t.length<=B)try{b=JSON.parse(t)}catch(e){b=t}window.__opencli_net.push({url:x._ou,method:x._om||'GET',status:x.status,size:t?t.length:0,ct:ct,body:b})}}catch(e){}});return S.apply(this,arguments)}})()`;
|
||||
|
||||
operate.command('open').argument('<url>').description('Open URL in automation window')
|
||||
.action(operateAction(async (page, url) => {
|
||||
await page.goto(url);
|
||||
await page.wait(2);
|
||||
// Auto-inject network interceptor for API discovery
|
||||
try { await page.evaluate(NETWORK_INTERCEPTOR_JS); } catch { /* non-fatal */ }
|
||||
console.log(`Navigated to: ${await page.getCurrentUrl?.() ?? url}`);
|
||||
}));
|
||||
|
||||
operate.command('back').description('Go back in browser history')
|
||||
.action(operateAction(async (page) => {
|
||||
await page.evaluate('history.back()');
|
||||
await page.wait(2);
|
||||
console.log('Navigated back');
|
||||
}));
|
||||
|
||||
operate.command('scroll').argument('<direction>', 'up or down').option('--amount <pixels>', 'Pixels to scroll', '500')
|
||||
.description('Scroll page')
|
||||
.action(operateAction(async (page, direction, opts) => {
|
||||
if (direction !== 'up' && direction !== 'down') {
|
||||
console.error(`Invalid direction "${direction}". Use "up" or "down".`);
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
await page.scroll(direction, parseInt(opts.amount, 10));
|
||||
console.log(`Scrolled ${direction}`);
|
||||
}));
|
||||
|
||||
// ── Inspect ──
|
||||
|
||||
operate.command('state').description('Page state: URL, title, interactive elements with [N] indices')
|
||||
.action(operateAction(async (page) => {
|
||||
const snapshot = await page.snapshot({ viewportExpand: 800 });
|
||||
const url = await page.getCurrentUrl?.() ?? '';
|
||||
console.log(`URL: ${url}\n`);
|
||||
console.log(typeof snapshot === 'string' ? snapshot : JSON.stringify(snapshot, null, 2));
|
||||
}));
|
||||
|
||||
operate.command('screenshot').argument('[path]', 'Save to file (base64 if omitted)')
|
||||
.description('Take screenshot')
|
||||
.action(operateAction(async (page, path) => {
|
||||
if (path) {
|
||||
await page.screenshot({ path });
|
||||
console.log(`Screenshot saved to: ${path}`);
|
||||
} else {
|
||||
console.log(await page.screenshot({ format: 'png' }));
|
||||
}
|
||||
}));
|
||||
|
||||
// ── Get commands (structured data extraction) ──
|
||||
|
||||
const get = operate.command('get').description('Get page properties');
|
||||
|
||||
get.command('title').description('Page title')
|
||||
.action(operateAction(async (page) => {
|
||||
console.log(await page.evaluate('document.title'));
|
||||
}));
|
||||
|
||||
get.command('url').description('Current page URL')
|
||||
.action(operateAction(async (page) => {
|
||||
console.log(await page.getCurrentUrl?.() ?? await page.evaluate('location.href'));
|
||||
}));
|
||||
|
||||
get.command('text').argument('<index>', 'Element index').description('Element text content')
|
||||
.action(operateAction(async (page, index) => {
|
||||
const text = await page.evaluate(`document.querySelector('[data-opencli-ref="${index}"]')?.textContent?.trim()`);
|
||||
console.log(text ?? '(empty)');
|
||||
}));
|
||||
|
||||
get.command('value').argument('<index>', 'Element index').description('Input/textarea value')
|
||||
.action(operateAction(async (page, index) => {
|
||||
const val = await page.evaluate(`document.querySelector('[data-opencli-ref="${index}"]')?.value`);
|
||||
console.log(val ?? '(empty)');
|
||||
}));
|
||||
|
||||
get.command('html').option('--selector <css>', 'CSS selector scope').description('Page HTML (or scoped)')
|
||||
.action(operateAction(async (page, opts) => {
|
||||
const sel = opts.selector ? JSON.stringify(opts.selector) : 'null';
|
||||
const html = await page.evaluate(`(${sel} ? document.querySelector(${sel})?.outerHTML : document.documentElement.outerHTML)?.slice(0, 50000)`);
|
||||
console.log(html ?? '(empty)');
|
||||
}));
|
||||
|
||||
get.command('attributes').argument('<index>', 'Element index').description('Element attributes')
|
||||
.action(operateAction(async (page, index) => {
|
||||
const attrs = await page.evaluate(`JSON.stringify(Object.fromEntries([...document.querySelector('[data-opencli-ref="${index}"]')?.attributes].map(a=>[a.name,a.value])))`);
|
||||
console.log(attrs ?? '{}');
|
||||
}));
|
||||
|
||||
// ── Interact ──
|
||||
|
||||
operate.command('click').argument('<index>', 'Element index from state').description('Click element by index')
|
||||
.action(operateAction(async (page, index) => {
|
||||
await page.click(index);
|
||||
console.log(`Clicked element [${index}]`);
|
||||
}));
|
||||
|
||||
operate.command('type').argument('<index>', 'Element index').argument('<text>', 'Text to type')
|
||||
.description('Click element, then type text')
|
||||
.action(operateAction(async (page, index, text) => {
|
||||
await page.click(index);
|
||||
await page.wait(0.3);
|
||||
await page.typeText(index, text);
|
||||
console.log(`Typed "${text}" into element [${index}]`);
|
||||
}));
|
||||
|
||||
operate.command('select').argument('<index>', 'Element index of <select>').argument('<option>', 'Option text')
|
||||
.description('Select dropdown option')
|
||||
.action(operateAction(async (page, index, option) => {
|
||||
const result = await page.evaluate(`
|
||||
(function() {
|
||||
var sel = document.querySelector('[data-opencli-ref="${index}"]');
|
||||
if (!sel || sel.tagName !== 'SELECT') return { error: 'Not a <select>' };
|
||||
var match = Array.from(sel.options).find(o => o.text.trim() === ${JSON.stringify(option)} || o.value === ${JSON.stringify(option)});
|
||||
if (!match) return { error: 'Option not found', available: Array.from(sel.options).map(o => o.text.trim()) };
|
||||
var setter = Object.getOwnPropertyDescriptor(HTMLSelectElement.prototype, 'value')?.set;
|
||||
if (setter) setter.call(sel, match.value); else sel.value = match.value;
|
||||
sel.dispatchEvent(new Event('input', {bubbles:true}));
|
||||
sel.dispatchEvent(new Event('change', {bubbles:true}));
|
||||
return { selected: match.text };
|
||||
})()
|
||||
`) as { error?: string; selected?: string; available?: string[] } | null;
|
||||
if (result?.error) {
|
||||
console.error(`Error: ${result.error}${result.available ? ` — Available: ${result.available.join(', ')}` : ''}`);
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
} else {
|
||||
console.log(`Selected "${result?.selected}" in element [${index}]`);
|
||||
}
|
||||
}));
|
||||
|
||||
operate.command('keys').argument('<key>', 'Key to press (Enter, Escape, Tab, Control+a)')
|
||||
.description('Press keyboard key')
|
||||
.action(operateAction(async (page, key) => {
|
||||
await page.pressKey(key);
|
||||
console.log(`Pressed: ${key}`);
|
||||
}));
|
||||
|
||||
// ── Wait commands ──
|
||||
|
||||
operate.command('wait')
|
||||
.argument('<type>', 'selector, text, or time')
|
||||
.argument('[value]', 'CSS selector, text string, or seconds')
|
||||
.option('--timeout <ms>', 'Timeout in milliseconds', '10000')
|
||||
.description('Wait for selector, text, or time (e.g. wait selector ".loaded", wait text "Success", wait time 3)')
|
||||
.action(operateAction(async (page, type, value, opts) => {
|
||||
const timeout = parseInt(opts.timeout, 10);
|
||||
if (type === 'time') {
|
||||
const seconds = parseFloat(value ?? '2');
|
||||
await page.wait(seconds);
|
||||
console.log(`Waited ${seconds}s`);
|
||||
} else if (type === 'selector') {
|
||||
if (!value) { console.error('Missing CSS selector'); process.exitCode = EXIT_CODES.USAGE_ERROR; return; }
|
||||
await page.wait({ selector: value, timeout: timeout / 1000 });
|
||||
console.log(`Element "${value}" appeared`);
|
||||
} else if (type === 'text') {
|
||||
if (!value) { console.error('Missing text'); process.exitCode = EXIT_CODES.USAGE_ERROR; return; }
|
||||
await page.wait({ text: value, timeout: timeout / 1000 });
|
||||
console.log(`Text "${value}" appeared`);
|
||||
} else {
|
||||
console.error(`Unknown wait type "${type}". Use: selector, text, or time`);
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
}
|
||||
}));
|
||||
|
||||
// ── Extract ──
|
||||
|
||||
operate.command('eval').argument('<js>', 'JavaScript code').description('Execute JS in page context, return result')
|
||||
.action(operateAction(async (page, js) => {
|
||||
const result = await page.evaluate(js);
|
||||
if (typeof result === 'string') console.log(result);
|
||||
else console.log(JSON.stringify(result, null, 2));
|
||||
}));
|
||||
|
||||
// ── Network (API discovery) ──
|
||||
|
||||
operate.command('network')
|
||||
.option('--detail <index>', 'Show full response body of request at index')
|
||||
.option('--all', 'Show all requests including static resources')
|
||||
.description('Show captured network requests (auto-captured since last open)')
|
||||
.action(operateAction(async (page, opts) => {
|
||||
const requests = await page.evaluate(`(function(){
|
||||
var reqs = window.__opencli_net || [];
|
||||
return JSON.stringify(reqs);
|
||||
})()`) as string;
|
||||
|
||||
let items: Array<{ url: string; method: string; status: number; size: number; ct: string; body: unknown }> = [];
|
||||
try { items = JSON.parse(requests); } catch { console.log('No network data captured. Run "operate open <url>" first.'); return; }
|
||||
|
||||
if (items.length === 0) { console.log('No requests captured.'); return; }
|
||||
|
||||
// Filter out static resources unless --all
|
||||
if (!opts.all) {
|
||||
items = items.filter(r =>
|
||||
(r.ct?.includes('json') || r.ct?.includes('xml') || r.ct?.includes('text/plain')) &&
|
||||
!/\.(js|css|png|jpg|gif|svg|woff|ico|map)(\?|$)/i.test(r.url) &&
|
||||
!/analytics|tracking|telemetry|beacon|pixel|gtag|fbevents/i.test(r.url)
|
||||
);
|
||||
}
|
||||
|
||||
if (opts.detail !== undefined) {
|
||||
const idx = parseInt(opts.detail, 10);
|
||||
const req = items[idx];
|
||||
if (!req) { console.error(`Request #${idx} not found. ${items.length} requests available.`); process.exitCode = EXIT_CODES.USAGE_ERROR; return; }
|
||||
console.log(`${req.method} ${req.url}`);
|
||||
console.log(`Status: ${req.status} | Size: ${req.size} | Type: ${req.ct}`);
|
||||
console.log('---');
|
||||
console.log(typeof req.body === 'string' ? req.body : JSON.stringify(req.body, null, 2));
|
||||
} else {
|
||||
console.log(`Captured ${items.length} API requests:\n`);
|
||||
items.forEach((r, i) => {
|
||||
const bodyPreview = r.body ? (typeof r.body === 'string' ? r.body.slice(0, 60) : JSON.stringify(r.body).slice(0, 60)) : '';
|
||||
console.log(` [${i}] ${r.method} ${r.status} ${r.url.slice(0, 80)}`);
|
||||
if (bodyPreview) console.log(` ${bodyPreview}...`);
|
||||
});
|
||||
console.log(`\nUse --detail <index> to see full response body.`);
|
||||
}
|
||||
}));
|
||||
|
||||
// ── Init (adapter scaffolding) ──
|
||||
|
||||
operate.command('init')
|
||||
.argument('<name>', 'Adapter name in site/command format (e.g. hn/top)')
|
||||
.description('Generate adapter scaffold in ~/.opencli/clis/')
|
||||
.action(async (name: string) => {
|
||||
try {
|
||||
const parts = name.split('/');
|
||||
if (parts.length !== 2 || !parts[0] || !parts[1]) {
|
||||
console.error('Name must be site/command format (e.g. hn/top)');
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
const [site, command] = parts;
|
||||
if (!/^[a-zA-Z0-9_-]+$/.test(site) || !/^[a-zA-Z0-9_-]+$/.test(command)) {
|
||||
console.error('Name parts must be alphanumeric/dash/underscore only');
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
|
||||
const os = await import('node:os');
|
||||
const fs = await import('node:fs');
|
||||
const path = await import('node:path');
|
||||
const dir = path.join(os.homedir(), '.opencli', 'clis', site);
|
||||
const filePath = path.join(dir, `${command}.ts`);
|
||||
|
||||
if (fs.existsSync(filePath)) {
|
||||
console.log(`Adapter already exists: ${filePath}`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Try to detect domain from last operate session
|
||||
let domain = site;
|
||||
try {
|
||||
const page = await getOperatePage();
|
||||
const url = await page.getCurrentUrl?.();
|
||||
if (url) { try { domain = new URL(url).hostname; } catch {} }
|
||||
} catch { /* no active session */ }
|
||||
|
||||
const template = `import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: '${site}',
|
||||
name: '${command}',
|
||||
description: '', // TODO: describe what this command does
|
||||
domain: '${domain}',
|
||||
strategy: Strategy.PUBLIC, // TODO: PUBLIC (no auth), COOKIE (needs login), UI (DOM interaction)
|
||||
browser: false, // TODO: set true if needs browser
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of items' },
|
||||
],
|
||||
columns: [], // TODO: field names for table output (e.g. ['title', 'score', 'url'])
|
||||
func: async (page, kwargs) => {
|
||||
// TODO: implement data fetching
|
||||
// Prefer API calls (fetch) over browser automation
|
||||
// page is available if browser: true
|
||||
return [];
|
||||
},
|
||||
});
|
||||
`;
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(filePath, template, 'utf-8');
|
||||
console.log(`Created: ${filePath}`);
|
||||
console.log(`Edit the file to implement your adapter, then run: opencli operate verify ${name}`);
|
||||
} catch (err) {
|
||||
console.error(`Error: ${err instanceof Error ? err.message : String(err)}`);
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
});
|
||||
|
||||
// ── Verify (test adapter) ──
|
||||
|
||||
operate.command('verify')
|
||||
.argument('<name>', 'Adapter name in site/command format (e.g. hn/top)')
|
||||
.description('Execute an adapter and show results')
|
||||
.action(async (name: string) => {
|
||||
try {
|
||||
const parts = name.split('/');
|
||||
if (parts.length !== 2) { console.error('Name must be site/command format'); process.exitCode = EXIT_CODES.USAGE_ERROR; return; }
|
||||
const [site, command] = parts;
|
||||
if (!/^[a-zA-Z0-9_-]+$/.test(site) || !/^[a-zA-Z0-9_-]+$/.test(command)) {
|
||||
console.error('Name parts must be alphanumeric/dash/underscore only');
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
|
||||
const { execSync } = await import('node:child_process');
|
||||
const os = await import('node:os');
|
||||
const path = await import('node:path');
|
||||
const filePath = path.join(os.homedir(), '.opencli', 'clis', site, `${command}.ts`);
|
||||
|
||||
const fs = await import('node:fs');
|
||||
if (!fs.existsSync(filePath)) {
|
||||
console.error(`Adapter not found: ${filePath}`);
|
||||
console.error(`Run "opencli operate init ${name}" to create it.`);
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`🔍 Verifying ${name}...\n`);
|
||||
console.log(` Loading: ${filePath}`);
|
||||
|
||||
try {
|
||||
const output = execSync(`node dist/main.js ${site} ${command} --limit 3`, {
|
||||
cwd: path.join(path.dirname(import.meta.url.replace('file://', '')), '..'),
|
||||
timeout: 30000,
|
||||
encoding: 'utf-8',
|
||||
env: process.env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
});
|
||||
console.log(` Executing: opencli ${site} ${command} --limit 3\n`);
|
||||
console.log(output);
|
||||
console.log(`\n ✓ Adapter works!`);
|
||||
} catch (err: any) {
|
||||
console.log(` Executing: opencli ${site} ${command} --limit 3\n`);
|
||||
if (err.stdout) console.log(err.stdout);
|
||||
if (err.stderr) console.error(err.stderr.slice(0, 500));
|
||||
console.log(`\n ✗ Adapter failed. Fix the code and try again.`);
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`Error: ${err instanceof Error ? err.message : String(err)}`);
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
});
|
||||
|
||||
// ── Session ──
|
||||
|
||||
operate.command('close').description('Close the automation window')
|
||||
.action(operateAction(async (page) => {
|
||||
await page.closeWindow?.();
|
||||
console.log('Automation window closed');
|
||||
}));
|
||||
|
||||
// ── Built-in: doctor / completion ──────────────────────────────────────────
|
||||
|
||||
program
|
||||
@@ -272,7 +667,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Error: ${getErrorMessage(err)}`));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
});
|
||||
|
||||
@@ -287,7 +682,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
console.log(chalk.green(`✅ Plugin "${name}" uninstalled.`));
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Error: ${getErrorMessage(err)}`));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
});
|
||||
|
||||
@@ -299,12 +694,12 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.action(async (name: string | undefined, opts: { all?: boolean }) => {
|
||||
if (!name && !opts.all) {
|
||||
console.error(chalk.red('Error: Please specify a plugin name or use the --all flag.'));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
if (name && opts.all) {
|
||||
console.error(chalk.red('Error: Cannot specify both a plugin name and --all.'));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -335,7 +730,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
console.log();
|
||||
if (hasErrors) {
|
||||
console.error(chalk.red('Completed with some errors.'));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
} else {
|
||||
console.log(chalk.green('✅ All plugins updated successfully.'));
|
||||
}
|
||||
@@ -348,7 +743,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
console.log(chalk.green(`✅ Plugin "${name}" updated successfully.`));
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Error: ${getErrorMessage(err)}`));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
});
|
||||
|
||||
@@ -438,10 +833,25 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
console.log(chalk.dim(` opencli ${name} hello`));
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Error: ${getErrorMessage(err)}`));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
});
|
||||
|
||||
// ── Built-in: daemon ──────────────────────────────────────────────────────
|
||||
const daemonCmd = program.command('daemon').description('Manage the opencli daemon');
|
||||
daemonCmd
|
||||
.command('status')
|
||||
.description('Show daemon status')
|
||||
.action(async () => { await daemonStatus(); });
|
||||
daemonCmd
|
||||
.command('stop')
|
||||
.description('Stop the daemon')
|
||||
.action(async () => { await daemonStop(); });
|
||||
daemonCmd
|
||||
.command('restart')
|
||||
.description('Restart the daemon')
|
||||
.action(async () => { await daemonRestart(); });
|
||||
|
||||
// ── External CLIs ─────────────────────────────────────────────────────────
|
||||
|
||||
const externalClis = loadExternalClis();
|
||||
@@ -454,7 +864,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
const ext = externalClis.find(e => e.name === name);
|
||||
if (!ext) {
|
||||
console.error(chalk.red(`External CLI '${name}' not found in registry.`));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
return;
|
||||
}
|
||||
installExternalCli(ext);
|
||||
@@ -480,7 +890,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
executeExternalCli(name, args, externalClis);
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Error: ${getErrorMessage(err)}`));
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.GENERIC_ERROR;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -525,7 +935,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
console.error(chalk.dim(` Tip: '${binary}' exists on your PATH. Use 'opencli register ${binary}' to add it as an external CLI.`));
|
||||
}
|
||||
program.outputHelp();
|
||||
process.exitCode = 1;
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
});
|
||||
|
||||
program.parse();
|
||||
|
||||
@@ -60,7 +60,7 @@ cli({
|
||||
|
||||
await page.installInterceptor('36kr.com/api');
|
||||
await page.goto(url);
|
||||
await page.wait(6);
|
||||
await page.waitForCapture(6);
|
||||
|
||||
// Scrape rendered article links from DOM (deduplicated)
|
||||
const domItems: any = await page.evaluate(`
|
||||
|
||||
@@ -24,7 +24,7 @@ cli({
|
||||
|
||||
await page.installInterceptor('36kr.com/api');
|
||||
await page.goto(`https://www.36kr.com/search/articles/${query}`);
|
||||
await page.wait(6);
|
||||
await page.waitForCapture(6);
|
||||
|
||||
const domItems: any = await page.evaluate(`
|
||||
(() => {
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
/**
|
||||
* Shared utilities for CLI adapters.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Clamp a numeric value to [min, max].
|
||||
* Matches the signature of lodash.clamp and Rust's clamp.
|
||||
*/
|
||||
export function clamp(value: number, min: number, max: number): number {
|
||||
return Math.max(min, Math.min(value, max));
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './bestsellers.js';
|
||||
|
||||
describe('amazon bestsellers normalization', () => {
|
||||
it('normalizes bestseller cards and infers review counts from card text', () => {
|
||||
const result = __test__.normalizeBestsellerCandidate({
|
||||
asin: 'B0DR31GC3D',
|
||||
title: '',
|
||||
href: 'https://www.amazon.com/NUTIKAS-Shelves-Desktop-Orgnizer-Shlef/dp/B0DR31GC3D/ref=zg_bs',
|
||||
price_text: '$25.92',
|
||||
rating_text: '4.3 out of 5 stars',
|
||||
review_count_text: '',
|
||||
card_text: 'Desk Shelves Desktop Organizer Shlef\n4.3 out of 5 stars\n435\n$25.92',
|
||||
}, 2, 'Amazon Best Sellers: Best Desktop & Off-Surface Shelves', 'https://www.amazon.com/example');
|
||||
|
||||
expect(result.rank).toBe(2);
|
||||
expect(result.asin).toBe('B0DR31GC3D');
|
||||
expect(result.title).toBe('Desk Shelves Desktop Organizer Shlef');
|
||||
expect(result.review_count).toBe(435);
|
||||
expect(result.list_title).toBe('Amazon Best Sellers: Best Desktop & Off-Surface Shelves');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,180 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
buildProvenance,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
extractReviewCountFromCardText,
|
||||
firstMeaningfulLine,
|
||||
normalizeProductUrl,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
resolveBestsellersUrl,
|
||||
uniqueNonEmpty,
|
||||
assertUsableState,
|
||||
gotoAndReadState,
|
||||
} from './shared.js';
|
||||
|
||||
interface BestsellersPagePayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
list_title?: string;
|
||||
cards?: Array<{
|
||||
rank_text?: string | null;
|
||||
asin?: string | null;
|
||||
title?: string | null;
|
||||
href?: string | null;
|
||||
price_text?: string | null;
|
||||
rating_text?: string | null;
|
||||
review_count_text?: string | null;
|
||||
card_text?: string | null;
|
||||
}>;
|
||||
page_links?: string[];
|
||||
}
|
||||
|
||||
function normalizeBestsellerCandidate(
|
||||
candidate: NonNullable<BestsellersPagePayload['cards']>[number],
|
||||
rank: number,
|
||||
listTitle: string | null,
|
||||
sourceUrl: string,
|
||||
): Record<string, unknown> {
|
||||
const productUrl = normalizeProductUrl(candidate.href);
|
||||
const asin = extractAsin(candidate.asin ?? '') ?? extractAsin(productUrl ?? '') ?? null;
|
||||
const title = cleanText(candidate.title) || firstMeaningfulLine(candidate.card_text);
|
||||
const price = parsePriceText(cleanText(candidate.price_text) || candidate.card_text);
|
||||
const ratingText = cleanText(candidate.rating_text) || null;
|
||||
const reviewCountText = cleanText(candidate.review_count_text)
|
||||
|| extractReviewCountFromCardText(candidate.card_text)
|
||||
|| null;
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
rank,
|
||||
asin,
|
||||
title: title || null,
|
||||
product_url: productUrl,
|
||||
list_title: listTitle,
|
||||
...provenance,
|
||||
price_text: price.price_text,
|
||||
price_value: price.price_value,
|
||||
currency: price.currency,
|
||||
rating_text: ratingText,
|
||||
rating_value: parseRatingValue(ratingText),
|
||||
review_count_text: reviewCountText,
|
||||
review_count: parseReviewCount(reviewCountText),
|
||||
};
|
||||
}
|
||||
|
||||
async function readBestsellersPage(page: IPage, url: string): Promise<BestsellersPagePayload> {
|
||||
const state = await gotoAndReadState(page, url, 2500, 'bestsellers');
|
||||
assertUsableState(state, 'bestsellers');
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
list_title:
|
||||
document.querySelector('#zg_banner_text')?.textContent
|
||||
|| document.querySelector('h1')?.textContent
|
||||
|| '',
|
||||
cards: Array.from(document.querySelectorAll('.p13n-sc-uncoverable-faceout'))
|
||||
.map((card) => ({
|
||||
rank_text:
|
||||
card.querySelector('.zg-bdg-text')?.textContent
|
||||
|| card.querySelector('[class*="rank"]')?.textContent
|
||||
|| '',
|
||||
asin: card.id || '',
|
||||
title:
|
||||
card.querySelector('[class*="line-clamp"]')?.textContent
|
||||
|| card.querySelector('img')?.getAttribute('alt')
|
||||
|| '',
|
||||
href: card.querySelector('a[href*="/dp/"]')?.href || '',
|
||||
price_text: card.querySelector('.a-price .a-offscreen')?.textContent || '',
|
||||
rating_text: card.querySelector('[aria-label*="out of 5 stars"]')?.getAttribute('aria-label') || '',
|
||||
review_count_text:
|
||||
card.querySelector('a[href*="#customerReviews"]')?.textContent
|
||||
|| card.querySelector('.a-size-small')?.textContent
|
||||
|| '',
|
||||
card_text: card.innerText || '',
|
||||
})),
|
||||
page_links: Array.from(document.querySelectorAll('li.a-normal a, li.a-selected a'))
|
||||
.map((anchor) => anchor.href || '')
|
||||
.filter((href) => /\\/zgbs\\//.test(href) && /(?:[?&]pg=|ref=zg_bs_pg_)/.test(href)),
|
||||
}))()
|
||||
`) as BestsellersPagePayload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'amazon',
|
||||
name: 'bestsellers',
|
||||
description: 'Amazon Best Sellers pages for category candidate discovery',
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
positional: true,
|
||||
help: 'Best sellers URL or /zgbs path. Omit to use the root Best Sellers page.',
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: 100,
|
||||
help: 'Maximum number of ranked items to return (default 100)',
|
||||
},
|
||||
],
|
||||
columns: ['rank', 'asin', 'title', 'price_text', 'rating_value', 'review_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = Math.max(1, Number(kwargs.limit) || 100);
|
||||
const initialUrl = resolveBestsellersUrl(typeof kwargs.input === 'string' ? kwargs.input : undefined);
|
||||
|
||||
const queue = [initialUrl];
|
||||
const visited = new Set<string>();
|
||||
const seenAsins = new Set<string>();
|
||||
const results: Record<string, unknown>[] = [];
|
||||
let listTitle: string | null = null;
|
||||
|
||||
while (queue.length > 0 && results.length < limit) {
|
||||
const nextUrl = queue.shift()!;
|
||||
if (visited.has(nextUrl)) continue;
|
||||
visited.add(nextUrl);
|
||||
|
||||
const payload = await readBestsellersPage(page, nextUrl);
|
||||
const sourceUrl = cleanText(payload.href) || nextUrl;
|
||||
listTitle = cleanText(payload.list_title) || cleanText(payload.title) || listTitle;
|
||||
const cards = payload.cards ?? [];
|
||||
|
||||
for (const card of cards) {
|
||||
const normalized = normalizeBestsellerCandidate(card, results.length + 1, listTitle, sourceUrl);
|
||||
const asin = cleanText(String(normalized.asin ?? ''));
|
||||
if (!asin || seenAsins.has(asin)) continue;
|
||||
seenAsins.add(asin);
|
||||
results.push(normalized);
|
||||
if (results.length >= limit) break;
|
||||
}
|
||||
|
||||
const pageLinks = uniqueNonEmpty(payload.page_links ?? []);
|
||||
for (const href of pageLinks) {
|
||||
if (!visited.has(href) && !queue.includes(href)) {
|
||||
queue.push(href);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (results.length === 0) {
|
||||
throw new CommandExecutionError(
|
||||
'amazon bestsellers did not expose any ranked items',
|
||||
'Open the same best sellers page in Chrome, verify it is a real Amazon ranking page, and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
return results.slice(0, limit);
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeBestsellerCandidate,
|
||||
};
|
||||
@@ -0,0 +1,38 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './discussion.js';
|
||||
|
||||
describe('amazon discussion normalization', () => {
|
||||
it('normalizes review summary and sample reviews', () => {
|
||||
const result = __test__.normalizeDiscussionPayload({
|
||||
href: 'https://www.amazon.com/product-reviews/B0FJS72893',
|
||||
average_rating_text: '3.9 out of 5',
|
||||
total_review_count_text: '27 global ratings',
|
||||
qa_links: [],
|
||||
review_samples: [
|
||||
{
|
||||
title: '5.0 out of 5 stars Great value and quality',
|
||||
rating_text: '5.0 out of 5 stars',
|
||||
author: 'GTreader2',
|
||||
date_text: 'Reviewed in the United States on February 21, 2026',
|
||||
body: 'Small but mighty.',
|
||||
verified: true,
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(result.asin).toBe('B0FJS72893');
|
||||
expect(result.average_rating_value).toBe(3.9);
|
||||
expect(result.total_review_count).toBe(27);
|
||||
expect(result.review_samples).toEqual([
|
||||
{
|
||||
title: 'Great value and quality',
|
||||
rating_text: '5.0 out of 5 stars',
|
||||
rating_value: 5,
|
||||
author: 'GTreader2',
|
||||
date_text: 'Reviewed in the United States on February 21, 2026',
|
||||
body: 'Small but mighty.',
|
||||
verified_purchase: true,
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
buildDiscussionUrl,
|
||||
buildProvenance,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
normalizeProductUrl,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
trimRatingPrefix,
|
||||
uniqueNonEmpty,
|
||||
assertUsableState,
|
||||
gotoAndReadState,
|
||||
} from './shared.js';
|
||||
|
||||
interface DiscussionPayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
average_rating_text?: string | null;
|
||||
total_review_count_text?: string | null;
|
||||
qa_links?: string[];
|
||||
review_samples?: Array<{
|
||||
title?: string | null;
|
||||
rating_text?: string | null;
|
||||
author?: string | null;
|
||||
date_text?: string | null;
|
||||
body?: string | null;
|
||||
verified?: boolean;
|
||||
}>;
|
||||
}
|
||||
|
||||
function normalizeDiscussionPayload(payload: DiscussionPayload): Record<string, unknown> {
|
||||
const sourceUrl = cleanText(payload.href) || buildDiscussionUrl(payload.href ?? '');
|
||||
const asin = extractAsin(payload.href ?? '') ?? null;
|
||||
const averageRatingText = cleanText(payload.average_rating_text) || null;
|
||||
const totalReviewCountText = cleanText(payload.total_review_count_text) || null;
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
asin,
|
||||
product_url: asin ? normalizeProductUrl(asin) : null,
|
||||
discussion_url: sourceUrl,
|
||||
...provenance,
|
||||
average_rating_text: averageRatingText,
|
||||
average_rating_value: parseRatingValue(averageRatingText),
|
||||
total_review_count_text: totalReviewCountText,
|
||||
total_review_count: parseReviewCount(totalReviewCountText),
|
||||
qa_urls: uniqueNonEmpty(payload.qa_links ?? []),
|
||||
review_samples: (payload.review_samples ?? []).map((sample) => ({
|
||||
title: trimRatingPrefix(sample.title) || null,
|
||||
rating_text: cleanText(sample.rating_text) || null,
|
||||
rating_value: parseRatingValue(sample.rating_text),
|
||||
author: cleanText(sample.author) || null,
|
||||
date_text: cleanText(sample.date_text) || null,
|
||||
body: cleanText(sample.body) || null,
|
||||
verified_purchase: sample.verified === true,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
async function readDiscussionPayload(page: IPage, input: string, limit: number): Promise<DiscussionPayload> {
|
||||
const url = buildDiscussionUrl(input);
|
||||
const state = await gotoAndReadState(page, url, 2500, 'discussion');
|
||||
assertUsableState(state, 'discussion');
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
average_rating_text: document.querySelector('[data-hook="rating-out-of-text"]')?.textContent || '',
|
||||
total_review_count_text: document.querySelector('[data-hook="total-review-count"]')?.textContent || '',
|
||||
qa_links: Array.from(document.querySelectorAll('a[href*="ask/questions"]')).map((anchor) => anchor.href || ''),
|
||||
review_samples: Array.from(document.querySelectorAll('[data-hook="review"]')).slice(0, ${limit}).map((card) => ({
|
||||
title: card.querySelector('[data-hook="review-title"]')?.textContent || '',
|
||||
rating_text:
|
||||
card.querySelector('[data-hook="review-star-rating"]')?.textContent
|
||||
|| card.querySelector('[data-hook="cmps-review-star-rating"]')?.textContent
|
||||
|| '',
|
||||
author: card.querySelector('.a-profile-name')?.textContent || '',
|
||||
date_text: card.querySelector('[data-hook="review-date"]')?.textContent || '',
|
||||
body: card.querySelector('[data-hook="review-body"]')?.textContent || '',
|
||||
verified: !!card.querySelector('[data-hook="avp-badge"]'),
|
||||
})),
|
||||
}))()
|
||||
`) as DiscussionPayload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'amazon',
|
||||
name: 'discussion',
|
||||
description: 'Amazon review summary and sample customer discussion from product review pages',
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: 'ASIN or product URL, for example B0FJS72893',
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: 10,
|
||||
help: 'Maximum number of review samples to return (default 10)',
|
||||
},
|
||||
],
|
||||
columns: ['asin', 'average_rating_value', 'total_review_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const input = String(kwargs.input ?? '');
|
||||
const limit = Math.max(1, Number(kwargs.limit) || 10);
|
||||
const payload = await readDiscussionPayload(page, input, limit);
|
||||
const normalized = normalizeDiscussionPayload(payload);
|
||||
|
||||
if (!normalized.average_rating_text && !normalized.total_review_count_text) {
|
||||
throw new CommandExecutionError(
|
||||
'amazon discussion page did not expose review summary',
|
||||
'The review page may have changed or hit a robot check. Open the review page in Chrome and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
return [normalized];
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeDiscussionPayload,
|
||||
};
|
||||
@@ -0,0 +1,35 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './offer.js';
|
||||
|
||||
describe('amazon offer normalization', () => {
|
||||
it('extracts sold-by and fulfillment facts from product offer text', () => {
|
||||
const result = __test__.normalizeOfferPayload({
|
||||
href: 'https://www.amazon.com/dp/B0FJS72893',
|
||||
price_text: '$15.99',
|
||||
merchant_info: '',
|
||||
sold_by: 'KUATUDIRECT',
|
||||
ships_from_text: 'Ships from Amazon',
|
||||
offer_link: null,
|
||||
review_url: 'https://www.amazon.com/dp/B0FJS72893#customerReviews',
|
||||
qa_url: null,
|
||||
});
|
||||
|
||||
expect(result.asin).toBe('B0FJS72893');
|
||||
expect(result.sold_by).toBe('KUATUDIRECT');
|
||||
expect(result.ships_from).toBe('Amazon');
|
||||
expect(result.is_amazon_sold).toBe(false);
|
||||
expect(result.is_amazon_fulfilled).toBe(true);
|
||||
});
|
||||
|
||||
it('parses merchant info fallback text', () => {
|
||||
expect(__test__.extractSoldBy('Sold by Example Seller and Fulfilled by Amazon.')).toBe('Example Seller');
|
||||
expect(__test__.extractShipsFrom('Ships from Amazon')).toBe('Amazon');
|
||||
});
|
||||
|
||||
it('detects delivery-location blocking in the buy box text', () => {
|
||||
expect(__test__.isDeliveryLocationBlocked(
|
||||
'This item cannot be shipped to your selected delivery location. Similar items shipping to Hong Kong',
|
||||
)).toBe(true);
|
||||
expect(__test__.isDeliveryLocationBlocked('Ships from Amazon')).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,185 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
buildProductUrl,
|
||||
buildProvenance,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
isAmazonEntity,
|
||||
normalizeProductUrl,
|
||||
PRIMARY_PRICE_SELECTORS,
|
||||
parsePriceText,
|
||||
assertUsableState,
|
||||
gotoAndReadState,
|
||||
} from './shared.js';
|
||||
|
||||
interface OfferPayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
price_text?: string | null;
|
||||
merchant_info?: string | null;
|
||||
sold_by?: string | null;
|
||||
ships_from_text?: string | null;
|
||||
offer_link?: string | null;
|
||||
review_url?: string | null;
|
||||
qa_url?: string | null;
|
||||
buybox_text?: string | null;
|
||||
}
|
||||
|
||||
const OFFER_FACT_SELECTOR = [
|
||||
'#sellerProfileTriggerId',
|
||||
'#shipsFromSoldByInsideBuyBox_feature_div',
|
||||
'#fulfillerInfoFeature_feature_div',
|
||||
'#merchantInfoFeature_feature_div',
|
||||
'#tabular-buybox-container',
|
||||
'#merchant-info',
|
||||
].join(', ');
|
||||
|
||||
function collapseAdjacentWords(text: string): string {
|
||||
const parts = cleanText(text).split(' ').filter(Boolean);
|
||||
const deduped: string[] = [];
|
||||
for (const part of parts) {
|
||||
if (deduped[deduped.length - 1] === part) continue;
|
||||
deduped.push(part);
|
||||
}
|
||||
return deduped.join(' ');
|
||||
}
|
||||
|
||||
function extractShipsFrom(text: string): string | null {
|
||||
const normalized = cleanText(text);
|
||||
const match = normalized.match(/Ships from\s+(.+?)(?=Sold by|and Fulfilled by|$)/i);
|
||||
return match ? collapseAdjacentWords(match[1].replace(/Ships from/ig, '')) : null;
|
||||
}
|
||||
|
||||
function extractSoldBy(text: string): string | null {
|
||||
const normalized = cleanText(text);
|
||||
const match = normalized.match(/Sold by\s+(.+?)(?=and Fulfilled by|Ships from|$)/i);
|
||||
return match ? collapseAdjacentWords(match[1]) : null;
|
||||
}
|
||||
|
||||
function isDeliveryLocationBlocked(text: string | null | undefined): boolean {
|
||||
const normalized = cleanText(text).toLowerCase();
|
||||
return normalized.includes('cannot be shipped to your selected delivery location')
|
||||
|| normalized.includes('similar items shipping to')
|
||||
|| normalized.includes('deliver to hong kong');
|
||||
}
|
||||
|
||||
function normalizeOfferPayload(payload: OfferPayload): Record<string, unknown> {
|
||||
const asin = extractAsin(payload.href ?? '') ?? null;
|
||||
const sourceUrl = cleanText(payload.href) || buildProductUrl(payload.href ?? '');
|
||||
const price = parsePriceText(payload.price_text);
|
||||
const merchantInfo = cleanText(payload.merchant_info) || null;
|
||||
const soldBy = cleanText(payload.sold_by)
|
||||
|| extractSoldBy(payload.ships_from_text ?? '')
|
||||
|| extractSoldBy(merchantInfo ?? '')
|
||||
|| null;
|
||||
const shipsFrom = extractShipsFrom(payload.ships_from_text ?? '')
|
||||
|| extractShipsFrom(merchantInfo ?? '')
|
||||
|| cleanText(payload.ships_from_text)
|
||||
|| null;
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
asin,
|
||||
product_url: normalizeProductUrl(payload.href),
|
||||
...provenance,
|
||||
price_text: price.price_text,
|
||||
price_value: price.price_value,
|
||||
currency: price.currency,
|
||||
merchant_info_text: merchantInfo,
|
||||
sold_by: soldBy,
|
||||
ships_from: shipsFrom,
|
||||
offer_listing_url: cleanText(payload.offer_link) || null,
|
||||
review_url: cleanText(payload.review_url) || null,
|
||||
qa_url: cleanText(payload.qa_url) || null,
|
||||
is_amazon_sold: isAmazonEntity(soldBy),
|
||||
is_amazon_fulfilled: isAmazonEntity(shipsFrom) || /fulfilled by amazon/i.test(merchantInfo ?? ''),
|
||||
};
|
||||
}
|
||||
|
||||
async function readOfferPayload(page: IPage, input: string): Promise<OfferPayload> {
|
||||
const url = buildProductUrl(input);
|
||||
const state = await gotoAndReadState(page, url, 2500, 'offer');
|
||||
assertUsableState(state, 'offer');
|
||||
|
||||
// Reconnecting to an existing Amazon target can surface the product page
|
||||
// before the buy-box / merchant blocks are reattached to the DOM.
|
||||
await page.wait({ selector: OFFER_FACT_SELECTOR, timeout: 6 }).catch(() => {});
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
price_text: (() => {
|
||||
const selectors = ${JSON.stringify(PRIMARY_PRICE_SELECTORS)};
|
||||
for (const selector of selectors) {
|
||||
const text = document.querySelector(selector)?.textContent || '';
|
||||
if (text.trim()) return text;
|
||||
}
|
||||
return '';
|
||||
})(),
|
||||
merchant_info: document.querySelector('#merchant-info')?.textContent || '',
|
||||
sold_by: document.querySelector('#sellerProfileTriggerId')?.textContent || '',
|
||||
ships_from_text:
|
||||
document.querySelector('#shipsFromSoldByInsideBuyBox_feature_div')?.textContent
|
||||
|| document.querySelector('#fulfillerInfoFeature_feature_div')?.textContent
|
||||
|| document.querySelector('#merchantInfoFeature_feature_div')?.textContent
|
||||
|| document.querySelector('#tabular-buybox-container')?.textContent
|
||||
|| '',
|
||||
offer_link: document.querySelector('a[href*="/gp/offer-listing/"]')?.href || '',
|
||||
review_url: document.querySelector('a[href*="#customerReviews"]')?.href || '',
|
||||
qa_url: document.querySelector('a[href*="ask/questions"]')?.href || '',
|
||||
buybox_text:
|
||||
document.querySelector('#desktop_qualifiedBuyBox')?.textContent
|
||||
|| document.querySelector('#buybox')?.textContent
|
||||
|| '',
|
||||
}))()
|
||||
`) as OfferPayload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'amazon',
|
||||
name: 'offer',
|
||||
description: 'Amazon seller, buy box, and fulfillment facts from the product page',
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: 'ASIN or product URL, for example B0FJS72893',
|
||||
},
|
||||
],
|
||||
columns: ['asin', 'price_text', 'sold_by', 'ships_from', 'is_amazon_sold', 'is_amazon_fulfilled'],
|
||||
func: async (page, kwargs) => {
|
||||
const input = String(kwargs.input ?? '');
|
||||
const payload = await readOfferPayload(page, input);
|
||||
const normalized = normalizeOfferPayload(payload);
|
||||
|
||||
if (!normalized.sold_by && !normalized.ships_from && !normalized.merchant_info_text) {
|
||||
if (isDeliveryLocationBlocked(payload.buybox_text)) {
|
||||
throw new CommandExecutionError(
|
||||
'amazon offer buy box is blocked by the current delivery location',
|
||||
'The shared Chrome profile is not set to the target US delivery address. Switch Amazon delivery location to the requested US destination, reopen the product page, and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
throw new CommandExecutionError(
|
||||
'amazon offer surface did not expose seller or fulfillment facts',
|
||||
'The product page may have changed. Open the product page in Chrome, make sure the buy box is visible, and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
return [normalized];
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
extractShipsFrom,
|
||||
extractSoldBy,
|
||||
isDeliveryLocationBlocked,
|
||||
normalizeOfferPayload,
|
||||
};
|
||||
@@ -0,0 +1,26 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './product.js';
|
||||
|
||||
describe('amazon product normalization', () => {
|
||||
it('normalizes product facts from the product page', () => {
|
||||
const result = __test__.normalizeProductPayload({
|
||||
href: 'https://www.amazon.com/dp/B0FJS72893',
|
||||
title: 'Amazon.com: KVTUKIAIT Desktop Shelf Organizer',
|
||||
product_title: 'White Desktop Shelf Organizer for Top of Desk',
|
||||
byline: 'Visit the KVTUKIAIT Store',
|
||||
price_text: '$15.99',
|
||||
rating_text: '3.9 out of 5 stars',
|
||||
review_count_text: '27 ratings',
|
||||
review_url: 'https://www.amazon.com/dp/B0FJS72893#customerReviews',
|
||||
qa_url: null,
|
||||
bullets: ['SPACE-SAVING DESK SHELF ORGANIZER', 'SMALL AND STYLISH AESTHETIC DECOR'],
|
||||
breadcrumbs: ['Office Products', 'Desktop & Off-Surface Shelves'],
|
||||
});
|
||||
|
||||
expect(result.asin).toBe('B0FJS72893');
|
||||
expect(result.price_value).toBe(15.99);
|
||||
expect(result.rating_value).toBe(3.9);
|
||||
expect(result.review_count).toBe(27);
|
||||
expect(result.breadcrumbs).toEqual(['Office Products', 'Desktop & Off-Surface Shelves']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
buildProductUrl,
|
||||
buildProvenance,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
PRIMARY_PRICE_SELECTORS,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
normalizeProductUrl,
|
||||
uniqueNonEmpty,
|
||||
assertUsableState,
|
||||
gotoAndReadState,
|
||||
} from './shared.js';
|
||||
|
||||
interface ProductPayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
product_title?: string | null;
|
||||
byline?: string | null;
|
||||
price_text?: string | null;
|
||||
rating_text?: string | null;
|
||||
review_count_text?: string | null;
|
||||
review_url?: string | null;
|
||||
qa_url?: string | null;
|
||||
bullets?: string[];
|
||||
breadcrumbs?: string[];
|
||||
}
|
||||
|
||||
const PRODUCT_TITLE_SELECTOR = '#productTitle, #title span, [data-feature-name="title"] h1 span';
|
||||
const BYLINE_SELECTOR = '#bylineInfo, [data-feature-name="bylineInfo"] #bylineInfo';
|
||||
|
||||
function normalizeProductPayload(payload: ProductPayload): Record<string, unknown> {
|
||||
const sourceUrl = cleanText(payload.href) || buildProductUrl(cleanText(payload.product_title) || cleanText(payload.href));
|
||||
const asin = extractAsin(payload.href ?? '') ?? null;
|
||||
const price = parsePriceText(payload.price_text);
|
||||
const ratingText = cleanText(payload.rating_text) || null;
|
||||
const reviewCountText = cleanText(payload.review_count_text) || null;
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
asin,
|
||||
title: cleanText(payload.product_title) || cleanText(payload.title) || null,
|
||||
product_url: normalizeProductUrl(payload.href),
|
||||
...provenance,
|
||||
brand_text: cleanText(payload.byline) || null,
|
||||
price_text: price.price_text,
|
||||
price_value: price.price_value,
|
||||
currency: price.currency,
|
||||
rating_text: ratingText,
|
||||
rating_value: parseRatingValue(ratingText),
|
||||
review_count_text: reviewCountText,
|
||||
review_count: parseReviewCount(reviewCountText),
|
||||
review_url: cleanText(payload.review_url) || null,
|
||||
qa_url: cleanText(payload.qa_url) || null,
|
||||
breadcrumbs: uniqueNonEmpty(payload.breadcrumbs ?? []),
|
||||
bullet_points: uniqueNonEmpty(payload.bullets ?? []),
|
||||
};
|
||||
}
|
||||
|
||||
async function readProductPayload(page: IPage, input: string): Promise<ProductPayload> {
|
||||
const url = buildProductUrl(input);
|
||||
const state = await gotoAndReadState(page, url, 2500, 'product');
|
||||
assertUsableState(state, 'product');
|
||||
|
||||
// Amazon can report a "stable" DOM before the product title block hydrates,
|
||||
// especially when reconnecting to an existing shared CDP target.
|
||||
await page.wait({ selector: PRODUCT_TITLE_SELECTOR, timeout: 6 }).catch(() => {});
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
product_title: document.querySelector(${JSON.stringify(PRODUCT_TITLE_SELECTOR)})?.textContent || '',
|
||||
byline: document.querySelector(${JSON.stringify(BYLINE_SELECTOR)})?.textContent || '',
|
||||
price_text: (() => {
|
||||
const selectors = ${JSON.stringify(PRIMARY_PRICE_SELECTORS)};
|
||||
for (const selector of selectors) {
|
||||
const text = document.querySelector(selector)?.textContent || '';
|
||||
if (text.trim()) return text;
|
||||
}
|
||||
return '';
|
||||
})(),
|
||||
rating_text:
|
||||
document.querySelector('#acrPopover')?.getAttribute('title')
|
||||
|| document.querySelector('#acrPopover')?.textContent
|
||||
|| '',
|
||||
review_count_text: document.querySelector('#acrCustomerReviewText')?.textContent || '',
|
||||
review_url: document.querySelector('a[href*="#customerReviews"]')?.href || '',
|
||||
qa_url: document.querySelector('a[href*="ask/questions"]')?.href || '',
|
||||
bullets: Array.from(document.querySelectorAll('#feature-bullets li .a-list-item')).map((node) => node.textContent || ''),
|
||||
breadcrumbs: Array.from(document.querySelectorAll('#wayfinding-breadcrumbs_feature_div a')).map((node) => node.textContent || ''),
|
||||
}))()
|
||||
`) as ProductPayload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'amazon',
|
||||
name: 'product',
|
||||
description: 'Amazon product page facts for candidate validation',
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: 'ASIN or product URL, for example B0FJS72893',
|
||||
},
|
||||
],
|
||||
columns: ['asin', 'title', 'price_text', 'rating_value', 'review_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const input = String(kwargs.input ?? '');
|
||||
const payload = await readProductPayload(page, input);
|
||||
if (!cleanText(payload.product_title)) {
|
||||
throw new CommandExecutionError(
|
||||
'amazon product page did not expose product content',
|
||||
'The product page may have changed or hit a robot check. Open the product page in Chrome and retry.',
|
||||
);
|
||||
}
|
||||
return [normalizeProductPayload(payload)];
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeProductPayload,
|
||||
};
|
||||
@@ -0,0 +1,24 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './search.js';
|
||||
|
||||
describe('amazon search normalization', () => {
|
||||
it('normalizes search cards into research-friendly fields', () => {
|
||||
const result = __test__.normalizeSearchCandidate({
|
||||
asin: 'B0FJS72893',
|
||||
title: 'White Desktop Shelf Organizer for Top of Desk',
|
||||
href: 'https://www.amazon.com/KVTUKIAIT-White-Desktop-Shelf-Organizer/dp/B0FJS72893/ref=sr_1_1',
|
||||
price_text: '$15.99',
|
||||
rating_text: '3.9 out of 5 stars, rating details',
|
||||
review_count_text: '(27)',
|
||||
sponsored: false,
|
||||
badge_texts: ['Limited time deal'],
|
||||
}, 1, 'https://www.amazon.com/s?k=desk+shelf+organizer');
|
||||
|
||||
expect(result.asin).toBe('B0FJS72893');
|
||||
expect(result.product_url).toBe('https://www.amazon.com/dp/B0FJS72893');
|
||||
expect(result.price_value).toBe(15.99);
|
||||
expect(result.rating_value).toBe(3.9);
|
||||
expect(result.review_count).toBe(27);
|
||||
expect(result.badges).toEqual(['Limited time deal']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,128 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
buildProvenance,
|
||||
buildSearchUrl,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
normalizeProductUrl,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
assertUsableState,
|
||||
gotoAndReadState,
|
||||
} from './shared.js';
|
||||
|
||||
interface SearchPayload {
|
||||
href?: string;
|
||||
cards?: Array<{
|
||||
asin?: string;
|
||||
title?: string;
|
||||
href?: string;
|
||||
price_text?: string | null;
|
||||
rating_text?: string | null;
|
||||
review_count_text?: string | null;
|
||||
sponsored?: boolean;
|
||||
badge_texts?: string[];
|
||||
}>;
|
||||
}
|
||||
|
||||
function normalizeSearchCandidate(
|
||||
candidate: NonNullable<SearchPayload['cards']>[number],
|
||||
rank: number,
|
||||
sourceUrl: string,
|
||||
): Record<string, unknown> {
|
||||
const productUrl = normalizeProductUrl(candidate.href);
|
||||
const asin = extractAsin(candidate.asin ?? '') ?? extractAsin(productUrl ?? '') ?? null;
|
||||
const price = parsePriceText(candidate.price_text);
|
||||
const ratingText = cleanText(candidate.rating_text) || null;
|
||||
const reviewCountText = cleanText(candidate.review_count_text) || null;
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
rank,
|
||||
asin,
|
||||
title: cleanText(candidate.title) || null,
|
||||
product_url: productUrl,
|
||||
...provenance,
|
||||
price_text: price.price_text,
|
||||
price_value: price.price_value,
|
||||
currency: price.currency,
|
||||
rating_text: ratingText,
|
||||
rating_value: parseRatingValue(ratingText),
|
||||
review_count_text: reviewCountText,
|
||||
review_count: parseReviewCount(reviewCountText),
|
||||
is_sponsored: candidate.sponsored === true,
|
||||
badges: (candidate.badge_texts ?? []).map((value) => cleanText(value)).filter(Boolean),
|
||||
};
|
||||
}
|
||||
|
||||
async function readSearchPayload(page: IPage, query: string): Promise<SearchPayload> {
|
||||
const url = buildSearchUrl(query);
|
||||
const state = await gotoAndReadState(page, url, 2500, 'search');
|
||||
assertUsableState(state, 'search');
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
cards: Array.from(document.querySelectorAll('[data-component-type="s-search-result"]'))
|
||||
.map((card) => ({
|
||||
asin: card.getAttribute('data-asin') || '',
|
||||
title: card.querySelector('h2')?.textContent || '',
|
||||
href: card.querySelector('a.a-link-normal[href*="/dp/"]')?.href || '',
|
||||
price_text: card.querySelector('.a-price .a-offscreen')?.textContent || '',
|
||||
rating_text: card.querySelector('[aria-label*="out of 5 stars"]')?.getAttribute('aria-label') || '',
|
||||
review_count_text: card.querySelector('a[href*="#customerReviews"]')?.textContent || '',
|
||||
sponsored: /sponsored/i.test(card.innerText || ''),
|
||||
badge_texts: Array.from(card.querySelectorAll('.a-badge-text')).map((node) => node.textContent || ''),
|
||||
})),
|
||||
}))()
|
||||
`) as SearchPayload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'amazon',
|
||||
name: 'search',
|
||||
description: 'Amazon search results for product discovery and coarse filtering',
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'query',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: 'Search query, for example "desk shelf organizer"',
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: 20,
|
||||
help: 'Maximum number of results to return (default 20)',
|
||||
},
|
||||
],
|
||||
columns: ['rank', 'asin', 'title', 'price_text', 'rating_value', 'review_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const query = String(kwargs.query ?? '');
|
||||
const limit = Math.max(1, Number(kwargs.limit) || 20);
|
||||
const payload = await readSearchPayload(page, query);
|
||||
const sourceUrl = cleanText(payload.href) || buildSearchUrl(query);
|
||||
const cards = (payload.cards ?? [])
|
||||
.filter((card) => cleanText(card.asin) && cleanText(card.title))
|
||||
.slice(0, limit);
|
||||
|
||||
if (cards.length === 0) {
|
||||
throw new CommandExecutionError(
|
||||
'amazon search did not expose any product cards',
|
||||
'The search page may have changed or hit a robot check. Open the same query in Chrome, verify the page is visible, and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
return cards.map((card, index) => normalizeSearchCandidate(card, index + 1, sourceUrl));
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeSearchCandidate,
|
||||
};
|
||||
@@ -0,0 +1,37 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './shared.js';
|
||||
|
||||
describe('amazon shared helpers', () => {
|
||||
it('builds canonical product and discussion URLs from ASINs and product URLs', () => {
|
||||
expect(__test__.buildProductUrl('B0FJS72893')).toBe('https://www.amazon.com/dp/B0FJS72893');
|
||||
expect(__test__.buildProductUrl('https://www.amazon.com/dp/B0FJS72893/ref=something')).toBe('https://www.amazon.com/dp/B0FJS72893');
|
||||
expect(__test__.buildDiscussionUrl('https://www.amazon.com/dp/B0FJS72893')).toBe('https://www.amazon.com/product-reviews/B0FJS72893');
|
||||
});
|
||||
|
||||
it('parses price, rating, and review-count text', () => {
|
||||
expect(__test__.parsePriceText('1 offer from $34.11')).toEqual({
|
||||
price_text: '$34.11',
|
||||
price_value: 34.11,
|
||||
currency: 'USD',
|
||||
});
|
||||
expect(__test__.parseRatingValue('3.9 out of 5 stars, rating details')).toBe(3.9);
|
||||
expect(__test__.parseReviewCount('27 global ratings')).toBe(27);
|
||||
expect(__test__.parseReviewCount('(2.9K)')).toBe(2900);
|
||||
expect(__test__.parseReviewCount('1.2M global ratings')).toBe(1200000);
|
||||
expect(__test__.extractReviewCountFromCardText('Desk Shelf\n4.3 out of 5 stars\n435\n$25.92')).toBe('435');
|
||||
});
|
||||
|
||||
it('recognizes robot checks and Amazon-owned merchants', () => {
|
||||
expect(__test__.isAmazonEntity('Ships from Amazon')).toBe(true);
|
||||
expect(__test__.trimRatingPrefix('5.0 out of 5 stars Great value and quality')).toBe('Great value and quality');
|
||||
expect(__test__.isRobotState({
|
||||
title: 'Robot Check',
|
||||
body_text: 'Sorry, we just need to make sure you\'re not a robot',
|
||||
})).toBe(true);
|
||||
});
|
||||
|
||||
it('requires a real best-sellers URL or path', () => {
|
||||
expect(__test__.resolveBestsellersUrl('/Best-Sellers/zgbs')).toBe('https://www.amazon.com/Best-Sellers/zgbs');
|
||||
expect(() => __test__.resolveBestsellersUrl('desk shelf organizer')).toThrow('amazon bestsellers expects a best sellers URL or /zgbs path');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,316 @@
|
||||
import { ArgumentError, CommandExecutionError } from '../../errors.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
|
||||
export const SITE = 'amazon';
|
||||
export const DOMAIN = 'amazon.com';
|
||||
export const HOME_URL = 'https://www.amazon.com/';
|
||||
export const BESTSELLERS_URL = 'https://www.amazon.com/Best-Sellers/zgbs';
|
||||
export const SEARCH_URL_PREFIX = 'https://www.amazon.com/s?k=';
|
||||
export const PRODUCT_URL_PREFIX = 'https://www.amazon.com/dp/';
|
||||
export const DISCUSSION_URL_PREFIX = 'https://www.amazon.com/product-reviews/';
|
||||
export const STRATEGY = 'cookie';
|
||||
export const PRIMARY_PRICE_SELECTORS = [
|
||||
'#corePrice_feature_div .a-offscreen',
|
||||
'#corePriceDisplay_desktop_feature_div .a-offscreen',
|
||||
'#corePrice_desktop .a-offscreen',
|
||||
'#apex_desktop .a-offscreen',
|
||||
'#newAccordionRow_0 .a-offscreen',
|
||||
'#price_inside_buybox',
|
||||
'#priceblock_ourprice',
|
||||
'#priceblock_dealprice',
|
||||
'#tp_price_block_total_price_ww',
|
||||
];
|
||||
|
||||
const ROBOT_TEXT_PATTERNS = [
|
||||
'Sorry, we just need to make sure you\'re not a robot',
|
||||
'Enter the characters you see below',
|
||||
'Type the characters you see in this image',
|
||||
'To discuss automated access to Amazon data please contact',
|
||||
];
|
||||
|
||||
export interface ProvenanceFields {
|
||||
source_url: string;
|
||||
fetched_at: string;
|
||||
strategy: string;
|
||||
}
|
||||
|
||||
export interface PageState {
|
||||
href: string;
|
||||
title: string;
|
||||
body_text: string;
|
||||
}
|
||||
|
||||
export interface PriceValue {
|
||||
price_text: string | null;
|
||||
price_value: number | null;
|
||||
currency: string | null;
|
||||
}
|
||||
|
||||
export function cleanText(value: unknown): string {
|
||||
return typeof value === 'string'
|
||||
? value.replace(/\u00a0/g, ' ').replace(/\s+/g, ' ').trim()
|
||||
: '';
|
||||
}
|
||||
|
||||
export function cleanMultilineText(value: unknown): string {
|
||||
return typeof value === 'string'
|
||||
? value
|
||||
.replace(/\u00a0/g, ' ')
|
||||
.split('\n')
|
||||
.map((line) => line.replace(/\s+/g, ' ').trim())
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
: '';
|
||||
}
|
||||
|
||||
export function uniqueNonEmpty(values: Array<string | null | undefined>): string[] {
|
||||
return [...new Set(values.map((value) => cleanText(value)).filter(Boolean))];
|
||||
}
|
||||
|
||||
export function buildProvenance(sourceUrl: string): ProvenanceFields {
|
||||
return {
|
||||
source_url: sourceUrl,
|
||||
fetched_at: new Date().toISOString(),
|
||||
strategy: STRATEGY,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildSearchUrl(query: string): string {
|
||||
const normalized = cleanText(query);
|
||||
if (!normalized) {
|
||||
throw new ArgumentError('amazon search query cannot be empty');
|
||||
}
|
||||
return `${SEARCH_URL_PREFIX}${encodeURIComponent(normalized)}`;
|
||||
}
|
||||
|
||||
export function extractAsin(input: string): string | null {
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) return null;
|
||||
if (/^[A-Z0-9]{10}$/i.test(normalized)) {
|
||||
return normalized.toUpperCase();
|
||||
}
|
||||
const match = normalized.match(/\/(?:dp|gp\/product|product-reviews)\/([A-Z0-9]{10})/i);
|
||||
return match ? match[1].toUpperCase() : null;
|
||||
}
|
||||
|
||||
export function buildProductUrl(input: string): string {
|
||||
const asin = extractAsin(input);
|
||||
if (!asin) {
|
||||
throw new ArgumentError(
|
||||
'amazon product expects an ASIN or product URL',
|
||||
'Example: opencli amazon product B0FJS72893',
|
||||
);
|
||||
}
|
||||
return `${PRODUCT_URL_PREFIX}${asin}`;
|
||||
}
|
||||
|
||||
export function buildDiscussionUrl(input: string): string {
|
||||
const asin = extractAsin(input);
|
||||
if (!asin) {
|
||||
throw new ArgumentError(
|
||||
'amazon discussion expects an ASIN or product URL',
|
||||
'Example: opencli amazon discussion B0FJS72893',
|
||||
);
|
||||
}
|
||||
return `${DISCUSSION_URL_PREFIX}${asin}`;
|
||||
}
|
||||
|
||||
export function resolveBestsellersUrl(input?: string): string {
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) return BESTSELLERS_URL;
|
||||
if (normalized === 'root') return BESTSELLERS_URL;
|
||||
if (normalized.startsWith('/')) {
|
||||
return new URL(normalized, HOME_URL).toString();
|
||||
}
|
||||
if (/^https?:\/\//i.test(normalized)) {
|
||||
return canonicalizeAmazonUrl(normalized);
|
||||
}
|
||||
if (normalized.includes('/zgbs/')) {
|
||||
return canonicalizeAmazonUrl(`https://${normalized.replace(/^\/+/, '')}`);
|
||||
}
|
||||
throw new ArgumentError(
|
||||
'amazon bestsellers expects a best sellers URL or /zgbs path',
|
||||
'Example: opencli amazon bestsellers https://www.amazon.com/Best-Sellers/zgbs',
|
||||
);
|
||||
}
|
||||
|
||||
export function canonicalizeAmazonUrl(input: string): string {
|
||||
try {
|
||||
const url = new URL(input);
|
||||
if (!url.hostname.endsWith(DOMAIN)) {
|
||||
throw new Error('not-amazon');
|
||||
}
|
||||
return url.toString();
|
||||
} catch {
|
||||
throw new ArgumentError('Invalid Amazon URL');
|
||||
}
|
||||
}
|
||||
|
||||
export function toAbsoluteAmazonUrl(value: string | null | undefined): string | null {
|
||||
const normalized = cleanText(value);
|
||||
if (!normalized) return null;
|
||||
try {
|
||||
return new URL(normalized, HOME_URL).toString();
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function normalizeProductUrl(value: string | null | undefined): string | null {
|
||||
const normalized = cleanText(value);
|
||||
const asin = extractAsin(normalized);
|
||||
if (asin) return buildProductUrl(asin);
|
||||
return toAbsoluteAmazonUrl(normalized);
|
||||
}
|
||||
|
||||
export function parsePriceText(text: string | null | undefined): PriceValue {
|
||||
const normalized = cleanText(text);
|
||||
const match = normalized.match(/([$€£])\s*(\d+(?:,\d{3})*(?:\.\d+)?)/);
|
||||
if (!match) {
|
||||
return {
|
||||
price_text: normalized || null,
|
||||
price_value: null,
|
||||
currency: null,
|
||||
};
|
||||
}
|
||||
|
||||
const currencyMap: Record<string, string> = {
|
||||
'$': 'USD',
|
||||
'€': 'EUR',
|
||||
'£': 'GBP',
|
||||
};
|
||||
|
||||
return {
|
||||
price_text: `${match[1]}${match[2]}`,
|
||||
price_value: Number.parseFloat(match[2].replace(/,/g, '')),
|
||||
currency: currencyMap[match[1]] ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
export function parseRatingValue(text: string | null | undefined): number | null {
|
||||
const normalized = cleanText(text);
|
||||
const match = normalized.match(/(\d+(?:\.\d+)?)\s*out of 5/i);
|
||||
return match ? Number.parseFloat(match[1]) : null;
|
||||
}
|
||||
|
||||
export function parseReviewCount(text: string | null | undefined): number | null {
|
||||
const normalized = cleanText(text);
|
||||
const compactMatch = normalized.match(/(\d+(?:\.\d+)?)\s*([kKmM])/);
|
||||
if (compactMatch) {
|
||||
const value = Number.parseFloat(compactMatch[1]);
|
||||
const multiplier = /m/i.test(compactMatch[2]) ? 1_000_000 : 1_000;
|
||||
return Number.isFinite(value) ? Math.round(value * multiplier) : null;
|
||||
}
|
||||
const match = normalized.match(/([\d,]+)/);
|
||||
return match ? Number.parseInt(match[1].replace(/,/g, ''), 10) : null;
|
||||
}
|
||||
|
||||
export function extractReviewCountFromCardText(text: string | null | undefined): string | null {
|
||||
const normalized = cleanMultilineText(text);
|
||||
const match = normalized.match(/out of 5 stars(?:, rating details)?\s*([\d,]+)/i);
|
||||
if (match) return match[1];
|
||||
|
||||
const numericLine = normalized
|
||||
.split('\n')
|
||||
.map((line) => cleanText(line))
|
||||
.find((line) => /^[\d,]+$/.test(line));
|
||||
return numericLine ?? null;
|
||||
}
|
||||
|
||||
export function isAmazonEntity(text: string | null | undefined): boolean {
|
||||
const normalized = cleanText(text).toLowerCase();
|
||||
return normalized.includes('amazon');
|
||||
}
|
||||
|
||||
export function firstMeaningfulLine(text: string | null | undefined): string {
|
||||
return cleanMultilineText(text)
|
||||
.split('\n')
|
||||
.map((line) => cleanText(line))
|
||||
.find(Boolean)
|
||||
?? '';
|
||||
}
|
||||
|
||||
export function trimRatingPrefix(text: string | null | undefined): string | null {
|
||||
const normalized = cleanText(text);
|
||||
if (!normalized) return null;
|
||||
return normalized.replace(/^\d+(?:\.\d+)?\s*out of 5 stars\s*/i, '').trim() || normalized;
|
||||
}
|
||||
|
||||
export function isRobotState(state: Partial<PageState>): boolean {
|
||||
const title = cleanText(state.title);
|
||||
const bodyText = cleanMultilineText(state.body_text);
|
||||
return ROBOT_TEXT_PATTERNS.some((pattern) => title.includes(pattern) || bodyText.includes(pattern));
|
||||
}
|
||||
|
||||
export function buildChallengeHint(action: string): string {
|
||||
return [
|
||||
`Open a clean Amazon ${action} page in the shared Chrome profile and clear any robot check first.`,
|
||||
'If you are using CDP, set OPENCLI_CDP_TARGET=amazon.com and avoid parallel Amazon commands against the same browser target.',
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
export async function readPageState(page: IPage): Promise<PageState> {
|
||||
const result = await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
body_text: document.body ? document.body.innerText || '' : '',
|
||||
}))()
|
||||
`) as Partial<PageState>;
|
||||
|
||||
return {
|
||||
href: cleanText(result.href),
|
||||
title: cleanText(result.title),
|
||||
body_text: cleanMultilineText(result.body_text),
|
||||
};
|
||||
}
|
||||
|
||||
export async function gotoAndReadState(
|
||||
page: IPage,
|
||||
url: string,
|
||||
settleMs: number = 2500,
|
||||
action: string = 'page',
|
||||
): Promise<PageState> {
|
||||
try {
|
||||
await page.goto(url, { settleMs });
|
||||
await page.wait(1.5);
|
||||
return await readPageState(page);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (
|
||||
message.includes('Inspected target navigated or closed')
|
||||
|| message.includes('Cannot find context with specified id')
|
||||
|| message.includes('Target closed')
|
||||
) {
|
||||
throw new CommandExecutionError(
|
||||
`amazon ${action} navigation lost the current browser target`,
|
||||
`${buildChallengeHint(action)} If CDP is attached to a stale tab, open a fresh Amazon tab and retry.`,
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export function assertUsableState(state: PageState, action: string): void {
|
||||
if (!isRobotState(state)) return;
|
||||
throw new CommandExecutionError(
|
||||
`amazon ${action} hit a robot check`,
|
||||
buildChallengeHint(action),
|
||||
);
|
||||
}
|
||||
|
||||
export const __test__ = {
|
||||
buildSearchUrl,
|
||||
extractAsin,
|
||||
buildProductUrl,
|
||||
buildDiscussionUrl,
|
||||
resolveBestsellersUrl,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
extractReviewCountFromCardText,
|
||||
isAmazonEntity,
|
||||
trimRatingPrefix,
|
||||
isRobotState,
|
||||
PRIMARY_PRICE_SELECTORS,
|
||||
};
|
||||
@@ -7,17 +7,10 @@ description: How to automate Antigravity using OpenCLI
|
||||
This skill allows AI agents to control the [Antigravity](https://github.com/chengazhen/Antigravity) desktop app (and any Electron app with CDP enabled) programmatically via OpenCLI.
|
||||
|
||||
## Requirements
|
||||
The target Electron application MUST be launched with the remote-debugging-port flag:
|
||||
\`\`\`bash
|
||||
/Applications/Antigravity.app/Contents/MacOS/Electron --remote-debugging-port=9224
|
||||
\`\`\`
|
||||
opencli automatically detects, launches (with `--remote-debugging-port=9234`), and connects to Antigravity.
|
||||
If Antigravity is already running without CDP, opencli will prompt to restart it.
|
||||
|
||||
The agent must configure the endpoint environment variable locally before invoking standard commands:
|
||||
\`\`\`bash
|
||||
export OPENCLI_CDP_ENDPOINT="http://127.0.0.1:9224"
|
||||
\`\`\`
|
||||
|
||||
If the endpoint exposes multiple inspectable targets, also set:
|
||||
If the endpoint exposes multiple inspectable targets, set:
|
||||
\`\`\`bash
|
||||
export OPENCLI_CDP_TARGET="antigravity"
|
||||
\`\`\`
|
||||
@@ -33,7 +26,6 @@ export OPENCLI_CDP_TARGET="antigravity"
|
||||
|
||||
### Generating and Saving Code
|
||||
\`\`\`bash
|
||||
export OPENCLI_CDP_ENDPOINT="http://127.0.0.1:9224"
|
||||
opencli antigravity send "Write a python script to fetch HN top stories"
|
||||
# wait ~10-15 seconds for output to render
|
||||
opencli antigravity extract-code > hn_fetcher.py
|
||||
@@ -42,6 +34,5 @@ opencli antigravity extract-code > hn_fetcher.py
|
||||
### Reading Real-time Logs
|
||||
Agents can run long-running streaming watch instances:
|
||||
\`\`\`bash
|
||||
export OPENCLI_CDP_ENDPOINT="http://127.0.0.1:9224"
|
||||
opencli antigravity watch
|
||||
\`\`\`
|
||||
|
||||
@@ -6,14 +6,15 @@
|
||||
* and returns it in Anthropic format.
|
||||
*
|
||||
* Usage:
|
||||
* OPENCLI_CDP_ENDPOINT=http://127.0.0.1:9224 opencli antigravity serve --port 8082
|
||||
* opencli antigravity serve --port 8082
|
||||
* ANTHROPIC_BASE_URL=http://localhost:8082 claude
|
||||
*/
|
||||
|
||||
import { createServer, type IncomingMessage, type ServerResponse } from 'node:http';
|
||||
import { CDPBridge } from '../../browser/cdp.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { getErrorMessage } from '../../errors.js';
|
||||
import { resolveElectronEndpoint } from '../../launcher.js';
|
||||
import { EXIT_CODES, getErrorMessage } from '../../errors.js';
|
||||
|
||||
// ─── Types ───────────────────────────────────────────────────────────
|
||||
|
||||
@@ -436,13 +437,7 @@ export async function startServe(opts: { port?: number } = {}): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
const endpoint = process.env.OPENCLI_CDP_ENDPOINT;
|
||||
if (!endpoint) {
|
||||
throw new Error(
|
||||
'OPENCLI_CDP_ENDPOINT is not set.\n' +
|
||||
'Usage: OPENCLI_CDP_ENDPOINT=http://127.0.0.1:9224 opencli antigravity serve'
|
||||
);
|
||||
}
|
||||
const endpoint = await resolveElectronEndpoint('antigravity');
|
||||
|
||||
// Note: Antigravity chat panel lives inside editor windows, not in Launchpad.
|
||||
// If multiple editor windows are open, set OPENCLI_CDP_TARGET to the window title.
|
||||
@@ -461,7 +456,7 @@ export async function startServe(opts: { port?: number } = {}): Promise<void> {
|
||||
console.error(`[serve] Connecting via CDP (target pattern: "${process.env.OPENCLI_CDP_TARGET}")...`);
|
||||
cdp = new CDPBridge();
|
||||
try {
|
||||
page = await cdp.connect({ timeout: 15_000 });
|
||||
page = await cdp.connect({ timeout: 15_000, cdpEndpoint: endpoint });
|
||||
} catch (err: unknown) {
|
||||
cdp = null;
|
||||
const errMsg = getErrorMessage(err);
|
||||
@@ -471,7 +466,7 @@ export async function startServe(opts: { port?: number } = {}): Promise<void> {
|
||||
isRefused
|
||||
? `Cannot connect to Antigravity at ${endpoint}.\n` +
|
||||
' 1. Make sure Antigravity is running\n' +
|
||||
' 2. Launch with: --remote-debugging-port=9224'
|
||||
' 2. Launch with: --remote-debugging-port=9234'
|
||||
: `CDP connection failed: ${errMsg}`
|
||||
);
|
||||
}
|
||||
@@ -594,7 +589,7 @@ export async function startServe(opts: { port?: number } = {}): Promise<void> {
|
||||
console.error('\n[serve] Shutting down...');
|
||||
cdp?.close().catch(() => {});
|
||||
server.close();
|
||||
process.exit(0);
|
||||
process.exit(EXIT_CODES.SUCCESS);
|
||||
};
|
||||
process.on('SIGTERM', shutdown);
|
||||
process.on('SIGINT', shutdown);
|
||||
|
||||
@@ -38,14 +38,14 @@ describe('apple-podcasts search command', () => {
|
||||
'https://itunes.apple.com/search?term=machine%20learning&media=podcast&limit=5',
|
||||
);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
expect.objectContaining({
|
||||
id: 42,
|
||||
title: 'Machine Learning Guide',
|
||||
author: 'OpenCLI',
|
||||
episodes: 12,
|
||||
genre: 'Technology',
|
||||
url: '',
|
||||
},
|
||||
}),
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -55,6 +55,30 @@ describe('apple-podcasts top command', () => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('adds a timeout signal to chart fetches', async () => {
|
||||
const cmd = getRegistry().get('apple-podcasts/top');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: () => Promise.resolve({
|
||||
feed: {
|
||||
results: [
|
||||
{ id: '100', name: 'Top Show', artistName: 'Host A' },
|
||||
],
|
||||
},
|
||||
}),
|
||||
});
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
await cmd!.func!(null as any, { country: 'US', limit: 1 });
|
||||
|
||||
const [, options] = fetchMock.mock.calls[0] ?? [];
|
||||
expect(options).toBeDefined();
|
||||
expect(options.signal).toBeDefined();
|
||||
expect(options.signal).toHaveProperty('aborted', false);
|
||||
});
|
||||
|
||||
it('uses the canonical Apple charts host and maps ranked results', async () => {
|
||||
const cmd = getRegistry().get('apple-podcasts/top');
|
||||
expect(cmd?.func).toBeTypeOf('function');
|
||||
@@ -76,6 +100,9 @@ describe('apple-podcasts top command', () => {
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
'https://rss.marketingtools.apple.com/api/v2/us/podcasts/top/2/podcasts.json',
|
||||
expect.objectContaining({
|
||||
signal: expect.any(Object),
|
||||
}),
|
||||
);
|
||||
expect(result).toEqual([
|
||||
{ rank: 1, title: 'Top Show', author: 'Host A', id: '100' },
|
||||
|
||||
@@ -3,6 +3,7 @@ import { CliError } from '../../errors.js';
|
||||
|
||||
// Apple Marketing Tools RSS API — public, no key required
|
||||
const CHARTS_URL = 'https://rss.marketingtools.apple.com/api/v2';
|
||||
const CHARTS_TIMEOUT_MS = 15_000;
|
||||
|
||||
cli({
|
||||
site: 'apple-podcasts',
|
||||
@@ -21,7 +22,9 @@ cli({
|
||||
const url = `${CHARTS_URL}/${country}/podcasts/top/${limit}/podcasts.json`;
|
||||
let resp: Response;
|
||||
try {
|
||||
resp = await fetch(url);
|
||||
resp = await fetch(url, {
|
||||
signal: AbortSignal.timeout(CHARTS_TIMEOUT_MS),
|
||||
});
|
||||
} catch (error: any) {
|
||||
const reason = error?.cause?.code ?? error?.message ?? 'unknown network error';
|
||||
throw new CliError(
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user