Compare commits
29 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1610cf7654 | |||
| b2f1f58a1b | |||
| 81308a474e | |||
| d818b5bed8 | |||
| e82649379b | |||
| e0a66af0f0 | |||
| c86e677b78 | |||
| 6364934423 | |||
| ef78aaf3a2 | |||
| 292b12d9b1 | |||
| 2c5066d1f4 | |||
| 8eefa3b1c9 | |||
| 052bf8bbf7 | |||
| c3c3abbbff | |||
| 81de69be3a | |||
| a39a858f0a | |||
| 855eaee04e | |||
| 1bcd96f38a | |||
| c2ac5525b3 | |||
| 748b09261d | |||
| 4b1153babe | |||
| 7aafd4af59 | |||
| a5abd3769f | |||
| 8070960444 | |||
| b1c0bcb464 | |||
| e18e0ed7a4 | |||
| c161f0f9f0 | |||
| dcad060230 | |||
| ff84d19ded |
@@ -23,3 +23,4 @@ docs/.vitepress/cache
|
||||
# Database files
|
||||
*.db
|
||||
autoresearch/results/
|
||||
autoresearch-results.tsv
|
||||
|
||||
-724
@@ -1,724 +0,0 @@
|
||||
# CLI-EXPLORER — 适配器探索式开发完全指南
|
||||
|
||||
> 本文档教你(或 AI Agent)如何为 OpenCLI 添加一个新网站的命令。
|
||||
> 从零到发布,覆盖 API 发现、方案选择、适配器编写、测试验证全流程。
|
||||
|
||||
> [!TIP]
|
||||
> **只想为一个具体页面快速生成一个命令?** 看 [CLI-ONESHOT.md](./CLI-ONESHOT.md)(~150 行,4 步搞定)。
|
||||
> 本文档适合从零探索一个新站点的完整流程。
|
||||
|
||||
---
|
||||
|
||||
## AI Agent 开发者必读:用浏览器探索
|
||||
|
||||
> [!CAUTION]
|
||||
> **你(AI Agent)必须通过浏览器打开目标网站去探索!**
|
||||
> 不要只靠 `opencli explore` 命令或静态分析来发现 API。
|
||||
> 你拥有浏览器工具,必须主动用它们浏览网页、观察网络请求、模拟用户交互。
|
||||
|
||||
### 为什么?
|
||||
|
||||
很多 API 是**懒加载**的(用户必须点击某个按钮/标签才会触发网络请求)。字幕、评论、关注列表等深层数据不会在页面首次加载时出现在 Network 面板中。**如果你不主动去浏览和交互页面,你永远发现不了这些 API。**
|
||||
|
||||
### AI Agent 探索工作流(必须遵循)
|
||||
|
||||
| 步骤 | 工具 | 做什么 |
|
||||
|------|------|--------|
|
||||
| 0. 打开浏览器 | `browser_navigate` | 导航到目标页面 |
|
||||
| 1. 观察页面 | `browser_snapshot` | 观察可交互元素(按钮/标签/链接) |
|
||||
| 2. 首次抓包 | `browser_network_requests` | 筛选 JSON API 端点,记录 URL pattern |
|
||||
| 3. 模拟交互 | `browser_click` + `browser_wait_for` | 点击"字幕""评论""关注"等按钮 |
|
||||
| 4. 二次抓包 | `browser_network_requests` | 对比步骤 2,找出新触发的 API |
|
||||
| 5. 验证 API | `browser_evaluate` | `fetch(url, {credentials:'include'})` 测试返回结构 |
|
||||
| 6. 写代码 | — | 基于确认的 API 写适配器 |
|
||||
|
||||
### 常犯错误
|
||||
|
||||
| ❌ 错误做法 | ✅ 正确做法 |
|
||||
|------------|------------|
|
||||
| 只用 `opencli explore` 命令,等结果自动出来 | 用浏览器工具打开页面,主动浏览 |
|
||||
| 直接在代码里 `fetch(url)`,不看浏览器实际请求 | 先在浏览器中确认 API 可用,再写代码 |
|
||||
| 页面打开后直接抓包,期望所有 API 都出现 | 模拟点击交互(展开评论/切换标签/加载更多) |
|
||||
| 遇到 HTTP 200 但空数据就放弃 | 检查是否需要 Wbi 签名或 Cookie 鉴权 |
|
||||
| 完全依赖 `__INITIAL_STATE__` 拿所有数据 | `__INITIAL_STATE__` 只有首屏数据,深层数据要调 API |
|
||||
|
||||
### 实战成功案例:5 分钟实现「关注列表」适配器
|
||||
|
||||
以下是用上述工作流实际发现 Bilibili 关注列表 API 的完整过程:
|
||||
|
||||
```
|
||||
1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
|
||||
2. browser_network_requests → 发现:
|
||||
GET /x/relation/followings?vmid={uid}&pn=1&ps=24 → [200]
|
||||
GET /x/relation/stat?vmid={uid} → [200]
|
||||
3. browser_evaluate → 验证 API:
|
||||
fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
|
||||
→ { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
|
||||
4. 结论:标准 Cookie API,无需 Wbi 签名
|
||||
5. 写 following.ts → 一次构建通过
|
||||
```
|
||||
|
||||
**关键决策点**:
|
||||
- 直接访问 `fans/follow` 页面(不是首页),页面加载就会触发 following API
|
||||
- 看到 URL 里没有 `/wbi/` → 不需要签名 → 直接用 `fetchJson` 而非 `apiGet`
|
||||
- API 返回 `code: 0` + 非空 `list` → Tier 2 Cookie 策略确认
|
||||
|
||||
---
|
||||
|
||||
## 核心流程
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌────────┐
|
||||
│ 1. 发现 API │ ──▶ │ 2. 选择策略 │ ──▶ │ 3. 写适配器 │ ──▶ │ 4. 测试 │
|
||||
└─────────────┘ └─────────────┘ └──────────────┘ └────────┘
|
||||
explore cascade YAML / TS run + verify
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1: 发现 API
|
||||
|
||||
### 1a. 自动化发现(推荐)
|
||||
|
||||
OpenCLI 内置 Deep Explore,自动分析网站网络请求:
|
||||
|
||||
```bash
|
||||
opencli explore https://www.example.com --site mysite
|
||||
```
|
||||
|
||||
输出到 `.opencli/explore/mysite/`:
|
||||
|
||||
| 文件 | 内容 |
|
||||
|------|------|
|
||||
| `manifest.json` | 站点元数据、框架检测(Vue2/3、React、Next.js、Pinia、Vuex) |
|
||||
| `endpoints.json` | 已发现的 API 端点,按评分排序,含 URL pattern、方法、响应类型 |
|
||||
| `capabilities.json` | 推理出的功能(`hot`、`search`、`feed`…),含置信度和推荐参数 |
|
||||
| `auth.json` | 认证方式检测(Cookie/Header/无认证),策略候选列表 |
|
||||
|
||||
### 1b. 手动抓包验证
|
||||
|
||||
Explore 的自动分析可能不完美,用 verbose 模式手动确认:
|
||||
|
||||
```bash
|
||||
# 在浏览器中打开目标页面,观察网络请求
|
||||
opencli explore https://www.example.com --site mysite -v
|
||||
|
||||
# 或直接用 evaluate 测试 API
|
||||
opencli bilibili hot -v # 查看已有命令的 pipeline 每步数据流
|
||||
```
|
||||
|
||||
关注抓包结果中的关键信息:
|
||||
- **URL pattern**: `/api/v2/hot?limit=20` → 这就是你要调用的端点
|
||||
- **Method**: `GET` / `POST`
|
||||
- **Request Headers**: Cookie? Bearer? 自定义签名头(X-s、X-t)?
|
||||
- **Response Body**: JSON 结构,特别是数据在哪个路径(`data.items`、`data.list`)
|
||||
|
||||
### 1c. 高阶 API 发现捷径法则 (Heuristics)
|
||||
|
||||
在开始死磕复杂的抓包拦截之前,按照以下优先级进行尝试:
|
||||
|
||||
1. **后缀爆破法 (`.json`)**: 像 Reddit 这样复杂的网站,只要在其 URL 后加上 `.json`(例如 `/r/all.json`),就能在带 Cookie 的情况下直接利用 `fetch` 拿到极其干净的 REST 数据(Tier 2 Cookie 策略极速秒杀)。另外如功能完备的**雪球 (xueqiu)** 也可以走这种纯 API 的方式极简获取,成为你构建简单 YAML 的黄金标杆。
|
||||
2. **全局状态查找法 (`__INITIAL_STATE__`)**: 许多服务端渲染 (SSR) 的网站(如小红书、Bilibili)会将首页或详情页的完整数据挂载到全局 window 对象上。与其去拦截网络请求,不如直接 `page.evaluate('() => window.__INITIAL_STATE__')` 获取整个数据树。
|
||||
3. **主动交互触发法 (Active Interaction)**: 很多深层 API(如视频字幕、评论下的回复)是懒加载的。在静态抓包找不到数据时,尝试在 `evaluate` 步骤或手动打断点时,主动去**点击(Click)页面上的对应按钮**(如"CC"、"展开全部"),从而诱发隐藏的 Network Fetch。
|
||||
4. **框架探测与 Store Action 截断**: 如果站点使用 Vue + Pinia,可以使用 `tap` 步骤调用 action,让前端框架代替你完成复杂的鉴权签名封装。
|
||||
5. **底层 XHR/Fetch 拦截**: 最后手段,当上述都不行时,使用 TypeScript 适配器进行无侵入式的请求抓取。
|
||||
|
||||
### 1d. 框架检测
|
||||
|
||||
Explore 自动检测前端框架。如果需要手动确认:
|
||||
|
||||
```bash
|
||||
# 在已打开目标网站的情况下
|
||||
opencli evaluate "(()=>{
|
||||
const vue3 = !!document.querySelector('#app')?.__vue_app__;
|
||||
const vue2 = !!document.querySelector('#app')?.__vue__;
|
||||
const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
|
||||
const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
|
||||
return JSON.stringify({vue3, vue2, react, pinia});
|
||||
})()"
|
||||
```
|
||||
|
||||
Vue + Pinia 的站点(如小红书)可以直接通过 Store Action 绕过签名。
|
||||
|
||||
---
|
||||
|
||||
## Step 2: 选择认证策略
|
||||
|
||||
OpenCLI 提供 5 级认证策略。使用 `cascade` 命令自动探测:
|
||||
|
||||
```bash
|
||||
opencli cascade https://api.example.com/hot
|
||||
```
|
||||
|
||||
### 策略决策树
|
||||
|
||||
```
|
||||
直接 fetch(url) 能拿到数据?
|
||||
→ ✅ Tier 1: public(公开 API,不需要浏览器)
|
||||
→ ❌ fetch(url, {credentials:'include'}) 带 Cookie 能拿到?
|
||||
→ ✅ Tier 2: cookie(最常见,evaluate 步骤内 fetch)
|
||||
→ ❌ → 加上 Bearer / CSRF header 后能拿到?
|
||||
→ ✅ Tier 3: header(如 Twitter ct0 + Bearer)
|
||||
→ ❌ → 网站有 Pinia/Vuex Store?
|
||||
→ ✅ Tier 4: intercept(Store Action + XHR 拦截)
|
||||
→ ❌ Tier 5: ui(UI 自动化,最后手段)
|
||||
```
|
||||
|
||||
### 各策略对比
|
||||
|
||||
| Tier | 策略 | 速度 | 复杂度 | 适用场景 | 实例 |
|
||||
|------|------|------|--------|---------|------|
|
||||
| 1 | `public` | ⚡ ~1s | 最简 | 公开 API,无需登录 | Hacker News, V2EX |
|
||||
| 2 | `cookie` | 🔄 ~7s | 简单 | Cookie 认证即可 | Bilibili, Zhihu, Reddit |
|
||||
| 3 | `header` | 🔄 ~7s | 中等 | 需要 CSRF token 或 Bearer | Twitter GraphQL |
|
||||
| 4 | `intercept` | 🔄 ~10s | 较高 | 请求有复杂签名 | 小红书 (Pinia + XHR) |
|
||||
| 5 | `ui` | 🐌 ~15s+ | 最高 | 无 API,纯 DOM 解析 | 遗留网站 |
|
||||
|
||||
---
|
||||
|
||||
## Step 2.5: 准备工作(写代码之前)
|
||||
|
||||
### 先找模板:从最相似的现有适配器开始
|
||||
|
||||
**不要从零开始写**。先看看同站点已有哪些适配器:
|
||||
|
||||
```bash
|
||||
ls src/clis/<site>/ # 看看已有什么
|
||||
cat src/clis/<site>/feed.ts # 读最相似的那个
|
||||
```
|
||||
|
||||
最高效的方式是 **复制最相似的适配器,然后改 3 个地方**:
|
||||
1. `name` → 新命令名
|
||||
2. API URL → 你在 Step 1 发现的端点
|
||||
3. 字段映射 → 对应新 API 的字段
|
||||
|
||||
### 平台 SDK 速查表
|
||||
|
||||
写 TS 适配器之前,先看看你的目标站点有没有**现成的 helper 函数**可以复用:
|
||||
|
||||
#### Bilibili (`src/clis/bilibili/utils.ts`)
|
||||
|
||||
| 函数 | 用途 | 何时使用 |
|
||||
|------|------|----------|
|
||||
| `fetchJson(page, url)` | 带 Cookie 的 fetch + JSON 解析 | 普通 Cookie-tier API |
|
||||
| `apiGet(page, path, {signed, params})` | 带 Wbi 签名的 API 调用 | URL 含 `/wbi/` 的接口 |
|
||||
| `getSelfUid(page)` | 获取当前登录用户的 UID | "我的xxx" 类命令 |
|
||||
| `resolveUid(page, input)` | 解析用户输入的 UID(支持数字/URL) | `--uid` 参数处理 |
|
||||
| `wbiSign(page, params)` | 底层 Wbi 签名生成 | 通常不直接用,`apiGet` 已封装 |
|
||||
| `stripHtml(s)` | 去除 HTML 标签 | 清理富文本字段 |
|
||||
|
||||
**如何判断需不需要 `apiGet`**?看 Network 请求 URL:
|
||||
- 含 `/wbi/` 或 `w_rid=` → 必须用 `apiGet(..., { signed: true })`
|
||||
- 不含 → 直接用 `fetchJson`
|
||||
|
||||
> 其他站点(Twitter、小红书等)暂无专用 SDK,直接用 `page.evaluate` + `fetch` 即可。
|
||||
|
||||
---
|
||||
|
||||
## Step 3: 编写适配器
|
||||
|
||||
### YAML vs TS?先看决策树
|
||||
|
||||
```
|
||||
你的 pipeline 里有 evaluate 步骤(内嵌 JS 代码)?
|
||||
→ ✅ 用 TypeScript (src/clis/<site>/<name>.ts),保存即自动动态注册
|
||||
→ ❌ 纯声明式(navigate + tap + map + limit)?
|
||||
→ ✅ 用 YAML (src/clis/<site>/<name>.yaml),保存即自动注册
|
||||
```
|
||||
|
||||
| 场景 | 选择 | 示例 |
|
||||
|------|------|------|
|
||||
| 纯 fetch/select/map/limit | YAML | `v2ex/hot.yaml`, `hackernews/top.yaml` |
|
||||
| navigate + evaluate(fetch) + map | YAML(评估复杂度) | `zhihu/hot.yaml` |
|
||||
| navigate + tap + map | YAML ✅ | `xiaohongshu/feed.yaml`, `xiaohongshu/notifications.yaml` |
|
||||
| 有复杂 JS 逻辑(Pinia state 读取、条件分支) | TS | `xiaohongshu/me.ts`, `bilibili/me.ts` |
|
||||
| XHR 拦截 + 签名 | TS | `xiaohongshu/search.ts` |
|
||||
| GraphQL / 分页 / Wbi 签名 | TS | `bilibili/search.ts`, `twitter/search.ts` |
|
||||
|
||||
> **经验法则**:如果你发现 YAML 里嵌了超过 10 行 JS,改用 TS 更可维护。
|
||||
|
||||
### 通用模式:分页 API
|
||||
|
||||
很多 API 使用 `pn`(页码)+ `ps`(每页数量)分页。标准处理模式:
|
||||
|
||||
```typescript
|
||||
args: [
|
||||
{ name: 'page', type: 'int', required: false, default: 1, help: '页码' },
|
||||
{ name: 'limit', type: 'int', required: false, default: 50, help: '每页数量 (最大 50)' },
|
||||
],
|
||||
func: async (page, kwargs) => {
|
||||
const pn = kwargs.page ?? 1;
|
||||
const ps = Math.min(kwargs.limit ?? 50, 50); // 尊重 API 的 ps 上限
|
||||
const payload = await fetchJson(page,
|
||||
`https://api.example.com/list?pn=${pn}&ps=${ps}`
|
||||
);
|
||||
return payload.data?.list || [];
|
||||
},
|
||||
```
|
||||
|
||||
> 大多数站点的 `ps` 上限是 20~50。超过会被静默截断或返回错误。
|
||||
|
||||
### 方式 A: YAML Pipeline(声明式,推荐)
|
||||
|
||||
文件路径: `src/clis/<site>/<name>.yaml`,放入即自动注册。
|
||||
|
||||
#### Tier 1 — 公开 API 模板
|
||||
|
||||
```yaml
|
||||
# src/clis/v2ex/hot.yaml
|
||||
site: v2ex
|
||||
name: hot
|
||||
description: V2EX 热门话题
|
||||
domain: www.v2ex.com
|
||||
strategy: public
|
||||
browser: false
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
pipeline:
|
||||
- fetch:
|
||||
url: https://www.v2ex.com/api/topics/hot.json
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
replies: ${{ item.replies }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, replies]
|
||||
```
|
||||
|
||||
#### Tier 2 — Cookie 认证模板(最常用)
|
||||
|
||||
```yaml
|
||||
# src/clis/zhihu/hot.yaml
|
||||
site: zhihu
|
||||
name: hot
|
||||
description: 知乎热榜
|
||||
domain: www.zhihu.com
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.zhihu.com # 先加载页面建立 session
|
||||
|
||||
- evaluate: | # 在浏览器内发请求,自动带 Cookie
|
||||
(async () => {
|
||||
const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
|
||||
credentials: 'include'
|
||||
});
|
||||
const d = await res.json();
|
||||
return (d?.data || []).map(item => {
|
||||
const t = item.target || {};
|
||||
return {
|
||||
title: t.title,
|
||||
heat: item.detail_text || '',
|
||||
answers: t.answer_count,
|
||||
};
|
||||
});
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
heat: ${{ item.heat }}
|
||||
answers: ${{ item.answers }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, heat, answers]
|
||||
```
|
||||
|
||||
> **关键**: `evaluate` 步骤内的 `fetch` 运行在浏览器页面内,自动携带 `credentials: 'include'`,无需手动处理 Cookie。
|
||||
|
||||
#### 进阶 — 带搜索参数
|
||||
|
||||
```yaml
|
||||
# src/clis/zhihu/search.yaml
|
||||
site: zhihu
|
||||
name: search
|
||||
description: 知乎搜索
|
||||
|
||||
args:
|
||||
query:
|
||||
type: str
|
||||
required: true
|
||||
positional: true
|
||||
description: Search query
|
||||
limit:
|
||||
type: int
|
||||
default: 10
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.zhihu.com
|
||||
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const q = encodeURIComponent('${{ args.query }}');
|
||||
const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${{ args.limit }}', {
|
||||
credentials: 'include'
|
||||
});
|
||||
const d = await res.json();
|
||||
return (d?.data || [])
|
||||
.filter(item => item.type === 'search_result')
|
||||
.map(item => ({
|
||||
title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
|
||||
type: item.object?.type || '',
|
||||
author: item.object?.author?.name || '',
|
||||
votes: item.object?.voteup_count || 0,
|
||||
}));
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
type: ${{ item.type }}
|
||||
author: ${{ item.author }}
|
||||
votes: ${{ item.votes }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, type, author, votes]
|
||||
```
|
||||
|
||||
#### Tier 4 — Store Action Bridge(`tap` 步骤,intercept 策略推荐)
|
||||
|
||||
适用于 Vue + Pinia/Vuex 的网站(如小红书),无须手动写 XHR 拦截代码:
|
||||
|
||||
```yaml
|
||||
# src/clis/xiaohongshu/notifications.yaml
|
||||
site: xiaohongshu
|
||||
name: notifications
|
||||
description: "小红书通知"
|
||||
domain: www.xiaohongshu.com
|
||||
strategy: intercept
|
||||
browser: true
|
||||
|
||||
args:
|
||||
type:
|
||||
type: str
|
||||
default: mentions
|
||||
description: "Notification type: mentions, likes, or connections"
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
columns: [rank, user, action, content, note, time]
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.xiaohongshu.com/notification
|
||||
- wait: 3
|
||||
- tap:
|
||||
store: notification # Pinia store name
|
||||
action: getNotification # Store action to call
|
||||
args: # Action arguments
|
||||
- ${{ args.type | default('mentions') }}
|
||||
capture: /you/ # URL pattern to capture response
|
||||
select: data.message_list # Extract sub-path from response
|
||||
timeout: 8
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
user: ${{ item.user_info.nickname }}
|
||||
action: ${{ item.title }}
|
||||
content: ${{ item.comment_info.content }}
|
||||
- limit: ${{ args.limit | default(20) }}
|
||||
```
|
||||
|
||||
> **`tap` 步骤自动完成**:注入 fetch+XHR 双拦截 → 查找 Pinia/Vuex store → 调用 action → 捕获匹配 URL 的响应 → 清理拦截。
|
||||
> 如果 store 或 action 找不到,会返回 `hint` 列出所有可用的 store actions,方便调试。
|
||||
|
||||
| tap 参数 | 必填 | 说明 |
|
||||
|---------|------|------|
|
||||
| `store` | ✅ | Pinia store 名称(如 `feed`, `search`, `notification`) |
|
||||
| `action` | ✅ | Store action 方法名 |
|
||||
| `capture` | ✅ | URL 子串匹配(匹配网络请求 URL) |
|
||||
| `args` | ❌ | 传给 action 的参数数组 |
|
||||
| `select` | ❌ | 从 captured JSON 中提取的路径(如 `data.items`) |
|
||||
| `timeout` | ❌ | 等待网络响应的超时秒数(默认 5s) |
|
||||
| `framework` | ❌ | `pinia` 或 `vuex`(默认自动检测) |
|
||||
|
||||
### 方式 B: TypeScript 适配器(编程式)
|
||||
|
||||
适用于需要嵌入 JS 代码读取 Pinia state、XHR 拦截、GraphQL、分页、复杂数据转换等场景。
|
||||
|
||||
文件路径: `src/clis/<site>/<name>.ts`。文件将会在运行时被动态扫描并注册(切勿在 `index.ts` 中手动 `import`)。
|
||||
|
||||
#### Tier 3 — Header 认证(Twitter)
|
||||
|
||||
```typescript
|
||||
// src/clis/twitter/search.ts
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'twitter',
|
||||
name: 'search',
|
||||
description: 'Search tweets',
|
||||
strategy: Strategy.HEADER,
|
||||
args: [{ name: 'query', required: true, positional: true }],
|
||||
columns: ['rank', 'author', 'text', 'likes'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://x.com');
|
||||
const data = await page.evaluate(`
|
||||
(async () => {
|
||||
// 从 Cookie 提取 CSRF token
|
||||
const ct0 = document.cookie.split(';')
|
||||
.map(c => c.trim())
|
||||
.find(c => c.startsWith('ct0='))?.split('=')[1];
|
||||
if (!ct0) return { error: 'Not logged in' };
|
||||
|
||||
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
|
||||
const headers = {
|
||||
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
|
||||
'X-Csrf-Token': ct0,
|
||||
'X-Twitter-Auth-Type': 'OAuth2Session',
|
||||
};
|
||||
|
||||
const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
|
||||
const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
|
||||
const res = await fetch(url, { headers, credentials: 'include' });
|
||||
return await res.json();
|
||||
})()
|
||||
`);
|
||||
// ... 解析 data
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
#### Tier 4 — XHR/Fetch 双重拦截 (Twitter/小红书 通用模式)
|
||||
|
||||
```typescript
|
||||
// src/clis/xiaohongshu/user.ts
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'xiaohongshu',
|
||||
name: 'user',
|
||||
description: '获取用户笔记',
|
||||
strategy: Strategy.INTERCEPT,
|
||||
args: [{ name: 'id', required: true }],
|
||||
columns: ['rank', 'title', 'likes', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
|
||||
await page.wait(5);
|
||||
|
||||
// XHR/Fetch 底层拦截:捕获所有包含 'v1/user/posted' 的请求
|
||||
await page.installInterceptor('v1/user/posted');
|
||||
|
||||
// 触发后端 API:模拟人类用户向底部滚动2次
|
||||
await page.autoScroll({ times: 2, delayMs: 2000 });
|
||||
|
||||
// 提取所有被拦截捕获的 JSON 响应体
|
||||
const requests = await page.getInterceptedRequests();
|
||||
if (!requests || requests.length === 0) return [];
|
||||
|
||||
let results = [];
|
||||
for (const req of requests) {
|
||||
if (req.data?.data?.notes) {
|
||||
for (const note of req.data.data.notes) {
|
||||
results.push({
|
||||
title: note.display_title || '',
|
||||
likes: note.interact_info?.liked_count || '0',
|
||||
url: `https://explore/${note.note_id || note.id}`
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results.slice(0, 20).map((item, i) => ({
|
||||
rank: i + 1, ...item,
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
> **拦截核心思路**:不自己构造签名,而是利用 `installInterceptor` 劫持网站自己的 `XMLHttpRequest` 和 `fetch`,让网站发请求,我们直接在底层取出解析好的 `response.json()`。
|
||||
|
||||
> **级联请求**(如 BVID→CID→字幕)的完整模板和要点见下方[进阶模式: 级联请求](#进阶模式-级联请求-cascading-requests)章节。
|
||||
|
||||
---
|
||||
|
||||
## Step 4: 测试
|
||||
|
||||
> **构建通过 ≠ 功能正常**。`npm run build` 只验证 TypeScript / YAML 语法,不验证运行时行为。
|
||||
> 每个新命令 **必须实际运行** 并确认输出正确后才算完成。
|
||||
|
||||
### 必做清单
|
||||
|
||||
```bash
|
||||
# 1. 构建(确认语法无误)
|
||||
npm run build
|
||||
|
||||
# 2. 确认命令已注册
|
||||
opencli list | grep mysite
|
||||
|
||||
# 3. 实际运行命令(最关键!)
|
||||
opencli mysite hot --limit 3 -v # verbose 查看每步数据流
|
||||
opencli mysite hot --limit 3 -f json # JSON 输出确认字段完整
|
||||
```
|
||||
|
||||
### tap 步骤调试(intercept 策略专用)
|
||||
|
||||
> **不要猜 store name / action name**。先用 evaluate 探索,再写 YAML。
|
||||
|
||||
#### Step 1: 列出所有 Pinia store
|
||||
|
||||
在浏览器中打开目标网站后:
|
||||
|
||||
```bash
|
||||
opencli evaluate "(() => {
|
||||
const app = document.querySelector('#app')?.__vue_app__;
|
||||
const pinia = app?.config?.globalProperties?.\$pinia;
|
||||
return [...pinia._s.keys()];
|
||||
})()"
|
||||
# 输出: ["user", "feed", "search", "notification", ...]
|
||||
```
|
||||
|
||||
#### Step 2: 查看 store 的 action 名称
|
||||
|
||||
故意写一个错误 action 名,tap 会返回所有可用 actions:
|
||||
|
||||
```
|
||||
⚠ tap: Action not found: wrongName on store notification
|
||||
💡 Available: getNotification, replyComment, getNotificationCount, reset
|
||||
```
|
||||
|
||||
#### Step 3: 用 network requests 确认 capture 模式
|
||||
|
||||
```bash
|
||||
# 在浏览器打开目标页面,查看网络请求
|
||||
# 找到目标 API 的 URL 特征(如 "/you/mentions"、"homefeed")
|
||||
```
|
||||
|
||||
#### 完整流程
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────┐
|
||||
│ 1. navigate │ ──▶ │ 2. 探索 store │ ──▶ │ 3. 写 YAML │ ──▶ │ 4. 测试 │
|
||||
│ 到目标页面 │ │ name/action │ │ tap 步骤 │ │ 运行验证 │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘ └────────┘
|
||||
```
|
||||
|
||||
### Verbose 模式 & 输出验证
|
||||
|
||||
```bash
|
||||
opencli bilibili hot --limit 1 -v # 查看 pipeline 每步数据流
|
||||
opencli mysite hot -f json | jq '.[0]' # 确认 JSON 可被解析
|
||||
opencli mysite hot -f csv > data.csv # 确认 CSV 可导入
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: 提交发布
|
||||
|
||||
文件放入 `src/clis/<site>/` 即自动注册(YAML 或 TS 无需手动 import),然后:
|
||||
|
||||
```bash
|
||||
opencli list | grep mysite # 确认注册
|
||||
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push
|
||||
```
|
||||
|
||||
> **架构理念**:OpenCLI 内建 **Zero-Dependency jq** 数据流 — 所有解析在 `evaluate` 的原生 JS 内完成,外层 YAML 用 `select`/`map` 提取,无需依赖系统 `jq` 二进制。
|
||||
|
||||
---
|
||||
|
||||
## 进阶模式: 级联请求 (Cascading Requests)
|
||||
|
||||
当目标数据需要多步 API 链式获取时(如 `BVID → CID → 字幕列表 → 字幕内容`),必须使用 **TS 适配器**。YAML 无法处理这种多步逻辑。
|
||||
|
||||
### 模板代码
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { apiGet } from './utils.js'; // 复用平台 SDK
|
||||
|
||||
cli({
|
||||
site: 'bilibili',
|
||||
name: 'subtitle',
|
||||
strategy: Strategy.COOKIE,
|
||||
args: [{ name: 'bvid', required: true }],
|
||||
columns: ['index', 'from', 'to', 'content'],
|
||||
func: async (page: IPage | null, kwargs: any) => {
|
||||
if (!page) throw new Error('Requires browser');
|
||||
|
||||
// Step 1: 建立 Session
|
||||
await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);
|
||||
|
||||
// Step 2: 从页面提取中间 ID (__INITIAL_STATE__)
|
||||
const cid = await page.evaluate(`(async () => {
|
||||
return window.__INITIAL_STATE__?.videoData?.cid;
|
||||
})()`);
|
||||
if (!cid) throw new Error('无法提取 CID');
|
||||
|
||||
// Step 3: 用中间 ID 调用下一级 API (自动 Wbi 签名)
|
||||
const payload = await apiGet(page, '/x/player/wbi/v2', {
|
||||
params: { bvid: kwargs.bvid, cid },
|
||||
signed: true, // ← 自动生成 w_rid
|
||||
});
|
||||
|
||||
// Step 4: 检测风控降级 (空值断言)
|
||||
const subtitles = payload.data?.subtitle?.subtitles || [];
|
||||
const url = subtitles[0]?.subtitle_url;
|
||||
if (!url) throw new Error('subtitle_url 为空,疑似风控降级');
|
||||
|
||||
// Step 5: 拉取最终数据 (CDN JSON)
|
||||
const items = await page.evaluate(`(async () => {
|
||||
const res = await fetch(${JSON.stringify('https:' + url)});
|
||||
const json = await res.json();
|
||||
return { data: json.body || json };
|
||||
})()`);
|
||||
|
||||
return items.data.map((item, idx) => ({ ... }));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### 关键要点
|
||||
|
||||
| 步骤 | 注意事项 |
|
||||
|------|----------|
|
||||
| 提取中间 ID | 优先从 `__INITIAL_STATE__` 拿,避免额外 API 调用 |
|
||||
| Wbi 签名 | B 站 `/wbi/` 接口**强制校验** `w_rid`,纯 `fetch` 会被 403 |
|
||||
| 空值断言 | 即使 HTTP 200,核心字段可能为空串(风控降级) |
|
||||
| CDN URL | 常以 `//` 开头,记得补 `https:` |
|
||||
| `JSON.stringify` | 拼接 URL 到 evaluate 时必须用它转义,避免注入 |
|
||||
|
||||
---
|
||||
|
||||
## 常见陷阱
|
||||
|
||||
| 陷阱 | 表现 | 解决方案 |
|
||||
|------|------|---------|
|
||||
| 缺少 `navigate` | evaluate 报 `Target page context` 错误 | 在 evaluate 前加 `navigate:` 步骤 |
|
||||
| 嵌套字段访问 | `${{ item.node?.title }}` 不工作 | 在 evaluate 中 flatten 数据,不在模板中用 optional chaining |
|
||||
| 缺少 `strategy: public` | 公开 API 也启动浏览器,7s → 1s | 公开 API 加上 `strategy: public` + `browser: false` |
|
||||
| evaluate 返回字符串 | map 步骤收到 `""` 而非数组 | pipeline 有 auto-parse,但建议在 evaluate 内 `.map()` 整形 |
|
||||
| 搜索参数被 URL 编码 | `${{ args.query }}` 被浏览器二次编码 | 在 evaluate 内用 `encodeURIComponent()` 手动编码 |
|
||||
| Cookie 过期 | 返回 401 / 空数据 | 在浏览器里重新登录目标站点 |
|
||||
| Extension tab 残留 | Chrome 多出 `chrome-extension://` tab | 已自动清理;若残留,手动关闭即可 |
|
||||
| TS evaluate 格式 | `() => {}` 报 `result is not a function` | TS 中 `page.evaluate()` 必须用 IIFE:`(async () => { ... })()` |
|
||||
| 页面异步加载 | evaluate 拿到空数据(store state 还没更新) | 在 evaluate 内用 polling 等待数据出现,或增加 `wait` 时间 |
|
||||
| YAML 内嵌大段 JS | 调试困难,字符串转义问题 | 超过 10 行 JS 的命令改用 TS adapter |
|
||||
| **风控被拦截(伪200)** | 获取到的 JSON 里核心数据是 `""` (空串) | 极易被误判。必须添加断言!无核心数据立刻要求升级鉴权 Tier 并重新配置 Cookie |
|
||||
| **API 没找见** | `explore` 工具打分出来的都拿不到深层数据 | 点击页面按钮诱发懒加载数据,再结合 `getInterceptedRequests` 获取 |
|
||||
|
||||
---
|
||||
|
||||
## 用 AI Agent 自动生成适配器
|
||||
|
||||
最快的方式是让 AI Agent 完成全流程:
|
||||
|
||||
```bash
|
||||
# 一键:探索 → 分析 → 合成 → 注册
|
||||
opencli generate https://www.example.com --goal "hot"
|
||||
|
||||
# 或分步执行:
|
||||
opencli explore https://www.example.com --site mysite # 发现 API
|
||||
opencli explore https://www.example.com --auto --click "字幕,CC" # 模拟点击触发懒加载 API
|
||||
opencli synthesize mysite # 生成候选 YAML
|
||||
opencli verify mysite/hot --smoke # 冒烟测试
|
||||
```
|
||||
|
||||
生成的候选 YAML 保存在 `.opencli/explore/mysite/candidates/`,可直接复制到 `src/clis/mysite/` 并微调。
|
||||
-216
@@ -1,216 +0,0 @@
|
||||
# CLI-ONESHOT — 单点快速 CLI 生成
|
||||
|
||||
> 给一个 URL + 一句话描述,4 步生成一个 CLI 命令。
|
||||
> 完整探索式开发请看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
|
||||
|
||||
---
|
||||
|
||||
## 输入
|
||||
|
||||
| 项目 | 示例 |
|
||||
|------|------|
|
||||
| **URL** | `https://x.com/jakevin7/lists` |
|
||||
| **Goal** | 获取我的 Twitter Lists |
|
||||
|
||||
---
|
||||
|
||||
## 流程
|
||||
|
||||
### Step 1: 打开页面 + 抓包
|
||||
|
||||
```
|
||||
1. browser_navigate → 打开目标 URL
|
||||
2. 等待 3-5 秒(让页面加载完、API 请求触发)
|
||||
3. browser_network_requests → 筛选 JSON API
|
||||
```
|
||||
|
||||
**关键**:只关注返回 `application/json` 的请求,忽略静态资源。
|
||||
如果没有自动触发 API,手动点击目标按钮/标签再抓一次。
|
||||
|
||||
### Step 2: 锁定一个接口
|
||||
|
||||
从抓包结果中找到**那个**目标 API。看这几个字段:
|
||||
|
||||
| 字段 | 关注什么 |
|
||||
|------|----------|
|
||||
| URL | API 路径 pattern(如 `/i/api/graphql/xxx/ListsManagePinTimeline`) |
|
||||
| Method | GET / POST |
|
||||
| Headers | 有 Cookie? Bearer? CSRF? 自定义签名? |
|
||||
| Response | 数据在哪个路径(如 `data.list.lists`) |
|
||||
|
||||
### Step 3: 验证接口能复现
|
||||
|
||||
在 `browser_evaluate` 中用 `fetch` 复现请求:
|
||||
|
||||
```javascript
|
||||
// Tier 2 (Cookie): 大多数情况
|
||||
fetch('/api/endpoint', { credentials: 'include' }).then(r => r.json())
|
||||
|
||||
// Tier 3 (Header): 如 Twitter 需要额外 header
|
||||
const ct0 = document.cookie.match(/ct0=([^;]+)/)?.[1];
|
||||
fetch('/api/endpoint', {
|
||||
headers: { 'Authorization': 'Bearer ...', 'X-Csrf-Token': ct0 },
|
||||
credentials: 'include'
|
||||
}).then(r => r.json())
|
||||
```
|
||||
|
||||
如果 fetch 能拿到数据 → 用 YAML 或简单 TS adapter。
|
||||
如果 fetch 拿不到(签名/风控)→ 用 intercept 策略。
|
||||
|
||||
### Step 4: 套模板,生成 adapter
|
||||
|
||||
根据 Step 3 判定的策略,选一个模板生成文件。
|
||||
|
||||
---
|
||||
|
||||
## 认证速查
|
||||
|
||||
```
|
||||
fetch(url) 直接能拿到? → Tier 1: public (YAML, browser: false)
|
||||
fetch(url, {credentials:'include'})? → Tier 2: cookie (YAML)
|
||||
加 Bearer/CSRF header 后拿到? → Tier 3: header (TS)
|
||||
都不行,但页面自己能请求成功? → Tier 4: intercept (TS, installInterceptor)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 模板
|
||||
|
||||
### YAML — Cookie/Public(最简)
|
||||
|
||||
```yaml
|
||||
# src/clis/<site>/<name>.yaml
|
||||
site: mysite
|
||||
name: mycommand
|
||||
description: "一句话描述"
|
||||
domain: www.example.com
|
||||
strategy: cookie # 或 public (加 browser: false)
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
|
||||
pipeline:
|
||||
- navigate: https://www.example.com/target-page
|
||||
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const res = await fetch('/api/target', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return (d.data?.items || []).map(item => ({
|
||||
title: item.title,
|
||||
value: item.value,
|
||||
}));
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
value: ${{ item.value }}
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, value]
|
||||
```
|
||||
|
||||
### TS — Intercept(抓包模式)
|
||||
|
||||
```typescript
|
||||
// src/clis/<site>/<name>.ts
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'mysite',
|
||||
name: 'mycommand',
|
||||
description: '一句话描述',
|
||||
domain: 'www.example.com',
|
||||
strategy: Strategy.INTERCEPT,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20 },
|
||||
],
|
||||
columns: ['rank', 'title', 'value'],
|
||||
func: async (page, kwargs) => {
|
||||
// 1. 导航
|
||||
await page.goto('https://www.example.com/target-page');
|
||||
await page.wait(3);
|
||||
|
||||
// 2. 注入拦截器(URL 子串匹配)
|
||||
await page.installInterceptor('target-api-keyword');
|
||||
|
||||
// 3. 触发 API(滚动/点击)
|
||||
await page.autoScroll({ times: 2, delayMs: 2000 });
|
||||
|
||||
// 4. 读取拦截的响应
|
||||
const requests = await page.getInterceptedRequests();
|
||||
if (!requests?.length) return [];
|
||||
|
||||
let results: any[] = [];
|
||||
for (const req of requests) {
|
||||
const items = req.data?.data?.items || [];
|
||||
results.push(...items);
|
||||
}
|
||||
|
||||
return results.slice(0, kwargs.limit).map((item, i) => ({
|
||||
rank: i + 1,
|
||||
title: item.title || '',
|
||||
value: item.value || '',
|
||||
}));
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### TS — Header(如 Twitter GraphQL)
|
||||
|
||||
```typescript
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'twitter',
|
||||
name: 'mycommand',
|
||||
description: '一句话描述',
|
||||
domain: 'x.com',
|
||||
strategy: Strategy.HEADER,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 20 },
|
||||
],
|
||||
columns: ['rank', 'name', 'value'],
|
||||
func: async (page, kwargs) => {
|
||||
await page.goto('https://x.com');
|
||||
const data = await page.evaluate(`(async () => {
|
||||
const ct0 = document.cookie.match(/ct0=([^;]+)/)?.[1];
|
||||
if (!ct0) return { error: 'Not logged in' };
|
||||
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
|
||||
const res = await fetch('/i/api/graphql/QUERY_ID/Endpoint', {
|
||||
headers: {
|
||||
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
|
||||
'X-Csrf-Token': ct0,
|
||||
'X-Twitter-Auth-Type': 'OAuth2Session',
|
||||
},
|
||||
credentials: 'include',
|
||||
});
|
||||
return res.json();
|
||||
})()`);
|
||||
// 解析 data...
|
||||
return [];
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 测试(必做)
|
||||
|
||||
```bash
|
||||
npm run build # 语法检查
|
||||
opencli list | grep mysite # 确认注册
|
||||
opencli mysite mycommand --limit 3 -v # 实际运行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 就这样,没了
|
||||
|
||||
写完文件 → build → run → 提交。有问题再看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
|
||||
+1
-1
@@ -109,7 +109,7 @@ cli({
|
||||
});
|
||||
```
|
||||
|
||||
Use `opencli explore <url>` to discover APIs and see [CLI-EXPLORER.md](./CLI-EXPLORER.md) if you need the full adapter workflow.
|
||||
Use `opencli explore <url>` to discover APIs and see [opencli-explorer skill](./skills/opencli-explorer/SKILL.md) if you need the full adapter workflow.
|
||||
|
||||
### Validate Your Adapter
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# OpenCLI
|
||||
|
||||
> **Make any website, Electron App, or Local Tool your CLI.**
|
||||
> Zero risk · Reuse Chrome login · AI-powered discovery · Universal CLI Hub
|
||||
> Zero risk · Reuse Chrome/Chromium login · AI-powered discovery · Universal CLI Hub
|
||||
|
||||
[](./README.zh-CN.md)
|
||||
[](https://www.npmjs.com/package/@jackwener/opencli)
|
||||
@@ -23,7 +23,7 @@ A CLI tool that turns **any website**, **Electron app**, or **local CLI tool** i
|
||||
- **CLI All Electron** — CLI-ify apps like Antigravity Ultra! Now AI can control itself natively.
|
||||
- **Browser Automation** — `operate` gives AI agents direct browser control: click, type, extract, screenshot — any interaction, fully scriptable.
|
||||
- **Website → CLI** — Turn any website into a deterministic CLI: 70+ pre-built adapters, or crystallize your own with `opencli record`.
|
||||
- **Account-safe** — Reuses Chrome's logged-in state; your credentials never leave the browser.
|
||||
- **Account-safe** — Reuses Chrome/Chromium logged-in state; your credentials never leave the browser.
|
||||
- **Anti-detection built-in** — Patches `navigator.webdriver`, stubs `window.chrome`, fakes plugin lists, cleans ChromeDriver/Playwright globals, and strips CDP frames from Error stack traces. Extensive anti-fingerprinting and risk-control evasion measures baked in at every layer.
|
||||
- **AI Agent ready** — `explore` discovers APIs, `synthesize` generates adapters, `cascade` finds auth strategies, `operate` controls the browser directly.
|
||||
- **External CLI Hub** — Discover, auto-install, and passthrough commands to any external CLI (gh, obsidian, docker, etc). Zero setup.
|
||||
@@ -39,7 +39,7 @@ A CLI tool that turns **any website**, **Electron app**, or **local CLI tool** i
|
||||
|
||||
### 1. Install Browser Bridge Extension
|
||||
|
||||
> OpenCLI connects to your browser through a lightweight **Browser Bridge** Chrome Extension + micro-daemon (zero config, auto-start).
|
||||
> OpenCLI connects to your browser through a lightweight **Browser Bridge** Chrome/Chromium extension + micro-daemon (zero config, auto-start).
|
||||
|
||||
1. Go to the GitHub [Releases page](https://github.com/jackwener/opencli/releases) and download the latest `opencli-extension.zip`.
|
||||
2. Unzip the file and open `chrome://extensions`, enable **Developer mode** (top-right toggle).
|
||||
@@ -51,6 +51,9 @@ A CLI tool that turns **any website**, **Electron app**, or **local CLI tool** i
|
||||
|
||||
```bash
|
||||
npm install -g @jackwener/opencli
|
||||
|
||||
# Install AI skills for Claude Code / Cursor
|
||||
npx skills add jackwener/opencli
|
||||
```
|
||||
|
||||
### 3. Verify & Try
|
||||
@@ -115,9 +118,9 @@ git clone git@github.com:jackwener/opencli.git && cd opencli && npm install && n
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js**: >= 20.0.0 (or **Bun** >= 1.0)
|
||||
- **Chrome** running **and logged into the target site** (e.g. bilibili.com, zhihu.com, xiaohongshu.com, goofish.com).
|
||||
- **Chrome or Chromium** running **and logged into the target site** (e.g. bilibili.com, zhihu.com, xiaohongshu.com, goofish.com).
|
||||
|
||||
> **⚠️ Important**: Browser commands reuse your Chrome login session. You must be logged into the target website in Chrome before running commands. If you get empty data or errors, check your login status first.
|
||||
> **⚠️ Important**: Browser commands reuse your Chrome/Chromium login session. You must be logged into the target website in Chrome or Chromium before running commands. If you get empty data or errors, check your login status first.
|
||||
|
||||
## Built-in Commands
|
||||
|
||||
@@ -126,14 +129,17 @@ git clone git@github.com:jackwener/opencli.git && cd opencli && npm install && n
|
||||
| **xiaohongshu** | `search` `note` `comments` `feed` `user` `download` `publish` `notifications` `creator-notes` `creator-notes-summary` `creator-note-detail` `creator-profile` `creator-stats` |
|
||||
| **bilibili** | `hot` `search` `history` `feed` `ranking` `download` `comments` `dynamic` `favorite` `following` `me` `subtitle` `user-videos` |
|
||||
| **tieba** | `hot` `posts` `search` `read` |
|
||||
| **hupu** | `hot` `search` `detail` `reply` `like` `unlike` |
|
||||
| **twitter** | `trending` `search` `timeline` `bookmarks` `post` `download` `profile` `article` `like` `likes` `notifications` `reply` `reply-dm` `thread` `follow` `unfollow` `followers` `following` `block` `unblock` `bookmark` `unbookmark` `delete` `hide-reply` `accept` |
|
||||
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `user` `user-posts` `user-comments` `read` `save` `saved` `subscribe` `upvote` `upvoted` `comment` |
|
||||
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` |
|
||||
| **1688** | `search` `item` `store` |
|
||||
| **gemini** | `new` `ask` `image` |
|
||||
| **yuanbao** | `new` `ask` |
|
||||
| **notebooklm** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` |
|
||||
| **spotify** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` |
|
||||
| **xianyu** | `search` `item` `chat` |
|
||||
| **xiaoe** | `courses` `detail` `catalog` `play-url` `content` |
|
||||
|
||||
73+ adapters in total — **[→ see all supported sites & commands](./docs/adapters/index.md)**
|
||||
|
||||
@@ -250,9 +256,9 @@ See [Plugins Guide](./docs/guide/plugins.md) for creating your own plugin.
|
||||
|
||||
## For AI Agents (Developer Guide)
|
||||
|
||||
> **Quick mode**: To generate a single command for a specific page URL, see [CLI-ONESHOT.md](./CLI-ONESHOT.md) — just a URL + one-line goal, 4 steps done.
|
||||
> **Quick mode**: To generate a single command for a specific page URL, see [opencli-oneshot skill](./skills/opencli-oneshot/SKILL.md) — just a URL + one-line goal, 4 steps done.
|
||||
|
||||
> **Full mode**: Before writing any adapter code, read [CLI-EXPLORER.md](./CLI-EXPLORER.md). It contains the complete browser exploration workflow, the 5-tier authentication strategy decision tree, and debugging guide.
|
||||
> **Full mode**: Before writing any adapter code, read [opencli-explorer skill](./skills/opencli-explorer/SKILL.md). It contains the complete browser exploration workflow, the 5-tier authentication strategy decision tree, and debugging guide.
|
||||
|
||||
```bash
|
||||
opencli explore https://example.com --site mysite # Discover APIs + capabilities
|
||||
@@ -267,9 +273,9 @@ See **[TESTING.md](./TESTING.md)** for how to run and write tests.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **"Extension not connected"** — Ensure the Browser Bridge extension is installed and **enabled** in `chrome://extensions`.
|
||||
- **"Extension not connected"** — Ensure the Browser Bridge extension is installed and **enabled** in `chrome://extensions` in Chrome or Chromium.
|
||||
- **"attach failed: Cannot access a chrome-extension:// URL"** — Another extension may be interfering. Try disabling other extensions temporarily.
|
||||
- **Empty data or 'Unauthorized' error** — Your Chrome login session may have expired. Navigate to the target site and log in again.
|
||||
- **Empty data or 'Unauthorized' error** — Your Chrome/Chromium login session may have expired. Navigate to the target site and log in again.
|
||||
- **Node API errors** — Ensure Node.js >= 20. Some dependencies require modern Node APIs.
|
||||
- **Daemon issues** — Check status: `curl localhost:19825/status` · View logs: `curl localhost:19825/logs`
|
||||
|
||||
|
||||
+17
-11
@@ -1,7 +1,7 @@
|
||||
# OpenCLI
|
||||
|
||||
> **把任何网站、本地工具、Electron 应用变成能够让 AI 调用的命令行!**
|
||||
> 零风控 · 复用 Chrome 登录 · AI 自动发现接口 · 全能 CLI 枢纽
|
||||
> 零风控 · 复用 Chrome/Chromium 登录 · AI 自动发现接口 · 全能 CLI 枢纽
|
||||
|
||||
[](./README.md)
|
||||
[](https://www.npmjs.com/package/@jackwener/opencli)
|
||||
@@ -25,7 +25,7 @@ CLI all electron!现在支持把所有 electron 应用 CLI 化,从而组合
|
||||
- **浏览器自动化** — `operate` 赋予 AI Agent 直接操控浏览器的能力:点击、输入、提取、截图,任意交互皆可脚本化
|
||||
- **网页转 CLI** — 将任意网站变成确定性命令行工具:73+ 预置适配器,或用 `opencli record` 沉淀自己的操作
|
||||
- **多站点覆盖** — 73+ 站点,横跨全球与中国平台(B站、知乎、小红书、Reddit、HackerNews 等),并支持通过 CDP 控制桌面 Electron 应用
|
||||
- **零风控** — 复用 Chrome 登录态,无需存储任何凭证
|
||||
- **零风控** — 复用 Chrome/Chromium 登录态,无需存储任何凭证
|
||||
- **外部 CLI 枢纽** — 统一发现、自动安装、透传执行 `gh`、`docker` 等本地 CLI
|
||||
- **自修复配置** — `opencli doctor` 自动启动 daemon,诊断扩展和浏览器连接状态
|
||||
- **AI 原生** — `explore` 自动发现 API,`synthesize` 生成适配器,`cascade` 探测认证策略,`operate` 直接控制浏览器
|
||||
@@ -35,11 +35,11 @@ CLI all electron!现在支持把所有 electron 应用 CLI 化,从而组合
|
||||
## 前置要求
|
||||
|
||||
- **Node.js**: >= 20.0.0
|
||||
- **Chrome** 浏览器正在运行,且**已登录目标网站**(如 bilibili.com、zhihu.com、xiaohongshu.com、goofish.com)
|
||||
- **Chrome 或 Chromium** 浏览器正在运行,且**已登录目标网站**(如 bilibili.com、zhihu.com、xiaohongshu.com、goofish.com)
|
||||
|
||||
> **⚠️ 重要**:大多数命令复用你的 Chrome 登录状态。运行命令前,你必须已在 Chrome 中打开目标网站并完成登录。如果获取到空数据或报错,请先检查你的浏览器登录状态。
|
||||
> **⚠️ 重要**:大多数命令复用你的 Chrome/Chromium 登录状态。运行命令前,你必须已在 Chrome 或 Chromium 中打开目标网站并完成登录。如果获取到空数据或报错,请先检查你的浏览器登录状态。
|
||||
|
||||
OpenCLI 通过轻量化的 **Browser Bridge** Chrome 扩展 + 微型 daemon 与浏览器通信(零配置,自动启动)。
|
||||
OpenCLI 通过轻量化的 **Browser Bridge** Chrome/Chromium 扩展 + 微型 daemon 与浏览器通信(零配置,自动启动)。
|
||||
|
||||
### Browser Bridge 扩展配置
|
||||
|
||||
@@ -47,7 +47,7 @@ OpenCLI 通过轻量化的 **Browser Bridge** Chrome 扩展 + 微型 daemon 与
|
||||
|
||||
**方式一:下载构建好的安装包(推荐)**
|
||||
1. 到 GitHub [Releases 页面](https://github.com/jackwener/opencli/releases) 下载最新的 `opencli-extension.zip`。
|
||||
2. 解压后打开 Chrome 的 `chrome://extensions`,启用右上角的 **开发者模式**。
|
||||
2. 解压后在 Chrome 或 Chromium 中打开 `chrome://extensions`,启用右上角的 **开发者模式**。
|
||||
3. 点击 **加载已解压的扩展程序**,选择解压后的文件夹。
|
||||
|
||||
**方式二:加载源码(针对开发者)**
|
||||
@@ -69,6 +69,9 @@ OpenCLI 通过轻量化的 **Browser Bridge** Chrome 扩展 + 微型 daemon 与
|
||||
|
||||
```bash
|
||||
npm install -g @jackwener/opencli
|
||||
|
||||
# 安装 AI Skills(Claude Code / Cursor)
|
||||
npx skills add jackwener/opencli
|
||||
```
|
||||
|
||||
直接使用:
|
||||
@@ -129,6 +132,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
|
||||
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | 浏览器 |
|
||||
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | 浏览器 |
|
||||
| **tieba** | `hot` `posts` `search` `read` | 浏览器 |
|
||||
| **hupu** | `hot` `search` `detail` `reply` `like` `unlike` | 浏览器 |
|
||||
| **cursor** | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` | 桌面端 |
|
||||
| **bilibili** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | 浏览器 |
|
||||
| **codex** | `status` `send` `read` `new` `dump` `extract-diff` `model` `ask` `screenshot` `history` `export` | 桌面端 |
|
||||
@@ -142,6 +146,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
|
||||
| **antigravity** | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` | 桌面端 |
|
||||
| **chatgpt** | `status` `new` `send` `read` `ask` `model` | 桌面端 |
|
||||
| **xiaohongshu** | `search` `notifications` `feed` `user` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | 浏览器 |
|
||||
| **xiaoe** | `courses` `detail` `catalog` `play-url` `content` | 浏览器 |
|
||||
| **apple-podcasts** | `search` `episodes` `top` | 公开 |
|
||||
| **xiaoyuzhou** | `podcast` `podcast-episodes` `episode` | 公开 |
|
||||
| **zhihu** | `hot` `search` `question` `download` | 浏览器 |
|
||||
@@ -181,6 +186,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
|
||||
| **facebook** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` | 浏览器 |
|
||||
| **google** | `news` `search` `suggest` `trends` | 公开 |
|
||||
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` | 浏览器 |
|
||||
| **1688** | `search` `item` `store` | 浏览器 |
|
||||
| **gemini** | `new` `ask` `image` | 浏览器 |
|
||||
| **spotify** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` | OAuth API |
|
||||
| **notebooklm** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` | 浏览器 |
|
||||
@@ -364,9 +370,9 @@ opencli plugin uninstall my-tool # 卸载
|
||||
|
||||
如果你是一个被要求查阅代码并编写新 `opencli` 适配器的 AI,请遵守以下工作流。
|
||||
|
||||
> **快速模式**:只想为某个页面快速生成一个命令?看 [CLI-ONESHOT.md](./CLI-ONESHOT.md) — 给一个 URL + 一句话描述,4 步搞定。
|
||||
> **快速模式**:只想为某个页面快速生成一个命令?看 [opencli-oneshot skill](./skills/opencli-oneshot/SKILL.md) — 给一个 URL + 一句话描述,4 步搞定。
|
||||
|
||||
> **完整模式**:在编写任何新代码前,先阅读 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。它包含完整的适配器探索开发指南、API 探测流程、5级认证策略以及常见陷阱。
|
||||
> **完整模式**:在编写任何新代码前,先阅读 [opencli-explorer skill](./skills/opencli-explorer/SKILL.md)。它包含完整的适配器探索开发指南、API 探测流程、5级认证策略以及常见陷阱。
|
||||
|
||||
```bash
|
||||
# 1. Deep Explore — 网络拦截 → 响应分析 → 能力推理 → 框架检测
|
||||
@@ -387,11 +393,11 @@ opencli cascade https://api.example.com/data
|
||||
## 常见问题排查
|
||||
|
||||
- **"Extension not connected" 报错**
|
||||
- 确保你当前的 Chrome 已安装且**开启了** opencli Browser Bridge 扩展(在 `chrome://extensions` 中检查)。
|
||||
- 确保你当前的 Chrome 或 Chromium 已安装且**开启了** opencli Browser Bridge 扩展(在 `chrome://extensions` 中检查)。
|
||||
- **"attach failed: Cannot access a chrome-extension:// URL" 报错**
|
||||
- 其他 Chrome 扩展(如 youmind、New Tab Override 或 AI 助手类扩展)可能产生冲突。请尝试**暂时禁用其他扩展**后重试。
|
||||
- 其他 Chrome/Chromium 扩展(如 youmind、New Tab Override 或 AI 助手类扩展)可能产生冲突。请尝试**暂时禁用其他扩展**后重试。
|
||||
- **返回空数据,或者报错 "Unauthorized"**
|
||||
- Chrome 里的登录态可能已经过期。请打开当前 Chrome 页面,在新标签页重新手工登录或刷新该页面。
|
||||
- Chrome/Chromium 里的登录态可能已经过期。请打开当前页面,在新标签页重新手工登录或刷新该页面。
|
||||
- **Node API 错误 (如 parseArgs, fs 等)**
|
||||
- 确保 Node.js 版本 `>= 20`。
|
||||
- **Daemon 问题**
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"name": "extract-title-iana",
|
||||
"steps": [
|
||||
"opencli operate open https://www.iana.org",
|
||||
"opencli operate eval \"document.querySelector('h1')?.textContent\""
|
||||
"opencli operate eval \"document.querySelector('h1')?.textContent || document.title || document.querySelector('title')?.textContent\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
@@ -67,7 +67,7 @@
|
||||
"name": "extract-github-readme-heading",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/vercel/next.js",
|
||||
"opencli operate eval \"document.querySelector('article h1, article h2')?.textContent?.trim()\""
|
||||
"opencli operate eval \"document.querySelector('[data-testid=readme] h1, [data-testid=readme] h2, #readme h1, #readme h2, article h1, article h2, .markdown-body h1, .markdown-body h2')?.textContent?.trim()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
@@ -143,7 +143,7 @@
|
||||
"name": "list-quotes-3",
|
||||
"steps": [
|
||||
"opencli operate open https://quotes.toscrape.com",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote')].slice(0,3).map(el=>({text:el.querySelector('.text')?.textContent,author:el.querySelector('.author')?.textContent})))\""
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote, [class*=quote]')].slice(0,3).map(el=>({text:(el.querySelector('.text, [class*=text]')?.textContent)||(el.querySelector('span')?.textContent),author:(el.querySelector('.author, [class*=author]')?.textContent)||(el.querySelector('small')?.textContent)})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
@@ -165,7 +165,7 @@
|
||||
"name": "list-github-trending",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/trending",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row')].slice(0,3).map(el=>({name:el.querySelector('h2 a')?.textContent?.trim().replace(/\\s+/g,' '),desc:el.querySelector('p')?.textContent?.trim()})))\""
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row, article[class*=Box-row], [data-hpc] article, .Box article')].slice(0,3).map(el=>({name:(el.querySelector('h2 a, h1 a')?.textContent?.trim().replace(/\\\\s+/g,' '))||(el.querySelector('a[href^=\\\"/\\\"]')?.textContent?.trim()),desc:el.querySelector('p')?.textContent?.trim()})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
@@ -176,7 +176,7 @@
|
||||
"name": "list-github-trending-lang",
|
||||
"steps": [
|
||||
"opencli operate open https://github.com/trending/python",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row')].slice(0,5).map(el=>({name:el.querySelector('h2 a')?.textContent?.trim().replace(/\\s+/g,' ')})))\""
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row, article[class*=Box-row], [data-hpc] article, .Box article')].slice(0,5).map(el=>({name:(el.querySelector('h2 a, h1 a')?.textContent?.trim().replace(/\\\\s+/g,' '))||(el.querySelector('a[href^=\\\"/\\\"]')?.textContent?.trim())})))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
@@ -208,8 +208,7 @@
|
||||
{
|
||||
"name": "search-google",
|
||||
"steps": [
|
||||
"opencli operate open https://www.google.com",
|
||||
"opencli operate eval \"document.querySelector('textarea[name=q], input[name=q]').value='opencli github';document.querySelector('form').submit();'submitted'\"",
|
||||
"opencli operate open https://www.google.com/search?q=opencli+github",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('h3')].slice(0,3).map(h=>h.textContent))\""
|
||||
],
|
||||
@@ -250,8 +249,7 @@
|
||||
{
|
||||
"name": "search-wiki",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org",
|
||||
"opencli operate eval \"document.querySelector('input[name=search]').value='Rust programming language';document.querySelector('form#searchform, form[role=search]').submit();'submitted'\"",
|
||||
"opencli operate open \"https://en.wikipedia.org/w/index.php?search=Rust+programming+language&title=Special:Search&go=Go\"",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const ps = document.querySelectorAll('#mw-content-text .mw-parser-output > p'); for (const p of ps) { const t = p.textContent?.trim(); if (t && t.length > 50) return t.slice(0,300); } return ''; })()\""
|
||||
],
|
||||
@@ -264,11 +262,9 @@
|
||||
{
|
||||
"name": "search-npm",
|
||||
"steps": [
|
||||
"opencli operate open https://www.npmjs.com",
|
||||
"opencli operate state",
|
||||
"opencli operate type 1 \"react\"",
|
||||
"opencli operate keys Enter",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=pkg-list-item] h3, section h3')].slice(0,3).map(h=>h.textContent?.trim()))\""
|
||||
"opencli operate open https://www.npmjs.com/search?q=react",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=pkg-list-item] h3, section h3, .package-list-item h3, a[class*=package] h3')].slice(0,3).map(h=>h.textContent?.trim()))\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
@@ -329,8 +325,8 @@
|
||||
"name": "nav-click-wiki-link",
|
||||
"steps": [
|
||||
"opencli operate open https://en.wikipedia.org/wiki/JavaScript",
|
||||
"opencli operate eval \"document.querySelector('#toc a, .toc a, [href=\\\"#History\\\"]')?.click(); 'clicked'\"",
|
||||
"opencli operate eval \"document.querySelector('#History, #History ~ p')?.textContent?.slice(0,100)\""
|
||||
"opencli operate eval \"document.querySelector('.vector-toc-contents a[href*=History], #toc a[href*=History], .toc a[href*=History], [href=\\\"#History\\\"]')?.click(); 'clicked'\"",
|
||||
"opencli operate eval \"document.querySelector('#History')?.textContent?.slice(0,100) || document.querySelector('[id*=History]')?.textContent?.slice(0,100)\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "nonEmpty"
|
||||
@@ -480,7 +476,7 @@
|
||||
"name": "form-textarea",
|
||||
"steps": [
|
||||
"opencli operate open https://httpbin.org/forms/post",
|
||||
"opencli operate eval \"var ta=document.querySelector('textarea[name=comments]');ta.value='AutoResearch test';ta.dispatchEvent(new Event('input',{bubbles:true}));ta.value\""
|
||||
"opencli operate eval \"var ta=document.querySelector('textarea[name=comments], textarea[name=delivery], textarea');ta.value='AutoResearch test';ta.dispatchEvent(new Event('input',{bubbles:true}));ta.value\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
|
||||
@@ -64,7 +64,7 @@ async function modify(ctx: ModifyContext, config: AutoResearchConfig): Promise<s
|
||||
`claude -p --dangerously-skip-permissions --allowedTools "Bash(npm:*),Bash(npx:*),Bash(git:*),Read,Edit,Write,Glob,Grep" --output-format text --no-session-persistence "${prompt.replace(/"/g, '\\"')}"`,
|
||||
{
|
||||
cwd: ROOT,
|
||||
timeout: 180_000,
|
||||
timeout: 300_000,
|
||||
encoding: 'utf-8',
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
env: process.env,
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
/**
|
||||
* Layer 5: Publish Testing — end-to-end content creation via operate commands
|
||||
*
|
||||
* Tests the full chain: read content → navigate to platform → fill title+body → (optionally) publish → verify → cleanup
|
||||
*
|
||||
* Task types:
|
||||
* fill-only: navigate + fill fields + verify content was entered (safe, no side effects)
|
||||
* publish: full publish + verify + cleanup (deletes the post after verification)
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx autoresearch/eval-publish.ts # Run all tasks
|
||||
* npx tsx autoresearch/eval-publish.ts --task twitter-fill # Run single task
|
||||
* npx tsx autoresearch/eval-publish.ts --type fill-only # Run only fill tasks (safe)
|
||||
* npx tsx autoresearch/eval-publish.ts --type publish # Run only publish tasks (destructive)
|
||||
* npx tsx autoresearch/eval-publish.ts --platform twitter # Run only twitter tasks
|
||||
*/
|
||||
|
||||
import { execSync } from 'node:child_process';
|
||||
import { readFileSync, writeFileSync, mkdirSync, readdirSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const PROJECT_ROOT = join(__dirname, '..');
|
||||
const TASKS_FILE = join(__dirname, 'publish-tasks.json');
|
||||
const RESULTS_DIR = join(__dirname, 'results');
|
||||
|
||||
interface PublishTask {
|
||||
name: string;
|
||||
platform: string;
|
||||
type: 'fill-only' | 'publish';
|
||||
description: string;
|
||||
steps: string[];
|
||||
judge: JudgeCriteria;
|
||||
cleanup?: string[];
|
||||
note?: string;
|
||||
}
|
||||
|
||||
type JudgeCriteria =
|
||||
| { type: 'contains'; value: string }
|
||||
| { type: 'arrayMinLength'; minLength: number }
|
||||
| { type: 'nonEmpty' }
|
||||
| { type: 'matchesPattern'; pattern: string };
|
||||
|
||||
interface TaskResult {
|
||||
name: string;
|
||||
platform: string;
|
||||
taskType: 'fill-only' | 'publish';
|
||||
passed: boolean;
|
||||
duration: number;
|
||||
cleanupResult?: string;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
function judge(criteria: JudgeCriteria, output: string): boolean {
|
||||
try {
|
||||
switch (criteria.type) {
|
||||
case 'contains':
|
||||
return output.toLowerCase().includes(criteria.value.toLowerCase());
|
||||
case 'arrayMinLength': {
|
||||
try {
|
||||
const arr = JSON.parse(output);
|
||||
if (Array.isArray(arr)) return arr.length >= criteria.minLength;
|
||||
} catch { /* not JSON */ }
|
||||
return false;
|
||||
}
|
||||
case 'nonEmpty':
|
||||
return output.trim().length > 0 && output.trim() !== 'null' && output.trim() !== 'undefined';
|
||||
case 'matchesPattern':
|
||||
return new RegExp(criteria.pattern, 'i').test(output);
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function runCommand(cmd: string, timeout = 30000): string {
|
||||
const localCmd = cmd.replace(/^opencli /, `node dist/main.js `);
|
||||
try {
|
||||
return execSync(localCmd, {
|
||||
cwd: PROJECT_ROOT,
|
||||
timeout,
|
||||
encoding: 'utf-8',
|
||||
env: process.env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
}).trim();
|
||||
} catch (err: any) {
|
||||
return err.stdout?.trim() || err.stderr?.trim() || '';
|
||||
}
|
||||
}
|
||||
|
||||
function runTask(task: PublishTask): TaskResult {
|
||||
const start = Date.now();
|
||||
|
||||
try {
|
||||
// Run main steps
|
||||
let lastOutput = '';
|
||||
for (let i = 0; i < task.steps.length; i++) {
|
||||
const step = task.steps[i];
|
||||
process.stderr.write(` step ${i + 1}/${task.steps.length}: ${step.slice(0, 60)}...\n`);
|
||||
lastOutput = runCommand(step, 45000);
|
||||
}
|
||||
|
||||
const passed = judge(task.judge, lastOutput);
|
||||
|
||||
// Run cleanup steps (if publish type and cleanup defined)
|
||||
let cleanupResult: string | undefined;
|
||||
if (task.cleanup && task.cleanup.length > 0) {
|
||||
process.stderr.write(` cleanup: ${task.cleanup.length} steps...\n`);
|
||||
let cleanupOutput = '';
|
||||
for (const step of task.cleanup) {
|
||||
cleanupOutput = runCommand(step, 30000);
|
||||
}
|
||||
cleanupResult = cleanupOutput.slice(0, 100);
|
||||
}
|
||||
|
||||
return {
|
||||
name: task.name,
|
||||
platform: task.platform,
|
||||
taskType: task.type,
|
||||
passed,
|
||||
duration: Date.now() - start,
|
||||
cleanupResult,
|
||||
error: passed ? undefined : `Output: ${lastOutput.slice(0, 150)}`,
|
||||
};
|
||||
} catch (err: any) {
|
||||
return {
|
||||
name: task.name,
|
||||
platform: task.platform,
|
||||
taskType: task.type,
|
||||
passed: false,
|
||||
duration: Date.now() - start,
|
||||
error: err.message?.slice(0, 150),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const singleTask = args.includes('--task') ? args[args.indexOf('--task') + 1] : null;
|
||||
const filterType = args.includes('--type') ? args[args.indexOf('--type') + 1] : null;
|
||||
const filterPlatform = args.includes('--platform') ? args[args.indexOf('--platform') + 1] : null;
|
||||
|
||||
const allTasks: PublishTask[] = JSON.parse(readFileSync(TASKS_FILE, 'utf-8'));
|
||||
let tasks = allTasks;
|
||||
|
||||
if (singleTask) tasks = tasks.filter(t => t.name === singleTask);
|
||||
if (filterType) tasks = tasks.filter(t => t.type === filterType);
|
||||
if (filterPlatform) tasks = tasks.filter(t => t.platform === filterPlatform);
|
||||
|
||||
if (tasks.length === 0) {
|
||||
console.error(`No tasks matched filters: task=${singleTask}, type=${filterType}, platform=${filterPlatform}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const fillTasks = tasks.filter(t => t.type === 'fill-only');
|
||||
const publishTasks = tasks.filter(t => t.type === 'publish');
|
||||
|
||||
console.log(`\n📝 Layer 5: Publish Testing — ${tasks.length} tasks`);
|
||||
console.log(` fill-only: ${fillTasks.length} | publish: ${publishTasks.length}`);
|
||||
console.log(` platforms: ${[...new Set(tasks.map(t => t.platform))].join(', ')}\n`);
|
||||
|
||||
const results: TaskResult[] = [];
|
||||
|
||||
for (let i = 0; i < tasks.length; i++) {
|
||||
const task = tasks[i];
|
||||
const icon = task.type === 'publish' ? '🚀' : '📋';
|
||||
process.stdout.write(` [${i + 1}/${tasks.length}] ${icon} ${task.name} (${task.platform})...`);
|
||||
|
||||
const result = runTask(task);
|
||||
results.push(result);
|
||||
|
||||
const status = result.passed ? '✓' : '✗';
|
||||
const cleanup = result.cleanupResult ? ` [cleanup: ${result.cleanupResult.slice(0, 30)}]` : '';
|
||||
console.log(` ${status} (${(result.duration / 1000).toFixed(1)}s)${cleanup}`);
|
||||
|
||||
// Close browser between tasks for clean state
|
||||
if (i < tasks.length - 1) {
|
||||
try { runCommand('opencli operate close'); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
|
||||
// Final close
|
||||
try { runCommand('opencli operate close'); } catch { /* ignore */ }
|
||||
|
||||
// Summary
|
||||
const totalPassed = results.filter(r => r.passed).length;
|
||||
const fillPassed = results.filter(r => r.taskType === 'fill-only' && r.passed).length;
|
||||
const publishPassed = results.filter(r => r.taskType === 'publish' && r.passed).length;
|
||||
const totalDuration = results.reduce((s, r) => s + r.duration, 0);
|
||||
|
||||
const fillTotal = results.filter(r => r.taskType === 'fill-only').length;
|
||||
const publishTotal = results.filter(r => r.taskType === 'publish').length;
|
||||
|
||||
console.log(`\n${'─'.repeat(50)}`);
|
||||
console.log(` Score: ${totalPassed}/${results.length}`);
|
||||
console.log(` fill-only: ${fillPassed}/${fillTotal}`);
|
||||
console.log(` publish: ${publishPassed}/${publishTotal}`);
|
||||
console.log(` Time: ${Math.round(totalDuration / 1000)}s`);
|
||||
|
||||
// Platform breakdown
|
||||
const platforms = [...new Set(results.map(r => r.platform))];
|
||||
for (const p of platforms) {
|
||||
const pr = results.filter(r => r.platform === p);
|
||||
const pp = pr.filter(r => r.passed).length;
|
||||
console.log(` ${p}: ${pp}/${pr.length}`);
|
||||
}
|
||||
|
||||
const failures = results.filter(r => !r.passed);
|
||||
if (failures.length > 0) {
|
||||
console.log(`\n Failures:`);
|
||||
for (const f of failures) {
|
||||
console.log(` ✗ ${f.name} [${f.platform}/${f.taskType}]: ${f.error ?? 'unknown'}`);
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Save result
|
||||
mkdirSync(RESULTS_DIR, { recursive: true });
|
||||
const existing = readdirSync(RESULTS_DIR).filter(f => f.startsWith('publish-')).length;
|
||||
const roundNum = String(existing + 1).padStart(3, '0');
|
||||
const resultPath = join(RESULTS_DIR, `publish-${roundNum}.json`);
|
||||
writeFileSync(resultPath, JSON.stringify({
|
||||
timestamp: new Date().toISOString(),
|
||||
score: `${totalPassed}/${results.length}`,
|
||||
fillScore: `${fillPassed}/${fillTotal}`,
|
||||
publishScore: `${publishPassed}/${publishTotal}`,
|
||||
duration: `${Math.round(totalDuration / 1000)}s`,
|
||||
tasks: results,
|
||||
}, null, 2), 'utf-8');
|
||||
console.log(` Results saved to: ${resultPath}`);
|
||||
console.log(`\nSCORE=${totalPassed}/${results.length}`);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
/**
|
||||
* Layer 4: Save as CLI Testing — "Save as CLI" Pipeline
|
||||
*
|
||||
* Tests the full operate init → write adapter → operate verify flow.
|
||||
* Validates that browser exploration can be crystallized into reusable CLI adapters.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx autoresearch/eval-save.ts # Run all tasks
|
||||
* npx tsx autoresearch/eval-save.ts --task hn-top # Run single task
|
||||
*/
|
||||
|
||||
import { execSync } from 'node:child_process';
|
||||
import { readFileSync, writeFileSync, mkdirSync, readdirSync, existsSync, rmSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { homedir } from 'node:os';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const TASKS_FILE = join(__dirname, 'save-tasks.json');
|
||||
const RESULTS_DIR = join(__dirname, 'results');
|
||||
const USER_CLIS_DIR = join(homedir(), '.opencli', 'clis');
|
||||
|
||||
interface SaveTask {
|
||||
name: string;
|
||||
site: string;
|
||||
command: string;
|
||||
/** Inline adapter code (simple tasks) */
|
||||
adapter?: string;
|
||||
/** Path to adapter file relative to autoresearch/ dir (complex tasks — avoids JSON escape issues) */
|
||||
adapterFile?: string;
|
||||
judge: JudgeCriteria;
|
||||
set?: 'test';
|
||||
note?: string;
|
||||
}
|
||||
|
||||
type JudgeCriteria =
|
||||
| { type: 'contains'; value: string }
|
||||
| { type: 'arrayMinLength'; minLength: number }
|
||||
| { type: 'nonEmpty' }
|
||||
| { type: 'matchesPattern'; pattern: string };
|
||||
|
||||
interface TaskResult {
|
||||
name: string;
|
||||
phase: 'init' | 'write' | 'verify' | 'judge';
|
||||
passed: boolean;
|
||||
duration: number;
|
||||
error?: string;
|
||||
set: 'train' | 'test';
|
||||
}
|
||||
|
||||
function judge(criteria: JudgeCriteria, output: string): boolean {
|
||||
try {
|
||||
switch (criteria.type) {
|
||||
case 'contains':
|
||||
return output.toLowerCase().includes(criteria.value.toLowerCase());
|
||||
case 'arrayMinLength': {
|
||||
// operate verify outputs table text; try JSON parse first, then count non-empty lines
|
||||
try {
|
||||
const arr = JSON.parse(output);
|
||||
if (Array.isArray(arr)) return arr.length >= criteria.minLength;
|
||||
} catch { /* not JSON — try line counting */ }
|
||||
// Table output: count data rows (skip header, separator, empty lines)
|
||||
const lines = output.split('\n').filter(l => l.trim() && !l.startsWith('─') && !l.startsWith('┌') && !l.startsWith('└') && !l.startsWith('├'));
|
||||
// Subtract header row
|
||||
const dataLines = lines.length > 1 ? lines.length - 1 : 0;
|
||||
return dataLines >= criteria.minLength;
|
||||
}
|
||||
case 'nonEmpty':
|
||||
return output.trim().length > 0 && output.trim() !== 'null' && output.trim() !== 'undefined';
|
||||
case 'matchesPattern':
|
||||
return new RegExp(criteria.pattern).test(output);
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
const PROJECT_ROOT = join(__dirname, '..');
|
||||
|
||||
/** Run a command, using local dist/main.js instead of global opencli for consistency */
|
||||
function runCommand(cmd: string, timeout = 30000): string {
|
||||
// Use local build so tests always run against the current source
|
||||
const localCmd = cmd.replace(/^opencli /, `node dist/main.js `);
|
||||
try {
|
||||
return execSync(localCmd, {
|
||||
cwd: PROJECT_ROOT,
|
||||
timeout,
|
||||
encoding: 'utf-8',
|
||||
env: process.env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
}).trim();
|
||||
} catch (err: any) {
|
||||
return err.stdout?.trim() || err.stderr?.trim() || '';
|
||||
}
|
||||
}
|
||||
|
||||
function cleanupAdapter(site: string, command: string): void {
|
||||
const siteDir = join(USER_CLIS_DIR, site);
|
||||
const filePath = join(siteDir, `${command}.ts`);
|
||||
try {
|
||||
if (existsSync(filePath)) rmSync(filePath);
|
||||
// Remove site dir if empty
|
||||
if (existsSync(siteDir)) {
|
||||
const remaining = readdirSync(siteDir);
|
||||
if (remaining.length === 0) rmSync(siteDir, { recursive: true });
|
||||
}
|
||||
} catch { /* best effort */ }
|
||||
}
|
||||
|
||||
function runTask(task: SaveTask): TaskResult {
|
||||
const start = Date.now();
|
||||
const { site, command } = task;
|
||||
const adapterDir = join(USER_CLIS_DIR, site);
|
||||
const adapterPath = join(adapterDir, `${command}.ts`);
|
||||
|
||||
// Cleanup any leftover from previous runs
|
||||
cleanupAdapter(site, command);
|
||||
|
||||
try {
|
||||
// Phase 1: init — create scaffold
|
||||
const initOutput = runCommand(`opencli operate init ${site}/${command}`);
|
||||
if (!existsSync(adapterPath)) {
|
||||
return {
|
||||
name: task.name, phase: 'init', passed: false,
|
||||
duration: Date.now() - start,
|
||||
error: `init failed: file not created. Output: ${initOutput.slice(0, 100)}`,
|
||||
set: task.set === 'test' ? 'test' : 'train',
|
||||
};
|
||||
}
|
||||
|
||||
// Phase 2: write — overwrite scaffold with real adapter code
|
||||
if (task.adapterFile) {
|
||||
// Read from file (complex adapters — avoids JSON string escape issues)
|
||||
const srcPath = join(__dirname, task.adapterFile);
|
||||
const code = readFileSync(srcPath, 'utf-8');
|
||||
writeFileSync(adapterPath, code, 'utf-8');
|
||||
} else if (task.adapter) {
|
||||
writeFileSync(adapterPath, task.adapter, 'utf-8');
|
||||
}
|
||||
|
||||
// Phase 3: verify — run the adapter via operate verify
|
||||
const verifyOutput = runCommand(
|
||||
`opencli operate verify ${site}/${command}`,
|
||||
45000, // longer timeout for network calls
|
||||
);
|
||||
|
||||
if (verifyOutput.includes('✗ Adapter failed')) {
|
||||
return {
|
||||
name: task.name, phase: 'verify', passed: false,
|
||||
duration: Date.now() - start,
|
||||
error: `verify failed: ${verifyOutput.slice(0, 200)}`,
|
||||
set: task.set === 'test' ? 'test' : 'train',
|
||||
};
|
||||
}
|
||||
|
||||
// Phase 4: judge — check output quality
|
||||
const passed = judge(task.judge, verifyOutput);
|
||||
|
||||
return {
|
||||
name: task.name,
|
||||
phase: 'judge',
|
||||
passed,
|
||||
duration: Date.now() - start,
|
||||
error: passed ? undefined : `Judge failed on output: ${verifyOutput.slice(0, 150)}`,
|
||||
set: task.set === 'test' ? 'test' : 'train',
|
||||
};
|
||||
} catch (err: any) {
|
||||
return {
|
||||
name: task.name, phase: 'verify', passed: false,
|
||||
duration: Date.now() - start,
|
||||
error: err.message?.slice(0, 150),
|
||||
set: task.set === 'test' ? 'test' : 'train',
|
||||
};
|
||||
} finally {
|
||||
// Always cleanup test adapters
|
||||
cleanupAdapter(site, command);
|
||||
}
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const singleTask = args.includes('--task') ? args[args.indexOf('--task') + 1] : null;
|
||||
|
||||
const allTasks: SaveTask[] = JSON.parse(readFileSync(TASKS_FILE, 'utf-8'));
|
||||
const tasks = singleTask ? allTasks.filter(t => t.name === singleTask) : allTasks;
|
||||
|
||||
if (tasks.length === 0) {
|
||||
console.error(`Task "${singleTask}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`\n🧪 Layer 4: Save as CLI — ${tasks.length} tasks\n`);
|
||||
|
||||
const results: TaskResult[] = [];
|
||||
|
||||
for (let i = 0; i < tasks.length; i++) {
|
||||
const task = tasks[i];
|
||||
process.stdout.write(` [${i + 1}/${tasks.length}] ${task.name}...`);
|
||||
|
||||
const result = runTask(task);
|
||||
results.push(result);
|
||||
|
||||
const icon = result.passed ? '✓' : '✗';
|
||||
const phase = result.passed ? '' : ` (${result.phase})`;
|
||||
console.log(` ${icon}${phase} (${(result.duration / 1000).toFixed(1)}s)`);
|
||||
}
|
||||
|
||||
// Summary
|
||||
const trainResults = results.filter(r => r.set === 'train');
|
||||
const testResults = results.filter(r => r.set === 'test');
|
||||
const totalPassed = results.filter(r => r.passed).length;
|
||||
const trainPassed = trainResults.filter(r => r.passed).length;
|
||||
const testPassed = testResults.filter(r => r.passed).length;
|
||||
const totalDuration = results.reduce((s, r) => s + r.duration, 0);
|
||||
|
||||
console.log(`\n${'─'.repeat(50)}`);
|
||||
console.log(` Score: ${totalPassed}/${results.length} (train: ${trainPassed}/${trainResults.length}, test: ${testPassed}/${testResults.length})`);
|
||||
console.log(` Time: ${Math.round(totalDuration / 1000)}s`);
|
||||
|
||||
const failures = results.filter(r => !r.passed);
|
||||
if (failures.length > 0) {
|
||||
console.log(`\n Failures:`);
|
||||
for (const f of failures) {
|
||||
console.log(` ✗ ${f.name} [${f.phase}]: ${f.error ?? 'unknown'}`);
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Save result
|
||||
mkdirSync(RESULTS_DIR, { recursive: true });
|
||||
const existing = readdirSync(RESULTS_DIR).filter(f => f.startsWith('save-')).length;
|
||||
const roundNum = String(existing + 1).padStart(3, '0');
|
||||
const resultPath = join(RESULTS_DIR, `save-${roundNum}.json`);
|
||||
writeFileSync(resultPath, JSON.stringify({
|
||||
timestamp: new Date().toISOString(),
|
||||
score: `${totalPassed}/${results.length}`,
|
||||
trainScore: `${trainPassed}/${trainResults.length}`,
|
||||
testScore: `${testPassed}/${testResults.length}`,
|
||||
duration: `${Math.round(totalDuration / 1000)}s`,
|
||||
tasks: results,
|
||||
}, null, 2), 'utf-8');
|
||||
console.log(` Results saved to: ${resultPath}`);
|
||||
console.log(`\nSCORE=${totalPassed}/${results.length}`);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -3,6 +3,7 @@ export { skillQuality } from './skill-quality.js';
|
||||
export { v2exReliability } from './v2ex-reliability.js';
|
||||
export { zhihuReliability } from './zhihu-reliability.js';
|
||||
export { combinedReliability } from './combined-reliability.js';
|
||||
export { saveReliability } from './save-reliability.js';
|
||||
|
||||
import type { AutoResearchConfig } from '../config.js';
|
||||
import { operateReliability } from './operate-reliability.js';
|
||||
@@ -10,6 +11,7 @@ import { skillQuality } from './skill-quality.js';
|
||||
import { v2exReliability } from './v2ex-reliability.js';
|
||||
import { zhihuReliability } from './zhihu-reliability.js';
|
||||
import { combinedReliability } from './combined-reliability.js';
|
||||
import { saveReliability } from './save-reliability.js';
|
||||
|
||||
export const PRESETS: Record<string, AutoResearchConfig> = {
|
||||
'operate-reliability': operateReliability,
|
||||
@@ -17,4 +19,5 @@ export const PRESETS: Record<string, AutoResearchConfig> = {
|
||||
'v2ex-reliability': v2exReliability,
|
||||
'zhihu-reliability': zhihuReliability,
|
||||
'combined': combinedReliability,
|
||||
'save-reliability': saveReliability,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* Preset: Save as CLI Reliability
|
||||
*
|
||||
* Optimizes the "Save as CLI" pipeline: operate init → write adapter → run.
|
||||
* Covers PUBLIC (no auth) and COOKIE (browser session) strategies.
|
||||
* Metric: number of passing save-tasks.
|
||||
*/
|
||||
|
||||
import type { AutoResearchConfig } from '../config.js';
|
||||
|
||||
export const saveReliability: AutoResearchConfig = {
|
||||
goal: 'Increase "Save as CLI" pipeline pass rate to 100%. The flow is: operate init creates a scaffold, user writes adapter code, opencli discovers and runs it. Covers both PUBLIC (fetch API) and COOKIE (browser session) strategies. Focus on: init template correctness, user CLI discovery, adapter loading, verify command robustness, and browser session handling.',
|
||||
scope: [
|
||||
'src/cli.ts',
|
||||
'src/discovery.ts',
|
||||
'src/registry.ts',
|
||||
'skills/opencli-operate/SKILL.md',
|
||||
'autoresearch/save-tasks.json',
|
||||
'autoresearch/save-adapters/*.ts',
|
||||
],
|
||||
metric: 'pass_count',
|
||||
direction: 'higher',
|
||||
verify: 'npx tsx autoresearch/eval-save.ts 2>&1 | tail -1',
|
||||
guard: 'npm run build',
|
||||
minDelta: 1,
|
||||
};
|
||||
@@ -0,0 +1,345 @@
|
||||
[
|
||||
{
|
||||
"name": "twitter-fill-compose",
|
||||
"platform": "twitter",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to tweet composer, fill in content (no publish)",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/compose/tweet",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]') ? 'composer-ready' : 'not-found'\"",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] OpenCLI publish eval - fill only test ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "3-step: open compose → paste text via ClipboardEvent → verify text in composer"
|
||||
},
|
||||
{
|
||||
"name": "twitter-post-and-delete",
|
||||
"platform": "twitter",
|
||||
"type": "publish",
|
||||
"description": "Post a tweet, verify success, then delete it",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/compose/tweet",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] OpenCLI publish eval ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const btn = document.querySelector('[data-testid=\\\"tweetButton\\\"]') || document.querySelector('[data-testid=\\\"tweetButtonInline\\\"]'); if (btn && !btn.disabled) { btn.click(); return 'clicked'; } return 'btn-not-ready'; })()\"",
|
||||
"opencli operate wait time 4",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"toast\\\"]')?.textContent || document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "post|sent|Your post|X"
|
||||
},
|
||||
"cleanup": [
|
||||
"opencli operate open https://x.com/home",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const tweets = document.querySelectorAll('[data-testid=\\\"tweet\\\"]'); for (const t of tweets) { if (t.textContent?.includes('[AutoTest]')) { const more = t.querySelector('[data-testid=\\\"caret\\\"]'); if (more) { more.click(); return 'found-menu'; } } } return 'no-autotest-tweet'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const items = document.querySelectorAll('[role=\\\"menuitem\\\"]'); for (const item of items) { if (item.textContent?.includes('Delete')) { item.click(); return 'clicked-delete'; } } return 'no-delete-option'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const confirm = document.querySelector('[data-testid=\\\"confirmationSheetConfirm\\\"]'); if (confirm) { confirm.click(); return 'deleted'; } return 'no-confirm'; })()\""
|
||||
],
|
||||
"note": "6-step chain: open compose → paste text → click post → wait → verify toast → cleanup: find tweet → menu → delete → confirm"
|
||||
},
|
||||
{
|
||||
"name": "twitter-read-hn-then-post",
|
||||
"platform": "twitter",
|
||||
"type": "publish",
|
||||
"description": "Read HN top story title, compose a tweet about it, post, then delete",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"document.querySelector('.titleline a')?.textContent?.trim() || 'no-title'\"",
|
||||
"opencli operate open https://x.com/compose/tweet",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const title = document.title || 'HN Story'; const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Interesting from HN: ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const btn = document.querySelector('[data-testid=\\\"tweetButton\\\"]') || document.querySelector('[data-testid=\\\"tweetButtonInline\\\"]'); if (btn && !btn.disabled) { btn.click(); return 'clicked'; } return 'btn-not-ready'; })()\"",
|
||||
"opencli operate wait time 4",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"toast\\\"]')?.textContent || document.title\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "matchesPattern",
|
||||
"pattern": "post|sent|Your post|X"
|
||||
},
|
||||
"cleanup": [
|
||||
"opencli operate open https://x.com/home",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const tweets = document.querySelectorAll('[data-testid=\\\"tweet\\\"]'); for (const t of tweets) { if (t.textContent?.includes('[AutoTest]')) { const more = t.querySelector('[data-testid=\\\"caret\\\"]'); if (more) { more.click(); return 'found-menu'; } } } return 'no-autotest-tweet'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const items = document.querySelectorAll('[role=\\\"menuitem\\\"]'); for (const item of items) { if (item.textContent?.includes('Delete')) { item.click(); return 'clicked-delete'; } } return 'no-delete-option'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const confirm = document.querySelector('[data-testid=\\\"confirmationSheetConfirm\\\"]'); if (confirm) { confirm.click(); return 'deleted'; } return 'no-confirm'; })()\""
|
||||
],
|
||||
"note": "9-step cross-site chain: read HN title → navigate to twitter compose → paste content → post → verify → cleanup delete"
|
||||
},
|
||||
{
|
||||
"name": "twitter-reply-to-own-tweet",
|
||||
"platform": "twitter",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to own profile, find latest tweet, open reply box, fill reply text",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/home",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const tweet = document.querySelector('[data-testid=\\\"tweet\\\"]'); if (!tweet) return 'no-tweet'; const reply = tweet.querySelector('[data-testid=\\\"reply\\\"]'); if (reply) { reply.click(); return 'reply-clicked'; } return 'no-reply-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Reply test ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "5-step: home → find first tweet → click reply → fill reply text → verify content"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-fill-answer",
|
||||
"platform": "zhihu",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to a popular question, open answer editor, fill in answer content (no publish)",
|
||||
"steps": [
|
||||
"opencli operate open https://www.zhihu.com/question/19550225",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const btn = document.querySelector('[data-zop-retarget=\\\"answer\\\"]') || Array.from(document.querySelectorAll('button')).find(b => b.textContent?.includes('写回答')); if (btn) { btn.click(); return 'editor-opened'; } return 'no-answer-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] 这是一个 OpenCLI 发文测试,时间戳: ' + Date.now() + '</p><p>这段内容用于验证 operate 命令链的完整性。</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "5-step: navigate to question → click '写回答' → find editor → fill rich content (title + body) → verify"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-fill-article",
|
||||
"platform": "zhihu",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to zhihu article editor (zhuanlan), fill title + body (no publish)",
|
||||
"steps": [
|
||||
"opencli operate open https://zhuanlan.zhihu.com/write",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const ta = document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder]'); if (!ta) return 'no-title-input'; ta.focus(); var nativeSetter = Object.getOwnPropertyDescriptor(window.HTMLTextAreaElement.prototype, 'value').set; nativeSetter.call(ta, '[AutoTest] OpenCLI 发文能力验证 ' + Date.now()); ta.dispatchEvent(new Event('input', { bubbles: true })); return 'title-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('[contenteditable=true]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>这是 OpenCLI autoresearch 发文测试集的一部分。</p><p>测试链路:导航 → 填写标题 → 填写正文 → 验证内容。</p><p>时间戳: ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'body-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const title = (document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder]'))?.value || ''; const body = document.querySelector('[contenteditable=true]')?.textContent || ''; return JSON.stringify({ title: title.slice(0, 50), body: body.slice(0, 50) }); })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "5-step: navigate to zhuanlan editor → fill title textarea → fill rich text body → verify both title and body content"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-read-hn-fill-answer",
|
||||
"platform": "zhihu",
|
||||
"type": "fill-only",
|
||||
"description": "Read HN top story, then navigate to zhihu question and fill an answer about it",
|
||||
"steps": [
|
||||
"opencli operate open https://news.ycombinator.com",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const a = document.querySelector('.titleline a'); return a ? a.textContent?.trim() : 'no-title'; })()\"",
|
||||
"opencli operate open https://www.zhihu.com/question/19550225",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const btn = Array.from(document.querySelectorAll('button')).find(b => b.textContent?.includes('写回答')) || document.querySelector('[data-zop-retarget=\\\"answer\\\"]'); if (btn) { btn.click(); return 'editor-opened'; } return 'no-answer-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=true]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] 分享一个来自 Hacker News 的有趣内容</p><p>这是一个跨平台内容搬运测试,时间戳: ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(document.querySelector('.ql-editor') || document.querySelector('[contenteditable=true]'))?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "8-step cross-site chain: read HN title → navigate zhihu question → click 写回答 → fill answer with HN content → verify"
|
||||
},
|
||||
{
|
||||
"name": "twitter-thread-compose",
|
||||
"platform": "twitter",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to compose, type first tweet, add thread tweet, type second tweet, verify both",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/compose/tweet",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]') ? 'composer-ready' : 'not-found'\"",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Thread tweet 1 - ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'first-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const addBtn = document.querySelector('[data-testid=\\\"addButton\\\"]') || document.querySelector('[aria-label=\\\"Add post\\\"]'); if (addBtn) { addBtn.click(); return 'thread-added'; } return 'no-add-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const boxes = document.querySelectorAll('[data-testid=\\\"tweetTextarea_0\\\"]'); const box = boxes[boxes.length - 1]; if (!box) return 'no-second-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Thread tweet 2 - continuation'); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'second-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const boxes = document.querySelectorAll('[data-testid=\\\"tweetTextarea_0\\\"]'); const t1 = boxes[0]?.textContent || ''; const t2 = boxes[boxes.length - 1]?.textContent || ''; return JSON.stringify({ tweet1: t1, tweet2: t2 }); })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Thread tweet 2"
|
||||
},
|
||||
"note": "10-step thread compose: open composer → fill tweet 1 → click add thread → fill tweet 2 → verify both tweets present"
|
||||
},
|
||||
{
|
||||
"name": "twitter-quote-retweet-fill",
|
||||
"platform": "twitter",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to home, find first tweet, open retweet menu, select Quote, fill quote text, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/home",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const tweet = document.querySelector('[data-testid=\\\"tweet\\\"]'); if (!tweet) return 'no-tweet'; const retweet = tweet.querySelector('[data-testid=\\\"retweet\\\"]'); if (retweet) { retweet.click(); return 'retweet-menu-opened'; } return 'no-retweet-btn'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const items = document.querySelectorAll('[role=\\\"menuitem\\\"]'); for (const item of items) { if (item.textContent?.includes('Quote') || item.textContent?.includes('引用')) { item.click(); return 'quote-selected'; } } return 'no-quote-option'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Quote retweet test ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'quote-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Quote retweet test"
|
||||
},
|
||||
"note": "8-step quote retweet: home → find tweet → click retweet → select Quote → fill quote text → verify"
|
||||
},
|
||||
{
|
||||
"name": "twitter-search-then-reply-fill",
|
||||
"platform": "twitter",
|
||||
"type": "fill-only",
|
||||
"description": "Search 'opencli' on twitter, find first result, click reply, fill reply text, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/search?q=opencli&src=typed_query&f=live",
|
||||
"opencli operate wait time 4",
|
||||
"opencli operate eval \"(() => { const tweets = document.querySelectorAll('[data-testid=\\\"tweet\\\"]'); if (tweets.length === 0) return 'no-results'; return 'found-' + tweets.length + '-results'; })()\"",
|
||||
"opencli operate eval \"(() => { const tweet = document.querySelector('[data-testid=\\\"tweet\\\"]'); if (!tweet) return 'no-tweet'; const reply = tweet.querySelector('[data-testid=\\\"reply\\\"]'); if (reply) { reply.click(); return 'reply-clicked'; } return 'no-reply-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Reply from search result ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'reply-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Reply from search"
|
||||
},
|
||||
"note": "8-step search-then-reply: navigate to search URL → verify results → click reply on first → fill reply → verify"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-search-then-fill-answer",
|
||||
"platform": "zhihu",
|
||||
"type": "fill-only",
|
||||
"description": "Search 'AI agent' on zhihu, click first question result, click 写回答, fill answer, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://www.zhihu.com/search?type=content&q=AI%20agent",
|
||||
"opencli operate wait time 4",
|
||||
"opencli operate eval \"(() => { const links = document.querySelectorAll('a[href*=\\\"/question/\\\"]'); if (links.length === 0) return 'no-question-links'; const link = links[0]; const href = link.getAttribute('href'); return 'found: ' + href; })()\"",
|
||||
"opencli operate eval \"(() => { const links = document.querySelectorAll('a[href*=\\\"/question/\\\"]'); if (links.length === 0) return 'no-links'; const link = links[0]; const href = link.getAttribute('href'); const match = href.match(/\\\\/question\\\\/(\\\\d+)/); if (match) { window.location.href = 'https://www.zhihu.com/question/' + match[1]; return 'navigating-to-question'; } link.click(); return 'clicked-link'; })()\"",
|
||||
"opencli operate wait time 4",
|
||||
"opencli operate eval \"(() => { const btn = document.querySelector('[data-zop-retarget=\\\"answer\\\"]') || Array.from(document.querySelectorAll('button')).find(b => b.textContent?.includes('写回答')) || Array.from(document.querySelectorAll('a')).find(a => a.textContent?.includes('写回答')); if (btn) { btn.click(); return 'editor-opened'; } return 'no-answer-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] AI agent 搜索后回答测试 ' + Date.now() + '</p><p>这是通过搜索 → 进入问题 → 填写回答的完整链路测试。</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "9-step search-then-answer: search zhihu → find question link → navigate → click 写回答 → fill answer → verify"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-read-question-fill-comment",
|
||||
"platform": "zhihu",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to question page, scroll to first answer, click comment, fill comment text, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://www.zhihu.com/question/19550225",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const answer = document.querySelector('[data-testid=\\\"answer\\\"]') || document.querySelector('.AnswerItem') || document.querySelector('.List-item'); if (answer) { answer.scrollIntoView({ behavior: 'smooth', block: 'center' }); return 'answer-scrolled'; } return 'no-answer'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const commentBtns = document.querySelectorAll('button'); for (const btn of commentBtns) { if (btn.textContent?.match(/评论|条评论|comment/i)) { btn.click(); return 'comment-opened: ' + btn.textContent.trim(); } } const commentIcons = document.querySelectorAll('[data-testid=\\\"comment\\\"]') || []; for (const icon of commentIcons) { icon.click(); return 'comment-icon-clicked'; } return 'no-comment-btn'; })()\"",
|
||||
"opencli operate wait time 2",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.CommentEditor textarea') || document.querySelector('[placeholder*=\\\"评论\\\"]') || document.querySelector('[placeholder*=\\\"comment\\\"]') || document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-comment-editor'; editor.focus(); if (editor.tagName === 'TEXTAREA' || editor.tagName === 'INPUT') { editor.value = '[AutoTest] 评论测试 ' + Date.now(); editor.dispatchEvent(new Event('input', { bubbles: true })); } else { editor.innerHTML = '<p>[AutoTest] 评论测试 ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); } return 'comment-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.CommentEditor textarea') || document.querySelector('[placeholder*=\\\"评论\\\"]') || document.querySelector('[placeholder*=\\\"comment\\\"]') || document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; return editor.value || editor.textContent || ''; })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "8-step comment chain: navigate question → scroll to answer → click comment → fill comment text → verify"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-article-with-formatting",
|
||||
"platform": "zhihu",
|
||||
"type": "fill-only",
|
||||
"description": "Navigate to zhuanlan editor, fill title, fill body with multiple paragraphs and bold text, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://zhuanlan.zhihu.com/write",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const titleInput = document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'); if (!titleInput) return 'no-title-input'; titleInput.focus(); titleInput.value = '[AutoTest] 格式化文章测试 ' + Date.now(); titleInput.dispatchEvent(new Event('input', { bubbles: true })); return 'title-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>这是第一段:OpenCLI 格式化发文测试。</p><p><strong>[AutoTest-Bold] 这是加粗的第二段,用于验证富文本格式。</strong></p><p>这是第三段,包含普通文本内容,时间戳: ' + Date.now() + '。</p><p>这是第四段,测试多段落填充能力。</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'body-filled-with-formatting'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; const hasBold = editor.querySelector('strong') || editor.querySelector('b'); const paragraphs = editor.querySelectorAll('p'); return JSON.stringify({ paragraphCount: paragraphs.length, hasBold: !!hasBold, preview: editor.textContent?.slice(0, 80) }); })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const title = (document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'))?.value || ''; const body = (document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''; return JSON.stringify({ title: title.slice(0, 60), bodyHasBold: body.includes('AutoTest-Bold'), bodyLength: body.length }); })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "AutoTest"
|
||||
},
|
||||
"note": "8-step formatted article: navigate editor → fill title → fill body with <strong> bold + 4 paragraphs → verify formatting + content"
|
||||
},
|
||||
{
|
||||
"name": "cross-zhihu-to-twitter",
|
||||
"platform": "cross",
|
||||
"type": "fill-only",
|
||||
"description": "Read zhihu hot topic title, navigate to twitter compose, fill tweet with zhihu content, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://www.zhihu.com/hot",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const hotItem = document.querySelector('.HotItem-content a') || document.querySelector('.HotList-item a') || document.querySelector('[data-testid=\\\"hot-item\\\"] a') || document.querySelector('.HotItem a'); if (hotItem) return hotItem.textContent?.trim()?.slice(0, 60) || 'no-text'; const titles = document.querySelectorAll('h2'); for (const t of titles) { if (t.textContent?.trim().length > 5) return t.textContent.trim().slice(0, 60); } return 'no-hot-topic'; })()\"",
|
||||
"opencli operate state save zhihu_hot_title",
|
||||
"opencli operate open https://x.com/compose/tweet",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]') ? 'composer-ready' : 'not-found'\"",
|
||||
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Zhihu热榜话题搬运: 知乎上正在热议的话题 - ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'tweet-filled-with-zhihu'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Zhihu热榜话题搬运"
|
||||
},
|
||||
"note": "10-step cross-platform: read zhihu hot → save state → navigate twitter compose → fill tweet with zhihu content → verify"
|
||||
},
|
||||
{
|
||||
"name": "cross-twitter-to-zhihu",
|
||||
"platform": "cross",
|
||||
"type": "fill-only",
|
||||
"description": "Read twitter trending/explore topic, navigate to zhihu zhuanlan editor, fill title and body, verify",
|
||||
"steps": [
|
||||
"opencli operate open https://x.com/explore/tabs/trending",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const trends = document.querySelectorAll('[data-testid=\\\"trend\\\"]'); if (trends.length > 0) { const first = trends[0]; return first.textContent?.trim()?.slice(0, 80) || 'no-text'; } const spans = document.querySelectorAll('span'); for (const s of spans) { if (s.textContent?.startsWith('#') || s.textContent?.includes('Trending')) { return s.textContent.trim().slice(0, 80); } } return 'no-trending-topic'; })()\"",
|
||||
"opencli operate state save twitter_trending",
|
||||
"opencli operate open https://zhuanlan.zhihu.com/write",
|
||||
"opencli operate wait time 3",
|
||||
"opencli operate eval \"(() => { const titleInput = document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'); if (!titleInput) return 'no-title-input'; titleInput.focus(); titleInput.value = '[AutoTest] Twitter热点搬运: 来自推特的热门话题 ' + Date.now(); titleInput.dispatchEvent(new Event('input', { bubbles: true })); return 'title-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] 这篇文章搬运自 Twitter 热门话题。</p><p>Twitter 上正在讨论的热门话题为大家带来了新的视角和思考。</p><p>时间戳: ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'body-filled'; })()\"",
|
||||
"opencli operate wait time 1",
|
||||
"opencli operate eval \"(() => { const title = (document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'))?.value || ''; const body = (document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''; return JSON.stringify({ title: title.slice(0, 60), body: body.slice(0, 60) }); })()\""
|
||||
],
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "Twitter热点搬运"
|
||||
},
|
||||
"note": "10-step cross-platform reverse: read twitter trending → save state → navigate zhihu editor → fill title + body → verify"
|
||||
}
|
||||
]
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# Layer 4: Save as CLI — test the full save pipeline
|
||||
# Tests: operate init → write adapter → operate verify
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
echo "=== Layer 4: Save as CLI ==="
|
||||
echo "Testing: init → write → verify pipeline"
|
||||
echo ""
|
||||
|
||||
npx tsx autoresearch/eval-save.ts "$@"
|
||||
@@ -0,0 +1,64 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'test-xhs',
|
||||
name: 'explore-deep',
|
||||
description: '小红书探索页深度提取 + 去重 + 按互动排序',
|
||||
domain: 'www.xiaohongshu.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 15, help: 'Number of items' },
|
||||
],
|
||||
columns: ['rank', 'title', 'author', 'likes', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = kwargs.limit ?? 15;
|
||||
// Step 1: Navigate to explore page
|
||||
await page.goto('https://www.xiaohongshu.com/explore');
|
||||
// Step 2: Wait for initial content via MutationObserver
|
||||
await page.evaluate(`new Promise(function(resolve) {
|
||||
var check = function() { return document.querySelectorAll('section.note-item').length > 0; };
|
||||
if (check()) return resolve(true);
|
||||
var observer = new MutationObserver(function(m, obs) { if (check()) { obs.disconnect(); resolve(true); } });
|
||||
observer.observe(document.body, { childList: true, subtree: true });
|
||||
setTimeout(function() { observer.disconnect(); resolve(false); }, 8000);
|
||||
})`);
|
||||
// Step 3: Multi-round adaptive scroll (early stop when no new content)
|
||||
let prevCount = 0;
|
||||
for (let round = 0; round < 5; round++) {
|
||||
await page.evaluate('window.scrollTo(0, document.body.scrollHeight)');
|
||||
await page.wait(1.5);
|
||||
const count = await page.evaluate('document.querySelectorAll("section.note-item").length') as number;
|
||||
if (count >= limit * 2 || count === prevCount) break;
|
||||
prevCount = count;
|
||||
}
|
||||
// Step 4: Extract with noteId deduplication + parse likes as integers
|
||||
const result = await page.evaluate(`(function() {
|
||||
var seen = {};
|
||||
var items = [];
|
||||
document.querySelectorAll('section.note-item').forEach(function(el) {
|
||||
var linkEl = el.querySelector('a[href]');
|
||||
var href = linkEl ? linkEl.getAttribute('href') || '' : '';
|
||||
var m = href.match(/explore\\/([a-f0-9]+)/);
|
||||
var noteId = m ? m[1] : '';
|
||||
if (!noteId || seen[noteId]) return;
|
||||
seen[noteId] = true;
|
||||
var titleEl = el.querySelector('.title span') || el.querySelector('a.title');
|
||||
var authorEl = el.querySelector('.author-wrapper .name') || el.querySelector('.author .name');
|
||||
var likesEl = el.querySelector('.like-wrapper .count') || el.querySelector('.interact-container .count');
|
||||
var title = (titleEl ? titleEl.textContent || '' : '').trim();
|
||||
var author = (authorEl ? authorEl.textContent || '' : '').trim();
|
||||
var likesRaw = (likesEl ? likesEl.textContent || '0' : '0').trim();
|
||||
var likes = parseInt(likesRaw.replace(/[^0-9]/g, '')) || 0;
|
||||
items.push({ title: title, author: author, likes: likes, url: 'https://www.xiaohongshu.com/explore/' + noteId });
|
||||
});
|
||||
return items;
|
||||
})()`);
|
||||
// Step 5: Sort by likes descending
|
||||
const sorted = (result as any[] || []).sort((a: any, b: any) => b.likes - a.likes);
|
||||
// Step 6: Slice and format
|
||||
return sorted.slice(0, limit).map((item: any, i: number) => ({
|
||||
rank: i + 1, title: item.title, author: item.author, likes: String(item.likes), url: item.url,
|
||||
}));
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,61 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'test-xhs',
|
||||
name: 'note-comments',
|
||||
description: '小红书笔记详情 + 评论(多步合并输出)',
|
||||
domain: 'www.xiaohongshu.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'id', type: 'string', default: '6745a82f000000000800b6ed', positional: true, help: 'Note ID' },
|
||||
{ name: 'limit', type: 'int', default: 5, help: 'Max comments' },
|
||||
],
|
||||
columns: ['section', 'title', 'author', 'likes', 'text'],
|
||||
func: async (page, kwargs) => {
|
||||
const noteId = kwargs.id ?? '6745a82f000000000800b6ed';
|
||||
const commentLimit = kwargs.limit ?? 5;
|
||||
// Step 1: Navigate to note detail page
|
||||
await page.goto('https://www.xiaohongshu.com/explore/' + noteId);
|
||||
await page.wait(3);
|
||||
// Step 2: Extract note metadata (title, author, likes)
|
||||
const meta = await page.evaluate(`(function() {
|
||||
return {
|
||||
title: (document.querySelector('#detail-title') || document.querySelector('.title') || {}).textContent?.trim() || '',
|
||||
author: (document.querySelector('.author-container .username') || document.querySelector('.user-nickname') || {}).textContent?.trim() || '',
|
||||
likes: (document.querySelector('[data-type="like"] .count') || document.querySelector('.like-wrapper .count') || {}).textContent?.trim() || '0',
|
||||
};
|
||||
})()`) as any;
|
||||
// Step 3: Scroll the note container to trigger comment loading
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await page.evaluate(`(function() {
|
||||
var scroller = document.querySelector('.note-scroller') || document.querySelector('.container');
|
||||
if (scroller && scroller.scrollTo) { scroller.scrollTo(0, 99999); } else { window.scrollTo(0, document.body.scrollHeight); }
|
||||
})()`);
|
||||
await page.wait(1);
|
||||
}
|
||||
// Step 4: Extract comments from DOM
|
||||
const comments = await page.evaluate(`(function() {
|
||||
var results = [];
|
||||
var commentEls = document.querySelectorAll('.parent-comment, .comment-item-root');
|
||||
commentEls.forEach(function(el) {
|
||||
var item = el.querySelector('.comment-item') || el.querySelector('.comment-inner');
|
||||
if (!item) return;
|
||||
var authorEl = item.querySelector('.author-wrapper .name') || item.querySelector('.user-name');
|
||||
var textEl = item.querySelector('.content') || item.querySelector('.note-text');
|
||||
var likesEl = item.querySelector('.count');
|
||||
var author = (authorEl ? authorEl.textContent || '' : '').trim();
|
||||
var text = (textEl ? textEl.textContent || '' : '').replace(/\\s+/g, ' ').trim();
|
||||
var likes = (likesEl ? likesEl.textContent || '0' : '0').trim();
|
||||
if (text) results.push({ author: author, text: text.slice(0, 80), likes: likes });
|
||||
});
|
||||
return results;
|
||||
})()`) as any[];
|
||||
// Step 5: Merge note meta + comments into unified output
|
||||
const rows: any[] = [{ section: 'note', title: meta.title, author: meta.author, likes: meta.likes, text: '' }];
|
||||
for (const c of (comments || []).slice(0, commentLimit)) {
|
||||
rows.push({ section: 'comment', title: '', author: c.author, likes: c.likes, text: c.text });
|
||||
}
|
||||
return rows;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,62 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'test-xhs',
|
||||
name: 'search-full',
|
||||
description: '小红书搜索 + 滚动加载 + 去重',
|
||||
domain: 'www.xiaohongshu.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'query', type: 'string', default: '咖啡', positional: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 10, help: 'Number of results' },
|
||||
],
|
||||
columns: ['rank', 'title', 'author', 'likes', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const query = encodeURIComponent(kwargs.query ?? '咖啡');
|
||||
const limit = kwargs.limit ?? 10;
|
||||
// Step 1: Navigate to search page
|
||||
await page.goto('https://www.xiaohongshu.com/search_result?keyword=' + query + '&source=web_search_result_notes');
|
||||
// Step 2: Wait for async render via MutationObserver
|
||||
await page.evaluate(`new Promise(function(resolve) {
|
||||
var check = function() { return document.querySelectorAll('section.note-item').length > 0; };
|
||||
if (check()) return resolve(true);
|
||||
var observer = new MutationObserver(function(m, obs) { if (check()) { obs.disconnect(); resolve(true); } });
|
||||
observer.observe(document.body, { childList: true, subtree: true });
|
||||
setTimeout(function() { observer.disconnect(); resolve(false); }, 8000);
|
||||
})`);
|
||||
// Step 3: Scroll 3x to load more content
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await page.evaluate('window.scrollTo(0, document.body.scrollHeight)');
|
||||
await page.wait(1);
|
||||
}
|
||||
// Step 4: Extract from DOM with deduplication
|
||||
const result = await page.evaluate(`(function() {
|
||||
var seen = {};
|
||||
var items = [];
|
||||
document.querySelectorAll('section.note-item').forEach(function(el) {
|
||||
var linkEl = el.querySelector('a[href]');
|
||||
var href = linkEl ? linkEl.getAttribute('href') || '' : '';
|
||||
var m = href.match(/explore\\/([a-f0-9]+)/);
|
||||
var noteId = m ? m[1] : href;
|
||||
if (!noteId || seen[noteId]) return;
|
||||
seen[noteId] = true;
|
||||
var titleEl = el.querySelector('.title span') || el.querySelector('a.title');
|
||||
var authorEl = el.querySelector('.author-wrapper .name') || el.querySelector('.author .name');
|
||||
var likesEl = el.querySelector('.like-wrapper .count') || el.querySelector('.interact-container .count');
|
||||
if (titleEl) {
|
||||
items.push({
|
||||
title: (titleEl.textContent || '').trim(),
|
||||
author: (authorEl ? authorEl.textContent || '' : '').trim(),
|
||||
likes: (likesEl ? likesEl.textContent || '0' : '0').trim(),
|
||||
url: 'https://www.xiaohongshu.com' + href,
|
||||
});
|
||||
}
|
||||
});
|
||||
return items;
|
||||
})()`);
|
||||
return (result as any[]).slice(0, limit).map((item: any, i: number) => ({
|
||||
rank: i + 1, title: item.title, author: item.author, likes: item.likes, url: item.url,
|
||||
}));
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,52 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'test-zhihu',
|
||||
name: 'hot-detail',
|
||||
description: '知乎热榜 + 每个问题的第一个回答摘要',
|
||||
domain: 'www.zhihu.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'limit', type: 'int', default: 5, help: 'Number of items' },
|
||||
],
|
||||
columns: ['rank', 'title', 'heat', 'top_answer_author', 'top_answer_excerpt'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = kwargs.limit ?? 5;
|
||||
// Step 1: Navigate
|
||||
await page.goto('https://www.zhihu.com');
|
||||
await page.wait(2);
|
||||
// Step 2: Fetch hot list (handle 16+ digit IDs)
|
||||
const hotList = await page.evaluate(`(async () => {
|
||||
const res = await fetch('https://www.zhihu.com/api/v3/feed/topstory/hot-lists/total?limit=50', { credentials: 'include' });
|
||||
const text = await res.text();
|
||||
const d = JSON.parse(text.replace(/("id"\\s*:\\s*)(\\d{16,})/g, '$1"$2"'));
|
||||
return (d?.data || []).map(item => {
|
||||
const t = item.target || {};
|
||||
return { qid: String(t.id || ''), title: t.title || '', heat: item.detail_text || '' };
|
||||
});
|
||||
})()`) as any[];
|
||||
// Step 3: For each hot question, fetch its top answer
|
||||
const items = hotList.slice(0, limit);
|
||||
const enriched = [];
|
||||
for (const item of items) {
|
||||
if (!item.qid) { enriched.push({ ...item, top_answer_author: '', top_answer_excerpt: '' }); continue; }
|
||||
const answer = await page.evaluate(`(async () => {
|
||||
const strip = (html) => (html || '').replace(/<[^>]+>/g, '').trim();
|
||||
try {
|
||||
const res = await fetch('https://www.zhihu.com/api/v4/questions/${item.qid}/answers?limit=1&offset=0&sort_by=default&include=data[*].content,voteup_count,author', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
const a = d?.data?.[0];
|
||||
if (!a) return { author: '', excerpt: '' };
|
||||
return { author: a.author?.name || 'anonymous', excerpt: strip(a.content || '').slice(0, 120) };
|
||||
} catch { return { author: '', excerpt: '' }; }
|
||||
})()`) as any;
|
||||
enriched.push({ ...item, top_answer_author: answer.author, top_answer_excerpt: answer.excerpt });
|
||||
}
|
||||
// Step 4: Format output
|
||||
return enriched.map((item, i) => ({
|
||||
rank: i + 1, title: item.title, heat: item.heat,
|
||||
top_answer_author: item.top_answer_author, top_answer_excerpt: item.top_answer_excerpt,
|
||||
}));
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,57 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'test-zhihu',
|
||||
name: 'question-full',
|
||||
description: '知乎问题 + 回答 + 相关推荐(三层数据合并)',
|
||||
domain: 'www.zhihu.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'id', type: 'string', default: '19550225', positional: true, help: 'Question ID' },
|
||||
{ name: 'limit', type: 'int', default: 3, help: 'Number of answers' },
|
||||
],
|
||||
columns: ['section', 'title', 'author', 'votes', 'excerpt'],
|
||||
func: async (page, kwargs) => {
|
||||
const qid = kwargs.id ?? '19550225';
|
||||
const limit = kwargs.limit ?? 3;
|
||||
// Step 1: Navigate to question page
|
||||
await page.goto('https://www.zhihu.com/question/' + qid);
|
||||
await page.wait(2);
|
||||
// Step 2: Fetch question detail
|
||||
const question = await page.evaluate(`(async () => {
|
||||
try {
|
||||
const res = await fetch('https://www.zhihu.com/api/v4/questions/${qid}', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return { title: d.title || '', follower_count: d.follower_count || 0, answer_count: d.answer_count || 0 };
|
||||
} catch { return { title: '', follower_count: 0, answer_count: 0 }; }
|
||||
})()`) as any;
|
||||
// Step 3: Fetch top answers
|
||||
const answers = await page.evaluate(`(async () => {
|
||||
const strip = (html) => (html || '').replace(/<[^>]+>/g, '').trim();
|
||||
try {
|
||||
const res = await fetch('https://www.zhihu.com/api/v4/questions/${qid}/answers?limit=${limit}&offset=0&sort_by=default&include=data[*].content,voteup_count,author', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return (d?.data || []).map(a => ({ author: a.author?.name || 'anonymous', votes: a.voteup_count || 0, excerpt: strip(a.content || '').slice(0, 120) }));
|
||||
} catch { return []; }
|
||||
})()`) as any[];
|
||||
// Step 4: Fetch related questions
|
||||
const related = await page.evaluate(`(async () => {
|
||||
try {
|
||||
const res = await fetch('https://www.zhihu.com/api/v4/questions/${qid}/similar?limit=3', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return (d?.data || []).map(q => ({ title: q.title || '', answer_count: q.answer_count || 0 }));
|
||||
} catch { return []; }
|
||||
})()`) as any[];
|
||||
// Step 5: Merge three layers into unified output
|
||||
const rows: any[] = [];
|
||||
rows.push({ section: 'question', title: question.title, author: '', votes: question.follower_count, excerpt: question.answer_count + ' answers' });
|
||||
for (const a of answers) {
|
||||
rows.push({ section: 'answer', title: '', author: a.author, votes: a.votes, excerpt: a.excerpt });
|
||||
}
|
||||
for (const r of related) {
|
||||
rows.push({ section: 'related', title: r.title, author: '', votes: 0, excerpt: r.answer_count + ' answers' });
|
||||
}
|
||||
return rows;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
import { cli, Strategy } from '@jackwener/opencli/registry';
|
||||
|
||||
cli({
|
||||
site: 'test-zhihu',
|
||||
name: 'search-detail',
|
||||
description: '知乎搜索 + 每条结果的问题统计',
|
||||
domain: 'www.zhihu.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
browser: true,
|
||||
args: [
|
||||
{ name: 'query', type: 'string', default: 'AI', positional: true, help: 'Search query' },
|
||||
{ name: 'limit', type: 'int', default: 5, help: 'Number of results' },
|
||||
],
|
||||
columns: ['rank', 'title', 'type', 'author', 'votes', 'answer_count', 'follower_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const query = kwargs.query ?? 'AI';
|
||||
const limit = kwargs.limit ?? 5;
|
||||
// Step 1: Navigate
|
||||
await page.goto('https://www.zhihu.com');
|
||||
await page.wait(2);
|
||||
// Step 2: Search API — filter results by type, extract question IDs
|
||||
const searchResults = await page.evaluate(`(async () => {
|
||||
const strip = (html) => (html || '').replace(/<[^>]+>/g, '').trim();
|
||||
const res = await fetch('https://www.zhihu.com/api/v4/search_v3?q=' + encodeURIComponent('${query}') + '&t=general&offset=0&limit=20', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return (d?.data || []).filter(item => item.type === 'search_result').map(item => {
|
||||
const obj = item.object || {};
|
||||
const q = obj.question || {};
|
||||
const questionId = obj.type === 'answer' ? String(q.id || '') : obj.type === 'question' ? String(obj.id || '') : '';
|
||||
return { type: obj.type || '', title: strip(obj.title || q.name || ''), author: obj.author?.name || '', votes: obj.voteup_count || 0, questionId };
|
||||
});
|
||||
})()`) as any[];
|
||||
// Step 3: For each result, fetch question stats (answer_count, follower_count)
|
||||
const items = searchResults.slice(0, limit);
|
||||
const enriched = [];
|
||||
for (const item of items) {
|
||||
if (!item.questionId) { enriched.push({ ...item, answer_count: 0, follower_count: 0 }); continue; }
|
||||
const stats = await page.evaluate(`(async () => {
|
||||
try {
|
||||
const res = await fetch('https://www.zhihu.com/api/v4/questions/${item.questionId}', { credentials: 'include' });
|
||||
const d = await res.json();
|
||||
return { answer_count: d.answer_count || 0, follower_count: d.follower_count || 0 };
|
||||
} catch { return { answer_count: 0, follower_count: 0 }; }
|
||||
})()`) as any;
|
||||
enriched.push({ ...item, answer_count: stats.answer_count, follower_count: stats.follower_count });
|
||||
}
|
||||
// Step 4: Format output
|
||||
return enriched.map((item, i) => ({
|
||||
rank: i + 1, title: item.title, type: item.type, author: item.author,
|
||||
votes: item.votes, answer_count: item.answer_count, follower_count: item.follower_count,
|
||||
}));
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,281 @@
|
||||
[
|
||||
{
|
||||
"name": "httpbin-get",
|
||||
"site": "test-httpbin",
|
||||
"command": "get",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-httpbin',\n name: 'get',\n description: 'httpbin echo test',\n domain: 'httpbin.org',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [],\n columns: ['origin', 'url'],\n func: async () => {\n const res = await fetch('https://httpbin.org/get');\n const d = await res.json();\n return [{ origin: d.origin, url: d.url }];\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 1
|
||||
},
|
||||
"note": "Simplest possible: httpbin echo, single row"
|
||||
},
|
||||
{
|
||||
"name": "jsonplaceholder-posts",
|
||||
"site": "test-jsonplaceholder",
|
||||
"command": "posts",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'posts',\n description: 'JSONPlaceholder posts',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of posts' },\n ],\n columns: ['id', 'title'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/posts');\n const posts = await res.json();\n return posts.slice(0, limit).map((p: any) => ({ id: p.id, title: p.title }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "jsonplaceholder-users",
|
||||
"site": "test-jsonplaceholder",
|
||||
"command": "users",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'users',\n description: 'JSONPlaceholder users',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of users' },\n ],\n columns: ['id', 'name', 'email'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/users');\n const users = await res.json();\n return users.slice(0, limit).map((u: any) => ({ id: u.id, name: u.name, email: u.email }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hn-top",
|
||||
"site": "test-hn",
|
||||
"command": "top",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'top',\n description: 'HackerNews top stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/topstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hn-ask",
|
||||
"site": "test-hn",
|
||||
"command": "ask",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'ask',\n description: 'HackerNews Ask HN stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/askstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "wiki-summary",
|
||||
"site": "test-wiki",
|
||||
"command": "summary",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-wiki',\n name: 'summary',\n description: 'Wikipedia article summary',\n domain: 'en.wikipedia.org',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'title', type: 'string', default: 'JavaScript', positional: true, help: 'Article title' },\n ],\n columns: ['title', 'extract'],\n func: async (_page, kwargs) => {\n const title = encodeURIComponent(kwargs.title);\n const res = await fetch(`https://en.wikipedia.org/api/rest_v1/page/summary/${title}`);\n const d = await res.json();\n return [{ title: d.title, extract: d.extract?.slice(0, 200) }];\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "contains",
|
||||
"value": "programming language"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "lobsters-hot",
|
||||
"site": "test-lobsters",
|
||||
"command": "hot",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-lobsters',\n name: 'hot',\n description: 'Lobsters hottest stories',\n domain: 'lobste.rs',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['title', 'score', 'url'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://lobste.rs/hottest.json');\n const stories = await res.json();\n return stories.slice(0, limit).map((s: any) => ({\n title: s.title, score: s.score, url: s.short_id_url,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "devto-top",
|
||||
"site": "test-devto",
|
||||
"command": "top",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-devto',\n name: 'top',\n description: 'DEV.to top articles',\n domain: 'dev.to',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of articles' },\n ],\n columns: ['title', 'user', 'reactions'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://dev.to/api/articles?per_page=' + limit);\n const articles = await res.json();\n return articles.map((a: any) => ({\n title: a.title, user: a.user?.username, reactions: a.positive_reactions_count,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "zhihu-hot-with-top-answer",
|
||||
"site": "test-zhihu",
|
||||
"command": "hot-detail",
|
||||
"adapterFile": "save-adapters/zhihu-hot-detail.ts",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "6-step chain: navigate → fetch hot list API → parse big-int IDs → loop N items → fetch answer API per question → strip HTML → merge"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-search-with-question-stats",
|
||||
"site": "test-zhihu",
|
||||
"command": "search-detail",
|
||||
"adapterFile": "save-adapters/zhihu-search-detail.ts",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "7-step chain: navigate → search API → filter by type → extract question IDs → fetch question detail per result → merge stats → format"
|
||||
},
|
||||
{
|
||||
"name": "xhs-search-scroll-extract",
|
||||
"site": "test-xhs",
|
||||
"command": "search-full",
|
||||
"adapterFile": "save-adapters/xhs-search-full.ts",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "6-step chain: navigate → MutationObserver wait → scroll 3x → DOM extract with URL dedup → slice + format"
|
||||
},
|
||||
{
|
||||
"name": "xhs-note-with-comments",
|
||||
"site": "test-xhs",
|
||||
"command": "note-comments",
|
||||
"adapterFile": "save-adapters/xhs-note-comments.ts",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 1
|
||||
},
|
||||
"note": "7-step chain: navigate → wait → extract note meta → scroll container 3x → extract comments DOM → merge note+comments → unified output"
|
||||
},
|
||||
{
|
||||
"name": "zhihu-question-with-related",
|
||||
"site": "test-zhihu",
|
||||
"command": "question-full",
|
||||
"adapterFile": "save-adapters/zhihu-question-full.ts",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 2
|
||||
},
|
||||
"note": "8-step chain: navigate → wait → fetch question detail → fetch answers → strip HTML → fetch related questions → merge 3 layers → format"
|
||||
},
|
||||
{
|
||||
"name": "xhs-explore-scroll-dedupe",
|
||||
"site": "test-xhs",
|
||||
"command": "explore-deep",
|
||||
"adapterFile": "save-adapters/xhs-explore-deep.ts",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "8-step chain: navigate → MutationObserver wait → adaptive scroll → DOM extract with dedup → parse likes → sort desc → slice → format"
|
||||
},
|
||||
{
|
||||
"name": "hn-new",
|
||||
"site": "test-hn",
|
||||
"command": "new",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'new',\n description: 'HackerNews newest stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/newstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score ?? 0,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: HackerNews new stories using same Firebase API as hn-top/hn-ask"
|
||||
},
|
||||
{
|
||||
"name": "jsonplaceholder-todos",
|
||||
"site": "test-jsonplaceholder",
|
||||
"command": "todos",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'todos',\n description: 'JSONPlaceholder todos',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of todos' },\n ],\n columns: ['id', 'title', 'completed'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/todos');\n const todos = await res.json();\n return todos.slice(0, limit).map((t: any) => ({ id: t.id, title: t.title, completed: t.completed }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: JSONPlaceholder todos — same base domain as posts/users, different endpoint"
|
||||
},
|
||||
{
|
||||
"name": "hn-show",
|
||||
"site": "test-hn",
|
||||
"command": "show",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'show',\n description: 'HackerNews Show HN stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/showstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score ?? 0,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: HackerNews show stories using same Firebase API as hn-top/hn-ask/hn-new"
|
||||
},
|
||||
{
|
||||
"name": "jsonplaceholder-comments",
|
||||
"site": "test-jsonplaceholder",
|
||||
"command": "comments",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'comments',\n description: 'JSONPlaceholder comments',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of comments' },\n ],\n columns: ['id', 'name', 'email'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/comments');\n const comments = await res.json();\n return comments.slice(0, limit).map((c: any) => ({ id: c.id, name: c.name, email: c.email }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: JSONPlaceholder comments — same base domain as posts/users/todos, different endpoint"
|
||||
},
|
||||
{
|
||||
"name": "jsonplaceholder-albums",
|
||||
"site": "test-jsonplaceholder",
|
||||
"command": "albums",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'albums',\n description: 'JSONPlaceholder albums',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of albums' },\n ],\n columns: ['id', 'userId', 'title'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/albums');\n const albums = await res.json();\n return albums.slice(0, limit).map((a: any) => ({ id: a.id, userId: a.userId, title: a.title }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: JSONPlaceholder albums — same base domain as posts/users/todos/comments, different endpoint"
|
||||
},
|
||||
{
|
||||
"name": "jsonplaceholder-photos",
|
||||
"site": "test-jsonplaceholder",
|
||||
"command": "photos",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'photos',\n description: 'JSONPlaceholder photos',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of photos' },\n ],\n columns: ['id', 'albumId', 'title'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/photos');\n const photos = await res.json();\n return photos.slice(0, limit).map((p: any) => ({ id: p.id, albumId: p.albumId, title: p.title }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: JSONPlaceholder photos — same base domain as posts/users/todos/comments/albums, different endpoint"
|
||||
},
|
||||
{
|
||||
"name": "hn-best",
|
||||
"site": "test-hn",
|
||||
"command": "best",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'best',\n description: 'HackerNews best stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/beststories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score ?? 0,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: HackerNews best stories using same Firebase API as hn-top/hn-ask/hn-new/hn-show"
|
||||
},
|
||||
{
|
||||
"name": "hn-jobs",
|
||||
"site": "test-hn",
|
||||
"command": "jobs",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'jobs',\n description: 'HackerNews job stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of jobs' },\n ],\n columns: ['rank', 'title', 'url'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/jobstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, url: item.url ?? '',\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: HackerNews job listings using same Firebase API as other HN adapters"
|
||||
},
|
||||
{
|
||||
"name": "restcountries-list",
|
||||
"site": "test-restcountries",
|
||||
"command": "list",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-restcountries',\n name: 'list',\n description: 'REST Countries list',\n domain: 'restcountries.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of countries' },\n ],\n columns: ['name', 'capital', 'region'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://restcountries.com/v3.1/all?fields=name,capital,region');\n const countries = await res.json();\n return countries.slice(0, limit).map((c: any) => ({\n name: c.name?.common ?? '',\n capital: c.capital?.[0] ?? '',\n region: c.region ?? '',\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: REST Countries API — stable, no-auth, returns 250 countries with name/capital/region"
|
||||
},
|
||||
{
|
||||
"name": "nager-holidays",
|
||||
"site": "test-nager",
|
||||
"command": "holidays",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-nager',\n name: 'holidays',\n description: 'US public holidays for current year',\n domain: 'date.nager.at',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of holidays' },\n ],\n columns: ['date', 'name', 'type'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const year = new Date().getFullYear();\n const res = await fetch(`https://date.nager.at/api/v3/PublicHolidays/${year}/US`);\n const holidays = await res.json();\n return holidays.slice(0, limit).map((h: any) => ({\n date: h.date,\n name: h.name,\n type: (h.types || []).join(','),\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: Nager public holidays API — stable, no-auth, returns US federal holidays by year"
|
||||
},
|
||||
{
|
||||
"name": "catfact-list",
|
||||
"site": "test-catfact",
|
||||
"command": "facts",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-catfact',\n name: 'facts',\n description: 'Random cat facts',\n domain: 'catfact.ninja',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of facts' },\n ],\n columns: ['fact', 'length'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch(`https://catfact.ninja/facts?limit=${limit}`);\n const d = await res.json();\n return d.data.map((item: any) => ({\n fact: item.fact.slice(0, 100),\n length: item.length,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: catfact.ninja facts API — stable, no-auth, returns random cat facts"
|
||||
},
|
||||
{
|
||||
"name": "opentdb-trivia",
|
||||
"site": "test-opentdb",
|
||||
"command": "easy",
|
||||
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-opentdb',\n name: 'easy',\n description: 'Easy trivia questions from Open Trivia DB',\n domain: 'opentdb.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of questions' },\n ],\n columns: ['category', 'question', 'answer'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch(`https://opentdb.com/api.php?amount=${limit}&difficulty=easy&type=multiple`);\n const d = await res.json();\n return d.results.map((q: any) => ({\n category: q.category,\n question: q.question.replace(/"/g, '\"').replace(/'/g, \"'\").slice(0, 80),\n answer: q.correct_answer,\n }));\n },\n});\n",
|
||||
"judge": {
|
||||
"type": "arrayMinLength",
|
||||
"minLength": 3
|
||||
},
|
||||
"note": "PUBLIC strategy: Open Trivia DB API — stable, no-auth, returns trivia questions with correct answers"
|
||||
}
|
||||
]
|
||||
@@ -54,6 +54,7 @@ export default defineConfig({
|
||||
{ text: 'Bilibili', link: '/adapters/browser/bilibili' },
|
||||
{ text: 'Zhihu', link: '/adapters/browser/zhihu' },
|
||||
{ text: 'Xiaohongshu', link: '/adapters/browser/xiaohongshu' },
|
||||
{ text: 'Xiaoe', link: '/adapters/browser/xiaoe' },
|
||||
{ text: 'Weibo', link: '/adapters/browser/weibo' },
|
||||
{ text: 'YouTube', link: '/adapters/browser/youtube' },
|
||||
{ text: 'Xueqiu', link: '/adapters/browser/xueqiu' },
|
||||
@@ -73,6 +74,7 @@ export default defineConfig({
|
||||
{ text: 'Chaoxing', link: '/adapters/browser/chaoxing' },
|
||||
{ text: 'Grok', link: '/adapters/browser/grok' },
|
||||
{ text: 'Amazon', link: '/adapters/browser/amazon' },
|
||||
{ text: '1688', link: '/adapters/browser/1688' },
|
||||
{ text: 'Gemini', link: '/adapters/browser/gemini' },
|
||||
{ text: 'Yuanbao', link: '/adapters/browser/yuanbao' },
|
||||
{ text: 'NotebookLM', link: '/adapters/browser/notebooklm' },
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# 1688
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `1688.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli 1688 search "<query>" --limit <n>` | Search public product candidates with price, MOQ, seller link, and visible badges |
|
||||
| `opencli 1688 item <url-or-offer-id>` | Read a public product detail page with price tiers, MOQ, delivery text, and seller basics |
|
||||
| `opencli 1688 store <url-or-member-id>` | Read a public supplier/store page with company info, years on platform, categories, and visible service signals |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Search products
|
||||
opencli 1688 search "桌面置物架 宿舍 收纳" --limit 10
|
||||
|
||||
# JSON output
|
||||
opencli 1688 search "桌面置物架 宿舍 收纳" --limit 10 -f json
|
||||
|
||||
# Read an item by offer id
|
||||
opencli 1688 item 841141931191 -f json
|
||||
|
||||
# Read an item by URL
|
||||
opencli 1688 item https://detail.1688.com/offer/841141931191.html -f json
|
||||
|
||||
# Read a supplier store
|
||||
opencli 1688 store https://shop52908bfw19166.1688.com/ -f json
|
||||
|
||||
# Read a supplier by member id
|
||||
opencli 1688 store b2b-22154705262941f196 -f json
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and **logged into** `1688.com`
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
- This adapter only returns fields visible on public pages. It does not send inquiries, place orders, or access seller back office data.
|
||||
- Prefer stable identifiers such as `offer_id`, `member_id`, and `shop_id` for follow-up workflows.
|
||||
- `search --limit` defaults to `20` and is capped at `100`.
|
||||
- `search` deduplicates with key priority: `offer_id` first, then canonical `item_url`.
|
||||
- `item` can be more sensitive to the active browser target than `search` or `store`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- If `opencli 1688 item` reports `did not expose product context`, first make sure the open page is a real `detail.1688.com` item page.
|
||||
- If the browser target is too broad, retry with `OPENCLI_CDP_TARGET=detail.1688.com`.
|
||||
- If you hit a slider or verification page, refresh the real page in Chrome and retry.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Hupu (虎扑)
|
||||
|
||||
**Mode**: 🌐 Public / 🔐 Browser · **Domain**: `bbs.hupu.com`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli hupu hot` | Read Hupu hot threads |
|
||||
| `opencli hupu search <keyword>` | Search Hupu threads by keyword |
|
||||
| `opencli hupu detail <tid>` | Read one thread and optional hot replies |
|
||||
| `opencli hupu reply <tid> <text>` | Reply to a thread or quote one reply |
|
||||
| `opencli hupu like <tid> <pid>` | Like one reply |
|
||||
| `opencli hupu unlike <tid> <pid>` | Cancel like on one reply |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Hot threads
|
||||
opencli hupu hot --limit 5
|
||||
|
||||
# Search threads
|
||||
opencli hupu search 湖人 --limit 10
|
||||
|
||||
# Read one thread and include hot replies
|
||||
opencli hupu detail 638234927 --replies true
|
||||
|
||||
# Reply to the thread
|
||||
opencli hupu reply 638234927 "hello from opencli" --topic_id 502
|
||||
|
||||
# Quote one hot reply by pid
|
||||
opencli hupu reply 638234927 "replying to this comment" --topic_id 502 --quote_id 174908
|
||||
|
||||
# Like / unlike one reply
|
||||
opencli hupu like 638234927 174908 --fid 4860
|
||||
opencli hupu unlike 638234927 174908 --fid 4860
|
||||
|
||||
# JSON output
|
||||
opencli hupu detail 638234927 -f json
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `reply --topic_id` maps to Hupu's API `topicId`, for example `502` for Basketball News
|
||||
- `reply --quote_id` is the quoted reply `pid`
|
||||
- `like` / `unlike --fid` uses the forum ID from thread metadata
|
||||
- `detail --replies true` appends top hot replies to the content field
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and able to open `bbs.hupu.com`
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
- For `reply`, `like`, and `unlike`, a valid Hupu login session in Chrome is required
|
||||
@@ -9,6 +9,7 @@
|
||||
| `opencli sinafinance news` | 新浪财经 7×24 小时实时快讯 | 🌐 Public |
|
||||
| `opencli sinafinance rolling-news` | 新浪财经滚动新闻 | 🔐 Browser |
|
||||
| `opencli sinafinance stock` | 新浪财经行情(A股/港股/美股) | 🌐 Public |
|
||||
| `opencli sinafinance stock-rank` | 新浪财经热搜榜 | 🔐 Browser |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
@@ -56,6 +57,28 @@ opencli sinafinance stock 招商证券
|
||||
opencli sinafinance stock 贵州茅台 -f json
|
||||
```
|
||||
|
||||
### stock-rank - 热搜榜
|
||||
|
||||
```bash
|
||||
# Default A股热搜榜
|
||||
opencli sinafinance stock-rank
|
||||
|
||||
# 港股热搜榜
|
||||
opencli sinafinance stock-rank --market hk
|
||||
|
||||
# 美股热搜榜
|
||||
opencli sinafinance stock-rank --market us
|
||||
|
||||
# 外汇热搜榜
|
||||
opencli sinafinance stock-rank --market ft
|
||||
|
||||
# 期货热搜榜
|
||||
opencli sinafinance stock-rank --market wh
|
||||
|
||||
# JSON output
|
||||
opencli sinafinance stock-rank -f json
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
### news
|
||||
@@ -71,11 +94,17 @@ opencli sinafinance stock 贵州茅台 -f json
|
||||
|--------|-------------|
|
||||
| `--market` | Market: `cn`, `hk`, `us`, `auto` (default: auto). When `auto`, searches in cn, hk, us order |
|
||||
|
||||
### stock-rank
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--market` | Market: `cn` (A股, 默认), `ft` (期货), `us` (美股), `wh` (外汇), `hk` (港股) |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `news` & `stock`: No browser required — uses public API
|
||||
- `rolling-news`: Chrome running and **logged into** `finance.sina.com.cn`
|
||||
- For `rolling-news`: [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
- `rolling-news` & `stock-rank`: Chrome running and **logged into** `finance.sina.com.cn`
|
||||
- For `rolling-news` & `stock-rank`: [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -83,3 +112,4 @@ opencli sinafinance stock 贵州茅台 -f json
|
||||
- `stock` supports Chinese names, Chinese codes, and ticker symbols; auto-detects market
|
||||
- Market priority for auto-detection: cn (A股) → hk (港股) → us (美股)
|
||||
- US stock `High`/`Low` columns show 52-week range; A股/港股 show today's range
|
||||
- `stock-rank` scrapes the hot search list from the Sina Finance homepage; requires browser login
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Xiaoe (小鹅通)
|
||||
|
||||
**Mode**: 🔐 Browser · **Domain**: `study.xiaoe-tech.com` / `*.h5.xet.citv.cn`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `opencli xiaoe courses` | List purchased courses with course URLs and shop names |
|
||||
| `opencli xiaoe detail <url>` | Read course metadata such as title, price, student count, and shop |
|
||||
| `opencli xiaoe catalog <url>` | Read the full course outline for normal courses, columns, and big columns |
|
||||
| `opencli xiaoe play-url <url>` | Resolve the M3U8 playback URL for video lessons or live replays |
|
||||
| `opencli xiaoe content <url>` | Extract rich-text lesson or page content as plain text |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# List purchased courses
|
||||
opencli xiaoe courses --limit 10
|
||||
|
||||
# Read course metadata
|
||||
opencli xiaoe detail "https://appxxxx.h5.xet.citv.cn/p/course/ecourse/v_xxxxx"
|
||||
|
||||
# Read the course outline
|
||||
opencli xiaoe catalog "https://appxxxx.h5.xet.citv.cn/p/course/ecourse/v_xxxxx"
|
||||
|
||||
# Resolve a lesson M3U8 URL
|
||||
opencli xiaoe play-url "https://appxxxx.h5.xet.citv.cn/v1/course/video/v_xxxxx?product_id=p_xxxxx" -f json
|
||||
|
||||
# Extract page content
|
||||
opencli xiaoe content "https://appxxxx.h5.xet.citv.cn/v1/course/text/t_xxxxx"
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Chrome running and **logged into** the target Xiaoe shop
|
||||
- [Browser Bridge extension](/guide/browser-bridge) installed
|
||||
|
||||
## Notes
|
||||
|
||||
- `courses` starts from `study.xiaoe-tech.com` and matches purchased course cards back to Vue data to recover shop names and course URLs
|
||||
- `catalog` supports normal courses, columns, and big columns by reading Vuex / Vue component state after the course page loads
|
||||
- `play-url` uses a direct API path for video lessons and falls back to runtime resource inspection for live replays
|
||||
- Cross-shop course URLs are preserved, so you can take a URL from `courses` and pass it directly into `detail`, `catalog`, `play-url`, or `content`
|
||||
@@ -9,9 +9,11 @@ Run `opencli list` for the live registry.
|
||||
| **[twitter](./browser/twitter)** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `likes` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | 🔐 Browser |
|
||||
| **[reddit](./browser/reddit)** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | 🔐 Browser |
|
||||
| **[tieba](./browser/tieba)** | `hot` `posts` `search` `read` | 🔐 Browser |
|
||||
| **[hupu](./browser/hupu)** | `hot` `search` `detail` `reply` `like` `unlike` | 🌐 / 🔐 |
|
||||
| **[bilibili](./browser/bilibili)** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | 🔐 Browser |
|
||||
| **[zhihu](./browser/zhihu)** | `hot` `search` `question` `download` | 🔐 Browser |
|
||||
| **[xiaohongshu](./browser/xiaohongshu)** | `search` `notifications` `feed` `user` `note` `comments` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | 🔐 Browser |
|
||||
| **[xiaoe](./browser/xiaoe)** | `courses` `detail` `catalog` `play-url` `content` | 🔐 Browser |
|
||||
| **[xueqiu](./browser/xueqiu)** | `feed` `hot-stock` `hot` `search` `stock` `comments` `watchlist` `earnings-date` `fund-holdings` `fund-snapshot` | 🔐 Browser |
|
||||
| **[youtube](./browser/youtube)** | `search` `video` `transcript` | 🔐 Browser |
|
||||
| **[v2ex](./browser/v2ex)** | `hot` `latest` `topic` `node` `user` `member` `replies` `nodes` `daily` `me` `notifications` | 🌐 / 🔐 |
|
||||
@@ -46,6 +48,7 @@ Run `opencli list` for the live registry.
|
||||
| **[google](./browser/google)** | `news` `search` `suggest` `trends` | 🌐 / 🔐 |
|
||||
| **[jd](./browser/jd)** | `item` | 🔐 Browser |
|
||||
| **[amazon](./browser/amazon)** | `bestsellers` `search` `product` `offer` `discussion` | 🔐 Browser |
|
||||
| **[1688](./browser/1688)** | `search` `item` `store` | 🔐 Browser |
|
||||
| **[web](./browser/web)** | `read` | 🔐 Browser |
|
||||
| **[weixin](./browser/weixin)** | `download` | 🔐 Browser |
|
||||
| **[36kr](./browser/36kr)** | `news` `hot` `search` `article` | 🌐 / 🔐 |
|
||||
|
||||
@@ -12,7 +12,7 @@ opencli generate https://example.com --goal "trending"
|
||||
|
||||
This runs: explore → synthesize → register in one shot.
|
||||
|
||||
For the complete one-shot workflow details, see [CLI-ONESHOT.md](https://github.com/jackwener/opencli/blob/main/CLI-ONESHOT.md).
|
||||
For the complete one-shot workflow details, see [opencli-oneshot skill](https://github.com/jackwener/opencli/blob/main/skills/opencli-oneshot/SKILL.md).
|
||||
|
||||
## Full Mode (Explorer Workflow)
|
||||
|
||||
@@ -63,4 +63,4 @@ The explorer uses a decision tree to determine the best authentication approach:
|
||||
4. **BROWSER** — Full browser automation
|
||||
5. **CDP** — Chrome DevTools Protocol for Electron apps
|
||||
|
||||
For the complete browser exploration workflow and debugging guide, see [CLI-EXPLORER.md](https://github.com/jackwener/opencli/blob/main/CLI-EXPLORER.md).
|
||||
For the complete browser exploration workflow and debugging guide, see [opencli-explorer skill](https://github.com/jackwener/opencli/blob/main/skills/opencli-explorer/SKILL.md).
|
||||
|
||||
@@ -131,6 +131,8 @@ npx vitest src/
|
||||
- `browser-public.test.ts` 使用 `tryBrowserCommand()`,站点反爬或地域限制导致空数据时会 warn + pass
|
||||
- `browser-auth.test.ts` 验证 **graceful failure**,重点是不 crash、不 hang、错误信息可控
|
||||
- 如需测试完整登录态,保持 Chrome 登录态并安装 Browser Bridge 扩展,再手动运行对应测试
|
||||
- 对依赖具体 host 页面上下文的 browser adapter,除了单测外,还应手动验证真实命令,并把必要的 target host 约束写进 adapter docs / troubleshooting
|
||||
- 对会主动导航页面的 browser commands,手动验证时优先串行执行;多个 CLI 进程同时连到同一个 CDP target 可能互相覆盖导航,制造假的 adapter 故障
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -12,6 +12,17 @@
|
||||
- Your login session in Chrome might have expired. Open a normal Chrome tab, navigate to the target site, and log in or refresh the page.
|
||||
- Some sites have geographic restrictions (e.g., Bilibili, Zhihu from outside China).
|
||||
|
||||
### Browser command opens the page but still cannot read context
|
||||
|
||||
- A healthy Browser Bridge connection does not guarantee that the current page target exposes the data your adapter expects.
|
||||
- Some browser adapters are sensitive to the active host or page context.
|
||||
- Example: `opencli 1688 item` may fail with `did not expose product context` if the target is too broad.
|
||||
- Retry on a real item page, refresh the page in Chrome, and if needed narrow the target, for example:
|
||||
|
||||
```bash
|
||||
OPENCLI_CDP_TARGET=detail.1688.com opencli 1688 item 841141931191 -f json
|
||||
```
|
||||
|
||||
### Node API errors
|
||||
|
||||
- Make sure you are using **Node.js >= 20**. Some dependencies require modern Node APIs.
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
{
|
||||
"lockfileVersion": 1,
|
||||
"configVersion": 0,
|
||||
"workspaces": {
|
||||
"": {
|
||||
"name": "opencli-extension",
|
||||
"devDependencies": {
|
||||
"@types/chrome": "^0.0.287",
|
||||
"typescript": "^5.7.0",
|
||||
"vite": "^6.0.0",
|
||||
},
|
||||
},
|
||||
},
|
||||
"packages": {
|
||||
"@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.25.12", "", { "os": "aix", "cpu": "ppc64" }, "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA=="],
|
||||
|
||||
"@esbuild/android-arm": ["@esbuild/android-arm@0.25.12", "", { "os": "android", "cpu": "arm" }, "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg=="],
|
||||
|
||||
"@esbuild/android-arm64": ["@esbuild/android-arm64@0.25.12", "", { "os": "android", "cpu": "arm64" }, "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg=="],
|
||||
|
||||
"@esbuild/android-x64": ["@esbuild/android-x64@0.25.12", "", { "os": "android", "cpu": "x64" }, "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg=="],
|
||||
|
||||
"@esbuild/darwin-arm64": ["@esbuild/darwin-arm64@0.25.12", "", { "os": "darwin", "cpu": "arm64" }, "sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg=="],
|
||||
|
||||
"@esbuild/darwin-x64": ["@esbuild/darwin-x64@0.25.12", "", { "os": "darwin", "cpu": "x64" }, "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA=="],
|
||||
|
||||
"@esbuild/freebsd-arm64": ["@esbuild/freebsd-arm64@0.25.12", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg=="],
|
||||
|
||||
"@esbuild/freebsd-x64": ["@esbuild/freebsd-x64@0.25.12", "", { "os": "freebsd", "cpu": "x64" }, "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ=="],
|
||||
|
||||
"@esbuild/linux-arm": ["@esbuild/linux-arm@0.25.12", "", { "os": "linux", "cpu": "arm" }, "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw=="],
|
||||
|
||||
"@esbuild/linux-arm64": ["@esbuild/linux-arm64@0.25.12", "", { "os": "linux", "cpu": "arm64" }, "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ=="],
|
||||
|
||||
"@esbuild/linux-ia32": ["@esbuild/linux-ia32@0.25.12", "", { "os": "linux", "cpu": "ia32" }, "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA=="],
|
||||
|
||||
"@esbuild/linux-loong64": ["@esbuild/linux-loong64@0.25.12", "", { "os": "linux", "cpu": "none" }, "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng=="],
|
||||
|
||||
"@esbuild/linux-mips64el": ["@esbuild/linux-mips64el@0.25.12", "", { "os": "linux", "cpu": "none" }, "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw=="],
|
||||
|
||||
"@esbuild/linux-ppc64": ["@esbuild/linux-ppc64@0.25.12", "", { "os": "linux", "cpu": "ppc64" }, "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA=="],
|
||||
|
||||
"@esbuild/linux-riscv64": ["@esbuild/linux-riscv64@0.25.12", "", { "os": "linux", "cpu": "none" }, "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w=="],
|
||||
|
||||
"@esbuild/linux-s390x": ["@esbuild/linux-s390x@0.25.12", "", { "os": "linux", "cpu": "s390x" }, "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg=="],
|
||||
|
||||
"@esbuild/linux-x64": ["@esbuild/linux-x64@0.25.12", "", { "os": "linux", "cpu": "x64" }, "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw=="],
|
||||
|
||||
"@esbuild/netbsd-arm64": ["@esbuild/netbsd-arm64@0.25.12", "", { "os": "none", "cpu": "arm64" }, "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg=="],
|
||||
|
||||
"@esbuild/netbsd-x64": ["@esbuild/netbsd-x64@0.25.12", "", { "os": "none", "cpu": "x64" }, "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ=="],
|
||||
|
||||
"@esbuild/openbsd-arm64": ["@esbuild/openbsd-arm64@0.25.12", "", { "os": "openbsd", "cpu": "arm64" }, "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A=="],
|
||||
|
||||
"@esbuild/openbsd-x64": ["@esbuild/openbsd-x64@0.25.12", "", { "os": "openbsd", "cpu": "x64" }, "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw=="],
|
||||
|
||||
"@esbuild/openharmony-arm64": ["@esbuild/openharmony-arm64@0.25.12", "", { "os": "none", "cpu": "arm64" }, "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg=="],
|
||||
|
||||
"@esbuild/sunos-x64": ["@esbuild/sunos-x64@0.25.12", "", { "os": "sunos", "cpu": "x64" }, "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w=="],
|
||||
|
||||
"@esbuild/win32-arm64": ["@esbuild/win32-arm64@0.25.12", "", { "os": "win32", "cpu": "arm64" }, "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg=="],
|
||||
|
||||
"@esbuild/win32-ia32": ["@esbuild/win32-ia32@0.25.12", "", { "os": "win32", "cpu": "ia32" }, "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ=="],
|
||||
|
||||
"@esbuild/win32-x64": ["@esbuild/win32-x64@0.25.12", "", { "os": "win32", "cpu": "x64" }, "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA=="],
|
||||
|
||||
"@rollup/rollup-android-arm-eabi": ["@rollup/rollup-android-arm-eabi@4.59.0", "", { "os": "android", "cpu": "arm" }, "sha512-upnNBkA6ZH2VKGcBj9Fyl9IGNPULcjXRlg0LLeaioQWueH30p6IXtJEbKAgvyv+mJaMxSm1l6xwDXYjpEMiLMg=="],
|
||||
|
||||
"@rollup/rollup-android-arm64": ["@rollup/rollup-android-arm64@4.59.0", "", { "os": "android", "cpu": "arm64" }, "sha512-hZ+Zxj3SySm4A/DylsDKZAeVg0mvi++0PYVceVyX7hemkw7OreKdCvW2oQ3T1FMZvCaQXqOTHb8qmBShoqk69Q=="],
|
||||
|
||||
"@rollup/rollup-darwin-arm64": ["@rollup/rollup-darwin-arm64@4.59.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-W2Psnbh1J8ZJw0xKAd8zdNgF9HRLkdWwwdWqubSVk0pUuQkoHnv7rx4GiF9rT4t5DIZGAsConRE3AxCdJ4m8rg=="],
|
||||
|
||||
"@rollup/rollup-darwin-x64": ["@rollup/rollup-darwin-x64@4.59.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-ZW2KkwlS4lwTv7ZVsYDiARfFCnSGhzYPdiOU4IM2fDbL+QGlyAbjgSFuqNRbSthybLbIJ915UtZBtmuLrQAT/w=="],
|
||||
|
||||
"@rollup/rollup-freebsd-arm64": ["@rollup/rollup-freebsd-arm64@4.59.0", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-EsKaJ5ytAu9jI3lonzn3BgG8iRBjV4LxZexygcQbpiU0wU0ATxhNVEpXKfUa0pS05gTcSDMKpn3Sx+QB9RlTTA=="],
|
||||
|
||||
"@rollup/rollup-freebsd-x64": ["@rollup/rollup-freebsd-x64@4.59.0", "", { "os": "freebsd", "cpu": "x64" }, "sha512-d3DuZi2KzTMjImrxoHIAODUZYoUUMsuUiY4SRRcJy6NJoZ6iIqWnJu9IScV9jXysyGMVuW+KNzZvBLOcpdl3Vg=="],
|
||||
|
||||
"@rollup/rollup-linux-arm-gnueabihf": ["@rollup/rollup-linux-arm-gnueabihf@4.59.0", "", { "os": "linux", "cpu": "arm" }, "sha512-t4ONHboXi/3E0rT6OZl1pKbl2Vgxf9vJfWgmUoCEVQVxhW6Cw/c8I6hbbu7DAvgp82RKiH7TpLwxnJeKv2pbsw=="],
|
||||
|
||||
"@rollup/rollup-linux-arm-musleabihf": ["@rollup/rollup-linux-arm-musleabihf@4.59.0", "", { "os": "linux", "cpu": "arm" }, "sha512-CikFT7aYPA2ufMD086cVORBYGHffBo4K8MQ4uPS/ZnY54GKj36i196u8U+aDVT2LX4eSMbyHtyOh7D7Zvk2VvA=="],
|
||||
|
||||
"@rollup/rollup-linux-arm64-gnu": ["@rollup/rollup-linux-arm64-gnu@4.59.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-jYgUGk5aLd1nUb1CtQ8E+t5JhLc9x5WdBKew9ZgAXg7DBk0ZHErLHdXM24rfX+bKrFe+Xp5YuJo54I5HFjGDAA=="],
|
||||
|
||||
"@rollup/rollup-linux-arm64-musl": ["@rollup/rollup-linux-arm64-musl@4.59.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-peZRVEdnFWZ5Bh2KeumKG9ty7aCXzzEsHShOZEFiCQlDEepP1dpUl/SrUNXNg13UmZl+gzVDPsiCwnV1uI0RUA=="],
|
||||
|
||||
"@rollup/rollup-linux-loong64-gnu": ["@rollup/rollup-linux-loong64-gnu@4.59.0", "", { "os": "linux", "cpu": "none" }, "sha512-gbUSW/97f7+r4gHy3Jlup8zDG190AuodsWnNiXErp9mT90iCy9NKKU0Xwx5k8VlRAIV2uU9CsMnEFg/xXaOfXg=="],
|
||||
|
||||
"@rollup/rollup-linux-loong64-musl": ["@rollup/rollup-linux-loong64-musl@4.59.0", "", { "os": "linux", "cpu": "none" }, "sha512-yTRONe79E+o0FWFijasoTjtzG9EBedFXJMl888NBEDCDV9I2wGbFFfJQQe63OijbFCUZqxpHz1GzpbtSFikJ4Q=="],
|
||||
|
||||
"@rollup/rollup-linux-ppc64-gnu": ["@rollup/rollup-linux-ppc64-gnu@4.59.0", "", { "os": "linux", "cpu": "ppc64" }, "sha512-sw1o3tfyk12k3OEpRddF68a1unZ5VCN7zoTNtSn2KndUE+ea3m3ROOKRCZxEpmT9nsGnogpFP9x6mnLTCaoLkA=="],
|
||||
|
||||
"@rollup/rollup-linux-ppc64-musl": ["@rollup/rollup-linux-ppc64-musl@4.59.0", "", { "os": "linux", "cpu": "ppc64" }, "sha512-+2kLtQ4xT3AiIxkzFVFXfsmlZiG5FXYW7ZyIIvGA7Bdeuh9Z0aN4hVyXS/G1E9bTP/vqszNIN/pUKCk/BTHsKA=="],
|
||||
|
||||
"@rollup/rollup-linux-riscv64-gnu": ["@rollup/rollup-linux-riscv64-gnu@4.59.0", "", { "os": "linux", "cpu": "none" }, "sha512-NDYMpsXYJJaj+I7UdwIuHHNxXZ/b/N2hR15NyH3m2qAtb/hHPA4g4SuuvrdxetTdndfj9b1WOmy73kcPRoERUg=="],
|
||||
|
||||
"@rollup/rollup-linux-riscv64-musl": ["@rollup/rollup-linux-riscv64-musl@4.59.0", "", { "os": "linux", "cpu": "none" }, "sha512-nLckB8WOqHIf1bhymk+oHxvM9D3tyPndZH8i8+35p/1YiVoVswPid2yLzgX7ZJP0KQvnkhM4H6QZ5m0LzbyIAg=="],
|
||||
|
||||
"@rollup/rollup-linux-s390x-gnu": ["@rollup/rollup-linux-s390x-gnu@4.59.0", "", { "os": "linux", "cpu": "s390x" }, "sha512-oF87Ie3uAIvORFBpwnCvUzdeYUqi2wY6jRFWJAy1qus/udHFYIkplYRW+wo+GRUP4sKzYdmE1Y3+rY5Gc4ZO+w=="],
|
||||
|
||||
"@rollup/rollup-linux-x64-gnu": ["@rollup/rollup-linux-x64-gnu@4.59.0", "", { "os": "linux", "cpu": "x64" }, "sha512-3AHmtQq/ppNuUspKAlvA8HtLybkDflkMuLK4DPo77DfthRb71V84/c4MlWJXixZz4uruIH4uaa07IqoAkG64fg=="],
|
||||
|
||||
"@rollup/rollup-linux-x64-musl": ["@rollup/rollup-linux-x64-musl@4.59.0", "", { "os": "linux", "cpu": "x64" }, "sha512-2UdiwS/9cTAx7qIUZB/fWtToJwvt0Vbo0zmnYt7ED35KPg13Q0ym1g442THLC7VyI6JfYTP4PiSOWyoMdV2/xg=="],
|
||||
|
||||
"@rollup/rollup-openbsd-x64": ["@rollup/rollup-openbsd-x64@4.59.0", "", { "os": "openbsd", "cpu": "x64" }, "sha512-M3bLRAVk6GOwFlPTIxVBSYKUaqfLrn8l0psKinkCFxl4lQvOSz8ZrKDz2gxcBwHFpci0B6rttydI4IpS4IS/jQ=="],
|
||||
|
||||
"@rollup/rollup-openharmony-arm64": ["@rollup/rollup-openharmony-arm64@4.59.0", "", { "os": "none", "cpu": "arm64" }, "sha512-tt9KBJqaqp5i5HUZzoafHZX8b5Q2Fe7UjYERADll83O4fGqJ49O1FsL6LpdzVFQcpwvnyd0i+K/VSwu/o/nWlA=="],
|
||||
|
||||
"@rollup/rollup-win32-arm64-msvc": ["@rollup/rollup-win32-arm64-msvc@4.59.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-V5B6mG7OrGTwnxaNUzZTDTjDS7F75PO1ae6MJYdiMu60sq0CqN5CVeVsbhPxalupvTX8gXVSU9gq+Rx1/hvu6A=="],
|
||||
|
||||
"@rollup/rollup-win32-ia32-msvc": ["@rollup/rollup-win32-ia32-msvc@4.59.0", "", { "os": "win32", "cpu": "ia32" }, "sha512-UKFMHPuM9R0iBegwzKF4y0C4J9u8C6MEJgFuXTBerMk7EJ92GFVFYBfOZaSGLu6COf7FxpQNqhNS4c4icUPqxA=="],
|
||||
|
||||
"@rollup/rollup-win32-x64-gnu": ["@rollup/rollup-win32-x64-gnu@4.59.0", "", { "os": "win32", "cpu": "x64" }, "sha512-laBkYlSS1n2L8fSo1thDNGrCTQMmxjYY5G0WFWjFFYZkKPjsMBsgJfGf4TLxXrF6RyhI60L8TMOjBMvXiTcxeA=="],
|
||||
|
||||
"@rollup/rollup-win32-x64-msvc": ["@rollup/rollup-win32-x64-msvc@4.59.0", "", { "os": "win32", "cpu": "x64" }, "sha512-2HRCml6OztYXyJXAvdDXPKcawukWY2GpR5/nxKp4iBgiO3wcoEGkAaqctIbZcNB6KlUQBIqt8VYkNSj2397EfA=="],
|
||||
|
||||
"@types/chrome": ["@types/chrome@0.0.287", "", { "dependencies": { "@types/filesystem": "*", "@types/har-format": "*" } }, "sha512-wWhBNPNXZHwycHKNYnexUcpSbrihVZu++0rdp6GEk5ZgAglenLx+RwdEouh6FrHS0XQiOxSd62yaujM1OoQlZQ=="],
|
||||
|
||||
"@types/estree": ["@types/estree@1.0.8", "", {}, "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w=="],
|
||||
|
||||
"@types/filesystem": ["@types/filesystem@0.0.36", "", { "dependencies": { "@types/filewriter": "*" } }, "sha512-vPDXOZuannb9FZdxgHnqSwAG/jvdGM8Wq+6N4D/d80z+D4HWH+bItqsZaVRQykAn6WEVeEkLm2oQigyHtgb0RA=="],
|
||||
|
||||
"@types/filewriter": ["@types/filewriter@0.0.33", "", {}, "sha512-xFU8ZXTw4gd358lb2jw25nxY9QAgqn2+bKKjKOYfNCzN4DKCFetK7sPtrlpg66Ywe3vWY9FNxprZawAh9wfJ3g=="],
|
||||
|
||||
"@types/har-format": ["@types/har-format@1.2.16", "", {}, "sha512-fluxdy7ryD3MV6h8pTfTYpy/xQzCFC7m89nOH9y94cNqJ1mDIDPut7MnRHI3F6qRmh/cT2fUjG1MLdCNb4hE9A=="],
|
||||
|
||||
"esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": "bin/esbuild" }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="],
|
||||
|
||||
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" } }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
|
||||
|
||||
"fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="],
|
||||
|
||||
"nanoid": ["nanoid@3.3.11", "", { "bin": "bin/nanoid.cjs" }, "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w=="],
|
||||
|
||||
"picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="],
|
||||
|
||||
"picomatch": ["picomatch@4.0.3", "", {}, "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q=="],
|
||||
|
||||
"postcss": ["postcss@8.5.8", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-OW/rX8O/jXnm82Ey1k44pObPtdblfiuWnrd8X7GJ7emImCOstunGbXUpp7HdBrFQX6rJzn3sPT397Wp5aCwCHg=="],
|
||||
|
||||
"rollup": ["rollup@4.59.0", "", { "dependencies": { "@types/estree": "1.0.8" }, "optionalDependencies": { "@rollup/rollup-android-arm-eabi": "4.59.0", "@rollup/rollup-android-arm64": "4.59.0", "@rollup/rollup-darwin-arm64": "4.59.0", "@rollup/rollup-darwin-x64": "4.59.0", "@rollup/rollup-freebsd-arm64": "4.59.0", "@rollup/rollup-freebsd-x64": "4.59.0", "@rollup/rollup-linux-arm-gnueabihf": "4.59.0", "@rollup/rollup-linux-arm-musleabihf": "4.59.0", "@rollup/rollup-linux-arm64-gnu": "4.59.0", "@rollup/rollup-linux-arm64-musl": "4.59.0", "@rollup/rollup-linux-loong64-gnu": "4.59.0", "@rollup/rollup-linux-loong64-musl": "4.59.0", "@rollup/rollup-linux-ppc64-gnu": "4.59.0", "@rollup/rollup-linux-ppc64-musl": "4.59.0", "@rollup/rollup-linux-riscv64-gnu": "4.59.0", "@rollup/rollup-linux-riscv64-musl": "4.59.0", "@rollup/rollup-linux-s390x-gnu": "4.59.0", "@rollup/rollup-linux-x64-gnu": "4.59.0", "@rollup/rollup-linux-x64-musl": "4.59.0", "@rollup/rollup-openbsd-x64": "4.59.0", "@rollup/rollup-openharmony-arm64": "4.59.0", "@rollup/rollup-win32-arm64-msvc": "4.59.0", "@rollup/rollup-win32-ia32-msvc": "4.59.0", "@rollup/rollup-win32-x64-gnu": "4.59.0", "@rollup/rollup-win32-x64-msvc": "4.59.0", "fsevents": "~2.3.2" }, "bin": "dist/bin/rollup" }, "sha512-2oMpl67a3zCH9H79LeMcbDhXW/UmWG/y2zuqnF2jQq5uq9TbM9TVyXvA4+t+ne2IIkBdrLpAaRQAvo7YI/Yyeg=="],
|
||||
|
||||
"source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
|
||||
|
||||
"tinyglobby": ["tinyglobby@0.2.15", "", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.3" } }, "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ=="],
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"vite": ["vite@6.4.1", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": "bin/vite.js" }, "sha512-+Oxm7q9hDoLMyJOYfUYBuHQo+dkAloi33apOPP56pzj+vsdJDzr+j1NISE5pyaAuKL4A3UD34qd0lx5+kfKp2g=="],
|
||||
}
|
||||
}
|
||||
Vendored
+1046
-658
File diff suppressed because it is too large
Load Diff
+144
-8
@@ -117,6 +117,8 @@ type AutomationSession = {
|
||||
windowId: number;
|
||||
idleTimer: ReturnType<typeof setTimeout> | null;
|
||||
idleDeadlineAt: number;
|
||||
owned: boolean;
|
||||
preferredTabId: number | null;
|
||||
};
|
||||
|
||||
const automationSessions = new Map<string, AutomationSession>();
|
||||
@@ -134,6 +136,11 @@ function resetWindowIdleTimer(workspace: string): void {
|
||||
session.idleTimer = setTimeout(async () => {
|
||||
const current = automationSessions.get(workspace);
|
||||
if (!current) return;
|
||||
if (!current.owned) {
|
||||
console.log(`[opencli] Borrowed workspace ${workspace} detached from window ${current.windowId} (idle timeout)`);
|
||||
automationSessions.delete(workspace);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await chrome.windows.remove(current.windowId);
|
||||
console.log(`[opencli] Automation window ${current.windowId} (${workspace}) closed (idle timeout)`);
|
||||
@@ -177,6 +184,8 @@ async function getAutomationWindow(workspace: string, initialUrl?: string): Prom
|
||||
windowId: win.id!,
|
||||
idleTimer: null,
|
||||
idleDeadlineAt: Date.now() + WINDOW_IDLE_TIMEOUT,
|
||||
owned: true,
|
||||
preferredTabId: null,
|
||||
};
|
||||
automationSessions.set(workspace, session);
|
||||
console.log(`[opencli] Created automation window ${session.windowId} (${workspace}, start=${startUrl})`);
|
||||
@@ -279,6 +288,14 @@ async function handleCommand(cmd: Command): Promise<Result> {
|
||||
return await handleSessions(cmd);
|
||||
case 'set-file-input':
|
||||
return await handleSetFileInput(cmd, workspace);
|
||||
case 'insert-text':
|
||||
return await handleInsertText(cmd, workspace);
|
||||
case 'bind-current':
|
||||
return await handleBindCurrent(cmd, workspace);
|
||||
case 'network-capture-start':
|
||||
return await handleNetworkCaptureStart(cmd, workspace);
|
||||
case 'network-capture-read':
|
||||
return await handleNetworkCaptureRead(cmd, workspace);
|
||||
default:
|
||||
return { id: cmd.id, ok: false, error: `Unknown action: ${cmd.action}` };
|
||||
}
|
||||
@@ -326,7 +343,31 @@ function isTargetUrl(currentUrl: string | undefined, targetUrl: string): boolean
|
||||
return normalizeUrlForComparison(currentUrl) === normalizeUrlForComparison(targetUrl);
|
||||
}
|
||||
|
||||
function setWorkspaceSession(workspace: string, session: Pick<AutomationSession, 'windowId'>): void {
|
||||
function matchesDomain(url: string | undefined, domain: string): boolean {
|
||||
if (!url) return false;
|
||||
try {
|
||||
const parsed = new URL(url);
|
||||
return parsed.hostname === domain || parsed.hostname.endsWith(`.${domain}`);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function matchesBindCriteria(tab: chrome.tabs.Tab, cmd: Command): boolean {
|
||||
if (!tab.id || !isDebuggableUrl(tab.url)) return false;
|
||||
if (cmd.matchDomain && !matchesDomain(tab.url, cmd.matchDomain)) return false;
|
||||
if (cmd.matchPathPrefix) {
|
||||
try {
|
||||
const parsed = new URL(tab.url!);
|
||||
if (!parsed.pathname.startsWith(cmd.matchPathPrefix)) return false;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function setWorkspaceSession(workspace: string, session: Omit<AutomationSession, 'idleTimer' | 'idleDeadlineAt'>): void {
|
||||
const existing = automationSessions.get(workspace);
|
||||
if (existing?.idleTimer) clearTimeout(existing.idleTimer);
|
||||
automationSessions.set(workspace, {
|
||||
@@ -348,9 +389,11 @@ async function resolveTab(tabId: number | undefined, workspace: string, initialU
|
||||
try {
|
||||
const tab = await chrome.tabs.get(tabId);
|
||||
const session = automationSessions.get(workspace);
|
||||
const matchesSession = session ? tab.windowId === session.windowId : false;
|
||||
const matchesSession = session
|
||||
? (session.preferredTabId !== null ? session.preferredTabId === tabId : tab.windowId === session.windowId)
|
||||
: false;
|
||||
if (isDebuggableUrl(tab.url) && matchesSession) return { tabId, tab };
|
||||
if (session && !matchesSession && isDebuggableUrl(tab.url)) {
|
||||
if (session && !matchesSession && session.preferredTabId === null && isDebuggableUrl(tab.url)) {
|
||||
// Tab drifted to another window but content is still valid.
|
||||
// Try to move it back instead of abandoning it.
|
||||
console.warn(`[opencli] Tab ${tabId} drifted to window ${tab.windowId}, moving back to ${session.windowId}`);
|
||||
@@ -371,6 +414,16 @@ async function resolveTab(tabId: number | undefined, workspace: string, initialU
|
||||
}
|
||||
}
|
||||
|
||||
const existingSession = automationSessions.get(workspace);
|
||||
if (existingSession?.preferredTabId !== null) {
|
||||
try {
|
||||
const preferredTab = await chrome.tabs.get(existingSession.preferredTabId);
|
||||
if (isDebuggableUrl(preferredTab.url)) return { tabId: preferredTab.id!, tab: preferredTab };
|
||||
} catch {
|
||||
automationSessions.delete(workspace);
|
||||
}
|
||||
}
|
||||
|
||||
// Get (or create) the automation window
|
||||
const windowId = await getAutomationWindow(workspace, initialUrl);
|
||||
|
||||
@@ -408,6 +461,14 @@ async function resolveTabId(tabId: number | undefined, workspace: string, initia
|
||||
async function listAutomationTabs(workspace: string): Promise<chrome.tabs.Tab[]> {
|
||||
const session = automationSessions.get(workspace);
|
||||
if (!session) return [];
|
||||
if (session.preferredTabId !== null) {
|
||||
try {
|
||||
return [await chrome.tabs.get(session.preferredTabId)];
|
||||
} catch {
|
||||
automationSessions.delete(workspace);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
try {
|
||||
return await chrome.tabs.query({ windowId: session.windowId });
|
||||
} catch {
|
||||
@@ -681,10 +742,12 @@ async function handleCdp(cmd: Command, workspace: string): Promise<Result> {
|
||||
async function handleCloseWindow(cmd: Command, workspace: string): Promise<Result> {
|
||||
const session = automationSessions.get(workspace);
|
||||
if (session) {
|
||||
try {
|
||||
await chrome.windows.remove(session.windowId);
|
||||
} catch {
|
||||
// Window may already be closed
|
||||
if (session.owned) {
|
||||
try {
|
||||
await chrome.windows.remove(session.windowId);
|
||||
} catch {
|
||||
// Window may already be closed
|
||||
}
|
||||
}
|
||||
if (session.idleTimer) clearTimeout(session.idleTimer);
|
||||
automationSessions.delete(workspace);
|
||||
@@ -705,6 +768,39 @@ async function handleSetFileInput(cmd: Command, workspace: string): Promise<Resu
|
||||
}
|
||||
}
|
||||
|
||||
async function handleInsertText(cmd: Command, workspace: string): Promise<Result> {
|
||||
if (typeof cmd.text !== 'string') {
|
||||
return { id: cmd.id, ok: false, error: 'Missing text payload' };
|
||||
}
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
await executor.insertText(tabId, cmd.text);
|
||||
return { id: cmd.id, ok: true, data: { inserted: true } };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
async function handleNetworkCaptureStart(cmd: Command, workspace: string): Promise<Result> {
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
await executor.startNetworkCapture(tabId, cmd.pattern);
|
||||
return { id: cmd.id, ok: true, data: { started: true } };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
async function handleNetworkCaptureRead(cmd: Command, workspace: string): Promise<Result> {
|
||||
const tabId = await resolveTabId(cmd.tabId, workspace);
|
||||
try {
|
||||
const data = await executor.readNetworkCapture(tabId);
|
||||
return { id: cmd.id, ok: true, data };
|
||||
} catch (err) {
|
||||
return { id: cmd.id, ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
async function handleSessions(cmd: Command): Promise<Result> {
|
||||
const now = Date.now();
|
||||
const data = await Promise.all([...automationSessions.entries()].map(async ([workspace, session]) => ({
|
||||
@@ -716,11 +812,49 @@ async function handleSessions(cmd: Command): Promise<Result> {
|
||||
return { id: cmd.id, ok: true, data };
|
||||
}
|
||||
|
||||
async function handleBindCurrent(cmd: Command, workspace: string): Promise<Result> {
|
||||
const activeTabs = await chrome.tabs.query({ active: true, lastFocusedWindow: true });
|
||||
const fallbackTabs = await chrome.tabs.query({ lastFocusedWindow: true });
|
||||
const allTabs = await chrome.tabs.query({});
|
||||
const boundTab = activeTabs.find((tab) => matchesBindCriteria(tab, cmd))
|
||||
?? fallbackTabs.find((tab) => matchesBindCriteria(tab, cmd))
|
||||
?? allTabs.find((tab) => matchesBindCriteria(tab, cmd));
|
||||
if (!boundTab?.id) {
|
||||
return {
|
||||
id: cmd.id,
|
||||
ok: false,
|
||||
error: cmd.matchDomain || cmd.matchPathPrefix
|
||||
? `No visible tab matching ${cmd.matchDomain ?? 'domain'}${cmd.matchPathPrefix ? ` ${cmd.matchPathPrefix}` : ''}`
|
||||
: 'No active debuggable tab found',
|
||||
};
|
||||
}
|
||||
|
||||
setWorkspaceSession(workspace, {
|
||||
windowId: boundTab.windowId,
|
||||
owned: false,
|
||||
preferredTabId: boundTab.id,
|
||||
});
|
||||
resetWindowIdleTimer(workspace);
|
||||
console.log(`[opencli] Workspace ${workspace} explicitly bound to tab ${boundTab.id} (${boundTab.url})`);
|
||||
return {
|
||||
id: cmd.id,
|
||||
ok: true,
|
||||
data: {
|
||||
tabId: boundTab.id,
|
||||
windowId: boundTab.windowId,
|
||||
url: boundTab.url,
|
||||
title: boundTab.title,
|
||||
workspace,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const __test__ = {
|
||||
handleNavigate,
|
||||
isTargetUrl,
|
||||
handleTabs,
|
||||
handleSessions,
|
||||
handleBindCurrent,
|
||||
resolveTabId,
|
||||
resetWindowIdleTimer,
|
||||
getSession: (workspace: string = 'default') => automationSessions.get(workspace) ?? null,
|
||||
@@ -734,9 +868,11 @@ export const __test__ = {
|
||||
}
|
||||
setWorkspaceSession(workspace, {
|
||||
windowId,
|
||||
owned: true,
|
||||
preferredTabId: null,
|
||||
});
|
||||
},
|
||||
setSession: (workspace: string, session: { windowId: number }) => {
|
||||
setSession: (workspace: string, session: { windowId: number; owned: boolean; preferredTabId: number | null }) => {
|
||||
setWorkspaceSession(workspace, session);
|
||||
},
|
||||
};
|
||||
|
||||
+178
-1
@@ -8,6 +8,27 @@
|
||||
|
||||
const attached = new Set<number>();
|
||||
|
||||
type NetworkCaptureEntry = {
|
||||
kind: 'cdp';
|
||||
url: string;
|
||||
method: string;
|
||||
requestHeaders?: Record<string, string>;
|
||||
requestBodyKind?: string;
|
||||
requestBodyPreview?: string;
|
||||
responseStatus?: number;
|
||||
responseContentType?: string;
|
||||
responseHeaders?: Record<string, string>;
|
||||
responsePreview?: string;
|
||||
timestamp: number;
|
||||
};
|
||||
|
||||
type NetworkCaptureState = {
|
||||
patterns: string[];
|
||||
entries: NetworkCaptureEntry[];
|
||||
requestToIndex: Map<string, number>;
|
||||
};
|
||||
|
||||
const networkCaptures = new Map<number, NetworkCaptureState>();
|
||||
/** Check if a URL can be attached via CDP — only allow http(s) and blank pages. */
|
||||
function isDebuggableUrl(url?: string): boolean {
|
||||
if (!url) return true; // empty/undefined = tab still loading, allow it
|
||||
@@ -241,18 +262,100 @@ export async function setFileInputFiles(
|
||||
});
|
||||
}
|
||||
|
||||
export async function insertText(
|
||||
tabId: number,
|
||||
text: string,
|
||||
): Promise<void> {
|
||||
await ensureAttached(tabId);
|
||||
await chrome.debugger.sendCommand({ tabId }, 'Input.insertText', { text });
|
||||
}
|
||||
|
||||
function normalizeCapturePatterns(pattern?: string): string[] {
|
||||
return String(pattern || '')
|
||||
.split('|')
|
||||
.map((part) => part.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function shouldCaptureUrl(url: string | undefined, patterns: string[]): boolean {
|
||||
if (!url) return false;
|
||||
if (!patterns.length) return true;
|
||||
return patterns.some((pattern) => url.includes(pattern));
|
||||
}
|
||||
|
||||
function normalizeHeaders(headers: unknown): Record<string, string> {
|
||||
if (!headers || typeof headers !== 'object') return {};
|
||||
const out: Record<string, string> = {};
|
||||
for (const [key, value] of Object.entries(headers as Record<string, unknown>)) {
|
||||
out[String(key)] = String(value);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function getOrCreateNetworkCaptureEntry(tabId: number, requestId: string, fallback?: {
|
||||
url?: string;
|
||||
method?: string;
|
||||
requestHeaders?: Record<string, string>;
|
||||
}): NetworkCaptureEntry | null {
|
||||
const state = networkCaptures.get(tabId);
|
||||
if (!state) return null;
|
||||
const existingIndex = state.requestToIndex.get(requestId);
|
||||
if (existingIndex !== undefined) {
|
||||
return state.entries[existingIndex] || null;
|
||||
}
|
||||
const url = fallback?.url || '';
|
||||
if (!shouldCaptureUrl(url, state.patterns)) return null;
|
||||
const entry: NetworkCaptureEntry = {
|
||||
kind: 'cdp',
|
||||
url,
|
||||
method: fallback?.method || 'GET',
|
||||
requestHeaders: fallback?.requestHeaders || {},
|
||||
timestamp: Date.now(),
|
||||
};
|
||||
state.entries.push(entry);
|
||||
state.requestToIndex.set(requestId, state.entries.length - 1);
|
||||
return entry;
|
||||
}
|
||||
|
||||
export async function startNetworkCapture(
|
||||
tabId: number,
|
||||
pattern?: string,
|
||||
): Promise<void> {
|
||||
await ensureAttached(tabId);
|
||||
await chrome.debugger.sendCommand({ tabId }, 'Network.enable');
|
||||
networkCaptures.set(tabId, {
|
||||
patterns: normalizeCapturePatterns(pattern),
|
||||
entries: [],
|
||||
requestToIndex: new Map(),
|
||||
});
|
||||
}
|
||||
|
||||
export async function readNetworkCapture(tabId: number): Promise<NetworkCaptureEntry[]> {
|
||||
const state = networkCaptures.get(tabId);
|
||||
if (!state) return [];
|
||||
const entries = state.entries.slice();
|
||||
state.entries = [];
|
||||
state.requestToIndex.clear();
|
||||
return entries;
|
||||
}
|
||||
|
||||
export async function detach(tabId: number): Promise<void> {
|
||||
if (!attached.has(tabId)) return;
|
||||
attached.delete(tabId);
|
||||
networkCaptures.delete(tabId);
|
||||
try { await chrome.debugger.detach({ tabId }); } catch { /* ignore */ }
|
||||
}
|
||||
|
||||
export function registerListeners(): void {
|
||||
chrome.tabs.onRemoved.addListener((tabId) => {
|
||||
attached.delete(tabId);
|
||||
networkCaptures.delete(tabId);
|
||||
});
|
||||
chrome.debugger.onDetach.addListener((source) => {
|
||||
if (source.tabId) attached.delete(source.tabId);
|
||||
if (source.tabId) {
|
||||
attached.delete(source.tabId);
|
||||
networkCaptures.delete(source.tabId);
|
||||
}
|
||||
});
|
||||
// Invalidate attached cache when tab URL changes to non-debuggable
|
||||
chrome.tabs.onUpdated.addListener(async (tabId, info) => {
|
||||
@@ -260,4 +363,78 @@ export function registerListeners(): void {
|
||||
await detach(tabId);
|
||||
}
|
||||
});
|
||||
chrome.debugger.onEvent.addListener(async (source, method, params) => {
|
||||
const tabId = source.tabId;
|
||||
if (!tabId) return;
|
||||
const state = networkCaptures.get(tabId);
|
||||
if (!state) return;
|
||||
|
||||
if (method === 'Network.requestWillBeSent') {
|
||||
const requestId = String(params?.requestId || '');
|
||||
const request = params?.request as {
|
||||
url?: string;
|
||||
method?: string;
|
||||
headers?: Record<string, unknown>;
|
||||
postData?: string;
|
||||
hasPostData?: boolean;
|
||||
} | undefined;
|
||||
const entry = getOrCreateNetworkCaptureEntry(tabId, requestId, {
|
||||
url: request?.url,
|
||||
method: request?.method,
|
||||
requestHeaders: normalizeHeaders(request?.headers),
|
||||
});
|
||||
if (!entry) return;
|
||||
entry.requestBodyKind = request?.hasPostData ? 'string' : 'empty';
|
||||
entry.requestBodyPreview = String(request?.postData || '').slice(0, 4000);
|
||||
try {
|
||||
const postData = await chrome.debugger.sendCommand({ tabId }, 'Network.getRequestPostData', { requestId }) as { postData?: string };
|
||||
if (postData?.postData) {
|
||||
entry.requestBodyKind = 'string';
|
||||
entry.requestBodyPreview = postData.postData.slice(0, 4000);
|
||||
}
|
||||
} catch {
|
||||
// Optional; some requests do not expose postData.
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (method === 'Network.responseReceived') {
|
||||
const requestId = String(params?.requestId || '');
|
||||
const response = params?.response as {
|
||||
url?: string;
|
||||
mimeType?: string;
|
||||
status?: number;
|
||||
headers?: Record<string, unknown>;
|
||||
} | undefined;
|
||||
const entry = getOrCreateNetworkCaptureEntry(tabId, requestId, {
|
||||
url: response?.url,
|
||||
});
|
||||
if (!entry) return;
|
||||
entry.responseStatus = response?.status;
|
||||
entry.responseContentType = response?.mimeType || '';
|
||||
entry.responseHeaders = normalizeHeaders(response?.headers);
|
||||
return;
|
||||
}
|
||||
|
||||
if (method === 'Network.loadingFinished') {
|
||||
const requestId = String(params?.requestId || '');
|
||||
const stateEntryIndex = state.requestToIndex.get(requestId);
|
||||
if (stateEntryIndex === undefined) return;
|
||||
const entry = state.entries[stateEntryIndex];
|
||||
if (!entry) return;
|
||||
try {
|
||||
const body = await chrome.debugger.sendCommand({ tabId }, 'Network.getResponseBody', { requestId }) as {
|
||||
body?: string;
|
||||
base64Encoded?: boolean;
|
||||
};
|
||||
if (typeof body?.body === 'string') {
|
||||
entry.responsePreview = body.base64Encoded
|
||||
? `base64:${body.body.slice(0, 4000)}`
|
||||
: body.body.slice(0, 4000);
|
||||
}
|
||||
} catch {
|
||||
// Optional; bodies are unavailable for some requests (e.g. uploads).
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -5,7 +5,20 @@
|
||||
* Everything else is just JS code sent via 'exec'.
|
||||
*/
|
||||
|
||||
export type Action = 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions' | 'set-file-input' | 'cdp';
|
||||
export type Action =
|
||||
| 'exec'
|
||||
| 'navigate'
|
||||
| 'tabs'
|
||||
| 'cookies'
|
||||
| 'screenshot'
|
||||
| 'close-window'
|
||||
| 'sessions'
|
||||
| 'set-file-input'
|
||||
| 'insert-text'
|
||||
| 'bind-current'
|
||||
| 'network-capture-start'
|
||||
| 'network-capture-read'
|
||||
| 'cdp';
|
||||
|
||||
export interface Command {
|
||||
/** Unique request ID */
|
||||
@@ -26,6 +39,10 @@ export interface Command {
|
||||
index?: number;
|
||||
/** Cookie domain filter */
|
||||
domain?: string;
|
||||
/** Optional hostname/domain to require for current-tab binding */
|
||||
matchDomain?: string;
|
||||
/** Optional pathname prefix to require for current-tab binding */
|
||||
matchPathPrefix?: string;
|
||||
/** Screenshot format: png (default) or jpeg */
|
||||
format?: 'png' | 'jpeg';
|
||||
/** JPEG quality (0-100), only for jpeg format */
|
||||
@@ -36,6 +53,10 @@ export interface Command {
|
||||
files?: string[];
|
||||
/** CSS selector for file input element (set-file-input action) */
|
||||
selector?: string;
|
||||
/** Raw text payload for insert-text action */
|
||||
text?: string;
|
||||
/** URL substring filter pattern for network capture actions */
|
||||
pattern?: string;
|
||||
/** CDP method name for 'cdp' action (e.g. 'Accessibility.getFullAXTree') */
|
||||
cdpMethod?: string;
|
||||
/** CDP method params for 'cdp' action */
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@jackwener/opencli",
|
||||
"version": "1.6.1",
|
||||
"version": "1.6.2",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@jackwener/opencli",
|
||||
"version": "1.6.1",
|
||||
"version": "1.6.2",
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@jackwener/opencli",
|
||||
"version": "1.6.1",
|
||||
"version": "1.6.2",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
|
||||
@@ -217,7 +217,7 @@ function main() {
|
||||
console.log(' \x1b[1mNext step — Browser Bridge setup\x1b[0m');
|
||||
console.log(' Browser commands (bilibili, zhihu, twitter...) require the extension:');
|
||||
console.log(' 1. Download: https://github.com/jackwener/opencli/releases');
|
||||
console.log(' 2. Open chrome://extensions → enable Developer Mode → Load unpacked');
|
||||
console.log(' 2. In Chrome or Chromium, open chrome://extensions → enable Developer Mode → Load unpacked');
|
||||
console.log('');
|
||||
console.log(' Then run \x1b[36mopencli doctor\x1b[0m to verify.');
|
||||
console.log('');
|
||||
|
||||
@@ -10,7 +10,7 @@ tags: [opencli, adapter, browser, api-discovery, cli, web-scraping, automation]
|
||||
> 从零到发布,覆盖 API 发现、方案选择、适配器编写、测试验证全流程。
|
||||
|
||||
> [!TIP]
|
||||
> **只想为一个具体页面快速生成一个命令?** 看 [CLI-ONESHOT.md](./CLI-ONESHOT.md)(~150 行,4 步搞定)。
|
||||
> **只想为一个具体页面快速生成一个命令?** 看 [opencli-oneshot skill](../opencli-oneshot/SKILL.md)(~150 行,4 步搞定)。
|
||||
> 本文档适合从零探索一个新站点的完整流程。
|
||||
|
||||
---
|
||||
|
||||
@@ -7,7 +7,7 @@ tags: [opencli, adapter, quick-start, yaml, cli, one-shot, automation]
|
||||
# CLI-ONESHOT — 单点快速 CLI 生成
|
||||
|
||||
> 给一个 URL + 一句话描述,4 步生成一个 CLI 命令。
|
||||
> 完整探索式开发请看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
|
||||
> 完整探索式开发请看 [opencli-explorer skill](../opencli-explorer/SKILL.md)。
|
||||
|
||||
---
|
||||
|
||||
@@ -219,4 +219,4 @@ opencli mysite mycommand --limit 3 -v # 实际运行
|
||||
|
||||
## 就这样,没了
|
||||
|
||||
写完文件 → build → run → 提交。有问题再看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
|
||||
写完文件 → build → run → 提交。有问题再看 [opencli-explorer skill](../opencli-explorer/SKILL.md)。
|
||||
|
||||
@@ -115,13 +115,17 @@ opencli operate keys "Enter" # Press key (Enter, Escape, Tab, Control
|
||||
|
||||
### Wait
|
||||
|
||||
Three variants — use the right one for the situation:
|
||||
|
||||
```bash
|
||||
opencli operate wait selector ".loaded" # Wait for element
|
||||
opencli operate wait selector ".spinner" --timeout 5000 # With timeout
|
||||
opencli operate wait text "Success" # Wait for text
|
||||
opencli operate wait time 3 # Wait N seconds
|
||||
opencli operate wait time 3 # Wait N seconds (fixed delay)
|
||||
opencli operate wait selector ".loaded" # Wait until element appears in DOM
|
||||
opencli operate wait selector ".spinner" --timeout 5000 # With timeout (default 30s)
|
||||
opencli operate wait text "Success" # Wait until text appears on page
|
||||
```
|
||||
|
||||
**When to wait**: After `open` on SPAs, after `click` that triggers async loading, before `eval` on dynamically rendered content.
|
||||
|
||||
### Extract (free & instant, read-only)
|
||||
|
||||
Use `eval` ONLY for reading data. Never use it to click, type, or navigate.
|
||||
@@ -134,6 +138,16 @@ opencli operate eval "JSON.stringify([...document.querySelectorAll('h2')].map(e
|
||||
opencli operate eval "(function(){ const items = [...document.querySelectorAll('.item')]; return JSON.stringify(items.map(e => e.textContent)); })()"
|
||||
```
|
||||
|
||||
**Selector safety**: Always use fallback selectors — `querySelector` returns `null` on miss:
|
||||
```bash
|
||||
# BAD: crashes if selector misses
|
||||
opencli operate eval "document.querySelector('.title').textContent"
|
||||
|
||||
# GOOD: fallback with || or ?.
|
||||
opencli operate eval "(document.querySelector('.title') || document.querySelector('h1') || {textContent:''}).textContent"
|
||||
opencli operate eval "document.querySelector('.title')?.textContent ?? 'not found'"
|
||||
```
|
||||
|
||||
### Network (API Discovery)
|
||||
|
||||
```bash
|
||||
@@ -145,10 +159,14 @@ opencli operate network --all # Include static resources
|
||||
### Sedimentation (Save as CLI)
|
||||
|
||||
```bash
|
||||
opencli operate init hn/top # Generate adapter scaffold
|
||||
opencli operate verify hn/top # Test the adapter
|
||||
opencli operate init hn/top # Generate adapter scaffold at ~/.opencli/clis/hn/top.ts
|
||||
opencli operate verify hn/top # Test the adapter (adds --limit 3 only if `limit` arg is defined)
|
||||
```
|
||||
|
||||
- `init` auto-detects the domain from the active browser session (no need to specify it)
|
||||
- `init` creates the file + populates `site`, `name`, `domain`, and `columns` from current page
|
||||
- `verify` runs the adapter end-to-end and prints output; if no `limit` arg exists in the adapter, it won't pass `--limit 3`
|
||||
|
||||
### Session
|
||||
|
||||
```bash
|
||||
@@ -252,6 +270,32 @@ Save to `~/.opencli/clis/<site>/<command>.ts` → immediately available as `open
|
||||
4. **Use `network` to find APIs** — JSON APIs are more reliable than DOM scraping
|
||||
5. **Alias**: `opencli op` is shorthand for `opencli operate`
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **`form.submit()` fails in automation** — Don't use `form.submit()` or `eval` to submit forms. Navigate directly to the search URL instead:
|
||||
```bash
|
||||
# BAD: form.submit() often silently fails
|
||||
opencli operate eval "document.querySelector('form').submit()"
|
||||
# GOOD: construct the URL and navigate
|
||||
opencli operate open "https://github.com/search?q=opencli&type=repositories"
|
||||
```
|
||||
|
||||
2. **GitHub DOM changes frequently** — Prefer `data-testid` attributes when available; they are more stable than class names or tag structure.
|
||||
|
||||
3. **SPA pages need `wait` before extraction** — After `open` or `click` on single-page apps, the DOM isn't ready immediately. Always `wait selector` or `wait text` before `eval`.
|
||||
|
||||
4. **Use `state` before clicking** — Run `opencli operate state` to inspect available interactive elements and their indices. Never guess indices from memory.
|
||||
|
||||
5. **`evaluate` runs in browser context** — `page.evaluate()` in adapters executes inside the browser. Node.js APIs (`fs`, `path`, `process`) are NOT available. Use `fetch()` for network calls, DOM APIs for page data.
|
||||
|
||||
6. **Backticks in `page.evaluate` break JSON storage** — When writing adapters that will be stored/transported as JSON, avoid template literals inside `page.evaluate`. Use string concatenation or function-style evaluate:
|
||||
```typescript
|
||||
// BAD: template literal backticks break when adapter is in JSON
|
||||
page.evaluate(`document.querySelector("${selector}")`)
|
||||
// GOOD: function-style evaluate
|
||||
page.evaluate((sel) => document.querySelector(sel), selector)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Error | Fix |
|
||||
|
||||
@@ -133,7 +133,7 @@ export abstract class BasePage implements IPage {
|
||||
|
||||
async snapshot(opts: SnapshotOptions = {}): Promise<unknown> {
|
||||
const snapshotJs = generateSnapshotJs({
|
||||
viewportExpand: opts.viewportExpand ?? 800,
|
||||
viewportExpand: opts.viewportExpand ?? 2000,
|
||||
maxDepth: Math.max(1, Math.min(Number(opts.maxDepth) || 50, 200)),
|
||||
interactiveOnly: opts.interactive ?? false,
|
||||
maxTextLength: opts.maxTextLength ?? 120,
|
||||
@@ -152,7 +152,11 @@ export abstract class BasePage implements IPage {
|
||||
// Non-fatal: diff is best-effort
|
||||
}
|
||||
return result;
|
||||
} catch {
|
||||
} catch (err) {
|
||||
// Log snapshot failure for debugging, then fallback to basic accessibility tree
|
||||
if (process.env.DEBUG_SNAPSHOT) {
|
||||
console.error('[snapshot] DOM snapshot failed, falling back to accessibility tree:', (err as Error)?.message?.slice(0, 200));
|
||||
}
|
||||
return this._basicSnapshot(opts);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -69,8 +69,8 @@ export class BrowserBridge implements IBrowserFactory {
|
||||
// Daemon running but no extension — wait for extension with progress
|
||||
if (status !== null) {
|
||||
if (process.env.OPENCLI_VERBOSE || process.stderr.isTTY) {
|
||||
process.stderr.write('⏳ Waiting for Chrome extension to connect...\n');
|
||||
process.stderr.write(' Make sure Chrome is open and the OpenCLI extension is enabled.\n');
|
||||
process.stderr.write('⏳ Waiting for Chrome/Chromium extension to connect...\n');
|
||||
process.stderr.write(' Make sure Chrome or Chromium is open and the OpenCLI extension is enabled.\n');
|
||||
}
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
@@ -79,7 +79,7 @@ export class BrowserBridge implements IBrowserFactory {
|
||||
}
|
||||
throw new Error(
|
||||
'Daemon is running but the Browser Extension is not connected.\n' +
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome.',
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome or Chromium.',
|
||||
);
|
||||
}
|
||||
|
||||
@@ -116,7 +116,7 @@ export class BrowserBridge implements IBrowserFactory {
|
||||
if ((await fetchDaemonStatus()) !== null) {
|
||||
throw new Error(
|
||||
'Daemon is running but the Browser Extension is not connected.\n' +
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome.',
|
||||
'Please install and enable the opencli Browser Bridge extension in Chrome or Chromium.',
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ function generateId(): string {
|
||||
|
||||
export interface DaemonCommand {
|
||||
id: string;
|
||||
action: 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions' | 'set-file-input' | 'cdp';
|
||||
action: 'exec' | 'navigate' | 'tabs' | 'cookies' | 'screenshot' | 'close-window' | 'sessions' | 'set-file-input' | 'insert-text' | 'bind-current' | 'network-capture-start' | 'network-capture-read' | 'cdp';
|
||||
tabId?: number;
|
||||
code?: string;
|
||||
workspace?: string;
|
||||
@@ -29,6 +29,8 @@ export interface DaemonCommand {
|
||||
op?: string;
|
||||
index?: number;
|
||||
domain?: string;
|
||||
matchDomain?: string;
|
||||
matchPathPrefix?: string;
|
||||
format?: 'png' | 'jpeg';
|
||||
quality?: number;
|
||||
fullPage?: boolean;
|
||||
@@ -37,6 +39,10 @@ export interface DaemonCommand {
|
||||
files?: string[];
|
||||
/** CSS selector for file input element (set-file-input action) */
|
||||
selector?: string;
|
||||
/** Raw text payload for insert-text action */
|
||||
text?: string;
|
||||
/** URL substring filter pattern for network capture */
|
||||
pattern?: string;
|
||||
cdpMethod?: string;
|
||||
cdpParams?: Record<string, unknown>;
|
||||
}
|
||||
@@ -163,3 +169,7 @@ export async function listSessions(): Promise<BrowserSessionInfo[]> {
|
||||
const result = await sendCommand('sessions');
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
|
||||
export async function bindCurrentTab(workspace: string, opts: { matchDomain?: string; matchPathPrefix?: string } = {}): Promise<unknown> {
|
||||
return sendCommand('bind-current', { workspace, ...opts });
|
||||
}
|
||||
|
||||
@@ -407,7 +407,8 @@ export function generateSnapshotJs(opts: DomSnapshotOptions = {}): string {
|
||||
|
||||
function isSearchElement(el) {
|
||||
// Check class names for search indicators
|
||||
const className = el.className?.toLowerCase() || '';
|
||||
// Note: SVG elements have className as SVGAnimatedString (not a string), use baseVal
|
||||
const className = (typeof el.className === 'string' ? el.className : el.className?.baseVal || '').toLowerCase();
|
||||
const classes = className.split(/\\s+/).filter(Boolean);
|
||||
for (const cls of classes) {
|
||||
const cleaned = cls.replace(/[^a-z0-9-]/g, '');
|
||||
|
||||
+26
-1
@@ -120,6 +120,9 @@ export class Page extends BasePage {
|
||||
await sendCommand('close-window', { ...this._wsOpt() });
|
||||
} catch {
|
||||
// Window may already be closed or daemon may be down
|
||||
} finally {
|
||||
this._tabId = undefined;
|
||||
this._lastUrl = null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -151,6 +154,19 @@ export class Page extends BasePage {
|
||||
return base64;
|
||||
}
|
||||
|
||||
async startNetworkCapture(pattern: string = ''): Promise<void> {
|
||||
await sendCommand('network-capture-start', {
|
||||
pattern,
|
||||
...this._cmdOpts(),
|
||||
});
|
||||
}
|
||||
|
||||
async readNetworkCapture(): Promise<unknown[]> {
|
||||
const result = await sendCommand('network-capture-read', {
|
||||
...this._cmdOpts(),
|
||||
});
|
||||
return Array.isArray(result) ? result : [];
|
||||
}
|
||||
/**
|
||||
* Set local file paths on a file input element via CDP DOM.setFileInputFiles.
|
||||
* Chrome reads the files directly from the local filesystem, avoiding the
|
||||
@@ -167,6 +183,16 @@ export class Page extends BasePage {
|
||||
}
|
||||
}
|
||||
|
||||
async insertText(text: string): Promise<void> {
|
||||
const result = await sendCommand('insert-text', {
|
||||
text,
|
||||
...this._cmdOpts(),
|
||||
}) as { inserted?: boolean };
|
||||
if (!result?.inserted) {
|
||||
throw new Error('insertText returned no inserted flag — command may not be supported by the extension');
|
||||
}
|
||||
}
|
||||
|
||||
async cdp(method: string, params: Record<string, unknown> = {}): Promise<unknown> {
|
||||
return sendCommand('cdp', {
|
||||
cdpMethod: method,
|
||||
@@ -287,4 +313,3 @@ export class Page extends BasePage {
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ export interface ManifestEntry {
|
||||
type?: string;
|
||||
default?: unknown;
|
||||
required?: boolean;
|
||||
valueRequired?: boolean;
|
||||
positional?: boolean;
|
||||
help?: string;
|
||||
choices?: string[];
|
||||
@@ -62,6 +63,7 @@ function toManifestArgs(args: CliCommand['args']): ManifestEntry['args'] {
|
||||
type: arg.type ?? 'str',
|
||||
default: arg.default,
|
||||
required: !!arg.required,
|
||||
valueRequired: !!arg.valueRequired || undefined,
|
||||
positional: arg.positional || undefined,
|
||||
help: arg.help ?? '',
|
||||
choices: arg.choices,
|
||||
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { IPage } from './types.js';
|
||||
|
||||
const {
|
||||
mockExploreUrl,
|
||||
mockRenderExploreSummary,
|
||||
mockGenerateCliFromUrl,
|
||||
mockRenderGenerateSummary,
|
||||
mockRecordSession,
|
||||
mockRenderRecordSummary,
|
||||
mockCascadeProbe,
|
||||
mockRenderCascadeResult,
|
||||
mockGetBrowserFactory,
|
||||
mockBrowserSession,
|
||||
} = vi.hoisted(() => ({
|
||||
mockExploreUrl: vi.fn(),
|
||||
mockRenderExploreSummary: vi.fn(),
|
||||
mockGenerateCliFromUrl: vi.fn(),
|
||||
mockRenderGenerateSummary: vi.fn(),
|
||||
mockRecordSession: vi.fn(),
|
||||
mockRenderRecordSummary: vi.fn(),
|
||||
mockCascadeProbe: vi.fn(),
|
||||
mockRenderCascadeResult: vi.fn(),
|
||||
mockGetBrowserFactory: vi.fn(() => ({ name: 'BrowserFactory' })),
|
||||
mockBrowserSession: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('./explore.js', () => ({
|
||||
exploreUrl: mockExploreUrl,
|
||||
renderExploreSummary: mockRenderExploreSummary,
|
||||
}));
|
||||
|
||||
vi.mock('./generate.js', () => ({
|
||||
generateCliFromUrl: mockGenerateCliFromUrl,
|
||||
renderGenerateSummary: mockRenderGenerateSummary,
|
||||
}));
|
||||
|
||||
vi.mock('./record.js', () => ({
|
||||
recordSession: mockRecordSession,
|
||||
renderRecordSummary: mockRenderRecordSummary,
|
||||
}));
|
||||
|
||||
vi.mock('./cascade.js', () => ({
|
||||
cascadeProbe: mockCascadeProbe,
|
||||
renderCascadeResult: mockRenderCascadeResult,
|
||||
}));
|
||||
|
||||
vi.mock('./runtime.js', () => ({
|
||||
getBrowserFactory: mockGetBrowserFactory,
|
||||
browserSession: mockBrowserSession,
|
||||
}));
|
||||
|
||||
import { createProgram } from './cli.js';
|
||||
|
||||
describe('built-in browser commands verbose wiring', () => {
|
||||
const consoleLogSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env.OPENCLI_VERBOSE;
|
||||
process.exitCode = undefined;
|
||||
|
||||
mockExploreUrl.mockReset().mockResolvedValue({ ok: true });
|
||||
mockRenderExploreSummary.mockReset().mockReturnValue('explore-summary');
|
||||
mockGenerateCliFromUrl.mockReset().mockResolvedValue({ ok: true });
|
||||
mockRenderGenerateSummary.mockReset().mockReturnValue('generate-summary');
|
||||
mockRecordSession.mockReset().mockResolvedValue({ candidateCount: 1 });
|
||||
mockRenderRecordSummary.mockReset().mockReturnValue('record-summary');
|
||||
mockCascadeProbe.mockReset().mockResolvedValue({ ok: true });
|
||||
mockRenderCascadeResult.mockReset().mockReturnValue('cascade-summary');
|
||||
mockGetBrowserFactory.mockClear();
|
||||
mockBrowserSession.mockReset().mockImplementation(async (_factory, fn) => {
|
||||
const page = {
|
||||
goto: vi.fn(),
|
||||
wait: vi.fn(),
|
||||
} as unknown as IPage;
|
||||
return fn(page);
|
||||
});
|
||||
});
|
||||
|
||||
it('enables OPENCLI_VERBOSE for explore via the real CLI command', async () => {
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'explore', 'https://example.com', '-v']);
|
||||
|
||||
expect(process.env.OPENCLI_VERBOSE).toBe('1');
|
||||
expect(mockExploreUrl).toHaveBeenCalledWith(
|
||||
'https://example.com',
|
||||
expect.objectContaining({ workspace: 'explore:example.com' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('enables OPENCLI_VERBOSE for generate via the real CLI command', async () => {
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'generate', 'https://example.com', '-v']);
|
||||
|
||||
expect(process.env.OPENCLI_VERBOSE).toBe('1');
|
||||
expect(mockGenerateCliFromUrl).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ url: 'https://example.com', workspace: 'generate:example.com' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('enables OPENCLI_VERBOSE for record via the real CLI command', async () => {
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'record', 'https://example.com', '-v']);
|
||||
|
||||
expect(process.env.OPENCLI_VERBOSE).toBe('1');
|
||||
expect(mockRecordSession).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ url: 'https://example.com' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('enables OPENCLI_VERBOSE for cascade via the real CLI command', async () => {
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'cascade', 'https://example.com', '-v']);
|
||||
|
||||
expect(process.env.OPENCLI_VERBOSE).toBe('1');
|
||||
expect(mockBrowserSession).toHaveBeenCalled();
|
||||
expect(mockCascadeProbe).toHaveBeenCalledWith(expect.any(Object), 'https://example.com');
|
||||
});
|
||||
|
||||
it('leaves OPENCLI_VERBOSE unset when verbose is omitted', async () => {
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'explore', 'https://example.com']);
|
||||
|
||||
expect(process.env.OPENCLI_VERBOSE).toBeUndefined();
|
||||
});
|
||||
|
||||
consoleLogSpy.mockClear();
|
||||
});
|
||||
+56
-10
@@ -25,7 +25,11 @@ async function getOperatePage(): Promise<import('./types.js').IPage> {
|
||||
return bridge.connect({ timeout: 30, workspace: 'operate:default' });
|
||||
}
|
||||
|
||||
export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
function applyVerbose(opts: { verbose?: boolean }): void {
|
||||
if (opts.verbose) process.env.OPENCLI_VERBOSE = '1';
|
||||
}
|
||||
|
||||
export function createProgram(BUILTIN_CLIS: string, USER_CLIS: string): Command {
|
||||
const program = new Command();
|
||||
// enablePositionalOptions: prevents parent from consuming flags meant for subcommands;
|
||||
// prerequisite for passThroughOptions to forward --help/--version to external binaries
|
||||
@@ -145,7 +149,16 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.option('--wait <s>', '', '3')
|
||||
.option('--auto', 'Enable interactive fuzzing')
|
||||
.option('--click <labels>', 'Comma-separated labels to click before fuzzing')
|
||||
.action(async (url, opts) => {
|
||||
.option('-v, --verbose', 'Debug output')
|
||||
.action(async (url: string, opts: {
|
||||
site?: string;
|
||||
goal?: string;
|
||||
wait: string;
|
||||
auto?: boolean;
|
||||
click?: string;
|
||||
verbose?: boolean;
|
||||
}) => {
|
||||
applyVerbose(opts);
|
||||
const { exploreUrl, renderExploreSummary } = await import('./explore.js');
|
||||
const clickLabels = opts.click
|
||||
? opts.click.split(',').map((s: string) => s.trim())
|
||||
@@ -168,7 +181,9 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.description('Synthesize CLIs from explore')
|
||||
.argument('<target>')
|
||||
.option('--top <n>', '', '3')
|
||||
.option('-v, --verbose', 'Debug output')
|
||||
.action(async (target, opts) => {
|
||||
applyVerbose(opts);
|
||||
const { synthesizeFromExplore, renderSynthesizeSummary } = await import('./synthesize.js');
|
||||
console.log(renderSynthesizeSummary(synthesizeFromExplore(target, { top: parseInt(opts.top) })));
|
||||
});
|
||||
@@ -179,7 +194,13 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.argument('<url>')
|
||||
.option('--goal <text>')
|
||||
.option('--site <name>')
|
||||
.action(async (url, opts) => {
|
||||
.option('-v, --verbose', 'Debug output')
|
||||
.action(async (url: string, opts: {
|
||||
goal?: string;
|
||||
site?: string;
|
||||
verbose?: boolean;
|
||||
}) => {
|
||||
applyVerbose(opts);
|
||||
const { generateCliFromUrl, renderGenerateSummary } = await import('./generate.js');
|
||||
const workspace = `generate:${inferHost(url, opts.site)}`;
|
||||
const r = await generateCliFromUrl({
|
||||
@@ -203,7 +224,15 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.option('--out <dir>', 'Output directory for candidates')
|
||||
.option('--poll <ms>', 'Poll interval in milliseconds', '2000')
|
||||
.option('--timeout <ms>', 'Auto-stop after N milliseconds (default: 60000)', '60000')
|
||||
.action(async (url, opts) => {
|
||||
.option('-v, --verbose', 'Debug output')
|
||||
.action(async (url: string, opts: {
|
||||
site?: string;
|
||||
out?: string;
|
||||
poll: string;
|
||||
timeout: string;
|
||||
verbose?: boolean;
|
||||
}) => {
|
||||
applyVerbose(opts);
|
||||
const { recordSession, renderRecordSummary } = await import('./record.js');
|
||||
const result = await recordSession({
|
||||
BrowserFactory: getBrowserFactory(),
|
||||
@@ -222,7 +251,12 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
.description('Strategy cascade: find simplest working strategy')
|
||||
.argument('<url>')
|
||||
.option('--site <name>')
|
||||
.action(async (url, opts) => {
|
||||
.option('-v, --verbose', 'Debug output')
|
||||
.action(async (url: string, opts: {
|
||||
site?: string;
|
||||
verbose?: boolean;
|
||||
}) => {
|
||||
applyVerbose(opts);
|
||||
const { cascadeProbe, renderCascadeResult } = await import('./cascade.js');
|
||||
const workspace = `cascade:${inferHost(url, opts.site)}`;
|
||||
const result = await browserSession(getBrowserFactory(), async (page) => {
|
||||
@@ -302,7 +336,7 @@ export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
|
||||
operate.command('state').description('Page state: URL, title, interactive elements with [N] indices')
|
||||
.action(operateAction(async (page) => {
|
||||
const snapshot = await page.snapshot({ viewportExpand: 800 });
|
||||
const snapshot = await page.snapshot({ viewportExpand: 2000 });
|
||||
const url = await page.getCurrentUrl?.() ?? '';
|
||||
console.log(`URL: ${url}\n`);
|
||||
console.log(typeof snapshot === 'string' ? snapshot : JSON.stringify(snapshot, null, 2));
|
||||
@@ -605,19 +639,25 @@ cli({
|
||||
console.log(`🔍 Verifying ${name}...\n`);
|
||||
console.log(` Loading: ${filePath}`);
|
||||
|
||||
// Read adapter to check if it defines a 'limit' arg
|
||||
const adapterSrc = fs.readFileSync(filePath, 'utf-8');
|
||||
const hasLimitArg = /['"]limit['"]/.test(adapterSrc);
|
||||
const limitFlag = hasLimitArg ? ' --limit 3' : '';
|
||||
const verifyCmd = `node dist/main.js ${site} ${command}${limitFlag}`;
|
||||
|
||||
try {
|
||||
const output = execSync(`node dist/main.js ${site} ${command} --limit 3`, {
|
||||
const output = execSync(verifyCmd, {
|
||||
cwd: path.join(path.dirname(import.meta.url.replace('file://', '')), '..'),
|
||||
timeout: 30000,
|
||||
encoding: 'utf-8',
|
||||
env: process.env,
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
});
|
||||
console.log(` Executing: opencli ${site} ${command} --limit 3\n`);
|
||||
console.log(` Executing: opencli ${site} ${command}${limitFlag}\n`);
|
||||
console.log(output);
|
||||
console.log(`\n ✓ Adapter works!`);
|
||||
} catch (err: any) {
|
||||
console.log(` Executing: opencli ${site} ${command} --limit 3\n`);
|
||||
console.log(` Executing: opencli ${site} ${command}${limitFlag}\n`);
|
||||
if (err.stdout) console.log(err.stdout);
|
||||
if (err.stderr) console.error(err.stderr.slice(0, 500));
|
||||
console.log(`\n ✗ Adapter failed. Fix the code and try again.`);
|
||||
@@ -644,7 +684,9 @@ cli({
|
||||
.description('Diagnose opencli browser bridge connectivity')
|
||||
.option('--no-live', 'Skip live browser connectivity test')
|
||||
.option('--sessions', 'Show active automation sessions', false)
|
||||
.option('-v, --verbose', 'Debug output')
|
||||
.action(async (opts) => {
|
||||
applyVerbose(opts);
|
||||
const { runBrowserDoctor, renderBrowserDoctorReport } = await import('./doctor.js');
|
||||
const report = await runBrowserDoctor({ live: opts.live, sessions: opts.sessions, cliVersion: PKG_VERSION });
|
||||
console.log(renderBrowserDoctorReport(report));
|
||||
@@ -954,7 +996,11 @@ cli({
|
||||
process.exitCode = EXIT_CODES.USAGE_ERROR;
|
||||
});
|
||||
|
||||
program.parse();
|
||||
return program;
|
||||
}
|
||||
|
||||
export function runCli(BUILTIN_CLIS: string, USER_CLIS: string): void {
|
||||
createProgram(BUILTIN_CLIS, USER_CLIS).parse();
|
||||
}
|
||||
|
||||
// ── Helpers ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './item.js';
|
||||
|
||||
describe('1688 item normalization', () => {
|
||||
it('normalizes public item payload into contract fields', () => {
|
||||
const result = __test__.normalizeItemPayload({
|
||||
href: 'https://detail.1688.com/offer/887904326744.html',
|
||||
title: '法式春季长袖开衫连衣裙女新款大码女装碎花吊带裙套装142077 - 阿里巴巴',
|
||||
bodyText: `
|
||||
青岛沁澜衣品服装有限公司
|
||||
入驻13年
|
||||
主营:大码女装
|
||||
店铺回头率
|
||||
87%
|
||||
山东青岛
|
||||
3套起批
|
||||
已售1600+套
|
||||
支持定制logo
|
||||
`,
|
||||
offerTitle: '法式春季长袖开衫连衣裙女新款大码女装碎花吊带裙套装142077',
|
||||
offerId: 887904326744,
|
||||
seller: {
|
||||
companyName: '青岛沁澜衣品服装有限公司',
|
||||
memberId: 'b2b-1641351767',
|
||||
winportUrl: 'https://yinuoweierfushi.1688.com/page/index.html?spm=a1',
|
||||
},
|
||||
trade: {
|
||||
beginAmount: 3,
|
||||
priceDisplay: '96.00-98.00',
|
||||
unit: '套',
|
||||
saleCount: 1655,
|
||||
offerIDatacenterSellInfo: {
|
||||
面料名称: '莫代尔',
|
||||
主面料成分: '莫代尔纤维',
|
||||
sellPointModel: '{"ignore":true}',
|
||||
},
|
||||
offerPriceModel: {
|
||||
currentPrices: [
|
||||
{ beginAmount: 3, price: '98.00' },
|
||||
{ beginAmount: 50, price: '97.00' },
|
||||
],
|
||||
},
|
||||
},
|
||||
gallery: {
|
||||
mainImage: ['https://example.com/1.jpg'],
|
||||
offerImgList: ['https://example.com/2.jpg'],
|
||||
wlImageInfos: [{ fullPathImageURI: 'https://example.com/3.jpg' }],
|
||||
},
|
||||
services: [
|
||||
{ serviceName: '延期必赔', agreeDeliveryHours: 360 },
|
||||
{ serviceName: '品质保障' },
|
||||
],
|
||||
});
|
||||
|
||||
expect(result.offer_id).toBe('887904326744');
|
||||
expect(result.member_id).toBe('b2b-1641351767');
|
||||
expect(result.shop_id).toBe('yinuoweierfushi');
|
||||
expect(result.seller_url).toBe('https://yinuoweierfushi.1688.com');
|
||||
expect(result.price_text).toBe('¥96.00-98.00');
|
||||
expect(result.moq_text).toBe('3套起批');
|
||||
expect(result.origin_place).toBe('山东青岛');
|
||||
expect(result.delivery_days_text).toBe('360小时内发货');
|
||||
expect(result.private_label_text).toBe('支持定制logo');
|
||||
expect(result.visible_attributes).toEqual([
|
||||
{ key: '面料名称', value: '莫代尔' },
|
||||
{ key: '主面料成分', value: '莫代尔纤维' },
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,282 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { isRecord } from '../../utils.js';
|
||||
import {
|
||||
assertAuthenticatedState,
|
||||
buildDetailUrl,
|
||||
buildProvenance,
|
||||
canonicalizeSellerUrl,
|
||||
cleanMultilineText,
|
||||
cleanText,
|
||||
extractLocation,
|
||||
extractMemberId,
|
||||
extractOfferId,
|
||||
extractShopId,
|
||||
gotoAndReadState,
|
||||
normalizePriceTiers,
|
||||
parseMoqText,
|
||||
parsePriceText,
|
||||
toNumber,
|
||||
uniqueNonEmpty,
|
||||
} from './shared.js';
|
||||
|
||||
interface BuyerProtectionModel {
|
||||
serviceName?: string;
|
||||
shortBuyerDesc?: string;
|
||||
packageBuyerDesc?: string;
|
||||
textDesc?: string;
|
||||
agreeDeliveryHours?: number;
|
||||
}
|
||||
|
||||
interface ItemBrowserPayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
bodyText?: string;
|
||||
offerTitle?: string;
|
||||
offerId?: string | number;
|
||||
seller?: {
|
||||
companyName?: string;
|
||||
memberId?: string;
|
||||
winportUrl?: string;
|
||||
sellerWinportUrlMap?: Record<string, string>;
|
||||
};
|
||||
trade?: {
|
||||
beginAmount?: string | number;
|
||||
priceDisplay?: string;
|
||||
unit?: string;
|
||||
saleCount?: string | number;
|
||||
offerIDatacenterSellInfo?: Record<string, unknown>;
|
||||
offerPriceModel?: {
|
||||
currentPrices?: Array<{ beginAmount?: string | number; price?: string | number }>;
|
||||
};
|
||||
};
|
||||
gallery?: {
|
||||
mainImage?: string[];
|
||||
offerImgList?: string[];
|
||||
wlImageInfos?: Array<{ fullPathImageURI?: string }>;
|
||||
};
|
||||
shipping?: {
|
||||
deliveryLimitText?: string;
|
||||
logisticsText?: string;
|
||||
protectionInfos?: BuyerProtectionModel[];
|
||||
buyerProtectionModel?: BuyerProtectionModel[];
|
||||
};
|
||||
services?: BuyerProtectionModel[];
|
||||
}
|
||||
|
||||
interface VisibleAttribute {
|
||||
key: string;
|
||||
value: string;
|
||||
}
|
||||
|
||||
function normalizeItemPayload(payload: ItemBrowserPayload): Record<string, unknown> {
|
||||
const href = cleanText(payload.href);
|
||||
const bodyText = cleanMultilineText(payload.bodyText);
|
||||
const sellerName = cleanText(payload.seller?.companyName);
|
||||
const sellerUrlRaw = cleanText(
|
||||
payload.seller?.winportUrl
|
||||
?? payload.seller?.sellerWinportUrlMap?.defaultUrl
|
||||
?? payload.seller?.sellerWinportUrlMap?.indexUrl,
|
||||
);
|
||||
const sellerUrl = canonicalizeSellerUrl(sellerUrlRaw);
|
||||
const offerId = cleanText(String(payload.offerId ?? '')) || extractOfferId(href) || null;
|
||||
const memberId = cleanText(payload.seller?.memberId) || extractMemberId(sellerUrlRaw || href) || null;
|
||||
const shopId = extractShopId(sellerUrl ?? href);
|
||||
const unit = cleanText(payload.trade?.unit);
|
||||
const priceDisplay = cleanText(payload.trade?.priceDisplay);
|
||||
const priceRange = parsePriceText(priceDisplay ? `¥${priceDisplay}` : bodyText);
|
||||
const moqText = extractMoqText(bodyText, payload.trade?.beginAmount, unit);
|
||||
const moq = parseMoqText(moqText);
|
||||
const services = uniqueServices(payload);
|
||||
const serviceBadges = uniqueNonEmpty(services.map((service) => cleanText(service.serviceName)));
|
||||
const attributes = normalizeVisibleAttributes(payload.trade?.offerIDatacenterSellInfo);
|
||||
const priceTiers = normalizePriceTiers(payload.trade?.offerPriceModel?.currentPrices ?? [], unit || null);
|
||||
const images = uniqueNonEmpty([
|
||||
...(payload.gallery?.mainImage ?? []),
|
||||
...(payload.gallery?.offerImgList ?? []),
|
||||
...((payload.gallery?.wlImageInfos ?? []).map((item) => item.fullPathImageURI ?? '')),
|
||||
]);
|
||||
|
||||
const detailUrl = offerId ? buildDetailUrl(offerId) : href;
|
||||
const provenance = buildProvenance(href || detailUrl);
|
||||
|
||||
return {
|
||||
offer_id: offerId,
|
||||
member_id: memberId,
|
||||
shop_id: shopId,
|
||||
title: cleanText(payload.offerTitle) || stripAlibabaSuffix(payload.title) || firstNonEmptyLine(bodyText) || null,
|
||||
item_url: detailUrl,
|
||||
main_images: images,
|
||||
price_text: priceRange.price_text || null,
|
||||
price_tiers: priceTiers,
|
||||
currency: priceRange.currency,
|
||||
moq_text: moq.moq_text || null,
|
||||
moq_value: moq.moq_value,
|
||||
seller_name: sellerName || null,
|
||||
seller_url: sellerUrl,
|
||||
shop_name: sellerName || null,
|
||||
origin_place: extractLocation(bodyText),
|
||||
delivery_days_text: extractDeliveryDaysText(bodyText, services, payload.shipping),
|
||||
customization_text: extractKeywordLine(bodyText, ['来样定制', '来图定制', '支持定制', '可定制', '定制']),
|
||||
private_label_text: extractKeywordLine(bodyText, ['贴牌', '贴标', '定制logo', '打logo', 'OEM', 'ODM']),
|
||||
visible_attributes: attributes,
|
||||
sales_text: extractSalesText(bodyText),
|
||||
service_badges: serviceBadges,
|
||||
stock_quantity: extractStockQuantity(bodyText),
|
||||
...provenance,
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeVisibleAttributes(raw: unknown): VisibleAttribute[] {
|
||||
if (!isRecord(raw)) return [];
|
||||
return Object.entries(raw)
|
||||
.filter(([key, value]) => key !== 'sellPointModel' && cleanText(key) && cleanText(String(value)))
|
||||
.map(([key, value]) => ({ key: cleanText(key), value: cleanText(String(value)) }));
|
||||
}
|
||||
|
||||
function uniqueServices(payload: ItemBrowserPayload): BuyerProtectionModel[] {
|
||||
const combined = [
|
||||
...(Array.isArray(payload.services) ? payload.services : []),
|
||||
...(Array.isArray(payload.shipping?.protectionInfos) ? payload.shipping.protectionInfos : []),
|
||||
...(Array.isArray(payload.shipping?.buyerProtectionModel) ? payload.shipping.buyerProtectionModel : []),
|
||||
];
|
||||
|
||||
const seen = new Set<string>();
|
||||
const result: BuyerProtectionModel[] = [];
|
||||
for (const service of combined) {
|
||||
const key = cleanText(service.serviceName);
|
||||
if (!key || seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
result.push(service);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function stripAlibabaSuffix(title: string | undefined): string {
|
||||
return cleanText(title).replace(/\s*-\s*阿里巴巴$/, '').trim();
|
||||
}
|
||||
|
||||
function firstNonEmptyLine(text: string): string {
|
||||
return text.split('\n').map((line) => cleanText(line)).find(Boolean) ?? '';
|
||||
}
|
||||
|
||||
function extractMoqText(bodyText: string, beginAmount: string | number | undefined, unit: string): string {
|
||||
const lineMatch = bodyText.match(/\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)\s*起批/);
|
||||
if (lineMatch) return lineMatch[0];
|
||||
|
||||
const moqValue = toNumber(beginAmount);
|
||||
if (moqValue !== null) {
|
||||
return `${moqValue}${unit || ''}起批`;
|
||||
}
|
||||
|
||||
return '';
|
||||
}
|
||||
|
||||
function extractDeliveryDaysText(
|
||||
bodyText: string,
|
||||
services: BuyerProtectionModel[],
|
||||
shipping: ItemBrowserPayload['shipping'],
|
||||
): string | null {
|
||||
const shippingText = cleanText(shipping?.deliveryLimitText) || cleanText(shipping?.logisticsText);
|
||||
if (shippingText) return shippingText;
|
||||
|
||||
const textMatch = bodyText.match(/\d+\s*(?:小时|天)(?:内)?发货/);
|
||||
if (textMatch) return textMatch[0];
|
||||
|
||||
const hourMatch = services.find((service) => typeof service.agreeDeliveryHours === 'number');
|
||||
if (hourMatch && typeof hourMatch.agreeDeliveryHours === 'number') {
|
||||
return `${hourMatch.agreeDeliveryHours}小时内发货`;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function extractKeywordLine(bodyText: string, keywords: string[]): string | null {
|
||||
const lines = bodyText.split('\n').map((line) => cleanText(line)).filter(Boolean);
|
||||
for (const line of lines) {
|
||||
if (keywords.some((keyword) => line.includes(keyword))) {
|
||||
return line;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function extractSalesText(bodyText: string): string | null {
|
||||
const match = bodyText.match(/(?:全网销量|已售)\s*\d+(?:\.\d+)?\+?\s*[件套个单]?/);
|
||||
return match ? cleanText(match[0]) : null;
|
||||
}
|
||||
|
||||
function extractStockQuantity(bodyText: string): number | null {
|
||||
const match = bodyText.match(/库存\s*(\d+)/);
|
||||
return match ? Number.parseInt(match[1], 10) : null;
|
||||
}
|
||||
|
||||
async function readItemPayload(page: IPage, itemUrl: string): Promise<ItemBrowserPayload> {
|
||||
const state = await gotoAndReadState(page, itemUrl, 2500, 'item');
|
||||
assertAuthenticatedState(state, 'item');
|
||||
|
||||
const payload = await page.evaluate(`
|
||||
(() => {
|
||||
const root = window.context ?? {};
|
||||
const model = root.result?.global?.globalData?.model ?? null;
|
||||
const toJson = (value) => JSON.parse(JSON.stringify(value ?? null));
|
||||
return {
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
bodyText: document.body ? document.body.innerText || '' : '',
|
||||
offerTitle: model?.offerTitleModel?.subject ?? '',
|
||||
offerId: model?.tradeModel?.offerId ?? '',
|
||||
seller: toJson(model?.sellerModel),
|
||||
trade: toJson(model?.tradeModel),
|
||||
gallery: toJson(root.result?.data?.gallery?.fields ?? null),
|
||||
shipping: toJson(root.result?.data?.shippingServices?.fields ?? null),
|
||||
services: toJson(root.result?.data?.shippingServices?.fields?.protectionInfos ?? []),
|
||||
};
|
||||
})()
|
||||
`) as ItemBrowserPayload;
|
||||
|
||||
const resolvedOfferId = cleanText(String(payload.offerId ?? '')) || extractOfferId(cleanText(payload.href));
|
||||
if (!resolvedOfferId) {
|
||||
throw new CommandExecutionError(
|
||||
'1688 item page did not expose product context',
|
||||
'当前 tab 非商品详情上下文,请切到 detail.1688.com 商品页并重试',
|
||||
);
|
||||
}
|
||||
|
||||
return payload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: '1688',
|
||||
name: 'item',
|
||||
description: '1688 商品详情(公开商品字段、价格阶梯、卖家基础信息)',
|
||||
domain: 'www.1688.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '1688 商品 URL 或 offer ID(如 887904326744)',
|
||||
},
|
||||
],
|
||||
columns: ['offer_id', 'title', 'price_text', 'moq_text', 'seller_name', 'origin_place'],
|
||||
func: async (page, kwargs) => {
|
||||
const itemUrl = buildDetailUrl(String(kwargs.input ?? ''));
|
||||
const payload = await readItemPayload(page, itemUrl);
|
||||
return [normalizeItemPayload(payload)];
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeItemPayload,
|
||||
normalizeVisibleAttributes,
|
||||
stripAlibabaSuffix,
|
||||
extractMoqText,
|
||||
extractDeliveryDaysText,
|
||||
extractKeywordLine,
|
||||
extractSalesText,
|
||||
extractStockQuantity,
|
||||
};
|
||||
@@ -0,0 +1,81 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './search.js';
|
||||
|
||||
describe('1688 search normalization', () => {
|
||||
it('normalizes search candidates into structured result rows', () => {
|
||||
const result = __test__.normalizeSearchCandidate({
|
||||
item_url: 'https://detail.1688.com/offer/887904326744.html',
|
||||
title: '宿舍置物架桌面加高架',
|
||||
container_text: '宿舍置物架桌面加高架 ¥56.00 2套起批 山东青岛 已售300+套',
|
||||
price_text: '¥ 56 .00',
|
||||
sales_text: '300+套',
|
||||
moq_text: '2套起批',
|
||||
tag_items: ['退货包运费', '回头率52%'],
|
||||
hover_items: ['验厂报告'],
|
||||
seller_name: '青岛沁澜衣品服装有限公司',
|
||||
seller_url: 'https://yinuoweierfushi.1688.com/page/index.html?spm=a123',
|
||||
}, 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=置物架');
|
||||
|
||||
expect(result.rank).toBe(0);
|
||||
expect(result.offer_id).toBe('887904326744');
|
||||
expect(result.shop_id).toBe('yinuoweierfushi');
|
||||
expect(result.item_url).toBe('https://detail.1688.com/offer/887904326744.html');
|
||||
expect(result.seller_url).toBe('https://yinuoweierfushi.1688.com');
|
||||
expect(result.price_text).toBe('¥56.00');
|
||||
expect(result.price_min).toBe(56);
|
||||
expect(result.price_max).toBe(56);
|
||||
expect(result.moq_value).toBe(2);
|
||||
expect(result.location).toBe('山东青岛');
|
||||
expect(result.sales_text).toBe('300+套');
|
||||
expect(result.badges).toEqual(expect.arrayContaining(['退货包运费', '验厂报告']));
|
||||
expect(result.return_rate_text).toBe('回头率52%');
|
||||
});
|
||||
|
||||
it('does not use hover_price_text as MOQ source', () => {
|
||||
const result = __test__.normalizeSearchCandidate({
|
||||
item_url: 'https://detail.1688.com/offer/887904326744.html',
|
||||
title: 'test',
|
||||
container_text: 'test ¥56.00',
|
||||
price_text: '¥ 56 .00',
|
||||
hover_price_text: '¥56.00 3件起批',
|
||||
moq_text: null,
|
||||
}, 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=test');
|
||||
// hover_price_text should not be used for MOQ extraction
|
||||
expect(result.moq_text).toBeNull();
|
||||
expect(result.moq_value).toBeNull();
|
||||
});
|
||||
|
||||
it('extracts offer id from mobile detail search links', () => {
|
||||
const result = __test__.normalizeSearchCandidate({
|
||||
item_url: 'http://detail.m.1688.com/page/index.html?offerId=910933345396&sortType=&pageId=',
|
||||
title: '',
|
||||
container_text: '桌面书桌办公室工位收纳展示新中式博古架多层茶具厨房摆放置物架 ¥24.3 已售20+件',
|
||||
price_text: '¥ 14 .28',
|
||||
sales_text: '1500+件',
|
||||
moq_text: '≥2个',
|
||||
seller_name: '泰商国际贸易(宁阳)有限公司',
|
||||
seller_url: 'http://tsgjmy.1688.com/',
|
||||
}, 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=桌面置物架');
|
||||
|
||||
expect(result.offer_id).toBe('910933345396');
|
||||
expect(result.shop_id).toBe('tsgjmy');
|
||||
expect(result.item_url).toBe('https://detail.1688.com/offer/910933345396.html');
|
||||
expect(result.title).toContain('桌面书桌办公室工位收纳展示');
|
||||
expect(result.price_text).toBe('¥14.28');
|
||||
expect(result.sales_text).toBe('1500+件');
|
||||
expect(result.moq_text).toBe('≥2个');
|
||||
expect(result.moq_value).toBe(2);
|
||||
});
|
||||
|
||||
it('prefers offer id and falls back to item url for dedupe key', () => {
|
||||
expect(__test__.buildDedupeKey({
|
||||
offer_id: '123456',
|
||||
item_url: 'https://detail.1688.com/offer/123456.html',
|
||||
})).toBe('offer:123456');
|
||||
expect(__test__.buildDedupeKey({
|
||||
offer_id: null,
|
||||
item_url: 'https://detail.1688.com/offer/123456.html',
|
||||
})).toBe('url:https://detail.1688.com/offer/123456.html');
|
||||
expect(__test__.buildDedupeKey({ offer_id: null, item_url: null })).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,402 @@
|
||||
import { CommandExecutionError, EmptyResultError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
FACTORY_BADGE_PATTERNS,
|
||||
SERVICE_BADGE_PATTERNS,
|
||||
assertAuthenticatedState,
|
||||
buildProvenance,
|
||||
buildSearchUrl,
|
||||
canonicalizeItemUrl,
|
||||
canonicalizeSellerUrl,
|
||||
cleanText,
|
||||
extractBadges,
|
||||
extractLocation,
|
||||
extractMemberId,
|
||||
extractOfferId,
|
||||
extractShopId,
|
||||
gotoAndReadState,
|
||||
parseMoqText,
|
||||
parsePriceText,
|
||||
SEARCH_LIMIT_DEFAULT,
|
||||
SEARCH_LIMIT_MAX,
|
||||
parseSearchLimit,
|
||||
uniqueNonEmpty,
|
||||
} from './shared.js';
|
||||
|
||||
interface SearchPayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
bodyText?: string;
|
||||
next_url?: string;
|
||||
candidates?: Array<{
|
||||
item_url?: string;
|
||||
title?: string;
|
||||
container_text?: string;
|
||||
desc_rows?: string[];
|
||||
price_text?: string | null;
|
||||
sales_text?: string | null;
|
||||
hover_price_text?: string | null;
|
||||
moq_text?: string | null;
|
||||
tag_items?: string[];
|
||||
hover_items?: string[];
|
||||
seller_name?: string | null;
|
||||
seller_url?: string | null;
|
||||
}>;
|
||||
}
|
||||
|
||||
interface SearchRow {
|
||||
rank: number;
|
||||
offer_id: string | null;
|
||||
member_id: string | null;
|
||||
shop_id: string | null;
|
||||
title: string | null;
|
||||
item_url: string | null;
|
||||
seller_name: string | null;
|
||||
seller_url: string | null;
|
||||
price_text: string | null;
|
||||
price_min: number | null;
|
||||
price_max: number | null;
|
||||
currency: string | null;
|
||||
moq_text: string | null;
|
||||
moq_value: number | null;
|
||||
location: string | null;
|
||||
badges: string[];
|
||||
sales_text: string | null;
|
||||
return_rate_text: string | null;
|
||||
source_url: string;
|
||||
fetched_at: string;
|
||||
strategy: string;
|
||||
}
|
||||
|
||||
const SEARCH_ITEM_URL_PATTERNS = [
|
||||
'detail.1688.com/offer/',
|
||||
'detail.m.1688.com/page/index.html?offerId=',
|
||||
];
|
||||
const MAX_SEARCH_PAGES = 12;
|
||||
|
||||
function normalizeSearchCandidate(
|
||||
candidate: NonNullable<SearchPayload['candidates']>[number],
|
||||
sourceUrl: string,
|
||||
): SearchRow {
|
||||
const canonicalItemUrl = canonicalizeItemUrl(cleanText(candidate.item_url));
|
||||
const containerText = cleanText(candidate.container_text);
|
||||
const priceText = firstNonEmpty([
|
||||
normalizeInlineText(candidate.price_text),
|
||||
normalizeInlineText(extractPriceText(candidate.hover_price_text)),
|
||||
]);
|
||||
const priceRange = parsePriceText(priceText || containerText);
|
||||
const moq = parseMoqText(firstNonEmpty([
|
||||
normalizeInlineText(candidate.moq_text),
|
||||
normalizeInlineText(extractMoqText(containerText)),
|
||||
]));
|
||||
const canonicalSellerUrl = canonicalizeSellerUrl(cleanText(candidate.seller_url));
|
||||
const evidenceText = uniqueNonEmpty([
|
||||
containerText,
|
||||
...(candidate.desc_rows ?? []),
|
||||
...(candidate.tag_items ?? []),
|
||||
...(candidate.hover_items ?? []),
|
||||
]).join('\n');
|
||||
const badges = extractBadges(evidenceText, [...FACTORY_BADGE_PATTERNS, ...SERVICE_BADGE_PATTERNS]);
|
||||
const salesText = firstNonEmpty([
|
||||
extractSalesText(candidate.sales_text),
|
||||
extractSalesText(containerText),
|
||||
]);
|
||||
const returnRateText = extractReturnRateText([...(candidate.tag_items ?? []), ...(candidate.hover_items ?? [])]);
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
rank: 0,
|
||||
offer_id: extractOfferId(canonicalItemUrl ?? '') ?? null,
|
||||
member_id: extractMemberId(canonicalSellerUrl ?? '') ?? null,
|
||||
shop_id: extractShopId(canonicalSellerUrl ?? '') ?? null,
|
||||
title: cleanText(candidate.title) || firstWord(containerText) || null,
|
||||
item_url: canonicalItemUrl,
|
||||
seller_name: cleanText(candidate.seller_name) || null,
|
||||
seller_url: canonicalSellerUrl,
|
||||
price_text: priceRange.price_text || null,
|
||||
price_min: priceRange.price_min,
|
||||
price_max: priceRange.price_max,
|
||||
currency: priceRange.currency,
|
||||
moq_text: moq.moq_text || null,
|
||||
moq_value: moq.moq_value,
|
||||
location: extractLocation(containerText),
|
||||
badges,
|
||||
sales_text: salesText || null,
|
||||
return_rate_text: returnRateText,
|
||||
source_url: provenance.source_url,
|
||||
fetched_at: provenance.fetched_at,
|
||||
strategy: provenance.strategy,
|
||||
};
|
||||
}
|
||||
|
||||
function extractMoqText(text: string | null | undefined): string {
|
||||
const normalized = normalizeInlineText(text);
|
||||
return normalized.match(/\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)\s*起批/i)?.[0]
|
||||
?? normalized.match(/≥\s*\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)?/i)?.[0]
|
||||
?? normalized.match(/\d+(?:\.\d+)?\s*(?:~|-|至|到)\s*\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)/i)?.[0]
|
||||
?? '';
|
||||
}
|
||||
|
||||
function extractPriceText(text: string | null | undefined): string {
|
||||
const normalized = normalizeInlineText(text);
|
||||
return normalized.match(/[¥$€]\s*\d+(?:\.\d+)?/)?.[0] ?? '';
|
||||
}
|
||||
|
||||
function extractSalesText(text: string | null | undefined): string {
|
||||
const normalized = normalizeInlineText(text);
|
||||
if (!normalized) return '';
|
||||
if (/^\d+(?:\.\d+)?\+?\s*(件|套|个|单)$/.test(normalized)) {
|
||||
return normalized;
|
||||
}
|
||||
const match = normalized.match(/(?:已售|销量|售)\s*\d+(?:\.\d+)?\+?\s*(件|套|个|单)?/);
|
||||
return match ? cleanText(match[0]) : '';
|
||||
}
|
||||
|
||||
function firstWord(text: string): string {
|
||||
return text.split(/\s+/).find(Boolean) ?? '';
|
||||
}
|
||||
|
||||
function firstNonEmpty(values: Array<string | null | undefined>): string {
|
||||
return values.map((value) => cleanText(value)).find(Boolean) ?? '';
|
||||
}
|
||||
|
||||
function normalizeInlineText(text: string | null | undefined): string {
|
||||
return cleanText(text)
|
||||
.replace(/([¥$€])\s+(?=\d)/g, '$1')
|
||||
.replace(/(\d)\s*\.\s*(\d)/g, '$1.$2')
|
||||
.replace(/\s*([~-])\s*/g, '$1')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function extractReturnRateText(values: string[]): string | null {
|
||||
return uniqueNonEmpty(values.map((value) => normalizeInlineText(value)))
|
||||
.find((value) => /^回头率\s*\d+(?:\.\d+)?%$/.test(value))
|
||||
?? null;
|
||||
}
|
||||
|
||||
function buildDedupeKey(row: Pick<SearchRow, 'offer_id' | 'item_url'>): string | null {
|
||||
if (row.offer_id) return `offer:${row.offer_id}`;
|
||||
if (row.item_url) return `url:${row.item_url}`;
|
||||
return null;
|
||||
}
|
||||
|
||||
async function readSearchPayload(page: IPage, url: string): Promise<SearchPayload> {
|
||||
const state = await gotoAndReadState(page, url, 2500, 'search');
|
||||
assertAuthenticatedState(state, 'search');
|
||||
|
||||
const payload = await page.evaluate(`
|
||||
(() => {
|
||||
const normalizeText = (value) => (value || '').replace(/\\s+/g, ' ').trim();
|
||||
const normalizeUrl = (href) => {
|
||||
if (!href) return '';
|
||||
try {
|
||||
return new URL(href, window.location.href).toString();
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
};
|
||||
const isItemHref = (href) => ${JSON.stringify(SEARCH_ITEM_URL_PATTERNS)}
|
||||
.some((pattern) => (href || '').includes(pattern));
|
||||
const uniqueTexts = (values) => [...new Set(values.map((value) => normalizeText(value)).filter(Boolean))];
|
||||
const collectTexts = (root, selector) => uniqueTexts(
|
||||
Array.from(root.querySelectorAll(selector)).map((node) => node.innerText || node.textContent || ''),
|
||||
);
|
||||
const firstText = (root, selectors) => {
|
||||
for (const selector of selectors) {
|
||||
const node = root.querySelector(selector);
|
||||
const value = normalizeText(node ? node.innerText || node.textContent || '' : '');
|
||||
if (value) return value;
|
||||
}
|
||||
return '';
|
||||
};
|
||||
const findMoqText = (values, priceText) => {
|
||||
const moqPattern = /(≥\\s*\\d+(?:\\.\\d+)?\\s*(件|个|套|箱|包|双|台|把|只)?)|(\\d+(?:\\.\\d+)?\\s*(?:~|-|至|到)\\s*\\d+(?:\\.\\d+)?\\s*(件|个|套|箱|包|双|台|把|只))|(\\d+(?:\\.\\d+)?\\s*(件|个|套|箱|包|双|台|把|只)\\s*起批)/i;
|
||||
return values.find((value) => moqPattern.test(value))
|
||||
|| normalizeText(priceText).match(moqPattern)?.[0]
|
||||
|| '';
|
||||
};
|
||||
const isSellerHref = (href) => {
|
||||
if (!href) return false;
|
||||
try {
|
||||
const url = new URL(href, window.location.href);
|
||||
const host = url.hostname || '';
|
||||
if (!host.endsWith('.1688.com')) return false;
|
||||
if (
|
||||
host === 's.1688.com'
|
||||
|| host === 'r.1688.com'
|
||||
|| host === 'air.1688.com'
|
||||
|| host === 'detail.1688.com'
|
||||
|| host === 'detail.m.1688.com'
|
||||
|| host === 'dj.1688.com'
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
};
|
||||
const pickContainer = (anchor) => {
|
||||
let node = anchor;
|
||||
while (node && node !== document.body) {
|
||||
const text = normalizeText(node.innerText || node.textContent || '');
|
||||
if (text.length >= 40 && text.length <= 2000) {
|
||||
return node;
|
||||
}
|
||||
node = node.parentElement;
|
||||
}
|
||||
return anchor;
|
||||
};
|
||||
const collectCandidates = () => {
|
||||
const anchors = Array.from(document.querySelectorAll('a')).filter((anchor) => isItemHref(anchor.href || ''));
|
||||
const seen = new Set();
|
||||
const items = [];
|
||||
for (const anchor of anchors) {
|
||||
const href = anchor.href || '';
|
||||
if (!href || seen.has(href)) continue;
|
||||
seen.add(href);
|
||||
|
||||
const container = pickContainer(anchor);
|
||||
const tagItems = collectTexts(container, '.offer-tag-row .offer-desc-item');
|
||||
const hoverItems = collectTexts(container, '.offer-hover-wrapper .offer-desc-item');
|
||||
const sellerAnchor = Array.from(container.querySelectorAll('a'))
|
||||
.find((link) => isSellerHref(link.href || ''));
|
||||
const hoverPriceText = firstText(container, [
|
||||
'.offer-hover-wrapper .hover-price-item',
|
||||
'.offer-hover-wrapper .price-item',
|
||||
]);
|
||||
|
||||
items.push({
|
||||
item_url: href,
|
||||
title: firstText(container, ['.offer-title-row .title-text', '.offer-title-row'])
|
||||
|| normalizeText(anchor.innerText || anchor.textContent || ''),
|
||||
container_text: normalizeText(container.innerText || container.textContent || ''),
|
||||
desc_rows: collectTexts(container, '.offer-desc-row'),
|
||||
price_text: firstText(container, ['.offer-price-row .price-item']),
|
||||
sales_text: firstText(container, ['.offer-price-row .col-desc_after', '.offer-desc-row .col-desc_after']),
|
||||
hover_price_text: hoverPriceText,
|
||||
moq_text: findMoqText(hoverItems, hoverPriceText),
|
||||
tag_items: tagItems,
|
||||
hover_items: hoverItems,
|
||||
seller_name: sellerAnchor ? normalizeText(sellerAnchor.innerText || sellerAnchor.textContent || '') : null,
|
||||
seller_url: sellerAnchor ? sellerAnchor.href : null,
|
||||
});
|
||||
}
|
||||
return items;
|
||||
};
|
||||
const findNextUrl = () => {
|
||||
const selectors = [
|
||||
'a.fui-next:not(.disabled)',
|
||||
'a.next-pagination-item:not(.disabled)',
|
||||
'a[rel="next"]:not(.disabled)',
|
||||
'a[data-role="next"]:not(.disabled)',
|
||||
];
|
||||
for (const selector of selectors) {
|
||||
const node = document.querySelector(selector);
|
||||
if (!node) continue;
|
||||
const href = normalizeUrl(node.getAttribute('href') || node.href || '');
|
||||
if (href) return href;
|
||||
}
|
||||
const textBased = Array.from(document.querySelectorAll('a'))
|
||||
.find((node) => /下一页|next/i.test(normalizeText(node.textContent || '')));
|
||||
if (!textBased) return '';
|
||||
return normalizeUrl(textBased.getAttribute('href') || textBased.href || '');
|
||||
};
|
||||
|
||||
return {
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
bodyText: document.body ? document.body.innerText || '' : '',
|
||||
next_url: findNextUrl(),
|
||||
candidates: collectCandidates(),
|
||||
};
|
||||
})()
|
||||
`) as SearchPayload;
|
||||
|
||||
if (!payload || typeof payload !== 'object') {
|
||||
throw new CommandExecutionError(
|
||||
'1688 search page did not return a readable payload',
|
||||
'Open the same query in Chrome and verify the page is fully loaded before retrying.',
|
||||
);
|
||||
}
|
||||
|
||||
return payload;
|
||||
}
|
||||
|
||||
async function collectSearchRows(page: IPage, query: string, limit: number): Promise<SearchRow[]> {
|
||||
const rowsByKey = new Map<string, SearchRow>();
|
||||
const seenPages = new Set<string>();
|
||||
let nextUrl = buildSearchUrl(query);
|
||||
let pageCount = 0;
|
||||
|
||||
while (nextUrl && rowsByKey.size < limit && pageCount < MAX_SEARCH_PAGES) {
|
||||
if (seenPages.has(nextUrl)) break;
|
||||
seenPages.add(nextUrl);
|
||||
pageCount += 1;
|
||||
|
||||
const payload = await readSearchPayload(page, nextUrl);
|
||||
const sourceUrl = cleanText(payload.href) || nextUrl;
|
||||
const candidates = Array.isArray(payload.candidates) ? payload.candidates : [];
|
||||
|
||||
for (const candidate of candidates) {
|
||||
const row = normalizeSearchCandidate(candidate, sourceUrl);
|
||||
const dedupeKey = buildDedupeKey(row);
|
||||
if (!dedupeKey || rowsByKey.has(dedupeKey)) continue;
|
||||
rowsByKey.set(dedupeKey, row);
|
||||
if (rowsByKey.size >= limit) break;
|
||||
}
|
||||
|
||||
const candidateNextUrl = cleanText(payload.next_url);
|
||||
if (!candidateNextUrl || candidateNextUrl === sourceUrl) break;
|
||||
nextUrl = candidateNextUrl;
|
||||
}
|
||||
|
||||
if (rowsByKey.size === 0) {
|
||||
throw new EmptyResultError(
|
||||
'1688 search',
|
||||
'No visible results were extracted. Retry with a different query or open the same search page in Chrome first.',
|
||||
);
|
||||
}
|
||||
|
||||
return [...rowsByKey.values()]
|
||||
.slice(0, limit)
|
||||
.map((row, index) => ({ ...row, rank: index + 1 }));
|
||||
}
|
||||
|
||||
cli({
|
||||
site: '1688',
|
||||
name: 'search',
|
||||
description: '1688 商品搜索(结果候选、卖家链接、价格/MOQ/销量文本)',
|
||||
domain: 'www.1688.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'query',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '搜索关键词,如 "置物架"',
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: SEARCH_LIMIT_DEFAULT,
|
||||
help: `结果数量上限(默认 ${SEARCH_LIMIT_DEFAULT},最大 ${SEARCH_LIMIT_MAX})`,
|
||||
},
|
||||
],
|
||||
columns: ['rank', 'title', 'price_text', 'moq_text', 'seller_name', 'location'],
|
||||
func: async (page, kwargs) => {
|
||||
const query = String(kwargs.query ?? '');
|
||||
const limit = parseSearchLimit(kwargs.limit);
|
||||
return collectSearchRows(page, query, limit);
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeSearchCandidate,
|
||||
extractMoqText,
|
||||
extractSalesText,
|
||||
firstWord,
|
||||
buildDedupeKey,
|
||||
};
|
||||
@@ -0,0 +1,75 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './shared.js';
|
||||
|
||||
describe('1688 shared helpers', () => {
|
||||
it('builds encoded search URLs and validates limit', () => {
|
||||
expect(__test__.buildSearchUrl('置物架')).toBe(
|
||||
'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=%E7%BD%AE%E7%89%A9%E6%9E%B6',
|
||||
);
|
||||
expect(() => __test__.buildSearchUrl(' ')).toThrowError(/cannot be empty/i);
|
||||
|
||||
expect(__test__.parseSearchLimit(3)).toBe(3);
|
||||
expect(__test__.parseSearchLimit('1000')).toBe(__test__.SEARCH_LIMIT_MAX);
|
||||
expect(() => __test__.parseSearchLimit('0')).toThrowError(/positive integer/i);
|
||||
});
|
||||
|
||||
it('extracts IDs and canonicalizes urls', () => {
|
||||
expect(__test__.extractOfferId('887904326744')).toBe('887904326744');
|
||||
expect(__test__.extractOfferId('https://detail.1688.com/offer/887904326744.html')).toBe('887904326744');
|
||||
expect(__test__.extractMemberId('https://winport.m.1688.com/page/index.html?memberId=b2b-1641351767')).toBe('b2b-1641351767');
|
||||
expect(__test__.extractMemberId('b2b-22154705262941f196')).toBe('b2b-22154705262941f196');
|
||||
expect(__test__.resolveStoreUrl('b2b-22154705262941f196')).toBe(
|
||||
'https://winport.m.1688.com/page/index.html?memberId=b2b-22154705262941f196',
|
||||
);
|
||||
expect(__test__.canonicalizeStoreUrl('https://yinuoweierfushi.1688.com/page/index.html?spm=foo')).toBe(
|
||||
'https://yinuoweierfushi.1688.com',
|
||||
);
|
||||
expect(__test__.canonicalizeItemUrl('http://detail.m.1688.com/page/index.html?offerId=910933345396&spm=x')).toBe(
|
||||
'https://detail.1688.com/offer/910933345396.html',
|
||||
);
|
||||
expect(__test__.canonicalizeSellerUrl('https://yinuoweierfushi.1688.com/page/contactinfo.html?tracelog=1')).toBe(
|
||||
'https://yinuoweierfushi.1688.com',
|
||||
);
|
||||
expect(__test__.extractShopId('https://yinuoweierfushi.1688.com/page/index.html')).toBe('yinuoweierfushi');
|
||||
});
|
||||
|
||||
it('parses price ranges and moq text', () => {
|
||||
expect(__test__.parsePriceText('¥96.00-98.00')).toEqual({
|
||||
price_text: '¥96.00-98.00',
|
||||
price_min: 96,
|
||||
price_max: 98,
|
||||
currency: 'CNY',
|
||||
});
|
||||
|
||||
expect(__test__.parsePriceText('¥ 14 .28')).toEqual({
|
||||
price_text: '¥14.28',
|
||||
price_min: 14.28,
|
||||
price_max: 14.28,
|
||||
currency: 'CNY',
|
||||
});
|
||||
|
||||
expect(__test__.parseMoqText('3套起批')).toEqual({
|
||||
moq_text: '3套起批',
|
||||
moq_value: 3,
|
||||
});
|
||||
|
||||
expect(__test__.parseMoqText('2~999个')).toEqual({
|
||||
moq_text: '2~999个',
|
||||
moq_value: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it('detects captcha and login states', () => {
|
||||
expect(__test__.extractLocation('山东青岛 送至 江苏苏州')).toBe('山东青岛');
|
||||
expect(__test__.isCaptchaState({
|
||||
href: 'https://s.1688.com/_____tmd_____/punish',
|
||||
title: '验证码拦截',
|
||||
body_text: '请拖动下方滑块完成验证',
|
||||
})).toBe(true);
|
||||
expect(__test__.isLoginState({
|
||||
href: 'https://login.taobao.com/member/login.jhtml',
|
||||
title: '账号登录',
|
||||
body_text: '请登录后继续',
|
||||
})).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,623 @@
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '../../errors.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
|
||||
export const SITE = '1688';
|
||||
export const HOME_URL = 'https://www.1688.com/';
|
||||
export const SEARCH_URL_PREFIX = 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=';
|
||||
export const DETAIL_URL_PREFIX = 'https://detail.1688.com/offer/';
|
||||
export const STORE_MOBILE_URL_PREFIX = 'https://winport.m.1688.com/page/index.html?memberId=';
|
||||
export const STRATEGY = 'cookie';
|
||||
export const SEARCH_LIMIT_DEFAULT = 20;
|
||||
export const SEARCH_LIMIT_MAX = 100;
|
||||
|
||||
const STORE_GENERIC_HOSTS = new Set(['www', 'detail', 's', 'winport', 'work', 'air', 'dj']);
|
||||
const TRACKING_QUERY_KEYS = new Set([
|
||||
'spm',
|
||||
'tracelog',
|
||||
'clickid',
|
||||
'source',
|
||||
'scene',
|
||||
'from',
|
||||
'src',
|
||||
'ns',
|
||||
'cna',
|
||||
'pvid',
|
||||
]);
|
||||
const CAPTCHA_URL_MARKER = '/_____tmd_____/punish';
|
||||
const CAPTCHA_TEXT_PATTERNS = [
|
||||
'请拖动下方滑块完成验证',
|
||||
'请按住滑块,拖动到最右边',
|
||||
'通过验证以确保正常访问',
|
||||
'验证码拦截',
|
||||
'访问验证',
|
||||
'滑动验证',
|
||||
];
|
||||
const LOGIN_TEXT_PATTERNS = [
|
||||
'请登录',
|
||||
'登录后',
|
||||
'账号登录',
|
||||
'手机登录',
|
||||
'立即登录',
|
||||
'扫码登录',
|
||||
'请先完成登录',
|
||||
'请先登录后查看',
|
||||
];
|
||||
const LOGIN_URL_PATTERNS = ['/member/login', 'passport', 'login.taobao.com', 'account.1688.com'];
|
||||
|
||||
export const FACTORY_BADGE_PATTERNS = [
|
||||
'源头工厂',
|
||||
'深度验厂',
|
||||
'实力工厂',
|
||||
'工厂档案',
|
||||
'加工专区',
|
||||
'验厂报告',
|
||||
'厂家直销',
|
||||
'生产厂家',
|
||||
'工厂直供',
|
||||
];
|
||||
export const SERVICE_BADGE_PATTERNS = [
|
||||
'延期必赔',
|
||||
'品质保障',
|
||||
'破损包赔',
|
||||
'退货包运费',
|
||||
'晚发必赔',
|
||||
'7*24小时响应',
|
||||
'48小时发货',
|
||||
'72小时发货',
|
||||
'后天达',
|
||||
'包邮',
|
||||
'闪电拿样',
|
||||
];
|
||||
|
||||
const CHINA_LOCATIONS = [
|
||||
'北京',
|
||||
'天津',
|
||||
'上海',
|
||||
'重庆',
|
||||
'河北',
|
||||
'山西',
|
||||
'辽宁',
|
||||
'吉林',
|
||||
'黑龙江',
|
||||
'江苏',
|
||||
'浙江',
|
||||
'安徽',
|
||||
'福建',
|
||||
'江西',
|
||||
'山东',
|
||||
'河南',
|
||||
'湖北',
|
||||
'湖南',
|
||||
'广东',
|
||||
'海南',
|
||||
'四川',
|
||||
'贵州',
|
||||
'云南',
|
||||
'陕西',
|
||||
'甘肃',
|
||||
'青海',
|
||||
'台湾',
|
||||
'内蒙古',
|
||||
'广西',
|
||||
'西藏',
|
||||
'宁夏',
|
||||
'新疆',
|
||||
'香港',
|
||||
'澳门',
|
||||
];
|
||||
|
||||
export interface ProvenanceFields {
|
||||
source_url: string;
|
||||
fetched_at: string;
|
||||
strategy: string;
|
||||
}
|
||||
|
||||
export interface PageState {
|
||||
href: string;
|
||||
title: string;
|
||||
body_text: string;
|
||||
}
|
||||
|
||||
export interface PriceRange {
|
||||
price_text: string;
|
||||
price_min: number | null;
|
||||
price_max: number | null;
|
||||
currency: string | null;
|
||||
}
|
||||
|
||||
export interface MoqValue {
|
||||
moq_text: string;
|
||||
moq_value: number | null;
|
||||
}
|
||||
|
||||
export interface PriceTier {
|
||||
quantity_text: string;
|
||||
quantity_min: number | null;
|
||||
price_text: string;
|
||||
price: number | null;
|
||||
currency: string | null;
|
||||
}
|
||||
|
||||
export interface SearchCandidate {
|
||||
item_url: string;
|
||||
title: string;
|
||||
container_text: string;
|
||||
seller_name: string | null;
|
||||
seller_url: string | null;
|
||||
}
|
||||
|
||||
export function cleanText(value: unknown): string {
|
||||
return typeof value === 'string'
|
||||
? value.replace(/\u00a0/g, ' ').replace(/\s+/g, ' ').trim()
|
||||
: '';
|
||||
}
|
||||
|
||||
export function cleanMultilineText(value: unknown): string {
|
||||
return typeof value === 'string'
|
||||
? value
|
||||
.replace(/\u00a0/g, ' ')
|
||||
.split('\n')
|
||||
.map((line) => line.replace(/\s+/g, ' ').trim())
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
: '';
|
||||
}
|
||||
|
||||
export function uniqueNonEmpty(values: Array<string | null | undefined>): string[] {
|
||||
return [...new Set(values.map((value) => cleanText(value)).filter(Boolean))];
|
||||
}
|
||||
|
||||
export function parseSearchLimit(input: unknown): number {
|
||||
const parsed = Number.parseInt(String(input ?? SEARCH_LIMIT_DEFAULT), 10);
|
||||
if (!Number.isFinite(parsed) || parsed < 1) {
|
||||
throw new ArgumentError(
|
||||
'1688 search --limit must be a positive integer',
|
||||
'Example: opencli 1688 search "桌面置物架" --limit 20',
|
||||
);
|
||||
}
|
||||
return Math.min(SEARCH_LIMIT_MAX, parsed);
|
||||
}
|
||||
|
||||
export function buildSearchUrl(query: string): string {
|
||||
const normalized = cleanText(query);
|
||||
if (!normalized) {
|
||||
throw new ArgumentError(
|
||||
'1688 search query cannot be empty',
|
||||
'Example: opencli 1688 search "桌面置物架" --limit 20',
|
||||
);
|
||||
}
|
||||
return `${SEARCH_URL_PREFIX}${encodeURIComponent(normalized)}`;
|
||||
}
|
||||
|
||||
export function buildDetailUrl(input: string): string {
|
||||
const offerId = extractOfferId(input);
|
||||
if (!offerId) {
|
||||
throw new ArgumentError(
|
||||
'1688 item expects an offer URL or offer ID',
|
||||
'Example: opencli 1688 item 887904326744',
|
||||
);
|
||||
}
|
||||
return `${DETAIL_URL_PREFIX}${offerId}.html`;
|
||||
}
|
||||
|
||||
export function resolveStoreUrl(input: string): string {
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) {
|
||||
throw new ArgumentError(
|
||||
'1688 store expects a store URL or member ID',
|
||||
'Example: opencli 1688 store https://yinuoweierfushi.1688.com/',
|
||||
);
|
||||
}
|
||||
|
||||
const memberId = extractMemberId(normalized);
|
||||
if (memberId) {
|
||||
return `${STORE_MOBILE_URL_PREFIX}${memberId}`;
|
||||
}
|
||||
|
||||
if (/^https?:\/\//i.test(normalized)) {
|
||||
return canonicalizeStoreUrl(normalized);
|
||||
}
|
||||
|
||||
if (normalized.endsWith('.1688.com')) {
|
||||
return canonicalizeStoreUrl(`https://${normalized}`);
|
||||
}
|
||||
|
||||
if (/^[a-z0-9-]+$/i.test(normalized)) {
|
||||
return canonicalizeStoreUrl(`https://${normalized}.1688.com`);
|
||||
}
|
||||
|
||||
throw new ArgumentError(
|
||||
'1688 store expects a store URL or member ID',
|
||||
'Example: opencli 1688 store b2b-22154705262941f196',
|
||||
);
|
||||
}
|
||||
|
||||
export function canonicalizeStoreUrl(input: string): string {
|
||||
const url = parse1688Url(input);
|
||||
const memberId = extractMemberId(url.toString());
|
||||
if (memberId) {
|
||||
return `${STORE_MOBILE_URL_PREFIX}${memberId}`;
|
||||
}
|
||||
|
||||
const host = normalizeStoreHost(url.hostname);
|
||||
if (!host) {
|
||||
throw new ArgumentError(
|
||||
'Invalid 1688 store URL',
|
||||
'Example: opencli 1688 store https://yinuoweierfushi.1688.com/',
|
||||
);
|
||||
}
|
||||
return `https://${host}`;
|
||||
}
|
||||
|
||||
export function canonicalizeItemUrl(input: string): string | null {
|
||||
const offerId = extractOfferId(input);
|
||||
if (offerId) {
|
||||
return `${DETAIL_URL_PREFIX}${offerId}.html`;
|
||||
}
|
||||
const url = parse1688UrlOrNull(input);
|
||||
if (!url) return null;
|
||||
stripTrackingParams(url);
|
||||
url.hash = '';
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
export function canonicalizeSellerUrl(input: string): string | null {
|
||||
const memberId = extractMemberId(input);
|
||||
if (memberId) {
|
||||
return `${STORE_MOBILE_URL_PREFIX}${memberId}`;
|
||||
}
|
||||
const url = parse1688UrlOrNull(input);
|
||||
if (!url) return null;
|
||||
const host = normalizeStoreHost(url.hostname);
|
||||
if (!host) return null;
|
||||
return `https://${host}`;
|
||||
}
|
||||
|
||||
export function extractOfferId(input: string): string | null {
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) return null;
|
||||
const directId = normalized.match(/^\d{6,}$/)?.[0];
|
||||
if (directId) return directId;
|
||||
const detailMatch = normalized.match(/\/offer\/(\d{6,})\.html/i);
|
||||
if (detailMatch) return detailMatch[1];
|
||||
const queryMatch = normalized.match(/[?&]offerId=(\d{6,})/i);
|
||||
if (queryMatch) return queryMatch[1];
|
||||
return null;
|
||||
}
|
||||
|
||||
export function extractMemberId(input: string): string | null {
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) return null;
|
||||
const direct = normalized.match(/\bb2b-[a-z0-9]+\b/i)?.[0];
|
||||
if (direct) return direct;
|
||||
const queryMatch = normalized.match(/[?&]memberId=(b2b-[a-z0-9]+)/i);
|
||||
if (queryMatch) return queryMatch[1];
|
||||
const mobileMatch = normalized.match(/\/winport\/(b2b-[a-z0-9]+)\.html/i);
|
||||
if (mobileMatch) return mobileMatch[1];
|
||||
return null;
|
||||
}
|
||||
|
||||
export function extractShopId(input: string): string | null {
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) return null;
|
||||
|
||||
try {
|
||||
const url = new URL(/^https?:\/\//i.test(normalized) ? normalized : `https://${normalized}`);
|
||||
const host = normalizeStoreHost(url.hostname);
|
||||
if (!host) return null;
|
||||
return host.split('.')[0] ?? null;
|
||||
} catch {
|
||||
return /^[a-z0-9-]+$/i.test(normalized) ? normalized : null;
|
||||
}
|
||||
}
|
||||
|
||||
export function buildProvenance(sourceUrl: string): ProvenanceFields {
|
||||
return {
|
||||
source_url: sourceUrl,
|
||||
fetched_at: new Date().toISOString(),
|
||||
strategy: STRATEGY,
|
||||
};
|
||||
}
|
||||
|
||||
export function parsePriceText(text: string): PriceRange {
|
||||
const normalized = normalizeNumericText(cleanText(text));
|
||||
const matches = normalized.match(/\d+(?:,\d{3})*(?:\.\d+)?/g) ?? [];
|
||||
const values = matches
|
||||
.map((value) => Number.parseFloat(value.replace(/,/g, '')))
|
||||
.filter((value) => Number.isFinite(value));
|
||||
|
||||
if (values.length === 0) {
|
||||
return {
|
||||
price_text: normalized,
|
||||
price_min: null,
|
||||
price_max: null,
|
||||
currency: null,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
price_text: normalized,
|
||||
price_min: values[0] ?? null,
|
||||
price_max: values[values.length - 1] ?? values[0] ?? null,
|
||||
currency: normalized.includes('¥') || normalized.includes('元') ? 'CNY' : null,
|
||||
};
|
||||
}
|
||||
|
||||
export function normalizePriceTiers(
|
||||
rawTiers: Array<{ beginAmount?: unknown; price?: unknown }>,
|
||||
unit: string | null,
|
||||
): PriceTier[] {
|
||||
return rawTiers
|
||||
.map((tier) => {
|
||||
const quantityMin = toNumber(tier.beginAmount);
|
||||
const priceText = cleanText(tier.price);
|
||||
const price = toNumber(tier.price);
|
||||
return {
|
||||
quantity_text: quantityMin !== null ? `${quantityMin}${unit ?? ''}` : '',
|
||||
quantity_min: quantityMin,
|
||||
price_text: priceText,
|
||||
price,
|
||||
currency: priceText ? 'CNY' : null,
|
||||
};
|
||||
})
|
||||
.filter((tier) => tier.price_text);
|
||||
}
|
||||
|
||||
export function parseMoqText(text: string): MoqValue {
|
||||
const normalized = normalizeNumericText(cleanText(text));
|
||||
const match = normalized.match(/(\d+(?:\.\d+)?)\s*(件|个|套|箱|包|双|台|把|只|pcs|piece|pieces)?\s*起批/i)
|
||||
?? normalized.match(/≥\s*(\d+(?:\.\d+)?)/);
|
||||
const rangeMatch = normalized.match(
|
||||
/(\d+(?:\.\d+)?)\s*(?:~|-|至|到)\s*\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只|pcs|piece|pieces)/i,
|
||||
);
|
||||
|
||||
if (!match && !rangeMatch) {
|
||||
return {
|
||||
moq_text: normalized,
|
||||
moq_value: null,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
moq_text: normalized,
|
||||
moq_value: Number.parseFloat((match ?? rangeMatch)![1]),
|
||||
};
|
||||
}
|
||||
|
||||
export function extractLocation(text: string): string | null {
|
||||
const normalized = cleanMultilineText(text);
|
||||
const primaryRegion = normalized.split(/送至|发往/)[0] ?? normalized;
|
||||
const lines = primaryRegion.split('\n');
|
||||
for (const line of lines) {
|
||||
const compact = cleanText(line);
|
||||
if (!compact || compact.length > 16) continue;
|
||||
if (CHINA_LOCATIONS.some((location) => compact.startsWith(location))) {
|
||||
return compact;
|
||||
}
|
||||
}
|
||||
|
||||
const locationPattern = new RegExp(`(${CHINA_LOCATIONS.join('|')})[\\u4e00-\\u9fa5]{0,8}`);
|
||||
return primaryRegion.match(locationPattern)?.[0] ?? null;
|
||||
}
|
||||
|
||||
export function extractAddress(text: string): string | null {
|
||||
const normalized = cleanMultilineText(text);
|
||||
const lineMatch = normalized.match(/地址[::]\s*([^\n]+)/);
|
||||
if (lineMatch) return cleanText(lineMatch[1]);
|
||||
return normalized
|
||||
.split('\n')
|
||||
.map((line) => cleanText(line))
|
||||
.find((line) => line.includes('省') || line.includes('市') || line.includes('区') || line.includes('县'))
|
||||
?? null;
|
||||
}
|
||||
|
||||
export function extractMetric(text: string, label: string): string | null {
|
||||
const normalized = cleanMultilineText(text);
|
||||
const direct = normalized.match(new RegExp(`(?:^|\\n)\\s*${escapeForRegex(label)}[::]?\\s*([^\\n]+)`));
|
||||
if (direct) return cleanText(direct[1]);
|
||||
|
||||
const lineBased = normalized.match(new RegExp(`(?:^|\\n)\\s*${escapeForRegex(label)}\\n([^\\n]+)`));
|
||||
return lineBased ? cleanText(lineBased[1]) : null;
|
||||
}
|
||||
|
||||
export function extractYearsOnPlatform(text: string): string | null {
|
||||
return text.match(/入驻\d+年/)?.[0] ?? null;
|
||||
}
|
||||
|
||||
export function extractMainBusiness(text: string): string | null {
|
||||
const value = extractMetric(text, '主营');
|
||||
return value ? value.replace(/^:/, '').trim() : null;
|
||||
}
|
||||
|
||||
export function extractBadges(text: string, candidates: string[]): string[] {
|
||||
return uniqueNonEmpty(candidates.filter((candidate) => cleanMultilineText(text).includes(candidate)));
|
||||
}
|
||||
|
||||
export function guessTopCategories(text: string): string[] {
|
||||
const mainBusiness = extractMainBusiness(text);
|
||||
if (!mainBusiness) return [];
|
||||
return uniqueNonEmpty(mainBusiness.split(/[、,/|]/).map((value) => value.trim()));
|
||||
}
|
||||
|
||||
export function isCaptchaState(state: Partial<PageState>): boolean {
|
||||
const href = cleanText(state.href).toLowerCase();
|
||||
const title = cleanText(state.title);
|
||||
const bodyText = cleanMultilineText(state.body_text);
|
||||
if (href.includes(CAPTCHA_URL_MARKER)) return true;
|
||||
return CAPTCHA_TEXT_PATTERNS.some((pattern) => title.includes(pattern) || bodyText.includes(pattern));
|
||||
}
|
||||
|
||||
export function isLoginState(state: Partial<PageState>): boolean {
|
||||
const href = cleanText(state.href).toLowerCase();
|
||||
const title = cleanText(state.title);
|
||||
const bodyText = cleanMultilineText(state.body_text);
|
||||
if (LOGIN_URL_PATTERNS.some((pattern) => href.includes(pattern))) return true;
|
||||
return LOGIN_TEXT_PATTERNS.some((pattern) => title.includes(pattern) || bodyText.includes(pattern));
|
||||
}
|
||||
|
||||
export function buildCaptchaHint(action: string): string {
|
||||
return [
|
||||
`Open a clean 1688 ${action} page in the shared Chrome profile and finish any slider challenge first.`,
|
||||
'If you run opencli via CDP, set OPENCLI_CDP_TARGET=1688.com or a more specific 1688 host before retrying.',
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
export async function readPageState(page: IPage): Promise<PageState> {
|
||||
const result = await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
body_text: document.body ? document.body.innerText || '' : '',
|
||||
}))()
|
||||
`) as Partial<PageState>;
|
||||
|
||||
return {
|
||||
href: cleanText(result.href),
|
||||
title: cleanText(result.title),
|
||||
body_text: cleanMultilineText(result.body_text),
|
||||
};
|
||||
}
|
||||
|
||||
export async function gotoAndReadState(
|
||||
page: IPage,
|
||||
url: string,
|
||||
settleMs: number = 2500,
|
||||
action: string = 'page',
|
||||
): Promise<PageState> {
|
||||
try {
|
||||
await page.goto(url, { settleMs });
|
||||
await page.wait(1.5);
|
||||
return readPageState(page);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (
|
||||
message.includes('Inspected target navigated or closed')
|
||||
|| message.includes('Cannot find context with specified id')
|
||||
|| message.includes('Target closed')
|
||||
) {
|
||||
throw new CommandExecutionError(
|
||||
`1688 ${action} navigation lost the current browser target`,
|
||||
`${buildCaptchaHint(action)} If CDP is attached to a stale or blocked tab, open a fresh 1688 tab and point OPENCLI_CDP_TARGET at that tab.`,
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export async function ensure1688Session(page: IPage): Promise<void> {
|
||||
const state = await gotoAndReadState(page, HOME_URL, 1500, 'homepage');
|
||||
assertAuthenticatedState(state, 'homepage');
|
||||
}
|
||||
|
||||
export function assertAuthenticatedState(state: PageState, action: string): void {
|
||||
if (!isCaptchaState(state) && !isLoginState(state)) return;
|
||||
throw new AuthRequiredError('1688.com', `请先在共享 Chrome 完成 1688 登录/验证,再重试(${action})`);
|
||||
}
|
||||
|
||||
export function assertNotCaptcha(state: PageState, action: string): void {
|
||||
assertAuthenticatedState(state, action);
|
||||
}
|
||||
|
||||
export function toNumber(value: unknown): number | null {
|
||||
if (typeof value === 'number' && Number.isFinite(value)) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.replace(/,/g, '').trim();
|
||||
if (!normalized) return null;
|
||||
const parsed = Number.parseFloat(normalized);
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function limitCandidates<T>(values: T[], limit: number): T[] {
|
||||
const normalizedLimit = Math.max(1, Math.trunc(limit) || 1);
|
||||
return values.slice(0, normalizedLimit);
|
||||
}
|
||||
|
||||
function normalizeNumericText(value: string): string {
|
||||
return value
|
||||
.replace(/([¥$€])\s+(?=\d)/g, '$1')
|
||||
.replace(/(\d)\s*\.\s*(\d)/g, '$1.$2')
|
||||
.replace(/\s*([~-])\s*/g, '$1')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function escapeForRegex(value: string): string {
|
||||
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
}
|
||||
|
||||
function parse1688Url(input: string): URL {
|
||||
const normalized = cleanText(input);
|
||||
try {
|
||||
const url = new URL(normalized);
|
||||
if (!url.hostname.endsWith('.1688.com') && url.hostname !== '1688.com' && url.hostname !== 'www.1688.com') {
|
||||
throw new Error('invalid-host');
|
||||
}
|
||||
stripTrackingParams(url);
|
||||
url.hash = '';
|
||||
return url;
|
||||
} catch {
|
||||
throw new ArgumentError(
|
||||
'Invalid 1688 URL',
|
||||
'Use a URL under 1688.com (for example: https://detail.1688.com/offer/887904326744.html)',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function parse1688UrlOrNull(input: string): URL | null {
|
||||
try {
|
||||
return parse1688Url(input);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeStoreHost(hostname: string): string | null {
|
||||
const lower = cleanText(hostname).toLowerCase();
|
||||
if (!lower.endsWith('.1688.com')) return null;
|
||||
const [subdomain] = lower.split('.');
|
||||
if (!subdomain || STORE_GENERIC_HOSTS.has(subdomain)) return null;
|
||||
return lower;
|
||||
}
|
||||
|
||||
function stripTrackingParams(url: URL): void {
|
||||
const keys = [...url.searchParams.keys()];
|
||||
for (const key of keys) {
|
||||
if (TRACKING_QUERY_KEYS.has(key) || key.toLowerCase().startsWith('utm_')) {
|
||||
url.searchParams.delete(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const __test__ = {
|
||||
SEARCH_LIMIT_DEFAULT,
|
||||
SEARCH_LIMIT_MAX,
|
||||
parseSearchLimit,
|
||||
buildSearchUrl,
|
||||
buildDetailUrl,
|
||||
resolveStoreUrl,
|
||||
canonicalizeStoreUrl,
|
||||
canonicalizeItemUrl,
|
||||
canonicalizeSellerUrl,
|
||||
extractOfferId,
|
||||
extractMemberId,
|
||||
extractShopId,
|
||||
parsePriceText,
|
||||
normalizePriceTiers,
|
||||
parseMoqText,
|
||||
extractLocation,
|
||||
extractAddress,
|
||||
extractMetric,
|
||||
extractYearsOnPlatform,
|
||||
extractMainBusiness,
|
||||
extractBadges,
|
||||
guessTopCategories,
|
||||
isCaptchaState,
|
||||
isLoginState,
|
||||
cleanText,
|
||||
cleanMultilineText,
|
||||
uniqueNonEmpty,
|
||||
limitCandidates,
|
||||
};
|
||||
@@ -0,0 +1,69 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './store.js';
|
||||
|
||||
describe('1688 store normalization', () => {
|
||||
it('merges store contact text with seller seed data', () => {
|
||||
const result = __test__.normalizeStorePayload({
|
||||
resolvedUrl: 'https://yinuoweierfushi.1688.com/?offerId=887904326744',
|
||||
explicitMemberId: null,
|
||||
storePayload: {
|
||||
href: 'https://yinuoweierfushi.1688.com/page/index.html',
|
||||
bodyText: `
|
||||
青岛沁澜衣品服装有限公司
|
||||
联系方式
|
||||
地址:山东省青岛市即墨区环秀街道办事处湘江二路97号甲
|
||||
`,
|
||||
offerLinks: ['https://detail.1688.com/offer/887904326744.html'],
|
||||
},
|
||||
contactPayload: {
|
||||
href: 'https://yinuoweierfushi.1688.com/page/contactinfo.html',
|
||||
bodyText: `
|
||||
青岛沁澜衣品服装有限公司
|
||||
电话:86 0532 86655366
|
||||
手机:15963238678
|
||||
地址:山东省青岛市即墨区环秀街道办事处湘江二路97号甲
|
||||
`,
|
||||
},
|
||||
seed: {
|
||||
bodyText: `
|
||||
入驻13年
|
||||
主营:大码女装
|
||||
店铺回头率
|
||||
87%
|
||||
延期必赔
|
||||
品质保障
|
||||
`,
|
||||
seller: {
|
||||
companyName: '青岛沁澜衣品服装有限公司',
|
||||
memberId: 'b2b-1641351767',
|
||||
winportUrl: 'https://yinuoweierfushi.1688.com/page/index.html?spm=abc',
|
||||
},
|
||||
services: [{ serviceName: '延期必赔' }, { serviceName: '品质保障' }],
|
||||
},
|
||||
});
|
||||
|
||||
expect(result.member_id).toBe('b2b-1641351767');
|
||||
expect(result.store_url).toBe('https://yinuoweierfushi.1688.com');
|
||||
expect(result.company_url).toBe('https://yinuoweierfushi.1688.com/page/contactinfo.html');
|
||||
expect(result.years_on_platform_text).toBe('入驻13年');
|
||||
expect(result.location).toBe('山东省青岛市即墨区环秀街道办事处湘江二路97号甲');
|
||||
expect(result.return_rate_text).toContain('87%');
|
||||
expect(result.top_categories).toEqual(['大码女装']);
|
||||
expect(result.service_badges).toEqual(['延期必赔', '品质保障']);
|
||||
});
|
||||
|
||||
it('builds contact urls and extracts offer ids', () => {
|
||||
expect(__test__.safeCanonicalStoreUrl('https://yinuoweierfushi.1688.com/page/index.html?spm=foo')).toBe(
|
||||
'https://yinuoweierfushi.1688.com',
|
||||
);
|
||||
expect(__test__.buildContactUrl('https://yinuoweierfushi.1688.com')).toBe(
|
||||
'https://yinuoweierfushi.1688.com/page/contactinfo.html',
|
||||
);
|
||||
expect(__test__.firstOfferId([
|
||||
'https://detail.1688.com/offer/887904326744.html',
|
||||
])).toBe('887904326744');
|
||||
expect(__test__.firstContactUrl([
|
||||
'https://yinuoweierfushi.1688.com/page/contactinfo.html?spm=1',
|
||||
])).toBe('https://yinuoweierfushi.1688.com/page/contactinfo.html');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,300 @@
|
||||
import { CommandExecutionError, EmptyResultError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
FACTORY_BADGE_PATTERNS,
|
||||
SERVICE_BADGE_PATTERNS,
|
||||
assertAuthenticatedState,
|
||||
buildDetailUrl,
|
||||
buildProvenance,
|
||||
canonicalizeSellerUrl,
|
||||
canonicalizeStoreUrl,
|
||||
cleanMultilineText,
|
||||
cleanText,
|
||||
extractAddress,
|
||||
extractBadges,
|
||||
extractMemberId,
|
||||
extractMetric,
|
||||
extractOfferId,
|
||||
extractShopId,
|
||||
extractYearsOnPlatform,
|
||||
gotoAndReadState,
|
||||
guessTopCategories,
|
||||
resolveStoreUrl,
|
||||
uniqueNonEmpty,
|
||||
} from './shared.js';
|
||||
|
||||
interface StoreBrowserPayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
bodyText?: string;
|
||||
offerLinks?: string[];
|
||||
contactLinks?: string[];
|
||||
}
|
||||
|
||||
interface StoreItemSeed {
|
||||
href?: string;
|
||||
bodyText?: string;
|
||||
seller?: {
|
||||
companyName?: string;
|
||||
memberId?: string;
|
||||
winportUrl?: string;
|
||||
sellerWinportUrlMap?: Record<string, string>;
|
||||
};
|
||||
services?: Array<{ serviceName?: string }>;
|
||||
}
|
||||
|
||||
function normalizeStorePayload(input: {
|
||||
resolvedUrl: string;
|
||||
storePayload: StoreBrowserPayload | null;
|
||||
contactPayload: StoreBrowserPayload | null;
|
||||
seed: StoreItemSeed | null;
|
||||
explicitMemberId: string | null;
|
||||
}): Record<string, unknown> {
|
||||
const storePayload = input.storePayload;
|
||||
const contactPayload = input.contactPayload;
|
||||
const seed = input.seed;
|
||||
|
||||
const contactText = cleanMultilineText(contactPayload?.bodyText);
|
||||
const storeText = cleanMultilineText(storePayload?.bodyText);
|
||||
const seedText = cleanMultilineText(seed?.bodyText);
|
||||
const combinedText = [contactText, storeText, seedText].filter(Boolean).join('\n');
|
||||
|
||||
const sellerUrlRaw = cleanText(
|
||||
seed?.seller?.winportUrl
|
||||
?? seed?.seller?.sellerWinportUrlMap?.defaultUrl
|
||||
?? storePayload?.href
|
||||
?? input.resolvedUrl,
|
||||
);
|
||||
const storeUrl = safeCanonicalStoreUrl(sellerUrlRaw || input.resolvedUrl) ?? input.resolvedUrl;
|
||||
const sellerUrl = canonicalizeSellerUrl(sellerUrlRaw) ?? storeUrl;
|
||||
const companyUrl = pickCompanyUrl(contactPayload?.href, storeUrl);
|
||||
const memberId = cleanText(seed?.seller?.memberId)
|
||||
|| input.explicitMemberId
|
||||
|| extractMemberId(input.resolvedUrl)
|
||||
|| extractMemberId(storePayload?.href ?? '')
|
||||
|| null;
|
||||
const shopId = extractShopId(sellerUrl) ?? extractShopId(storeUrl);
|
||||
const companyName = cleanText(seed?.seller?.companyName)
|
||||
|| firstNamedLine(contactText)
|
||||
|| firstNamedLine(storeText)
|
||||
|| null;
|
||||
const serviceBadges = uniqueNonEmpty([
|
||||
...extractBadges(combinedText, SERVICE_BADGE_PATTERNS),
|
||||
...((seed?.services ?? []).map((service) => cleanText(service.serviceName))),
|
||||
]);
|
||||
const factoryBadges = extractBadges(combinedText, FACTORY_BADGE_PATTERNS);
|
||||
|
||||
return {
|
||||
member_id: memberId,
|
||||
shop_id: shopId,
|
||||
store_name: companyName,
|
||||
store_url: storeUrl,
|
||||
company_name: companyName,
|
||||
company_url: companyUrl,
|
||||
business_model_text: firstMetric(combinedText, ['经营模式', '生产加工', '主营产品']),
|
||||
years_on_platform_text: extractYearsOnPlatform(combinedText),
|
||||
location: extractAddress(contactText) ?? extractAddress(storeText),
|
||||
staff_size_text: firstMetric(combinedText, ['员工人数', '员工总数']),
|
||||
factory_badges: factoryBadges,
|
||||
service_badges: serviceBadges,
|
||||
response_rate_text: firstMetric(combinedText, ['响应率', '回复率', '响应速度']),
|
||||
return_rate_text: extractReturnRate(combinedText),
|
||||
top_categories: guessTopCategories(combinedText),
|
||||
phone_text: extractMetric(contactText, '电话'),
|
||||
mobile_text: extractMetric(contactText, '手机'),
|
||||
...buildProvenance(cleanText(contactPayload?.href) || cleanText(storePayload?.href) || input.resolvedUrl),
|
||||
};
|
||||
}
|
||||
|
||||
function safeCanonicalStoreUrl(url: string): string | null {
|
||||
try {
|
||||
return canonicalizeStoreUrl(url);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function pickCompanyUrl(contactHref: string | undefined, storeUrl: string): string | null {
|
||||
const fromPage = cleanText(contactHref);
|
||||
if (fromPage) {
|
||||
const normalized = buildContactUrl(fromPage);
|
||||
if (normalized) return normalized;
|
||||
}
|
||||
return buildContactUrl(storeUrl);
|
||||
}
|
||||
|
||||
function buildContactUrl(storeUrl: string): string | null {
|
||||
try {
|
||||
const parsed = new URL(storeUrl);
|
||||
if (!parsed.hostname.endsWith('.1688.com')) return null;
|
||||
return `${parsed.protocol}//${parsed.hostname}/page/contactinfo.html`;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function firstNamedLine(text: string): string | null {
|
||||
return text
|
||||
.split('\n')
|
||||
.map((line) => cleanText(line))
|
||||
.find((line) => line.includes('有限公司') || line.includes('商行') || line.includes('工厂'))
|
||||
?? null;
|
||||
}
|
||||
|
||||
function firstMetric(text: string, labels: string[]): string | null {
|
||||
for (const label of labels) {
|
||||
const value = extractMetric(text, label);
|
||||
if (value) return value;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function extractReturnRate(text: string): string | null {
|
||||
const inline = text.match(/回头率\s*([0-9.]+%)/);
|
||||
if (inline) return cleanText(inline[0]);
|
||||
const multiline = text.match(/回头率\s*\n\s*([0-9.]+%)/);
|
||||
if (!multiline) return null;
|
||||
return `回头率${cleanText(multiline[1])}`;
|
||||
}
|
||||
|
||||
function firstOfferId(links: string[]): string | null {
|
||||
for (const link of links) {
|
||||
const offerId = extractOfferId(link);
|
||||
if (offerId) return offerId;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function firstContactUrl(links: string[]): string | null {
|
||||
for (const link of links) {
|
||||
const url = buildContactUrl(link);
|
||||
if (url) return url;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function readStorePayload(page: IPage, url: string, action: string): Promise<StoreBrowserPayload> {
|
||||
const state = await gotoAndReadState(page, url, 2500, action);
|
||||
assertAuthenticatedState(state, action);
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
bodyText: document.body ? document.body.innerText || '' : '',
|
||||
offerLinks: Array.from(document.querySelectorAll('a[href*="detail.1688.com/offer/"], a[href*="offerId="]'))
|
||||
.map((anchor) => anchor.href)
|
||||
.filter(Boolean),
|
||||
contactLinks: Array.from(document.querySelectorAll('a[href*="contactinfo"]'))
|
||||
.map((anchor) => anchor.href)
|
||||
.filter(Boolean),
|
||||
}))()
|
||||
`) as StoreBrowserPayload;
|
||||
}
|
||||
|
||||
async function readItemSeed(page: IPage, offerId: string): Promise<StoreItemSeed> {
|
||||
const itemUrl = buildDetailUrl(offerId);
|
||||
const state = await gotoAndReadState(page, itemUrl, 2500, 'store seed item');
|
||||
assertAuthenticatedState(state, 'store seed item');
|
||||
|
||||
const seed = await page.evaluate(`
|
||||
(() => {
|
||||
const model = window.context?.result?.global?.globalData?.model ?? null;
|
||||
const toJson = (value) => JSON.parse(JSON.stringify(value ?? null));
|
||||
return {
|
||||
href: window.location.href,
|
||||
bodyText: document.body ? document.body.innerText || '' : '',
|
||||
seller: toJson(model?.sellerModel),
|
||||
services: toJson(model?.shippingServices?.fields?.buyerProtectionModel ?? []),
|
||||
};
|
||||
})()
|
||||
`) as StoreItemSeed;
|
||||
|
||||
const hasSellerContext = !!cleanText(seed?.seller?.memberId) || !!cleanText(seed?.seller?.winportUrl);
|
||||
if (!hasSellerContext) {
|
||||
throw new CommandExecutionError(
|
||||
'1688 store seed item did not expose seller context',
|
||||
'当前 tab 非商品详情上下文,请切到 detail.1688.com 商品页并重试',
|
||||
);
|
||||
}
|
||||
|
||||
return seed;
|
||||
}
|
||||
|
||||
function hasAnyEvidence(
|
||||
storePayload: StoreBrowserPayload | null,
|
||||
contactPayload: StoreBrowserPayload | null,
|
||||
seed: StoreItemSeed | null,
|
||||
): boolean {
|
||||
return !!cleanText(storePayload?.bodyText)
|
||||
|| !!cleanText(contactPayload?.bodyText)
|
||||
|| !!cleanText(seed?.bodyText);
|
||||
}
|
||||
|
||||
cli({
|
||||
site: '1688',
|
||||
name: 'store',
|
||||
description: '1688 店铺/供应商公开信息(联系方式、主营、入驻年限、公开服务信号)',
|
||||
domain: 'www.1688.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '1688 店铺 URL 或 member ID(如 b2b-22154705262941f196)',
|
||||
},
|
||||
],
|
||||
columns: ['store_name', 'years_on_platform_text', 'location', 'return_rate_text'],
|
||||
func: async (page, kwargs) => {
|
||||
const rawInput = String(kwargs.input ?? '');
|
||||
const resolvedUrl = resolveStoreUrl(rawInput);
|
||||
const explicitMemberId = extractMemberId(rawInput);
|
||||
|
||||
const storePayload = await readStorePayload(page, resolvedUrl, 'store');
|
||||
const contactUrl = firstContactUrl(storePayload.contactLinks ?? []) || buildContactUrl(storePayload.href || resolvedUrl);
|
||||
const contactPayload = contactUrl ? await readStorePayload(page, contactUrl, 'store contact') : null;
|
||||
const offerId = extractOfferId(rawInput)
|
||||
|| firstOfferId(storePayload.offerLinks ?? [])
|
||||
|| firstOfferId(contactPayload?.offerLinks ?? []);
|
||||
|
||||
let seed: StoreItemSeed | null = null;
|
||||
if (offerId) {
|
||||
try {
|
||||
seed = await readItemSeed(page, offerId);
|
||||
} catch (error) {
|
||||
if (!(error instanceof CommandExecutionError)) throw error;
|
||||
}
|
||||
}
|
||||
|
||||
if (!hasAnyEvidence(storePayload, contactPayload, seed)) {
|
||||
throw new EmptyResultError(
|
||||
'1688 store',
|
||||
'Store page is reachable but no visible fields were extracted. Open the store page in Chrome and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
return [
|
||||
normalizeStorePayload({
|
||||
resolvedUrl,
|
||||
storePayload,
|
||||
contactPayload,
|
||||
seed,
|
||||
explicitMemberId,
|
||||
}),
|
||||
];
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeStorePayload,
|
||||
safeCanonicalStoreUrl,
|
||||
buildContactUrl,
|
||||
firstNamedLine,
|
||||
firstMetric,
|
||||
extractReturnRate,
|
||||
firstOfferId,
|
||||
firstContactUrl,
|
||||
};
|
||||
@@ -1,9 +1,9 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './bestsellers.js';
|
||||
import { __test__ } from './rankings.js';
|
||||
|
||||
describe('amazon bestsellers normalization', () => {
|
||||
it('normalizes bestseller cards and infers review counts from card text', () => {
|
||||
const result = __test__.normalizeBestsellerCandidate({
|
||||
const result = __test__.normalizeRankingCandidate({
|
||||
asin: 'B0DR31GC3D',
|
||||
title: '',
|
||||
href: 'https://www.amazon.com/NUTIKAS-Shelves-Desktop-Orgnizer-Shlef/dp/B0DR31GC3D/ref=zg_bs',
|
||||
@@ -11,7 +11,16 @@ describe('amazon bestsellers normalization', () => {
|
||||
rating_text: '4.3 out of 5 stars',
|
||||
review_count_text: '',
|
||||
card_text: 'Desk Shelves Desktop Organizer Shlef\n4.3 out of 5 stars\n435\n$25.92',
|
||||
}, 2, 'Amazon Best Sellers: Best Desktop & Off-Surface Shelves', 'https://www.amazon.com/example');
|
||||
}, {
|
||||
listType: 'bestsellers',
|
||||
rankFallback: 2,
|
||||
listTitle: 'Amazon Best Sellers: Best Desktop & Off-Surface Shelves',
|
||||
sourceUrl: 'https://www.amazon.com/example',
|
||||
categoryTitle: null,
|
||||
categoryUrl: 'https://www.amazon.com/example',
|
||||
categoryPath: [],
|
||||
visibleCategoryLinks: [],
|
||||
});
|
||||
|
||||
expect(result.rank).toBe(2);
|
||||
expect(result.asin).toBe('B0DR31GC3D');
|
||||
|
||||
@@ -1,180 +1,8 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
buildProvenance,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
extractReviewCountFromCardText,
|
||||
firstMeaningfulLine,
|
||||
normalizeProductUrl,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
resolveBestsellersUrl,
|
||||
uniqueNonEmpty,
|
||||
assertUsableState,
|
||||
gotoAndReadState,
|
||||
} from './shared.js';
|
||||
import { cli } from '../../registry.js';
|
||||
import { createRankingCliOptions } from './rankings.js';
|
||||
|
||||
interface BestsellersPagePayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
list_title?: string;
|
||||
cards?: Array<{
|
||||
rank_text?: string | null;
|
||||
asin?: string | null;
|
||||
title?: string | null;
|
||||
href?: string | null;
|
||||
price_text?: string | null;
|
||||
rating_text?: string | null;
|
||||
review_count_text?: string | null;
|
||||
card_text?: string | null;
|
||||
}>;
|
||||
page_links?: string[];
|
||||
}
|
||||
|
||||
function normalizeBestsellerCandidate(
|
||||
candidate: NonNullable<BestsellersPagePayload['cards']>[number],
|
||||
rank: number,
|
||||
listTitle: string | null,
|
||||
sourceUrl: string,
|
||||
): Record<string, unknown> {
|
||||
const productUrl = normalizeProductUrl(candidate.href);
|
||||
const asin = extractAsin(candidate.asin ?? '') ?? extractAsin(productUrl ?? '') ?? null;
|
||||
const title = cleanText(candidate.title) || firstMeaningfulLine(candidate.card_text);
|
||||
const price = parsePriceText(cleanText(candidate.price_text) || candidate.card_text);
|
||||
const ratingText = cleanText(candidate.rating_text) || null;
|
||||
const reviewCountText = cleanText(candidate.review_count_text)
|
||||
|| extractReviewCountFromCardText(candidate.card_text)
|
||||
|| null;
|
||||
const provenance = buildProvenance(sourceUrl);
|
||||
|
||||
return {
|
||||
rank,
|
||||
asin,
|
||||
title: title || null,
|
||||
product_url: productUrl,
|
||||
list_title: listTitle,
|
||||
...provenance,
|
||||
price_text: price.price_text,
|
||||
price_value: price.price_value,
|
||||
currency: price.currency,
|
||||
rating_text: ratingText,
|
||||
rating_value: parseRatingValue(ratingText),
|
||||
review_count_text: reviewCountText,
|
||||
review_count: parseReviewCount(reviewCountText),
|
||||
};
|
||||
}
|
||||
|
||||
async function readBestsellersPage(page: IPage, url: string): Promise<BestsellersPagePayload> {
|
||||
const state = await gotoAndReadState(page, url, 2500, 'bestsellers');
|
||||
assertUsableState(state, 'bestsellers');
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
list_title:
|
||||
document.querySelector('#zg_banner_text')?.textContent
|
||||
|| document.querySelector('h1')?.textContent
|
||||
|| '',
|
||||
cards: Array.from(document.querySelectorAll('.p13n-sc-uncoverable-faceout'))
|
||||
.map((card) => ({
|
||||
rank_text:
|
||||
card.querySelector('.zg-bdg-text')?.textContent
|
||||
|| card.querySelector('[class*="rank"]')?.textContent
|
||||
|| '',
|
||||
asin: card.id || '',
|
||||
title:
|
||||
card.querySelector('[class*="line-clamp"]')?.textContent
|
||||
|| card.querySelector('img')?.getAttribute('alt')
|
||||
|| '',
|
||||
href: card.querySelector('a[href*="/dp/"]')?.href || '',
|
||||
price_text: card.querySelector('.a-price .a-offscreen')?.textContent || '',
|
||||
rating_text: card.querySelector('[aria-label*="out of 5 stars"]')?.getAttribute('aria-label') || '',
|
||||
review_count_text:
|
||||
card.querySelector('a[href*="#customerReviews"]')?.textContent
|
||||
|| card.querySelector('.a-size-small')?.textContent
|
||||
|| '',
|
||||
card_text: card.innerText || '',
|
||||
})),
|
||||
page_links: Array.from(document.querySelectorAll('li.a-normal a, li.a-selected a'))
|
||||
.map((anchor) => anchor.href || '')
|
||||
.filter((href) => /\\/zgbs\\//.test(href) && /(?:[?&]pg=|ref=zg_bs_pg_)/.test(href)),
|
||||
}))()
|
||||
`) as BestsellersPagePayload;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'amazon',
|
||||
name: 'bestsellers',
|
||||
cli(createRankingCliOptions({
|
||||
commandName: 'bestsellers',
|
||||
listType: 'bestsellers',
|
||||
description: 'Amazon Best Sellers pages for category candidate discovery',
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
positional: true,
|
||||
help: 'Best sellers URL or /zgbs path. Omit to use the root Best Sellers page.',
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: 100,
|
||||
help: 'Maximum number of ranked items to return (default 100)',
|
||||
},
|
||||
],
|
||||
columns: ['rank', 'asin', 'title', 'price_text', 'rating_value', 'review_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = Math.max(1, Number(kwargs.limit) || 100);
|
||||
const initialUrl = resolveBestsellersUrl(typeof kwargs.input === 'string' ? kwargs.input : undefined);
|
||||
|
||||
const queue = [initialUrl];
|
||||
const visited = new Set<string>();
|
||||
const seenAsins = new Set<string>();
|
||||
const results: Record<string, unknown>[] = [];
|
||||
let listTitle: string | null = null;
|
||||
|
||||
while (queue.length > 0 && results.length < limit) {
|
||||
const nextUrl = queue.shift()!;
|
||||
if (visited.has(nextUrl)) continue;
|
||||
visited.add(nextUrl);
|
||||
|
||||
const payload = await readBestsellersPage(page, nextUrl);
|
||||
const sourceUrl = cleanText(payload.href) || nextUrl;
|
||||
listTitle = cleanText(payload.list_title) || cleanText(payload.title) || listTitle;
|
||||
const cards = payload.cards ?? [];
|
||||
|
||||
for (const card of cards) {
|
||||
const normalized = normalizeBestsellerCandidate(card, results.length + 1, listTitle, sourceUrl);
|
||||
const asin = cleanText(String(normalized.asin ?? ''));
|
||||
if (!asin || seenAsins.has(asin)) continue;
|
||||
seenAsins.add(asin);
|
||||
results.push(normalized);
|
||||
if (results.length >= limit) break;
|
||||
}
|
||||
|
||||
const pageLinks = uniqueNonEmpty(payload.page_links ?? []);
|
||||
for (const href of pageLinks) {
|
||||
if (!visited.has(href) && !queue.includes(href)) {
|
||||
queue.push(href);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (results.length === 0) {
|
||||
throw new CommandExecutionError(
|
||||
'amazon bestsellers did not expose any ranked items',
|
||||
'Open the same best sellers page in Chrome, verify it is a real Amazon ranking page, and retry.',
|
||||
);
|
||||
}
|
||||
|
||||
return results.slice(0, limit);
|
||||
},
|
||||
});
|
||||
|
||||
export const __test__ = {
|
||||
normalizeBestsellerCandidate,
|
||||
};
|
||||
}));
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
import { cli } from '../../registry.js';
|
||||
import { createRankingCliOptions } from './rankings.js';
|
||||
|
||||
cli(createRankingCliOptions({
|
||||
commandName: 'movers-shakers',
|
||||
listType: 'movers_shakers',
|
||||
description: 'Amazon Movers & Shakers pages for short-term growth signals',
|
||||
}));
|
||||
@@ -0,0 +1,8 @@
|
||||
import { cli } from '../../registry.js';
|
||||
import { createRankingCliOptions } from './rankings.js';
|
||||
|
||||
cli(createRankingCliOptions({
|
||||
commandName: 'new-releases',
|
||||
listType: 'new_releases',
|
||||
description: 'Amazon New Releases pages for early momentum discovery',
|
||||
}));
|
||||
@@ -0,0 +1,47 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { __test__ } from './rankings.js';
|
||||
|
||||
describe('amazon rankings helpers', () => {
|
||||
it('normalizes ranking candidates with unified schema', () => {
|
||||
const result = __test__.normalizeRankingCandidate(
|
||||
{
|
||||
rank_text: '#3',
|
||||
asin: 'B0DR31GC3D',
|
||||
title: 'Desk Shelves Desktop Organizer',
|
||||
href: 'https://www.amazon.com/dp/B0DR31GC3D/ref=zg_bs',
|
||||
price_text: '$25.92',
|
||||
rating_text: '4.3 out of 5 stars',
|
||||
review_count_text: '435',
|
||||
},
|
||||
{
|
||||
listType: 'new_releases',
|
||||
rankFallback: 3,
|
||||
listTitle: 'Amazon New Releases',
|
||||
sourceUrl: 'https://www.amazon.com/gp/new-releases',
|
||||
categoryTitle: 'Home & Kitchen',
|
||||
categoryUrl: 'https://www.amazon.com/gp/new-releases/home-garden',
|
||||
categoryPath: ['Home & Kitchen'],
|
||||
visibleCategoryLinks: [{ title: 'Storage', url: 'https://www.amazon.com/gp/new-releases/storage', node_id: null }],
|
||||
},
|
||||
);
|
||||
|
||||
expect(result.list_type).toBe('new_releases');
|
||||
expect(result.rank).toBe(3);
|
||||
expect(result.asin).toBe('B0DR31GC3D');
|
||||
expect(result.product_url).toBe('https://www.amazon.com/dp/B0DR31GC3D');
|
||||
expect(result.category_title).toBe('Home & Kitchen');
|
||||
expect(result.visible_category_links).toEqual([
|
||||
{ title: 'Storage', url: 'https://www.amazon.com/gp/new-releases/storage', node_id: null },
|
||||
]);
|
||||
});
|
||||
|
||||
it('deduplicates category links and parses rank fallback', () => {
|
||||
const links = __test__.normalizeVisibleCategoryLinks([
|
||||
{ title: 'Kitchen', url: '/gp/new-releases/home-garden' },
|
||||
{ title: 'Kitchen', url: 'https://www.amazon.com/gp/new-releases/home-garden' },
|
||||
{ title: 'Storage', url: '/gp/new-releases/storage', node_id: '1064954' },
|
||||
]);
|
||||
expect(links.length).toBe(2);
|
||||
expect(__test__.parseRank('N/A', 8)).toBe(8);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,312 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import { Strategy, type CliOptions } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
assertUsableState,
|
||||
buildProvenance,
|
||||
cleanText,
|
||||
extractAsin,
|
||||
extractCategoryNodeId,
|
||||
extractReviewCountFromCardText,
|
||||
firstMeaningfulLine,
|
||||
gotoAndReadState,
|
||||
isRankingPaginationUrl,
|
||||
normalizeProductUrl,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
resolveRankingUrl,
|
||||
toAbsoluteAmazonUrl,
|
||||
uniqueNonEmpty,
|
||||
type AmazonRankingListType,
|
||||
} from './shared.js';
|
||||
|
||||
export interface RankingCardPayload {
|
||||
rank_text?: string | null;
|
||||
asin?: string | null;
|
||||
title?: string | null;
|
||||
href?: string | null;
|
||||
price_text?: string | null;
|
||||
rating_text?: string | null;
|
||||
review_count_text?: string | null;
|
||||
card_text?: string | null;
|
||||
}
|
||||
|
||||
interface RankingPagePayload {
|
||||
href?: string;
|
||||
title?: string;
|
||||
list_title?: string;
|
||||
category_title?: string;
|
||||
category_path?: string[];
|
||||
cards?: RankingCardPayload[];
|
||||
page_links?: string[];
|
||||
visible_category_links?: Array<{
|
||||
title?: string | null;
|
||||
url?: string | null;
|
||||
node_id?: string | null;
|
||||
}>;
|
||||
}
|
||||
|
||||
interface RankingCommandDefinition {
|
||||
commandName: string;
|
||||
listType: AmazonRankingListType;
|
||||
description: string;
|
||||
}
|
||||
|
||||
interface RankingNormalizeContext {
|
||||
listType: AmazonRankingListType;
|
||||
rankFallback: number;
|
||||
listTitle: string | null;
|
||||
sourceUrl: string;
|
||||
categoryTitle: string | null;
|
||||
categoryUrl: string | null;
|
||||
categoryPath: string[];
|
||||
visibleCategoryLinks: Array<{ title: string; url: string; node_id: string | null }>;
|
||||
}
|
||||
|
||||
function parseRank(rawRank: string | null | undefined, fallback: number): number {
|
||||
const normalized = cleanText(rawRank);
|
||||
const match = normalized.match(/(\d{1,4})/);
|
||||
if (!match) return fallback;
|
||||
const parsed = Number.parseInt(match[1], 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function normalizeVisibleCategoryLinks(
|
||||
links: RankingPagePayload['visible_category_links'],
|
||||
): Array<{ title: string; url: string; node_id: string | null }> {
|
||||
const normalized = (links ?? [])
|
||||
.map((entry) => ({
|
||||
title: cleanText(entry?.title),
|
||||
url: toAbsoluteAmazonUrl(entry?.url) ?? '',
|
||||
node_id: cleanText(entry?.node_id) || extractCategoryNodeId(entry?.url) || null,
|
||||
}))
|
||||
.filter((entry) => Boolean(entry.title) && Boolean(entry.url));
|
||||
|
||||
const seen = new Set<string>();
|
||||
const deduped: Array<{ title: string; url: string; node_id: string | null }> = [];
|
||||
for (const entry of normalized) {
|
||||
if (seen.has(entry.url)) continue;
|
||||
seen.add(entry.url);
|
||||
deduped.push(entry);
|
||||
}
|
||||
return deduped;
|
||||
}
|
||||
|
||||
export function normalizeRankingCandidate(
|
||||
candidate: RankingCardPayload,
|
||||
context: RankingNormalizeContext,
|
||||
): Record<string, unknown> {
|
||||
const productUrl = normalizeProductUrl(candidate.href);
|
||||
const asin = extractAsin(candidate.asin ?? '') ?? extractAsin(productUrl ?? '') ?? null;
|
||||
const title = cleanText(candidate.title) || firstMeaningfulLine(candidate.card_text);
|
||||
const price = parsePriceText(cleanText(candidate.price_text) || candidate.card_text);
|
||||
const ratingText = cleanText(candidate.rating_text) || null;
|
||||
const reviewCountText = cleanText(candidate.review_count_text)
|
||||
|| extractReviewCountFromCardText(candidate.card_text)
|
||||
|| null;
|
||||
const provenance = buildProvenance(context.sourceUrl);
|
||||
const categoryUrl = context.categoryUrl || context.sourceUrl;
|
||||
|
||||
return {
|
||||
list_type: context.listType,
|
||||
rank: parseRank(candidate.rank_text, context.rankFallback),
|
||||
asin,
|
||||
title: title || null,
|
||||
product_url: productUrl,
|
||||
price_text: price.price_text,
|
||||
price_value: price.price_value,
|
||||
currency: price.currency,
|
||||
rating_text: ratingText,
|
||||
rating_value: parseRatingValue(ratingText),
|
||||
review_count_text: reviewCountText,
|
||||
review_count: parseReviewCount(reviewCountText),
|
||||
list_title: context.listTitle,
|
||||
category_title: context.categoryTitle,
|
||||
category_url: categoryUrl,
|
||||
category_node_id: extractCategoryNodeId(categoryUrl),
|
||||
category_path: context.categoryPath,
|
||||
visible_category_links: context.visibleCategoryLinks,
|
||||
...provenance,
|
||||
};
|
||||
}
|
||||
|
||||
async function readRankingPage(
|
||||
page: IPage,
|
||||
listType: AmazonRankingListType,
|
||||
url: string,
|
||||
): Promise<RankingPagePayload> {
|
||||
const state = await gotoAndReadState(page, url, 2500, listType);
|
||||
assertUsableState(state, listType);
|
||||
|
||||
return await page.evaluate(`
|
||||
(() => ({
|
||||
href: window.location.href,
|
||||
title: document.title || '',
|
||||
list_title:
|
||||
document.querySelector('#zg_banner_text')?.textContent
|
||||
|| document.querySelector('h1')?.textContent
|
||||
|| '',
|
||||
category_title:
|
||||
document.querySelector('#zg_browseRoot .zg_selected')?.textContent
|
||||
|| document.querySelector('#wayfinding-breadcrumbs_feature_div ul li:last-child')?.textContent
|
||||
|| document.querySelector('#wayfinding-breadcrumbs_container ul li:last-child')?.textContent
|
||||
|| '',
|
||||
category_path: Array.from(document.querySelectorAll(
|
||||
'#zg_browseRoot ul li a, #zg_browseRoot ul li span, ' +
|
||||
'#wayfinding-breadcrumbs_feature_div ul li a, #wayfinding-breadcrumbs_feature_div ul li span.a-list-item, ' +
|
||||
'#wayfinding-breadcrumbs_container ul li a, #wayfinding-breadcrumbs_container ul li span.a-list-item'
|
||||
))
|
||||
.map((entry) => (entry.textContent || '').trim())
|
||||
.filter(Boolean),
|
||||
cards: Array.from(document.querySelectorAll(
|
||||
'.p13n-sc-uncoverable-faceout, .zg-grid-general-faceout, [data-asin][class*="p13n"]'
|
||||
)).map((card) => ({
|
||||
rank_text:
|
||||
card.querySelector('.zg-bdg-text')?.textContent
|
||||
|| card.querySelector('[class*="rank"]')?.textContent
|
||||
|| '',
|
||||
asin:
|
||||
card.getAttribute('data-asin')
|
||||
|| card.getAttribute('id')
|
||||
|| '',
|
||||
title:
|
||||
card.querySelector('[class*="line-clamp"]')?.textContent
|
||||
|| card.querySelector('img')?.getAttribute('alt')
|
||||
|| card.querySelector('a[href*="/dp/"]')?.textContent
|
||||
|| '',
|
||||
href:
|
||||
card.querySelector('a[href*="/dp/"], a[href*="/gp/product/"]')?.href
|
||||
|| '',
|
||||
price_text:
|
||||
card.querySelector('.a-price .a-offscreen')?.textContent
|
||||
|| card.querySelector('.a-color-price')?.textContent
|
||||
|| '',
|
||||
rating_text:
|
||||
card.querySelector('[aria-label*="out of 5 stars"]')?.getAttribute('aria-label')
|
||||
|| '',
|
||||
review_count_text:
|
||||
card.querySelector('a[href*="#customerReviews"]')?.textContent
|
||||
|| card.querySelector('.a-size-small')?.textContent
|
||||
|| '',
|
||||
card_text: card.innerText || '',
|
||||
})),
|
||||
page_links: Array.from(document.querySelectorAll('.a-pagination a[href], li.a-normal a[href], li.a-selected a[href]'))
|
||||
.map((anchor) => anchor.href || '')
|
||||
.filter(Boolean),
|
||||
visible_category_links: Array.from(document.querySelectorAll(
|
||||
'#zg_browseRoot a[href], #zg-left-col a[href], [class*="zg-browse"] a[href]'
|
||||
)).map((anchor) => ({
|
||||
title: (anchor.textContent || '').trim(),
|
||||
url: anchor.href || '',
|
||||
node_id:
|
||||
anchor.getAttribute('data-node-id')
|
||||
|| anchor.dataset?.nodeid
|
||||
|| '',
|
||||
}))
|
||||
.filter((entry) => entry.title && entry.url),
|
||||
}))()
|
||||
`) as RankingPagePayload;
|
||||
}
|
||||
|
||||
function createEmptyResultHint(commandName: string): string {
|
||||
return [
|
||||
`Open the same Amazon ${commandName} page in shared Chrome and verify ranked items are visible.`,
|
||||
'If the page shows a robot check, clear it manually and retry.',
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
export function createRankingCliOptions(definition: RankingCommandDefinition): CliOptions {
|
||||
return {
|
||||
site: 'amazon',
|
||||
name: definition.commandName,
|
||||
description: definition.description,
|
||||
domain: 'amazon.com',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'input',
|
||||
positional: true,
|
||||
help: 'Ranking URL or supported Amazon path. Omit to use the list root.',
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: 100,
|
||||
help: 'Maximum number of ranked items to return (default 100)',
|
||||
},
|
||||
],
|
||||
columns: ['list_type', 'rank', 'asin', 'title', 'price_text', 'rating_value', 'review_count'],
|
||||
func: async (page, kwargs) => {
|
||||
const limit = Math.max(1, Number(kwargs.limit) || 100);
|
||||
const initialUrl = resolveRankingUrl(definition.listType, typeof kwargs.input === 'string' ? kwargs.input : undefined);
|
||||
|
||||
const queue = [initialUrl];
|
||||
const visited = new Set<string>();
|
||||
const seenEntityKeys = new Set<string>();
|
||||
const results: Record<string, unknown>[] = [];
|
||||
let listTitle: string | null = null;
|
||||
|
||||
while (queue.length > 0 && results.length < limit) {
|
||||
const nextUrl = queue.shift()!;
|
||||
if (visited.has(nextUrl)) continue;
|
||||
visited.add(nextUrl);
|
||||
|
||||
const payload = await readRankingPage(page, definition.listType, nextUrl);
|
||||
const sourceUrl = cleanText(payload.href) || nextUrl;
|
||||
listTitle = cleanText(payload.list_title) || cleanText(payload.title) || listTitle;
|
||||
const categoryPath = uniqueNonEmpty(payload.category_path ?? []);
|
||||
const categoryTitle = cleanText(payload.category_title)
|
||||
|| (categoryPath.length > 0 ? categoryPath[categoryPath.length - 1] : '');
|
||||
const visibleCategoryLinks = normalizeVisibleCategoryLinks(payload.visible_category_links);
|
||||
const cards = payload.cards ?? [];
|
||||
|
||||
for (const card of cards) {
|
||||
const normalized = normalizeRankingCandidate(card, {
|
||||
listType: definition.listType,
|
||||
rankFallback: results.length + 1,
|
||||
listTitle,
|
||||
sourceUrl,
|
||||
categoryTitle: categoryTitle || null,
|
||||
categoryUrl: sourceUrl,
|
||||
categoryPath,
|
||||
visibleCategoryLinks,
|
||||
});
|
||||
|
||||
const dedupeKey = cleanText(String(normalized.asin ?? ''))
|
||||
|| cleanText(String(normalized.product_url ?? ''));
|
||||
if (dedupeKey && seenEntityKeys.has(dedupeKey)) continue;
|
||||
if (dedupeKey) seenEntityKeys.add(dedupeKey);
|
||||
|
||||
results.push(normalized);
|
||||
if (results.length >= limit) break;
|
||||
}
|
||||
|
||||
const pageLinks = uniqueNonEmpty(payload.page_links ?? []);
|
||||
for (const href of pageLinks) {
|
||||
const absolute = toAbsoluteAmazonUrl(href);
|
||||
if (!absolute || !isRankingPaginationUrl(definition.listType, absolute)) continue;
|
||||
if (!visited.has(absolute) && !queue.includes(absolute)) {
|
||||
queue.push(absolute);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (results.length === 0) {
|
||||
throw new CommandExecutionError(
|
||||
`amazon ${definition.commandName} did not expose any ranked items`,
|
||||
createEmptyResultHint(definition.commandName),
|
||||
);
|
||||
}
|
||||
|
||||
return results.slice(0, limit);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const __test__ = {
|
||||
parseRank,
|
||||
normalizeVisibleCategoryLinks,
|
||||
normalizeRankingCandidate,
|
||||
};
|
||||
@@ -34,4 +34,20 @@ describe('amazon shared helpers', () => {
|
||||
expect(__test__.resolveBestsellersUrl('/Best-Sellers/zgbs')).toBe('https://www.amazon.com/Best-Sellers/zgbs');
|
||||
expect(() => __test__.resolveBestsellersUrl('desk shelf organizer')).toThrow('amazon bestsellers expects a best sellers URL or /zgbs path');
|
||||
});
|
||||
|
||||
it('resolves and validates all ranking list URLs', () => {
|
||||
expect(__test__.resolveRankingUrl('new_releases')).toBe('https://www.amazon.com/gp/new-releases');
|
||||
expect(__test__.resolveRankingUrl('movers_shakers')).toBe('https://www.amazon.com/gp/movers-and-shakers');
|
||||
expect(__test__.resolveRankingUrl('new_releases', '/gp/new-releases/kitchen')).toBe('https://www.amazon.com/gp/new-releases/kitchen');
|
||||
expect(__test__.resolveRankingUrl(
|
||||
'bestsellers',
|
||||
'https://www.amazon.com/Best-Sellers/zgbs/ref=zg_bsnr_tab_bs',
|
||||
)).toBe('https://www.amazon.com/Best-Sellers/zgbs');
|
||||
expect(() => __test__.resolveRankingUrl('movers_shakers', 'https://example.com/gp/movers-and-shakers')).toThrow('Invalid Amazon URL');
|
||||
});
|
||||
|
||||
it('extracts category node id from URL best effort', () => {
|
||||
expect(__test__.extractCategoryNodeId('https://www.amazon.com/Best-Sellers-Home-Kitchen/zgbs/home-garden/3744371')).toBe('3744371');
|
||||
expect(__test__.extractCategoryNodeId('https://www.amazon.com/s?k=desk+organizer&rh=n%3A1064954')).toBe('1064954');
|
||||
});
|
||||
});
|
||||
|
||||
+134
-12
@@ -5,6 +5,8 @@ export const SITE = 'amazon';
|
||||
export const DOMAIN = 'amazon.com';
|
||||
export const HOME_URL = 'https://www.amazon.com/';
|
||||
export const BESTSELLERS_URL = 'https://www.amazon.com/Best-Sellers/zgbs';
|
||||
export const NEW_RELEASES_URL = 'https://www.amazon.com/gp/new-releases';
|
||||
export const MOVERS_SHAKERS_URL = 'https://www.amazon.com/gp/movers-and-shakers';
|
||||
export const SEARCH_URL_PREFIX = 'https://www.amazon.com/s?k=';
|
||||
export const PRODUCT_URL_PREFIX = 'https://www.amazon.com/dp/';
|
||||
export const DISCUSSION_URL_PREFIX = 'https://www.amazon.com/product-reviews/';
|
||||
@@ -28,6 +30,40 @@ const ROBOT_TEXT_PATTERNS = [
|
||||
'To discuss automated access to Amazon data please contact',
|
||||
];
|
||||
|
||||
export type AmazonRankingListType = 'bestsellers' | 'new_releases' | 'movers_shakers';
|
||||
|
||||
interface AmazonRankingSpec {
|
||||
commandName: string;
|
||||
rootUrl: string;
|
||||
pathPattern: RegExp;
|
||||
invalidInputMessage: string;
|
||||
invalidInputHint: string;
|
||||
}
|
||||
|
||||
const AMAZON_RANKING_SPECS: Record<AmazonRankingListType, AmazonRankingSpec> = {
|
||||
bestsellers: {
|
||||
commandName: 'bestsellers',
|
||||
rootUrl: BESTSELLERS_URL,
|
||||
pathPattern: /(?:^|\/)zgbs(?:\/|$)/i,
|
||||
invalidInputMessage: 'amazon bestsellers expects a best sellers URL or /zgbs path',
|
||||
invalidInputHint: 'Example: opencli amazon bestsellers https://www.amazon.com/Best-Sellers/zgbs',
|
||||
},
|
||||
new_releases: {
|
||||
commandName: 'new-releases',
|
||||
rootUrl: NEW_RELEASES_URL,
|
||||
pathPattern: /\/gp\/new-releases(?:\/|$)/i,
|
||||
invalidInputMessage: 'amazon new-releases expects a new releases URL or /gp/new-releases path',
|
||||
invalidInputHint: 'Example: opencli amazon new-releases https://www.amazon.com/gp/new-releases',
|
||||
},
|
||||
movers_shakers: {
|
||||
commandName: 'movers-shakers',
|
||||
rootUrl: MOVERS_SHAKERS_URL,
|
||||
pathPattern: /\/gp\/movers-and-shakers(?:\/|$)/i,
|
||||
invalidInputMessage: 'amazon movers-shakers expects a movers-and-shakers URL or /gp/movers-and-shakers path',
|
||||
invalidInputHint: 'Example: opencli amazon movers-shakers https://www.amazon.com/gp/movers-and-shakers',
|
||||
},
|
||||
};
|
||||
|
||||
export interface ProvenanceFields {
|
||||
source_url: string;
|
||||
fetched_at: string;
|
||||
@@ -115,23 +151,105 @@ export function buildDiscussionUrl(input: string): string {
|
||||
return `${DISCUSSION_URL_PREFIX}${asin}`;
|
||||
}
|
||||
|
||||
export function resolveBestsellersUrl(input?: string): string {
|
||||
function getRankingSpec(listType: AmazonRankingListType): AmazonRankingSpec {
|
||||
return AMAZON_RANKING_SPECS[listType];
|
||||
}
|
||||
|
||||
export function isSupportedRankingPath(listType: AmazonRankingListType, inputUrl: string): boolean {
|
||||
try {
|
||||
const url = new URL(inputUrl);
|
||||
return getRankingSpec(listType).pathPattern.test(url.pathname);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function resolveRankingUrl(listType: AmazonRankingListType, input?: string): string {
|
||||
const spec = getRankingSpec(listType);
|
||||
const normalized = cleanText(input);
|
||||
if (!normalized) return BESTSELLERS_URL;
|
||||
if (normalized === 'root') return BESTSELLERS_URL;
|
||||
if (!normalized || normalized === 'root') return spec.rootUrl;
|
||||
|
||||
let candidateUrl: string;
|
||||
if (normalized.startsWith('/')) {
|
||||
return new URL(normalized, HOME_URL).toString();
|
||||
candidateUrl = new URL(normalized, HOME_URL).toString();
|
||||
} else if (/^https?:\/\//i.test(normalized)) {
|
||||
candidateUrl = canonicalizeAmazonUrl(normalized);
|
||||
} else if (normalized.includes('amazon.') && normalized.includes('/')) {
|
||||
candidateUrl = canonicalizeAmazonUrl(`https://${normalized.replace(/^\/+/, '')}`);
|
||||
} else {
|
||||
throw new ArgumentError(spec.invalidInputMessage, spec.invalidInputHint);
|
||||
}
|
||||
if (/^https?:\/\//i.test(normalized)) {
|
||||
return canonicalizeAmazonUrl(normalized);
|
||||
|
||||
if (!isSupportedRankingPath(listType, candidateUrl)) {
|
||||
throw new ArgumentError(spec.invalidInputMessage, spec.invalidInputHint);
|
||||
}
|
||||
if (normalized.includes('/zgbs/')) {
|
||||
return canonicalizeAmazonUrl(`https://${normalized.replace(/^\/+/, '')}`);
|
||||
return normalizeRankingInputUrl(candidateUrl);
|
||||
}
|
||||
|
||||
function normalizeRankingInputUrl(inputUrl: string): string {
|
||||
try {
|
||||
const url = new URL(inputUrl);
|
||||
const normalizedPathSegments = url.pathname
|
||||
.split('/')
|
||||
.filter(Boolean)
|
||||
.filter((segment) => !/^ref=/i.test(segment));
|
||||
url.pathname = `/${normalizedPathSegments.join('/')}`;
|
||||
url.hash = '';
|
||||
// Ranking pages are frequently shared with tracking refs that can land on unstable variants.
|
||||
// Dropping ref keeps the canonical ranking path while preserving useful params (for example pg=2).
|
||||
url.searchParams.delete('ref');
|
||||
return url.toString();
|
||||
} catch {
|
||||
return inputUrl;
|
||||
}
|
||||
throw new ArgumentError(
|
||||
'amazon bestsellers expects a best sellers URL or /zgbs path',
|
||||
'Example: opencli amazon bestsellers https://www.amazon.com/Best-Sellers/zgbs',
|
||||
);
|
||||
}
|
||||
|
||||
export function isRankingPaginationUrl(listType: AmazonRankingListType, inputUrl: string): boolean {
|
||||
const absolute = toAbsoluteAmazonUrl(inputUrl);
|
||||
if (!absolute || !isSupportedRankingPath(listType, absolute)) return false;
|
||||
|
||||
try {
|
||||
const url = new URL(absolute);
|
||||
const ref = cleanText(url.searchParams.get('ref')).toLowerCase();
|
||||
// pg= query param is the most reliable pagination indicator across all ranking lists
|
||||
return url.searchParams.has('pg')
|
||||
|| /(?:^|_)pg(?:_|$)/.test(ref)
|
||||
// Amazon ranking pagination refs: zg_bs_pg_ (bestsellers), zg_bsnr_pg_ (new releases), zg_bsms_pg_ (movers & shakers)
|
||||
|| /zg_bs(?:nr|ms)?_pg_/.test(ref);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function extractCategoryNodeId(inputUrl: string | null | undefined): string | null {
|
||||
const absolute = toAbsoluteAmazonUrl(inputUrl);
|
||||
if (!absolute) return null;
|
||||
|
||||
try {
|
||||
const url = new URL(absolute);
|
||||
|
||||
for (const key of ['node', 'nodeid', 'nodeId', 'browseNode']) {
|
||||
const value = cleanText(url.searchParams.get(key));
|
||||
if (/^\d{4,}$/.test(value)) return value;
|
||||
}
|
||||
|
||||
const rhValue = cleanText(url.searchParams.get('rh'));
|
||||
const rhMatch = decodeURIComponent(rhValue).match(/(?:^|,)\s*n:(\d{4,})(?:,|$)/i);
|
||||
if (rhMatch) return rhMatch[1];
|
||||
|
||||
const pathMatches = [...url.pathname.matchAll(/\/(\d{4,})(?=\/|$)/g)];
|
||||
if (pathMatches.length > 0) {
|
||||
return pathMatches[pathMatches.length - 1][1];
|
||||
}
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export function resolveBestsellersUrl(input?: string): string {
|
||||
return resolveRankingUrl('bestsellers', input);
|
||||
}
|
||||
|
||||
export function canonicalizeAmazonUrl(input: string): string {
|
||||
@@ -305,6 +423,10 @@ export const __test__ = {
|
||||
buildProductUrl,
|
||||
buildDiscussionUrl,
|
||||
resolveBestsellersUrl,
|
||||
resolveRankingUrl,
|
||||
isSupportedRankingPath,
|
||||
isRankingPaginationUrl,
|
||||
extractCategoryNodeId,
|
||||
parsePriceText,
|
||||
parseRatingValue,
|
||||
parseReviewCount,
|
||||
|
||||
@@ -4,7 +4,8 @@ const { mockApiGet } = vi.hoisted(() => ({
|
||||
mockApiGet: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('./utils.js', () => ({
|
||||
vi.mock('./utils.js', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('./utils.js')>()),
|
||||
apiGet: mockApiGet,
|
||||
}));
|
||||
|
||||
@@ -58,8 +59,8 @@ describe('bilibili comments', () => {
|
||||
it('throws when aid cannot be resolved', async () => {
|
||||
mockApiGet.mockResolvedValueOnce({ data: {} }); // no aid
|
||||
|
||||
await expect(command!.func!({} as any, { bvid: 'BV_invalid', limit: 5 })).rejects.toThrow(
|
||||
'Cannot resolve aid for bvid: BV_invalid',
|
||||
await expect(command!.func!({} as any, { bvid: 'BVinvalid123', limit: 5 })).rejects.toThrow(
|
||||
'Cannot resolve aid for bvid: BVinvalid123',
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
*/
|
||||
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { apiGet } from './utils.js';
|
||||
import { apiGet, resolveBvid } from './utils.js';
|
||||
|
||||
cli({
|
||||
site: 'bilibili',
|
||||
@@ -18,7 +18,7 @@ cli({
|
||||
],
|
||||
columns: ['rank', 'author', 'text', 'likes', 'replies', 'time'],
|
||||
func: async (page, kwargs) => {
|
||||
const bvid = String(kwargs.bvid).trim();
|
||||
const bvid = await resolveBvid(kwargs.bvid);
|
||||
const limit = Math.min(Number(kwargs.limit) || 20, 50);
|
||||
|
||||
// Resolve bvid → aid (required by reply API)
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { checkYtdlp, sanitizeFilename } from '../../download/index.js';
|
||||
import { downloadMedia } from '../../download/media-download.js';
|
||||
import { resolveBvid } from './utils.js';
|
||||
|
||||
cli({
|
||||
site: 'bilibili',
|
||||
@@ -25,7 +26,7 @@ cli({
|
||||
],
|
||||
columns: ['bvid', 'title', 'status', 'size'],
|
||||
func: async (page, kwargs) => {
|
||||
const bvid = kwargs.bvid;
|
||||
const bvid = await resolveBvid(kwargs.bvid);
|
||||
const output = kwargs.output;
|
||||
const quality = kwargs.quality;
|
||||
|
||||
|
||||
@@ -6,7 +6,8 @@ const { mockApiGet } = vi.hoisted(() => ({
|
||||
mockApiGet: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('./utils.js', () => ({
|
||||
vi.mock('./utils.js', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('./utils.js')>()),
|
||||
apiGet: mockApiGet,
|
||||
}));
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { AuthRequiredError, CommandExecutionError, EmptyResultError, SelectorError } from '../../errors.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { apiGet } from './utils.js';
|
||||
import { apiGet, resolveBvid } from './utils.js';
|
||||
|
||||
cli({
|
||||
site: 'bilibili',
|
||||
@@ -15,8 +15,9 @@ cli({
|
||||
columns: ['index', 'from', 'to', 'content'],
|
||||
func: async (page: IPage | null, kwargs: any) => {
|
||||
if (!page) throw new CommandExecutionError('Browser session required for bilibili subtitle');
|
||||
const bvid = await resolveBvid(kwargs.bvid);
|
||||
// 1. 先前往视频详情页 (建立有鉴权的 Session,且这里不需要加载完整个视频)
|
||||
await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);
|
||||
await page.goto(`https://www.bilibili.com/video/${bvid}/`);
|
||||
|
||||
// 2. 利用 __INITIAL_STATE__ 获取基础信息,拿 CID
|
||||
const cid = await page.evaluate(`(async () => {
|
||||
@@ -31,7 +32,7 @@ cli({
|
||||
// 3. 在 Node 端使用 apiGet 获取带 Wbi 签名的字幕列表
|
||||
// 之前纯靠 evaluate 里的 fetch 会失败,因为 B 站 /wbi/ 开头的接口强校验 w_rid,未签名直接被风控返回 403 HTML
|
||||
const payload = await apiGet(page, '/x/player/wbi/v2', {
|
||||
params: { bvid: kwargs.bvid, cid },
|
||||
params: { bvid, cid },
|
||||
signed: true, // 开启 wbi_sign 自动签名
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { resolveBvid } from './utils.js';
|
||||
|
||||
describe('resolveBvid', () => {
|
||||
it('passes through a valid BV ID', async () => {
|
||||
expect(await resolveBvid('BV1MV9NBtENN')).toBe('BV1MV9NBtENN');
|
||||
});
|
||||
|
||||
it('passes through BV ID with surrounding whitespace', async () => {
|
||||
expect(await resolveBvid(' BV1MV9NBtENN ')).toBe('BV1MV9NBtENN');
|
||||
});
|
||||
|
||||
it('handles non-string input via String() coercion', async () => {
|
||||
expect(await resolveBvid('BV123abc' as any)).toBe('BV123abc');
|
||||
});
|
||||
|
||||
it('rejects invalid input that cannot be resolved', async () => {
|
||||
// A random string that b23.tv won't resolve — should timeout or fail
|
||||
await expect(resolveBvid('not-a-valid-code-99999')).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -2,9 +2,36 @@
|
||||
* Bilibili shared helpers: WBI signing, authenticated fetch, nav data, UID resolution.
|
||||
*/
|
||||
|
||||
import https from 'node:https';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { AuthRequiredError, EmptyResultError } from '../../errors.js';
|
||||
|
||||
/**
|
||||
* Resolve Bilibili short URL / short code to BV ID.
|
||||
* Supports: BV1MV9NBtENN, XYzsqGa, b23.tv/XYzsqGa, https://b23.tv/XYzsqGa
|
||||
*/
|
||||
export function resolveBvid(input: unknown): Promise<string> {
|
||||
const trimmed = String(input).trim();
|
||||
if (/^BV[A-Za-z0-9]+$/i.test(trimmed)) {
|
||||
return Promise.resolve(trimmed);
|
||||
}
|
||||
const shortCode = trimmed.replace(/^https?:\/\//, '').replace(/^(www\.)?b23\.tv\//, '');
|
||||
const url = 'https://b23.tv/' + shortCode;
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = https.get(url, (res) => {
|
||||
const location = res.headers.location;
|
||||
if (location) {
|
||||
const match = location.match(/\/video\/(BV[A-Za-z0-9]+)/);
|
||||
if (match) { res.resume(); resolve(match[1]); return; }
|
||||
}
|
||||
res.resume();
|
||||
reject(new Error(`Cannot resolve BV ID from short URL: ${trimmed}`));
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.setTimeout(5000, () => { req.destroy(); reject(new Error(`Timeout resolving short URL: ${trimmed}`)); });
|
||||
});
|
||||
}
|
||||
|
||||
const MIXIN_KEY_ENC_TAB = [
|
||||
46,47,18,2,53,8,23,32,15,50,10,31,58,3,45,35,27,43,5,49,
|
||||
33,9,42,19,29,28,14,39,12,38,41,13,37,48,7,16,24,55,40,
|
||||
|
||||
@@ -50,7 +50,7 @@ async function fetchMarks(
|
||||
): Promise<DoubanMark[]> {
|
||||
const marks: DoubanMark[] = [];
|
||||
let offset = 0;
|
||||
const pageSize = 30;
|
||||
const pageSize = 15;
|
||||
|
||||
while (true) {
|
||||
const url = `https://movie.douban.com/people/${uid}/${status}?start=${offset}&sort=time&rating=all&filter=all&mode=grid`;
|
||||
|
||||
@@ -18,46 +18,75 @@ pipeline:
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
const id = '${{ args.id }}';
|
||||
|
||||
|
||||
// Wait for page to load
|
||||
await new Promise(r => setTimeout(r, 2000));
|
||||
|
||||
// Extract title
|
||||
|
||||
// Extract title - v:itemreviewed contains "中文名 OriginalName"
|
||||
const titleEl = document.querySelector('span[property="v:itemreviewed"]');
|
||||
const title = titleEl?.textContent?.trim() || '';
|
||||
|
||||
// Extract original title
|
||||
const ogTitleEl = document.querySelector('span[property="v:originalTitle"]');
|
||||
const originalTitle = ogTitleEl?.textContent?.trim() || '';
|
||||
|
||||
const fullTitle = titleEl?.textContent?.trim() || '';
|
||||
|
||||
// Split title and originalTitle
|
||||
// Douban format: "中文名 OriginalName" - split by first space that separates CJK from non-CJK
|
||||
let title = fullTitle;
|
||||
let originalTitle = '';
|
||||
const titleMatch = fullTitle.match(/^([\u4e00-\u9fff\u3000-\u303f\uff00-\uffef]+(?:\s*[\u4e00-\u9fff\u3000-\u303f\uff00-\uffef·::!?]+)*)\s+(.+)$/);
|
||||
if (titleMatch) {
|
||||
title = titleMatch[1].trim();
|
||||
originalTitle = titleMatch[2].trim();
|
||||
}
|
||||
|
||||
// Extract year
|
||||
const yearEl = document.querySelector('.year');
|
||||
const year = yearEl?.textContent?.trim() || '';
|
||||
|
||||
const year = yearEl?.textContent?.trim().replace(/[()()]/g, '') || '';
|
||||
|
||||
// Extract rating
|
||||
const ratingEl = document.querySelector('strong[property="v:average"]');
|
||||
const rating = parseFloat(ratingEl?.textContent || '0');
|
||||
|
||||
|
||||
// Extract rating count
|
||||
const ratingCountEl = document.querySelector('span[property="v:votes"]');
|
||||
const ratingCount = parseInt(ratingCountEl?.textContent || '0', 10);
|
||||
|
||||
|
||||
// Extract genres
|
||||
const genreEls = document.querySelectorAll('span[property="v:genre"]');
|
||||
const genres = Array.from(genreEls).map(el => el.textContent?.trim()).filter(Boolean).join(',');
|
||||
|
||||
|
||||
// Extract directors
|
||||
const directorEls = document.querySelectorAll('a[rel="v:directedBy"]');
|
||||
const directors = Array.from(directorEls).map(el => el.textContent?.trim()).filter(Boolean).join(',');
|
||||
|
||||
|
||||
// Extract casts
|
||||
const castEls = document.querySelectorAll('a[rel="v:starring"]');
|
||||
const casts = Array.from(castEls).slice(0, 5).map(el => el.textContent?.trim()).filter(Boolean).join(',');
|
||||
|
||||
const casts = Array.from(castEls).slice(0, 5).map(el => el.textContent?.trim()).filter(Boolean);
|
||||
|
||||
// Extract info section for country and duration
|
||||
const infoEl = document.querySelector('#info');
|
||||
const infoText = infoEl?.textContent || '';
|
||||
|
||||
// Extract country/region from #info as list
|
||||
let country = [];
|
||||
const countryMatch = infoText.match(/制片国家\/地区:\s*([^\n]+)/);
|
||||
if (countryMatch) {
|
||||
country = countryMatch[1].trim().split(/\s*\/\s*/).filter(Boolean);
|
||||
}
|
||||
|
||||
// Extract duration from #info as pure number in min
|
||||
const durationEl = document.querySelector('span[property="v:runtime"]');
|
||||
let durationRaw = durationEl?.textContent?.trim() || '';
|
||||
if (!durationRaw) {
|
||||
const durationMatch = infoText.match(/片长:\s*([^\n]+)/);
|
||||
if (durationMatch) {
|
||||
durationRaw = durationMatch[1].trim();
|
||||
}
|
||||
}
|
||||
const durationNumMatch = durationRaw.match(/(\d+)/);
|
||||
const duration = durationNumMatch ? parseInt(durationNumMatch[1], 10) : null;
|
||||
|
||||
// Extract summary
|
||||
const summaryEl = document.querySelector('span[property="v:summary"]');
|
||||
const summary = summaryEl?.textContent?.trim() || '';
|
||||
|
||||
|
||||
return [{
|
||||
id,
|
||||
title,
|
||||
@@ -68,9 +97,11 @@ pipeline:
|
||||
genres,
|
||||
directors,
|
||||
casts,
|
||||
country,
|
||||
duration,
|
||||
summary: summary.substring(0, 200),
|
||||
url: `https://movie.douban.com/subject/${id}`
|
||||
}];
|
||||
})()
|
||||
|
||||
columns: [id, title, originalTitle, year, rating, ratingCount, genres, directors, casts, summary, url]
|
||||
columns: [id, title, originalTitle, year, rating, ratingCount, genres, directors, casts, country, duration, summary, url]
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { IPage } from '../../types.js';
|
||||
|
||||
const baseline = {
|
||||
turns: [{ Role: 'Assistant', Text: '旧回答' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
};
|
||||
|
||||
const submission = {
|
||||
snapshot: {
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
},
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: { Role: 'User', Text: '请只回复:OK' },
|
||||
reason: 'user_turn' as const,
|
||||
};
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
readGeminiSnapshot: vi.fn(),
|
||||
sendGeminiMessage: vi.fn(),
|
||||
startNewGeminiChat: vi.fn(),
|
||||
waitForGeminiSubmission: vi.fn(),
|
||||
waitForGeminiResponse: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('./utils.js', async () => {
|
||||
const actual = await vi.importActual<typeof import('./utils.js')>('./utils.js');
|
||||
return {
|
||||
...actual,
|
||||
readGeminiSnapshot: mocks.readGeminiSnapshot,
|
||||
sendGeminiMessage: mocks.sendGeminiMessage,
|
||||
startNewGeminiChat: mocks.startNewGeminiChat,
|
||||
waitForGeminiSubmission: mocks.waitForGeminiSubmission,
|
||||
waitForGeminiResponse: mocks.waitForGeminiResponse,
|
||||
};
|
||||
});
|
||||
|
||||
import { askCommand } from './ask.js';
|
||||
|
||||
function createPageMock(): IPage {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(),
|
||||
getCookies: vi.fn().mockResolvedValue([]),
|
||||
snapshot: vi.fn().mockResolvedValue(undefined),
|
||||
click: vi.fn().mockResolvedValue(undefined),
|
||||
typeText: vi.fn().mockResolvedValue(undefined),
|
||||
pressKey: vi.fn().mockResolvedValue(undefined),
|
||||
scrollTo: vi.fn().mockResolvedValue(undefined),
|
||||
getFormState: vi.fn().mockResolvedValue({}),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
tabs: vi.fn().mockResolvedValue([]),
|
||||
selectTab: vi.fn().mockResolvedValue(undefined),
|
||||
networkRequests: vi.fn().mockResolvedValue([]),
|
||||
consoleMessages: vi.fn().mockResolvedValue([]),
|
||||
scroll: vi.fn().mockResolvedValue(undefined),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
screenshot: vi.fn().mockResolvedValue(''),
|
||||
nativeType: vi.fn().mockResolvedValue(undefined),
|
||||
nativeKeyPress: vi.fn().mockResolvedValue(undefined),
|
||||
} as unknown as IPage;
|
||||
}
|
||||
|
||||
describe('gemini ask orchestration', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it('captures baseline, sends, waits for confirmed submission, then waits with the remaining timeout', async () => {
|
||||
vi.spyOn(Date, 'now')
|
||||
.mockReturnValueOnce(0)
|
||||
.mockReturnValueOnce(2000);
|
||||
|
||||
const page = createPageMock();
|
||||
mocks.readGeminiSnapshot.mockResolvedValueOnce(baseline);
|
||||
mocks.sendGeminiMessage.mockResolvedValueOnce('button');
|
||||
mocks.waitForGeminiSubmission.mockResolvedValueOnce(submission);
|
||||
mocks.waitForGeminiResponse.mockResolvedValueOnce('OK');
|
||||
|
||||
const result = await askCommand.func!(page, { prompt: '请只回复:OK', timeout: '20', new: 'false' });
|
||||
|
||||
expect(mocks.readGeminiSnapshot).toHaveBeenCalledWith(page);
|
||||
expect(mocks.waitForGeminiSubmission).toHaveBeenCalledWith(page, baseline, 20);
|
||||
expect(mocks.waitForGeminiResponse).toHaveBeenCalledWith(page, submission, '请只回复:OK', 18);
|
||||
expect(result).toEqual([{ response: '💬 OK' }]);
|
||||
});
|
||||
|
||||
it('does not spend extra response wait time after submission has already consumed the full timeout budget', async () => {
|
||||
vi.spyOn(Date, 'now')
|
||||
.mockReturnValueOnce(0)
|
||||
.mockReturnValueOnce(20000);
|
||||
|
||||
const page = createPageMock();
|
||||
mocks.readGeminiSnapshot.mockResolvedValueOnce(baseline);
|
||||
mocks.sendGeminiMessage.mockResolvedValueOnce('button');
|
||||
mocks.waitForGeminiSubmission.mockResolvedValueOnce(submission);
|
||||
mocks.waitForGeminiResponse.mockResolvedValueOnce('');
|
||||
|
||||
await askCommand.func!(page, { prompt: '请只回复:OK', timeout: '20', new: 'false' });
|
||||
|
||||
expect(mocks.waitForGeminiResponse).toHaveBeenCalledWith(page, submission, '请只回复:OK', 0);
|
||||
});
|
||||
});
|
||||
+10
-3
@@ -1,6 +1,6 @@
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import { GEMINI_DOMAIN, getGeminiTranscriptLines, sendGeminiMessage, startNewGeminiChat, waitForGeminiResponse } from './utils.js';
|
||||
import { GEMINI_DOMAIN, readGeminiSnapshot, sendGeminiMessage, startNewGeminiChat, waitForGeminiResponse, waitForGeminiSubmission } from './utils.js';
|
||||
|
||||
function normalizeBooleanFlag(value: unknown): boolean {
|
||||
if (typeof value === 'boolean') return value;
|
||||
@@ -33,9 +33,16 @@ export const askCommand = cli({
|
||||
|
||||
if (startFresh) await startNewGeminiChat(page);
|
||||
|
||||
const beforeLines = await getGeminiTranscriptLines(page);
|
||||
const before = await readGeminiSnapshot(page);
|
||||
await sendGeminiMessage(page, prompt);
|
||||
const response = await waitForGeminiResponse(page, beforeLines, prompt, timeout);
|
||||
const submissionStartedAt = Date.now();
|
||||
const submitted = await waitForGeminiSubmission(page, before, timeout);
|
||||
if (!submitted) {
|
||||
return [{ response: `💬 ${NO_RESPONSE_PREFIX} No Gemini response within ${timeout}s.` }];
|
||||
}
|
||||
|
||||
const remainingTimeoutSeconds = Math.max(0, timeout - Math.ceil((Date.now() - submissionStartedAt) / 1000));
|
||||
const response = await waitForGeminiResponse(page, submitted, prompt, remainingTimeoutSeconds);
|
||||
|
||||
if (!response) {
|
||||
return [{ response: `💬 ${NO_RESPONSE_PREFIX} No Gemini response within ${timeout}s.` }];
|
||||
|
||||
@@ -0,0 +1,708 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type { IPage } from '../../types.js';
|
||||
import type { GeminiSnapshot } from './utils.js';
|
||||
import { __test__, waitForGeminiResponse, waitForGeminiSubmission } from './utils.js';
|
||||
|
||||
function snapshot(overrides: Partial<GeminiSnapshot> = {}): GeminiSnapshot {
|
||||
return {
|
||||
turns: [],
|
||||
transcriptLines: [],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function createPageMock(): IPage {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(),
|
||||
getCookies: vi.fn().mockResolvedValue([]),
|
||||
snapshot: vi.fn().mockResolvedValue(undefined),
|
||||
click: vi.fn().mockResolvedValue(undefined),
|
||||
typeText: vi.fn().mockResolvedValue(undefined),
|
||||
pressKey: vi.fn().mockResolvedValue(undefined),
|
||||
scrollTo: vi.fn().mockResolvedValue(undefined),
|
||||
getFormState: vi.fn().mockResolvedValue({}),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
tabs: vi.fn().mockResolvedValue([]),
|
||||
selectTab: vi.fn().mockResolvedValue(undefined),
|
||||
networkRequests: vi.fn().mockResolvedValue([]),
|
||||
consoleMessages: vi.fn().mockResolvedValue([]),
|
||||
scroll: vi.fn().mockResolvedValue(undefined),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
screenshot: vi.fn().mockResolvedValue(''),
|
||||
nativeType: vi.fn().mockResolvedValue(undefined),
|
||||
nativeKeyPress: vi.fn().mockResolvedValue(undefined),
|
||||
} as unknown as IPage;
|
||||
}
|
||||
|
||||
describe('Gemini snapshot diff helpers', () => {
|
||||
it('reports appended trusted turns when the current snapshot extends the baseline', () => {
|
||||
const before = snapshot({
|
||||
turns: [{ Role: 'Assistant', Text: '旧回答' }],
|
||||
});
|
||||
const current = snapshot({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
],
|
||||
});
|
||||
|
||||
expect(__test__.diffTrustedStructuredTurns(before, current)).toEqual({
|
||||
appendedTurns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
hasTrustedAppend: true,
|
||||
hasNewUserTurn: true,
|
||||
hasNewAssistantTurn: false,
|
||||
});
|
||||
});
|
||||
|
||||
it('treats restored structured turns as untrusted when the pre-send snapshot had no trustworthy turns', () => {
|
||||
const before = snapshot({
|
||||
turns: [],
|
||||
transcriptLines: ['旧问题', '旧回答'],
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
const current = snapshot({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '旧问题' },
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
],
|
||||
transcriptLines: ['旧问题', '旧回答'],
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
expect(__test__.diffTrustedStructuredTurns(before, current)).toEqual({
|
||||
appendedTurns: [],
|
||||
hasTrustedAppend: false,
|
||||
hasNewUserTurn: false,
|
||||
hasNewAssistantTurn: false,
|
||||
});
|
||||
});
|
||||
|
||||
it('keeps transcript delta lines raw for later conservative fallback checks', () => {
|
||||
const before = snapshot({
|
||||
transcriptLines: ['baseline'],
|
||||
});
|
||||
const current = snapshot({
|
||||
transcriptLines: ['baseline', '关于“请只回复:OK”,这里是解释。'],
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
|
||||
expect(__test__.diffTranscriptLines(before, current)).toEqual([
|
||||
'关于“请只回复:OK”,这里是解释。',
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Gemini submission state', () => {
|
||||
it('confirms submission from a trusted appended user turn', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
turns: [{ Role: 'Assistant', Text: '旧回答' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}), 4);
|
||||
|
||||
expect(result).toEqual({
|
||||
snapshot: {
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
},
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: { Role: 'User', Text: '请只回复:OK' },
|
||||
reason: 'user_turn',
|
||||
});
|
||||
});
|
||||
|
||||
it('confirms submission from composer cleared plus generating even when transcript has not changed yet', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
structuredTurnsTrusted: false,
|
||||
}), 2);
|
||||
|
||||
expect(result).toEqual({
|
||||
snapshot: {
|
||||
turns: [],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: false,
|
||||
},
|
||||
preSendAssistantCount: 0,
|
||||
userAnchorTurn: null,
|
||||
reason: 'composer_generating',
|
||||
});
|
||||
});
|
||||
|
||||
it('confirms submission from generating state even when the pre-send baseline composer was empty', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:DBG2' },
|
||||
{ Role: 'User', Text: '请只回复:DBG2' },
|
||||
],
|
||||
transcriptLines: ['baseline', '请只回复:DBG2'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
turns: [{ Role: 'Assistant', Text: '需要我为你做些什么?' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
structuredTurnsTrusted: true,
|
||||
}), 2);
|
||||
|
||||
expect(result).toEqual({
|
||||
snapshot: {
|
||||
turns: [
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:DBG2' },
|
||||
{ Role: 'User', Text: '请只回复:DBG2' },
|
||||
],
|
||||
transcriptLines: ['baseline', '请只回复:DBG2'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
},
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: { Role: 'User', Text: '请只回复:DBG2' },
|
||||
reason: 'composer_generating',
|
||||
});
|
||||
});
|
||||
|
||||
it('confirms submission from composer cleared plus transcript growth when generation state is unavailable', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
// This transcript delta may be only a prompt echo. It is allowed to confirm
|
||||
// submission only because the composer has already cleared, and it must never
|
||||
// be reused later as reply ownership evidence.
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
structuredTurnsTrusted: false,
|
||||
}), 2);
|
||||
|
||||
expect(result).toEqual({
|
||||
snapshot: {
|
||||
turns: [],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: false,
|
||||
},
|
||||
preSendAssistantCount: 0,
|
||||
userAnchorTurn: null,
|
||||
reason: 'composer_transcript',
|
||||
});
|
||||
});
|
||||
|
||||
it('does not confirm submission when old structured turns only reappear after an untrusted pre-send snapshot', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '旧问题' },
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
],
|
||||
transcriptLines: ['旧问题', '旧回答'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '旧问题' },
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
],
|
||||
transcriptLines: ['旧问题', '旧回答'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
turns: [],
|
||||
transcriptLines: ['旧问题', '旧回答'],
|
||||
composerHasText: true,
|
||||
structuredTurnsTrusted: false,
|
||||
}), 2);
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('does not confirm submission from transcript growth alone when the composer never clears', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: true,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: false,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: true,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
structuredTurnsTrusted: false,
|
||||
}), 2);
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('keeps polling past ten seconds when the overall timeout budget still allows submission confirmation', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
for (let index = 0; index < 10; index += 1) {
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
}
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [],
|
||||
transcriptLines: ['baseline', '请只回复:OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: false,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiSubmission(page, snapshot({
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: true,
|
||||
structuredTurnsTrusted: false,
|
||||
}), 12);
|
||||
|
||||
expect(result?.reason).toBe('composer_transcript');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Gemini reply state', () => {
|
||||
it('does not reuse an older identical reply when the submission baseline has no structured user anchor', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'Assistant', Text: 'OK' }],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [{ Role: 'Assistant', Text: 'OK' }],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: null,
|
||||
reason: 'composer_generating',
|
||||
}, '请只回复:OK', 6);
|
||||
|
||||
expect(result).toBe('OK');
|
||||
});
|
||||
|
||||
it('does not treat prepended older history as the current round reply when reply ownership has no user anchor', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '更早的问题' },
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '更早的问题' },
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '更早的问题' },
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [{ Role: 'Assistant', Text: '旧回答' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: null,
|
||||
reason: 'composer_generating',
|
||||
}, '请只回复:OK', 6);
|
||||
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
it('accepts a reply when the submission snapshot contains only the current round user turns and later appends a new assistant', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:DBGREG' },
|
||||
{ Role: 'User', Text: '请只回复:DBGREG' },
|
||||
{ Role: 'Assistant', Text: 'DBGREG' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:DBGREG' },
|
||||
{ Role: 'User', Text: '请只回复:DBGREG' },
|
||||
{ Role: 'Assistant', Text: 'DBGREG' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:DBGREG' },
|
||||
{ Role: 'User', Text: '请只回复:DBGREG' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: { Role: 'User', Text: '请只回复:DBGREG' },
|
||||
reason: 'composer_generating',
|
||||
}, '请只回复:DBGREG', 6);
|
||||
|
||||
expect(result).toBe('DBGREG');
|
||||
});
|
||||
|
||||
it('does not trust an assistant-only submission snapshot without a stable post-submission owner', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'Assistant', Text: '完整回答' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'Assistant', Text: '完整回答' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [{ Role: 'Assistant', Text: '半截回答' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 0,
|
||||
userAnchorTurn: null,
|
||||
reason: 'composer_generating',
|
||||
}, '请解释', 4);
|
||||
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
it('accepts an assistant reply that appears after a structured user anchor only after it stabilizes and generation stops', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请解释' },
|
||||
{ Role: 'Assistant', Text: '半截回答' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请解释' },
|
||||
{ Role: 'Assistant', Text: '完整回答' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请解释' },
|
||||
{ Role: 'Assistant', Text: '完整回答' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [
|
||||
{ Role: 'Assistant', Text: '旧回答' },
|
||||
{ Role: 'User', Text: '请解释' },
|
||||
],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 1,
|
||||
userAnchorTurn: { Role: 'User', Text: '请解释' },
|
||||
reason: 'user_turn',
|
||||
}, '请解释', 6);
|
||||
|
||||
expect(result).toBe('完整回答');
|
||||
});
|
||||
|
||||
it('uses transcript fallback only after two identical post-submission deltas and after generation stops', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 0,
|
||||
userAnchorTurn: { Role: 'User', Text: '请只回复:OK' },
|
||||
reason: 'user_turn',
|
||||
}, '请只回复:OK', 6);
|
||||
|
||||
expect(result).toBe('OK');
|
||||
});
|
||||
|
||||
it('ignores transcript lines that appeared before submission confirmation and only accepts post-submission transcript deltas', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', '早到的提示词回声', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', '早到的提示词回声', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
})
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', '早到的提示词回声', 'OK'],
|
||||
composerHasText: false,
|
||||
isGenerating: false,
|
||||
structuredTurnsTrusted: true,
|
||||
});
|
||||
|
||||
const result = await waitForGeminiResponse(page, {
|
||||
snapshot: snapshot({
|
||||
turns: [{ Role: 'User', Text: '请只回复:OK' }],
|
||||
transcriptLines: ['baseline', '早到的提示词回声'],
|
||||
composerHasText: false,
|
||||
isGenerating: true,
|
||||
structuredTurnsTrusted: true,
|
||||
}),
|
||||
preSendAssistantCount: 0,
|
||||
userAnchorTurn: { Role: 'User', Text: '请只回复:OK' },
|
||||
reason: 'composer_transcript',
|
||||
}, '请只回复:OK', 6);
|
||||
|
||||
expect(result).toBe('OK');
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,39 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { collectGeminiTranscriptAdditions, sanitizeGeminiResponseText } from './utils.js';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type { IPage } from '../../types.js';
|
||||
import type { GeminiTurn } from './utils.js';
|
||||
import {
|
||||
__test__,
|
||||
collectGeminiTranscriptAdditions,
|
||||
sanitizeGeminiResponseText,
|
||||
sendGeminiMessage,
|
||||
} from './utils.js';
|
||||
|
||||
function createPageMock(): IPage {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn(),
|
||||
getCookies: vi.fn().mockResolvedValue([]),
|
||||
snapshot: vi.fn().mockResolvedValue(undefined),
|
||||
click: vi.fn().mockResolvedValue(undefined),
|
||||
typeText: vi.fn().mockResolvedValue(undefined),
|
||||
pressKey: vi.fn().mockResolvedValue(undefined),
|
||||
scrollTo: vi.fn().mockResolvedValue(undefined),
|
||||
getFormState: vi.fn().mockResolvedValue({}),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
tabs: vi.fn().mockResolvedValue([]),
|
||||
selectTab: vi.fn().mockResolvedValue(undefined),
|
||||
networkRequests: vi.fn().mockResolvedValue([]),
|
||||
consoleMessages: vi.fn().mockResolvedValue([]),
|
||||
scroll: vi.fn().mockResolvedValue(undefined),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
screenshot: vi.fn().mockResolvedValue(''),
|
||||
nativeType: vi.fn().mockResolvedValue(undefined),
|
||||
nativeKeyPress: vi.fn().mockResolvedValue(undefined),
|
||||
} as unknown as IPage;
|
||||
}
|
||||
|
||||
describe('sanitizeGeminiResponseText', () => {
|
||||
it('strips a prompt echo only when it appears as a prefixed block', () => {
|
||||
@@ -33,4 +67,152 @@ describe('collectGeminiTranscriptAdditions', () => {
|
||||
const current = ['Previous', 'Tell me a haiku', 'Tell me a haiku\n\nSoft spring rain arrives'];
|
||||
expect(collectGeminiTranscriptAdditions(before, current, prompt)).toBe('Soft spring rain arrives');
|
||||
});
|
||||
|
||||
it('keeps a reply line that quotes the prompt inside the answer body', () => {
|
||||
const prompt = '请只回复:OK';
|
||||
const before = ['baseline'];
|
||||
const current = ['baseline', '关于“请只回复:OK”,这里是解释。'];
|
||||
expect(collectGeminiTranscriptAdditions(before, current, prompt)).toBe('关于“请只回复:OK”,这里是解释。');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gemini send strategy', () => {
|
||||
it('includes structural composer selectors instead of relying only on english aria labels', () => {
|
||||
expect(__test__.GEMINI_COMPOSER_SELECTORS).toContain('.ql-editor[contenteditable="true"]');
|
||||
expect(__test__.GEMINI_COMPOSER_SELECTORS).toContain('.ql-editor[role="textbox"]');
|
||||
});
|
||||
|
||||
it('prefers native text insertion before submitting the composer', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
const nativeType = vi.mocked(page.nativeType!);
|
||||
const nativeKeyPress = vi.mocked(page.nativeKeyPress!);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ hasText: true })
|
||||
.mockResolvedValueOnce('button');
|
||||
|
||||
const result = await sendGeminiMessage(page, '你好');
|
||||
|
||||
expect(nativeType).toHaveBeenCalledWith('你好');
|
||||
expect(nativeKeyPress).not.toHaveBeenCalled();
|
||||
expect(result).toBe('button');
|
||||
});
|
||||
|
||||
it('falls back when native insertion does not update the composer', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
const nativeType = vi.mocked(page.nativeType!);
|
||||
const nativeKeyPress = vi.mocked(page.nativeKeyPress!);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ hasText: false })
|
||||
.mockResolvedValueOnce({ hasText: true })
|
||||
.mockResolvedValueOnce('enter');
|
||||
|
||||
const result = await sendGeminiMessage(page, '你好');
|
||||
|
||||
expect(nativeType).toHaveBeenCalledWith('你好');
|
||||
expect(nativeKeyPress).toHaveBeenCalledWith('Enter');
|
||||
expect(evaluate).toHaveBeenCalledTimes(5);
|
||||
expect(result).toBe('enter');
|
||||
});
|
||||
|
||||
it('falls back when native insertion throws', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
const nativeType = vi.mocked(page.nativeType!);
|
||||
|
||||
nativeType.mockRejectedValueOnce(new Error('Unknown action: cdp'));
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ hasText: true })
|
||||
.mockResolvedValueOnce('button');
|
||||
|
||||
const result = await sendGeminiMessage(page, '你好');
|
||||
|
||||
expect(nativeType).toHaveBeenCalledWith('你好');
|
||||
expect(result).toBe('button');
|
||||
});
|
||||
|
||||
it('retries composer preparation until a slow-loading composer appears', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
const wait = vi.mocked(page.wait);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({ ok: false, reason: 'Could not find Gemini composer' })
|
||||
.mockResolvedValueOnce({ ok: false, reason: 'Could not find Gemini composer' })
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ hasText: true })
|
||||
.mockResolvedValueOnce('button');
|
||||
|
||||
const result = await sendGeminiMessage(page, '你好');
|
||||
|
||||
expect(result).toBe('button');
|
||||
expect(wait.mock.calls.filter(([value]) => value === 1)).toHaveLength(3);
|
||||
});
|
||||
|
||||
it('keeps retrying until a composer that appears on the fourth attempt is ready', async () => {
|
||||
const page = createPageMock();
|
||||
const evaluate = vi.mocked(page.evaluate);
|
||||
const wait = vi.mocked(page.wait);
|
||||
|
||||
evaluate
|
||||
.mockResolvedValueOnce('https://gemini.google.com/app')
|
||||
.mockResolvedValueOnce({ ok: false, reason: 'Could not find Gemini composer' })
|
||||
.mockResolvedValueOnce({ ok: false, reason: 'Could not find Gemini composer' })
|
||||
.mockResolvedValueOnce({ ok: false, reason: 'Could not find Gemini composer' })
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ hasText: true })
|
||||
.mockResolvedValueOnce('button');
|
||||
|
||||
const result = await sendGeminiMessage(page, '你好');
|
||||
|
||||
expect(result).toBe('button');
|
||||
expect(wait.mock.calls.filter(([value]) => value === 1)).toHaveLength(4);
|
||||
});
|
||||
|
||||
it('avoids innerHTML in the fallback insertion path for trusted types pages', () => {
|
||||
expect(__test__.insertComposerTextFallbackScript('你好')).not.toContain('innerHTML');
|
||||
expect(__test__.insertComposerTextFallbackScript('你好')).toContain('replaceChildren');
|
||||
});
|
||||
|
||||
it('keeps a button submit path in the generated submit script', () => {
|
||||
expect(__test__.submitComposerScript()).toContain('.click()');
|
||||
});
|
||||
|
||||
it('supports localized new chat labels in the generated new-chat script', () => {
|
||||
expect(__test__.clickNewChatScript()).toContain('发起新对话');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gemini turn normalization', () => {
|
||||
it('collapses only adjacent duplicate turns so identical replies across rounds remain visible', () => {
|
||||
const turns: GeminiTurn[] = [
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:OK' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:OK' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
];
|
||||
|
||||
expect(__test__.collapseAdjacentGeminiTurns(turns)).toEqual([
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:OK' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
{ Role: 'User', Text: '你说\n\n请只回复:OK' },
|
||||
{ Role: 'User', Text: '请只回复:OK' },
|
||||
{ Role: 'Assistant', Text: 'OK' },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
+594
-66
@@ -1,3 +1,4 @@
|
||||
import { CommandExecutionError } from '../../errors.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
|
||||
export const GEMINI_DOMAIN = 'gemini.google.com';
|
||||
@@ -16,12 +17,88 @@ export interface GeminiTurn {
|
||||
Text: string;
|
||||
}
|
||||
|
||||
export interface GeminiSnapshot {
|
||||
turns: GeminiTurn[];
|
||||
transcriptLines: string[];
|
||||
composerHasText: boolean;
|
||||
isGenerating: boolean;
|
||||
structuredTurnsTrusted: boolean;
|
||||
}
|
||||
|
||||
export interface GeminiStructuredAppend {
|
||||
appendedTurns: GeminiTurn[];
|
||||
hasTrustedAppend: boolean;
|
||||
hasNewUserTurn: boolean;
|
||||
hasNewAssistantTurn: boolean;
|
||||
}
|
||||
|
||||
export interface GeminiSubmissionBaseline {
|
||||
snapshot: GeminiSnapshot;
|
||||
preSendAssistantCount: number;
|
||||
userAnchorTurn: GeminiTurn | null;
|
||||
reason: 'user_turn' | 'composer_generating' | 'composer_transcript';
|
||||
}
|
||||
|
||||
const GEMINI_RESPONSE_NOISE_PATTERNS = [
|
||||
/Gemini can make mistakes\.?/gi,
|
||||
/Google Terms/gi,
|
||||
/Google Privacy Policy/gi,
|
||||
/Opens in a new window/gi,
|
||||
];
|
||||
const GEMINI_TRANSCRIPT_CHROME_MARKERS = ['gemini', '我的内容', '对话', 'google terms', 'google privacy policy'];
|
||||
|
||||
const GEMINI_COMPOSER_SELECTORS = [
|
||||
'.ql-editor[contenteditable="true"]',
|
||||
'.ql-editor[role="textbox"]',
|
||||
'.ql-editor[aria-label*="Gemini"]',
|
||||
'[contenteditable="true"][aria-label*="Gemini"]',
|
||||
'[aria-label="Enter a prompt for Gemini"]',
|
||||
'[aria-label*="prompt for Gemini"]',
|
||||
];
|
||||
|
||||
const GEMINI_COMPOSER_MARKER_ATTR = 'data-opencli-gemini-composer';
|
||||
const GEMINI_COMPOSER_PREPARE_ATTEMPTS = 4;
|
||||
const GEMINI_COMPOSER_PREPARE_WAIT_SECONDS = 1;
|
||||
|
||||
function buildGeminiComposerLocatorScript(): string {
|
||||
const selectorsJson = JSON.stringify(GEMINI_COMPOSER_SELECTORS);
|
||||
const markerAttrJson = JSON.stringify(GEMINI_COMPOSER_MARKER_ATTR);
|
||||
return `
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
if (style.display === 'none' || style.visibility === 'hidden') return false;
|
||||
const rect = el.getBoundingClientRect();
|
||||
return rect.width > 0 && rect.height > 0;
|
||||
};
|
||||
|
||||
const markerAttr = ${markerAttrJson};
|
||||
const clearComposerMarkers = (active) => {
|
||||
document.querySelectorAll('[' + markerAttr + ']').forEach((node) => {
|
||||
if (node !== active) node.removeAttribute(markerAttr);
|
||||
});
|
||||
};
|
||||
|
||||
const markComposer = (node) => {
|
||||
if (!(node instanceof HTMLElement)) return null;
|
||||
clearComposerMarkers(node);
|
||||
node.setAttribute(markerAttr, '1');
|
||||
return node;
|
||||
};
|
||||
|
||||
const findComposer = () => {
|
||||
const marked = document.querySelector('[' + markerAttr + '="1"]');
|
||||
if (marked instanceof HTMLElement && isVisible(marked)) return marked;
|
||||
|
||||
const selectors = ${selectorsJson};
|
||||
for (const selector of selectors) {
|
||||
const node = Array.from(document.querySelectorAll(selector)).find((candidate) => candidate instanceof HTMLElement && isVisible(candidate));
|
||||
if (node instanceof HTMLElement) return markComposer(node);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
`;
|
||||
}
|
||||
|
||||
export function sanitizeGeminiResponseText(value: string, promptText: string): string {
|
||||
let sanitized = value;
|
||||
@@ -52,33 +129,160 @@ export function collectGeminiTranscriptAdditions(
|
||||
const beforeSet = new Set(beforeLines);
|
||||
const additions = currentLines
|
||||
.filter((line) => !beforeSet.has(line))
|
||||
.map((line) => sanitizeGeminiResponseText(line, promptText))
|
||||
.map((line) => extractGeminiTranscriptLineCandidate(line, promptText))
|
||||
.filter((line) => line && line !== promptText);
|
||||
|
||||
return additions.join('\n').trim();
|
||||
}
|
||||
|
||||
export function collapseAdjacentGeminiTurns(turns: GeminiTurn[]): GeminiTurn[] {
|
||||
const collapsed: GeminiTurn[] = [];
|
||||
|
||||
for (const turn of turns) {
|
||||
if (!turn || typeof turn.Role !== 'string' || typeof turn.Text !== 'string') continue;
|
||||
const previous = collapsed.at(-1);
|
||||
if (previous?.Role === turn.Role && previous.Text === turn.Text) continue;
|
||||
collapsed.push(turn);
|
||||
}
|
||||
|
||||
return collapsed;
|
||||
}
|
||||
|
||||
function hasGeminiTurnPrefix(before: GeminiTurn[], current: GeminiTurn[]): boolean {
|
||||
if (before.length > current.length) return false;
|
||||
return before.every((turn, index) => (
|
||||
turn.Role === current[index]?.Role
|
||||
&& turn.Text === current[index]?.Text
|
||||
));
|
||||
}
|
||||
|
||||
function findLastMatchingGeminiTurnIndex(turns: GeminiTurn[], target: GeminiTurn | null): number | null {
|
||||
if (!target) return null;
|
||||
for (let index = turns.length - 1; index >= 0; index -= 1) {
|
||||
const turn = turns[index];
|
||||
if (turn?.Role === target.Role && turn.Text === target.Text) {
|
||||
return index;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function diffTrustedStructuredTurns(
|
||||
before: GeminiSnapshot,
|
||||
current: GeminiSnapshot,
|
||||
): GeminiStructuredAppend {
|
||||
if (!before.structuredTurnsTrusted || !current.structuredTurnsTrusted) {
|
||||
return {
|
||||
appendedTurns: [],
|
||||
hasTrustedAppend: false,
|
||||
hasNewUserTurn: false,
|
||||
hasNewAssistantTurn: false,
|
||||
};
|
||||
}
|
||||
|
||||
if (!hasGeminiTurnPrefix(before.turns, current.turns)) {
|
||||
return {
|
||||
appendedTurns: [],
|
||||
hasTrustedAppend: false,
|
||||
hasNewUserTurn: false,
|
||||
hasNewAssistantTurn: false,
|
||||
};
|
||||
}
|
||||
|
||||
const appendedTurns = current.turns.slice(before.turns.length);
|
||||
return {
|
||||
appendedTurns,
|
||||
hasTrustedAppend: appendedTurns.length > 0,
|
||||
hasNewUserTurn: appendedTurns.some((turn) => turn.Role === 'User'),
|
||||
hasNewAssistantTurn: appendedTurns.some((turn) => turn.Role === 'Assistant'),
|
||||
};
|
||||
}
|
||||
|
||||
function diffTranscriptLines(before: GeminiSnapshot, current: GeminiSnapshot): string[] {
|
||||
const beforeLines = new Set(before.transcriptLines);
|
||||
return current.transcriptLines.filter((line) => !beforeLines.has(line));
|
||||
}
|
||||
|
||||
function isLikelyGeminiTranscriptChrome(line: string): boolean {
|
||||
const lower = line.toLowerCase();
|
||||
const markerHits = GEMINI_TRANSCRIPT_CHROME_MARKERS.filter((marker) => lower.includes(marker)).length;
|
||||
return markerHits >= 2;
|
||||
}
|
||||
|
||||
function extractGeminiTranscriptLineCandidate(transcriptLine: string, promptText: string): string {
|
||||
const candidate = transcriptLine.trim();
|
||||
if (!candidate) return '';
|
||||
|
||||
const prompt = promptText.trim();
|
||||
const sanitized = sanitizeGeminiResponseText(candidate, promptText);
|
||||
|
||||
if (!prompt) return sanitized;
|
||||
if (!candidate.includes(prompt)) return sanitized;
|
||||
if (sanitized && sanitized !== prompt && sanitized !== candidate) return sanitized;
|
||||
if (isLikelyGeminiTranscriptChrome(candidate)) return '';
|
||||
|
||||
// Some transcript snapshots flatten "prompt + answer" into a single line.
|
||||
// Recover the answer only when the line starts with the current prompt.
|
||||
if (candidate.startsWith(prompt)) {
|
||||
const tail = candidate.slice(prompt.length).replace(/^[\s::,,-]+/, '').trim();
|
||||
return tail ? sanitizeGeminiResponseText(tail, '') : '';
|
||||
}
|
||||
|
||||
return sanitized;
|
||||
}
|
||||
|
||||
function getStateScript(): string {
|
||||
return `
|
||||
(() => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
|
||||
const signInNode = Array.from(document.querySelectorAll('a, button')).find((node) => {
|
||||
const text = (node.textContent || '').trim().toLowerCase();
|
||||
const aria = (node.getAttribute('aria-label') || '').trim().toLowerCase();
|
||||
const href = node.getAttribute('href') || '';
|
||||
return text === 'sign in'
|
||||
|| aria === 'sign in'
|
||||
|| text === '登录'
|
||||
|| aria === '登录'
|
||||
|| href.includes('accounts.google.com/ServiceLogin');
|
||||
});
|
||||
|
||||
const composer = document.querySelector('[aria-label="Enter a prompt for Gemini"], [aria-label*="prompt for Gemini"], .ql-editor[aria-label*="Gemini"], [contenteditable="true"][aria-label*="Gemini"]');
|
||||
const sendButton = document.querySelector('button[aria-label="Send message"]');
|
||||
const composer = findComposer();
|
||||
|
||||
return {
|
||||
url: window.location.href,
|
||||
title: document.title || '',
|
||||
isSignedIn: signInNode ? false : (composer ? true : null),
|
||||
composerLabel: composer?.getAttribute('aria-label') || '',
|
||||
canSend: !!(sendButton && !sendButton.disabled),
|
||||
canSend: !!composer,
|
||||
};
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
function readGeminiSnapshotScript(): string {
|
||||
return `
|
||||
(() => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
const composer = findComposer();
|
||||
const composerText = composer?.textContent?.replace(/\\u00a0/g, ' ').trim() || '';
|
||||
const isGenerating = !!Array.from(document.querySelectorAll('button, [role="button"]')).find((node) => {
|
||||
const text = (node.textContent || '').trim().toLowerCase();
|
||||
const aria = (node.getAttribute('aria-label') || '').trim().toLowerCase();
|
||||
return text === 'stop response'
|
||||
|| aria === 'stop response'
|
||||
|| text === '停止回答'
|
||||
|| aria === '停止回答';
|
||||
});
|
||||
const turns = ${getTurnsScript().trim()};
|
||||
const transcriptLines = ${getTranscriptLinesScript().trim()};
|
||||
|
||||
return {
|
||||
turns,
|
||||
transcriptLines,
|
||||
composerHasText: composerText.length > 0,
|
||||
isGenerating,
|
||||
structuredTurnsTrusted: turns.length > 0 || transcriptLines.length === 0,
|
||||
};
|
||||
})()
|
||||
`;
|
||||
@@ -186,7 +390,16 @@ function getTurnsScript(): string {
|
||||
];
|
||||
|
||||
const roots = selectors.flatMap((selector) => Array.from(document.querySelectorAll(selector)));
|
||||
const unique = roots.filter((el, index, all) => all.indexOf(el) === index).filter(isVisible);
|
||||
const unique = roots
|
||||
.filter((el, index, all) => all.indexOf(el) === index)
|
||||
.filter(isVisible)
|
||||
.sort((left, right) => {
|
||||
if (left === right) return 0;
|
||||
const relation = left.compareDocumentPosition(right);
|
||||
if (relation & Node.DOCUMENT_POSITION_FOLLOWING) return -1;
|
||||
if (relation & Node.DOCUMENT_POSITION_PRECEDING) return 1;
|
||||
return 0;
|
||||
});
|
||||
|
||||
const turns = unique.map((el) => {
|
||||
const text = clean(el.innerText || el.textContent || '');
|
||||
@@ -206,53 +419,189 @@ function getTurnsScript(): string {
|
||||
return role ? { Role: role, Text: text } : null;
|
||||
}).filter(Boolean);
|
||||
|
||||
const deduped = [];
|
||||
const seen = new Set();
|
||||
for (const turn of turns) {
|
||||
const key = turn.Role + '::' + turn.Text;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
deduped.push(turn);
|
||||
}
|
||||
return deduped;
|
||||
return turns;
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
function fillAndSubmitComposerScript(text: string): string {
|
||||
function prepareComposerScript(): string {
|
||||
return `
|
||||
((inputText) => {
|
||||
const cleanInsert = (el) => {
|
||||
if (!(el instanceof HTMLElement)) throw new Error('Composer is not editable');
|
||||
el.focus();
|
||||
(() => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
const composer = findComposer();
|
||||
|
||||
if (!(composer instanceof HTMLElement)) {
|
||||
return { ok: false, reason: 'Could not find Gemini composer' };
|
||||
}
|
||||
|
||||
try {
|
||||
composer.focus();
|
||||
const selection = window.getSelection();
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(el);
|
||||
range.selectNodeContents(composer);
|
||||
range.collapse(false);
|
||||
selection?.removeAllRanges();
|
||||
selection?.addRange(range);
|
||||
el.textContent = '';
|
||||
document.execCommand('insertText', false, inputText);
|
||||
el.dispatchEvent(new InputEvent('input', { bubbles: true, data: inputText, inputType: 'insertText' }));
|
||||
};
|
||||
composer.textContent = '';
|
||||
composer.dispatchEvent(new InputEvent('input', { bubbles: true, data: '', inputType: 'deleteContentBackward' }));
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: error instanceof Error ? error.message : String(error),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
label: composer.getAttribute('aria-label') || '',
|
||||
};
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
function composerHasTextScript(): string {
|
||||
return `
|
||||
(() => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
const composer = findComposer();
|
||||
|
||||
return {
|
||||
hasText: !!(composer && ((composer.textContent || '').trim() || (composer.innerText || '').trim())),
|
||||
};
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
function insertComposerTextFallbackScript(text: string): string {
|
||||
return `
|
||||
((inputText) => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
const composer = findComposer();
|
||||
|
||||
if (!(composer instanceof HTMLElement)) {
|
||||
return { hasText: false, reason: 'Could not find Gemini composer' };
|
||||
}
|
||||
|
||||
const selection = window.getSelection();
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(composer);
|
||||
range.collapse(false);
|
||||
selection?.removeAllRanges();
|
||||
selection?.addRange(range);
|
||||
|
||||
composer.focus();
|
||||
composer.textContent = '';
|
||||
const execResult = typeof document.execCommand === 'function'
|
||||
? document.execCommand('insertText', false, inputText)
|
||||
: false;
|
||||
|
||||
if (!execResult) {
|
||||
const paragraph = document.createElement('p');
|
||||
const lines = String(inputText).split(/\\n/);
|
||||
for (const [index, line] of lines.entries()) {
|
||||
if (index > 0) paragraph.appendChild(document.createElement('br'));
|
||||
paragraph.appendChild(document.createTextNode(line));
|
||||
}
|
||||
composer.replaceChildren(paragraph);
|
||||
}
|
||||
|
||||
composer.dispatchEvent(new InputEvent('beforeinput', { bubbles: true, data: inputText, inputType: 'insertText' }));
|
||||
composer.dispatchEvent(new InputEvent('input', { bubbles: true, data: inputText, inputType: 'insertText' }));
|
||||
composer.dispatchEvent(new Event('change', { bubbles: true }));
|
||||
|
||||
return {
|
||||
hasText: !!((composer.textContent || '').trim() || (composer.innerText || '').trim()),
|
||||
};
|
||||
})(${JSON.stringify(text)})
|
||||
`;
|
||||
}
|
||||
|
||||
function submitComposerScript(): string {
|
||||
return `
|
||||
(() => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
const composer = findComposer();
|
||||
|
||||
const composer = document.querySelector('[aria-label="Enter a prompt for Gemini"], [aria-label*="prompt for Gemini"], .ql-editor[aria-label*="Gemini"], [contenteditable="true"][aria-label*="Gemini"]');
|
||||
if (!(composer instanceof HTMLElement)) {
|
||||
throw new Error('Could not find Gemini composer');
|
||||
}
|
||||
|
||||
cleanInsert(composer);
|
||||
const composerRect = composer.getBoundingClientRect();
|
||||
const rootCandidates = [
|
||||
composer.closest('form'),
|
||||
composer.closest('[role="form"]'),
|
||||
composer.closest('.input-area-container'),
|
||||
composer.closest('.textbox-container'),
|
||||
composer.closest('.input-wrapper'),
|
||||
composer.parentElement,
|
||||
composer.parentElement?.parentElement,
|
||||
].filter(Boolean);
|
||||
|
||||
const sendButton = document.querySelector('button[aria-label="Send message"]');
|
||||
if (sendButton instanceof HTMLButtonElement && !sendButton.disabled) {
|
||||
sendButton.click();
|
||||
const seen = new Set();
|
||||
const buttons = [];
|
||||
for (const root of rootCandidates) {
|
||||
root.querySelectorAll('button, [role="button"]').forEach((node) => {
|
||||
if (!(node instanceof HTMLElement)) return;
|
||||
if (seen.has(node)) return;
|
||||
seen.add(node);
|
||||
buttons.push(node);
|
||||
});
|
||||
}
|
||||
|
||||
const excludedPattern = /main menu|主菜单|microphone|麦克风|upload|上传|mode|模式|tools|工具|settings|临时对话|new chat|新对话/i;
|
||||
const submitPattern = /send|发送|submit|提交/i;
|
||||
let bestButton = null;
|
||||
let bestScore = -1;
|
||||
|
||||
for (const button of buttons) {
|
||||
if (!isVisible(button)) continue;
|
||||
if (button instanceof HTMLButtonElement && button.disabled) continue;
|
||||
if (button.getAttribute('aria-disabled') === 'true') continue;
|
||||
|
||||
const label = ((button.getAttribute('aria-label') || '') + ' ' + ((button.textContent || '').trim())).trim();
|
||||
if (excludedPattern.test(label)) continue;
|
||||
|
||||
const rect = button.getBoundingClientRect();
|
||||
const verticalDistance = Math.abs((rect.top + rect.bottom) / 2 - (composerRect.top + composerRect.bottom) / 2);
|
||||
if (verticalDistance > 160) continue;
|
||||
|
||||
let score = 0;
|
||||
if (submitPattern.test(label)) score += 10;
|
||||
if (rect.left >= composerRect.right - 160) score += 3;
|
||||
if (rect.left >= composerRect.left) score += 1;
|
||||
if (rect.width <= 96 && rect.height <= 96) score += 1;
|
||||
|
||||
if (score > bestScore) {
|
||||
bestScore = score;
|
||||
bestButton = button;
|
||||
}
|
||||
}
|
||||
|
||||
if (bestButton instanceof HTMLElement && bestScore >= 3) {
|
||||
bestButton.click();
|
||||
return 'button';
|
||||
}
|
||||
|
||||
composer.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', code: 'Enter', keyCode: 13, bubbles: true }));
|
||||
composer.dispatchEvent(new KeyboardEvent('keyup', { key: 'Enter', code: 'Enter', keyCode: 13, bubbles: true }));
|
||||
return 'enter';
|
||||
})(${JSON.stringify(text)})
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
function dispatchComposerEnterScript(): string {
|
||||
return `
|
||||
(() => {
|
||||
${buildGeminiComposerLocatorScript()}
|
||||
const composer = findComposer();
|
||||
|
||||
if (!(composer instanceof HTMLElement)) {
|
||||
throw new Error('Could not find Gemini composer');
|
||||
}
|
||||
|
||||
composer.focus();
|
||||
composer.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', code: 'Enter', keyCode: 13, which: 13, bubbles: true }));
|
||||
composer.dispatchEvent(new KeyboardEvent('keyup', { key: 'Enter', code: 'Enter', keyCode: 13, which: 13, bubbles: true }));
|
||||
return 'enter';
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
@@ -270,7 +619,14 @@ function clickNewChatScript(): string {
|
||||
const candidates = Array.from(document.querySelectorAll('button, a')).filter((node) => {
|
||||
const text = (node.textContent || '').trim().toLowerCase();
|
||||
const aria = (node.getAttribute('aria-label') || '').trim().toLowerCase();
|
||||
return isVisible(node) && (text === 'new chat' || aria === 'new chat');
|
||||
return isVisible(node) && (
|
||||
text === 'new chat'
|
||||
|| aria === 'new chat'
|
||||
|| text === '发起新对话'
|
||||
|| aria === '发起新对话'
|
||||
|| text === '新对话'
|
||||
|| aria === '新对话'
|
||||
);
|
||||
});
|
||||
|
||||
const target = candidates.find((node) => !node.hasAttribute('disabled')) || candidates[0];
|
||||
@@ -321,27 +677,150 @@ export async function startNewGeminiChat(page: IPage): Promise<'clicked' | 'navi
|
||||
}
|
||||
|
||||
export async function getGeminiVisibleTurns(page: IPage): Promise<GeminiTurn[]> {
|
||||
await ensureGeminiPage(page);
|
||||
const turns = await page.evaluate(getTurnsScript()) as GeminiTurn[];
|
||||
const turns = await getGeminiStructuredTurns(page);
|
||||
if (Array.isArray(turns) && turns.length > 0) return turns;
|
||||
|
||||
const lines = await getGeminiTranscriptLines(page);
|
||||
return lines.map((line) => ({ Role: 'System', Text: line }));
|
||||
}
|
||||
|
||||
async function getGeminiStructuredTurns(page: IPage): Promise<GeminiTurn[]> {
|
||||
await ensureGeminiPage(page);
|
||||
const turns = collapseAdjacentGeminiTurns(await page.evaluate(getTurnsScript()) as GeminiTurn[]);
|
||||
return Array.isArray(turns) ? turns : [];
|
||||
}
|
||||
|
||||
export async function getGeminiTranscriptLines(page: IPage): Promise<string[]> {
|
||||
await ensureGeminiPage(page);
|
||||
return await page.evaluate(getTranscriptLinesScript()) as string[];
|
||||
}
|
||||
|
||||
export async function sendGeminiMessage(page: IPage, text: string): Promise<'button' | 'enter'> {
|
||||
export async function readGeminiSnapshot(page: IPage): Promise<GeminiSnapshot> {
|
||||
await ensureGeminiPage(page);
|
||||
const submittedBy = await page.evaluate(fillAndSubmitComposerScript(text)) as 'button' | 'enter';
|
||||
await page.wait(1);
|
||||
return submittedBy;
|
||||
return await page.evaluate(readGeminiSnapshotScript()) as GeminiSnapshot;
|
||||
}
|
||||
|
||||
function findLastUserTurnIndex(turns: GeminiTurn[]): number | null {
|
||||
for (let index = turns.length - 1; index >= 0; index -= 1) {
|
||||
if (turns[index]?.Role === 'User') return index;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function findLastUserTurn(turns: GeminiTurn[]): GeminiTurn | null {
|
||||
const index = findLastUserTurnIndex(turns);
|
||||
return index === null ? null : turns[index] ?? null;
|
||||
}
|
||||
|
||||
export async function waitForGeminiSubmission(
|
||||
page: IPage,
|
||||
before: GeminiSnapshot,
|
||||
timeoutSeconds: number,
|
||||
): Promise<GeminiSubmissionBaseline | null> {
|
||||
const preSendAssistantCount = before.turns.filter((turn) => turn.Role === 'Assistant').length;
|
||||
const maxPolls = Math.max(1, Math.ceil(timeoutSeconds));
|
||||
|
||||
for (let index = 0; index < maxPolls; index += 1) {
|
||||
await page.wait(index === 0 ? 0.5 : 1);
|
||||
const current = await readGeminiSnapshot(page);
|
||||
const structuredAppend = diffTrustedStructuredTurns(before, current);
|
||||
const transcriptDelta = diffTranscriptLines(before, current);
|
||||
|
||||
if (structuredAppend.hasTrustedAppend && structuredAppend.hasNewUserTurn) {
|
||||
return {
|
||||
snapshot: current,
|
||||
preSendAssistantCount,
|
||||
userAnchorTurn: findLastUserTurn(current.turns),
|
||||
reason: 'user_turn',
|
||||
};
|
||||
}
|
||||
|
||||
if (!current.composerHasText && current.isGenerating) {
|
||||
return {
|
||||
snapshot: current,
|
||||
preSendAssistantCount,
|
||||
userAnchorTurn: findLastUserTurn(current.turns),
|
||||
reason: 'composer_generating',
|
||||
};
|
||||
}
|
||||
|
||||
if (!current.composerHasText && transcriptDelta.length > 0) {
|
||||
return {
|
||||
snapshot: current,
|
||||
preSendAssistantCount,
|
||||
userAnchorTurn: findLastUserTurn(current.turns),
|
||||
reason: 'composer_transcript',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function sendGeminiMessage(page: IPage, text: string): Promise<'button' | 'enter'> {
|
||||
await ensureGeminiPage(page);
|
||||
let prepared: { ok?: boolean; reason?: string } | undefined;
|
||||
for (let attempt = 0; attempt < GEMINI_COMPOSER_PREPARE_ATTEMPTS; attempt += 1) {
|
||||
prepared = await page.evaluate(prepareComposerScript()) as { ok?: boolean; reason?: string };
|
||||
if (prepared?.ok) break;
|
||||
if (attempt < GEMINI_COMPOSER_PREPARE_ATTEMPTS - 1) await page.wait(GEMINI_COMPOSER_PREPARE_WAIT_SECONDS);
|
||||
}
|
||||
if (!prepared?.ok) {
|
||||
throw new CommandExecutionError(prepared?.reason || 'Could not find Gemini composer');
|
||||
}
|
||||
|
||||
let hasText = false;
|
||||
if (page.nativeType) {
|
||||
try {
|
||||
await page.nativeType(text);
|
||||
await page.wait(0.2);
|
||||
const nativeState = await page.evaluate(composerHasTextScript()) as { hasText?: boolean };
|
||||
hasText = !!nativeState?.hasText;
|
||||
} catch {}
|
||||
}
|
||||
|
||||
if (!hasText) {
|
||||
const fallbackState = await page.evaluate(insertComposerTextFallbackScript(text)) as { hasText?: boolean };
|
||||
hasText = !!fallbackState?.hasText;
|
||||
}
|
||||
|
||||
if (!hasText) {
|
||||
throw new CommandExecutionError('Failed to insert text into Gemini composer');
|
||||
}
|
||||
|
||||
const submitAction = await page.evaluate(submitComposerScript()) as 'button' | 'enter';
|
||||
if (submitAction === 'button') {
|
||||
await page.wait(1);
|
||||
return 'button';
|
||||
}
|
||||
|
||||
if (page.nativeKeyPress) {
|
||||
try {
|
||||
await page.nativeKeyPress('Enter');
|
||||
} catch {
|
||||
await page.evaluate(dispatchComposerEnterScript());
|
||||
}
|
||||
} else {
|
||||
await page.evaluate(dispatchComposerEnterScript());
|
||||
}
|
||||
|
||||
await page.wait(1);
|
||||
return 'enter';
|
||||
}
|
||||
|
||||
export const __test__ = {
|
||||
GEMINI_COMPOSER_SELECTORS,
|
||||
GEMINI_COMPOSER_MARKER_ATTR,
|
||||
collapseAdjacentGeminiTurns,
|
||||
clickNewChatScript,
|
||||
diffTranscriptLines,
|
||||
diffTrustedStructuredTurns,
|
||||
hasGeminiTurnPrefix,
|
||||
readGeminiSnapshot,
|
||||
readGeminiSnapshotScript,
|
||||
submitComposerScript,
|
||||
insertComposerTextFallbackScript,
|
||||
};
|
||||
|
||||
export async function getGeminiVisibleImageUrls(page: IPage): Promise<string[]> {
|
||||
await ensureGeminiPage(page);
|
||||
@@ -484,40 +963,89 @@ export async function exportGeminiImages(page: IPage, urls: string[]): Promise<G
|
||||
}
|
||||
export async function waitForGeminiResponse(
|
||||
page: IPage,
|
||||
beforeLines: string[],
|
||||
baseline: GeminiSubmissionBaseline,
|
||||
promptText: string,
|
||||
timeoutSeconds: number,
|
||||
): Promise<string> {
|
||||
const getCandidate = async (): Promise<string> => {
|
||||
const turns = await getGeminiVisibleTurns(page);
|
||||
const assistantCandidate = [...turns].reverse().find((turn) => turn.Role === 'Assistant');
|
||||
const visibleCandidate = assistantCandidate
|
||||
? sanitizeGeminiResponseText(assistantCandidate.Text, promptText)
|
||||
: '';
|
||||
if (visibleCandidate && visibleCandidate !== promptText) return visibleCandidate;
|
||||
if (timeoutSeconds <= 0) return '';
|
||||
|
||||
const lines = await getGeminiTranscriptLines(page);
|
||||
return collectGeminiTranscriptAdditions(beforeLines, lines, promptText);
|
||||
};
|
||||
// Reply ownership must survive Gemini prepending older history later.
|
||||
// Re-anchor on the submitted user turn when possible, and otherwise only
|
||||
// accept assistants that are appended to the exact submission snapshot.
|
||||
const pickStructuredReplyCandidate = (current: GeminiSnapshot): string => {
|
||||
if (!current.structuredTurnsTrusted) return '';
|
||||
|
||||
const pollIntervalSeconds = 2;
|
||||
const maxPolls = Math.max(1, Math.ceil(timeoutSeconds / pollIntervalSeconds));
|
||||
let lastCandidate = '';
|
||||
let stableCount = 0;
|
||||
|
||||
for (let index = 0; index < maxPolls; index += 1) {
|
||||
await page.wait(index === 0 ? 1.5 : pollIntervalSeconds);
|
||||
const candidate = await getCandidate();
|
||||
if (!candidate) continue;
|
||||
|
||||
if (candidate === lastCandidate) stableCount += 1;
|
||||
else {
|
||||
lastCandidate = candidate;
|
||||
stableCount = 1;
|
||||
const userAnchorTurnIndex = findLastMatchingGeminiTurnIndex(current.turns, baseline.userAnchorTurn);
|
||||
if (userAnchorTurnIndex !== null) {
|
||||
const candidate = current.turns
|
||||
.slice(userAnchorTurnIndex + 1)
|
||||
.filter((turn) => turn.Role === 'Assistant')
|
||||
.at(-1);
|
||||
return candidate ? sanitizeGeminiResponseText(candidate.Text, promptText) : '';
|
||||
}
|
||||
|
||||
if (stableCount >= 2 || index === maxPolls - 1) return candidate;
|
||||
if (hasGeminiTurnPrefix(baseline.snapshot.turns, current.turns)) {
|
||||
const appendedAssistant = current.turns
|
||||
.slice(baseline.snapshot.turns.length)
|
||||
.filter((turn) => turn.Role === 'Assistant')
|
||||
.at(-1);
|
||||
if (appendedAssistant) {
|
||||
return sanitizeGeminiResponseText(appendedAssistant.Text, promptText);
|
||||
}
|
||||
}
|
||||
|
||||
return '';
|
||||
};
|
||||
|
||||
const pickFallbackGeminiTranscriptReply = (current: GeminiSnapshot): string => current.transcriptLines
|
||||
.filter((line) => !baseline.snapshot.transcriptLines.includes(line))
|
||||
.map((line) => extractGeminiTranscriptLineCandidate(line, promptText))
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
.trim();
|
||||
|
||||
const maxPolls = Math.max(1, Math.ceil(timeoutSeconds / 2));
|
||||
let lastStructured = '';
|
||||
let structuredStableCount = 0;
|
||||
let lastTranscript = '';
|
||||
let transcriptStableCount = 0;
|
||||
let transcriptMissCount = 0;
|
||||
|
||||
for (let index = 0; index < maxPolls; index += 1) {
|
||||
await page.wait(index === 0 ? 1 : 2);
|
||||
const current = await readGeminiSnapshot(page);
|
||||
const structuredCandidate = pickStructuredReplyCandidate(current);
|
||||
|
||||
if (structuredCandidate) {
|
||||
if (structuredCandidate === lastStructured) structuredStableCount += 1;
|
||||
else {
|
||||
lastStructured = structuredCandidate;
|
||||
structuredStableCount = 1;
|
||||
}
|
||||
|
||||
if (!current.isGenerating && structuredStableCount >= 2) {
|
||||
return structuredCandidate;
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
transcriptMissCount += 1;
|
||||
if (transcriptMissCount < 2) continue;
|
||||
|
||||
const transcriptCandidate = pickFallbackGeminiTranscriptReply(current);
|
||||
if (!transcriptCandidate) continue;
|
||||
|
||||
if (transcriptCandidate === lastTranscript) transcriptStableCount += 1;
|
||||
else {
|
||||
lastTranscript = transcriptCandidate;
|
||||
transcriptStableCount = 1;
|
||||
}
|
||||
|
||||
if (!current.isGenerating && transcriptStableCount >= 2) {
|
||||
return transcriptCandidate;
|
||||
}
|
||||
}
|
||||
|
||||
return lastCandidate;
|
||||
return '';
|
||||
}
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { getHupuThreadUrl, readHupuNextData, stripHtml } from './utils.js';
|
||||
|
||||
// JSON数据结构(对应Next.js的__NEXT_DATA__)
|
||||
interface NextData {
|
||||
props: Props;
|
||||
}
|
||||
|
||||
interface Props {
|
||||
pageProps: PageProps;
|
||||
}
|
||||
|
||||
interface PageProps {
|
||||
detail?: Detail;
|
||||
detail_error_info?: DetailError;
|
||||
}
|
||||
|
||||
interface DetailError {
|
||||
code: number;
|
||||
message: string;
|
||||
}
|
||||
|
||||
interface Detail {
|
||||
thread?: Thread;
|
||||
lights?: ReplyData[];
|
||||
}
|
||||
|
||||
interface Thread {
|
||||
tid: string;
|
||||
title: string;
|
||||
content: string;
|
||||
lights?: number;
|
||||
replies?: number;
|
||||
author?: Author;
|
||||
}
|
||||
|
||||
interface Author {
|
||||
puname?: string;
|
||||
}
|
||||
|
||||
interface ReplyData {
|
||||
pid: string;
|
||||
author?: Author;
|
||||
content: string;
|
||||
allLightCount?: number; // 修复:正确的字段名
|
||||
created_at_format?: string;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'hupu',
|
||||
name: 'detail',
|
||||
description: '获取虎扑帖子详情 (使用Next.js JSON数据)',
|
||||
domain: 'bbs.hupu.com',
|
||||
strategy: Strategy.PUBLIC,
|
||||
browser: true,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'tid',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '帖子ID(9位数字)'
|
||||
},
|
||||
{
|
||||
name: 'replies',
|
||||
type: 'boolean',
|
||||
default: false,
|
||||
help: '是否包含热门回复'
|
||||
}
|
||||
],
|
||||
columns: ['title', 'author', 'content', 'replies', 'lights', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const { tid, replies: includeReplies = false } = kwargs;
|
||||
|
||||
const url = getHupuThreadUrl(tid).replace(/-1\.html$/, '.html');
|
||||
const data = await readHupuNextData<NextData>(page, url, 'Read Hupu thread detail', {
|
||||
expectedTid: String(tid),
|
||||
});
|
||||
|
||||
// 检查错误信息(只有当code不是200时才报错)
|
||||
const errorInfo = data.props.pageProps.detail_error_info;
|
||||
if (errorInfo && errorInfo.code !== 200) {
|
||||
throw new Error(`帖子访问失败: ${errorInfo.message} (code: ${errorInfo.code})`);
|
||||
}
|
||||
|
||||
// 获取帖子信息
|
||||
const thread = data.props.pageProps.detail?.thread;
|
||||
if (!thread) {
|
||||
throw new Error('帖子不存在或已被删除');
|
||||
}
|
||||
|
||||
const authorName = thread.author?.puname || '未知作者';
|
||||
const content = stripHtml(thread.content);
|
||||
const contentPreview = content.length > 300 ? content.substring(0, 300) + '...' : content;
|
||||
|
||||
// 构建结果
|
||||
const result: any = {
|
||||
title: thread.title,
|
||||
author: authorName,
|
||||
content: contentPreview,
|
||||
replies: thread.replies || 0,
|
||||
lights: thread.lights || 0,
|
||||
url: `https://bbs.hupu.com/${tid}.html`
|
||||
};
|
||||
|
||||
// 如果需要包含回复,添加回复信息到内容中
|
||||
if (includeReplies) {
|
||||
const replyList = data.props.pageProps.detail?.lights || [];
|
||||
const topReplies = replyList.slice(0, 3);
|
||||
|
||||
if (topReplies.length > 0) {
|
||||
let replyText = '\n\n【热门回复】\n';
|
||||
topReplies.forEach((reply, index) => {
|
||||
const userName = reply.author?.puname || '未知用户';
|
||||
const replyContent = stripHtml(reply.content).substring(0, 100);
|
||||
const replyLights = reply.allLightCount || 0; // 修复:使用正确的字段名
|
||||
const replyTime = reply.created_at_format || '未知时间';
|
||||
replyText += `${index + 1}. ${userName} (亮${replyLights} ${replyTime}):\n ${replyContent}\n\n`;
|
||||
});
|
||||
result.content = contentPreview + replyText;
|
||||
}
|
||||
}
|
||||
|
||||
return [result];
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,43 @@
|
||||
site: hupu
|
||||
name: hot
|
||||
description: 虎扑热门帖子
|
||||
domain: bbs.hupu.com
|
||||
|
||||
args:
|
||||
limit:
|
||||
type: int
|
||||
default: 20
|
||||
description: Number of hot posts
|
||||
|
||||
pipeline:
|
||||
- navigate: https://bbs.hupu.com/
|
||||
|
||||
- evaluate: |
|
||||
(async () => {
|
||||
// 从HTML中提取帖子信息(适配新的HTML结构)
|
||||
const html = document.documentElement.outerHTML;
|
||||
const posts = [];
|
||||
|
||||
// 匹配当前虎扑页面结构的正则表达式
|
||||
// 结构: <a href="/638249612.html"...><span class="t-title">标题</span></a>
|
||||
const regex = /<a[^>]*href="\/(\d{9})\.html"[^>]*><span[^>]*class="t-title"[^>]*>([^<]+)<\/span><\/a>/g;
|
||||
let match;
|
||||
|
||||
while ((match = regex.exec(html)) !== null && posts.length < ${{ args.limit }}) {
|
||||
posts.push({
|
||||
tid: match[1],
|
||||
title: match[2].trim()
|
||||
});
|
||||
}
|
||||
|
||||
return posts;
|
||||
})()
|
||||
|
||||
- map:
|
||||
rank: ${{ index + 1 }}
|
||||
title: ${{ item.title }}
|
||||
url: https://bbs.hupu.com/${{ item.tid }}.html
|
||||
|
||||
- limit: ${{ args.limit }}
|
||||
|
||||
columns: [rank, title, url]
|
||||
@@ -0,0 +1,76 @@
|
||||
import { CliError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { postHupuJson } from './utils.js';
|
||||
|
||||
cli({
|
||||
site: 'hupu',
|
||||
name: 'like',
|
||||
description: '点赞虎扑回复 (需要登录)',
|
||||
domain: 'bbs.hupu.com',
|
||||
strategy: Strategy.COOKIE, // 需要Cookie认证
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'tid',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '帖子ID(9位数字)'
|
||||
},
|
||||
{
|
||||
name: 'pid',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '回复ID'
|
||||
},
|
||||
{
|
||||
name: 'fid',
|
||||
required: true,
|
||||
help: '板块ID(如278汽车区)'
|
||||
}
|
||||
],
|
||||
columns: ['status', 'message'],
|
||||
func: async (page, kwargs) => {
|
||||
const { tid, pid, fid } = kwargs;
|
||||
|
||||
const url = 'https://bbs.hupu.com/pcmapi/pc/bbs/v1/reply/light';
|
||||
|
||||
// 构建请求体
|
||||
const body = {
|
||||
tid,
|
||||
pid,
|
||||
puid: '',
|
||||
fid,
|
||||
shumei_id: '',
|
||||
deviceid: ''
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await postHupuJson(page, tid, url, body, 'Like Hupu reply');
|
||||
|
||||
// 处理响应
|
||||
if (result.code === 1) {
|
||||
return [{
|
||||
status: '✅ 点赞成功',
|
||||
message: ''
|
||||
}];
|
||||
} else if (result.code === 0 && result.msg === '你已经点亮过这个回帖了') {
|
||||
return [{
|
||||
status: '⚠️ 已经点赞过了',
|
||||
message: result.msg || ''
|
||||
}];
|
||||
} else if (result.code === 0) {
|
||||
return [{
|
||||
status: '⚠️ 操作未执行',
|
||||
message: result.msg || result.message || ''
|
||||
}];
|
||||
} else {
|
||||
throw new Error(`接口错误 code=${result.code}: ${result.msg || result.message}`);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
if (error instanceof CliError) throw error;
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
throw new Error(`点赞失败: ${errorMessage}`);
|
||||
}
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import { CliError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { postHupuJson } from './utils.js';
|
||||
|
||||
cli({
|
||||
site: 'hupu',
|
||||
name: 'reply',
|
||||
description: '回复虎扑帖子 (需要登录)',
|
||||
domain: 'bbs.hupu.com',
|
||||
strategy: Strategy.COOKIE, // 需要Cookie认证
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'tid',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '帖子ID(9位数字)'
|
||||
},
|
||||
{
|
||||
name: 'topic_id',
|
||||
required: true,
|
||||
help: '板块ID,即接口中的 topicId(如 502 篮球资讯)'
|
||||
},
|
||||
{
|
||||
name: 'text',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '回复内容'
|
||||
},
|
||||
{
|
||||
name: 'quote_id',
|
||||
help: '被引用回复的 pid;填写后会以“回复某条热门回复”的方式发言'
|
||||
}
|
||||
],
|
||||
columns: ['status', 'message'],
|
||||
func: async (page, kwargs) => {
|
||||
const { tid, topic_id, text, quote_id } = kwargs;
|
||||
|
||||
const url = 'https://bbs.hupu.com/pcmapi/pc/bbs/v1/createReply';
|
||||
|
||||
// 虎扑内容用 <p> 包裹
|
||||
const content = `<p>${text}</p>`;
|
||||
|
||||
// 构建请求体
|
||||
const body: Record<string, unknown> = {
|
||||
topicId: topic_id,
|
||||
content,
|
||||
shumeiId: '',
|
||||
deviceid: '',
|
||||
tid
|
||||
};
|
||||
|
||||
// 如果有引用回复ID,添加到请求体
|
||||
if (quote_id) {
|
||||
body.quoteId = quote_id;
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await postHupuJson(page, tid, url, body, 'Reply to Hupu thread', 'reply');
|
||||
|
||||
if (result.code === 1) {
|
||||
return [{
|
||||
status: '✅ 回复成功',
|
||||
message: result.msg || result.message || ''
|
||||
}];
|
||||
} else {
|
||||
throw new Error(`接口错误 code=${result.code}: ${result.msg || result.message}`);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
if (error instanceof CliError) throw error;
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
throw new Error(`回复失败: ${errorMessage}`);
|
||||
}
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,95 @@
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { decodeHtmlEntities, getHupuSearchUrl, readHupuSearchData, stripHtml } from './utils.js';
|
||||
|
||||
// 搜索结果数据结构
|
||||
interface SearchResult {
|
||||
id: string;
|
||||
title: string;
|
||||
content?: string;
|
||||
username?: string;
|
||||
addTimeDisplay?: string;
|
||||
replies?: string;
|
||||
lights?: string;
|
||||
recNum?: string;
|
||||
forum_name?: string;
|
||||
fid?: string;
|
||||
}
|
||||
|
||||
// 虎扑搜索响应数据结构
|
||||
interface HupuSearchResponse {
|
||||
init?: {
|
||||
redirect?: string;
|
||||
};
|
||||
env?: string;
|
||||
query?: {
|
||||
q?: string;
|
||||
page?: string;
|
||||
};
|
||||
searchRes?: {
|
||||
count: number;
|
||||
totalPage: number;
|
||||
type?: string;
|
||||
data: SearchResult[];
|
||||
};
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'hupu',
|
||||
name: 'search',
|
||||
description: '搜索虎扑帖子 (使用官方API)',
|
||||
domain: 'bbs.hupu.com',
|
||||
strategy: Strategy.PUBLIC, // 公开API,不需要Cookie
|
||||
browser: true,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'query',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '搜索关键词'
|
||||
},
|
||||
{
|
||||
name: 'page',
|
||||
type: 'int',
|
||||
default: 1,
|
||||
help: '结果页码'
|
||||
},
|
||||
{
|
||||
name: 'limit',
|
||||
type: 'int',
|
||||
default: 20,
|
||||
help: '返回结果数量'
|
||||
},
|
||||
{
|
||||
name: 'forum',
|
||||
help: '板块ID过滤 (可选)'
|
||||
},
|
||||
{
|
||||
name: 'sort',
|
||||
default: 'general',
|
||||
help: '排序方式: general/createtime/replytime/light/reply'
|
||||
}
|
||||
],
|
||||
columns: ['rank', 'title', 'author', 'replies', 'lights', 'forum', 'url'],
|
||||
func: async (page, kwargs) => {
|
||||
const { query, page: pageNum = 1, limit = 20, forum, sort = 'general' } = kwargs;
|
||||
const searchUrl = getHupuSearchUrl(query, pageNum, forum, sort);
|
||||
const data = await readHupuSearchData<HupuSearchResponse>(page, searchUrl, 'Search Hupu threads');
|
||||
|
||||
// 提取搜索结果
|
||||
const results = data.searchRes?.data || [];
|
||||
|
||||
// 处理结果:清理HTML标签,解码HTML实体
|
||||
const processedResults = results.slice(0, Number(limit)).map((item, index) => ({
|
||||
rank: index + 1,
|
||||
title: decodeHtmlEntities(stripHtml(item.title)),
|
||||
author: item.username || '未知用户',
|
||||
replies: item.replies || '0',
|
||||
lights: item.lights || '0',
|
||||
forum: item.forum_name || '未知板块',
|
||||
url: `https://bbs.hupu.com/${item.id}.html`
|
||||
}));
|
||||
|
||||
return processedResults;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import { CliError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { postHupuJson } from './utils.js';
|
||||
|
||||
cli({
|
||||
site: 'hupu',
|
||||
name: 'unlike',
|
||||
description: '取消点赞虎扑回复 (需要登录)',
|
||||
domain: 'bbs.hupu.com',
|
||||
strategy: Strategy.COOKIE, // 需要Cookie认证
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{
|
||||
name: 'tid',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '帖子ID(9位数字)'
|
||||
},
|
||||
{
|
||||
name: 'pid',
|
||||
required: true,
|
||||
positional: true,
|
||||
help: '回复ID'
|
||||
},
|
||||
{
|
||||
name: 'fid',
|
||||
required: true,
|
||||
help: '板块ID(如278汽车区)'
|
||||
}
|
||||
],
|
||||
columns: ['status', 'message'],
|
||||
func: async (page, kwargs) => {
|
||||
const { tid, pid, fid } = kwargs;
|
||||
|
||||
const url = 'https://bbs.hupu.com/pcmapi/pc/bbs/v1/reply/cancelLight';
|
||||
|
||||
// 构建请求体(与点赞相同)
|
||||
const body = {
|
||||
tid,
|
||||
pid,
|
||||
puid: '',
|
||||
fid,
|
||||
shumei_id: '',
|
||||
deviceid: ''
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await postHupuJson(page, tid, url, body, 'Unlike Hupu reply');
|
||||
|
||||
// 处理响应
|
||||
if (result.code === 1) {
|
||||
return [{
|
||||
status: '✅ 取消点赞成功',
|
||||
message: ''
|
||||
}];
|
||||
} else if (result.code === 0 && result.msg === '你还没有点亮过这个回帖') {
|
||||
return [{
|
||||
status: '⚠️ 你还没点赞过',
|
||||
message: result.msg || ''
|
||||
}];
|
||||
} else if (result.code === 0) {
|
||||
return [{
|
||||
status: '⚠️ 操作未执行',
|
||||
message: result.msg || result.message || ''
|
||||
}];
|
||||
} else {
|
||||
throw new Error(`接口错误 code=${result.code}: ${result.msg || result.message}`);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
if (error instanceof CliError) throw error;
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
throw new Error(`取消点赞失败: ${errorMessage}`);
|
||||
}
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,381 @@
|
||||
import { AuthRequiredError, CommandExecutionError } from '../../errors.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
|
||||
export interface HupuApiResponse {
|
||||
code?: number;
|
||||
msg?: string;
|
||||
message?: string;
|
||||
}
|
||||
|
||||
interface BrowserFetchResult {
|
||||
ok?: boolean;
|
||||
status?: number;
|
||||
data?: HupuApiResponse | null;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
interface BrowserDataResult<T> {
|
||||
ok?: boolean;
|
||||
data?: T;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export function stripHtml(html: string): string {
|
||||
if (!html) return '';
|
||||
const decoded = html
|
||||
.replace(/\\u003c/g, '<')
|
||||
.replace(/\\u003e/g, '>')
|
||||
.replace(/\\n/g, '\n')
|
||||
.replace(/\\r/g, '');
|
||||
return decoded.replace(/<[^>]+>/g, '').trim();
|
||||
}
|
||||
|
||||
export function decodeHtmlEntities(html: string): string {
|
||||
if (!html) return '';
|
||||
return html.replace(/ /g, ' ')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>')
|
||||
.replace(/&/g, '&')
|
||||
.replace(/"/g, '"')
|
||||
.replace(/'/g, "'");
|
||||
}
|
||||
|
||||
export function getHupuThreadUrl(tid: unknown): string {
|
||||
return `https://bbs.hupu.com/${encodeURIComponent(String(tid))}-1.html`;
|
||||
}
|
||||
|
||||
export function getHupuSearchUrl(query: unknown, page: unknown, forum?: unknown, sort?: unknown): string {
|
||||
const searchParams = new URLSearchParams();
|
||||
searchParams.append('q', String(query));
|
||||
searchParams.append('page', String(page));
|
||||
|
||||
if (forum) {
|
||||
searchParams.append('topicId', String(forum));
|
||||
}
|
||||
|
||||
if (sort) {
|
||||
searchParams.append('sortby', String(sort));
|
||||
}
|
||||
|
||||
return `https://bbs.hupu.com/search?${searchParams.toString()}`;
|
||||
}
|
||||
|
||||
export async function readHupuNextData<T>(
|
||||
page: IPage,
|
||||
url: string,
|
||||
actionLabel: string,
|
||||
options: {
|
||||
expectedTid?: string;
|
||||
timeoutMs?: number;
|
||||
} = {},
|
||||
): Promise<T> {
|
||||
await page.goto(url);
|
||||
|
||||
const result = await page.evaluate(`
|
||||
(async () => {
|
||||
const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
const expectedTid = ${JSON.stringify(options.expectedTid || '')};
|
||||
const timeoutMs = ${JSON.stringify(options.timeoutMs ?? 5000)};
|
||||
let lastSeenTid = '';
|
||||
let lastSeenHref = '';
|
||||
|
||||
const waitFor = async (predicate, limitMs = timeoutMs) => {
|
||||
const start = Date.now();
|
||||
while (Date.now() - start < limitMs) {
|
||||
if (predicate()) return true;
|
||||
await wait(100);
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
const ready = await waitFor(() => {
|
||||
const script = document.getElementById('__NEXT_DATA__');
|
||||
if (!script?.textContent) return false;
|
||||
|
||||
lastSeenHref = location.href;
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(script.textContent);
|
||||
const threadTid = parsed?.props?.pageProps?.detail?.thread?.tid;
|
||||
lastSeenTid = typeof threadTid === 'string' ? threadTid : '';
|
||||
|
||||
if (!expectedTid) return true;
|
||||
return threadTid === expectedTid;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
});
|
||||
if (!ready) {
|
||||
return {
|
||||
ok: false,
|
||||
error: expectedTid
|
||||
? \`帖子数据未就绪或tid不匹配(expected=\${expectedTid}, actual=\${lastSeenTid || 'unknown'}, href=\${lastSeenHref || location.href})\`
|
||||
: '无法找到帖子数据'
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
const text = document.getElementById('__NEXT_DATA__')?.textContent || '';
|
||||
return {
|
||||
ok: true,
|
||||
data: JSON.parse(text)
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
};
|
||||
}
|
||||
})()
|
||||
`) as BrowserDataResult<T>;
|
||||
|
||||
if (!result || typeof result !== 'object' || !result.ok) {
|
||||
throw new CommandExecutionError(`${actionLabel} failed: ${result?.error || 'invalid browser response'}`);
|
||||
}
|
||||
|
||||
return result.data as T;
|
||||
}
|
||||
|
||||
export async function readHupuSearchData<T>(page: IPage, url: string, actionLabel: string): Promise<T> {
|
||||
await page.goto(url);
|
||||
|
||||
const result = await page.evaluate(`
|
||||
(async () => {
|
||||
const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
const waitFor = async (predicate, timeoutMs = 5000) => {
|
||||
const start = Date.now();
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
if (predicate()) return true;
|
||||
await wait(100);
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
const extractFromScript = () => {
|
||||
const marker = 'window.$$data=';
|
||||
for (const script of Array.from(document.scripts)) {
|
||||
const text = script.textContent || '';
|
||||
const dataIndex = text.indexOf(marker);
|
||||
if (dataIndex === -1) continue;
|
||||
|
||||
const jsonStart = dataIndex + marker.length;
|
||||
let braceCount = 0;
|
||||
let jsonEnd = jsonStart;
|
||||
let inString = false;
|
||||
let escapeNext = false;
|
||||
|
||||
for (let i = jsonStart; i < text.length; i++) {
|
||||
const char = text[i];
|
||||
|
||||
if (escapeNext) {
|
||||
escapeNext = false;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (char === '\\\\') {
|
||||
escapeNext = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (char === '"') {
|
||||
inString = !inString;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!inString) {
|
||||
if (char === '{') {
|
||||
braceCount++;
|
||||
} else if (char === '}') {
|
||||
braceCount--;
|
||||
if (braceCount === 0) {
|
||||
jsonEnd = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (jsonEnd > jsonStart) {
|
||||
return text.substring(jsonStart, jsonEnd + 1);
|
||||
}
|
||||
}
|
||||
return '';
|
||||
};
|
||||
|
||||
const ready = await waitFor(() => {
|
||||
return typeof window.$$data !== 'undefined' || Boolean(extractFromScript());
|
||||
});
|
||||
if (!ready) {
|
||||
return { ok: false, error: '无法找到搜索数据' };
|
||||
}
|
||||
|
||||
try {
|
||||
if (typeof window.$$data !== 'undefined') {
|
||||
return {
|
||||
ok: true,
|
||||
data: JSON.parse(JSON.stringify(window.$$data))
|
||||
};
|
||||
}
|
||||
|
||||
const jsonString = extractFromScript();
|
||||
return {
|
||||
ok: true,
|
||||
data: JSON.parse(jsonString)
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
};
|
||||
}
|
||||
})()
|
||||
`) as BrowserDataResult<T>;
|
||||
|
||||
if (!result || typeof result !== 'object' || !result.ok) {
|
||||
throw new CommandExecutionError(`${actionLabel} failed: ${result?.error || 'invalid browser response'}`);
|
||||
}
|
||||
|
||||
return result.data as T;
|
||||
}
|
||||
|
||||
function buildBrowserJsonPostScript(
|
||||
apiUrl: string,
|
||||
body: Record<string, unknown>,
|
||||
mode: 'default' | 'reply',
|
||||
): string {
|
||||
return `
|
||||
(async () => {
|
||||
const url = ${JSON.stringify(apiUrl)};
|
||||
const payload = ${JSON.stringify(body)};
|
||||
const mode = ${JSON.stringify(mode)};
|
||||
const getCookie = (name) => document.cookie
|
||||
.split('; ')
|
||||
.find((item) => item.startsWith(name + '='))
|
||||
?.slice(name.length + 1) || '';
|
||||
|
||||
const findThumbcacheValue = () => {
|
||||
const rawEntry = document.cookie
|
||||
.split('; ')
|
||||
.find((item) => item.startsWith('.thumbcache_'));
|
||||
if (rawEntry && rawEntry.includes('=')) {
|
||||
const rawValue = rawEntry.slice(rawEntry.indexOf('=') + 1);
|
||||
try {
|
||||
return decodeURIComponent(rawValue);
|
||||
} catch {
|
||||
return rawValue;
|
||||
}
|
||||
}
|
||||
|
||||
const storageKey = Object.keys(localStorage).find((key) => key.startsWith('.thumbcache_'));
|
||||
if (!storageKey) return '';
|
||||
return localStorage.getItem(storageKey) || '';
|
||||
};
|
||||
|
||||
const resolveDefaultPayload = (input) => {
|
||||
const next = { ...input };
|
||||
const sensorsRaw = decodeURIComponent(getCookie('sensorsdata2015jssdkcross') || '');
|
||||
let deviceid = '';
|
||||
try {
|
||||
const sensors = JSON.parse(sensorsRaw);
|
||||
deviceid = sensors?.props?.['$device_id'] || sensors?.distinct_id || '';
|
||||
} catch {}
|
||||
|
||||
if ((next.puid === '' || next.puid == null) && getCookie('ua')) {
|
||||
next.puid = getCookie('ua');
|
||||
}
|
||||
if ((next.shumei_id === '' || next.shumei_id == null) && getCookie('smidV2')) {
|
||||
next.shumei_id = getCookie('smidV2');
|
||||
}
|
||||
if ((next.deviceid === '' || next.deviceid == null) && deviceid) {
|
||||
next.deviceid = deviceid;
|
||||
}
|
||||
return next;
|
||||
};
|
||||
|
||||
const resolveReplyPayload = (input) => {
|
||||
const next = { ...input };
|
||||
const thumbcache = findThumbcacheValue();
|
||||
if ((next.shumeiId === '' || next.shumeiId == null) && thumbcache) {
|
||||
next.shumeiId = thumbcache;
|
||||
}
|
||||
if ((next.deviceid === '' || next.deviceid == null) && thumbcache) {
|
||||
next.deviceid = thumbcache;
|
||||
}
|
||||
return next;
|
||||
};
|
||||
|
||||
const resolvedPayload = mode === 'reply'
|
||||
? resolveReplyPayload(payload)
|
||||
: resolveDefaultPayload(payload);
|
||||
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
credentials: 'include',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify(resolvedPayload)
|
||||
});
|
||||
|
||||
const text = await response.text();
|
||||
let data = null;
|
||||
try {
|
||||
data = text ? JSON.parse(text) : null;
|
||||
} catch {
|
||||
data = text ? { message: text } : null;
|
||||
}
|
||||
|
||||
return {
|
||||
ok: response.ok,
|
||||
status: response.status,
|
||||
data
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
};
|
||||
}
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute authenticated Hupu JSON requests inside the browser page so
|
||||
* cookies and the thread referer come from the live logged-in session.
|
||||
*/
|
||||
export async function postHupuJson(
|
||||
page: IPage,
|
||||
tid: unknown,
|
||||
apiUrl: string,
|
||||
body: Record<string, unknown>,
|
||||
actionLabel: string,
|
||||
mode: 'default' | 'reply' = 'default',
|
||||
): Promise<HupuApiResponse> {
|
||||
const referer = getHupuThreadUrl(tid);
|
||||
await page.goto(referer);
|
||||
|
||||
const result = await page.evaluate(
|
||||
buildBrowserJsonPostScript(apiUrl, body, mode),
|
||||
) as BrowserFetchResult;
|
||||
|
||||
if (!result || typeof result !== 'object') {
|
||||
throw new CommandExecutionError(`${actionLabel} failed: invalid browser response`);
|
||||
}
|
||||
|
||||
if (result.status === 401 || result.status === 403) {
|
||||
throw new AuthRequiredError('bbs.hupu.com', `${actionLabel} failed: please log in to Hupu first`);
|
||||
}
|
||||
|
||||
if (result.error) {
|
||||
throw new CommandExecutionError(`${actionLabel} failed: ${result.error}`);
|
||||
}
|
||||
|
||||
if (!result.ok) {
|
||||
const detail = result.data?.msg || result.data?.message || `HTTP ${result.status ?? 'unknown'}`;
|
||||
throw new CommandExecutionError(`${actionLabel} failed: ${detail}`);
|
||||
}
|
||||
|
||||
return result.data ?? {};
|
||||
}
|
||||
@@ -0,0 +1,827 @@
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import { afterAll, describe, expect, it } from 'vitest';
|
||||
|
||||
import type { InstagramProtocolCaptureEntry } from './protocol-capture.js';
|
||||
import {
|
||||
buildConfigureBody,
|
||||
buildConfigureSidecarPayload,
|
||||
buildConfigureToStoryPhotoPayload,
|
||||
buildConfigureToStoryVideoPayload,
|
||||
deriveInstagramJazoest,
|
||||
derivePrivateApiContextFromCapture,
|
||||
extractInstagramRuntimeInfo,
|
||||
getInstagramFeedNormalizedDimensions,
|
||||
getInstagramStoryNormalizedDimensions,
|
||||
isInstagramFeedAspectRatioAllowed,
|
||||
isInstagramStoryAspectRatioAllowed,
|
||||
publishStoryViaPrivateApi,
|
||||
publishMediaViaPrivateApi,
|
||||
publishImagesViaPrivateApi,
|
||||
readImageAsset,
|
||||
resolveInstagramPrivatePublishConfig,
|
||||
} from './private-publish.js';
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function createTempFile(name: string, bytes: Buffer): string {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-instagram-private-'));
|
||||
tempDirs.push(dir);
|
||||
const filePath = path.join(dir, name);
|
||||
fs.writeFileSync(filePath, bytes);
|
||||
return filePath;
|
||||
}
|
||||
|
||||
afterAll(() => {
|
||||
for (const dir of tempDirs) {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe('instagram private publish helpers', () => {
|
||||
it('derives the private API context from captured instagram request headers', () => {
|
||||
const entries: InstagramProtocolCaptureEntry[] = [
|
||||
{
|
||||
kind: 'cdp' as never,
|
||||
url: 'https://www.instagram.com/api/v1/feed/timeline/',
|
||||
method: 'GET',
|
||||
requestHeaders: {
|
||||
'X-ASBD-ID': '359341',
|
||||
'X-CSRFToken': 'csrf-token',
|
||||
'X-IG-App-ID': '936619743392459',
|
||||
'X-IG-WWW-Claim': 'hmac.claim',
|
||||
'X-Instagram-AJAX': '1036517563',
|
||||
'X-Web-Session-ID': 'abc:def:ghi',
|
||||
},
|
||||
timestamp: Date.now(),
|
||||
},
|
||||
];
|
||||
|
||||
expect(derivePrivateApiContextFromCapture(entries)).toEqual({
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
});
|
||||
});
|
||||
|
||||
it('derives jazoest from the csrf token', () => {
|
||||
expect(deriveInstagramJazoest('SJ_btbvfkpAVFKCN_tJstW')).toBe('22047');
|
||||
});
|
||||
|
||||
it('extracts app id, rollout hash, and csrf token from instagram html', () => {
|
||||
const html = `
|
||||
<html>
|
||||
<head>
|
||||
<script type="application/json">
|
||||
{"csrf_token":"csrf-from-html","rollout_hash":"1036523242","X-IG-App-ID":"936619743392459"}
|
||||
</script>
|
||||
</head>
|
||||
</html>
|
||||
`;
|
||||
expect(extractInstagramRuntimeInfo(html)).toEqual({
|
||||
appId: '936619743392459',
|
||||
csrfToken: 'csrf-from-html',
|
||||
instagramAjax: '1036523242',
|
||||
});
|
||||
});
|
||||
|
||||
it('resolves private publish config from capture, runtime html, and cookies', async () => {
|
||||
const entries: InstagramProtocolCaptureEntry[] = [
|
||||
{
|
||||
kind: 'cdp' as never,
|
||||
url: 'https://www.instagram.com/api/v1/feed/timeline/',
|
||||
method: 'GET',
|
||||
requestHeaders: {
|
||||
'X-ASBD-ID': '359341',
|
||||
'X-IG-WWW-Claim': 'hmac.claim',
|
||||
'X-Web-Session-ID': 'abc:def:ghi',
|
||||
},
|
||||
timestamp: Date.now(),
|
||||
},
|
||||
];
|
||||
const page = {
|
||||
goto: async () => undefined,
|
||||
wait: async () => undefined,
|
||||
getCookies: async () => [{ name: 'csrftoken', value: 'csrf-cookie', domain: 'instagram.com' }],
|
||||
startNetworkCapture: async () => undefined,
|
||||
readNetworkCapture: async () => entries,
|
||||
evaluate: async () => ({
|
||||
appId: '936619743392459',
|
||||
csrfToken: 'csrf-from-html',
|
||||
instagramAjax: '1036523242',
|
||||
}),
|
||||
} as any;
|
||||
|
||||
await expect(resolveInstagramPrivatePublishConfig(page)).resolves.toEqual({
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-from-html',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036523242',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: deriveInstagramJazoest('csrf-from-html'),
|
||||
});
|
||||
});
|
||||
|
||||
it('retries transient private publish config resolution failures and then succeeds', async () => {
|
||||
const entries: InstagramProtocolCaptureEntry[] = [
|
||||
{
|
||||
kind: 'cdp' as never,
|
||||
url: 'https://www.instagram.com/api/v1/feed/timeline/',
|
||||
method: 'GET',
|
||||
requestHeaders: {
|
||||
'X-ASBD-ID': '359341',
|
||||
'X-IG-WWW-Claim': 'hmac.claim',
|
||||
'X-Web-Session-ID': 'abc:def:ghi',
|
||||
},
|
||||
timestamp: Date.now(),
|
||||
},
|
||||
];
|
||||
let evaluateAttempts = 0;
|
||||
const page = {
|
||||
goto: async () => undefined,
|
||||
wait: async () => undefined,
|
||||
getCookies: async () => [{ name: 'csrftoken', value: 'csrf-cookie', domain: 'instagram.com' }],
|
||||
startNetworkCapture: async () => undefined,
|
||||
readNetworkCapture: async () => entries,
|
||||
evaluate: async () => {
|
||||
evaluateAttempts += 1;
|
||||
if (evaluateAttempts === 1) {
|
||||
throw new TypeError('fetch failed');
|
||||
}
|
||||
return {
|
||||
appId: '936619743392459',
|
||||
csrfToken: 'csrf-from-html',
|
||||
instagramAjax: '1036523242',
|
||||
};
|
||||
},
|
||||
} as any;
|
||||
|
||||
await expect(resolveInstagramPrivatePublishConfig(page)).resolves.toEqual({
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-from-html',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036523242',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: deriveInstagramJazoest('csrf-from-html'),
|
||||
});
|
||||
expect(evaluateAttempts).toBe(2);
|
||||
});
|
||||
|
||||
it('builds the single-image configure form body', () => {
|
||||
expect(buildConfigureBody({
|
||||
uploadId: '1775134280303',
|
||||
caption: 'hello private route',
|
||||
jazoest: '22047',
|
||||
})).toBe(
|
||||
'archive_only=false&caption=hello+private+route&clips_share_preview_to_feed=1'
|
||||
+ '&disable_comments=0&disable_oa_reuse=false&igtv_share_preview_to_feed=1'
|
||||
+ '&is_meta_only_post=0&is_unified_video=1&like_and_view_counts_disabled=0'
|
||||
+ '&media_share_flow=creation_flow&share_to_facebook=&share_to_fb_destination_type=USER'
|
||||
+ '&source_type=library&upload_id=1775134280303&video_subtitles_enabled=0&jazoest=22047'
|
||||
);
|
||||
});
|
||||
|
||||
it('builds the carousel configure_sidecar JSON payload', () => {
|
||||
expect(buildConfigureSidecarPayload({
|
||||
uploadIds: ['1', '3', '2'],
|
||||
caption: 'hello carousel',
|
||||
clientSidecarId: '1775134574348',
|
||||
jazoest: '22047',
|
||||
})).toEqual({
|
||||
archive_only: false,
|
||||
caption: 'hello carousel',
|
||||
children_metadata: [
|
||||
{ upload_id: '1' },
|
||||
{ upload_id: '3' },
|
||||
{ upload_id: '2' },
|
||||
],
|
||||
client_sidecar_id: '1775134574348',
|
||||
disable_comments: '0',
|
||||
is_meta_only_post: false,
|
||||
is_open_to_public_submission: false,
|
||||
like_and_view_counts_disabled: 0,
|
||||
media_share_flow: 'creation_flow',
|
||||
share_to_facebook: '',
|
||||
share_to_fb_destination_type: 'USER',
|
||||
source_type: 'library',
|
||||
jazoest: '22047',
|
||||
});
|
||||
});
|
||||
|
||||
it('reads png and jpeg image assets with mime type and dimensions', () => {
|
||||
const png = createTempFile('sample.png', Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000000030000000508060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
));
|
||||
const jpeg = createTempFile('sample.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
|
||||
expect(readImageAsset(png)).toMatchObject({
|
||||
mimeType: 'image/png',
|
||||
width: 3,
|
||||
height: 5,
|
||||
});
|
||||
expect(readImageAsset(jpeg)).toMatchObject({
|
||||
mimeType: 'image/jpeg',
|
||||
width: 6,
|
||||
height: 4,
|
||||
});
|
||||
});
|
||||
|
||||
it('computes feed-safe aspect-ratio normalization targets', () => {
|
||||
expect(isInstagramFeedAspectRatioAllowed(1080, 1350)).toBe(true);
|
||||
expect(isInstagramFeedAspectRatioAllowed(1179, 2556)).toBe(false);
|
||||
expect(getInstagramFeedNormalizedDimensions(1179, 2556)).toEqual({
|
||||
width: 2045,
|
||||
height: 2556,
|
||||
});
|
||||
expect(getInstagramFeedNormalizedDimensions(2120, 1140)).toBeNull();
|
||||
});
|
||||
|
||||
it('computes story-safe aspect-ratio normalization targets', () => {
|
||||
expect(isInstagramStoryAspectRatioAllowed(1080, 1920)).toBe(true);
|
||||
expect(isInstagramStoryAspectRatioAllowed(1080, 1080)).toBe(false);
|
||||
expect(getInstagramStoryNormalizedDimensions(1080, 1080)).toEqual({
|
||||
width: 1080,
|
||||
height: 1440,
|
||||
});
|
||||
});
|
||||
|
||||
it('builds the single-photo configure_to_story payload', () => {
|
||||
expect(buildConfigureToStoryPhotoPayload({
|
||||
uploadId: '1775134280303',
|
||||
width: 1080,
|
||||
height: 1920,
|
||||
now: () => 1_775_134_280_303,
|
||||
jazoest: '22047',
|
||||
})).toMatchObject({
|
||||
source_type: '4',
|
||||
upload_id: '1775134280303',
|
||||
configure_mode: 1,
|
||||
edits: {
|
||||
crop_original_size: [1080, 1920],
|
||||
crop_center: [0, 0],
|
||||
crop_zoom: 1.3333334,
|
||||
},
|
||||
extra: {
|
||||
source_width: 1080,
|
||||
source_height: 1920,
|
||||
},
|
||||
jazoest: '22047',
|
||||
});
|
||||
});
|
||||
|
||||
it('builds the single-video configure_to_story payload', () => {
|
||||
expect(buildConfigureToStoryVideoPayload({
|
||||
uploadId: '1775134280303',
|
||||
width: 1080,
|
||||
height: 1920,
|
||||
durationMs: 12500,
|
||||
now: () => 1_775_134_280_303,
|
||||
jazoest: '22047',
|
||||
})).toMatchObject({
|
||||
source_type: '4',
|
||||
upload_id: '1775134280303',
|
||||
configure_mode: 1,
|
||||
poster_frame_index: 0,
|
||||
length: 12.5,
|
||||
extra: {
|
||||
source_width: 1080,
|
||||
source_height: 1920,
|
||||
},
|
||||
jazoest: '22047',
|
||||
});
|
||||
});
|
||||
|
||||
it('publishes a single image through rupload + configure', async () => {
|
||||
const jpeg = createTempFile('private-single.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const calls: Array<{ url: string; init?: { method?: string; headers?: Record<string, string>; body?: unknown } }> = [];
|
||||
const fetcher = async (url: string | URL, init?: { method?: string; headers?: Record<string, string>; body?: unknown }) => {
|
||||
calls.push({ url: String(url), init });
|
||||
if (String(url).includes('/rupload_igphoto/')) {
|
||||
return new Response('{"upload_id":"111","status":"ok"}', { status: 200 });
|
||||
}
|
||||
return new Response('{"media":{"code":"ABC123"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const response = await publishImagesViaPrivateApi({
|
||||
page: {} as never,
|
||||
imagePaths: [jpeg],
|
||||
caption: 'private single',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 111,
|
||||
fetcher,
|
||||
});
|
||||
|
||||
expect(calls).toHaveLength(2);
|
||||
expect(calls[0]?.url).toContain('https://i.instagram.com/rupload_igphoto/fb_uploader_111');
|
||||
expect(calls[0]?.init?.headers).toMatchObject({
|
||||
'Content-Type': 'image/jpeg',
|
||||
'X-Entity-Length': String(fs.statSync(jpeg).size),
|
||||
'X-Entity-Name': 'fb_uploader_111',
|
||||
'X-IG-App-ID': '936619743392459',
|
||||
});
|
||||
expect(calls[1]?.url).toBe('https://www.instagram.com/api/v1/media/configure/');
|
||||
expect(String(calls[1]?.init?.body || '')).toContain('upload_id=111');
|
||||
expect(response).toEqual({ code: 'ABC123', uploadIds: ['111'] });
|
||||
});
|
||||
|
||||
it('publishes a single image story through rupload + configure_to_story', async () => {
|
||||
const jpeg = createTempFile('private-story.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080010000903012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const calls: Array<{ url: string; init?: { method?: string; headers?: Record<string, string>; body?: unknown } }> = [];
|
||||
const fetcher = async (url: string | URL, init?: { method?: string; headers?: Record<string, string>; body?: unknown }) => {
|
||||
calls.push({ url: String(url), init });
|
||||
if (String(url).includes('/rupload_igphoto/')) {
|
||||
return new Response('{"upload_id":"111","status":"ok"}', { status: 200 });
|
||||
}
|
||||
return new Response('{"media":{"pk":"1234567890"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const response = await publishStoryViaPrivateApi({
|
||||
page: {} as never,
|
||||
mediaItem: { type: 'image', filePath: jpeg },
|
||||
content: '',
|
||||
currentUserId: '61236465677',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 111,
|
||||
fetcher,
|
||||
prepareMediaAsset: async () => ({
|
||||
type: 'image',
|
||||
asset: {
|
||||
filePath: jpeg,
|
||||
fileName: path.basename(jpeg),
|
||||
mimeType: 'image/jpeg',
|
||||
width: 1080,
|
||||
height: 1920,
|
||||
byteLength: fs.statSync(jpeg).size,
|
||||
bytes: fs.readFileSync(jpeg),
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
expect(calls).toHaveLength(2);
|
||||
expect(calls[0]?.url).toContain('/rupload_igphoto/fb_uploader_111');
|
||||
expect(calls[1]?.url).toBe('https://i.instagram.com/api/v1/media/configure_to_story/');
|
||||
expect(String(calls[1]?.init?.body || '')).toContain('signed_body=');
|
||||
expect(response).toEqual({ mediaPk: '1234567890', uploadId: '111' });
|
||||
});
|
||||
|
||||
it('publishes a single video story through rupload + cover + configure_to_story?video=1', async () => {
|
||||
const video = createTempFile('private-story.mp4', Buffer.from('story-video'));
|
||||
const coverBytes = Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080010000903012200021101031101FFD9',
|
||||
'hex',
|
||||
);
|
||||
const calls: Array<{ url: string; init?: { method?: string; headers?: Record<string, string>; body?: unknown } }> = [];
|
||||
const fetcher = async (url: string | URL, init?: { method?: string; headers?: Record<string, string>; body?: unknown }) => {
|
||||
calls.push({ url: String(url), init });
|
||||
if (String(url).includes('/rupload_igvideo/')) {
|
||||
return new Response('{"upload_id":"222","status":"ok"}', { status: 200 });
|
||||
}
|
||||
if (String(url).includes('/rupload_igphoto/')) {
|
||||
return new Response('{"upload_id":"222","status":"ok"}', { status: 200 });
|
||||
}
|
||||
return new Response('{"media":{"pk":"9988776655"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const response = await publishStoryViaPrivateApi({
|
||||
page: {} as never,
|
||||
mediaItem: { type: 'video', filePath: video },
|
||||
content: '',
|
||||
currentUserId: '61236465677',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 222,
|
||||
fetcher,
|
||||
prepareMediaAsset: async () => ({
|
||||
type: 'video',
|
||||
asset: {
|
||||
filePath: video,
|
||||
fileName: path.basename(video),
|
||||
mimeType: 'video/mp4',
|
||||
width: 1080,
|
||||
height: 1920,
|
||||
durationMs: 12500,
|
||||
byteLength: fs.statSync(video).size,
|
||||
bytes: fs.readFileSync(video),
|
||||
coverImage: {
|
||||
filePath: '/tmp/cover.jpg',
|
||||
fileName: 'cover.jpg',
|
||||
mimeType: 'image/jpeg',
|
||||
width: 1080,
|
||||
height: 1920,
|
||||
byteLength: coverBytes.length,
|
||||
bytes: coverBytes,
|
||||
},
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
expect(calls).toHaveLength(4);
|
||||
expect(calls[0]?.url).toContain('/rupload_igvideo/fb_uploader_222');
|
||||
expect(calls[1]?.url).toContain('/rupload_igphoto/fb_uploader_222');
|
||||
expect(calls[2]?.url).toBe('https://i.instagram.com/api/v1/media/configure_to_story/');
|
||||
expect(calls[3]?.url).toBe('https://i.instagram.com/api/v1/media/configure_to_story/?video=1');
|
||||
expect(String(calls[2]?.init?.body || '')).toContain('signed_body=');
|
||||
expect(String(calls[3]?.init?.body || '')).toContain('signed_body=');
|
||||
expect(response).toEqual({ mediaPk: '9988776655', uploadId: '222' });
|
||||
});
|
||||
|
||||
it('publishes a carousel through rupload + configure_sidecar', async () => {
|
||||
const first = createTempFile('private-carousel-1.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const second = createTempFile('private-carousel-2.png', Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000000030000000508060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
));
|
||||
const calls: Array<{ url: string; init?: { method?: string; headers?: Record<string, string>; body?: unknown } }> = [];
|
||||
let uploadCounter = 0;
|
||||
const fetcher = async (url: string | URL, init?: { method?: string; headers?: Record<string, string>; body?: unknown }) => {
|
||||
calls.push({ url: String(url), init });
|
||||
if (String(url).includes('/rupload_igphoto/')) {
|
||||
uploadCounter += 1;
|
||||
return new Response(JSON.stringify({ upload_id: String(200 + uploadCounter), status: 'ok' }), { status: 200 });
|
||||
}
|
||||
return new Response('{"media":{"code":"SIDE123"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const response = await publishImagesViaPrivateApi({
|
||||
page: {} as never,
|
||||
imagePaths: [first, second],
|
||||
caption: 'private carousel',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 200,
|
||||
fetcher,
|
||||
prepareAsset: async (filePath) => readImageAsset(filePath),
|
||||
});
|
||||
|
||||
expect(calls).toHaveLength(3);
|
||||
expect(calls[2]?.url).toBe('https://www.instagram.com/api/v1/media/configure_sidecar/');
|
||||
expect(JSON.parse(String(calls[2]?.init?.body || '{}'))).toMatchObject({
|
||||
caption: 'private carousel',
|
||||
client_sidecar_id: '200',
|
||||
children_metadata: [{ upload_id: '201' }, { upload_id: '202' }],
|
||||
});
|
||||
expect(response).toEqual({ code: 'SIDE123', uploadIds: ['201', '202'] });
|
||||
});
|
||||
|
||||
it('uses prepared assets when private carousel upload needs aspect-ratio normalization', async () => {
|
||||
const first = createTempFile('private-carousel-normalize-1.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const second = createTempFile('private-carousel-normalize-2.png', Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000000030000000508060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
));
|
||||
const calls: Array<{ url: string; init?: { method?: string; headers?: Record<string, string>; body?: unknown } }> = [];
|
||||
let uploadCounter = 0;
|
||||
const fetcher = async (url: string | URL, init?: { method?: string; headers?: Record<string, string>; body?: unknown }) => {
|
||||
calls.push({ url: String(url), init });
|
||||
if (String(url).includes('/rupload_igphoto/')) {
|
||||
uploadCounter += 1;
|
||||
return new Response(JSON.stringify({ upload_id: String(400 + uploadCounter), status: 'ok' }), { status: 200 });
|
||||
}
|
||||
return new Response('{"media":{"code":"SIDEPAD"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const preparedBytes = Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000007FD000009FC08060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
);
|
||||
|
||||
const response = await publishImagesViaPrivateApi({
|
||||
page: {} as never,
|
||||
imagePaths: [first, second],
|
||||
caption: 'private carousel normalized',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 400,
|
||||
fetcher,
|
||||
prepareAsset: async (filePath) => {
|
||||
if (filePath === second) {
|
||||
return {
|
||||
filePath: '/tmp/normalized.png',
|
||||
fileName: 'normalized.png',
|
||||
mimeType: 'image/png',
|
||||
width: 2045,
|
||||
height: 2556,
|
||||
byteLength: preparedBytes.length,
|
||||
bytes: preparedBytes,
|
||||
cleanupPath: '/tmp/normalized.png',
|
||||
};
|
||||
}
|
||||
return readImageAsset(filePath);
|
||||
},
|
||||
});
|
||||
|
||||
const secondUploadHeaders = calls[1]?.init?.headers ?? {};
|
||||
expect(JSON.parse(String(secondUploadHeaders['X-Instagram-Rupload-Params'] || '{}'))).toMatchObject({
|
||||
upload_media_width: 2045,
|
||||
upload_media_height: 2556,
|
||||
});
|
||||
expect(response).toEqual({ code: 'SIDEPAD', uploadIds: ['401', '402'] });
|
||||
});
|
||||
|
||||
it('includes the response body when configure_sidecar returns a 400', async () => {
|
||||
const first = createTempFile('private-carousel-error-1.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const second = createTempFile('private-carousel-error-2.png', Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000000030000000508060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
));
|
||||
let uploadCounter = 0;
|
||||
const fetcher = async (url: string | URL) => {
|
||||
if (String(url).includes('/rupload_igphoto/')) {
|
||||
uploadCounter += 1;
|
||||
return new Response(JSON.stringify({ upload_id: String(300 + uploadCounter), status: 'ok' }), { status: 200 });
|
||||
}
|
||||
return new Response('{"message":"children_metadata invalid"}', { status: 400 });
|
||||
};
|
||||
|
||||
await expect(publishImagesViaPrivateApi({
|
||||
page: {} as never,
|
||||
imagePaths: [first, second],
|
||||
caption: 'private carousel',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 300,
|
||||
fetcher,
|
||||
prepareAsset: async (filePath) => readImageAsset(filePath),
|
||||
})).rejects.toThrow('children_metadata invalid');
|
||||
});
|
||||
|
||||
it('retries transient rupload fetch failures and still completes the carousel publish', async () => {
|
||||
const first = createTempFile('private-carousel-retry-1.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const second = createTempFile('private-carousel-retry-2.png', Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000000030000000508060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
));
|
||||
const calls: string[] = [];
|
||||
let firstUploadAttempts = 0;
|
||||
let uploadCounter = 0;
|
||||
const fetcher = async (url: string | URL) => {
|
||||
const value = String(url);
|
||||
calls.push(value);
|
||||
if (value.includes('/rupload_igphoto/')) {
|
||||
firstUploadAttempts += value.includes('fb_uploader_501') ? 1 : 0;
|
||||
if (value.includes('fb_uploader_501') && firstUploadAttempts === 1) {
|
||||
throw new TypeError('fetch failed');
|
||||
}
|
||||
uploadCounter += 1;
|
||||
return new Response(JSON.stringify({ upload_id: String(500 + uploadCounter), status: 'ok' }), { status: 200 });
|
||||
}
|
||||
return new Response('{"media":{"code":"SIDERETRY"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const response = await publishImagesViaPrivateApi({
|
||||
page: {} as never,
|
||||
imagePaths: [first, second],
|
||||
caption: 'private carousel retry',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 500,
|
||||
fetcher,
|
||||
prepareAsset: async (filePath) => readImageAsset(filePath),
|
||||
});
|
||||
|
||||
expect(calls.filter((url) => url.includes('fb_uploader_501'))).toHaveLength(2);
|
||||
expect(response).toEqual({ code: 'SIDERETRY', uploadIds: ['501', '502'] });
|
||||
});
|
||||
|
||||
it('does not retry transient configure_sidecar fetch failures to avoid duplicate posts', async () => {
|
||||
const first = createTempFile('private-carousel-no-retry-1.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const second = createTempFile('private-carousel-no-retry-2.png', Buffer.from(
|
||||
'89504E470D0A1A0A0000000D49484452000000030000000508060000008D6F26E50000000049454E44AE426082',
|
||||
'hex',
|
||||
));
|
||||
const calls: string[] = [];
|
||||
let uploadCounter = 0;
|
||||
const fetcher = async (url: string | URL) => {
|
||||
const value = String(url);
|
||||
calls.push(value);
|
||||
if (value.includes('/rupload_igphoto/')) {
|
||||
uploadCounter += 1;
|
||||
return new Response(JSON.stringify({ upload_id: String(600 + uploadCounter), status: 'ok' }), { status: 200 });
|
||||
}
|
||||
throw new TypeError('fetch failed');
|
||||
};
|
||||
|
||||
await expect(publishImagesViaPrivateApi({
|
||||
page: {} as never,
|
||||
imagePaths: [first, second],
|
||||
caption: 'private no retry configure',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 600,
|
||||
fetcher,
|
||||
prepareAsset: async (filePath) => readImageAsset(filePath),
|
||||
})).rejects.toThrow('fetch failed');
|
||||
|
||||
expect(calls.filter((url) => url.includes('configure_sidecar'))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('publishes a mixed image/video carousel and polls configure_sidecar until transcoding finishes', async () => {
|
||||
const image = createTempFile('mixed-private-image.jpg', Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080004000603012200021101031101FFD9',
|
||||
'hex',
|
||||
));
|
||||
const video = createTempFile('mixed-private-video.mp4', Buffer.from('video-binary'));
|
||||
const coverBytes = Buffer.from(
|
||||
'FFD8FFE000104A46494600010100000100010000FFC00011080168028003012200021101031101FFD9',
|
||||
'hex',
|
||||
);
|
||||
const calls: Array<{ url: string; init?: { method?: string; headers?: Record<string, string>; body?: unknown } }> = [];
|
||||
let configureAttempts = 0;
|
||||
const fetcher = async (url: string | URL, init?: { method?: string; headers?: Record<string, string>; body?: unknown }) => {
|
||||
const value = String(url);
|
||||
calls.push({ url: value, init });
|
||||
if (value.includes('/rupload_igphoto/') && value.includes('fb_uploader_701')) {
|
||||
return new Response('{"upload_id":"701","status":"ok"}', { status: 200 });
|
||||
}
|
||||
if (value.includes('/rupload_igvideo/') && value.includes('fb_uploader_702')) {
|
||||
return new Response('{"media_id":17944674009157009,"status":"ok"}', { status: 200 });
|
||||
}
|
||||
if (value.includes('/rupload_igphoto/') && value.includes('fb_uploader_702')) {
|
||||
return new Response('{"upload_id":"702","status":"ok"}', { status: 200 });
|
||||
}
|
||||
configureAttempts += 1;
|
||||
if (configureAttempts === 1) {
|
||||
return new Response('{"message":"Transcode not finished yet.","status":"fail"}', { status: 202 });
|
||||
}
|
||||
return new Response('{"status":"ok","media":{"code":"MIXEDSIDE123"}}', { status: 200 });
|
||||
};
|
||||
|
||||
const response = await publishMediaViaPrivateApi({
|
||||
page: {} as never,
|
||||
mediaItems: [
|
||||
{ type: 'image', filePath: image },
|
||||
{ type: 'video', filePath: video },
|
||||
],
|
||||
caption: 'mixed private carousel',
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'hmac.claim',
|
||||
instagramAjax: '1036517563',
|
||||
webSessionId: 'abc:def:ghi',
|
||||
},
|
||||
jazoest: '22047',
|
||||
now: () => 700,
|
||||
fetcher,
|
||||
prepareMediaAsset: async (item) => {
|
||||
if (item.type === 'image') {
|
||||
return {
|
||||
type: 'image' as const,
|
||||
asset: readImageAsset(item.filePath),
|
||||
};
|
||||
}
|
||||
return {
|
||||
type: 'video' as const,
|
||||
asset: {
|
||||
filePath: item.filePath,
|
||||
fileName: 'mixed-private-video.mp4',
|
||||
mimeType: 'video/mp4',
|
||||
width: 640,
|
||||
height: 360,
|
||||
durationMs: 28245,
|
||||
byteLength: 12,
|
||||
bytes: Buffer.from('video-binary'),
|
||||
coverImage: {
|
||||
filePath: '/tmp/mixed-private-cover.jpg',
|
||||
fileName: 'mixed-private-cover.jpg',
|
||||
mimeType: 'image/jpeg',
|
||||
width: 640,
|
||||
height: 360,
|
||||
byteLength: coverBytes.length,
|
||||
bytes: coverBytes,
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
waitMs: async () => undefined,
|
||||
});
|
||||
|
||||
expect(calls).toHaveLength(5);
|
||||
expect(calls[0]?.url).toContain('/rupload_igphoto/fb_uploader_701');
|
||||
expect(calls[1]?.url).toContain('/rupload_igvideo/fb_uploader_702');
|
||||
expect(calls[2]?.url).toContain('/rupload_igphoto/fb_uploader_702');
|
||||
expect(JSON.parse(String(calls[1]?.init?.headers?.['X-Instagram-Rupload-Params'] || '{}'))).toMatchObject({
|
||||
media_type: 2,
|
||||
upload_id: '702',
|
||||
upload_media_width: 640,
|
||||
upload_media_height: 360,
|
||||
upload_media_duration_ms: 28245,
|
||||
video_edit_params: {
|
||||
crop_width: 360,
|
||||
crop_height: 360,
|
||||
crop_x1: 140,
|
||||
crop_y1: 0,
|
||||
trim_start: 0,
|
||||
trim_end: 28.245,
|
||||
mute: false,
|
||||
},
|
||||
});
|
||||
expect(JSON.parse(String(calls[2]?.init?.headers?.['X-Instagram-Rupload-Params'] || '{}'))).toMatchObject({
|
||||
media_type: 2,
|
||||
upload_id: '702',
|
||||
upload_media_width: 640,
|
||||
upload_media_height: 360,
|
||||
});
|
||||
expect(JSON.parse(String(calls[3]?.init?.body || '{}'))).toMatchObject({
|
||||
caption: 'mixed private carousel',
|
||||
client_sidecar_id: '700',
|
||||
children_metadata: [{ upload_id: '701' }, { upload_id: '702' }],
|
||||
});
|
||||
expect(response).toEqual({ code: 'MIXEDSIDE123', uploadIds: ['701', '702'] });
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,148 @@
|
||||
import * as fs from 'node:fs';
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import type { BrowserCookie, IPage } from '../../../types.js';
|
||||
import {
|
||||
buildInstallInstagramProtocolCaptureJs,
|
||||
buildReadInstagramProtocolCaptureJs,
|
||||
dumpInstagramProtocolCaptureIfEnabled,
|
||||
instagramPrivateApiFetch,
|
||||
installInstagramProtocolCapture,
|
||||
readInstagramProtocolCapture,
|
||||
} from './protocol-capture.js';
|
||||
|
||||
describe('instagram protocol capture helpers', () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
delete process.env.OPENCLI_INSTAGRAM_CAPTURE;
|
||||
try { fs.rmSync('/tmp/instagram_post_protocol_trace.json', { force: true }); } catch {}
|
||||
});
|
||||
|
||||
it('installs the protocol capture patch in page context', async () => {
|
||||
const evaluate = vi.fn().mockResolvedValue({ ok: true });
|
||||
const page = { evaluate } as unknown as IPage;
|
||||
|
||||
await installInstagramProtocolCapture(page);
|
||||
|
||||
expect(evaluate).toHaveBeenCalledTimes(1);
|
||||
expect(String(evaluate.mock.calls[0]?.[0] || '')).toContain('__opencli_ig_protocol_capture');
|
||||
expect(String(evaluate.mock.calls[0]?.[0] || '')).toContain('/media/configure_sidecar/');
|
||||
});
|
||||
|
||||
it('prefers native page network capture when available', async () => {
|
||||
const startNetworkCapture = vi.fn().mockResolvedValue(undefined);
|
||||
const evaluate = vi.fn();
|
||||
const page = { startNetworkCapture, evaluate } as unknown as IPage;
|
||||
|
||||
await installInstagramProtocolCapture(page);
|
||||
|
||||
expect(startNetworkCapture).toHaveBeenCalledTimes(1);
|
||||
expect(evaluate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('reads and normalizes captured protocol entries', async () => {
|
||||
const evaluate = vi.fn().mockResolvedValue({
|
||||
data: [{ kind: 'fetch', url: 'https://www.instagram.com/api/v1/media/configure/' }],
|
||||
errors: ['ignored'],
|
||||
});
|
||||
const page = { evaluate } as unknown as IPage;
|
||||
|
||||
const result = await readInstagramProtocolCapture(page);
|
||||
|
||||
expect(String(evaluate.mock.calls[0]?.[0] || '')).toContain('__opencli_ig_protocol_capture');
|
||||
expect(result).toEqual({
|
||||
data: [{ kind: 'fetch', url: 'https://www.instagram.com/api/v1/media/configure/' }],
|
||||
errors: ['ignored'],
|
||||
});
|
||||
});
|
||||
|
||||
it('prefers native page network capture reads when available', async () => {
|
||||
const readNetworkCapture = vi.fn().mockResolvedValue([
|
||||
{ kind: 'cdp', url: 'https://www.instagram.com/rupload_igphoto/test', method: 'POST' },
|
||||
]);
|
||||
const evaluate = vi.fn();
|
||||
const page = { readNetworkCapture, evaluate } as unknown as IPage;
|
||||
|
||||
const result = await readInstagramProtocolCapture(page);
|
||||
|
||||
expect(readNetworkCapture).toHaveBeenCalledTimes(1);
|
||||
expect(evaluate).not.toHaveBeenCalled();
|
||||
expect(result).toEqual({
|
||||
data: [{ kind: 'cdp', url: 'https://www.instagram.com/rupload_igphoto/test', method: 'POST' }],
|
||||
errors: [],
|
||||
});
|
||||
});
|
||||
|
||||
it('dumps protocol traces to /tmp only when capture env is enabled', async () => {
|
||||
process.env.OPENCLI_INSTAGRAM_CAPTURE = '1';
|
||||
const page = {
|
||||
evaluate: vi.fn().mockResolvedValue({
|
||||
data: [{ kind: 'fetch', url: 'https://www.instagram.com/rupload_igphoto/test' }],
|
||||
errors: [],
|
||||
}),
|
||||
} as unknown as IPage;
|
||||
|
||||
await dumpInstagramProtocolCaptureIfEnabled(page);
|
||||
|
||||
const raw = fs.readFileSync('/tmp/instagram_post_protocol_trace.json', 'utf8');
|
||||
expect(raw).toContain('rupload_igphoto');
|
||||
});
|
||||
|
||||
it('does not dump protocol traces when capture env is disabled', async () => {
|
||||
const page = {
|
||||
evaluate: vi.fn(),
|
||||
} as unknown as IPage;
|
||||
|
||||
await dumpInstagramProtocolCaptureIfEnabled(page);
|
||||
|
||||
expect(page.evaluate).not.toHaveBeenCalled();
|
||||
expect(fs.existsSync('/tmp/instagram_post_protocol_trace.json')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('instagram private api fetch', () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('uses browser cookies to build instagram private api requests', async () => {
|
||||
const getCookies = vi.fn()
|
||||
.mockResolvedValueOnce([{ name: 'sessionid', value: 'sess', domain: '.instagram.com' } satisfies BrowserCookie])
|
||||
.mockResolvedValueOnce([
|
||||
{ name: 'csrftoken', value: 'csrf', domain: '.instagram.com' } satisfies BrowserCookie,
|
||||
{ name: 'sessionid', value: 'sess', domain: '.instagram.com' } satisfies BrowserCookie,
|
||||
]);
|
||||
const evaluate = vi.fn().mockResolvedValue({
|
||||
appId: 'dynamic-app-id',
|
||||
csrfToken: 'csrf',
|
||||
instagramAjax: 'dynamic-rollout',
|
||||
});
|
||||
const page = { getCookies, evaluate } as unknown as IPage;
|
||||
const fetchMock = vi.fn().mockResolvedValue(new Response('{}', { status: 200 }));
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
await instagramPrivateApiFetch(page, 'https://www.instagram.com/api/v1/media/configure/', {
|
||||
method: 'POST',
|
||||
body: 'caption=test',
|
||||
});
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
'https://www.instagram.com/api/v1/media/configure/',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
headers: expect.objectContaining({
|
||||
'X-CSRFToken': 'csrf',
|
||||
'X-IG-App-ID': 'dynamic-app-id',
|
||||
'Cookie': expect.stringContaining('sessionid=sess'),
|
||||
}),
|
||||
body: 'caption=test',
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('exposes stable browser-side JS builders', () => {
|
||||
expect(buildInstallInstagramProtocolCaptureJs()).toContain('/rupload_igphoto/');
|
||||
expect(buildReadInstagramProtocolCaptureJs()).toContain('__opencli_ig_protocol_capture');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,321 @@
|
||||
import * as fs from 'node:fs';
|
||||
|
||||
import type { BrowserCookie, IPage } from '../../../types.js';
|
||||
import { resolveInstagramRuntimeInfo } from './runtime-info.js';
|
||||
|
||||
const DEFAULT_CAPTURE_VAR = '__opencli_ig_protocol_capture';
|
||||
const DEFAULT_CAPTURE_ERRORS_VAR = '__opencli_ig_protocol_capture_errors';
|
||||
const TRACE_OUTPUT_PATH = '/tmp/instagram_post_protocol_trace.json';
|
||||
const INSTAGRAM_PROTOCOL_CAPTURE_PATTERN = [
|
||||
'/rupload_igphoto/',
|
||||
'/rupload_igvideo/',
|
||||
'/api/v1/',
|
||||
'/media/configure/',
|
||||
'/media/configure_sidecar/',
|
||||
'/media/configure_to_story/',
|
||||
'/api/graphql/',
|
||||
].join('|');
|
||||
|
||||
export interface InstagramProtocolCaptureEntry {
|
||||
kind: 'fetch' | 'xhr';
|
||||
url: string;
|
||||
method: string;
|
||||
requestHeaders?: Record<string, string>;
|
||||
requestBodyKind?: string;
|
||||
requestBodyPreview?: string;
|
||||
responseStatus?: number;
|
||||
responseContentType?: string;
|
||||
responsePreview?: string;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
export function buildInstallInstagramProtocolCaptureJs(
|
||||
captureVar: string = DEFAULT_CAPTURE_VAR,
|
||||
captureErrorsVar: string = DEFAULT_CAPTURE_ERRORS_VAR,
|
||||
): string {
|
||||
return `
|
||||
(() => {
|
||||
const CAPTURE_VAR = ${JSON.stringify(captureVar)};
|
||||
const CAPTURE_ERRORS_VAR = ${JSON.stringify(captureErrorsVar)};
|
||||
const PATCH_GUARD = CAPTURE_VAR + '_patched';
|
||||
const FILTERS = [
|
||||
'/rupload_igphoto/',
|
||||
'/rupload_igvideo/',
|
||||
'/api/v1/',
|
||||
'/media/configure/',
|
||||
'/media/configure_sidecar/',
|
||||
'/media/configure_to_story/',
|
||||
'/api/graphql/',
|
||||
];
|
||||
|
||||
const shouldCapture = (url) => {
|
||||
const value = String(url || '');
|
||||
return FILTERS.some((filter) => value.includes(filter));
|
||||
};
|
||||
|
||||
const normalizeHeaders = (headersLike) => {
|
||||
const out = {};
|
||||
try {
|
||||
if (!headersLike) return out;
|
||||
if (headersLike instanceof Headers) {
|
||||
headersLike.forEach((value, key) => { out[key] = value; });
|
||||
return out;
|
||||
}
|
||||
if (Array.isArray(headersLike)) {
|
||||
for (const pair of headersLike) {
|
||||
if (Array.isArray(pair) && pair.length >= 2) out[String(pair[0])] = String(pair[1]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
if (typeof headersLike === 'object') {
|
||||
for (const [key, value] of Object.entries(headersLike)) out[key] = String(value);
|
||||
}
|
||||
} catch {}
|
||||
return out;
|
||||
};
|
||||
|
||||
const summarizeBody = async (body) => {
|
||||
if (body == null) return { kind: 'empty', preview: '' };
|
||||
try {
|
||||
if (typeof body === 'string') {
|
||||
return { kind: 'string', preview: body.slice(0, 1000) };
|
||||
}
|
||||
if (body instanceof URLSearchParams) {
|
||||
return { kind: 'urlencoded', preview: body.toString().slice(0, 1000) };
|
||||
}
|
||||
if (body instanceof FormData) {
|
||||
const parts = [];
|
||||
for (const [key, value] of body.entries()) {
|
||||
if (value instanceof File) {
|
||||
parts.push(key + '=File(' + value.name + ',' + value.type + ',' + value.size + ')');
|
||||
} else {
|
||||
parts.push(key + '=' + String(value));
|
||||
}
|
||||
}
|
||||
return { kind: 'formdata', preview: parts.join('&').slice(0, 2000) };
|
||||
}
|
||||
if (body instanceof Blob) {
|
||||
return { kind: 'blob', preview: 'Blob(' + body.type + ',' + body.size + ')' };
|
||||
}
|
||||
if (body instanceof ArrayBuffer) {
|
||||
return { kind: 'arraybuffer', preview: 'ArrayBuffer(' + body.byteLength + ')' };
|
||||
}
|
||||
if (ArrayBuffer.isView(body)) {
|
||||
return { kind: 'typed-array', preview: body.constructor.name + '(' + body.byteLength + ')' };
|
||||
}
|
||||
return { kind: typeof body, preview: String(body).slice(0, 1000) };
|
||||
} catch (error) {
|
||||
return { kind: 'unknown', preview: 'body-preview-error:' + String(error) };
|
||||
}
|
||||
};
|
||||
|
||||
const capture = async (kind, url, method, headers, body, response) => {
|
||||
if (!shouldCapture(url)) return;
|
||||
try {
|
||||
const bodyInfo = await summarizeBody(body);
|
||||
const contentType = response?.headers?.get?.('content-type') || '';
|
||||
let responsePreview = '';
|
||||
try {
|
||||
if (response && typeof response.clone === 'function') {
|
||||
const clone = response.clone();
|
||||
responsePreview = (await clone.text()).slice(0, 4000);
|
||||
}
|
||||
} catch (error) {
|
||||
responsePreview = 'response-preview-error:' + String(error);
|
||||
}
|
||||
window[CAPTURE_VAR].push({
|
||||
kind,
|
||||
url: String(url || ''),
|
||||
method: String(method || 'GET').toUpperCase(),
|
||||
requestHeaders: normalizeHeaders(headers),
|
||||
requestBodyKind: bodyInfo.kind,
|
||||
requestBodyPreview: bodyInfo.preview,
|
||||
responseStatus: response?.status,
|
||||
responseContentType: contentType,
|
||||
responsePreview,
|
||||
timestamp: Date.now(),
|
||||
});
|
||||
} catch (error) {
|
||||
window[CAPTURE_ERRORS_VAR].push(String(error));
|
||||
}
|
||||
};
|
||||
|
||||
if (!Array.isArray(window[CAPTURE_VAR])) window[CAPTURE_VAR] = [];
|
||||
if (!Array.isArray(window[CAPTURE_ERRORS_VAR])) window[CAPTURE_ERRORS_VAR] = [];
|
||||
if (window[PATCH_GUARD]) return { ok: true };
|
||||
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function(...args) {
|
||||
const input = args[0];
|
||||
const init = args[1] || {};
|
||||
const url = typeof input === 'string'
|
||||
? input
|
||||
: input instanceof Request
|
||||
? input.url
|
||||
: String(input || '');
|
||||
const method = init.method || (input instanceof Request ? input.method : 'GET');
|
||||
const headers = init.headers || (input instanceof Request ? input.headers : undefined);
|
||||
const body = init.body || (input instanceof Request ? input.body : undefined);
|
||||
const response = await origFetch.apply(this, args);
|
||||
capture('fetch', url, method, headers, body, response);
|
||||
return response;
|
||||
};
|
||||
|
||||
const origOpen = XMLHttpRequest.prototype.open;
|
||||
const origSend = XMLHttpRequest.prototype.send;
|
||||
const origSetRequestHeader = XMLHttpRequest.prototype.setRequestHeader;
|
||||
|
||||
XMLHttpRequest.prototype.open = function(method, url) {
|
||||
this.__opencli_method = method;
|
||||
this.__opencli_url = url;
|
||||
this.__opencli_headers = {};
|
||||
return origOpen.apply(this, arguments);
|
||||
};
|
||||
XMLHttpRequest.prototype.setRequestHeader = function(name, value) {
|
||||
try {
|
||||
this.__opencli_headers = this.__opencli_headers || {};
|
||||
this.__opencli_headers[String(name)] = String(value);
|
||||
} catch {}
|
||||
return origSetRequestHeader.apply(this, arguments);
|
||||
};
|
||||
XMLHttpRequest.prototype.send = function(body) {
|
||||
this.addEventListener('load', () => {
|
||||
if (!shouldCapture(this.__opencli_url)) return;
|
||||
try {
|
||||
window[CAPTURE_VAR].push({
|
||||
kind: 'xhr',
|
||||
url: String(this.__opencli_url || ''),
|
||||
method: String(this.__opencli_method || 'GET').toUpperCase(),
|
||||
requestHeaders: this.__opencli_headers || {},
|
||||
requestBodyKind: body == null ? 'empty' : (body instanceof FormData ? 'formdata' : typeof body),
|
||||
requestBodyPreview: body == null ? '' : (body instanceof FormData ? '[formdata]' : String(body).slice(0, 2000)),
|
||||
responseStatus: this.status,
|
||||
responseContentType: this.getResponseHeader('content-type') || '',
|
||||
responsePreview: String(this.responseText || '').slice(0, 4000),
|
||||
timestamp: Date.now(),
|
||||
});
|
||||
} catch (error) {
|
||||
window[CAPTURE_ERRORS_VAR].push(String(error));
|
||||
}
|
||||
});
|
||||
return origSend.apply(this, arguments);
|
||||
};
|
||||
|
||||
window[PATCH_GUARD] = true;
|
||||
return { ok: true };
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
export function buildReadInstagramProtocolCaptureJs(
|
||||
captureVar: string = DEFAULT_CAPTURE_VAR,
|
||||
captureErrorsVar: string = DEFAULT_CAPTURE_ERRORS_VAR,
|
||||
): string {
|
||||
return `
|
||||
(() => {
|
||||
const data = Array.isArray(window[${JSON.stringify(captureVar)}]) ? window[${JSON.stringify(captureVar)}] : [];
|
||||
const errors = Array.isArray(window[${JSON.stringify(captureErrorsVar)}]) ? window[${JSON.stringify(captureErrorsVar)}] : [];
|
||||
window[${JSON.stringify(captureVar)}] = [];
|
||||
window[${JSON.stringify(captureErrorsVar)}] = [];
|
||||
return { data, errors };
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
export async function installInstagramProtocolCapture(page: IPage): Promise<void> {
|
||||
if (typeof page.startNetworkCapture === 'function') {
|
||||
try {
|
||||
await page.startNetworkCapture(INSTAGRAM_PROTOCOL_CAPTURE_PATTERN);
|
||||
return;
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (!message.includes('Unknown action') && !message.includes('network-capture')) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
await page.evaluate(buildInstallInstagramProtocolCaptureJs());
|
||||
}
|
||||
|
||||
export async function readInstagramProtocolCapture(page: IPage): Promise<{
|
||||
data: InstagramProtocolCaptureEntry[];
|
||||
errors: string[];
|
||||
}> {
|
||||
if (typeof page.readNetworkCapture === 'function') {
|
||||
try {
|
||||
const data = await page.readNetworkCapture();
|
||||
return {
|
||||
data: Array.isArray(data) ? data as InstagramProtocolCaptureEntry[] : [],
|
||||
errors: [],
|
||||
};
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (!message.includes('Unknown action') && !message.includes('network-capture')) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
const result = await page.evaluate(buildReadInstagramProtocolCaptureJs()) as {
|
||||
data?: InstagramProtocolCaptureEntry[];
|
||||
errors?: string[];
|
||||
};
|
||||
return {
|
||||
data: Array.isArray(result?.data) ? result.data : [],
|
||||
errors: Array.isArray(result?.errors) ? result.errors : [],
|
||||
};
|
||||
}
|
||||
|
||||
export async function dumpInstagramProtocolCaptureIfEnabled(page: IPage): Promise<void> {
|
||||
if (process.env.OPENCLI_INSTAGRAM_CAPTURE !== '1') return;
|
||||
const payload = await readInstagramProtocolCapture(page);
|
||||
fs.writeFileSync(TRACE_OUTPUT_PATH, JSON.stringify(payload, null, 2));
|
||||
}
|
||||
|
||||
function buildCookieHeader(cookies: BrowserCookie[]): string {
|
||||
return cookies
|
||||
.filter((cookie) => cookie?.name && cookie?.value)
|
||||
.map((cookie) => `${cookie.name}=${cookie.value}`)
|
||||
.join('; ');
|
||||
}
|
||||
|
||||
export async function instagramPrivateApiFetch(
|
||||
page: IPage,
|
||||
input: string | URL,
|
||||
init: {
|
||||
method?: 'GET' | 'POST';
|
||||
headers?: Record<string, string>;
|
||||
body?: unknown;
|
||||
} = {},
|
||||
): Promise<Response> {
|
||||
const url = String(input);
|
||||
const [urlCookies, domainCookies] = await Promise.all([
|
||||
page.getCookies({ url }),
|
||||
page.getCookies({ domain: 'instagram.com' }),
|
||||
]);
|
||||
const merged = new Map<string, BrowserCookie>();
|
||||
for (const cookie of domainCookies) merged.set(cookie.name, cookie);
|
||||
for (const cookie of urlCookies) merged.set(cookie.name, cookie);
|
||||
const cookieHeader = buildCookieHeader(Array.from(merged.values()));
|
||||
const csrf = merged.get('csrftoken')?.value || '';
|
||||
const initHeaders = init.headers ?? {};
|
||||
const requestedAppIdHeader = Object.entries(initHeaders).find(([key]) => key.toLowerCase() === 'x-ig-app-id')?.[1] || '';
|
||||
const runtimeInfo = requestedAppIdHeader ? null : await resolveInstagramRuntimeInfo(page);
|
||||
const appId = requestedAppIdHeader || runtimeInfo?.appId || '';
|
||||
const hasContentType = Object.keys(init.headers ?? {}).some((key) => key.toLowerCase() === 'content-type');
|
||||
|
||||
return fetch(url, {
|
||||
method: init.method ?? 'GET',
|
||||
headers: {
|
||||
'Accept': 'application/json, text/plain, */*',
|
||||
'X-CSRFToken': csrf,
|
||||
'X-Requested-With': 'XMLHttpRequest',
|
||||
'Origin': 'https://www.instagram.com',
|
||||
'Referer': 'https://www.instagram.com/',
|
||||
...(appId ? { 'X-IG-App-ID': appId } : {}),
|
||||
...(cookieHeader ? { 'Cookie': cookieHeader } : {}),
|
||||
...(typeof init.body === 'string' && !hasContentType ? { 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8' } : {}),
|
||||
...initHeaders,
|
||||
},
|
||||
...(init.body !== undefined ? { body: init.body as BodyInit } : {}),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
import type { BrowserCookie, IPage } from '../../../types.js';
|
||||
|
||||
export interface InstagramRuntimeInfo {
|
||||
appId: string;
|
||||
csrfToken: string;
|
||||
instagramAjax: string;
|
||||
}
|
||||
|
||||
function pickMatch(input: string, patterns: RegExp[]): string {
|
||||
for (const pattern of patterns) {
|
||||
const match = input.match(pattern);
|
||||
if (!match) continue;
|
||||
for (let index = 1; index < match.length; index += 1) {
|
||||
if (match[index]) return match[index]!;
|
||||
}
|
||||
return match[0] || '';
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
export function extractInstagramRuntimeInfo(html: string): InstagramRuntimeInfo {
|
||||
return {
|
||||
appId: pickMatch(html, [
|
||||
/"X-IG-App-ID":"(\d+)"/,
|
||||
/"appId":"(\d+)"/,
|
||||
/"app_id":"(\d+)"/,
|
||||
/"instagramWebAppId":"(\d+)"/,
|
||||
]),
|
||||
csrfToken: pickMatch(html, [
|
||||
/"csrf_token":"([^"]+)"/,
|
||||
/"csrfToken":"([^"]+)"/,
|
||||
]),
|
||||
instagramAjax: pickMatch(html, [
|
||||
/"rollout_hash":"([^"]+)"/,
|
||||
/"X-Instagram-AJAX":"([^"]+)"/,
|
||||
/"Instagram-AJAX":"([^"]+)"/,
|
||||
]),
|
||||
};
|
||||
}
|
||||
|
||||
export function buildReadInstagramRuntimeInfoJs(): string {
|
||||
return `
|
||||
(() => {
|
||||
const html = document.documentElement?.outerHTML || '';
|
||||
const pick = (patterns) => {
|
||||
for (const pattern of patterns) {
|
||||
const match = html.match(new RegExp(pattern, 'i'));
|
||||
if (!match) continue;
|
||||
for (let index = 1; index < match.length; index += 1) {
|
||||
if (match[index]) return match[index];
|
||||
}
|
||||
return match[0] || '';
|
||||
}
|
||||
return '';
|
||||
};
|
||||
return {
|
||||
appId: pick([
|
||||
'"X-IG-App-ID":"(\\\\d+)"',
|
||||
'"appId":"(\\\\d+)"',
|
||||
'"app_id":"(\\\\d+)"',
|
||||
'"instagramWebAppId":"(\\\\d+)"',
|
||||
]),
|
||||
csrfToken: pick([
|
||||
'"csrf_token":"([^"]+)"',
|
||||
'"csrfToken":"([^"]+)"',
|
||||
]),
|
||||
instagramAjax: pick([
|
||||
'"rollout_hash":"([^"]+)"',
|
||||
'"X-Instagram-AJAX":"([^"]+)"',
|
||||
'"Instagram-AJAX":"([^"]+)"',
|
||||
]),
|
||||
};
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
function getCookieValue(cookies: BrowserCookie[], name: string): string {
|
||||
return cookies.find((cookie) => cookie.name === name)?.value || '';
|
||||
}
|
||||
|
||||
export async function resolveInstagramRuntimeInfo(page: IPage): Promise<InstagramRuntimeInfo> {
|
||||
const [runtime, cookies] = await Promise.all([
|
||||
page.evaluate(buildReadInstagramRuntimeInfoJs()) as Promise<InstagramRuntimeInfo>,
|
||||
page.getCookies({ domain: 'instagram.com' }),
|
||||
]);
|
||||
return {
|
||||
appId: runtime?.appId || '',
|
||||
csrfToken: runtime?.csrfToken || getCookieValue(cookies, 'csrftoken') || '',
|
||||
instagramAjax: runtime?.instagramAjax || '',
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ArgumentError } from '../../errors.js';
|
||||
import { getRegistry } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import './note.js';
|
||||
|
||||
function createPageMock(): IPage {
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate: vi.fn().mockResolvedValue(undefined),
|
||||
getCookies: vi.fn().mockResolvedValue([]),
|
||||
snapshot: vi.fn().mockResolvedValue(undefined),
|
||||
click: vi.fn().mockResolvedValue(undefined),
|
||||
typeText: vi.fn().mockResolvedValue(undefined),
|
||||
pressKey: vi.fn().mockResolvedValue(undefined),
|
||||
scrollTo: vi.fn().mockResolvedValue(undefined),
|
||||
getFormState: vi.fn().mockResolvedValue({ forms: [], orphanFields: [] }),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
tabs: vi.fn().mockResolvedValue([]),
|
||||
closeTab: vi.fn().mockResolvedValue(undefined),
|
||||
newTab: vi.fn().mockResolvedValue(undefined),
|
||||
selectTab: vi.fn().mockResolvedValue(undefined),
|
||||
networkRequests: vi.fn().mockResolvedValue([]),
|
||||
consoleMessages: vi.fn().mockResolvedValue([]),
|
||||
scroll: vi.fn().mockResolvedValue(undefined),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
screenshot: vi.fn().mockResolvedValue(''),
|
||||
setFileInput: vi.fn().mockResolvedValue(undefined),
|
||||
insertText: vi.fn().mockResolvedValue(undefined),
|
||||
getCurrentUrl: vi.fn().mockResolvedValue(null),
|
||||
};
|
||||
}
|
||||
|
||||
describe('instagram note registration', () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('registers the note command with a required positional content arg', () => {
|
||||
const cmd = getRegistry().get('instagram/note');
|
||||
expect(cmd).toBeDefined();
|
||||
expect(cmd?.browser).toBe(true);
|
||||
expect(cmd?.args.some((arg) => arg.name === 'content' && arg.positional && arg.required)).toBe(true);
|
||||
});
|
||||
|
||||
it('rejects missing note content before browser work', async () => {
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/note');
|
||||
|
||||
await expect(cmd!.func!(page, {})).rejects.toThrow(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects blank note content before browser work', async () => {
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/note');
|
||||
|
||||
await expect(cmd!.func!(page, { content: ' ' })).rejects.toThrow(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects note content longer than 60 characters before browser work', async () => {
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/note');
|
||||
|
||||
await expect(cmd!.func!(page, { content: 'x'.repeat(61) })).rejects.toThrow(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('publishes a note through the web inbox mutation', async () => {
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/note');
|
||||
vi.mocked(page.evaluate).mockResolvedValue({
|
||||
ok: true,
|
||||
noteId: '17849203563031468',
|
||||
});
|
||||
|
||||
const rows = await cmd!.func!(page, { content: 'hello note' }) as Array<Record<string, string>>;
|
||||
|
||||
expect(page.goto).toHaveBeenCalledWith('https://www.instagram.com/direct/inbox/');
|
||||
expect(page.evaluate).toHaveBeenCalledTimes(1);
|
||||
expect(rows).toEqual([{
|
||||
status: '✅ Posted',
|
||||
detail: 'Instagram note published successfully',
|
||||
noteId: '17849203563031468',
|
||||
}]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,254 @@
|
||||
import { ArgumentError, CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
|
||||
type InstagramNoteSuccessRow = {
|
||||
status: string;
|
||||
detail: string;
|
||||
noteId: string;
|
||||
};
|
||||
|
||||
type BrowserNoteResult = {
|
||||
ok?: boolean;
|
||||
stage?: string;
|
||||
status?: number;
|
||||
text?: string;
|
||||
noteId?: string;
|
||||
};
|
||||
|
||||
const INSTAGRAM_INBOX_URL = 'https://www.instagram.com/direct/inbox/';
|
||||
const INSTAGRAM_NOTE_DOC_ID = '25155183657506484';
|
||||
const INSTAGRAM_NOTE_MUTATION_NAME = 'usePolarisCreateInboxTrayItemSubmitMutation';
|
||||
const INSTAGRAM_NOTE_ROOT_FIELD = 'xdt_create_inbox_tray_item';
|
||||
|
||||
function requirePage(page: IPage | null): IPage {
|
||||
if (!page) throw new CommandExecutionError('Browser session required for instagram note');
|
||||
return page;
|
||||
}
|
||||
|
||||
function validateInstagramNoteArgs(kwargs: Record<string, unknown>): void {
|
||||
if (kwargs.content === undefined) {
|
||||
throw new ArgumentError(
|
||||
'Argument "content" is required.',
|
||||
'Provide a note text, for example: opencli instagram note "hello"',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeInstagramNoteContent(kwargs: Record<string, unknown>): string {
|
||||
const content = String(kwargs.content ?? '').trim();
|
||||
if (!content) {
|
||||
throw new ArgumentError(
|
||||
'Instagram note content cannot be empty.',
|
||||
'Provide a non-empty note text, for example: opencli instagram note "hello"',
|
||||
);
|
||||
}
|
||||
if (Array.from(content).length > 60) {
|
||||
throw new ArgumentError(
|
||||
'Instagram note content must be 60 characters or fewer.',
|
||||
'Shorten the note text and try again.',
|
||||
);
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
||||
function buildNoteSuccessResult(noteId: string): InstagramNoteSuccessRow[] {
|
||||
return [{
|
||||
status: '✅ Posted',
|
||||
detail: 'Instagram note published successfully',
|
||||
noteId,
|
||||
}];
|
||||
}
|
||||
|
||||
function buildPublishInstagramNoteJs(content: string): string {
|
||||
return `
|
||||
(async () => {
|
||||
const input = ${JSON.stringify({ content })};
|
||||
const html = document.documentElement?.outerHTML || '';
|
||||
const scripts = Array.from(document.scripts || [])
|
||||
.map((script) => script.textContent || '')
|
||||
.join('\\n');
|
||||
const source = html + '\\n' + scripts;
|
||||
const pick = (patterns) => {
|
||||
for (const pattern of patterns) {
|
||||
const match = source.match(pattern);
|
||||
if (!match) continue;
|
||||
for (let index = 1; index < match.length; index += 1) {
|
||||
if (match[index]) return match[index];
|
||||
}
|
||||
return match[0] || '';
|
||||
}
|
||||
return '';
|
||||
};
|
||||
const readCookie = (name) => {
|
||||
const prefix = name + '=';
|
||||
const part = document.cookie
|
||||
.split('; ')
|
||||
.find((cookie) => cookie.startsWith(prefix));
|
||||
return part ? decodeURIComponent(part.slice(prefix.length)) : '';
|
||||
};
|
||||
const actorId = pick([
|
||||
/"actorID":"(\\d+)"/,
|
||||
/"actor_id":"(\\d+)"/,
|
||||
/"viewerId":"(\\d+)"/,
|
||||
]);
|
||||
const fbDtsg = pick([
|
||||
/(NAF[a-zA-Z0-9:_-]{20,})/,
|
||||
/(NAf[a-zA-Z0-9:_-]{20,})/,
|
||||
]);
|
||||
const lsd = pick([
|
||||
/"LSD",\\[\\],\\{"token":"([^"]+)"\\}/,
|
||||
/"lsd":"([^"]+)"/,
|
||||
]);
|
||||
const appId = pick([
|
||||
/"X-IG-App-ID":"(\\d+)"/,
|
||||
/"instagramWebAppId":"(\\d+)"/,
|
||||
/"appId":"(\\d+)"/,
|
||||
]);
|
||||
const asbdId = pick([
|
||||
/"X-ASBD-ID":"(\\d+)"/,
|
||||
/"asbd_id":"(\\d+)"/,
|
||||
]);
|
||||
const spinR = pick([/"__spin_r":(\\d+)/]);
|
||||
const spinB = pick([/"__spin_b":"([^"]+)"/]);
|
||||
const spinT = pick([/"__spin_t":(\\d+)/]);
|
||||
const csrfToken = readCookie('csrftoken') || pick([
|
||||
/"csrf_token":"([^"]+)"/,
|
||||
/"csrfToken":"([^"]+)"/,
|
||||
]);
|
||||
const jazoest = fbDtsg
|
||||
? '2' + Array.from(fbDtsg).reduce((total, char) => total + char.charCodeAt(0), 0)
|
||||
: '';
|
||||
|
||||
if (!actorId || !fbDtsg || !lsd || !appId || !csrfToken || !spinR || !spinB || !spinT || !jazoest) {
|
||||
return {
|
||||
ok: false,
|
||||
stage: 'config',
|
||||
text: JSON.stringify({
|
||||
actorId: Boolean(actorId),
|
||||
fbDtsg: Boolean(fbDtsg),
|
||||
lsd: Boolean(lsd),
|
||||
appId: Boolean(appId),
|
||||
csrfToken: Boolean(csrfToken),
|
||||
spinR: Boolean(spinR),
|
||||
spinB: Boolean(spinB),
|
||||
spinT: Boolean(spinT),
|
||||
jazoest: Boolean(jazoest),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
const variables = {
|
||||
input: {
|
||||
actor_id: actorId,
|
||||
client_mutation_id: '1',
|
||||
additional_params: {
|
||||
note_create_params: {
|
||||
note_style: 0,
|
||||
text: input.content,
|
||||
},
|
||||
},
|
||||
audience: 0,
|
||||
inbox_tray_item_type: 'note',
|
||||
},
|
||||
};
|
||||
|
||||
const body = new URLSearchParams();
|
||||
body.set('av', actorId);
|
||||
body.set('__user', '0');
|
||||
body.set('__a', '1');
|
||||
body.set('__req', '1');
|
||||
body.set('__hs', '');
|
||||
body.set('dpr', String(window.devicePixelRatio || 1));
|
||||
body.set('__ccg', 'UNKNOWN');
|
||||
body.set('__rev', spinR);
|
||||
body.set('__s', '');
|
||||
body.set('__hsi', '');
|
||||
body.set('__dyn', '');
|
||||
body.set('__csr', '');
|
||||
body.set('__comet_req', '7');
|
||||
body.set('fb_dtsg', fbDtsg);
|
||||
body.set('jazoest', jazoest);
|
||||
body.set('lsd', lsd);
|
||||
body.set('__spin_r', spinR);
|
||||
body.set('__spin_b', spinB);
|
||||
body.set('__spin_t', spinT);
|
||||
body.set('fb_api_caller_class', 'RelayModern');
|
||||
body.set('fb_api_req_friendly_name', ${JSON.stringify(INSTAGRAM_NOTE_MUTATION_NAME)});
|
||||
body.set('variables', JSON.stringify(variables));
|
||||
body.set('server_timestamps', 'true');
|
||||
body.set('doc_id', ${JSON.stringify(INSTAGRAM_NOTE_DOC_ID)});
|
||||
|
||||
const headers = {
|
||||
Accept: '*/*',
|
||||
'Content-Type': 'application/x-www-form-urlencoded',
|
||||
'X-ASBD-ID': asbdId || undefined,
|
||||
'X-CSRFToken': csrfToken,
|
||||
'X-FB-Friendly-Name': ${JSON.stringify(INSTAGRAM_NOTE_MUTATION_NAME)},
|
||||
'X-FB-LSD': lsd,
|
||||
'X-IG-App-ID': appId,
|
||||
'X-Root-Field-Name': ${JSON.stringify(INSTAGRAM_NOTE_ROOT_FIELD)},
|
||||
};
|
||||
|
||||
const response = await fetch('/graphql/query', {
|
||||
method: 'POST',
|
||||
credentials: 'include',
|
||||
headers,
|
||||
body: body.toString(),
|
||||
});
|
||||
const text = await response.text();
|
||||
const normalizedText = text.replace(/^for \\(;;\\);?/, '').trim();
|
||||
let data = null;
|
||||
try {
|
||||
data = JSON.parse(normalizedText);
|
||||
} catch {}
|
||||
|
||||
const rootField = ${JSON.stringify(INSTAGRAM_NOTE_ROOT_FIELD)};
|
||||
const note = data?.data?.[rootField]?.inbox_tray_item;
|
||||
const noteId = String(note?.inbox_tray_item_id || note?.id || '');
|
||||
if (response.ok && noteId) {
|
||||
return {
|
||||
ok: true,
|
||||
stage: 'publish',
|
||||
noteId,
|
||||
text: String(note?.note_dict?.text || input.content || ''),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
ok: false,
|
||||
stage: 'publish',
|
||||
status: response.status,
|
||||
text: normalizedText || text,
|
||||
};
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'instagram',
|
||||
name: 'note',
|
||||
description: 'Publish a text Instagram note',
|
||||
domain: 'www.instagram.com',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
timeoutSeconds: 120,
|
||||
args: [
|
||||
{ name: 'content', positional: true, required: true, help: 'Note text (max 60 characters)' },
|
||||
],
|
||||
columns: ['status', 'detail', 'noteId'],
|
||||
validateArgs: validateInstagramNoteArgs,
|
||||
func: async (page: IPage | null, kwargs) => {
|
||||
const browserPage = requirePage(page);
|
||||
const content = normalizeInstagramNoteContent(kwargs as Record<string, unknown>);
|
||||
await browserPage.goto(INSTAGRAM_INBOX_URL);
|
||||
await browserPage.wait({ time: 2 });
|
||||
const result = await browserPage.evaluate(buildPublishInstagramNoteJs(content)) as BrowserNoteResult;
|
||||
if (!result?.ok) {
|
||||
throw new CommandExecutionError(
|
||||
`Instagram note publish failed at ${String(result?.stage || 'unknown')}: ${String(result?.text || 'unknown error')}`,
|
||||
);
|
||||
}
|
||||
return buildNoteSuccessResult(String(result.noteId || ''));
|
||||
},
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,191 @@
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ArgumentError } from '../../errors.js';
|
||||
import { getRegistry } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import './reel.js';
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function createTempVideo(name = 'demo.mp4', bytes = Buffer.from('video')): string {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-instagram-reel-'));
|
||||
tempDirs.push(dir);
|
||||
const filePath = path.join(dir, name);
|
||||
fs.writeFileSync(filePath, bytes);
|
||||
return filePath;
|
||||
}
|
||||
|
||||
function createPageMock(evaluateResults: unknown[], overrides: Partial<IPage> = {}): IPage {
|
||||
const evaluate = vi.fn();
|
||||
for (const result of evaluateResults) {
|
||||
evaluate.mockResolvedValueOnce(result);
|
||||
}
|
||||
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
getCookies: vi.fn().mockResolvedValue([]),
|
||||
snapshot: vi.fn().mockResolvedValue(undefined),
|
||||
click: vi.fn().mockResolvedValue(undefined),
|
||||
typeText: vi.fn().mockResolvedValue(undefined),
|
||||
pressKey: vi.fn().mockResolvedValue(undefined),
|
||||
scrollTo: vi.fn().mockResolvedValue(undefined),
|
||||
getFormState: vi.fn().mockResolvedValue({ forms: [], orphanFields: [] }),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
tabs: vi.fn().mockResolvedValue([]),
|
||||
closeTab: vi.fn().mockResolvedValue(undefined),
|
||||
newTab: vi.fn().mockResolvedValue(undefined),
|
||||
selectTab: vi.fn().mockResolvedValue(undefined),
|
||||
networkRequests: vi.fn().mockResolvedValue([]),
|
||||
consoleMessages: vi.fn().mockResolvedValue([]),
|
||||
scroll: vi.fn().mockResolvedValue(undefined),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
screenshot: vi.fn().mockResolvedValue(''),
|
||||
setFileInput: vi.fn().mockResolvedValue(undefined),
|
||||
insertText: vi.fn().mockResolvedValue(undefined),
|
||||
getCurrentUrl: vi.fn().mockResolvedValue(null),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
afterAll(() => {
|
||||
for (const dir of tempDirs) {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe('instagram reel registration', () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('registers the reel command with a required-value video arg', () => {
|
||||
const cmd = getRegistry().get('instagram/reel');
|
||||
expect(cmd).toBeDefined();
|
||||
expect(cmd?.browser).toBe(true);
|
||||
expect(cmd?.args.some((arg) => arg.name === 'video' && !arg.required && arg.valueRequired)).toBe(true);
|
||||
expect(cmd?.args.some((arg) => arg.name === 'content' && arg.positional && !arg.required)).toBe(true);
|
||||
});
|
||||
|
||||
it('rejects missing --video before browser work', async () => {
|
||||
const page = createPageMock([]);
|
||||
const cmd = getRegistry().get('instagram/reel');
|
||||
|
||||
await expect(cmd!.func!(page, { content: 'hello reel' })).rejects.toThrow(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects unsupported video formats', async () => {
|
||||
const videoPath = createTempVideo('demo.mov');
|
||||
const page = createPageMock([]);
|
||||
const cmd = getRegistry().get('instagram/reel');
|
||||
|
||||
await expect(cmd!.func!(page, { video: videoPath })).rejects.toThrow('Unsupported video format');
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('uploads a reel video without caption and shares it', async () => {
|
||||
const videoPath = createTempVideo();
|
||||
const page = createPageMock([
|
||||
{ ok: false }, // dismiss residual dialogs
|
||||
{ ok: true }, // ensure composer open
|
||||
{ ok: true }, // composer upload input ready
|
||||
{ ok: true, selectors: ['[data-opencli-reel-upload-index="0"]', '[data-opencli-reel-upload-index="1"]'] }, // resolve upload selector
|
||||
{ count: 1 }, // file bound to input
|
||||
{ state: 'preview', detail: 'Crop Back Next' }, // preview detected
|
||||
{ ok: true, label: 'OK' }, // dismiss reels nux
|
||||
{ ok: true, label: 'Next' }, // move from crop to edit
|
||||
{ state: 'edit' }, // edit stage
|
||||
{ ok: true, label: 'Next' }, // move from edit to composer
|
||||
{ state: 'composer' }, // composer stage
|
||||
{ ok: true, label: 'Share' }, // share
|
||||
{ ok: true, url: 'https://www.instagram.com/reel/REEL123/' }, // success
|
||||
]);
|
||||
|
||||
const cmd = getRegistry().get('instagram/reel');
|
||||
const result = await cmd!.func!(page, { video: videoPath });
|
||||
|
||||
expect(page.setFileInput).toHaveBeenCalledWith([videoPath], '[data-opencli-reel-upload-index="0"]');
|
||||
expect(page.insertText).not.toHaveBeenCalled();
|
||||
expect(result).toEqual([
|
||||
{
|
||||
status: '✅ Posted',
|
||||
detail: 'Single reel shared successfully',
|
||||
url: 'https://www.instagram.com/reel/REEL123/',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('copies query-style local video filenames to a safe temp upload path before setFileInput', async () => {
|
||||
const videoPath = createTempVideo('demo.mp4?sign=abc&t=123video.MP4');
|
||||
const page = createPageMock([
|
||||
{ ok: false },
|
||||
{ ok: true },
|
||||
{ ok: true },
|
||||
{ ok: true, selectors: ['[data-opencli-reel-upload-index="0"]'] },
|
||||
{ count: 1 },
|
||||
{ state: 'preview', detail: 'Crop Back Next' },
|
||||
{ ok: true, label: 'OK' },
|
||||
{ ok: true, label: 'Next' },
|
||||
{ state: 'edit' },
|
||||
{ ok: true, label: 'Next' },
|
||||
{ state: 'composer' },
|
||||
{ ok: true, label: 'Share' },
|
||||
{ ok: true, url: 'https://www.instagram.com/reel/REELSAFE123/' },
|
||||
]);
|
||||
|
||||
const cmd = getRegistry().get('instagram/reel');
|
||||
await cmd!.func!(page, { video: videoPath });
|
||||
|
||||
const uploadPaths = (page.setFileInput as any).mock.calls[0]?.[0] ?? [];
|
||||
expect(uploadPaths).toHaveLength(1);
|
||||
expect(uploadPaths[0]).not.toBe(videoPath);
|
||||
expect(String(uploadPaths[0])).toContain('opencli-instagram-video-real');
|
||||
expect(String(uploadPaths[0]).toLowerCase()).toContain('.mp4');
|
||||
});
|
||||
|
||||
it('uploads a reel video with caption and shares it', async () => {
|
||||
const videoPath = createTempVideo('captioned.mp4');
|
||||
const page = createPageMock([
|
||||
{ ok: false }, // dismiss residual dialogs
|
||||
{ ok: true }, // ensure composer open
|
||||
{ ok: true }, // composer upload input ready
|
||||
{ ok: true, selectors: ['[data-opencli-reel-upload-index="0"]'] }, // resolve upload selector
|
||||
{ count: 1 }, // file bound to input
|
||||
{ state: 'preview', detail: 'Crop Back Next' }, // preview detected
|
||||
{ ok: true, label: 'OK' }, // dismiss reels nux
|
||||
{ ok: true, label: 'Next' }, // move from crop to edit
|
||||
{ state: 'edit' }, // edit stage
|
||||
{ ok: true, label: 'Next' }, // move from edit to composer
|
||||
{ state: 'composer' }, // composer stage
|
||||
{ ok: true }, // focus caption editor
|
||||
{ ok: true }, // post-insert event dispatch
|
||||
{ ok: true }, // caption matches
|
||||
{ ok: true, label: 'Share' }, // share
|
||||
{ ok: true, url: 'https://www.instagram.com/reel/REEL456/' }, // success
|
||||
]);
|
||||
|
||||
const cmd = getRegistry().get('instagram/reel');
|
||||
const result = await cmd!.func!(page, { video: videoPath, content: 'hello reel' });
|
||||
|
||||
expect(page.insertText).toHaveBeenCalledWith('hello reel');
|
||||
expect(result).toEqual([
|
||||
{
|
||||
status: '✅ Posted',
|
||||
detail: 'Single reel shared successfully',
|
||||
url: 'https://www.instagram.com/reel/REEL456/',
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,886 @@
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import { Page as BrowserPage } from '../../browser/page.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '../../errors.js';
|
||||
import type { BrowserCookie, IPage } from '../../types.js';
|
||||
import {
|
||||
buildClickActionJs,
|
||||
buildEnsureComposerOpenJs,
|
||||
buildInspectUploadStageJs,
|
||||
} from './post.js';
|
||||
import { resolveInstagramRuntimeInfo } from './_shared/runtime-info.js';
|
||||
|
||||
const INSTAGRAM_HOME_URL = 'https://www.instagram.com/';
|
||||
const SUPPORTED_VIDEO_EXTENSIONS = new Set(['.mp4']);
|
||||
const INSTAGRAM_REEL_TIMEOUT_SECONDS = 600;
|
||||
|
||||
type InstagramReelSuccessRow = {
|
||||
status: string;
|
||||
detail: string;
|
||||
url: string;
|
||||
};
|
||||
|
||||
type ReelStageState = {
|
||||
state: 'crop' | 'edit' | 'composer' | 'failed' | 'pending';
|
||||
detail?: string;
|
||||
};
|
||||
|
||||
type PreparedVideoUpload = {
|
||||
originalPath: string;
|
||||
uploadPath: string;
|
||||
cleanupPath?: string;
|
||||
};
|
||||
|
||||
function requirePage(page: IPage | null): IPage {
|
||||
if (!page) throw new CommandExecutionError('Browser session required for instagram reel');
|
||||
return page;
|
||||
}
|
||||
|
||||
async function gotoInstagramHome(page: IPage, forceReload = false): Promise<void> {
|
||||
if (forceReload) {
|
||||
await page.goto(`${INSTAGRAM_HOME_URL}?__opencli_reset=${Date.now()}`);
|
||||
await page.wait({ time: 1 });
|
||||
}
|
||||
await page.goto(INSTAGRAM_HOME_URL);
|
||||
}
|
||||
|
||||
function validateVideoPath(input: unknown): string {
|
||||
const resolved = path.resolve(String(input || '').trim());
|
||||
if (!resolved) {
|
||||
throw new ArgumentError('Video path cannot be empty');
|
||||
}
|
||||
if (!fs.existsSync(resolved)) {
|
||||
throw new ArgumentError(`Video file not found: ${resolved}`);
|
||||
}
|
||||
const ext = path.extname(resolved).toLowerCase();
|
||||
if (!SUPPORTED_VIDEO_EXTENSIONS.has(ext)) {
|
||||
throw new ArgumentError(`Unsupported video format: ${ext}`, 'Supported formats: .mp4');
|
||||
}
|
||||
return resolved;
|
||||
}
|
||||
|
||||
function validateInstagramReelArgs(kwargs: Record<string, unknown>): void {
|
||||
if (kwargs.video === undefined) {
|
||||
throw new ArgumentError(
|
||||
'Argument "video" is required.',
|
||||
'Provide --video /path/to/file.mp4',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function buildInstagramReelSuccessResult(url: string): InstagramReelSuccessRow[] {
|
||||
return [{
|
||||
status: '✅ Posted',
|
||||
detail: 'Single reel shared successfully',
|
||||
url,
|
||||
}];
|
||||
}
|
||||
|
||||
function isRecoverableReelSessionError(error: unknown): boolean {
|
||||
if (!(error instanceof CommandExecutionError)) return false;
|
||||
return error.message === 'Instagram reel upload input not found'
|
||||
|| error.message === 'Instagram reel preview did not appear after upload'
|
||||
|| error.message === 'Instagram reel upload failed';
|
||||
}
|
||||
|
||||
function buildSafeTempVideoPath(filePath: string): string {
|
||||
const ext = path.extname(filePath).toLowerCase() || '.mp4';
|
||||
return path.join(os.tmpdir(), `opencli-instagram-video-real${ext}`);
|
||||
}
|
||||
|
||||
function prepareVideoUpload(filePath: string): PreparedVideoUpload {
|
||||
const baseName = path.basename(filePath);
|
||||
if (/^[a-zA-Z0-9._-]+$/.test(baseName)) {
|
||||
return { originalPath: filePath, uploadPath: filePath };
|
||||
}
|
||||
const uploadPath = buildSafeTempVideoPath(filePath);
|
||||
fs.copyFileSync(filePath, uploadPath);
|
||||
return {
|
||||
originalPath: filePath,
|
||||
uploadPath,
|
||||
cleanupPath: uploadPath,
|
||||
};
|
||||
}
|
||||
|
||||
async function ensureComposerOpen(page: IPage): Promise<void> {
|
||||
const result = await page.evaluate(buildEnsureComposerOpenJs()) as { ok?: boolean; reason?: string };
|
||||
if (!result?.ok) {
|
||||
if (result?.reason === 'auth') {
|
||||
throw new AuthRequiredError('www.instagram.com', 'Instagram login required before posting a reel');
|
||||
}
|
||||
throw new CommandExecutionError('Failed to open Instagram reel composer');
|
||||
}
|
||||
for (let attempt = 0; attempt < 12; attempt += 1) {
|
||||
const ready = await page.evaluate(`
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const inputs = Array.from(document.querySelectorAll('input[type="file"]'))
|
||||
.filter((el) => el instanceof HTMLInputElement)
|
||||
.filter((el) => {
|
||||
const dialog = el.closest('[role="dialog"]');
|
||||
return dialog instanceof HTMLElement && isVisible(dialog);
|
||||
});
|
||||
return { ok: inputs.length > 0 };
|
||||
})()
|
||||
`) as { ok?: boolean };
|
||||
if (ready?.ok) return;
|
||||
if (attempt < 11) await page.wait({ time: 0.5 });
|
||||
}
|
||||
throw new CommandExecutionError('Instagram reel upload input not found', 'Open the new-post composer in a logged-in browser session and retry');
|
||||
}
|
||||
|
||||
async function dismissResidualDialogs(page: IPage): Promise<void> {
|
||||
for (let attempt = 0; attempt < 4; attempt += 1) {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]'))
|
||||
.filter((el) => el instanceof HTMLElement && isVisible(el));
|
||||
for (const dialog of dialogs) {
|
||||
const text = (dialog.textContent || '').replace(/\\s+/g, ' ').trim().toLowerCase();
|
||||
if (!text) continue;
|
||||
if (
|
||||
text.includes('post shared')
|
||||
|| text.includes('your post has been shared')
|
||||
|| text.includes('your reel has been shared')
|
||||
|| text.includes('video posts are now reels')
|
||||
|| text.includes('something went wrong')
|
||||
|| text.includes('sharing')
|
||||
|| text.includes('create new post')
|
||||
|| text.includes('new reel')
|
||||
|| text.includes('crop')
|
||||
|| text.includes('edit')
|
||||
) {
|
||||
const close = dialog.querySelector('[aria-label="Close"], button[aria-label="Close"], div[role="button"][aria-label="Close"]');
|
||||
if (close instanceof HTMLElement && isVisible(close)) {
|
||||
close.click();
|
||||
return { ok: true };
|
||||
}
|
||||
}
|
||||
}
|
||||
return { ok: false };
|
||||
})()
|
||||
`) as { ok?: boolean };
|
||||
|
||||
if (!result?.ok) return;
|
||||
await page.wait({ time: 0.5 });
|
||||
}
|
||||
}
|
||||
|
||||
async function resolveUploadSelectors(page: IPage): Promise<string[]> {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]'))
|
||||
.filter((el) => el instanceof HTMLElement && isVisible(el));
|
||||
const roots = dialogs.length ? dialogs : [document.body];
|
||||
const selectors = [];
|
||||
let index = 0;
|
||||
|
||||
for (const root of roots) {
|
||||
const inputs = Array.from(root.querySelectorAll('input[type="file"]'));
|
||||
for (const input of inputs) {
|
||||
if (!(input instanceof HTMLInputElement)) continue;
|
||||
if (input.disabled) continue;
|
||||
const accept = (input.getAttribute('accept') || '').toLowerCase();
|
||||
if (accept && !accept.includes('video') && !accept.includes('.mp4')) continue;
|
||||
input.setAttribute('data-opencli-reel-upload-index', String(index));
|
||||
selectors.push('[data-opencli-reel-upload-index="' + index + '"]');
|
||||
index += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return { ok: selectors.length > 0, selectors };
|
||||
})()
|
||||
`) as { ok?: boolean; selectors?: string[] };
|
||||
|
||||
if (!result?.ok || !Array.isArray(result.selectors) || result.selectors.length === 0) {
|
||||
throw new CommandExecutionError(
|
||||
'Instagram reel upload input not found',
|
||||
'Open the new-post composer in a logged-in browser session and retry',
|
||||
);
|
||||
}
|
||||
return result.selectors;
|
||||
}
|
||||
|
||||
async function uploadVideo(page: IPage, videoPath: string, selector: string): Promise<void> {
|
||||
if (!page.setFileInput) {
|
||||
throw new CommandExecutionError(
|
||||
'Instagram reel upload requires Browser Bridge file upload support',
|
||||
'Use Browser Bridge or another browser mode that supports setFileInput',
|
||||
);
|
||||
}
|
||||
await page.setFileInput([videoPath], selector);
|
||||
}
|
||||
|
||||
async function readSelectedFileCount(page: IPage, selector: string): Promise<number | null> {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
const input = document.querySelector(${JSON.stringify(selector)});
|
||||
if (!(input instanceof HTMLInputElement)) return { count: null };
|
||||
return { count: input.files?.length || 0 };
|
||||
})()
|
||||
`) as { count?: number | null };
|
||||
if (result?.count === null || result?.count === undefined) return null;
|
||||
return Number(result.count);
|
||||
}
|
||||
|
||||
async function waitForVideoPreview(page: IPage, maxWaitSeconds = 20): Promise<void> {
|
||||
let lastDetail = '';
|
||||
for (let attempt = 0; attempt < maxWaitSeconds * 2; attempt += 1) {
|
||||
const result = await page.evaluate(buildInspectUploadStageJs()) as { state?: string; detail?: string };
|
||||
lastDetail = String(result?.detail || '').trim();
|
||||
if (result?.state === 'preview') return;
|
||||
if (result?.state === 'failed') {
|
||||
throw new CommandExecutionError(
|
||||
'Instagram reel upload failed',
|
||||
result.detail ? `Instagram rejected the reel upload: ${result.detail}` : 'Instagram rejected the reel upload before the preview stage',
|
||||
);
|
||||
}
|
||||
if (attempt < maxWaitSeconds * 2 - 1) await page.wait({ time: 0.5 });
|
||||
}
|
||||
await page.screenshot({ path: '/tmp/instagram_reel_preview_debug.png' });
|
||||
throw new CommandExecutionError(
|
||||
'Instagram reel preview did not appear after upload',
|
||||
lastDetail
|
||||
? `Inspect /tmp/instagram_reel_preview_debug.png. Last visible dialog text: ${lastDetail}`
|
||||
: 'Inspect /tmp/instagram_reel_preview_debug.png for the upload state',
|
||||
);
|
||||
}
|
||||
|
||||
async function clickAction(page: IPage, labels: string[], scope: 'any' | 'media' | 'caption' = 'any'): Promise<string> {
|
||||
const result = await page.evaluate(buildClickActionJs(labels, scope)) as { ok?: boolean; label?: string };
|
||||
if (!result?.ok) {
|
||||
throw new CommandExecutionError(`Instagram action button not found: ${labels.join(' / ')}`);
|
||||
}
|
||||
return result.label || labels[0] || '';
|
||||
}
|
||||
|
||||
async function clickActionMaybe(page: IPage, labels: string[], scope: 'any' | 'media' | 'caption' = 'any'): Promise<boolean> {
|
||||
const result = await page.evaluate(buildClickActionJs(labels, scope)) as { ok?: boolean };
|
||||
return !!result?.ok;
|
||||
}
|
||||
|
||||
function buildInspectReelStageJs(): string {
|
||||
return `
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]')).filter((el) => isVisible(el));
|
||||
const text = dialogs.map((el) => (el.textContent || '').replace(/\\s+/g, ' ').trim()).join(' ');
|
||||
const lower = text.toLowerCase();
|
||||
const hasVisibleButton = (labels) => dialogs.some((dialog) =>
|
||||
Array.from(dialog.querySelectorAll('button, div[role="button"]')).some((el) => {
|
||||
const value = (el.textContent || '').replace(/\\s+/g, ' ').trim().toLowerCase();
|
||||
return isVisible(el) && labels.includes(value);
|
||||
})
|
||||
);
|
||||
if (/something went wrong|please try again|share failed|couldn['’]t be shared|could not be shared|失败|出错/.test(lower)) {
|
||||
return { state: 'failed', detail: text };
|
||||
}
|
||||
if (/new reel|write a caption|add location|tag people/.test(lower) && hasVisibleButton(['share'])) {
|
||||
return { state: 'composer', detail: text };
|
||||
}
|
||||
if (/edit|cover photo|trim|video has no audio/.test(lower) && hasVisibleButton(['next'])) {
|
||||
return { state: 'edit', detail: text };
|
||||
}
|
||||
if (/crop|select crop|open media gallery/.test(lower) && hasVisibleButton(['next'])) {
|
||||
return { state: 'crop', detail: text };
|
||||
}
|
||||
return { state: 'pending', detail: text };
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
async function waitForReelStage(page: IPage, expected: ReelStageState['state'], maxWaitSeconds = 20): Promise<void> {
|
||||
for (let attempt = 0; attempt < maxWaitSeconds * 2; attempt += 1) {
|
||||
const result = await page.evaluate(buildInspectReelStageJs()) as ReelStageState;
|
||||
if (result?.state === expected) return;
|
||||
if (result?.state === 'failed') {
|
||||
throw new CommandExecutionError(
|
||||
'Instagram reel editor did not appear',
|
||||
result.detail ? `Instagram reel flow failed: ${result.detail}` : 'Instagram reel flow failed before the next editor stage',
|
||||
);
|
||||
}
|
||||
if (attempt < maxWaitSeconds * 2 - 1) await page.wait({ time: 0.5 });
|
||||
}
|
||||
throw new CommandExecutionError(`Instagram reel ${expected} editor did not appear`);
|
||||
}
|
||||
|
||||
async function focusCaptionEditor(page: IPage): Promise<boolean> {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]')).filter((el) => isVisible(el));
|
||||
for (const dialog of dialogs) {
|
||||
const textarea = dialog.querySelector('[aria-label="Write a caption..."], textarea');
|
||||
if (textarea instanceof HTMLTextAreaElement && isVisible(textarea)) {
|
||||
textarea.focus();
|
||||
textarea.select();
|
||||
return { ok: true, kind: 'textarea' };
|
||||
}
|
||||
|
||||
const editor = dialog.querySelector('[aria-label="Write a caption..."][contenteditable="true"]')
|
||||
|| dialog.querySelector('[contenteditable="true"]');
|
||||
if (editor instanceof HTMLElement && isVisible(editor)) {
|
||||
const lexical = editor.__lexicalEditor;
|
||||
try {
|
||||
if (lexical && typeof lexical.getEditorState === 'function' && typeof lexical.parseEditorState === 'function') {
|
||||
const emptyState = {
|
||||
root: {
|
||||
children: [{
|
||||
children: [],
|
||||
direction: null,
|
||||
format: '',
|
||||
indent: 0,
|
||||
textFormat: 0,
|
||||
textStyle: '',
|
||||
type: 'paragraph',
|
||||
version: 1,
|
||||
}],
|
||||
direction: null,
|
||||
format: '',
|
||||
indent: 0,
|
||||
type: 'root',
|
||||
version: 1,
|
||||
},
|
||||
};
|
||||
const nextState = lexical.parseEditorState(JSON.stringify(emptyState));
|
||||
try {
|
||||
lexical.setEditorState(nextState, { tag: 'history-merge', discrete: true });
|
||||
} catch {
|
||||
lexical.setEditorState(nextState);
|
||||
}
|
||||
} else {
|
||||
editor.textContent = '';
|
||||
}
|
||||
} catch {
|
||||
editor.textContent = '';
|
||||
}
|
||||
|
||||
editor.focus();
|
||||
const selection = window.getSelection();
|
||||
if (selection) {
|
||||
selection.removeAllRanges();
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(editor);
|
||||
range.collapse(false);
|
||||
selection.addRange(range);
|
||||
}
|
||||
return { ok: true, kind: 'contenteditable' };
|
||||
}
|
||||
}
|
||||
return { ok: false };
|
||||
})()
|
||||
`) as { ok?: boolean };
|
||||
return !!result?.ok;
|
||||
}
|
||||
|
||||
async function captionMatches(page: IPage, content: string): Promise<boolean> {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
const target = ${JSON.stringify(content.trim())}.replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const readLexicalText = (node) => {
|
||||
if (!node || typeof node !== 'object') return '';
|
||||
if (node.type === 'text' && typeof node.text === 'string') return node.text;
|
||||
if (!Array.isArray(node.children)) return '';
|
||||
if (node.type === 'root') return node.children.map((child) => readLexicalText(child)).join('\\n');
|
||||
if (node.type === 'paragraph') return node.children.map((child) => readLexicalText(child)).join('');
|
||||
return node.children.map((child) => readLexicalText(child)).join('');
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]')).filter((el) => isVisible(el));
|
||||
for (const dialog of dialogs) {
|
||||
const textarea = dialog.querySelector('[aria-label="Write a caption..."], textarea');
|
||||
if (textarea instanceof HTMLTextAreaElement && isVisible(textarea)) {
|
||||
if (textarea.value.replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim() === target) {
|
||||
return { ok: true };
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const editor = dialog.querySelector('[aria-label="Write a caption..."][contenteditable="true"]')
|
||||
|| dialog.querySelector('[contenteditable="true"]');
|
||||
if (!(editor instanceof HTMLElement) || !isVisible(editor)) continue;
|
||||
|
||||
const lexical = editor.__lexicalEditor;
|
||||
if (lexical && typeof lexical.getEditorState === 'function') {
|
||||
const currentState = lexical.getEditorState();
|
||||
const pendingState = lexical._pendingEditorState;
|
||||
const current = currentState && typeof currentState.toJSON === 'function' ? currentState.toJSON() : null;
|
||||
const pending = pendingState && typeof pendingState.toJSON === 'function' ? pendingState.toJSON() : null;
|
||||
const currentText = readLexicalText(current && current.root).replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
|
||||
const pendingText = readLexicalText(pending && pending.root).replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
|
||||
if (currentText === target || pendingText === target) {
|
||||
return { ok: true };
|
||||
}
|
||||
}
|
||||
|
||||
const value = (editor.textContent || '').replace(/\\u00a0/g, ' ').replace(/\\s+/g, ' ').trim();
|
||||
if (value === target) {
|
||||
return { ok: true };
|
||||
}
|
||||
}
|
||||
return { ok: false };
|
||||
})()
|
||||
`) as { ok?: boolean };
|
||||
return !!result?.ok;
|
||||
}
|
||||
|
||||
async function fillCaption(page: IPage, content: string): Promise<void> {
|
||||
const focused = await focusCaptionEditor(page);
|
||||
if (!focused) {
|
||||
throw new CommandExecutionError('Instagram reel caption editor did not appear');
|
||||
}
|
||||
if (page.insertText) {
|
||||
try {
|
||||
await page.insertText(content);
|
||||
await page.wait({ time: 0.3 });
|
||||
await page.evaluate(`
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]')).filter((el) => isVisible(el));
|
||||
for (const dialog of dialogs) {
|
||||
const textarea = dialog.querySelector('[aria-label="Write a caption..."], textarea');
|
||||
if (textarea instanceof HTMLTextAreaElement && isVisible(textarea)) {
|
||||
textarea.dispatchEvent(new InputEvent('input', { bubbles: true, composed: true, inputType: 'insertText' }));
|
||||
textarea.dispatchEvent(new Event('change', { bubbles: true, composed: true }));
|
||||
textarea.blur();
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
const editor = dialog.querySelector('[aria-label="Write a caption..."][contenteditable="true"]')
|
||||
|| dialog.querySelector('[contenteditable="true"]');
|
||||
if (!(editor instanceof HTMLElement) || !isVisible(editor)) continue;
|
||||
try {
|
||||
editor.dispatchEvent(new InputEvent('input', { bubbles: true, composed: true, inputType: 'insertText' }));
|
||||
} catch {
|
||||
editor.dispatchEvent(new Event('input', { bubbles: true, composed: true }));
|
||||
}
|
||||
editor.dispatchEvent(new Event('change', { bubbles: true, composed: true }));
|
||||
editor.blur();
|
||||
return { ok: true };
|
||||
}
|
||||
return { ok: false };
|
||||
})()
|
||||
`);
|
||||
return;
|
||||
} catch {
|
||||
// Fall back to browser-side editor manipulation below.
|
||||
}
|
||||
}
|
||||
await page.evaluate(`
|
||||
((content) => {
|
||||
const createParagraph = (text) => ({
|
||||
children: text
|
||||
? [{ detail: 0, format: 0, mode: 'normal', style: '', text, type: 'text', version: 1 }]
|
||||
: [],
|
||||
direction: null,
|
||||
format: '',
|
||||
indent: 0,
|
||||
textFormat: 0,
|
||||
textStyle: '',
|
||||
type: 'paragraph',
|
||||
version: 1,
|
||||
});
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]')).filter((el) => isVisible(el));
|
||||
for (const dialog of dialogs) {
|
||||
const textarea = dialog.querySelector('[aria-label="Write a caption..."], textarea');
|
||||
if (textarea instanceof HTMLTextAreaElement && isVisible(textarea)) {
|
||||
textarea.focus();
|
||||
const dt = new DataTransfer();
|
||||
dt.setData('text/plain', content);
|
||||
textarea.dispatchEvent(new ClipboardEvent('paste', {
|
||||
clipboardData: dt,
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
}));
|
||||
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, 'value')?.set;
|
||||
setter?.call(textarea, content);
|
||||
textarea.dispatchEvent(new Event('input', { bubbles: true, composed: true }));
|
||||
textarea.dispatchEvent(new Event('change', { bubbles: true, composed: true }));
|
||||
textarea.blur();
|
||||
return { ok: true, mode: 'textarea' };
|
||||
}
|
||||
|
||||
const editor = dialog.querySelector('[aria-label="Write a caption..."][contenteditable="true"]')
|
||||
|| dialog.querySelector('[contenteditable="true"]');
|
||||
if (!(editor instanceof HTMLElement) || !isVisible(editor)) continue;
|
||||
|
||||
editor.focus();
|
||||
const lexical = editor.__lexicalEditor;
|
||||
if (lexical && typeof lexical.getEditorState === 'function' && typeof lexical.parseEditorState === 'function') {
|
||||
const currentState = lexical.getEditorState && lexical.getEditorState();
|
||||
const base = currentState && typeof currentState.toJSON === 'function' ? currentState.toJSON() : {};
|
||||
const lines = String(content).split(/\\r?\\n/);
|
||||
const paragraphs = lines.map((line) => createParagraph(line));
|
||||
base.root = {
|
||||
children: paragraphs.length ? paragraphs : [createParagraph('')],
|
||||
direction: null,
|
||||
format: '',
|
||||
indent: 0,
|
||||
type: 'root',
|
||||
version: 1,
|
||||
};
|
||||
|
||||
const nextState = lexical.parseEditorState(JSON.stringify(base));
|
||||
try {
|
||||
lexical.setEditorState(nextState, { tag: 'history-merge', discrete: true });
|
||||
} catch {
|
||||
lexical.setEditorState(nextState);
|
||||
}
|
||||
|
||||
editor.dispatchEvent(new Event('input', { bubbles: true, composed: true }));
|
||||
editor.dispatchEvent(new Event('change', { bubbles: true, composed: true }));
|
||||
editor.blur();
|
||||
return { ok: true, mode: 'lexical' };
|
||||
}
|
||||
|
||||
const selection = window.getSelection();
|
||||
if (selection) {
|
||||
selection.removeAllRanges();
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(editor);
|
||||
selection.addRange(range);
|
||||
}
|
||||
const dt = new DataTransfer();
|
||||
dt.setData('text/plain', content);
|
||||
editor.dispatchEvent(new ClipboardEvent('paste', {
|
||||
clipboardData: dt,
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
}));
|
||||
editor.blur();
|
||||
return { ok: true, mode: 'contenteditable' };
|
||||
}
|
||||
return { ok: false };
|
||||
})(${JSON.stringify(content)})
|
||||
`);
|
||||
}
|
||||
|
||||
async function ensureCaptionFilled(page: IPage, content: string): Promise<void> {
|
||||
for (let attempt = 0; attempt < 6; attempt += 1) {
|
||||
if (await captionMatches(page, content)) return;
|
||||
if (attempt < 5) await page.wait({ time: 0.5 });
|
||||
}
|
||||
throw new CommandExecutionError('Instagram reel caption did not stick before sharing');
|
||||
}
|
||||
|
||||
function buildReelPublishStatusProbeJs(): string {
|
||||
return `
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const dialogs = Array.from(document.querySelectorAll('[role="dialog"]')).filter((el) => isVisible(el));
|
||||
const dialogText = dialogs.map((el) => (el.textContent || '').replace(/\\s+/g, ' ').trim()).join(' ');
|
||||
const lower = dialogText.toLowerCase();
|
||||
const url = window.location.href;
|
||||
const sharingVisible = /sharing/.test(lower);
|
||||
const shared = /your reel has been shared|reel shared|已分享|已发布/.test(lower) || /\\/reel\\//.test(url);
|
||||
const failed = !shared && !sharingVisible && (
|
||||
/couldn['’]t be shared|could not be shared|share failed|无法分享|分享失败/.test(lower)
|
||||
|| (/something went wrong/.test(lower) && /try again/.test(lower))
|
||||
);
|
||||
const composerOpen = dialogs.some((dialog) =>
|
||||
!!dialog.querySelector('textarea, [contenteditable="true"], input[type="file"]')
|
||||
|| /new reel|cover photo|trim|select from computer|crop|sharing/.test((dialog.textContent || '').toLowerCase())
|
||||
);
|
||||
const settled = !shared && !composerOpen && !sharingVisible;
|
||||
return { ok: shared, failed, settled, url: /\\/reel\\//.test(url) ? url : '' };
|
||||
})()
|
||||
`;
|
||||
}
|
||||
|
||||
async function waitForPublishSuccess(page: IPage): Promise<string> {
|
||||
let settledStreak = 0;
|
||||
for (let attempt = 0; attempt < 120; attempt += 1) {
|
||||
const result = await page.evaluate(buildReelPublishStatusProbeJs()) as { ok?: boolean; failed?: boolean; settled?: boolean; url?: string };
|
||||
if (result?.failed) {
|
||||
throw new CommandExecutionError('Instagram reel share failed');
|
||||
}
|
||||
if (result?.ok) {
|
||||
return result.url || '';
|
||||
}
|
||||
if (result?.settled) {
|
||||
settledStreak += 1;
|
||||
if (settledStreak >= 3) return '';
|
||||
} else {
|
||||
settledStreak = 0;
|
||||
}
|
||||
if (attempt < 119) await page.wait({ time: 1 });
|
||||
}
|
||||
throw new CommandExecutionError('Instagram reel share confirmation did not appear');
|
||||
}
|
||||
|
||||
async function resolveCurrentUserId(page: IPage): Promise<string> {
|
||||
const cookies = await page.getCookies({ domain: 'instagram.com' });
|
||||
return cookies.find((cookie: BrowserCookie) => cookie.name === 'ds_user_id')?.value || '';
|
||||
}
|
||||
|
||||
async function resolveProfileUrl(page: IPage, currentUserId = ''): Promise<string> {
|
||||
if (currentUserId) {
|
||||
const runtimeInfo = await resolveInstagramRuntimeInfo(page);
|
||||
const apiResult = await page.evaluate(`
|
||||
(async () => {
|
||||
const userId = ${JSON.stringify(currentUserId)};
|
||||
const appId = ${JSON.stringify(runtimeInfo.appId || '')};
|
||||
try {
|
||||
const res = await fetch(
|
||||
'https://www.instagram.com/api/v1/users/' + encodeURIComponent(userId) + '/info/',
|
||||
{
|
||||
credentials: 'include',
|
||||
headers: appId ? { 'X-IG-App-ID': appId } : {},
|
||||
},
|
||||
);
|
||||
if (!res.ok) return { ok: false };
|
||||
const data = await res.json();
|
||||
const username = data?.user?.username || '';
|
||||
return { ok: !!username, username };
|
||||
} catch {
|
||||
return { ok: false };
|
||||
}
|
||||
})()
|
||||
`) as { ok?: boolean; username?: string };
|
||||
|
||||
if (apiResult?.ok && apiResult.username) {
|
||||
return new URL(`/${apiResult.username}/`, INSTAGRAM_HOME_URL).toString();
|
||||
}
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
async function collectVisibleProfileMediaPaths(page: IPage): Promise<string[]> {
|
||||
const result = await page.evaluate(`
|
||||
(() => {
|
||||
const isVisible = (el) => {
|
||||
if (!(el instanceof HTMLElement)) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none'
|
||||
&& style.visibility !== 'hidden'
|
||||
&& rect.width > 0
|
||||
&& rect.height > 0;
|
||||
};
|
||||
const hrefs = Array.from(document.querySelectorAll('a[href*="/reel/"], a[href*="/p/"]'))
|
||||
.filter((el) => el instanceof HTMLAnchorElement && isVisible(el))
|
||||
.map((el) => el.getAttribute('href') || '')
|
||||
.filter((href) => /^\\/(?:[^/?#]+\\/)?(?:reel|p)\\/[^/?#]+\\/?$/.test(href))
|
||||
.filter((href, index, arr) => arr.indexOf(href) === index);
|
||||
return { hrefs };
|
||||
})()
|
||||
`) as { hrefs?: string[] };
|
||||
return Array.isArray(result?.hrefs) ? result.hrefs.filter(Boolean) : [];
|
||||
}
|
||||
|
||||
async function captureExistingProfileMediaPaths(page: IPage): Promise<Set<string>> {
|
||||
const currentUserId = await resolveCurrentUserId(page);
|
||||
if (!currentUserId) return new Set();
|
||||
const profileUrl = await resolveProfileUrl(page, currentUserId);
|
||||
if (!profileUrl) return new Set();
|
||||
try {
|
||||
await page.goto(profileUrl);
|
||||
await page.wait({ time: 3 });
|
||||
return new Set(await collectVisibleProfileMediaPaths(page));
|
||||
} catch {
|
||||
return new Set();
|
||||
}
|
||||
}
|
||||
|
||||
async function resolveLatestReelUrl(page: IPage, existingPaths: ReadonlySet<string>): Promise<string> {
|
||||
const currentUrl = await page.getCurrentUrl?.();
|
||||
if (currentUrl && /\/reel\//.test(currentUrl)) return currentUrl;
|
||||
|
||||
const currentUserId = await resolveCurrentUserId(page);
|
||||
const profileUrl = await resolveProfileUrl(page, currentUserId);
|
||||
if (!profileUrl) return '';
|
||||
|
||||
await page.goto(profileUrl);
|
||||
await page.wait({ time: 4 });
|
||||
|
||||
for (let attempt = 0; attempt < 8; attempt += 1) {
|
||||
const hrefs = await collectVisibleProfileMediaPaths(page);
|
||||
const href = hrefs.find((candidate) => candidate.includes('/reel/') && !existingPaths.has(candidate))
|
||||
|| hrefs.find((candidate) => !existingPaths.has(candidate))
|
||||
|| '';
|
||||
if (href) {
|
||||
return new URL(href, INSTAGRAM_HOME_URL).toString();
|
||||
}
|
||||
if (attempt < 7) await page.wait({ time: 1 });
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'instagram',
|
||||
name: 'reel',
|
||||
description: 'Post an Instagram reel video',
|
||||
domain: 'www.instagram.com',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
timeoutSeconds: INSTAGRAM_REEL_TIMEOUT_SECONDS,
|
||||
args: [
|
||||
{ name: 'video', required: false, valueRequired: true, help: 'Path to a single .mp4 video file' },
|
||||
{ name: 'content', positional: true, required: false, help: 'Caption text' },
|
||||
],
|
||||
columns: ['status', 'detail', 'url'],
|
||||
validateArgs: validateInstagramReelArgs,
|
||||
func: async (page: IPage | null, kwargs) => {
|
||||
const browserPage = requirePage(page);
|
||||
const videoPath = validateVideoPath(kwargs.video);
|
||||
const content = String(kwargs.content ?? '').trim();
|
||||
const preparedUpload = prepareVideoUpload(videoPath);
|
||||
|
||||
const run = async (
|
||||
activePage: IPage,
|
||||
existingMediaPaths: ReadonlySet<string> = new Set(),
|
||||
): Promise<InstagramReelSuccessRow[]> => {
|
||||
if (typeof activePage.startNetworkCapture === 'function') {
|
||||
await activePage.startNetworkCapture('/rupload_igvideo/|/api/v1/|/reel/|/clips/|/media/|/configure|/upload');
|
||||
}
|
||||
await gotoInstagramHome(activePage, true);
|
||||
await activePage.wait({ time: 2 });
|
||||
await dismissResidualDialogs(activePage);
|
||||
await ensureComposerOpen(activePage);
|
||||
await activePage.wait({ time: 2 });
|
||||
const selectors = await resolveUploadSelectors(activePage);
|
||||
let uploaded = false;
|
||||
let uploadError: unknown;
|
||||
for (const selector of selectors) {
|
||||
try {
|
||||
await uploadVideo(activePage, preparedUpload.uploadPath, selector);
|
||||
const selectedFileCount = await readSelectedFileCount(activePage, selector);
|
||||
if (selectedFileCount === 0) {
|
||||
throw new CommandExecutionError('Instagram reel upload failed', 'The selected reel input never received the video file');
|
||||
}
|
||||
await waitForVideoPreview(activePage, 10);
|
||||
uploaded = true;
|
||||
break;
|
||||
} catch (error) {
|
||||
uploadError = error;
|
||||
}
|
||||
}
|
||||
if (!uploaded) {
|
||||
throw uploadError instanceof Error
|
||||
? uploadError
|
||||
: new CommandExecutionError('Instagram reel preview did not appear after upload');
|
||||
}
|
||||
await clickActionMaybe(activePage, ['OK'], 'any');
|
||||
await clickAction(activePage, ['Next', '下一步'], 'media');
|
||||
await waitForReelStage(activePage, 'edit', 20);
|
||||
await clickAction(activePage, ['Next', '下一步'], 'media');
|
||||
await waitForReelStage(activePage, 'composer', 20);
|
||||
|
||||
if (content) {
|
||||
await fillCaption(activePage, content);
|
||||
await ensureCaptionFilled(activePage, content);
|
||||
}
|
||||
|
||||
await clickAction(activePage, ['Share', '分享'], 'caption');
|
||||
const sharedUrl = await waitForPublishSuccess(activePage);
|
||||
const url = sharedUrl || await resolveLatestReelUrl(activePage, existingMediaPaths);
|
||||
return buildInstagramReelSuccessResult(url);
|
||||
};
|
||||
|
||||
try {
|
||||
if (!process.env.VITEST) {
|
||||
const runIsolated = async (): Promise<InstagramReelSuccessRow[]> => {
|
||||
const isolatedPage = new BrowserPage(`site:instagram-reel-${Date.now()}`);
|
||||
try {
|
||||
return await run(isolatedPage, new Set());
|
||||
} finally {
|
||||
await isolatedPage.closeWindow?.();
|
||||
}
|
||||
};
|
||||
|
||||
try {
|
||||
return await runIsolated();
|
||||
} catch (error) {
|
||||
if (!isRecoverableReelSessionError(error)) throw error;
|
||||
return await runIsolated();
|
||||
}
|
||||
}
|
||||
|
||||
const existingMediaPaths = await captureExistingProfileMediaPaths(browserPage);
|
||||
return await run(browserPage, existingMediaPaths);
|
||||
} finally {
|
||||
if (preparedUpload.cleanupPath) {
|
||||
fs.rmSync(preparedUpload.cleanupPath, { force: true });
|
||||
}
|
||||
}
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,191 @@
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ArgumentError } from '../../errors.js';
|
||||
import { getRegistry } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import * as privatePublish from './_shared/private-publish.js';
|
||||
import './story.js';
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function createTempFile(name: string, bytes = Buffer.from('story-media')): string {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-instagram-story-'));
|
||||
tempDirs.push(dir);
|
||||
const filePath = path.join(dir, name);
|
||||
fs.writeFileSync(filePath, bytes);
|
||||
return filePath;
|
||||
}
|
||||
|
||||
function createPageMock(evaluateResults: unknown[] = [], overrides: Partial<IPage> = {}): IPage {
|
||||
const evaluate = vi.fn();
|
||||
for (const result of evaluateResults) {
|
||||
evaluate.mockResolvedValueOnce(result);
|
||||
}
|
||||
return {
|
||||
goto: vi.fn().mockResolvedValue(undefined),
|
||||
evaluate,
|
||||
getCookies: vi.fn().mockResolvedValue([]),
|
||||
snapshot: vi.fn().mockResolvedValue(undefined),
|
||||
click: vi.fn().mockResolvedValue(undefined),
|
||||
typeText: vi.fn().mockResolvedValue(undefined),
|
||||
pressKey: vi.fn().mockResolvedValue(undefined),
|
||||
scrollTo: vi.fn().mockResolvedValue(undefined),
|
||||
getFormState: vi.fn().mockResolvedValue({ forms: [], orphanFields: [] }),
|
||||
wait: vi.fn().mockResolvedValue(undefined),
|
||||
tabs: vi.fn().mockResolvedValue([]),
|
||||
closeTab: vi.fn().mockResolvedValue(undefined),
|
||||
newTab: vi.fn().mockResolvedValue(undefined),
|
||||
selectTab: vi.fn().mockResolvedValue(undefined),
|
||||
networkRequests: vi.fn().mockResolvedValue([]),
|
||||
consoleMessages: vi.fn().mockResolvedValue([]),
|
||||
scroll: vi.fn().mockResolvedValue(undefined),
|
||||
autoScroll: vi.fn().mockResolvedValue(undefined),
|
||||
installInterceptor: vi.fn().mockResolvedValue(undefined),
|
||||
getInterceptedRequests: vi.fn().mockResolvedValue([]),
|
||||
waitForCapture: vi.fn().mockResolvedValue(undefined),
|
||||
screenshot: vi.fn().mockResolvedValue(''),
|
||||
setFileInput: vi.fn().mockResolvedValue(undefined),
|
||||
insertText: vi.fn().mockResolvedValue(undefined),
|
||||
getCurrentUrl: vi.fn().mockResolvedValue(null),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
afterAll(() => {
|
||||
for (const dir of tempDirs) {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe('instagram story registration', () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('registers the story command with a required-value media arg', () => {
|
||||
const cmd = getRegistry().get('instagram/story');
|
||||
expect(cmd).toBeDefined();
|
||||
expect(cmd?.browser).toBe(true);
|
||||
expect(cmd?.args.some((arg) => arg.name === 'media' && !arg.required && arg.valueRequired)).toBe(true);
|
||||
expect(cmd?.args.some((arg) => arg.name === 'content')).toBe(false);
|
||||
});
|
||||
|
||||
it('rejects missing --media before browser work', async () => {
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/story');
|
||||
|
||||
await expect(cmd!.func!(page, {})).rejects.toThrow(ArgumentError);
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects multiple media inputs for a single story', async () => {
|
||||
const first = createTempFile('one.jpg');
|
||||
const second = createTempFile('two.mp4');
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/story');
|
||||
|
||||
await expect(cmd!.func!(page, { media: `${first},${second}` })).rejects.toThrow('single media');
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects unsupported story formats', async () => {
|
||||
const filePath = createTempFile('story.mov');
|
||||
const page = createPageMock();
|
||||
const cmd = getRegistry().get('instagram/story');
|
||||
|
||||
await expect(cmd!.func!(page, { media: filePath })).rejects.toThrow('Unsupported story media format');
|
||||
expect(page.goto).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('publishes a single image story through the private route', async () => {
|
||||
const imagePath = createTempFile('story.jpg');
|
||||
const page = createPageMock([
|
||||
{ appId: '936619743392459', csrfToken: '', instagramAjax: 'ajax' },
|
||||
{ ok: true, username: 'tsezi_ray' },
|
||||
], {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'ds_user_id', value: '123', domain: 'instagram.com' }]),
|
||||
});
|
||||
const cmd = getRegistry().get('instagram/story');
|
||||
|
||||
vi.spyOn(privatePublish, 'resolveInstagramPrivatePublishConfig').mockResolvedValue({
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'claim',
|
||||
instagramAjax: 'ajax',
|
||||
webSessionId: 'session',
|
||||
},
|
||||
jazoest: '22047',
|
||||
});
|
||||
vi.spyOn(privatePublish, 'publishStoryViaPrivateApi').mockResolvedValue({
|
||||
mediaPk: '1234567890',
|
||||
uploadId: '1234567890',
|
||||
});
|
||||
|
||||
const result = await cmd!.func!(page, { media: imagePath });
|
||||
|
||||
expect(privatePublish.publishStoryViaPrivateApi).toHaveBeenCalledWith(expect.objectContaining({
|
||||
page,
|
||||
mediaItem: { type: 'image', filePath: imagePath },
|
||||
content: '',
|
||||
}));
|
||||
expect(result).toEqual([
|
||||
{
|
||||
status: '✅ Posted',
|
||||
detail: 'Single story shared successfully',
|
||||
url: 'https://www.instagram.com/stories/tsezi_ray/1234567890/',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('publishes a single video story through the private route', async () => {
|
||||
const videoPath = createTempFile('story.mp4');
|
||||
const page = createPageMock([
|
||||
{ appId: '936619743392459', csrfToken: '', instagramAjax: 'ajax' },
|
||||
{ ok: true, username: 'tsezi_ray' },
|
||||
], {
|
||||
getCookies: vi.fn().mockResolvedValue([{ name: 'ds_user_id', value: '123', domain: 'instagram.com' }]),
|
||||
});
|
||||
const cmd = getRegistry().get('instagram/story');
|
||||
|
||||
vi.spyOn(privatePublish, 'resolveInstagramPrivatePublishConfig').mockResolvedValue({
|
||||
apiContext: {
|
||||
asbdId: '359341',
|
||||
csrfToken: 'csrf-token',
|
||||
igAppId: '936619743392459',
|
||||
igWwwClaim: 'claim',
|
||||
instagramAjax: 'ajax',
|
||||
webSessionId: 'session',
|
||||
},
|
||||
jazoest: '22047',
|
||||
});
|
||||
vi.spyOn(privatePublish, 'publishStoryViaPrivateApi').mockResolvedValue({
|
||||
mediaPk: '9988776655',
|
||||
uploadId: '9988776655',
|
||||
});
|
||||
|
||||
const result = await cmd!.func!(page, { media: videoPath });
|
||||
|
||||
expect(privatePublish.publishStoryViaPrivateApi).toHaveBeenCalledWith(expect.objectContaining({
|
||||
page,
|
||||
mediaItem: { type: 'video', filePath: videoPath },
|
||||
content: '',
|
||||
}));
|
||||
expect(result).toEqual([
|
||||
{
|
||||
status: '✅ Posted',
|
||||
detail: 'Single video story shared successfully',
|
||||
url: 'https://www.instagram.com/stories/tsezi_ray/9988776655/',
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,151 @@
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import { ArgumentError, CommandExecutionError } from '../../errors.js';
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
import type { IPage } from '../../types.js';
|
||||
import {
|
||||
publishStoryViaPrivateApi,
|
||||
resolveInstagramPrivatePublishConfig,
|
||||
} from './_shared/private-publish.js';
|
||||
import { resolveInstagramRuntimeInfo } from './_shared/runtime-info.js';
|
||||
|
||||
const INSTAGRAM_HOME_URL = 'https://www.instagram.com/';
|
||||
const SUPPORTED_STORY_IMAGE_EXTENSIONS = new Set(['.jpg', '.jpeg', '.png', '.webp']);
|
||||
const SUPPORTED_STORY_VIDEO_EXTENSIONS = new Set(['.mp4']);
|
||||
|
||||
type InstagramStoryMediaItem = {
|
||||
type: 'image' | 'video';
|
||||
filePath: string;
|
||||
};
|
||||
|
||||
type InstagramStorySuccessRow = {
|
||||
status: string;
|
||||
detail: string;
|
||||
url: string;
|
||||
};
|
||||
|
||||
function requirePage(page: IPage | null): IPage {
|
||||
if (!page) throw new CommandExecutionError('Browser session required for instagram story');
|
||||
return page;
|
||||
}
|
||||
|
||||
function validateInstagramStoryArgs(kwargs: Record<string, unknown>): void {
|
||||
if (kwargs.media === undefined) {
|
||||
throw new ArgumentError(
|
||||
'Argument "media" is required.',
|
||||
'Provide --media /path/to/file.jpg or --media /path/to/file.mp4',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeStoryMediaItem(kwargs: Record<string, unknown>): InstagramStoryMediaItem {
|
||||
const raw = String(kwargs.media ?? '').trim();
|
||||
const parts = raw.split(',').map((part) => part.trim()).filter(Boolean);
|
||||
if (parts.length === 0) {
|
||||
throw new ArgumentError(
|
||||
'Argument "media" is required.',
|
||||
'Provide --media /path/to/file.jpg or --media /path/to/file.mp4',
|
||||
);
|
||||
}
|
||||
if (parts.length > 1) {
|
||||
throw new ArgumentError(
|
||||
'Instagram story currently supports a single media item.',
|
||||
'Provide one image or one video path with --media',
|
||||
);
|
||||
}
|
||||
|
||||
const resolved = path.resolve(parts[0]!);
|
||||
if (!fs.existsSync(resolved)) {
|
||||
throw new ArgumentError(`Story media file not found: ${resolved}`);
|
||||
}
|
||||
const ext = path.extname(resolved).toLowerCase();
|
||||
if (SUPPORTED_STORY_IMAGE_EXTENSIONS.has(ext)) {
|
||||
return { type: 'image', filePath: resolved };
|
||||
}
|
||||
if (SUPPORTED_STORY_VIDEO_EXTENSIONS.has(ext)) {
|
||||
return { type: 'video', filePath: resolved };
|
||||
}
|
||||
throw new ArgumentError(
|
||||
`Unsupported story media format: ${ext}`,
|
||||
'Supported formats: images (.jpg, .jpeg, .png, .webp) and videos (.mp4)',
|
||||
);
|
||||
}
|
||||
|
||||
async function resolveCurrentUserId(page: IPage): Promise<string> {
|
||||
const cookies = await page.getCookies({ domain: 'instagram.com' });
|
||||
return cookies.find((cookie) => cookie.name === 'ds_user_id')?.value || '';
|
||||
}
|
||||
|
||||
async function resolveCurrentUsername(page: IPage, currentUserId = ''): Promise<string> {
|
||||
if (!currentUserId) return '';
|
||||
const runtimeInfo = await resolveInstagramRuntimeInfo(page);
|
||||
const apiResult = await page.evaluate(`
|
||||
(async () => {
|
||||
const userId = ${JSON.stringify(currentUserId)};
|
||||
const appId = ${JSON.stringify(runtimeInfo.appId || '')};
|
||||
try {
|
||||
const res = await fetch(
|
||||
'https://www.instagram.com/api/v1/users/' + encodeURIComponent(userId) + '/info/',
|
||||
{
|
||||
credentials: 'include',
|
||||
headers: appId ? { 'X-IG-App-ID': appId } : {},
|
||||
},
|
||||
);
|
||||
if (!res.ok) return { ok: false };
|
||||
const data = await res.json();
|
||||
const username = data?.user?.username || '';
|
||||
return { ok: !!username, username };
|
||||
} catch {
|
||||
return { ok: false };
|
||||
}
|
||||
})()
|
||||
`) as { ok?: boolean; username?: string };
|
||||
|
||||
return apiResult?.ok && apiResult.username ? apiResult.username : '';
|
||||
}
|
||||
|
||||
function buildStorySuccessResult(mediaItem: InstagramStoryMediaItem, url: string): InstagramStorySuccessRow[] {
|
||||
return [{
|
||||
status: '✅ Posted',
|
||||
detail: mediaItem.type === 'video'
|
||||
? 'Single video story shared successfully'
|
||||
: 'Single story shared successfully',
|
||||
url,
|
||||
}];
|
||||
}
|
||||
|
||||
cli({
|
||||
site: 'instagram',
|
||||
name: 'story',
|
||||
description: 'Post a single Instagram story image or video',
|
||||
domain: 'www.instagram.com',
|
||||
strategy: Strategy.UI,
|
||||
browser: true,
|
||||
timeoutSeconds: 300,
|
||||
args: [
|
||||
{ name: 'media', required: false, valueRequired: true, help: 'Path to a single story image or video file' },
|
||||
],
|
||||
columns: ['status', 'detail', 'url'],
|
||||
validateArgs: validateInstagramStoryArgs,
|
||||
func: async (page: IPage | null, kwargs) => {
|
||||
const browserPage = requirePage(page);
|
||||
const mediaItem = normalizeStoryMediaItem(kwargs as Record<string, unknown>);
|
||||
const currentUserId = await resolveCurrentUserId(browserPage);
|
||||
const privateConfig = await resolveInstagramPrivatePublishConfig(browserPage);
|
||||
const storyResult = await publishStoryViaPrivateApi({
|
||||
page: browserPage,
|
||||
mediaItem,
|
||||
content: '',
|
||||
apiContext: privateConfig.apiContext,
|
||||
jazoest: privateConfig.jazoest,
|
||||
currentUserId,
|
||||
});
|
||||
const username = await resolveCurrentUsername(browserPage, currentUserId);
|
||||
const mediaPk = storyResult.mediaPk || storyResult.uploadId;
|
||||
const url = username && mediaPk
|
||||
? new URL(`/stories/${username}/${mediaPk}/`, INSTAGRAM_HOME_URL).toString()
|
||||
: '';
|
||||
return buildStorySuccessResult(mediaItem, url);
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,68 @@
|
||||
/**
|
||||
* Sinafinance stock rank
|
||||
*/
|
||||
|
||||
import { cli, Strategy } from '../../registry.js';
|
||||
|
||||
cli({
|
||||
site: 'sinafinance',
|
||||
name: 'stock-rank',
|
||||
description: '新浪财经热搜榜',
|
||||
domain: 'finance.sina.cn',
|
||||
strategy: Strategy.COOKIE,
|
||||
navigateBefore: false,
|
||||
args: [
|
||||
{ name: 'market', type: 'string', default: 'cn', choices: ['cn', 'hk', 'us', 'wh', 'ft'], help: 'Market: cn (A股), hk (港股), us (美股), wh (外汇), ft (期货)' },
|
||||
],
|
||||
columns: ['rank', 'name', 'symbol', 'market', 'price', 'change', 'url'],
|
||||
func: async (page, _args) => {
|
||||
const market = _args.market || 'cn';
|
||||
|
||||
await page.goto('https://finance.sina.cn/');
|
||||
await page.wait({ selector: '#actionSearch', timeout: 10000 });
|
||||
|
||||
const payload = await page.evaluate(`
|
||||
(async () => {
|
||||
const wait = (ms) => new Promise(r => setTimeout(r, ms));
|
||||
const cleanText = (value) => (value || '').replace(/\\s+/g, ' ').trim();
|
||||
const marketType = ${JSON.stringify(market)};
|
||||
|
||||
const searchBtn = document.querySelector('#actionSearch');
|
||||
if (searchBtn) {
|
||||
searchBtn.dispatchEvent(new Event('tap', { bubbles: true }));
|
||||
await wait(3000);
|
||||
}
|
||||
|
||||
const tabEl = document.querySelector('[data-type="' + marketType + '"]');
|
||||
const marketName = tabEl?.textContent || marketType;
|
||||
if (marketType !== 'cn' && tabEl) {
|
||||
tabEl.click();
|
||||
await wait(2000);
|
||||
}
|
||||
|
||||
const results = [];
|
||||
document.querySelectorAll('#stock-list .j-stock-row').forEach(el => {
|
||||
const rankEl = el.querySelector('.rank');
|
||||
const nameEl = el.querySelector('.j-sname');
|
||||
const codeEl = el.querySelector('.stock-code');
|
||||
const priceEl = el.querySelector('.j-price');
|
||||
const changeEl = el.querySelector('.j-change');
|
||||
const openUrl = el.getAttribute('open-url') || '';
|
||||
const fullUrl = openUrl ? 'https:' + openUrl : '';
|
||||
results.push({
|
||||
rank: cleanText(rankEl?.textContent || ''),
|
||||
name: cleanText(nameEl?.textContent || ''),
|
||||
symbol: cleanText(codeEl?.textContent || ''),
|
||||
market: cleanText(marketName),
|
||||
price: cleanText(priceEl?.textContent || ''),
|
||||
change: cleanText(changeEl?.textContent || ''),
|
||||
url: fullUrl,
|
||||
});
|
||||
});
|
||||
return results;
|
||||
})()
|
||||
`);
|
||||
if (!Array.isArray(payload)) return [];
|
||||
return payload;
|
||||
},
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user