Compare commits

...

57 Commits

Author SHA1 Message Date
jakevin 60c92e1150 chore: bump version to 1.6.8 (#825)
Build Chrome Extension / build (push) Has been cancelled
Release / release (push) Has been cancelled
2026-04-06 03:24:52 +08:00
jakevin 009c25b955 refactor: remove scoring heuristic, use noise filter + structured metadata (#824)
* refactor: remove scoring heuristic, replace with noise filter + metadata

The scoring mechanism was a pre-LLM heuristic that compressed rich endpoint
metadata into a single number. Since this project is designed for AI Agents,
the agent can reason about structured metadata directly.

Changes:
- Remove scoreEndpoint/scoreRequest/scoreWriteRequest and all score fields
- Replace with isNoiseUrl() filter (tracking/beacon/pixel) + isUsefulEndpoint()
- Remove artificial confidence percentages (was score/20)
- Sort by itemCount (transparent, observable) instead of weighted score
- Endpoints now expose full structured metadata for agent consumption
- Net reduction: -43 lines

* fix: widen endpoint filter to keep single-object JSON and stats/metric URLs

- Remove stats/metric from noise pattern — these are often business APIs
- Relax isUsefulEndpoint to keep any JSON endpoint, not just arrays
  (preserves /me, /profile, /detail and other single-object APIs)

* fix: add deterministic endpoint ordering for generate/synthesize path

The AI agent path doesn't need ranking, but generate/synthesize still
pick candidates[0] as default — this needs a stable, explainable order.

- Add endpointSortKey() with transparent observable signals: array items,
  detected fields, API path patterns, query params
- Update synthesize chooseEndpoint fallback to use itemCount + field count
- Sort key is internal only; not exposed as score to external consumers
2026-04-06 03:21:15 +08:00
tiaot33 5553300597 feat(linux-do): split topic content into a dedicated command (#821)
* feat(linux-do): split topic content into a dedicated command

Move the old main-post path out of linux-do topic so topic stays a summarized first-page reader while topic-content becomes the Markdown-focused entrypoint for full post bodies.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* feat(linux-do): update topic content handling to include YAML front matter

* fix(linux-do): update default output format to plain for topic-content rendering

* fix(linux-do): replace js-yaml with inline YAML serialization for topic-content

Adapters must only import node builtins, relative modules, or opencli
public APIs. Hand-roll the simple front matter serialization to remove
the third-party js-yaml dependency.

* fix(linux-do): refine YAML quoting to only escape colons followed by space

Colons in URLs (e.g. https://) are valid unquoted YAML values. Only
quote when a colon is followed by a space or appears at end of line.

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-06 02:59:56 +08:00
jakevin d61dd7be0e refactor: extract shared scoring + consolidate time utils (#823)
* refactor: extract shared scoring logic and consolidate time format utils

- Extract applyUrlScoreAdjustments() and scoreArrayResponse() to analysis.ts,
  eliminating duplicated endpoint scoring between explore.ts and record.ts
- Consolidate formatDuration/formatUptime into a single formatDuration(ms)
  in download/progress.ts, reused by commands/daemon.ts

* fix: preserve explore scoring semantics and round daemon uptime

- Revert explore.ts scoreEndpoint to original inline /api/ /x/ bonus
  without record's tracking/analytics penalty (blocker from review)
- Math.round uptime*1000 to avoid floating-point noise in daemon status
2026-04-06 02:50:37 +08:00
jakevin e9867dcab0 feat(extension): v1.6.8 — fix scripting permission + refresh icons (#822)
* feat(extension): v1.6.8 — remove unused scripting permission, refresh icons

- Remove unused `scripting` permission (Chrome Web Store rejection fix)
- Bump version 1.6.7 → 1.6.8
- Redesign icons with neon gradient style (pure SVG paths, glow effect)

* revert icons to original, fix package.json version to 1.6.8

* test: remove stale scripting permission assertion
2026-04-06 02:07:00 +08:00
williamxie1989 bb7b26bc4a feat(xueqiu): add kline and groups adapters (#809)
* feat(xueqiu): add kline and groups adapters

Add kline.yaml: fetch candlestick/OHLCV data from Xueqiu v5 chart API.
Supports custom days lookback and outputs date, open, high, low, close,
volume, percent.

Add groups.yaml: list Xueqiu portfolio/group entries.


* fix(xueqiu): correct groups.yaml to use /portfolio/list.json API

The previous implementation used /portfolio/stock/list.json which only
returns stocks in a single group and does not return the group list.
Switch to /portfolio/list.json which returns all portfolio groups
including 实盘, 沪深, 港股, 美股, 模拟(pid=-4), 持仓 etc.


* fix(xueqiu): replace watchlist category param with pid selector

- Remove the unused 'category' parameter (the API ignores it;
  all groups live under category=1 regardless)
- Replace with 'pid' parameter to allow fetching any group:
  -4=simulated, -5=SH/SZ, -6=US stocks, -7=HK stocks, etc.
- API path still uses category=1 but pid is now user-controllable


---------
2026-04-06 02:01:29 +08:00
BruceLoveDecimal d2ae28786c feat:add 1688 assets downloadable (#820)
* feat:add 1688 assets downloadable

* fix: remove duplicate visitedRoots declaration and fix normalizeMediaUrl re-export

- Remove unused outer `visitedRoots` variable in scriptToReadAssets()
- Move normalizeMediaUrl test to import from shared.ts where it's defined
- Remove unused normalizeMediaUrl import from assets.ts

---------

Co-authored-by: 刘启灏 <liuqihao@liuqihaodeMacBook-Pro.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-06 01:59:39 +08:00
jakevin 1abff0cc1d feat(operate): unify network capture + implement CDP consoleMessages (#816)
* feat(operate): unify network capture + implement CDP consoleMessages

- operate open: start session capture before navigation (catches initial requests)
- operate network: prefer readNetworkCapture() over JS interceptor
- CDPPage: implement consoleMessages() via Runtime.consoleAPICalled

Part of #810

* fix(operate): use correct daemon/CDP entry field names for network capture

Daemon and CDP capture entries use responseStatus/responseContentType/
responsePreview (not status/contentType/responseBody). Fix the
normalization in operate network to match the actual entry shape from
extension/src/cdp.ts.

* fix(cdp): capture Runtime.exceptionThrown in consoleMessages

- Register Runtime.exceptionThrown handler to capture uncaught exceptions
  as error-level messages (most valuable diagnostic signal)
- 'error' filter now returns both console.error() and warning/exception
  entries, matching typical severity-based logging semantics
2026-04-05 22:55:03 +08:00
jakevin 1d46c5f934 feat(cdp): implement session-level network capture for CDPPage (#815)
* feat(cdp): implement session-level network capture for CDPPage

Implements startNetworkCapture() and readNetworkCapture() on CDPPage using
CDP Network domain events. Updates explore.ts to prefer session capture
over Performance API networkRequests().

Closes part of #810

* fix(cdp): use Network.loadingFinished for reliable body capture

- Move getResponseBody call from responseReceived to loadingFinished,
  matching the extension's implementation pattern
- Use extension-compatible entry shape (responseStatus, responseContentType,
  responsePreview) instead of custom field names
- Remove unreliable 100ms sleep hack in readNetworkCapture()
- Align with extension/src/cdp.ts:419-437 for consistency

* fix(cdp): drain buffer on readNetworkCapture to match daemon contract

readNetworkCapture() must clear the buffer after reading, matching the
daemon Page's read-and-drain behavior. Without this, repeated reads
would return stale entries.

* fix(cdp): await in-flight body fetches before returning from readNetworkCapture

Track all pending getResponseBody promises and await them in
readNetworkCapture() before draining the buffer. This ensures
explore/diagnostic consumers always get entries with responsePreview
populated, not empty shells where the body fetch hasn't resolved yet.

* fix(explore): handle both legacy and capture entry field names

parseNetworkRequests now maps both shapes:
- Legacy: status, contentType, responseBody
- Capture (extension/CDP): responseStatus, responseContentType, responsePreview

Also clears _pendingBodyFetches on startNetworkCapture reset.
2026-04-05 22:50:55 +08:00
jakevin d7fe7a7ffa fix(scaffold): replace non-existent extract step with select in YAML template (#814) 2026-04-05 22:50:43 +08:00
jakevin 7b55b8c595 test: remove flaky bloomberg e2e tests (#818)
* test: remove flaky bloomberg e2e tests

Bloomberg RSS endpoints are unreliable in CI, causing intermittent
e2e-headed failures unrelated to code changes.

* test: remove flaky bloomberg e2e tests

Bloomberg RSS feeds are unreliable in CI, causing false failures
in e2e-headed runs. Remove bloomberg tests from both
public-commands.test.ts and browser-public-extended.test.ts.
2026-04-05 22:50:24 +08:00
jakevin 15268da8f3 fix: add safety boundaries to diagnostic output (#806)
* fix: add safety boundaries to diagnostic output

- Redact sensitive headers (Authorization, Cookie, etc.) from network requests
- Redact sensitive URL query parameters (token, key, secret, etc.)
- Cap individual fields: snapshot (100K chars), adapter source (50K chars),
  network requests (50 entries, 4K body each), stack trace (5K chars)
- Enforce 256KB total output budget with graceful degradation:
  drops snapshot first, then page state entirely
- Export truncate/redactUrl helpers for testing

* fix: add free-text redaction for all diagnostic string channels

Addresses review feedback: snapshot, consoleErrors, error message/hint/stack
could contain inline secrets (Bearer tokens, JWTs, cookie values, token=value
patterns). All string channels now pass through redactText() before emission.

- Add redactText() with patterns for Bearer tokens, JWTs, cookie values,
  and inline key=value secrets
- Apply redactText to: error.message, error.hint, error.stack,
  page.snapshot, page.consoleErrors
- Add 6 new test cases for redactText and error message redaction

* fix: resolve adapter source path and add page state collection timeout

Fixes #808 items 1 and 3:

1. adapter.source was missing for all command types because buildRepairContext
   only checked cmd._modulePath (set only for manifest lazy-loaded TS).
   Now resolveAdapterSourcePath() checks cmd.source first, skips manifest:
   pseudo-paths, and maps dist/clis/*.js back to source clis/*.ts.

3. collectPageState() had no timeout — a hung CDP connection would block
   error propagation indefinitely. Now wrapped with 5s Promise.race timeout,
   falling back to emitting diagnostic without page state.

* fix: track sourceFile in manifest for YAML adapter source resolution

YAML commands inlined in the manifest previously lost their original file
path, causing resolveAdapterSourcePath() to return undefined. Add
sourceFile field to ManifestEntry so discovery can reconstruct the
editable source path for both YAML and TS commands.
2026-04-05 19:50:03 +08:00
jakevin a3efdc16de refactor: centralize build path resolution (#807) 2026-04-05 19:46:58 +08:00
jakevin d51338cbf3 chore: bump version to 1.6.7
Build Chrome Extension / build (push) Has been cancelled
Release / release (push) Has been cancelled
2026-04-05 18:07:55 +08:00
jakevin 664a971ed5 feat: structured diagnostic output for AI-driven adapter repair (#802)
* feat: add structured diagnostic output for AI-driven adapter repair

When OPENCLI_DIAGNOSTIC=1 is set, failed commands emit a RepairContext
JSON to stderr containing the error, adapter source, and browser state
(DOM snapshot, network requests, console errors). AI Agents consume
this to diagnose and fix adapters when websites change.

Also adds the opencli-repair skill guide for AI Agents.

* fix: correct e2e test binary path to dist/src/main.js

The e2e helpers pointed to dist/main.js but the actual build output
is at dist/src/main.js (matching package.json "main" field). This
caused all e2e-headed tests to fail with "Cannot find module".

* fix: correct dist/main.js path in autoresearch scripts

* fix: emit diagnostic for pre-session browser failures

When browser connection fails before the session callback runs
(e.g., BrowserConnectError), the inner diagnostic catch never fires.
Use a flag to ensure the outer catch emits diagnostic as a fallback.

* test: tolerate unavailable Bloomberg RSS feeds in e2e

* test: skip flaky bloomberg businessweek e2e test

The Bloomberg Businessweek RSS feed is intermittently unavailable,
causing CI failures unrelated to code changes.

* revert: restore bloomberg businessweek e2e coverage
2026-04-05 18:04:49 +08:00
Kai 97a547c6c5 fix: avoid inserting completion config inside multi-line shell commands (#796)
* fix: avoid inserting completion config inside multi-line shell commands

The postinstall zshrc insertion logic splits backslash-continued blocks
(e.g. zinit stanzas) when it finds a compinit match inside them, which
breaks the user's shell config. Walk backward past continuation lines
so the insertion lands before the entire logical command.

* fix: append zsh completion to end of .zshrc instead of splicing

Replace the fragile compinit-searching splice logic with a simple
append, matching the strategy already used for bash. This avoids
breaking multi-line commands (e.g. zinit blocks with zicompinit).

Still detects existing compinit to avoid adding a duplicate call.

* fix: stop modifying shell rc files in postinstall

Replace the fragile .zshrc/.bashrc modification logic with a safer
approach: only write completion files and print setup instructions.

The previous approach tried to parse and splice into rc files, which
broke multi-line shell commands (e.g. zinit blocks with backslash
continuations matching /compinit/). Instead of attempting to fix the
parser, remove rc modification entirely — this matches the approach
used by rustup, homebrew, and other CLI tools.

Closes #788

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-05 17:39:23 +08:00
jakevin eedc47aa26 chore: bump version to 1.6.6
Build Chrome Extension / build (push) Has been cancelled
Release / release (push) Has been cancelled
2026-04-05 17:34:57 +08:00
jakevin 8c41411860 fix: route copied adapters through opencli exports
* fix(discovery): expose runtime deps to user adapters

* fix: route copied adapters through opencli exports

* refactor: route adapter status output through logger
2026-04-05 17:31:45 +08:00
jakevin ed69e839ab chore: bump version to 1.6.5 (#797)
Build Chrome Extension / build (push) Has been cancelled
Release / release (push) Has been cancelled
2026-04-05 16:07:02 +08:00
jakevin 4fe9a73ebc refactor: migrate adapter imports to package exports (#795)
* refactor: migrate adapter imports to package exports

Replace all relative imports (../../src/registry.js, ../../browser/cdp.js, etc.)
with package exports (@jackwener/opencli/registry, @jackwener/opencli/errors, etc.)
across all 484 adapter files.

This decouples adapter import resolution from directory structure:
- User CLIs in ~/.opencli/clis/ resolve via node_modules symlink
- Internal adapters resolve via Node.js self-referencing
- No more shim files needed for import resolution

Changes:
- package.json: add sub-path exports for all public modules
- clis/**: replace relative imports with @jackwener/opencli/...
- discovery.ts: simplify ensureUserCliCompatShims to symlink-only
- registry-api.ts: export CommandArgs type
- Remove root-level shim directories (browser/, download/, pipeline/)
- Remove shim entries from tsconfig.json include and package.json files

* test: add regression tests for package exports

Prevents regressions like #788/#791 by:
1. Scanning all adapter files for forbidden relative imports
   (../../src/, ../../browser/, etc.) — fails if any remain
2. Verifying every package.json export maps to an existing source file

18 new test cases.

* fix: use junction on Windows + broaden test patterns

- discovery.ts: use 'junction' symlink type on Windows (no admin required)
- package-exports.test.ts: generalize forbidden patterns to catch any
  depth of ../ traversal (not just ../../ and ../../../)

* fix: update stale vi.mock/importActual paths in adapter tests

Test files still used old relative paths for vi.mock() and
vi.importActual() calls. Updated 5 test files to use package exports.
Also broadened regression test patterns to catch mock/importActual paths.

* fix: use rm instead of unlink for symlink cleanup, add warn on failure

Addresses review feedback from Astro-Han:
- rm() handles both symlinks and stale directories (unlink fails on dirs)
- Log a warning when symlink creation fails instead of silent catch

* docs: update import examples to use package exports

Update all documentation, contributing guides, and skills to use
@jackwener/opencli/registry instead of ../../src/registry.js.

Without this, users following the docs would write adapters with
broken imports since the old shim files are no longer created.
2026-04-05 16:02:17 +08:00
jakevin a1dd817886 chore: bump version to 1.6.4 (#794)
Build Chrome Extension / build (push) Has been cancelled
Release / release (push) Has been cancelled
2026-04-05 14:57:45 +08:00
AstroHan 20957cbc7b fix: resolve version 0.0.0 and user CLI load failures (#788) (#789)
Bug 1: version.ts used a single-level parent lookup for package.json,
which broke after #784 changed rootDir from "src" to "." (version.js
now lives in dist/src/ instead of dist/).  Walk up until package.json
is found — works in both dev (src/) and prod (dist/src/).

Bug 2: adapters copied to ~/.opencli/clis/ import ../../src/registry.js
etc., which resolves to ~/.opencli/src/.  Derive src/ compat shims from
the existing rootShims list so these imports resolve correctly.
2026-04-05 14:54:30 +08:00
jakevin ab58aa4098 fix(docs): update outdated paths and command lists across all docs (#787)
- Replace 140 instances of `src/clis/` → `clis/` across 12 doc files
  (path changed after repo restructure)
- Remove non-command `rpc` and `rankings` from notebooklm/amazon
  command lists in README, README.zh-CN, SKILL.md, and adapters index
  (these are internal utility modules, not user-facing commands)
- Add `deep-research` and `deep-research-result` to gemini adapter doc
- Add `movers-shakers` and `new-releases` to amazon adapter doc
- Update notebooklm doc examples to use canonical commands instead of
  deprecated aliases (`metadata` → `get`, `notes-list` → `note-list`)
- Bump SKILL.md version to 1.6.3
2026-04-05 03:53:52 +08:00
jakevin 76b5d53bb5 chore: bump version to 1.6.3 (#786)
Build Chrome Extension / build (push) Has been cancelled
Release / release (push) Has been cancelled
2026-04-05 03:46:13 +08:00
jakevin 00f6062e74 docs: fix outdated commands and adapter counts (#785)
* docs: fix outdated commands and adapter counts in README and skills

- Add gemini deep-research and deep-research-result commands
- Fix notebooklm: remove non-existent select/metadata/notes-list, add rpc
- Add amazon movers-shakers, new-releases, rankings commands
- Add missing weibo commands in zh-CN README
- Fix linux-do missing hot/latest/category in zh-CN README
- Update adapter count from 73+ to 79+
- Update skills version to 1.6.2
- Add full spotify command list in skills SKILL.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(test): update xiaohongshu note tests for search_result URL change

buildNoteUrl now uses /search_result/<id> instead of /explore/<id> for
bare note IDs. Update test expectations to match:
- buildNoteUrl test: expect /search_result/ not /explore/
- goto URL assertion: expect /search_result/ not /explore/
- empty shell hint: match actual error message text

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(test): update xiaohongshu comments test for search_result URL change

- Bare note ID now navigates to /search_result/ not /explore/
- Full URL inputs are preserved as-is (including /explore/ URLs)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 03:39:50 +08:00
jakevin 12176de1a5 refactor: simplify core modules (#784)
* refactor: simplify core modules — remove root shims, consolidate error classification, streamline cascade/interceptor, clean up synthesize

1. Remove root-level shim files (errors.ts, logger.ts, registry.ts, types.ts, utils.ts, launcher.ts) — update all ~840 adapter imports to reference src/ directly
2. Consolidate interceptor: reuse shared DISGUISE_FN in tap interceptor instead of reimplementing
3. Unify error classification: single ClassifiedError type with icon/exitCode/hint lookup table, eliminating duplicated pattern matching between resolveExitCode and renderError
4. Simplify cascade probe: replace repetitive switch cases with PROBE_OPTIONS lookup map
12. Clean up synthesize.ts: remove deprecated snake_case field aliases (recommended_args, recommended_columns, recommendedColumnsLegacy) and unnecessary constant aliases

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: update import paths in contributor docs and skill templates

Update all documentation and skill files to reference src/ directly,
matching the shim removal in the previous commit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update new Gemini adapter imports to use src/ paths

Fix imports in newly added deep-research adapter files that were
still referencing the deleted root shim files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update xiaohongshu tests for /search_result/ URL change

Tests now expect /search_result/<id> for bare note IDs (matching
the note-helpers.ts change from PR #774) and updated empty-shell
hint assertion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update LessWrong and hupu adapter imports to use src/ paths

Fix imports in newly merged LessWrong and hupu/mentions adapter
files that were still referencing the deleted root shim files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(xueqiu): mock logger via src path

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 03:35:12 +08:00
kevin.zhang 8bd36aaa37 fix(hupu): add mentions command (#757)
* feat(hupu): add hupu cli adapter

* fix(hupu): prevent detail from returning the wrong thread

* refactor: deduplicate shared utilities in hupu adapter

- Merge postHupuJson and postHupuReplyJson into single function with mode parameter
- Move stripHtml and decodeHtmlEntities to utils.ts, remove duplicate definitions

* fix(hupu): add mentions command

* fix: move mentions.ts to clis/hupu/, remove src/clis/hupu duplicates

Post PR #782 restructure: adapter files live at root clis/, not src/clis/.

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-05 03:26:40 +08:00
Xule Lin d2051cdab7 feat(lesswrong): add LessWrong adapter (#773)
* feat(lesswrong): add LessWrong adapter

15 commands for the LessWrong rationality and AI alignment community:
- Post listings: curated, frontpage, new, top, top-week/month/year
- Content: read (full post), comments, shortform (quick takes)
- Discovery: tag, tags, sequences
- Users: user (profile), user-posts

All commands use the public GraphQL API (no browser required).
Time-filtered views use the `after` date parameter.
Tag lookup resolves slugs to IDs via the `tagBySlug` view.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: move lesswrong adapter to clis/ (post PR #782 restructure)

New adapter files were created at src/clis/lesswrong/ but PR #782 moved
all adapters to root clis/. Move to correct location.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-05 03:21:48 +08:00
kingOfSoySauce cfb915b1d7 fix(xiaohongshu): use /search_result/<id> for bare note IDs (#774)
XHS blocks /explore/<id> without a valid xsec_token, causing code 66
(empty result) when passing bare note IDs. The /search_result/<id> path
works without xsec_token when the user is logged in via cookies.

Changes:
- note-helpers.ts: buildNoteUrl now uses /search_result/<id> for bare IDs
- note.ts: remove isBareNoteId branching and simplify empty shell error
2026-04-05 03:14:49 +08:00
backtomyfuture 4e2b314930 feat(gemini): add deep-research workflow and docs export result (#778)
* feat(gemini): add deep-research workflow and docs export result

* fix(gemini): improve deep-research submit and confirm flow

* fix(gemini): return waiting state when deep research is in progress

* fix(gemini): avoid false submit detection on root app transcript changes

* fix(gemini): return pending state when deep-research export is not ready

---------

Co-authored-by: f1480022 <f148002@163.com>
2026-04-05 03:09:46 +08:00
gucasbrg a0a4dd68ef fix(36kr): replace waitForCapture with DOM polling for search/hot (#779)
* fix(36kr): replace waitForCapture with DOM polling for search/hot

waitForCapture(6) always times out on 36kr because the API intercept
never captures a matching request. However, the DOM is already fully
rendered with search/hot results by the time the timeout fires.

Replace the 6-second intercept wait with a DOM polling loop that checks
for article links (a[href*="/p/"]) every 300ms, returning immediately
once content is available (typically ~1s vs 6s timeout + error).

Tested on opencli 1.6.2 with both CDP and Browser Bridge modes.

* fix: rebase onto main, remove unused interceptor, fix strategy

- Rebase onto main after clis/ move (PR #782)
- Remove installInterceptor calls (no longer used after waitForCapture removal)
- Change strategy from INTERCEPT to PUBLIC (browser: true) to match actual behavior
- Improve polling loop readability

---------

Co-authored-by: buruguo <buruguo@lambdafintech.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-05 03:04:37 +08:00
Inori333 97ae87ccee fix(cli): make operate verify work in source checkouts (#777)
* fix(cli): make operate verify work in source checkouts

* fix(cli): resolve operate verify entry from package metadata

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-05 02:56:10 +08:00
Howard 174ef75a54 docs: add Android Chrome usage guide (#687) 2026-04-05 02:53:10 +08:00
jakevin 639a31fc84 fix: review follow-ups for monorepo adapter separation (#783)
* fix: review follow-ups — better first-run log, OPENCLI_FETCH=1 skips version check

- Clarify first-run log message: "copying adapters (one-time setup)"
- Add comment explaining why scriptPath uses two levels of ../
- OPENCLI_FETCH=1 now bypasses version-skip to allow forced refresh

* fix: update doc-coverage script path after clis/ move

check-doc-coverage.sh still referenced src/clis/ after PR #782 moved
adapters to root clis/. This caused CI to fail with "0/1 documented".

* fix: resolve package root dynamically for symlink and first-run paths

The symlink at ~/.opencli/node_modules/@jackwener/opencli pointed to
dist/ instead of the package root in prod mode, breaking user TS CLIs
that import from '@jackwener/opencli/registry'.

The first-run scriptPath also resolved incorrectly in dev mode.

Extract findPackageRoot() that walks up to find package.json, fixing
both paths for dev (src/) and prod (dist/src/) layouts.
2026-04-05 02:25:53 +08:00
jakevin 80eef46b4e refactor: monorepo adapter separation (clis/ at root) (#782)
* refactor: move adapters from src/clis/ to root clis/ for monorepo separation

Separates CLI adapters from the core runtime to prepare for independent
adapter distribution via postinstall fetch.

Key changes:
- Move src/clis/ → clis/ (adapters at repo root)
- Change tsconfig rootDir from "src" to "." so tsc compiles both
- Create root-level shim files (registry.ts, errors.ts, etc.) so adapter
  relative imports (../../registry.js) resolve correctly
- Update build-manifest.ts, main.ts paths for new dist/src/ structure
- Expand ensureUserCliCompatShims() to cover all adapter import targets
  (types, utils, logger, launcher, browser/*, download/*, pipeline/*)
- Add scripts/fetch-adapters.js postinstall for ~/.opencli/clis/ sync
- Update vitest.config.ts adapter test paths
- Add package.json files field to exclude adapters from npm package

Official adapter files are unconditionally overwritten on update;
user-created files not in the manifest are preserved.

* fix: add dist/clis/ and cli-manifest.json to npm files, harden fetch-adapters

- Add dist/clis/ and dist/cli-manifest.json to package.json files field
  so built-in adapters and manifest ship with the npm package
- Replace execSync with execFileSync to prevent command injection
- Add version check to skip redundant adapter fetches
- Track tmpRoot explicitly for reliable cleanup

* fix: address review blockers — manifest-based updates, global-only fetch, first-run fallback

1. Manifest-based update strategy:
   - Read old manifest to identify previously-official files
   - Clean up files removed upstream (in old manifest but not new)
   - User-created files (never in any manifest) remain untouched

2. Only run fetch-adapters on global install (npm_config_global=true)
   or explicit OPENCLI_FETCH=1, preventing heavy side effects for
   local/dev installs

3. First-run fallback in discovery.ts:
   - ensureUserAdapters() checks for adapter-manifest.json
   - If missing and ~/.opencli/clis/ is empty, spawns fetch-adapters.js
   - Guarantees adapters are available even with --ignore-scripts

* fix: remove OPENCLI_FETCH env var, use internal _OPENCLI_FIRST_RUN instead

* feat: also support OPENCLI_FETCH=1 for explicit adapter fetch trigger

* simplify: replace git clone with local copy from dist/clis/

Adapters already ship in the npm package (dist/clis/), so there's no
need to clone from GitHub. Copy directly from the installed package:

- Eliminates git, curl, tar dependencies
- No network calls in postinstall
- No timeout/offline issues
- Version always matches the installed CLI
- ~65 lines of clone/download code replaced by one cpSync loop
2026-04-05 01:46:36 +08:00
TennyZhuang 60bee91650 fix: match the requested tweet before deleting on X (#781)
* fix(twitter): match target tweet before deleting

* review: normalize invalid twitter delete URLs

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 23:57:01 +08:00
jakevin cf9e9f0137 chore: update lock files for v1.6.2 (#776) 2026-04-04 21:32:27 +08:00
jakevin b2f1f58a1b chore: bump version to 1.6.2 (#775)
Release / release (push) Has been cancelled
Build Chrome Extension / build (push) Has been cancelled
2026-04-04 21:28:37 +08:00
jakevin 81308a474e docs: add skills install command to Quick Start section (#772)
Add `npx skills add jackwener/opencli` to the install step in both
English and Chinese READMEs so users discover AI skills during setup.
2026-04-04 18:46:27 +08:00
Inori333 d818b5bed8 fix(completion): sync top-level command suggestions (#588) 2026-04-04 16:26:31 +08:00
Ray e82649379b feat(instagram): add post, reel, story, and note publishing (#671)
* Add draft Instagram posting flow

* Refine Instagram post flow

* Add dynamic Instagram posting routes

* Retry transient Instagram private setup failures

* Add Instagram reel posting command

* Add Instagram mixed-media carousel posting

* Unify Instagram post media input

* Add Instagram story posting command

* Add Instagram note publishing command

* fix(instagram): use JSON.stringify for constants in note evaluate string

Replace template literal interpolation of Node-side constants with
JSON.stringify() for consistency with codebase evaluate patterns.
Use bracket notation for dynamic property access instead of template
interpolation into a property chain.

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 16:22:22 +08:00
kevin.zhang e0a66af0f0 feat(hupu): add Hupu adapter (#751)
* feat(hupu): add hupu cli adapter

* fix(hupu): prevent detail from returning the wrong thread

* refactor: deduplicate shared utilities in hupu adapter

- Merge postHupuJson and postHupuReplyJson into single function with mode parameter
- Move stripHtml and decodeHtmlEntities to utils.ts, remove duplicate definitions

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 16:03:14 +08:00
YoungCan-Wang c86e677b78 推特支持回复图片 (#756)
* feat: 推特新增回复图片能力支持本地路径和网络路径

* fix(twitter/reply): fix image upload fallback, restore execCommand, add size limit

- Fix attachReplyImage fallback: use uploaded flag instead of checking
  page.setFileInput existence, so base64 fallback actually runs when
  CDP setFileInput throws "Unknown action"
- Restore execCommand('insertText') as primary text input method for
  Twitter's Draft.js editor, with paste event as fallback
- Add 20MB size limit for remote image downloads to prevent OOM
- Remove unsafe buttons[0] fallback that could click invisible buttons

* fix(twitter/reply): add local image size check and base64 fallback warning

Local images were not validated for size — a 100MB file would fail only
at upload time. Remote images already had MAX_IMAGE_SIZE_BYTES checks.
Also add a console.warn when using the base64 fallback with large
payloads, consistent with xiaohongshu/publish.ts behavior.

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 16:00:35 +08:00
yulin7645 6364934423 feat(xiaoe): add 小鹅通 (Xiaoe-tech) student platform adapter (#617)
* feat(xiaoe): add 小鹅通 (Xiaoe-tech) student platform adapter

Add 5 YAML adapters for 小鹅通 (xiaoe-tech.com), the leading Chinese
online education platform:

- courses: list purchased courses with URLs and shop names
- detail: course info (name, price, user count, shop)
- catalog: full course outline supporting normal courses (type 50),
  columns (type 6), and big columns (type 8)
- play-url: get M3U8 play URL via direct API for video courses,
  and Vue component tree search + Performance API polling for
  live replay courses
- content: extract rich-text page content as plain text

Technical notes:
- Strategy: cookie (reuses Chrome login session)
- Framework: Vue 2 + Vuex Store (SPA)
- Video courses use a two-step API chain:
  detail_info.get → play_sign → getPlayUrl → M3U8
- Live replays use Performance API + Vue data tree polling
- Catalog expands chapters via Vue component method getSecitonList()
- Supports multiple stores (cross-domain cookie sharing via
  study.xiaoe-tech.com)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review: stop truncating xiaoe content

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 15:54:14 +08:00
AstroHan ef78aaf3a2 fix: add -v/--verbose to built-in browser commands (#719)
* fix: add -v/--verbose to explore, record, generate, cascade

Built-in browser commands were registered directly in cli.ts and
missed the -v/--verbose flag that commanderAdapter.ts wires up for
adapter commands. Also switch explore's lone log.debug() call to
log.verbose() so the flag has visible effect.

Closes #716

* refactor(cli): make builtin command wiring testable

* refactor(cli): simplify verbose wiring, use normal Commander pattern

Replace registerVerboseAction wrapper with simple applyVerbose() helper.
The wrapper broke Commander's builder chain and created awkward
indentation. Now each command uses standard .option().action() with
applyVerbose(opts) as the first line — easier to read and maintain.

* fix(cli): add -v/--verbose to doctor and synthesize commands

These commands were also missing verbose support, same root cause as
explore/record/generate/cascade — registered directly in cli.ts,
bypassing commanderAdapter's automatic -v wiring.

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 15:44:23 +08:00
artshooter 292b12d9b1 feat(twitter): add --images flag to post command (#666)
* feat(twitter): add --images flag to post command

Support attaching up to 4 images when posting tweets via
`opencli twitter post "text" --images /path/a.png,/path/b.jpg`.

Uses the existing CDP DOM.setFileInputFiles mechanism (page.setFileInput)
to inject files into Twitter's file input. Includes proper file validation,
graceful error handling for older extensions, and polling-based upload
readiness detection instead of fixed delays.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(twitter): use attachments DOM signal for upload detection, add tests

Replace unreliable tweet-button-only polling with dual-condition check:
wait for [data-testid="attachments"] with correct [role="group"] count
AND button enabled. Increase timeout to 30s. Add 8 unit tests covering
image upload flow, file validation, and error paths.

Addresses PR #666 review feedback.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(twitter): use top-level imports, fix test mocks, faster upload poll

- Use top-level fs/path imports instead of dynamic imports inside func
- Fix test statSync mock to return undefined (not null) for missing files
- Fix test path mock to preserve other exports via importOriginal
- Fix null type error in no-browser-session test
- Reduce upload poll interval from 1s to 500ms for faster detection
- Use JSON.stringify for imageCount interpolation for consistency

* refactor(twitter): extract validation, fail-fast, reduce duplication

- Extract validateImagePaths() with extension validation (jpg/png/gif/webp)
  matching xiaohongshu publish pattern
- Validate images before browser navigation (fail-fast on bad input)
- Remove try/catch wrapper around setFileInput — let errors propagate
  naturally instead of masking the original error
- Deduplicate tweetButton/tweetButtonInline lookups using fallback OR
- Use constants for MAX_IMAGES, UPLOAD_POLL_MS, UPLOAD_TIMEOUT_MS
- Add tests: unsupported format, validates-before-navigating

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 15:31:01 +08:00
AstroHan 2c5066d1f4 fix(gemini): stabilize ask reply state handling (#735)
* fix(gemini): stabilize ask reply state handling

* fix: use CommandExecutionError for composer failures and clean up formatting

- Replace raw Error with CommandExecutionError for Node-side composer
  failures (prepareComposer, insertText) to match adapter error conventions
- Remove extra blank lines after __test__ export

* refactor: remove dead code and add Chinese sign-in label

- Remove unused areGeminiTurnsEqual and areGeminiLinesEqual functions
- Add Chinese sign-in label (登录) to sign-in detection for consistency
  with other Chinese labels already added in this PR

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 15:26:58 +08:00
yichuanzhao99-ctrl 8eefa3b1c9 增加新浪财经热搜股票榜 (#736)
* 增加新浪财经热搜股票榜

* fix: address review issues in stock-rank adapter

- Fix string interpolation injection: use JSON.stringify for market param
- Add choices validation for market arg (cn/hk/us/wh/ft)
- Normalize column names to lowercase (rank/name/symbol/market/price/change/url)
- Add navigateBefore: false to avoid redundant navigation
- Add null safety on tabEl with optional chaining
- Remove unused waitForElement helper and unnecessary await on querySelectorAll
- Remove unrelated ?from=opencli tracking change from rolling-news.ts
- Remove ?from=opencli tracking from stock-rank URLs

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 15:16:05 +08:00
jakevin 052bf8bbf7 refactor(zhihu): simplify question evaluate to follow pixivFetch pattern (#754)
Move data processing (HTML stripping, answer mapping) from browser-side
evaluate to Node-side, keeping the evaluate minimal: just fetch + status
check. Uses __httpError sentinel consistent with pixivFetch convention.
2026-04-04 15:06:16 +08:00
jakevin c3c3abbbff fix(1688): remove MOQ extraction from price field, rename firstLine to firstWord, fix sales regex (#755)
- Remove hover_price_text as MOQ source in search normalizeSearchCandidate
  to prevent price fields from being misinterpreted as MOQ data
- Rename firstLine() to firstWord() to match its actual behavior (splits
  by whitespace, not newlines)
- Add missing "单" unit to item.ts extractSalesText regex
- Add test case verifying hover_price_text is not used for MOQ
2026-04-04 14:53:42 +08:00
GanFanNewOrder 81de69be3a feat(1688): add browser adapter and docs (#650)
* feat(1688): add browser adapter and docs

* fix(1688): retry alternate store seed offers

* feat(1688): harden adapter contracts and search pagination

---------

Co-authored-by: 泽加武 <zejiawu@zejiawudeMac-mini.local>
2026-04-04 14:45:32 +08:00
jakevin a39a858f0a feat(autoresearch): improve operate success rate + complex publish chains (#753)
* chore(autoresearch): format save-tasks.json

* feat(autoresearch): add Layer 5 Publish testing for twitter/zhihu

New eval-publish.ts tests end-to-end content creation via operate commands:
- 7 tasks: 5 fill-only (safe) + 2 publish (post + delete)
- Twitter: compose fill, reply fill, post+delete, cross-site HN→tweet
- Zhihu: answer fill, article fill (title+body), cross-site HN→answer
- Supports --type fill-only/publish and --platform twitter/zhihu filters
- Cleanup steps auto-delete published content after verification
- fill-only: 5/5 passing

* feat(autoresearch): improve operate success rate + complex publish chains

Iteration round 1 results:
- Browse: 50/59 → 58/59 (+8) — fixed 8 broken selectors, 1 remaining (DDG images anti-crawl)
- Publish fill-only: 5/5 → 12/13 → 13/13 — added 8 complex tasks, fixed selectors
- Save as CLI: 26/26 (maintained)

Changes:
- browse-tasks.json: fix 8 broken selectors (iana, github, quotes, trending, google, wiki, npm, httpbin)
- publish-tasks.json: add 8 complex multi-step tasks (thread compose, quote RT, search→reply, cross-platform)
- skills/opencli-operate/SKILL.md: add Common Pitfalls section, improve save-as-CLI guidance
- Fix twitter thread compose (use querySelectorAll for 2nd textarea)
- Fix zhihu editor selectors (WriteIndex-titleInput, contenteditable)
2026-04-04 14:37:40 +08:00
Kyrie Cai 855eaee04e fix(zhihu): make question runtime-compatible (#732)
* fix(zhihu): make question runtime-compatible

* fix: validate questionId is numeric to prevent interpolation issues

* refactor: simplify evaluate string and harden against injection

- Build URL in Node.js, embed via JSON.stringify for safety-by-design
- Remove unnecessary (page as any) cast — IPage already has evaluate
- Simplify error message construction (no nested ternaries)
- Replace implementation-detail test with numeric ID validation test

* refactor: simplify zhihu question — move stripHtml into evaluate, return clean data

* fix: add colon separator in fetch error message for readability

"request failed Failed to fetch" → "request failed: Failed to fetch"

---------

Co-authored-by: Kyrie <kyrie@mallab.world>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-04 14:36:27 +08:00
ykfnxx 1bcd96f38a fix(douban): fix marks pagination and improve subject data extraction (#752)
1. marks: correct pageSize from 30 to 15 — douban grid mode shows 15
   items per page, causing pagination to stop after the first page.

2. subject: split title/originalTitle correctly — v:itemreviewed contains
   both Chinese and original titles concatenated.

3. subject: extract country/region from #info as list, split by "/".

4. subject: extract duration as pure number (min) from v:runtime or #info.

5. subject: return casts as list instead of comma-joined string.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-04 14:32:51 +08:00
jakevin c2ac5525b3 chore(autoresearch): format save-tasks.json (#750) 2026-04-04 02:12:26 +08:00
jakevin 748b09261d fix: handle missing electron executable gracefully (#747)
* fix: handle missing electron executable gracefully

* fix: support antigravity electron executable fallback
2026-04-04 02:03:30 +08:00
jakevin 4b1153babe docs: remove duplicated root cli workflow guides (#748) 2026-04-04 01:57:30 +08:00
812 changed files with 23450 additions and 3582 deletions
-724
View File
@@ -1,724 +0,0 @@
# CLI-EXPLORER — 适配器探索式开发完全指南
> 本文档教你(或 AI Agent)如何为 OpenCLI 添加一个新网站的命令。
> 从零到发布,覆盖 API 发现、方案选择、适配器编写、测试验证全流程。
> [!TIP]
> **只想为一个具体页面快速生成一个命令?** 看 [CLI-ONESHOT.md](./CLI-ONESHOT.md)~150 行,4 步搞定)。
> 本文档适合从零探索一个新站点的完整流程。
---
## AI Agent 开发者必读:用浏览器探索
> [!CAUTION]
> **你(AI Agent)必须通过浏览器打开目标网站去探索!**
> 不要只靠 `opencli explore` 命令或静态分析来发现 API。
> 你拥有浏览器工具,必须主动用它们浏览网页、观察网络请求、模拟用户交互。
### 为什么?
很多 API 是**懒加载**的(用户必须点击某个按钮/标签才会触发网络请求)。字幕、评论、关注列表等深层数据不会在页面首次加载时出现在 Network 面板中。**如果你不主动去浏览和交互页面,你永远发现不了这些 API。**
### AI Agent 探索工作流(必须遵循)
| 步骤 | 工具 | 做什么 |
|------|------|--------|
| 0. 打开浏览器 | `browser_navigate` | 导航到目标页面 |
| 1. 观察页面 | `browser_snapshot` | 观察可交互元素(按钮/标签/链接) |
| 2. 首次抓包 | `browser_network_requests` | 筛选 JSON API 端点,记录 URL pattern |
| 3. 模拟交互 | `browser_click` + `browser_wait_for` | 点击"字幕""评论""关注"等按钮 |
| 4. 二次抓包 | `browser_network_requests` | 对比步骤 2,找出新触发的 API |
| 5. 验证 API | `browser_evaluate` | `fetch(url, {credentials:'include'})` 测试返回结构 |
| 6. 写代码 | — | 基于确认的 API 写适配器 |
### 常犯错误
| ❌ 错误做法 | ✅ 正确做法 |
|------------|------------|
| 只用 `opencli explore` 命令,等结果自动出来 | 用浏览器工具打开页面,主动浏览 |
| 直接在代码里 `fetch(url)`,不看浏览器实际请求 | 先在浏览器中确认 API 可用,再写代码 |
| 页面打开后直接抓包,期望所有 API 都出现 | 模拟点击交互(展开评论/切换标签/加载更多) |
| 遇到 HTTP 200 但空数据就放弃 | 检查是否需要 Wbi 签名或 Cookie 鉴权 |
| 完全依赖 `__INITIAL_STATE__` 拿所有数据 | `__INITIAL_STATE__` 只有首屏数据,深层数据要调 API |
### 实战成功案例:5 分钟实现「关注列表」适配器
以下是用上述工作流实际发现 Bilibili 关注列表 API 的完整过程:
```
1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
2. browser_network_requests → 发现:
GET /x/relation/followings?vmid={uid}&pn=1&ps=24 → [200]
GET /x/relation/stat?vmid={uid} → [200]
3. browser_evaluate → 验证 API:
fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
→ { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
4. 结论:标准 Cookie API,无需 Wbi 签名
5. 写 following.ts → 一次构建通过
```
**关键决策点**
- 直接访问 `fans/follow` 页面(不是首页),页面加载就会触发 following API
- 看到 URL 里没有 `/wbi/` → 不需要签名 → 直接用 `fetchJson` 而非 `apiGet`
- API 返回 `code: 0` + 非空 `list` → Tier 2 Cookie 策略确认
---
## 核心流程
```
┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌────────┐
│ 1. 发现 API │ ──▶ │ 2. 选择策略 │ ──▶ │ 3. 写适配器 │ ──▶ │ 4. 测试 │
└─────────────┘ └─────────────┘ └──────────────┘ └────────┘
explore cascade YAML / TS run + verify
```
---
## Step 1: 发现 API
### 1a. 自动化发现(推荐)
OpenCLI 内置 Deep Explore,自动分析网站网络请求:
```bash
opencli explore https://www.example.com --site mysite
```
输出到 `.opencli/explore/mysite/`
| 文件 | 内容 |
|------|------|
| `manifest.json` | 站点元数据、框架检测(Vue2/3、React、Next.js、Pinia、Vuex |
| `endpoints.json` | 已发现的 API 端点,按评分排序,含 URL pattern、方法、响应类型 |
| `capabilities.json` | 推理出的功能(`hot``search``feed`…),含置信度和推荐参数 |
| `auth.json` | 认证方式检测(Cookie/Header/无认证),策略候选列表 |
### 1b. 手动抓包验证
Explore 的自动分析可能不完美,用 verbose 模式手动确认:
```bash
# 在浏览器中打开目标页面,观察网络请求
opencli explore https://www.example.com --site mysite -v
# 或直接用 evaluate 测试 API
opencli bilibili hot -v # 查看已有命令的 pipeline 每步数据流
```
关注抓包结果中的关键信息:
- **URL pattern**: `/api/v2/hot?limit=20` → 这就是你要调用的端点
- **Method**: `GET` / `POST`
- **Request Headers**: Cookie? Bearer? 自定义签名头(X-s、X-t?
- **Response Body**: JSON 结构,特别是数据在哪个路径(`data.items``data.list`
### 1c. 高阶 API 发现捷径法则 (Heuristics)
在开始死磕复杂的抓包拦截之前,按照以下优先级进行尝试:
1. **后缀爆破法 (`.json`)**: 像 Reddit 这样复杂的网站,只要在其 URL 后加上 `.json`(例如 `/r/all.json`),就能在带 Cookie 的情况下直接利用 `fetch` 拿到极其干净的 REST 数据(Tier 2 Cookie 策略极速秒杀)。另外如功能完备的**雪球 (xueqiu)** 也可以走这种纯 API 的方式极简获取,成为你构建简单 YAML 的黄金标杆。
2. **全局状态查找法 (`__INITIAL_STATE__`)**: 许多服务端渲染 (SSR) 的网站(如小红书、Bilibili)会将首页或详情页的完整数据挂载到全局 window 对象上。与其去拦截网络请求,不如直接 `page.evaluate('() => window.__INITIAL_STATE__')` 获取整个数据树。
3. **主动交互触发法 (Active Interaction)**: 很多深层 API(如视频字幕、评论下的回复)是懒加载的。在静态抓包找不到数据时,尝试在 `evaluate` 步骤或手动打断点时,主动去**点击(Click)页面上的对应按钮**(如"CC"、"展开全部"),从而诱发隐藏的 Network Fetch。
4. **框架探测与 Store Action 截断**: 如果站点使用 Vue + Pinia,可以使用 `tap` 步骤调用 action,让前端框架代替你完成复杂的鉴权签名封装。
5. **底层 XHR/Fetch 拦截**: 最后手段,当上述都不行时,使用 TypeScript 适配器进行无侵入式的请求抓取。
### 1d. 框架检测
Explore 自动检测前端框架。如果需要手动确认:
```bash
# 在已打开目标网站的情况下
opencli evaluate "(()=>{
const vue3 = !!document.querySelector('#app')?.__vue_app__;
const vue2 = !!document.querySelector('#app')?.__vue__;
const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
return JSON.stringify({vue3, vue2, react, pinia});
})()"
```
Vue + Pinia 的站点(如小红书)可以直接通过 Store Action 绕过签名。
---
## Step 2: 选择认证策略
OpenCLI 提供 5 级认证策略。使用 `cascade` 命令自动探测:
```bash
opencli cascade https://api.example.com/hot
```
### 策略决策树
```
直接 fetch(url) 能拿到数据?
→ ✅ Tier 1: public(公开 API,不需要浏览器)
→ ❌ fetch(url, {credentials:'include'}) 带 Cookie 能拿到?
→ ✅ Tier 2: cookie(最常见,evaluate 步骤内 fetch
→ ❌ → 加上 Bearer / CSRF header 后能拿到?
→ ✅ Tier 3: header(如 Twitter ct0 + Bearer
→ ❌ → 网站有 Pinia/Vuex Store
→ ✅ Tier 4: interceptStore Action + XHR 拦截)
→ ❌ Tier 5: ui(UI 自动化,最后手段)
```
### 各策略对比
| Tier | 策略 | 速度 | 复杂度 | 适用场景 | 实例 |
|------|------|------|--------|---------|------|
| 1 | `public` | ⚡ ~1s | 最简 | 公开 API,无需登录 | Hacker News, V2EX |
| 2 | `cookie` | 🔄 ~7s | 简单 | Cookie 认证即可 | Bilibili, Zhihu, Reddit |
| 3 | `header` | 🔄 ~7s | 中等 | 需要 CSRF token 或 Bearer | Twitter GraphQL |
| 4 | `intercept` | 🔄 ~10s | 较高 | 请求有复杂签名 | 小红书 (Pinia + XHR) |
| 5 | `ui` | 🐌 ~15s+ | 最高 | 无 API,纯 DOM 解析 | 遗留网站 |
---
## Step 2.5: 准备工作(写代码之前)
### 先找模板:从最相似的现有适配器开始
**不要从零开始写**。先看看同站点已有哪些适配器:
```bash
ls src/clis/<site>/ # 看看已有什么
cat src/clis/<site>/feed.ts # 读最相似的那个
```
最高效的方式是 **复制最相似的适配器,然后改 3 个地方**
1. `name` → 新命令名
2. API URL → 你在 Step 1 发现的端点
3. 字段映射 → 对应新 API 的字段
### 平台 SDK 速查表
写 TS 适配器之前,先看看你的目标站点有没有**现成的 helper 函数**可以复用:
#### Bilibili (`src/clis/bilibili/utils.ts`)
| 函数 | 用途 | 何时使用 |
|------|------|----------|
| `fetchJson(page, url)` | 带 Cookie 的 fetch + JSON 解析 | 普通 Cookie-tier API |
| `apiGet(page, path, {signed, params})` | 带 Wbi 签名的 API 调用 | URL 含 `/wbi/` 的接口 |
| `getSelfUid(page)` | 获取当前登录用户的 UID | "我的xxx" 类命令 |
| `resolveUid(page, input)` | 解析用户输入的 UID(支持数字/URL) | `--uid` 参数处理 |
| `wbiSign(page, params)` | 底层 Wbi 签名生成 | 通常不直接用,`apiGet` 已封装 |
| `stripHtml(s)` | 去除 HTML 标签 | 清理富文本字段 |
**如何判断需不需要 `apiGet`**?看 Network 请求 URL
-`/wbi/``w_rid=` → 必须用 `apiGet(..., { signed: true })`
- 不含 → 直接用 `fetchJson`
> 其他站点(Twitter、小红书等)暂无专用 SDK,直接用 `page.evaluate` + `fetch` 即可。
---
## Step 3: 编写适配器
### YAML vs TS?先看决策树
```
你的 pipeline 里有 evaluate 步骤(内嵌 JS 代码)?
→ ✅ 用 TypeScript (src/clis/<site>/<name>.ts),保存即自动动态注册
→ ❌ 纯声明式(navigate + tap + map + limit)?
→ ✅ 用 YAML (src/clis/<site>/<name>.yaml),保存即自动注册
```
| 场景 | 选择 | 示例 |
|------|------|------|
| 纯 fetch/select/map/limit | YAML | `v2ex/hot.yaml`, `hackernews/top.yaml` |
| navigate + evaluate(fetch) + map | YAML(评估复杂度) | `zhihu/hot.yaml` |
| navigate + tap + map | YAML ✅ | `xiaohongshu/feed.yaml`, `xiaohongshu/notifications.yaml` |
| 有复杂 JS 逻辑(Pinia state 读取、条件分支) | TS | `xiaohongshu/me.ts`, `bilibili/me.ts` |
| XHR 拦截 + 签名 | TS | `xiaohongshu/search.ts` |
| GraphQL / 分页 / Wbi 签名 | TS | `bilibili/search.ts`, `twitter/search.ts` |
> **经验法则**:如果你发现 YAML 里嵌了超过 10 行 JS,改用 TS 更可维护。
### 通用模式:分页 API
很多 API 使用 `pn`(页码)+ `ps`(每页数量)分页。标准处理模式:
```typescript
args: [
{ name: 'page', type: 'int', required: false, default: 1, help: '页码' },
{ name: 'limit', type: 'int', required: false, default: 50, help: '每页数量 (最大 50)' },
],
func: async (page, kwargs) => {
const pn = kwargs.page ?? 1;
const ps = Math.min(kwargs.limit ?? 50, 50); // 尊重 API 的 ps 上限
const payload = await fetchJson(page,
`https://api.example.com/list?pn=${pn}&ps=${ps}`
);
return payload.data?.list || [];
},
```
> 大多数站点的 `ps` 上限是 20~50。超过会被静默截断或返回错误。
### 方式 A: YAML Pipeline(声明式,推荐)
文件路径: `src/clis/<site>/<name>.yaml`,放入即自动注册。
#### Tier 1 — 公开 API 模板
```yaml
# src/clis/v2ex/hot.yaml
site: v2ex
name: hot
description: V2EX 热门话题
domain: www.v2ex.com
strategy: public
browser: false
args:
limit:
type: int
default: 20
pipeline:
- fetch:
url: https://www.v2ex.com/api/topics/hot.json
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
replies: ${{ item.replies }}
- limit: ${{ args.limit }}
columns: [rank, title, replies]
```
#### Tier 2 — Cookie 认证模板(最常用)
```yaml
# src/clis/zhihu/hot.yaml
site: zhihu
name: hot
description: 知乎热榜
domain: www.zhihu.com
pipeline:
- navigate: https://www.zhihu.com # 先加载页面建立 session
- evaluate: | # 在浏览器内发请求,自动带 Cookie
(async () => {
const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
credentials: 'include'
});
const d = await res.json();
return (d?.data || []).map(item => {
const t = item.target || {};
return {
title: t.title,
heat: item.detail_text || '',
answers: t.answer_count,
};
});
})()
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
heat: ${{ item.heat }}
answers: ${{ item.answers }}
- limit: ${{ args.limit }}
columns: [rank, title, heat, answers]
```
> **关键**: `evaluate` 步骤内的 `fetch` 运行在浏览器页面内,自动携带 `credentials: 'include'`,无需手动处理 Cookie。
#### 进阶 — 带搜索参数
```yaml
# src/clis/zhihu/search.yaml
site: zhihu
name: search
description: 知乎搜索
args:
query:
type: str
required: true
positional: true
description: Search query
limit:
type: int
default: 10
pipeline:
- navigate: https://www.zhihu.com
- evaluate: |
(async () => {
const q = encodeURIComponent('${{ args.query }}');
const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${{ args.limit }}', {
credentials: 'include'
});
const d = await res.json();
return (d?.data || [])
.filter(item => item.type === 'search_result')
.map(item => ({
title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
type: item.object?.type || '',
author: item.object?.author?.name || '',
votes: item.object?.voteup_count || 0,
}));
})()
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
type: ${{ item.type }}
author: ${{ item.author }}
votes: ${{ item.votes }}
- limit: ${{ args.limit }}
columns: [rank, title, type, author, votes]
```
#### Tier 4 — Store Action Bridge`tap` 步骤,intercept 策略推荐)
适用于 Vue + Pinia/Vuex 的网站(如小红书),无须手动写 XHR 拦截代码:
```yaml
# src/clis/xiaohongshu/notifications.yaml
site: xiaohongshu
name: notifications
description: "小红书通知"
domain: www.xiaohongshu.com
strategy: intercept
browser: true
args:
type:
type: str
default: mentions
description: "Notification type: mentions, likes, or connections"
limit:
type: int
default: 20
columns: [rank, user, action, content, note, time]
pipeline:
- navigate: https://www.xiaohongshu.com/notification
- wait: 3
- tap:
store: notification # Pinia store name
action: getNotification # Store action to call
args: # Action arguments
- ${{ args.type | default('mentions') }}
capture: /you/ # URL pattern to capture response
select: data.message_list # Extract sub-path from response
timeout: 8
- map:
rank: ${{ index + 1 }}
user: ${{ item.user_info.nickname }}
action: ${{ item.title }}
content: ${{ item.comment_info.content }}
- limit: ${{ args.limit | default(20) }}
```
> **`tap` 步骤自动完成**:注入 fetch+XHR 双拦截 → 查找 Pinia/Vuex store → 调用 action → 捕获匹配 URL 的响应 → 清理拦截。
> 如果 store 或 action 找不到,会返回 `hint` 列出所有可用的 store actions,方便调试。
| tap 参数 | 必填 | 说明 |
|---------|------|------|
| `store` | ✅ | Pinia store 名称(如 `feed`, `search`, `notification` |
| `action` | ✅ | Store action 方法名 |
| `capture` | ✅ | URL 子串匹配(匹配网络请求 URL) |
| `args` | ❌ | 传给 action 的参数数组 |
| `select` | ❌ | 从 captured JSON 中提取的路径(如 `data.items` |
| `timeout` | ❌ | 等待网络响应的超时秒数(默认 5s) |
| `framework` | ❌ | `pinia``vuex`(默认自动检测) |
### 方式 B: TypeScript 适配器(编程式)
适用于需要嵌入 JS 代码读取 Pinia state、XHR 拦截、GraphQL、分页、复杂数据转换等场景。
文件路径: `src/clis/<site>/<name>.ts`。文件将会在运行时被动态扫描并注册(切勿在 `index.ts` 中手动 `import`)。
#### Tier 3 — Header 认证(Twitter
```typescript
// src/clis/twitter/search.ts
import { cli, Strategy } from '../../registry.js';
cli({
site: 'twitter',
name: 'search',
description: 'Search tweets',
strategy: Strategy.HEADER,
args: [{ name: 'query', required: true, positional: true }],
columns: ['rank', 'author', 'text', 'likes'],
func: async (page, kwargs) => {
await page.goto('https://x.com');
const data = await page.evaluate(`
(async () => {
// 从 Cookie 提取 CSRF token
const ct0 = document.cookie.split(';')
.map(c => c.trim())
.find(c => c.startsWith('ct0='))?.split('=')[1];
if (!ct0) return { error: 'Not logged in' };
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
const headers = {
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
'X-Csrf-Token': ct0,
'X-Twitter-Auth-Type': 'OAuth2Session',
};
const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
const res = await fetch(url, { headers, credentials: 'include' });
return await res.json();
})()
`);
// ... 解析 data
},
});
```
#### Tier 4 — XHR/Fetch 双重拦截 (Twitter/小红书 通用模式)
```typescript
// src/clis/xiaohongshu/user.ts
import { cli, Strategy } from '../../registry.js';
cli({
site: 'xiaohongshu',
name: 'user',
description: '获取用户笔记',
strategy: Strategy.INTERCEPT,
args: [{ name: 'id', required: true }],
columns: ['rank', 'title', 'likes', 'url'],
func: async (page, kwargs) => {
await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
await page.wait(5);
// XHR/Fetch 底层拦截:捕获所有包含 'v1/user/posted' 的请求
await page.installInterceptor('v1/user/posted');
// 触发后端 API:模拟人类用户向底部滚动2次
await page.autoScroll({ times: 2, delayMs: 2000 });
// 提取所有被拦截捕获的 JSON 响应体
const requests = await page.getInterceptedRequests();
if (!requests || requests.length === 0) return [];
let results = [];
for (const req of requests) {
if (req.data?.data?.notes) {
for (const note of req.data.data.notes) {
results.push({
title: note.display_title || '',
likes: note.interact_info?.liked_count || '0',
url: `https://explore/${note.note_id || note.id}`
});
}
}
}
return results.slice(0, 20).map((item, i) => ({
rank: i + 1, ...item,
}));
},
});
```
> **拦截核心思路**:不自己构造签名,而是利用 `installInterceptor` 劫持网站自己的 `XMLHttpRequest` 和 `fetch`,让网站发请求,我们直接在底层取出解析好的 `response.json()`。
> **级联请求**(如 BVID→CID→字幕)的完整模板和要点见下方[进阶模式: 级联请求](#进阶模式-级联请求-cascading-requests)章节。
---
## Step 4: 测试
> **构建通过 ≠ 功能正常**。`npm run build` 只验证 TypeScript / YAML 语法,不验证运行时行为。
> 每个新命令 **必须实际运行** 并确认输出正确后才算完成。
### 必做清单
```bash
# 1. 构建(确认语法无误)
npm run build
# 2. 确认命令已注册
opencli list | grep mysite
# 3. 实际运行命令(最关键!)
opencli mysite hot --limit 3 -v # verbose 查看每步数据流
opencli mysite hot --limit 3 -f json # JSON 输出确认字段完整
```
### tap 步骤调试(intercept 策略专用)
> **不要猜 store name / action name**。先用 evaluate 探索,再写 YAML。
#### Step 1: 列出所有 Pinia store
在浏览器中打开目标网站后:
```bash
opencli evaluate "(() => {
const app = document.querySelector('#app')?.__vue_app__;
const pinia = app?.config?.globalProperties?.\$pinia;
return [...pinia._s.keys()];
})()"
# 输出: ["user", "feed", "search", "notification", ...]
```
#### Step 2: 查看 store 的 action 名称
故意写一个错误 action 名,tap 会返回所有可用 actions
```
⚠ tap: Action not found: wrongName on store notification
💡 Available: getNotification, replyComment, getNotificationCount, reset
```
#### Step 3: 用 network requests 确认 capture 模式
```bash
# 在浏览器打开目标页面,查看网络请求
# 找到目标 API 的 URL 特征(如 "/you/mentions"、"homefeed"
```
#### 完整流程
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────┐
│ 1. navigate │ ──▶ │ 2. 探索 store │ ──▶ │ 3. 写 YAML │ ──▶ │ 4. 测试 │
│ 到目标页面 │ │ name/action │ │ tap 步骤 │ │ 运行验证 │
└──────────────┘ └──────────────┘ └──────────────┘ └────────┘
```
### Verbose 模式 & 输出验证
```bash
opencli bilibili hot --limit 1 -v # 查看 pipeline 每步数据流
opencli mysite hot -f json | jq '.[0]' # 确认 JSON 可被解析
opencli mysite hot -f csv > data.csv # 确认 CSV 可导入
```
---
## Step 5: 提交发布
文件放入 `src/clis/<site>/` 即自动注册(YAML 或 TS 无需手动 import),然后:
```bash
opencli list | grep mysite # 确认注册
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push
```
> **架构理念**OpenCLI 内建 **Zero-Dependency jq** 数据流 — 所有解析在 `evaluate` 的原生 JS 内完成,外层 YAML 用 `select`/`map` 提取,无需依赖系统 `jq` 二进制。
---
## 进阶模式: 级联请求 (Cascading Requests)
当目标数据需要多步 API 链式获取时(如 `BVID → CID → 字幕列表 → 字幕内容`),必须使用 **TS 适配器**。YAML 无法处理这种多步逻辑。
### 模板代码
```typescript
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { apiGet } from './utils.js'; // 复用平台 SDK
cli({
site: 'bilibili',
name: 'subtitle',
strategy: Strategy.COOKIE,
args: [{ name: 'bvid', required: true }],
columns: ['index', 'from', 'to', 'content'],
func: async (page: IPage | null, kwargs: any) => {
if (!page) throw new Error('Requires browser');
// Step 1: 建立 Session
await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);
// Step 2: 从页面提取中间 ID (__INITIAL_STATE__)
const cid = await page.evaluate(`(async () => {
return window.__INITIAL_STATE__?.videoData?.cid;
})()`);
if (!cid) throw new Error('无法提取 CID');
// Step 3: 用中间 ID 调用下一级 API (自动 Wbi 签名)
const payload = await apiGet(page, '/x/player/wbi/v2', {
params: { bvid: kwargs.bvid, cid },
signed: true, // ← 自动生成 w_rid
});
// Step 4: 检测风控降级 (空值断言)
const subtitles = payload.data?.subtitle?.subtitles || [];
const url = subtitles[0]?.subtitle_url;
if (!url) throw new Error('subtitle_url 为空,疑似风控降级');
// Step 5: 拉取最终数据 (CDN JSON)
const items = await page.evaluate(`(async () => {
const res = await fetch(${JSON.stringify('https:' + url)});
const json = await res.json();
return { data: json.body || json };
})()`);
return items.data.map((item, idx) => ({ ... }));
},
});
```
### 关键要点
| 步骤 | 注意事项 |
|------|----------|
| 提取中间 ID | 优先从 `__INITIAL_STATE__` 拿,避免额外 API 调用 |
| Wbi 签名 | B 站 `/wbi/` 接口**强制校验** `w_rid`,纯 `fetch` 会被 403 |
| 空值断言 | 即使 HTTP 200,核心字段可能为空串(风控降级) |
| CDN URL | 常以 `//` 开头,记得补 `https:` |
| `JSON.stringify` | 拼接 URL 到 evaluate 时必须用它转义,避免注入 |
---
## 常见陷阱
| 陷阱 | 表现 | 解决方案 |
|------|------|---------|
| 缺少 `navigate` | evaluate 报 `Target page context` 错误 | 在 evaluate 前加 `navigate:` 步骤 |
| 嵌套字段访问 | `${{ item.node?.title }}` 不工作 | 在 evaluate 中 flatten 数据,不在模板中用 optional chaining |
| 缺少 `strategy: public` | 公开 API 也启动浏览器,7s → 1s | 公开 API 加上 `strategy: public` + `browser: false` |
| evaluate 返回字符串 | map 步骤收到 `""` 而非数组 | pipeline 有 auto-parse,但建议在 evaluate 内 `.map()` 整形 |
| 搜索参数被 URL 编码 | `${{ args.query }}` 被浏览器二次编码 | 在 evaluate 内用 `encodeURIComponent()` 手动编码 |
| Cookie 过期 | 返回 401 / 空数据 | 在浏览器里重新登录目标站点 |
| Extension tab 残留 | Chrome 多出 `chrome-extension://` tab | 已自动清理;若残留,手动关闭即可 |
| TS evaluate 格式 | `() => {}``result is not a function` | TS 中 `page.evaluate()` 必须用 IIFE`(async () => { ... })()` |
| 页面异步加载 | evaluate 拿到空数据(store state 还没更新) | 在 evaluate 内用 polling 等待数据出现,或增加 `wait` 时间 |
| YAML 内嵌大段 JS | 调试困难,字符串转义问题 | 超过 10 行 JS 的命令改用 TS adapter |
| **风控被拦截(伪200)** | 获取到的 JSON 里核心数据是 `""` (空串) | 极易被误判。必须添加断言!无核心数据立刻要求升级鉴权 Tier 并重新配置 Cookie |
| **API 没找见** | `explore` 工具打分出来的都拿不到深层数据 | 点击页面按钮诱发懒加载数据,再结合 `getInterceptedRequests` 获取 |
---
## 用 AI Agent 自动生成适配器
最快的方式是让 AI Agent 完成全流程:
```bash
# 一键:探索 → 分析 → 合成 → 注册
opencli generate https://www.example.com --goal "hot"
# 或分步执行:
opencli explore https://www.example.com --site mysite # 发现 API
opencli explore https://www.example.com --auto --click "字幕,CC" # 模拟点击触发懒加载 API
opencli synthesize mysite # 生成候选 YAML
opencli verify mysite/hot --smoke # 冒烟测试
```
生成的候选 YAML 保存在 `.opencli/explore/mysite/candidates/`,可直接复制到 `src/clis/mysite/` 并微调。
-216
View File
@@ -1,216 +0,0 @@
# CLI-ONESHOT — 单点快速 CLI 生成
> 给一个 URL + 一句话描述,4 步生成一个 CLI 命令。
> 完整探索式开发请看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
---
## 输入
| 项目 | 示例 |
|------|------|
| **URL** | `https://x.com/jakevin7/lists` |
| **Goal** | 获取我的 Twitter Lists |
---
## 流程
### Step 1: 打开页面 + 抓包
```
1. browser_navigate → 打开目标 URL
2. 等待 3-5 秒(让页面加载完、API 请求触发)
3. browser_network_requests → 筛选 JSON API
```
**关键**:只关注返回 `application/json` 的请求,忽略静态资源。
如果没有自动触发 API,手动点击目标按钮/标签再抓一次。
### Step 2: 锁定一个接口
从抓包结果中找到**那个**目标 API。看这几个字段:
| 字段 | 关注什么 |
|------|----------|
| URL | API 路径 pattern(如 `/i/api/graphql/xxx/ListsManagePinTimeline` |
| Method | GET / POST |
| Headers | 有 Cookie? Bearer? CSRF? 自定义签名? |
| Response | 数据在哪个路径(如 `data.list.lists` |
### Step 3: 验证接口能复现
`browser_evaluate` 中用 `fetch` 复现请求:
```javascript
// Tier 2 (Cookie): 大多数情况
fetch('/api/endpoint', { credentials: 'include' }).then(r => r.json())
// Tier 3 (Header): 如 Twitter 需要额外 header
const ct0 = document.cookie.match(/ct0=([^;]+)/)?.[1];
fetch('/api/endpoint', {
headers: { 'Authorization': 'Bearer ...', 'X-Csrf-Token': ct0 },
credentials: 'include'
}).then(r => r.json())
```
如果 fetch 能拿到数据 → 用 YAML 或简单 TS adapter。
如果 fetch 拿不到(签名/风控)→ 用 intercept 策略。
### Step 4: 套模板,生成 adapter
根据 Step 3 判定的策略,选一个模板生成文件。
---
## 认证速查
```
fetch(url) 直接能拿到? → Tier 1: public (YAML, browser: false)
fetch(url, {credentials:'include'}) → Tier 2: cookie (YAML)
加 Bearer/CSRF header 后拿到? → Tier 3: header (TS)
都不行,但页面自己能请求成功? → Tier 4: intercept (TS, installInterceptor)
```
---
## 模板
### YAML — Cookie/Public(最简)
```yaml
# src/clis/<site>/<name>.yaml
site: mysite
name: mycommand
description: "一句话描述"
domain: www.example.com
strategy: cookie # 或 public (加 browser: false)
args:
limit:
type: int
default: 20
pipeline:
- navigate: https://www.example.com/target-page
- evaluate: |
(async () => {
const res = await fetch('/api/target', { credentials: 'include' });
const d = await res.json();
return (d.data?.items || []).map(item => ({
title: item.title,
value: item.value,
}));
})()
- map:
rank: ${{ index + 1 }}
title: ${{ item.title }}
value: ${{ item.value }}
- limit: ${{ args.limit }}
columns: [rank, title, value]
```
### TS — Intercept(抓包模式)
```typescript
// src/clis/<site>/<name>.ts
import { cli, Strategy } from '../../registry.js';
cli({
site: 'mysite',
name: 'mycommand',
description: '一句话描述',
domain: 'www.example.com',
strategy: Strategy.INTERCEPT,
browser: true,
args: [
{ name: 'limit', type: 'int', default: 20 },
],
columns: ['rank', 'title', 'value'],
func: async (page, kwargs) => {
// 1. 导航
await page.goto('https://www.example.com/target-page');
await page.wait(3);
// 2. 注入拦截器(URL 子串匹配)
await page.installInterceptor('target-api-keyword');
// 3. 触发 API(滚动/点击)
await page.autoScroll({ times: 2, delayMs: 2000 });
// 4. 读取拦截的响应
const requests = await page.getInterceptedRequests();
if (!requests?.length) return [];
let results: any[] = [];
for (const req of requests) {
const items = req.data?.data?.items || [];
results.push(...items);
}
return results.slice(0, kwargs.limit).map((item, i) => ({
rank: i + 1,
title: item.title || '',
value: item.value || '',
}));
},
});
```
### TS — Header(如 Twitter GraphQL
```typescript
import { cli, Strategy } from '../../registry.js';
cli({
site: 'twitter',
name: 'mycommand',
description: '一句话描述',
domain: 'x.com',
strategy: Strategy.HEADER,
browser: true,
args: [
{ name: 'limit', type: 'int', default: 20 },
],
columns: ['rank', 'name', 'value'],
func: async (page, kwargs) => {
await page.goto('https://x.com');
const data = await page.evaluate(`(async () => {
const ct0 = document.cookie.match(/ct0=([^;]+)/)?.[1];
if (!ct0) return { error: 'Not logged in' };
const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
const res = await fetch('/i/api/graphql/QUERY_ID/Endpoint', {
headers: {
'Authorization': 'Bearer ' + decodeURIComponent(bearer),
'X-Csrf-Token': ct0,
'X-Twitter-Auth-Type': 'OAuth2Session',
},
credentials: 'include',
});
return res.json();
})()`);
// 解析 data...
return [];
},
});
```
---
## 测试(必做)
```bash
npm run build # 语法检查
opencli list | grep mysite # 确认注册
opencli mysite mycommand --limit 3 -v # 实际运行
```
---
## 就这样,没了
写完文件 → build → run → 提交。有问题再看 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。
+5 -5
View File
@@ -30,7 +30,7 @@ This is the most common type of contribution. Start with YAML when possible, and
### YAML Adapter (Recommended for data-fetching commands)
Create a file like `src/clis/<site>/<command>.yaml`:
Create a file like `clis/<site>/<command>.yaml`:
```yaml
site: mysite
@@ -66,14 +66,14 @@ pipeline:
columns: [rank, title, score, url]
```
See [`hackernews/top.yaml`](src/clis/hackernews/top.yaml) for a real example.
See [`hackernews/top.yaml`](clis/hackernews/top.yaml) for a real example.
### TypeScript Adapter (For complex browser interactions)
Create a file like `src/clis/<site>/<command>.ts`:
Create a file like `clis/<site>/<command>.ts`:
```typescript
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: 'mysite',
@@ -109,7 +109,7 @@ cli({
});
```
Use `opencli explore <url>` to discover APIs and see [CLI-EXPLORER.md](./CLI-EXPLORER.md) if you need the full adapter workflow.
Use `opencli explore <url>` to discover APIs and see [opencli-explorer skill](./skills/opencli-explorer/SKILL.md) if you need the full adapter workflow.
### Validate Your Adapter
+16 -8
View File
@@ -31,7 +31,7 @@ A CLI tool that turns **any website**, **Electron app**, or **local CLI tool** i
- **Dynamic Loader** — Simply drop `.ts` or `.yaml` adapters into the `clis/` folder for auto-registration.
- **Zero LLM cost** — No tokens consumed at runtime. Run 10,000 times and pay nothing.
- **Deterministic** — Same command, same output schema, every time. Pipeable, scriptable, CI-friendly.
- **Broad coverage** — 73+ sites across global and Chinese platforms (Bilibili, Zhihu, Xiaohongshu, Reddit, HackerNews, and more), plus desktop Electron apps via CDP.
- **Broad coverage** — 79+ sites across global and Chinese platforms (Bilibili, Zhihu, Xiaohongshu, Reddit, HackerNews, and more), plus desktop Electron apps via CDP.
---
@@ -51,6 +51,9 @@ A CLI tool that turns **any website**, **Electron app**, or **local CLI tool** i
```bash
npm install -g @jackwener/opencli
# Install AI skills for Claude Code / Cursor
npx skills add jackwener/opencli
```
### 3. Verify & Try
@@ -126,16 +129,19 @@ git clone git@github.com:jackwener/opencli.git && cd opencli && npm install && n
| **xiaohongshu** | `search` `note` `comments` `feed` `user` `download` `publish` `notifications` `creator-notes` `creator-notes-summary` `creator-note-detail` `creator-profile` `creator-stats` |
| **bilibili** | `hot` `search` `history` `feed` `ranking` `download` `comments` `dynamic` `favorite` `following` `me` `subtitle` `user-videos` |
| **tieba** | `hot` `posts` `search` `read` |
| **hupu** | `hot` `search` `detail` `mentions` `reply` `like` `unlike` |
| **twitter** | `trending` `search` `timeline` `bookmarks` `post` `download` `profile` `article` `like` `likes` `notifications` `reply` `reply-dm` `thread` `follow` `unfollow` `followers` `following` `block` `unblock` `bookmark` `unbookmark` `delete` `hide-reply` `accept` |
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `user` `user-posts` `user-comments` `read` `save` `saved` `subscribe` `upvote` `upvoted` `comment` |
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` |
| **gemini** | `new` `ask` `image` |
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `upvoted` `save` `saved` `comment` `subscribe` |
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` `movers-shakers` `new-releases` |
| **1688** | `search` `item` `assets` `download` `store` |
| **gemini** | `new` `ask` `image` `deep-research` `deep-research-result` |
| **yuanbao** | `new` `ask` |
| **notebooklm** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` |
| **notebooklm** | `status` `list` `open` `current` `get` `history` `summary` `note-list` `notes-get` `source-list` `source-get` `source-fulltext` `source-guide` |
| **spotify** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` |
| **xianyu** | `search` `item` `chat` |
| **xiaoe** | `courses` `detail` `catalog` `play-url` `content` |
73+ adapters in total — **[→ see all supported sites & commands](./docs/adapters/index.md)**
79+ adapters in total — **[→ see all supported sites & commands](./docs/adapters/index.md)**
## CLI Hub
@@ -185,6 +191,7 @@ OpenCLI supports downloading images, videos, and articles from supported platfor
| **twitter** | Images, Videos | From user media tab or single tweet |
| **douban** | Images | Poster / still image lists |
| **pixiv** | Images | Original-quality illustrations, multi-page |
| **1688** | Images, Videos | Downloads page-visible product media from item pages |
| **zhihu** | Articles (Markdown) | Exports with optional image download |
| **weixin** | Articles (Markdown) | WeChat Official Account articles |
@@ -194,6 +201,7 @@ For video downloads, install `yt-dlp` first: `brew install yt-dlp`
opencli xiaohongshu download abc123 --output ./xhs
opencli bilibili download BV1xxx --output ./bilibili
opencli twitter download elonmusk --limit 20 --output ./twitter
opencli 1688 download 841141931191 --output ./1688-downloads
```
## Output Formats
@@ -250,9 +258,9 @@ See [Plugins Guide](./docs/guide/plugins.md) for creating your own plugin.
## For AI Agents (Developer Guide)
> **Quick mode**: To generate a single command for a specific page URL, see [CLI-ONESHOT.md](./CLI-ONESHOT.md) — just a URL + one-line goal, 4 steps done.
> **Quick mode**: To generate a single command for a specific page URL, see [opencli-oneshot skill](./skills/opencli-oneshot/SKILL.md) — just a URL + one-line goal, 4 steps done.
> **Full mode**: Before writing any adapter code, read [CLI-EXPLORER.md](./CLI-EXPLORER.md). It contains the complete browser exploration workflow, the 5-tier authentication strategy decision tree, and debugging guide.
> **Full mode**: Before writing any adapter code, read [opencli-explorer skill](./skills/opencli-explorer/SKILL.md). It contains the complete browser exploration workflow, the 5-tier authentication strategy decision tree, and debugging guide.
```bash
opencli explore https://example.com --site mysite # Discover APIs + capabilities
+20 -10
View File
@@ -23,8 +23,8 @@ CLI all electron!现在支持把所有 electron 应用 CLI 化,从而组合
- **CLI All Electron** — 支持把所有 electron 应用(如 Antigravity UltraCLI 化,让 AI 控制自己!
- **浏览器自动化** — `operate` 赋予 AI Agent 直接操控浏览器的能力:点击、输入、提取、截图,任意交互皆可脚本化
- **网页转 CLI** — 将任意网站变成确定性命令行工具:73+ 预置适配器,或用 `opencli record` 沉淀自己的操作
- **多站点覆盖** — 73+ 站点,横跨全球与中国平台(B站、知乎、小红书、Reddit、HackerNews 等),并支持通过 CDP 控制桌面 Electron 应用
- **网页转 CLI** — 将任意网站变成确定性命令行工具:79+ 预置适配器,或用 `opencli record` 沉淀自己的操作
- **多站点覆盖** — 79+ 站点,横跨全球与中国平台(B站、知乎、小红书、Reddit、HackerNews 等),并支持通过 CDP 控制桌面 Electron 应用
- **零风控** — 复用 Chrome/Chromium 登录态,无需存储任何凭证
- **外部 CLI 枢纽** — 统一发现、自动安装、透传执行 `gh``docker` 等本地 CLI
- **自修复配置** — `opencli doctor` 自动启动 daemon,诊断扩展和浏览器连接状态
@@ -69,6 +69,9 @@ OpenCLI 通过轻量化的 **Browser Bridge** Chrome/Chromium 扩展 + 微型 da
```bash
npm install -g @jackwener/opencli
# 安装 AI SkillsClaude Code / Cursor
npx skills add jackwener/opencli
```
直接使用:
@@ -129,6 +132,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
| **twitter** | `trending` `bookmarks` `profile` `search` `timeline` `thread` `following` `followers` `notifications` `post` `reply` `delete` `like` `article` `follow` `unfollow` `bookmark` `unbookmark` `download` `accept` `reply-dm` `block` `unblock` `hide-reply` | 浏览器 |
| **reddit** | `hot` `frontpage` `popular` `search` `subreddit` `read` `user` `user-posts` `user-comments` `upvote` `save` `comment` `subscribe` `saved` `upvoted` | 浏览器 |
| **tieba** | `hot` `posts` `search` `read` | 浏览器 |
| **hupu** | `hot` `search` `detail` `mentions` `reply` `like` `unlike` | 浏览器 |
| **cursor** | `status` `send` `read` `new` `dump` `composer` `model` `extract-code` `ask` `screenshot` `history` `export` | 桌面端 |
| **bilibili** | `hot` `search` `me` `favorite` `history` `feed` `subtitle` `dynamic` `ranking` `following` `user-videos` `download` | 浏览器 |
| **codex** | `status` `send` `read` `new` `dump` `extract-diff` `model` `ask` `screenshot` `history` `export` | 桌面端 |
@@ -142,6 +146,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
| **antigravity** | `status` `send` `read` `new` `dump` `extract-code` `model` `watch` | 桌面端 |
| **chatgpt** | `status` `new` `send` `read` `ask` `model` | 桌面端 |
| **xiaohongshu** | `search` `notifications` `feed` `user` `download` `publish` `creator-notes` `creator-note-detail` `creator-notes-summary` `creator-profile` `creator-stats` | 浏览器 |
| **xiaoe** | `courses` `detail` `catalog` `play-url` `content` | 浏览器 |
| **apple-podcasts** | `search` `episodes` `top` | 公开 |
| **xiaoyuzhou** | `podcast` `podcast-episodes` `episode` | 公开 |
| **zhihu** | `hot` `search` `question` `download` | 浏览器 |
@@ -163,7 +168,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
| **reuters** | `search` | 浏览器 |
| **smzdm** | `search` | 浏览器 |
| **web** | `read` | 浏览器 |
| **weibo** | `hot` `search` | 浏览器 |
| **weibo** | `hot` `search` `feed` `user` `me` `post` `comments` | 浏览器 |
| **yahoo-finance** | `quote` | 浏览器 |
| **sinafinance** | `news` | 🌐 公开 |
| **barchart** | `quote` `options` `greeks` `flow` | 浏览器 |
@@ -173,17 +178,18 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
| **jike** | `feed` `search` `create` `like` `comment` `repost` `notifications` `post` `topic` `user` | 浏览器 |
| **jimeng** | `generate` `history` | 浏览器 |
| **yollomi** | `generate` `video` `edit` `upload` `models` `remove-bg` `upscale` `face-swap` `restore` `try-on` `background` `object-remover` | 浏览器 |
| **linux-do** | `feed` `categories` `tags` `search` `topic` `user-topics` `user-posts` | 浏览器 |
| **linux-do** | `hot` `latest` `feed` `search` `categories` `category` `tags` `topic` `topic-content` `user-posts` `user-topics` | 浏览器 |
| **stackoverflow** | `hot` `search` `bounties` `unanswered` | 公开 |
| **steam** | `top-sellers` | 公开 |
| **weread** | `shelf` `search` `book` `highlights` `notes` `notebooks` `ranking` | 浏览器 |
| **douban** | `search` `top250` `subject` `photos` `download` `marks` `reviews` `movie-hot` `book-hot` | 浏览器 |
| **facebook** | `feed` `profile` `search` `friends` `groups` `events` `notifications` `memories` `add-friend` `join-group` | 浏览器 |
| **google** | `news` `search` `suggest` `trends` | 公开 |
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` | 浏览器 |
| **gemini** | `new` `ask` `image` | 浏览器 |
| **amazon** | `bestsellers` `search` `product` `offer` `discussion` `movers-shakers` `new-releases` | 浏览器 |
| **1688** | `search` `item` `assets` `download` `store` | 浏览器 |
| **gemini** | `new` `ask` `image` `deep-research` `deep-research-result` | 浏览器 |
| **spotify** | `auth` `status` `play` `pause` `next` `prev` `volume` `search` `queue` `shuffle` `repeat` | OAuth API |
| **notebooklm** | `status` `list` `open` `select` `current` `get` `metadata` `source-list` `source-get` `source-fulltext` `source-guide` `history` `note-list` `notes-list` `notes-get` `summary` | 浏览器 |
| **notebooklm** | `status` `list` `open` `current` `get` `history` `summary` `note-list` `notes-get` `source-list` `source-get` `source-fulltext` `source-guide` | 浏览器 |
| **36kr** | `news` `hot` `search` `article` | 公开 / 浏览器 |
| **imdb** | `search` `title` `top` `trending` `person` `reviews` | 公开 |
| **producthunt** | `posts` `today` `hot` `browse` | 公开 / 浏览器 |
@@ -199,7 +205,7 @@ npx skills add jackwener/opencli --skill opencli-oneshot # 快速命令参
| **douyin** | `videos` `publish` `drafts` `draft` `delete` `stats` `profile` `update` `hashtag` `location` `activities` `collections` | 浏览器 |
| **yuanbao** | `new` `ask` | 浏览器 |
73+ 适配器 — **[→ 查看完整命令列表](./docs/adapters/index.md)**
79+ 适配器 — **[→ 查看完整命令列表](./docs/adapters/index.md)**
### 外部 CLI 枢纽
@@ -252,6 +258,7 @@ OpenCLI 支持从各平台下载图片、视频和文章。
| **B站** | 视频 | 需要安装 `yt-dlp` |
| **Twitter/X** | 图片、视频 | 从用户媒体页或单条推文下载 |
| **Pixiv** | 图片 | 下载原始画质插画,支持多页作品 |
| **1688** | 图片、视频 | 下载商品页中可见的商品素材 |
| **知乎** | 文章(Markdown) | 导出文章,可选下载图片到本地 |
| **微信公众号** | 文章(Markdown | 导出微信公众号文章为 Markdown |
| **豆瓣** | 图片 | 下载电影条目的海报 / 剧照图片 |
@@ -286,6 +293,9 @@ opencli twitter download --tweet-url "https://x.com/user/status/123" --output ./
# 下载豆瓣电影海报 / 剧照
opencli douban download 30382501 --output ./douban
# 下载 1688 商品页中的图片 / 视频素材
opencli 1688 download 841141931191 --output ./1688-downloads
# 导出知乎文章为 Markdown
opencli zhihu download "https://zhuanlan.zhihu.com/p/xxx" --output ./zhihu
@@ -364,9 +374,9 @@ opencli plugin uninstall my-tool # 卸载
如果你是一个被要求查阅代码并编写新 `opencli` 适配器的 AI,请遵守以下工作流。
> **快速模式**:只想为某个页面快速生成一个命令?看 [CLI-ONESHOT.md](./CLI-ONESHOT.md) — 给一个 URL + 一句话描述,4 步搞定。
> **快速模式**:只想为某个页面快速生成一个命令?看 [opencli-oneshot skill](./skills/opencli-oneshot/SKILL.md) — 给一个 URL + 一句话描述,4 步搞定。
> **完整模式**:在编写任何新代码前,先阅读 [CLI-EXPLORER.md](./CLI-EXPLORER.md)。它包含完整的适配器探索开发指南、API 探测流程、5级认证策略以及常见陷阱。
> **完整模式**:在编写任何新代码前,先阅读 [opencli-explorer skill](./skills/opencli-explorer/SKILL.md)。它包含完整的适配器探索开发指南、API 探测流程、5级认证策略以及常见陷阱。
```bash
# 1. Deep Explore — 网络拦截 → 响应分析 → 能力推理 → 框架检测
+5 -5
View File
@@ -49,7 +49,7 @@ src/
|---|---|
| 核心运行时与输出 | `src/browser.test.ts`, `src/browser/dom-snapshot.test.ts`, `src/build-manifest.test.ts`, `src/capabilityRouting.test.ts`, `src/doctor.test.ts`, `src/engine.test.ts`, `src/interceptor.test.ts`, `src/output.test.ts`, `src/plugin.test.ts`, `src/registry.test.ts`, `src/snapshotFormatter.test.ts` |
| pipeline 与下载 | `src/download/index.test.ts`, `src/pipeline/executor.test.ts`, `src/pipeline/template.test.ts`, `src/pipeline/transform.test.ts` |
| 站点 / adapter 逻辑 | `src/clis/apple-podcasts/commands.test.ts`, `src/clis/apple-podcasts/utils.test.ts`, `src/clis/bloomberg/utils.test.ts`, `src/clis/chaoxing/utils.test.ts`, `src/clis/coupang/utils.test.ts`, `src/clis/google/utils.test.ts`, `src/clis/grok/ask.test.ts`, `src/clis/twitter/timeline.test.ts`, `src/clis/weread/utils.test.ts`, `src/clis/xiaohongshu/creator-note-detail.test.ts`, `src/clis/xiaohongshu/creator-notes-summary.test.ts`, `src/clis/xiaohongshu/creator-notes.test.ts`, `src/clis/xiaohongshu/search.test.ts`, `src/clis/xiaohongshu/user-helpers.test.ts`, `src/clis/xiaoyuzhou/utils.test.ts`, `src/clis/youtube/transcript-group.test.ts`, `src/clis/zhihu/download.test.ts` |
| 站点 / adapter 逻辑 | `clis/apple-podcasts/commands.test.ts`, `clis/apple-podcasts/utils.test.ts`, `clis/bloomberg/utils.test.ts`, `clis/chaoxing/utils.test.ts`, `clis/coupang/utils.test.ts`, `clis/google/utils.test.ts`, `clis/grok/ask.test.ts`, `clis/twitter/timeline.test.ts`, `clis/weread/utils.test.ts`, `clis/xiaohongshu/creator-note-detail.test.ts`, `clis/xiaohongshu/creator-notes-summary.test.ts`, `clis/xiaohongshu/creator-notes.test.ts`, `clis/xiaohongshu/search.test.ts`, `clis/xiaohongshu/user-helpers.test.ts`, `clis/xiaoyuzhou/utils.test.ts`, `clis/youtube/transcript-group.test.ts`, `clis/zhihu/download.test.ts` |
这些测试覆盖的重点包括:
@@ -94,7 +94,7 @@ find tests/smoke -name '*.test.ts' | sort
```bash
npm ci # 安装依赖
npm run build # 编译(E2E / smoke 测试需要 dist/main.js
npm run build # 编译(E2E / smoke 测试需要 dist/src/main.js
```
### 运行命令
@@ -110,7 +110,7 @@ npx vitest run tests/e2e/
npx vitest run tests/smoke/
# 单个测试文件
npx vitest run src/clis/apple-podcasts/commands.test.ts
npx vitest run clis/apple-podcasts/commands.test.ts
npx vitest run tests/e2e/management.test.ts
# 全部测试
@@ -123,7 +123,7 @@ npx vitest src/
### 浏览器命令本地测试须知
- opencli 通过 Browser Bridge 扩展连接已运行的 Chrome 浏览器
- E2E 测试通过 `tests/e2e/helpers.ts` 里的 `runCli()` 调用已构建的 `dist/main.js`
- E2E 测试通过 `tests/e2e/helpers.ts` 里的 `runCli()` 调用已构建的 `dist/src/main.js`
- `browser-public.test.ts` 使用 `tryBrowserCommand()`,站点反爬或地域限制导致空数据时会 warn + pass
- `browser-auth.test.ts` 验证 **graceful failure**,重点是不 crash、不 hang、错误信息可控
- 如需测试完整登录态,保持 Chrome 登录态并安装 Browser Bridge 扩展,再手动运行对应测试
@@ -132,7 +132,7 @@ npx vitest src/
## 如何添加新测试
### 新增 YAML Adapter(如 `src/clis/producthunt/trending.yaml`
### 新增 YAML Adapter(如 `clis/producthunt/trending.yaml`
1. `opencli validate` 的 E2E / smoke 测试会覆盖 adapter 结构校验
2. 根据 adapter 类型,在对应测试文件补一个 `it()` block
+13 -17
View File
@@ -14,7 +14,7 @@
"name": "extract-title-iana",
"steps": [
"opencli operate open https://www.iana.org",
"opencli operate eval \"document.querySelector('h1')?.textContent\""
"opencli operate eval \"document.querySelector('h1')?.textContent || document.title || document.querySelector('title')?.textContent\""
],
"judge": {
"type": "nonEmpty"
@@ -67,7 +67,7 @@
"name": "extract-github-readme-heading",
"steps": [
"opencli operate open https://github.com/vercel/next.js",
"opencli operate eval \"document.querySelector('article h1, article h2')?.textContent?.trim()\""
"opencli operate eval \"document.querySelector('[data-testid=readme] h1, [data-testid=readme] h2, #readme h1, #readme h2, article h1, article h2, .markdown-body h1, .markdown-body h2')?.textContent?.trim()\""
],
"judge": {
"type": "nonEmpty"
@@ -143,7 +143,7 @@
"name": "list-quotes-3",
"steps": [
"opencli operate open https://quotes.toscrape.com",
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote')].slice(0,3).map(el=>({text:el.querySelector('.text')?.textContent,author:el.querySelector('.author')?.textContent})))\""
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('.quote, [class*=quote]')].slice(0,3).map(el=>({text:(el.querySelector('.text, [class*=text]')?.textContent)||(el.querySelector('span')?.textContent),author:(el.querySelector('.author, [class*=author]')?.textContent)||(el.querySelector('small')?.textContent)})))\""
],
"judge": {
"type": "arrayMinLength",
@@ -165,7 +165,7 @@
"name": "list-github-trending",
"steps": [
"opencli operate open https://github.com/trending",
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row')].slice(0,3).map(el=>({name:el.querySelector('h2 a')?.textContent?.trim().replace(/\\s+/g,' '),desc:el.querySelector('p')?.textContent?.trim()})))\""
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row, article[class*=Box-row], [data-hpc] article, .Box article')].slice(0,3).map(el=>({name:(el.querySelector('h2 a, h1 a')?.textContent?.trim().replace(/\\\\s+/g,' '))||(el.querySelector('a[href^=\\\"/\\\"]')?.textContent?.trim()),desc:el.querySelector('p')?.textContent?.trim()})))\""
],
"judge": {
"type": "arrayMinLength",
@@ -176,7 +176,7 @@
"name": "list-github-trending-lang",
"steps": [
"opencli operate open https://github.com/trending/python",
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row')].slice(0,5).map(el=>({name:el.querySelector('h2 a')?.textContent?.trim().replace(/\\s+/g,' ')})))\""
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('article.Box-row, article[class*=Box-row], [data-hpc] article, .Box article')].slice(0,5).map(el=>({name:(el.querySelector('h2 a, h1 a')?.textContent?.trim().replace(/\\\\s+/g,' '))||(el.querySelector('a[href^=\\\"/\\\"]')?.textContent?.trim())})))\""
],
"judge": {
"type": "arrayMinLength",
@@ -208,8 +208,7 @@
{
"name": "search-google",
"steps": [
"opencli operate open https://www.google.com",
"opencli operate eval \"document.querySelector('textarea[name=q], input[name=q]').value='opencli github';document.querySelector('form').submit();'submitted'\"",
"opencli operate open https://www.google.com/search?q=opencli+github",
"opencli operate wait time 3",
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('h3')].slice(0,3).map(h=>h.textContent))\""
],
@@ -250,8 +249,7 @@
{
"name": "search-wiki",
"steps": [
"opencli operate open https://en.wikipedia.org",
"opencli operate eval \"document.querySelector('input[name=search]').value='Rust programming language';document.querySelector('form#searchform, form[role=search]').submit();'submitted'\"",
"opencli operate open \"https://en.wikipedia.org/w/index.php?search=Rust+programming+language&title=Special:Search&go=Go\"",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const ps = document.querySelectorAll('#mw-content-text .mw-parser-output > p'); for (const p of ps) { const t = p.textContent?.trim(); if (t && t.length > 50) return t.slice(0,300); } return ''; })()\""
],
@@ -264,11 +262,9 @@
{
"name": "search-npm",
"steps": [
"opencli operate open https://www.npmjs.com",
"opencli operate state",
"opencli operate type 1 \"react\"",
"opencli operate keys Enter",
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=pkg-list-item] h3, section h3')].slice(0,3).map(h=>h.textContent?.trim()))\""
"opencli operate open https://www.npmjs.com/search?q=react",
"opencli operate wait time 3",
"opencli operate eval \"JSON.stringify([...document.querySelectorAll('[data-testid=pkg-list-item] h3, section h3, .package-list-item h3, a[class*=package] h3')].slice(0,3).map(h=>h.textContent?.trim()))\""
],
"judge": {
"type": "arrayMinLength",
@@ -329,8 +325,8 @@
"name": "nav-click-wiki-link",
"steps": [
"opencli operate open https://en.wikipedia.org/wiki/JavaScript",
"opencli operate eval \"document.querySelector('#toc a, .toc a, [href=\\\"#History\\\"]')?.click(); 'clicked'\"",
"opencli operate eval \"document.querySelector('#History, #History ~ p')?.textContent?.slice(0,100)\""
"opencli operate eval \"document.querySelector('.vector-toc-contents a[href*=History], #toc a[href*=History], .toc a[href*=History], [href=\\\"#History\\\"]')?.click(); 'clicked'\"",
"opencli operate eval \"document.querySelector('#History')?.textContent?.slice(0,100) || document.querySelector('[id*=History]')?.textContent?.slice(0,100)\""
],
"judge": {
"type": "nonEmpty"
@@ -480,7 +476,7 @@
"name": "form-textarea",
"steps": [
"opencli operate open https://httpbin.org/forms/post",
"opencli operate eval \"var ta=document.querySelector('textarea[name=comments]');ta.value='AutoResearch test';ta.dispatchEvent(new Event('input',{bubbles:true}));ta.value\""
"opencli operate eval \"var ta=document.querySelector('textarea[name=comments], textarea[name=delivery], textarea');ta.value='AutoResearch test';ta.dispatchEvent(new Event('input',{bubbles:true}));ta.value\""
],
"judge": {
"type": "contains",
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env npx tsx
/**
* Layer 5: Publish Testing — end-to-end content creation via operate commands
*
* Tests the full chain: read content → navigate to platform → fill title+body → (optionally) publish → verify → cleanup
*
* Task types:
* fill-only: navigate + fill fields + verify content was entered (safe, no side effects)
* publish: full publish + verify + cleanup (deletes the post after verification)
*
* Usage:
* npx tsx autoresearch/eval-publish.ts # Run all tasks
* npx tsx autoresearch/eval-publish.ts --task twitter-fill # Run single task
* npx tsx autoresearch/eval-publish.ts --type fill-only # Run only fill tasks (safe)
* npx tsx autoresearch/eval-publish.ts --type publish # Run only publish tasks (destructive)
* npx tsx autoresearch/eval-publish.ts --platform twitter # Run only twitter tasks
*/
import { execSync } from 'node:child_process';
import { readFileSync, writeFileSync, mkdirSync, readdirSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
const __dirname = dirname(fileURLToPath(import.meta.url));
const PROJECT_ROOT = join(__dirname, '..');
const TASKS_FILE = join(__dirname, 'publish-tasks.json');
const RESULTS_DIR = join(__dirname, 'results');
interface PublishTask {
name: string;
platform: string;
type: 'fill-only' | 'publish';
description: string;
steps: string[];
judge: JudgeCriteria;
cleanup?: string[];
note?: string;
}
type JudgeCriteria =
| { type: 'contains'; value: string }
| { type: 'arrayMinLength'; minLength: number }
| { type: 'nonEmpty' }
| { type: 'matchesPattern'; pattern: string };
interface TaskResult {
name: string;
platform: string;
taskType: 'fill-only' | 'publish';
passed: boolean;
duration: number;
cleanupResult?: string;
error?: string;
}
function judge(criteria: JudgeCriteria, output: string): boolean {
try {
switch (criteria.type) {
case 'contains':
return output.toLowerCase().includes(criteria.value.toLowerCase());
case 'arrayMinLength': {
try {
const arr = JSON.parse(output);
if (Array.isArray(arr)) return arr.length >= criteria.minLength;
} catch { /* not JSON */ }
return false;
}
case 'nonEmpty':
return output.trim().length > 0 && output.trim() !== 'null' && output.trim() !== 'undefined';
case 'matchesPattern':
return new RegExp(criteria.pattern, 'i').test(output);
default:
return false;
}
} catch {
return false;
}
}
function runCommand(cmd: string, timeout = 30000): string {
const localCmd = cmd.replace(/^opencli /, `node dist/src/main.js `);
try {
return execSync(localCmd, {
cwd: PROJECT_ROOT,
timeout,
encoding: 'utf-8',
env: process.env,
stdio: ['pipe', 'pipe', 'pipe'],
}).trim();
} catch (err: any) {
return err.stdout?.trim() || err.stderr?.trim() || '';
}
}
function runTask(task: PublishTask): TaskResult {
const start = Date.now();
try {
// Run main steps
let lastOutput = '';
for (let i = 0; i < task.steps.length; i++) {
const step = task.steps[i];
process.stderr.write(` step ${i + 1}/${task.steps.length}: ${step.slice(0, 60)}...\n`);
lastOutput = runCommand(step, 45000);
}
const passed = judge(task.judge, lastOutput);
// Run cleanup steps (if publish type and cleanup defined)
let cleanupResult: string | undefined;
if (task.cleanup && task.cleanup.length > 0) {
process.stderr.write(` cleanup: ${task.cleanup.length} steps...\n`);
let cleanupOutput = '';
for (const step of task.cleanup) {
cleanupOutput = runCommand(step, 30000);
}
cleanupResult = cleanupOutput.slice(0, 100);
}
return {
name: task.name,
platform: task.platform,
taskType: task.type,
passed,
duration: Date.now() - start,
cleanupResult,
error: passed ? undefined : `Output: ${lastOutput.slice(0, 150)}`,
};
} catch (err: any) {
return {
name: task.name,
platform: task.platform,
taskType: task.type,
passed: false,
duration: Date.now() - start,
error: err.message?.slice(0, 150),
};
}
}
function main() {
const args = process.argv.slice(2);
const singleTask = args.includes('--task') ? args[args.indexOf('--task') + 1] : null;
const filterType = args.includes('--type') ? args[args.indexOf('--type') + 1] : null;
const filterPlatform = args.includes('--platform') ? args[args.indexOf('--platform') + 1] : null;
const allTasks: PublishTask[] = JSON.parse(readFileSync(TASKS_FILE, 'utf-8'));
let tasks = allTasks;
if (singleTask) tasks = tasks.filter(t => t.name === singleTask);
if (filterType) tasks = tasks.filter(t => t.type === filterType);
if (filterPlatform) tasks = tasks.filter(t => t.platform === filterPlatform);
if (tasks.length === 0) {
console.error(`No tasks matched filters: task=${singleTask}, type=${filterType}, platform=${filterPlatform}`);
process.exit(1);
}
const fillTasks = tasks.filter(t => t.type === 'fill-only');
const publishTasks = tasks.filter(t => t.type === 'publish');
console.log(`\n📝 Layer 5: Publish Testing — ${tasks.length} tasks`);
console.log(` fill-only: ${fillTasks.length} | publish: ${publishTasks.length}`);
console.log(` platforms: ${[...new Set(tasks.map(t => t.platform))].join(', ')}\n`);
const results: TaskResult[] = [];
for (let i = 0; i < tasks.length; i++) {
const task = tasks[i];
const icon = task.type === 'publish' ? '🚀' : '📋';
process.stdout.write(` [${i + 1}/${tasks.length}] ${icon} ${task.name} (${task.platform})...`);
const result = runTask(task);
results.push(result);
const status = result.passed ? '✓' : '✗';
const cleanup = result.cleanupResult ? ` [cleanup: ${result.cleanupResult.slice(0, 30)}]` : '';
console.log(` ${status} (${(result.duration / 1000).toFixed(1)}s)${cleanup}`);
// Close browser between tasks for clean state
if (i < tasks.length - 1) {
try { runCommand('opencli operate close'); } catch { /* ignore */ }
}
}
// Final close
try { runCommand('opencli operate close'); } catch { /* ignore */ }
// Summary
const totalPassed = results.filter(r => r.passed).length;
const fillPassed = results.filter(r => r.taskType === 'fill-only' && r.passed).length;
const publishPassed = results.filter(r => r.taskType === 'publish' && r.passed).length;
const totalDuration = results.reduce((s, r) => s + r.duration, 0);
const fillTotal = results.filter(r => r.taskType === 'fill-only').length;
const publishTotal = results.filter(r => r.taskType === 'publish').length;
console.log(`\n${'─'.repeat(50)}`);
console.log(` Score: ${totalPassed}/${results.length}`);
console.log(` fill-only: ${fillPassed}/${fillTotal}`);
console.log(` publish: ${publishPassed}/${publishTotal}`);
console.log(` Time: ${Math.round(totalDuration / 1000)}s`);
// Platform breakdown
const platforms = [...new Set(results.map(r => r.platform))];
for (const p of platforms) {
const pr = results.filter(r => r.platform === p);
const pp = pr.filter(r => r.passed).length;
console.log(` ${p}: ${pp}/${pr.length}`);
}
const failures = results.filter(r => !r.passed);
if (failures.length > 0) {
console.log(`\n Failures:`);
for (const f of failures) {
console.log(`${f.name} [${f.platform}/${f.taskType}]: ${f.error ?? 'unknown'}`);
}
}
console.log('');
// Save result
mkdirSync(RESULTS_DIR, { recursive: true });
const existing = readdirSync(RESULTS_DIR).filter(f => f.startsWith('publish-')).length;
const roundNum = String(existing + 1).padStart(3, '0');
const resultPath = join(RESULTS_DIR, `publish-${roundNum}.json`);
writeFileSync(resultPath, JSON.stringify({
timestamp: new Date().toISOString(),
score: `${totalPassed}/${results.length}`,
fillScore: `${fillPassed}/${fillTotal}`,
publishScore: `${publishPassed}/${publishTotal}`,
duration: `${Math.round(totalDuration / 1000)}s`,
tasks: results,
}, null, 2), 'utf-8');
console.log(` Results saved to: ${resultPath}`);
console.log(`\nSCORE=${totalPassed}/${results.length}`);
}
main();
+2 -2
View File
@@ -80,10 +80,10 @@ function judge(criteria: JudgeCriteria, output: string): boolean {
const PROJECT_ROOT = join(__dirname, '..');
/** Run a command, using local dist/main.js instead of global opencli for consistency */
/** Run a command, using the local built entrypoint instead of global opencli for consistency */
function runCommand(cmd: string, timeout = 30000): string {
// Use local build so tests always run against the current source
const localCmd = cmd.replace(/^opencli /, `node dist/main.js `);
const localCmd = cmd.replace(/^opencli /, `node dist/src/main.js `);
try {
return execSync(localCmd, {
cwd: PROJECT_ROOT,
+345
View File
@@ -0,0 +1,345 @@
[
{
"name": "twitter-fill-compose",
"platform": "twitter",
"type": "fill-only",
"description": "Navigate to tweet composer, fill in content (no publish)",
"steps": [
"opencli operate open https://x.com/compose/tweet",
"opencli operate wait time 3",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]') ? 'composer-ready' : 'not-found'\"",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] OpenCLI publish eval - fill only test ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "3-step: open compose → paste text via ClipboardEvent → verify text in composer"
},
{
"name": "twitter-post-and-delete",
"platform": "twitter",
"type": "publish",
"description": "Post a tweet, verify success, then delete it",
"steps": [
"opencli operate open https://x.com/compose/tweet",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] OpenCLI publish eval ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const btn = document.querySelector('[data-testid=\\\"tweetButton\\\"]') || document.querySelector('[data-testid=\\\"tweetButtonInline\\\"]'); if (btn && !btn.disabled) { btn.click(); return 'clicked'; } return 'btn-not-ready'; })()\"",
"opencli operate wait time 4",
"opencli operate eval \"document.querySelector('[data-testid=\\\"toast\\\"]')?.textContent || document.title\""
],
"judge": {
"type": "matchesPattern",
"pattern": "post|sent|Your post|X"
},
"cleanup": [
"opencli operate open https://x.com/home",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const tweets = document.querySelectorAll('[data-testid=\\\"tweet\\\"]'); for (const t of tweets) { if (t.textContent?.includes('[AutoTest]')) { const more = t.querySelector('[data-testid=\\\"caret\\\"]'); if (more) { more.click(); return 'found-menu'; } } } return 'no-autotest-tweet'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const items = document.querySelectorAll('[role=\\\"menuitem\\\"]'); for (const item of items) { if (item.textContent?.includes('Delete')) { item.click(); return 'clicked-delete'; } } return 'no-delete-option'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const confirm = document.querySelector('[data-testid=\\\"confirmationSheetConfirm\\\"]'); if (confirm) { confirm.click(); return 'deleted'; } return 'no-confirm'; })()\""
],
"note": "6-step chain: open compose → paste text → click post → wait → verify toast → cleanup: find tweet → menu → delete → confirm"
},
{
"name": "twitter-read-hn-then-post",
"platform": "twitter",
"type": "publish",
"description": "Read HN top story title, compose a tweet about it, post, then delete",
"steps": [
"opencli operate open https://news.ycombinator.com",
"opencli operate wait time 2",
"opencli operate eval \"document.querySelector('.titleline a')?.textContent?.trim() || 'no-title'\"",
"opencli operate open https://x.com/compose/tweet",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const title = document.title || 'HN Story'; const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Interesting from HN: ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const btn = document.querySelector('[data-testid=\\\"tweetButton\\\"]') || document.querySelector('[data-testid=\\\"tweetButtonInline\\\"]'); if (btn && !btn.disabled) { btn.click(); return 'clicked'; } return 'btn-not-ready'; })()\"",
"opencli operate wait time 4",
"opencli operate eval \"document.querySelector('[data-testid=\\\"toast\\\"]')?.textContent || document.title\""
],
"judge": {
"type": "matchesPattern",
"pattern": "post|sent|Your post|X"
},
"cleanup": [
"opencli operate open https://x.com/home",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const tweets = document.querySelectorAll('[data-testid=\\\"tweet\\\"]'); for (const t of tweets) { if (t.textContent?.includes('[AutoTest]')) { const more = t.querySelector('[data-testid=\\\"caret\\\"]'); if (more) { more.click(); return 'found-menu'; } } } return 'no-autotest-tweet'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const items = document.querySelectorAll('[role=\\\"menuitem\\\"]'); for (const item of items) { if (item.textContent?.includes('Delete')) { item.click(); return 'clicked-delete'; } } return 'no-delete-option'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const confirm = document.querySelector('[data-testid=\\\"confirmationSheetConfirm\\\"]'); if (confirm) { confirm.click(); return 'deleted'; } return 'no-confirm'; })()\""
],
"note": "9-step cross-site chain: read HN title → navigate to twitter compose → paste content → post → verify → cleanup delete"
},
{
"name": "twitter-reply-to-own-tweet",
"platform": "twitter",
"type": "fill-only",
"description": "Navigate to own profile, find latest tweet, open reply box, fill reply text",
"steps": [
"opencli operate open https://x.com/home",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const tweet = document.querySelector('[data-testid=\\\"tweet\\\"]'); if (!tweet) return 'no-tweet'; const reply = tweet.querySelector('[data-testid=\\\"reply\\\"]'); if (reply) { reply.click(); return 'reply-clicked'; } return 'no-reply-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Reply test ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "5-step: home → find first tweet → click reply → fill reply text → verify content"
},
{
"name": "zhihu-fill-answer",
"platform": "zhihu",
"type": "fill-only",
"description": "Navigate to a popular question, open answer editor, fill in answer content (no publish)",
"steps": [
"opencli operate open https://www.zhihu.com/question/19550225",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const btn = document.querySelector('[data-zop-retarget=\\\"answer\\\"]') || Array.from(document.querySelectorAll('button')).find(b => b.textContent?.includes('写回答')); if (btn) { btn.click(); return 'editor-opened'; } return 'no-answer-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] 这是一个 OpenCLI 发文测试,时间戳: ' + Date.now() + '</p><p>这段内容用于验证 operate 命令链的完整性。</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "5-step: navigate to question → click '写回答' → find editor → fill rich content (title + body) → verify"
},
{
"name": "zhihu-fill-article",
"platform": "zhihu",
"type": "fill-only",
"description": "Navigate to zhihu article editor (zhuanlan), fill title + body (no publish)",
"steps": [
"opencli operate open https://zhuanlan.zhihu.com/write",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const ta = document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder]'); if (!ta) return 'no-title-input'; ta.focus(); var nativeSetter = Object.getOwnPropertyDescriptor(window.HTMLTextAreaElement.prototype, 'value').set; nativeSetter.call(ta, '[AutoTest] OpenCLI 发文能力验证 ' + Date.now()); ta.dispatchEvent(new Event('input', { bubbles: true })); return 'title-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const editor = document.querySelector('[contenteditable=true]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>这是 OpenCLI autoresearch 发文测试集的一部分。</p><p>测试链路:导航 → 填写标题 → 填写正文 → 验证内容。</p><p>时间戳: ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'body-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const title = (document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder]'))?.value || ''; const body = document.querySelector('[contenteditable=true]')?.textContent || ''; return JSON.stringify({ title: title.slice(0, 50), body: body.slice(0, 50) }); })()\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "5-step: navigate to zhuanlan editor → fill title textarea → fill rich text body → verify both title and body content"
},
{
"name": "zhihu-read-hn-fill-answer",
"platform": "zhihu",
"type": "fill-only",
"description": "Read HN top story, then navigate to zhihu question and fill an answer about it",
"steps": [
"opencli operate open https://news.ycombinator.com",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const a = document.querySelector('.titleline a'); return a ? a.textContent?.trim() : 'no-title'; })()\"",
"opencli operate open https://www.zhihu.com/question/19550225",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const btn = Array.from(document.querySelectorAll('button')).find(b => b.textContent?.includes('写回答')) || document.querySelector('[data-zop-retarget=\\\"answer\\\"]'); if (btn) { btn.click(); return 'editor-opened'; } return 'no-answer-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=true]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] 分享一个来自 Hacker News 的有趣内容</p><p>这是一个跨平台内容搬运测试,时间戳: ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(document.querySelector('.ql-editor') || document.querySelector('[contenteditable=true]'))?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "8-step cross-site chain: read HN title → navigate zhihu question → click 写回答 → fill answer with HN content → verify"
},
{
"name": "twitter-thread-compose",
"platform": "twitter",
"type": "fill-only",
"description": "Navigate to compose, type first tweet, add thread tweet, type second tweet, verify both",
"steps": [
"opencli operate open https://x.com/compose/tweet",
"opencli operate wait time 3",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]') ? 'composer-ready' : 'not-found'\"",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Thread tweet 1 - ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'first-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const addBtn = document.querySelector('[data-testid=\\\"addButton\\\"]') || document.querySelector('[aria-label=\\\"Add post\\\"]'); if (addBtn) { addBtn.click(); return 'thread-added'; } return 'no-add-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const boxes = document.querySelectorAll('[data-testid=\\\"tweetTextarea_0\\\"]'); const box = boxes[boxes.length - 1]; if (!box) return 'no-second-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Thread tweet 2 - continuation'); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'second-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const boxes = document.querySelectorAll('[data-testid=\\\"tweetTextarea_0\\\"]'); const t1 = boxes[0]?.textContent || ''; const t2 = boxes[boxes.length - 1]?.textContent || ''; return JSON.stringify({ tweet1: t1, tweet2: t2 }); })()\""
],
"judge": {
"type": "contains",
"value": "Thread tweet 2"
},
"note": "10-step thread compose: open composer → fill tweet 1 → click add thread → fill tweet 2 → verify both tweets present"
},
{
"name": "twitter-quote-retweet-fill",
"platform": "twitter",
"type": "fill-only",
"description": "Navigate to home, find first tweet, open retweet menu, select Quote, fill quote text, verify",
"steps": [
"opencli operate open https://x.com/home",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const tweet = document.querySelector('[data-testid=\\\"tweet\\\"]'); if (!tweet) return 'no-tweet'; const retweet = tweet.querySelector('[data-testid=\\\"retweet\\\"]'); if (retweet) { retweet.click(); return 'retweet-menu-opened'; } return 'no-retweet-btn'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const items = document.querySelectorAll('[role=\\\"menuitem\\\"]'); for (const item of items) { if (item.textContent?.includes('Quote') || item.textContent?.includes('引用')) { item.click(); return 'quote-selected'; } } return 'no-quote-option'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Quote retweet test ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'quote-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "Quote retweet test"
},
"note": "8-step quote retweet: home → find tweet → click retweet → select Quote → fill quote text → verify"
},
{
"name": "twitter-search-then-reply-fill",
"platform": "twitter",
"type": "fill-only",
"description": "Search 'opencli' on twitter, find first result, click reply, fill reply text, verify",
"steps": [
"opencli operate open https://x.com/search?q=opencli&src=typed_query&f=live",
"opencli operate wait time 4",
"opencli operate eval \"(() => { const tweets = document.querySelectorAll('[data-testid=\\\"tweet\\\"]'); if (tweets.length === 0) return 'no-results'; return 'found-' + tweets.length + '-results'; })()\"",
"opencli operate eval \"(() => { const tweet = document.querySelector('[data-testid=\\\"tweet\\\"]'); if (!tweet) return 'no-tweet'; const reply = tweet.querySelector('[data-testid=\\\"reply\\\"]'); if (reply) { reply.click(); return 'reply-clicked'; } return 'no-reply-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Reply from search result ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'reply-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "Reply from search"
},
"note": "8-step search-then-reply: navigate to search URL → verify results → click reply on first → fill reply → verify"
},
{
"name": "zhihu-search-then-fill-answer",
"platform": "zhihu",
"type": "fill-only",
"description": "Search 'AI agent' on zhihu, click first question result, click 写回答, fill answer, verify",
"steps": [
"opencli operate open https://www.zhihu.com/search?type=content&q=AI%20agent",
"opencli operate wait time 4",
"opencli operate eval \"(() => { const links = document.querySelectorAll('a[href*=\\\"/question/\\\"]'); if (links.length === 0) return 'no-question-links'; const link = links[0]; const href = link.getAttribute('href'); return 'found: ' + href; })()\"",
"opencli operate eval \"(() => { const links = document.querySelectorAll('a[href*=\\\"/question/\\\"]'); if (links.length === 0) return 'no-links'; const link = links[0]; const href = link.getAttribute('href'); const match = href.match(/\\\\/question\\\\/(\\\\d+)/); if (match) { window.location.href = 'https://www.zhihu.com/question/' + match[1]; return 'navigating-to-question'; } link.click(); return 'clicked-link'; })()\"",
"opencli operate wait time 4",
"opencli operate eval \"(() => { const btn = document.querySelector('[data-zop-retarget=\\\"answer\\\"]') || Array.from(document.querySelectorAll('button')).find(b => b.textContent?.includes('写回答')) || Array.from(document.querySelectorAll('a')).find(a => a.textContent?.includes('写回答')); if (btn) { btn.click(); return 'editor-opened'; } return 'no-answer-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] AI agent 搜索后回答测试 ' + Date.now() + '</p><p>这是通过搜索 → 进入问题 → 填写回答的完整链路测试。</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "9-step search-then-answer: search zhihu → find question link → navigate → click 写回答 → fill answer → verify"
},
{
"name": "zhihu-read-question-fill-comment",
"platform": "zhihu",
"type": "fill-only",
"description": "Navigate to question page, scroll to first answer, click comment, fill comment text, verify",
"steps": [
"opencli operate open https://www.zhihu.com/question/19550225",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const answer = document.querySelector('[data-testid=\\\"answer\\\"]') || document.querySelector('.AnswerItem') || document.querySelector('.List-item'); if (answer) { answer.scrollIntoView({ behavior: 'smooth', block: 'center' }); return 'answer-scrolled'; } return 'no-answer'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const commentBtns = document.querySelectorAll('button'); for (const btn of commentBtns) { if (btn.textContent?.match(/评论|条评论|comment/i)) { btn.click(); return 'comment-opened: ' + btn.textContent.trim(); } } const commentIcons = document.querySelectorAll('[data-testid=\\\"comment\\\"]') || []; for (const icon of commentIcons) { icon.click(); return 'comment-icon-clicked'; } return 'no-comment-btn'; })()\"",
"opencli operate wait time 2",
"opencli operate eval \"(() => { const editor = document.querySelector('.CommentEditor textarea') || document.querySelector('[placeholder*=\\\"评论\\\"]') || document.querySelector('[placeholder*=\\\"comment\\\"]') || document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-comment-editor'; editor.focus(); if (editor.tagName === 'TEXTAREA' || editor.tagName === 'INPUT') { editor.value = '[AutoTest] 评论测试 ' + Date.now(); editor.dispatchEvent(new Event('input', { bubbles: true })); } else { editor.innerHTML = '<p>[AutoTest] 评论测试 ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); } return 'comment-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const editor = document.querySelector('.CommentEditor textarea') || document.querySelector('[placeholder*=\\\"评论\\\"]') || document.querySelector('[placeholder*=\\\"comment\\\"]') || document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; return editor.value || editor.textContent || ''; })()\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "8-step comment chain: navigate question → scroll to answer → click comment → fill comment text → verify"
},
{
"name": "zhihu-article-with-formatting",
"platform": "zhihu",
"type": "fill-only",
"description": "Navigate to zhuanlan editor, fill title, fill body with multiple paragraphs and bold text, verify",
"steps": [
"opencli operate open https://zhuanlan.zhihu.com/write",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const titleInput = document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'); if (!titleInput) return 'no-title-input'; titleInput.focus(); titleInput.value = '[AutoTest] 格式化文章测试 ' + Date.now(); titleInput.dispatchEvent(new Event('input', { bubbles: true })); return 'title-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>这是第一段:OpenCLI 格式化发文测试。</p><p><strong>[AutoTest-Bold] 这是加粗的第二段,用于验证富文本格式。</strong></p><p>这是第三段,包含普通文本内容,时间戳: ' + Date.now() + '。</p><p>这是第四段,测试多段落填充能力。</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'body-filled-with-formatting'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; const hasBold = editor.querySelector('strong') || editor.querySelector('b'); const paragraphs = editor.querySelectorAll('p'); return JSON.stringify({ paragraphCount: paragraphs.length, hasBold: !!hasBold, preview: editor.textContent?.slice(0, 80) }); })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const title = (document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'))?.value || ''; const body = (document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''; return JSON.stringify({ title: title.slice(0, 60), bodyHasBold: body.includes('AutoTest-Bold'), bodyLength: body.length }); })()\""
],
"judge": {
"type": "contains",
"value": "AutoTest"
},
"note": "8-step formatted article: navigate editor → fill title → fill body with <strong> bold + 4 paragraphs → verify formatting + content"
},
{
"name": "cross-zhihu-to-twitter",
"platform": "cross",
"type": "fill-only",
"description": "Read zhihu hot topic title, navigate to twitter compose, fill tweet with zhihu content, verify",
"steps": [
"opencli operate open https://www.zhihu.com/hot",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const hotItem = document.querySelector('.HotItem-content a') || document.querySelector('.HotList-item a') || document.querySelector('[data-testid=\\\"hot-item\\\"] a') || document.querySelector('.HotItem a'); if (hotItem) return hotItem.textContent?.trim()?.slice(0, 60) || 'no-text'; const titles = document.querySelectorAll('h2'); for (const t of titles) { if (t.textContent?.trim().length > 5) return t.textContent.trim().slice(0, 60); } return 'no-hot-topic'; })()\"",
"opencli operate state save zhihu_hot_title",
"opencli operate open https://x.com/compose/tweet",
"opencli operate wait time 3",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]') ? 'composer-ready' : 'not-found'\"",
"opencli operate eval \"(() => { const box = document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]'); if (!box) return 'no-box'; box.focus(); const dt = new DataTransfer(); dt.setData('text/plain', '[AutoTest] Zhihu热榜话题搬运: 知乎上正在热议的话题 - ' + Date.now()); box.dispatchEvent(new ClipboardEvent('paste', { clipboardData: dt, bubbles: true, cancelable: true })); return 'tweet-filled-with-zhihu'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"document.querySelector('[data-testid=\\\"tweetTextarea_0\\\"]')?.textContent || ''\""
],
"judge": {
"type": "contains",
"value": "Zhihu热榜话题搬运"
},
"note": "10-step cross-platform: read zhihu hot → save state → navigate twitter compose → fill tweet with zhihu content → verify"
},
{
"name": "cross-twitter-to-zhihu",
"platform": "cross",
"type": "fill-only",
"description": "Read twitter trending/explore topic, navigate to zhihu zhuanlan editor, fill title and body, verify",
"steps": [
"opencli operate open https://x.com/explore/tabs/trending",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const trends = document.querySelectorAll('[data-testid=\\\"trend\\\"]'); if (trends.length > 0) { const first = trends[0]; return first.textContent?.trim()?.slice(0, 80) || 'no-text'; } const spans = document.querySelectorAll('span'); for (const s of spans) { if (s.textContent?.startsWith('#') || s.textContent?.includes('Trending')) { return s.textContent.trim().slice(0, 80); } } return 'no-trending-topic'; })()\"",
"opencli operate state save twitter_trending",
"opencli operate open https://zhuanlan.zhihu.com/write",
"opencli operate wait time 3",
"opencli operate eval \"(() => { const titleInput = document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'); if (!titleInput) return 'no-title-input'; titleInput.focus(); titleInput.value = '[AutoTest] Twitter热点搬运: 来自推特的热门话题 ' + Date.now(); titleInput.dispatchEvent(new Event('input', { bubbles: true })); return 'title-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const editor = document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'); if (!editor) return 'no-editor'; editor.focus(); editor.innerHTML = '<p>[AutoTest] 这篇文章搬运自 Twitter 热门话题。</p><p>Twitter 上正在讨论的热门话题为大家带来了新的视角和思考。</p><p>时间戳: ' + Date.now() + '</p>'; editor.dispatchEvent(new Event('input', { bubbles: true })); return 'body-filled'; })()\"",
"opencli operate wait time 1",
"opencli operate eval \"(() => { const title = (document.querySelector('.WriteIndex-titleInput textarea') || document.querySelector('textarea[placeholder*=\\\"标题\\\"]'))?.value || ''; const body = (document.querySelector('.ql-editor') || document.querySelector('[contenteditable=\\\"true\\\"]'))?.textContent || ''; return JSON.stringify({ title: title.slice(0, 60), body: body.slice(0, 60) }); })()\""
],
"judge": {
"type": "contains",
"value": "Twitter热点搬运"
},
"note": "10-step cross-platform reverse: read twitter trending → save state → navigate zhihu editor → fill title + body → verify"
}
]
+119 -26
View File
@@ -4,7 +4,10 @@
"site": "test-httpbin",
"command": "get",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-httpbin',\n name: 'get',\n description: 'httpbin echo test',\n domain: 'httpbin.org',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [],\n columns: ['origin', 'url'],\n func: async () => {\n const res = await fetch('https://httpbin.org/get');\n const d = await res.json();\n return [{ origin: d.origin, url: d.url }];\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 1 },
"judge": {
"type": "arrayMinLength",
"minLength": 1
},
"note": "Simplest possible: httpbin echo, single row"
},
{
@@ -12,57 +15,80 @@
"site": "test-jsonplaceholder",
"command": "posts",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'posts',\n description: 'JSONPlaceholder posts',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of posts' },\n ],\n columns: ['id', 'title'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/posts');\n const posts = await res.json();\n return posts.slice(0, limit).map((p: any) => ({ id: p.id, title: p.title }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 }
"judge": {
"type": "arrayMinLength",
"minLength": 3
}
},
{
"name": "jsonplaceholder-users",
"site": "test-jsonplaceholder",
"command": "users",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'users',\n description: 'JSONPlaceholder users',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of users' },\n ],\n columns: ['id', 'name', 'email'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/users');\n const users = await res.json();\n return users.slice(0, limit).map((u: any) => ({ id: u.id, name: u.name, email: u.email }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 }
"judge": {
"type": "arrayMinLength",
"minLength": 3
}
},
{
"name": "hn-top",
"site": "test-hn",
"command": "top",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'top',\n description: 'HackerNews top stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/topstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 }
"judge": {
"type": "arrayMinLength",
"minLength": 3
}
},
{
"name": "hn-ask",
"site": "test-hn",
"command": "ask",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'ask',\n description: 'HackerNews Ask HN stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/askstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 }
"judge": {
"type": "arrayMinLength",
"minLength": 3
}
},
{
"name": "wiki-summary",
"site": "test-wiki",
"command": "summary",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-wiki',\n name: 'summary',\n description: 'Wikipedia article summary',\n domain: 'en.wikipedia.org',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'title', type: 'string', default: 'JavaScript', positional: true, help: 'Article title' },\n ],\n columns: ['title', 'extract'],\n func: async (_page, kwargs) => {\n const title = encodeURIComponent(kwargs.title);\n const res = await fetch(`https://en.wikipedia.org/api/rest_v1/page/summary/${title}`);\n const d = await res.json();\n return [{ title: d.title, extract: d.extract?.slice(0, 200) }];\n },\n});\n",
"judge": { "type": "contains", "value": "programming language" }
"judge": {
"type": "contains",
"value": "programming language"
}
},
{
"name": "lobsters-hot",
"site": "test-lobsters",
"command": "hot",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-lobsters',\n name: 'hot',\n description: 'Lobsters hottest stories',\n domain: 'lobste.rs',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['title', 'score', 'url'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://lobste.rs/hottest.json');\n const stories = await res.json();\n return stories.slice(0, limit).map((s: any) => ({\n title: s.title, score: s.score, url: s.short_id_url,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 }
"judge": {
"type": "arrayMinLength",
"minLength": 3
}
},
{
"name": "devto-top",
"site": "test-devto",
"command": "top",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-devto',\n name: 'top',\n description: 'DEV.to top articles',\n domain: 'dev.to',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of articles' },\n ],\n columns: ['title', 'user', 'reactions'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://dev.to/api/articles?per_page=' + limit);\n const articles = await res.json();\n return articles.map((a: any) => ({\n title: a.title, user: a.user?.username, reactions: a.positive_reactions_count,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 }
"judge": {
"type": "arrayMinLength",
"minLength": 3
}
},
{
"name": "zhihu-hot-with-top-answer",
"site": "test-zhihu",
"command": "hot-detail",
"adapterFile": "save-adapters/zhihu-hot-detail.ts",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "6-step chain: navigate → fetch hot list API → parse big-int IDs → loop N items → fetch answer API per question → strip HTML → merge"
},
{
@@ -70,7 +96,10 @@
"site": "test-zhihu",
"command": "search-detail",
"adapterFile": "save-adapters/zhihu-search-detail.ts",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "7-step chain: navigate → search API → filter by type → extract question IDs → fetch question detail per result → merge stats → format"
},
{
@@ -78,7 +107,10 @@
"site": "test-xhs",
"command": "search-full",
"adapterFile": "save-adapters/xhs-search-full.ts",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "6-step chain: navigate → MutationObserver wait → scroll 3x → DOM extract with URL dedup → slice + format"
},
{
@@ -86,7 +118,10 @@
"site": "test-xhs",
"command": "note-comments",
"adapterFile": "save-adapters/xhs-note-comments.ts",
"judge": { "type": "arrayMinLength", "minLength": 1 },
"judge": {
"type": "arrayMinLength",
"minLength": 1
},
"note": "7-step chain: navigate → wait → extract note meta → scroll container 3x → extract comments DOM → merge note+comments → unified output"
},
{
@@ -94,7 +129,10 @@
"site": "test-zhihu",
"command": "question-full",
"adapterFile": "save-adapters/zhihu-question-full.ts",
"judge": { "type": "arrayMinLength", "minLength": 2 },
"judge": {
"type": "arrayMinLength",
"minLength": 2
},
"note": "8-step chain: navigate → wait → fetch question detail → fetch answers → strip HTML → fetch related questions → merge 3 layers → format"
},
{
@@ -102,7 +140,10 @@
"site": "test-xhs",
"command": "explore-deep",
"adapterFile": "save-adapters/xhs-explore-deep.ts",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "8-step chain: navigate → MutationObserver wait → adaptive scroll → DOM extract with dedup → parse likes → sort desc → slice → format"
},
{
@@ -110,7 +151,10 @@
"site": "test-hn",
"command": "new",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'new',\n description: 'HackerNews newest stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/newstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score ?? 0,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: HackerNews new stories using same Firebase API as hn-top/hn-ask"
},
{
@@ -118,7 +162,10 @@
"site": "test-jsonplaceholder",
"command": "todos",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'todos',\n description: 'JSONPlaceholder todos',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of todos' },\n ],\n columns: ['id', 'title', 'completed'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/todos');\n const todos = await res.json();\n return todos.slice(0, limit).map((t: any) => ({ id: t.id, title: t.title, completed: t.completed }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: JSONPlaceholder todos — same base domain as posts/users, different endpoint"
},
{
@@ -126,7 +173,10 @@
"site": "test-hn",
"command": "show",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'show',\n description: 'HackerNews Show HN stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/showstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score ?? 0,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: HackerNews show stories using same Firebase API as hn-top/hn-ask/hn-new"
},
{
@@ -134,7 +184,10 @@
"site": "test-jsonplaceholder",
"command": "comments",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'comments',\n description: 'JSONPlaceholder comments',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of comments' },\n ],\n columns: ['id', 'name', 'email'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/comments');\n const comments = await res.json();\n return comments.slice(0, limit).map((c: any) => ({ id: c.id, name: c.name, email: c.email }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: JSONPlaceholder comments — same base domain as posts/users/todos, different endpoint"
},
{
@@ -142,7 +195,10 @@
"site": "test-jsonplaceholder",
"command": "albums",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'albums',\n description: 'JSONPlaceholder albums',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of albums' },\n ],\n columns: ['id', 'userId', 'title'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/albums');\n const albums = await res.json();\n return albums.slice(0, limit).map((a: any) => ({ id: a.id, userId: a.userId, title: a.title }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: JSONPlaceholder albums — same base domain as posts/users/todos/comments, different endpoint"
},
{
@@ -150,7 +206,10 @@
"site": "test-jsonplaceholder",
"command": "photos",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-jsonplaceholder',\n name: 'photos',\n description: 'JSONPlaceholder photos',\n domain: 'jsonplaceholder.typicode.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of photos' },\n ],\n columns: ['id', 'albumId', 'title'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://jsonplaceholder.typicode.com/photos');\n const photos = await res.json();\n return photos.slice(0, limit).map((p: any) => ({ id: p.id, albumId: p.albumId, title: p.title }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: JSONPlaceholder photos — same base domain as posts/users/todos/comments/albums, different endpoint"
},
{
@@ -158,7 +217,10 @@
"site": "test-hn",
"command": "best",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'best',\n description: 'HackerNews best stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of stories' },\n ],\n columns: ['rank', 'title', 'score'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/beststories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, score: item.score ?? 0,\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: HackerNews best stories using same Firebase API as hn-top/hn-ask/hn-new/hn-show"
},
{
@@ -166,7 +228,10 @@
"site": "test-hn",
"command": "jobs",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-hn',\n name: 'jobs',\n description: 'HackerNews job stories',\n domain: 'news.ycombinator.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of jobs' },\n ],\n columns: ['rank', 'title', 'url'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch('https://hacker-news.firebaseio.com/v0/jobstories.json');\n const ids = await res.json();\n const items = await Promise.all(ids.slice(0, limit).map(async (id: number) => {\n const r = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);\n return r.json();\n }));\n return items.map((item: any, i: number) => ({\n rank: i + 1, title: item.title, url: item.url ?? '',\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: HackerNews job listings using same Firebase API as other HN adapters"
},
{
@@ -174,7 +239,10 @@
"site": "test-restcountries",
"command": "list",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-restcountries',\n name: 'list',\n description: 'REST Countries list',\n domain: 'restcountries.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of countries' },\n ],\n columns: ['name', 'capital', 'region'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch('https://restcountries.com/v3.1/all?fields=name,capital,region');\n const countries = await res.json();\n return countries.slice(0, limit).map((c: any) => ({\n name: c.name?.common ?? '',\n capital: c.capital?.[0] ?? '',\n region: c.region ?? '',\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: REST Countries API — stable, no-auth, returns 250 countries with name/capital/region"
},
{
@@ -182,7 +250,32 @@
"site": "test-nager",
"command": "holidays",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-nager',\n name: 'holidays',\n description: 'US public holidays for current year',\n domain: 'date.nager.at',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of holidays' },\n ],\n columns: ['date', 'name', 'type'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const year = new Date().getFullYear();\n const res = await fetch(`https://date.nager.at/api/v3/PublicHolidays/${year}/US`);\n const holidays = await res.json();\n return holidays.slice(0, limit).map((h: any) => ({\n date: h.date,\n name: h.name,\n type: (h.types || []).join(','),\n }));\n },\n});\n",
"judge": { "type": "arrayMinLength", "minLength": 3 },
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: Nager public holidays API — stable, no-auth, returns US federal holidays by year"
},
{
"name": "catfact-list",
"site": "test-catfact",
"command": "facts",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-catfact',\n name: 'facts',\n description: 'Random cat facts',\n domain: 'catfact.ninja',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of facts' },\n ],\n columns: ['fact', 'length'],\n func: async (_page, kwargs) => {\n const limit = kwargs.limit ?? 5;\n const res = await fetch(`https://catfact.ninja/facts?limit=${limit}`);\n const d = await res.json();\n return d.data.map((item: any) => ({\n fact: item.fact.slice(0, 100),\n length: item.length,\n }));\n },\n});\n",
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: catfact.ninja facts API — stable, no-auth, returns random cat facts"
},
{
"name": "opentdb-trivia",
"site": "test-opentdb",
"command": "easy",
"adapter": "import { cli, Strategy } from '@jackwener/opencli/registry';\n\ncli({\n site: 'test-opentdb',\n name: 'easy',\n description: 'Easy trivia questions from Open Trivia DB',\n domain: 'opentdb.com',\n strategy: Strategy.PUBLIC,\n browser: false,\n args: [\n { name: 'limit', type: 'int', default: 5, help: 'Number of questions' },\n ],\n columns: ['category', 'question', 'answer'],\n func: async (_page, kwargs) => {\n const limit = Math.min(kwargs.limit ?? 5, 10);\n const res = await fetch(`https://opentdb.com/api.php?amount=${limit}&difficulty=easy&type=multiple`);\n const d = await res.json();\n return d.results.map((q: any) => ({\n category: q.category,\n question: q.question.replace(/&quot;/g, '\"').replace(/&#039;/g, \"'\").slice(0, 80),\n answer: q.correct_answer,\n }));\n },\n});\n",
"judge": {
"type": "arrayMinLength",
"minLength": 3
},
"note": "PUBLIC strategy: Open Trivia DB API — stable, no-auth, returns trivia questions with correct answers"
}
]
]
+42
View File
@@ -0,0 +1,42 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './assets.js';
import { __test__ as sharedTest } from './shared.js';
describe('1688 assets normalization', () => {
it('normalizes gallery and scanned assets into grouped media lists', () => {
const result = __test__.normalizeAssets({
href: 'https://detail.1688.com/offer/887904326744.html',
title: '测试商品 - 阿里巴巴',
offerTitle: '测试商品',
offerId: 887904326744,
gallery: {
mainImage: ['//img.example.com/main-1.jpg'],
offerImgList: ['https://img.example.com/main-2.jpg'],
wlImageInfos: [{ fullPathImageURI: 'https://img.example.com/main-3.jpg' }],
},
scannedAssets: [
{ type: 'image', group: 'sku', url: 'https://img.example.com/sku-1.png', source: 'dom:.sku' },
{ type: 'image', group: 'detail', url: 'https://img.example.com/detail-1.jpg', source: 'dom:.detail' },
{ type: 'video', group: 'video', url: 'https://video.example.com/demo.mp4', source: 'script' },
{ type: 'image', group: 'detail', url: 'blob:https://detail.1688.com/1', source: 'ignore' },
],
});
expect(result.offer_id).toBe('887904326744');
expect(result.main_images).toEqual([
'https://img.example.com/main-1.jpg',
'https://img.example.com/main-2.jpg',
'https://img.example.com/main-3.jpg',
]);
expect(result.sku_images).toEqual(['https://img.example.com/sku-1.png']);
expect(result.detail_images).toEqual(['https://img.example.com/detail-1.jpg']);
expect(result.videos).toEqual(['https://video.example.com/demo.mp4']);
expect(result.main_count).toBe(3);
expect(result.video_count).toBe(1);
});
it('normalizes media urls from style syntax and protocol-relative URLs', () => {
expect(sharedTest.normalizeMediaUrl('url("//img.example.com/1.jpg")')).toBe('https://img.example.com/1.jpg');
expect(sharedTest.normalizeMediaUrl('blob:https://detail.1688.com/1')).toBe('');
});
});
+257
View File
@@ -0,0 +1,257 @@
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
assertAuthenticatedState,
buildDetailUrl,
buildProvenance,
cleanText,
extractOfferId,
gotoAndReadState,
type MediaSource,
uniqueMediaSources,
} from './shared.js';
interface AssetBrowserPayload {
href?: string;
title?: string;
offerTitle?: string;
offerId?: string | number;
gallery?: {
mainImage?: string[];
offerImgList?: string[];
wlImageInfos?: Array<{ fullPathImageURI?: string }>;
[key: string]: unknown;
};
scannedAssets?: MediaSource[];
}
export interface Normalized1688Assets {
offer_id: string | null;
title: string | null;
item_url: string;
main_images: string[];
sku_images: string[];
detail_images: string[];
videos: string[];
other_images: string[];
raw_assets: MediaSource[];
source: string[];
main_count: number;
sku_count: number;
detail_count: number;
video_count: number;
source_url: string;
fetched_at: string;
strategy: string;
}
function scriptToReadAssets(): string {
return `
(() => {
const root = window.context ?? {};
const model = root.result?.global?.globalData?.model ?? null;
const gallery = root.result?.data?.gallery?.fields ?? null;
const defaultSrcProps = ['data-lazyload-src', 'data-src', 'data-ks-lazyload', 'currentSrc', 'src'];
const groups = [
{ key: 'main', type: 'image', selectors: ['#dt-tab img', '.detail-gallery-turn img.detail-gallery-img', '.img-list-wrapper img.od-gallery-img', '.od-scroller-item span'] },
{ key: 'video', type: 'video', selectors: ['.lib-video video', 'video[src]', 'video source[src]'] },
{ key: 'sku', type: 'image', selectors: ['.pc-sku-wrapper .prop-item-inner-wrapper', '.sku-item-wrapper', '.specification-cell', '.sku-filter-button', '.expand-view-item', '.feature-item img'], srcProps: ['backgroundImage'] },
{ key: 'detail', type: 'image', selectors: ['.de-description-detail img', '#detailContentContainer img', '.html-description img', '.html-description source', '.desc-lazyload-container img'] },
];
const assets = [];
const seen = new Set();
const normalizeUrl = (value) => {
if (typeof value !== 'string') return '';
let next = value
.replace(/^url\\((.*)\\)$/i, '$1')
.replace(/^['"]|['"]$/g, '')
.replace(/\\\\u002F/g, '/')
.replace(/&amp;/g, '&')
.trim();
if (!next || next.startsWith('blob:') || next.startsWith('data:')) return '';
if (next.startsWith('//')) next = 'https:' + next;
try {
return new URL(next, location.href).toString();
} catch {
return '';
}
};
const push = (type, group, url, source) => {
const normalized = normalizeUrl(url);
if (!normalized) return;
const key = type + ':' + normalized;
if (seen.has(key)) return;
seen.add(key);
assets.push({ type, group, url: normalized, source });
};
const queryAllDeep = (selector) => {
const results = [];
const visitedRoots = new Set();
const walkRoots = (root, fn) => {
if (!root || visitedRoots.has(root)) return;
visitedRoots.add(root);
fn(root);
const childElements = root.querySelectorAll ? Array.from(root.querySelectorAll('*')) : [];
for (const child of childElements) {
if (child && child.shadowRoot) {
walkRoots(child.shadowRoot, fn);
}
}
};
walkRoots(document, (root) => {
if (root.querySelectorAll) {
results.push(...Array.from(root.querySelectorAll(selector)));
}
});
return results;
};
const valuesFromElement = (element, srcProps) => {
const values = [];
const props = srcProps && srcProps.length ? srcProps : defaultSrcProps;
for (const prop of props) {
try {
if (prop === 'backgroundImage') {
const bg = getComputedStyle(element).backgroundImage || '';
const matches = bg.match(/url\\(([^)]+)\\)/g) || [];
for (const match of matches) {
const clean = match.replace(/^url\\(/, '').replace(/\\)$/, '');
values.push(clean);
}
continue;
}
const direct = element[prop];
if (typeof direct === 'string' && direct) values.push(direct);
const attr = element.getAttribute ? element.getAttribute(prop) : '';
if (attr) values.push(attr);
} catch {}
}
if (element.tagName === 'SOURCE' && element.parentElement?.tagName === 'VIDEO') {
values.push(element.src || element.getAttribute('src') || '');
}
if (element.tagName === 'VIDEO') {
values.push(element.currentSrc || '');
values.push(element.src || '');
}
return values;
};
for (const group of groups) {
for (const selector of group.selectors) {
for (const element of queryAllDeep(selector)) {
for (const value of valuesFromElement(element, group.srcProps)) {
push(group.type, group.key, value, 'dom:' + selector);
}
}
}
}
const scriptTexts = Array.from(document.scripts).map((script) => script.textContent || '');
const videoRegex = /https?:\\/\\/[^"'\\s]+\\.(?:mp4|m3u8)(?:\\?[^"'\\s]*)?/gi;
for (const scriptText of scriptTexts) {
const matches = scriptText.match(videoRegex) || [];
for (const match of matches) {
push('video', 'video', match, 'script');
}
}
const toJson = (value) => JSON.parse(JSON.stringify(value ?? null));
return {
href: window.location.href,
title: document.title || '',
offerTitle: model?.offerTitleModel?.subject ?? '',
offerId: model?.tradeModel?.offerId ?? '',
gallery: toJson(gallery),
scannedAssets: assets,
};
})()
`;
}
function normalizeAssets(payload: AssetBrowserPayload): Normalized1688Assets {
const offerId = cleanText(String(payload.offerId ?? '')) || extractOfferId(cleanText(payload.href)) || null;
const itemUrl = offerId ? buildDetailUrl(offerId) : cleanText(payload.href);
const seededAssets: MediaSource[] = [
...((payload.gallery?.mainImage ?? []).map((url) => ({ type: 'image' as const, group: 'main' as const, url, source: 'page_state:mainImage' }))),
...((payload.gallery?.offerImgList ?? []).map((url) => ({ type: 'image' as const, group: 'main' as const, url, source: 'page_state:offerImgList' }))),
...((payload.gallery?.wlImageInfos ?? []).map((item) => ({
type: 'image' as const,
group: 'main' as const,
url: item?.fullPathImageURI ?? '',
source: 'page_state:wlImageInfos',
}))),
];
const assets = uniqueMediaSources([...seededAssets, ...(payload.scannedAssets ?? [])]);
const mainImages = assets.filter((item) => item.type === 'image' && item.group === 'main').map((item) => item.url);
const skuImages = assets.filter((item) => item.type === 'image' && item.group === 'sku').map((item) => item.url);
const detailImages = assets.filter((item) => item.type === 'image' && item.group === 'detail').map((item) => item.url);
const videos = assets.filter((item) => item.type === 'video').map((item) => item.url);
const otherImages = assets
.filter((item) => item.type === 'image' && !['main', 'sku', 'detail'].includes(item.group))
.map((item) => item.url);
return {
offer_id: offerId,
title: cleanText(payload.offerTitle) || cleanText(payload.title) || null,
item_url: itemUrl,
main_images: mainImages,
sku_images: skuImages,
detail_images: detailImages,
videos,
other_images: otherImages,
raw_assets: assets,
source: [...new Set(assets.map((item) => cleanText(item.source)).filter(Boolean))],
main_count: mainImages.length,
sku_count: skuImages.length,
detail_count: detailImages.length,
video_count: videos.length,
...buildProvenance(cleanText(payload.href) || itemUrl),
};
}
async function readAssetsPayload(page: IPage, itemUrl: string): Promise<AssetBrowserPayload> {
const state = await gotoAndReadState(page, itemUrl, 2500, 'assets');
assertAuthenticatedState(state, 'assets');
await page.autoScroll({ times: 3, delayMs: 400 });
await page.wait(1);
return await page.evaluate(scriptToReadAssets()) as AssetBrowserPayload;
}
export async function extractAssetsForInput(page: IPage, input: string): Promise<Normalized1688Assets> {
const itemUrl = buildDetailUrl(String(input ?? ''));
const payload = await readAssetsPayload(page, itemUrl);
return normalizeAssets(payload);
}
cli({
site: '1688',
name: 'assets',
description: '列出 1688 商品页可提取的图片/视频素材',
domain: 'www.1688.com',
strategy: Strategy.COOKIE,
args: [
{
name: 'input',
required: true,
positional: true,
help: '1688 商品 URL 或 offer ID(如 887904326744',
},
],
columns: ['offer_id', 'title', 'main_count', 'sku_count', 'detail_count', 'video_count'],
func: async (page, kwargs) => {
return [await extractAssetsForInput(page, String(kwargs.input ?? ''))];
},
});
export const __test__ = {
normalizeAssets,
};
+33
View File
@@ -0,0 +1,33 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './download.js';
describe('1688 download helpers', () => {
it('builds stable filenames for grouped assets', () => {
const items = __test__.toDownloadItems('887904326744', {
offer_id: '887904326744',
title: '测试商品',
item_url: 'https://detail.1688.com/offer/887904326744.html',
main_images: ['https://img.example.com/a.jpg'],
sku_images: ['https://img.example.com/b.png'],
detail_images: ['https://img.example.com/c.webp'],
videos: ['https://video.example.com/d.mp4'],
other_images: [],
raw_assets: [],
source: [],
main_count: 1,
sku_count: 1,
detail_count: 1,
video_count: 1,
source_url: 'https://detail.1688.com/offer/887904326744.html',
fetched_at: new Date().toISOString(),
strategy: 'cookie',
});
expect(items.map((item) => item.filename)).toEqual([
'887904326744_main_01.jpg',
'887904326744_sku_01.png',
'887904326744_detail_01.webp',
'887904326744_video_01.mp4',
]);
});
});
+83
View File
@@ -0,0 +1,83 @@
import * as path from 'node:path';
import { formatCookieHeader } from '@jackwener/opencli/download';
import { downloadMedia, type MediaItem } from '@jackwener/opencli/download/media-download';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { cleanText } from './shared.js';
import { extractAssetsForInput } from './assets.js';
function extFromUrl(url: string, fallback: string): string {
try {
const ext = path.extname(new URL(url).pathname).toLowerCase();
if (ext && ext.length <= 8) return ext;
} catch {
// ignore
}
return fallback;
}
function toDownloadItems(offerId: string, assets: Awaited<ReturnType<typeof extractAssetsForInput>>): MediaItem[] {
const items: MediaItem[] = [];
const pushImages = (urls: string[], prefix: string) => {
urls.forEach((url, index) => {
items.push({
type: 'image',
url,
filename: `${offerId}_${prefix}_${String(index + 1).padStart(2, '0')}${extFromUrl(url, '.jpg')}`,
});
});
};
pushImages(assets.main_images, 'main');
pushImages(assets.sku_images, 'sku');
pushImages(assets.detail_images, 'detail');
pushImages(assets.other_images, 'other');
assets.videos.forEach((url, index) => {
items.push({
type: 'video',
url,
filename: `${offerId}_video_${String(index + 1).padStart(2, '0')}${extFromUrl(url, '.mp4')}`,
});
});
return items;
}
cli({
site: '1688',
name: 'download',
description: '批量下载 1688 商品页可提取的图片和视频素材',
domain: 'www.1688.com',
strategy: Strategy.COOKIE,
args: [
{
name: 'input',
required: true,
positional: true,
help: '1688 商品 URL 或 offer ID(如 887904326744',
},
{ name: 'output', default: './1688-downloads', help: '输出目录' },
],
columns: ['index', 'type', 'status', 'size'],
func: async (page, kwargs) => {
const assets = await extractAssetsForInput(page, String(kwargs.input ?? ''));
const offerId = cleanText(assets.offer_id) || '1688';
const items = toDownloadItems(offerId, assets);
const browserCookies = await page.getCookies({ domain: '1688.com' });
return downloadMedia(items, {
output: String(kwargs.output || './1688-downloads'),
subdir: offerId,
cookies: formatCookieHeader(browserCookies),
browserCookies,
filenamePrefix: offerId,
timeout: 60000,
});
},
});
export const __test__ = {
extFromUrl,
toDownloadItems,
};
+69
View File
@@ -0,0 +1,69 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './item.js';
describe('1688 item normalization', () => {
it('normalizes public item payload into contract fields', () => {
const result = __test__.normalizeItemPayload({
href: 'https://detail.1688.com/offer/887904326744.html',
title: '法式春季长袖开衫连衣裙女新款大码女装碎花吊带裙套装142077 - 阿里巴巴',
bodyText: `
青岛沁澜衣品服装有限公司
入驻13年
主营:大码女装
店铺回头率
87%
山东青岛
3套起批
已售1600+套
支持定制logo
`,
offerTitle: '法式春季长袖开衫连衣裙女新款大码女装碎花吊带裙套装142077',
offerId: 887904326744,
seller: {
companyName: '青岛沁澜衣品服装有限公司',
memberId: 'b2b-1641351767',
winportUrl: 'https://yinuoweierfushi.1688.com/page/index.html?spm=a1',
},
trade: {
beginAmount: 3,
priceDisplay: '96.00-98.00',
unit: '套',
saleCount: 1655,
offerIDatacenterSellInfo: {
: '莫代尔',
: '莫代尔纤维',
sellPointModel: '{"ignore":true}',
},
offerPriceModel: {
currentPrices: [
{ beginAmount: 3, price: '98.00' },
{ beginAmount: 50, price: '97.00' },
],
},
},
gallery: {
mainImage: ['https://example.com/1.jpg'],
offerImgList: ['https://example.com/2.jpg'],
wlImageInfos: [{ fullPathImageURI: 'https://example.com/3.jpg' }],
},
services: [
{ serviceName: '延期必赔', agreeDeliveryHours: 360 },
{ serviceName: '品质保障' },
],
});
expect(result.offer_id).toBe('887904326744');
expect(result.member_id).toBe('b2b-1641351767');
expect(result.shop_id).toBe('yinuoweierfushi');
expect(result.seller_url).toBe('https://yinuoweierfushi.1688.com');
expect(result.price_text).toBe('¥96.00-98.00');
expect(result.moq_text).toBe('3套起批');
expect(result.origin_place).toBe('山东青岛');
expect(result.delivery_days_text).toBe('360小时内发货');
expect(result.private_label_text).toBe('支持定制logo');
expect(result.visible_attributes).toEqual([
{ key: '面料名称', value: '莫代尔' },
{ key: '主面料成分', value: '莫代尔纤维' },
]);
});
});
+282
View File
@@ -0,0 +1,282 @@
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import { isRecord } from '@jackwener/opencli/utils';
import {
assertAuthenticatedState,
buildDetailUrl,
buildProvenance,
canonicalizeSellerUrl,
cleanMultilineText,
cleanText,
extractLocation,
extractMemberId,
extractOfferId,
extractShopId,
gotoAndReadState,
normalizePriceTiers,
parseMoqText,
parsePriceText,
toNumber,
uniqueNonEmpty,
} from './shared.js';
interface BuyerProtectionModel {
serviceName?: string;
shortBuyerDesc?: string;
packageBuyerDesc?: string;
textDesc?: string;
agreeDeliveryHours?: number;
}
interface ItemBrowserPayload {
href?: string;
title?: string;
bodyText?: string;
offerTitle?: string;
offerId?: string | number;
seller?: {
companyName?: string;
memberId?: string;
winportUrl?: string;
sellerWinportUrlMap?: Record<string, string>;
};
trade?: {
beginAmount?: string | number;
priceDisplay?: string;
unit?: string;
saleCount?: string | number;
offerIDatacenterSellInfo?: Record<string, unknown>;
offerPriceModel?: {
currentPrices?: Array<{ beginAmount?: string | number; price?: string | number }>;
};
};
gallery?: {
mainImage?: string[];
offerImgList?: string[];
wlImageInfos?: Array<{ fullPathImageURI?: string }>;
};
shipping?: {
deliveryLimitText?: string;
logisticsText?: string;
protectionInfos?: BuyerProtectionModel[];
buyerProtectionModel?: BuyerProtectionModel[];
};
services?: BuyerProtectionModel[];
}
interface VisibleAttribute {
key: string;
value: string;
}
function normalizeItemPayload(payload: ItemBrowserPayload): Record<string, unknown> {
const href = cleanText(payload.href);
const bodyText = cleanMultilineText(payload.bodyText);
const sellerName = cleanText(payload.seller?.companyName);
const sellerUrlRaw = cleanText(
payload.seller?.winportUrl
?? payload.seller?.sellerWinportUrlMap?.defaultUrl
?? payload.seller?.sellerWinportUrlMap?.indexUrl,
);
const sellerUrl = canonicalizeSellerUrl(sellerUrlRaw);
const offerId = cleanText(String(payload.offerId ?? '')) || extractOfferId(href) || null;
const memberId = cleanText(payload.seller?.memberId) || extractMemberId(sellerUrlRaw || href) || null;
const shopId = extractShopId(sellerUrl ?? href);
const unit = cleanText(payload.trade?.unit);
const priceDisplay = cleanText(payload.trade?.priceDisplay);
const priceRange = parsePriceText(priceDisplay ? `¥${priceDisplay}` : bodyText);
const moqText = extractMoqText(bodyText, payload.trade?.beginAmount, unit);
const moq = parseMoqText(moqText);
const services = uniqueServices(payload);
const serviceBadges = uniqueNonEmpty(services.map((service) => cleanText(service.serviceName)));
const attributes = normalizeVisibleAttributes(payload.trade?.offerIDatacenterSellInfo);
const priceTiers = normalizePriceTiers(payload.trade?.offerPriceModel?.currentPrices ?? [], unit || null);
const images = uniqueNonEmpty([
...(payload.gallery?.mainImage ?? []),
...(payload.gallery?.offerImgList ?? []),
...((payload.gallery?.wlImageInfos ?? []).map((item) => item.fullPathImageURI ?? '')),
]);
const detailUrl = offerId ? buildDetailUrl(offerId) : href;
const provenance = buildProvenance(href || detailUrl);
return {
offer_id: offerId,
member_id: memberId,
shop_id: shopId,
title: cleanText(payload.offerTitle) || stripAlibabaSuffix(payload.title) || firstNonEmptyLine(bodyText) || null,
item_url: detailUrl,
main_images: images,
price_text: priceRange.price_text || null,
price_tiers: priceTiers,
currency: priceRange.currency,
moq_text: moq.moq_text || null,
moq_value: moq.moq_value,
seller_name: sellerName || null,
seller_url: sellerUrl,
shop_name: sellerName || null,
origin_place: extractLocation(bodyText),
delivery_days_text: extractDeliveryDaysText(bodyText, services, payload.shipping),
customization_text: extractKeywordLine(bodyText, ['来样定制', '来图定制', '支持定制', '可定制', '定制']),
private_label_text: extractKeywordLine(bodyText, ['贴牌', '贴标', '定制logo', '打logo', 'OEM', 'ODM']),
visible_attributes: attributes,
sales_text: extractSalesText(bodyText),
service_badges: serviceBadges,
stock_quantity: extractStockQuantity(bodyText),
...provenance,
};
}
function normalizeVisibleAttributes(raw: unknown): VisibleAttribute[] {
if (!isRecord(raw)) return [];
return Object.entries(raw)
.filter(([key, value]) => key !== 'sellPointModel' && cleanText(key) && cleanText(String(value)))
.map(([key, value]) => ({ key: cleanText(key), value: cleanText(String(value)) }));
}
function uniqueServices(payload: ItemBrowserPayload): BuyerProtectionModel[] {
const combined = [
...(Array.isArray(payload.services) ? payload.services : []),
...(Array.isArray(payload.shipping?.protectionInfos) ? payload.shipping.protectionInfos : []),
...(Array.isArray(payload.shipping?.buyerProtectionModel) ? payload.shipping.buyerProtectionModel : []),
];
const seen = new Set<string>();
const result: BuyerProtectionModel[] = [];
for (const service of combined) {
const key = cleanText(service.serviceName);
if (!key || seen.has(key)) continue;
seen.add(key);
result.push(service);
}
return result;
}
function stripAlibabaSuffix(title: string | undefined): string {
return cleanText(title).replace(/\s*-\s*阿里巴巴$/, '').trim();
}
function firstNonEmptyLine(text: string): string {
return text.split('\n').map((line) => cleanText(line)).find(Boolean) ?? '';
}
function extractMoqText(bodyText: string, beginAmount: string | number | undefined, unit: string): string {
const lineMatch = bodyText.match(/\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)\s*起批/);
if (lineMatch) return lineMatch[0];
const moqValue = toNumber(beginAmount);
if (moqValue !== null) {
return `${moqValue}${unit || ''}起批`;
}
return '';
}
function extractDeliveryDaysText(
bodyText: string,
services: BuyerProtectionModel[],
shipping: ItemBrowserPayload['shipping'],
): string | null {
const shippingText = cleanText(shipping?.deliveryLimitText) || cleanText(shipping?.logisticsText);
if (shippingText) return shippingText;
const textMatch = bodyText.match(/\d+\s*(?:小时|天)(?:内)?发货/);
if (textMatch) return textMatch[0];
const hourMatch = services.find((service) => typeof service.agreeDeliveryHours === 'number');
if (hourMatch && typeof hourMatch.agreeDeliveryHours === 'number') {
return `${hourMatch.agreeDeliveryHours}小时内发货`;
}
return null;
}
function extractKeywordLine(bodyText: string, keywords: string[]): string | null {
const lines = bodyText.split('\n').map((line) => cleanText(line)).filter(Boolean);
for (const line of lines) {
if (keywords.some((keyword) => line.includes(keyword))) {
return line;
}
}
return null;
}
function extractSalesText(bodyText: string): string | null {
const match = bodyText.match(/(?:全网销量|已售)\s*\d+(?:\.\d+)?\+?\s*[件套个单]?/);
return match ? cleanText(match[0]) : null;
}
function extractStockQuantity(bodyText: string): number | null {
const match = bodyText.match(/库存\s*(\d+)/);
return match ? Number.parseInt(match[1], 10) : null;
}
async function readItemPayload(page: IPage, itemUrl: string): Promise<ItemBrowserPayload> {
const state = await gotoAndReadState(page, itemUrl, 2500, 'item');
assertAuthenticatedState(state, 'item');
const payload = await page.evaluate(`
(() => {
const root = window.context ?? {};
const model = root.result?.global?.globalData?.model ?? null;
const toJson = (value) => JSON.parse(JSON.stringify(value ?? null));
return {
href: window.location.href,
title: document.title || '',
bodyText: document.body ? document.body.innerText || '' : '',
offerTitle: model?.offerTitleModel?.subject ?? '',
offerId: model?.tradeModel?.offerId ?? '',
seller: toJson(model?.sellerModel),
trade: toJson(model?.tradeModel),
gallery: toJson(root.result?.data?.gallery?.fields ?? null),
shipping: toJson(root.result?.data?.shippingServices?.fields ?? null),
services: toJson(root.result?.data?.shippingServices?.fields?.protectionInfos ?? []),
};
})()
`) as ItemBrowserPayload;
const resolvedOfferId = cleanText(String(payload.offerId ?? '')) || extractOfferId(cleanText(payload.href));
if (!resolvedOfferId) {
throw new CommandExecutionError(
'1688 item page did not expose product context',
'当前 tab 非商品详情上下文,请切到 detail.1688.com 商品页并重试',
);
}
return payload;
}
cli({
site: '1688',
name: 'item',
description: '1688 商品详情(公开商品字段、价格阶梯、卖家基础信息)',
domain: 'www.1688.com',
strategy: Strategy.COOKIE,
navigateBefore: false,
args: [
{
name: 'input',
required: true,
positional: true,
help: '1688 商品 URL 或 offer ID(如 887904326744',
},
],
columns: ['offer_id', 'title', 'price_text', 'moq_text', 'seller_name', 'origin_place'],
func: async (page, kwargs) => {
const itemUrl = buildDetailUrl(String(kwargs.input ?? ''));
const payload = await readItemPayload(page, itemUrl);
return [normalizeItemPayload(payload)];
},
});
export const __test__ = {
normalizeItemPayload,
normalizeVisibleAttributes,
stripAlibabaSuffix,
extractMoqText,
extractDeliveryDaysText,
extractKeywordLine,
extractSalesText,
extractStockQuantity,
};
+81
View File
@@ -0,0 +1,81 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './search.js';
describe('1688 search normalization', () => {
it('normalizes search candidates into structured result rows', () => {
const result = __test__.normalizeSearchCandidate({
item_url: 'https://detail.1688.com/offer/887904326744.html',
title: '宿舍置物架桌面加高架',
container_text: '宿舍置物架桌面加高架 ¥56.00 2套起批 山东青岛 已售300+套',
price_text: '¥ 56 .00',
sales_text: '300+套',
moq_text: '2套起批',
tag_items: ['退货包运费', '回头率52%'],
hover_items: ['验厂报告'],
seller_name: '青岛沁澜衣品服装有限公司',
seller_url: 'https://yinuoweierfushi.1688.com/page/index.html?spm=a123',
}, 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=置物架');
expect(result.rank).toBe(0);
expect(result.offer_id).toBe('887904326744');
expect(result.shop_id).toBe('yinuoweierfushi');
expect(result.item_url).toBe('https://detail.1688.com/offer/887904326744.html');
expect(result.seller_url).toBe('https://yinuoweierfushi.1688.com');
expect(result.price_text).toBe('¥56.00');
expect(result.price_min).toBe(56);
expect(result.price_max).toBe(56);
expect(result.moq_value).toBe(2);
expect(result.location).toBe('山东青岛');
expect(result.sales_text).toBe('300+套');
expect(result.badges).toEqual(expect.arrayContaining(['退货包运费', '验厂报告']));
expect(result.return_rate_text).toBe('回头率52%');
});
it('does not use hover_price_text as MOQ source', () => {
const result = __test__.normalizeSearchCandidate({
item_url: 'https://detail.1688.com/offer/887904326744.html',
title: 'test',
container_text: 'test ¥56.00',
price_text: '¥ 56 .00',
hover_price_text: '¥56.00 3件起批',
moq_text: null,
}, 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=test');
// hover_price_text should not be used for MOQ extraction
expect(result.moq_text).toBeNull();
expect(result.moq_value).toBeNull();
});
it('extracts offer id from mobile detail search links', () => {
const result = __test__.normalizeSearchCandidate({
item_url: 'http://detail.m.1688.com/page/index.html?offerId=910933345396&sortType=&pageId=',
title: '',
container_text: '桌面书桌办公室工位收纳展示新中式博古架多层茶具厨房摆放置物架 ¥24.3 已售20+件',
price_text: '¥ 14 .28',
sales_text: '1500+件',
moq_text: '≥2个',
seller_name: '泰商国际贸易(宁阳)有限公司',
seller_url: 'http://tsgjmy.1688.com/',
}, 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=桌面置物架');
expect(result.offer_id).toBe('910933345396');
expect(result.shop_id).toBe('tsgjmy');
expect(result.item_url).toBe('https://detail.1688.com/offer/910933345396.html');
expect(result.title).toContain('桌面书桌办公室工位收纳展示');
expect(result.price_text).toBe('¥14.28');
expect(result.sales_text).toBe('1500+件');
expect(result.moq_text).toBe('≥2个');
expect(result.moq_value).toBe(2);
});
it('prefers offer id and falls back to item url for dedupe key', () => {
expect(__test__.buildDedupeKey({
offer_id: '123456',
item_url: 'https://detail.1688.com/offer/123456.html',
})).toBe('offer:123456');
expect(__test__.buildDedupeKey({
offer_id: null,
item_url: 'https://detail.1688.com/offer/123456.html',
})).toBe('url:https://detail.1688.com/offer/123456.html');
expect(__test__.buildDedupeKey({ offer_id: null, item_url: null })).toBeNull();
});
});
+402
View File
@@ -0,0 +1,402 @@
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
FACTORY_BADGE_PATTERNS,
SERVICE_BADGE_PATTERNS,
assertAuthenticatedState,
buildProvenance,
buildSearchUrl,
canonicalizeItemUrl,
canonicalizeSellerUrl,
cleanText,
extractBadges,
extractLocation,
extractMemberId,
extractOfferId,
extractShopId,
gotoAndReadState,
parseMoqText,
parsePriceText,
SEARCH_LIMIT_DEFAULT,
SEARCH_LIMIT_MAX,
parseSearchLimit,
uniqueNonEmpty,
} from './shared.js';
interface SearchPayload {
href?: string;
title?: string;
bodyText?: string;
next_url?: string;
candidates?: Array<{
item_url?: string;
title?: string;
container_text?: string;
desc_rows?: string[];
price_text?: string | null;
sales_text?: string | null;
hover_price_text?: string | null;
moq_text?: string | null;
tag_items?: string[];
hover_items?: string[];
seller_name?: string | null;
seller_url?: string | null;
}>;
}
interface SearchRow {
rank: number;
offer_id: string | null;
member_id: string | null;
shop_id: string | null;
title: string | null;
item_url: string | null;
seller_name: string | null;
seller_url: string | null;
price_text: string | null;
price_min: number | null;
price_max: number | null;
currency: string | null;
moq_text: string | null;
moq_value: number | null;
location: string | null;
badges: string[];
sales_text: string | null;
return_rate_text: string | null;
source_url: string;
fetched_at: string;
strategy: string;
}
const SEARCH_ITEM_URL_PATTERNS = [
'detail.1688.com/offer/',
'detail.m.1688.com/page/index.html?offerId=',
];
const MAX_SEARCH_PAGES = 12;
function normalizeSearchCandidate(
candidate: NonNullable<SearchPayload['candidates']>[number],
sourceUrl: string,
): SearchRow {
const canonicalItemUrl = canonicalizeItemUrl(cleanText(candidate.item_url));
const containerText = cleanText(candidate.container_text);
const priceText = firstNonEmpty([
normalizeInlineText(candidate.price_text),
normalizeInlineText(extractPriceText(candidate.hover_price_text)),
]);
const priceRange = parsePriceText(priceText || containerText);
const moq = parseMoqText(firstNonEmpty([
normalizeInlineText(candidate.moq_text),
normalizeInlineText(extractMoqText(containerText)),
]));
const canonicalSellerUrl = canonicalizeSellerUrl(cleanText(candidate.seller_url));
const evidenceText = uniqueNonEmpty([
containerText,
...(candidate.desc_rows ?? []),
...(candidate.tag_items ?? []),
...(candidate.hover_items ?? []),
]).join('\n');
const badges = extractBadges(evidenceText, [...FACTORY_BADGE_PATTERNS, ...SERVICE_BADGE_PATTERNS]);
const salesText = firstNonEmpty([
extractSalesText(candidate.sales_text),
extractSalesText(containerText),
]);
const returnRateText = extractReturnRateText([...(candidate.tag_items ?? []), ...(candidate.hover_items ?? [])]);
const provenance = buildProvenance(sourceUrl);
return {
rank: 0,
offer_id: extractOfferId(canonicalItemUrl ?? '') ?? null,
member_id: extractMemberId(canonicalSellerUrl ?? '') ?? null,
shop_id: extractShopId(canonicalSellerUrl ?? '') ?? null,
title: cleanText(candidate.title) || firstWord(containerText) || null,
item_url: canonicalItemUrl,
seller_name: cleanText(candidate.seller_name) || null,
seller_url: canonicalSellerUrl,
price_text: priceRange.price_text || null,
price_min: priceRange.price_min,
price_max: priceRange.price_max,
currency: priceRange.currency,
moq_text: moq.moq_text || null,
moq_value: moq.moq_value,
location: extractLocation(containerText),
badges,
sales_text: salesText || null,
return_rate_text: returnRateText,
source_url: provenance.source_url,
fetched_at: provenance.fetched_at,
strategy: provenance.strategy,
};
}
function extractMoqText(text: string | null | undefined): string {
const normalized = normalizeInlineText(text);
return normalized.match(/\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)\s*起批/i)?.[0]
?? normalized.match(/≥\s*\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)?/i)?.[0]
?? normalized.match(/\d+(?:\.\d+)?\s*(?:~|-|至|到)\s*\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只)/i)?.[0]
?? '';
}
function extractPriceText(text: string | null | undefined): string {
const normalized = normalizeInlineText(text);
return normalized.match(/[¥$€]\s*\d+(?:\.\d+)?/)?.[0] ?? '';
}
function extractSalesText(text: string | null | undefined): string {
const normalized = normalizeInlineText(text);
if (!normalized) return '';
if (/^\d+(?:\.\d+)?\+?\s*(件|套|个|单)$/.test(normalized)) {
return normalized;
}
const match = normalized.match(/(?:已售|销量|售)\s*\d+(?:\.\d+)?\+?\s*(件|套|个|单)?/);
return match ? cleanText(match[0]) : '';
}
function firstWord(text: string): string {
return text.split(/\s+/).find(Boolean) ?? '';
}
function firstNonEmpty(values: Array<string | null | undefined>): string {
return values.map((value) => cleanText(value)).find(Boolean) ?? '';
}
function normalizeInlineText(text: string | null | undefined): string {
return cleanText(text)
.replace(/([¥$€])\s+(?=\d)/g, '$1')
.replace(/(\d)\s*\.\s*(\d)/g, '$1.$2')
.replace(/\s*([~-])\s*/g, '$1')
.trim();
}
function extractReturnRateText(values: string[]): string | null {
return uniqueNonEmpty(values.map((value) => normalizeInlineText(value)))
.find((value) => /^回头率\s*\d+(?:\.\d+)?%$/.test(value))
?? null;
}
function buildDedupeKey(row: Pick<SearchRow, 'offer_id' | 'item_url'>): string | null {
if (row.offer_id) return `offer:${row.offer_id}`;
if (row.item_url) return `url:${row.item_url}`;
return null;
}
async function readSearchPayload(page: IPage, url: string): Promise<SearchPayload> {
const state = await gotoAndReadState(page, url, 2500, 'search');
assertAuthenticatedState(state, 'search');
const payload = await page.evaluate(`
(() => {
const normalizeText = (value) => (value || '').replace(/\\s+/g, ' ').trim();
const normalizeUrl = (href) => {
if (!href) return '';
try {
return new URL(href, window.location.href).toString();
} catch {
return '';
}
};
const isItemHref = (href) => ${JSON.stringify(SEARCH_ITEM_URL_PATTERNS)}
.some((pattern) => (href || '').includes(pattern));
const uniqueTexts = (values) => [...new Set(values.map((value) => normalizeText(value)).filter(Boolean))];
const collectTexts = (root, selector) => uniqueTexts(
Array.from(root.querySelectorAll(selector)).map((node) => node.innerText || node.textContent || ''),
);
const firstText = (root, selectors) => {
for (const selector of selectors) {
const node = root.querySelector(selector);
const value = normalizeText(node ? node.innerText || node.textContent || '' : '');
if (value) return value;
}
return '';
};
const findMoqText = (values, priceText) => {
const moqPattern = /(≥\\s*\\d+(?:\\.\\d+)?\\s*(件|个|套|箱|包|双|台|把|只)?)|(\\d+(?:\\.\\d+)?\\s*(?:~|-|至|到)\\s*\\d+(?:\\.\\d+)?\\s*(件|个|套|箱|包|双|台|把|只))|(\\d+(?:\\.\\d+)?\\s*(件|个|套|箱|包|双|台|把|只)\\s*起批)/i;
return values.find((value) => moqPattern.test(value))
|| normalizeText(priceText).match(moqPattern)?.[0]
|| '';
};
const isSellerHref = (href) => {
if (!href) return false;
try {
const url = new URL(href, window.location.href);
const host = url.hostname || '';
if (!host.endsWith('.1688.com')) return false;
if (
host === 's.1688.com'
|| host === 'r.1688.com'
|| host === 'air.1688.com'
|| host === 'detail.1688.com'
|| host === 'detail.m.1688.com'
|| host === 'dj.1688.com'
) {
return false;
}
return true;
} catch {
return false;
}
};
const pickContainer = (anchor) => {
let node = anchor;
while (node && node !== document.body) {
const text = normalizeText(node.innerText || node.textContent || '');
if (text.length >= 40 && text.length <= 2000) {
return node;
}
node = node.parentElement;
}
return anchor;
};
const collectCandidates = () => {
const anchors = Array.from(document.querySelectorAll('a')).filter((anchor) => isItemHref(anchor.href || ''));
const seen = new Set();
const items = [];
for (const anchor of anchors) {
const href = anchor.href || '';
if (!href || seen.has(href)) continue;
seen.add(href);
const container = pickContainer(anchor);
const tagItems = collectTexts(container, '.offer-tag-row .offer-desc-item');
const hoverItems = collectTexts(container, '.offer-hover-wrapper .offer-desc-item');
const sellerAnchor = Array.from(container.querySelectorAll('a'))
.find((link) => isSellerHref(link.href || ''));
const hoverPriceText = firstText(container, [
'.offer-hover-wrapper .hover-price-item',
'.offer-hover-wrapper .price-item',
]);
items.push({
item_url: href,
title: firstText(container, ['.offer-title-row .title-text', '.offer-title-row'])
|| normalizeText(anchor.innerText || anchor.textContent || ''),
container_text: normalizeText(container.innerText || container.textContent || ''),
desc_rows: collectTexts(container, '.offer-desc-row'),
price_text: firstText(container, ['.offer-price-row .price-item']),
sales_text: firstText(container, ['.offer-price-row .col-desc_after', '.offer-desc-row .col-desc_after']),
hover_price_text: hoverPriceText,
moq_text: findMoqText(hoverItems, hoverPriceText),
tag_items: tagItems,
hover_items: hoverItems,
seller_name: sellerAnchor ? normalizeText(sellerAnchor.innerText || sellerAnchor.textContent || '') : null,
seller_url: sellerAnchor ? sellerAnchor.href : null,
});
}
return items;
};
const findNextUrl = () => {
const selectors = [
'a.fui-next:not(.disabled)',
'a.next-pagination-item:not(.disabled)',
'a[rel="next"]:not(.disabled)',
'a[data-role="next"]:not(.disabled)',
];
for (const selector of selectors) {
const node = document.querySelector(selector);
if (!node) continue;
const href = normalizeUrl(node.getAttribute('href') || node.href || '');
if (href) return href;
}
const textBased = Array.from(document.querySelectorAll('a'))
.find((node) => /下一页|next/i.test(normalizeText(node.textContent || '')));
if (!textBased) return '';
return normalizeUrl(textBased.getAttribute('href') || textBased.href || '');
};
return {
href: window.location.href,
title: document.title || '',
bodyText: document.body ? document.body.innerText || '' : '',
next_url: findNextUrl(),
candidates: collectCandidates(),
};
})()
`) as SearchPayload;
if (!payload || typeof payload !== 'object') {
throw new CommandExecutionError(
'1688 search page did not return a readable payload',
'Open the same query in Chrome and verify the page is fully loaded before retrying.',
);
}
return payload;
}
async function collectSearchRows(page: IPage, query: string, limit: number): Promise<SearchRow[]> {
const rowsByKey = new Map<string, SearchRow>();
const seenPages = new Set<string>();
let nextUrl = buildSearchUrl(query);
let pageCount = 0;
while (nextUrl && rowsByKey.size < limit && pageCount < MAX_SEARCH_PAGES) {
if (seenPages.has(nextUrl)) break;
seenPages.add(nextUrl);
pageCount += 1;
const payload = await readSearchPayload(page, nextUrl);
const sourceUrl = cleanText(payload.href) || nextUrl;
const candidates = Array.isArray(payload.candidates) ? payload.candidates : [];
for (const candidate of candidates) {
const row = normalizeSearchCandidate(candidate, sourceUrl);
const dedupeKey = buildDedupeKey(row);
if (!dedupeKey || rowsByKey.has(dedupeKey)) continue;
rowsByKey.set(dedupeKey, row);
if (rowsByKey.size >= limit) break;
}
const candidateNextUrl = cleanText(payload.next_url);
if (!candidateNextUrl || candidateNextUrl === sourceUrl) break;
nextUrl = candidateNextUrl;
}
if (rowsByKey.size === 0) {
throw new EmptyResultError(
'1688 search',
'No visible results were extracted. Retry with a different query or open the same search page in Chrome first.',
);
}
return [...rowsByKey.values()]
.slice(0, limit)
.map((row, index) => ({ ...row, rank: index + 1 }));
}
cli({
site: '1688',
name: 'search',
description: '1688 商品搜索(结果候选、卖家链接、价格/MOQ/销量文本)',
domain: 'www.1688.com',
strategy: Strategy.COOKIE,
navigateBefore: false,
args: [
{
name: 'query',
required: true,
positional: true,
help: '搜索关键词,如 "置物架"',
},
{
name: 'limit',
type: 'int',
default: SEARCH_LIMIT_DEFAULT,
help: `结果数量上限(默认 ${SEARCH_LIMIT_DEFAULT},最大 ${SEARCH_LIMIT_MAX}`,
},
],
columns: ['rank', 'title', 'price_text', 'moq_text', 'seller_name', 'location'],
func: async (page, kwargs) => {
const query = String(kwargs.query ?? '');
const limit = parseSearchLimit(kwargs.limit);
return collectSearchRows(page, query, limit);
},
});
export const __test__ = {
normalizeSearchCandidate,
extractMoqText,
extractSalesText,
firstWord,
buildDedupeKey,
};
+75
View File
@@ -0,0 +1,75 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './shared.js';
describe('1688 shared helpers', () => {
it('builds encoded search URLs and validates limit', () => {
expect(__test__.buildSearchUrl('置物架')).toBe(
'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=%E7%BD%AE%E7%89%A9%E6%9E%B6',
);
expect(() => __test__.buildSearchUrl(' ')).toThrowError(/cannot be empty/i);
expect(__test__.parseSearchLimit(3)).toBe(3);
expect(__test__.parseSearchLimit('1000')).toBe(__test__.SEARCH_LIMIT_MAX);
expect(() => __test__.parseSearchLimit('0')).toThrowError(/positive integer/i);
});
it('extracts IDs and canonicalizes urls', () => {
expect(__test__.extractOfferId('887904326744')).toBe('887904326744');
expect(__test__.extractOfferId('https://detail.1688.com/offer/887904326744.html')).toBe('887904326744');
expect(__test__.extractMemberId('https://winport.m.1688.com/page/index.html?memberId=b2b-1641351767')).toBe('b2b-1641351767');
expect(__test__.extractMemberId('b2b-22154705262941f196')).toBe('b2b-22154705262941f196');
expect(__test__.resolveStoreUrl('b2b-22154705262941f196')).toBe(
'https://winport.m.1688.com/page/index.html?memberId=b2b-22154705262941f196',
);
expect(__test__.canonicalizeStoreUrl('https://yinuoweierfushi.1688.com/page/index.html?spm=foo')).toBe(
'https://yinuoweierfushi.1688.com',
);
expect(__test__.canonicalizeItemUrl('http://detail.m.1688.com/page/index.html?offerId=910933345396&spm=x')).toBe(
'https://detail.1688.com/offer/910933345396.html',
);
expect(__test__.canonicalizeSellerUrl('https://yinuoweierfushi.1688.com/page/contactinfo.html?tracelog=1')).toBe(
'https://yinuoweierfushi.1688.com',
);
expect(__test__.extractShopId('https://yinuoweierfushi.1688.com/page/index.html')).toBe('yinuoweierfushi');
});
it('parses price ranges and moq text', () => {
expect(__test__.parsePriceText('¥96.00-98.00')).toEqual({
price_text: '¥96.00-98.00',
price_min: 96,
price_max: 98,
currency: 'CNY',
});
expect(__test__.parsePriceText('¥ 14 .28')).toEqual({
price_text: '¥14.28',
price_min: 14.28,
price_max: 14.28,
currency: 'CNY',
});
expect(__test__.parseMoqText('3套起批')).toEqual({
moq_text: '3套起批',
moq_value: 3,
});
expect(__test__.parseMoqText('2~999个')).toEqual({
moq_text: '2~999个',
moq_value: 2,
});
});
it('detects captcha and login states', () => {
expect(__test__.extractLocation('山东青岛 送至 江苏苏州')).toBe('山东青岛');
expect(__test__.isCaptchaState({
href: 'https://s.1688.com/_____tmd_____/punish',
title: '验证码拦截',
body_text: '请拖动下方滑块完成验证',
})).toBe(true);
expect(__test__.isLoginState({
href: 'https://login.taobao.com/member/login.jhtml',
title: '账号登录',
body_text: '请登录后继续',
})).toBe(true);
});
});
+672
View File
@@ -0,0 +1,672 @@
import { ArgumentError, AuthRequiredError, CommandExecutionError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
export const SITE = '1688';
export const HOME_URL = 'https://www.1688.com/';
export const SEARCH_URL_PREFIX = 'https://s.1688.com/selloffer/offer_search.htm?charset=utf8&keywords=';
export const DETAIL_URL_PREFIX = 'https://detail.1688.com/offer/';
export const STORE_MOBILE_URL_PREFIX = 'https://winport.m.1688.com/page/index.html?memberId=';
export const STRATEGY = 'cookie';
export const SEARCH_LIMIT_DEFAULT = 20;
export const SEARCH_LIMIT_MAX = 100;
const STORE_GENERIC_HOSTS = new Set(['www', 'detail', 's', 'winport', 'work', 'air', 'dj']);
const TRACKING_QUERY_KEYS = new Set([
'spm',
'tracelog',
'clickid',
'source',
'scene',
'from',
'src',
'ns',
'cna',
'pvid',
]);
const CAPTCHA_URL_MARKER = '/_____tmd_____/punish';
const CAPTCHA_TEXT_PATTERNS = [
'请拖动下方滑块完成验证',
'请按住滑块,拖动到最右边',
'通过验证以确保正常访问',
'验证码拦截',
'访问验证',
'滑动验证',
];
const LOGIN_TEXT_PATTERNS = [
'请登录',
'登录后',
'账号登录',
'手机登录',
'立即登录',
'扫码登录',
'请先完成登录',
'请先登录后查看',
];
const LOGIN_URL_PATTERNS = ['/member/login', 'passport', 'login.taobao.com', 'account.1688.com'];
export const FACTORY_BADGE_PATTERNS = [
'源头工厂',
'深度验厂',
'实力工厂',
'工厂档案',
'加工专区',
'验厂报告',
'厂家直销',
'生产厂家',
'工厂直供',
];
export const SERVICE_BADGE_PATTERNS = [
'延期必赔',
'品质保障',
'破损包赔',
'退货包运费',
'晚发必赔',
'7*24小时响应',
'48小时发货',
'72小时发货',
'后天达',
'包邮',
'闪电拿样',
];
const CHINA_LOCATIONS = [
'北京',
'天津',
'上海',
'重庆',
'河北',
'山西',
'辽宁',
'吉林',
'黑龙江',
'江苏',
'浙江',
'安徽',
'福建',
'江西',
'山东',
'河南',
'湖北',
'湖南',
'广东',
'海南',
'四川',
'贵州',
'云南',
'陕西',
'甘肃',
'青海',
'台湾',
'内蒙古',
'广西',
'西藏',
'宁夏',
'新疆',
'香港',
'澳门',
];
export interface ProvenanceFields {
source_url: string;
fetched_at: string;
strategy: string;
}
export interface PageState {
href: string;
title: string;
body_text: string;
}
export interface PriceRange {
price_text: string;
price_min: number | null;
price_max: number | null;
currency: string | null;
}
export interface MoqValue {
moq_text: string;
moq_value: number | null;
}
export interface PriceTier {
quantity_text: string;
quantity_min: number | null;
price_text: string;
price: number | null;
currency: string | null;
}
export interface SearchCandidate {
item_url: string;
title: string;
container_text: string;
seller_name: string | null;
seller_url: string | null;
}
export interface MediaSource {
type: 'image' | 'video';
group: 'main' | 'sku' | 'detail' | 'video' | 'unknown';
url: string;
source?: string;
}
export function cleanText(value: unknown): string {
return typeof value === 'string'
? value.replace(/\u00a0/g, ' ').replace(/\s+/g, ' ').trim()
: '';
}
export function cleanMultilineText(value: unknown): string {
return typeof value === 'string'
? value
.replace(/\u00a0/g, ' ')
.split('\n')
.map((line) => line.replace(/\s+/g, ' ').trim())
.filter(Boolean)
.join('\n')
: '';
}
export function uniqueNonEmpty(values: Array<string | null | undefined>): string[] {
return [...new Set(values.map((value) => cleanText(value)).filter(Boolean))];
}
export function parseSearchLimit(input: unknown): number {
const parsed = Number.parseInt(String(input ?? SEARCH_LIMIT_DEFAULT), 10);
if (!Number.isFinite(parsed) || parsed < 1) {
throw new ArgumentError(
'1688 search --limit must be a positive integer',
'Example: opencli 1688 search "桌面置物架" --limit 20',
);
}
return Math.min(SEARCH_LIMIT_MAX, parsed);
}
export function buildSearchUrl(query: string): string {
const normalized = cleanText(query);
if (!normalized) {
throw new ArgumentError(
'1688 search query cannot be empty',
'Example: opencli 1688 search "桌面置物架" --limit 20',
);
}
return `${SEARCH_URL_PREFIX}${encodeURIComponent(normalized)}`;
}
export function buildDetailUrl(input: string): string {
const offerId = extractOfferId(input);
if (!offerId) {
throw new ArgumentError(
'1688 item expects an offer URL or offer ID',
'Example: opencli 1688 item 887904326744',
);
}
return `${DETAIL_URL_PREFIX}${offerId}.html`;
}
export function resolveStoreUrl(input: string): string {
const normalized = cleanText(input);
if (!normalized) {
throw new ArgumentError(
'1688 store expects a store URL or member ID',
'Example: opencli 1688 store https://yinuoweierfushi.1688.com/',
);
}
const memberId = extractMemberId(normalized);
if (memberId) {
return `${STORE_MOBILE_URL_PREFIX}${memberId}`;
}
if (/^https?:\/\//i.test(normalized)) {
return canonicalizeStoreUrl(normalized);
}
if (normalized.endsWith('.1688.com')) {
return canonicalizeStoreUrl(`https://${normalized}`);
}
if (/^[a-z0-9-]+$/i.test(normalized)) {
return canonicalizeStoreUrl(`https://${normalized}.1688.com`);
}
throw new ArgumentError(
'1688 store expects a store URL or member ID',
'Example: opencli 1688 store b2b-22154705262941f196',
);
}
export function canonicalizeStoreUrl(input: string): string {
const url = parse1688Url(input);
const memberId = extractMemberId(url.toString());
if (memberId) {
return `${STORE_MOBILE_URL_PREFIX}${memberId}`;
}
const host = normalizeStoreHost(url.hostname);
if (!host) {
throw new ArgumentError(
'Invalid 1688 store URL',
'Example: opencli 1688 store https://yinuoweierfushi.1688.com/',
);
}
return `https://${host}`;
}
export function canonicalizeItemUrl(input: string): string | null {
const offerId = extractOfferId(input);
if (offerId) {
return `${DETAIL_URL_PREFIX}${offerId}.html`;
}
const url = parse1688UrlOrNull(input);
if (!url) return null;
stripTrackingParams(url);
url.hash = '';
return url.toString();
}
export function canonicalizeSellerUrl(input: string): string | null {
const memberId = extractMemberId(input);
if (memberId) {
return `${STORE_MOBILE_URL_PREFIX}${memberId}`;
}
const url = parse1688UrlOrNull(input);
if (!url) return null;
const host = normalizeStoreHost(url.hostname);
if (!host) return null;
return `https://${host}`;
}
export function extractOfferId(input: string): string | null {
const normalized = cleanText(input);
if (!normalized) return null;
const directId = normalized.match(/^\d{6,}$/)?.[0];
if (directId) return directId;
const detailMatch = normalized.match(/\/offer\/(\d{6,})\.html/i);
if (detailMatch) return detailMatch[1];
const queryMatch = normalized.match(/[?&]offerId=(\d{6,})/i);
if (queryMatch) return queryMatch[1];
return null;
}
export function extractMemberId(input: string): string | null {
const normalized = cleanText(input);
if (!normalized) return null;
const direct = normalized.match(/\bb2b-[a-z0-9]+\b/i)?.[0];
if (direct) return direct;
const queryMatch = normalized.match(/[?&]memberId=(b2b-[a-z0-9]+)/i);
if (queryMatch) return queryMatch[1];
const mobileMatch = normalized.match(/\/winport\/(b2b-[a-z0-9]+)\.html/i);
if (mobileMatch) return mobileMatch[1];
return null;
}
export function extractShopId(input: string): string | null {
const normalized = cleanText(input);
if (!normalized) return null;
try {
const url = new URL(/^https?:\/\//i.test(normalized) ? normalized : `https://${normalized}`);
const host = normalizeStoreHost(url.hostname);
if (!host) return null;
return host.split('.')[0] ?? null;
} catch {
return /^[a-z0-9-]+$/i.test(normalized) ? normalized : null;
}
}
export function buildProvenance(sourceUrl: string): ProvenanceFields {
return {
source_url: sourceUrl,
fetched_at: new Date().toISOString(),
strategy: STRATEGY,
};
}
export function parsePriceText(text: string): PriceRange {
const normalized = normalizeNumericText(cleanText(text));
const matches = normalized.match(/\d+(?:,\d{3})*(?:\.\d+)?/g) ?? [];
const values = matches
.map((value) => Number.parseFloat(value.replace(/,/g, '')))
.filter((value) => Number.isFinite(value));
if (values.length === 0) {
return {
price_text: normalized,
price_min: null,
price_max: null,
currency: null,
};
}
return {
price_text: normalized,
price_min: values[0] ?? null,
price_max: values[values.length - 1] ?? values[0] ?? null,
currency: normalized.includes('¥') || normalized.includes('元') ? 'CNY' : null,
};
}
export function normalizePriceTiers(
rawTiers: Array<{ beginAmount?: unknown; price?: unknown }>,
unit: string | null,
): PriceTier[] {
return rawTiers
.map((tier) => {
const quantityMin = toNumber(tier.beginAmount);
const priceText = cleanText(tier.price);
const price = toNumber(tier.price);
return {
quantity_text: quantityMin !== null ? `${quantityMin}${unit ?? ''}` : '',
quantity_min: quantityMin,
price_text: priceText,
price,
currency: priceText ? 'CNY' : null,
};
})
.filter((tier) => tier.price_text);
}
export function parseMoqText(text: string): MoqValue {
const normalized = normalizeNumericText(cleanText(text));
const match = normalized.match(/(\d+(?:\.\d+)?)\s*(件|个|套|箱|包|双|台|把|只|pcs|piece|pieces)?\s*起批/i)
?? normalized.match(/≥\s*(\d+(?:\.\d+)?)/);
const rangeMatch = normalized.match(
/(\d+(?:\.\d+)?)\s*(?:~|-|至|到)\s*\d+(?:\.\d+)?\s*(件|个|套|箱|包|双|台|把|只|pcs|piece|pieces)/i,
);
if (!match && !rangeMatch) {
return {
moq_text: normalized,
moq_value: null,
};
}
return {
moq_text: normalized,
moq_value: Number.parseFloat((match ?? rangeMatch)![1]),
};
}
export function extractLocation(text: string): string | null {
const normalized = cleanMultilineText(text);
const primaryRegion = normalized.split(/送至|发往/)[0] ?? normalized;
const lines = primaryRegion.split('\n');
for (const line of lines) {
const compact = cleanText(line);
if (!compact || compact.length > 16) continue;
if (CHINA_LOCATIONS.some((location) => compact.startsWith(location))) {
return compact;
}
}
const locationPattern = new RegExp(`(${CHINA_LOCATIONS.join('|')})[\\u4e00-\\u9fa5]{0,8}`);
return primaryRegion.match(locationPattern)?.[0] ?? null;
}
export function extractAddress(text: string): string | null {
const normalized = cleanMultilineText(text);
const lineMatch = normalized.match(/地址[:]\s*([^\n]+)/);
if (lineMatch) return cleanText(lineMatch[1]);
return normalized
.split('\n')
.map((line) => cleanText(line))
.find((line) => line.includes('省') || line.includes('市') || line.includes('区') || line.includes('县'))
?? null;
}
export function extractMetric(text: string, label: string): string | null {
const normalized = cleanMultilineText(text);
const direct = normalized.match(new RegExp(`(?:^|\\n)\\s*${escapeForRegex(label)}[:]?\\s*([^\\n]+)`));
if (direct) return cleanText(direct[1]);
const lineBased = normalized.match(new RegExp(`(?:^|\\n)\\s*${escapeForRegex(label)}\\n([^\\n]+)`));
return lineBased ? cleanText(lineBased[1]) : null;
}
export function extractYearsOnPlatform(text: string): string | null {
return text.match(/入驻\d+年/)?.[0] ?? null;
}
export function extractMainBusiness(text: string): string | null {
const value = extractMetric(text, '主营');
return value ? value.replace(/^/, '').trim() : null;
}
export function extractBadges(text: string, candidates: string[]): string[] {
return uniqueNonEmpty(candidates.filter((candidate) => cleanMultilineText(text).includes(candidate)));
}
export function guessTopCategories(text: string): string[] {
const mainBusiness = extractMainBusiness(text);
if (!mainBusiness) return [];
return uniqueNonEmpty(mainBusiness.split(/[、,/|]/).map((value) => value.trim()));
}
export function isCaptchaState(state: Partial<PageState>): boolean {
const href = cleanText(state.href).toLowerCase();
const title = cleanText(state.title);
const bodyText = cleanMultilineText(state.body_text);
if (href.includes(CAPTCHA_URL_MARKER)) return true;
return CAPTCHA_TEXT_PATTERNS.some((pattern) => title.includes(pattern) || bodyText.includes(pattern));
}
export function isLoginState(state: Partial<PageState>): boolean {
const href = cleanText(state.href).toLowerCase();
const title = cleanText(state.title);
const bodyText = cleanMultilineText(state.body_text);
if (LOGIN_URL_PATTERNS.some((pattern) => href.includes(pattern))) return true;
return LOGIN_TEXT_PATTERNS.some((pattern) => title.includes(pattern) || bodyText.includes(pattern));
}
export function buildCaptchaHint(action: string): string {
return [
`Open a clean 1688 ${action} page in the shared Chrome profile and finish any slider challenge first.`,
'If you run opencli via CDP, set OPENCLI_CDP_TARGET=1688.com or a more specific 1688 host before retrying.',
].join(' ');
}
export async function readPageState(page: IPage): Promise<PageState> {
const result = await page.evaluate(`
(() => ({
href: window.location.href,
title: document.title || '',
body_text: document.body ? document.body.innerText || '' : '',
}))()
`) as Partial<PageState>;
return {
href: cleanText(result.href),
title: cleanText(result.title),
body_text: cleanMultilineText(result.body_text),
};
}
export async function gotoAndReadState(
page: IPage,
url: string,
settleMs: number = 2500,
action: string = 'page',
): Promise<PageState> {
try {
await page.goto(url, { settleMs });
await page.wait(1.5);
return readPageState(page);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
if (
message.includes('Inspected target navigated or closed')
|| message.includes('Cannot find context with specified id')
|| message.includes('Target closed')
) {
throw new CommandExecutionError(
`1688 ${action} navigation lost the current browser target`,
`${buildCaptchaHint(action)} If CDP is attached to a stale or blocked tab, open a fresh 1688 tab and point OPENCLI_CDP_TARGET at that tab.`,
);
}
throw error;
}
}
export async function ensure1688Session(page: IPage): Promise<void> {
const state = await gotoAndReadState(page, HOME_URL, 1500, 'homepage');
assertAuthenticatedState(state, 'homepage');
}
export function assertAuthenticatedState(state: PageState, action: string): void {
if (!isCaptchaState(state) && !isLoginState(state)) return;
throw new AuthRequiredError('1688.com', `请先在共享 Chrome 完成 1688 登录/验证,再重试(${action}`);
}
export function assertNotCaptcha(state: PageState, action: string): void {
assertAuthenticatedState(state, action);
}
export function toNumber(value: unknown): number | null {
if (typeof value === 'number' && Number.isFinite(value)) {
return value;
}
if (typeof value === 'string') {
const normalized = value.replace(/,/g, '').trim();
if (!normalized) return null;
const parsed = Number.parseFloat(normalized);
return Number.isFinite(parsed) ? parsed : null;
}
return null;
}
export function limitCandidates<T>(values: T[], limit: number): T[] {
const normalizedLimit = Math.max(1, Math.trunc(limit) || 1);
return values.slice(0, normalizedLimit);
}
export function normalizeMediaUrl(input: unknown): string {
const raw = cleanText(input);
if (!raw) return '';
let value = raw
.replace(/^url\((.*)\)$/i, '$1')
.replace(/^['"]|['"]$/g, '')
.replace(/\\u002F/g, '/')
.replace(/&amp;/g, '&')
.trim();
if (!value || value.startsWith('data:') || value.startsWith('blob:')) return '';
if (value.startsWith('//')) value = `https:${value}`;
try {
const url = new URL(value);
return url.toString();
} catch {
return '';
}
}
export function uniqueMediaSources(values: MediaSource[]): MediaSource[] {
const seen = new Set<string>();
const result: MediaSource[] = [];
for (const value of values) {
const url = normalizeMediaUrl(value.url);
if (!url) continue;
const key = `${value.type}:${url}`;
if (seen.has(key)) continue;
seen.add(key);
result.push({
...value,
url,
source: cleanText(value.source) || undefined,
});
}
return result;
}
function normalizeNumericText(value: string): string {
return value
.replace(/([¥$€])\s+(?=\d)/g, '$1')
.replace(/(\d)\s*\.\s*(\d)/g, '$1.$2')
.replace(/\s*([~-])\s*/g, '$1')
.trim();
}
function escapeForRegex(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
function parse1688Url(input: string): URL {
const normalized = cleanText(input);
try {
const url = new URL(normalized);
if (!url.hostname.endsWith('.1688.com') && url.hostname !== '1688.com' && url.hostname !== 'www.1688.com') {
throw new Error('invalid-host');
}
stripTrackingParams(url);
url.hash = '';
return url;
} catch {
throw new ArgumentError(
'Invalid 1688 URL',
'Use a URL under 1688.com (for example: https://detail.1688.com/offer/887904326744.html)',
);
}
}
function parse1688UrlOrNull(input: string): URL | null {
try {
return parse1688Url(input);
} catch {
return null;
}
}
function normalizeStoreHost(hostname: string): string | null {
const lower = cleanText(hostname).toLowerCase();
if (!lower.endsWith('.1688.com')) return null;
const [subdomain] = lower.split('.');
if (!subdomain || STORE_GENERIC_HOSTS.has(subdomain)) return null;
return lower;
}
function stripTrackingParams(url: URL): void {
const keys = [...url.searchParams.keys()];
for (const key of keys) {
if (TRACKING_QUERY_KEYS.has(key) || key.toLowerCase().startsWith('utm_')) {
url.searchParams.delete(key);
}
}
}
export const __test__ = {
SEARCH_LIMIT_DEFAULT,
SEARCH_LIMIT_MAX,
parseSearchLimit,
buildSearchUrl,
buildDetailUrl,
resolveStoreUrl,
canonicalizeStoreUrl,
canonicalizeItemUrl,
canonicalizeSellerUrl,
extractOfferId,
extractMemberId,
extractShopId,
parsePriceText,
normalizePriceTiers,
parseMoqText,
extractLocation,
extractAddress,
extractMetric,
extractYearsOnPlatform,
extractMainBusiness,
extractBadges,
guessTopCategories,
isCaptchaState,
isLoginState,
cleanText,
cleanMultilineText,
uniqueNonEmpty,
normalizeMediaUrl,
uniqueMediaSources,
limitCandidates,
};
+69
View File
@@ -0,0 +1,69 @@
import { describe, expect, it } from 'vitest';
import { __test__ } from './store.js';
describe('1688 store normalization', () => {
it('merges store contact text with seller seed data', () => {
const result = __test__.normalizeStorePayload({
resolvedUrl: 'https://yinuoweierfushi.1688.com/?offerId=887904326744',
explicitMemberId: null,
storePayload: {
href: 'https://yinuoweierfushi.1688.com/page/index.html',
bodyText: `
青岛沁澜衣品服装有限公司
联系方式
地址:山东省青岛市即墨区环秀街道办事处湘江二路97号甲
`,
offerLinks: ['https://detail.1688.com/offer/887904326744.html'],
},
contactPayload: {
href: 'https://yinuoweierfushi.1688.com/page/contactinfo.html',
bodyText: `
青岛沁澜衣品服装有限公司
电话:86 0532 86655366
手机:15963238678
地址:山东省青岛市即墨区环秀街道办事处湘江二路97号甲
`,
},
seed: {
bodyText: `
入驻13年
主营:大码女装
店铺回头率
87%
延期必赔
品质保障
`,
seller: {
companyName: '青岛沁澜衣品服装有限公司',
memberId: 'b2b-1641351767',
winportUrl: 'https://yinuoweierfushi.1688.com/page/index.html?spm=abc',
},
services: [{ serviceName: '延期必赔' }, { serviceName: '品质保障' }],
},
});
expect(result.member_id).toBe('b2b-1641351767');
expect(result.store_url).toBe('https://yinuoweierfushi.1688.com');
expect(result.company_url).toBe('https://yinuoweierfushi.1688.com/page/contactinfo.html');
expect(result.years_on_platform_text).toBe('入驻13年');
expect(result.location).toBe('山东省青岛市即墨区环秀街道办事处湘江二路97号甲');
expect(result.return_rate_text).toContain('87%');
expect(result.top_categories).toEqual(['大码女装']);
expect(result.service_badges).toEqual(['延期必赔', '品质保障']);
});
it('builds contact urls and extracts offer ids', () => {
expect(__test__.safeCanonicalStoreUrl('https://yinuoweierfushi.1688.com/page/index.html?spm=foo')).toBe(
'https://yinuoweierfushi.1688.com',
);
expect(__test__.buildContactUrl('https://yinuoweierfushi.1688.com')).toBe(
'https://yinuoweierfushi.1688.com/page/contactinfo.html',
);
expect(__test__.firstOfferId([
'https://detail.1688.com/offer/887904326744.html',
])).toBe('887904326744');
expect(__test__.firstContactUrl([
'https://yinuoweierfushi.1688.com/page/contactinfo.html?spm=1',
])).toBe('https://yinuoweierfushi.1688.com/page/contactinfo.html');
});
});
+300
View File
@@ -0,0 +1,300 @@
import { CommandExecutionError, EmptyResultError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
FACTORY_BADGE_PATTERNS,
SERVICE_BADGE_PATTERNS,
assertAuthenticatedState,
buildDetailUrl,
buildProvenance,
canonicalizeSellerUrl,
canonicalizeStoreUrl,
cleanMultilineText,
cleanText,
extractAddress,
extractBadges,
extractMemberId,
extractMetric,
extractOfferId,
extractShopId,
extractYearsOnPlatform,
gotoAndReadState,
guessTopCategories,
resolveStoreUrl,
uniqueNonEmpty,
} from './shared.js';
interface StoreBrowserPayload {
href?: string;
title?: string;
bodyText?: string;
offerLinks?: string[];
contactLinks?: string[];
}
interface StoreItemSeed {
href?: string;
bodyText?: string;
seller?: {
companyName?: string;
memberId?: string;
winportUrl?: string;
sellerWinportUrlMap?: Record<string, string>;
};
services?: Array<{ serviceName?: string }>;
}
function normalizeStorePayload(input: {
resolvedUrl: string;
storePayload: StoreBrowserPayload | null;
contactPayload: StoreBrowserPayload | null;
seed: StoreItemSeed | null;
explicitMemberId: string | null;
}): Record<string, unknown> {
const storePayload = input.storePayload;
const contactPayload = input.contactPayload;
const seed = input.seed;
const contactText = cleanMultilineText(contactPayload?.bodyText);
const storeText = cleanMultilineText(storePayload?.bodyText);
const seedText = cleanMultilineText(seed?.bodyText);
const combinedText = [contactText, storeText, seedText].filter(Boolean).join('\n');
const sellerUrlRaw = cleanText(
seed?.seller?.winportUrl
?? seed?.seller?.sellerWinportUrlMap?.defaultUrl
?? storePayload?.href
?? input.resolvedUrl,
);
const storeUrl = safeCanonicalStoreUrl(sellerUrlRaw || input.resolvedUrl) ?? input.resolvedUrl;
const sellerUrl = canonicalizeSellerUrl(sellerUrlRaw) ?? storeUrl;
const companyUrl = pickCompanyUrl(contactPayload?.href, storeUrl);
const memberId = cleanText(seed?.seller?.memberId)
|| input.explicitMemberId
|| extractMemberId(input.resolvedUrl)
|| extractMemberId(storePayload?.href ?? '')
|| null;
const shopId = extractShopId(sellerUrl) ?? extractShopId(storeUrl);
const companyName = cleanText(seed?.seller?.companyName)
|| firstNamedLine(contactText)
|| firstNamedLine(storeText)
|| null;
const serviceBadges = uniqueNonEmpty([
...extractBadges(combinedText, SERVICE_BADGE_PATTERNS),
...((seed?.services ?? []).map((service) => cleanText(service.serviceName))),
]);
const factoryBadges = extractBadges(combinedText, FACTORY_BADGE_PATTERNS);
return {
member_id: memberId,
shop_id: shopId,
store_name: companyName,
store_url: storeUrl,
company_name: companyName,
company_url: companyUrl,
business_model_text: firstMetric(combinedText, ['经营模式', '生产加工', '主营产品']),
years_on_platform_text: extractYearsOnPlatform(combinedText),
location: extractAddress(contactText) ?? extractAddress(storeText),
staff_size_text: firstMetric(combinedText, ['员工人数', '员工总数']),
factory_badges: factoryBadges,
service_badges: serviceBadges,
response_rate_text: firstMetric(combinedText, ['响应率', '回复率', '响应速度']),
return_rate_text: extractReturnRate(combinedText),
top_categories: guessTopCategories(combinedText),
phone_text: extractMetric(contactText, '电话'),
mobile_text: extractMetric(contactText, '手机'),
...buildProvenance(cleanText(contactPayload?.href) || cleanText(storePayload?.href) || input.resolvedUrl),
};
}
function safeCanonicalStoreUrl(url: string): string | null {
try {
return canonicalizeStoreUrl(url);
} catch {
return null;
}
}
function pickCompanyUrl(contactHref: string | undefined, storeUrl: string): string | null {
const fromPage = cleanText(contactHref);
if (fromPage) {
const normalized = buildContactUrl(fromPage);
if (normalized) return normalized;
}
return buildContactUrl(storeUrl);
}
function buildContactUrl(storeUrl: string): string | null {
try {
const parsed = new URL(storeUrl);
if (!parsed.hostname.endsWith('.1688.com')) return null;
return `${parsed.protocol}//${parsed.hostname}/page/contactinfo.html`;
} catch {
return null;
}
}
function firstNamedLine(text: string): string | null {
return text
.split('\n')
.map((line) => cleanText(line))
.find((line) => line.includes('有限公司') || line.includes('商行') || line.includes('工厂'))
?? null;
}
function firstMetric(text: string, labels: string[]): string | null {
for (const label of labels) {
const value = extractMetric(text, label);
if (value) return value;
}
return null;
}
function extractReturnRate(text: string): string | null {
const inline = text.match(/回头率\s*([0-9.]+%)/);
if (inline) return cleanText(inline[0]);
const multiline = text.match(/回头率\s*\n\s*([0-9.]+%)/);
if (!multiline) return null;
return `回头率${cleanText(multiline[1])}`;
}
function firstOfferId(links: string[]): string | null {
for (const link of links) {
const offerId = extractOfferId(link);
if (offerId) return offerId;
}
return null;
}
function firstContactUrl(links: string[]): string | null {
for (const link of links) {
const url = buildContactUrl(link);
if (url) return url;
}
return null;
}
async function readStorePayload(page: IPage, url: string, action: string): Promise<StoreBrowserPayload> {
const state = await gotoAndReadState(page, url, 2500, action);
assertAuthenticatedState(state, action);
return await page.evaluate(`
(() => ({
href: window.location.href,
title: document.title || '',
bodyText: document.body ? document.body.innerText || '' : '',
offerLinks: Array.from(document.querySelectorAll('a[href*="detail.1688.com/offer/"], a[href*="offerId="]'))
.map((anchor) => anchor.href)
.filter(Boolean),
contactLinks: Array.from(document.querySelectorAll('a[href*="contactinfo"]'))
.map((anchor) => anchor.href)
.filter(Boolean),
}))()
`) as StoreBrowserPayload;
}
async function readItemSeed(page: IPage, offerId: string): Promise<StoreItemSeed> {
const itemUrl = buildDetailUrl(offerId);
const state = await gotoAndReadState(page, itemUrl, 2500, 'store seed item');
assertAuthenticatedState(state, 'store seed item');
const seed = await page.evaluate(`
(() => {
const model = window.context?.result?.global?.globalData?.model ?? null;
const toJson = (value) => JSON.parse(JSON.stringify(value ?? null));
return {
href: window.location.href,
bodyText: document.body ? document.body.innerText || '' : '',
seller: toJson(model?.sellerModel),
services: toJson(model?.shippingServices?.fields?.buyerProtectionModel ?? []),
};
})()
`) as StoreItemSeed;
const hasSellerContext = !!cleanText(seed?.seller?.memberId) || !!cleanText(seed?.seller?.winportUrl);
if (!hasSellerContext) {
throw new CommandExecutionError(
'1688 store seed item did not expose seller context',
'当前 tab 非商品详情上下文,请切到 detail.1688.com 商品页并重试',
);
}
return seed;
}
function hasAnyEvidence(
storePayload: StoreBrowserPayload | null,
contactPayload: StoreBrowserPayload | null,
seed: StoreItemSeed | null,
): boolean {
return !!cleanText(storePayload?.bodyText)
|| !!cleanText(contactPayload?.bodyText)
|| !!cleanText(seed?.bodyText);
}
cli({
site: '1688',
name: 'store',
description: '1688 店铺/供应商公开信息(联系方式、主营、入驻年限、公开服务信号)',
domain: 'www.1688.com',
strategy: Strategy.COOKIE,
navigateBefore: false,
args: [
{
name: 'input',
required: true,
positional: true,
help: '1688 店铺 URL 或 member ID(如 b2b-22154705262941f196',
},
],
columns: ['store_name', 'years_on_platform_text', 'location', 'return_rate_text'],
func: async (page, kwargs) => {
const rawInput = String(kwargs.input ?? '');
const resolvedUrl = resolveStoreUrl(rawInput);
const explicitMemberId = extractMemberId(rawInput);
const storePayload = await readStorePayload(page, resolvedUrl, 'store');
const contactUrl = firstContactUrl(storePayload.contactLinks ?? []) || buildContactUrl(storePayload.href || resolvedUrl);
const contactPayload = contactUrl ? await readStorePayload(page, contactUrl, 'store contact') : null;
const offerId = extractOfferId(rawInput)
|| firstOfferId(storePayload.offerLinks ?? [])
|| firstOfferId(contactPayload?.offerLinks ?? []);
let seed: StoreItemSeed | null = null;
if (offerId) {
try {
seed = await readItemSeed(page, offerId);
} catch (error) {
if (!(error instanceof CommandExecutionError)) throw error;
}
}
if (!hasAnyEvidence(storePayload, contactPayload, seed)) {
throw new EmptyResultError(
'1688 store',
'Store page is reachable but no visible fields were extracted. Open the store page in Chrome and retry.',
);
}
return [
normalizeStorePayload({
resolvedUrl,
storePayload,
contactPayload,
seed,
explicitMemberId,
}),
];
},
});
export const __test__ = {
normalizeStorePayload,
safeCanonicalStoreUrl,
buildContactUrl,
firstNamedLine,
firstMetric,
extractReturnRate,
firstOfferId,
firstContactUrl,
};
@@ -3,9 +3,9 @@
*
* Fetches the full content of a 36kr article given its ID or URL.
*/
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import type { IPage } from '../../types.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
/** Extract article ID from a full URL or a bare numeric ID string */
function parseArticleId(input: string): string {
+12 -7
View File
@@ -1,12 +1,12 @@
/**
* 36kr hot-list INTERCEPT strategy.
* 36kr hot-list DOM scraping.
*
* Navigates to the 36kr hot-list page and scrapes rendered article links.
* Supports category types: renqi (), zonghe (), shoucang (), catalog ().
*/
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import type { IPage } from '../../types.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
const TYPE_MAP: Record<string, string> = {
renqi: '人气榜',
@@ -34,7 +34,8 @@ cli({
name: 'hot',
description: '36氪热榜 — trending articles (renqi/zonghe/shoucang/catalog)',
domain: 'www.36kr.com',
strategy: Strategy.INTERCEPT,
strategy: Strategy.PUBLIC,
browser: true,
args: [
{ name: 'limit', type: 'int', default: 20, help: 'Number of items (max 50)' },
{
@@ -58,9 +59,13 @@ cli({
const url = buildHotListUrl(listType);
await page.installInterceptor('36kr.com/api');
await page.goto(url);
await page.waitForCapture(6);
// Poll DOM until article links appear (36kr renders client-side)
const deadline = Date.now() + 5000;
while (Date.now() < deadline) {
if (await page.evaluate('document.querySelectorAll("a[href*=\\"/p/\\"]").length')) break;
await new Promise(r => setTimeout(r, 300));
}
// Scrape rendered article links from DOM (deduplicated)
const domItems: any = await page.evaluate(`
+1 -1
View File
@@ -1,7 +1,7 @@
/**
* 36kr latest news public RSS feed, no browser needed.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: '36kr',
@@ -1,18 +1,19 @@
/**
* 36kr article search INTERCEPT strategy.
* 36kr article search DOM scraping.
*
* Navigates to the 36kr search results page and scrapes rendered articles.
*/
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import type { IPage } from '../../types.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
cli({
site: '36kr',
name: 'search',
description: '搜索36氪文章',
domain: 'www.36kr.com',
strategy: Strategy.INTERCEPT,
strategy: Strategy.PUBLIC,
browser: true,
args: [
{ name: 'query', positional: true, required: true, help: 'Search keyword (e.g. "AI", "OpenAI")' },
{ name: 'limit', type: 'int', default: 20, help: 'Number of results (max 50)' },
@@ -22,9 +23,13 @@ cli({
const count = Math.min(Number(args.limit) || 20, 50);
const query = encodeURIComponent(String(args.query ?? ''));
await page.installInterceptor('36kr.com/api');
await page.goto(`https://www.36kr.com/search/articles/${query}`);
await page.waitForCapture(6);
// Poll DOM until article links appear (36kr renders client-side)
const deadline = Date.now() + 5000;
while (Date.now() < deadline) {
if (await page.evaluate('document.querySelectorAll("a[href*=\\"/p/\\"]").length')) break;
await new Promise(r => setTimeout(r, 300));
}
const domItems: any = await page.evaluate(`
(() => {
@@ -5,9 +5,9 @@
*/
import * as fs from 'node:fs';
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import type { CliOptions } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import type { CliOptions } from '@jackwener/opencli/registry';
/**
* Factory: capture DOM HTML + accessibility snapshot.
@@ -1,4 +1,4 @@
import { cli } from '../../registry.js';
import { cli } from '@jackwener/opencli/registry';
import { createRankingCliOptions } from './rankings.js';
cli(createRankingCliOptions({
@@ -1,6 +1,6 @@
import { CommandExecutionError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
buildDiscussionUrl,
buildProvenance,
@@ -1,4 +1,4 @@
import { cli } from '../../registry.js';
import { cli } from '@jackwener/opencli/registry';
import { createRankingCliOptions } from './rankings.js';
cli(createRankingCliOptions({
@@ -1,4 +1,4 @@
import { cli } from '../../registry.js';
import { cli } from '@jackwener/opencli/registry';
import { createRankingCliOptions } from './rankings.js';
cli(createRankingCliOptions({
@@ -1,6 +1,6 @@
import { CommandExecutionError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
buildProductUrl,
buildProvenance,
@@ -1,6 +1,6 @@
import { CommandExecutionError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
buildProductUrl,
buildProvenance,
@@ -1,6 +1,6 @@
import { CommandExecutionError } from '../../errors.js';
import { Strategy, type CliOptions } from '../../registry.js';
import type { IPage } from '../../types.js';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { Strategy, type CliOptions } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
assertUsableState,
buildProvenance,
@@ -1,6 +1,6 @@
import { CommandExecutionError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import {
buildProvenance,
buildSearchUrl,
@@ -1,5 +1,5 @@
import { ArgumentError, CommandExecutionError } from '../../errors.js';
import type { IPage } from '../../types.js';
import { ArgumentError, CommandExecutionError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
export const SITE = 'amazon';
export const DOMAIN = 'amazon.com';
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import * as fs from 'node:fs';
export const dumpCommand = cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const extractCodeCommand = cli({
site: 'antigravity',
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const modelCommand = cli({
site: 'antigravity',
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const newCommand = cli({
site: 'antigravity',
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const readCommand = cli({
site: 'antigravity',
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const sendCommand = cli({
site: 'antigravity',
@@ -11,10 +11,10 @@
*/
import { createServer, type IncomingMessage, type ServerResponse } from 'node:http';
import { CDPBridge } from '../../browser/cdp.js';
import type { IPage } from '../../types.js';
import { resolveElectronEndpoint } from '../../launcher.js';
import { EXIT_CODES, getErrorMessage } from '../../errors.js';
import { CDPBridge } from '@jackwener/opencli/browser/cdp';
import type { IPage } from '@jackwener/opencli/types';
import { resolveElectronEndpoint } from '@jackwener/opencli/launcher';
import { EXIT_CODES, getErrorMessage } from '@jackwener/opencli/errors';
// ─── Types ───────────────────────────────────────────────────────────
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const statusCommand = cli({
site: 'antigravity',
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
export const watchCommand = cli({
site: 'antigravity',
@@ -1,5 +1,5 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { getRegistry } from '../../registry.js';
import { getRegistry } from '@jackwener/opencli/registry';
import './search.js';
import './top.js';
@@ -1,5 +1,5 @@
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import { itunesFetch, formatDuration, formatDate } from './utils.js';
cli({
@@ -1,5 +1,5 @@
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import { itunesFetch } from './utils.js';
cli({
@@ -1,5 +1,5 @@
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
// Apple Marketing Tools RSS API — public, no key required
const CHARTS_URL = 'https://rss.marketingtools.apple.com/api/v2';
@@ -5,7 +5,7 @@
* https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/iTuneSearchAPI/
*/
import { CliError } from '../../errors.js';
import { CliError } from '@jackwener/opencli/errors';
const BASE = 'https://itunes.apple.com';
@@ -1,5 +1,5 @@
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import { arxivFetch, parseEntries } from './utils.js';
cli({
@@ -1,5 +1,5 @@
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import { arxivFetch, parseEntries } from './utils.js';
cli({
@@ -5,7 +5,7 @@
* https://info.arxiv.org/help/api/index.html
*/
import { CliError } from '../../errors.js';
import { CliError } from '@jackwener/opencli/errors';
export const ARXIV_BASE = 'https://export.arxiv.org/api/query';
@@ -1,5 +1,5 @@
import { AuthRequiredError, EmptyResultError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import { AuthRequiredError, EmptyResultError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
/**
* band bands List all Bands you belong to.
@@ -1,5 +1,5 @@
import { AuthRequiredError, EmptyResultError, SelectorError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import { AuthRequiredError, EmptyResultError, SelectorError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
/**
* band mentions Show Band notifications where you were @mentioned.
+4 -4
View File
@@ -1,7 +1,7 @@
import { AuthRequiredError, EmptyResultError } from '../../errors.js';
import { formatCookieHeader } from '../../download/index.js';
import { downloadMedia } from '../../download/media-download.js';
import { cli, Strategy } from '../../registry.js';
import { AuthRequiredError, EmptyResultError } from '@jackwener/opencli/errors';
import { formatCookieHeader } from '@jackwener/opencli/download';
import { downloadMedia } from '@jackwener/opencli/download/media-download';
import { cli, Strategy } from '@jackwener/opencli/registry';
/**
* band post Export full content of a Band post: body, comments, and optional photo download.
@@ -1,5 +1,5 @@
import { AuthRequiredError, EmptyResultError } from '../../errors.js';
import { cli, Strategy } from '../../registry.js';
import { AuthRequiredError, EmptyResultError } from '@jackwener/opencli/errors';
import { cli, Strategy } from '@jackwener/opencli/registry';
/**
* band posts List posts from a specific Band.
@@ -3,7 +3,7 @@
* Shows high volume/OI ratio trades that may indicate institutional activity.
* Auth: CSRF token from <meta name="csrf-token"> + session cookies.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: 'barchart',
@@ -3,7 +3,7 @@
* for near-the-money options on a given symbol.
* Auth: CSRF token from <meta name="csrf-token"> + session cookies.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: 'barchart',
@@ -2,7 +2,7 @@
* Barchart options chain strike, bid/ask, volume, OI, greeks, IV.
* Auth: CSRF token from <meta name="csrf-token"> + session cookies.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: 'barchart',
@@ -2,7 +2,7 @@
* Barchart stock quote price, volume, market cap, P/E, EPS, and key metrics.
* Auth: CSRF token from <meta name="csrf-token"> + session cookies.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: 'barchart',
+1 -1
View File
@@ -1,7 +1,7 @@
/**
* BBC News headlines public RSS feed, no browser needed.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
cli({
site: 'bbc',
@@ -9,7 +9,7 @@ vi.mock('./utils.js', async (importOriginal) => ({
apiGet: mockApiGet,
}));
import { getRegistry } from '../../registry.js';
import { getRegistry } from '@jackwener/opencli/registry';
import './comments.js';
describe('bilibili comments', () => {
@@ -3,7 +3,7 @@
* Uses the /x/v2/reply/main endpoint which is stable and doesn't depend on DOM structure.
*/
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, resolveBvid } from './utils.js';
cli({
@@ -8,9 +8,9 @@
* - yt-dlp must be installed: pip install yt-dlp
*/
import { cli, Strategy } from '../../registry.js';
import { checkYtdlp, sanitizeFilename } from '../../download/index.js';
import { downloadMedia } from '../../download/media-download.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { checkYtdlp, sanitizeFilename } from '@jackwener/opencli/download';
import { downloadMedia } from '@jackwener/opencli/download/media-download';
import { resolveBvid } from './utils.js';
cli({
@@ -8,7 +8,7 @@ vi.mock('./utils.js', () => ({
apiGet: mockApiGet,
}));
import { getRegistry } from '../../registry.js';
import { getRegistry } from '@jackwener/opencli/registry';
import './dynamic.js';
describe('bilibili dynamic adapter', () => {
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, payloadData, getSelfUid } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, payloadData, getSelfUid, stripHtml } from './utils.js';
cli({
@@ -1,6 +1,6 @@
import { cli, Strategy } from '../../registry.js';
import { CommandExecutionError } from '../../errors.js';
import type { IPage } from '../../types.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CommandExecutionError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
import { fetchJson, getSelfUid, resolveUid } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, payloadData } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, getSelfUid } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, stripHtml } from './utils.js';
cli({
@@ -1,6 +1,6 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AuthRequiredError, EmptyResultError } from '../../errors.js';
import { AuthRequiredError, EmptyResultError } from '@jackwener/opencli/errors';
const { mockApiGet } = vi.hoisted(() => ({
mockApiGet: vi.fn(),
@@ -11,7 +11,7 @@ vi.mock('./utils.js', async (importOriginal) => ({
apiGet: mockApiGet,
}));
import { getRegistry } from '../../registry.js';
import { getRegistry } from '@jackwener/opencli/registry';
import './subtitle.js';
describe('bilibili subtitle', () => {
@@ -1,6 +1,6 @@
import { cli, Strategy } from '../../registry.js';
import { AuthRequiredError, CommandExecutionError, EmptyResultError, SelectorError } from '../../errors.js';
import type { IPage } from '../../types.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { AuthRequiredError, CommandExecutionError, EmptyResultError, SelectorError } from '@jackwener/opencli/errors';
import type { IPage } from '@jackwener/opencli/types';
import { apiGet, resolveBvid } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { apiGet, payloadData, resolveUid } from './utils.js';
cli({
@@ -3,8 +3,8 @@
*/
import https from 'node:https';
import type { IPage } from '../../types.js';
import { AuthRequiredError, EmptyResultError } from '../../errors.js';
import type { IPage } from '@jackwener/opencli/types';
import { AuthRequiredError, EmptyResultError } from '@jackwener/opencli/errors';
/**
* Resolve Bilibili short URL / short code to BV ID.
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { fetchBloombergFeed } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { fetchBloombergFeed } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { BLOOMBERG_FEEDS } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { fetchBloombergFeed } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { fetchBloombergFeed } from './utils.js';
cli({
@@ -1,4 +1,4 @@
import { cli, Strategy } from '../../registry.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { fetchBloombergFeed } from './utils.js';
cli({
@@ -1,5 +1,5 @@
import { cli, Strategy } from '../../registry.js';
import { CliError } from '../../errors.js';
import { cli, Strategy } from '@jackwener/opencli/registry';
import { CliError } from '@jackwener/opencli/errors';
import {
extractStoryMediaLinks,
renderStoryBody,

Some files were not shown because too many files have changed in this diff Show More