refactor(trace): retire diagnostic repair path (#1257)

* refactor(trace): retire diagnostic repair path

* chore(trace): clarify artifact summary guidance

* chore(trace): version trace receipt schema
This commit is contained in:
jakevin
2026-05-03 15:19:44 +08:00
committed by GitHub
parent 5f0cce7b22
commit 4ceb3314fe
21 changed files with 703 additions and 866 deletions
+1
View File
@@ -5,6 +5,7 @@
### Features
* **observation** — add trace artifact primitives, `browser console`, `browser network --since/--follow/--failed`, and adapter `--trace=retain-on-failure` for failure-retained browser evidence.
* **autofix** — retire `OPENCLI_DIAGNOSTIC`; adapter repair now uses `--trace retain-on-failure`, trace `summary.md`, and error-envelope trace metadata.
* **browser** — `bind` attaches `bound:*` workspaces to user-owned Chrome tabs without taking over window lifecycle; `sessions` reports `idleMsRemaining: null` for bound workspaces because they do not schedule idle close timers. ([#1169](https://github.com/jackwener/opencli/issues/1169), [#929](https://github.com/jackwener/opencli/issues/929))
* **browser lifecycle** — owned browser workspaces now lease tabs inside a shared dedicated automation container instead of owning one Chrome window per workspace; lease state is persisted for MV3 service-worker reconciliation and idle cleanup is backed by alarms.
* **web read** — make page extraction render-aware: same-origin iframe content is merged into the Markdown source, `--wait-for` can wait inside main/iframe documents, `--wait-until networkidle` waits for captured requests to settle, and `--diagnose` reports frames, empty containers, and API-like XHRs for shell/AJAX pages.
-1
View File
@@ -205,7 +205,6 @@ OpenCLI is not only for websites. It can also:
| `OPENCLI_CDP_ENDPOINT` | — | Chrome DevTools Protocol endpoint for remote browser or Electron apps |
| `OPENCLI_CDP_TARGET` | — | Filter CDP targets by URL substring (e.g. `detail.1688.com`) |
| `OPENCLI_VERBOSE` | `false` | Enable verbose logging (`-v` flag also works) |
| `OPENCLI_DIAGNOSTIC` | `false` | Set to `1` to capture structured diagnostic context on failures |
| `DEBUG_SNAPSHOT` | — | Set to `1` for DOM snapshot debug output |
`--focus` works for both `opencli browser *` and browser-backed adapter commands. `--live` is mainly for adapter commands: browser subcommands already keep the automation lease open until you run `opencli browser close` or the idle timeout expires.
-1
View File
@@ -188,7 +188,6 @@ OpenCLI 不只是网站 CLI,还可以:
| `OPENCLI_CDP_ENDPOINT` | — | Chrome DevTools Protocol 端点,用于远程浏览器或 Electron 应用 |
| `OPENCLI_CDP_TARGET` | — | 按 URL 子串过滤 CDP target(如 `detail.1688.com` |
| `OPENCLI_VERBOSE` | `false` | 启用详细日志(`-v` 也可以) |
| `OPENCLI_DIAGNOSTIC` | `false` | 设为 `1` 时在失败时输出结构化诊断上下文 |
| `DEBUG_SNAPSHOT` | — | 设为 `1` 输出 DOM 快照调试信息 |
`--focus` 同时适用于 `opencli browser *` 和浏览器型 adapter 命令。`--live` 主要是给 adapter 命令用的:`browser` 子命令本来就会一直保留 automation window,直到你手动执行 `opencli browser close` 或等空闲超时。
+71 -73
View File
@@ -1,28 +1,28 @@
# Self-Repair Protocol — Design Document
**Authors**: @opus0, @codex-mini0
**Date**: 2026-04-07
**Status**: Approved
**Authors**: @opus0, @codex-mini0
**Date**: 2026-04-07
**Status**: Approved, updated for trace-based repair
**Supersedes**: `designs/autofix-incident-repair.md` (PR #863, deferred to Phase 2)
---
## Problem Statement
When an AI agent uses `opencli <site> <command>` and the command fails (site changed DOM, API, or response schema), the agent should **automatically repair the adapter and retry** without human intervention or pre-written spec files.
When an AI agent uses `opencli <site> <command>` and the command fails because the site changed DOM, API, or response schema, the agent should automatically repair the adapter and retry without human intervention or pre-written spec files.
### Why the simpler approach
From first principles, the agent needs five things:
The previous design (PR #863) required pre-authoring `command-specs.json` with verify checks, safety profiles, and failure taxonomy before any command could be repaired. This created a chicken-and-egg problem: you can only repair commands you've already written specs for.
1. The failing command it just ran.
2. The structured error envelope from stderr.
3. The adapter source path.
4. Browser runtime evidence: actions, page state, network, console, screenshot.
5. A verify oracle: re-run the same command.
From first principles, the agent already has everything it needs:
1. **The failing command** — it just ran it
2. **The error output** — stdout/stderr
3. **The adapter source** — resolved via `RepairContext.adapter.sourcePath`
4. **Diagnostic context** — DOM snapshot, network requests (via `OPENCLI_DIAGNOSTIC=1`)
5. **A verify oracle** — re-run the same command
No spec file needed. The command itself is the spec.
The command itself is the spec. The trace artifact is the evidence channel.
---
@@ -30,31 +30,34 @@ No spec file needed. The command itself is the spec.
### Core Protocol
```
```text
Agent runs: opencli <site> <command> [args...]
Command succeeds continue task
Command fails
1. Re-run with OPENCLI_DIAGNOSTIC=1 to collect RepairContext
2. Read adapter source from RepairContext.adapter.sourcePath
3. Analyze: error code + DOM snapshot + network requests → root cause
4. Edit the adapter file at RepairContext.adapter.sourcePath
5. Retry the original command
6. If still failing → repeat (max 3 rounds)
7. If 3 rounds exhausted → report failure, do not loop further
-> Command succeeds -> continue task
-> Command fails ->
1. Re-run with --trace retain-on-failure to collect a trace artifact
2. Read trace.summaryPath from the error envelope
3. Read adapterSourcePath from summary.md front matter
4. Analyze: error code + failed network + console + state/action timeline -> root cause
5. Edit the adapter file at adapterSourcePath
6. Retry the original command
7. If still failing -> repeat (max 3 rounds)
8. If 3 rounds exhausted -> report failure, do not loop further
```
### Scope Constraint
**Only modify the adapter file identified by `RepairContext.adapter.sourcePath`.**
Only modify the adapter file identified by `adapterSourcePath` in trace `summary.md` front matter.
The diagnostic resolves the actual editable source path at runtime — it may be:
- `clis/<site>/*.js` — repo-local adapters (dev/source checkout)
- `~/.opencli/clis/<site>/*.js` — user-local adapters (npm install scenario)
That path may be:
The agent must use the path from the diagnostic, not guess a repo-relative path. This is critical for npm-installed users where `clis/` is not in the repo.
- `clis/<site>/*.js` — repo-local adapters in a source checkout
- `~/.opencli/clis/<site>/*.js` — user-local adapters in npm install scenarios
**Never modify:**
- `src/**` — core runtime (npm package, requires version release)
The agent must use the trace summary path, not guess a repo-relative path. This matters for npm-installed users where `clis/` may not be in the working directory.
Never modify:
- `src/**` — core runtime
- `extension/**` — browser extension
- `autoresearch/**` — research infrastructure
- `tests/**` — test files
@@ -62,8 +65,6 @@ The agent must use the path from the diagnostic, not guess a repo-relative path.
### When NOT to Self-Repair
The agent should recognize non-repairable failures and stop:
| Signal | Meaning | Action |
|--------|---------|--------|
| Auth/login error | Not logged into site in Chrome | Tell user to log in, don't modify code |
@@ -74,63 +75,59 @@ The agent should recognize non-repairable failures and stop:
### Retry Budget
- **Max 3 repair rounds per command failure**
- Each round: diagnose → edit adapter retry command
- If the error is identical after a repair attempt, the fix didn't work — try a different approach
- After 3 rounds, stop and report what was tried
- Max 3 repair rounds per command failure.
- Each round: trace -> edit adapter -> retry command.
- If the error is identical after a repair attempt, the fix didn't work. Try a different approach.
- After 3 rounds, stop and report what was tried.
---
## Implementation
### What Already Exists
| Component | Status | Location |
|-----------|--------|----------|
| Diagnostic output (RepairContext) | ✅ Done | `src/diagnostic.ts` |
| Diagnostic wiring in execution | Done | `src/execution.ts` |
| Error taxonomy (CliError codes) | Done | `src/errors.ts` |
| Adapter source resolution | ✅ Done | `src/diagnostic.ts:resolveAdapterSourcePath` |
| Trace artifact output | Done | `src/observation/` |
| Error envelope trace metadata | Done | `src/errors.ts`, `src/execution.ts` |
| Adapter source resolution | Done | `src/adapter-source.ts` |
| AutoFix skill protocol | Done | `skills/opencli-autofix/SKILL.md` |
### What's New (This Design)
### Delivery Mechanism
| Component | Description |
|-----------|-------------|
| `skills/opencli-autofix/SKILL.md` (renamed from `opencli-repair`) | AutoFix skill with safety boundaries, sourcePath-based scope, 3-round limit. The primary delivery mechanism for the self-repair protocol. |
| `skills/opencli-usage/SKILL.md` (updated) | Self-Repair section for discoverability |
The `opencli-autofix` skill is the portable self-repair protocol. Any AI agent can load this skill to get the workflow.
### Delivery mechanism
No separate diagnostic env var is required. The runtime has two control axes:
The `opencli-autofix` skill is the portable self-repair protocol. Any AI agent — regardless of framework, provider, or working directory — can load this skill to get the full autofix workflow. It is not tied to any specific agent framework or repo location.
- **No new runtime code** — the diagnostic infrastructure already exists
- **No CLAUDE.md dependency** — the skill is the protocol, not a repo-local file
```text
-v / OPENCLI_VERBOSE human-readable logs
--trace off|on|retain-on-failure machine-readable browser evidence artifact
```
---
## The AutoFix Protocol (in the skill)
## The AutoFix Protocol
The `opencli-autofix` skill instructs agents:
1. When `opencli <site> <command>` fails, **don't just report the error**
2. Re-run with `OPENCLI_DIAGNOSTIC=1` to get structured context
3. Parse the RepairContext (error code, adapter source, DOM snapshot)
4. Read and fix the adapter at `RepairContext.adapter.sourcePath`
5. Retry the original command
6. If the retry passes, ask whether to file an upstream GitHub issue for `jackwener/OpenCLI`
7. If approved and `gh` is available, file the issue using a structured summary
8. Max 3 repair rounds, then stop
1. When `opencli <site> <command>` fails, don't just report the error.
2. Re-run with `--trace retain-on-failure`.
3. Read the error envelope `trace.summaryPath`.
4. Parse `summary.md` front matter for `adapterSourcePath`.
5. Read and fix the adapter at that exact path.
6. Retry the original command.
7. If the retry passes, ask whether to file an upstream GitHub issue for `jackwener/OpenCLI`.
8. If approved and `gh` is available, file the issue using a structured summary.
9. Max 3 repair rounds, then stop.
---
## Relationship to PR #863
PR #863 (spec/runner/incident framework) is **not needed for Phase 1**. It becomes useful later as a "hardening layer":
PR #863 (spec/runner/incident framework) is not needed for Phase 1. It becomes useful later as a hardening layer:
- **Phase 1 (now)**: Self-Repair via `opencli-autofix` skill — agent repairs on the fly
- **Phase 2 (later)**: High-frequency failures get hardened into `command-specs.json` for offline regression testing and CI
- Phase 1: self-repair via `opencli-autofix` skill and trace artifacts.
- Phase 2: high-frequency failures get hardened into command specs for offline regression testing and CI.
The spec/runner framework is the "asset layer" — it turns ad-hoc repairs into reusable, verifiable test cases. But it's not the entry point.
The spec/runner framework is the asset layer. It turns ad-hoc repairs into reusable tests, but it is not the entry point.
---
@@ -143,11 +140,12 @@ No new commands. No new scripts. The agent loads the `opencli-autofix` skill and
opencli weibo hot --limit 5 -f json
# If it fails, the agent automatically:
# 1. Runs OPENCLI_DIAGNOSTIC=1 opencli weibo hot --limit 5 -f json 2>diag.json
# 2. Reads the diagnostic context
# 3. Fixes the adapter at RepairContext.adapter.sourcePath
# 4. Retries: opencli weibo hot --limit 5 -f json
# 5. If retry passes, asks whether to file an upstream issue
# 6. If approved, runs `gh issue create --repo jackwener/OpenCLI ...`
# 7. Continues with the task
# 1. Runs opencli weibo hot --limit 5 -f json --trace retain-on-failure 2>trace-error.yaml
# 2. Reads trace.summaryPath from trace-error.yaml
# 3. Reads adapterSourcePath from summary.md
# 4. Fixes the adapter at adapterSourcePath
# 5. Retries: opencli weibo hot --limit 5 -f json
# 6. If retry passes, asks whether to file an upstream issue
# 7. If approved, runs `gh issue create --repo jackwener/OpenCLI ...`
# 8. Continues with the task
```
+1 -1
View File
@@ -48,7 +48,7 @@ opencli CLI
- `src/runtime.ts` — shared command runtime and target resolution
- `src/daemon.ts` — lifecycle and bridge behavior for the local daemon
- `src/doctor.ts` — browser bridge diagnostics
- `src/diagnostic.ts`structured failure context
- `src/observation/` — trace artifacts, redaction, and structured runtime evidence
- `src/interceptor.ts` — interception helpers for browser-backed strategies
- `src/browser/` — Browser Bridge connection and browser-side primitives
+3 -3
View File
@@ -10,7 +10,7 @@ allowed-tools: Bash(opencli:*), Read, Edit, Write, Grep
全程用现有工具:`opencli browser *` / `opencli doctor` / `opencli browser init` / `opencli browser verify`。没有新命令。
调试浏览器型 adapter 时,优先直接带上 `--live --focus`这样命令跑完后 automation lease 还在,而且容器在前台,方便核对最终页面状态,而不是猜是抓数错了还是页面走偏了
调试浏览器型 adapter 时,优先直接带上 `--trace on --live --focus``--trace on` 每轮都落 trace artifact`summary.md` 是失败/成功复盘入口;`--live --focus` automation lease 保留且容器在前台,方便核对最终页面状态。
---
@@ -82,7 +82,7 @@ START
┌──────────────────────────┐
│ opencli browser verify │── 失败 ──→ autofix skill,回对应步骤
│ opencli browser verify │── 失败 ──→ autofix skill用 --trace retain-on-failure 回对应步骤
└──────────────────────────┘
│ 成功
@@ -163,7 +163,7 @@ DONE
| | 200 但 `data: []` 空 | 参数传错 / 接口换版,回 §1 看 network 里真实请求头 |
| Step 7 字段解码 | 排序键对比推不出 | field-decode-playbook.md §3 结构差分 |
| | 还推不出 | 先输出 rawadapter 跑起来再迭代 |
| Step 10 verify 失败 | `fltt` 漏了 / 字段映射错 | autofix skill |
| Step 10 verify 失败 | `fltt` 漏了 / 字段映射错 | autofix skill;复现命令加 `--trace retain-on-failure` |
| | 某列永远是 `null` | 字段路径错了,回 Step 7 |
| Step 10 verify fixture mismatch | `[pattern]` row[i] 报错 | 先肉眼比对网页值;值对 → 是 fixture pattern 太严,放宽;值不对 → 字段映射错 |
| | `[column] missing column "X"` | 实际 response 没这列(站点改版 or args 影响);重新 `--update-fixture` 或修 adapter |
+61 -47
View File
@@ -1,6 +1,6 @@
---
name: opencli-autofix
description: Automatically fix broken OpenCLI adapters when commands fail. Load this skill when an opencli command fails — it guides you through diagnosing the failure via OPENCLI_DIAGNOSTIC, patching the adapter, retrying, and filing an upstream GitHub issue after a verified fix. Works with any AI agent.
description: Automatically fix broken OpenCLI adapters when commands fail. Load this skill when an opencli command fails — it guides you through collecting a trace artifact, patching the adapter, retrying, and filing an upstream GitHub issue after a verified fix. Works with any AI agent.
allowed-tools: Bash(opencli:*), Bash(gh:*), Read, Edit, Write
---
@@ -17,7 +17,7 @@ When an `opencli` command fails because a website changed its DOM, API, or respo
- **CAPTCHA / rate limiting** — **STOP.** Not an adapter issue.
**Scope constraint:**
- **Only modify the file at `RepairContext.adapter.sourcePath`** — this is the authoritative adapter location (may be `clis/<site>/` in repo or `~/.opencli/clis/<site>/` for npm installs)
- **Only modify the file at `adapterSourcePath` in the trace `summary.md` front matter** — this is the authoritative adapter location (may be `clis/<site>/` in repo or `~/.opencli/clis/<site>/` for npm installs)
- **Never modify** `src/`, `extension/`, `tests/`, `package.json`, or `tsconfig.json`
**Retry budget:** Max **3 repair rounds** per failure. If 3 rounds of diagnose → fix → retry don't resolve it, stop and report what was tried.
@@ -49,48 +49,65 @@ Use when `opencli <site> <command>` fails with repairable errors:
Only proceed to Step 1 if the empty/selector-missing result is **reproducible across retries and alternative entry points**. Otherwise you're patching a working adapter to chase noise, and the patched version will break the next working path.
## Step 1: Collect Diagnostic Context
## Step 1: Collect Trace Context
Run the failing command with diagnostic mode enabled:
Run the failing command with failure-retained trace enabled:
```bash
OPENCLI_DIAGNOSTIC=1 opencli <site> <command> [args...] 2>diagnostic.json
opencli <site> <command> [args...] --trace retain-on-failure 2>trace-error.yaml
```
This outputs a `RepairContext` JSON between `___OPENCLI_DIAGNOSTIC___` markers in stderr:
On failure, stderr contains the normal error envelope plus a small `trace` block:
```json
{
"error": {
"code": "SELECTOR",
"message": "Could not find element: .old-selector",
"hint": "The page UI may have changed."
},
"adapter": {
"site": "example",
"command": "example/search",
"sourcePath": "/path/to/clis/example/search.js",
"source": "// full adapter source code"
},
"page": {
"url": "https://example.com/search",
"snapshot": "// DOM snapshot with [N] indices",
"networkRequests": [],
"consoleErrors": []
},
"timestamp": "2025-01-01T00:00:00.000Z"
}
```yaml
ok: false
error:
code: SELECTOR
message: "Could not find element: .old-selector"
trace:
schemaVersion: 1
opencliVersion: "..."
traceId: "..."
dir: "/path/to/.opencli/profiles/default/traces/..."
summaryPath: "/path/to/.opencli/profiles/default/traces/.../summary.md"
receiptPath: "/path/to/.opencli/profiles/default/traces/.../receipt.json"
```
**Parse it:**
```bash
# Extract JSON between markers from stderr output
cat diagnostic.json | sed -n '/___OPENCLI_DIAGNOSTIC___/{n;p;}'
Read `summaryPath` first. It is the LLM-oriented entry point and includes front matter:
```yaml
---
schemaVersion: 1
opencliVersion: "..."
traceId: "..."
status: failure
site: "example"
command: "example/search"
adapterSourcePath: "/path/to/clis/example/search.js"
errorCode: "SELECTOR"
errorMessage: "Could not find element: .old-selector"
---
```
The artifact directory contains:
```text
summary.md # start here
receipt.json # machine-readable trace receipt
trace.jsonl # full redacted timeline
network.jsonl # redacted network events
console.jsonl # redacted console events
state/ # final snapshots when available
screenshots/ # final screenshots when available
```
If you redirected stderr to a file, read that file and copy `trace.summaryPath`.
Do not ask the user to rerun with legacy diagnostic env vars. Trace is the repair evidence path.
## Step 2: Analyze the Failure
Read the diagnostic context and the adapter source. Classify the root cause:
Read the trace summary and the adapter source. Classify the root cause:
| Error Code | Likely Cause | Repair Strategy |
|-----------|-------------|-----------------|
@@ -102,9 +119,9 @@ Read the diagnostic context and the adapter source. Classify the root cause:
| PAGE_CHANGED | Major redesign | May need full adapter rewrite |
**Key questions to answer:**
1. What is the adapter trying to do? (Read the `source` field)
2. What did the page look like when it failed? (Read the `snapshot` field)
3. What network requests happened? (Read `networkRequests`)
1. What is the adapter trying to do? (Read the file at `adapterSourcePath`)
2. What did the page look like when it failed? (Read `summary.md`, then `state/` if needed)
3. What network requests happened? (Read `Failed Network` in `summary.md`, then `network.jsonl` if needed)
4. What's the gap between what the adapter expects and what the page provides?
## Step 3: Explore the Current Website
@@ -139,12 +156,9 @@ opencli browser network --detail <key>
## Step 4: Patch the Adapter
Read the adapter source file at the path from `RepairContext.adapter.sourcePath` and make targeted fixes. This path is authoritative — it may be in the repo (`clis/`) or user-local (`~/.opencli/clis/`).
Read the adapter source file at `adapterSourcePath` from the trace summary front matter and make targeted fixes. This path is authoritative — it may be in the repo (`clis/`) or user-local (`~/.opencli/clis/`).
```bash
# Read the adapter (use the exact path from diagnostic)
cat <RepairContext.adapter.sourcePath>
```
Use the `Read` tool on the exact path from summary.md front matter.
### Common Fixes
@@ -184,11 +198,11 @@ cat <RepairContext.adapter.sourcePath>
## Step 5: Verify the Fix
```bash
# Run the command normally (without diagnostic mode)
# Run the command normally
opencli <site> <command> [args...]
```
If it still fails, go back to Step 1 and collect fresh diagnostics. You have a budget of **3 repair rounds** (diagnose → fix → retry). If the same error persists after a fix, try a different approach. After 3 rounds, stop and report what was tried.
If it still fails, go back to Step 1 and collect a fresh trace. You have a budget of **3 repair rounds** (trace → fix → retry). If the same error persists after a fix, try a different approach. After 3 rounds, stop and report what was tried.
## Step 6: File an Upstream Issue
@@ -203,7 +217,7 @@ If the retry **passes**, the local adapter has drifted from upstream. File a Git
**Procedure:**
1. Prepare the issue content from the RepairContext you already have:
1. Prepare the issue content from the trace summary you already have:
- **Title:** `[autofix] <site>/<command>: <error_code>` (e.g. `[autofix] zhihu/hot: SELECTOR`)
- **Body** (use this template):
@@ -264,15 +278,15 @@ In all stop cases, clearly communicate the situation to the user rather than mak
1. User runs: opencli zhihu hot
→ Fails: SELECTOR "Could not find element: .HotList-item"
2. AI runs: OPENCLI_DIAGNOSTIC=1 opencli zhihu hot 2>diag.json
→ Gets RepairContext with DOM snapshot showing page loaded
2. AI runs: opencli zhihu hot --trace retain-on-failure 2>trace-error.yaml
→ Gets trace summary with final state and failed action evidence
3. AI reads diagnostic: snapshot shows the page loaded but uses ".HotItem" instead of ".HotList-item"
3. AI reads summary/state: page loaded but uses ".HotItem" instead of ".HotList-item"
4. AI explores: opencli browser open https://www.zhihu.com/hot && opencli browser state
→ Confirms new class name ".HotItem" with child ".HotItem-content"
5. AI patches: Edit adapter at RepairContext.adapter.sourcePath — replace ".HotList-item" with ".HotItem"
5. AI patches: Edit adapter at `adapterSourcePath` — replace ".HotList-item" with ".HotItem"
6. AI verifies: opencli zhihu hot
→ Success: returns hot topics
+1 -1
View File
@@ -380,4 +380,4 @@ opencli browser eval "(() => document.querySelector('input[name=cardnumber]')?.v
## See also
- `opencli-adapter-author` — turning what you just figured out into a reusable `~/.opencli/clis/<site>/<command>.js`.
- `opencli-autofix` — when an existing adapter breaks, this skill walks you through `OPENCLI_DIAGNOSTIC` and filing a fix.
- `opencli-autofix` — when an existing adapter breaks, this skill walks you through `--trace retain-on-failure` evidence and filing a fix.
+2 -3
View File
@@ -85,11 +85,10 @@ A few commands override the default via `cmd.defaultFormat` (e.g. chat commands
| `OPENCLI_CACHE_DIR` | `~/.opencli/cache` | Network capture + browser-state cache. |
| `OPENCLI_WINDOW_FOCUSED` | `false` | `1` → automation window opens in the foreground. |
| `OPENCLI_VERBOSE` | `false` | Verbose logging (also triggered by `-v`). |
| `OPENCLI_DIAGNOSTIC` | `false` | `1` → emit structured `RepairContext` JSON on adapter failure. Required for `opencli-autofix`. |
## Self-repair
When an adapter command fails because the site changed (selectors drifted, API rotated, response schema shifted), the CLI emits a hint: `# AutoFix: re-run with OPENCLI_DIAGNOSTIC=1 ...`. Do that, read the `RepairContext`, patch the adapter at `RepairContext.adapter.sourcePath`, and retry. Max 3 repair rounds. The full flow is in `opencli-autofix`.
When an adapter command fails because the site changed (selectors drifted, API rotated, response schema shifted), re-run with `--trace retain-on-failure`. The error envelope includes a `trace` block pointing at `summary.md`; patch only the `adapterSourcePath` from that summary and retry. Max 3 repair rounds. The full flow is in `opencli-autofix`.
## Writing your own adapter
@@ -166,4 +165,4 @@ The following were removed in the PR #1094 consolidation — don't try to invoke
- Don't paste this skill's command list into your plan; it will rot. Call `opencli list -f json` at the start of a task instead.
- Don't assume every adapter needs a browser — strategy `PUBLIC` and `LOCAL` don't. Check the `strategy` field.
- Don't silently fall back from a failing adapter to a hand-rolled `fetch``OPENCLI_DIAGNOSTIC=1` almost always tells you exactly what to change in the adapter. Do that first.
- Don't silently fall back from a failing adapter to a hand-rolled `fetch``--trace retain-on-failure` gives you the browser evidence and adapter source path. Do that first.
+35
View File
@@ -0,0 +1,35 @@
import { describe, expect, it } from 'vitest';
import type { InternalCliCommand } from './registry.js';
import { resolveAdapterSourcePath } from './adapter-source.js';
function makeCmd(overrides: Partial<InternalCliCommand> = {}): InternalCliCommand {
return {
site: 'test-site',
name: 'test-cmd',
description: 'test',
args: [],
...overrides,
} as InternalCliCommand;
}
describe('resolveAdapterSourcePath', () => {
it('returns source when it is a real file path (not manifest:)', () => {
const cmd = makeCmd({ source: '/home/user/.opencli/clis/arxiv/search.js' });
expect(resolveAdapterSourcePath(cmd)).toBe('/home/user/.opencli/clis/arxiv/search.js');
});
it('skips manifest: pseudo-paths and falls back to _modulePath', () => {
const cmd = makeCmd({ source: 'manifest:arxiv/search', _modulePath: '/pkg/clis/arxiv/search.js' });
expect(resolveAdapterSourcePath(cmd)).toBe('/pkg/clis/arxiv/search.js');
});
it('returns undefined when only manifest: pseudo-path and no _modulePath', () => {
const cmd = makeCmd({ source: 'manifest:test/cmd' });
expect(resolveAdapterSourcePath(cmd)).toBeUndefined();
});
it('returns _modulePath when it is the only path available', () => {
const cmd = makeCmd({ _modulePath: '/project/clis/site/cmd.js' });
expect(resolveAdapterSourcePath(cmd)).toBe('/project/clis/site/cmd.js');
});
});
+28
View File
@@ -0,0 +1,28 @@
import * as fs from 'node:fs';
import type { InternalCliCommand } from './registry.js';
/**
* Resolve the editable source file path for an adapter.
*
* Priority:
* 1. cmd.source (set for FS-scanned JS and manifest lazy-loaded JS)
* 2. cmd._modulePath (set for manifest lazy-loaded JS)
*
* Skip manifest: prefixed pseudo-paths (YAML commands inlined in manifest).
*/
export function resolveAdapterSourcePath(cmd: InternalCliCommand): string | undefined {
const candidates: string[] = [];
if (cmd.source && !cmd.source.startsWith('manifest:')) {
candidates.push(cmd.source);
}
if (cmd._modulePath) {
candidates.push(cmd._modulePath);
}
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return candidates[0];
}
+69 -1
View File
@@ -1,7 +1,7 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { Command } from 'commander';
import type { CliCommand } from './registry.js';
import { EmptyResultError, selectorError } from './errors.js';
import { attachTraceReceipt, EmptyResultError, selectorError } from './errors.js';
const { mockExecuteCommand, mockRenderOutput } = vi.hoisted(() => ({
mockExecuteCommand: vi.fn(),
@@ -360,6 +360,9 @@ describe('commanderAdapter error envelope output', () => {
expect(output).toContain('ok: false');
expect(output).toContain('code: EMPTY_RESULT');
expect(output).toContain('xsec_token');
expect(output).toContain('--trace=retain-on-failure');
expect(output).toContain('opencli xiaohongshu note --trace retain-on-failure');
expect(output).not.toContain('OPENCLI_DIAGNOSTIC');
stderrSpy.mockRestore();
});
@@ -380,6 +383,71 @@ describe('commanderAdapter error envelope output', () => {
expect(output).toContain('ok: false');
expect(output).toContain('code: SELECTOR');
expect(output).toContain('selector no longer matches');
expect(output).toContain('--trace=retain-on-failure');
stderrSpy.mockRestore();
});
it('does not add an AutoFix rerun hint when trace is already enabled', async () => {
const program = new Command();
const siteCmd = program.command('xiaohongshu');
registerCommandToProgram(siteCmd, cmd);
const stderrSpy = vi.spyOn(process.stderr, 'write').mockImplementation(() => true);
mockExecuteCommand.mockRejectedValueOnce(selectorError('.note-title'));
await program.parseAsync([
'node',
'opencli',
'xiaohongshu',
'note',
'69ca3927000000001a020fd5',
'--trace',
'retain-on-failure',
]);
const output = stderrSpy.mock.calls.map(c => String(c[0])).join('');
expect(output).toContain('code: SELECTOR');
expect(output).not.toContain('AutoFix: re-run');
stderrSpy.mockRestore();
});
it('includes trace metadata from the error envelope when execution attached it', async () => {
const program = new Command();
const siteCmd = program.command('xiaohongshu');
registerCommandToProgram(siteCmd, cmd);
const stderrSpy = vi.spyOn(process.stderr, 'write').mockImplementation(() => true);
const err = selectorError('.note-title');
attachTraceReceipt(err, {
schemaVersion: 1,
opencliVersion: '1.7.8',
traceId: 'trace-1',
traceDir: '/tmp/opencli/profiles/default/traces/trace-1',
summaryPath: '/tmp/opencli/profiles/default/traces/trace-1/summary.md',
receiptPath: '/tmp/opencli/profiles/default/traces/trace-1/receipt.json',
status: 'failure',
createdAt: '2026-05-03T00:00:00.000Z',
error: { code: 'SELECTOR', message: 'Could not find element: .note-title' },
});
mockExecuteCommand.mockRejectedValueOnce(err);
await program.parseAsync([
'node',
'opencli',
'xiaohongshu',
'note',
'69ca3927000000001a020fd5',
'--trace',
'retain-on-failure',
]);
const output = stderrSpy.mock.calls.map(c => String(c[0])).join('');
expect(output).toContain('trace:');
expect(output).toContain('dir: /tmp/opencli/profiles/default/traces/trace-1');
expect(output).toContain('summaryPath: /tmp/opencli/profiles/default/traces/trace-1/summary.md');
expect(output).toContain('receiptPath: /tmp/opencli/profiles/default/traces/trace-1/receipt.json');
stderrSpy.mockRestore();
});
+10 -8
View File
@@ -22,7 +22,6 @@ import {
EXIT_CODES,
toEnvelope,
} from './errors.js';
import { isDiagnosticEnabled } from './diagnostic.js';
/**
* Register a single CliCommand as a Commander subcommand.
@@ -125,7 +124,7 @@ export function registerCommandToProgram(siteCmd: Command, cmd: CliCommand): voi
footerExtra: resolved.footerExtra?.(kwargs),
});
} catch (err) {
renderError(err, fullName(cmd), optionsRecord.verbose === true);
renderError(err, fullName(cmd), optionsRecord.verbose === true, optionsRecord.trace);
process.exitCode = resolveExitCode(err);
}
});
@@ -140,13 +139,16 @@ function resolveExitCode(err: unknown): number {
// ── Error rendering ─────────────────────────────────────────────────────────
/** Emit AutoFix hint for repairable adapter errors (skipped if already in diagnostic mode). */
function emitAutoFixHint(envelope: string, cmdName: string): string {
if (isDiagnosticEnabled()) return envelope;
return envelope + `# AutoFix: re-run with OPENCLI_DIAGNOSTIC=1 for repair context\n# OPENCLI_DIAGNOSTIC=1 ${cmdName}\n`;
/** Emit AutoFix hint for repairable adapter errors (skipped if trace already exported). */
function emitAutoFixHint(envelope: string, cmdName: string, traceMode: unknown): string {
if (traceMode === 'on' || traceMode === 'retain-on-failure') return envelope;
const runnable = cmdName.replace('/', ' ');
return envelope
+ `# AutoFix: re-run with --trace=retain-on-failure for trace artifact\n`
+ `# opencli ${runnable} --trace retain-on-failure\n`;
}
function renderError(err: unknown, cmdName: string, verbose: boolean): void {
function renderError(err: unknown, cmdName: string, verbose: boolean, traceMode?: unknown): void {
const envelope = toEnvelope(err);
// In verbose mode, include stack trace for debugging
@@ -159,7 +161,7 @@ function renderError(err: unknown, cmdName: string, verbose: boolean): void {
// Append AutoFix hint for repairable errors
const code = envelope.error.code;
if (code === 'SELECTOR' || code === 'EMPTY_RESULT' || code === 'ADAPTER_LOAD' || code === 'UNKNOWN') {
output = emitAutoFixHint(output, cmdName);
output = emitAutoFixHint(output, cmdName, traceMode);
}
process.stderr.write(output);
-371
View File
@@ -1,371 +0,0 @@
import { describe, it, expect, vi, afterEach } from 'vitest';
import {
buildRepairContext, collectDiagnostic, isDiagnosticEnabled, emitDiagnostic,
truncate, redactUrl, redactText, resolveAdapterSourcePath, MAX_DIAGNOSTIC_BYTES,
type RepairContext,
} from './diagnostic.js';
import { selectorError, CommandExecutionError } from './errors.js';
import type { InternalCliCommand } from './registry.js';
import type { IPage } from './types.js';
function makeCmd(overrides: Partial<InternalCliCommand> = {}): InternalCliCommand {
return {
site: 'test-site',
name: 'test-cmd',
description: 'test',
args: [],
...overrides,
} as InternalCliCommand;
}
describe('isDiagnosticEnabled', () => {
const origEnv = process.env.OPENCLI_DIAGNOSTIC;
afterEach(() => {
if (origEnv === undefined) delete process.env.OPENCLI_DIAGNOSTIC;
else process.env.OPENCLI_DIAGNOSTIC = origEnv;
});
it('returns false when env not set', () => {
delete process.env.OPENCLI_DIAGNOSTIC;
expect(isDiagnosticEnabled()).toBe(false);
});
it('returns true when env is "1"', () => {
process.env.OPENCLI_DIAGNOSTIC = '1';
expect(isDiagnosticEnabled()).toBe(true);
});
it('returns false for other values', () => {
process.env.OPENCLI_DIAGNOSTIC = 'true';
expect(isDiagnosticEnabled()).toBe(false);
});
});
describe('truncate', () => {
it('returns short strings unchanged', () => {
expect(truncate('hello', 100)).toBe('hello');
});
it('truncates long strings with marker', () => {
const long = 'a'.repeat(200);
const result = truncate(long, 50);
expect(result.length).toBeLessThan(200);
expect(result).toContain('...[truncated,');
expect(result).toContain('150 chars omitted]');
});
});
describe('redactUrl', () => {
it('redacts sensitive query parameters', () => {
expect(redactUrl('https://api.com/v1?token=abc123&q=test'))
.toBe('https://api.com/v1?token=[REDACTED]&q=test');
});
it('redacts multiple sensitive params', () => {
const url = 'https://api.com?api_key=xxx&secret=yyy&page=1';
const result = redactUrl(url);
expect(result).toContain('api_key=[REDACTED]');
expect(result).toContain('secret=[REDACTED]');
expect(result).toContain('page=1');
});
it('leaves clean URLs unchanged', () => {
expect(redactUrl('https://example.com/page?q=test')).toBe('https://example.com/page?q=test');
});
});
describe('redactText', () => {
it('redacts Bearer tokens', () => {
expect(redactText('Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.test'))
.toContain('Bearer [REDACTED]');
});
it('redacts JWT tokens', () => {
const jwt = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dozjgNryP4J3jVmNHl0w5N_XgL0n3I9PlFUP0THsR8U';
expect(redactText(`token is ${jwt}`)).toContain('[REDACTED_JWT]');
expect(redactText(`token is ${jwt}`)).not.toContain('eyJhbGci');
});
it('redacts inline token=value patterns', () => {
expect(redactText('failed with token=abc123def456')).toContain('token=[REDACTED]');
});
it('redacts cookie values', () => {
const result = redactText('cookie: session=abc123; user=xyz789; path=/');
expect(result).toContain('[REDACTED]');
expect(result).not.toContain('session=abc123');
});
it('leaves normal text unchanged', () => {
expect(redactText('Error: element not found')).toBe('Error: element not found');
});
});
describe('resolveAdapterSourcePath', () => {
it('returns source when it is a real file path (not manifest:)', () => {
const cmd = makeCmd({ source: '/home/user/.opencli/clis/arxiv/search.js' });
expect(resolveAdapterSourcePath(cmd as InternalCliCommand)).toBe('/home/user/.opencli/clis/arxiv/search.js');
});
it('skips manifest: pseudo-paths and falls back to _modulePath', () => {
const cmd = makeCmd({ source: 'manifest:arxiv/search', _modulePath: '/pkg/clis/arxiv/search.js' });
// Should try to map to source, but since files don't exist on disk, returns _modulePath
const result = resolveAdapterSourcePath(cmd as InternalCliCommand);
expect(result).toBeDefined();
expect(result).not.toContain('manifest:');
});
it('returns undefined when only manifest: pseudo-path and no _modulePath', () => {
const cmd = makeCmd({ source: 'manifest:test/cmd' });
expect(resolveAdapterSourcePath(cmd as InternalCliCommand)).toBeUndefined();
});
it('returns _modulePath when it is the only path available', () => {
const cmd = makeCmd({ _modulePath: '/project/clis/site/cmd.js' });
const result = resolveAdapterSourcePath(cmd as InternalCliCommand);
// Since file doesn't exist, returns _modulePath as best guess
expect(result).toBe('/project/clis/site/cmd.js');
});
});
describe('buildRepairContext', () => {
it('captures CliError fields', () => {
const err = selectorError('.missing-element', 'Element removed');
const ctx = buildRepairContext(err, makeCmd());
expect(ctx.error.code).toBe('SELECTOR');
expect(ctx.error.message).toContain('.missing-element');
expect(ctx.error.hint).toBe('Element removed');
expect(ctx.error.stack).toBeDefined();
expect(ctx.adapter.site).toBe('test-site');
expect(ctx.adapter.command).toBe('test-site/test-cmd');
expect(ctx.timestamp).toMatch(/^\d{4}-\d{2}-\d{2}T/);
});
it('handles non-CliError errors', () => {
const err = new TypeError('Cannot read property "x" of undefined');
const ctx = buildRepairContext(err, makeCmd());
expect(ctx.error.code).toBe('UNKNOWN');
expect(ctx.error.message).toContain('Cannot read property');
expect(ctx.error.hint).toBeUndefined();
});
it('includes page state when provided', () => {
const pageState: RepairContext['page'] = {
url: 'https://example.com/page',
snapshot: '<div>...</div>',
networkRequests: [{ url: '/api/data', status: 200 }],
consoleErrors: ['Uncaught TypeError'],
};
const ctx = buildRepairContext(new CommandExecutionError('boom'), makeCmd(), pageState);
expect(ctx.page).toEqual(pageState);
});
it('includes trace artifact metadata when provided', () => {
const ctx = buildRepairContext(new CommandExecutionError('boom'), makeCmd(), undefined, {
traceId: 'trace-1',
dir: '/tmp/opencli/profiles/default/traces/trace-1',
summaryPath: '/tmp/opencli/profiles/default/traces/trace-1/summary.md',
});
expect(ctx.trace).toEqual({
traceId: 'trace-1',
dir: '/tmp/opencli/profiles/default/traces/trace-1',
summaryPath: '/tmp/opencli/profiles/default/traces/trace-1/summary.md',
});
});
it('omits page when not provided', () => {
const ctx = buildRepairContext(new Error('boom'), makeCmd());
expect(ctx.page).toBeUndefined();
});
it('truncates long stack traces', () => {
const err = new Error('boom');
err.stack = 'x'.repeat(60_000);
const ctx = buildRepairContext(err, makeCmd());
expect(ctx.error.stack!.length).toBeLessThan(60_000);
expect(ctx.error.stack).toContain('truncated');
});
it('redacts sensitive data in error message and stack', () => {
const err = new Error('Request failed with Bearer eyJhbGciOiJIUzI1NiJ9.test.sig');
const ctx = buildRepairContext(err, makeCmd());
expect(ctx.error.message).toContain('Bearer [REDACTED]');
expect(ctx.error.message).not.toContain('eyJhbGci');
// Stack also gets redacted
expect(ctx.error.stack).toContain('Bearer [REDACTED]');
});
});
describe('emitDiagnostic', () => {
it('writes delimited JSON to stderr', () => {
const writeSpy = vi.spyOn(process.stderr, 'write').mockReturnValue(true);
const ctx = buildRepairContext(new CommandExecutionError('test error'), makeCmd());
emitDiagnostic(ctx);
const output = writeSpy.mock.calls.map(c => c[0]).join('');
expect(output).toContain('___OPENCLI_DIAGNOSTIC___');
expect(output).toContain('"code":"COMMAND_EXEC"');
expect(output).toContain('"message":"test error"');
// Verify JSON is parseable between markers
const match = output.match(/___OPENCLI_DIAGNOSTIC___\n(.*)\n___OPENCLI_DIAGNOSTIC___/);
expect(match).toBeTruthy();
const parsed = JSON.parse(match![1]);
expect(parsed.error.code).toBe('COMMAND_EXEC');
writeSpy.mockRestore();
});
it('drops page snapshot when over size budget', () => {
const writeSpy = vi.spyOn(process.stderr, 'write').mockReturnValue(true);
const ctx: RepairContext = {
error: { code: 'COMMAND_EXEC', message: 'boom' },
adapter: { site: 'test', command: 'test/cmd' },
page: {
url: 'https://example.com',
snapshot: 'x'.repeat(MAX_DIAGNOSTIC_BYTES + 1000),
networkRequests: [],
consoleErrors: [],
},
timestamp: new Date().toISOString(),
};
emitDiagnostic(ctx);
const output = writeSpy.mock.calls.map(c => c[0]).join('');
const match = output.match(/___OPENCLI_DIAGNOSTIC___\n(.*)\n___OPENCLI_DIAGNOSTIC___/);
expect(match).toBeTruthy();
const parsed = JSON.parse(match![1]);
// Page snapshot should be replaced or page dropped entirely
expect(parsed.page?.snapshot !== ctx.page!.snapshot || parsed.page === undefined).toBe(true);
expect(match![1].length).toBeLessThanOrEqual(MAX_DIAGNOSTIC_BYTES);
writeSpy.mockRestore();
});
it('redacts sensitive headers in network requests', () => {
const pageState: RepairContext['page'] = {
url: 'https://example.com',
snapshot: '<div/>',
networkRequests: [{
url: 'https://api.com/data?token=secret123',
headers: { authorization: 'Bearer xyz', 'content-type': 'application/json' },
body: '{"data": "ok"}',
}],
consoleErrors: [],
};
// Build context manually to test redaction via collectPageState
// Since collectPageState is private, test the output of buildRepairContext
// with already-collected page state — redaction happens in collectPageState.
// For unit test, verify redactUrl directly (tested above) and trust integration.
expect(redactUrl('https://api.com/data?token=secret123')).toContain('[REDACTED]');
});
});
function makePage(overrides: Partial<IPage> = {}): IPage {
return {
goto: vi.fn(),
evaluate: vi.fn(),
getCookies: vi.fn(),
snapshot: vi.fn().mockResolvedValue('<div>...</div>'),
click: vi.fn(),
typeText: vi.fn(),
pressKey: vi.fn(),
scrollTo: vi.fn(),
getFormState: vi.fn(),
wait: vi.fn(),
tabs: vi.fn(),
selectTab: vi.fn(),
networkRequests: vi.fn().mockResolvedValue([]),
consoleMessages: vi.fn().mockResolvedValue([]),
scroll: vi.fn(),
autoScroll: vi.fn(),
installInterceptor: vi.fn(),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
waitForCapture: vi.fn(),
screenshot: vi.fn(),
getCurrentUrl: vi.fn().mockResolvedValue('https://example.com/page'),
...overrides,
} as IPage;
}
describe('collectDiagnostic', () => {
it('keeps intercepted payloads in a dedicated capturedPayloads field', async () => {
const page = makePage({
networkRequests: vi.fn().mockResolvedValue([{ url: '/api/data', status: 200 }]),
getInterceptedRequests: vi.fn().mockResolvedValue([{ items: [{ id: 1 }] }]),
});
const ctx = await collectDiagnostic(new Error('boom'), makeCmd(), page);
expect(ctx.page?.networkRequests).toEqual([
{ url: '/api/data', status: 200 },
]);
expect(ctx.page?.capturedPayloads).toEqual([
{ source: 'interceptor', responseBody: { items: [{ id: 1 }] } },
]);
});
it('preserves the previous network request output when interception is empty', async () => {
const page = makePage({
networkRequests: vi.fn().mockResolvedValue([{ url: '/api/data', status: 200 }]),
getInterceptedRequests: vi.fn().mockResolvedValue([]),
});
const ctx = await collectDiagnostic(new Error('boom'), makeCmd(), page);
expect(ctx.page?.networkRequests).toEqual([{ url: '/api/data', status: 200 }]);
expect(ctx.page?.capturedPayloads).toEqual([]);
});
it('swallows intercepted request failures and still returns page state', async () => {
const page = makePage({
networkRequests: vi.fn().mockResolvedValue([{ url: '/api/data', status: 200 }]),
getInterceptedRequests: vi.fn().mockRejectedValue(new Error('interceptor unavailable')),
});
const ctx = await collectDiagnostic(new Error('boom'), makeCmd(), page);
expect(ctx.page).toEqual({
url: 'https://example.com/page',
snapshot: '<div>...</div>',
networkRequests: [{ url: '/api/data', status: 200 }],
capturedPayloads: [],
consoleErrors: [],
});
});
it('redacts and truncates intercepted payloads recursively', async () => {
const page = makePage({
getInterceptedRequests: vi.fn().mockResolvedValue([{
token: 'token=abc123def456ghi789',
nested: {
cookie: 'cookie: session=super-secret-cookie-value',
body: 'x'.repeat(60_000),
},
}]),
});
const ctx = await collectDiagnostic(new Error('boom'), makeCmd(), page);
const payload = ctx.page?.capturedPayloads?.[0] as Record<string, unknown>;
const body = ((payload.responseBody as Record<string, unknown>).nested as Record<string, unknown>).body as string;
expect(payload).toEqual({
source: 'interceptor',
responseBody: {
token: '[REDACTED]',
nested: {
cookie: '[REDACTED]',
body,
},
},
});
expect(body).toContain('[truncated,');
expect(body.length).toBeLessThan(60_000);
});
});
-309
View File
@@ -1,309 +0,0 @@
/**
* Structured diagnostic output for AI-driven adapter repair.
*
* When OPENCLI_DIAGNOSTIC=1, failed commands emit a JSON RepairContext to stderr
* containing the error, adapter source, and browser state (DOM snapshot, network
* requests, console errors). AI Agents consume this to diagnose and fix adapters.
*
* Safety boundaries:
* - Sensitive headers/cookies are redacted before emission
* - Individual fields are capped to prevent unbounded output
* - Network response bodies from authenticated requests are stripped
* - Total output is capped to MAX_DIAGNOSTIC_BYTES
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import type { IPage } from './types.js';
import { CliError, getErrorMessage } from './errors.js';
import type { InternalCliCommand } from './registry.js';
import { fullName } from './registry.js';
import type { ObservationExportResult } from './observation/index.js';
import {
redactHeaders as redactObservationHeaders,
redactText as redactObservationText,
redactUrl as redactObservationUrl,
redactValue as redactObservationValue,
} from './observation/redaction.js';
// ── Size budgets ─────────────────────────────────────────────────────────────
/** Maximum bytes for the entire diagnostic JSON output. */
export const MAX_DIAGNOSTIC_BYTES = 256 * 1024; // 256 KB
/** Maximum characters for any single diagnostic text field. */
const MAX_DIAGNOSTIC_FIELD_CHARS = 50_000;
/** Maximum entries to keep from diagnostic collections. */
const MAX_DIAGNOSTIC_COLLECTION_ITEMS = 50;
// ── Types ────────────────────────────────────────────────────────────────────
export interface RepairContext {
error: {
code: string;
message: string;
hint?: string;
stack?: string;
};
adapter: {
site: string;
command: string;
sourcePath?: string;
source?: string;
};
page?: {
url: string;
snapshot: string;
networkRequests: unknown[];
capturedPayloads?: unknown[];
consoleErrors: unknown[];
};
trace?: {
traceId: string;
dir: string;
summaryPath: string;
};
timestamp: string;
}
// ── Redaction helpers ────────────────────────────────────────────────────────
/** Truncate a string to maxLen, appending a truncation marker. */
export function truncate(str: string, maxLen: number): string {
if (str.length <= maxLen) return str;
return str.slice(0, maxLen) + `\n...[truncated, ${str.length - maxLen} chars omitted]`;
}
/** Redact sensitive query parameters from a URL. */
export function redactUrl(url: string): string {
return redactObservationUrl(url);
}
/** Redact inline secrets from free-text strings (error messages, stack traces, console output, DOM). */
export function redactText(text: string): string {
return redactObservationText(text, { maxStringLength: MAX_DIAGNOSTIC_FIELD_CHARS });
}
/** Redact sensitive headers from a headers object. */
function redactHeaders(headers: Record<string, string> | undefined): Record<string, string> | undefined {
if (!headers || typeof headers !== 'object') return headers;
return redactObservationHeaders(headers, {
maxStringLength: MAX_DIAGNOSTIC_FIELD_CHARS,
maxArrayItems: MAX_DIAGNOSTIC_COLLECTION_ITEMS,
maxObjectFields: MAX_DIAGNOSTIC_COLLECTION_ITEMS,
}) as Record<string, string>;
}
/** Recursively sanitize arbitrary captured response content for diagnostic output. */
function sanitizeCapturedValue(value: unknown): unknown {
return redactObservationValue(value, {
maxStringLength: MAX_DIAGNOSTIC_FIELD_CHARS,
maxArrayItems: MAX_DIAGNOSTIC_COLLECTION_ITEMS,
maxObjectFields: MAX_DIAGNOSTIC_COLLECTION_ITEMS,
maxDepth: 4,
});
}
/** Redact sensitive data from a single network request entry. */
function redactNetworkRequest(req: unknown): unknown {
if (!req || typeof req !== 'object') return req;
const r = req as Record<string, unknown>;
const redacted: Record<string, unknown> = { ...r };
// Redact URL
if (typeof redacted.url === 'string') {
redacted.url = redactUrl(redacted.url);
}
// Redact headers
if (redacted.headers && typeof redacted.headers === 'object') {
redacted.headers = redactHeaders(redacted.headers as Record<string, string>);
}
if (redacted.requestHeaders && typeof redacted.requestHeaders === 'object') {
redacted.requestHeaders = redactHeaders(redacted.requestHeaders as Record<string, string>);
}
if (redacted.responseHeaders && typeof redacted.responseHeaders === 'object') {
redacted.responseHeaders = redactHeaders(redacted.responseHeaders as Record<string, string>);
}
// Redact and truncate response body
if (typeof redacted.body === 'string') {
redacted.body = redactText(truncate(redacted.body, MAX_DIAGNOSTIC_FIELD_CHARS));
}
if ('responseBody' in redacted) {
redacted.responseBody = sanitizeCapturedValue(redacted.responseBody);
}
if ('responsePreview' in redacted) {
redacted.responsePreview = sanitizeCapturedValue(redacted.responsePreview);
}
return redacted;
}
// ── Timeout helper ───────────────────────────────────────────────────────────
/** Timeout for page state collection (prevents hang when CDP connection is stuck). */
const PAGE_STATE_TIMEOUT_MS = 5_000;
function withTimeout<T>(promise: Promise<T>, ms: number, fallback: T): Promise<T> {
return Promise.race([
promise,
new Promise<T>(resolve => setTimeout(() => resolve(fallback), ms)),
]);
}
// ── Source path resolution ───────────────────────────────────────────────────
/**
* Resolve the editable source file path for an adapter.
*
* Priority:
* 1. cmd.source (set for FS-scanned JS and manifest lazy-loaded JS)
* 2. cmd._modulePath (set for manifest lazy-loaded JS)
*
* Skip manifest: prefixed pseudo-paths (YAML commands inlined in manifest).
*/
export function resolveAdapterSourcePath(cmd: InternalCliCommand): string | undefined {
const candidates: string[] = [];
// cmd.source may be a real file path or 'manifest:site/name'
if (cmd.source && !cmd.source.startsWith('manifest:')) {
candidates.push(cmd.source);
}
if (cmd._modulePath) {
candidates.push(cmd._modulePath);
}
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return candidates[0]; // Return best guess even if file doesn't exist
}
// ── Diagnostic collection ────────────────────────────────────────────────────
/** Whether diagnostic mode is enabled. */
export function isDiagnosticEnabled(): boolean {
return process.env.OPENCLI_DIAGNOSTIC === '1';
}
function normalizeInterceptedRequests(interceptedRequests: unknown[]): unknown[] {
return interceptedRequests.slice(0, MAX_DIAGNOSTIC_COLLECTION_ITEMS).map(responseBody => ({
source: 'interceptor',
responseBody: sanitizeCapturedValue(responseBody),
}));
}
/** Safely collect page diagnostic state with redaction, size caps, and timeout. */
async function collectPageState(page: IPage): Promise<RepairContext['page'] | undefined> {
const collect = async (): Promise<RepairContext['page'] | undefined> => {
try {
const [url, snapshot, networkRequests, interceptedRequests, consoleErrors] = await Promise.all([
page.getCurrentUrl?.().catch(() => null) ?? Promise.resolve(null),
page.snapshot().catch(() => '(snapshot unavailable)'),
page.networkRequests().catch(() => []),
page.getInterceptedRequests().catch(() => []),
page.consoleMessages('error').catch(() => []),
]);
const rawUrl = url ?? 'unknown';
const capturedResponses = normalizeInterceptedRequests(interceptedRequests as unknown[]);
return {
url: redactUrl(rawUrl),
snapshot: redactText(truncate(snapshot, MAX_DIAGNOSTIC_FIELD_CHARS)),
networkRequests: (networkRequests as unknown[])
.slice(0, MAX_DIAGNOSTIC_COLLECTION_ITEMS)
.map(redactNetworkRequest),
capturedPayloads: capturedResponses,
consoleErrors: (consoleErrors as unknown[])
.slice(0, MAX_DIAGNOSTIC_COLLECTION_ITEMS)
.map(e => typeof e === 'string' ? redactText(e) : e),
};
} catch {
return undefined;
}
};
return withTimeout(collect(), PAGE_STATE_TIMEOUT_MS, undefined);
}
/** Read adapter source file content with size cap. */
function readAdapterSource(sourcePath: string | undefined): string | undefined {
if (!sourcePath) return undefined;
try {
const content = fs.readFileSync(sourcePath, 'utf-8');
return truncate(content, MAX_DIAGNOSTIC_FIELD_CHARS);
} catch {
return undefined;
}
}
/** Build a RepairContext from an error, command metadata, and optional page state. */
export function buildRepairContext(
err: unknown,
cmd: InternalCliCommand,
pageState?: RepairContext['page'],
trace?: ObservationExportResult,
): RepairContext {
const isCliError = err instanceof CliError;
const sourcePath = resolveAdapterSourcePath(cmd);
return {
error: {
code: isCliError ? err.code : 'UNKNOWN',
message: redactText(getErrorMessage(err)),
hint: isCliError && err.hint ? redactText(err.hint) : undefined,
stack: err instanceof Error ? redactText(truncate(err.stack ?? '', MAX_DIAGNOSTIC_FIELD_CHARS)) : undefined,
},
adapter: {
site: cmd.site,
command: fullName(cmd),
sourcePath,
source: readAdapterSource(sourcePath),
},
page: pageState,
trace: trace ? {
traceId: trace.traceId,
dir: trace.dir,
summaryPath: trace.summaryPath,
} : undefined,
timestamp: new Date().toISOString(),
};
}
/** Collect full diagnostic context including page state (with timeout). */
export async function collectDiagnostic(
err: unknown,
cmd: InternalCliCommand,
page: IPage | null,
trace?: ObservationExportResult,
): Promise<RepairContext> {
const pageState = page ? await collectPageState(page) : undefined;
return buildRepairContext(err, cmd, pageState, trace);
}
/** Emit diagnostic JSON to stderr, enforcing total size cap. */
export function emitDiagnostic(ctx: RepairContext): void {
const marker = '___OPENCLI_DIAGNOSTIC___';
let json = JSON.stringify(ctx);
// Enforce total output budget — drop page state (largest section) first if over budget
if (json.length > MAX_DIAGNOSTIC_BYTES && ctx.page) {
const trimmed = {
...ctx,
page: {
...ctx.page,
snapshot: '[omitted: over size budget]',
networkRequests: [],
capturedPayloads: [],
},
};
json = JSON.stringify(trimmed);
}
// If still over budget, drop page entirely
if (json.length > MAX_DIAGNOSTIC_BYTES) {
const minimal = { ...ctx, page: undefined };
json = JSON.stringify(minimal);
}
process.stderr.write(`\n${marker}\n${json}\n${marker}\n`);
}
+39
View File
@@ -19,6 +19,7 @@
* 78 Configuration error (ConfigError)
* 130 Interrupted by Ctrl-C (set by tui.ts SIGINT handler)
*/
import type { ObservationTraceReceipt } from './observation/events.js';
// ── Exit code table ──────────────────────────────────────────────────────────
@@ -55,6 +56,27 @@ export class CliError extends Error {
}
}
const TRACE_RECEIPT_SYMBOL = Symbol.for('opencli.traceReceipt');
export function attachTraceReceipt(err: unknown, receipt: ObservationTraceReceipt): void {
if (!err || (typeof err !== 'object' && typeof err !== 'function')) return;
try {
Object.defineProperty(err, TRACE_RECEIPT_SYMBOL, {
value: receipt,
enumerable: false,
configurable: true,
});
} catch {
// Non-extensible thrown objects are rare; trace export should never mask the
// original adapter error just because metadata attachment failed.
}
}
export function getTraceReceipt(err: unknown): ObservationTraceReceipt | undefined {
if (!err || (typeof err !== 'object' && typeof err !== 'function')) return undefined;
return (err as Record<PropertyKey, unknown>)[TRACE_RECEIPT_SYMBOL] as ObservationTraceReceipt | undefined;
}
// ── Typed subclasses ─────────────────────────────────────────────────────────
export type BrowserConnectKind = 'daemon-not-running' | 'extension-not-connected' | 'profile-required' | 'profile-disconnected' | 'command-failed' | 'unknown';
@@ -152,6 +174,13 @@ export interface ErrorEnvelope {
stack?: string;
cause?: string;
};
trace?: {
traceId: string;
dir: string;
summaryPath: string;
receiptPath: string;
status: ObservationTraceReceipt['status'];
};
}
// ── Utilities ───────────────────────────────────────────────────────────────
@@ -175,6 +204,14 @@ function serializeCause(cause: unknown, depth: number = 0): string {
/** Build an ErrorEnvelope from any caught value. */
export function toEnvelope(err: unknown): ErrorEnvelope {
const cause = err instanceof Error && err.cause ? serializeCause(err.cause) : undefined;
const traceReceipt = getTraceReceipt(err);
const trace = traceReceipt ? {
traceId: traceReceipt.traceId,
dir: traceReceipt.traceDir,
summaryPath: traceReceipt.summaryPath,
receiptPath: traceReceipt.receiptPath,
status: traceReceipt.status,
} : undefined;
if (err instanceof CliError) {
return {
ok: false,
@@ -185,6 +222,7 @@ export function toEnvelope(err: unknown): ErrorEnvelope {
exitCode: err.exitCode,
...(cause ? { cause } : {}),
},
...(trace ? { trace } : {}),
};
}
const msg = getErrorMessage(err);
@@ -196,5 +234,6 @@ export function toEnvelope(err: unknown): ErrorEnvelope {
exitCode: EXIT_CODES.GENERIC_ERROR,
...(cause ? { cause } : {}),
},
...(trace ? { trace } : {}),
};
}
+73 -3
View File
@@ -4,7 +4,7 @@ import * as os from 'node:os';
import * as path from 'node:path';
import type { CliCommand } from './registry.js';
import { executeCommand, prepareCommandArgs } from './execution.js';
import { TimeoutError } from './errors.js';
import { TimeoutError, toEnvelope } from './errors.js';
import { cli, Strategy } from './registry.js';
import { withTimeoutMs } from './runtime.js';
import * as runtime from './runtime.js';
@@ -193,17 +193,87 @@ describe('executeCommand — non-browser timeout', () => {
func: async () => { throw new Error('adapter failure'); },
});
await expect(executeCommand(cmd, {}, false, { trace: 'retain-on-failure' })).rejects.toThrow('adapter failure');
const thrown = await executeCommand(cmd, {}, false, { trace: 'retain-on-failure' }).catch((err) => err);
expect(thrown).toBeInstanceOf(Error);
expect((thrown as Error).message).toContain('adapter failure');
const tracesRoot = path.join(baseDir, 'profiles', 'default', 'traces');
const traceId = fs.readdirSync(tracesRoot)[0];
const traceDir = path.join(tracesRoot, traceId);
expect(fs.existsSync(path.join(traceDir, 'trace.jsonl'))).toBe(true);
expect(fs.existsSync(path.join(traceDir, 'receipt.json'))).toBe(true);
const trace = fs.readFileSync(path.join(traceDir, 'trace.jsonl'), 'utf-8');
expect(trace).toContain('token=[REDACTED]');
expect(trace).toContain('"authorization":"[REDACTED]"');
expect(trace).not.toContain('password=secret');
expect(stderrSpy.mock.calls.flat().join('\n')).toContain('OpenCLI trace artifact:');
expect(stderrSpy.mock.calls.flat().join('\n')).not.toContain('___OPENCLI_TRACE___');
expect(toEnvelope(thrown).trace).toMatchObject({
traceId,
dir: traceDir,
summaryPath: path.join(traceDir, 'summary.md'),
receiptPath: path.join(traceDir, 'receipt.json'),
status: 'failure',
});
expect(closeWindow).toHaveBeenCalledTimes(1);
} finally {
if (prevConfigDir === undefined) delete process.env.OPENCLI_CONFIG_DIR;
else process.env.OPENCLI_CONFIG_DIR = prevConfigDir;
stderrSpy.mockRestore();
fs.rmSync(baseDir, { recursive: true, force: true });
vi.restoreAllMocks();
}
});
it('exports a trace receipt on browser command success when trace is on', async () => {
const baseDir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-exec-trace-success-'));
const prevConfigDir = process.env.OPENCLI_CONFIG_DIR;
process.env.OPENCLI_CONFIG_DIR = baseDir;
const stderrSpy = vi.spyOn(process.stderr, 'write').mockImplementation(() => true);
const onTraceExport = vi.fn();
const closeWindow = vi.fn().mockResolvedValue(undefined);
const mockPage = {
closeWindow,
startNetworkCapture: vi.fn().mockResolvedValue(true),
readNetworkCapture: vi.fn().mockResolvedValue([]),
consoleMessages: vi.fn().mockResolvedValue([]),
snapshot: vi.fn().mockResolvedValue('snapshot'),
screenshot: vi.fn().mockResolvedValue(Buffer.from('png').toString('base64')),
getCurrentUrl: vi.fn().mockResolvedValue('https://example.com'),
getActivePage: vi.fn().mockReturnValue('tab-1'),
} as any;
vi.spyOn(capRouting, 'shouldUseBrowserSession').mockReturnValue(true);
vi.spyOn(runtime, 'browserSession').mockImplementation(async (_Factory, fn) => fn(mockPage));
try {
const cmd = cli({
site: 'test-execution',
name: 'browser-trace-success',
description: 'test trace export on success',
browser: true,
strategy: Strategy.PUBLIC,
func: async () => [{ ok: true }],
});
await expect(executeCommand(cmd, {}, false, { trace: 'on', onTraceExport })).resolves.toEqual([{ ok: true }]);
const stderr = stderrSpy.mock.calls.flat().join('\n');
expect(stderr).toContain('OpenCLI trace artifact:');
const tracesRoot = path.join(baseDir, 'profiles', 'default', 'traces');
const traceId = fs.readdirSync(tracesRoot)[0];
const receipt = JSON.parse(fs.readFileSync(path.join(tracesRoot, traceId, 'receipt.json'), 'utf-8'));
expect(receipt.status).toBe('success');
expect(receipt.traceDir).toContain(path.join(baseDir, 'profiles', 'default', 'traces'));
expect(receipt.scope).toMatchObject({
site: 'test-execution',
command: 'test-execution/browser-trace-success',
});
expect(receipt.error).toBeUndefined();
expect(onTraceExport).toHaveBeenCalledWith(expect.objectContaining({
traceId,
receipt: expect.objectContaining({ status: 'success' }),
}));
expect(closeWindow).toHaveBeenCalledTimes(1);
} finally {
if (prevConfigDir === undefined) delete process.env.OPENCLI_CONFIG_DIR;
+43 -26
View File
@@ -24,8 +24,7 @@ import { pathToFileURL } from 'node:url';
import * as fs from 'node:fs';
import * as os from 'node:os';
import { executePipeline } from './pipeline/index.js';
import { adapterLoadError, ArgumentError, CommandExecutionError, getErrorMessage } from './errors.js';
import { isDiagnosticEnabled, collectDiagnostic, emitDiagnostic } from './diagnostic.js';
import { adapterLoadError, ArgumentError, CommandExecutionError, attachTraceReceipt, getErrorMessage } from './errors.js';
import { shouldUseBrowserSession } from './capabilityRouting.js';
import { getBrowserFactory, browserSession, runWithTimeout, DEFAULT_BROWSER_COMMAND_TIMEOUT } from './runtime.js';
import { resolveProfileContextId } from './browser/profile.js';
@@ -33,7 +32,8 @@ import { emitHook, type HookContext } from './hooks.js';
import { log } from './logger.js';
import { isElectronApp } from './electron-apps.js';
import { probeCDP, resolveElectronEndpoint } from './launcher.js';
import { ObservationSession, exportObservationSession, type ObservationExportResult } from './observation/index.js';
import { ObservationSession, exportObservationSession, type ObservationExportResult, type ObservationExportStatus } from './observation/index.js';
import { resolveAdapterSourcePath } from './adapter-source.js';
const _loadedModules = new Map<string, Promise<void>>();
/** Track mtime of loaded user adapter files for hot-reload in daemon mode. */
@@ -179,7 +179,12 @@ export async function executeCommand(
cmd: CliCommand,
rawKwargs: CommandArgs,
debug: boolean = false,
opts: { prepared?: boolean; profile?: string; trace?: string } = {},
opts: {
prepared?: boolean;
profile?: string;
trace?: string;
onTraceExport?: (trace: ObservationExportResult) => void;
} = {},
): Promise<unknown> {
let kwargs: CommandArgs;
try {
@@ -199,7 +204,6 @@ export async function executeCommand(
await emitHook('onBeforeExecute', hookCtx);
let result: unknown;
let diagnosticEmitted = false;
try {
if (shouldUseBrowserSession(cmd)) {
const electron = isElectronApp(cmd.site);
@@ -225,6 +229,7 @@ export async function executeCommand(
ensureRequiredEnv(cmd);
const BrowserFactory = getBrowserFactory(cmd.site);
const contextId = resolveProfileContextId(opts.profile);
const internal = cmd as InternalCliCommand;
result = await browserSession(BrowserFactory, async (page) => {
const observation = traceMode === 'off'
? null
@@ -235,6 +240,7 @@ export async function executeCommand(
target: page.getActivePage?.(),
site: cmd.site,
command: fullName(cmd),
adapterSourcePath: resolveAdapterSourcePath(internal),
},
});
if (observation) {
@@ -274,10 +280,22 @@ export async function executeCommand(
phase: 'error',
data: { url: preNavUrl, error: err instanceof Error ? err.message : String(err) },
});
throw new CommandExecutionError(
const wrapped = new CommandExecutionError(
`Pre-navigation to ${preNavUrl} failed: ${err instanceof Error ? err.message : err}`,
'Check that the site is reachable and the browser extension is running.',
);
if (observation && (traceMode === 'on' || traceMode === 'retain-on-failure')) {
observation.record({
stream: 'error',
message: wrapped.message,
stack: wrapped.stack,
code: wrapped.code,
hint: wrapped.hint,
});
await collectObservationEvidence(observation, page).catch(() => {});
exportTraceArtifact(observation, 'failure', wrapped, opts.onTraceExport);
}
throw wrapped;
}
}
// --live / OPENCLI_LIVE=1 keeps the automation window open after the
@@ -295,14 +313,13 @@ export async function executeCommand(
});
if (observation && traceMode === 'on') {
await collectObservationEvidence(observation, page).catch(() => {});
exportTraceArtifact(observation);
exportTraceArtifact(observation, 'success', undefined, opts.onTraceExport);
}
// Adapter commands are one-shot — close the automation window immediately
// instead of waiting for the 30s idle timeout.
if (!keepOpen) await page.closeWindow?.().catch(() => {});
return result;
} catch (err) {
let trace: ObservationExportResult | undefined;
if (observation) {
observation.record({
stream: 'action',
@@ -317,16 +334,9 @@ export async function executeCommand(
});
if (traceMode === 'on' || traceMode === 'retain-on-failure') {
await collectObservationEvidence(observation, page).catch(() => {});
trace = exportTraceArtifact(observation, err);
exportTraceArtifact(observation, 'failure', err, opts.onTraceExport);
}
}
// Collect diagnostic while page is still alive (before closing the window).
if (isDiagnosticEnabled()) {
const internal = cmd as InternalCliCommand;
const ctx = await collectDiagnostic(err, internal, page, trace);
emitDiagnostic(ctx);
diagnosticEmitted = true;
}
// Close the automation window on failure too — without this, the window
// lingers until the extension's idle timer fires (unreliable on Windows
// where MV3 service workers may be suspended before setTimeout triggers).
@@ -348,13 +358,6 @@ export async function executeCommand(
}
}
} catch (err) {
// Emit diagnostic if not already emitted (browser session emits with page state;
// this fallback covers non-browser commands and pre-session failures like BrowserConnectError).
if (isDiagnosticEnabled() && !diagnosticEmitted) {
const internal = cmd as InternalCliCommand;
const ctx = await collectDiagnostic(err, internal, null);
emitDiagnostic(ctx);
}
hookCtx.error = err;
hookCtx.finishedAt = Date.now();
await emitHook('onAfterExecute', hookCtx);
@@ -413,10 +416,24 @@ async function collectObservationEvidence(session: ObservationSession, page: IPa
}
}
function exportTraceArtifact(session: ObservationSession, error?: unknown): ObservationExportResult | undefined {
function exportTraceArtifact(
session: ObservationSession,
status: ObservationExportStatus,
error?: unknown,
onTraceExport?: (trace: ObservationExportResult) => void,
): ObservationExportResult | undefined {
try {
const trace = exportObservationSession(session, { error });
process.stderr.write(`OpenCLI trace artifact: ${trace.dir}\n`);
const trace = exportObservationSession(session, { error, status });
if (status === 'failure' && error !== undefined) {
attachTraceReceipt(error, trace.receipt);
} else {
process.stderr.write(`OpenCLI trace artifact: ${trace.dir}\n`);
}
try {
onTraceExport?.(trace);
} catch (err) {
log.warn(`[trace] Trace export callback failed: ${err instanceof Error ? err.message : String(err)}`);
}
return trace;
} catch (err) {
log.warn(`[trace] Failed to export trace artifact: ${err instanceof Error ? err.message : String(err)}`);
+55 -3
View File
@@ -2,7 +2,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { exportObservationSession, getTraceDirectory } from './artifact.js';
import { buildTraceReceipt, exportObservationSession, getTraceDirectory } from './artifact.js';
import { ObservationSession } from './session.js';
describe('observation artifact', () => {
@@ -19,7 +19,13 @@ describe('observation artifact', () => {
it('writes artifacts under profile-scoped trace directory', () => {
const session = new ObservationSession({
id: 'trace-1',
scope: { contextId: 'work', workspace: 'site:demo', site: 'demo', command: 'demo/run' },
scope: {
contextId: 'work',
workspace: 'site:demo',
site: 'demo',
command: 'demo/run',
adapterSourcePath: '/tmp/clis/demo/run.js',
},
now: () => 1_700_000_000_000,
});
session.record({ stream: 'action', name: 'command', phase: 'start' });
@@ -39,6 +45,7 @@ describe('observation artifact', () => {
expect(fs.existsSync(path.join(result.dir, 'trace.jsonl'))).toBe(true);
expect(fs.existsSync(path.join(result.dir, 'network.jsonl'))).toBe(true);
expect(fs.existsSync(path.join(result.dir, 'console.jsonl'))).toBe(true);
expect(fs.existsSync(result.receiptPath)).toBe(true);
expect(fs.readFileSync(path.join(result.dir, 'screenshots', '0001.png'), 'utf-8')).toBe('png-bytes');
const trace = fs.readFileSync(path.join(result.dir, 'trace.jsonl'), 'utf-8');
@@ -47,7 +54,52 @@ describe('observation artifact', () => {
expect(trace).not.toContain('supersecret');
const summary = fs.readFileSync(result.summaryPath, 'utf-8');
expect(summary).toContain('contextId: work');
expect(summary).toContain('schemaVersion: 1');
expect(summary).toContain('opencliVersion:');
expect(summary).toContain('status: failure');
expect(summary).toContain('contextId: "work"');
expect(summary).toContain('adapterSourcePath: "/tmp/clis/demo/run.js"');
expect(summary).toContain('adapterSourcePathExists: false');
expect(summary).toContain('## Failed Network');
expect(summary).toContain('500 GET https://api.test/data?token=[REDACTED]');
expect(summary).toContain('network: 1');
const receipt = JSON.parse(fs.readFileSync(result.receiptPath, 'utf-8'));
expect(receipt).toMatchObject({
schemaVersion: 1,
opencliVersion: expect.any(String),
traceId: 'trace-1',
traceDir: result.dir,
summaryPath: result.summaryPath,
receiptPath: result.receiptPath,
status: 'failure',
scope: {
contextId: 'work',
workspace: 'site:demo',
site: 'demo',
command: 'demo/run',
adapterSourcePath: '/tmp/clis/demo/run.js',
},
error: { message: 'failed' },
});
});
it('builds a compact trace receipt', () => {
const receipt = buildTraceReceipt({
traceId: 'trace-1',
dir: '/tmp/opencli/profiles/work/traces/trace-1',
summaryPath: '/tmp/opencli/profiles/work/traces/trace-1/summary.md',
receiptPath: '/tmp/opencli/profiles/work/traces/trace-1/receipt.json',
}, 'failure', new Error('failed with token=secret'));
expect(receipt).toMatchObject({
schemaVersion: 1,
opencliVersion: expect.any(String),
traceId: 'trace-1',
traceDir: '/tmp/opencli/profiles/work/traces/trace-1',
receiptPath: '/tmp/opencli/profiles/work/traces/trace-1/receipt.json',
status: 'failure',
});
expect(receipt.error?.message).toContain('token=[REDACTED]');
});
});
+187 -15
View File
@@ -1,13 +1,16 @@
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import type { ObservationEvent, ObservationExportResult } from './events.js';
import type { ObservationEvent, ObservationExportResult, ObservationExportStatus, ObservationTraceReceipt } from './events.js';
import { ObservationSession } from './session.js';
import { redactValue } from './redaction.js';
import { CliError, getErrorMessage } from '../errors.js';
import { PKG_VERSION } from '../version.js';
export interface ExportObservationOptions {
baseDir?: string;
error?: unknown;
status?: ObservationExportStatus;
}
function baseOpenCliDir(): string {
@@ -25,6 +28,8 @@ export function getTraceDirectory(contextId: string | undefined, traceId: string
export function exportObservationSession(session: ObservationSession, opts: ExportObservationOptions = {}): ObservationExportResult {
const dir = getTraceDirectory(session.scope.contextId, session.id, opts.baseDir);
const status = opts.status ?? (opts.error === undefined ? 'success' : 'failure');
const createdAt = new Date().toISOString();
fs.mkdirSync(dir, { recursive: true });
fs.mkdirSync(path.join(dir, 'screenshots'), { recursive: true });
fs.mkdirSync(path.join(dir, 'state'), { recursive: true });
@@ -65,37 +70,204 @@ export function exportObservationSession(session: ObservationSession, opts: Expo
fs.writeFileSync(path.join(dir, 'console.jsonl'), consoleLines.join('\n') + (consoleLines.length ? '\n' : ''), 'utf-8');
const summaryPath = path.join(dir, 'summary.md');
fs.writeFileSync(summaryPath, renderSummary(session, sanitizedEvents, opts.error), 'utf-8');
return { traceId: session.id, dir, summaryPath };
fs.writeFileSync(summaryPath, renderSummary(session, sanitizedEvents, {
error: opts.error,
status,
dir,
createdAt,
}), 'utf-8');
const receiptPath = path.join(dir, 'receipt.json');
const resultBase = { traceId: session.id, dir, summaryPath, receiptPath };
const receipt = buildTraceReceipt(resultBase, status, opts.error, {
createdAt,
scope: session.scope,
});
fs.writeFileSync(receiptPath, JSON.stringify(receipt, null, 2), 'utf-8');
return { ...resultBase, receipt };
}
function redactObservationEvent(event: ObservationEvent): ObservationEvent {
return redactValue(event) as ObservationEvent;
}
function renderSummary(session: ObservationSession, events: ObservationEvent[], error: unknown): string {
export function buildTraceReceipt(
result: Pick<ObservationExportResult, 'traceId' | 'dir' | 'summaryPath' | 'receiptPath'>,
status: ObservationExportStatus,
error?: unknown,
opts: { createdAt?: string; scope?: ObservationSession['scope'] } = {},
): ObservationTraceReceipt {
const maybeCliError = error instanceof CliError ? error : undefined;
return {
schemaVersion: 1,
opencliVersion: PKG_VERSION,
traceId: result.traceId,
traceDir: result.dir,
summaryPath: result.summaryPath,
receiptPath: result.receiptPath,
status,
createdAt: opts.createdAt ?? new Date().toISOString(),
...(opts.scope ? { scope: opts.scope } : {}),
...(error === undefined ? {} : {
error: {
...(error instanceof Error ? { name: error.name } : {}),
...(maybeCliError ? { code: maybeCliError.code, hint: maybeCliError.hint, exitCode: maybeCliError.exitCode } : {}),
message: String(redactValue(getErrorMessage(error))),
},
}),
};
}
function renderSummary(
session: ObservationSession,
events: ObservationEvent[],
opts: { error?: unknown; status: ObservationExportStatus; dir: string; createdAt: string },
): string {
const counts = events.reduce<Record<string, number>>((acc, event) => {
acc[event.stream] = (acc[event.stream] ?? 0) + 1;
return acc;
}, {});
const errorMessage = error instanceof Error ? error.message : (error === undefined ? undefined : String(error));
const error = serializeSummaryError(opts.error);
const errorEvents = events.filter((event) => event.stream === 'error').slice(-20).reverse();
const failedNetwork = events
.filter((event): event is Extract<ObservationEvent, { stream: 'network' }> => event.stream === 'network')
.filter((event) => event.status === undefined || event.status === 0 || event.status >= 400)
.slice(-20)
.reverse();
const suspiciousConsole = events
.filter((event): event is Extract<ObservationEvent, { stream: 'console' }> => event.stream === 'console')
.filter((event) => /^(error|warning|warn|assert)$/i.test(event.level))
.slice(-20)
.reverse();
const actions = events
.filter((event): event is Extract<ObservationEvent, { stream: 'action' }> => event.stream === 'action')
.slice(-30);
const lines = [
'# OpenCLI Trace',
'---',
'schemaVersion: 1',
`opencliVersion: ${yamlScalar(PKG_VERSION)}`,
`traceId: ${yamlScalar(session.id)}`,
`status: ${opts.status}`,
`contextId: ${yamlScalar(session.scope.contextId ?? 'default')}`,
`workspace: ${yamlScalar(session.scope.workspace)}`,
...(session.scope.target ? [`target: ${yamlScalar(session.scope.target)}`] : []),
...(session.scope.site ? [`site: ${yamlScalar(session.scope.site)}`] : []),
...(session.scope.command ? [`command: ${yamlScalar(session.scope.command)}`] : []),
...(session.scope.adapterSourcePath ? [`adapterSourcePath: ${yamlScalar(session.scope.adapterSourcePath)}`] : []),
...(session.scope.adapterSourcePath ? [`adapterSourcePathExists: ${fs.existsSync(session.scope.adapterSourcePath)}`] : []),
`traceDir: ${yamlScalar(opts.dir)}`,
`startedAt: ${yamlScalar(new Date(session.startedAt).toISOString())}`,
`exportedAt: ${yamlScalar(opts.createdAt)}`,
...(error ? [
`errorCode: ${yamlScalar(error.code ?? 'UNKNOWN')}`,
`errorMessage: ${yamlScalar(error.message)}`,
] : []),
'---',
'',
`- traceId: ${session.id}`,
`- contextId: ${session.scope.contextId ?? 'default'}`,
`- workspace: ${session.scope.workspace}`,
...(session.scope.target ? [`- target: ${session.scope.target}`] : []),
...(session.scope.site ? [`- site: ${session.scope.site}`] : []),
...(session.scope.command ? [`- command: ${session.scope.command}`] : []),
`- startedAt: ${new Date(session.startedAt).toISOString()}`,
`- exportedAt: ${new Date().toISOString()}`,
...(errorMessage ? [`- error: ${String(redactValue(errorMessage))}`] : []),
'# OpenCLI Trace Summary',
'',
'## How To Use',
'',
'- Start with this summary, then inspect `trace.jsonl` only when the evidence below is insufficient.',
'- For adapter repair policy and retry limits, use the `opencli-autofix` skill.',
'- `adapterSourcePathExists: false` means the path is a best-effort hint, not a confirmed editable file.',
'',
'## Error',
'',
...renderErrorSection(error, errorEvents),
'',
'## Failed Network',
'',
...renderNetworkSection(failedNetwork),
'',
'## Suspicious Console',
'',
...renderConsoleSection(suspiciousConsole),
'',
'## Action Timeline',
'',
...renderActionSection(actions),
'',
'## Event Counts',
'',
...Object.entries(counts).map(([stream, count]) => `- ${stream}: ${count}`),
'',
'## Artifact Files',
'',
'- `trace.jsonl`: full redacted event timeline',
'- `network.jsonl`: redacted network events',
'- `console.jsonl`: redacted console events',
'- `state/`: final state snapshots when available',
'- `screenshots/`: final screenshots when available',
'',
];
return lines.join('\n');
}
function serializeSummaryError(error: unknown): { code?: string; message: string; hint?: string } | undefined {
if (error === undefined) return undefined;
if (error instanceof CliError) {
return {
code: error.code,
message: String(redactValue(error.message)),
...(error.hint ? { hint: String(redactValue(error.hint)) } : {}),
};
}
return { message: String(redactValue(getErrorMessage(error))) };
}
function yamlScalar(value: string): string {
return JSON.stringify(value);
}
function renderErrorSection(
error: { code?: string; message: string; hint?: string } | undefined,
errorEvents: ObservationEvent[],
): string[] {
const lines: string[] = [];
if (error) {
lines.push(`- ${error.code ?? 'UNKNOWN'}: ${error.message}`);
if (error.hint) lines.push(`- hint: ${error.hint}`);
}
for (const event of errorEvents) {
if (event.stream !== 'error') continue;
lines.push(`- ${formatTs(event.ts)} ${event.code ?? 'ERROR'}: ${event.message}`);
}
return lines.length ? lines : ['- none'];
}
function renderNetworkSection(events: Extract<ObservationEvent, { stream: 'network' }>[]): string[] {
if (!events.length) return ['- none'];
return events.map((event) => {
const status = event.status ?? 'unknown';
const method = event.method ?? 'GET';
const contentType = event.contentType ? ` ${event.contentType}` : '';
return `- ${formatTs(event.ts)} ${status} ${method} ${event.url}${contentType}`;
});
}
function renderConsoleSection(events: Extract<ObservationEvent, { stream: 'console' }>[]): string[] {
if (!events.length) return ['- none'];
return events.map((event) => `- ${formatTs(event.ts)} ${event.level}: ${trimLine(event.text, 240)}`);
}
function renderActionSection(events: Extract<ObservationEvent, { stream: 'action' }>[]): string[] {
if (!events.length) return ['- none'];
return events.map((event) => {
const phase = event.phase ? ` ${event.phase}` : '';
const data = event.data && Object.keys(event.data).length
? ` ${trimLine(JSON.stringify(redactValue(event.data)), 240)}`
: '';
return `- ${formatTs(event.ts)} ${event.name}${phase}${data}`;
});
}
function formatTs(ts: number): string {
return new Date(ts).toISOString();
}
function trimLine(value: string, max: number): string {
const compact = value.replace(/\s+/g, ' ').trim();
return compact.length > max ? `${compact.slice(0, max)}...` : compact;
}
+24
View File
@@ -6,6 +6,7 @@ export interface ObservationScope {
target?: string;
site?: string;
command?: string;
adapterSourcePath?: string;
}
interface BaseObservationEvent {
@@ -83,4 +84,27 @@ export interface ObservationExportResult {
traceId: string;
dir: string;
summaryPath: string;
receiptPath: string;
receipt: ObservationTraceReceipt;
}
export type ObservationExportStatus = 'success' | 'failure';
export interface ObservationTraceReceipt {
schemaVersion: 1;
opencliVersion: string;
traceId: string;
traceDir: string;
summaryPath: string;
receiptPath: string;
status: ObservationExportStatus;
createdAt: string;
scope?: ObservationScope;
error?: {
name?: string;
code?: string;
message: string;
hint?: string;
exitCode?: number;
};
}