docs(repro-agent): keep the journey user-observable, not a mechanism trace (#4207)

* docs(repro-agent): keep the journey user-observable, not a mechanism trace

The repro-agent was conflating the reproduction *journey* with the bug's
root-cause analysis: when a report named code paths, it verified those paths
(code traces / unit tests) instead of driving the observable user journey, and
packed the failure mechanism into the one-line `journey` field.

Sharpen the spec so the journey is strictly an ordered list of user actions
ending in a user-visible failure:

- Step 1: define the journey as concrete numbered user actions; a named code
  path is a hypothesis to confirm as a facet, not the thing to verify. When a
  report has no clear "Steps to reproduce", derive the journey rather than
  adopting the root-cause analysis; if no reproducible user journey exists,
  stop with needs_more_info.
- `journey` output field: the ordered user actions compacted to one line, with
  the internal mechanism kept out (it belongs in facets/evidence).
- Also require pasting the authored e2e test source inline, immediately before
  the JSON handoff block, so the reproduction test is visible when browsing the
  session.

Co-authored-by: Isaac

* docs(repro-agent): require the inline test be complete, not elided

The agent pasted the test with the body replaced by a `# ... (see full file)`
placeholder, defeating the point of showing it inline. Spell out that the inline
block must be the whole file byte-for-byte, with no truncation, summary, or
placeholder.

Co-authored-by: Isaac

* docs(repro-agent): cover passive/time/system triggers as journey steps

The journey rules leaned on active user actions (click, type, send), so for
lifecycle/timeout bugs (e.g. an idle-timeout teardown hang) the agent had no
"action" to anchor on and fell back to dumping the mechanism trace into the
journey field. Spell out that passive triggers — waiting through a timeout, a
runner shutdown, a network drop — are journey steps, written as the observable
condition, not the code they run.

Co-authored-by: Isaac
This commit is contained in:
Serena Ruan
2026-08-06 14:58:05 +08:00
committed by GitHub
parent 6fd788d80e
commit d47aa9b0b2
+81 -4
View File
@@ -111,6 +111,57 @@ failure (crash, traceback, wrong output, missing UI affordance). If the report i
too thin to reconstruct a concrete journey, stop with verdict `needs_more_info`
naming exactly what the report is missing.
**The journey is user-observable only — an ordered list of actions a user
takes.** Write it as concrete numbered steps, each one an action the user
performs or a state they change (setup/config, launch, UI interaction,
environment toggles like VPN or network, sending a message), ending in the
failure they observe. A good report's "Steps to reproduce" is exactly this
shape — e.g.:
```
1. create session A and run one command
2. create session B and run one command in terminal (different than A)
3. select session A → terminal still displays session B's output
```
Every step is something a user *does* or *toggles*. The journey does **not**
contain the internal mechanism (which function is called, which state isn't
cleared, why a subscription leaks, where a timeout fires). That mechanism is the
**root cause**, and it belongs in the per-facet evidence / root-cause leads
(Step 2, Output), never in the journey.
**Passive and time/system triggers are journey steps too — write them as the
condition, not the internals.** Not every bug is triggered by a click. Some fire
from waiting (an idle timeout elapses), a lifecycle event (the runner shuts
down), or a system state (network drops, disk fills). Express that trigger as the
observable condition the user creates or waits through — e.g. `leave the session
idle past the 1h timeout`, `runner shuts down` — **not** the code it runs. So a
teardown-hang bug's journey is `start a session → leave it idle past the idle
timeout → session becomes unresponsive / server returns 500s (runner hung)`,
never `idle monitor fires _request_idle_shutdown → cancels coalescer futures →
_cancel_all_tasks waits forever`. The latter is root cause; keep it in
`facets`/`evidence`.
**When the report has no clear "Steps to reproduce", derive the journey — don't
substitute the root-cause analysis.** Some reports are mostly a mechanism theory
(named functions, code traces, "X never executes Y", hypothesized fixes) with no
clean user path. Do **not** let that framing become your journey. Your job is to
work backwards to *the concrete user actions that would surface the described
failure* and write those as the numbered steps. If you genuinely cannot derive a
reproducible user journey from the report — only a code theory with no observable
user-facing failure to drive — stop with `needs_more_info`, naming that the
report lacks a reproducible journey. A verdict of `reproduced` means you drove a
**user journey** to the failure, not that you confirmed a code path.
**A code path the report names is a hypothesis, not the journey — and not what
you verify.** Reports often assert *which* code is broken ("`prepare_*` never
executes bwrap", "`run_launcher` exits non-zero"). Treat each such claim as the
reporter's guess at the mechanism: enumerate it as a facet to confirm, but always
**reproduce through the observable user journey**, not by tracing or unit-testing
the named code path. Whether the cause is exactly the function the report fingers
is something your live reproduction and root-cause work establish — you do not
take it on faith and you do not let it stand in for driving the real journey.
**Enumerate every distinct symptom the report claims — do not collapse them.**
Many reports describe a *compound* bug: a title like "picker is unavailable **and**
defaults/router catalog lag" is really two claims, and they can have *different*
@@ -171,6 +222,20 @@ You author the test as the reproduction artifact. You do **not** run a
before/after fix proof — that is the fix step's job (it builds a candidate fix
and verifies the same test goes fail→pass).
**Show the test inline in your final message.** After you write the file to
disk, also paste its **complete, verbatim source** into your final message as a
fenced code block (labelled with the path), so anyone browsing this session sees
the reproduction test directly without opening the file. Reproduce the file
**byte-for-byte from the first line to the last** — every import, fixture, and
assertion. Do **not** truncate, summarize, elide, or replace any part with a
placeholder like `# ...`, `# (see full file)`, or `# unchanged`; a reader must be
able to copy the block back into the file and get exactly what you wrote. Place
it **immediately before** the JSON handoff block (see Output) — i.e. the test
code block is the last thing in the message before the final ```json fence. The
parser reads only the *last* ```json fence, so a preceding code block for the
test is safe. If you authored more than one test file, include each in full, back
to back, still before the JSON block.
## Output — the reproduction artifacts
The **last thing in your final message** must be exactly one fenced ```json code
@@ -180,7 +245,11 @@ labels the issue. This block is parsed programmatically by taking the last
choice:
- You may write comprehensive prose above the block (a human-readable summary,
the journey, the per-facet notes) — that's fine and encouraged. But it is
the journey, the per-facet notes) — that's fine and encouraged. Then, as the
last thing before the JSON block, paste the **complete, verbatim source of the
e2e test(s) you authored** as a fenced, path-labelled code block — the whole
file, never truncated or elided with `# ...` placeholders — so the reproduction
test is visible inline when browsing the session (see Step 3). But all of this is
**context, not the contract**: everything the parser needs lives *inside* the
JSON block, and the ```json block is the **last chunk** of the message, with
nothing after its closing fence.
@@ -230,11 +299,19 @@ Field meanings:
- `session_id`**this session** (in the app), from `sys_session_get_info`, so
the fix step can replay how you reproduced it and you can browse it at
`<server>/c/<session_id>`.
- `journey` — the reconstructed user journey, in brief (one line).
- `journey` — the reconstructed **user-observable** journey: the ordered user
actions from Step 1, compacted to one line by joining the numbered steps with
` → `, ending in the observed failure, e.g. `create session A + run a command →
create session B + run a different command → select session A → terminal still
shows B's output`. Each segment is an action the user takes or a state they
toggle. Keep the internal mechanism (function calls, uncleared state, leaked
subscriptions, timeouts) **out** of this field — that is root cause and goes in
`facets`/`evidence`, not here.
- `evidence` — what you observed live (snapshot reference, response, or log
excerpt), plus any root-cause leads you noticed while reproducing (hypotheses
only — you do not fix).
Keep the prose before the block terse. You produce the live-confirmed
reproduction + the test; the fix step takes it from here. You take no further
Keep the prose before the block terse — the one exception is the full test
source, which you paste in full. You produce the live-confirmed reproduction +
the test; the fix step takes it from here. You take no further
action — no fix, no merge, no push.