docs(repro-agent): keep the journey user-observable, not a mechanism trace (#4207)
* docs(repro-agent): keep the journey user-observable, not a mechanism trace The repro-agent was conflating the reproduction *journey* with the bug's root-cause analysis: when a report named code paths, it verified those paths (code traces / unit tests) instead of driving the observable user journey, and packed the failure mechanism into the one-line `journey` field. Sharpen the spec so the journey is strictly an ordered list of user actions ending in a user-visible failure: - Step 1: define the journey as concrete numbered user actions; a named code path is a hypothesis to confirm as a facet, not the thing to verify. When a report has no clear "Steps to reproduce", derive the journey rather than adopting the root-cause analysis; if no reproducible user journey exists, stop with needs_more_info. - `journey` output field: the ordered user actions compacted to one line, with the internal mechanism kept out (it belongs in facets/evidence). - Also require pasting the authored e2e test source inline, immediately before the JSON handoff block, so the reproduction test is visible when browsing the session. Co-authored-by: Isaac * docs(repro-agent): require the inline test be complete, not elided The agent pasted the test with the body replaced by a `# ... (see full file)` placeholder, defeating the point of showing it inline. Spell out that the inline block must be the whole file byte-for-byte, with no truncation, summary, or placeholder. Co-authored-by: Isaac * docs(repro-agent): cover passive/time/system triggers as journey steps The journey rules leaned on active user actions (click, type, send), so for lifecycle/timeout bugs (e.g. an idle-timeout teardown hang) the agent had no "action" to anchor on and fell back to dumping the mechanism trace into the journey field. Spell out that passive triggers — waiting through a timeout, a runner shutdown, a network drop — are journey steps, written as the observable condition, not the code they run. Co-authored-by: Isaac
This commit is contained in:
@@ -111,6 +111,57 @@ failure (crash, traceback, wrong output, missing UI affordance). If the report i
|
||||
too thin to reconstruct a concrete journey, stop with verdict `needs_more_info`
|
||||
naming exactly what the report is missing.
|
||||
|
||||
**The journey is user-observable only — an ordered list of actions a user
|
||||
takes.** Write it as concrete numbered steps, each one an action the user
|
||||
performs or a state they change (setup/config, launch, UI interaction,
|
||||
environment toggles like VPN or network, sending a message), ending in the
|
||||
failure they observe. A good report's "Steps to reproduce" is exactly this
|
||||
shape — e.g.:
|
||||
|
||||
```
|
||||
1. create session A and run one command
|
||||
2. create session B and run one command in terminal (different than A)
|
||||
3. select session A → terminal still displays session B's output
|
||||
```
|
||||
|
||||
Every step is something a user *does* or *toggles*. The journey does **not**
|
||||
contain the internal mechanism (which function is called, which state isn't
|
||||
cleared, why a subscription leaks, where a timeout fires). That mechanism is the
|
||||
**root cause**, and it belongs in the per-facet evidence / root-cause leads
|
||||
(Step 2, Output), never in the journey.
|
||||
|
||||
**Passive and time/system triggers are journey steps too — write them as the
|
||||
condition, not the internals.** Not every bug is triggered by a click. Some fire
|
||||
from waiting (an idle timeout elapses), a lifecycle event (the runner shuts
|
||||
down), or a system state (network drops, disk fills). Express that trigger as the
|
||||
observable condition the user creates or waits through — e.g. `leave the session
|
||||
idle past the 1h timeout`, `runner shuts down` — **not** the code it runs. So a
|
||||
teardown-hang bug's journey is `start a session → leave it idle past the idle
|
||||
timeout → session becomes unresponsive / server returns 500s (runner hung)`,
|
||||
never `idle monitor fires _request_idle_shutdown → cancels coalescer futures →
|
||||
_cancel_all_tasks waits forever`. The latter is root cause; keep it in
|
||||
`facets`/`evidence`.
|
||||
|
||||
**When the report has no clear "Steps to reproduce", derive the journey — don't
|
||||
substitute the root-cause analysis.** Some reports are mostly a mechanism theory
|
||||
(named functions, code traces, "X never executes Y", hypothesized fixes) with no
|
||||
clean user path. Do **not** let that framing become your journey. Your job is to
|
||||
work backwards to *the concrete user actions that would surface the described
|
||||
failure* and write those as the numbered steps. If you genuinely cannot derive a
|
||||
reproducible user journey from the report — only a code theory with no observable
|
||||
user-facing failure to drive — stop with `needs_more_info`, naming that the
|
||||
report lacks a reproducible journey. A verdict of `reproduced` means you drove a
|
||||
**user journey** to the failure, not that you confirmed a code path.
|
||||
|
||||
**A code path the report names is a hypothesis, not the journey — and not what
|
||||
you verify.** Reports often assert *which* code is broken ("`prepare_*` never
|
||||
executes bwrap", "`run_launcher` exits non-zero"). Treat each such claim as the
|
||||
reporter's guess at the mechanism: enumerate it as a facet to confirm, but always
|
||||
**reproduce through the observable user journey**, not by tracing or unit-testing
|
||||
the named code path. Whether the cause is exactly the function the report fingers
|
||||
is something your live reproduction and root-cause work establish — you do not
|
||||
take it on faith and you do not let it stand in for driving the real journey.
|
||||
|
||||
**Enumerate every distinct symptom the report claims — do not collapse them.**
|
||||
Many reports describe a *compound* bug: a title like "picker is unavailable **and**
|
||||
defaults/router catalog lag" is really two claims, and they can have *different*
|
||||
@@ -171,6 +222,20 @@ You author the test as the reproduction artifact. You do **not** run a
|
||||
before/after fix proof — that is the fix step's job (it builds a candidate fix
|
||||
and verifies the same test goes fail→pass).
|
||||
|
||||
**Show the test inline in your final message.** After you write the file to
|
||||
disk, also paste its **complete, verbatim source** into your final message as a
|
||||
fenced code block (labelled with the path), so anyone browsing this session sees
|
||||
the reproduction test directly without opening the file. Reproduce the file
|
||||
**byte-for-byte from the first line to the last** — every import, fixture, and
|
||||
assertion. Do **not** truncate, summarize, elide, or replace any part with a
|
||||
placeholder like `# ...`, `# (see full file)`, or `# unchanged`; a reader must be
|
||||
able to copy the block back into the file and get exactly what you wrote. Place
|
||||
it **immediately before** the JSON handoff block (see Output) — i.e. the test
|
||||
code block is the last thing in the message before the final ```json fence. The
|
||||
parser reads only the *last* ```json fence, so a preceding code block for the
|
||||
test is safe. If you authored more than one test file, include each in full, back
|
||||
to back, still before the JSON block.
|
||||
|
||||
## Output — the reproduction artifacts
|
||||
|
||||
The **last thing in your final message** must be exactly one fenced ```json code
|
||||
@@ -180,7 +245,11 @@ labels the issue. This block is parsed programmatically by taking the last
|
||||
choice:
|
||||
|
||||
- You may write comprehensive prose above the block (a human-readable summary,
|
||||
the journey, the per-facet notes) — that's fine and encouraged. But it is
|
||||
the journey, the per-facet notes) — that's fine and encouraged. Then, as the
|
||||
last thing before the JSON block, paste the **complete, verbatim source of the
|
||||
e2e test(s) you authored** as a fenced, path-labelled code block — the whole
|
||||
file, never truncated or elided with `# ...` placeholders — so the reproduction
|
||||
test is visible inline when browsing the session (see Step 3). But all of this is
|
||||
**context, not the contract**: everything the parser needs lives *inside* the
|
||||
JSON block, and the ```json block is the **last chunk** of the message, with
|
||||
nothing after its closing fence.
|
||||
@@ -230,11 +299,19 @@ Field meanings:
|
||||
- `session_id` — **this session** (in the app), from `sys_session_get_info`, so
|
||||
the fix step can replay how you reproduced it and you can browse it at
|
||||
`<server>/c/<session_id>`.
|
||||
- `journey` — the reconstructed user journey, in brief (one line).
|
||||
- `journey` — the reconstructed **user-observable** journey: the ordered user
|
||||
actions from Step 1, compacted to one line by joining the numbered steps with
|
||||
` → `, ending in the observed failure, e.g. `create session A + run a command →
|
||||
create session B + run a different command → select session A → terminal still
|
||||
shows B's output`. Each segment is an action the user takes or a state they
|
||||
toggle. Keep the internal mechanism (function calls, uncleared state, leaked
|
||||
subscriptions, timeouts) **out** of this field — that is root cause and goes in
|
||||
`facets`/`evidence`, not here.
|
||||
- `evidence` — what you observed live (snapshot reference, response, or log
|
||||
excerpt), plus any root-cause leads you noticed while reproducing (hypotheses
|
||||
only — you do not fix).
|
||||
|
||||
Keep the prose before the block terse. You produce the live-confirmed
|
||||
reproduction + the test; the fix step takes it from here. You take no further
|
||||
Keep the prose before the block terse — the one exception is the full test
|
||||
source, which you paste in full. You produce the live-confirmed reproduction +
|
||||
the test; the fix step takes it from here. You take no further
|
||||
action — no fix, no merge, no push.
|
||||
|
||||
Reference in New Issue
Block a user