Compare commits

...

3 Commits

Author SHA1 Message Date
Hubert Zub 9a6a0f1851 test(e2e-ui): stop visual regen from writing a duplicate baseline
playwright-visual-snapshot already rewrites a drifting baseline IN PLACE under
snapshots/ when GITHUB_ACTIONS is set (and creates a missing one there), while it
writes actual/expected/diff into snapshot_failures/<test>[browser][platform]/ --
a DIFFERENT subdir scheme than the baseline's snapshots/<test>/. The old "adopt"
steps reconstructed a snapshots/ path from that failures subdir, so every regen
wrote a parallel snapshots/<test>[chromium][linux]/ baseline that nothing reads.

- ui-snapshot-update.yml: drop the redundant adopt step; the in-CI in-place
  update already leaves snapshots/ holding exactly the changed PNGs.
- regen_baseline_docker.sh: set GITHUB_ACTIONS=true so the local Docker render
  updates baselines in place like the gate does; drop the adopt path-munging.
- update_baseline_from_pr.sh: restore the artifact's snapshots/ tree verbatim
  instead of reconstructing paths from snapshot_failures/.
- Delete the stray duplicate chat baseline dir created by the old logic.
- README: document the in-place update + the simplified fork path.
2026-06-24 06:26:28 +00:00
omnigent-ci[bot] 74838e71f8 test(e2e-ui): regenerate visual baselines 2026-06-24 05:59:01 +00:00
Hubert Zub 5928e29898 test(e2e-ui): shorten chat snapshot sample to fix wrap-boundary flake
The assistant code sample's longest line landed exactly on the code box's
overflow boundary, so subpixel rendering differences flipped the SPA between
"fits" (clipped, no wrap toggle) and "overflows" (wraps + shows a wrap toggle).
The extra wrapped row shifted the whole transcript below it, producing a large
diff with no UI change behind it. Shorten every line well clear of the box width
so nothing reflows at the edge. Baseline regenerated in the pinned image.
2026-06-24 05:54:42 +00:00
7 changed files with 50 additions and 75 deletions
+5 -25
View File
@@ -113,37 +113,17 @@ jobs:
- name: Compare the baselines (no --update-snapshots)
# Deliberately NOT --update-snapshots: that rewrites EVERY PNG, churning
# baselines that already pass (a sub-threshold re-render still changes the
# bytes). In plain compare mode the plugin instead leaves passing
# baselines untouched and surfaces only the drift -- a mismatching
# baseline's fresh render lands in snapshot_failures/.../actual_*.png (the
# committed PNG is left in place), and a MISSING baseline is created
# directly under snapshots/. The run "fails" by design on any drift, so
# don't gate on its exit code.
# bytes). In plain compare mode the plugin leaves passing baselines
# untouched and rewrites only the drift: under GitHub Actions it updates a
# mismatching baseline IN PLACE (and creates a MISSING one) under
# snapshots/, so the tree below already holds exactly the changed PNGs.
# The run "fails" by design on any drift, so don't gate on its exit code.
run: |
uv run pytest tests/e2e_ui/visual -m visual \
-v --tb=long --log-level=INFO -r a \
-p no:rerunfailures \
--ui-skip-build || true
- name: Adopt only the changed renders over their baselines
# Copy each mismatching test's actual_<name>.png over its committed
# baseline; previously-missing baselines were already written under
# snapshots/ by the compare above. Baselines that passed are not in
# snapshot_failures, so they stay byte-for-byte unchanged.
run: |
fail_dir=tests/e2e_ui/visual/snapshot_failures
if [ -d "$fail_dir" ]; then
while IFS= read -r src; do
rel=${src#"$fail_dir"/} # <module>/<test>/actual_<name>.png
dest="tests/e2e_ui/visual/snapshots/$(dirname "$rel")/$(basename "$rel" | sed 's/^actual_//')"
mkdir -p "$(dirname "$dest")"
cp "$src" "$dest"
echo "adopted: $dest"
done < <(find "$fail_dir" -type f -name 'actual_*.png')
else
echo "No snapshot_failures dir -- no existing baseline drifted (only new baselines, if any, were created)."
fi
# Tar the snapshots tree (paths intact) so the commit job can restore it
# wholesale. Only genuinely-changed/created PNGs differ from the committed
# tree, so git add in the commit job stages exactly those.
+8 -7
View File
@@ -99,18 +99,19 @@ build.
### Fork PR without Docker — adopt the run's render
The failing compare run already rendered your change in the pinned image, so pull
that image into the baseline:
The failing compare run already rendered your change in the pinned image, and
because it runs under GitHub Actions the plugin rewrote each drifting baseline
**in place** under `snapshots/`. Pull that tree in:
```bash
tests/e2e_ui/visual/update_baseline_from_pr.sh <pr-number>
```
It finds the PR's UI Snapshot run, downloads the artifact, and copies the
runner-rendered `actual_` PNG over the committed baseline. **Review the image**,
then commit and push. (Manual equivalent: download the `ui-snapshot-<run_id>`
artifact, take `snapshot_failures/.../actual_*.png`, and commit it over the
baseline path above.)
It finds the PR's UI Snapshot run, downloads the artifact, and restores its
runner-rendered `snapshots/` tree over the committed baselines — only the
drifting ones differ. **Review the image(s)**, then commit and push. (Manual
equivalent: download the `ui-snapshot-<run_id>` artifact and commit its
`snapshots/` tree over `tests/e2e_ui/visual/snapshots/`.)
### Workflow dispatch (non-PR branches)
+9 -21
View File
@@ -65,13 +65,15 @@ fi
echo "Rendering + comparing the baselines in the pinned Playwright image ..."
# Deliberately NOT --update-snapshots: that rewrites every PNG, churning
# baselines that still pass (a sub-threshold re-render changes the bytes). Plain
# compare leaves passing baselines alone and writes only the drift to
# snapshot_failures/.../actual_*.png (a missing baseline is created directly in
# snapshots/). The run "fails" by design on any drift, so `|| true` is expected --
# the adopt step + git diff below are the real signal. UV_PROJECT_ENVIRONMENT
# lives in the container (not the mounted repo) so no root-owned .venv leaks out.
# compare leaves passing baselines alone and rewrites only the drift. GITHUB_ACTIONS
# is set so the plugin behaves exactly as the CI gate does: it updates a mismatching
# baseline IN PLACE (and creates a missing one) under snapshots/, so the git diff
# below is the real signal. The run "fails" by design on any drift, so `|| true` is
# expected. UV_PROJECT_ENVIRONMENT lives in the container (not the mounted repo) so
# no root-owned .venv leaks out.
docker run --rm --platform "$PLATFORM" -v "$PWD":/work -w /work \
-e CI=1 \
-e GITHUB_ACTIONS=true \
-e OMNIGENT_PW_NO_SANDBOX=1 \
-e OMNIGENT_SKIP_WEB_UI=true \
-e UV_PYTHON_PREFERENCE=only-system \
@@ -83,25 +85,11 @@ docker run --rm --platform "$PLATFORM" -v "$PWD":/work -w /work \
-p no:rerunfailures --ui-skip-build
' || true
# Files Docker wrote are root-owned; hand them back so the adopt + git add work
# unprivileged. Includes ap-web (node_modules + build intermediates the Node
# container wrote).
# Files Docker wrote are root-owned; hand them back so git add works unprivileged.
# Includes ap-web (node_modules + build intermediates the Node container wrote).
docker run --rm --platform "$PLATFORM" -v "$PWD":/work "$PW_IMAGE" \
chown -R "$(id -u):$(id -g)" /work/tests/e2e_ui/visual /work/"$BUILD_OUTPUT" /work/ap-web 2>/dev/null || true
# Adopt only the changed renders: copy each mismatching test's actual_ PNG over
# its committed baseline (missing baselines were already created in snapshots/).
# Passing baselines are not in snapshot_failures, so they stay untouched.
FAIL_DIR="tests/e2e_ui/visual/snapshot_failures"
if [ -d "$FAIL_DIR" ]; then
while IFS= read -r src; do
rel=${src#"$FAIL_DIR"/}
dest="$SNAP_ROOT/$(dirname "$rel")/$(basename "$rel" | sed 's/^actual_//')"
mkdir -p "$(dirname "$dest")"
cp "$src" "$dest"
done < <(find "$FAIL_DIR" -type f -name 'actual_*.png')
fi
echo
if git diff --quiet -- "$SNAP_ROOT"; then
echo "Baselines unchanged — they already match this render (or the render failed; check the output above)."
Binary file not shown.

Before

Width:  |  Height:  |  Size: 106 KiB

After

Width:  |  Height:  |  Size: 92 KiB

+12 -6
View File
@@ -71,16 +71,22 @@ _HOSTS_BODY = {
}
_EMPTY_LIST_BODY = {"object": "list", "data": [], "has_more": False}
_USER_TEXT = "How do I read a file in Python and print its contents?"
_USER_TEXT = "How do I read a file in Python?"
# Keep every line comfortably SHORT. The code box is narrow, and a line that
# lands near its wrap/overflow boundary renders nondeterministically: subpixel
# differences flip the SPA between "fits" (clipped, no wrap toggle) and
# "overflows" (wraps + shows a wrap toggle), which adds a row and shifts the
# whole transcript below it -- a snapshot flake with no UI change behind it. So
# no line here should approach the box width.
_ASSISTANT_TEXT = (
"Use a `with` block so the file is closed automatically:\n\n"
"Use a `with` block so the file closes itself:\n\n"
"```python\n"
"with open('notes.txt', encoding='utf-8') as f:\n"
"with open('notes.txt') as f:\n"
" print(f.read())\n"
"```\n\n"
"A couple of things to keep in mind:\n\n"
"- `with` releases the handle even if an error is raised mid-read.\n"
"- Pass `encoding='utf-8'` explicitly so the read is platform-independent.\n"
"A couple of notes:\n\n"
"- `with` frees the handle even on an error.\n"
"- Use `f.read()` for all of it, or loop to stream.\n"
)
# Server returns newest-first; the client reverses to chronological, so the
+16 -16
View File
@@ -58,27 +58,27 @@ if ! gh run download "$RUN_ID" --repo "$REPO" -n "ui-snapshot-$RUN_ID" -D "$TMP"
gh run download "$RUN_ID" --repo "$REPO" -D "$TMP" # fall back to all artifacts
fi
# On a mismatch each failing snapshot's runner render is
# snapshot_failures/<module>/<test>/actual_<name>.png; its committed baseline is
# snapshots/<module>/<test>/<name>.png. Adopt every actual_ over its baseline so
# a multi-snapshot failure is fixed in one pass.
updated=0
while IFS= read -r src; do
rel=${src##*/snapshot_failures/} # <module>/<test>/actual_<name>.png
dest="$SNAP_ROOT/$(dirname "$rel")/$(basename "$rel" | sed 's/^actual_//')"
mkdir -p "$(dirname "$dest")"
cp "$src" "$dest"
echo " updated: $dest"
updated=$((updated + 1))
done < <(find "$TMP" -type f -path '*/snapshot_failures/*' -name 'actual_*.png')
# The gate runs under GitHub Actions, where the plugin rewrites a mismatching
# baseline IN PLACE under snapshots/ (and creates a missing one there); passing
# baselines are left untouched. The artifact carries that whole snapshots/ tree,
# so just restore it over the committed one -- git add then stages exactly the
# baselines that drifted. (Reconstructing paths from snapshot_failures/ instead
# is fragile: that tree keys subdirs by <test>[browser][platform], not the
# baseline's <test>, so it would write a parallel, never-read duplicate.)
src_root=$(find "$TMP" -type d -name snapshots | head -1)
if [ -z "$src_root" ]; then
echo "error: artifact has no snapshots/ tree -- nothing to restore." >&2
exit 1
fi
cp -R "$src_root/." "$SNAP_ROOT/"
if [ "$updated" -eq 0 ]; then
echo "No rendered diffs in the artifact -- the gate likely PASSED, so the baselines already match. Nothing to update." >&2
if git diff --quiet -- "$SNAP_ROOT"; then
echo "No baseline drift in the artifact -- the gate render already matches the committed baselines. Nothing to update." >&2
exit 0
fi
echo
echo "Updated $updated baseline(s)."
echo "Updated baseline(s) from the gate render:"
git --no-pager diff --stat -- "$SNAP_ROOT" || true
echo
echo "Next: review the image(s), then commit + push:"