fix(integ): widen the truth matchers the runners disagree with

Three literals were host-dependent and CI proved it: the disk example's
stat line carries the mtime actions/checkout stamps on data/example.json,
its restore root is /tmp on Linux where it was /var/folders on macOS, and
the TS redis cache example echoes $REDIS_URL, which is db 1 locally and
db 0 on the runners.

All three got captured as literals because --emit runs both of its
captures on one machine, so it sees run-to-run volatility and is blind to
host-to-host volatility. That limitation is now in its docstring with the
three classes to widen by hand.

The workflow loops now keep going after a failure and fail the step at
the end, so one stale truth file reports every other one in the same run
instead of hiding the examples listed below it. Both runs this round
aborted early and hid nine Python and two TypeScript files.
This commit is contained in:
Zecheng Zhang
2026-08-16 22:48:27 -07:00
parent b810ff931f
commit eb1c404609
5 changed files with 40 additions and 7 deletions
+9 -2
View File
@@ -97,12 +97,18 @@ jobs:
run: uv run pytest --ignore=tests/agents/camel
# Each truth file owns the command it was captured against, so this
# list is paths only and cannot drift from what is asserted.
# list is paths only and cannot drift from what is asserted. The
# runner keeps going after a failure and fails the step at the end,
# so one stale truth file reports every other one in the same run
# instead of hiding the examples listed below it.
- name: Examples (output checked against integ/ truth files)
env:
REDIS_URL: redis://localhost:6379/0
run: |
run() { python3 integ/check_example.py "$1" --variant python; }
bad=0
run() {
python3 integ/check_example.py "$1" --variant python || bad=1
}
run integ/truth/python/ram.json
run integ/truth/python/ram_vfs.json
run integ/truth/python/ram_python.json
@@ -116,6 +122,7 @@ jobs:
run integ/truth/python/redis_cache.json
run integ/truth/python/redis_vfs.json
run integ/truth/version_branching.json
exit $bad
- name: Sync Python dependencies for camel
working-directory: python
+9 -2
View File
@@ -92,12 +92,18 @@ jobs:
run: pnpm test
# Each truth file owns the command it was captured against, so this
# list is paths only and cannot drift from what is asserted.
# list is paths only and cannot drift from what is asserted. The
# runner keeps going after a failure and fails the step at the end,
# so one stale truth file reports every other one in the same run
# instead of hiding the examples listed below it.
- name: Examples (output checked against integ/ truth files)
env:
REDIS_URL: redis://localhost:6379/0
run: |
run() { python3 integ/check_example.py "$1" --variant typescript; }
bad=0
run() {
python3 integ/check_example.py "$1" --variant typescript || bad=1
}
run integ/truth/typescript/ram.json
run integ/truth/typescript/ram_vfs.json
run integ/truth/typescript/disk.json
@@ -115,6 +121,7 @@ jobs:
run integ/truth/typescript/redis_cache.json
run integ/truth/typescript/redis_vfs.json
run integ/truth/version_branching.json
exit $bad
python-fs:
needs: changes
+13
View File
@@ -37,6 +37,19 @@ Line matchers, in the order of preference an author should reach for:
differs between the runs becomes a `re` with digit runs generalized, or
a `volatile` when even that does not hold. Tighten a `volatile` by hand
into an `re` whenever the line has a stable shape worth pinning.
Both of those runs happen on ONE machine, so `--emit` sees run-to-run
volatility and is blind to host-to-host volatility: a line that is
stable on the author's Mac and different on CI's Ubuntu is emitted as a
literal. Three classes of line have to be widened by hand after an
emit, all of which reached CI as literals before this was written down:
a temp root mkdtemp is /var/folders/... on macOS, /tmp on Linux
a checkout mtime actions/checkout stamps the repo files on clone
an env value REDIS_URL is db 1 locally and db 0 on the runners
The rule of thumb: a literal earns its place when the example itself
determines it, and has to become an `re` when the machine does.
"""
import argparse
+6 -2
View File
@@ -38,7 +38,9 @@
" 257 1370 15533 /data/example.json",
"",
"=== stat /data/example.json ===",
"name=example.json size=15533 modified=2026-08-17T03:59:32Z type=json",
{
"re": "name=example\\.json size=15533 modified=\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}Z type=json"
},
"",
"=== tree /data/ ===",
"/data/",
@@ -134,7 +136,9 @@
"example.jsonl",
"example.orc",
"example.parque…",
" loaded (root=/var/folders/4d/s_lrmhbd7xb9jglnmy13sz_w…) ls: example.feather",
{
"re": " loaded \\(root=[^)]+…\\) ls: example\\.feather"
},
"example.h5",
"example.json",
"example.jsonl",
+3 -1
View File
@@ -15,7 +15,9 @@
"lines": [
"=== RedisFileCacheStore: FileCache backed by Redis ===",
"",
" url: redis://localhost:6379/1",
{
"re": " url: redis://localhost:6379/\\d+"
},
" keyPrefix: mirage:example:cache:",
" cacheLimit: 67108864 bytes",
"",