chore: cleanup items 31+32 — nested-function rule, layout, CI audit, ls facets

Closes the two remaining small items of Block E. Both were largely
recon: much of what they filed had already been fixed, and the part
that had not turned up a real GNU divergence.

Item 31 (T3-7), rule/doc reconciliation
---------------------------------------

The `object`-parameter half is already done: #825's mypy inversion took
the package from the filed 28 down to 7, of which 3 are docstring prose
and the remaining 4 are protocol methods (`__contains__`, `pop`) whose
signatures typeshed dictates, all already listed in the no-object gate.
`utils/errors.py` reads `str | PathSpec` today, and `builders/sed.py` no
longer holds flag values at all -- item 22 moved that to the generic.

That left the nested-function rule, which the plan asked to decide
before gating. Measured: 39 nested defs, and **every one of them closes
over its enclosing scope** -- 38 by free variable, and `sed_helper._repl`
by binding through parameter defaults, the loop-variable idiom, with a
comment saying so. A flat "do not nest" is a rule the architecture
cannot keep (op factories, provision builders, the read-through cache,
every decorator's wrapper), so gating it as written would have meant 39
standing violations.

So the rule now says what it was reaching for: a nested def must capture
the scope around it, or it belongs at module level. That is mechanical
and enforceable -- `tests/test_nested_functions_are_closures.py` reads
free variables from `symtable` and parameter defaults from the AST. It
passes on the tree as-is and fires on a helper that reads only its own
arguments.

Layout: the eight test-only TS directories are flattened
(`ram/{cat,cut,grep,head,ls,tail,wc}/`, `ssh/ls/`), `awk_helper.ts` moves
up beside its nine siblings, and the `provision.ts` / `_provision.ts`
split is settled on Python's convention -- `_provision` for a backend's
own, bare `provision` for the shared ones (cli, generic_bind). That last
one was not cosmetic: Python already had `_provision.py` for github,
gmail, redis and email, so renaming **closed 4 real parity divergences**
(layout baseline 296 -> 292). The `findEval`/`findParse` line was
already done in #827.

Item 32 (T3-8), test/CI tidiness
--------------------------------

The two asymmetric `ls` conformance matrices are raised to symmetry, and
they pass: python `[ram,disk,redis]` vs typescript `[ram]` was stale
caution, not a divergence. Both runners now reject an asymmetric matrix
at load time unless the case carries a `divergence` key explaining why --
a case is a parity claim, and narrowing one side reads as coverage while
the side that still lists the backend goes green. Loading the corpus
under the new assertion proves no other case was asymmetric. The README
documented the override as "not yet needed"; it exists now.

Adds the `ts-audit` job, mirroring `test_python.yml`'s exactly (same
`continue-on-error`, same out-of-gate placement). Nothing ran `pnpm
audit` for the TS tree while `typescript/package.json` carried 27
hand-written CVE overrides that only a person remembering to check kept
current.

Integ facets for `ls`: `-A`, `-d`, `-r`, `-S`, `-h`, all pinned against
GNU coreutils 9.7 in docker. `-S` uses regular files only, because GNU
sorts a directory by its inode size while mirage counts it as 0 -- a
divergence CLAUDE.md documents deliberately.

The `-h` facet found a real bug
-------------------------------

`ls -h` disagreed with GNU three ways, and because the flag had zero
cross-backend coverage nothing caught it. GNU prints a count below one
unit with no suffix (`24`); mirage printed `24B`. GNU rounds *up* to the
precision shown (1025 bytes is `1.1K`); mirage gave `1.0K`. GNU drops
the decimal once the value reaches ten (`10K`); mirage gave `10.0K`.

One shared engine replaces both formatters in both languages -- GNU runs
`-h` and `-H` through one `human_readable`, and so do we now. Rounding
up can carry past the base (1048575 ceils to 1024K, which GNU shows as
`1.0M`), so the unit is re-chosen after rounding rather than once up
front. 21 GNU-read points are pinned as a table in each language.

Blast radius, all re-pinned: five integ cases that had recorded the `B`
suffix (du, discord, langfuse, email, gmail), and the du fan-out tests.
Those last ones needed care rather than a new number: they exist to
prove the total is humanized once instead of twice, and their 1500+1500
stops discriminating under correct rounding, since 3000 bytes and two
`1.5K` readings both render `3.0K`. They now use 1025+1025, where
single-rounding gives `2.1K` and double-rounding would give `2.2K`.

Verified: pre-commit clean, integ 8054/0 on ram+disk+redis in both
languages, conformance green in both, and the layout, spec, barrel,
docs, case-target and PathSpec gates all pass.

Not covered, still open on item 32: integ facets for `ls -t` (mtime
order is not stable across backends without a seeded fixture), mktemp,
gzip, the checksum `--check` companions, `df -H/-k/-a`, and `cmp
-n/-b/-i`; and the T2-7 helper unit-suite mirroring. Item 31 leaves the
provision *presence* sets unreconciled (Python has lancedb/qdrant, TS has
trello) -- that is a question about which backends should provision at
all, not about naming.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
bytecii
2026-08-16 06:57:08 -07:00
parent e4d72adeb1
commit 73f5bd7c21
57 changed files with 837 additions and 141 deletions
+29
View File
@@ -150,6 +150,35 @@ jobs:
node --import tsx/esm pyodide/vfs.ts 2>&1 \
| bash ../../integ/check_lines.sh ../../integ/truth/typescript/pyodide_vfs.txt
audit:
needs: changes
if: ${{ !cancelled() && (github.event_name != 'pull_request' || needs.changes.outputs.hit == 'true') }}
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v7
- name: Install pnpm
uses: pnpm/action-setup@v6.0.10
with:
version: 10.32.1
- name: Set up Node
uses: actions/setup-node@v7
with:
node-version: "22"
cache: pnpm
cache-dependency-path: typescript/pnpm-lock.yaml
# Advisories read against the resolved lockfile, which is what the 27
# hand-written pnpm.overrides in typescript/package.json pin. Those
# pins were the only thing between a published advisory and this
# tree, and nothing re-checked them. Informational like the Python
# audit: a fresh CVE reports without blocking a merge.
- name: Audit dependencies for known vulnerabilities (informational)
working-directory: typescript
run: pnpm audit --no-color
gate:
name: test-typescript-gate
runs-on: ubuntu-latest
+1 -1
View File
@@ -619,7 +619,7 @@ Invoke the venv's `pre-commit` binary directly (not via `uv --directory python r
- **Do not add `__init__.py` files under `tests/`.** Tests are namespace packages and pytest discovers them without `__init__.py`. Don't create one when adding a new test directory.
- **Monkeypatching a backend command module in tests:** the command imports its helpers by value (`from mirage.core.<backend>.read import read_bytes`), so to intercept them you must rebind the name inside the command module, not the core source module. But the command module is hard to reach: the backend package re-exports the command function in `__init__.py` (`from .cat import cat`), which shadows the submodule of the same name, so `import mirage.commands.builtin.<backend>.cat as mod`, `from ...<backend> import cat as mod`, and even pytest's string target `monkeypatch.setattr("mirage.commands.builtin.<backend>.cat.read_bytes", fake)` all resolve to the function, not the module (`AttributeError`). The command is also wrapped by `@command`, so `cat.__globals__` is the decorator's module. Reach the real command-module namespace through the unwrapped function and patch the dict: `monkeypatch.setitem(cat.__wrapped__.__globals__, "read_bytes", fake)`.
- Avoid add any comments or docstrings on the top of the file.
- Do not create nested functions.
- **A nested function must capture the scope around it.** Nesting is for closures: the def reads a name from the enclosing function, or binds one through a parameter default (`def f(m, _n=n)`, the loop-variable idiom). Everything the architecture nests is that shape — op factories (`ops/generic/factory.py`), provision builders, the read-through cache, and every decorator's wrapper — and those stay. What is banned is the nesting that buys nothing: a helper written inside a function although it reads only its own arguments, which rebuilds a function object per call and hides a testable unit where no test can reach it. Put that one at module level. Enforced for Python by `tests/test_nested_functions_are_closures.py`.
- Add type to Args for docstring.
- Do not add comment after each line of code in the format of "# 10MB - trigger segmentation for files larger than this". The most you can add is "# 10MB".
- For all imports you need to put to the top of the file. Don't have imports within each function.
+10 -2
View File
@@ -57,13 +57,21 @@ CI runs it as the `integ-shared-parity` job.
- `matrix` is explicit: a case runs only on the listed backends. Listing a
backend is a claim of support — a missing command there is a failure, not a
skip. A case whose matrix is empty is a load-time error in both runners.
- The two languages must list the same backends. A case is a parity claim, so
narrowing one side reads as coverage while it is really an unexamined
divergence — and it goes unnoticed, because the side that still lists the
backend passes. Both runners reject an asymmetric matrix at load time.
- `divergence` is the override: a string saying why one language cannot run
the case everywhere the other does. Setting it permits an asymmetric matrix,
and it is the only thing that does.
## Policy
- One expected value per case. If a backend or language legitimately diverges,
that divergence is triaged first: either it is a bug (fix the
implementation) or it is intended semantics (document it and add an explicit
per-backend override mechanism — not yet needed).
implementation) or it is intended semantics, recorded in the case's
`divergence` key so the narrowing is stated rather than inferred from a
short matrix.
- This spec is an acceptance/parity net, not a replacement for backend tests.
API call counts, pushdown/fallback, cache invalidation, error injection, and
concurrency stay in hand-written per-backend tests.
+6 -2
View File
@@ -53,7 +53,9 @@
"redis"
],
"typescript": [
"ram"
"ram",
"disk",
"redis"
]
},
"expect": {
@@ -72,7 +74,9 @@
"redis"
],
"typescript": [
"ram"
"ram",
"disk",
"redis"
]
},
"expect": {
+1 -1
View File
@@ -230,7 +230,7 @@
"command": "du -sh '/discord/Mirage HQ__100000000000000001/channels/general__200000000000000001/2026-06-01/files'",
"expect": {
"exit": 0,
"stdout": "29B\t/discord/Mirage HQ__100000000000000001/channels/general__200000000000000001/2026-06-01/files\n",
"stdout": "29\t/discord/Mirage HQ__100000000000000001/channels/general__200000000000000001/2026-06-01/files\n",
"stderr": ""
}
},
+1 -1
View File
@@ -522,7 +522,7 @@
"command": "du -h /mail/INBOX/2026-01-05/Q2_Budget_Review__1/budget.csv",
"expect": {
"exit": 0,
"stdout": "36B\t/mail/INBOX/2026-01-05/Q2_Budget_Review__1/budget.csv\n",
"stdout": "36\t/mail/INBOX/2026-01-05/Q2_Budget_Review__1/budget.csv\n",
"stderr": ""
}
},
+1 -1
View File
@@ -510,7 +510,7 @@
"command": "du -h /mail/INBOX/2026-01-05/Q2_Budget_Review__msg0001/budget.csv",
"expect": {
"exit": 0,
"stdout": "36B\t/mail/INBOX/2026-01-05/Q2_Budget_Review__msg0001/budget.csv\n",
"stdout": "36\t/mail/INBOX/2026-01-05/Q2_Budget_Review__msg0001/budget.csv\n",
"stderr": ""
}
},
@@ -282,7 +282,7 @@
"command": "du -sh /lf/traces",
"expect": {
"exit": 0,
"stdout": "0B\t/lf/traces\n",
"stdout": "0\t/lf/traces\n",
"stderr": ""
}
},
+1 -1
View File
@@ -30,7 +30,7 @@
"command": "du -h /data/a.txt",
"expect": {
"exit": 0,
"stdout": "24B\t/data/a.txt\n",
"stdout": "24\t/data/a.txt\n",
"stderr": ""
},
"flags": [
+79
View File
@@ -0,0 +1,79 @@
{
"cases": [
{
"id": "ls_almost_all",
"seq": 353,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -A /data/sub",
"expect": {
"exit": 0,
"stdout": "deep\nnested.txt\n",
"stderr": ""
},
"flags": [
"A"
]
},
{
"id": "ls_almost_all_matches_all",
"seq": 960061,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "[ \"$(ls -A /data/sub)\" = \"$(ls -a /data/sub)\" ] && echo same",
"expect": {
"exit": 0,
"stdout": "same\n",
"stderr": ""
},
"flags": [
"A",
"a"
]
}
]
}
+78
View File
@@ -0,0 +1,78 @@
{
"cases": [
{
"id": "ls_d_directory_itself",
"seq": 960070,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -d /data/sub",
"expect": {
"exit": 0,
"stdout": "/data/sub\n",
"stderr": ""
},
"flags": [
"d"
]
},
{
"id": "ls_d_keeps_the_typed_slash",
"seq": 960071,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -d /data/sub/",
"expect": {
"exit": 0,
"stdout": "/data/sub/\n",
"stderr": ""
},
"flags": [
"d"
]
}
]
}
+80
View File
@@ -0,0 +1,80 @@
{
"cases": [
{
"id": "ls_h_sub_kilobyte_has_no_suffix",
"seq": 960090,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -lh /data/a.txt | awk '{print $5}'",
"expect": {
"exit": 0,
"stdout": "24\n",
"stderr": ""
},
"flags": [
"l",
"h"
]
},
{
"id": "ls_h_matches_plain_size_under_1k",
"seq": 960091,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "[ \"$(ls -lh /data/a.txt | awk '{print $5}')\" = \"$(ls -l /data/a.txt | awk '{print $5}')\" ] && echo same",
"expect": {
"exit": 0,
"stdout": "same\n",
"stderr": ""
},
"flags": [
"l",
"h"
]
}
]
}
+119
View File
@@ -0,0 +1,119 @@
{
"cases": [
{
"id": "ls_r_reverses_name_order",
"seq": 354,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -1r /data/sub",
"expect": {
"exit": 0,
"stdout": "nested.txt\ndeep\n",
"stderr": ""
},
"flags": [
"1",
"r"
]
},
{
"id": "ls_S_sorts_by_size_desc",
"seq": 960081,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -1S /data/a.txt /data/b.txt /data/c.txt /data/one_byte.txt",
"expect": {
"exit": 0,
"stdout": "/data/a.txt\n/data/c.txt\n/data/b.txt\n/data/one_byte.txt\n",
"stderr": ""
},
"flags": [
"1",
"S"
]
},
{
"id": "ls_rS_reverses_the_size_sort",
"seq": 960082,
"targets": [
"ram",
"disk",
"redis",
"opfs",
"s3",
"databricks",
"gridfs",
"s3-prefix",
"databricks-prefix",
"gridfs-prefix",
"hf",
"hf-prefix",
"dropbox",
"dropbox-root",
"onedrive",
"sharepoint",
"sharepoint-prefix",
"ssh",
"nextcloud",
"gdrive",
"gdrive-folder",
"box"
],
"command": "ls -1rS /data/a.txt /data/b.txt /data/c.txt /data/one_byte.txt",
"expect": {
"exit": 0,
"stdout": "/data/one_byte.txt\n/data/b.txt\n/data/c.txt\n/data/a.txt\n",
"stderr": ""
},
"flags": [
"1",
"r",
"S"
]
}
]
}
@@ -21,15 +21,43 @@ from mirage.commands.builtin.utils.constants import (DEFAULT_MODES,
from mirage.types import LINK_TARGET_KEY, FileStat, FileType
def _human_size(n: int) -> str:
units = ("B", "K", "M", "G", "T")
value = float(n)
i = 0
while value >= 1024 and i < len(units) - 1:
value /= 1024
def human_scaled(n: int, base: int, units: tuple[str, ...]) -> str:
"""GNU's ``human_readable`` rounding, shared by ``-h`` and ``-H``.
Three rules, none of which fall out of a plain divide-and-format.
Below one unit GNU prints the count alone -- ``24``, never ``24B``.
Above it the value is rounded *up* to the precision shown, so 1025
bytes is ``1.1K`` rather than ``1.0K``. And the decimal is dropped
once the scaled value reaches ten, giving ``10K`` rather than
``10.0K``. Rounding up can carry past the base (1048575 bytes ceils
to 1024K, which GNU shows as ``1.0M``), so the unit is re-chosen
after rounding instead of once up front.
Args:
n (int): byte count.
base (int): 1024 for ``-h``, 1000 for ``-H``.
units (tuple[str, ...]): suffixes indexed by power; index 0 is
unused because a sub-unit count carries no suffix at all.
Returns:
str: the size as GNU would print it.
"""
if n < base:
return str(n)
i, divisor = 1, base
while True:
tenths = -(-n * 10 // divisor)
if tenths < 100:
return f"{tenths // 10}.{tenths % 10}{units[i]}"
whole = -(-n // divisor)
if whole < base or i == len(units) - 1:
return f"{whole}{units[i]}"
i += 1
text = str(round(value)) if i == 0 else f"{value:.1f}"
return f"{text}{units[i]}"
divisor *= base
def _human_size(n: int) -> str:
return human_scaled(n, 1024, ("", "K", "M", "G", "T", "P", "E"))
def _perm_triplet(bits: int, special: str | None = None) -> str:
@@ -14,7 +14,7 @@
import math
from mirage.commands.builtin.utils.formatting import _human_size
from mirage.commands.builtin.utils.formatting import _human_size, human_scaled
from mirage.runtime.types import DispatchFn
from mirage.types import CapacityResult, CapacityState, PathSpec
from mirage.utils.path import resolve_path
@@ -26,7 +26,7 @@ from mirage.workspace.mount.registry import MountRegistry
from mirage.workspace.session import Session
_BLOCK_SUFFIX = {"K": 1024, "M": 1024**2, "G": 1024**3, "T": 1024**4}
_SI_UNITS = ("B", "K", "M", "G", "T")
_SI_UNITS = ("", "K", "M", "G", "T", "P", "E")
def _parse_block(text: str) -> tuple[int, str] | None:
@@ -87,19 +87,17 @@ def _last_format(args: list[str | PathSpec]) -> str | None:
def _human_si(n: int) -> str:
"""Human-readable size in powers of 1000 (df -H), mirroring the 1024
``_human_size`` shape used by df -h / du -h.
"""Human-readable size in powers of 1000 (df -H).
Same rounding as ``-h``; GNU runs both through one ``human_readable``.
Args:
n (int): byte count.
Returns:
str: the size as GNU would print it.
"""
value = float(n)
i = 0
while value >= 1000 and i < len(_SI_UNITS) - 1:
value /= 1000
i += 1
text = str(round(value)) if i == 0 else f"{value:.1f}"
return f"{text}{_SI_UNITS[i]}"
return human_scaled(n, 1000, _SI_UNITS)
def _scale(nbytes: int, block: int) -> str:
@@ -41,19 +41,21 @@ def testdu_total_strips_per_run_totals_and_sums():
def testdu_total_humanizes_from_exact_bytes_without_rounding_twice():
# run_fanout forces -h off on the native runs, so the rows arrive in
# bytes: 1500 + 1500 is 2.9K, not the 3.0K that summing two "1.5K"
# readings back through parse_size would give.
# bytes: 1025 + 1025 is 2.1K, not the 2.2K that summing two "1.1K"
# readings back through parse_size would give. 1500 would not show
# the difference -- GNU rounds up, so 3000 bytes and two 1.5K
# readings both render 3.0K.
runs = [
_op(b"1500\t/a/x\n1500\ttotal\n"),
_op(b"1500\t/b/z\n1500\ttotal\n"),
_op(b"1025\t/a/x\n1025\ttotal\n"),
_op(b"1025\t/b/z\n1025\ttotal\n"),
]
out = du_total(runs, human=True).decode()
assert out == "1.5K\t/a/x\n1.5K\t/b/z\n2.9K\ttotal\n"
assert out == "1.1K\t/a/x\n1.1K\t/b/z\n2.1K\ttotal\n"
def testdu_total_leaves_a_row_without_a_tab_alone():
out = du_total([_op(b"odd-row\n0\ttotal\n")], human=True).decode()
assert out == "odd-row\n0B\ttotal\n"
assert out == "odd-row\n0\ttotal\n"
def testmerge_du_totals_takes_rendered_blocks():
@@ -12,22 +12,46 @@
# limitations under the License.
# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import pytest
from mirage.commands.builtin.utils.formatting import (_human_size,
format_ls_long)
from mirage.types import FileStat, FileType
def test_human_size_bytes():
assert _human_size(500) == "500B"
# Read off GNU coreutils 9.7 (`ls -lh` on a file of each size, debian
# stable-slim). The three rows that matter are the ones a plain
# divide-and-format gets wrong: no suffix under 1024, rounding *up* to
# the shown precision (1025 -> 1.1K), and the decimal dropping once the
# value reaches ten (10240 -> 10K). 1048575 pins the carry: it ceils to
# 1024K, which GNU re-scales to 1.0M.
GNU_HUMAN_SIZES = [
(0, "0"),
(1, "1"),
(24, "24"),
(500, "500"),
(999, "999"),
(1000, "1000"),
(1023, "1023"),
(1024, "1.0K"),
(1025, "1.1K"),
(1126, "1.1K"),
(1127, "1.2K"),
(1536, "1.5K"),
(2048, "2.0K"),
(10188, "10K"),
(10240, "10K"),
(10241, "11K"),
(11263, "11K"),
(1048575, "1.0M"),
(1048576, "1.0M"),
(1024 * 1024 + 512 * 1024, "1.5M"),
(1073741824, "1.0G"),
]
def test_human_size_kilobytes():
assert _human_size(1024) == "1.0K"
def test_human_size_fractional_rounds_not_floored():
assert _human_size(1536) == "1.5K"
assert _human_size(1024 * 1024 + 512 * 1024) == "1.5M"
@pytest.mark.parametrize(("size", "expected"), GNU_HUMAN_SIZES)
def test_human_size_matches_gnu(size: int, expected: str):
assert _human_size(size) == expected
def test_format_ls_long_regular_file():
@@ -74,6 +74,26 @@ def _validate_matrix(case: dict, spec_name: str) -> None:
raise ValueError(
f"case {case['id']} in {spec_name} applies to no backend")
# A case is a parity claim, so the two languages have to be asked the
# same question. Dropping a backend from one side reads as coverage
# while it is really an unexamined divergence -- and it is invisible,
# because the side that still lists the backend goes green. Anything
# genuinely language-specific says so in a `divergence` key, which the
# README calls the per-backend override.
if "divergence" not in case:
python_backends = set(matrix.get("python", []))
typescript_backends = set(matrix.get("typescript", []))
if python_backends != typescript_backends:
only_python = ", ".join(
sorted(python_backends - typescript_backends)) or "none"
only_typescript = ", ".join(
sorted(typescript_backends - python_backends)) or "none"
raise ValueError(
f"case {case['id']} in {spec_name} has an asymmetric "
f"matrix (python-only: {only_python}; typescript-only: "
f"{only_typescript}). Run it on both, or record why it "
f"cannot with a `divergence` key.")
def _load_cases() -> list[dict]:
cases = []
@@ -185,3 +205,27 @@ def test_validate_matrix_accepts_supported_targets() -> None:
},
}
_validate_matrix(case, "valid.json")
def test_validate_matrix_rejects_an_asymmetric_matrix() -> None:
case = {
"id": "narrowed_matrix",
"matrix": {
"python": ["ram", "disk", "redis"],
"typescript": ["ram"],
},
}
with pytest.raises(ValueError, match="asymmetric matrix"):
_validate_matrix(case, "narrowed.json")
def test_validate_matrix_allows_an_asymmetric_matrix_that_says_why() -> None:
case = {
"id": "declared_divergence",
"matrix": {
"python": ["ram", "disk", "redis"],
"typescript": ["ram"],
},
"divergence": "TypeScript has no redis-backed foo yet (#1234).",
}
_validate_matrix(case, "declared.json")
@@ -0,0 +1,107 @@
# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import ast
import symtable
from pathlib import Path
SOURCE = Path(__file__).resolve().parents[1] / "mirage"
# symtable gives every comprehension and lambda a scope of type "function"
# under one of these reserved names. They are expressions, not definitions,
# and the rule is about `def`.
SYNTHETIC = frozenset({"genexpr", "lambda", "listcomp", "setcomp", "dictcomp"})
def _defs_by_position(tree: ast.Module) -> dict[tuple[str, int], ast.AST]:
found: dict[tuple[str, int], ast.AST] = {}
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
found[(node.name, node.lineno)] = node
return found
def _binds_through_defaults(node: ast.AST, enclosing: frozenset[str]) -> bool:
"""Whether a def captures enclosing state through its parameter defaults.
The loop-variable idiom (``def f(m, _n=n)``) binds the enclosing value
at definition time instead of reading it at call time, which is the
whole point when the def is created inside a loop. symtable sees no
free variable, because by the time the body runs the name is a
parameter -- but the closure is real and the capture is deliberate.
Args:
node: The nested function definition.
enclosing: Names local to the function that contains it.
Returns:
True when a default expression reads an enclosing local.
"""
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
return False
defaults = [
*node.args.defaults,
*[d for d in node.args.kw_defaults if d is not None]
]
return any(
isinstance(name, ast.Name) and name.id in enclosing
for default in defaults for name in ast.walk(default))
def _offenders(path: Path, rel: str) -> list[str]:
source = path.read_text()
positions = _defs_by_position(ast.parse(source))
found: list[str] = []
def visit(table: symtable.SymbolTable, enclosing: frozenset[str] | None):
for child in table.get_children():
is_def = (child.get_type() == "function"
and child.get_name() not in SYNTHETIC)
if is_def and enclosing is not None and not child.get_frees():
node = positions.get((child.get_name(), child.get_lineno()))
if node is None or not _binds_through_defaults(
node, enclosing):
found.append(
f"{rel}:{child.get_lineno()} {child.get_name()}")
locals_ = frozenset(child.get_locals()) if is_def else enclosing
visit(child, locals_ if is_def else enclosing)
visit(symtable.symtable(source, str(path), "exec"), None)
return found
def test_nested_functions_capture_enclosing_state():
"""A nested def exists to close over its enclosing scope, or not at all.
The flat rule -- never nest -- is one the architecture cannot keep:
every op factory, every provision builder, and the read-through cache
are closure factories, and a decorator has nowhere else to put its
wrapper. Forbidding the shape outright would mean 39 standing
violations and a rule nobody could enforce.
What the rule was reaching for is the nesting that buys nothing: a
helper written inside a function although it reads only its own
arguments. That one costs a rebuilt function object per call, hides a
testable unit inside a scope no test can reach, and grows the body it
sits in. Capture is the line between the two, and it is mechanical:
either the def reads a name from the scope above it (or binds one
through a parameter default), or it belongs at module level.
"""
offenders = []
for path in sorted(SOURCE.rglob("*.py")):
offenders.extend(_offenders(path, path.relative_to(SOURCE).as_posix()))
assert not offenders, (
"these nested functions capture nothing from the scope around "
"them, so nesting them only hides them -- move them to module "
"level:\n" + "\n".join(offenders))
@@ -326,11 +326,13 @@ def test_du_sc_fanout_prints_one_total():
def test_du_ch_fanout_humanizes_the_total_once():
# Summing each mount's already-humanized total would round twice and
# report 3.0K; the sub-runs render exact bytes and only the merge
# humanizes.
ws = _shadowed_workspace(top=1500, real=1500)
# report 2.2K; the sub-runs render exact bytes and only the merge
# humanizes. 1025 bytes rather than 1500 because GNU rounds up: 1500
# doubles to 3000, which single- and double-rounding both render
# 3.0K, so those sizes could no longer tell the two apart.
ws = _shadowed_workspace(top=1025, real=1025)
io = asyncio.run(ws.execute("du -ch /base"))
assert _stdout(io) == "1.5K\t/base/inner\n2.9K\t/base\n2.9K\ttotal\n"
assert _stdout(io) == "1.1K\t/base/inner\n2.1K\t/base\n2.1K\ttotal\n"
def test_du_max_depth_prunes_printing_not_accounting():
+1 -1
View File
@@ -1,5 +1,5 @@
{
"baseline": 296,
"baseline": 292,
"baseline_reason": "Every divergence below the excused ones predates the gate and each needs its own decision, so --strict fails on a rise rather than demanding zero. Lower this number whenever a divergence is closed; the gate fails on a drop too, so an improvement cannot be silently spent. Items 31-37 of the cleanup plan are scoped from this report. The unit is one module, never one directory: a one-sided directory counts once per module inside it, so it cannot absorb new modules without moving the number. An excused directory is the deliberate exception -- it excuses its whole subtree, because the excuse is that there is no counterpart to mirror, which makes growth inside it expected rather than drift.",
"directories": {
"python_only": {
@@ -12,9 +12,9 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { AsyncLineIterator } from '../../../io/async_line_iterator.ts'
import { UsageError } from '../../errors.ts'
import { toNumber } from '../utils/formatting.ts'
import { AsyncLineIterator } from '../../io/async_line_iterator.ts'
import { UsageError } from '../errors.ts'
import { toNumber } from './utils/formatting.ts'
import {
AwkBlock,
AwkBoolOp,
@@ -23,7 +23,7 @@ import {
CMP_OP_PATTERN,
FIELD_PREFIX,
PRINT_STMT,
} from './awk_types.ts'
} from './generic/awk_types.ts'
const ENC = new TextEncoder()
const DEC = new TextDecoder('utf-8', { fatal: false })
@@ -17,7 +17,7 @@ import { ResourceName } from '../../../types.ts'
import type { ProvisionFn, RegisteredCommand } from '../../config.ts'
import { makeGenericCommands } from '../generic_bind/index.ts'
import { GDOCS_IO } from './io.ts'
import { fileReadProvision } from './provision.ts'
import { fileReadProvision } from './_provision.ts'
import { GDOCS_RM } from './rm.ts'
// Docs verbs and API passthroughs live in the gws CLI
@@ -18,7 +18,7 @@ import { mountKey, mountPrefixOf } from '../../../utils/key_prefix.ts'
import { IOResult, materialize } from '../../../io/types.ts'
import { PathSpec } from '../../../types.ts'
import type { CommandFnResult, CommandOpts } from '../../config.ts'
import { awkStream, validateAwkProgram } from './awk_helper.ts'
import { awkStream, validateAwkProgram } from '../awk_helper.ts'
import { USAGE, type AwkFlags } from './awk_types.ts'
import { isMissingPath } from '../../../utils/errors.ts'
import { resolvePath } from '../../../utils/path.ts'
@@ -46,16 +46,18 @@ describe('duTotal', () => {
it('humanizes from exact bytes without rounding twice', () => {
// runFanout forces -h off on the native runs, so the rows arrive in
// bytes: 1500 + 1500 is 2.9K, not the 3.0K that summing two "1.5K"
// readings back through parseSize would give.
// bytes: 1025 + 1025 is 2.1K, not the 2.2K that summing two "1.1K"
// readings back through parseSize would give. 1500 would not show
// the difference -- GNU rounds up, so 3000 bytes and two 1.5K
// readings both render 3.0K.
const out = DEC.decode(
duTotal([op('1500\t/a/x\n1500\ttotal\n'), op('1500\t/b/z\n1500\ttotal\n')], true),
duTotal([op('1025\t/a/x\n1025\ttotal\n'), op('1025\t/b/z\n1025\ttotal\n')], true),
)
expect(out).toBe('1.5K\t/a/x\n1.5K\t/b/z\n2.9K\ttotal\n')
expect(out).toBe('1.1K\t/a/x\n1.1K\t/b/z\n2.1K\ttotal\n')
})
it('leaves a row without a tab alone', () => {
expect(DEC.decode(duTotal([op('odd-row\n0\ttotal\n')], true))).toBe('odd-row\n0B\ttotal\n')
expect(DEC.decode(duTotal([op('odd-row\n0\ttotal\n')], true))).toBe('odd-row\n0\ttotal\n')
})
})
@@ -167,10 +167,14 @@ describe('du walk fallback (no native du op)', () => {
).rejects.toThrow('403 Forbidden')
})
// GNU prints a count below one unit with no suffix at all, so -h and
// the plain form agree on this tree. The scaling and rounding rules
// are pinned against GNU in utils/utils.test.ts; here -h only has to
// reach the formatter.
it('-h renders human-readable sizes', async () => {
expect(await runDu([PathSpec.fromStrPath('/db')], { h: true })).toEqual([
'2B\t/db/sub',
'5B\t/db',
'2\t/db/sub',
'5\t/db',
])
})
})
@@ -19,7 +19,7 @@ import { GITHUB_IO } from './io.ts'
import { ResourceName, type PathSpec } from '../../../types.ts'
import { command, type CommandFnResult, type CommandOpts } from '../../config.ts'
import { specOf } from '../../spec/builtins.ts'
import { metadataProvision } from './provision.ts'
import { metadataProvision } from './_provision.ts'
import { IOResult } from '../../../io/types.ts'
import { runDu } from '../generic/du.ts'
@@ -18,7 +18,7 @@ import { ResourceName, type PathSpec } from '../../../types.ts'
import { command, type CommandFnResult, type CommandOpts } from '../../config.ts'
import { specOf } from '../../spec/builtins.ts'
import { findGeneric } from '../generic/find.ts'
import { metadataProvision } from './provision.ts'
import { metadataProvision } from './_provision.ts'
async function findCommand(
accessor: GitHubAccessor,
@@ -20,7 +20,7 @@ import { GITHUB_DU } from './du.ts'
import { GITHUB_FIND } from './find.ts'
import { GITHUB_GREP } from './grep.ts'
import { GITHUB_IO } from './io.ts'
import { metadataProvision } from './provision.ts'
import { metadataProvision } from './_provision.ts'
import { GITHUB_RG } from './rg.ts'
const GITHUB_OVERRIDES = new Set(['du', 'find', 'grep', 'rg'])
@@ -28,7 +28,7 @@ import { command, type CommandFnResult, type CommandOpts } from '../../config.ts
import { specOf } from '../../spec/builtins.ts'
import { grepGeneric } from '../generic/grep.ts'
import { patternArg } from '../grep_helper.ts'
import { fileReadProvision } from './provision.ts'
import { fileReadProvision } from './_provision.ts'
import { FlagView } from '../../spec/types.ts'
const resolveGlob = resolveGlobOf(GMAIL_IO)
@@ -18,7 +18,7 @@ import type { ProvisionFn, RegisteredCommand } from '../../config.ts'
import { makeGenericCommands } from '../generic_bind/index.ts'
import { GMAIL_GREP } from './grep.ts'
import { GMAIL_IO } from './io.ts'
import { metadataProvision } from './provision.ts'
import { metadataProvision } from './_provision.ts'
import { GMAIL_RG } from './rg.ts'
const GMAIL_OVERRIDES = new Set(['grep', 'rg'])
@@ -17,7 +17,7 @@ import { ResourceName } from '../../../types.ts'
import type { ProvisionFn, RegisteredCommand } from '../../config.ts'
import { makeGenericCommands } from '../generic_bind/index.ts'
import { GSHEETS_IO } from './io.ts'
import { fileReadProvision } from './provision.ts'
import { fileReadProvision } from './_provision.ts'
import { GSHEETS_RM } from './rm.ts'
// Sheets verbs and API passthroughs live in the gws CLI
@@ -17,7 +17,7 @@ import { ResourceName } from '../../../types.ts'
import type { ProvisionFn, RegisteredCommand } from '../../config.ts'
import { makeGenericCommands } from '../generic_bind/index.ts'
import { GSLIDES_IO } from './io.ts'
import { fileReadProvision } from './provision.ts'
import { fileReadProvision } from './_provision.ts'
import { GSLIDES_RM } from './rm.ts'
// Slides API passthroughs live in the gws CLI
@@ -12,11 +12,11 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { PathSpec } from '../../../../types.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
import { PathSpec } from '../../../types.ts'
const RAM_CAT = RAM_COMMANDS.filter((c) => c.name === 'cat' && c.filetype == null)
const DEC = new TextDecoder()
@@ -12,10 +12,10 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
const RAM_CUT = RAM_COMMANDS.filter((c) => c.name === 'cut' && c.filetype == null)
const ENC = new TextEncoder()
@@ -12,11 +12,11 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { PathSpec } from '../../../../types.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
import { PathSpec } from '../../../types.ts'
const RAM_GREP = RAM_COMMANDS.filter((c) => c.name === 'grep' && c.filetype == null)
const ENC = new TextEncoder()
@@ -12,11 +12,11 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { PathSpec } from '../../../../types.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
import { PathSpec } from '../../../types.ts'
const RAM_HEAD = RAM_COMMANDS.filter((c) => c.name === 'head' && c.filetype == null)
const ENC = new TextEncoder()
@@ -12,11 +12,11 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { PathSpec } from '../../../../types.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
import { PathSpec } from '../../../types.ts'
const RAM_LS = RAM_COMMANDS.filter((c) => c.name === 'ls' && c.filetype == null)
const ENC = new TextEncoder()
@@ -12,11 +12,11 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { PathSpec } from '../../../../types.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
import { PathSpec } from '../../../types.ts'
const RAM_TAIL = RAM_COMMANDS.filter((c) => c.name === 'tail' && c.filetype == null)
const ENC = new TextEncoder()
@@ -12,11 +12,11 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { RAM_COMMANDS } from '../index.ts'
import { RAM_COMMANDS } from './index.ts'
import { describe, expect, it } from 'vitest'
import { materialize } from '../../../../io/types.ts'
import { RAMResource } from '../../../../resource/ram/ram.ts'
import { PathSpec } from '../../../../types.ts'
import { materialize } from '../../../io/types.ts'
import { RAMResource } from '../../../resource/ram/ram.ts'
import { PathSpec } from '../../../types.ts'
const RAM_WC = RAM_COMMANDS.filter((c) => c.name === 'wc' && c.filetype == null)
const ENC = new TextEncoder()
@@ -15,16 +15,43 @@
import { FileType, LINK_TARGET_KEY, type FileStat } from '../../../types.ts'
import { DEFAULT_MODES, EPOCH_LS_TIME, MONTHS, NUMERIC_PREFIX, TYPE_CHARS } from './constants.ts'
export function humanSize(n: number): string {
const units = ['B', 'K', 'M', 'G', 'T']
let value = n
let i = 0
while (value >= 1024 && i < units.length - 1) {
value /= 1024
/**
* GNU's `human_readable` rounding, shared by `-h` and `-H`.
*
* Three rules, none of which fall out of a plain divide-and-format.
* Below one unit GNU prints the count alone -- `24`, never `24B`. Above
* it the value is rounded *up* to the precision shown, so 1025 bytes is
* `1.1K` rather than `1.0K`. And the decimal is dropped once the scaled
* value reaches ten, giving `10K` rather than `10.0K`. Rounding up can
* carry past the base (1048575 bytes ceils to 1024K, which GNU shows as
* `1.0M`), so the unit is re-chosen after rounding instead of once up
* front.
*
* @param n byte count
* @param base 1024 for `-h`, 1000 for `-H`
* @param units suffixes indexed by power; index 0 is unused because a
* sub-unit count carries no suffix at all
*/
export function humanScaled(n: number, base: number, units: readonly string[]): string {
if (n < base) return String(n)
let i = 1
let divisor = base
for (;;) {
const tenths = Math.ceil((n * 10) / divisor)
if (tenths < 100) {
const unit = Math.floor(tenths / 10).toString()
const decimal = (tenths % 10).toString()
return `${unit}.${decimal}${units[i] ?? ''}`
}
const whole = Math.ceil(n / divisor)
if (whole < base || i === units.length - 1) return `${whole.toString()}${units[i] ?? ''}`
i += 1
divisor *= base
}
const s = i === 0 ? Math.round(value).toString() : value.toFixed(1)
return `${s}${units[i] ?? ''}`
}
export function humanSize(n: number): string {
return humanScaled(n, 1024, ['', 'K', 'M', 'G', 'T', 'P', 'E'])
}
function permTriplet(bits: number, special?: string): string {
@@ -25,20 +25,39 @@ function decode(bytes: Uint8Array): string {
return new TextDecoder().decode(bytes)
}
// Read off GNU coreutils 9.7 (`ls -lh` on a file of each size, debian
// stable-slim). The three rows that matter are the ones a plain
// divide-and-format gets wrong: no suffix under 1024, rounding *up* to
// the shown precision (1025 -> 1.1K), and the decimal dropping once the
// value reaches ten (10240 -> 10K). 1048575 pins the carry: it ceils to
// 1024K, which GNU re-scales to 1.0M.
const GNU_HUMAN_SIZES: readonly (readonly [number, string])[] = [
[0, '0'],
[1, '1'],
[24, '24'],
[500, '500'],
[999, '999'],
[1000, '1000'],
[1023, '1023'],
[1024, '1.0K'],
[1025, '1.1K'],
[1126, '1.1K'],
[1127, '1.2K'],
[1536, '1.5K'],
[2048, '2.0K'],
[10188, '10K'],
[10240, '10K'],
[10241, '11K'],
[11263, '11K'],
[1048575, '1.0M'],
[1048576, '1.0M'],
[1024 * 1024 + 512 * 1024, '1.5M'],
[1073741824, '1.0G'],
]
describe('humanSize', () => {
it('bytes below 1K', () => {
expect(humanSize(500)).toBe('500B')
})
it('K/M/G units', () => {
expect(humanSize(1024)).toBe('1.0K')
expect(humanSize(1024 * 1024)).toBe('1.0M')
expect(humanSize(1024 * 1024 * 1024)).toBe('1.0G')
})
it('fractional sizes round to one decimal (GNU, not floored)', () => {
expect(humanSize(1536)).toBe('1.5K')
expect(humanSize(1024 * 1024 + 512 * 1024)).toBe('1.5M')
it.each(GNU_HUMAN_SIZES)('formats %i as GNU does', (size, expected) => {
expect(humanSize(size)).toBe(expected)
})
})
@@ -12,7 +12,7 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { humanSize } from '../../../commands/builtin/utils/formatting.ts'
import { humanScaled, humanSize } from '../../../commands/builtin/utils/formatting.ts'
import { CapacityState, FileStat, PathSpec } from '../../../types.ts'
import type { CapacityResult } from '../../../types.ts'
import { isMissingPath } from '../../../utils/errors.ts'
@@ -25,7 +25,7 @@ import type { Session } from '../../session/session.ts'
import { fail, ok, operandText, splitValueFlags, type Result } from './shared.ts'
import { compareCodePoints } from '../../../utils/sort.ts'
const SI_UNITS = ['B', 'K', 'M', 'G', 'T']
const SI_UNITS = ['', 'K', 'M', 'G', 'T', 'P', 'E']
const BLOCK_SUFFIX: Record<string, number> = {
K: 1024,
M: 1024 ** 2,
@@ -94,17 +94,10 @@ async function pathExists(dispatch: DispatchFn, spec: PathSpec): Promise<boolean
}
}
// Human-readable size in powers of 1000 (df -H), mirroring the 1024
// `humanSize` shape used by df -h / du -h.
// Human-readable size in powers of 1000 (df -H). Same rounding as -h;
// GNU runs both through one `human_readable`.
function humanSi(n: number): string {
let value = n
let i = 0
while (value >= 1000 && i < SI_UNITS.length - 1) {
value /= 1000
i += 1
}
const text = i === 0 ? String(Math.round(value)) : value.toFixed(1)
return `${text}${SI_UNITS[i] ?? ''}`
return humanScaled(n, 1000, SI_UNITS)
}
// Bytes as a count of `block`-byte units, rounded up like GNU df.
@@ -318,11 +318,13 @@ describe('fanOutTraversal du at a descendant mount boundary', () => {
})
// Summing each mount's already-humanized total would round twice and
// report 3.0K; the sub-runs render exact bytes and only the merge
// humanizes.
// report 2.2K; the sub-runs render exact bytes and only the merge
// humanizes. 1025 bytes rather than 1500 because GNU rounds up: 1500
// doubles to 3000, which single- and double-rounding both render 3.0K,
// so those sizes could no longer tell the two apart.
it('humanizes the total once under -ch', async () => {
expect(await runLines(['du -ch /base'], 1500, 1500)).toBe(
'1.5K\t/base/inner\n2.9K\t/base\n2.9K\ttotal\n',
expect(await runLines(['du -ch /base'], 1025, 1025)).toBe(
'1.1K\t/base/inner\n2.1K\t/base\n2.1K\ttotal\n',
)
})
@@ -24,7 +24,7 @@ import type { EmailAccessor } from '../../../accessor/email.ts'
import { readdir as emailReaddir } from '../../../core/email/readdir.ts'
import { stat as emailStat } from '../../../core/email/stat.ts'
import { EMAIL_IO } from './io.ts'
import { metadataProvision } from './provision.ts'
import { metadataProvision } from './_provision.ts'
const resolveGlob = resolveGlobOf(EMAIL_IO)
@@ -33,7 +33,7 @@ import { stat as emailStat } from '../../../core/email/stat.ts'
import { detectScope } from '../../../core/email/scope.ts'
import { searchAndFormat } from '../../../core/email/search.ts'
import { EMAIL_IO } from './io.ts'
import { fileReadProvision } from './provision.ts'
import { fileReadProvision } from './_provision.ts'
const resolveGlob = resolveGlobOf(EMAIL_IO)
@@ -19,7 +19,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { RedisAccessor } from '../../../accessor/redis.ts'
import { writeBytes } from '../../../core/redis/write.ts'
import { RedisStore } from '../../../resource/redis/store.ts'
import { fileReadProvision, headTailProvision, metadataProvision } from './provision.ts'
import { fileReadProvision, headTailProvision, metadataProvision } from './_provision.ts'
const REDIS_URL = process.env.REDIS_URL
const skip = REDIS_URL === undefined
@@ -15,9 +15,9 @@
import { describe, expect, it } from 'vitest'
import { materialize } from '@struktoai/mirage-core/io/types'
import { PathSpec } from '@struktoai/mirage-core/types'
import type { SSHAccessor } from '../../../../accessor/ssh.ts'
import { makeFakeAccessor } from '../../../../core/ssh/_test_utils.ts'
import { SSH_COMMANDS } from '../index.ts'
import type { SSHAccessor } from '../../../accessor/ssh.ts'
import { makeFakeAccessor } from '../../../core/ssh/_test_utils.ts'
import { SSH_COMMANDS } from './index.ts'
const SSH_LS = SSH_COMMANDS.filter((c) => c.name === 'ls' && c.filetype == null)
const DEC = new TextDecoder()
@@ -51,6 +51,7 @@ interface ConformanceCase {
stdin_text?: string
stdin_base64?: string
matrix: Record<string, string[]>
divergence?: string
expect: ConformanceExpect
}
@@ -108,6 +109,27 @@ function validateMatrix(c: ConformanceCase, specName: string): void {
if (!Object.values(c.matrix).some((backends) => backends.length > 0)) {
throw new Error(`case ${c.id} in ${specName} applies to no backend`)
}
// A case is a parity claim, so the two languages have to be asked the
// same question. Dropping a backend from one side reads as coverage
// while it is really an unexamined divergence -- and it is invisible,
// because the side that still lists the backend goes green. Anything
// genuinely language-specific says so in a `divergence` key, which the
// README calls the per-backend override.
if (c.divergence === undefined) {
const python = new Set(c.matrix.python ?? [])
const typescript = new Set(c.matrix.typescript ?? [])
const onlyPython = [...python].filter((b) => !typescript.has(b)).sort()
const onlyTypescript = [...typescript].filter((b) => !python.has(b)).sort()
if (onlyPython.length > 0 || onlyTypescript.length > 0) {
throw new Error(
`case ${c.id} in ${specName} has an asymmetric matrix ` +
`(python-only: ${onlyPython.join(', ') || 'none'}; ` +
`typescript-only: ${onlyTypescript.join(', ') || 'none'}). ` +
'Run it on both, or record why it cannot with a `divergence` key.',
)
}
}
}
function loadCases(): ConformanceCase[] {
@@ -254,4 +276,29 @@ describe('command conformance matrix validation', () => {
validateMatrix(c, 'valid.json')
}).not.toThrow()
})
it('rejects an asymmetric matrix', () => {
const c = {
id: 'narrowed_matrix',
cmd: 'true',
matrix: { python: ['ram', 'disk', 'redis'], typescript: ['ram'] },
expect: { exit: 0, stdout_text: '', stderr_text: '' },
}
expect(() => {
validateMatrix(c, 'narrowed.json')
}).toThrow('asymmetric matrix')
})
it('allows an asymmetric matrix that says why', () => {
const c = {
id: 'declared_divergence',
cmd: 'true',
matrix: { python: ['ram', 'disk', 'redis'], typescript: ['ram'] },
divergence: 'TypeScript has no redis-backed foo yet (#1234).',
expect: { exit: 0, stdout_text: '', stderr_text: '' },
}
expect(() => {
validateMatrix(c, 'declared.json')
}).not.toThrow()
})
})
+1 -1
View File
@@ -57,7 +57,7 @@ export {
headTailProvision,
metadataProvision,
type RedisResourceLike,
} from './commands/builtin/redis/provision.ts'
} from './commands/builtin/redis/_provision.ts'
export { RedisFileCacheStore, type RedisFileCacheOptions } from './cache/redis/file.ts'
export { FuseManager } from './workspace/fuse.ts'
export { MirageFS, type MirageFSOptions, type FuseAttr } from './fuse/fs.ts'