20 Commits

Author	SHA1	Message	Date
bytecii	3293cb5ff4	feat(py): mirage mcp over stdio, and invert the mypy allowlist ... Two cleanup-plan items, both Python-side. Item 29 -- `mirage mcp`. TypeScript shipped a six-tool stdio MCP server; Python had none, so a pip-install user could not point Cursor or Claude Desktop at a workspace and `mirage --help` differed by distribution. Adding the entry point alone would have duplicated the tools, because this side kept them private inside the Claude Agent SDK integration, so the shared layer comes first: - agents/tool_descriptions.py -- the six strings, one copy. - agents/tool_operations.py -- MirageToolOperations, lifted out of the SDK server's private _MirageTools. - agents/file_version.py -- stale-write protection, which this side lacked entirely. TS stamps stored bytes; here the stamp covers the rendered bytes, because this read tool has always rendered and an edit must search what the agent was actually shown. - agents/mcp/server.py + cli/mcp.py -- the server and `mirage mcp`. - server/workspace_config.py -- config discovery (candidates, env names, walk up from cwd), which Python had nowhere, so every entry point had to be handed an explicit path. The server is the low-level MCP Server rather than FastMCP: FastMCP does not forward a version, and TS advertises one. Handlers are bound methods, not decorated closures, so nothing nests. Item 28 -- the mypy allowlist. 54 modules opted *in* to annotation checking against 1826, so the default was unchecked and every new file joined the unchecked side. The default is now strict, with a list of what is not yet annotated that only shrinks. 166 annotations cleared along the way; the remainder is named module by module. Two real defects surfaced by the annotations, neither of them typing: - Workspace._original_open / _original_os were invented by assignment in lifecycle.patch_process, so unpatch without a patch raised AttributeError. Declared, and the restore is guarded. - sed_generic declared a non-optional writer while its own docstring and its `write_bytes is None` branch said otherwise; the builder passes None whenever the backend cannot write. Tests keep the PathSpec rule instead of full strict: measured, full strict on python/tests is 2374 errors, of which 634 are `str` where a pydantic field declares SecretStr -- which pydantic coerces at runtime -- and most of the rest is the monkeypatched-fake pattern CLAUDE.md sanctions. The rule that is violated for real is PathSpec, 19 times, and scripts/check_test_pathspec.py now holds that line. One of the 19 was a latent AttributeError: tests/e2e passes a str to s3 write_bytes, which reads .mount_path, and the test skips without a live versioned bucket. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 03:33:16 -07:00
Zecheng Zhang	6748e66c55	fix(stat): render the symlink arrow for %N (#779) ... * fix(stat): render the symlink arrow for %N stat -c '%N' on a symlink printed only the quoted name. GNU prints 'name' -> 'target'. The target was already on FileStat.extra under LINK_TARGET_KEY, which is what ls -l and file read, so %N now reads the same fact. Pinned against GNU coreutils 9.7 on debian:stable-slim, which showed two more rules: any format modifier drops the quoting (only a bare %N quotes), and a width or precision applies to the name and the target separately rather than to the joined line. That second rule had to be implemented either way to render %20N on a link, and it also fixes %30N on a plain file, which used to keep the quotes. Tests in both languages plus six integ cases across all 22 targets. * fix(stat): quote %N the way gnulib shell-escape does The helper picked double quotes for any name holding an apostrophe, so a'b$c rendered as "a'b$c" and replaying the advertised shell-safe output expanded $c. It also passed control characters through raw. Probed GNU 9.7 across ASCII 32-126 and the C0 range: single quotes are the default with '\'' escaping, double quotes are an optimization for a name whose only awkward character is an apostrophe, and a control character becomes a $'..' segment spliced between single-quoted runs. Now applies to a symlink target too, since %N quotes each field on its own.	2026-08-13 11:13:54 -07:00
Zecheng Zhang	0d5915457f	feat(workspace): state doors and per-plane views (#771) ... * feat(workspace): state doors and per-plane views * fix(workspace): one general session door for every variable write Codex round: shell readlink dispatches through the op door; array-shaped assignments and subscripted unsets can no longer bypass pre_session. SessionView.set is now general over variable shapes (str | ShellArray), pre_session_gate is private to the door and takes a SessionContext that carries session_id, staged declaration arrays store through the builtins, for/select loop variables write through the view, and readonly pre-checks are separated from policy-denial catches at every shell site. Also fixes export ARR=(x) printing the export listing and CI formatter drift. * test(integ): pin the state-door fixes; readonly declaration arrays abandon the line * ci(integ): key the WinFsp install on the DLL, not choco's exit code	2026-08-13 06:59:17 -07:00
bytecii	ef99849b94	refactor(seam): dispatcher-built CommandOpts, typed adapter ops, named ParsedCommand, one DispatchFn home (#609 items 26+23) (#772) ... * wip(seam): py dispatcher builds CommandOpts; handlers take (accessor, paths, texts, opts); adapter op protocols; DispatchFn typing Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(seam): dispatcher-built CommandOpts, typed adapter ops, named ParsedCommand, one DispatchFn home (#609 items 26+23) Item 26 (T2-8 seam typing), all four pieces, plus task 23's static FlagView query-name gate: - The python dispatcher (Mount.execute_cmd) now constructs one CommandOpts per invocation and calls every handler as fn(accessor, paths, texts, opts) — the TS convention. Flags stop sharing a bag with injected context, accepts_kwarg opt-in dies, and all 72 builders, 75 bespoke wrappers and ~30 provision functions become pure wiring; the provision path builds the same bag with command/spec set. Generics that took trailing fact params (ls, find, tree, zip, tar, du, file, stat) now read opts.links/opts.mounts/... like their TS twins. - adapter.py gains per-slot op protocols (ReaddirOp/StatOp/WriteOp/...) mirroring adapter.ts, so wiring readdir where stat belongs no longer type-checks; Builder.fn/provision and CommandIO fields are typed. Surfaced and fixed real drift: hf/databricks mkdir had index before parents (a positional parents call would land in index), wget -O dropped its PathSpec value under as_str. - TS parseFlags returns a named ParsedCommand (twin of the py NamedTuple) instead of a 15-slot positional tuple; optionError takes (cmdName, parsed) like python's option_error. - DispatchFn moves to runtime/types.ts (python's home for the same protocol); the crossmount re-export inversion and the CommandDispatch duplicate are gone; 38 python files stop spelling it Callable[..., Any]. The dead CommandOpts.resource field (zero readers) is deleted along with the CrossResourceStub that existed to fill it. - Task 23: tests/commands/test_flag_query_names.py and commands/flag_query_names.test.ts fail on a FlagView query naming a dest no spec bound in the module declares, without needing the code path to run (both verified to fire on a planted typo). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(examples): redis example calls provisions with the 4-positional shape file_read_provision/head_tail_provision/metadata_provision are (accessor, paths, texts, opts) now; the direct demo calls still passed the old command= kwarg, crashing the example before the persistence section (CI examples gate). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(integ): pin the #772 fixes where the harness can reach them - find -empty on chroma + github (the wrappers used to drop the flag, which would flood every path; -empty now rides find_generic), plus a -not -empty composition case on each so the pin has positive output - mkdir -p gains databricks/databricks-prefix (zero prior mkdir coverage on that backend; the param-order bug itself is only reachable positionally, so MkdirOp+mypy is its real gate) - wget -O / curl -o were already pinned by integ/resources/http Both hosts green: py chroma+github 83 ok, ts all five targets 4070 ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: bytecii <bytecii@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 05:53:25 -07:00
bytecii	ae36ac41ee	test(integ): report-and-continue pins + builder wiring gate ... postgres/mongodb mixed good+missing operand facets, history cross-mount continue, test_builders_declare_only_dispatcher_params ratchet; history ls/tree/stat/find and dify find/cat wrappers ride the generics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-12 19:02:28 -07:00
bytecii	836047e286	refactor(commands): builders become wiring; generics own the flags (T2-2) ... 26 builders forward **flags wholesale; each generic parses once into a frozen struct via a spec-bound FlagView (ls/du/find/sed/tar/tree + 20 mechanical). rm/rmdir/ln/touch are builder-is-implementation and read the bag through FlagView inline, mirroring the TS builders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-12 18:19:56 -07:00
Zecheng Zhang	a16684823d	fix(symlinks): report links across commands, and add -R to chmod/chown/chgrp (#688) ... * fix(symlinks): report and follow links the way GNU does Symlinks live in the namespace, so no backend readdir or stat can see them and every command was blind to them unless wired by hand. Six GNU divergences, each pinned against debian:stable-slim first, mirrored in python and typescript. - ls -l on a dangling link failed the whole listing with exit 2; GNU prints the link row and exits 0. ls -l/-d on a live link silently followed it, and ls -R and -F omitted links entirely. - readlink -e printed a dangling target and exited 0; GNU requires the whole resolved path to exist. -f now requires only the parent. - file followed the link and relabelled the operand as its target; GNU prints "symbolic link to <target verbatim>", or "broken symbolic link to ..." when the target is absent. - du omitted links; GNU lists one line per link under -a and does not follow a link operand without -L. - find followed a symlink start point; GNU's default is -P, which reports the link and stops. -P/-H/-L are leading options, not predicates, so the expression splitter consumes them first. - -L has to apply below the operand too, not just to it: ls -L stats each link child and reports the target under the link's own name, and du -L stops counting links as entries of their own. Generalize the seam while it is being touched: - A command opts in by naming a `links: LinkView` parameter. execute_cmd offers the fact to every handler and accepts_kwarg reads the signature to decide delivery, so there is no allowlist to keep in step. This also removes the hardcoded stat_overlay command list. TypeScript needs no equivalent: a command that does not read `links` off its context ignores it. - LinkView bundles all five link facts, so a new need is a field read rather than a keyword threaded through four layers. - Link merging lives in the generic above the native-op/walk fork, so a mount's symlink behavior cannot depend on whether its backend ships a native find or du op. - ls merges links as stat rows instead of appending to its own stdout, which is what makes -R, -F, -l and sorting work. Deletes inject_links and its latent dedup bug, where splitting a GNU long row on tabs returned the whole line. Also moves data shapes and constant tables to the modules the layout calls for (ops/types, commands/builtin/constants, utils/constants) in both languages, and adds 33 integ cases that run in both runners. du sizes a link at its target length, which is not a divergence: mirage counts bytes, GNU's --apparent-size mode, and there GNU reports len(target) too. Two known gaps, both documented rather than papered over: du -L undercounts a link pointing outside the operand's own subtree, and ls -RL does not descend a directory link. A version of the latter that only worked on RAM was removed rather than shipped, because descending needs the child read re-keyed onto the resolved target and that only resolves on backends with real directory entries. Symlink behavior that depends on which backend is mounted is the exact failure this branch adds a rule against. A link operand is caught on both readdir shapes. Backends without real directories (s3, nextcloud) answer readdir on a link path with an empty list rather than raising, so the link row has to be resolved there too or the operand renders as an empty directory. Pinned by unit tests against the generic in both languages, so the case is covered without needing a remote backend to reproduce it. Fixes #414 * fix(integ): drop the empty-directory dependency from the symlink fixtures * fix(symlinks): give bespoke stat wrappers the link table, and make find's link options last-wins s3 and gridfs ship their own stat command, and neither declared the links parameter, so signature-based injection silently withheld the namespace link table and stat on a symlink reported ENOENT on exactly those two backends. TypeScript is unaffected: its wrappers forward the whole opts object, so the generic reads opts.links regardless. find states its link policy as a leading option and GNU takes the last one, so find -L -P x must not follow while find -P -L x must. The scanner returned true on seeing any -L or -H anywhere. * feat(metadata): walk the subtree for chmod, chown and chgrp -R All three refused -R with exit 2. GNU splits three ways here: chmod -R skips a traversed link but follows a command-line link to a directory, while chown -R and chgrp -R change the link node itself because POSIX gives -R an implicit -P. Two bugs surfaced while pinning it. chown -h wrote uid onto the node but link_stat dropped ownership when rendering, so nothing ever showed it; only mode is meaningless on a link. And TypeScript rebuilt stat rows field by field where Python uses model_copy, dropping uid/gid on the link row, so FileStat gains a with() method and the four rename sites use it. * fix(find): classify a link by its target under -L find -L must test a link as what it points at, not as a link: a link to a file is -type f, a link to a directory is -type d, and only a dangling link stays -type l. link_results classified every namespace link as l regardless, so find -L d -type f missed the links and find -L d -type l wrongly named the live ones. Both find paths reach link_results, so the classification lands above the native-op/walk fork rather than in one branch. * fix(symlinks): stop backends losing links and -L one wrapper at a time `find -L` classifies a link by statting its target, and that stat was reached through the callable wired for the -mtime post-filter, which is gated on ops.local. Post-filtering costs one stat per result, so gating it is right; classification costs one stat per link, so gating it was not. Off the local backends nothing was statted and every link stayed typed `l`: -type f and -type d missed them, -type l still listed them. The fact belongs on LinkView, not on a threaded argument, so it is now `target_stat` / `targetStat`. It resolves through the op dispatcher rather than a backend key, which also fixes a link pointing into another mount: the old code built the key from the search root's mount prefix and could never resolve one. Pinned in integ/crossmount/find/deref.json. The same omission then turned up in ten bespoke commands that shadow a link-aware generic without declaring `links`, and in the same ten for `-L`, which lands in the wrapper's opaque flag bag when unnamed. Both failures run fine and exit 0, so nothing notices until someone makes a link on that backend. tests/commands/test_links_optin.py asserts it instead, deriving from the builders which link parameters each family requires; it caught history/ls on the first run. TypeScript needs no guard because wrappers forward the whole opts object. Its one exception, email/find, walks its own tree and never reaches a generic, so it now calls linkResults directly. * fix(find): trim slashes without a regex, and derive the link guard from the router Exporting linkResults made its input library-reachable, which surfaced the polynomial-ReDoS trim inside it (CodeQL js/polynomial-redos, high). Both `/^\/+|\/+$/g` trims now use the loop-based stripSlash helper. The guard's list of required link options was hardcoded. It now reads DEREFERENCE_FLAGS and LAST_WINS_LINK_OPTIONS, so a new link option taught to the router starts being required of bespoke wrappers too. * fix(symlinks): let real backend failures surface while statting a link target link_target_stat caught OSError around _stat_or_none, which already maps the missing-target case to None. The extra catch could therefore only swallow a genuine failure: a permission, connection or I/O error came back as a dangling link, printed as -type l with exit 0. Only the two legitimate no-target cases stay None now, a loop from namespace.follow and a missing target from _stat_or_none. Everything else propagates. Same in TypeScript, where the bare catch did the same.	2026-08-03 05:01:05 -07:00
Zecheng Zhang	6e1b800c2a	Fix dependency and CodeQL security alerts (#639) ... * Fix dependency and CodeQL security alerts * fix(stat): port linear-time format parser to Python (CodeQL #247) The TypeScript fix left the identical backtracking regex live in the Python mirror, so stat -c FORMAT could still hang on a crafted format. Replace _FORMAT_RE with the same cursor-based directive scanner.	2026-07-25 23:21:54 -07:00
bytecii	426090484a	feat(stat): full GNU stat -c directive set + agent-aware ownership (#609 Tier 1) (#626)	2026-07-23 19:08:49 -07:00
Zecheng Zhang	1c193bc4e8	Optional-typing sweep: path-only generic ops, NULL_INDEX, no bare generics (#541) ... * refactor(types): stdin annotations use the ByteSource alias * refactor(generics): injected ops are path-only, accessor+index bind at the wrapper Generic commands no longer take accessor (or a dead index) just to thread them back into injected callables. Builders and bespoke wrappers bind both via bound_op (None-passthrough) or partial for write-side ops, mirroring the TS builders' closures. call_*/resolve_pattern/relay helpers and the cache read-through wrappers gain path-first forms; CommandIO-level ops keep the raw (accessor, path, index) shape. * refactor(index): NULL_INDEX everywhere, no index-is-None branches index is a required IndexCacheStore on every op and wrapper; callers with no real index pass NULL_INDEX (the null object built for exactly this). Dead is-None guards drop out of trello/linear/github_ci/google readdir+stat, CacheManager, and the github wrappers. * style(types): parameterize every bare generic, gate with disallow_any_generics dict -> dict[str, Any] (payloads) / dict[str, object] (flag bags), Callable -> Callable[..., Any], plus list/tuple/Awaitable/Pattern/ Task/Future/Token/AsyncMongoClient. The mypy gate now carries disallow_any_generics so bare generics cannot come back. * style: pre-commit formatting + parameterize main's new bare dicts * fix: post-merge gate fixes + make security barriers analyzer-recognizable - resolve_within_root: startswith(root + sep) guard instead of commonpath (same semantics; the shape CodeQL models as a path-injection barrier) - cli table output: trimEnd() instead of the polynomial /\s+$/ trim - example proxies: re-encode validated endpoint segments before URL construction (recognized SSRF barrier; no-op for the allowed charset) - parameterize bare generics arriving from #540 (the new mypy gate caught them) * fix(server): plain startswith barrier in resolve_within_root	2026-07-17 20:45:19 -07:00
Zecheng Zhang	5244558e29	Fix latent type-safety and lifecycle bugs (#528) ... * Fix latent type and lifecycle bugs * fix: close ownership for shared resources, gate find -empty, gdrive populate parity, drop mongo DriverInfo * style: yapf/isort formatting on merged test files * fix: fan-out grep keeps GNU any-match exit 0, mock github_ci stat populate readdir, current mock s3 mtime * fix(ts): mirror close ownership, shared stores and copy-shared resources stay open * refactor: shared op-fn aliases and copy/move strategies in mirage.types, tar package with types and constants * refactor: single mtime-window primitive, drop msgraph.time re-export, github declares supportsSnapshot * fix: find -mtime keeps unknown-mtime dirs (s3 integ), mktemp -p PATH-kind spec parity * fix(find): remote -mtime reverts to TS parity, onedrive root stat carries mtime - generic_bind find builder: stat filter only for local backends (matches main + TS; remote backend find ops ignore mtime like the TS s3 core) - s3/onedrive core find: drop in-core mtime filtering, document the deliberate no-op in the s3 docstring - onedrive stat: mount root fetches the real Graph root item so size/modified are populated (du/find no longer see modified=None) - integ: fake Graph stamps items at run time like moto; bespoke onedrive suite freezes the clock for deterministic ls -l; stat_dir case prints stable fields only * test: drop remote find -mtime tests reverted to TS-parity semantics	2026-07-17 17:58:39 -07:00
Zecheng Zhang	ea8096f155	Drive Python mypy to zero and enable it as a hard gate (#522) ... * chore(py): shrink mypy baseline 117->84 (resource accessor typing) Narrow each resource's accessor attribute to its concrete type so the backend resolve_glob/stat calls type-check; wrap fingerprint stat() paths with PathSpec.from_str_path (fixes a latent str-to-stat bug); retype the shared hf_buckets core to _HfAccessor (common base of the hf siblings); make DevStore a RAMStore subclass (_DevFiles a dict subclass); narrow the sync redis get_state() reads (redis-py ResponseT is an async/sync union). Full Python test suite green (9139 passed; fuse excluded, macOS limit). * chore(py): shrink mypy baseline 84->74 (workspace/executor/shell/resource typing) * chore(py): clear non-generic mypy errors, shrink baseline 74->56 * chore(py): clear all generic command-layer mypy errors (127->0) * chore(py): enable mypy as a hard gate (pre-commit + CI), drop ratchet baseline * fix(py): sed -i guards write availability; dedupe mypy pre-commit hook post-merge * fix(py): pyproject trailing newline; run mypy on hosted pre-commit too	2026-07-16 12:57:40 -07:00
Zecheng Zhang	e1df638356	Read family keeps partial output past missing operands (#464) ... * fix(commands): read family keeps partial output past missing operands cat/head/tail/wc abort on the first missing operand: the error reaches the executor chokepoint before any output is emitted (or the lazy drain discards it), so the good operands' output is lost. GNU keeps going: partial stdout, one stderr line per failed operand, exit 1. Builders now partition operands eagerly (split_readable / splitReadable) and stream only the readable ones; wc's format_multi reports failed operands and still prints the total row (0 total when none resolve). Cross-mount run_operands catches filesystem errors at materialize so a lazy drain failure no longer escapes to the route catch-all, which dropped every operand's output and printed a line with no strerror. Single-mount and cross-mount are byte-identical in both languages; integ covers cat/wc/head/tail good+missing on ram, redis, and s3. * refactor(errors): FS_ERRORS single catch set; document grep exit rule The recoverable filesystem exception tuple was spelled out at five catch sites; derive FS_ERRORS from the strerror table in utils/errors so the catch set and the formatter cannot drift (mirrors TS isFsError). Hoist runOperands' TextEncoder to module scope. Document at combinedExit that grep's exit 0 on match-despite-read-error is a deliberate GNU divergence pinned by integ. * fix(commands): read family sweep processes every operand, keeps partial output cut, tac, zcat, and strings processed only paths[0], silently dropping every other operand (cut even exited 0 with a missing operand). They now handle each operand independently and concatenate in operand order, and nl's counter continues across operands instead of resetting per file, all per GNU. The remaining multi-file read builders (nl, md5, sha256sum, strings, tac, rev, cut, expand, unexpand, fold, fmt, zcat) and the stat generic now continue past a missing operand: partial output, one stderr line per failed operand, exit 1 (resolve_readable/withReadable wrap the shared split). sort keeps its GNU-style abort. Cross-mount STREAM errors respell the internal cat sub-run's prefix to the real command name, so cross-mount bytes match single-mount. Integ pins nl (STREAM respell), md5 (FANOUT), and the previously uncovered cross-mount stat on ram, redis, and s3. * test(integ): partial-read battery on every backend; sed keeps going, sort aborts Add PARTIAL_READ_CASES to the shared battery (both langs, truth.txt): multi-operand correctness for cut/tac/nl/strings/zcat plus good+missing for the whole read family, so all 13 shared-truth backends pin partial output, per-operand stderr, and exit codes. cross_commands adds cut/tac/sed/sort and the missing cross-mount stat coverage. Pinning exposed two more divergences, fixed both langs: sed aborted single-mount but GNU keeps going past a missing operand (generic_sed now skips and reports per operand, including -i, exit 1 where GNU exits 2); sort kept partial output cross-mount but GNU aborts (run_stream now drops the body for sort when any operand failed, matching single-mount). * fix(ts): missing-operand handling moves into the read-family generics The withReadable builder wrapper only protected factory-built backends; OPFS's bespoke wrappers call the generics directly, so integ-ts lost the partial output on every swept command. The per-operand read handling now lives inside the generics themselves (readOperands/operandsIo in utils/operands), like cat/head/tail/wc/sed already do, and the factory builders go back to thin wiring. nl, cut, rev, and sha256sum read their operands eagerly in paths mode so the IOResult is sealed before the output stream is handed back (tail/wc already buffer this way). * refactor(crossmount): respell helper named for the fetch, prefix from Cmd.CAT The rewrite is an artifact of the STREAM strategy fetching operands via a native Cmd.CAT sub-run, not anything about the cat command itself, so the name says what it means and the stripped prefix derives from the enum instead of a hardcoded literal.	2026-07-11 15:48:10 -07:00
Zecheng Zhang	ce39e351ce	chore(generic): type accessor as Accessor, document probe except-pass sites	2026-07-10 22:31:25 -07:00
Zecheng Zhang	31340ad9ca	refactor(shell): raw_path owns word spelling; respell lazy drain errors ... - relative_spec: one constructor for typed word + cwd -> PathSpec - raw_path is total (defaults to virtual); PathSpec.display deleted - word_text/wordText: single str-or-spelling coercion helper - rebase_display -> rebase_raw, nullable param dropped - lazy-stream drain errors respell operands as typed via ExecutionNode.paths (TS drain now also formats GNU lines) - integ: relspell battery (walker labels, error spelling, ./ glob, du trailing slash)	2026-07-10 03:17:21 -07:00
Zecheng Zhang	e80af8fe0b	feat: GNU-aligned cwd/env, subshell isolation, and relative-path display	2026-06-16 11:29:36 -07:00
Zecheng Zhang	98e9d8bb22	fix: stat -c %n prints the operand path like GNU stat	2026-06-10 22:44:50 -07:00
Sonny	2671ad7517	fix(commands): terminate generated record output with newlines (#147) ... * fix(commands): terminate generated record output with newlines Add a shared record-output helper for commands that build logical text rows, and use it across affected find/grep/rg/ls/tree/stat/du/md5/cmp/file/wc call sites. This keeps byte-preserving commands untouched while making generated command records end with POSIX-style trailing newlines. Update focused generic, RAM/Redis, and mocked Discord tests for the new output contract. * fix(commands): terminate du and rm verbose record outputs * test(commands): update assertions for terminated command output The newline-termination fix makes find/grep/rg/ls/stat/tree/wc/du emit a trailing newline on the last record. These assertions still encoded the old separator-style output (no final newline) and failed in CI: - dify/discord/github/workspace: expect a trailing "\n" - notion: switch to splitlines() so the terminator doesn't add an empty member to the compared set - github wc: rstrip the newline before splitting on tab - s3 integration: expect the blank separator line that `echo >>` always intended — the missing grep terminator had previously swallowed it Test-only; no production code changed. * fix(commands): terminate grep/rg -c count output * refactor: use format_records at remaining grep/rg/wc sites; add header to output.py DRY-only: every changed site is guaranteed non-empty (or guards the empty case separately), so output is byte-identical. discord stderr helper uses format_optional_records to preserve None-on-empty. --------- Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>	2026-06-01 03:51:59 -07:00
Zecheng Zhang	0ac9cecca9	refactor(commands): consolidate cloud backend wrappers (s3/gdrive/google workspaces) (#70) ... * refactor(commands): consolidate s3/gdrive/gdocs/gsheets/gslides/gmail wrappers Apply the same thin-wrapper + generic-dispatch pattern that landed for ram/disk/redis in #68. Each cloud backend wrapper now does just three things: glob-resolve paths, inject backend-specific callables (read_bytes / read_stream / readdir_fn / stat_fn / mkdir_fn), and pass through stdin/options to the generic implementation. Backends touched (84 wrappers, net -2487 lines): - s3: 34 wrappers - gdrive: 33 wrappers - gdocs / gsheets / gslides / gmail: 4 each (basename, dirname, realpath, tree) Kept resource-specific (per "use resource-specific when generic doesn't fit" guidance): - s3/du, s3/find: custom S3 listing semantics - gdrive/du, gdrive/find, gdrive/sed: no core/gdrive/du, find, or write - gdocs/gsheets/gslides/gmail jq, nl, find: no core/stream or core/find - Folder-based commands (cat/, head/, tail/, ls/, stat/, file/, cut/, grep/, wc/): dispatch over format variants (parquet/feather/etc.) - gws_* domain-specific Google commands: no generic equivalent For Google backends that only expose `read` (no `read_bytes`), the wrappers alias `from mirage.core.<b>.read import read as read_bytes`. This makes gdrive's special-format handling (.gdoc.json / .gsheet.json / .gslide.json -> JSON bytes) transparent to the generic commands: they see "just bytes" regardless of source format. gdrive/__init__.py renamed `sort_cmd` import to `sort` for consistency with other backends. Streaming semantics preserved verbatim: read_stream callables continue to chunk-iterate S3 GetObject responses, backpressure propagates through multi-stage pipes (verified end-to-end on s3 and gcs -- `cat | tr | grep | head -n 1` downloads only 8 KB of a 10 MB object). Two pre-existing example bugs fixed while verifying: - examples/python/s3/s3.py: Workspace.load was passing only /s3/ override when the workspace had both /s3/ and /deep/ mounts. Added the missing /deep/ override. - examples/python/gcs/gcs.py: called ws.reset() which never existed on Workspace. Replaced with ws.cache.clear() (which is what the surrounding code clearly intended given the cache.get() assertions). Also extended the example with streaming + multi-stage pipe backpressure demonstrations. .gitignore: added docs/learnings/ alongside docs/plans/ for local-only notes. Tests: 6203 passed, 266 skipped (no agent tests run). Pre-commit: all hooks pass. * fix(gdrive): thread index into delegated command closures (gdrive/gdocs/gsheets/gslides/gmail) * test(integ): add moto-based S3/GCS cross-backend integ harness * refactor(commands): promote index to an explicit param in backend command wrappers * refactor(filetype): redirect ls/file for parquet/orc/feather to shared core handlers * refactor(commands): delegate cloud/s3 content commands to generic layer - thread index through generic ls/stat so delegated cloud stat works in walk - migrate gdocs/gdrive/gmail/gsheets/gslides/s3 cat/head/wc/ls/stat/jq/nl off inlined logic - remove dead package-shadowed cat/head modules * test(integ): add streaming byte-accounting cases to s3/gcs harness Clear cache, run, and report bytes pulled from the backend for early-exit commands (head -c/-n, grep -m) vs a full read, per mount. Output is deterministic (no timing) and identical across s3/gcs, proving parity. * refactor(integ): iterate s3/gcs cases over a single MOUNTS loop * fix(commands): use index fast-path for stat in tree/grep/rg - generic tree threads index to stat (uniform with ls); cloud tree wrappers stop binding index in the stat partial - s3 grep/rg bind index into call_stat so per-path stat hits the index - integ/s3.py asserts API-call counts: ls -l and tree do 0 HeadObject, proving stats resolve from the index readdir populated * fix(ls): use index fast-path for filetype rendering in directory ls -l render_long_entry passed a prefix-less str to the filetype handler, so cloud handlers hit the wrong key and silently fell back to standard format while wasting HeadObject/GetObject calls. Build a prefixed PathSpec and thread index through; s3 ls filetype handlers now stat via the index. Adds parquet/orc/feather cases to the s3/gcs integ harness. * chore(commands): remove stray no-op expression statements Drop leftover bare statements: 'index' in email/rg.py and 'accessor.store' in ram/redis stat provision. No behavior change.	2026-05-23 22:08:13 -07:00
Zecheng Zhang	870016c5b4	refactor(commands): generic dedup + cross-backend integ harness (#68) ... * feat(utils): add stream/bytes conversion helpers * feat(commands): add generic cat as async iterator * refactor(commands): ram cat wrapper uses generic * test(commands): expand generic cat coverage; fix $ on trailing partial line Old _number_lines_stream had loose tests that missed three POSIX edge cases: multi-digit line number alignment (%6d vs literal 5-space prefix), trailing newline preservation, and $-marker placement on partial last lines. Added unit coverage for all three, plus combined-flag scenarios and a one-byte-at-a-time chunking test. The trailing-$ test caught a real bug: my generic was emitting "hello$" for `cat -E` on input without a final \n, but BSD/GNU cat only emit $ before \n. Fixed by dropping the suffix from the trailing-partial-line branch. * test(ram): add 1:1 test file for ram cat wrapper Establishes src↔test correspondence (every src file has a matching test_*.py) for the ram cat wrapper. Covers the wrapper's behavior end-to-end via Workspace.execute, including regression tests for the two POSIX bug fixes that landed via the generic refactor: - cat -n multi-digit line number alignment (lines 1-12) - no-trailing-newline preservation for both cat and cat -n Plus multi-file concatenation, empty input, and the all-blank-lines edge case. * refactor(commands): disk cat wrapper uses generic + 1:1 tests The active disk cat is at disk/cat/cat.py (the disk/cat package shadows the top-level disk/cat.py file, which is dead code). Wires through generic_cat, preserves CachableAsyncIterator/IOResult shape, drops _number_lines_stream. Tests cover the same POSIX edge cases as ram cat: multi-digit alignment, no-trailing-newline preservation, empty input, all-blank-lines. * refactor(commands): redis cat wrapper uses generic + 1:1 tests Structurally identical to ram cat migration. Tests use the same REDIS_URL skip pattern as tests/core/redis/conftest.py — they run when REDIS_URL is set in the environment, skip otherwise. * fix(commands): preserve cachable identity in cat wrappers for cache to populate Bug: wrapping cachable in generic_cat unconditionally caused stdout and io.reads[path] to reference different objects. mount.py:_wrap then wrapped them separately, and wrap_cachable_streams only updated stdout when identity matched (stdout is stream). The cache cachable was never iterated by stdout materialization, so the background drain saw an exhausted source and cached empty bytes. Fix: only wrap with generic_cat when a flag (currently just -n) actually requires line processing. For the plain 'cat path' case, return cachable directly — matching the pre-refactor behavior and keeping stdout identical to the value in io.reads[path]. Tests caught: test_cache_hit_serves_from_ram, test_grep_uses_cache, test_fuse_read_uses_cache_when_populated (workspace cache layer). * feat(commands): add generic head with -c fast path and -n line-buffered path * refactor(commands): ram head wrapper uses generic + 1:1 tests * refactor(commands): disk head wrapper uses generic + 1:1 tests * refactor(commands): redis head wrapper uses generic + 1:1 tests * test(commands): backfill generic cat/head coverage missing from old helpers Audited tests against legacy head_helper/cat_helper test files. Added: cat: - empty input passthrough (no flags) - binary passthrough across full 0..255 byte range - show_ends only marks newlines, not other binary bytes head: - n=1 single-line output - empty input (with and without -c) - single line no newline with default n - c=negative emits nothing * refactor(commands): consolidate ram/disk/redis logic into generic modules Phases K through Z extracted shared per-command logic into mirage/commands/builtin/generic/<name>.py modules that accept VFS callables (read_bytes, read_stream, write_bytes, stat, readdir, mkdir) via dependency injection. Per-backend wrappers (ram, disk, redis) are now thin shims that resolve globs, supply backend-specific callables, and delegate to the generic. Scope: - 57 generic modules - 170 backend wrappers reduced to shims - 67 new generic-level tests under tests/commands/builtin/generic/ - Net -7,772 lines (178 files changed: +1,891 / -9,663) Bug fixes converged during refactor (no backward compat needed): - disk/md5 emitted only the digest; now matches coreutils `{digest} {path}` - disk/sort and disk/nl only operated on paths[0]; now process all paths - disk/awk lacked BEGIN/END/accumulator support; converged on ram/redis - disk/unzip flattened archive directory structure; now preserves it - disk/patch used offset-splice algorithm; converged on lockstep walk (closer to real patch(1) semantics) - rg failed to auto-recurse on bare directories; now matches ripgrep - file command silently swallowed read errors; now logs to logger.debug Cleanup: - Deleted 3 dead top-level head.py files shadowed by head/ packages in ram/disk/redis (Python package always wins over .py at same level) - Removed 4 stray __init__.py files under tests/ per CLAUDE.md rule - Replaced inline string literals with enums (FindType, LsSortBy) - Removed bare `try/except: pass` for mkdir using idempotent mkdir(parents=True) instead * test(integ): add cross-backend equivalence harness with 240 cases New /integ folder runs identical command suite across ram/disk/redis and diffs each backend's output against committed integ/truth.txt canonical fixture. CI workflow (.github/workflows/test_integ.yml) boots a redis service and runs all three backends. Also fixes bugs surfaced by running the suite: - disk/cat: chain multiple files (was processing only paths[0]) - sha256sum: use original path string; -c mode resolves mount prefix - md5: support stdin when no paths given - jq: iterate per-record on .jsonl files (real jq semantics) - xxd -r: strip offset prefix and ASCII suffix before unhexlify - find -mindepth/-maxdepth: off-by-one across ram/disk/redis/s3 - find -o vs -or: handle both spellings - awk: \$NF/\$NR resolve via field indirection; bare condition programs (e.g. '\$2 > 28') treated as condition with default print \$0; && and || compound conditions; BEGIN block executes; END can read NR - diff: default to GNU normal format (1,5c1,3 with < / > / ---); -u flag required for unified output - cmp: use 'char N' instead of 'byte N' (matches GNU/BSD) Extracts: - DiffOpTag StrEnum (equal/delete/insert/replace) in diff_types.py - AwkCmpOp / AwkBoolOp / AwkBlock / AwkBuiltin StrEnums plus FIELD_PREFIX / PRINT_STMT / CMP_OP_PATTERN constants in generic/awk_types.py * fix(integ): preserve trailing whitespace in truth.txt + redis find test The trailing-whitespace pre-commit hook stripped tabs at line-end in integ/truth.txt, which broke the diff against backends (paste pads shorter columns with trailing tabs). Exclude integ/truth.txt from end-of-file-fixer, mixed-line-ending, and trailing-whitespace hooks. Also fix tests/core/redis/test_find.py {maxdepth, mindepth} that asserted the previously-buggy depth math (matches the existing ram/disk test updates).	2026-05-19 14:03:05 -07:00