12 Commits

Author	SHA1	Message	Date
bytecii	3293cb5ff4	feat(py): mirage mcp over stdio, and invert the mypy allowlist ... Two cleanup-plan items, both Python-side. Item 29 -- `mirage mcp`. TypeScript shipped a six-tool stdio MCP server; Python had none, so a pip-install user could not point Cursor or Claude Desktop at a workspace and `mirage --help` differed by distribution. Adding the entry point alone would have duplicated the tools, because this side kept them private inside the Claude Agent SDK integration, so the shared layer comes first: - agents/tool_descriptions.py -- the six strings, one copy. - agents/tool_operations.py -- MirageToolOperations, lifted out of the SDK server's private _MirageTools. - agents/file_version.py -- stale-write protection, which this side lacked entirely. TS stamps stored bytes; here the stamp covers the rendered bytes, because this read tool has always rendered and an edit must search what the agent was actually shown. - agents/mcp/server.py + cli/mcp.py -- the server and `mirage mcp`. - server/workspace_config.py -- config discovery (candidates, env names, walk up from cwd), which Python had nowhere, so every entry point had to be handed an explicit path. The server is the low-level MCP Server rather than FastMCP: FastMCP does not forward a version, and TS advertises one. Handlers are bound methods, not decorated closures, so nothing nests. Item 28 -- the mypy allowlist. 54 modules opted *in* to annotation checking against 1826, so the default was unchecked and every new file joined the unchecked side. The default is now strict, with a list of what is not yet annotated that only shrinks. 166 annotations cleared along the way; the remainder is named module by module. Two real defects surfaced by the annotations, neither of them typing: - Workspace._original_open / _original_os were invented by assignment in lifecycle.patch_process, so unpatch without a patch raised AttributeError. Declared, and the restore is guarded. - sed_generic declared a non-optional writer while its own docstring and its `write_bytes is None` branch said otherwise; the builder passes None whenever the backend cannot write. Tests keep the PathSpec rule instead of full strict: measured, full strict on python/tests is 2374 errors, of which 634 are `str` where a pydantic field declares SecretStr -- which pydantic coerces at runtime -- and most of the rest is the monkeypatched-fake pattern CLAUDE.md sanctions. The rule that is violated for real is PathSpec, 19 times, and scripts/check_test_pathspec.py now holds that line. One of the 19 was a latent AttributeError: tests/e2e passes a str to s3 write_bytes, which reads .mount_path, and the test skips without a live versioned bucket. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 03:33:16 -07:00
bytecii	ae36ac41ee	test(integ): report-and-continue pins + builder wiring gate ... postgres/mongodb mixed good+missing operand facets, history cross-mount continue, test_builders_declare_only_dispatcher_params ratchet; history ls/tree/stat/find and dify find/cat wrappers ride the generics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-12 19:02:28 -07:00
bytecii	836047e286	refactor(commands): builders become wiring; generics own the flags (T2-2) ... 26 builders forward **flags wholesale; each generic parses once into a frozen struct via a spec-bound FlagView (ls/du/find/sed/tar/tree + 20 mechanical). rm/rmdir/ln/touch are builder-is-implementation and read the bag through FlagView inline, mirroring the TS builders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-12 18:19:56 -07:00
Zecheng Zhang	daddf11965	Add Tier 2 coreutils support (#619) ... * Add Tier 2 coreutils support * Fix Tier 2 CI checks * Fix PR review findings and CI failures for Tier 2 coreutils CI: - regenerate spec/ JSON for the new and changed command specs - bump the TypeScript BUILTIN_SPECS count to 92 - wire the truncate op for redis, ssh and gridfs on the Python side (TypeScript already had all three), fixing native truncate on redis Review findings, applied to both languages: - checksum: parse BSD/--tag lines under --check, and honor -z for --tag - join: bounds-check --check-order and --header field reads so blank or short lines no longer raise IndexError, and honor --nocheck-order - mktemp: move -q into the generic, narrow it to OSError, and suppress the creation diagnostic instead of emitting it (GNU); add -q to the TypeScript generic, which had none - split: keep the empty record a second trailing separator terminates - paste: decode -d escapes with a single left-to-right scan so \0 means the empty delimiter and \\ stays a literal backslash - executor: merge positional and path-flag scopes instead of dropping the flag scopes whenever a positional path is present, so a path-flag target on another mount is routed and rejected rather than misrouted - mkdir: apply -m/--mode through set_attrs; parse_mode moves to a leaf util shared by the command layer and the executor builtins - numfmt: convert only the first field of each stdin record and copy the rest through verbatim - od: concatenate every FILE operand as one input - nl: write an empty line in place of each logical-page delimiter, and pad a one-character -d with ':' as GNU does GNU behavior for nl, join, paste and numfmt was pinned against coreutils 9.8. The nl integ expectations encoded the old delimiter handling and are corrected; six integ cases are added for findings that had no coverage. * Regenerate truncate spec after wiring redis, ssh and gridfs	2026-07-23 01:44:59 -07:00
Zecheng Zhang	1c193bc4e8	Optional-typing sweep: path-only generic ops, NULL_INDEX, no bare generics (#541) ... * refactor(types): stdin annotations use the ByteSource alias * refactor(generics): injected ops are path-only, accessor+index bind at the wrapper Generic commands no longer take accessor (or a dead index) just to thread them back into injected callables. Builders and bespoke wrappers bind both via bound_op (None-passthrough) or partial for write-side ops, mirroring the TS builders' closures. call_*/resolve_pattern/relay helpers and the cache read-through wrappers gain path-first forms; CommandIO-level ops keep the raw (accessor, path, index) shape. * refactor(index): NULL_INDEX everywhere, no index-is-None branches index is a required IndexCacheStore on every op and wrapper; callers with no real index pass NULL_INDEX (the null object built for exactly this). Dead is-None guards drop out of trello/linear/github_ci/google readdir+stat, CacheManager, and the github wrappers. * style(types): parameterize every bare generic, gate with disallow_any_generics dict -> dict[str, Any] (payloads) / dict[str, object] (flag bags), Callable -> Callable[..., Any], plus list/tuple/Awaitable/Pattern/ Task/Future/Token/AsyncMongoClient. The mypy gate now carries disallow_any_generics so bare generics cannot come back. * style: pre-commit formatting + parameterize main's new bare dicts * fix: post-merge gate fixes + make security barriers analyzer-recognizable - resolve_within_root: startswith(root + sep) guard instead of commonpath (same semantics; the shape CodeQL models as a path-injection barrier) - cli table output: trimEnd() instead of the polynomial /\s+$/ trim - example proxies: re-encode validated endpoint segments before URL construction (recognized SSRF barrier; no-op for the allowed charset) - parameterize bare generics arriving from #540 (the new mypy gate caught them) * fix(server): plain startswith barrier in resolve_within_root	2026-07-17 20:45:19 -07:00
Zecheng Zhang	ea8096f155	Drive Python mypy to zero and enable it as a hard gate (#522) ... * chore(py): shrink mypy baseline 117->84 (resource accessor typing) Narrow each resource's accessor attribute to its concrete type so the backend resolve_glob/stat calls type-check; wrap fingerprint stat() paths with PathSpec.from_str_path (fixes a latent str-to-stat bug); retype the shared hf_buckets core to _HfAccessor (common base of the hf siblings); make DevStore a RAMStore subclass (_DevFiles a dict subclass); narrow the sync redis get_state() reads (redis-py ResponseT is an async/sync union). Full Python test suite green (9139 passed; fuse excluded, macOS limit). * chore(py): shrink mypy baseline 84->74 (workspace/executor/shell/resource typing) * chore(py): clear non-generic mypy errors, shrink baseline 74->56 * chore(py): clear all generic command-layer mypy errors (127->0) * chore(py): enable mypy as a hard gate (pre-commit + CI), drop ratchet baseline * fix(py): sed -i guards write availability; dedupe mypy pre-commit hook post-merge * fix(py): pyproject trailing newline; run mypy on hosted pre-commit too	2026-07-16 12:57:40 -07:00
Zecheng Zhang	753f005154	GNU gaps: EISDIR on directories, bash arithmetic, cross-mount paste/comm/join, operand arity (#477) ... * fix(read): directory operands refuse with GNU EISDIR, implicit dirs probed via readdir (#457) * feat(shell): bash arithmetic evaluator, (( )) command and $(( )) assignments (#369) * feat(crossmount): paste/comm/join relay over the shared generics, awk streams (#445) * test(integ): arith, EISDIR, and relay-interleave coverage; OPFS dirs raise EISDIR * feat(spec): commands own operand arity, extra operands refuse with GNU errors (#452) * fix(read): implicit-dir probe confirms against the parent listing, TS stream stats first The ENOENT readdir probe trusted the operand's own listing: postgres fabricates schema children for any name (Is a directory for missing files) and lancedb raises driver errors that escaped the probe. The parent listing decides instead, and any probe failure keeps the original ENOENT. TS streamRefusingDirs now stats before reading like the Python twin, so sftp directory reads report EISDIR instead of an opaque Failure. Also updates the stale parser overflow test (#452 pass-through) and the databricks tr file-operand test (GNU tr takes no file operands). * refactor(spec,shell): CommandName enum, arith constants/errors modules, ArithParser/ArithEvaluator Arith tokenizer/limit constants move to shell/constants and ArithError to shell/errors in both languages; the parser and evaluator classes get their public names. CommandName (StrEnum / TS string enum) lands in spec/types next to OperandKind, used by the arity guards and the usage message shapes; the diff/cmp hint grouping becomes USAGE_HINT_PREFIX in spec/constants. USAGE_EXIT keys stay plain strings: types.py imports constants for flag_kwarg_name, so the table cannot import the enum without a cycle.	2026-07-11 22:00:48 -07:00
Zecheng Zhang	ce39e351ce	chore(generic): type accessor as Accessor, document probe except-pass sites	2026-07-10 22:31:25 -07:00
Zecheng Zhang	4da17edb3f	refactor(paths): pure-virtual PathSpec, mount stamps resource_path ... - rename original -> virtual, as_typed -> raw_path - remove prefix field and strip_prefix/key properties - add required resource_path stamped by the mount at dispatch - add strip_mount/mount_key/rekey/mount_prefix_of in utils/key_prefix - mirror in TypeScript (resourcePath, rawPath, mountKey et al)	2026-07-02 04:50:31 -07:00
Zecheng Zhang	82a1d15460	fix: align no-input behavior of generic commands with GNU /dev/null semantics (#279)	2026-06-11 19:02:26 -07:00
Zecheng Zhang	12b975d5af	fix(io): sweep remaining original-keyed records; ln/zip integ coverage ... Reviewer follow-ups on the same bug class: ln (6 py backends + ts s3) and zip (4 py copies + ts s3) writes keys; dify cat/wc reads; generic base64/tr/tac/xxd/diff/sha256sum cache keys; local_audio reads/cache (py s3 + ts disk/ram). apply_io now uses the public buffered_chunks. chain_cachables unit tests cover live pull, drained replay, partial consumption, and early-stop. Integ adds lnzip_* cases (warm ls, ln -s, zip, re-ls, read back); truth regenerated and all nine backends verified byte-identical.	2026-06-10 04:20:33 -07:00
Zecheng Zhang	870016c5b4	refactor(commands): generic dedup + cross-backend integ harness (#68) ... * feat(utils): add stream/bytes conversion helpers * feat(commands): add generic cat as async iterator * refactor(commands): ram cat wrapper uses generic * test(commands): expand generic cat coverage; fix $ on trailing partial line Old _number_lines_stream had loose tests that missed three POSIX edge cases: multi-digit line number alignment (%6d vs literal 5-space prefix), trailing newline preservation, and $-marker placement on partial last lines. Added unit coverage for all three, plus combined-flag scenarios and a one-byte-at-a-time chunking test. The trailing-$ test caught a real bug: my generic was emitting "hello$" for `cat -E` on input without a final \n, but BSD/GNU cat only emit $ before \n. Fixed by dropping the suffix from the trailing-partial-line branch. * test(ram): add 1:1 test file for ram cat wrapper Establishes src↔test correspondence (every src file has a matching test_*.py) for the ram cat wrapper. Covers the wrapper's behavior end-to-end via Workspace.execute, including regression tests for the two POSIX bug fixes that landed via the generic refactor: - cat -n multi-digit line number alignment (lines 1-12) - no-trailing-newline preservation for both cat and cat -n Plus multi-file concatenation, empty input, and the all-blank-lines edge case. * refactor(commands): disk cat wrapper uses generic + 1:1 tests The active disk cat is at disk/cat/cat.py (the disk/cat package shadows the top-level disk/cat.py file, which is dead code). Wires through generic_cat, preserves CachableAsyncIterator/IOResult shape, drops _number_lines_stream. Tests cover the same POSIX edge cases as ram cat: multi-digit alignment, no-trailing-newline preservation, empty input, all-blank-lines. * refactor(commands): redis cat wrapper uses generic + 1:1 tests Structurally identical to ram cat migration. Tests use the same REDIS_URL skip pattern as tests/core/redis/conftest.py — they run when REDIS_URL is set in the environment, skip otherwise. * fix(commands): preserve cachable identity in cat wrappers for cache to populate Bug: wrapping cachable in generic_cat unconditionally caused stdout and io.reads[path] to reference different objects. mount.py:_wrap then wrapped them separately, and wrap_cachable_streams only updated stdout when identity matched (stdout is stream). The cache cachable was never iterated by stdout materialization, so the background drain saw an exhausted source and cached empty bytes. Fix: only wrap with generic_cat when a flag (currently just -n) actually requires line processing. For the plain 'cat path' case, return cachable directly — matching the pre-refactor behavior and keeping stdout identical to the value in io.reads[path]. Tests caught: test_cache_hit_serves_from_ram, test_grep_uses_cache, test_fuse_read_uses_cache_when_populated (workspace cache layer). * feat(commands): add generic head with -c fast path and -n line-buffered path * refactor(commands): ram head wrapper uses generic + 1:1 tests * refactor(commands): disk head wrapper uses generic + 1:1 tests * refactor(commands): redis head wrapper uses generic + 1:1 tests * test(commands): backfill generic cat/head coverage missing from old helpers Audited tests against legacy head_helper/cat_helper test files. Added: cat: - empty input passthrough (no flags) - binary passthrough across full 0..255 byte range - show_ends only marks newlines, not other binary bytes head: - n=1 single-line output - empty input (with and without -c) - single line no newline with default n - c=negative emits nothing * refactor(commands): consolidate ram/disk/redis logic into generic modules Phases K through Z extracted shared per-command logic into mirage/commands/builtin/generic/<name>.py modules that accept VFS callables (read_bytes, read_stream, write_bytes, stat, readdir, mkdir) via dependency injection. Per-backend wrappers (ram, disk, redis) are now thin shims that resolve globs, supply backend-specific callables, and delegate to the generic. Scope: - 57 generic modules - 170 backend wrappers reduced to shims - 67 new generic-level tests under tests/commands/builtin/generic/ - Net -7,772 lines (178 files changed: +1,891 / -9,663) Bug fixes converged during refactor (no backward compat needed): - disk/md5 emitted only the digest; now matches coreutils `{digest} {path}` - disk/sort and disk/nl only operated on paths[0]; now process all paths - disk/awk lacked BEGIN/END/accumulator support; converged on ram/redis - disk/unzip flattened archive directory structure; now preserves it - disk/patch used offset-splice algorithm; converged on lockstep walk (closer to real patch(1) semantics) - rg failed to auto-recurse on bare directories; now matches ripgrep - file command silently swallowed read errors; now logs to logger.debug Cleanup: - Deleted 3 dead top-level head.py files shadowed by head/ packages in ram/disk/redis (Python package always wins over .py at same level) - Removed 4 stray __init__.py files under tests/ per CLAUDE.md rule - Replaced inline string literals with enums (FindType, LsSortBy) - Removed bare `try/except: pass` for mkdir using idempotent mkdir(parents=True) instead * test(integ): add cross-backend equivalence harness with 240 cases New /integ folder runs identical command suite across ram/disk/redis and diffs each backend's output against committed integ/truth.txt canonical fixture. CI workflow (.github/workflows/test_integ.yml) boots a redis service and runs all three backends. Also fixes bugs surfaced by running the suite: - disk/cat: chain multiple files (was processing only paths[0]) - sha256sum: use original path string; -c mode resolves mount prefix - md5: support stdin when no paths given - jq: iterate per-record on .jsonl files (real jq semantics) - xxd -r: strip offset prefix and ASCII suffix before unhexlify - find -mindepth/-maxdepth: off-by-one across ram/disk/redis/s3 - find -o vs -or: handle both spellings - awk: \$NF/\$NR resolve via field indirection; bare condition programs (e.g. '\$2 > 28') treated as condition with default print \$0; && and || compound conditions; BEGIN block executes; END can read NR - diff: default to GNU normal format (1,5c1,3 with < / > / ---); -u flag required for unified output - cmp: use 'char N' instead of 'byte N' (matches GNU/BSD) Extracts: - DiffOpTag StrEnum (equal/delete/insert/replace) in diff_types.py - AwkCmpOp / AwkBoolOp / AwkBlock / AwkBuiltin StrEnums plus FIELD_PREFIX / PRINT_STMT / CMP_OP_PATTERN constants in generic/awk_types.py * fix(integ): preserve trailing whitespace in truth.txt + redis find test The trailing-whitespace pre-commit hook stripped tabs at line-end in integ/truth.txt, which broke the diff against backends (paste pads shorter columns with trailing tabs). Exclude integ/truth.txt from end-of-file-fixer, mixed-line-ending, and trailing-whitespace hooks. Also fix tests/core/redis/test_find.py {maxdepth, mindepth} that asserted the previously-buggy depth math (matches the existing ram/disk test updates).	2026-05-19 14:03:05 -07:00