3293cb5ff40c3eb18a599ce9113ed74d3d8d1607
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bytecii
|
3293cb5ff4 |
feat(py): mirage mcp over stdio, and invert the mypy allowlist
Two cleanup-plan items, both Python-side.
Item 29 -- `mirage mcp`. TypeScript shipped a six-tool stdio MCP server;
Python had none, so a pip-install user could not point Cursor or Claude
Desktop at a workspace and `mirage --help` differed by distribution.
Adding the entry point alone would have duplicated the tools, because
this side kept them private inside the Claude Agent SDK integration, so
the shared layer comes first:
- agents/tool_descriptions.py -- the six strings, one copy.
- agents/tool_operations.py -- MirageToolOperations, lifted out of the
SDK server's private _MirageTools.
- agents/file_version.py -- stale-write protection, which this side
lacked entirely. TS stamps stored bytes; here the stamp covers the
rendered bytes, because this read tool has always rendered and an
edit must search what the agent was actually shown.
- agents/mcp/server.py + cli/mcp.py -- the server and `mirage mcp`.
- server/workspace_config.py -- config discovery (candidates, env
names, walk up from cwd), which Python had nowhere, so every entry
point had to be handed an explicit path.
The server is the low-level MCP Server rather than FastMCP: FastMCP does
not forward a version, and TS advertises one. Handlers are bound methods,
not decorated closures, so nothing nests.
Item 28 -- the mypy allowlist. 54 modules opted *in* to annotation
checking against 1826, so the default was unchecked and every new file
joined the unchecked side. The default is now strict, with a list of
what is not yet annotated that only shrinks. 166 annotations cleared
along the way; the remainder is named module by module.
Two real defects surfaced by the annotations, neither of them typing:
- Workspace._original_open / _original_os were invented by assignment
in lifecycle.patch_process, so unpatch without a patch raised
AttributeError. Declared, and the restore is guarded.
- sed_generic declared a non-optional writer while its own docstring
and its `write_bytes is None` branch said otherwise; the builder
passes None whenever the backend cannot write.
Tests keep the PathSpec rule instead of full strict: measured, full
strict on python/tests is 2374 errors, of which 634 are `str` where a
pydantic field declares SecretStr -- which pydantic coerces at runtime --
and most of the rest is the monkeypatched-fake pattern CLAUDE.md
sanctions. The rule that is violated for real is PathSpec, 19 times, and
scripts/check_test_pathspec.py now holds that line. One of the 19 was a
latent AttributeError: tests/e2e passes a str to s3 write_bytes, which
reads .mount_path, and the test skips without a live versioned bucket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
bytecii
|
ae36ac41ee |
test(integ): report-and-continue pins + builder wiring gate
postgres/mongodb mixed good+missing operand facets, history cross-mount continue, test_builders_declare_only_dispatcher_params ratchet; history ls/tree/stat/find and dify find/cat wrappers ride the generics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
836047e286 |
refactor(commands): builders become wiring; generics own the flags (T2-2)
26 builders forward **flags wholesale; each generic parses once into a frozen struct via a spec-bound FlagView (ls/du/find/sed/tar/tree + 20 mechanical). rm/rmdir/ln/touch are builder-is-implementation and read the bag through FlagView inline, mirroring the TS builders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
d82c6e9478 | refactor(flags): delete the dead short-flag params, name the bag's type, spec-bind the TS write-side builders (#292) (#733) | ||
|
bytecii
|
58a00116e7 |
feat: coreutils Tier 2 — chgrp, realpath -e, ln -r, rm safety flags (#609) (#617)
* feat: coreutils Tier 2 — chgrp, realpath -e, ln -r, rm safety flags (#609) Fill the genuine gaps in #609 Tier 2 (medium priority), scoped to what fits mirage's async control-plane model. Most of Tier 2 (touch -c/-r/-d, ln -s, readlink -f/-e/-m, chmod, chown) already lives in the executor-builtins layer; this adds the missing pieces, py ≡ ts: - chgrp (new): group ownership via the namespace-overlay set_attrs path (the group half of chown; -h no-deref, rejects -R). - realpath -e: fix a doubled ENOENT message — raise a plain error rather than a filesystem error type so format_fs_error emits it verbatim, matching the TS realpath throw. - ln -r/--relative: store the target relative to the link's directory; -n/-T accepted as no-ops (a namespace link name is never dereferenced nor treated as a directory to descend into). - rm -i/-I/--preserve-root/--no-preserve-root/--one-file-system: accepted no-ops — mirage is a non-interactive control plane (no prompt), mount roots and / are structurally unremovable so the root failsafe is always on and cannot be disabled, and recursion never crosses a mount boundary so --one-file-system already matches the default. Complete integ coverage (integ/unix/{chgrp,ln,realpath,rm}) verified on ram/disk/opfs across both hosts, plus py/ts unit tests. Regenerated the rm spec JSON for all three surfaces. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address PR review — cwd-resolve chmod/chown/chgrp operands, canonicalize ln -r Two P2 review findings on #617: - chmod/chown/chgrp had no CommandSpec, so a relative FILE operand with a non-root cwd (`cd /data/sub && chmod 600 f.txt`) reached the handler as a bare string and resolved against the mount root, not the session cwd (ENOENT / wrong path). Add specs for all three (leading MODE/OWNER/GROUP as TEXT, FILE operands as PATH) so the classifier cwd-resolves them; the handlers keep self-parsing their flags (declared options are not stripped from the operand list). Fixes chgrp and the pre-existing chmod/chown bug. - ln -r computed the relative target lexically, storing an aliased path instead of the canonical one (`ln -sr /alias/a/f /b/link` -> ../alias/a/f vs GNU ../real/a/f). Canonicalize the target and link directory through the symlink table before computing the relative path. Spec count 86 -> 89: update the Python key-set test and the TS builtins count test; regenerate the new chmod/chown/chgrp spec JSON. New integ regression cases (chgrp relative-cwd, ln -r via a symlinked dir), verified ram + disk on both hosts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: bytecii <bytecii@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
1c193bc4e8 |
Optional-typing sweep: path-only generic ops, NULL_INDEX, no bare generics (#541)
* refactor(types): stdin annotations use the ByteSource alias * refactor(generics): injected ops are path-only, accessor+index bind at the wrapper Generic commands no longer take accessor (or a dead index) just to thread them back into injected callables. Builders and bespoke wrappers bind both via bound_op (None-passthrough) or partial for write-side ops, mirroring the TS builders' closures. call_*/resolve_pattern/relay helpers and the cache read-through wrappers gain path-first forms; CommandIO-level ops keep the raw (accessor, path, index) shape. * refactor(index): NULL_INDEX everywhere, no index-is-None branches index is a required IndexCacheStore on every op and wrapper; callers with no real index pass NULL_INDEX (the null object built for exactly this). Dead is-None guards drop out of trello/linear/github_ci/google readdir+stat, CacheManager, and the github wrappers. * style(types): parameterize every bare generic, gate with disallow_any_generics dict -> dict[str, Any] (payloads) / dict[str, object] (flag bags), Callable -> Callable[..., Any], plus list/tuple/Awaitable/Pattern/ Task/Future/Token/AsyncMongoClient. The mypy gate now carries disallow_any_generics so bare generics cannot come back. * style: pre-commit formatting + parameterize main's new bare dicts * fix: post-merge gate fixes + make security barriers analyzer-recognizable - resolve_within_root: startswith(root + sep) guard instead of commonpath (same semantics; the shape CodeQL models as a path-injection barrier) - cli table output: trimEnd() instead of the polynomial /\s+$/ trim - example proxies: re-encode validated endpoint segments before URL construction (recognized SSRF barrier; no-op for the allowed charset) - parameterize bare generics arriving from #540 (the new mypy gate caught them) * fix(server): plain startswith barrier in resolve_within_root |
||
|
Zecheng Zhang
|
ea8096f155 |
Drive Python mypy to zero and enable it as a hard gate (#522)
* chore(py): shrink mypy baseline 117->84 (resource accessor typing) Narrow each resource's accessor attribute to its concrete type so the backend resolve_glob/stat calls type-check; wrap fingerprint stat() paths with PathSpec.from_str_path (fixes a latent str-to-stat bug); retype the shared hf_buckets core to _HfAccessor (common base of the hf siblings); make DevStore a RAMStore subclass (_DevFiles a dict subclass); narrow the sync redis get_state() reads (redis-py ResponseT is an async/sync union). Full Python test suite green (9139 passed; fuse excluded, macOS limit). * chore(py): shrink mypy baseline 84->74 (workspace/executor/shell/resource typing) * chore(py): clear non-generic mypy errors, shrink baseline 74->56 * chore(py): clear all generic command-layer mypy errors (127->0) * chore(py): enable mypy as a hard gate (pre-commit + CI), drop ratchet baseline * fix(py): sed -i guards write availability; dedupe mypy pre-commit hook post-merge * fix(py): pyproject trailing newline; run mypy on hosted pre-commit too |
||
|
Zecheng Zhang
|
0b560b96f3 |
refactor(core): PathSpec-only core, drop str coercion shims (#496)
* test: construct PathSpec at core-call sites instead of raw strings Sixty-five call sites across eighteen files passed string paths directly to core functions, leaning on the per-function coercion shim the next commit deletes. * refactor(core): PathSpec-only core, drop str coercion shims Core ops now require PathSpec; the Ops facade remains the only str-to-PathSpec boundary (mount-aware), and helper-injected fns keep the to_pathspec adapter in utils/wrap. Deletes 191 isinstance-str coercion blocks across 166 core modules, the str branch in resolve_glob_with, and the sharepoint resolver str fork. Production fixes surfaced by the removal: ram/redis read_stream now forward the PathSpec to stream() instead of a computed string key; generic tar keeps the archive operand as a PathSpec and builds specs for extracted members; generic patch reads targets via PathSpec and builds specs for parsed patch paths. Tests that exercised the deleted coercion are removed; remaining test call sites construct PathSpec explicitly or bind the production call_readdir/call_stat/call_read_bytes adapters. * fix(core): forward PathSpec in redis/ram read and realpath/iconv operands The coercion-shim removal exposed three more production sites that passed computed strings to spec-only fns, all caught by CI (redis integ, notion integ, snapshot interop, redis-gated unit jobs): - redis stream() recorded the op with the PathSpec instead of the stripped key; redis/ram read() unwrapped to mount_path before calling read_bytes, which reads path.virtual. - realpath -e normalized the operand to a string before the injected stat; it now rebuilds a spec for the normalized path (notion realpath_dotdot case). - generic iconv passed output_path.mount_path to write_bytes. Adds workspace-level regressions for realpath -e with .. on a mount and iconv -o. * fix(commands): wrap derived output paths in archive generics as PathSpec gzip/gunzip/split/csplit/unzip build output paths by string arithmetic (suffixes, part names, entry names) and passed them raw to the injected write_bytes/mkdir. The deleted per-backend coercion shims used to absorb that; backends whose ops read the spec unconditionally (onedrive, nextcloud in the shared-truth suites) crashed with 'str' object has no attribute 'virtual'. Wrap each derived path with PathSpec.from_str_path in mount-relative space, matching the tar/patch fix in this branch. io.writes keys are unchanged. |
||
|
Zecheng Zhang
|
ce39e351ce | chore(generic): type accessor as Accessor, document probe except-pass sites | ||
|
Zecheng Zhang
|
4da17edb3f |
refactor(paths): pure-virtual PathSpec, mount stamps resource_path
- rename original -> virtual, as_typed -> raw_path - remove prefix field and strip_prefix/key properties - add required resource_path stamped by the mount at dispatch - add strip_mount/mount_key/rekey/mount_prefix_of in utils/key_prefix - mirror in TypeScript (resourcePath, rawPath, mountKey et al) |
||
|
Zecheng Zhang
|
870016c5b4 |
refactor(commands): generic dedup + cross-backend integ harness (#68)
* feat(utils): add stream/bytes conversion helpers
* feat(commands): add generic cat as async iterator
* refactor(commands): ram cat wrapper uses generic
* test(commands): expand generic cat coverage; fix $ on trailing partial line
Old _number_lines_stream had loose tests that missed three POSIX edge cases:
multi-digit line number alignment (%6d vs literal 5-space prefix), trailing
newline preservation, and $-marker placement on partial last lines. Added
unit coverage for all three, plus combined-flag scenarios and a
one-byte-at-a-time chunking test.
The trailing-$ test caught a real bug: my generic was emitting "hello$" for
`cat -E` on input without a final \n, but BSD/GNU cat only emit $ before \n.
Fixed by dropping the suffix from the trailing-partial-line branch.
* test(ram): add 1:1 test file for ram cat wrapper
Establishes src↔test correspondence (every src file has a matching test_*.py)
for the ram cat wrapper. Covers the wrapper's behavior end-to-end via
Workspace.execute, including regression tests for the two POSIX bug fixes
that landed via the generic refactor:
- cat -n multi-digit line number alignment (lines 1-12)
- no-trailing-newline preservation for both cat and cat -n
Plus multi-file concatenation, empty input, and the all-blank-lines edge case.
* refactor(commands): disk cat wrapper uses generic + 1:1 tests
The active disk cat is at disk/cat/cat.py (the disk/cat package shadows the
top-level disk/cat.py file, which is dead code). Wires through generic_cat,
preserves CachableAsyncIterator/IOResult shape, drops _number_lines_stream.
Tests cover the same POSIX edge cases as ram cat: multi-digit alignment,
no-trailing-newline preservation, empty input, all-blank-lines.
* refactor(commands): redis cat wrapper uses generic + 1:1 tests
Structurally identical to ram cat migration. Tests use the same REDIS_URL
skip pattern as tests/core/redis/conftest.py — they run when REDIS_URL is
set in the environment, skip otherwise.
* fix(commands): preserve cachable identity in cat wrappers for cache to populate
Bug: wrapping cachable in generic_cat unconditionally caused stdout and
io.reads[path] to reference different objects. mount.py:_wrap then wrapped
them separately, and wrap_cachable_streams only updated stdout when
identity matched (stdout is stream). The cache cachable was never iterated
by stdout materialization, so the background drain saw an exhausted source
and cached empty bytes.
Fix: only wrap with generic_cat when a flag (currently just -n) actually
requires line processing. For the plain 'cat path' case, return cachable
directly — matching the pre-refactor behavior and keeping stdout identical
to the value in io.reads[path].
Tests caught: test_cache_hit_serves_from_ram, test_grep_uses_cache,
test_fuse_read_uses_cache_when_populated (workspace cache layer).
* feat(commands): add generic head with -c fast path and -n line-buffered path
* refactor(commands): ram head wrapper uses generic + 1:1 tests
* refactor(commands): disk head wrapper uses generic + 1:1 tests
* refactor(commands): redis head wrapper uses generic + 1:1 tests
* test(commands): backfill generic cat/head coverage missing from old helpers
Audited tests against legacy head_helper/cat_helper test files. Added:
cat:
- empty input passthrough (no flags)
- binary passthrough across full 0..255 byte range
- show_ends only marks newlines, not other binary bytes
head:
- n=1 single-line output
- empty input (with and without -c)
- single line no newline with default n
- c=negative emits nothing
* refactor(commands): consolidate ram/disk/redis logic into generic modules
Phases K through Z extracted shared per-command logic into
mirage/commands/builtin/generic/<name>.py modules that accept VFS
callables (read_bytes, read_stream, write_bytes, stat, readdir, mkdir)
via dependency injection. Per-backend wrappers (ram, disk, redis) are
now thin shims that resolve globs, supply backend-specific callables,
and delegate to the generic.
Scope:
- 57 generic modules
- 170 backend wrappers reduced to shims
- 67 new generic-level tests under tests/commands/builtin/generic/
- Net -7,772 lines (178 files changed: +1,891 / -9,663)
Bug fixes converged during refactor (no backward compat needed):
- disk/md5 emitted only the digest; now matches coreutils `{digest} {path}`
- disk/sort and disk/nl only operated on paths[0]; now process all paths
- disk/awk lacked BEGIN/END/accumulator support; converged on ram/redis
- disk/unzip flattened archive directory structure; now preserves it
- disk/patch used offset-splice algorithm; converged on lockstep walk
(closer to real patch(1) semantics)
- rg failed to auto-recurse on bare directories; now matches ripgrep
- file command silently swallowed read errors; now logs to logger.debug
Cleanup:
- Deleted 3 dead top-level head.py files shadowed by head/ packages in
ram/disk/redis (Python package always wins over .py at same level)
- Removed 4 stray __init__.py files under tests/ per CLAUDE.md rule
- Replaced inline string literals with enums (FindType, LsSortBy)
- Removed bare `try/except: pass` for mkdir using idempotent
mkdir(parents=True) instead
* test(integ): add cross-backend equivalence harness with 240 cases
New /integ folder runs identical command suite across ram/disk/redis
and diffs each backend's output against committed integ/truth.txt
canonical fixture. CI workflow (.github/workflows/test_integ.yml)
boots a redis service and runs all three backends.
Also fixes bugs surfaced by running the suite:
- disk/cat: chain multiple files (was processing only paths[0])
- sha256sum: use original path string; -c mode resolves mount prefix
- md5: support stdin when no paths given
- jq: iterate per-record on .jsonl files (real jq semantics)
- xxd -r: strip offset prefix and ASCII suffix before unhexlify
- find -mindepth/-maxdepth: off-by-one across ram/disk/redis/s3
- find -o vs -or: handle both spellings
- awk: \$NF/\$NR resolve via field indirection; bare condition
programs (e.g. '\$2 > 28') treated as condition with default
print \$0; && and || compound conditions; BEGIN block executes;
END can read NR
- diff: default to GNU normal format (1,5c1,3 with < / > / ---);
-u flag required for unified output
- cmp: use 'char N' instead of 'byte N' (matches GNU/BSD)
Extracts:
- DiffOpTag StrEnum (equal/delete/insert/replace) in diff_types.py
- AwkCmpOp / AwkBoolOp / AwkBlock / AwkBuiltin StrEnums plus
FIELD_PREFIX / PRINT_STMT / CMP_OP_PATTERN constants in
generic/awk_types.py
* fix(integ): preserve trailing whitespace in truth.txt + redis find test
The trailing-whitespace pre-commit hook stripped tabs at line-end in
integ/truth.txt, which broke the diff against backends (paste pads
shorter columns with trailing tabs). Exclude integ/truth.txt from
end-of-file-fixer, mixed-line-ending, and trailing-whitespace hooks.
Also fix tests/core/redis/test_find.py {maxdepth, mindepth} that
asserted the previously-buggy depth math (matches the existing
ram/disk test updates).
|