strukto-ai--mirage/CLAUDE.md

CLAUDE.md

MIRAGE is a package that allows you to mount anything as a filesystem and make it usable by AI Agents.

Repo Layout

This monorepo hosts two sibling implementations:

• python/ — the Python package (mirage/, tests/, pyproject.toml, uv.lock).
• typescript/ — the TypeScript monorepo (packages/core, packages/browser, packages/node, etc.).
• docs/, examples/, .github/ — shared across both.

Run Python commands from python/, TypeScript commands from typescript/.

TypeScript packages

• typescript/packages/core contains runtime-agnostic primitives and shared logic. Code in core must work in both browser and Node.js runtimes; do not put browser-only or Node-only APIs there.
• typescript/packages/browser contains browser-only resources, commands, and workspace wiring. It depends on @struktoai/mirage-core.
• typescript/packages/node contains Node.js-only resources, commands, and workspace wiring. It depends on @struktoai/mirage-core.
• Put shared TypeScript behavior in core only when it works in both runtime packages. Put runtime-specific behavior in browser or node.

Python/TypeScript Parity

• Keep Python and TypeScript layout, architecture, and semantics mirrored as much as practical.
• When changing one implementation, check the other for the matching pattern or feature. If one side is more correct, use it to improve the weaker side instead of copying a bad design.
• For major Python or TypeScript changes, consider adding or updating integration coverage under integ/.
• The layout half of that rule is gated. scripts/check_layout_parity.py diffs the module-name sets of every mirage/<pkg>/ against its TypeScript counterpart (core/node/browser unioned onto one namespace, plus cli/server/agents by prefix), folding camelCase, hyphens and a leading underscore so a rename reads as a rename rather than a missing module; __init__.py and index.ts are skipped because only Python needs one per directory. Intentional differences live in spec/layout_exceptions.json with a reason, and a stale entry fails as loudly as a new gap. --strict (what CI runs) does not demand zero: it fails when the count moves off the committed baseline in either direction, so new drift is blocked and closing a divergence has to be locked in by lowering the number. Run it without --strict for the full advisory report; that report is how layout work gets scoped.
• mirage ships no filetype renderers, and no factory for them. Parquet, ORC, feather/arrow/ipc and hdf5/h5 rendering are gone, along with the parquet/hdf5/pdf extras, the hyparquet/apache-arrow/h5wasm dependencies, and the whole commands/builtin/filetype_factory/ package in both languages (with its filetype_read / filetypeRead op knobs). A file with an unregistered extension is read as raw bytes. The one surviving extension point is registration on a mount: a command or op carrying a filetype resolves as (name, filetype) before (name, resource) before (name,). examples/{python,typescript}/filetype/ register a .tally renderer end to end and are gated in CI against integ/truth/*/filetype.json; tests/commands/custom/test_filetype_fns.py and test_unregister_removes_all_filetypes cover the unit path.

Module Layout

Packages split by role, one module per concern, the same way in both languages:

• types.py — data shapes only: frozen dataclasses, type aliases, Literal unions (e.g. runtime/types.py holds RunArgs/RunResult/EvalValue/EvalResult/ScriptSource). No logic.
• errors.py — the package's exception types (e.g. runtime/errors.py holds EvalError, policy/errors.py holds PolicyError).
• config.py — configuration knobs and their coercion (e.g. runtime/config.py holds RuntimeConfig, which fails loud on unknown fields).
• mixin.py — opt-in capability mixins: stateless, no constructor, abstract methods only (e.g. runtime/mixin.py holds EvaluatorMixin). Capability is detected by type (isinstance), never by probing for a method.
• base.py — the package's core ABC and nothing else (e.g. runtime/base.py is just Runtime).

History

Command history is a recording, not a command log. A hidden Observer records every top-level command as timestamp-ordered events (COMMAND, CLEAR, DELETE, op events); the user-facing surfaces are just views of those events.

• Observer + ObserverStore. The Observer owns a storage-agnostic ObserverStore (append/write/readAll/readMatching/clear/close), not a mount. Stores: RAMObserverStore (core, default), DiskObserverStore and RedisObserverStore (node). RAM is just the default, history can persist to disk or Redis.
• Two views over the same events. /.bash_history is a read-only view mount (HistoryViewResource) rendered in GNU bash histfile format (#<epoch> line then the command), so cat/grep/tail/find work on it for free. The history shell builtin (GNU -c -d -a -n -r -w -s -p + count) routes through the same mount, so file and builtin never disagree.
• Recording scope. Top-level lines record; nested evals ($(), eval, source, xargs) run with record: false, so their inner ops bubble to the parent and no spurious command is logged (mirrors GNU's line reader).
• Snapshots. History is captured as events into snapshot state and restored on load.
• Format is GNU bash, not zsh (#<epoch>, not : <ts>:<dur>;<cmd>).

CLIs

An installed CLI is a typed program tree (CLISpec) bound to a head word on the workspace. It is dispatched by name, never by operand path: the VFS is how an agent discovers state, the CLI is how it acts.

• An account CLI consults no mount; git is the one that does. The two are different tiers, told apart by config_model. An account CLI (slack, linear, gws, …) declares one, initializes from it, and reaches a service, so a mount would be a second source of truth for the same account. git declares none, because there is nothing to authenticate to, and its subject is a repository that lives on a mount, so it reads that repository through the op dispatcher like any command. That is what makes git work on a RAM mount, a disk mount or an object store without knowing which. It reaches the dispatcher through CLIDoors, one door per state plane: dispatch and stat_path for data, ns for the name plane (mount boundaries, links, attr overlay), and session_view for session state. The mount prefix is ns.mounts.root_of, not a fourth field. handle_cli/handleCli puts the record on inv.doors; the field is None/absent outside a workspace, and a verb that never reads it cannot touch a mount, so this stays opt-in per verb rather than ambient. The door is one field read rather than a parameter list the dispatcher inspects, because every leaf takes exactly one CLIInvocation and nothing is threaded through keyword injection. Every field is spelled the way CommandOpts (commands/config.py) spells it, so one fact reached from a CLI leaf and reached from a command handler has one name; meta-tests pin that (tests/commands/cli/test_doors_parity.py, and a compile-time mapped type in commands/cli/types.test.ts). Do not give an account CLI a mount, and do not give git a config_model. Nuance: an account CLI verb MAY read an unrelated workspace file the user named on the line through inv.doors.dispatch (himalaya's --attach reads the attachment path this way); what it must not do is treat a mount as a second view of its own account's data.

• The lifecycle is host-side only. register_cli/unregister_cli (workspace.py, workspace.ts) are called by the embedding program, never by a line the agent types, and there is no install/uninstall shell builtin. Keep it that way: an agent must not be able to take away the tools it was given. Shadowing is the one thing it can do (define a shell function with the same name), which is bash's own rule, reversible with unset -f, bypassable with command <name>, and visible in type -a. A deployment that needs a head word pinned enforces that in the policy layer's pre_execute, not in the CLI registry.

• Precedence is written down once, in _layers/layers (workspace/route/route.py, route.ts): shell builtin, namespace command, function, CLI, mount. route takes the first match (the winner, which is what dispatch runs) and route_all/routeAll takes all of them (every layer, which is what type -a prints). The generator is lazy so the winner still costs one probe. Do not add a second precedence list.

• A CLI that mimics a real program is gated against that program. ntn is the worked example: every case in integ/cli/ntn.json is asserted twice, once by the shared battery inside a mirage workspace and once by integ/ntn_conformance.ts, which runs the same shell line with the real npm ntn binary pointed at the same fake through NOTION_API_BASE_URL. A golden both agree on is by construction what the official CLI prints, so the grammar cannot drift from upstream without a red build. Four rules keep it honest. The binary's version is pinned (the script refuses any other). Both runs share one environment, declared as env on the target in integ/targets.json and read from there by both hosts and by the conformance runner: an option that reads a variable renders differently with and without it, so two environments would mean two different lines were being compared. That map must carry NOTION_API_VERSION, and the runner fails loudly if it does not, because unset ntn resolves the newest version by fetching developers.notion.com at startup and every case starts depending on the network. A case the upstream binary cannot answer carries a conformance_skip string saying why, never a silent omission; there are none today, and the only two cases the runner passes over are the ones whose line reads a /notion mount path, which no bare binary can have. And the harness spawns asynchronously: the fake is served by the same event loop, so a synchronous spawn deadlocks the loop that has to answer the child's request, which is the FUSE self-touch trap in another costume.

• UsageStyle is how a mimicking CLI answers in its original's voice, and it is read off the ROOT spec at every level, never off the node, because a program answers in one voice throughout. It lives in commands/spec/types.py (types.ts), not beside the CLI tree, because the help renderer is the spec's and cannot import upward. ARGPARSE is the default; GIT rewords the unknown-option refusal and its exit code; CLAP additionally governs help layout (bare description line, [OPTIONS]/<COMMAND>, Options: not Flags:, subcommands in declaration order rather than sorted) and the missing-operand refusal. ntn is the CLAP one. Adding a dialect means adding a member here, not a second knob: help layout, refusal wording and exit code are one decision about whose program this imitates.

  Three spec fields exist only to serve those foreign usage lines, and all three hold bare names with no brackets, because only the renderer knows the dialect that wraps them: Operand.name (PAGE_ID, rendered <PAGE_ID> required and [PAGE_ID] not), Operand.required (the parser reports the empty slot rather than each leaf re-discovering and rewording it), and Option.metavar (VERSION, rendered --notion-version <VERSION>; derived from the long spelling when absent, which covers most options and is why only the four upstream overrides declare one). Option.env is a fourth and is not a synonym for default: an env-sourced value counts as supplied, so clap echoes it in a usage line where a defaulted one is invisible, and it is read from the session rather than frozen into the spec. The executor fills it, so a leaf reads one flag instead of a flag and a fallback.

• Imitating a Rust CLI means imitating serde_json, so the engine parser does not get to decide. ntn api echoes its JSON parse failures verbatim, and those words are serde_json's (EOF while parsing an object at line 1 column 1), which neither json.loads nor JSON.parse can produce and which the two of them word differently from each other anyway. So commands/cli/builtin/ntn/serde.py (serde.ts) is a scanner whose only job is that message, and it is the authority on validity too: the engine parser runs only on input the scanner already accepted, because the two disagree about what JSON is (python accepts NaN, serde refuses it, and silently sending NaN is worse than either). Three rules that are easy to get wrong and are pinned by a probed table in test_serde.py / serde.test.ts: columns count bytes, not characters (["é"x fails at column 6); a newline advances the line and zeroes the column; and the recursion limit is 128, refused at the opening bracket of the 128th container. Do not "simplify" this to a try/except around the engine parser.

• ntn api has three body sources, and both the order and the exit codes are observable. stdin, --data and inline path=value / path:=json inputs are each validated in that order, and only then is "more than one source" reported, so a malformed pipe outranks a malformed --data and both outrank the conflict. The exit codes are two families, neither of them argparse's 2: a body that arrived malformed is 1 (error: Invalid JSON from stdin), while a line the CLI will not interpret at all is 5 (the conflict, an unparseable inline input, an empty --data), and those carry a second hint: line. There is no object check anywhere: --data '[]' posts the array, and any body source at all makes the call a POST even when what it carries is empty, so --data '{}' posts rather than falling back to GET on falsiness. name==value stays a query parameter whatever the method is.

• Discoverability is part of shipping a CLI, and it comes from the spec, so it works for a user's own registered CLI exactly as for a builtin one. man <cli> and man <cli> <verb>... render through node_help/nodeHelp, the same renderer --help uses, so a manual cannot drift from the program; bare man lists installs under # clis. type reports an installed CLI as its own kind (type -t prints cli, a sixth word beside bash's five, because reusing file would promise type -p a path that does not exist). which prints the bare name, never a fabricated path.

Notion data sources

The notion backend speaks Notion-Version: 2025-09-03, the generation that split a database into a container plus one or more data sources. The split is not cosmetic and it is not optional:

• The database object no longer carries properties. The column schema lives only on the data source, so GET /v1/databases/{id} answers with data_sources: [{id, name}] and nothing to render a schema from. Do not synthesize one back onto the container; the fake deliberately omits it so anything that still reads a column list off a database fails loudly.
• The mount nests accordingly, because that is where the data is: databases/<Title>__<db-id>/database.json is the container, and databases/<Title>__<db-id>/<Name>__<ds-id>/data_source.json is the schema, with the row pages under the data source. A row page therefore sits at depth 4 under databases/, not 3; readdir/read/stat all key on that in both languages. The name stutters for a single-source database because Notion names the auto-created data source after its database, which is honest and disappears the moment a database holds two.
• A row's parent is its data source: {type: "data_source_id", data_source_id, database_id}. The database id rides along because Notion kept emitting it, and the fake stores rows keyed by database, which is the same fact one derivation away.
• /search rejects filter.value = "database" at this version; the searchable schema-bearing object is data_source. Listing databases is therefore "search data sources, take the distinct parents, retrieve each", which costs one extra call per database and is the only way to get the title and url the directory is named and rendered from.
• ids are not interchangeable: a data source id is not its database id. The fake derives one from the other (d5000000 + the database's tail) rather than reusing it, precisely so an id mix-up cannot pass.

Mount boundaries

A mount root is not an ordinary directory, and a mount nested inside another mount's tree is invisible to the backend that owns the parent: the child's keys live in a different resource, so the parent's readdir never lists it. Two mechanisms follow from that, and they are separate.

• MountView is how a command sees the boundaries (ops/types.py, ops/types.ts), and it is offered the way LinkView is: a command opts in by naming a mounts parameter, execute_cmd/executeCmd delivers it only to handlers that do, and there is no list of boundary-aware commands anywhere. It carries descendants (mount roots strictly under a path), is_root, and root_of. A traversal command that renders lines does not need it: the executor's fan-out (workspace/executor/fanout.py) already reruns find/du/tree/grep -r per mount and concatenates the output. A command whose output is one binary object cannot be merged that way, which is why tar reads the boundaries itself.
• Crossing into a descendant mount is refused, not attempted. tar and zip both keep the mountpoint as a directory entry and drop its contents with GNU's own --one-file-system wording (<name>/: file is on a different filesystem; not dumped; Info-ZIP has no message of its own for this, so zip borrows the wording under its own zip warning: prefix). This is deliberate: descending would archive by accident exactly what the mount-root refusal below forbids on purpose.
• MountRootPolicy refuses a mount root in a source slot for tar -c, zip and cp, on top of the POSIX EBUSY rules it already enforces for rm/rmdir/mv/mkdir/touch/ln. Real tar and cp allow it; mirage does not, because the mount table is the deployment's configuration and reading a whole backend into one object is neither what the operand looks like it costs nor something an agent should be able to do to data it was given a view of. Only positional operands are tested, which is why CommandContext carries operands beside paths: tar -xf a.tar -C /mnt extracts INTO a mount and must stay legal, while tar -cf a.tar /mnt must not. positional_scopes / positionalScopes (executor/command/routing) is what tells the two apart, since classification turns every path-shaped word into a PathSpec whether it filled an operand slot or a flag's value. Mode matters too: only tar -c reads its operands from the filesystem, so is_create_mode / isCreateMode gates the refusal. Under -t and -x an operand is a member selector matched inside the archive, and refusing one that happens to spell a mount root would deny an ordinary listing.

CommandSpec and Operand are not a scratchpad

Both dataclasses are shared by every command in the repo, so a field added for one command is a field every other command's author has to read past, and a fourth one turns the grammar into a pile of per-command dialects. Do not add a field to CommandSpec, Operand or Option unless POSIX and argparse both already have the concept, and then name it after theirs, not after the mechanism it trips inside the parser.

Both, not either. The test is literal, not a judgement call: write the equivalent line in argparse and run it. If argparse parses it, the concept is borrowed and the field is allowed. If argparse cannot express it, the behavior belongs to the one command that needs it and must be handled in that command, not in the shared grammar.

POSIX alone is not enough, and python3 is exactly why: POSIX specifies an option whose argument is a program (sh -c command_string [command_name [argument...]], and python3's own synopsis python [option] ... [-c cmd | -m mod | file | -] [arg] ...), so an either-or rule would license the Option field this section exists to refuse. argparse is the narrower gate because it is the parser this spec layer is modelled on, so a concept it cannot express is one the grammar has no shape for.

Both halves of python3's command line are the worked example, and they come out on opposite sides:

• Allowed: Operand.remainder. It is nargs=argparse.REMAINDER, which is POSIX's own option order (the first operand ends option parsing; GNU's permuting default is the extension, and POSIXLY_CORRECT=1 ls a -1 shows the difference). argparse spells it on the positional slot, so mirage spells it on Operand too, and CommandSpec does not change at all.
• Not allowed: an option whose argument is a program. python3 -c 'code' -u x must hand -u to the code, but add_argument("-c"); add_argument("rest", nargs=REMAINDER) answers unrecognized arguments: -u. CPython's own command line is parsed in C for exactly this reason. There is no concept to borrow, so this does not become an Option field.

A CLISpec is a CommandSpec (python: subclass; TypeScript: extends), and every level of a CLI tree parses with the ordinary spec machinery. Moving a command to the CLI tier therefore does not exempt it from this rule, because it still parses with the same Option and Operand. The CLI tier is for a program tree (a verb the line selects, like git status or ntn api), not for a program with an unusual option grammar.

An option that chdirs: operand_base

tar -C is not a flag the command reads once, it is a chdir for the path operands typed after it, and it is cumulative (-C d1 x -C ../d2 y reads d1/x and d1/../d2/y). That is a property of the line, so it is declared in the spec (CommandSpec.operand_base / operandBase, tar's only) and resolved by the one component that walks the line positionally: parse_command / parseCommand tracks the base as it scans and reports it per word as word_bases / wordBases, which classify_parts then resolves each operand against. Doing it anywhere later is too late: the classifier has already produced absolute PathSpecs, and an operand resolved against the wrong base makes the router see a phantom cross-mount span (which is what tar -czf /work/out.tgz -C /work/check my_paper used to fail as). Only path operands and the option's own value move; every other path-valued flag keeps resolving against the session cwd, which is what GNU does with -f.

Symlinks

Symlinks are namespace state, not backend state. The Namespace node table owns them (target stored verbatim as typed), no resource stores or reports one, and no backend readdir or stat can see one. Three consequences, in the order they bite:

• A new resource needs no symlink code at all. Links live above every backend, so a backend author never implements, stores, or forwards one. This is the whole point of keeping them in the namespace; do not push link awareness down into a resource or an accessor.
• A command consumes a fact by reading the opts field, nothing else. Every namespace fact (links, stat_overlay, stat_path, readdir_path, child_mounts, mounts) rides CommandOpts into every handler, identically in both languages; the generic that wants one reads opts.links and the rest ignore it, so there is no opt-in registry, spec field, or signature convention that can fall out of step. LinkView bundles every link fact (stat_at, children, subtree, resolve, exists, target_stat) so a command that grows a new need adds a field read, not a new keyword threaded through execute_cmd, the builder and the generic. Families reading links today: ls, stat, find, du, file — in the generics, so a bespoke wrapper that delegates (as all of them now do) inherits link awareness for free. exists and target_stat answer through the op dispatcher, not one backend's stat, so a link that points into another mount resolves correctly.
• A CLI verb that walks a tree itself has to lstat through the same door, which on that tier is CLIDoors.ns.links. A walk built on stat_path alone dereferences, so a link reads as its target and a broken one reads as absent. git's worktree walk did both until it opted in, which is why git add stored a symlink as a regular file holding its target's bytes (real git stores mode 120000 with the target as the blob) and why git status never listed a broken one.
• Merge links in the generic, above the native-op/walk fork. find and du each have two paths: a backend with a native op (find_core, du_size/ du_entries) and a backend walked by readdir. Link merging lives in one shared place per family (link_results in generic/find.py, link_leaves in generic/du.py) that both paths call. Merging inside only one path makes a mount's symlink behavior depend on whether its backend happens to ship a native op, which is the worst kind of divergence to debug.

Follow policy is two symmetric tables in workspace/route/constants.py, both read off the raw command line (operand rewriting happens before flag parsing): NO_FOLLOW_COMMANDS lists commands that lstat (rm, mv, ln, readlink, rmdir, unlink, stat, file, du, find, tar, zip), with DEREFERENCE_FLAGS naming the flag that turns following back on (-L). tar and zip are in that list for a different reason and deliberately carry no DEREFERENCE_FLAGS entry: they dereference too, but their planner has to be the one doing it. Rewriting the operand in the router hands the planner a target it can no longer tell was reached through a link, so tar stored a regular file where GNU stores a symlink member, and neither archiver could apply its own cross-mount refusal or ELOOP wording. tar -h and zip -y are read by scan_operand instead. find states its policy as a leading -P/-H/-L option instead, last one wins, so it lives in LAST_WINS_LINK_OPTIONS; NO_FOLLOW_FLAGS is the mirror, for a following command that a flag makes lstat (ls -l and ls -d report a command-line link itself, while a bare ls dereferences a link to a directory, and ls -L overrides both).

Those tables only cover the operand; honoring -L below it is the generic's job, not the router's.

Rendering derives from one fact: link_stat builds the row with FileType.SYMLINK and the target under FileStat.extra (LINK_TARGET_KEY), so lrwxrwxrwx, the name -> target column, -F's @, and file's "symbolic link to" all follow from it without a second lookup.

du sizes a link at its target string's length. This is not a divergence: mirage's du counts bytes, which is GNU's --apparent-size --block-size=1 (du -b) mode, and in that mode GNU reports a symlink as len(target) too. The familiar 0 comes from GNU's default 1 KiB block mode, where a short target sits inline in the inode and occupies no data blocks (stat reports size=15 blocks=0 for it). mirage has no block mode at all, so 0 is not an option it can express; comparing against it would also make every regular file look wrong (a 6-byte file is 6 in bytes, 4 in 1 KiB blocks).

FUSE

• The mount layer is split core/adapter in both languages. MountCore (python/mirage/fuse/core.py, typescript/packages/node/src/fuse/core.ts) owns all filesystem semantics in POSIX terms and imports nothing from mfusepy or @zkochan/fuse-native; MirageFS in fs.py/fs.ts is the libfuse adapter and owns only the callback signatures plus errno translation. Core methods raise ordinary exceptions; adapters classify them through classify_error/classifyErrno (fuse/errors.py, fuse/errors.ts), which is one shared table, not per-method except arms. Put new filesystem behavior in the core, not the adapter.
• One backend field per mount: vfs | fuse | fskit (MountBackend in mirage/types.py, beside MountMode). vfs is the default and means the mount lives only inside mirage's own filesystem; fuse and fskit also register a real mountpoint, with mountpoint pinning where. There is no fuse=True boolean any more, and no auto backend. Mount(..., backend=MountBackend.FSKIT) routes through macFUSE 5.x's FSKit shim (no kernel extension). Rules live in fuse/backend.py and are enforced at mount time: macOS-only, mountpoint must be under /Volumes, and every mounted resource must set SIZES_ALWAYS_KNOWN (FSKit has no direct_io, so a size-unknown resource would serve silent empty files). resolve_backend rejects vfs: reaching it means a kernel mount was requested. In YAML the keys are backend: and mountpoint:. TypeScript serves fskit too: fuse.node links /usr/local/lib/libfuse.2.dylib by absolute path (the bundled libosxfuse.2.dylib is a stub with that install name), so backend=fskit + volname reach macFUSE 5.x's own libfuse via appendMountOptions in mount.ts. Verified live. Caveat: a TS fskit mount intermittently wedged on a write op in testing (probed from child processes), and a dead FSKit volume blocks mount-table enumeration system-wide until the macFUSE appex process is killed; treat fskit from TS as read-mostly and experimental. A mount is ready only when os.path.ismount says so, never when the /Volumes entry merely exists: macFUSE creates that directory while mounting and leaves it behind if the FSKit handoff fails, which reads as a live mount and then fails with ENOENT on the first read. Python fskit mounts have the full write surface, and only because of fuse/darwin.py. The FSKit shim finalizes every created item through macFUSE's Darwin-only setattr_x and routes rename through renamex; mfusepy leaves those fuse_operations slots as reserved NULLs, so without the extension module create/mkdir fail with ENOSYS after the op already applied and rename never reaches userspace (verified by libfuse wire trace: CREATE success, then SETATTR -78). Do not remove the install_macfuse_extensions() call in mount.py, and keep the struct tail in sync with macFUSE's fuse.h if mfusepy changes layout. Pinned in integ/fuse/truth_fskit.json on macFUSE 5.3.3 / macOS 26. TS fskit stays read-mostly: fuse-native's compiled op table cannot gain new C callbacks from JS. Upstream shim caveats that remain for both: exec-until-first-read fails (macfuse#1181) and root readdir cache cannot be invalidated (macfuse#1165). integ/fuse/fskit.py is the only coverage of a real FSKit mount, and it only runs on a Mac with macFUSE 5.x: on the integ-fskit-macos job a hosted runner installs and enables everything but the mount request never reaches macFUSE, so the job reports that one known timeout signature as a skip and stays green; any other failure there is real.
• Directory and unknown sizes. getattr reports st_size 0 for directories and for API-backed size-unknown files that have not been opened recently. Reads stay correct because Python mounts with direct_io (kernel reads to EOF regardless of st_size) AND attr_timeout=0 (post-open fstat routes to getattr(path, fh), which serves the real size of the open-hydrated content); prefetched bytes live in a 30s TTL cache (PREFETCH_TTL) so release-then-stat does not refetch. All three pieces are load-bearing: without attr_timeout=0, wc -c prints 0, BSD cp copies 0 bytes, and tail -c dumps the whole file; without direct_io, cat reads 0 bytes on macOS. Do not "fix" getattr to report real sizes eagerly (one API fetch per ls -l entry), and do not report fake sizes: stat-only tools (tar, rsync, test -s) seeing 0 matches procfs precedent. TypeScript uses the same recipe: @zkochan/fuse-native doesn't serialize a direct_io option, so mount.ts appends it to the option string at runtime (appendDirectIO; a pnpm patch would not reach consumers), plus attrTimeout: '0' + fgetattr. The old 100 MiB sentinel is gone; do not reintroduce it.
• TS FUSE mounts are served by the mounting process's event loop. Never touch your own mountpoint synchronously (readFileSync, statSync, execFileSync) from the process that created the mount: the call deadlocks the loop that must answer it, the kernel times out, and every later op fails with Device not configured/ENOTCONN — which looks exactly like a broken mount. Probe from a child process or use async APIs (see examples/typescript/fuse/helper.ts). Python is immune (FUSE loop runs on a thread).
• FileStat.size must be the rendered content's byte length or None, never a storage-side or source-side number. A confidently wrong size is worse than an unknown one: over FUSE it makes wc -c/ls -l lie and risks truncated copies, while None rides the unknown-size machinery above. Postgres rows.jsonl (on-disk table_size_bytes vs rendered JSONL), Dify documents (uploaded source size vs rendered segment text), Gmail messages (sizeEstimate vs rendered .gmail.json), Drive-rendered google-apps files (Drive storage size vs rendered .gdoc/.gsheet/.gslide JSON; raw binary downloads keep Drive's size), and Microsoft Graph folders (OneDrive/SharePoint report an aggregate subtree size on the folder facet, not any content length) made this mistake; their storage/source numbers now live in extra (size_bytes / source_size / size_estimate). Do not reintroduce it in new backends. Graph's folder.childCount rides along in extra too, and is what find -empty reads for a directory.
• macOS allows only one FUSE mount per process. The second mount dies with fuse: cannot register signal source (mfusepy registers libfuse signal handlers, which only the first mount in a process can claim). Multi-mount scenarios (integ/fuse/fuse.py mounts two) pass only on Linux; do not debug them as regressions on macOS. A failed run leaks the first mount: list with mount | grep MirageFS, clean with umount <mountpoint>.
• Windows (WinFsp) conventions differ in three ways, all handled in the python mount path (_prepare_mountpoint, _await_ready, the teardown branches): the mountpoint must NOT exist (WinFsp creates it; an existing dir fails with "mount point in use"), os.path.ismount never sees WinFsp directory mounts (readiness = bare existence), and there is no fusermount (WinFsp unmounts when the serving process exits). Ownership: mount with uid=-1,gid=-1 (WinFsp builtin: files owned by the mounting user); never report raw POSIX ids into the SFU/Cygwin SID mapping, and os.getuid does not exist there (MirageFS caches a guarded uid/gid once). Behavior quirk: Windows cannot stat without opening a handle, so size-unknown files hydrate on first stat and report their real size even "pre-open" (multi-mount per process works). The integ-fuse-windows job is advisory (not in the gate).

Development Setup

This project uses uv for Python dependency management. Install dependencies with:

cd python && uv sync --all-extras --no-extra camel

camel is declared as conflicting with openai (and other extras) in pyproject.toml, so uv sync --all-extras fails. Exclude camel to keep the openai stack.

Running examples

Examples under examples/python/ load .env.development from the repo root (cwd-relative). To keep cwd at the root while using the python/ venv, invoke the venv interpreter directly:

./python/.venv/bin/python examples/python/s3/s3.py

Avoid uv --directory python run ... for examples — it changes cwd to python/ and breaks load_dotenv(".env.development").

Backward Compatibility

• No need to consider backward compatibility for the code.

Create a PR

When asked to create a PR, please follow the following steps:

1. Run pre-commit run --all-files from the repo root to lint and format the code.
2. Run cd python && uv run pytest to run the Python tests.
3. Run git add -A to add all changes.
4. Run git checkout -b <branch-name> to create a new branch.
5. Run git commit -m "<commit-message>" to commit the changes.
6. Run git push origin <branch-name> to push the changes to the remote repository.
7. Run gh pr create --title "<pr-title>" --body "<pr-body>" to create a PR.

Commands

Linting and Formatting

After making major changes, run pre-commit from the repo root to ensure code quality:

./python/.venv/bin/pre-commit run --all-files

Invoke the venv's pre-commit binary directly (not via uv --directory python run) so cwd stays at the repo root — otherwise git ls-files only lists files under python/ and examples/ gets silently skipped.

Type Conventions

• Paths must always be represented as PathSpec, never raw strings. All functions that accept or return paths use list[str | PathSpec] where str is for text arguments and PathSpec is for paths. Never pass a path as a plain str — wrap it in PathSpec.

Rules

• Shell-style commands (cat, grep, du, find, head, tail, wc, ls, etc.) follow POSIX / Unix coreutils semantics as much as possible; match BSD/GNU behavior and document any deliberate divergence. Pin exact GNU behavior with docker (debian:stable-slim) before changing command semantics.
• find -size is strict and rounds up. GNU +N keeps ceil(size/unit) > N, -N keeps ceil(size/unit) < N, bare N keeps ceil(size/unit) == N (so -size -1k matches only empty files and +0c excludes empty ones). The parsers (_parse_size / parseSize) translate this once into inclusive byte bounds; backend cores just keep min_size <= size <= max_size and must not re-interpret the spec. Deliberate divergence: directories count as size 0 (GNU compares the inode size, e.g. 4096 on ext4), which matches what find sees over a mirage FUSE mount.
• find's row for the start point is the generic's, not the backend's. GNU lists a start point before descending into it, and every native find op used to decide that row from its own listing, which is only a proxy for existence: an object store holding no keys under the prefix and no directory marker reported nothing at all for a directory test -d and tree both saw, and ssh called every directory non-empty so -empty never matched one. The generic stats the start point through the dispatcher (resolve_start, which asks both channels a backend can answer on via resolve_path_stat, since on a prefix store a directory is the set of keys under it rather than an object), takes one readdir for -empty (dir_empty, wired by the builder), and then replaces whatever row the backend produced for it (with_root_row / withRootRow). A native find op may still emit the start path and they all do, because it is the answer when no dispatcher probe is wired (a command constructed outside a workspace); with one wired it is discarded. A new backend only has to report descendants.
• find classifies walked entries through stat, never by name. The walk's one in-band proof is a trailing slash on a cold listing (box, gdrive and dropbox mark folders that way, and no backend renders a file with one); every other entry is classified by the index-backed stat that the same readdir just populated, so the lookup is RAM, not another API call. There is no per-backend is_dir_name/isDirName hint: those heuristics guessed wrong as soon as a child's name was user-controlled (email and gmail attachments and slack uploads carry whatever name the sender gave them, and were reported as directories, so find -type f missed them). Do not reintroduce name-based classification in a backend; if stat misclassifies an entry, fix that backend's stat.
• An archiver walks a directory operand; it does not read it. tar and zip decide every member first (plan_create / planCreate in generic/tar/create.*, plan_zip / planZip in generic/zip_cmd.*) and only then write, which is what lets an exclusion prune a whole subtree and keeps the ordering stable. Both plans are built on one traversal, scan_operand / scanOperand (generic/archive/walk.*), which merges three sources no single one can see: the backend walk (reusing find's walk_find / walkFind, so an archiver classifies an entry through stat exactly as find does, never by name), the namespace's symlinks, and the mount table. It reports paths, never names, because naming is exactly where the two formats disagree; the two things they disagree about in the traversal itself are parameters (dereference, recurse), so a third archiver adds a caller, not a second walk. Members are named from PathSpec.raw_path, so tar -C d x stores x, not d/x. A directory is its own member, with GNU's trailing slash and no content, which is the only record an empty directory leaves and the reason extraction has to mkdir for one. A symlink is a symlink member (SYMTYPE, target in linkname), never a file of its target's bytes, unless -h says to follow it. Two links to one target are not a loop, and both are archived; the only loop is one resolve refuses to resolve, since the namespace already walks the chain under a hop limit and raises CycleError at the end of it. That arrives as a fatal Problem carrying GNU's Too many levels of symbolic links, reported per member with the directory entry kept, rather than as an exception that aborts the plan. Every -C is checked, not just the last: GNU chdirs at each one and fails at the first it cannot enter, so the option accumulates (multiple=True) and the planner walks the list. Two deliberate divergences from GNU, both documented in place: siblings are sorted rather than emitted in readdir order (the same choice du makes, for the same reason), and a descendant mount is never crossed (see "Mount boundaries"). Everything else is pinned against GNU tar 1.35 on debian:stable-slim: the leading-slash warning, Cowardly refusing to create an empty archive (exit 2), a per-operand Cannot stat plus one trailer (exit 2, and the other operands still archive), a -C it cannot enter (exit 2, no archive written), and archive cannot contain itself; not dumped (exit 0). A backend error must never reach the user as itself: an unreadable operand is reported in virtual path space with tar's wording, because the raw IsADirectoryError leaked the host path behind a disk mount.
• zip is Info-ZIP, which inverts tar's two defaults. A directory operand contributes only its own entry unless -r says to descend, and a symlink is followed unless -y says to store the link, where tar always descends and always stores unless -h. Both are just the recurse / dereference arguments to the shared scan. The rest is pinned against Info-ZIP 3.0 on debian:stable-slim: a leading slash is stripped in silence (tar warns, zip does not), -j junks to the basename and drops directory entries entirely, -x is anchored on the whole stored name (d/sub/* matches, sub/* does not) where tar's --exclude is unanchored, an unreachable operand is \tzip warning: name not matched: <name> and does not stop the run, and a run that matched nothing prints zip error: Nothing to do!, exits 12, and writes no archive. -q silences the warnings but never that error. Two deliberate divergences: -x takes one pattern per occurrence (mirage's spec has no variadic option value, and -x a -x b says the same thing), and the adding: line carries no (deflated N%) suffix, since the ratio depends on the compressor and would differ between the two languages.
• Tar formats come from a library, not from hand-rolled block code, and bzip2 is read-only in TypeScript. Python builds archives with stdlib tarfile; TypeScript uses modern-tar behind tar_helper.ts, which keeps the TarEntry shape (name/data/isFile/isDir/linkname) the two call sites already speak and is the only place the dependency is named. Do not go back to writing ustar blocks by hand: the version that did truncated any name past 100 bytes instead of using the ustar prefix field or a PAX header (so a deep member extracted to the wrong path), and read a PAX/GNU extension block as if it were a member (so tar -t on any archive GNU or Python wrote listed a phantom ././@PaxHeader row). writeTar/readTar are async for this reason. Compression is a registry (registerCompressionCodec): gzip is built in via CompressionStream, and a codec may be decompress-only, which is the one deliberate py/ts divergence here. Python reads and writes .tar.bz2 because bz2 is stdlib; TypeScript only reads one, because every JavaScript bzip2 compressor is GPL (compressjs, archive-wasm) and an Apache-2.0 package cannot ship that, while seek-bzip (MIT) decodes. tar -cj therefore exits 1 with tar: bzip2 not supported, the same answer browser core already gives for an unregistered codec. If a permissively licensed bzip2 compressor appears, adding compress to that one codec closes the gap with no other change.
• The TypeScript walkFind answers in mount-relative keys; the Python walk_find answers in virtual paths. TS's stands in for a backend's native find op, so a caller that needs virtual paths (tar does, to name members and compare against mount prefixes) lifts them with mountPrefixOf the way findGeneric does. That lift lives once, in generic_bind/archive_io.*, which is where both archivers get their walk. This asymmetry is real and has bitten once: a unit test on an unprefixed mount cannot see it, so cover a prefixed mount too.
• du has one backend contract: size and entries. Each backend exposes core/<backend>/du/size.py (recursive byte total for one path) and core/<backend>/du/entries.py (per-file breakdown), wired as du_size / du_entries on the adapter. entries returns (entries, total) where entries are leaf files only, in mount-relative path space, with no summary row; the generic lifts them onto virtual paths (to_virtual, via mount_prefix_of) and re-spells them as the operand was typed (respell_raw). A backend that returns backend-key paths, or appends its own roll-up row, makes two mounts holding the same filename render identical lines. Do not reintroduce a second shape; the old flat-list du_multi contract is gone.
• du prints a line per directory, derived not walked. GNU prints one line per directory with its recursive total, post-order (children before parents), plus one per file under -a. Backends only ever report leaf files, so the generic derives the directory rows by summing each leaf into every ancestor (rollup, same name both languages), then emits post-order with siblings sorted. Two deliberate divergences: GNU orders siblings by readdir (filesystem-dependent), mirage sorts them; and an empty directory is invisible to mirage because no leaf points at it. Sizes are bytes, not GNU's 1 KiB blocks, since an object store has no block size. --max-depth prunes only what is printed, never the walk, because every printed total still covers the whole subtree. Verify changes with the differential harness against debian:stable-slim: paths, exit codes and stderr must match GNU exactly.
• du usage errors exit 1, not 2. du is absent from USAGE_EXIT, which is correct: GNU du exits 1 for -s with -a ("cannot both summarize and show all entries"), -s with --max-depth ("warning: summarizing conflicts with --max-depth=N"), and a bad depth ("invalid maximum depth 'x'"). All three are raised by parse_flags / parseDuFlags before any I/O, mirroring GNU's option-parse order: the depth is parsed as the option is read, so a bad depth wins over the conflict checks. An unreadable operand is not a usage error: GNU names it (du: cannot access 'x': No such file or directory), prints every other operand, and exits 1, and still prints 0 total under -c when every operand failed. With no operand at all, du measures the working directory; it never says "missing operand".
• du walks are bounded. Backends with no native du op are walked one readdir at a time, which on an API tree is one request per directory. CommandIO.max_du_entries caps that walk; when it trips, du prints what it accounted for, writes a notice to stderr and exits 1 (GNU's behavior for a tree it could not fully read), rather than hanging or silently reporting a wrong number. Slack sets a low cap (DU_MAX_ENTRIES) because it exposes a directory per conversation per day against a ~50/minute rate limit.
• Async-native by default. I/O uses aiofiles / redis.asyncio / aioboto3, and command pipelines are async generators.
• Python unit tests mirror src 1:1 where reasonable. Try to have a matching tests/<path>/test_a.py for each source file mirage/<path>/a.py. __init__.py, pure type-stub modules, and trivial re-exports are fine to skip; modules with real logic should have one.
• Do not add __init__.py files under tests/. Tests are namespace packages and pytest discovers them without __init__.py. Don't create one when adding a new test directory.
• Monkeypatching a backend command module in tests: the command imports its helpers by value (from mirage.core.<backend>.read import read_bytes), so to intercept them you must rebind the name inside the command module, not the core source module. But the command module is hard to reach: the backend package re-exports the command function in __init__.py (from .cat import cat), which shadows the submodule of the same name, so import mirage.commands.builtin.<backend>.cat as mod, from ...<backend> import cat as mod, and even pytest's string target monkeypatch.setattr("mirage.commands.builtin.<backend>.cat.read_bytes", fake) all resolve to the function, not the module (AttributeError). The command is also wrapped by @command, so cat.__globals__ is the decorator's module. Reach the real command-module namespace through the unwrapped function and patch the dict: monkeypatch.setitem(cat.__wrapped__.__globals__, "read_bytes", fake).
• Avoid add any comments or docstrings on the top of the file.
• A nested function must capture the scope around it. Nesting is for closures: the def reads a name from the enclosing function, or binds one through a parameter default (def f(m, _n=n), the loop-variable idiom). Everything the architecture nests is that shape — op factories (ops/generic/factory.py), provision builders, the read-through cache, and every decorator's wrapper — and those stay. What is banned is the nesting that buys nothing: a helper written inside a function although it reads only its own arguments, which rebuilds a function object per call and hides a testable unit where no test can reach it. Put that one at module level. Enforced for Python by tests/test_nested_functions_are_closures.py.
• Add type to Args for docstring.
• Do not add comment after each line of code in the format of "# 10MB - trigger segmentation for files larger than this". The most you can add is "# 10MB".
• For all imports you need to put to the top of the file. Don't have imports within each function.
• No circular imports. If putting an import at the top would cause a cycle, that's a sign the dependency direction is wrong — fix the design (dependency injection, splitting modules, moving the shared piece to a leaf), don't paper over it with function-local lazy imports. Verify by checking that running cd python && uv run python -c "import <every changed module>" succeeds without ImportError.
• Never silently swallow exceptions. try: ... except: pass (or except SomeError: pass) hides real bugs. If you genuinely need to ignore an error, log it (logger.debug(...)) or document loudly why it's safe. Default behavior should be: let the exception propagate. Especially never silently swallow RuntimeError — it usually signals something deeper (event loop in wrong state, recursion limit, etc.) that you need to actually fix.
• Never call asyncio.run() inside a sync function that might be invoked under an outer event loop. It will raise RuntimeError: asyncio.run() cannot be called from a running event loop. If you need async behavior from a sync API, either: (a) make the calling function async, (b) operate on the underlying sync state directly (e.g. write to a dict instead of calling an async setter), or (c) use a sync alternative of the same library (e.g. redis.Redis instead of redis.asyncio.Redis). Do NOT wrap with try/except RuntimeError: pass — that masks the bug AND leaks the unawaited coroutine.
• Please don't change any file name unless I ask you to do so.
• Don't add too many printings or comments in the code.
• Don't add README.md unless I ask you to do so.
• Use uv add to install new dependencies.
• Command handlers take (accessor, paths, texts, opts) — the same four positionals in both languages. The dispatcher (Mount.execute_cmd / Mount.executeCmd) constructs one CommandOpts per invocation carrying stdin, the flag bag, cwd, the mount prefix, the index, and every namespace fact; the provision path builds the same bag with command/spec set (ProvisionFn has the same four-positional shape). Handlers never declare a flag or an injected fact as a parameter — tests/commands/test_no_dead_flag_params.py pins the exact signature. When a wrapper genuinely needs a flag value itself (e.g. a search push-down), read it through a spec-bound FlagView (fl = FlagView(opts.flags, spec=SPECS["grep"]) then fl.as_bool("F"), fl.as_int("m"), fl.as_str("type"), fl.as_list("e")) or a shared domain accessor like pattern_arg — never raw flags.get(...) / isinstance chains, and never a raw kwargs/_extra read either (tests/commands/test_no_raw_flag_reads.py). To override a flag before delegating, pass dataclasses.replace(opts, flags=bag) down, never a hand-built CommandOpts. A PATH-typed flag reaches a python command as a PathSpec, not a string — the executor promotes it (workspace/executor/command/flags.py), so read it with fl.as_paths(name); as_str reads it as absent and the operand is silently never used. TypeScript's bag carries the resolved virtual-path string instead, so its twin is fl.asStr(name).
• Generic commands own flag interpretation. Backend wrappers are wiring only (glob resolution, backend I/O injection, pass-through of texts and opts); all flag semantics live in the generic command for that family, mirroring the TS generics. Adding or changing a flag should touch the spec and the generic, not N wrappers. Every literal FlagView query name must be a dest of a spec bound in the same module — tests/commands/test_flag_query_names.py and commands/flag_query_names.test.ts fail on a typo'd spelling without needing the code path to run.
• Generics parse flags once into a frozen struct. Each generic defines a @dataclass(frozen=True, slots=True) flag struct plus a module-level parse_flags(fl, ...) (mirroring the TS parseFlags struct); the function body reads only struct attributes, never string keys. Construct the FlagView with the command's spec (FlagView(flags, spec=SPECS["grep"])) so a typo in a flag name raises KeyError instead of silently reading as False/None.
• Never annotate anything as object — not a parameter, not a return, not a type argument (dict[str, object], Callable[..., object]). object reads as "we did not decide": it accepts bytes where JSON was meant and a PathSpec where a flag value was meant, so every use site pays for it with an isinstance chain back to the set the author had in mind. Name the real type instead:
  • a parsed command-line flag is FlagValue (mirage.commands.spec.types) — **flags: FlagValue, Mapping[str, FlagValue], FlagValue | None for a single raw read. The TypeScript side has always called it FlagValue too (commands/spec/types.ts); keep the two spellings identical.
  • a decoded JSON payload or an API field is JsonValue (mirage.types). It is recursive, so it is spelled as a forward-reference string until the floor is 3.12 — a union with it must be quoted: Awaitable["JsonValue | X"].
  • a path is str | PathSpec, a backend handle is accessor: Accessor (mirage.accessor.base), an index is index: IndexCacheStore | None (mirage.cache.index), a stat function is StatFn (mirage.types). Ignored variadics are still typed (*texts: str).
  • a sentinel is a one-member Enum, never object(); that keeps it distinguishable from the real values sharing the variable. tests/commands/test_no_object_annotations.py enforces this and carries the only exemptions: four Python protocol methods (__setattr__, __contains__, Mapping.pop) whose signatures the language fixes, and one guard whose whole job is to catch a value the annotations already claim cannot arrive. Adding to that allowlist needs the same kind of reason.
• Every session write goes through SessionView.set. A deployment refuses a name (AWS_*, a credential) with a pre_session rule, and a writer that reaches session.env directly makes that rule advisory. export and a plain assignment always cleared the gate; the expansion-time writers did not, so ${X:=d}, $((X=5)), (( )), printf -v and for ((X=0; ...)) each wrote past every policy. The view is threaded into expansion explicitly rather than read from ambient state, and defaults to None (correct outside a workspace, where the env is the only state there is). Reads ($X) stay sync; only the writers await. tests/workspace/expand/test_session_gate.py and integ/session/writers.json hold the line in both languages.
• The record client is a substrate, not a session detail. Sessions, the namespace node table and workspace metadata are three tables that persist the same way, so the keyed-record clients live in workspace/record/ (DiskRecordClient with an O_CREAT|O_EXCL lockfile and rename(2), S3RecordClient with If-Match CAS) and import none of the three. They used to live inside the session package, which made the other two import upward into it; a layering test in each language fails if that returns. The two clients deliberately do not merge: different concurrency contract, different blob layout, different key shape.