gws, notion and github each take a --port and then hardcode 127.0.0.1.
That is right for every direct invocation and wrong inside a container:
the published port accepts the connection and then answers nothing, which
reaches the caller as an empty reply with no error anywhere explaining it.
--host leaves the default alone, so nothing that runs these on a host
changes. github also takes --advertise, because the base URL it writes
into html_url and clone_url is not always where it bound -- a container
binds 0.0.0.0, and 0.0.0.0 in a URL sends the caller nowhere.
The remaining integ servers carry the same hardcoded bind and are left
alone here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Also share the file-open umask rule between the plain and exec redirect
paths, and route the four new builtins through the existing option
scanner, refusal helper and single-quote renderer instead of their own.
integ-runtime sets INTEG_RUNTIME_STRICT=1, which turns an unmet suite
requirement into a failure unless the suite declares itself optional.
Every existing suite is satisfiable on a GitHub runner -- the workflow
even starts the docker container the docker suite needs -- so these two
are the first that genuinely are not: smolvm needs a hypervisor
(/dev/kvm) the hosted runners do not have, and sandlock needs its CLI
plus Landlock ABI v6 (Linux 6.12+), with no apt package to add cheaply.
Each carries an optional_reason naming what a host would need and when
the flag should come off, and the runner still prints the skip, so the
dropped coverage stays visible rather than reading as a pass.
Two codex P2 findings, both verified against GNU before acting.
`humanScaled` computed `n * 10` as a double. That leaves the safe-integer
range a little under a petabyte, and the product rounds down before the
ceiling, landing on the wrong tenth: 1914029841632461 bytes rendered
`1.7P` in TypeScript where GNU (`numfmt --to=iec --round=up`) and the
integer-exact Python say `1.8P`. A real py/ts divergence reachable from
`ls -h`, `du -h` and `df -h` on a large enough tree. The scaling now runs
in BigInt, which is what Python's arbitrary-precision ints were doing all
along, so the two sides compute the same way rather than agreeing by
luck. Both GNU tables gain the row; reverting to the float product fails
it.
The `divergence` exemption tested only for the key's presence, so
`"divergence": null` or `""` bought an asymmetric matrix while explaining
nothing -- the one thing the key exists to supply, and it would read as a
considered decision in review. Both runners now require a non-empty
string, with the six falsy/ill-typed shapes pinned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex P1: a python-only module under runtime/python left
check_layout_parity.py --strict red (297 against a 296 baseline). Making
sandlock a package the way monty already is moves the divergence from a
module to a directory, which the exceptions file excuses as a subtree --
the honest shape here, since there is no TypeScript counterpart to
mirror and growth inside it is expected rather than drift.
Constants and config move out of the runtime module to match monty's and
smolvm's layout: constants.py carries the env var, the CLI hint, and the
system read set; config.py carries SandlockConfig.
Docs: smolvm joins the sandbox provider table and gets its own section on
both language pages, covering the refused-state probe and the fact that
one machine is one guest, so concurrent lines share a filesystem and a
process table. sandlock joins the python runtime table with a section on
what "process" reach does and does not buy, and on the environment it
deliberately does not inherit. Vendor marks added under the existing
docs/images/<name>-logo.svg convention.
The job was copied from the `test` job beside it, which pins 22. It
runs no project code -- `pnpm audit` resolves the lockfile and queries
the advisory API -- so it is free to track the newer line `test_integ`
is already on, and there is no reason for it to trail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the priority items of #608 in both languages.
- let: (( )) as a builtin, ordered writes, last-value status
- umask: session mask applied to touch/mkdir/redirect creates
- shopt: full option table, plus nullglob/failglob/dotglob/globstar;
extglob is refused rather than stored, since the parser has no mode
- alias/unalias: definition, expansion (expand_aliases, same-line rule,
trailing-blank chaining), type/command -v reporting
- mapfile/readarray with -d -n -O -s -t -u -C -c
- read -a -d -n -N -t, plus the non-tty no-ops -p -s -e -i
- declare -g and declare -n: references resolve on every read and write
- disown, and wait -f/-n/-p
- exec: redirect-only form installs on the shell; the command form is
refused, having no OS process to replace
Two sandboxed execution options, both using the host CLI as transport so
neither adds an SDK dependency.
smolvm joins the RemoteSandbox family as a fourth provider alongside
docker/e2b/daytona, mirroring DockerRuntime's shape: `machine status
--json` is the liveness probe and each line is one `smolvm machine exec`
with the merged env, rebased cwd, real stdin, and separated stderr.
`machine exec` would start a stopped machine itself, but this family
never manages sandbox lifecycle, so connect() refuses anything but
"running" and names why — "unreachable" (VMM alive, guest agent dead)
and "frozen" (a fork base, paused by design) get their own recovery
hints rather than a bare not-running.
sandlock is a new python-tier engine: the same real CPython `local`
spawns, wrapped in `sandlock run -r ... -w ...` so Landlock and seccomp
bound where its I/O can land. It sits between the bridged engines and
`local` in the preference order — reach stays "process", because
confinement narrows which host paths the code touches while the
workspace gate still never sees the I/O, but a "process" runtime need no
longer be an unbounded one. Unlike `local` it does not inherit
os.environ, which would hand every live backend credential to confined
code.
Both stay out of DEFAULT_ENTRIES: sandlock is Linux-only, and binding it
ahead of monty would silently reroute python3 by host OS.
A node_modules that predates the lockfile does not announce itself. It
surfaces as a type error inside whichever package the drifted dependency
types, in a file nobody touched, which reads as a source bug.
The worked example: `pnpm -r build` failing at
packages/dsh/src/fs.ts:88 with
TS2345: Argument of type '"FS_TOO_LARGE"' is not assignable to
parameter of type 'FsErrorCode'
on a checkout whose install predated 3bbb79121, the commit that bumped
@deepseek-ai/dsh-fs 0.0.1-rc.1 -> 0.1.0-rc.6 and with it the FsErrorCode
union that 88698fa08 then used. Source and lockfile were both correct;
only the install was behind, and the message pointed at neither.
pnpm can say that itself, so let it. `pnpm install` is the whole fix.
The setting has to live here rather than in .npmrc: inside a workspace
pnpm overwrites .npmrc settings from this file, so a
verify-deps-before-run= line there resolves in `pnpm config get` and is
still never consulted.
No effect on CI, which installs from the lockfile before it builds, and
none on a pull: verifyDepsBeforeRun is absent from pnpm's workspace-state
settings snapshot, so taking this commit does not itself invalidate an
up-to-date node_modules.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`df -H` is powers of 1000, and GNU prints that kilo as `k` -- mirage
printed `K`, which is the 1024 unit. Confirmed against a real `df -H` on
a 100 KiB tmpfs, which GNU renders `103k` while `df -h` on the same
mount renders `100K`; `numfmt --to=si` and `--to=iec` agree. Only kilo
differs in case, M and up are capitals in both tables.
Nothing pinned `df -H` at all, in either language, which is why the
letter could be wrong since the flag was written. Both sides now assert
the quota fixture's three numbers under `-h` and `-H` against GNU, and
reverting the unit table fails the `-H` assertion.
This is the SI half of the same `human_readable` engine the previous
commit fixed for `-h`; the rounding was already shared and already
correct here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the two remaining small items of Block E. Both were largely
recon: much of what they filed had already been fixed, and the part
that had not turned up a real GNU divergence.
Item 31 (T3-7), rule/doc reconciliation
---------------------------------------
The `object`-parameter half is already done: #825's mypy inversion took
the package from the filed 28 down to 7, of which 3 are docstring prose
and the remaining 4 are protocol methods (`__contains__`, `pop`) whose
signatures typeshed dictates, all already listed in the no-object gate.
`utils/errors.py` reads `str | PathSpec` today, and `builders/sed.py` no
longer holds flag values at all -- item 22 moved that to the generic.
That left the nested-function rule, which the plan asked to decide
before gating. Measured: 39 nested defs, and **every one of them closes
over its enclosing scope** -- 38 by free variable, and `sed_helper._repl`
by binding through parameter defaults, the loop-variable idiom, with a
comment saying so. A flat "do not nest" is a rule the architecture
cannot keep (op factories, provision builders, the read-through cache,
every decorator's wrapper), so gating it as written would have meant 39
standing violations.
So the rule now says what it was reaching for: a nested def must capture
the scope around it, or it belongs at module level. That is mechanical
and enforceable -- `tests/test_nested_functions_are_closures.py` reads
free variables from `symtable` and parameter defaults from the AST. It
passes on the tree as-is and fires on a helper that reads only its own
arguments.
Layout: the eight test-only TS directories are flattened
(`ram/{cat,cut,grep,head,ls,tail,wc}/`, `ssh/ls/`), `awk_helper.ts` moves
up beside its nine siblings, and the `provision.ts` / `_provision.ts`
split is settled on Python's convention -- `_provision` for a backend's
own, bare `provision` for the shared ones (cli, generic_bind). That last
one was not cosmetic: Python already had `_provision.py` for github,
gmail, redis and email, so renaming **closed 4 real parity divergences**
(layout baseline 296 -> 292). The `findEval`/`findParse` line was
already done in #827.
Item 32 (T3-8), test/CI tidiness
--------------------------------
The two asymmetric `ls` conformance matrices are raised to symmetry, and
they pass: python `[ram,disk,redis]` vs typescript `[ram]` was stale
caution, not a divergence. Both runners now reject an asymmetric matrix
at load time unless the case carries a `divergence` key explaining why --
a case is a parity claim, and narrowing one side reads as coverage while
the side that still lists the backend goes green. Loading the corpus
under the new assertion proves no other case was asymmetric. The README
documented the override as "not yet needed"; it exists now.
Adds the `ts-audit` job, mirroring `test_python.yml`'s exactly (same
`continue-on-error`, same out-of-gate placement). Nothing ran `pnpm
audit` for the TS tree while `typescript/package.json` carried 27
hand-written CVE overrides that only a person remembering to check kept
current.
Integ facets for `ls`: `-A`, `-d`, `-r`, `-S`, `-h`, all pinned against
GNU coreutils 9.7 in docker. `-S` uses regular files only, because GNU
sorts a directory by its inode size while mirage counts it as 0 -- a
divergence CLAUDE.md documents deliberately.
The `-h` facet found a real bug
-------------------------------
`ls -h` disagreed with GNU three ways, and because the flag had zero
cross-backend coverage nothing caught it. GNU prints a count below one
unit with no suffix (`24`); mirage printed `24B`. GNU rounds *up* to the
precision shown (1025 bytes is `1.1K`); mirage gave `1.0K`. GNU drops
the decimal once the value reaches ten (`10K`); mirage gave `10.0K`.
One shared engine replaces both formatters in both languages -- GNU runs
`-h` and `-H` through one `human_readable`, and so do we now. Rounding
up can carry past the base (1048575 ceils to 1024K, which GNU shows as
`1.0M`), so the unit is re-chosen after rounding rather than once up
front. 21 GNU-read points are pinned as a table in each language.
Blast radius, all re-pinned: five integ cases that had recorded the `B`
suffix (du, discord, langfuse, email, gmail), and the du fan-out tests.
Those last ones needed care rather than a new number: they exist to
prove the total is humanized once instead of twice, and their 1500+1500
stops discriminating under correct rounding, since 3000 bytes and two
`1.5K` readings both render `3.0K`. They now use 1025+1025, where
single-rounding gives `2.1K` and double-rounding would give `2.2K`.
Verified: pre-commit clean, integ 8054/0 on ram+disk+redis in both
languages, conformance green in both, and the layout, spec, barrel,
docs, case-target and PathSpec gates all pass.
Not covered, still open on item 32: integ facets for `ls -t` (mtime
order is not stable across backends without a seeded fixture), mktemp,
gzip, the checksum `--check` companions, `df -H/-k/-a`, and `cmp
-n/-b/-i`; and the T2-7 helper unit-suite mirroring. Item 31 leaves the
provision *presence* sets unreconciled (Python has lancedb/qdrant, TS has
trello) -- that is a question about which backends should provision at
all, not about naming.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Conflicts, both from ratchets/config the two branches moved independently:
- python/pyproject.toml: #825 inverted the mypy config (global strict +
an opt-out list of not-yet-annotated modules) and deleted the old
opt-in allowlist my last commit had edited. Took upstream's file
wholesale -- github.pushdown now gets strict from the global default
and passes it, so the retarget commit is subsumed. Retargeted the one
stale entry the rename left in the NEW list:
generic.rm_command -> generic.rm_cmd (the module still needs its
exemption: one no-any-return at rm_cmd.py:89), keeping its exempt
status exactly as it was under the old name.
- spec/layout_exceptions.json: both sides lowered the baseline from 299
(mine to 293, upstream's to 296). Neither value is right after the
merge; ran the gate, which reports 290 -- the two closure sets are
disjoint (299 - 6 - 3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The core/github/scope.py -> pushdown.py rename left the pyproject
per-module mypy override pointing at the old name, silently dropping
disallow_untyped_defs/warn_return_any coverage for the module.
(codex P2 on #827)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four review findings, three of which the TypeScript side shared, so both
sides are fixed and tested together.
- file_version stamped the bytes handed to `write` while every later
check reads them back. On a mount whose read op renders, the agent's
own next write was refused as somebody else's change. Both sides now
stamp what a later read returns.
- version stamps were keyed by the caller's spelling, but read and write
follow the namespace symlink table, so `/alias` and `/target` got
separate stamps and an edit through the other name skipped the
staleness check. Keyed through `namespace.follow` now.
- the grep tool reported every exit code as success. Exit 1 is "no
match" and stays a success; >1 (an unreadable path) is now an error.
- `Server[object, object]` discarded the SDK's context types. It is
`Server[dict[str, Any], Any]`: the default lifespan yields an empty
dict and nothing here reads a request payload. The no-object gate only
swept function signatures, so it could not see a type argument in a
variable annotation; extended to annotated assignments, which found
one more (`sort_helper.seen`, now `set[DedupKey]`).
Also narrows `Resource.resolve_glob` from `list[str | PathSpec]` to
`list[PathSpec]`. The union is the argv type -- a command line really
does mix text words and paths -- and it had leaked a layer past the
split into a function that only ever takes paths. TypeScript's twin is
already `readonly PathSpec[]`, all 39 implementations take PathSpec, and
every call site passes one. Nothing checked that, because
`make_resolve_glob` returned `Callable[..., Any]`; it now returns a
`ResolveGlobFn` protocol, and `ResolveGlobOp` (the consumer side) drops
the same union. Re-widening any override now fails two ways.
`tests/cli/test_mcp.py` asserted on rendered `--help` text, which is laid
out to the terminal's width; it passed locally and wrapped away on CI.
It reads the declarations instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- coerce -i/-l/-u before the pre_session gate, so a rule judges the value that lands
- ${m[k]:=v} and ${a[3]:=v} land on the named element; [@]/[*] and a bad index refuse
- arithmetic writes land in evaluation order across bare and subscripted targets, and a bare array name reads as element 0
- -i resolves element references (resolver moved from session/elements into session/state, no cycle)
- test -v unquotes an associative subscript
3bbb79121 rewrote agents/langchain/backend.py to follow deepagents' new grep
cap -- GrepResult.truncated and the keyword-only max_count on grep/agrep -- and
bumped python/uv.lock and typescript/packages/agents/package.json to match. It
left python/pyproject.toml at deepagents>=0.6.12.
0.6.12 is the last release before that API. Its GrepResult is a two-field
dataclass, so constructing one with truncated=True raises TypeError, and
BackendProtocol.grep has no max_count for the override to match. Both landed in
0.7.0, not 0.7.6 as that commit's message says -- checked by reading
backends/protocol.py out of every published 0.6.x and 0.7.x wheel.
Nothing in CI resolves the floor, which is why it stayed green: uv.lock pins
0.7.6, and Test (Install) only exercises a base install with no extras. The gap
is reachable from a downstream pip install mirage-ai[deepagents].
Raised to >=0.7.6, matching the lock and what the TypeScript side already got.
The floor is the version we test against, which is 3bbb79121's own rationale.
No behavior change: backend.py is untouched and already conforms to the 0.7.x
contract on every clause it specifies -- a total cap rather than a per-file one,
truncated only when matches were actually dropped, and exactly max_count with
none dropped reported complete.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two cleanup-plan items, both Python-side.
Item 29 -- `mirage mcp`. TypeScript shipped a six-tool stdio MCP server;
Python had none, so a pip-install user could not point Cursor or Claude
Desktop at a workspace and `mirage --help` differed by distribution.
Adding the entry point alone would have duplicated the tools, because
this side kept them private inside the Claude Agent SDK integration, so
the shared layer comes first:
- agents/tool_descriptions.py -- the six strings, one copy.
- agents/tool_operations.py -- MirageToolOperations, lifted out of the
SDK server's private _MirageTools.
- agents/file_version.py -- stale-write protection, which this side
lacked entirely. TS stamps stored bytes; here the stamp covers the
rendered bytes, because this read tool has always rendered and an
edit must search what the agent was actually shown.
- agents/mcp/server.py + cli/mcp.py -- the server and `mirage mcp`.
- server/workspace_config.py -- config discovery (candidates, env
names, walk up from cwd), which Python had nowhere, so every entry
point had to be handed an explicit path.
The server is the low-level MCP Server rather than FastMCP: FastMCP does
not forward a version, and TS advertises one. Handlers are bound methods,
not decorated closures, so nothing nests.
Item 28 -- the mypy allowlist. 54 modules opted *in* to annotation
checking against 1826, so the default was unchecked and every new file
joined the unchecked side. The default is now strict, with a list of
what is not yet annotated that only shrinks. 166 annotations cleared
along the way; the remainder is named module by module.
Two real defects surfaced by the annotations, neither of them typing:
- Workspace._original_open / _original_os were invented by assignment
in lifecycle.patch_process, so unpatch without a patch raised
AttributeError. Declared, and the restore is guarded.
- sed_generic declared a non-optional writer while its own docstring
and its `write_bytes is None` branch said otherwise; the builder
passes None whenever the backend cannot write.
Tests keep the PathSpec rule instead of full strict: measured, full
strict on python/tests is 2374 errors, of which 634 are `str` where a
pydantic field declares SecretStr -- which pydantic coerces at runtime --
and most of the rest is the monkeypatched-fake pattern CLAUDE.md
sanctions. The rule that is violated for real is PathSpec, 19 times, and
scripts/check_test_pathspec.py now holds that line. One of the 19 was a
latent AttributeError: tests/e2e passes a str to s3 write_bytes, which
reads .mount_path, and the test skips without a live versioned bucket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
readonly -f freezes a function: either definition syntax and unset (-f
or auto) refuse in GNU's voice, exit 1, and the old body stays; a
non-function operand is 'not a function'; the bare form lists the
frozen set. The set is a session field beside functions, inherited by
forks and child shells. declare -rf freezes the same way, -F prints
names, and -f prints a name row where GNU pretty-prints the body, which
mirage does not carry.
jobs gains -l -n -p -r -s, a jobspec filter (%N or N), 'no such job'
exit 1, and the usage refusal exit 2, all applied to mirage's own row
shape: -p prints the table id where GNU prints a pid, and -s lists
nothing since mirage jobs never stop.
Pinned against bash 5.2.37; 21 unit cases per language, 3 battery
cases on 22 targets, both hosts byte-identical.
Phase 1 of the core/ restructure plan: introduce mirage/core/render/json.py
and core/render/json.ts (json_text/json_bytes, compact_json_text/bytes,
jsonl_bytes) with a cross-language byte-parity fixture, and migrate every
core/ backend render site onto the kit: langfuse + jaeger (generic render
modules deleted), mem0, discord, slack (history/search/users), gcal, gdocs,
gsheets, gslides, gmail, qdrant, email, and postgres-TS. linear/notion/trello
normalize helpers become thin delegates (their created_at sort stays local;
the ~90 CLI call sites keep their names for a follow-up audit). jq/format
and py-postgres (orjson) deliberately stay local.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
awk splits statements at brace depth zero and outside quotes, and a
`{ ... }` compound contributes its inner statements in place, so
`{{print $1}}` prints the first word the way gawk does instead of being
refused; the chroma golden that had pinned the old empty output is
corrected.
find -printf renders %m/%M from a reported mode (chmod overlay) with the
per-kind constants as the fallback, and %Y on a symlink row reports the
target's kind through the namespace (N when dangling).
TS parseDateExpr truncates fractional seconds instead of rounding into
the next second, matching Date parsing and Python.
The Notion fake validates every child of a block append before shifting
siblings or inserting, so a refused request mutates nothing.
The value-shaping attributes apply in the session door on every write,
which is where bash applies them: -i evaluates the incoming text as
arithmetic (a bad expression refuses in the evaluator's voice, fatal for
a plain assignment and exit 1 for a builtin), -l/-u fold case, and each
applies per element on an array. A declaration pre-marks the name so its
own value coerces too, after the local snapshot so a function scope is
not leaked. -l and -u are exclusive; a cluster naming both sets neither.
n+=3 on an integer name adds rather than concatenates.
+attr turns an attribute off through the gated mark door; +r on a
readonly name and +a/+A on an array refuse as GNU does, and an unknown
letter refuses with the usage line, exit 2. -t and -n are stored and
printed; nameref aliasing is deliberately not wired, since it is a seam
through every expansion site and a partial alias is worse than none.
Pinned against bash 5.2.37; 51 unit cases per language, 7 battery cases
on 22 targets, both hosts byte-identical.
The loader read a script before it stat'd it, and only a real
filesystem answers a directory read with EISDIR: WebDAV serves the
collection's HTML as bytes (nextcloud ran it as a script), ssh's raw
error carries no errno, and a keyed store answers ENOENT. The stat
probe now runs first in both languages. The directory cases give hf a
key to see (hf holds no empty directory), and the multi-line grep -r
cases sort, since box lists folders first.
sed -i writes for every command, not only s and d; multi-file output is
concatenated raw; an escaped delimiter inside an address regex no longer
cuts the address short, and \%re% custom delimiters parse.
awk refuses constructs it cannot run (exit 2, "unsupported construct")
instead of echoing their source; simple assignment and OFS now execute
and unset names render empty.
date -d parses ISO, @epoch and the gnulib relative grammar in both
languages (TS printed NaN with exit 0), invalid input is exit 1 with GNU
wording; TS strftime learns %F %c %C %g %G %V %U %W %h %k %l %n %P %q %r
%R %t %x %X, and Python expands %q ahead of strftime.
find -printf is implemented (%p %P %f %h %d %s %y %Y %m %M %T escapes),
registered on the CommandSpec so the format is not read as a path, and
stats through the dispatcher when no overlay stat is wired.
Fakes: the GitHub server strips the internal files list from list-commits,
accepts refs/heads/x on /contents and /commits, and reports files as
objects on one commit; the GWS Drive server evaluates fullText contains
and answers 400 for an unknown query field; the Notion server honors
after on block append with the live API's response and validation shape.
Integ: new unix cases for each command fix, awk_ofs golden corrected,
and cli-gh/cli-gws/cli-ntn cases that drive every fake fix through the
CLIs; ntn conformance re-pinned against the real binary.
declare -A / local -A / readonly -A build a real map: keyed and pair
literals, element and += writes, unset of one key, declare -p rendering
with GNU's trailing space and key quoting, kind conversion refusals,
[[ -v ]] on elements, and read/printf -v into an element. Arithmetic
gains element lvalues (a[i]++, m[k]+=2) through an ElementOps seam the
session fills, so $((a[i])) and ((m[k]=5)) resolve without the
evaluator learning what a session is. Two pinned divergences: keys walk
in sorted order, and a bare declare -A prints =().
Pinned against bash 5.2.37; 49 unit cases per language, 15 battery
cases on 22 targets, both hosts byte-identical.
grep --include/--exclude resolve by command-line order (last matching
rule decides, no-match default follows the first rule), pinned against
GNU grep 3.11; the bag's insertion order carries it through a new
FlagView.typed_order. is_create_mode stops at --, so a member named
-c after the terminator no longer reads as create mode. env -S and
--split-string are consumed on a shebang line instead of naming the
interpreter. A slash-carrying head word joins the admission context
so a path-pattern guard sees the executed file.
The revert missed integ/, so two CI jobs failed on code the unit suite
never imports: integ/root.py awaited to_workspace_kwargs twice (once
split across lines, which is why the line-based grep missed it), and the
github watch builder and the shared-battery adapter still called
GitHubResource.build.
Swept with an AST walk over every .py in the repo rather than a grep this
time, matching an Await whose call resolves to one of the reverted names;
it now reports none. Verified by rerunning the two failing steps:
integ/root.py exits 0, and the github battery is 23 passed 0 failed
against the fake.
Push mode showed only the hand-written FileEvent path. It now leads
with the three mappers that exist, says why you import one rather than
ask the mount for it, and keeps the hand-written version for a backend
without one.
Slack gets the two traps written down: the day is bucketed in UTC while
the client shows local time, and a thread reply belongs to the parent's
day because chat.jsonl renders conversations.history, which returns
parents only. The matrix drops Slack from planned and gives redis its
own row.
Hosts the endpoint Slack posts to, unwraps the event_callback envelope,
maps it with SlackEventHook and notifies the workspace, with a live
watch stream printing what comes out.
Verified against a real workspace: a message maps to
channels/incident__C0B0DB9K11T/2026-08-13/chat.jsonl, file_shared to
that day's files directory, and the mount serves both.
The type and the method bought one line: reaching a hook from a mount
instead of from an import. Nothing needs that, because a push payload
has no vendor-neutral shape, so the caller always knows the backend it
is decoding for. The integ battery proves it: the event mode has to
build a watchdog payload before it can call anything, so the call site
was disk-specific with or without a generic accessor.
Consumers construct the hook they want:
from mirage.core.slack.watch import SlackEventHook
hook = SlackEventHook(resource.accessor)
build_event_hook goes with them, since the resource method was its only
caller. What stays is to_events, which is the whole feature: the path
arithmetic that has to live beside the backend.
delta_hook stays as it is. pull(root, checkpoint) is the same call for
every backend, so one case set drives ten of them in integ; that is the
generic reach event_hook never had.
Both from review on #820.
ensure_tree decided hydration by whether accessor.tree held anything,
but an empty repository (or one holding only excluded gitlinks) hydrates
to {}. That read as never-hydrated, so every direct-tree command
refetched it, and twice per call once an index was wired: ensure_live_index
seeds an empty root, then the fallback fetch runs because the tree still
looks empty. Measured 2 fetches on the first ensure_tree and 2 more on
each later one; now 1 in every combination of empty/non-empty and
index/no-index. Tracked by accessor.tree_loaded, set by the constructor
when a tree is supplied and by every site that reseats one (refill_index,
the watch delta hook).
is_default_branch compared ref against a default_branch that is None
until something hydrates it, so it answered False for a mount pinned to
the default branch. A bare read hydrates only the tree, so that wrong
answer could last the life of the mount. It now returns None for not
known yet, and callers needing a definite answer await
ensure_default_branch first. TypeScript keeps a plain bool because
construction there fetches the fact; noted in the docstring.
The pull and push batteries both hand the watcher a FileEvent the
harness built, so the mapping half of the feature had no integ coverage
at all: nothing exercised to_events. Event mode reuses the same cases
and the same freshness checks, but triggers by handing the backend the
watchdog notification a real filesystem watcher would emit, so a
delivered change can only have come from the hook.
The per-case modes default was the pair [pull, push], which silently
skipped every case of the new mode and still reported OK. It is now
every mode, since a per-case list is how a case opts out.
chat.jsonl renders conversations.history, which returns parents only,
so a thread reply appears in no day file at all. What changes is the
parent's row, whose reply_count and latest_reply the same listing
carries. Bucketing by the reply's own ts refreshed today's directory
and left the parent's day stale, silently, because notifying a path
that exists but did not change looks like success.
affected_ts now returns the parent's ts for a reply, both for a
broadcast reply (Slack puts those in the channel history too), and the
message's own otherwise. Deletes and edits read thread_ts from the
message they describe, not the envelope.
Also records why file_change and file_deleted stay unmapped: neither
carries a conversation or the share day, and files.info on a deleted
file errors, but the index already stores each blob's Slack id, so a
reverse lookup would name the path once the hook is handed the index.
SupportsEvents had exactly one caller in the repo and it was a test
asserting the protocol itself; SupportsChanges had none at all, not
even a test. TypeScript never had either one and its Resource just
declares deltaHook?() / eventHook?(), which is the same feature working
without them.
BaseResource now answers delta_hook() / event_hook() with None, so a
consumer checks the return value instead of the resource's type. None
is the common case, not an edge: 52 of 55 resources have no event hook
and 26 have no delta hook.
`.some(isMountBlock)` is not a narrowing TypeScript can follow, so the
else branch still types `config.mounts` as holding a `MirageMountBlock`
and the assertion is what carries the fact. It was removed by an
eslint --fix run whose type information was momentarily unresolvable, so
the rule read the assertion as redundant; the build then failed in a
package the change had no business touching.
Our mypy gate reads the source tree, so it proves nothing to anyone who
installs us: without the PEP 561 marker a type checker refuses to read an
installed package's annotations and every mirage import resolves to Any.
Verified against the haystack integration, whose pyproject carries an
ignore_missing_imports override for exactly this reason; with the marker
its mypy run typechecks against the real signatures and passes.
A prefix assignment is a session write like any other and the form
exports the name for the command, but the pre-check only ran the hidden
half and the seeding goes through seed_var, the ungated door. A
deployment refusing SECRET_* still saw `SECRET_K=leak printenv SECRET_K`
print the secret. The loop now asks pre_session with the value.
`declare -x NAME` on a name that already exists writes no value, so the
export mark was the only session write there was and it was stamped
directly, which let an agent export a host-seeded credential the
deployment had refused. It goes through view.mark now.
GNU keeps the valid operands of a declaration and reports the invalid
one, so `declare -x GOOD=1 1BAD=x` exits 1 and still answers
`declare -x GOOD="1"`. The stamp was gated on the aggregate status, so
GOOD came back unexported; it is driven by what actually stored.
Session payloads always carry var_attrs, since its presence is what
tells a session record from a bare process environment.
Adds a meta-test in each language that enumerates every ungated
set_attr/setAttr call site and requires each to name the gated write
that covers it, so the next one fails a test rather than a review.
GoogleConfig demanded client_id + refresh_token and minted its own
tokens, while MsGraphConfig already took access_token as either a
SecretStr or a provider callable. Same OAuth problem, two answers in one
codebase, and the Google half locked out anything that cannot produce a
refresh token: a service account, or a host application that already
owns the OAuth dance. A consumer worked around it by monkeypatching
refresh_access_token, a private function.
GoogleConfig now takes the same access_token field. A supplied token
short-circuits the grant in TokenManager.get_token, and a provider is
read every call rather than cached, since the provider is the caller's
own cache. client_id and refresh_token become optional, with a validator
refusing a config that names no credential at all.
The second half is a bug that predates this and is already reachable on
shipped code: pydantic cannot serialize a function, so redacted_config_dump
raised PydanticSerializationError on any config holding a provider
callable, and took the whole snapshot with it. Verified against a
OneDrive mount, which is exactly the shape the haystack integration
builds. Secret fields are now excluded from the pydantic dump and written
back by the walk, and a callable redacts in both modes rather than being
revealed, which would freeze one expiring token into a snapshot that
outlives it. Redaction is already the contract that makes
requires_resource_override demand a fresh resource at load.
0.0.5 made build_resource a coroutine so GitHubResource could fetch the
repo tree before returning. That fixed a real defect (the two fetches ran
in __init__ over a blocking urlopen and froze the daemon's loop) but paid
for it with the one function every caller who describes a mount as data
comes through: the YAML loader, the daemon's create/load routes, clone,
and every embedder reaching it through mirage.sdk. The haystack
integration, our first outside consumer, broke on it.
Hydrating lazily removes a round trip instead of adding one. Nothing
seeded the index at build time, so the first readdir ran
ensure_live_index and refetched the whole tree, discarding the one the
constructor had just paid for; measured two git/trees calls where one
does. GitHubResource now names the repository and contacts nothing, and
the tree and default branch arrive through ensure_tree and
ensure_default_branch on first use, each behind a lock so concurrent
first reads cost one request. readdir and read already hydrated through
ensure_live_index; only find, du and grep's narrow read accessor.tree
directly and needed wiring.
normalize_resources now refuses a non-resource and names the mount. The
old failure was 'coroutine' object has no attribute 'set_index', raised
two frames away in install_mounts, naming a method the caller never
called and no mount.
TypeScript stays async: its factory type is uniformly
(config) => Promise<Resource> and two of its backends need it. Recorded
as a deliberate divergence rather than mirrored.
A message, reaction, pin or shared file names a channel by id and a
message by ts; the mount names a directory <name>__<C-id> and buckets
the day in UTC. That arithmetic is the backend's, so the hook lives
beside it rather than in the consumer, which would otherwise reimplement
it and get the UTC rule wrong for a fifth of every day with no error to
notice.
core/slack/watch/ constants types payload hook
Resolution is memoized: an event carries the id, the directory carries
the name, so conversations.info (plus users.info for a DM) runs once per
conversation instead of once per message against a ~50/minute tier. A
rename drops the entry, since the subtree moved.
Coarser answers where the notification cannot say more: a listing change
is UNKNOWN on channels/dms/users, and a shared file is UNKNOWN on that
day's files directory, because the rendered name needs metadata the
event does not carry.
Each was one file holding constants, helpers and a class, and disk also
held both halves of the feature: the delta-hook walk and the event hook
shared a file with nothing in common but the word watch.
core/disk/watch/ constants walk hook
core/redis/watch/ constants hook
Same module names in both languages. The node barrels export only the
build hooks the resources import; tests reach the classes directly.
_virtual returned the mount-absolute key both caches are keyed by, but
read like it returned PathSpec.virtual verbatim, so it hid that its job
is re-deriving that key against the manager's own prefix. Renamed to
_cache_key / cacheKey, with the locals following.
Four Args docstrings said the parameter was a resource-relative path.
That was already stale and became wrong when the method started reading
only virtual.
Each open is visible to the next, so `set -C; echo x > a > a` creates
`a` on the first redirect and refuses the second. Probing every target
against one pre-command snapshot passed both and wrote the output. The
opens a refused statement already performed are applied, since bash
leaves the file existing and empty.
`declare -rx X=1` carries both attributes. Readonly answers first, so
the export stamp moved into a helper both branches call.
`export -n ARR=(b)` takes the attribute off an array. The store keeps
whatever the name carried, so an unapplied mark left it exported.
`set -v` echoes comments and blank lines, which the reader consumes
like any other line.
- disk: read watchdog's `src_path`, not `path`. The docstring already
claimed watchdog's field names and `dest_path` already matched, so
every forwarded watchdog event mapped to nothing.
- disk: a move whose source is outside the mount is a CREATE at the
destination. Discarding the event left a file in the mount that no
listing knew about.
- disk: the mount root itself normalizes to `/`, not `/.`, matching
what the TypeScript side already did.
- redis: an external mkdir/rmdir touches the single `<prefix>dir` set,
and the message names the key, never the member. An empty directory
has no `file:` key at all, so those changes were invisible. They now
map to UNKNOWN at the mount root.
Adds the push half of change detection, alongside the existing delta_hook
pull half.
- EventHook / SupportsEvents: a resource can map one service notification
to FileEvents. Mirage owns no transport; the consumer runs the socket or
webhook receiver and passes what arrived, so only the path arithmetic
lives beside the backend.
- invalidate_subtree on CacheManager plus invalidate_prefix on every index
store, wired to FileChangeKind.UNKNOWN in the watcher. A notification
that names only a scope now drops everything below it, which is what
UNKNOWN already promised in its docstring.
- disk and redis event hooks, both languages, with workspace-level tests.
Also fixes a pre-existing bug found on the way. CacheManager._virtual
inferred whether a caller's path was already mount-absolute by comparing
strings, so a mount-relative /day under a /d mount was read as absolute
and every eviction targeted the wrong key. An eviction that hits no key is
silent, which is why it survived.
The fix is to stop inferring: read only path.virtual and rebuild the key
against the manager's own prefix, exactly as Mount.execute_op already
does. That also explains why the roughly fifty from_str_path sites that
fabricate a resource_path are harmless: the dispatcher passes
path.virtual and re-derives, so a fabricated key only matters to a
consumer that reads the pair without going through a mount.
New adversarial suites in both languages cover mount/child pairs whose
names collide by prefix (/d holding day, /d holding d, /data holding
database), which is how the second case was found.
The refused-open case ends its line with ls on a file that correctly does
not exist, and GNU ls exits 2 for a missing operand, so the case expected
0 where both hosts rightly answer 2. Pinned against bash 5.2.37.
Excuse the two cases that cannot run on hf, which keys no directory
marker, so a case that makes a directory and then looks it up finds
nothing there. That puts the coverage gate back on its baseline.
Persist variable attributes instead of promoting every scalar to
exported on reload, and keep an attribute-free payload readable as a
process environment.
Route `readonly NAME` and `local`/`declare` through the gated mark door,
so a pre_session rule can refuse the name and a bare declaration no
longer invents an empty value.
Refuse a noclobber redirect before the command runs, not after, and stop
noexec at execute_node plus each loop driver so it holds one construct
deep without spinning.
Restore the process view for prefix assignments and SessionProfile env,
which narrowing env_snapshot to the exported set had broken.
43 GNU-pinned cases: `assign/export_attr.json`, plus noclobber, noexec,
verbose and braceexpand under `setopt/`, which had errexit, noglob,
nounset, pipefail and xtrace but none of stage 1. The noexec cases are
what caught the subshell bug.
Both runners and `state_store` wrote into `ws.env`, which the read-only
projection refuses, so the battery could not start on either host. A
meta-test now scans for that write shape across both trees: mypy runs on
`mirage/` alone, so `integ/` and `tests/` are exactly where it recurs
unseen.
`export`, `export -n`, `declare -x`, `declare -p` and `set -a` now agree
on one attribute stored on the variable record.
`env` and `export -p` carry exported names only, which is what GNU does.
`$X`, arithmetic and `[[ ]]` still resolve against every variable, so the
process view and the shell view are separate functions in both languages;
TypeScript had collapsed them into one. Adds bash's third variable state,
declared and exported but unset, which `declare -x Z` reports and `env`
omits. Attribute writes with no value go through a gated door so the
policy layer still sees them.
`export`, `readonly`, `local` and `declare` refuse a word that is not a
valid identifier, each in its own voice, instead of exiting 0 and storing
a name that `$1BAD` can never spell back. Quoting decides whether an
empty operand survives to be refused.
Aligns TypeScript to Python where Python was right, including `env -i
FOO=bar`, which built the child environment without the attribute. Python
was wrong about PWD across `fork`, and both were wrong about OLDPWD.
`session.env` is a read-only projection of the variable records, so
`Object.assign(ws.env, ...)` throws instead of landing. dsh and the two
version tests still wrote through it.
Mirrors the Python side of tier 2 stage 1: noclobber with `>|` as the
override, noexec, verbose, the `set -o`/`+o` listing and the remaining
option letters.
Also fixes `set -n` inside a subshell in both languages. `handle_subshell`
runs a second statement loop beside `execute_program`'s, so the check had
to be stated there too. Without it `set -n` worked at top level and did
nothing one paren deep.
Same design, same names, same semantics as the Python commit before it:
Session.vars holds one readonly ShellVar per name, env/arrays/readonlyVars
become projections, and the two local save/restore stacks collapse into
one that shadows the whole record.
The projections are frozen for the same reason Python's are mappingproxy:
an assignment into a plain object lands in a throwaway and vanishes, which
is the failure the store exists to remove. TypeScript catches those at
compile time rather than at runtime, which is how the seedVar(ws, ...)
mis-targeting that Python only surfaced under test was found here before
running anything.
attrLetters is pinned against the same bash 5.2.37 clusters in
variable.test.ts, so the two languages cannot drift on print order.
Full core suite 8266 passed, tsc/eslint/knip/prettier clean, layout parity
at baseline.
A variable's facts were spread across session.env, session.arrays and
session.readonly_vars, with "scalar or array" encoded as which dict the
name lived in and mutual exclusion maintained by hand at every write.
The remaining #608 attributes would each have added another container
(-x -i -l -u -n -t, plus associative values and a third local stack):
thirteen name-keyed containers, any two of which could disagree about
one name. That has already cost bugs here twice -- printf -v wiping an
array, readonly_vars sitting permanently empty.
Session.vars now holds one frozen ShellVar per name (value plus
attributes) and env/arrays/readonly_vars are read-only projections of
it. Frozen because every writer already computed on a copy and handed
the result to the session door; the type now enforces what was
convention, so the door is the only way to change a variable and an
attribute write can finally be gated the way a value write is.
Two things the migration surfaced:
- The projections return mappingproxy, not a plain dict. A dict made
session.env[k] = v write into a throwaway and vanish silently, and
the suite went green on it -- the exact failure the record exists to
remove. Failing loud is what located all 125 write sites.
- bash has a third state, declared-but-unset: readonly ONLY prints
declare -r ONLY and ${ONLY-d} still expands to d, while readonly
EMPTY= prints declare -r EMPTY="" and does not. ShellVar.value is
therefore optional; None is that state, and export FOO and declare -i
n need it too.
Attribute print order is pinned exhaustively against bash 5.2.37 over
all 72 ordered pairs: a/A, then i n r t x, then l/u. -a and -A are
derived from the value's type rather than stored, so they cannot
contradict the value they describe.
Also adds the Stage 1 tests that were missing: a leftover reference
broke set -C outright and the whole suite stayed green.
set -C was unreachable by its own letter: only five letters were mapped to
option names, so -C fell into the ignore-unknown-letter branch while
set -o noclobber worked. Maps all nineteen bash letters, then implements
the three options that silently did nothing:
- -C refuses a truncating open onto an existing target (empty counts),
leaves the file intact, exit 1. >| is a new operator that overrides it
for one redirect; >> is unaffected. Existence is asked of the op
dispatcher, and only when the option is on, so the ordinary redirect
path costs no extra round trip.
- -n reads without executing, checked in the top-level statement loop,
which gives bash's one-way trip for free: a later set +n never runs.
- -v echoes input as the reader consumes it. The unit is a line, not a
statement, so set -v; echo a echoes nothing while set -v\necho a does.
set -o and set +o with no name print the option table in both spellings,
and +B is honoured rather than merely stored so the listing does not
report a state nothing acts on.
All pinned against GNU bash 5.2.37 on debian:stable-slim.
With the barrel down to its public surface, knip can finally see core's
unused exports. 233 findings: 74 are the surviving index.ts lines, whose
consumers live in examples/, docs/ and integ/, outside knip's project
root, and are gated by scripts/check_barrel_surface.py instead.
Of the remaining 159, six turned out to be false as well. knip strips a
barrel line and then cascades into whatever that line alone kept alive,
so CommandContext, ExecuteResultContext, RAMWatchQueue and Watcher are
still reachable through index.ts, and S3BrowserOperation is re-exported
by packages/browser.
The other 153 are gone: 61 dead re-export lines in the sub-barrels, 52
symbols that lost the export keyword but are still used in their own
file, and 40 deleted outright.
knip.json is unchanged. `pnpm exec knip` is a pre-commit gate, and
includeEntryExports would fail it on the 74 barrel lines.
The Files API answers a honored Range with no Content-Range and the SDK
surfaces no status, so taking that header as the proof sliced an already
windowed body a second time. A body longer than the window is the only
proof available and the only case that matters.
Past the end huggingface echoes `bytes 99-2/3` and OpenDAL refuses to
parse it, which reached the caller as a crash rather than an empty read.
#805 shrank core's barrel to its public surface and moved every in-repo
import onto a module path, which collided with this branch in five files.
The barrel line this branch added for `listingError` is dropped: the new
gate refuses a name that only package source imports, and the four node
readdirs that wanted it now name `@struktoai/mirage-core/utils/errors`
like their siblings. `enoent` goes with it in hf and ssh, and
`enotdir`/`readdirError` in gridfs -- each was left over from the private
`listingError` wrapper this branch deleted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hf was left out of #809, which fixed this for s3, gridfs and nextcloud.
readdir answered a missing path with an empty listing, so ls exited 0
where stat and every other backend report ENOENT.
What proves existence differs per store and was probed rather than
assumed. hf is a third rule: the tree API lists children only, with no
self-entry, and the service refuses a directory marker client-side
(create_dir=false), so a bucket directory exists exactly while it holds
a key and any entry at all proves it. The Hub answers a missing subpath
with 200 and [], which the lister reports as an empty result rather than
raising, so the existing NotFound arm never fired.
readdir now tracks whether the listing saw anything and refuses when it
saw nothing, with the mount root exempt and the errno coming from the
shared readdir_error / readdirError helper.
The TypeScript fake rejected with NotFound where the service answers [],
which would have made the new tests vacuous; it now matches. That fake
was also why the existing "raises ENOENT for a missing directory" test
passed without reaching the code under it.
integ: hf and hf-prefix join the seven ls/missing cases and their
exceptions are deleted. The baseline stays at 2297 because it counts
unexcused gaps only, and those pairs were excused.
Three cases only passed before because this bug masked them, all from hf
having no empty directory: rm_interactive_noops and rm_root_and_fs_flags
end with the directory emptied, and xm_mv_no_copy_source_kept lists a
directory that mkdir -p never keyed. hf is dropped from those three, with
a reason recorded for the one whose directory reports it as a gap.
Sending a Range is a request, not an instruction: a server may answer
200 with the whole representation, and box, dropbox, gdrive, onedrive,
sharepoint and databricks all handed that back whole for what the
caller asked to be a window. They now carry the window as one value
and check the answer, the way slack and discord already did.
The node mongo driver validates a window against the file and refuses
every POSIX-shaped answer: a start past the end, an end past the end,
and an omitted end, which it defaults to 0 and then reads as ending
before the start. That last one broke every read-to-EOF on gridfs.
Python is unaffected because its driver seeks through a file object.
Three P1s, each of which was present in both languages while the review
saw only one half.
github: the walk fetched a complete tree, diffed it and threw it away.
find, du and grep's scope counter read accessor.tree directly since the
re-key, so they answered from the tree the mount was built with until an
unrelated read refilled the index, and a pull that reported a CREATE was
followed by a find that could not see the file. A truncated tree is
still refused rather than adopted.
dropbox: path_display carries the server's casing and root_path the
user's, and dropbox paths are case-insensitive, so a configured /team
whose displayed path is /Team matched nothing, kept the root on the
front of every virtual path, and put every event outside the watch
scope. The comparison now folds case; the slice keeps the server's
casing below the root.
disk: an unreadable directory is not an empty one. The typescript walk
caught every readdir error and python's os.walk swallowed them by
default, so EACCES or EIO diffed into a DELETE for every child and a
CREATE for each once access returned. Only ENOENT is swallowed now.
Tests pin all three in both languages.
Rebasing onto #812 broke the build, and the break was one my own gate
could not see. #812 rewrote the redis example to take (accessor, paths,
texts, opts) and added a typecheck script for examples/typescript, so
tsc now says:
redis/redis.ts(18,42): error TS2305: Module
'"@struktoai/mirage-core"' has no exported member 'CommandOpts'.
check_barrel_surface.py only ever asked whether every line in index.ts
has a consumer. The other direction -- a consumer importing a name no
line carries -- it never asked, so the shrunken barrel could go stale
under any consumer change and only tsc would notice. tsc is not enough
on its own either: it reads examples and integ, but a name in a docs
page is prose to it. The gate now owns both halves.
Adding the second rule needed the first one's bookkeeping fixed.
own_exports() counted a runtime package's own names only when they came
from a relative module, so a name the index re-exports from a core
subpath was credited to core's barrel instead. That mattered because
node and browser both name DiscordConfig, LinearConfig, NotionConfig and
SlackConfig from their core subpaths -- every TypeScript consumer gets
them from the runtime package, and the four lines in index.ts were dead.
The rule is now "any module but the bare barrel supplies the name
itself", which drops those four.
Net: 80 names -> 77. Verified the new rule fires rather than trusting a
green run: deleting the CommandOpts line reproduces the failure with the
name reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shortcut folded into readdir_error assumed a path that exists proves
its ancestors are directories. A flat store breaks that: ram and redis
rename without creating the destination's ancestors, so they can hold
/missing/a.txt with /missing absent, and readdir of the orphan itself
came back ENOTDIR where resolution stops at the gap and the answer is
ENOENT. One level below (/missing/a.txt/x) was already right, which is
why the existing orphan test did not catch it.
readdir_error goes back to walking every component, the listed path
included. The shortcut moves to `listing_error` / `listingError` beside
it, whose docstring states the premise it needs -- a store that cannot
hold an orphan -- and which delegates to the walk for everything it does
not settle. The eight backends that qualify (s3, gridfs, nextcloud,
dropbox, databricks, msgraph, ssh, hf) call it, so the fast path is
still one shared implementation rather than one copy each; ram and redis
call readdir_error directly.
Both new behaviors are pinned in both languages: the orphan case against
readdir_error, and the one-probe count against listing_error.
Two OpenDAL behaviours the python binding never exposes, both fatal to the
typescript integ run and neither visible from a python-host run.
Its node binding takes an exact byte range and refuses to return fewer
bytes than asked for ("reader got too little data", expect 100 / actual 3),
where a POSIX read comes back short. Python reads through a file object, so
f.read(n) stops at EOF like any file. Both node backends now retry the read
unbounded and take the window themselves.
Past that, an offset beyond EOF answers 416, and OpenDAL reports the
store's whole response as message text rather than fields: the status is
only readable inside the string, the code is OpenDAL's own, and SabreDAV's
RequestedRangeNotSatisfiable has no spaces for the spaced spelling to
match. Widened isUnsatisfiableRange / is_unsatisfiable_range to that shape
in both languages, so the read comes back empty as POSIX expects.
Two codex findings on #805.
P2: node and browser lost Mem0Config, OneDriveConfig and SharePointConfig
from their public API. Stage 1 deleted that block from both barrels as a
duplicate of `export * from '@struktoai/mirage-core'`, which was true when
core exported 1457 names; stage 4 shrank core to 80 and left the premise
false. The three resources still arrive through the star -- they are among
the 80 -- so only the config types went missing. Both barrels name them
module-by-module now. Measured: node and browser each shed ~1356 further
names through that star, which is the intended core shrink, and every one
is still reachable by module path; these three were the only names their
own barrels had ever listed.
P1: `with { type: 'json' }` in version.ts parses from Node 20.10, and the
tsup build used to inline package.json so any Node could load it. Nothing
declared a floor -- no package had an `engines` field at all -- so this
would have surfaced as a SyntaxError at import instead of an install
warning. All eight published packages now declare `node: ">=20.10.0"` and
the three docs lines saying "Node.js 20" say 20.10. Hard-coding the
version instead would break a deliberate parity: mirage/version.py reads
importlib.metadata for the same reason this reads the shipped
package.json.
index.ts goes from 1589 lines / 1457 names to 102 / 80. What survives is
what the repo's own consumers -- examples, docs, integ, the spec
generator -- actually import; everything else is reached by module path,
as `mirage.resource.s3` is in Python.
gen-specs.ts stops enumerating `Object.keys(Core)` for command groups and
reads them from the modules that declare them, so core no longer needs
the reachability assertion: a directory scan cannot miss a group. node
and browser keep it, being bundled behind one entry. Spec output is
byte-identical.
The gate is scripts/check_barrel_surface.py, not knip's
includeEntryExports. knip's project root is typescript/, so the barrel's
only remaining consumers sit outside its graph -- it calls 74 of the 80
survivors unused -- and core's `./*` exports map makes every module an
entry, turning the flag into a repo-wide dead-export check (233 findings,
159 of them real and left for their own sweep). The script enforces the
two rules that matter instead: no package source imports the barrel, and
every name in it has a consumer. Both were checked to fail when violated.
Also fixes two examples that imported a type that does not exist
(HfDatasetsConfig, HfSpacesConfig; the real name is HfRepoConfig).
Nothing caught it because examples/typescript is never type-checked.
Every cross-package import of `@struktoai/mirage-core` now names the
module it wants, the way `mirage.resource.s3` does in Python. The barrel
only existed because the package had one door; stage 2 gave it a subpath
map, so the door count no longer forces the name list.
Two things the barrel was hiding come out with it:
- The seven bundled CLI specs registered themselves at import time, so
which CLIs a caller could resolve depended on what it happened to
import. The registry now seeds them, and specs.test.ts pins that they
resolve from the registry module alone.
- browser and node reached zod through core's re-export. They keep doing
so, now through `resource/secrets` -- the module whose helpers are
typed against that instance. Declaring their own zod resolved a
second copy (4.4.3 next to core's 4.3.6), which type-checks but takes
134s on one config file.
`core/src/index.ts` is a 1589-line hand-written list because the package
had exactly one door: tsup bundled `src/index.ts` and nothing else was
reachable. This gives it the door Python already has — `./*` maps onto the
built tree, so a consumer can import one module the way Python does
(`from mirage.resource.s3 import S3Resource`) instead of going through a
barrel that has to name it first.
The map is a single wildcard, so a new module needs no build config and no
barrel edit. The `.` entry is unchanged, so nothing has to migrate yet.
tsup cannot do this: `bundle: false` leaves `from "../x.ts"` in the output
rather than rewriting to `.js`, and its dts step fails outright. `tsc`
already rewrites correctly (`rewriteRelativeImportExtensions` is on in
tsconfig.base), so core builds with `tsc -p tsconfig.build.json` plus a
clean step, since tsc overwrites but never deletes. Tests and their
helpers are excluded from the build but still checked by `tsc --noEmit`.
Unbundling surfaced three defects the bundler was hiding, each invisible
while esbuild stood between the source and the runtime:
- Core imports `node:module`, `node:path` and `node:worker_threads` but
never declared `@types/node`; it typechecked only because core's own
test files dragged the types in, so CLAUDE.md's rule that core work in
both runtimes was unenforced. Declaring those three modules locally with
only the members used makes the boundary compiler-enforced — adding
`@types/node` would have let any module here reach for `node:fs` and
still pass.
- `version.ts` imported `../package.json` with no `with { type: 'json' }`.
tsup inlined the JSON; Node refuses it. Every CLI end-to-end test failed
on this.
- One extensionless relative import (`./constants`), legal under
`moduleResolution: Bundler` and unresolvable by Node.
tsup is no longer a core devDependency, which knip confirms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for giving core a subpath `exports` map, but each of these is
a defect on its own: a module exported from two places in one barrel is
two lines to keep in sync, and the second is easy to miss.
Six modules in `core/src/index.ts` were exported from two blocks each
(io/types, generic/cat, generic/head, grep_helper, runtime/table,
cache/index/config) — merged into one block apiece, no name added or
removed. `sed_helper` exported three symbols under both their own names
and a `sed`-prefixed alias; the aliases have zero consumers anywhere in
the monorepo, so they are gone.
`utils/key_prefix.ts` looked like a seventh, but its flat block and
`export * as keyPrefix` are both live — `node/src/core/gridfs/_client.ts`
imports the namespace. Left alone.
The runtime barrels each re-exported names their own
`export * from '@struktoai/mirage-core'` already covers: `MountMode` one
line below the star in browser, and the Mem0/OneDrive/SharePoint block in
both. Nothing shadows those names locally, so the star is the only
source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The spec is regenerated by a CI step of its own, not by a pre-commit hook,
so adding the slot to those two command_io tables left the committed
inventory a commit behind with nothing local to say so.
Two ways a windowed read was quietly answered with the whole file.
A backend that registers its own read op does not go through
makeGenericOps, so neither the native-range dispatch nor the read-and-slice
fallback reaches it. opfs declared readRange on its CommandIO and still
returned every byte through the op path, which is what the integ battery
caught; gcal, gdocs, gsheets and gslides had the same hole in both
languages and no coverage pointing at it. Each now reads the window off
kwargs: opfs forwards it (File is a Blob, so it slices natively), the four
rendered ones slice after building their bytes.
Sending a Range is a request, not an instruction. RFC 9110 lets a server
answer 200 with the whole representation, and slack and discord fetch
attachments from a CDN, so the header alone was never proof. downloadFile
now takes the window rather than a prepared header and trims anything that
did not come back 206 (window_if_unranged / windowIfUnranged). Reported by
codex.
The slack and discord fakes returned full bodies whatever the request, so
they could not have caught either half. Both now serve 206 with
Content-Range, and 416 past the end.
Every agent SDK was resolving to the floor of its own range, so the adapters
were being tested against builds a month older than the pins implied. Two were
outside their caret entirely (@openai/agents 0.13.5 with 0.16.0 published,
pi-coding-agent 0.80.10 with 0.84.2). The examples workspace carries the same
pin set and was bumped with it.
deepagents 0.7.6 added max_count to BackendProtocol.grep/agrep. mypy caught the
incompatible override; TypeScript accepted the shorter signature structurally
and typechecked clean, so both sides are fixed. max_count is a total cap across
files, which is not what grep -m means, so it is applied to the collected
matches rather than pushed onto the command line, and truncated is only set
when matches were actually dropped.
dsh-shell moves to the 0.1.0 line to match dsh-fs. DeepSeek relicensed between
the two (BSD-3-Clause and restricted, to MIT and public), and the published
types are identical. That family tags its 0.1.0 line as next while latest still
points at 0.0.x, so the siblings are pinned in pnpm.overrides.
Clears a low advisory on @ai-sdk/provider-utils that reached examples through
mastra 1.51's @ai-sdk/ui-utils, which 1.59 drops. Both audits are clean.
Note: @opencode-ai/plugin 1.18.18 pulls ini@7, which needs node >= 24.15.0.
CI already resolves above that floor.
Adds an optional read_range slot to the op table. Backends that can fetch a
window do; the generic read op falls back to read-and-slice for the rest, so
no backend has to implement one. Wired in both languages for box, databricks,
dify, discord, disk, dropbox, gdrive, gridfs, hf, nextcloud, onedrive, opfs,
ram, redis, s3, sharepoint, slack and ssh.
Shared helpers live in utils/ranges (range_header for a raw push-down,
slice_window for rendered content, is_unsatisfiable_range to normalise a 416).
Backends that render their bytes take the window right after building them,
which is why dify and ram are on the native list too: a windowed read is
answered the same way everywhere, whatever is behind the mount.
Fixes a crash on hf and nextcloud: OpenDAL's reader seeks rather than sending
a header, so a window past EOF raised from the seek instead of surfacing a
416. Both now read as empty. Caught by the integ battery, not by unit tests.
Removes the github_ci resource, its commands, ops, docs and examples.
Integ: a shared ranges/read.json across 21 targets, plus per-backend cases for
slack, discord and dify, reached through ws.dispatch since no shell command
asks for a window.
The re-key made the index key mount-absolute, but TypeScript's stat kept
prepending the mount prefix when it derived the parent to list, so a
lookup that missed asked for /repo/repo and never populated the entry.
stat then reported ENOENT for a file that exists, and the read family's
implicit-directory probe found the name in the parent listing and
answered EISDIR, so cat on a freshly written file said 'Is a directory'.
Python computed the parent from the key directly and was already right.
`ls <file>/x` reported "No such file or directory" on every API-backed
backend, where GNU (and mirage's own ram/disk/redis/opfs/s3/gridfs)
report "Not a directory". Each of those readdirs raised a flat
`enoent(path)` when its API 404'd, instead of deciding the errno the way
the store-backed backends do.
The API cannot make that call for them: Dropbox answers `path/not_found`
for a path under a file exactly as it does for a name that is simply
absent, Graph 404s both children listings alike, SFTP 3 has one code for
every unresolvable name, and the Databricks Files API answers 404 to
both. So the errno has to come from walking the ancestors, which is what
`readdir_error` / `readdirError` already does for ram, redis, disk, s3,
gridfs and nextcloud. Wired onto that helper: dropbox, databricks,
onedrive, sharepoint (through the shared msgraph `readdir_items`), ssh
and hf. The probes stay on the failure path, at one request per
component.
gdrive and box do not use the helper, and should not: both resolve a
path by walking it through the index cache, which IS the ancestor walk,
already done and already paid for. box already refused a non-folder
entry there; gdrive did not, so listing a file's id returned an empty
child set and the recursion above reported ENOENT. It now makes the same
check box makes, at no extra request.
Two side effects worth naming:
- The one-probe shortcut the three keyed stores had each copied into
their own `_listing_error` moves into `readdir_error` itself, so all
nine backends get it from one place and the three copies are deleted.
It is behavior-preserving (a path that exists implies every ancestor
of it is a directory, so the walk cannot reach any answer but ENOTDIR)
and strictly cheaper: a `readdir` on a plain file is now one probe
rather than one per component, and a missing path is one fewer.
- hf gets the ENOTDIR half only. Unlike s3 it can store no directory
marker at all (see core/hf_buckets/mkdir), so "no keys under this
prefix" covers both a path the repo does not have and a directory
whose last file was removed. A stored key above the operand proves
ENOTDIR outright; ENOENT would be a guess, and guessing it would turn
`mkdir d; touch d/a; rm d/a; ls d` into an error. Dropped rather than
guessed, so an emptied directory still lists as empty.
`integ/unix/ls/missing.json::ls_operand_under_a_file` regains the 13
targets it was missing. Verified on both hosts: every target runs its
whole battery with 0 failures. Adding those targets made the case-target
gate surface the siblings' gaps; box turned out to answer all six
sibling cases and both `operands.json` cases exactly as GNU does and was
simply absent from them, so it joins those too (baseline 2297 -> 2295).
hf's six sibling gaps are recorded in target_exceptions.json with the
reason above rather than closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The case file #809 added drops 31 (case, target) pairs its ls siblings
still exercise, which fails check_case_targets.py --strict on main.
- box passes all 7 cases on both hosts, so it was an omission; add it
- hf/hf-prefix report a missing path as an empty listing, so ls exits 0
instead of ENOENT: the defect #809 fixed for s3/gridfs/nextcloud,
recorded as an exception naming the bug
- ls_operand_under_a_file wants ENOTDIR, which the path-addressed
backends cannot distinguish from ENOENT; excused with a reason
Back to the 2297 baseline with no baseline change.
Add a typecheck script so `pnpm -r typecheck` covers the examples
workspace, and fix the 32 errors it reports.
- 21 FUSE examples used MountBackend without importing it, so they
failed at runtime with ReferenceError
- hf_datasets/hf_spaces imported config types that do not exist; both
resources take HfRepoConfig
- redis provision helpers take (accessor, paths, texts, opts), matching
the Python example
- postgres_fuse/postgres_vfs read a module-level env const inside main,
where the undefined guard does not narrow
- s3_browser/presigner passed Record<string, unknown> as ListObjectsV2 input
- lancedb: registry.get may return undefined, and TextEmbeddingFunction
.sourceField takes no argument
Adds pull change detection to s3, github, dropbox, disk, gridfs, the hf
family, ssh, gdrive, graph and box, so eleven resource families now
answer "what changed under this root since my checkpoint".
Two shared helpers keep a new backend from writing its own loop:
synth_dirs builds the directory rows a prefix store implies, and
ReaddirWalk descends through a backend's own readdir with a fresh
private index per pull.
Adding github to the integ battery surfaced two real bugs. Its index was
keyed repo-relative while the other eleven backends and CacheManager key
mount-absolute, so cache invalidation never reached a github mount at
all, silently, since evicting an absent key succeeds. And invalidate_dir
drops a row rather than expiring it, so github's readers took a dropped
index for real absence. Github now keys mount-absolute like everyone
else, its repo-relative path logic reads the git tree on the accessor
the way TypeScript's always has, and ensure_live_index refetches when
the index holds no listing.
Box fingerprinted on modified_at while the same listing already carried
sha1, so two writes in one second were indistinguishable. It now prefers
sha1.
The integ watch battery covers eleven targets at 238 cases, and the disk
example is gated in both languages against one shared truth file.
* fix(s3,gridfs,nextcloud): a prefix with no keys is ENOENT, not an empty directory
`ls` on a path that does not exist printed nothing and exited 0 on the
keyed stores and on nextcloud, where every other backend exits 2 with
GNU's diagnostic. `stat` has always been right here; `readdir` was not.
It lists what is under a prefix and returns whatever it finds, and for
a prefix holding nothing that is an empty list rather than an error,
which `ls`'s probe_operand reads as an empty directory. So `ls` and
`stat` disagreed about whether the same path existed, and `ls` is the
one command in the family that reads the listing directly.
Seven lines diverged from ram/disk/GNU, all from that one hole: a
missing operand, a missing nested operand, a path below a file, the
same under -l, a second operand beside a good one (which also printed
a phantom header for the missing path), an unmatched glob, and a
directory listed after rm -r.
Each readdir now tracks whether the listing saw anything at all and
refuses when it saw nothing. What proves existence differs by store
and both were probed against a live server rather than assumed:
s3, gridfs an empty directory is a zero-byte marker keyed at the
prefix itself, which the entry loop skips as a
non-child, so the flag is set from the raw page before
that skip
nextcloud PROPFIND lists the collection itself, so an empty
directory yields one entry and only a missing path
yields none. OpenDAL's WebDAV lister reports that as an
empty result rather than raising, which is why the
existing `except NotFound` never fired
The mount root is exempt everywhere: it exists because it is mounted.
The errno comes from the shared readdir_error / readdirError helper
that ram, redis and disk already use, so ENOTDIR-vs-ENOENT stays
decided in one place. One probe runs ahead of the walk: a path that is
itself a file is ENOTDIR whatever the ancestors say, because every
ancestor of a stored key is a prefix by construction. No probe runs on
a successful listing.
is_not_found moves from a private copy in s3/stat.py into s3/_client.py
so readdir can share it, mirroring TypeScript's isNotFoundError, which
has always lived in _client.ts.
Both nextcloud test fakes were lower fidelity than the server and would
have made the new tests vacuous: the python one omitted the collection
self-entry, and the TypeScript one rejected with NotFound where the
server answers []. Both now match what op.list actually returns.
integ/unix/ls/missing.json pins all seven, each oracled against GNU
coreutils on debian:stable-slim first. `ls <file>/x` carries a narrower
target list: databricks, dropbox, onedrive, sharepoint, ssh and gdrive
answer ENOENT there where GNU says ENOTDIR, a separate pre-existing gap
in backends this change does not touch.
test_ls_does_not_show_removed_file was pinning the bug. It wrote
/data/arch/c.txt with no mkdir, so no marker for arch/ ever existed and
removing the last key removes the directory too -- which is what stat
and ram have always reported. It now mkdirs explicitly and keeps a
sibling, so it pins invalidation rather than directory lifetime, and a
companion test pins the emptied-implicit-directory answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(errors): a path component resolves as the directory when both exist
readdir_error walked the ancestors asking is_file before is_dir, which
is not a distinction ram, redis or disk can make -- a path there is one
or the other. A keyed store can hold both an object `a` and a prefix
`a/`, which is a coexistence s3's stat already handles deliberately,
and traversal only ever reaches an intermediate component through the
directory. With an object `a` and a key `a/x.txt`, `ls /a/never` has to
report ENOENT because `never` is absent, not ENOTDIR because `a` is
also an object.
Fixed in the shared helper rather than the three keyed-store callers,
so s3, gridfs and nextcloud all get it. The order is immaterial where
the two are mutually exclusive, so ram, redis and disk are unaffected;
it is also one probe cheaper for a directory ancestor, which now stops
the walk without a second lookup.
Pinned in both languages at the helper (the coexistence case and its
mirror, where nothing coexists and ENOTDIR must survive) and at the
s3 and gridfs readdirs.
Reported by codex on #809.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(rename): directories move whole, on every backend that fakes them (py+ts)
The reported bug was dropbox refusing `mv <srcdir> <dstdir>` when
`<dstdir>/<srcname>` existed as an EMPTY directory: move_v2 rejects any
existing destination, and the conflict handler delete-and-retried only
for a destination file. rename(2) replaces an empty directory, and the
generic mv layer has already proved it empty before calling the backend,
so dropbox now mirrors msgraph's rename_replace -- list the folder one
entry deep, and keep the original error only when a child is there, so a
non-empty destination still reports GNU's "Directory not empty".
Sweeping every backend for the same question turned up three more, and
they are exactly the synthetic-directory keyed stores, where a directory
is not an object but a fact derived from keys and someone has to walk
the subtree by hand. In all four that walk handled files only:
s3 no directory rename at all; raw botocore NoSuchKey reached the
user. Now a prefix walk carries the mkdir marker and the whole
subtree, and a source that is neither object nor prefix is
ENOENT rather than backend text.
gridfs no directory rename at all; ENOENT for a directory ls had just
listed. Now a prefix retag across every revision.
ram moved files, orphaned nested subdirectories. The orphans imply
a phantom source tree, so the old name reappeared in its
parent's listing and then stat as missing -- the shape
check_dest_parents already refuses on the way in.
redis same code shape, same bug.
ram and redis get one _move_subtree that re-keys dirs, files, modified
and attrs uniformly; that also picked up nested files' mtimes, which
were being dropped across a directory rename.
Backends with real directories were already correct and are untouched:
disk (rename(2)), ssh (posix_rename), nextcloud (WebDAV MOVE), box,
gdrive, onedrive, sharepoint and dropbox (server-side folder move),
databricks (recursive copy + rm), opfs (recursive copy).
s3 classifies the source with HeadObject before copying anything rather
than falling back off a failed CopyObject. Stores disagree about a
missing source and a lenient S3-compatible one accepts the copy while
writing nothing -- the repo's own mock does exactly that -- so an
error-driven fallback would have deleted a source whose copy never
landed.
None of this was visible because a case file's `targets` is an opt-in
allowlist: a backend that cannot pass is deleted from the list and the
divergence becomes an omission nobody can see. dropbox was parked that
way, and s3/gridfs directory rename was never tested at all.
check_case_targets.py compares each case file against the modal target
set of its own directory and reports what it drops, on the same ratchet
as check_layout_parity.py: advisory by default, --strict in CI fails
when the count moves off the baseline in either direction, and a stale
excuse fails as loudly as a new gap.
empty_dir.json regains all six dropped targets, and dir_rename.json is
new -- nothing in the battery renamed a directory before. The ram and
redis unit tests already had a test_rename_dir_moves_children, but it
nested only one level, which is how the orphan survived; the new cases
add a subdirectory and fail without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(s3): a refused DeleteObjects key is not a successful move; gate per case
Two codex findings on #808, both real.
DeleteObjects answers 200 with per-key results and reports a key it
refused under "Errors" rather than raising, so reading only the absence
of an exception called the move a success while the whole source tree
survived beside the fresh copy -- a duplicated directory nobody is told
about. Both languages now collect those keys and fail. PermissionError /
EACCES because a refused delete is a lock or a policy in practice, and
because it is an fs error, so mv reports the operand and keeps going
rather than aborting the command line. Both trees are left in place,
which is what GNU mv leaves behind when the unlink half fails after the
copy half landed.
The mock's delete_objects returned None, which is why the gap was
invisible to a unit test; it now answers with Deleted/Errors the way the
real API does, and `undeletable` is how a test asks for the refused half.
The coverage gate keyed on the union of a file's cases, which hides the
exact omission it exists to catch: a sequence-style file states one
scenario across prep/act/verify cases, so dropping a backend from only
the verifying case leaves the union unchanged while the runner silently
skips the assertion. The unit is now one (case, target) pair, and the
baseline moves 836 -> 2297 with it. A "files" excuse still covers every
case in that file, since the usual reason is a property of the file.
Verified against the reported scenario: removing s3 from only
mv_dir_rename_subtree_followed now fails the gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Seventeen pages still documented parquet, ORC, HDF5 and feather
rendering, which neither language ships any more: the ops are absent
from the source, the extras are gone from pyproject, and the two example
files the pages linked to do not exist. Ten carried a whole Data Format
Support section, the rest advertised cat_parquet and friends inline.
Filenames and find examples stay. A .parquet object still sits on these
backends and still reads as raw bytes, so only the rendering claims are
untrue.
Also: home/python.mdx was a lone redirect stub with no nav entry and no
inbound links, four images had no reference anywhere in the repo, and
four pages carried no description.
The sidebar rendered six databases, three cloud stores and every CLI as
the same generic glyph, so the icon carried no information. Thirteen
brand marks now stand in: notion, linear, mongodb (also gridfs, which is
mongodb's), postgres, qdrant, chroma, redis, supabase, databricks, box,
cloudflare for r2, nextcloud and langfuse, applied across the home,
python and typescript trees.
Twelve come from simple-icons at the pinned 16.28.0 with the brand hex
filled in, which is the convention the existing logos already follow;
regenerating minio-logo.svg that way reproduces the committed file byte
for byte. Notion's brand hex is #000000, so it takes the neutral gray
the OpenAI mark already uses rather than vanishing in dark mode. Chroma
and langfuse are not in simple-icons and come from each project's own
mark instead.
Oracle, LanceDB and QingStor keep their generic icons: oracle was
removed from simple-icons upstream and the other two publish no usable
svg.
github_ci is gone from the docs, with its nav entries, its resource
matrix row and the resource index bullet.
PR-0e from the restructure audit (bugs 8+9+G5). The new mv-onto-empty-dir
integ pin caught the rename write-flavor staleness in disk, ssh, onedrive,
and sharepoint; every backend rename now takes the unlink flavor on dst.
Dropbox folder-conflict mv and s3/gridfs directory rename are pre-existing
divergences excluded from the pin and filed as follow-ups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gws was left as the only quoted description, which is the shape that
invites the next unquoted colon. Quoting the rest makes the frontmatter
uniform and immune to it. No parsed value changes.
This also re-targets the CLI pages for the docs build. The icons from
#801 never went live: that deploy aborted on the gws parse error and
dropped all sixteen files with it, and #803 only changed the two gws
pages, so the incremental build republished those alone.
The description held an unquoted ": ", which YAML reads as a nested
mapping key, so Mintlify failed the whole deploy on both gws pages.
Nothing checked frontmatter before CI, so a one-character mistake only
surfaced as a failed deploy. check_docs_frontmatter.py parses every page
the way the docs build does and runs beside the other repo-root checks.
Both from the codex review of #800, both verified against Python first.
P2 — a null secret was being masked. `redactValueWithSchema` returns
null *before* it asks whether a field is secret, mirroring Python's
`if value is None: continue` in `_walk_config_dump`. The hand-written
qdrant and lancedb redactors masked unconditionally, and both keys are
`string | null`, so a keyless local Qdrant or an on-disk LanceDB got a
`<REDACTED>` marker for a credential it never had — making
`Workspace.load` demand a fresh config for a self-contained snapshot.
`RedactedConfig<T, K>` now keeps null in the redacted twin when the
source allows it, so the type says this too.
P1 — chroma has no credential, so its state carried no marker, so
`buildMountArgs` handed back a `RAMResource` and a /chroma mount loaded
as an empty local directory. The cause is broader than chroma:
TypeScript rebuilds *nothing* from state, because core cannot import
`buildResource` (it lives in node/browser), while Python reconstructs the
class from `resource_state["type"]` via its registry
(`_resource_class_for`). Until core gets a resource factory,
`resourceStateRequiresOverride` also honors an explicit
`needs_override`, and the six config-backed resources set it — so the
mount refuses to load rather than coming back empty. Python already
writes that field on four resources and reads it nowhere; TypeScript now
reads it, which also protects Python-written snapshots loaded in TS.
Tests: keyless lancedb/qdrant cases on both sides (Python passes as-is,
which is the proof the null rule was already its behavior), and the
registry sweep now asserts every config-backed backend demands a
resource at load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(readme): fix the hero snippet and resync every mirror
The hero snippet called ws.command(...), which exists in neither
language: registration is the standalone command() plus mount.register.
It is replaced with a Python example that mounts ram, redis and slack
side by side, captures python with monty, and installs a CLI, all of it
run against the published 0.0.5 packages first.
Two more corrections. The filetype sentence promised parsed PDF pages,
which the filetype removal took away, so it now says a format renders
however you register it. DeepSeek Harness joins the coding agents row.
The eleven mirrors are regenerated from the root rather than patched,
which also closes drift they had accumulated: a stale backend list, the
old CLI + daemon integrations line, a missing Grok Build entry and a
Codex link pointing at the wrong docs path.
* docs(cli): give each CLI page its own icon
Every CLI page shared icon: terminal, so the sidebar was nine identical
rows. Each now takes the icon its service already uses elsewhere in the
docs: slack, discord, github for gh, google for gws, envelope for
himalaya, book for ntn and chart-gantt for linear (matching the notion
and linear setup pages, since Font Awesome carries no brand mark for
either), and git-alt for git. gws and himalaya also get their names
spelled GWS and Himalaya; the rest stay lowercase because that is the
head word you type.
* examples(filetype): register through the public mount accessor
The example reached into ws._registry.mount_for, but ws.mount is public
and returns the same MountEntry. Output is unchanged, so the CI truth
file still matches.
The interface I added to `resource/base.ts` was a byte-identical copy of
a private one that already sat in `workspace/snapshot/types.ts` feeding
`ResourceState`. Snapshot types now import the exported one, so the
Resource contract and the snapshot format name one shape.
The doc comment now says what the two keys are for rather than
paraphrasing: `type` is the registry name Python's `_resource_class_for`
looks up before falling back to the mount's `resource_class` import path,
and `config` is what `resourceStateRequiresOverride` scans. It also
records why the literal twin of Python's `dict[str, Any]` does not work
here — a TS interface has no implicit index signature, so
`Record<string, unknown>` would reject every named `XResourceState`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`snapshot/state.ts` cast every mount's resource into `{ getState }` and
called it unconditionally, so a resource without one was a runtime crash
at save time rather than a compile error. postgres and mongodb — both
registry-mountable, in node and browser — had none, and snapshotting such
a mount died with `resource.getState is not a function`.
Structure follows Python's, which already had all of this:
- `BaseResource` gains `getState()` returning `{type: this.kind}` and a
no-op `loadState()`, the twins of `BaseResource.get_state` /
`load_state`. `kind` moves onto the base as abstract so the default can
spell itself, mirroring Python's `name`.
- `Resource` declares both non-optionally, so the two casts in
`snapshot/state.ts` go away.
- The eight config-backed resources that had no state — chroma, dify,
qdrant (core), postgres, mongodb (node + browser), lancedb — now carry
their config, redacted. Inheriting the bare default would be worse than
crashing: with no `<REDACTED>` marker,`resourceStateRequiresOverride`
returns false and `buildMountArgs` substitutes an empty RAMResource,
turning a live database mount into an empty directory on load.
- Redacted fields were taken from Python's own `secret_field_names`, not
guessed: qdrant/lancedb `apiKey`, postgres `dsn`, mongodb `uri`, dify
`apiKey` (Python masks it in `get_state` rather than on the model),
chroma none.
- `HfResource` re-narrows `getState` to abstract so the bare default
cannot reach a Hub resource. Python has no shared Hub base; its four
resources each spell `get_state`.
- Test fakes that stood up a `Resource` by hand now extend `BaseResource`
like real resources do, so they inherit the default instead of carrying
a stub. The three that cannot (two object literals, and the bare class
that exists to have no `storageId`) spell the pair.
- mem0/onedrive/sharepoint swap `Record<string, unknown>` for named state
types, joining the ~35 resources that already had them.
Tests: a registry-derived sweep over the six database backends asserts
getState/loadState exist and the credential is masked, plus an end-to-end
`toStateDict` over a postgres mount. Python gains the twin sweep in
`tests/resource/test_state_round_trip.py` — it passes as-is, which is the
point: this closes a TypeScript-only gap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(discord): parity batch — dirname-parsed scope, tombstoned attachments, date-less grep paths, smart head revival (py+ts)
Phase 0 PR-0d from the restructure plan (bugs 5+6+7 plus one found
during the scope investigation).
1. Scope detection contract (bug 6). Python's detect_scope was async
and resolved guild/channel ids through the index cache; the TypeScript
twin parses them out of the `name__id` dirnames the tree itself mints.
The index store is an exact-key lookup, so the async machinery could
never resolve anything the dirname doesn't already carry — but it
COULD miss (cold cache), and then grep/rg raised "cannot resolve guild
ID" internally and fell back to a per-file scan with a spurious
stderr warning, on paths whose ids were sitting right in the operand.
Python now mirrors slack-py/discord-ts: sync, dirname-parsed, with the
same field set (level, use_native, names, container). Consequences
picked up on purpose:
- members-container paths no longer push down to guild message search
(use_native false, matching TS) — message search cannot answer a
grep over member JSONs.
- the "root-level search not yet supported" special case is gone; a
root operand falls through to the generic scan like TS and errors
GNU-shaped.
- bare names (no `__id`) now skip the push-down silently instead of
depending on cache temperature.
2. coalesce_scopes ported to TypeScript. Python (like slack) coalesces
concrete same-channel chat.jsonl operands into one channel-wide native
search under -w; TS discord never had it, so the same command produced
formatted search lines on python and raw JSONL scan lines on TS.
TS gains coalesceScopes with python's exact semantics.
3. Tombstoned attachments filtered (bug 5). Discord listed any
attachment with an id; slack requires id + download URL + size because
tombstoned/access-restricted payloads carry an id but nothing to read
— they showed as phantom files that ENOENT on read, with unknown
sizes. Both discord twins now carry slack's three-part guard, and the
integ fake gained a `tombstoned` seed knob to prove it end to end.
4. Date-less grep result paths (bug 7). A search hit without a
timestamp rendered `.../channel//chat.jsonl`. Both twins now derive
the day from the message snowflake (the same day readdir buckets it
under); a hit with neither timestamp nor parseable id points at the
channel dir, slack-style.
5. Smart head was dead code (found during 6). head.py gated its
native first-N-messages fetch on scope.level == "file" — a level no
detect_scope in either language has ever produced — so every head ran
the full-day fetch. Revived on level "messages", plus the end-of-day
bound the branch always needed: with `after`, a short day spills into
the next one until `limit` is met, so unbounded revival would print
lines chat.jsonl does not contain. Output is byte-identical to the
generic path by construction (single renderer). TS has no head
override; its scan produces the same bytes.
Also removed the vestigial 'file' member from the TS DiscordLevel
union.
Tests
- py unit: test_scope.py rewritten against the sync API (mirrors
scope.test.ts case-for-case), test_grep_pushdown.py reworked to
`name__id` paths with new cold-cache + bare-name pins, test_head.py
new (smart gate, day-boundary spill, generic fallbacks), readdir +
formatters guards. Full suite green.
- ts unit: scope.test.ts +4 (coalesce), readdir.test.ts +1
(tombstone), search.test.ts +2 (snowflake fallback, date-less),
grep.test.ts single-file scan repointed + coalesce-native case.
Core suite 8205 passed.
- integ: fixture gains a tombstoned attachment and a second releases
day; new scope.json cases pin channel-native grep -w with empty
stderr, coalesced single-file grep -w, and head day boundaries.
Post-fix: discord + cli-discord = 62/62 on BOTH hosts. Pre-fix
(sources stashed, pins kept): TS fails 3 (ghost listed in ls/find,
raw-line grep), python fails 2 (ghost in ls/find) — python's grep
pins pass pre-fix only because the battery's shared workspace
pre-warms the index; the cold-cache symptom is pinned at unit level.
- pnpm -r typecheck clean, pre-commit clean (mypy: no issues in 1819
files).
Known family-wide follow-ups deliberately not taken here: native
search push-down ignores -r on directory operands (slack too), and a
coalesced/date-level push-down can return hits outside the operand
date set (slack too). Both predate this change and need one decision
across the chat family.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(discord): codex round — coalesce only message files, smart-head guards (py+ts)
Review round on #797, all four findings taken:
- P1: coalescing keyed on `not use_native` swallowed file_blob operands
— grep/rg -w on an attachment became a channel-wide MESSAGE search
that says nothing about the requested bytes (pre-existing python
behavior the port had copied into TS). coalesce_scopes/coalesceScopes
now require every operand to be a chat.jsonl messages scope, and the
call sites gate on level == "messages".
- P2: smart head forwarded any count as one Discord page; the endpoint
caps limit at 100, and zero/negative (all-but-last-N) forms cannot be
a single page. Counts outside 1..100 keep the generic path.
- P2: an empty day rendered "\n" instead of b"" — the branch now reuses
history_jsonl_bytes, the same renderer read() uses, so empty is empty
and the bytes match the generic path by construction.
- P2: head -v bypassed the ==> path <== header; verbose keeps the
generic path.
Pins: py unit +5 (file-blob scan, -v, >100/negative counts, empty-day
b""), ts unit +3 (coalesce blob/mixed null, blob-scan no-search),
integ +3 (attachment grep -w scans bytes, head -v header, empty-day
head). Pre-fix: TS fails the attachment pin; python additionally fails
the -v and empty-day pins (got the predicted lone "\n"). Post-fix:
discord + cli-discord = 65/65 on both hosts; typecheck + pre-commit
clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Release Python 0.0.5a1
* Release TypeScript 0.0.5-alpha.1
* Release TypeScript dsh 0.0.1-alpha.1
* test(version): let the --version assertions accept a prerelease
The five TS assertions pinned the output to \d+\.\d+\.\d+, which cannot
express a prerelease, so every one of them failed the moment the version
became 0.0.5-alpha.1. Python asserts the same output with startswith and
was unaffected. The shape is still asserted, now with an optional
prerelease suffix.
* Release 0.0.5 (mirage-dsh 0.0.5 line at 0.0.1)
* feat(ts): install a CLI from a file reference in yaml
`clis: {tally: {cli: ./tally.mjs:TALLY}}` now resolves in TypeScript, so
a deployment can ship its own program tree without a host program
calling registerCli. Mirrors the `":" in name` branch of Python's
`cli_spec_for`, one layer up: `cliSpecFor` lives in core, which has no
filesystem and is synchronous, so the ref resolves in the config layer.
The referenced file is imported by Node, so .mjs, .js and .ts all work;
a .ts spec rides Node's strip-only type stripping and is refused with a
hint when it uses a construct that needs codegen.
* fix(ts): node's specifier rule for cli refs, and a real spec check
Both from review.
A bare specifier carries slashes in Node (`@scope/pkg`, `pkg/subpath`),
so copying Python's `"/" in source` test read every scoped or subpath
package as a file, rebased it under the config directory and reported it
missing. A path is now what Node calls a path: relative or absolute,
plus a bare filename carrying a module suffix.
A class name is not proof a tree is usable. Dispatch reads subcommands,
aliases and options at every level, so a value that only answers to the
name crashed on the first line an agent typed instead of failing the
create. The shape is checked recursively instead.
* refactor(runtime): replace confined with a three-value reach marker
A runtime now declares reach: vfs | process | remote, stating whether
the workspace dispatch gate is its only door or the code can act
around it (host process doors, another machine). The default is
process, the no-promise claim, so a custom runtime must narrow its
reach explicitly. The dsh sandbox claim reads the aggregate: every
runtime at vfs means workspace-write, anything wider means no claim.
Mirrored in Python (RuntimeReach in runtime/types.py) with reach
declared on monty, quickjs, wasi, vfs, local, and RemoteSandbox.
* fix(pyodide): seal the js module so the guest has no host door
Pyodide's default exposes the host globalThis as the `js` module,
which under Node handed guest code js.process (host env, confirmed
reading HOME) and js.fetch (network) — doors around the workspace
bridge that made the runtime's reach='vfs' claim false and
contradicted its own 'no network' docstring. Pass a null-prototype
jsglobals to loadPyodide: `import js` still resolves but the host
globals are unreachable through it, while pyodide's internals (which
capture their globals at load time) and the FS bridge are unaffected.
Pinned by jsglobals.test.ts.
* feat(dsh): stamp ShellSandboxInfo on run results and process handles
When the world is fully workspace-bound (vfsOnly), run() and start()
now fill dsh's optional sandbox field: mode workspace-write,
enforcement 'full' (the VFS gate is unbypassable, so unlike an OS
sandbox on an old kernel there is no promised effect it fails to
govern), denied false (mirage has no out-of-band denial channel; a
refused write fails in-band as an ordinary command error), and
runnerFailed false (the executor is the runner). The process handle
stamps it on settle. Omitted when any runtime reaches beyond the
workspace.
* feat(dsh): stream background command output through a JobConsole
Adds a public ExecuteOptions.sink: pass a JobConsole and the line's
output streams into it as each statement finishes, instead of being
returned whole (the result then carries only the exit code). This
reuses the executor's existing internal sink mechanism, so a compound
line flushes per statement and stdout/stderr keep their channels.
MirageShellProcess is rewritten over that seam: start() runs the
command with a console as its sink and a follow loop drains it into
the read buffer, so readOutput() delivers output incrementally and
stdout/stderr interleave in order (stderr opened by a marker) rather
than stderr being concatenated at the end. The unread backlog is
bounded to stdoutMaxBytes (tail kept, lossy flagged) so a reader that
never drains cannot grow it without limit.
* feat(dsh): spill the full stream to a workspace file on overrun
Adds an opt-in spillDir config. When a background command's streamed
output overruns its delta budget, the full stdout and stderr are
written to files under that workspace directory and readOutput()
points at them (stdoutSpillPath/stderrSpillPath), so a reader can
recover what the delta dropped by reading the spill through the same
VFS. Memory stays bounded: each channel buffers only until the first
overrun, then flushes to its file and appends from there. A write
failure (no writable mount at the path) disables the sink and leaves
the paths undefined, the honest 'no safe path' answer. Default unset,
so nothing spills unless a deployment asks for it.
* docs(dsh): custom backends, background streaming, and the reach model
Corrects the sandbox-claim wording to the reach model (workspace-write
when every runtime reaches only the vfs, dropped when one reaches the
host), and adds a Custom backends section (registerResourceFactory,
host-side before the workspace builds) and a Background commands
section (per-statement streaming, bounded backlog, spillDir).
* test(dsh,core): satisfy lint on the streaming tests
Narrow spill paths with an explicit guard instead of a non-null
assertion (forbidden in the dsh package), and drop the now-unnecessary
ExecuteResult casts the sink overload already implies.
* style: prettier formatting on the streaming changes
* fix(core): drain a buffered line into the sink
A sink only saw output the command-tree walk emitted, so a whole-line
runtime, the syntax gate, a policy denial and a failed line all answered
with bytes in hand that a streaming caller never read. executeLine now
moves any buffered result into the console on every path, in one place
rather than five, and the result stays empty as it already did when the
line streamed.
* fix(dsh): bound the console store, make the spill dir idempotent
Capping the delta did not bound memory: reading a chunk advances a
cursor but frees nothing, so an uncapped store held every chunk of a
noisy background command for the life of the process. The store now
carries a retention budget, and the drain reports a trimmed chunk as
lossy and stops the spill, since a file missing the middle of a stream
is worse than no file.
The spill directory is created through ensureDirPath, which walks the
ancestors and accepts a refusal for a directory that now exists, so two
commands overrunning at once do not cost the loser its spill.
* feat(shell): redis console store with cross-language streaming integ
* feat(shell): console config block, shared console battery, redis store package
* refactor(shell): job_table package, and codex fixes on the redis console
Split job_table into types/constants/table in both languages, mirroring
the console package.
Redis console review fixes: the minted key prefix is public (JobConsole
.store plus RedisConsoleStore.key_prefix) so an external reader can be
handed a console's address; keys expire ttl_seconds after the last
append (default one day) instead of accumulating; the ending chunk is
terminal in the store itself, so an emit racing a kill past the local
guard is dropped server-side; and the TS loader validates the console
block's value types the way Pydantic already did.
* refactor(ts): four shared factories mirroring the Python originals (#609 item 24 / T2-9)
TypeScript had grown four copy-drift sites where Python already has one
shared module. Each is extracted onto the Python shape, so the two trees
line up module-for-module.
builtins/shared.ts (twin of workspace/executor/builtins/shared.py). Three
independent errorResult()s with three different signatures — capacity.ts
hardcoded the 'df' command, links.ts hardcoded exit 1, only metadata.ts
took both — so a GNU exit-code fix could not be shared. They collapse onto
result/ok/fail/finish, plus operandText/absPath/splitFlags/splitValueFlags/
expandOperands. Drops links.ts's re-implemented abs+splitFlags and
capacity.ts's import of splitValueFlags from metadata.ts (df reaching into
chmod/touch). `Result` moves here from scope.ts, where Python keeps it in
shared.py; the other 12 builtins change only their import line.
The chmod/chown/chgrp/touch handlers each tracked an exitCode variable
that was only ever set to 1 next to an errors.push, so finish() derives it
from `errors` exactly as Python does and the variable is gone.
workspace/executor/interpreter.ts. python/handle.ts and js/handle.ts were
the same 130 lines apart from the label, the unavailable-error class, the
payload flag, Python's -m guard and -x, and js's .mjs module flag.
makeInterpreterHandler takes those as a spec; both handle.ts keep their
path, their exported name and their signature, so callers and
runtime.test.ts's dynamic import are untouched. The .mjs check moves into
the js wrapper, which is where Python's js.py keeps it. Kept in
workspace/executor rather than beside commands/builtin/general/
interpreter.ts because no commands/ module imports workspace/ today and
this should not be the first.
makeRm. generic/rm_command.ts already existed and gcal already used it;
the gdocs/gsheets/gslides bodies were verified character-identical to it
modulo indentation, so the three 79-line files become 20-line bindings.
core/google/tree_ops.ts (twin of core/google/tree_ops.py). makeStat and
makeUnlink over a backend's readdir; the six modules stay as real files
exporting `stat`/`unlink`, so the barrel and the provision/io importers do
not move. The three local eisdir() copies give way to the shared one in
utils/errors.ts: isFsError keys on `code`, which is unchanged, and its
message (the bare virtual path) is what Python's IsADirectoryError(raw)
already produced.
No behavior change intended. -995/+147 across the 26 edited files.
* fix(js): honor the explicit `-` stdin operand, like python3 and node
`js - a b` read a file literally named `-` and failed with
"js: /-: No such file", while `python3 - a b` on the same tree read the
program from stdin. Python has no such split: resolve_source's
STDIN_OPERAND branch (interpreter.py) is unconditional, so both
interpreters honor the operand, and js.py inherits it by taking the
default Argv0Rules. Real `node -` agrees — it reads the program from
stdin and keeps the following words as argv.
Verified against Python by calling resolve_source directly for the js
call shape: `js - a b` yields mode=stdin, args=['a', 'b'], no script
path. Both quickjs hosts then build scriptArgs the same way (an absent
prog leaves the args alone), so the operand lands identically.
Found while extracting the shared interpreter handler in the previous
commit; kept separate because it is a behavior change, not a move.
The unit test fails without this fix (exit 1, not 0). The integ case
pins py=ts.
* test(ts): mirror the Python suites for the two new shared modules
Python has tests/workspace/executor/builtins/test_shared.py and
tests/core/google/test_tree_ops.py; TypeScript had neither twin, and both
modules are now shared chokepoints rather than per-command code.
shared.test.ts ports test_shared.py case for case: the result triple
(ok/fail/finish, including finish keeping a carried IOResult's writes and
deriving the exit code from `errors`), operandText/absPath, the three
splitFlags cases, the three splitValueFlags cases, and the expandOperands
glob expansion over a real RAM mount.
tree_ops.test.ts covers makeStat over a stub readdir: the synthetic
owned/shared roots answer as directories without a readdir round trip, an
indexed document is served straight from the index, a miss warms the
parent and then raises ENOENT, and a call with no index raises ENOENT
without reaching the backend.
It deliberately does not port test_tree_ops.py's one case — that a readdir
raising a non-fs error propagates out of stat. TypeScript swallows it in a
bare catch and reports ENOENT instead, which is a real pre-existing
divergence in the T1-G "honest errors" family; pinning current TS behavior
here would entrench it. Filed separately.
* refactor(ts): keep the interpreter opt interfaces module-private
Nothing outside interpreter.ts imports them — both handle.ts wrappers pass
object literals — so knip reported them as unused exports.
* fix(google): only an absent parent may collapse into ENOENT (py+ts)
Every Drive-family stat/read/unlink warms a cold index by listing the
parent, then retries. Python narrows that call to FileNotFoundError, so
an auth failure or a dropped connection reaches the caller; TypeScript
swallowed it in a bare `catch {}` and reported "No such file" instead.
Five TS sites did this (google/tree_ops stat+unlink, gdrive stat, gdrive
read, gdocs/gsheets/gslides read) — seven catch blocks in all.
tree_ops.unlink was wrong in the other direction: Python called readdir
completely unguarded, so removing a path under a missing directory
reported the parent rather than the operand. GNU names the operand:
$ rm nodir/file.txt
rm: cannot remove 'nodir/file.txt': No such file or directory
(pinned against coreutils in debian:stable-slim). Both sides now use the
shape Python's make_stat already had: swallow a not-found parent, let
everything else through. TypeScript spells it with the existing
isEnoent(), the twin of `except FileNotFoundError`.
gdrive/readdir warms a parent unguarded in both languages — already
aligned, left alone.
Tests fail without the fix. Two of them mirror Python suites that had
pinned this since before the TS port (test_stat_propagates_parent_
refresh_failure, test_read_propagates_parent_refresh_failure); the
tree_ops pair is new on both sides.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(cache): one entry_or_warm for every cold-index lookup (py+ts)
Twelve call sites across eight backends had grown their own copy of
"ask the index, list the parent if cold, retry, else ENOENT", each with
its own idea of what a failed listing means. They now share one function
in cache/index/warm, so exactly one catch in the tree decides that.
The previous commit narrowed seven of those catches by hand; this makes
the narrowing structural instead, and sweeps the five that commit did
not reach:
box/read (x2) bare catch, py narrowed -> divergence
gmail/read bare catch, py narrowed -> divergence
dropbox/read (x2) bare catch, py caught (FileNotFoundError,
DropboxApiError) -- but readdir already turns the
API's 409 for a missing path into ENOENT, so that
second arm only ever swallowed live API failures.
Python had the same bug here, from the other side.
gdrive/readdir warms a parent unguarded in BOTH languages; already
aligned, deliberately left alone.
Also folds eight private eisdir() copies into the shared helper. The
copies built `EISDIR: ${p}` and never stamped virtualPath, and
execute.ts renders errorVirtualPath(err) straight into stderr, so a
directory read printed
cat: EISDIR: /docs/owned: Is a directory
where Python printed the GNU form. Two TS tests asserted on that
message with /EISDIR/ and so had been holding the bug in place; they now
assert the stamped code and virtualPath, which is what Python's
IsADirectoryError(virtual) carries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): pin the Drive-family cold-lookup error paths on both hosts
Five cases over the gapps and gdrive targets, all curated by running the
battery and reading the real output rather than guessing the strings:
gapp_cat_dir_is_gnu cat /docs/owned -> Is a directory
gapp_cat_missing_parent cat .../nope/x.gdoc.json -> names the operand
gapp_ls_missing_parent ls .../nope -> ENOENT, exit 2
gapp_rm_missing_parent rm .../nope/x.gdoc.json -> names the operand
gdrive_cat_dir_is_gnu cat /data/sub -> Is a directory
clear_cache forces the cold index, which is the branch entry_or_warm
now owns. Both wordings match GNU pinned in debian:stable-slim
(`cat: <path>: Is a directory`, exit 1).
Two things these cases do NOT prove, so nobody reads more into them:
- rm builds its own diagnostic from p.rawPath, so it reports the
operand whatever the backend error carries. The unlink
operand-naming fix is pinned by test_tree_ops.py, not here.
- `cat` on a directory is answered by the read generic off stat, so
the eisdir consolidation is pinned by the gdrive/dropbox unit tests
(which assert virtualPath, the field errorVirtualPath reads), not by
these.
They sit at seq 930030+ deliberately: at 520100 their clear_cache wiped
the index that gdocs_prov_cat (930023) expects warm, and that case
started failing on hits=1 vs hits=0.
Verified: gapps + gdrive, 2193 passed / 0 failed, on the typescript-node
host and again on the python host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* style: formatters over the warm-helper sweep, + the py eisdir message pin
isort regroups the hand-inserted `partial` / `entry_or_warm` imports and
prettier rewraps one dropbox thunk; no behavior change.
Also completes the eisdir work on the Python side: the gdrive read test
asserted only the exception type, so it would not have caught the "EISDIR: "
prefix the deleted TS copies were adding. It now asserts the message is the
bare operand, which is the string the TS twin pins through virtualPath.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(gate): lower the layout-parity baseline to 301
The ratchet fails on a drop as well as a rise, so a closed divergence has
to be locked in. Net -1 from three real movements:
-1 core/google: python-only ['tree_ops'] tree_ops.ts now exists
-1 workspace/executor/builtins: python-only ['shared'] shared.ts now exists
+1 workspace/executor: typescript-only [... 'interpreter']
The +1 is the deliberate layering call from this PR: the shared interpreter
handler lives under workspace/ rather than beside Python's twin in
commands/builtin/general/, because no commands/ module imports workspace/
today and this should not be the first. Recording it here so the net -1 does
not hide that one improvement paid for one new divergence.
cache/index/warm does not appear: it landed on both sides, so it is neutral.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(spec): js/node end option parsing at the first operand, like python
`node - -e x` ran `x` instead of the piped program, and `node s.js -m`
flipped the interpreter into module mode instead of handing s.js its own
-m. Both real interpreters disagree, pinned locally:
$ echo 'console.log(process.argv.slice(1))' | node - -e 'nope'
[ '-', '-e', 'nope' ]
$ echo 'import sys; print(sys.argv)' | python3 - -c 'nope'
['-', '-c', 'nope']
(node 22.8.0, CPython 3.13)
`Operand.remainder` (argparse nargs=REMAINDER) already exists and both
parsers already honor it in one line — the first operand ends option
parsing, so every word after it is an operand verbatim. `python`/`python3`
have carried it all along; js/node simply never got it, in EITHER
language, so this is a symmetric py+ts spec fix rather than a TS-only one.
Verified by calling parse_command directly: js/node/python/python3 now
agree on `- -e PROG` and on `s.js -m --module`, while a flag BEFORE the
first operand is still the interpreter's (`js -m -e CODE a`).
One consequence beyond the reported case: remainder also turns off
lenient_dash_operands, so `js -Z foo` now reports an invalid option
instead of silently treating `-Z` as the script path (which produced
"js: /-Z: No such file"). Real node agrees — `node -Z foo` prints
"node: bad option: -Z". Nothing in integ or the suites relied on the old
leniency.
Reported by codex on #792. Its premise was wrong in one respect — it said
to mark the rest operand "as the Python specs do", but Python's js/node
spec is Operand(type="str") with no remainder, so Python had the same bug.
Fixing TS alone would have created a divergence.
Tests: 4 parametrized parser cases per language over the real js/node/
python/python3 specs, plus a quickjs end-to-end case. All 4 py and all 5
TS fail without the spec change (python/python3 pass either way, which is
what makes them the control). integ/runtime/quickjs.json `stdin_operand`
gains the -e and -m steps.
Verified: pytest 14159 passed / 0 failed; pnpm -r typecheck clean;
core suite 8129 passed (the one python_repl timeout is the known
Pyodide-boot-under-load flake, 10/10 alone); runtime/run.ts quickjs
11 passed / 0 failed on the typescript host; spec drift, spec parity
(93 match), layout parity (301 = baseline) and width table all clean.
The python host's quickjs suite needs the MIRAGE_QUICKJS_HOME WASI blob
that CI downloads, so that leg is covered by test_integ.yml rather than
locally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(quickjs): end qjs's own option parsing before the program's argv
The python host's quickjs runtime appends the program's arguments
straight onto the qjs command line:
argv += ["-e", args.code, *named, *args.args]
Unlike CPython's -c, qjs's -e does NOT end option parsing, so any
program argument that spells a qjs switch was read as one. Pinned
against the build CI uses (quickjs-ng v0.15.1 qjs.c): the option loop
runs while `*argv[optind] == '-'`, breaks only on a bare `--`, and
scriptArgs start at `argv + optind`.
That is what made integ-runtime red on the previous commit, and only on
the python host:
node - -e 'console.log("flag")' -> flag (second -e wins)
node - -m -> (empty) (module mode flipped)
Both are now `--`-separated, so they land in scriptArgs where the parser
already put them. qjs consumes the `--` itself, and the -e branch still
wins over any filename, so a named program keeps scriptArgs[0].
Surfaced by the remainder fix in the previous commit rather than caused
by it: before that, mirage's own parser swallowed those words, so they
never reached the runtime. Any program argument spelling a qjs switch
had this hole -- `node script.js -m` included.
TypeScript needs no twin. Its quickjs runtime is in-process and assigns
scriptArgs as a global (`quickjs.ts:387`), never building a qjs command
line, which is why the TS host passed the same integ steps. Checked the
other python runtimes too: wasi.py and local.py both go through CPython
`-c`, which DOES end option parsing (pinned: `python3 -c prog -e FLAG -m`
puts `-e FLAG -m` in sys.argv), and monty passes argv as a data global.
quickjs was the only one.
Verified with the real engine, after downloading the pinned
qjs-wasi.wasm: `runtime/run.py quickjs` on the python host is 11 passed
/ 0 failed with this change and reproduces CI's exact two-line failure
without it; the whole python runtime battery is 99 passed / 0 failed
(bridge/docker/wasi skip on unmet env). pytest 14192 passed / 0 failed
-- 33 more than the previous run, because the @live quickjs tests now
execute instead of skipping.
The three new unit tests pin the argv shape without needing the build,
and all three fail without this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(find): s3+gridfs synthesize implicit dirs, honest -empty start (py+ts)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cache,find): refresh ancestor listings, keep negated -name exact, dedup shadowed paths
A deep write on a keyed store materializes every missing level of the key at
once, but only the immediate parent's listing was invalidated, so `ls` served
the pre-write view until the 600s index TTL expired. That needed both
invalidation surfaces: `cache.context.invalidate_ancestors` only fires when a
cache manager is pushed (the command path: tee, cp, touch), while a shell
redirect, FUSE, and the ops facade go through the dispatcher's own post-write
bookkeeping. s3/gridfs `write` gets the first, `CacheManager.invalidate_ancestors`
the second. databricks_volume is left alone deliberately: its write requires
the parent to exist, so it cannot materialize a level.
Nextcloud compiles `-name` to a SQL LIKE, a deliberate superset that the
client-side keep() re-filters. Negation has nothing to re-filter with:
NOT(superset) withholds rows the true predicate keeps, so `! -name '*_b.txt'`
dropped `axb.txt`, which GNU keeps. Only an exact comparison is negatable now,
and a `[` or `\` pattern is not pushed down in either direction.
find on s3/gridfs could print one path twice when a file key and a synthesized
directory name it (`data/a` beside `data/a/b.txt`), which these stores allow
and a filesystem does not; all four implementations dedup like hf already did.
The TS integ runner grows the generic consistency adapter python already had:
adapters expose a shadow workspace over the same store, so scenario cases stop
silently skipping for s3/gridfs/dropbox/ssh/hf, and a target that cannot build
one says so instead of dropping its cases in silence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ssh): fingerprint the TS stat by mtime so ALWAYS revalidates
The python ssh stat carries `fingerprint=mod_str or None` (the remote
mtime); the TypeScript twin hard-coded `fingerprint: null`. Under
ConsistencyPolicy.ALWAYS, reconcile.probe reads that fingerprint, so
TypeScript returned Verdict.UNKNOWN, mayServeCached said yes, and the
cached copy kept being served after the server had replaced it.
Surfaced by the generic consistency adapter in this PR: with only
onedrive/sharepoint registered, `consistency_always_revalidates` had
never run against ssh on the TypeScript host. It now does, and failed
with "v1\nv1\n" against python's "v1\nv2\n".
The remote mtime is the only cheap change token SFTP offers, which is
why python uses it; directories get it too, matching python's single
FileStat. Audited the rest of the class: the other hard-coded
`fingerprint: null` sites (core/dify/stat.ts, node/nextcloud/watch.ts)
both match python's `fingerprint=None`.
Not fixed here: the same function formats `modified`/`atime` with
`new Date().toISOString()` (`.000Z`) where python emits second-precision
`Z` via epoch_to_iso. Aligning that means exporting epochToIso from
core's public index, which is a wider surface change than this fix
needs.
integ `consistency_always_revalidates` on ssh is the regression test —
it fails on the old code and passes on the new; ran the ssh target 3x
(2250 passed, 0 failed each) since the fingerprint is a 1-second-
resolution mtime and I wanted the granularity risk ruled out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(shell): job console for streaming background output
Background jobs stored their output in two byte fields that were only
readable once the job ended, so nothing could watch a running job and a
reader had no way to say where it had got to.
Replace them with a JobConsole: an append-only log of timestamped chunks
addressed by sequence number, where a reader's whole state is one
integer. Readers can join late, read from any position, and disappear
without the console noticing. The job's ending is an in-band CONTROL
chunk, so "is it done" and "what did it print" are the same question.
- mirage/shell/console/: config, store protocol, RAM store, JobConsole.
The RAM store keeps a waiter registry keyed by event loop, so a
reader parked on another thread and loop is woken safely.
- Job drops stdout/stderr for a console. All writes to a job's status,
exit code, and ending happen in one place, so the table has a single
writer and the polling _refresh is gone.
- kill and kill_all are async and join on the console, so a killed job
is settled before kill returns.
- Thread the console down the executor as an output sink. Sequencing
constructs (loops, lists, groups, subshells, conditionals) pass it to
their children, so each statement lands as it finishes instead of the
whole construct arriving at the end. Capture sites (command
substitution, pipe stages, redirects) do not inherit it and keep
receiving their output as a value.
- TypeScript: connect the job's AbortController to the executor, which
already checked the signal but was never given one. kill now stops a
running job, so it joins like Python instead of settling the job
itself.
- Drop the unreachable "no job table" branch in handle_background: the
workspace always has one, so the console is never optional.
Snapshots keep the existing stdout/stderr byte format; a restored job
rebuilds a finished console from it.
* feat(shell): bare `wait` adopts job output, plus compound-job integ coverage
The battery had no case that backgrounded a compound construct: all
eight existing job cases background a simple command, so the path this
branch rewrote was untested. Adding those cases surfaced a real bug.
`wait` with no operand waited for every job and then discarded what they
printed, while `wait <id>` returned it. A real shell has nothing to
adopt because its jobs share the terminal and have already printed;
mirage jobs print to their console, so the shell has to surface it or
the output is stranded. This was pre-existing, not introduced here.
- Bare `wait` now concatenates every unreaped job's console in job-id
order, then reaps. Not just the running ones: a job that finished
before the line was reached still has output nobody has read, and
whether it finished in time is a scheduling accident. Reaping keeps a
second `wait` from reprinting, and matches GNU, where `jobs` prints
nothing once `wait` has returned.
- 18 integ cases in bash/jobs/compound.json covering backgrounded for,
while, if, &&, subshell and brace groups, plus the capture sites that
must not leak (command substitution, pipe stages, redirects) and
stderr routing. Both hosts: 1629 passed, 0 failed.
- Unit coverage for bare `wait` in both languages.
The existing "completed jobs cleared after listing" test now uses
`wait %1`, which waits without reaping, so it still exercises what it
was written for; bare `wait` gets its own case.
* fix(shell): reparse `((` as nested subshells when it is not arithmetic
`((echo a); echo b)` failed with a syntax error. tree-sitter-bash lexes
`((` as the arithmetic opener and the lexer cannot back out, so a
subshell that immediately opens another subshell never parses. Bash
resolves the same ambiguity by trying the arithmetic command first and
reparsing as nested subshells when that fails.
Do the same: on a tree that already has an error, split the `((` openers
sitting inside the error and keep the retry only if it parses cleanly.
Commands that parse today are untouched, so no working command's offsets
move, and the original error is preserved when no reparse helps.
The guard is the subtle part. Splitting only openers inside an ERROR
subtree is NOT sufficient: tree-sitter's error region swallows
neighbouring tokens, so a valid `((i++))` next to a bad opener also
reports as errored, and splitting it silently turns arithmetic into a
subshell running `i++` as a command. That is a wrong parse rather than a
rejected one, which is worse. An arithmetic construct has to close with
`))`, so a line containing no `))` anywhere cannot hold a real
arithmetic opener and splitting it is provably safe; lines that mix both
keep their error.
Covered by unit tests in both languages, including the mixed line that
must stay rejected, plus four integ cases in bash/subshell/nested.json.
Both hosts: 1633 passed, 0 failed.
* fix(shell): judge each `((` opener on its own span, not on the error region
The first cut bailed out whenever the line contained `))` anywhere,
which left `i=1; ((i++)); ((echo x); echo $i)` rejected even though bash
runs it. That guard was sound but far too broad.
Judge each opener separately instead, by parsing its balanced span on
its own: `((i++))` stands alone cleanly and is left untouched, while
`((echo x); echo $i)` does not and gets split. This is what makes a
valid arithmetic command safe when it shares a line with a broken
opener, which scope alone cannot do because tree-sitter's error region
covers both. The span scan skips parens inside quotes and backslash
escapes, so a literal `")"` cannot throw off the depth, and an
unbalanced span is assumed arithmetic and left alone.
Newly working, all matching GNU:
i=1; ((i++)); ((echo x); echo $i) -> x 2
((echo ")"); echo b) -> ) b
((echo a); echo b); ((echo c); echo d) -> a b c d
((echo a) && (echo b)) -> a b
`(((echo d)))` still errors, which matches GNU: bash rejects it too.
Four more integ cases in bash/subshell/nested.json, both hosts 1637
passed 0 failed, plus unit tests in both languages.
* fix: address codex review — wait reaping, control-chunk retention, parse byte offsets
- Targeted `wait %N`/`fg` now reap the job after adopting its output, so a
later bare `wait` cannot replay the same console (GNU bash deletes a job
waited on by id; pinned with docker bash 5.2). Job numbering restarts at 1
once the table empties, matching GNU, so repeated `cmd & wait %1` keeps
working after reaps.
- RAM console stores never trim the terminal CONTROL chunk: evicting it left
wait_finished()/follow() blocked forever under byte budgets smaller than
the outcome payload.
- Python parse: _balanced_end/_is_arithmetic now scan the encoded bytes, so
tree-sitter's byte offsets index correctly when multibyte text precedes an
ambiguous `((` opener (e.g. `echo é; ((echo a); echo b)`).
- TS parse: hoist isArithmetic to module scope (no nested functions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: rename misnamed left_stderr to right_stderr in handle_background
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(shell): kill settles the job instead of joining a runner that may never notice the abort
`kill` aborted the job then joined on the console's finish. But the abort
is only observed where something checks it (the executor between nodes,
and commands that take the signal), so a job sitting inside one long
command never sees it and `kill` waits forever on exactly the runaway job
it is trying to stop. Every existing kill test used a runner that does
observe the abort, so the join looked safe.
kill now settles the job itself — abort, mark KILLED/137, emit `Killed`,
finish the console — and does not join. The console drops emits after the
ending chunk, so a runner still unwinding cannot append past its own
death. Python also gains the `status != RUNNING` guards in `_settle` that
TypeScript already had, so a runner completing in the window after `kill`
(cancel is deferred via call_soon_threadsafe) cannot relabel a killed job.
Regression test added in both languages with a runner that ignores the
abort; it times out on the pre-fix code and passes after.
* fix(shell): settle jobs at teardown, and stop close() stranding readers
Two of the four codex findings on the merge were real.
Teardown settled jobs on main, where kill was sync. This branch made
kill async and teardown dropped to a bare cancel, so a job stayed
RUNNING with no ending chunk and anyone parked on wait_finished waited
forever on a workspace that was already gone. Both close paths are
async, and kill no longer joins the runner, so they await kill_all
instead. It runs before any resource closes, so a job cannot keep
touching one that is already gone. The sync half keeps the bare cancel
as a last resort for a caller with no loop at all.
ConsoleStore.close promised to wake blocked readers, but follow() and
wait_finished() loop: woken, they re-read, find no CONTROL chunk, and
wait again on a console nobody will write to. Stores now carry closed
state, wait() returns immediately once set, and both loops end on it.
Regression tests in both languages, each verified to fail without its
fix. They use an abort-ignoring runner rather than sleep, which is the
one command that consumes the signal and would pass either way, and
they release that runner in a finally so a failure cannot turn into a
hang at loop teardown.
* refactor(io): exit-code reads delegate to the stream source
A streaming command's status can depend on its content (grep's
exit_on_empty settles the origin only at drain time), so merge() no
longer copies the number: the merged result links to its right-hand
original and reads through the link, always fresh. An explicit write
stores locally and severs the link, so aggregated or overridden
statuses still win (issue #43). Deletes sync_exit_code/syncExitCode
and every call site in both languages; the old one-shot sync also
detached on first use, so a sync before the drain froze a stale 0
forever, a hazard class the delegating read removes.
* refactor(console): split config into constants, types and utils
KILLED_OUTCOME moves to constants, Channel/ConsoleChunk/ReadResult to
types (ReadResult joins from store.py, mirroring where TS already kept
it), exit_outcome to utils, per the module-layout convention; config.py
held no configuration. Same split in TypeScript.
* fix(jobs): wait joins the console's ending chunk, not the status field
kill and settle both flip the status before their final appends, so a
status-based return let a waiter snapshot and reap a killed job before
the Killed marker or ending chunk was persisted: a waiter on another
loop today, any store that suspends tomorrow, and in TypeScript any
concurrent waiter, since every await yields a microtask. wait now joins
wait_finished for live jobs (restored jobs keep the no-task fast path),
and wait_all joins every job rather than only the running ones, since
bare wait snapshots killed jobs too. Gated-store regression tests in
both languages, verified failing against the old wait.
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The LLM-facing Notion and Trello mount prompts described trees that no
backend serves, identically in both languages.
Notion omitted the data source level entirely. Since Notion-Version
2025-09-03 a database is a container plus one or more data sources, and
both readdirs have served three levels under `databases/` for a while:
`database.json` plus one dir per data source at depth 2, `data_source.json`
plus row pages at depth 3, row page dirs from depth 4. The prompt still
showed row pages hanging directly off the database.
The prose was wrong the same way, and in a way that would send an agent
looking for a key that is not there: it claimed `database.json` carries
"its typed property schema". It does not. `normalize_database` emits no
`properties` at all -- the schema lives on `normalize_data_source` -- and
`ls` on a database dir lists data sources, not rows.
Trello dropped two directories that readdir has always seeded on every
board: `members/` and `labels/`, each holding one `<name>__<id>.json` per
entry. Only `board.json` and `lists/` were shown.
Both prompts' Notion ntn guidance had also drifted from the CLI, which is
itself gated against real npm ntn@0.21.9 by integ/ntn_conformance.ts. It
advertised `ntn blocks append`, a verb that does not exist, told the agent
to pass ids as `--page`/`--block`/`--datasource`, which are not options,
and used `--json` as a request body when it is a boolean output flag. The
docs already said all of this (docs/*/cli/ntn.mdx: "Ids are positional,
not flags"); only the prompts were stale. They now teach the five real
verbs, positional ids, `pages create --parent/--content`, and blocks,
comments and search through `ntn api`.
The Notion prompts render byte-identical across the two languages; the
ids sentence was previously Python-only.
Docs needed no change -- they were already correct on every count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(cli): one door per state plane for CLI verbs
CLIVerbOpts becomes CLIDoors on inv.doors, carrying dispatch and
stat_path for the data plane, ns for the name plane and session_view
for session state. mount_root is gone: it was the same callable as
ns.mounts.root_of, wired identically at both call sites. Field names
now match CommandOpts, pinned by a meta-test in each language.
Growing the doors closed four gaps in git:
- git add stored a symlink as a regular file holding its target's
bytes; it now writes mode 120000 with the target as the blob.
- a broken symlink was invisible, because the walk followed links
through stat_path; it lstats through ns.links first.
- dispatch("unlink") could not remove a namespace link, so checkout
to a branch without one left it behind.
- commit ignored GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL and EMAIL.
* refactor(workspace): move the record client out of the session package
Sessions, the namespace node table and workspace metadata are three
tables that persist the same way, so the keyed-record clients belong
under workspace/record/ rather than inside one of their consumers.
Before this the namespace and the state store imported upward into the
session package for a client that knows nothing about sessions.
A layering test in each language fails if the record tier grows an
import back into a tier that persists through it.
* fix(shell): route every session write through the policy gate
export and a plain assignment cleared pre_session; the expansion-time
writers did not, so ${X:=d}, $((X=5)), (( )), printf -v and
for ((X=0; ...)) wrote straight to the session env past every policy.
A deployment refusing AWS_* was one ${AWS_PROFILE:=x} away from being
advisory.
All five now write through SessionView.set. The view is threaded into
expansion explicitly and defaults to None, which is correct outside a
workspace. Reads stay sync.
* docs: CLIDoors, the record tier and the session write gate
CLAUDE.md still described CLIVerbOpts with mount_root on inv.ops.
* fix(shell): gate a subscripted printf target, and keep the array
`printf -v 'name[i]'` wrote the session's array table directly, so a
pre_session rule refusing the name never saw it. The door speaks in
whole variables and has since it was written, so the element write now
builds the array on a copy and hands that through view.set, which is
what set_var already asks writers with subscripts to do.
An expansion-time write to a name holding an array was storing a
scalar, which discarded every element after the first. bash writes
element 0 and keeps the tail: `a=(1 2 3)` then `$((a=5))` leaves
`5 2 3`, and `c=("" 9)` then `${c:=x}` leaves `x 9`. Both pinned
against bash 5.2.
* fix(git): replace a link and a file across a checkout
A path that is a symlink on one branch and a regular file on the other
was restored by writing the blob through the link. The dispatcher
follows it, so the content landed in the file the link pointed at,
which no branch touched, and the link stayed in the working tree while
HEAD and the index recorded a file. Linking over a regular file left
that file behind the link the same way.
git replaces the entry in both directions, pinned against git 2.47.
The kind that is there now is a namespace lookup, so a file restored
over a file still costs one write.
* fix(glob): a match named like the glob word is not the nullglob literal
Pathname expansion dropped a real match whose name equalled the word
that globbed for it. With `*a.txt` and `xa.txt` in a directory,
`echo /data/*a.txt` printed only `/data/xa.txt`; GNU bash 5.2.37
(debian:stable-slim) prints both, because the live `*` matches the
literal `*` in the first name.
The merge that unions a backend's matches with the namespace's (nested
mount roots and symlinks, which no backend can see) has to discard the
backend's nullglob-off fallback: a zero-match backend answers with the
literal word, and that is "no match", not an entry to merge against.
It identified the fallback by comparing the returned path to the word.
That comparison cannot work. After quote removal a zero-match glob and
a file genuinely named `*a.txt` are the same string, so the returned
value carries no evidence of which happened, and the real match was
thrown away. A `?` pattern was unaffected only because its word never
equals the name it matches.
resolve_glob_with already answers two different questions depending on
the spec it is handed: a word-shaped spec gets bash's own answer with
the literal reinstated, a directory-shaped one (PathSpec.dir) gets
matches alone. The expander now asks the dir-shaped question, so an
empty list means nothing matched, the comparison is gone, and
resolve_globs stays the only place that reinstates the literal -- which
it has to be, since only it can see whether the backend and the
namespace between them matched anything. That shape contract was an
incidental comment and is now stated in resolve_glob_with /
resolveGlobWith, because the fix rests on it.
TypeScript issued its backend glob call before branching, so a mid-path
word paid for a request whose result was then discarded; python's
if/elif/else never did. The call now lives in the branch that uses it.
_echo_resolve_glob in test_execute_node.py echoed back whatever spec it
was given, commented as matching real resolve_glob behavior. That was
only ever true on the word-shaped zero-match branch, so three tests
asserting the zero-match literal were passing through the unsound
filter. It now answers as a backend holding nothing the pattern
matches. The same latent echo in test_globs.py's default mock is fixed
too; no test exercises it today, but it would mislead the next one.
Pinned against GNU bash 5.2.37 on debian:stable-slim:
echo /data/*a.txt -> /data/*a.txt /data/xa.txt
echo /data/*d/f -> /data/*d/f /data/xd/f
cd /data; echo *a.txt -> *a.txt xa.txt
integ `glob_matched_name_is_the_word` covers it on the backends that
allow `*` in a name; both hosts report 4999 passed, 0 failed on
ram+disk, and the case goes red without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(glob): a match is a child of the directory it was globbed in
Codex review on #790: `resolve_glob` is a public extension point, so the
spec shape the expander sends is not something it can assume a resource
honors. Every in-repo resource funnels through `resolve_glob_with`,
which reinstates a literal only for a word-shaped spec, but a resource
that implements nullglob-off on its own answers a no-match ask with the
spec it was handed -- now the directory. With the word comparison gone,
`/mount/sub/*.nope` could have expanded to `/mount/sub/` and handed a
command the directory.
The merge now keeps only specs strictly under the globbed directory,
which is the guard `_level_matches` already applies for the same reason
one level down. The distinction from the comparison it replaces is the
point: that one tested against the WORD, which a real match can be
spelled exactly like, and was unsound for it. This one tests against the
DIRECTORY, which no match can equal, because a match is strictly longer
than the directory holding it. So the public hook is defended without
reintroducing the ambiguity.
The directory it tests against is the unmarked one #783 already derives
for `_namespace_children`. A match is a real path a backend listed, so a
glob character quoted in the directory's own name is a character of it;
comparing against the marked spelling would reject every match under a
directory like `'/data/*d'/`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`mountForPrefix` throws on a miss and returns a non-nullable MountEntry;
`tryMountForPrefix` is the one that returns null (the #781 mount-lookup
contract). The `?.` here predates that split, so it is unreachable, and
`no-unnecessary-condition` started failing the lint gate when #788
landed -- on main as well as on this branch, so it is not this PR's
change. Unblocking CI here rather than leaving both red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A word-level flag cannot say which metacharacter was quoted, and bash
decides that one occurrence at a time: `'/data/*'?.txt` still globs, on
the `?` alone, against a literal star. With files `*a.txt` and `xa.txt`,
GNU `rm '/data/*'?.txt` removes exactly `*a.txt`; reading the word as a
whole activated both metacharacters and removed both files.
Quoting now rides with the characters. `expand_words` returns the word
with every quoted glob character replaced by its own mark, and the
marks are produced by the expander itself, per node: a quoted node marks
everything it encloses (so `"$p"?.txt` differs from `$p?.txt`), a bare
word marks only what a backslash quoted, and a concatenation just joins
its children. At the matcher, `glob_pattern` hands a marked character
over as its own one-character class, which is what `escape_glob`
already does for quoted text in pattern contexts.
Three properties make this cheap rather than invasive. No mark is a
glob character, so `has_glob` already answers whether a word still
globs -- which retires the separate pattern-stripping pass and the `--`
index alignment it needed, and makes `set -f` just "mark every word".
A mark is one character wide, so every length relation a spec's
virtual/directory/resource_path/raw_path already depended on keeps
holding. And the marks come off at two places only: when resolution
finishes, and at the no-match fallback, where bash's own answer is the
word after quote removal.
Pinned against GNU bash 5.2.37 / coreutils 9.7 on debian:stable-slim,
including the `'*'*.txt` case previously documented as a deliberate
divergence -- bash matches only `*a.txt` there, and so does mirage now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bash applies pathname expansion only to unquoted metacharacters, but
quote removal happened before classification, so classify_word saw the
bare text of '/data/*.txt' and stamped a pattern: chmod 644 '/data/*.txt'
chmodded a.txt and exited 0 where GNU fails with 'cannot access', and
touch '/data/*.txt' touched a.txt instead of creating the literal name.
rm '/data/*.txt' could silently delete every match.
Expansion is the last layer that still sees quoting, so expand_parts
gains a metadata twin expand_words returning ExpandedWord(text,
globbable): a word is globbable only when it carries a glob character
from an unquoted region. A plain word is scanned raw (has_unescaped_glob
honors backslash quoting), a quoted node never contributes, an unquoted
expansion's words are live the way bash treats them, a concatenation is
live when any child is, and a brace template counts an atom only when
its child was an unquoted expansion holding glob characters
(template_globbable). expand_argv and expand_and_classify then reuse the
noglob shape: a non-globbable pattern PathSpec drops its pattern, so
every consumer (shell resolve, backend pushdown, per-builtin re-glob,
for/select word lists) reads the literal name.
Pinned against GNU bash 5.2.37 / coreutils 9.7 on debian:stable-slim:
'...'/"..."/backslash forms all literal; "/data/"*.txt still globs;
unquoted $p globs while "$p" does not; {'*',x} stays literal;
touch/rm address the literal name; for f in '...' iterates once.
Both languages, kept in lockstep: unit tests mirror across
test_parts.py / parts.test.ts, test_brace.py / brace.test.ts,
test_glob_walk.py / glob_walk.test.ts, quoting coverage suites, plus an
integ battery (integ/bash/glob/quoted.json) asserting both hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(shell): ambient session for nested evals, whole-body cmdsub
A re-entrant execute ($(), backticks, eval, source, xargs, timeout,
command, env, process substitution) now continues in the live ambient
session instead of re-resolving an id through the session manager, so
per-call cwd/env forks and restricted named sessions confine every
nested line. Background jobs rebind the ambient session around the
job task, so a nested eval inside '... &' runs in the job's fork and
can no longer read or move the parent session. Command substitution
executes its whole body: the old node filter ran only the first
command node, so $(a; b) ran only a, and declarations, assignments
and control flow ran nothing at all.
Both languages, red-first tests, plus six shared integ cases
(integ/bash/jobs/bg.json, integ/bash/cmdsub/body.json) pinned
against real bash.
* fix(shell): scope the ambient session to its workspace
Codex review on #788.
The ambient session is now owned by the workspace that published it,
so a callback fired mid-line that reaches a SECOND workspace resolves
that workspace's own session instead of adopting this one: a session
carries one workspace's cwd, env and mount grants, and an unrestricted
session must never stand in for a restricted one. A nested bind (a
background job's fork) inherits the owner, so it stays attributed to
the workspace running it.
The TypeScript policy router now decides with the session the line
actually runs, matching Python: it re-resolved the registered session
for cwd and env, so a policy could authorize a relative command
against one directory and have it execute in another. The router no
longer holds a session manager at all.
A background job binds its fork ambiently only where the async context
isolates tasks. The fallback storage a browser gets is one global slot
restored when the scope settles, which would show the job's fork to
the foreground for the job's whole life; there the job's inner evals
resolve by id as they did before.
* fix(links): resolve an operand's link prefix, and honor a trailing slash
POSIX resolves a path one component at a time, and only the last one is
exempt for a command with lstat semantics. mirage resolved either all of
an operand or none of it, so a no-follow command never resolved the
prefix either: `stat /data/dlink/f2` answered "No such file or
directory" and `du` reported 0.
Always resolve the directory prefix; resolve the last component when the
command follows or the operand was typed with a trailing slash, which
POSIX reads as `dlink/.`. A slash then requires a directory, so
`cat reg/` is "Not a directory" rather than the file's contents.
tar is the one command a slash does not reach (GNU strips it and
archives the link); zip is the counter-example and honors it. rm, rmdir,
mv and unlink refuse instead of following, so a slash no longer deletes
the link it was protecting. mkdir now lstats the name it creates, so
`mkdir -p dangle` collides with the link instead of creating its missing
target.
Every expectation is probed against GNU coreutils 9.4 / tar 1.35 on
debian:stable-slim. Python and TypeScript answer identically on all 73
probed cases.
* fix(links): honor a trailing slash on keyed and synthetic backends
A listing never reaches the stat guard, and on a keyed store it cannot
tell "not a directory" from "no keys under this prefix" on its own, so
`ls flink/` answered with an empty listing and exit 0. Guard readdir
too, but only refuse on a stat that actually answers: on a prefix or
synthetic store a directory is the set of keys under it rather than an
object, so a miss is not evidence of a non-directory.
s3 and gridfs ship their own rm and mkdir, so the generic builders'
link handling never ran for them. The refusals move to one shared
helper both tiers call, rather than a second copy per backend.
Drop the anchored-repetition trims CodeQL flags as polynomial (ReDoS)
for the loop-based rstripSlash helper the repo already has.
hf has no rmdir op at all, so the command is not registered on that
mount and answers ENOTSUP before any operand rule applies; its two
rmdir cases are scoped away from it. A keyed store cannot hold an
empty directory either, so the battery no longer asserts one.
* fix(links): validate a removal line before it drops a link, and refuse a slashed mv source
The dispatcher removes a symlink operand from the namespace before the
command layer parses, so a line that layer rejects has to leave the link
alone: GNU answers `unlink a b` with "extra operand" and `rm --bogus x`
with "unrecognized option", both with everything still in place.
mv gets the same treatment for a link typed with a trailing slash.
rename(2) never follows, so POSIX's `dlink/` == `dlink/.` asks for a
directory the call will not resolve, and GNU refuses in four wordings
depending on what the link leads to and what the destination is.
Also pass the index through the slash guard's readdir probe: a synthetic
backend resolves a path through it and cannot stat without one, so chroma
answered "missing index" for `ls /knowledge/`.
All wordings pinned against GNU coreutils 9.7 on debian:stable-slim.
* feat(dsh): ship the package as a dsh bundle with declarative mounts
* fix(dsh): claim workspace-write only for a confined runtime world, recheck aborts after the ready wait
* fix(runtime): declare confined on the monty test double
* refactor: one mount-lookup contract — mount_for raises, try_mount_for returns None (py+ts)
Cleanup-plan item 23 (T1-K). Python's mount_for raised bare ValueError
while TypeScript's mountFor returned null, so every call site coped its
own way and the two languages drifted. Both registries now expose the
same explicit pair:
- mount_for / mountFor: the path must be mounted; a miss raises the
typed NoMountError (py) / noMount-stamped Error (ts). Never caught —
callers with a fallback use the try variant instead.
- try_mount_for / tryMountFor (+ _prefix twins): None/null on a miss.
All ~70 call sites swept with per-site intent: expected-miss sites
(classify, globs, storage keys, drift, provision, cross-mount scopes,
history, df, man, find actions, cache gates) take the try variant with
their existing fallbacks; post-routing invariant sites (metadata
builtins, expand_operands, spec_hints, watch) keep the raising lookup.
Blanket `except ValueError` catches around lookups are gone, so backend
ValueErrors can no longer be swallowed as "no mount".
Cross-language divergences fixed along the contract:
- TS groupByMount silently dropped unmounted paths; it now propagates
like Python.
- Python mount_for_command could route a command to the /dev/ mount;
it now skips DEV_PREFIX like TypeScript.
- TS applyStateDict resolved snapshot prefixes by longest-prefix match,
so a stale nested-mount prefix would load resource state into the
ancestor mount; it now uses the exact-prefix lookup like Python.
- TS specForCommand masked a broken-registry miss with ?? null; it now
propagates like Python's documented behavior.
- TS Workspace.mount() returned null where Python raised; both raise.
- TS watch layer's private throwing wrapper is gone; the WatchRegistry
interface carries the raising contract directly.
- Deleted the duplicated hasGlob(r) method probes (metadata.ts,
expand/globs.ts) in favor of the plain resource.glob !== undefined
narrowing mount.ts already used.
Verified no user-visible behavior change: an 18-case unmounted-path
probe (chmod/touch/chown/chgrp/cat/ls/stat/cd/rm/cp/du/mkdir/redirect)
is byte-identical before/after on both languages and matches GNU
(docker debian:stable-slim). New integ file integ/unix/meta/
unmounted.json pins the GNU-exact errors on 22 targets; ram battery
2438/2438 on both hosts.
Related #74 (the /dev/ skip removes one wrong-pick class; the
first-wins ambiguity remains) and #30 (contract groundwork; cross-mount
command alignment still open).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: normalize the prefix argument in py try_mount_for_prefix (codex P2)
The new try variant inherited the old exact-string compare, so the
registration spelling ("/data", "data/") missed the stored "/data/"
entry only in Python; the TS twin normalizes. Normalize with the same
idiom mount/unmount use, delegate is_mount_root through it like the TS
registry does, and pin the spelling set in both languages' tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* fix(shell): don't unescape an already-expanded word in classification
classify_word ran a second POSIX quote-removal pass over words the
expansion layer had already unquoted, so a backslash that survived
expansion (a literal name character) was stripped again. printf's own
format arg is path-shaped, so `printf '/data/x\ty'` lost the tab before
printf ran, and `ln -s "$(...)"` stored a target with no tab.
Delete _unescape_path / unescapePath and both call sites in each
language. The relative-branch call site was unreachable: _RELATIVE_PATH
admits no backslash.
* test(integ): give the symv directory a real file so hf can remove it
hf's mkdir is a no-op, so a directory holding only symlinks (namespace
state, never backend keys) has no backend existence and rm -r reports
ENOENT. Write a file at setup the way sym/setup.json does; rm -r takes
it along with the directory.
* docs(hf): say why hf mkdir cannot write a directory marker
The comment claimed object stores have no real directories, but s3 and
gridfs are object stores and both write a zero-byte `key/` marker. The
real reason is the client: OpenDAL's hf service reports create_dir
unsupported and rejects a write to a slash-terminated path, both
client-side, in the python and node bindings alike.
* feat(session): per-session hidden paths and vars via SessionProfile
* fix(session): close the hidden-var write doors, HOME channel, hydration, and native-find bypass
* fix(session): filter hidden arrays from expansion and gate the raw env writers
GNU spells find's negation two ways and mirage only understood one.
`find /data -name '*.txt' ! -empty` exited 1 with `unknown predicate
'/!'`, because find's spec declares `rest=Operand(type="path")`, so
every word without a leading dash is classified as a start point and
`!` was cwd-prefixed into the path `/!`. `-not` worked, and so did
`\( ... \)`, because the parens were already exempted through
`ignore_tokens`; `!` was simply missing from that set. Nothing
downstream needed teaching -- `!` has always been in the expression
parser's operator table.
Fixing the classification made a second bug reachable: a line that
ends on an operator. `find /data !` used to print the whole tree and
then a bogus "No such file or directory" for `/!`; with the operand
gone it would have negated nothing in silence. Both parsers now refuse
it in GNU's words, and the check runs where the operator is consumed
rather than in the primary, because by then the token that needed a
right-hand side is no longer on the line:
find: expected an expression after '!'
find: expected an expression between '!' and ')'
Two py/ts divergences surfaced on the way and are closed here:
- The TypeScript parser skipped an ignore token silently, leaving its
`wordKinds` slot null, which means "guess from the shape" -- and the
shape of a grammar token says nothing about it. Python stated TEXT.
TypeScript states it now too.
- `spec_hints.ts` carried a by-value override that re-nulled ignore
tokens after the fact. It was redundant once the parser states the
kind, and it was position-blind: it matched an option's value as
readily as a grammar token, so `find /d -name '!'` lost the TEXT the
parser had correctly given the pattern. Deleted.
The new cases also caught a live bug in the Nextcloud backend, fixed
here. Its find pushes the predicate tree down as a WebDAV SEARCH, and
Nextcloud inverts a <d:not> by swapping the comparison inside it for
its opposite, so a <d:not> around anything but a single comparison
raises server-side:
Binary operators inside "not" is not supported (HTTP 500)
Probed against nextcloud:30-apache (30.0.17), the image CI runs:
not(like) and not(is-collection) answer 207, while not(or(..)),
not(and(..)) and not(not(..)) all 500. The compiler now tracks whether
a compiled condition is a bare comparison and refuses to negate one
that is not, which falls that find back to the scan walk, which has no
such limit. `-not` reached this first and has been answering 500 all
along: `find /data -not -type f` compiles to not(not(is-collection)),
unnoticed because find_not_name was the battery's only negation case
and negates a bare -name. find_bang_type_f now pins it.
Pinned against GNU findutils 4.10.0 on debian:stable-slim as a
three-way byte-identical differential (GNU vs python vs typescript)
over 25 output cases and 15 error cases on one shared seed tree. One
known divergence is normalized and commented: GNU quotes as `x' where
mirage quotes 'x' repo-wide, which is pre-existing and already pinned
in integ/unix/find/error.json. Also left alone as pre-existing: GNU's
wording for a binary operator with nothing before it, the
paren-specific messages, and "paths must precede expression".
integ gains five output cases in find/not.json and three error cases
in find/error.json. The output cases carry seq 295, the same seq as
find_not_name whose expected output they derive from: the battery runs
one shared session ordered by seq, and every one of the eight fixtures
that would otherwise intrude (/data/fs at 302, /data/dr at 317,
/data/rgm at 459, /data/arch at 467, /data/xm2ctree at 755, /data/g4
at 500200, /data/f4 at 500300, /data/roots1 at 785) is created after
295 and never cleaned up. The one case already at 295,
find_empty_dir_holding_only_a_link, removes its own tree in the same
line, which is what leaves the slot usable.
chroma and github each gain the `!` twin of the `-not` case added in
#772, placed immediately after it in both file order and seq
(chroma_find_empty_composes_with_name -> 630056,
github_find_empty_composes_with_name -> 530014). Both verified against
a locally run fake on both hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(stat): render the symlink arrow for %N
stat -c '%N' on a symlink printed only the quoted name. GNU prints
'name' -> 'target'. The target was already on FileStat.extra under
LINK_TARGET_KEY, which is what ls -l and file read, so %N now reads
the same fact.
Pinned against GNU coreutils 9.7 on debian:stable-slim, which showed
two more rules: any format modifier drops the quoting (only a bare
%N quotes), and a width or precision applies to the name and the
target separately rather than to the joined line. That second rule
had to be implemented either way to render %20N on a link, and it
also fixes %30N on a plain file, which used to keep the quotes.
Tests in both languages plus six integ cases across all 22 targets.
* fix(stat): quote %N the way gnulib shell-escape does
The helper picked double quotes for any name holding an apostrophe, so
a'b$c rendered as "a'b$c" and replaying the advertised shell-safe output
expanded $c. It also passed control characters through raw.
Probed GNU 9.7 across ASCII 32-126 and the C0 range: single quotes are
the default with '\'' escaping, double quotes are an optimization for a
name whose only awkward character is an apostrophe, and a control
character becomes a $'..' segment spliced between single-quoted runs.
Now applies to a symlink target too, since %N quotes each field on its
own.
* fix(glob): let glob expansion see nested mount roots and symlinks
Glob expansion enumerated candidates from a single backend's readdir.
A nested mount's keys live in another resource and no resource stores a
symlink, so neither was ever offered as a candidate, even though both
already show up in a plain listing via merge_readdir.
Both glob tiers now merge the same namespace_names union a listing uses,
session-filtered, and matches sort like bash. A glob whose directory
holds a child mount is expanded before routing so its matches route per
mount, and that happens before the follow policy so a matched symlink is
followed exactly as a typed one is.
Pinned against GNU coreutils 9.7 on debian:stable-slim with a tmpfs at
base/inner and the same symlink.
* fix(glob): expand boundary globs before admission, follow linked dirs
Codex review on #775.
Run the boundary expansion before pre_command, not just before the
follow policy: a word left unexpanded reaches MountRootPolicy as the
literal pattern, so `tar -cf out.tar /base/*` archived a mount root the
same operand typed by hand is refused for. cp -r and zip too.
Compare operands as words rather than by count, so a glob matching
exactly one name still installs. `du /base/i*` failed while
`du /base/inner` worked.
Resolve a glob's parent directory through the namespace link table
before a backend lists it, spelling matches back under the typed name.
Pinned against GNU bash on debian:stable-slim: `echo base/d*/f2` ->
`base/dlink/f2`, `echo base/*/f2` -> `base/dlink/f2 base/sub/f2`.
Drop a walk match that is the listed directory itself. A backend asked
to list a path that is really a file answers with that file, which
walked back out as a doubled segment (`/base/f*/f1` -> `/base/base/f1`)
where bash keeps the literal.
* test(integ): cover glob expansion across mount and link boundaries
Seven cases on ram-nested for the boundary glob rules, and one on
ram/disk for a glob whose parent is a symlinked directory. Five of the
eight fail against the previous commit, one per fix.
Expectations are pinned against GNU on debian:stable-slim with a tmpfs
at data/inner, except the mount-root refusal, which is mirage's own
documented divergence.
* feat(dsh): DeepSeek Harness fs and shell providers over a mirage workspace
* docs(dsh): monty python run and multi-source mounts in the dsh example and page
* docs(dsh): tidy the example into sectioned helpers
* docs(dsh): trim the page to install, compose, python, one world
* docs(dsh): use mirage as the example report owner
* docs(dsh): slack-hosted python script example
* docs(dsh): rename the slack-hosted script to example.py
* fix(dsh): match Slack upload names and newest-first date dirs in the example
* fix(dsh): address codex review on lock retention, pre-aborted signals, cyclic links, and byte limits
* docs(dsh): generalize the page, compose sample on redis + ram
* docs(dsh): keep slack in the compose sample
* docs(dsh): state the python3 capture forms
* docs(dsh): show the runtimes capture spec with an inline python sample
* docs(dsh): monty is the workspace python, capturing python and python3
* docs(dsh): one continuous example, no second workspace
* docs(dsh): show monty explicitly capturing python and python3
* docs(dsh): one example end to end — ram + redis + slack, monty capturing python
* docs(dsh): drop the one execution world section
* docs(dsh): state the in-memory property with measured capacity
* docs(dsh): drop the capacity numbers
* docs(dsh): drop the in-memory paragraph
* docs(dsh): add a harness comparison table
* docs(dsh): add concurrency column on a 2 vcpu / 8 gb server
* docs(dsh): correct the live python engine footprint to 27 MB
* docs(dsh): no em dashes, dsh row name, serving scenario before the table
* docs(dsh): tmp mount, redis-bound report, numeric concurrency table with startup
* docs(dsh): note the sandbox provisioning cost the cli sessions would add
* docs(dsh): sandbox startup cost in the cli startup cells
`du /empty` reported `cannot access '/empty': No such file or directory`
and exited 1 for the implied parent of a nested mount, even though it
printed the right rows. `/empty` is a real directory, but no backend
holds an entry for it: the content lives in the descendant mount's own
resource.
du ran bound to one mount and decided existence from that mount's stat,
so it could not see either fact that makes such a path a directory: a
mount nested below it, or a symlink below it. Both are namespace state.
The dispatcher already answers correctly for them (namespace_stat behind
the backend's miss), and that answer is offered to every handler as
`stat_path`; `find` was the only consumer, which is why find was the one
sibling that got this right.
du_operands now asks that probe when the backend stat fails, ahead of the
has_content walk, which is both cheaper and authoritative. has_content
stays behind it for a backend with no directory entry for its own mount
root.
stat_path rather than registry.descendant_mounts on purpose: the registry
lookup is not session-filtered, so proving presence from the mount table
would answer `0 /empty` for a session that may not see /empty/hole and
confirm a walled-off mount's parent. It also misses the link-implied
case, which has no descendant mount at all. Both are pinned by tests.
Pinned against GNU coreutils 9.7 on debian:stable-slim with a tmpfs
mounted at /empty/hole: both rows, no diagnostic, exit 0.
* feat(du): honor -S/--separate-dirs (py + ts)
GNU -S makes each directory total count only files that sit directly in
it, leaving subdirectory sizes out of the parent. Shared rollup and
summarize paths now match coreutils 9.7, with unit and integ coverage.
* fix(du): apply -S across nested mounts, keep the -c total recursive
The traversal fan-out re-derives du's tree centrally from the per-mount
blocks, so -S was silently dropped there: du -S over a mount printed the
recursive total. Teach the merge about it, and split the operand's own
row from what it contributes to -c, which GNU keeps recursive.
Add a guard that fails when a new du flag is neither applied centrally
nor classified as per-run, which is how -S went missing.
* chore(spec): regenerate du spec json for -S
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* feat(workspace): state doors and per-plane views
* fix(workspace): one general session door for every variable write
Codex round: shell readlink dispatches through the op door; array-shaped
assignments and subscripted unsets can no longer bypass pre_session.
SessionView.set is now general over variable shapes (str | ShellArray),
pre_session_gate is private to the door and takes a SessionContext that
carries session_id, staged declaration arrays store through the builtins,
for/select loop variables write through the view, and readonly pre-checks
are separated from policy-denial catches at every shell site. Also fixes
export ARR=(x) printing the export listing and CI formatter drift.
* test(integ): pin the state-door fixes; readonly declaration arrays abandon the line
* ci(integ): key the WinFsp install on the DLL, not choco's exit code
* wip(seam): py dispatcher builds CommandOpts; handlers take (accessor, paths, texts, opts); adapter op protocols; DispatchFn typing
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(seam): dispatcher-built CommandOpts, typed adapter ops, named ParsedCommand, one DispatchFn home (#609 items 26+23)
Item 26 (T2-8 seam typing), all four pieces, plus task 23's static
FlagView query-name gate:
- The python dispatcher (Mount.execute_cmd) now constructs one
CommandOpts per invocation and calls every handler as
fn(accessor, paths, texts, opts) — the TS convention. Flags stop
sharing a bag with injected context, accepts_kwarg opt-in dies, and
all 72 builders, 75 bespoke wrappers and ~30 provision functions
become pure wiring; the provision path builds the same bag with
command/spec set. Generics that took trailing fact params (ls, find,
tree, zip, tar, du, file, stat) now read opts.links/opts.mounts/...
like their TS twins.
- adapter.py gains per-slot op protocols (ReaddirOp/StatOp/WriteOp/...)
mirroring adapter.ts, so wiring readdir where stat belongs no longer
type-checks; Builder.fn/provision and CommandIO fields are typed.
Surfaced and fixed real drift: hf/databricks mkdir had index before
parents (a positional parents call would land in index), wget -O
dropped its PathSpec value under as_str.
- TS parseFlags returns a named ParsedCommand (twin of the py
NamedTuple) instead of a 15-slot positional tuple; optionError takes
(cmdName, parsed) like python's option_error.
- DispatchFn moves to runtime/types.ts (python's home for the same
protocol); the crossmount re-export inversion and the CommandDispatch
duplicate are gone; 38 python files stop spelling it
Callable[..., Any]. The dead CommandOpts.resource field (zero readers)
is deleted along with the CrossResourceStub that existed to fill it.
- Task 23: tests/commands/test_flag_query_names.py and
commands/flag_query_names.test.ts fail on a FlagView query naming a
dest no spec bound in the module declares, without needing the code
path to run (both verified to fire on a planted typo).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(examples): redis example calls provisions with the 4-positional shape
file_read_provision/head_tail_provision/metadata_provision are
(accessor, paths, texts, opts) now; the direct demo calls still passed
the old command= kwarg, crashing the example before the persistence
section (CI examples gate).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): pin the #772 fixes where the harness can reach them
- find -empty on chroma + github (the wrappers used to drop the flag,
which would flood every path; -empty now rides find_generic), plus a
-not -empty composition case on each so the pin has positive output
- mkdir -p gains databricks/databricks-prefix (zero prior mkdir
coverage on that backend; the param-order bug itself is only
reachable positionally, so MkdirOp+mypy is its real gate)
- wget -O / curl -o were already pinned by integ/resources/http
Both hosts green: py chroma+github 83 ok, ts all five targets 4070 ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(ts): derive config types from their zod schemas
Every resource and account config declared its shape twice, once as an
interface and once as the zod schema that actually validates it, plus a
redacted twin that re-listed every field by hand. Two type helpers in
resource/secrets.ts replace both: ConfigOf reads the shape off the schema,
and RedactedConfig builds the twin from the config plus the names of its
secret fields.
ConfigOf is not a bare z.infer: that renders an optional field as
`k?: T | undefined`, which under exactOptionalPropertyTypes is wider than
the `k?: T` these configs were written with and stops an S3 alias's state
from matching its base's.
The twin is reached through an `as unknown as` cast, so nothing checked it
and it had drifted. gcs, oci, r2, supabase, wasabi, hf_buckets, node's s3
and gridfs all typed redacted credentials as plain strings, and box dropped
a field the schema carries.
Ten files keep a hand-written config type because their schema is not the
config's own shape, each saying so in place: s3 has a profile the schema
lacks, google/box/dropbox omit callbacks no snapshot can carry, and the
S3-alias family and hf_buckets describe the resolved config the redactor
fills in.
Browser trello and linear declared readonly arrays their schemas did not,
which is the only runtime change here.
* fix(ts): keep readonly arrays on core linear and node trello configs
The old interfaces declared teamIds/boardIds as readonly string[], and
deriving from the schema narrowed both to string[], so an as const value
a consumer used to pass no longer typechecks. Mirror the .readonly() the
browser twins already carry.
The whole sweep removed exactly four readonly array fields, in browser
linear/trello and these two; the browser pair was already fixed.
Two CI-only gate failures the local pre-commit hooks never run:
- check_layout_parity --strict is a two-way ratchet, and pairing
commands/builtin/utils/operands.py with its TS twin closed one
divergence (305 -> 304); lower the committed baseline to lock the
improvement in.
- The directory-list checksum pin assumed `mkdir -p` leaves a statable
empty directory, which huggingface's keyed backend does not have (no
marker object, nothing in the parent listing), so hf/hf-prefix
reported ENOENT instead of GNU's "read error". Narrow the case's
targets, matching the existing unlink_dir precedent; the
missing-list pin stays on hf, where it passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(integ/github): repo, contents, issues and compare routes
The fake served the read path the `github` resource mounts -- repo
metadata, git/trees, git/blobs -- plus the Actions dataset `github_ci`
needs. A task that *acts* on a repository had nothing to call.
Adds the routes such a task uses: GET /user; POST /user/repos; DELETE
/repos/{owner}/{repo}; POST .../forks; GET .../branches and
.../branches/{branch}; GET .../commits; GET and PUT .../contents/{path};
GET and POST .../issues; GET .../compare/{basehead}.
Writes are visible to the next read, which is the point -- an issue
filed shows up in the listing, a committed file reads back with its new
bytes and a new blob sha, and the commit is recorded so `compare`
can answer which files moved. `FakeRepo` grows `issues` and `commits`
for that; a repository with no writes yet still reports one synthetic
root commit, so "the latest commit" is answerable before the agent has
done anything.
Three behaviours worth pinning rather than leaving to chance:
- A fork deep-copies. A task that forks an archive repo per run and then
commits to the fork must not write through to the source.
- PUT /contents enforces GitHub's sha rule in both directions: replacing
an existing file without the current blob sha is 409, and supplying
one for a file that does not exist is 422. A task that reads before
writing is doing so for this reason.
- A compare against a base the repository has never seen is 404, not an
empty file list. Answering "nothing changed" to a question about an
unrelated commit is the shape of wrongness that reads as success.
`_commit_list` is newest-first, so `compare` collects the commits
*before* it reaches the base; walking past the base instead would report
the commits the base already contains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(integ/github): git data API, rename, raw content and GHES paths
The write half a task that builds a repository needs, plus the two shapes
a real client insists on:
- PATCH /repos/{owner}/{repo} renames, carrying the content with it, and
POST /forks takes a name so a fork can be named in one step.
- The git data API -- get commit, create tree, create commit, move ref --
so a multi-file push lands as one commit. A staged tree is invisible
until a ref points at it, which is what git does and what keeps an
abandoned tree from showing up in a read.
- GET /git/ref/{ref}, and trees by commit sha, because a client resolves
a ref to a commit before reading anything.
- /raw/{owner}/{repo}/{ref}/{path}, with a text content type for text, so
a reader does not base64 the whole repository.
- Every route is served under /api/v3 as well as at the root: a client
pointed at a host that is not github.com talks Enterprise.
- GET /search/repositories, and an unrouted fallback that names the
method and path on stderr. Both exist for the same reason: a 404 is an
answer to a real client, so an unimplemented endpoint reads as a
negative result rather than as an error.
- Seeding takes a per-repo default branch, since a template whose branch
is master is graded at that ref by name.
* feat(gh): a GitHub CLI, and the write routes the fake needs behind it
The `github` mount is the read half -- a repository is a tree, so listing
and reading it is `ls` and `cat`. There was no write half at all:
GITHUB_IO carries readdir/read/stat where GDRIVE_IO carries the full set,
so nothing could commit a file, and forking and renaming are account
operations a filesystem has no shape for in any case.
`gh` is that half, spelled as cli.github.com spells it:
gh repo view [OWNER/REPO]
gh repo fork OWNER/REPO [--fork-name NAME]
gh repo rename NEW-NAME -R OWNER/REPO
gh api ENDPOINT [-X METHOD] [-f key=value] [-F key=value]
`rename` takes the new name as the operand and the repository as -R, which
is the reverse of what the shape of the line suggests and is upstream's.
`-f` sends a string and `-F` reads true/false/null/integers as their JSON
types, which is gh's own --raw-field / --field split; a call with no fields
is a GET, one with fields a POST unless -X says otherwise. Committing a
file is `gh api ... -X PUT -f content="$(base64 -w0 f)"`, which is what
real gh and real GitHub require of each other -- there is no file verb to
mimic.
The transport was GET-only, so it grows a `request`, and an empty body (204,
an empty 202) decodes to null rather than throwing on a call that worked.
Fake, in support: DELETE /contents with the same sha rule as the replace,
and GET /commits/{ref}, which is not /git/commits/{sha} -- it takes a branch
name and reports the file list. Both were found by the unrouted logger
during a run: an agent issued 35 deletes, every one 404'd, and it reported
the files as removed, because to a real client a 404 is an answer.
* feat(integ/github): repository metadata, star-sorted search and readme
A task that picks *between* repositories reads their descriptions and star
counts and never opens one, so seeding gains a second mode: --metadata takes
a JSON file keyed by owner/name and creates repositories that are metadata
only. The tree and contents endpoints answer 404 for them, which is what
GitHub says about a repository with no files.
search/repositories honours sort=stars and matches the description as well
as the name, since a repository is found by what it says it does at least as
often as by what it is called. Terms OR rather than AND, which is looser
than GitHub and errs towards showing a caller the row it wants.
GET /repos/{owner}/{repo}/readme, found by the unrouted logger during a run.
* feat(integ/github): branches
The fake kept exactly one branch, which rules out every task whose premise
is a difference between two of them. FakeRepo now holds a file map and a
commit list per branch:
- `files` and `commits` stay bound to the default branch, so every route
that does not name a ref reads what it always read.
- `branch_for` resolves a ref -- a branch name, HEAD, or a commit sha
belonging to one branch's history -- and `tree_of` returns that
branch's files, so a bad ref is a 404 rather than a silent read of the
default branch.
- contents, tree, raw, readme, branches, branch, commits, commit and
git/ref are ref-aware; PUT and DELETE /contents write to the branch
their body names; PATCH /git/refs moves the branch it names.
- Each branch has its own synthetic root commit, so two branches of a
fixture do not share a head.
- A fork copies every branch, not just the default one.
- Seeding takes `into=<branch>`, so a two-branch fixture is two
directories and two --repo flags naming one repository.
The default-branch binding is what keeps this small: 18 call sites read
`repo.files` and none of them had to change.
* feat(integ/github): GET /git/refs/{prefix}
The plural is a different endpoint from the singular: git/ref/<full-ref>
returns one object, git/refs/<prefix> a list of everything beneath it. A
caller picks whichever it expects, so serving only the singular made the
plural read as 'no such ref'. Found by the unrouted logger during a run.
* feat(integ/github): account repo listing, bare /contents, mirror-stable shas
Three things a task that reasons across several repositories needs.
GET /users/{login}/repos and GET /user/repos. A task that asks "is there a
repository for this on my GitHub" answers it here, and a 404 read as "the
account has none" -- a wrong answer rather than an error, which is the
failure mode the unrouted logger exists for.
GET /repos/{owner}/{repo}/contents, with no trailing slash. GitHub serves
both spellings of the root and a caller picks either; only one was routed.
A branch's root commit sha is now derived from its content rather than from
the repository's name. That is what `git clone --mirror` followed by a push
actually gives you -- the same shas -- and a grader that reads an initial
sha from an upstream repository and the latest from a local copy, then asks
what changed between them, depends on it. Two branches still differ exactly
when their trees differ.
* feat(integ/github): POST /git/refs, so a branch can be created
The last write a task can ask for that the fake had no answer to. A new
branch starts as a copy of whatever the base sha resolves to -- which is
what a branch is, another name for one commit and everything reachable from
it -- with its own empty history, so the two do not share future commits.
422 on a ref that is not under refs/heads/, on a name that already exists,
and on a base sha that resolves to nothing.
* feat(integ/github): GET /, the API root
github.com answers the root with a map of endpoint URL templates. Serving
only the endpoints beneath it meant a client probing the root got a 404,
which reads as the host not being there at all rather than as an unfamiliar
API. Found by the unrouted logger during a run.
* feat(integ/github): seed a branch's commit history, not just its tree
Seeding a directory filled a tree and nothing else, so every seeded
branch answered GET /commits with one synthetic root. That is enough for
a task that reads files and wrong for one that asks when something
arrived. `task-tracker` wants the tasks added by the *most recent*
commit on each of fifteen developer branches, and each branch has five
or six -- so a history of one does not make the task hard, it makes it
unanswerable while looking answerable.
--commits owner/name=<file> reads a manifest keyed by branch, each an
array oldest first, and is applied after every tree: a manifest names
branches, and a branch no directory was seeded into would otherwise be
created here as an empty tree with a history, which reads as a branch
whose files were all deleted.
Shas are derived from full_name, branch, date and message rather than
carried, for the same reason the root's is derived from its tree: a
fixture has to answer the same way twice.
Commits now carry commit.author/committer (name, email, date) and
author.login, which is what a client sorting or filtering by date reads,
and the synthetic root carries them too against a fixed epoch -- it
stands for everything before the fixture rather than for a moment, so
"now" would make it sort ahead of the commits it precedes.
GET /commits/{ref} now renders `files` as {filename, status} objects the
way GitHub does. The list endpoint keeps paths, which is also GitHub's
split: `files` is absent from a commit in a list and detailed when one
commit is asked for. _commit_files is shared with the compare route,
which was building the same objects inline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(github): back the transport with octokit
Wraps @octokit/core behind the unchanged GitHubTransport, so the github
mount and the gh CLI both gain retry and throttling. Handles three
octokit behaviours: {} in a url is a route template that silently eats
the segment, the "METHOD /path" form splits on whitespace, and 204
decodes to '' rather than null. The write throttle is github.com's own
secondary rate limit, so it is off for any other host.
* fix(gh): name the mount its writes invalidate, and parse the host segment
serves was empty, so the executor's post-write cache drop did nothing and
a committed file still read back as its pre-write bytes. parseRepo took
the first two segments of [HOST/]OWNER/REPO, so github.com/acme/tools
resolved to github.com/acme -- a different repository, reported as
success.
* fix(cache): invalidate the index instead of clearing it
A cleared index reads exactly like one that was never filled, so github --
whose index is the whole listing rather than a cache in front of one --
could not tell a cache drop from an empty repository, and ls reported the
mount root missing. IndexCacheStore grows invalidate(), which expires
entries in place, and github refetches once on an EXPIRED lookup. Also
fixes a latent bug with no CLI involved: after the 24h index TTL lapsed a
github mount answered ls with exit 0 and no output. Redis cannot express
stale (an expired key is gone), so it still clears; the comment says why.
* feat(gh): a python gh CLI, mirroring the typescript one
Same four verbs and the same grammar. The python client was GET-only, so
it grows github_request alongside github_get. Layout divergence 305 -> 303.
* test(integ): a gh CLI battery on both hosts
15 cases on a new cli-gh target, covering the write-then-read path through
the mount, the refusals, and the reversed rename grammar. The fake grows
POST /reset, because the cli facet runs both hosts against one process and
writes would otherwise leak between them; it also stops rejecting wrapped
base64, which real GitHub accepts. The unrouted-request logger is now a CI
failure rather than a log line.
* fix(github): probe the parent listing, so an update is not read back stale
The index tracks freshness per directory, never per entry, so get()
never answers EXPIRED and the refill it guarded was unreachable: after a
write invalidated the index the blob's row survived carrying the
pre-write sha, and the read served the old bytes. Create and delete went
through readdir and worked; an update did not.
Readdir on the TypeScript side refilled on any miss, which spent a full
recursive-tree call on every ENOENT and diverged from python. Both now
refill only on EXPIRED.
* test(cache): pin what invalidate keeps that clear discards
A cleared store reads exactly like one that was never filled, which is
what made a github mount report an empty repository. Assert the two
answer differently.
* fix(spec): render an operand's declared name outside the clap dialect
A named slot printed as <text>, so 'man gh api' said <text> where gh
says <endpoint>. argparse prints the dest too. gh is the only spec this
changes; ntn is clap and already rendered it.
* test(integ): cover the gh surface the battery missed
Update-then-read is the case that found the stale-read bug. Also the
help and discovery surface, a leading-slash endpoint, typed fields on a
GET, and a missing endpoint.
* docs(gh): a page per host for the gh CLI
Install, the mount-reads/CLI-acts split, the api rules, and a table of
what differs from gh 2.85.
* fix(github): keep an api field from steering the request
Octokit reads loose parameters off the same object that carries url,
method and headers, so `gh api X -f url=...` retargeted the call
instead of sending the field. The query is spelled into the url and the
body travels as `data`. The python client already separated them.
* fix(cli): let a handler say it did not write
A leaf declares `write` statically because for almost every verb it is
static, but `gh api` carries its method on the line, so a read like
`gh api /user` expired every github mount the install serves. A result
may now report what the spec cannot know.
* feat(gh): expand the owner, repo and branch placeholders
gh's own examples are written with them, so `gh api
repos/{owner}/{repo}/releases` asked for literal braces and 404'd. An
install's repo and branch stand in for the current checkout. Any other
brace pair still reaches the wire, which is gh's behavior too.
* feat(gh): render repo view the way gh renders it
A name line, a description line, then the README, with the separator
omitted when there is none. Probed against gh 2.85. The REST object is
still one `gh api repos/OWNER/REPO` away.
* test(integ): cover placeholders and the new repo view
Also carries a branch on the gh install, which is what {branch}
expands from.
* fix(gh): declare branch on the GhConfig type, and type the test trees
tsc runs in CI but not in pre-commit: GhConfig's zod schema grew
`branch` while its interface did not, and the new github tests built
tree items with a `mode` key the type has no room for.
* refactor(gh): derive GhConfig from its schema instead of declaring it twice
The zod schema is the one doing real work: it validates an install's
config and carries the secretStr marker redaction reads. A hand-written
interface beside it only adds a shape that can drift, which is how
branch reached the schema and not the type.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GNU 9.7 pinned on debian:stable-slim: each operand is its own checksum
list, verified in order; an unreadable one reports its strerror and the
run continues with exit 1, a directory operand reports the literal
"read error" (fopen succeeds, the read fails), and --status keeps the
strerror lines while silencing OK/summaries. The defect predated this
PR in the TS generic (paths[0] since #731), so both sides are fixed
together: py hashsum loops with WALK_ERRORS translation, TS
checksumGeneric loops over checkFile with the new shared isEisdir
(twin of isEnoent), and the five py checksum builders bind
dir_aware_stream like their TS twins so implicit keyed-backend
directories classify as EISDIR on the check path. Mirrored unit tests
plus two unix/sha256sum integ pins (seq 960093/960094), verified on
both hosts over ram+disk (4823/4823 each).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cat/head/tail/wc x postgres+mongodb mixed-operand pins, history tail
cross-mount continue, and a postgres directory-operand EISDIR pin that
exercises the implicit-dir parent-listing probe end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A lazy async-generator wrapper deferred the factory reader call to
drain time, past the command's cache-manager contextvar scope, so a
warm S3 LAZY cat re-read the backend instead of serving the cache.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
postgres/mongodb mixed good+missing operand facets, history cross-mount
continue, test_builders_declare_only_dispatcher_params ratchet; history
ls/tree/stat/find and dify find/cat wrappers ride the generics.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
26 builders forward **flags wholesale; each generic parses once into a
frozen struct via a spec-bound FlagView (ls/du/find/sed/tar/tree + 20
mechanical). rm/rmdir/ln/touch are builder-is-implementation and read
the bag through FlagView inline, mirroring the TS builders.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
operands.py + dir-aware stat/stream seam; cat/head/tail/wc/checksum5/
cut/expand/unexpand/fmt/fold/nl/rev/tac/strings/md5/zcat generics own
split_readable + flags; 16 builders and 12 bespoke wrappers become wiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(runtime): name the default python engine (#609 item 19 / T1-N)
The default world was an opaque positional triple on both sides
(`("monty", "quickjs", vfs)` in Python, `['pyodide','quickjs','vfs']`
in TypeScript), so the single fact the two implementations disagree on
— which python engine a default world registers — was readable only by
diffing two literals in two languages.
Lift it to a named registration, `DEFAULT_PYTHON`, on each side, and
derive `DEFAULT_ENTRIES` from it. Python stays on monty, TypeScript
stays on pyodide: no behavior change, but the divergence is now one
greppable constant carrying twin comments that name the reason
(`@pydantic/monty` cannot answer builtin `open()` calls yet, while
`pydantic-monty` can) and point at each other.
Also drops the second hardcoded copy of the same fact in the TS
`Runtimes` constructor, where `options.python` was forwarded on a
literal `name === 'pyodide'` test.
Recorded as an executable pin rather than prose: the new
`integ/runtime/defaults.json` reads the split back out of a live
default world on each host — `python3 -m nosuchmodule` is refused by
name on Python (`-m is not supported by the 'monty' runtime`) and
answered by CPython's own "No module named" on TypeScript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): pin that the -m refusal is byte-identical on both hosts
The defaults case shows Python's default world refusing `python3 -m`
and TypeScript's answering it, which reads like a message divergence.
It is not: it is purely which engine each default world registers.
Pin the actual parity claim ungated, so it is checked on both hosts:
run monty explicitly on each and assert the same bytes, including the
label normalization that makes the `python` alias report as `python3`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(runtime): one verb casing on the TS bridge
* feat(runtime): create and truncate join the TS bridge, quickjs open dispatches the real op
* refactor(runtime): one fopen-mode parser, chmod parser renamed parse_chmod
* feat(runtime): python readdir answers resolved entries like the TS bridge
* style: formatter output and lint fixes
* refactor(ops): WorkspaceFS becomes Ops in ops/ops.ts with the ledger on it
* feat(ops): union the facade surface across languages
* style: prettier reformat in fuse core
* test(integ): guest op-mode battery + ledger pins; record create/truncate in direct cores, rebind the recorder across the hop
* fix(ops): exclusivity outranks truncate for wx, and only a confirmed absence establishes in the quickjs shim
* fix(integ): cli.sh seeds nested parents and skips facade steps as sdk-only
CI caught this on hf and hf-prefix, identically on both language hosts:
expected "/data/cdp/lnk\n/data/cdp/t\n", got "/\n"
stderr: "cd: /lnk: No such file or directory"
The case created `/data/cdp/t` with `mkdir -p` and then looked inside it.
hf's `stat` short-circuits on a cached parent listing and raises ENOENT
without consulting the backend, and creating a nested directory does not
invalidate the ancestor that gained a child -- so the `$CDPATH` candidate
was rejected and `cd` fell through to the bare `/lnk`.
Reuses the symlink the setup case already makes, the way cdmode.json was
rebuilt onto `disptree/d` for the same reason. `CDPATH=/data` with the
operand `lkq` exercises the identical path -- the announcement is the
selected spelling, the cwd is the target -- without creating anything.
The hf caching bug itself is untouched and deserves its own issue; this
is the second fixture it has forced a rewrite of.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two behaviours from the `set -P` commit that its own integ cases missed.
A `$CDPATH` hit under `-P` announces the spelling it selected and lands
on the target, so the printed path and the resulting cwd disagree --
the same split as `cd -P -`, reached by a different route.
`set -o` applying everything named before the name it rejects is pinned
through pipefail's own effect rather than by reading the option back:
`set -o pipefail -o bogus` exits 2, and the pipeline after it still
reports its failing left-hand side.
Kept out of the `$PWD` commit so dropping that one leaves this in place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`$PWD` was answered by the expander before it reached the environment,
so the three things bash lets you do to it all silently did nothing:
PWD=/clobber; echo "$PWD" GNU /clobber mirage the cwd
unset PWD; echo "$PWD" GNU (empty) mirage the cwd
env | grep ^PWD GNU PWD=... mirage nothing
`$OLDPWD` was already a real variable written by `cd`, and `$HOME` reads
the env, which left `$PWD` the only one faked. It is now seeded at
session construction (bash exports it from startup) and re-stated by
every `cd`, and the special case in the expander is gone -- the fix is a
deletion, not a branch.
Two couplings are load-bearing and pinned:
* `$OLDPWD` is a straight copy of `$PWD` as it stands, not of the
shell's own record. After `PWD=/clobber; cd /data` a following
`cd -` tries /clobber and fails; after `unset PWD; cd /data` it is
empty. `cd` then re-states `$PWD`, so it always repairs itself.
* The shell does NOT read `$PWD` back when deciding where to go.
Clobbering it and running `cd ..` from /data/lk still lands on
/data, so `logical_cwd` stays the shell's own record.
`fork` carries `$PWD` onto a caller-supplied cwd for the same reason it
drops the logical name there.
Consequences worth knowing: `export -p`, `env` and bare `set` now list
`PWD`, and a CLI subprocess inherits it -- both of which is what bash
does, and both of which reach past `cd`, so this is kept as its own
commit.
Pinned against bash 5.2 on debian:stable-slim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four fixes to the logical-cwd work, two from review and two from a sweep
for what the design did not reach.
`fork` now drops the logical name when a caller supplies `cwd`. An
`execute(cwd=...)` call forks with the cwd overridden but inherited the
persistent session's logical spelling, so `pwd`, `$PWD` and a logical
`cd ..` inside that call all described the directory the session used to
be in while operands resolved against the requested one. Deciding it
inside `fork` rather than at the call site is what keeps the next caller
from forgetting; TypeScript needed it spelled out because `??` cannot
choose `undefined`.
`cd -P` announces the path as selected, not the one it resolves to. GNU
prints the spelling and lands on the target, and the two deliberately
disagree:
cd /tmp/lk; cd /tmp/other; cd -P - -> prints /tmp/lk, pwd /tmp/deep/real
CDPATH=/opt/c cd -P lnk -> prints /opt/c/lnk, pwd /opt/c/t
Overwriting `logical` under -P had collapsed both into the target.
`set -P` / `set -o physical` now work. They are the session-wide version
of the flag this branch added, and GNU applies them to `cd` and `pwd`
alike -- but `P` named no option, so it was dropped as an unknown cluster
letter, and while `set -o physical` did store the option nothing read it.
Both were silent, exit 0. `pwd -L` under `set -P` reports the physical
path for free: no logical name is ever recorded, so the pair collapses.
`set -o` rejects a name bash does not have, with GNU's `set: NAME:
invalid option name` and exit 2, keeping the settings named before it and
dropping the rest of the line. Accepting anything is what let
`set -o physical` look supported while doing nothing.
Pinned against bash 5.2 on debian:stable-slim, including the full 27-name
`set -o` table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shell kept one name for the working directory. bash keeps two, and
every surface that reports a cwd picks one of them:
cd /data/lk pwd = /data/lk pwd -P = /data/deep/real
cd /data/lk; cd .. pwd = /data (logical parent)
cd /data/lk; ls .. real (physical parent)
Not just a spelling difference: after following a symlink, `cd ..` and
`ls ..` land in different directories, and bash is internally
inconsistent on purpose -- its `cd` uses the name it remembers, its `ls`
asks the kernel. mirage was self-consistently physical, which is tidier
and not what bash does. It is also why `pwd -L`/`-P` were declared in the
spec and silently ignored: with one path there is no second answer.
`Session.cwd` keeps meaning physical -- it is what every operand resolves
against, so the ~58 readers outside the shell are untouched -- and gains
a `logical_cwd`, None whenever it would be equal. That collapsed
representation is the whole trick: a session that never walks a symlink
carries no new state, and `-P` collapses the pair rather than tracking a
second path forever.
One writer, four readers. `change_dir` maintains the pair (and stores the
logical name in $OLDPWD, which is what `cd -` returns to); `pwd`, `$PWD`
and `cd`'s own join read it. Under -L a relative operand joins the
logical name, under -P the physical one -- that is the `cd -P ..` row.
`set_cwd` is the seam for the four callers that move a session from
outside the shell (snapshot restore, session-store handoff, the
`workspace.cwd` setter). They have no typed spelling, so they drop the
logical name rather than leave it describing where the session used to
be, and they leave $OLDPWD alone because no `cd` ran.
pwd also grows the flag handling it always claimed: -L/-P last-wins
(`pwd -L -P` is physical) sharing cd's splitter, `pwd -x` exiting 2 with
GNU's usage line, and operands ignored. bash never re-validates the
logical name -- removing the link it was spelled through leaves `pwd`
still printing it -- so nothing here checks it either.
Every row pinned in GNU bash 5.2 (debian:stable-slim) and asserted in
both languages: 16 e2e rows each, plus 9 integ cases across 22 targets.
Not persisted across a session-store round trip: `to_dict`/`toJSON` is
the cross-language wire and a restored session collapses to physical,
which is the honest answer for a cwd nobody typed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings, both confirmed against GNU tar 1.35 on
debian:stable-slim.
An operand that cannot be read still owes its notice. `scan.missing`
short-circuited before the prefix was collected, so
`tar -cf out.tar sub/../missing` reported only the stat failure. GNU
announces the prefix first and then fails:
tar: Removing leading `sub/../' from member names
tar: sub/../missing: Cannot stat: No such file or directory
The prefix now comes from the operand's own spelling, before the scan's
problems are reported, so it survives an operand with no entries.
And the notices belong in operand order, not at the front. Collecting
them and prepending put a later operand's notice ahead of an earlier
operand's error; GNU emits diagnostics as it walks:
missing ../file -> Cannot stat missing then notice `../'
../file missing -> notice `../' then Cannot stat missing
`_announce_prefix` / `announcePrefix` now emits in place on first sight
of each distinct prefix, so per-name deduplication is kept without the
reordering.
The TypeScript side kept the old global prepend alongside the new
in-place emission for a moment, which doubled every notice -- caught by
the existing "warns once" assertion before it left the branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keeping an absolute operand's typed spelling (the commit before this)
reached further than cd: tar builds member names from `raw_path`, and
`member_name` only removed a leading slash, so `tar -cf out.tar
/data/sub/../file` began storing `data/sub/../file` -- a name that
extracts somewhere other than where it says.
GNU refuses to store any name that could climb out of the extraction
directory. It drops everything through the *last* `..` segment and falls
back to the leading slash only when there is no `..`:
/data/sub/../file -> file notice `/data/sub/../'
x/../y/f3 -> y/f3 notice `x/../'
../../file -> file notice `../../'
/data/file -> data/file notice `/'
./file -> ./file no notice
A `.` climbs nowhere, so it survives. Stripping is per name rather than
per operand, which is why `tar -cf a.tar ..` owes two notices: `..` for
the directory itself and `../` for everything under it. An operand that
is all traversal leaves nothing to name and GNU stores it as `./`.
The notice now names the prefix it removed instead of always saying `/',
and each distinct prefix is reported once.
Info-ZIP makes the opposite choice -- `zip sub/../g.txt` stores
`sub/../g.txt` verbatim -- so zip_cmd deliberately does not share this,
and an integ case pins that the two stay different.
All rows pinned against GNU tar 1.35 and Info-ZIP on debian:stable-slim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mkdir -p /data/lkd/real` created a directory whose only child was another
directory, and hf could not see it. Its stat short-circuits on a cached
parent listing:
parent_listing = await index.list_dir(parent)
if parent_listing.entries is not None:
raise enoent(path) # never consults the backend
/data's listing was already cached without `lkd`, and writing a file two
levels down did not invalidate the ancestor that gained a new child, so
`cd -P /data/lkp/..` got ENOENT on hf while `-L` passed by landing on the
mount root. Adding a file to the fixture did not help, for the same
reason -- the stale listing is consulted before any of it matters.
Point the link at `disptree/d` instead. It is already in the `files/v1`
fixture that all 22 targets mount at /data, so the case now creates
nothing but the symlink: no mkdir, no backend asked to make an empty
directory real, no cached listing asked to learn a new child.
The ancestor-invalidation gap is a real hf bug and is untouched here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GNU echoes a path operand back exactly as the user spelled it. Pinned in
debian:stable-slim:
du -b /k/ -> 7 /k/
find /k/guides/ -> /k/guides/
ls /k/nope/ -> ls: cannot access '/k/nope/': ...
md5sum /k/guides/../guides/a -> <hash> /k/guides/../guides/a
tree /k/guides/ -> /k/guides/
tree -f /k/guides/ -> /k/guides (-f is the exception)
Nine cases captured the pre-fix spelling, where the classifier normalized
an absolute operand before any command saw it and the trailing slash was
gone by the time du/find/ls/tree echoed it. Every one of these commands
takes a trailing-slash operand, so all nine move to the GNU spelling.
Also give the cd-mode fixture a file. `mkdir -p /data/lkd/real` leaves an
empty directory, which backends with no empty-directory concept (hf) do
not materialize, so `cd -P /data/lkp/..` could not reach /data/lkd there
while `-L` passed by landing on /data. Writing f.txt makes both parents
real on every target, which keeps all 22 rather than dropping hf.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`seed` waited for the query API on 16686 and then pushed spans to the
OTLP receiver on 4318. Those are separate ports on the same container,
bound at different times, so the push could meet a docker-proxy socket
that accepts the connection and resets it -- exactly the failure `query`
already documents and guards against for its own port. It killed
integ-shared-py on PR #763's first run, during setup, before a single
case executed:
ConnectionResetError: [Errno 104] Connection reset by peer
jaeger_seed.py:186 in post_spans
Retry the push itself, on the same POLL_ATTEMPTS/POLL_DELAY cadence, so
the readiness check is the endpoint that has to be ready. Catch OSError
for the reason `query` gives -- the reset arrives raw, past urllib -- and
raise with the last error when the receiver never accepts, rather than
failing confusingly further along.
Verified against a real `jaegertracing/jaeger:latest` container from a
cold start, and against a dead port (fails loudly after the full window).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`cd -P` was unreachable. `_resolve_target`/`resolveTarget` implement it
correctly -- given `/link/..` with `/link -> /deep/real` they return
`/deep` for -P and `/` for -L -- but nothing could reach the physical
branch, so the two modes returned the same answer for every input.
Two layers each normalized the `..` away before `cd` saw it:
- `_cd_candidates` built every candidate with `resolve_path`, which
normpaths. Join without normalizing instead and let `_resolve_target`
normalize, which it already does for both modes; when the workspace has
no symlink table at all, `handle_cd` normalizes the candidate itself.
- the operand arrived already collapsed. `heuristic.py`'s absolute branch
built its PathSpec with `virtual=normpath(word)` and no `raw_path`, so
`raw_path` defaulted to the normalized form and the typed spelling was
gone before any command ran. Pass `raw_path=word`, which is what the
field is documented to hold and what `relative_spec` has always done for
the relative case; `cd` reads it through `_typed_path`.
GNU bash 5.2, pinned in debian:stable-slim with /link -> /deep/real:
cd -L /link/.. PWD=/ cd -P /link/.. PWD=/deep
cd -L /link/sub/.. PWD=/link cd -P /link/sub/.. PWD=/deep/real
All four now match, and `cd` lands in the directory bash lands in for
every pinned case. One divergence is left, and it is a separate design
question: bash's -L keeps the *logical* name in `$PWD` (`cd -L /link`
leaves PWD=/link), while mirage resolves and stores the physical path.
That needs a logical cwd on the session, which would change what every
command resolves against -- and is why `pwd -L`/`-P` are declared in the
spec and ignored today.
Verified: full battery on ram and disk, 4755 cases, 0 failures on each
side, and `parity.py` diffs 4755 rows with 0 divergences. New
`integ/unix/sym/cdmode.json` is what caught that the first cut of this
fix worked in a unit test and not through the real pipeline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(errors): one condition vocabulary, one classifier per language
A condition is named once (FsCondition) and every boundary keeps only
a table from that name to its own number: posix, wasi preview1, and
the monty guest view. CycleError gets its ELOOP seat (was EIO at every
kernel and guest boundary), the wasi wire's EXDEV is 75 per wasi-libc
(18 is EDOM there), and the cross-mount-rename-is-ENOENT decision
moves from an adapter arm into the wasi table row.
* fix(stat): one stat view per language, offset-less stamps read as UTC
mtime, dir-ness, content size and the mode bits live once per language
(utils/stat_view), delegating to the existing naive-stamp-is-UTC
parsers. Three of the four translators (node FUSE, the runtime bridge,
py wasm) read an offset-less backend stamp as local time, so python
FUSE and node FUSE disagreed by the host's UTC offset for the same
stamp; all four now answer the same epoch, pinned by tests.
* refactor(errors): move the wasi and cpython tables into their runtimes
* refactor(errors): classifier arm tables into constants modules
* fix(errors): address review on the ValueError arm and epoch-zero mtime
* fix(himalaya): file the sender's own copy of a sent message
`--send` opened one SMTP conversation and stopped, so mail was delivered
but the sender's Sent mailbox stayed empty. A Sent folder is not produced
by sending: SMTP keeps no record of itself, and the copy is a second,
separate IMAP APPEND that every mail client makes on its own. Four call
sites, not one: `message send` plus the route compose, reply and forward
share.
Upstream v2.0.0 spells that APPEND as `--save <MAILBOX>` on all four
verbs, having replaced 1.x's automatic `message.send.save-copy` config.
Both are here, because they answer different needs:
- `--save <MAILBOX>` is v2 parity. On its own, with no `--send`, it files
the message without sending it, which is how a draft is written.
- `save_copy` is an account-level default, on, and is mirage's own
divergence: an agent that never learned the flag still leaves the
record a human sender would. `sent_folder` pins the mailbox.
The mailbox is asked for rather than guessed. A server implementing RFC
6154 tags one mailbox \Sent in its LIST reply, which is `[Gmail]/Sent
Mail` on Gmail and `Sent Items` on Exchange; failing that the configured
name, failing that `Sent`. This is ahead of upstream, whose v2 wizard
notes IMAP pins the reserved INBOX alone while it waits on LIST RETURN
(SPECIAL-USE) in io-imap.
Failure splits on whether a send already happened. A copy that fails
after a successful send is a warning on stderr with exit 0, because the
message is already gone and a non-zero exit invites a retry that sends it
twice. A `--save` that sends nothing fails loudly, because nothing
happened yet and retrying is safe. Upstream propagates in both cases.
Two bugs found on the way:
- aioimaplib joins command arguments with spaces verbatim and quotes
nothing, so an unquoted mailbox holding a space arrives as two
arguments and the server reads only the first word. That breaks on
exactly the two providers above. `quote_mailbox` now covers the APPEND
and the pre-existing `select_folder`, which had the same latent bug.
imapflow needs none of it, building commands from typed attributes.
- The TypeScript test CONFIG was an untyped literal, so a new required
field did not fail tsc and those tests passed vacuously through the
`!saveCopy` early return. It is annotated now, and a global beforeEach
stubs the accessor so no --send test reaches for a real connection.
GreenMail hands a new account nothing but an INBOX, where a real provider
ships a sent mailbox already made, so both integ seeders create one.
Verified against a live GreenMail on both hosts: the email and
cli-himalaya targets pass 86 and 74 cases including six new ones, and
compose, reply, forward and send each land in Sent with \Seen. Python
suite and TypeScript node suite green, pre-commit clean.
* fix(email): parse a LIST mailbox as an astring, not as a quoted string
Codex review on #761. A LIST mailbox is an astring, so a name needing
no quoting may legally arrive bare, and some servers emit it that way.
Splitting the line on quotes then reads the hierarchy delimiter as the
name: `(\HasNoChildren \Sent) "/" Sent` answered `/`.
That splitter predates this branch and already mis-listed every folder
on such a server, but this branch is what made it consequential: the
name now feeds an APPEND, so a sent copy would have been filed into a
mailbox called `/`.
The three tokens are walked in order instead. The atom form and the NIL
delimiter both parse, and a quoted name is now unescaped rather than cut
at the first inner quote, which the splitter also got wrong.
Verified against a live GreenMail on both hosts, whose own wire format
is the empty-attribute `() "." "INBOX"` shape.
* fix(integ/gws): resolve quoted worksheet names and implement updateCells
Two fidelity gaps in the fake Google Workspace server, both found by
driving a real client at it.
`parseA1` stripped the quotes around a sheet name only on the `!` branch,
so `'Jun-Jul_2025'!A1:G9` resolved but a bare `'Jun-Jul_2025'` did not.
That is not a corner: gspread quotes unconditionally
(`utils.quote_sheet_name`) and sends the bare quoted name to read a whole
worksheet, so `Worksheet.get_all_records()` answered
`Unable to parse range` against any sheet at all. Quoting is now handled
before the `!` split, in `splitRange`, with doubled apostrophes unescaped
the way the real grammar spells an embedded quote -- and, because the
name is consumed first, a `!` inside a quoted name no longer confuses
`lastIndexOf`.
`updateCells` was unimplemented and answered `Unsupported request`. It is
how a caller shortens a sheet it previously wrote longer -- the real
request clears whatever `fields` names across the range that `rows` does
not cover -- and there is no other way to do that through batchUpdate.
Both forms are served: `range` (clear, then write what is supplied) and
`start` (write at a coordinate). Only `userEnteredValue` is kept, since
the fake stores strings and formatting has nowhere to go; a number
renders as its decimal, a bool as TRUE/FALSE, and a formula as its text,
which is consistent with the fake not evaluating formulas.
Three cases in integ/cli/gws.json cover them, each self-contained on a
spreadsheet it creates. All three fail on the unfixed server -- the read
with `Unable to parse range`, the two writes with
`Unsupported request: updateCells` -- and the target goes 65 to 68 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(integ/gws): read a whole-column range like A:Z as cells, not a tab name
`parseA1` decided an unquoted range with no `!` was a tab name unless it
matched `^[A-Z]+\d`, which wants a digit right after the column letters.
`A:Z` has none, so it resolved as a sheet called "A:Z", found nothing, and
came back `Unable to parse range: A:Z`.
That is the range gspread asks for when a caller wants every row of the
first worksheet -- `spreadsheets().values().get(range='A:Z')` is the
ordinary way to read a sheet whose length you do not know -- so a whole
family of readers could not read the fake at all.
The test is now a single pattern covering every A1 form the real API takes
with a half open side: `A1`, `A1:G9`, a whole column span `A:Z`, and a
whole row span `1:5`. A name a tab actually has still wins first, as
before, and the old `includes(' ')` special case is subsumed -- a name with
a space cannot match. `parseCell` already answered `{row: null}` for a
bare column and the range builder already read that as unbounded, so
nothing downstream changed.
One more case in integ/cli/gws.json; the target goes 68 to 69 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(integ/gws): honor the updateCells field mask before clearing values
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the two grep_helper suites produced a ten-name import, and the
two formatters disagree about how to wrap one: yapf wants a hanging
indent, isort wants the names aligned under the paren. Each undoes the
other on every run, so `pre-commit run --all-files` reported both hooks
as "files were modified" forever while the tree kept returning to the
same bytes.
Pin yapf's form and take the import out of isort's hands, which is what
the tree already does at `tests/commands/builtin/generic/test_split.py`
and eight other sites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`control.py` (537 lines) and `jobs.py` (233) had no test on the Python
side while `control.test.ts` and `jobs.test.ts` sit beside their sources;
the only suite references were two `ReturnSignal` imports. Port both,
then cover the three `builtins/` modules neither tree tested at all.
- `test_control.py` ports the TypeScript assertions for if/for/while/
until/case, and adds what only Python has a twin for: the `for ((;;))`
handler, a multi-level `break` decrementing on its way out, and the
`_MAX_WHILE` cap emitting its stderr warning.
- `test_jobs.py` ports wait/kill/jobs/ps and adds `handle_fg`, which has
no TypeScript twin: the no-job message, the unknown id echoed as typed,
and the command-line header prepended to the adopted stdout.
- `test_history.py` pins `_parse_args`, whose bash-getopt corners were
documented in a docstring and asserted nowhere: `-d3` attached, `-dc`
reading "c" as the offset rather than another option, `history -1` as
an invalid option, `--` and the first operand both ending option
parsing, and a bare `-` as an operand.
- `test_dirs.py` covers `handle_cd`: CDPATH search order and the
announce-the-destination rule, the mount-root fallback for a path the
backend cannot stat, symlink following through a prefix and a chain,
and ELOOP.
- `test_scope.py` covers the two PathSpec helpers.
Also move `test_find_action_dispatch.py` beside its source, which lives
at `executor/find_action_dispatch.py`, not `workspace/`.
Not covered on purpose: `cd -P`. `_resolve_target` implements it
correctly -- called directly, `/link/..` gives `/deep` under `-P` and `/`
under `-L` -- but nothing can reach that branch, because every candidate
comes from `_cd_candidates` -> `resolve_path`, which normpaths `..` away
first, so `-L` and `-P` return the same answer for every input. The
TypeScript twin is structurally identical (`resolvePath` then
`resolveTarget`), so this is a shared divergence from bash rather than a
parity gap. Fixing it is a behavior change to `cd`, out of scope here;
asserting today's answer would pin the bug.
Mirror baseline 211 -> 207.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
isort folds the two same-module imports the merge left behind into one
and drops the duplicated `compile_pattern`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit's cleanup only holds if something refuses the next
drift, and the project already enforces comparable rules with
source-walking meta-tests (`test_no_raw_flag_reads.py`,
`test_no_dead_flag_params.py`). The absence of one here is why the
`__init__.py` count reached 87 and why a 24-file directory could sit
beside 19 packages it did not name.
Three hard checks and one ratchet:
- no `__init__.py` anywhere under `tests/`;
- every test directory mirrors a `mirage/` package, or is one of eight
entries in `UNMIRRORED_DIRS` with a written reason -- harnesses and
fixture trees (`commands/native`, `e2e`, `fixtures`, ...) that mirror
nothing on purpose;
- no stale exemption: an entry whose directory is gone, or whose source
twin now exists, fails instead of lingering;
- a ratchet on the number of source modules no test file is named for,
moving in either direction only deliberately. It counts the weaker of
the two readings -- `test_<name>.py` anywhere rather than beside its
twin -- because CLAUDE.md asks for a mirror only "where reasonable"
and several suites cover a family in one file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tests/` is a namespace tree -- pyproject already sets
`--import-mode=importlib`, which resolves the same-basename collisions
(34 different `test_find.py`) an `__init__.py` would otherwise be needed
for -- so CLAUDE.md forbids them. Without a check the count reached 87,
all of them content-free: 81 carried only the license header, 6 were
empty. Delete them; `tests/commands/postgres/` held nothing else and
goes with them.
Then move the three directories that mirrored no source package, because
that is what hid two coverage gaps:
- `tests/resource/object_storage/` collected 24 suites for 19 sibling
packages, so `mirage/resource/s3/s3.py` looked untested. Split it one
directory per source package, the five s3 suites into `tests/resource/s3/`,
and the cross-alias prefix suite up to `tests/resource/` beside
`test_s3_aliases.py` -- renamed to name its subject, since `test_prefix.py`
says nothing at that level.
- `tests/commands/builtin/helpers/` shadowed the flat `*_helper.py` sources.
Its conftest was a byte-identical copy of its parent's, so it just goes.
Three of its five suites collided with a same-named flat suite and were
merged: they test disjoint halves of one helper (`number_flag_error` vs
`parse_counts`; the `sort` command vs `parse_keydef`/`build_config`;
`grep_lines`/`grep_recursive` vs `compile_pattern`/`search_query`). The
sort merge renames a local `sort_lines` wrapper to `_run_sort`, which
otherwise shadows the `sort_helper` import of the same name.
- `tests/commands/ssh/` sat beside `commands/builtin/ssh/`'s twin instead
of under it.
Also flatten the ten single-file `{ram,redis}/{cat,head,ls,tail,wc}/`
directories onto their already-flat siblings, and move `test_man.py` under
`executor/builtins/` where `man.py` lives.
Collection is unchanged: 13863 tests before, 13863 after the moves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A directory present in only one implementation was recorded as a single
finding and its module set discarded, so the 21 one-sided directories
counted 21 rather than the 74 modules they hold. That made the ratchet
blind exactly where drift is easiest: a new module under a directory
already in the baseline -- `agents/mcp`, `ops/postgres`,
`core/onedrive` -- left the count untouched and `--strict` passed.
A one-sided directory now contributes every module inside it. Adding a
module under `agents/mcp` moves the count 310 -> 311 where it used to
sit at 257 forever. Directories become presentational: they are printed
once with their module count and excluded from `total()`, so they are
not double-counted against the modules they now contribute.
An excused directory stays a deliberate blind spot, and keeps excusing
its whole subtree. The excuse is precisely that there is no counterpart
to mirror -- agno is python-only, opfs is browser-only -- so a new
module inside one is expected growth, not drift. Verified both ways: a
probe under `agents/mcp` (unexcused) moves the count, a probe under
`core/opfs` (excused) does not, and stale-entry detection still fires.
Baseline 257 -> 310. The number is larger because the unit is now
consistently one module; no divergence was added.
Reported by codex on #759.
The #370 sweep added `compareCodePoints` to cli/src/config.ts and
cli/src/settings.ts, the first time anything under packages/cli/src
imported @struktoai/mirage-core. The package never declared that
dependency, and tsup externalizes only what package.json declares -- so
rather than leave it an import, tsup inlined the whole core barrel into
the CLI bundle, dragging isomorphic-git's CJS entry along with it.
esbuild rewrites a CJS `require()` it cannot resolve into a shim that
throws, so every `mirage` invocation died at module load with
`Dynamic require of "buffer" is not supported`.
Nothing local caught it: tsc resolves the import through the pnpm
workspace link and the build succeeds. Only the bundled binary breaks,
which is why it surfaced as 89 integ-runtime failures, the cli e2e
suite, and a cross-language snapshot job whose python client could not
reach a daemon that never started.
Declaring the dependency restores externalization: the CLI dist now
carries a plain `from "@struktoai/mirage-core"` and no isomorphic-git.
The unrelated follow-redirects hunk in the lockfile is pnpm's own
peer-dep dedup, emitted by the same install.
CLAUDE.md asks for the two trees to keep a mirrored layout and nothing
checked it, so seven helper modules went missing and two module homes
swapped without anything noticing. scripts/check_layout_parity.py diffs
the module-name sets directory by directory: core/node/browser union onto
one python namespace because CLAUDE.md puts runtime-specific code in
node/browser, while cli, server and agents are separate npm packages
whose python twins are subpackages, so they map by prefix.
Two normalizations keep the report honest. __init__.py and index.ts are
skipped, since only python requires a package file per directory and
comparing them reports a gap across most of the tree that nobody intends
to close. Module and directory names fold camelCase, hyphens and a
leading underscore, so findEval.ts against find_eval.py reads as a
rename rather than two missing modules -- 15 renames separated out from
207 genuinely absent modules that way.
--strict does not demand zero. 257 divergences predate the gate and each
needs its own decision, so it fails when the count moves off the
committed baseline in *either* direction: new drift is blocked, and
closing a divergence has to be locked in by lowering the number rather
than being silently spent. A stale exception fails just as loudly, which
is the failure mode every hand-maintained allowlist in this repo has
already hit.
The report reproduces five findings the audit had filed separately:
findEval/findParse as the only camelCase modules, _provision.ts vs
provision.ts, workspace/snapshot missing keys and types on the python
side, no builtins/shared.ts, and node's cache/redis/file.ts sitting where
python has cache/file/redis.py. That is what it is for -- items 31 to 37
get scoped from its output instead of from taste.
Also gates scripts/gen_width_table.py against its committed output, the
same way the spec trees are gated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent bugs, one shared table.
`-L` is the length of the longest line in *columns*, not characters, and
neither tree measured columns. Pinned against coreutils 9.7:
`printf 'a\tb' | wc -L` is 9, because a tab jumps to the next multiple of
8 -- both trees said 3. `printf 'a\rb' | wc -L` is 1, because carriage
return rewinds the cursor without ending the line: Python counted CR as a
column and said 3, TypeScript split on /\r?\n/ and reached 1 by treating
CR as a line break, which is a different answer that happens to match on
this input. A CJK character is 2 columns, a combining mark is 0, and a
control character is 0 (that is how wc accounts for wcwidth's -1).
`-w` splits on glibc's iswspace, which is Unicode White_Space *minus*
U+0085. Both trees approximated it with a builtin and both were wrong, in
opposite directions: Python's str.isspace() also reports U+001C-U+001F
and U+0085, so `printf 'a\x1cb' | wc -w` counted 2 words; JavaScript's
\s also matches U+FEFF, so `printf 'ab' | wc -w` counted 2. The
plan named Python correct on this row; it is not.
The width data now comes from scripts/gen_width_table.py, which walks
unicodedata and emits both trees plus integ/fixtures/wc/width.json. The
fixture is the contract, the same way integ/fixtures/filetype/tables.json
is: regenerating one tree without the other fails both suites. The
generated tables live under utils/generated/, exempted from yapf and
isort here and already covered by typescript/.prettierignore, because a
formatter that repacks 349 ranges fights the generator on every run.
Cf is not uniformly zero-width, which is the one place raw Unicode
categories would have been wrong: U+200B and U+FEFF measure 0 while
U+00AD, U+0600-U+0605 and U+110BD measure 1. Documented divergence: an
unassigned code point measures 1 here and 0 in glibc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JavaScript orders strings by UTF-16 code unit. An astral character
(U+10000 and up) is a surrogate pair in 0xD800-0xDFFF, so it sorts before
every BMP name from U+E000 up, while Python's `sorted` puts it after.
`ls` on a directory holding a U+E000 name and an emoji listed them in
opposite orders across the two trees. GNU agrees with Python here, in
both C and C.UTF-8: UTF-8 byte order *is* code-point order, so
TypeScript was the only one of the three disagreeing.
Adds utils/sort.ts (compareCodePoints, sortedByCodePoints) and routes
every string sort through it. Three classes of site, only the first of
which was obvious:
- 131 bare `.sort()` calls across 94 files.
- ~40 explicit `a < b ? -1 : a > b ? 1 : 0` comparators, which are the
same code-unit comparison wearing a comparator's clothes.
core/gdrive/readdir.ts even carried the comment "Codepoint compare, not
localeCompare: python sorts by codepoint" directly above one.
- 3 localeCompare calls, whose Python twins are all plain code-point
sorts. ICU collation reorders ASCII too, so those diverged on far more
than astral names.
An integ pin on `ls` and `find` is what caught generic/ls.ts, which the
bare-`.sort()` sweep could not see because it already passed a
comparator. eslint now rejects a zero-argument `.sort()` outright (it is
also wrong for numbers and tuples, which it compares as strings) so the
default cannot come back; typecheck caught the four sites where the
sweep had reached numbers and bigints.
Two latent bugs fixed on the way: git/reset.ts sorted [path, letter]
tuples by their `toString()`, ordering `a+b` before `a` because ',' has
a higher code point than '+'; sort_helper.ts's compareLines is the
`sort` command itself, so the C-locale byte order it should have had is
now what it computes.
Closes#370
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(ops): one door for VFS/FUSE ops in both languages
Ops (python) and WorkspaceFS (typescript) now delegate every op to
Dispatcher.dispatch, so FUSE and programmatic access walk the same
pipeline as a shell command: link follow, session grants, admission
policies, cache read-through, namespace structure and write
invalidation each fire once. The facade keeps only the typed surface
and op recording.
- PolicyDenied.completed distinguishes a pre-deny from a post-deny, so
a completed-then-denied op is still recorded before it propagates.
- A cross-mount rename is refused at the facade with EXDEV, so mv falls
back to copy+unlink. TypeScript classifies FUSE errnos by code
string, so EXDEV had to join the table or it degraded to EIO.
- TypeScript's MountCore was a split surface (reads via ws.fs, writes
via ws.dispatch), so drainOps reported no writes at all. Every
mutation now goes through the facade.
- Retired the forwardIndex knob so TypeScript forwards the index to
every backend the way python always has.
- Deleted the workspace-less Ops pipeline: dispatch is now required.
It was a second door with different behavior (no cache, no namespace
structure, gates only when a caller passed policies), and only tests
reached it.
- Ops.read gains raw, mirroring readFile(path, {raw: true}). FUSE's
read-modify-write hands its merged buffer to write, which stores, so
the read that feeds it must be the stored bytes. Raw reads also skip
the file cache, which is keyed on the path alone and holds whatever a
command rendered.
python 13185 passed, typescript core 7232 and node 2282.
* fix(ops): slice a warm cache for ranged reads, record denied read bytes
Two review findings from codex on the collapse.
A ranged read served from the file cache returned the whole object
instead of the window: the cache holds the full file, and the warm
path handed it back untouched. The dispatcher now slices it with the
same helper the cold path falls back to, so warm and cold agree. This
was already reachable on main through git, which reads pack indexes as
a few bytes at a known offset, and routing Ops.read(offset, size)
through the same door widened it.
A post_ops deny on a read recorded zero bytes, because the suppressed
result is the only place a read's byte count lived (a write's is still
in its own arguments). PolicyDenied now carries completed_bytes,
stamped where it is raised, so records and network_bytes stop
under-reporting traffic that actually happened.
* fix(ops): keep transfer bytes and cache origin through the op door
Two more review findings from codex, both cases where the deny and
limit paths lose a fact the success path has.
A post_ops Limit truncates what the caller receives, but the transfer
already happened, so recording the capped length under-reported
network_bytes by whatever the cap removed. IOResult gains op_bytes,
which the door sets only when a limit truncated a read, and the facade
prefers it over the delivered length.
A post_ops deny on a warm cache hit was recorded against the backend,
so a read served entirely from the file cache counted as network
traffic that never happened. PolicyDenied gains from_cache, stamped
where the warm branch raises, and the facade reads it exactly as the
success path reads a cache-served result.
* fix(ops): a namespace answer is not the parent backend's op
A directory that exists only because a mount or a link sits below it is
served from the namespace without contacting anything, but the facade
attributed it to the mount that lexically owns the path. On a remote
parent that invented a network record for every such lookup, which a
session granted only a nested mount hits on every walk down to its
grant.
The door now names the server explicitly through IOResult.op_source
rather than leaving the facade to infer it from a non-empty reads map:
"ram" for a warm cache hit and for a synthetic namespace answer, unset
when the owning mount served it. That replaces the inference, so the
two cases the door already knew about are now stated rather than
guessed.
The backend keeps the record when it was actually reached and answered
ENOENT, which is a real round trip on a remote mount.
* fix(ops): errors raised after an op ran carry its source and bytes
* style: match CI yapf on the door-error test
* fix(ops): stamp an OpReport at completion, drain drift at the door
Both trees resolved tr's SET operands with a reader built for `echo -e`.
Python's was a regex over eight named escapes, so `tr '\0' -` and every
octal spelling silently matched nothing. TypeScript's was echo's scanner
outright: it read `\x41` as `A`, stopped the whole scan at `\c`, and only
took octal behind a leading zero.
TypeScript did not drift there by accident. workspace/executor/escapes.test.ts
is a port of Python's *echo* test that imported tr's reader, so every
echo-only rule was pinned against the wrong command — the test was holding
the bug in place. It now imports interpretEchoEscapes, which text.ts
exports for it the way builtins/__init__.py already re-exports
_interpret_escapes.
tr's grammar, pinned against coreutils 9.7 (debian:stable-slim): octal is
\NNN with no leading zero required, greedy to three digits, so \0141 is
\014 then a literal 1; a three-digit value over 255 backs off to two
digits (GNU also warns, which a pure reader has no channel for); an
unknown escape drops its backslash, so `tr '\x41' -` deletes x, 4 and 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: GNU byte pins across tail, split, numfmt, sed, cp and find (#609 T1-F)
Six of the eight T1-F rows, each re-pinned against debian:stable-slim
(coreutils 9.7, sed 4.9) before it was touched. Four of them were wrong
in both languages, not one, and two were larger than the audit recorded.
tail -c takes the same sign grammar as -n. TypeScript did
`raw.slice(-bytesMode)`, so `-c -3` dropped the FIRST three bytes;
Python's `abs(c)` turned `-c +3` into the last three. Both now route -c
through one shared parse_counts/parseCounts, which is what the three
hand-written call sites had drifted apart from.
split -t is one byte or the two-character spelling \0. TypeScript took
`ENC.encode(sep)[0]` and Python kept the whole byte string, so `-t XY`
split on 'X' in one tree and on 'XY' in the other where GNU refuses to
run. An empty value is now a report, not a silent newline.
numfmt: --to=none renders in fixed notation from an exact value, so
`--from=si 1Y` prints twenty-five digits instead of TypeScript's `1e+24`
-- and `--from=iec 1Q` at all, which the 28-digit default decimal
context could not hold. The suffix grammar is now GNU's: one unit
letter, lowercase only for kilo, with the trailing i required under
iec-i, optional under auto and refused elsewhere. `1KiB` used to read as
a kilobyte in both trees. Errors carry GNU's four message shapes and
exit 2 instead of raising decimal.InvalidOperation.
sed's missing-script and no-input paths had four spellings across the
two languages, and Python raised where TypeScript returned exit 1. Both
now share two constants, return an IOResult, and use GNU's `no input
files` with its exit 4. GNU's usage block is deliberately not
reproduced.
cp -rv ordered sibling directories by length alone, which is
PYTHONHASHSEED-dependent for equal-length names; both sides key on
(len, path) now.
find -mtime parsed timestamps inline on both sides instead of using the
helper each tree already had: Python stamped UTC over a real reported
offset and let a malformed value raise out of the walk, while
TypeScript's bare Date.parse produced a NaN that slipped past the null
guard and KEPT the entry Python dropped.
Deliberate divergences, documented in place: GNU's quotearg escaping in
the split message (matching the existing truncate choice), and the long
double GNU reads numfmt input into, which prints 1.10 as 1.11 while 1.20
and 1.30 round-trip.
Verified: 4695 integ cases pass on ram and disk in both hosts, py+ts
unit suites green, pnpm -r typecheck clean, spec parity 93/93.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(tail): read a TailCounts field through ?? null, not !== null (codex)
Codex flagged the tailBytes signature change as a public API break.
Backward compatibility is explicitly out of scope here (CLAUDE.md), and
the described failure was wrong -- `tailBytes(data, 2)` does not fall
back to ten lines, it returns the whole input, because `(2).fromByte` is
undefined, `undefined !== null` is true, and `slice(NaN)` is `slice(0)`.
But that is a real hole and it is worth closing on its own: TailCounts
is a plain object, so any field a caller omits arrives as undefined and
a bare `!== null` guard waves it into the wrong branch. Python's twin is
a dataclass whose four fields all default to None and cannot do this.
`normalizeCounts` gives the TypeScript side the same floor, and both
`tailBytes` and the cache gate in the generic read through it.
Four tests pin it, including the exact shape codex described.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(shell): run `sh FILE` / `bash FILE` as a script, not as the program text
handle_bash treated the first non-flag operand as the program text, so
`sh /work/run.sh` executed the literal string `/work/run.sh` and reported
`/work/run.sh: command not found`.
- read the operand through the op dispatcher and run its contents
- positional parameters from the operands, and from `-c 'prog' name a b`
- `$0` is the script (new Session.script_name, read by `$0` and `${@:0}`)
- `-e -u -x -f` were silent no-ops; map them through SET_FLAG_TO_OPTION
and restore afterwards
- GNU diagnostics: missing 127, unreadable 126, directory 126, prefixed
with the head word so `sh` no longer says `bash:`
* refactor(shell): state the bash/sh startup rules once instead of case by case
The first commit fixed the reported case; this one replaces the case
lists it left behind with the rule each one was standing in for. Every
change below closes a divergence pinned against GNU bash 5.2 on
debian:stable-slim.
- The startup option grammar is `set`'s grammar, so both now read it from
one place (`shell/options`). `-o name` was parsed only to be thrown
away, which silently dropped `bash -o pipefail` even though pipefail is
implemented; `+x` / `+o` were not option words at all, so `bash +x f`
looked for a file called `+x`; and the options were a list of names to
enable, which cannot express `bash -e +e` (last one wins). `set -eo
pipefail` was broken by the same gap and is fixed by the same change.
- bash's long options carry arity now, not just names: `--rcfile FILE`
consumes FILE, and an unknown `--long` is a usage error rather than a
script path (`bash --version` used to report a missing file).
- The program comes from stdin whenever no operand names one. `-s` is one
case of that rule, not the rule, so `echo prog | bash` ran nothing and
`bash -s A B` ran the file `A`.
- One reader for a script file, shared by `source` and a nested shell,
refining a keyed backend's ENOENT for a directory into EISDIR. `source`
reported a directory as missing, spelled the path resolved rather than
as typed, and had a third copy of the payload drain in TypeScript.
Bare `source` now gives GNU's usage error instead of statting the cwd.
- The exit code comes from the errno: 127 only for ENOENT, 126 for
everything else. It was `PermissionError ? 126 : 127` in Python and
`EACCES` in TypeScript, so ENOTDIR was reported as 127 in both.
- One child-shell snapshot, shared by `( … )` and a nested `bash`/`sh`.
The nested shell isolated three hand-picked fields, so a `cd`, an
`export` or a function definition inside `bash -c` leaked into the
caller; the subshell's own list had never been told about `script_name`
and neither had Python's `Session.fork`, which is a hand-written
literal whose docstring promised it could not happen. The field list is
declared once and a test fails until every dataclass field is
classified. `source` deliberately does not snapshot: a sourced `set -x`
does leak in bash.
- `$0` reads one `Session.argv0` accessor; Python's `or` treated an empty
`-c` name as absent where TypeScript's `??` did not.
* fix(integ): keep the cwd case self-contained, and pin symlinked script operands
Two follow-ups from review.
`ctl_sh_script_file_cwd_isolated` changed the session's working directory
and never changed it back. The battery shares one session per target, so
that re-rooted every relative operand in every later case. It runs in a
subshell now, which restores the directory and exercises the other half
of the same snapshot.
Review also asked whether a script operand that is a namespace symlink
reaches the backend unresolved, since the backend cannot see a link the
namespace owns. It does not: the operand resolves before dispatch, and
absolute, relative and cross-mount links all run their target while a
broken one reports the link at exit 127. Pinned in both languages and in
the battery, since nothing covered it.
* fix(shell): start a child shell outside every source its caller is inside
`session.source_depth` was transient, so the child-shell snapshot neither
reset nor restored it. Running `bash FILE` from inside a sourced script
therefore left the depth positive, and a top-level `return` in the child
was absorbed as if the child were itself sourced: GNU prints "can only
`return' from a function or sourced script", keeps going and reports 0,
where mirage ended the child early with the returned status. The field
joins CHILD_SHELL_FIELDS so the same snapshot that already restores it
carries it, and the nested shell sets it to 0 the way it sets `$0`.
`OptionWord` also moves to `shell/types.py` / `shell/types.ts`, next to
the SET_FLAG_TO_OPTION table it is the shape of, per the module layout.
* fix(traversal): cross a nested mount correctly in du, find, ls -R and tree
A mount nested inside another mount's tree is invisible to the parent
backend's readdir, so the parent's keys under that path are shadowed.
Six bugs followed from that, all fixed here in both languages.
- du counted the shadowed keys. The generic now opts into MountView and
prunes leaves under a descendant mount after the link merge, then
recomputes the total from the survivors.
- du -c printed one total row per mount. The bytes-level re-total is
extracted as merge_du_totals so the executor fan-out reuses the same
code, with -h humanizing once at the merge instead of per block.
- Fan-out sub-runs never received LinkView, so symlinks vanished from
find/du/tree/grep -r exactly when a nested mount existed.
- A line whose operands span mounts never fanned out inside an operand,
because the cross-mount runner executes one single-mount native run
per operand. run_with_fanout wraps that seam; it forwards resolve_hint
as well as stdin, which the STREAM strategy needs.
- ls -R was filtered a line at a time. Its format is PATH: plus bare
names, so the path-first filter dropped the shadowed header and kept
its entries, and ate the blank separator. It is filtered a group at a
time now.
- tree is no longer fanned out at all: its output is one document, so a
per-mount block printed a second root, drawing and summary. It crosses
inside the generic through a new injected fact, readdir_path, a
dispatcher-backed listing offered beside stat_path.
Pinned against GNU coreutils 9.7 and tree 2.2.1 on debian:stable-slim.
* fix(traversal): fold nested du into ancestors, and stop tree disclosing an ungranted mount
Three review findings, all reproduced before fixing.
tree crossed a boundary from the mount table alone: a crossing entry's
row is synthesized as a directory without asking any backend, so the
dispatcher never saw it and could not refuse it. `tree /base` drew and
counted an ungranted `/base/private` while ls, find and du all hid it.
The child-mount lookup is now session-filtered, the way ls filters the
same fact. du keeps the raw list on purpose: an ungranted mount still
shadows the parent's keys.
du concatenated per-mount blocks, so every ancestor row carried only the
parent backend's own bytes. Pinned on coreutils 9.7 over a real tmpfs
mount, GNU folds: `du base` prints `7 base/inner` then `17 base`, and
`du -s base` prints one row. The blocks are now merged at the leaf level
and the tree derived by the same rollup a single-mount run uses, so
folding, post-order, --max-depth and -a come from one implementation.
rollup grows a `dirs` hint so an empty mount keeps GNU's `0` row, and
the mount roots are stat'ed rather than assumed to be directories,
because /.bash_history is a mount serving a single file.
tree's cross-mount stat threw a bare Error, which isWalkError does not
recognize, so a vanished entry rejected the whole run instead of being
skipped. It throws a stamped enoent now, matching python's
FileNotFoundError.
The three nested integ truths encoded the pre-fix output: ls -R had no
blank line between groups (GNU 9.7 prints one) and tree showed two roots
and two summaries (real tree 2.2.1 over a mount prints one document).
* feat(gws): add calendar and forms API passthrough, and mock routes
Adds `gws calendar` (calendarList, calendars, events, freebusy) and
`gws forms` (forms, responses) in both languages, plus routeCalendar
and routeForms in the integ GWS mock.
* feat(gcal): mount Google Calendar as day directories
Adds the gcal resource: one directory per calendar, one per local day,
one JSON file per event, named <eventId>__<HHMM-HHMM>_<Title>.gcal.json.
rm deletes the event; everything else goes through `gws calendar`.
Day bucketing uses one mount-wide zone, defaulting to the primary
calendar's, with bounds computed as consecutive local midnights so a DST
day is 23 or 25 hours rather than a fixed 24.
TypeScript mirror still to come, so spec parity is red until it lands.
* test(integ): cover gws calendar and forms, fix the service count
Adds a calendar/v1 fixture seeded through events.insert on both hosts,
and 12 cases: overlap querying, all-day end-exclusivity, q search,
freebusy, delete, form create/batchUpdate, and that a form's id is
reachable through Drive (the Forms API has no list method, so that is
the only path an agent has to an existing form).
gws --help now lists 8 services, not 6.
* fix(gcal): address codex review
- percent-encode Discovery path params, so a holiday calendar id's "#"
no longer truncates the URL at /calendars/en.usa (py + ts)
- resolve a zone-less dateTime through its declared timeZone instead of
leaving it naive, which raised TypeError when bucketed (py + mock)
- register commands and ops at module scope; there is no cycle
- validate a day is a real date, not merely date-shaped, so 2026-02-30
is ENOENT rather than a directory readdir then fails on
- centre the default window in the bucket zone, not the host's
* test(integ): cover calendar and forms mutations, fix the stale service count
* feat(gcal): mirror the calendar mount in typescript
* fix(gcal): drop the redundant close override and the polynomial regexes
* fix(notion): add GET /v1/comments, version the database object, align ntn API errors
The fake had no GET /v1/comments, only POST, so nothing could read a
comment back. Comments were also seeded with one global position while
createComment numbers by the parent's count, so a new comment could sort
ahead of seeded ones; seeding is now per parent.
databaseJson was version-blind and always answered 2025-09-03. A caller
sending Notion-Version: 2022-06-28 now gets the column schema under
properties. The MCP arm keeps the default: a tool call carries no version
header and must render identically to the REST arm.
ntn reported every API failure through the executor's generic fallback
(ntn <verb>: <message>, exit 1), dropping the status, reason phrase and
Notion's code. failure.py / failure.ts render upstream's wording, probed
against ntn 0.21.9, and every leaf is registered wrapped so a new verb
cannot miss it. datasources query now reports the data source miss with
upstream's hint instead of the database fallback's own 404.
* fix(ntn): narrow the guarded result before destructuring it
CommandFnResult is a nullable union, so annotating the awaited value as a
pair failed tsc --noEmit. The test now narrows instead: guarded returning
null would mean it swallowed the failure, which is worth failing on.
Three review findings, all of them cases where TypeScript accepted a
config Python refuses — or built something other than what was asked.
**The s3 store group was exempt from key validation.** The exemption
generalized `mounts.*.config`, which Python types as a bare dict, to
every credential-carrying block. That does not hold: an s3 group is
`S3StoreBlock(S3Config)` with extra="forbid", a strict model, so
`endpoint_urll` is a load error in Python and silently ignored here.
It now has a key table like every other block; only the camelizing is
skipped, which is all the backend's own spellings ever needed.
**`store.namespace`/`store.observer` accepted `type: s3`.** Both
languages took the config and failed later, at plane construction, with
`S3WorkspaceStateStore` raising for the planes it does not host. Both
now refuse it at the loader, with the message the store used to raise.
**A built cache store passed as `options.cache` became a RAM cache.**
Every `CacheConfig` field is optional, so a store is structurally
assignable to it; the workspace then built a cache of its own and the
supplied store never saw a read, a write, or a close. Structural typing
cannot refuse this, so `buildFileCache` says so instead. The test that
covered this path only asserted construction succeeded, which is why it
kept passing — it now asserts the rejection.
The two loader rules are pinned in the shared fixture, so both languages
must agree; each was proven red before the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ownership was spelled two different ways, and TypeScript was missing
the half Python has.
Python names ownership: `Workspace(store=..., owns_store=True)`, set by
the config loader and the daemon route beside the store they build.
TypeScript had no such knob — `owned = options.store === undefined` —
so a store built from a `store:` block, or the daemon's own disk
default, was always treated as borrowed and never closed. For redis or
s3 that is a client that is never quit, once per workspace the daemon
creates. Adds `ownsStore` and sets it at both build sites.
The cache had the mirror-image problem: `options.cache` accepted a
built store *or* a config and inferred ownership by duck-typing on
`.get`, which is neither Python's model nor a rule anything else here
follows. It now takes config only, as Python does, so the workspace
always builds the cache and always closes it — `resolveFileCache` and
`ownsCache` go away with it. A store core cannot build still reaches
`buildFileCache` through `registerFileCacheStore`, the seam node's
redis store already uses.
`WorkspaceArgs.options` was a hand-copied subset of `WorkspaceOptions`;
it is now the type itself. That list is what once dropped `clis` and
`guards` between the config and the daemon.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Self-review pass over the PR, for special cases that should have been
general rules and for shapes that stopped earning their keep:
- The s3 exemption was spelled `block.type === 's3'` in two places
(skip-camelize, skip-key-check) that encode one rule: a group whose
type names a backend carries the backend's config, not ours. Named
once as BACKEND_CONFIG_TYPES; a second such backend now joins a set
instead of needing both sites found again.
- `buildCache` had decayed to an identity cast once the workspace took
over building the store, and RamCacheBlock/RedisCacheBlock duplicated
core's CacheConfig/RedisCacheConfig field for field. Both deleted;
WorkspaceArgs.options.cache narrows to what it now actually carries.
- buildCliEntries kept its own copy of the cli key list this PR had just
added as CLI_KEYS; the store group names were spelled three times.
- absolutizeScripts: isPlainObject guards for the casts it grew, and a
comment about running before or after normalization that only one of
the two is now true.
The key tables are copied by hand from Python's pydantic models, so a
field added there would be refused here until someone hit it. Pinned the
way rejected.json pins the other direction: integ/fixtures/config/
accepted.json exercises every key of every block and both suites read
it. Proved it bites by dropping `root` from STORE_KEYS — the store case
fails with `unknown store key \`root\``.
Docs showed TypeScript constructing a RedisFileCacheStore beside a
declarative `index:`, which is the divergence this PR removed; the
snippet is now the config form, matching its own index line and Python.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two P1 review findings, both real, both introduced by this PR's own new
surfaces.
**Close declaratively constructed cache stores.** `closeWorkspace`
cleared the cache but never closed it — harmless while the only store it
could build was RAM, wrong the moment `cache: {type: redis}` became
expressible, since `clear()` itself connects and nothing then quits the
client. Python's `close_async` pairs them in a try/finally; TS now does
the same, behind an `ownsCache` flag so a store the caller built stays
theirs to close (the split `ownsStateStore` already draws).
That exposed a pre-existing leak one layer up: `configToWorkspaceArgs`
*built* the store and handed it over as `options.cache`, leaving the
daemon's redis client with no owner at shutdown. The loader now passes
the block through as a `CacheConfig` — which is what it already was —
so the workspace builds, owns and closes it.
**Normalize s3 store configuration.** An s3 store group IS an S3Config,
whose snake_case spellings do not camelize into the TS field names:
`aws_access_key_id` is `accessKeyId`, not `awsAccessKeyId`, and
`timeout` is seconds where `timeoutMs` is milliseconds. Casting the
camelized block silently dropped every credential and the endpoint, so a
configured store would have authenticated against the wrong service.
Fixed on both sides of the seam: `normalizeConfigKeys` now leaves an s3
group alone (as it already does for `mounts.*.config`, for exactly this
reason), and `buildStoreGroup` runs it through `normalizeS3Config` —
the same translation the s3 mount uses.
Both regressions are pinned by tests proven to fail without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(runtime): make the embedded python literals raw so TS stops eating their escapes
A template literal processes backslash escapes before Python sees the
text, so a lone \n inside an embedded Python string became a real
newline and truncated the literal. The symptom was remote from the
cause: every pyodide run died with pyodide_fatal_error, print(42)
included.
String.raw makes the escapes mean what they say; the emitted programs
are byte-identical, which is how the conversion was checked. The new
test compiles each wrapper on the interpreter that runs it, so a future
break reports a named SyntaxError with a line number instead. Reading
the .ts and un-escaping it by hand is not a substitute; that models the
escaping wrong and reports a false pass.
* fix(shell): splice a quoted "$@" into its word instead of replacing it
A quoted string holding a bare "$@" was replaced wholesale by the
positional parameters, so every literal in the word was discarded:
`f() { echo "saw: $@"; }; f a b` printed `a b`. Both languages.
The braced `${a[@]}` splat already spliced correctly, so `$@` and
`${@}` now take that path rather than carrying rules of their own,
which also fixes `${@}` not word-splitting at all and `${@/x/y}` acting
on the joined value. Three consequences worth naming:
- an empty splat that is the whole word yields no word, where it used
to yield an empty one (`x "${empty[@]}" y` is two words, not three)
- a slice numbers the parameters from 1, so `${@:0}` yields $0 first
- `${#@}` is untouched and still wrong (it measures the joined string);
it is a different code path and stays for its own change
Pinned against bash 5.2.37 in debian:stable-slim, not the macOS bash
3.2, which drops $0 from `${@:0}` and would have pinned the wrong rule.
* fix(shell): keep a splat whose single element is empty
`fragments == [""]` decided "no word" from the rendered text, but zero
elements and one empty element render the same, so `set -- ""` passed no
argument where bash passes one empty one. `${@}` and `${arr[@]}` too.
Track whether a splat yielded elements instead of inferring it.
The word survives iff the joined text is non-empty OR some splat yielded
at least one element. An empty expansion beside it does not count: with
no parameters, "$u$@" is no word at all, even though "$u" alone is an
empty word. All 16 combinations pinned on bash 5.2.37 in
debian:stable-slim.
The TypeScript CLI loaded the YAML, camelized it, and POSTed the
result; the daemon then ran `loadWorkspaceConfig` on that. With the
loader now refusing keys Python refuses, its own output no longer
round-tripped — `default_session_id` arrived as `defaultSessionId` and
was rejected.
Camelizing before sending was the actual defect: it made the wire
format differ from Python's, whose CLI sends `model_dump()` (snake_case)
and whose daemon validates the same spelling. Splits the loader into
`checkWorkspaceConfig{,File}` — interpolate, validate, resolve script
paths, and stop — from the camelizing `loadWorkspaceConfig{,File}`. The
CLI sends the checked mapping; the daemon does the single authoritative
load. Env interpolation stays client-side so a missing var still fails
before the round trip.
Caught by the cli-gate and cross-language interop jobs, which are the
only callers that cross that boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two Tier 1 items, both cases of a contract Python enforces and
TypeScript does not.
T1-M — core lifecycle
- `RedisIndexCacheStore.close()` had zero callers, so a mount
configured `index: {type: redis}` kept its client open past
`closeWorkspace` and the Node process never exited. `BaseResource`
now owns a `close()` that releases the index behind a `#closed`
guard, mirroring `resource/base.py`. The 29 hand-written
`close() { return Promise.resolve() }` overrides are deleted (they
existed only to satisfy the interface, and each one silently opted
its backend out of the teardown); the seven with real work call
`super.close()`. `noImplicitOverride` makes forgetting it a compile
error.
- `installDriftState` fired `void cache.remove(path)` per entry from a
sync function, so under `DriftPolicy.OFF` a read issued right after
`fromState` could still be served the snapshot bytes OFF exists to
bypass — and a rejected remove was an unhandled rejection. Adds the
sync `FileCache.evictPaths` seam Python has had (`evict_paths`):
implemented in the RAM store, a documented no-op in redis.
- `DiskObserverStore.walk` swallowed every readdir failure, so an
EACCES mid-tree yielded a silently partial `/.bash_history` and a
`clear()` that reported success while leaving files. Errors now
propagate; only a missing root reads as "nothing recorded yet".
Python's root probe used `os.path.isdir`, which has the same blind
spot, so it moves to the same explicit check. Adds
`ObserverStoreBase` (twin of Python's) so `readAll`/`close` stop
being copy-pasted into three stores.
T1-L — TS workspace-config validation and cache-as-config
- Python sets `extra="forbid"` on 13 config blocks; TS validated
unknown keys in two places and copied everything else through, so
`mount_point:`, `consistancy:` and `limitt:` were hard errors in one
language and silently ignored in the other. Adds per-block
allowed-key tables for the top level, mounts, cache, index, store
(+ groups), clis and guards, checked before normalization and
against Python's canonical snake_case spellings — a camelCase alias
TS would accept for free is just as unportable as a typo. Guards now
fail at load, as in Python. The local `snakeToCamel` (which diverged
from core's on uppercase-after-underscore and trailing underscore)
is replaced by the exported one.
- `store: {type: disk}` silently built a RAM state store, so state a
user believed was persisted was not; disk and s3 groups are now
wired, matching `_build_state_store`.
- `WorkspaceOptions.cache` took a built store where `index` takes a
config, so a programmatic TS consumer could not ask for a redis file
cache declaratively. Adds `core/workspace/workspace/cache.ts`
(`buildFileCache`, twin of `workspace/workspace/cache.py`) plus a
`registerFileCacheStore` seam — core cannot import node, so the
redis store registers itself the way runtimes already do — and
widens the option to `FileCache | CacheConfig`.
Verified with a shared `integ/fixtures/config/rejected.json` both
suites assert against, so a config accepted by one loader and refused
by the other fails a test until they agree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two call sites outside python/ that the first pass missed, both required
CI jobs:
- `runners/python/adapters.py` built its github mount with
`GitHubResource(GitHubConfig(...))`, which now raises TypeError against
the tree-taking constructor before a single case runs — the whole python
github battery was unusable. `GitHubService.resource` and `build_github`
are async; `build_mounts` awaits whatever the builder table returns, so
the other forty builders stay plain sync functions.
- `integ/root.py` indexed and splatted the coroutine from
`to_workspace_kwargs`, failing the virtual-root job with
`TypeError: 'coroutine' object is not subscriptable`.
Also corrects two comments (py adapters + the ts twin) that justified the
out-of-process github fake by "GitHubResource fetches the repo tree with a
blocking urlopen from its constructor". That is no longer true; it stays
external because both hosts share the one fake.
Verified locally against the CI fake (`--port 5098`, same --repo args):
github 20/20, github_ci 11/11, ram 2332/2332, disk 2297/2297,
ram-alias 8/8 (the alias_of branch), and `integ/root.py` all checks OK.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`GitHubResource.__init__` fetched the repo's default branch and its
recursive git tree before returning, over `urllib.request.urlopen` — the
only blocking HTTP call in the package. A constructor cannot await, which
is the entire reason that sync client existed.
It is reachable on a live event loop: `server/routers/workspaces.py`'s
`async def load_workspace` calls `build_resource` synchronously, so
mounting a github repo froze the daemon's loop for two GitHub round trips
— every other mount's in-flight I/O and the FUSE queue stalled with it.
Measured against a deliberately slow local GitHub, a 1.24s build let a
50ms heartbeat coroutine tick once; it now ticks 24 times over the same
1.24s.
TypeScript already had the answer, but not the one the audit recorded:
`GitHubResource.open()` there is a no-op, and the work lives in
`static async create` behind a private constructor, with the whole
`ResourceFactory` type declared `(config) => Promise<Resource>`. Mirror
that rather than the `open()` hook — a lazy hook would move the failure
from build time to first path resolution, which is a new divergence, not
a fix.
- `BaseResource.build`: async classmethod factory, default just calls the
constructor. Named `build`, not TypeScript's `create`, because `create`
is already an op name (make an empty file, what `touch` calls) and ops
are served by `__getattr__` — a real `create` on the class shadows every
backend's create op. Caught by the databricks_volume suite.
- `build_resource` is now async and awaits that factory through
`_instantiate`, which falls back to the plain constructor: registered
and entry-point resources need not subclass `BaseResource`.
- `WorkspaceConfig.to_workspace_kwargs` follows, matching TypeScript's
already-async `configToWorkspaceArgs`. `Workspace(**kwargs)` stays sync.
- `GitHubResource.__init__` takes the fetched tree and touches no network;
`GitHubResource.build` does the two fetches over the existing async
`github_get`. `github_get_sync`, `fetch_tree_sync` and
`fetch_default_branch_sync` are deleted.
- New `tests/resource/test_no_blocking_http.py` fails on any `urlopen`,
`urlretrieve` or `requests` import under `mirage/`, including
function-local ones.
- Docs updated on both sides; the TypeScript github pages showed
`new GitHubResource({...})`, which has never compiled against its
private constructor.
Snapshot load is untouched: github's state redacts its token, so
`requires_resource_override` always sends it down the live-override path
and never reaches `_construct_resource`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DELETE /v1/blocks/{id} is the only delete verb the public API has and the
only one the MCP tool surface exposes, but the fake had no route for it
and `ntn api` had no DELETE method, so nothing could remove anything.
`archived` is upstream's deprecated alias for `in_trash` and always
returns the same value; the fake stored two columns, so a PATCH of
`archived` left the row in database queries while `ntn pages trash`
(in_trash) worked. Now one stored bit, both spellings on the wire.
A database row's cells now ride in its page.json under `properties`.
Also: trashing a child page moved only the page row, leaving the
child_page block in the parent listing; `datasources query` took its
TSV columns from the schema where upstream takes them from the returned
rows; and two MCP-parity cases pointed at pre-data-source paths, so they
compared two identical errors and asserted nothing.
Docs: both ntn.mdx files documented the pre-#740 grammar (--page flags,
ntn blocks/comments/search). notion.mdx, the TS setup doc and both
examples predated the data-source split; the examples were broken.
* fix(ids): stamp uuid7 from the clock, not from the previous id
uuid6's uuid7 orders ids inside one millisecond by bumping the
timestamp past the previous one. The borrowed value becomes the next
call's floor, so the error compounds: a burst of 5000 ids stamps every
later one 4991 ms into the future, and it stays there until the clock
catches up. These are workspace and session ids, used as time-ordered
database keys.
Mint the id here instead, with RFC 9562's dedicated counter in rand_a:
48-bit timestamp, 12-bit intra-millisecond counter, 62 random bits. The
timestamp is only ever a real clock reading, and an exhausted
millisecond waits for the next rather than borrowing.
Drops the uuid6 dependency, which nothing else used. TypeScript already
had this right, via the uuid package's counter.
* feat(python3): parse CPython's option table, and seed sys.path on pyodide
python3's spec carried one option, -c, and a free-text rest, so every
other switch fell through to the operand slot: 'python3 -u s.py' ran
'/-u' as the script and failed, 'python3 -zz -c ...' exited 0 with the
flag swallowed, and -V and -h were read as paths.
Two spec knobs fix the class rather than the command. stop_at_operand
turns off the parser's dash-operand leniency and ends option parsing at
the first operand, which is CPython's own rule; ends_options marks the
options that carry a program (-c, -m), so the words after them are the
program's argv rather than python3's. Both mirror in TypeScript.
On top of that: -m runs through runpy, with a find_spec probe so a
missing module is CPython's one line instead of a traceback; the four
source doors each report the argv[0] CPython gives them; and the
interpreter-init switches ride in RunArgs.flags, honored where the
engine can and reported on stderr where it cannot.
argv[0] needed one more piece. The local and wasi tiers hand CPython
the program through -c, so it hardcoded argv[0] to '-c' and named every
frame '<string>'. bootstrap() re-compiles under the real name, which
fixes argv[0] and the traceback filename together.
Pyodide gains sysPath, which glob-expands and prepends mount paths once
the mounts are in place, so a vendored wheel imports without the
sys.path.append incantation. A glob that matches nothing is reported on
the run's stderr; the seed cannot report it itself, since it runs
before stderr is captured. packages, packageBaseUrl and lockFileURL
reach a prebuilt distribution, and the wrapper honors -O, -B, -W, -X,
-E and -I.
Verified against CPython 3.13.7: all 14 probed lines match on stdout,
stderr and exit code.
* refactor(spec): borrow argparse's REMAINDER instead of two bespoke parser knobs
python3's line needs opposite treatment of an unknown dash word on
either side of the script: `-zz` before it is python3's usage error,
`--foo` after it is data the script must receive. The first pass added
CommandSpec.stop_at_operand and Option.ends_options to say that, which
put two per-command dialect fields on dataclasses every command shares.
The first half is not ours to invent: it is argparse's
nargs=argparse.REMAINDER, and POSIX's own option order (GNU's permuting
default is the extension, which `POSIXLY_CORRECT=1 ls a -1` shows).
argparse spells it on the positional slot, so it goes on Operand under
argparse's name, and CommandSpec is untouched.
The second half argparse cannot express at all: `add_argument('-c')`
beside nargs=REMAINDER answers `unrecognized arguments: -u`, and CPython
parses its own command line in C for that reason. So it stays out of the
grammar. POSIX already spells the handoff `--`, and the parser already
consumes one, so a table in workspace/route inserts it after -c/-m's
value, beside the other command-name-keyed line rules. The marker is
added unconditionally: CPython passes a typed `--` through as data
(`python3 -c p -- -u` gives the program ['-c','--','-u']), so the parser
eats exactly the one added here.
Net: Option and CommandSpec return to what they were, Operand gains one
borrowed field. CLAUDE.md now states the rule as a literal test, since a
CLISpec is a CommandSpec and cannot be used to escape it.
* docs: require POSIX and argparse, not either, for a shared grammar field
POSIX alone specifies an option whose argument is a program (sh -c, and
python3's own synopsis), so an either-or rule would license the Option
field the section exists to refuse.
* fix(python3): address review on init switches, and declare the four missing ones
- pyodide restores warnings.filters after a run, so -W error no longer
follows every later line on the warm interpreter
- -OOO saturates at compile()'s max of 2 rather than raising ValueError
- -E/-I stop deleting PYTHON* from os.environ: CPython's -E stops those
variables configuring startup, it does not hide them from the program
- pyodide reports -E/-I/-s/-S through unhonoredNotice instead of faking
them; unhonored() grew a honored subset, and now reports -OO too
- declare -b, -P, -x and --check-hash-based-pycs, which CPython accepts
and mirage was refusing. -x is answered by the source resolver, since
handing it to an engine that runs code via -c would honor it nowhere
- end_options_after_program walks a short cluster letter by letter, so
-uc 'p' -v hands -v to the program, and steps over long value options
* fix(integ): argv[0] for a payload is -c on every engine, monty included
Monty's DEFAULT_PROG placeholder only applied because nothing ever told
it a name. The python3 command now supplies CPython's own answer, so
these four policy-routing steps read -c. They still distinguish monty
from local, since argv is a monty global and CPython would NameError.
Also picks up the formatting of the new flags test: it was untracked
when pre-commit ran, and --all-files goes through git ls-files.
* fix(python3): do not rewrite a shadowed name, and stop over-claiming -W and -X
- the `--` handoff is skipped when a shell function shadows python3.
bash's rule gives the function the line, and it has no CPython option
table to read a marker with. `command python3` masks the function for
its inner run, so the rewrite applies there again. A CLI cannot reach
this at all: register_cli refuses a shell builtin's name.
- an invalid -W filter is reported as CPython reports it and the program
still runs, instead of _OptionError escaping the wrapper and killing a
line every other runtime completes.
- a known -X name is reported as unhonored, since populating
sys._xoptions is all a warm interpreter can do for one. An arbitrary
name stays silent, which is all CPython does with it either.
- -O's reach is written down: the payload is compiled at the requested
level, a module imported from sys.path is not, and sys.flags stays 0.
`write_output` caught `OSError`, so a write that failed with anything
else aborted the whole command and the remaining operands were never
attempted — even in the default warn mode, where GNU keeps going. That
is every remote backend: `core/s3/write.py` forwards botocore's
`ClientError` out of `put_object` and `core/gridfs/write.py` pymongo's
`PyMongoError` out of `upload_from_stream`, and neither is an `OSError`.
The TypeScript twin catches whatever is thrown and classifies it, which
is why only python had the hole.
`error_line` already told the two apart (shared strerror for a
recognized filesystem refusal, the exception's own message otherwise),
so only the catch had to widen. `Exception`, not `BaseException`, so
cancellation still propagates. Not a swallow: every caught error is
named on stderr and the command exits 1.
The test sink now raises a non-OSError, mirroring the TypeScript sink's
plain `Error` — that alone turns four existing cases red on the old
catch — plus a case pinning the multi-operand diagnostic for an SDK
error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finishes T1-I, the `CommandIO` slots that are wired on one side only.
du (python gains it): `core/box/du/` was written and unreachable —
imported by its own `__init__` and its tests and nothing else — and
dropbox had no python du at all, so both took `DU_BUILDER`'s walk bounded
by `max_du_entries`. Past 10 000 entries that prints a partial total, a
stderr notice and exit 1. Ports dropbox's du beside box's and wires both,
matching the typescript tables.
find (typescript loses it): `core/box/find.ts` and `core/dropbox/find.ts`
were not pushdowns. Neither API has a search call to push to, so both
were the *same* readdir/stat walk the find and cp builders already run —
minus three things the builders pass and an op cannot:
- the mount's own index (they allocated a scratch `RAMIndexCacheStore`,
so every walk started cold and warmed nothing for the next command),
- the namespace stat overlay, because the native branch only applies it
when `ops.local === true` and both backends are remote,
- the symlink table `-empty` counts entries from.
The overlay half is a live divergence: `touch -d 2020-01-01 f` then
`find -mtime +5 f` finds it in python and does not in typescript. Pinned
by extending find/mtime.json's four mtprobe cases to box and dropbox;
they fail on pre-fix typescript on both targets and pass after.
The stale comment that justified the ops ("the cp builder may call find
without threading an index, unlike python") is refuted by cp.ts:47, which
reads `opts.index` and passes it to its own fallback walk.
Leaves box/dropbox with one command_io exception each (python range-reads,
typescript's core read takes no window) instead of three.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wiring `append` into python's ssh CommandIO makes the ops factory emit
an `append` op for ssh, which is what typescript has emitted all along
(`node/.../ssh/io.ts:55`). The expected table is a hand-kept literal, so
the new row has to be added by hand; the other three backends that wire
append (disk, ram, redis) already carry theirs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tee`'s spec declares `rest: {type: "path"}`, so the parser accepts N file
operands — and both generics wrote only `paths[0]` and silently dropped the
rest. `tee a b` put nothing in `b`. Identical in both languages, which is
exactly why no parity gate saw it: a shared bug is not a divergence.
GNU 9.7, pinned in docker: `printf x | tee a b c` writes all three; a failing
operand is diagnosed and skipped while the others still get the data (exit 1);
only `--output-error=exit` stops at the first failure. That last mode had never
been implemented because with one sink it was unobservable — the old comment
said as much — so it was parsed and ignored.
Deliberate divergence, documented at the call site: GNU opens every operand up
front, so under `exit` an *open* failure aborts before anything is written. A
mount has no open/write split, so operands before the failure are already
written. The two agree whenever the failure is at write time, which is what a
remote backend reports.
The `append` CommandIO slot is now wired into the generic, per the decision to
wire rather than delete it. It was declared on both adapters, wired by five
backends, and read by no builder, so `tee -a` was read-modify-write everywhere
— on ssh that meant downloading and re-uploading the whole file plus a
lost-update window. A backend with the slot now appends; the rest keep the
fallback. Where the native path runs, the resulting content is not in hand, so
the path is reported in `writes` but not in `cache`: that is how apply_io is
told to drop the stale entry instead of caching a wrong one. The slot was typed
`(...args: any[]) => unknown`, which is what let five backends wire something no
builder could consume; it is typed like `write` now.
Found while wiring it: python's `core/ssh/append.py` existed with no importer
outside the resource, so python ssh took the read-modify-write path its
typescript twin did not. Wired.
Two more real bugs on the way through:
- TS `tee -a missing-file` threw instead of creating the file. The not-found
test was `/not found/i.test(err.message)`, but `enoent()` puts the *path* in
the message, so it never matched — verified by running it. Python's
`except FileNotFoundError` was always correct. That broken test is why s3 and
gridfs had bespoke tee wrappers carrying an `exists()` pre-check (with a
`catch {}` that swallowed everything) to route around the generic. All four
wrappers — s3 and gridfs in both languages — are now wiring only, which fixes
their multi-operand bug in the same edit.
- `isEnoent` existed in three copies (`core/generic/find.ts`,
`workspace/reconcile.ts`, `executor/builtins/metadata.ts`) and in none of them
was it next to `isFsError`. One copy now, in `utils/errors.ts`.
Verified: three integ cases across 17 targets, each proven red on pre-fix code
and green after, on both hosts — the multi-operand pair fails as `DUP|` (first
operand only) and `tee -a` on a missing file fails TS-only. `xm_tee_multi`
passed all along because it is cross-mount: one operand per mount, so each
backend tee wrote its single `paths[0]`. Full core/node/browser vitest and the
python suite green; specs regenerate with no drift and the parity gate passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(notion): normalize written properties against the data source schema
The fake echoed the client's property object back, so a PATCH that omitted
`type` (which the API allows) stored an untyped value that its own reader
rendered blank. Fill id and type from the column and resolve a select name to
the whole option.
* fix(notion): persist a minted select option into the schema
The answer's option id was only usable once. A later write naming that id
alone missed the schema lookup and came back with an empty name, which is the
blank cell this PR is about. Add the option to its column when it is minted.
* fix(notion): queue a workspace's writes at the door
Two writes to one workspace interleave at any await, so both could read the
same schema and the later write would drop the other's minted option while the
page that minted it kept the id. Serialize every mutating request per
workspace; GET stays concurrent.
Resolves both codex findings on #742, plus the regression they framed and a
divergence found alongside it.
The presigned-fetch shim (core/s3/_client_browser.ts) reads only `Key` off a
GetObject input and drops `Range`, so #742's `readRange: rangeOf(s3Read)`
returned the *whole object* as the requested window — on all fifteen browser
S3-family resources (s3, r2, gcs, minio, ceph, backblaze, wasabi,
digitalocean, scaleway, supabase, aliyun, tencent, oci, qingstor, seaweedfs).
Before #742 the ops factory took the slice fallback and answered correctly.
`core/s3/stream.ts` now owns a `readRange` that branches: with the AWS SDK it
asks for one ranged GET as before, and on the presigned path it streams and
stops at `offset + size`. Adding a real Range header there is not the fix —
the header makes the request non-simple and trips a CORS preflight presigned
deployments generally do not allow — so the shim now *refuses* a Range it
cannot serve instead of answering wrongly. Stopping early only saves the bytes
if the transfer stops too: `bodyFromResponse` released the reader's lock
without cancelling, so the rest of the object arrived anyway.
Codex C2, unsatisfiable ranges: a window at or past EOF is an empty read under
POSIX and a 416 over HTTP, so wiring a native range changed the answer for the
same call. Normalized once per language in the ops factory rather than in each
reader, keyed on a shared `isUnsatisfiableRange` / `is_unsatisfiable_range`
that reads the botocore, aiohttp, httpx and aws-sdk-js error shapes. The node
S3 mock sliced silently instead of refusing, which would have let the fix look
correct without being tested; it now answers 416 like the real thing.
Codex C1, node vs browser: `merge_variants` resolves the two runtimes by
preferring node, which turned a divergence into a silently discarded value.
`check_variant_facts` now diffs the two variants for both resource-fact tables
*before* that merge, with its own stale-checked `variant_resource_facts`
exemption table. It found thirty real divergences: the browser S3 resource
declared neither `sizesAlwaysKnown` nor `storageId` where its node twin
declared both, so a browser `mv` between two mounts of one bucket saw two
separate storages. Fixed rather than exempted — the browser class gains both,
and the identity now comes from a shared `s3StorageId` so the runtimes cannot
compute different answers for one bucket. Zero exemptions needed.
Found alongside: TS `rangeRead`'s fourth argument is end-exclusive on ssh, hf,
nextcloud and databricks_volume, and in every python twin, but s3 and gridfs
read it as a length — `range_read(p, 10, 20)` returned twenty bytes from
offset ten where python returned ten. Both corrected. s3's inline Range
builder also now goes through the shared `rangeHeader`.
Verified: core/node/browser vitest and the full python suite green, both spec
trees regenerate with no drift, and the parity gate proven red on a planted
node/browser divergence and on a stale variant exemption.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(workspace): merge mount structure into the door's readdir and stat (R1)
* fix(structure): gate synthetic door answers, tighten bridge LIST errors, drop the flagged regexes
* test(integ): root ls truths carry the trailing newline the door renderer emits
* fix(structure): synthesize link ancestors like mount prefixes, feed one union to the door and the ls fact
* fix(structure): filter link names by owning-mount grant, serve granted structure under an ungranted parent
* fix(structure): ls lists structure-only dirs, nested-mount preload fails loud, fs.ts fallback gates with the synthetic prefix
* fix: fan-out gates on raw descendants, nested-mount rename raises EXDEV in pyodide
* refactor: rename ops/structure to ops/namespace_view, promote norm_dir into path utils
* fix: ls serves namespace-only directories under -R and -d, fan-out threads the child-mounts fact
* feat(notion): rebuild the fake on Prisma, migrate to data sources, align ntn to upstream
The fake Notion server is rebuilt on Prisma and its writes now land, so a
create is readable back through the mount and through the CLI. Slack's
writes never landed either; fixed the same way.
The client moves to Notion-Version 2025-09-03, the generation that split a
database into a container plus one or more data sources. The database object
no longer carries `properties`, so the mount nests to match: the schema and
the rows live under `databases/<Title>__<id>/<Name>__<ds-id>/`.
`ntn` is realigned to the real CLI verb for verb. `blocks`, `comments` and
`search` were mirage inventions and are gone; they are `ntn api v1/...`,
which subsumes the whole REST surface offline. Added `api`, `whoami`,
`auth token` and `datasources resolve`.
Every case in integ/cli/ntn.json is now asserted twice: once inside a mirage
workspace and once as the same shell line run by the real npm ntn binary
pointed at the same fake. A golden both agree on is by construction what the
official CLI prints. It caught five wrong goldens and one real fake bug
before they could be committed as correct.
* chore(spec): regenerate the spec snapshots for the four new fields
Operand gained `name` and `required`, Option gained `metavar` and `env`, and
all four serialize into every command's spec JSON. The TypeScript generator
had to learn to emit them too, or python and typescript specs diverge on all
93 commands.
* fix(ntn): address the codex review, and correct the fix it suggested
Four P2s, all real.
- TypeScript declared --filter-file and never read it, so a query with one
silently returned unfiltered rows. It now reads the path through the op
dispatcher, the way Python does and the way himalaya reads --attach.
- Both languages recognised `Header:Value` inputs and then discarded them.
Probed on the wire against the real binary: it sends them. Threaded through
as per-request headers in both.
- Malformed --data. Codex asked Python to converge on TypeScript's exit 2
usage error; probing the real binary shows neither was right, so both now
answer exit 1 with `error: Invalid JSON from --data`, and a conformance
case pins it.
- Env-backed options were filled after validation, so an option declaring
both `env` and `required` could not be satisfied by a populated variable.
The fill moved ahead of validation in both languages.
ntn api had an empty test file, which is why the first two survived. It now
covers classification precedence, headers, the body sources and the output
serializer.
* fix(ntn): satisfy CI's flake8 and the test-inclusive typecheck
E129 on a wrapped return annotation, and a ByteSource|null decode in the new
api test. Both are CI-only: the repo's flake8 is stricter than the local hook,
and `pnpm -r typecheck` covers test files that a bare -p tsconfig run skips.
* fix(ntn): match the real CLI on api body sources and JSON parse errors
`ntn api` diverged from the upstream binary in ways the conformance
harness could not see, because no case covered them.
- inline `path:=json` failures now carry serde_json's own message, from
a new scanner that also decides validity (python's json accepts NaN
and serde does not). Columns count bytes, newlines reset them, and
the recursion limit is 128.
- no object check on a body: `--data '[]'` is sent, not refused, and
any body source makes the call a POST, so `--data '{}'` no longer
falls back to GET on falsiness.
- stdin is a body source, and the three sources are validated in
upstream's order before the conflict is reported.
- refusals exit 1 for a malformed body and 5 for a line the CLI will
not interpret, both with upstream's hint lines.
- `name==value` survives a non-GET instead of being dropped.
- typescript gains PUT, which python already had.
Every byte is probed against ntn 0.21.9 and gated by 7 new conformance
cases, which the real binary is run through too.
* test(ntn): pin that the scanner never hands the engine parser a failure
`api` calls json.loads / JSON.parse only on input the serde scanner
accepted, so a document the scanner passes has to parse. A gap there
would put a raw parse error back on the command's error path, which is
what the scanner was introduced to remove. Checked over 8,145 generated
documents; the list here is the readable subset.
* fix(cli): answer group refusals in the CLI's own dialect, and let the environment through the parse
Three findings from the codex review.
- A group-level option refusal always exited 129 with git's wording,
even for a CLAP program whose own leaves exit 2. The style now picks
both: clap answers `error: unexpected argument '--x' found`, one usage
line and a footer, at every level of the tree. Probed against ntn
0.21.9 and pinned by three conformance cases.
- An option's declared environment variable was filled after the parse,
so an env-backed int went unvalidated, an env-backed choice untested
and an env-backed path stayed a raw string that FlagView.as_paths()
could not see. It now lands inside the parse, exactly where a default
does and just ahead of one, so it gets the same coercion and the same
required credit. apply_env is gone.
- `ntn pages edit --json` threw the PATCH response away and printed the
page id, on the one flag that asks for the response.
The clap renderer moved to the spec layer: the walk needs it and cannot
import a module that reaches the workspace.
Registry membership only ever said a backend could be *built*. What it
does once mounted — how stale a listing may be, whether reads are cached,
whether a `du` is pushed down to the API — was a second hand-maintained
surface with no gate at all, which is how python served up-to-ten-minute
stale listings of a live postgres schema while typescript pinned 0.
`spec/*/resources.json` now carries two more tables, diffed by
`check_spec_parity.py` with the same one-fact-per-exemption rule the
command checks use:
- `capabilities`: per registry name, `index_ttl` / `caches_reads` /
`supports_snapshot` / `sizes_always_known`, plus whether the class
overrides `storage_id` and `statfs`.
- `command_io`: per backend command package, the wired `CommandIO` slots
plus `local` / `max_glob_matches` / `max_du_entries`.
Python reads both off its classes and dataclasses. TypeScript reads them
from the source declarations (`scripts/resource_facts.ts`), because the
twins are instance fields and observing them at runtime would mean
constructing the resource — and construction is not inert:
`buildResource('github', {})` issues an HTTP request and `postgres` opens
a connection. A value the extractor cannot read as a literal is dumped as
`<expr:…>` rather than guessed.
The gate found 15 live divergences on its first run. Fixed here:
- `readRange` was wired on `disk` alone in typescript while python pushed
the window down on twelve backends, so `head -c 100` on a large object
downloaded the whole thing and sliced. Wired on the five whose read
already takes an `{offset, size}` window (s3, gridfs, nextcloud, hf,
databricks_volume) via a new `rangeOf` adapter, and `gen-specs` now
refuses to emit when a reader takes a window with no slot to hand it —
the twin of python's `test_read_range_optin.py`, which is why python
was fully wired and typescript was not. The check keys on `offset` *and*
`size`: postgres pairs `offset` with `limit` to mean SQL rows.
- python github ignored `SCOPE_ERROR` and refused globs at 10001 matches
instead of 5001; the constant had no importers.
The remaining ten are documented in `parity_exceptions.json`, each naming
one resource and one key: box/dropbox `du`+`find` and the seven readers
with no window argument (T1-I), github `supports_snapshot` (snapshot
redesign), lancedb `caches_reads` (computed per URI), notion/hf `find`
(slot on one side, bespoke command on the other), ssh `append` (wired in
typescript, read by no builder on either side).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(tar): drop the GPL bzip2 dependency, and parse tar with modern-tar
compressjs is GPLv2-or-later and sat in dependencies of the Apache-2.0
mirage-node package. Replace it with seek-bzip (MIT), which only decodes,
so a CompressionCodec may now be decompress-only and tar -cj exits 1 with
"tar: bzip2 not supported" the way browser core already answers for an
unregistered codec. Python keeps both directions through stdlib bz2.
Replace the hand-rolled ustar reader and writer with modern-tar (MIT,
zero dependency, web streams) behind the existing tar_helper seam. The
hand-rolled version truncated any name past 100 bytes instead of using
the ustar prefix field or a PAX header, so a deep member extracted to the
wrong path, and it read a PAX extension block as a member, so tar -t on
any archive GNU or Python wrote listed a phantom ././@PaxHeader row.
writeTar and readTar are async as a result.
* test(tar): cover -h, and round-trip bodies through the xz codec
-h had no test at all: the name a link member carries is the same whether
tar stored the link or followed it, so the existing symlink test could not
tell them apart. Read the archive back instead and assert the member kind,
in both directions, plus the unreachable-target case.
The xz test created and listed, which only proves headers survived. Extract
too, so the decompressed bodies are checked byte for byte.
* fix(archive): tar and zip archive a directory, and -C re-bases the operands after it
tar -cf d.tar d exited 1 with "tar: <hostpath>/d: Is a directory": every
operand went straight to read_bytes, with no isdir check and no
recursion, so the most common tar invocation there is could not work.
zip -r <dir> had the identical defect. Both now walk.
Create is a two-phase pass in each: decide every member, then write.
Both plans are built on one traversal, scan_operand / scanOperand
(generic/archive/walk), which merges three sources no single one can
see: the backend walk (find's walk_find, so an entry is classified
through stat and never by name), the namespace's symlinks, and the
mount table. It reports paths, never names, because naming is where the
two formats part company; the two things they disagree about in the
traversal itself are parameters, so tar passes recurse=True and
dereference=-h while zip passes recurse=-r and dereference=not -y.
A directory is its own member, so an empty one survives a round trip,
and both extractors now mkdir for one. A symlink is a symlink member
under tar (SYMTYPE, target in linkname) and under zip -y (mode 0120777).
An unreadable operand is reported in virtual path space with the
archiver's own wording, because the raw IsADirectoryError was leaking
the host path behind a disk mount.
tar -C was ignored: `tar -czf /work/out.tgz -C /work/check my_paper`
failed as "paths span multiple mounts", the same defect class as unzip
-p in #725, because the operand resolved against the session cwd and
the router then saw a phantom mount span. -C is not a flag the command
reads once, it is a chdir for the operands typed after it, so it is now
declared in the spec (CommandSpec.operand_base) and resolved by the one
component that walks the line positionally: the parser reports a base
per word and the classifier resolves each operand against it.
MountView is how a command sees mount boundaries, offered the way
LinkView is (name a `mounts` parameter, nothing else). A traversal that
renders lines gets this free from the executor's fan-out; one that
emits a single binary object cannot, which is why the archivers read
the table themselves. A descendant mount is not crossed: the mountpoint
stays an entry and its contents are dropped with GNU's
--one-file-system wording, since descending would archive by accident
what MountRootPolicy now refuses on purpose for tar, zip and cp in a
source slot.
Semantics pinned against GNU tar 1.35 and Info-ZIP 3.0 on
debian:stable-slim, including Info-ZIP's inverted defaults, its
anchored -x, its silent leading-slash strip, and "Nothing to do!"
exiting 12 with no archive written.
* chore(spec): the dumps state what a command declares, not what it defaulted to
Both generators emitted every field of every dataclass, so `truncate`,
which declares one thing (`-s`/`--size` takes a string), spent 27 lines
of spec body restating 21 defaults that read identically in all 93
files. `zip` was 198 lines for five flags, 11 of the 13 keys on each
option being defaults.
Anything equal to its default is now dropped on both sides. `type`
survives even at its default, because what a token is is the first
thing a reader looks for, and `"rest": {}` says less than
`"rest": {"type": "path"}`.
The defaults come from the dataclass fields in python and from a
default-constructed instance in typescript, rather than from a table
either side could let drift. The two must drop exactly the same keys,
and the parity gate reports every command if they do not.
This narrows that gate's blast radius rather than widening it. When
`operand_base` was missing from gen-specs.ts, python emitted the key in
all 93 files and typescript in none, so parity reported ~95 divergences
to sift. The same bug now reports one, on tar, which is the only
command that sets it.
check_spec_parity's option-diff renderer keyed options by `o["long"] or
o["short"]`, which raises once an option carries only the spelling it
declared.
279 files, 22113 lines of restated defaults gone.
* fix(archive): strip the mount prefix by scan, not by a backtracking regex
childSpec measured the backend key with `.replace(/^\/+|\/+$/g, '')`.
The trailing alternative backtracks on a run of slashes, which CodeQL
flags as js/polynomial-redos, and the input is a resource path a mount
supplies.
utils/slash.ts already has stripSlash, which walks the two ends by
charCode and cannot backtrack. Use it. slash.test.ts pins it against
the regex it replaces.
* fix(archive): the five symlink and mode bugs the codex review found
All five reproduced against a live workspace first, then pinned against
GNU tar 1.35 and Info-ZIP 3.0 on debian:stable-slim.
A symlink operand never reached the planner as a link. tar and zip were
absent from NO_FOLLOW_COMMANDS, so the router rewrote the operand
through the link table and `tar -cf o.tar link` stored a regular file
holding the target's bytes where GNU stores a symlink member of size 0.
It also skipped the planner's cross-mount refusal, since by then there
was no link left to refuse. Both archivers now lstat, and carry no
DEREFERENCE_FLAGS entry on purpose: -h and -y are the planner's to read.
Only the last -C was checked. `tar -cf m.tar -C missing x -C good y`
reported `x: Cannot stat` and still wrote an archive holding y, where
GNU chdirs at each -C and dies at the first it cannot enter. The option
accumulates now and the planner walks the list, so the first bad one is
fatal and no members are written.
Two links to one target were called a loop, and a real loop was not
caught at all. Both were the same mistake: an operand-wide `seen` set
doing detection the namespace already does properly under a hop limit.
Deleting it archives both names, as GNU and Info-ZIP do, and catching
the CycleError that resolve raises turns a genuine cycle into one fatal
problem per member with GNU's "Too many levels of symbolic links",
keeping the directory entry and exiting 2 instead of throwing out of
the planner for a bare exit 1.
The mount-root refusal denied member selectors. Under -t and -x an
operand names something inside the archive, so `tar -tf a.tar data`
was refused as busy when `data` happened to spell a mount. Gated on
create mode now, dashless first word included.
A test asserted the loop bug rather than catching it; it is replaced.
Five integ cases cover the lot end to end in both languages.
* test(archive): build the test LinkView's stat through FileStat, not a cast
integ-shared-py passes build-packages: false — the action's own note says
the python host needs node only for the tsx fake servers, which import no
mirage package. The typescript *runner* does import them, so the self-test
died on ERR_MODULE_NOT_FOUND there. It moves to integ-shared-ts, the job
that builds the packages; the setup action installs the python venv
unconditionally, so that job has both halves.
Reproducing it locally exposed a false pass: `strict (ts)` asserts a
non-zero exit, and a crash-on-import is also non-zero, so it went green
for the wrong reason. A smoke check now proves the runner starts (an
unknown facet exits 2 without running a case) before any exit code is read
as a verdict, and run_typescript keeps the first stderr line mentioning an
error so a failure names its cause instead of printing a bare exit code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
P1 — the database battery ran without --strict. My sweep that added the
flag matched single-line invocations, and integ-database's two commands
use a folded YAML scalar, so they were missed: losing mongodb, postgres,
chroma or qdrant to a renamed variable would still leave both jobs green.
Both now pass --strict; a re-scan that handles multi-line invocations
confirms every runner call in the workflow now carries it.
P2 — the gate self-test silently skipped its typescript half. The `integ`
job installs python but neither node nor integ/node_modules, so the
typescript assertions never ran while the step reported success — exactly
the report-green-having-tested-nothing failure this file exists to catch.
Adds --require-ts, which turns an absent tsx into a failure, and moves the
step to integ-shared-py, the job that installs both hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dify/find, mongodb/rg and postgres/rg had no provision while their generic
siblings on the same backend did; routing the hand-written command tails
through the family catalog gives them one, so `has_provision` flips to true
for those three in all three spec trees.
Spec parity: 93 commands match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught two things the local run could not.
pre-commit: `pre-commit run --all-files` only checks TRACKED files, so
gate_selftest.py was never linted locally while it was untracked. autoflake
drops an unused import and yapf reformats it.
--strict: the core facet is deliberately split across jobs (postgres and
mongodb run in integ-database, chroma/qdrant/lancedb in integ-data, notion
and nextcloud in their own steps), so integ-shared-{py,ts} skip six
services by design and --strict had no way to hear that. Adds --allow-skip,
which tolerates a declared skip, still fails an undeclared one, and rejects
a service name that does not exist so the list cannot rot into silently
widening what --strict accepts. The two --facet core invocations name the
six; every other --strict invocation needs none. gate_selftest covers both
directions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Items 11, 12 and 14 of the parity plan, plus the backend half of item 11.
Item 14: both runners' floor guard is facet-only and all-or-nothing, so a
facet that loses SOME targets still exits 0 — measured at 161 -> 71 passing
with TRELLO_ENDPOINT unset, still green. Adds a `services` table to
targets.json replacing both hand-rolled service->env if-chains (13 py / 21
ts, already drifted), `--strict` on all 16 CI invocations, load-time gates
for duplicate case ids and unknown target refs, a zero-pairs floor in
parity.py, and gate_selftest.py asserting all of it on both hosts in CI.
`check` becomes additive (expect.check) so a case can pin stdout AND the
post-condition; the 15 existing cases are migrated.
The duplicate-id gate found a live bug: unix/lookup/type.json and
bash/type/* shared six ids, and parity.py keys rows by (target, id), so one
of each pair was silently dropped from the py/ts diff.
Item 11: TS Limit.aggr gets a mapped-type field table (a fifth bound is now
a compile error); with_default_provisions lands in python and is applied to
history and then to every backend. Of 39 hand-named provisions per
language, 21 were pure catalog duplication and now come from the catalog
(verified 21/21 land on the identical estimator); 14 are genuinely bespoke
and stay. mongodb/rg, postgres/rg and dify/find had no provision at all
while their generic siblings did — now pinned by new integ cases.
Item 12: cut's legacy_flags shim, cp's find_type knob, grep's unreachable
files_only arm, OpRecord.rel_path + mount_prefix, the duplicate
chainStreams, the unreachable root vitest.config.ts (which also carried
passWithNoTests: true), and the dead mongodb/postgres _provision modules in
both languages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(flags): every command reads the bag through a spec-bound FlagView (#292)
The 152 raw `opts.flags.x` reads under `commands/builtin/` now go through
`new FlagView(opts.flags, spec)`, which throws on a name the spec does not
declare instead of reading it as false. The eslint rule that already
enforced this for `generic/*.ts` now covers `commands/builtin/**`, so the
next one fails CI rather than shipping.
Three things the guard caught on the way in:
- `bc -l`, `date -I`, `wget -O` and `email/rg -l` each read a key the
parser can never emit -- those shorts are disambiguated to `args_*`
(`AMBIGUOUS_NAMES`) -- and three test files pinned the same dead shape.
- `generic_bind/provision.ts` carried the TypeScript twin of the Python
`touch -r` bug fixed in #733: the shared write-metadata estimator read
`r`/`R` blindly, so `touch -r REF` (a reference file, not recursion)
reported UNKNOWN precision. Provisions now receive the invoked command's
spec (`CommandOpts.spec`, mirroring Python's `spec=` keyword) and resolve
by spelling, so head's `bytes` and tail's `c` need no hardcoded guess.
- `seq -w`'s string branch was unreachable: the spec declares it boolean.
cp/mv stop declaring their own flag-bag type. `parseCpFlags`/`parseMvFlags`
and the four helpers take a `FlagView` the way `parse_cp_flags` does in
Python, the local `export type Flags` is gone, and the builders and the
crossmount relays build the view with `specOf('cp'|'mv')`. The bag's value
type is the already-exported `FlagValue` everywhere -- 70 inline copies of
`string | boolean | number | string[]` across 43 files, plus the private
alias in `cross_mount.ts`, now name it.
Deliberately not widened to include PathSpec: Python's executor promotes
PATH-typed flag values, TypeScript's `command/flags.ts:91` keeps the
resolved virtual-path string, and no site in the tree writes a PathSpec
into the bag. cp's PathSpec-tolerant read stays, with a comment pointing
at the Python promotion it is reserved for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(flags): three Python bugs the raw bag reads were hiding (#292)
Converting the TypeScript wrappers to a spec-bound FlagView surfaced the
same rule violations on the Python side, and each one hid a live defect.
- **`--desc_file`/`--text_file` never worked on trello.** A PATH-typed
flag reaches a Python command as a PathSpec (the executor promotes it),
so `fl.as_str("desc_file")` read it as absent while the guard beside it
used the raw `_extra.get` and saw the value -- the branch was entered
and the file was never read. The eight trello wrappers now build the
view and read path flags through a shared `file_operand` helper.
- **`gzip -1` compressed at zlib's default.** `extract_level` looked the
bag up by the bare digit, but `-1` is the one digit the parser
disambiguates to `args_1`. `gzip -2`..`-9` worked, `-1` silently did
not. The native test passed because it only asserted `len(-9) <= len(-1)`.
- **`cp --backup --suffix=` used an empty suffix**, naming the original as
its own backup, where GNU 9.7 falls back to `~` (pinned with docker).
TypeScript already had this right, so the fix goes to Python and both
sides now pin it.
`test_no_raw_flag_reads.py` only matched `flags.get(` -- the `kwargs` and
`_extra` names a wrapper collects the same bag into slipped past, which is
how all of the above survived. It now matches all three (and exempts
`config.py`, which answers --help/--version off the bag as spec layer).
qdrant/lancedb stop fishing `cwd` out of `_extra`, curl's `-o` and unzip's
`-d` stop claiming to be `str`, and CLAUDE.md records the PathSpec rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): pin gzip -1, cp/mv empty --suffix=, touch -r provision (#292)
The three flag fixes earlier in this branch were covered by unit tests
only, so nothing proved them across the 20 backends the battery sweeps.
- `unix/gzip/1.json` mirrors `9.json` for the one digit the parser
disambiguates to `args_1`, which is the one that was broken.
- `cp`/`mv` grow an empty `--suffix=` case pinning the GNU 9.7 fallback
to `~` (docker-pinned), next to the existing `.bak` cases.
- `unix/provision/basic.json` grows `prov_touch_r`, which sits between
`prov_rm_r` (unknown) and `prov_mkdir` (exact): before the fix the
reference file read as recursion and the estimate degraded to
UNKNOWN. Its op count is 2 -- the provision layer scopes every
PathSpec in the command line, reference file included -- and both
hosts agree on that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): serve trello from a shared Prisma fake, the way slack is (#292)
Trello was the last write-capable service whose fake ran per-host: the
Python runner started `server/trello_server.py` in-process while CI
started a second copy of the same file for the TypeScript runner. Two
processes seeded from one fixture agree only as long as nobody edits
one of them, and neither host could observe what the other wrote.
`server/trello.ts` replaces it, built the way `server/slack.ts` is:
Prisma + SQLite, `db push` into a fresh file per instance, seeded from
`fixtures/trello/v1.json`, one shared process both hosts call over
TRELLO_ENDPOINT with POST /reset rolling back to the fixture. Writes now
persist in a store rather than a dict, so a card created by one host is
visible to the next command instead of dying with the process.
Every collection the API returns has a caller-visible order, so each
table carries an explicit `seq` and is read back with `orderBy: seq`;
membership (card members, card labels) is rows rather than JSON arrays
so an idempotent add is a `findFirst`, matching the endpoints. Writes
draw ids from one counter and stamp one date, exactly as the fake they
replace did, so `crd_new_1`/`cmt_new_2` still number the same way.
Verified by diffing both servers endpoint for endpoint -- 27 requests
covering every route, the 404 bodies, a create/update/assign/label/
comment/unlabel write sequence, an idempotent re-add, a unicode card
name, and the reads after those writes -- all byte-identical.
The datasource env var is now INTEG_DB_URL (it backs two services, not
one) and `slack:setup` is `prisma:setup`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(trello): --desc_file/--text_file address the mount, not the URL (#292)
`resources/trello/cmd.json` only ever ran reads, so no trello write was
covered on either host. Adding that coverage found the other half of the
`--desc_file` bug this branch already fixed once: reading the flag now
works, but the path handed to the backend was still the virtual one.
`core/trello/read` keys off the mount-relative path and quotes the
virtual one in an ENOENT. Both hosts were passing the virtual path as
the key, so `/board/workspaces/...` looked for a `board` collection that
does not exist and every read missed -- an in-mount operand died with
ENOENT, an off-mount one with "Operation not supported". Python now
carries the whole PathSpec through `file_operand` and reads
`resource_path`; TypeScript, whose bag holds only a string, strips the
prefix with `mountKey(filePath, opts.mountPrefix)`. The shared
`resolve_text_input` is generic over the path shape so the reader can
take whichever one its backend is addressed by.
The same new cases caught a divergence in the API error text: TypeScript
named the failing endpoint, Python did not. Python now names it too --
an agent reading a 404 needs to know which call made it.
`resources/trello/write.json` covers create (inline desc), create into a
list, update by `--desc_file`, update by inline desc, comment by
`--text_file`, comment by stdin, comment append, assign/label/move,
unlabel, the off-mount refusal, and the 404. They run after every read
pin (seq 940010+) because they mutate the board the size pins measure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(tar): accept GNU's old option style
`tar xzf a.tgz` failed with "must specify -c, -x, or -t" because a first
word with no leading dash was read as an operand. It is a cluster of
option letters whose arguments follow as separate words, expanded now by
expand_old_style / expandOldStyle before anything else reads the line.
Per letter, not one -xzf token: a cluster may hold several
argument-taking letters (tar xfC a.tgz out) and may continue past one
(tar cfz a.tgz f gzips), neither of which getopt clustering can express.
* chore(spec): regenerate spec dumps for old_option_style
* refactor(pyodide): serve mounts from an Emscripten filesystem, not a python shim
Replace the 425-line python source string the runtime injected with a
filesystem mounted through pyodide.FS.mount, so interception moves below
the interpreter instead of rebinding names inside it.
This fixes two bugs the shim shipped. os.open/os.write/os.fdopen and a
bare os.truncate recorded nothing, because the shim rebound 13 names and
those were not among them, so a low-level write applied to guest memory
and was dropped at exit 0. And a cross-mount rename raised errno 18,
which is EDOM under pyodide's musl numbering, so a guest comparing
against errno.EXDEV (75) never matched; the old test asserted the
message string python rendered from that same literal, so it could not
fail.
Every prefix is re-seeded on every run. The conformance rows covering
post-boot seeding only passed before because the shim's lazy backfill
used run_sync, and vitest enables JSPI while production does not.
A file the mount lists but will not serve now becomes a node that
refuses to open with EIO, rather than a hole an append would fill by
replacing the file.
* refactor(pyodide): name the filesystem package vfs, not fs
Mirrors the python side, where the module holding the guest filesystem
is vfs.py beside runtime/vfs.py. Leaving the package as fs/ while its
entry module became vfs.ts would have kept the mixed spelling the
rename is removing.
* refactor(pyodide): name the sentinel and the whence values
Number.MAX_SAFE_INTEGER appeared five times as the has-not-written-yet
sentinel, and llseek compared whence against bare 1 and 2. Both are now
in constants.ts beside the mode bits, matching the python side where
READONLY_HINT moved to wasm/constants.py.
* refactor(runtime): one mount op vocabulary for the typescript sandbox runtimes
New runtime/vfs.ts holds RuntimeVFS, planFlush and the mount routing that
pyodide, quickjs and monty each had their own copy of, mirroring the python
core. BridgeDispatchFn gains APPEND, so a handle that only extended a file
ships its tail: eight 3-byte appends now cost 24 bytes rather than 536, and
the two amplification rows in the conformance suite flip from it.fails to
it. runtime/config.ts takes coerceRuntimeConfig out of base.ts so a typed
config has the same home in both languages.
monty.ts becomes a monty/ package (binding, constants, errors, osaccess,
runtime, vfs) matching the python split, and MontyVFS gains the negative
cache python already had: monty asks whether a path exists on nearly every
guest expression, and each miss was costing a fresh listing.
mirage_bridge.ts splits into vfs/journal.ts and vfs/preload.ts, and
js/mirage_fs.ts becomes js/vfs.ts, so every module holding a guest
filesystem is named vfs. Dead MontyVFS.mountOf dropped in both languages.
* fix(pyodide): refuse a root mount, mount nested prefixes parent-first
Three review findings.
A `/` prefix stripped to the empty string, which Emscripten takes as a
detached pseudo-mount no path reaches: the guest kept reading and writing
MEMFS, and a write reported success the resource never saw. `/` is already
MEMFS's own mount root holding the stdlib, so there is nowhere to put it;
the prefix is now skipped with one warning rather than silently dropped.
prefixes() is longest-first, which is what routing wants and the reverse of
what the mount table wants: with /data/ and /data/inner/ the child mounted
first and the parent then mounted over it, orphaning the child. Unmount
deepest-first, mount shallowest-first.
MontyVFS lives as long as the runtime while python rebuilds its
MirageOSAccess per run, so the negative cache added here outlived the
command it belonged to and hid a file another writer created between two
monty runs. Reset it at the top of run and eval.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
pytest only honors pytestmark in test modules, so the redis suite
crashed on RedisStore's URL parse instead of skipping on machines
without REDIS_URL. The gate moves into the shared store fixture, and
the registry raw-kwargs test gets the same skipif plus the env URL
instead of a hardcoded localhost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two edge cases pinned against coreutils 9.7 and fixed in both
languages: 'no properly formatted checksum lines found' keys on the
parsed-line count (a malformed line next to an --ignore-missing skip
misread as it) and still prints under --status; 'no file was verified'
fires whenever --ignore-missing leaves zero OK lines — mismatches
included — follows the WARNING block, and --status silences the text
while its exit 1 stands. Also hoists the dev workspace-test helper to
module scope per the no-nested-functions rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dict.pop override gains the supertype's overloads (mypy), the dify
zulu formatter narrows to its number/string domain instead of
String(object) (eslint no-base-to-string), and the awk propagation
test throws from the stream factory instead of a yield-less generator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TS dify had ported only python's stat_light: no document-detail fetch,
so document_id, refreshed tokens/indexing_status, detail-derived
source_size, and the updated_at mtime were missing. The heavy stat now
mirrors python field-for-field (second-precision zulu mtime included),
with the same cost containment: a new opsOverrides on the generic
command factory (the TS twin of python's ops_overrides) pins ls to the
light stat, and find upgrades to the heavy stat only under -mtime.
df_prov_ls's net=0 pin stays green, and a new df_stat_mtime case pins
'stat -c %y' on both hosts.
rm /dev/null raised an uncaught KeyError('/null') in python and
silently no-opped in TS while rm -v printed a false removed claim.
Both stores are now a synthetic null/zero overlay with tombstones over
a real backing map — root-on-linux semantics, pinned in docker: the
removal succeeds and the path is genuinely gone, a later redirect
recreates it as a regular file, and writes to a live /dev/null still
vanish. /dev stays excluded from snapshot capture on both sides.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TS ls/tree/rg/checksum/awk caught every error class where python
narrows to (OSError, ValueError)/FileNotFoundError, so auth failures
and transport errors on those paths vanished from listings or were
laundered into GNU-shaped 'cannot access' lines. A new isWalkError
(isFsError + the noMount stamp, python's ValueError twin) narrows all
seven ls sites, tree's walk, rg's dir probe, and checksum -c; awk -f
mirrors python's absence-only catch. Python's own loose sites tighten
the other way: grep/rg per-entry warnings catch WALK_ERRORS instead of
Exception, and the existence probes in builders/common catch
MISS_ERRORS.
checksum -c also becomes GNU-complete (coreutils 9.7 pins): recorded
names resolve against the cwd as PathSpecs — a relative name used to
reach backend readers as a bare str and die on .virtual — and stderr
now carries the per-file strerror lines (kept under --status), the
prefixed WARNING block in GNU's order, per-line --warn diagnostics,
and the no-properly-formatted-lines / no-file-was-verified fatals.
awk -f resolves relative program files against the cwd too instead of
the mount root. The two tier2 integ pins move to the GNU behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python's generic find read paths[0] and dropped the rest, so
`find /a /b -name x` returned only /a's matches on all 12 backends
that wire a native find op (the readdir walk already looped). The
generic now owns the per-operand loop — GNU semantics pinned on
findutils 4.10.0: operand order, duplicates walk twice, a missing
operand gets its diagnostic and exit 1 while every other operand's
rows still print (the old path discarded them with the stdout).
The per-root -path tree is stamped into a local, so the shared
FindArgs stays unprefixed, and bespoke wrappers (github, chroma,
dify, history) inherit the loop through generic_find.
TypeScript looped already but re-sorted across roots at the end,
interleaving operands; the cross-root sort is gone.
integ/unix/find/roots.json pins operand order, duplicate roots, and
partial output past a missing middle operand on both hosts (22
targets; the order case provisions its own subtree since shared
directories accumulate per-target state).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five private TS guessType copies (box/gdrive/dropbox/gmail/email) drift
from the shared table py already uses: 8 typed extensions read as BINARY
(jsonl/tsv/py/js/ts/yaml/yml/toml), so the same file stats differently
per language. Py gmail/email guess through stdlib mimetypes, whose
platform tables make .ts video/mp2t and .gz unknown. All seven sites now
import the shared table; log->TEXT and gzip->GZIP (the private copies'
two extra keys) join it on both sides. The qdrant/lancedb 4-entry image
maps fold into a shared image_type_for_extension, and FILE_MIME_MAP
drops the parquet/orc/feather/hdf5 keys unreachable since #651.
integ/fixtures/filetype/tables.json now pins every table; both unit
suites assert equality so one side cannot drift alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex review on #728.
A relative operand after os.chdir into a mount missed every prefix check,
so the mutation (and, already on main, the write) stayed in MEMFS and the
run exited 0. Paths now resolve through abspath before the mount test.
An append whose base read failed for any reason treated the base as empty,
so a file that exists but is momentarily unreadable would be replaced by
the tail alone. Only a confirmed absence starts empty now, via the
repo's own isMissingPath.
The replay continued past a failed mutation, which could apply a later
entry against a prerequisite that never landed (a rename moving a stale
temp file onto its destination). It stops at the first failure and reports
how many entries it dropped.
The shim patched only open, listdir, stat and scandir, so every mutation
spelling mutated MEMFS and never reached the mount, and an append-mode
open on a file MEMFS had not seen flushed its whole buffer over content
the run never read.
Patch the os primitives instead of a spelling list (makedirs, pathlib and
shutil all reach the mutation through os), record an append as its tail
rather than the whole file, and backfill on non-truncating writable opens.
Guest mutations now go into an ordered journal that the runtime replays
after the script, so a write-then-rename lands the way it ran. None of it
consults the bridge inline, so mutations need no JSPI.
RFC 3501 compares SENTON/SENTBEFORE/SENTSINCE (and ON/BEFORE/SINCE)
"disregarding time and timezone", so a message written 05 Jan 23:30 -0500
answers a search for the 5th. Converting to UTC filed it under the 6th, and
`himalaya envelope search date 2026-01-05` then selected mail its own
directory did not contain.
Python drops the UTC conversion, restoring what it did before this branch.
TypeScript was always UTC and is the side that was wrong: `Date` keeps only
the instant, so the stated offset is parsed back out of the string (numeric
plus RFC 5322's obsolete zone names) and added before reading the fields. A
header with no zone at all reads its local fields, which hands back the wall
clock exactly as written.
Verified identical output between the two languages across nine timestamps
under TZ=UTC, America/New_York and Asia/Tokyo.
A message with no parseable Date: header fell straight to 1970-01-01, so
every undated message collapsed into one directory and /mail/<FOLDER>/<date>/
carried no information. The header still wins (it is what himalaya's SENT*
date conditions search on); INTERNALDATE fills the hole, epoch only if
neither parses.
Metadata is now requested ahead of the body in the FETCH item list: a server
may answer items in the requested order, and anything after BODY[] lands on
the line behind the literal where the response parsers never look.
Python normalizes offsets to UTC, matching TypeScript and both gmail
backends; it previously kept the sender's offset, so 23:30 -0500 bucketed a
day apart between hosts.
INTERNALDATE is transport metadata, not part of the message, so it is
dropped in the single renderer. The five other places that serialized a
fetched message by hand (himalaya read/list/search, email rg, searchAndFormat)
now route through that renderer too, so `himalaya message read` cannot drift
from `cat` of the same file.
* refactor(runtime): declare the default captures once per language tier
* test(runtime): conformance suite for runtime mount access
* chore(deps): bump pypdf to 6.15.0 for two DoS advisories
uv audit rejects pypdf 6.14.2 over GHSA-fp3f-mc75-235c (large
/ToUnicode streams) and GHSA-fwg2-594c-jp42 (large CID font width
ranges), both fixed in 6.15.0. pypdf is transitive only —
openhands-tools -> browser-use -> pypdf, nothing in the tree imports
it — so this is a lock-only bump that leaves the rest of the
resolution untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* style: wrap the monty rmdir command under the column limit
flake8 caught the one-liner at 80 columns; yapf cannot split a string
literal, so it passed the formatter and failed Check PEP8. Split it at
the same point as the neighbouring rename rows, which leaves the
concatenated command byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(runtime): assert the verification command itself succeeded
An absence want was read off the stdout of a check that was never
required to succeed, so a mutation that damaged the mount could certify
itself: `ls /nope` answers exit 2 and empty stdout, and "gone.txt is
absent" holds for every x in an empty string. Verified against the
suite before the fix — a row whose verifier cannot succeed passed.
Both helpers now require exit 0 from each check before inspecting its
output. Positive wants were already safe, since empty stdout fails the
substring, but they get the clearer message too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(unzip): select archive members, exit 11 on unmatched filespecs
* fix(unzip): match filespecs bytewise, read each selected ZipInfo
Pinned against Info-ZIP 6.00 on debian:stable-slim:
- `?` stands for one byte, not one code point, so `?.txt` misses
`é.txt` and `??.txt` hits it. Python matched code points and
TypeScript matched UTF-16 units; both now match encoded bytes.
- A name lookup resolves every duplicate member to the last one, so
`unzip -p a.zip dup.txt` served the same payload twice. Read the
selected ZipInfo instead.
Test callbacks move to module scope.
* feat: $'...' quoting, git log/show formats, himalaya --attach + byte-exact MIME parity (py+ts)
Shell:
- $'...' ANSI-C strings decode per bash 5.2 (docker-pinned): full escape
table, octal/hex/unicode/control forms, segment-only NUL truncation;
$"..." keeps plain double-quote semantics. New shell/escapes decoder,
expansion branches, allowlist sites, dollar-quote-aware backtick scan.
- Quoted case patterns match like bash: patterns stay nodes and a new
expand_case_pattern renders quoting into the fnmatch dialect (quoted
segments literal, unquoted globs live, backslash escapes, expansion
results live unless double-quoted, no word-splitting).
- export/local/declare/readonly accept quoted assignment operands; a
bare $ word survives as a literal argument (echo $).
Git:
- log gains --all (multi-root walk with annotated-tag peeling) and
--format/--pretty: oneline/short/medium/full/fuller presets,
format:/tformat: templates, ~20 placeholders incl %d decorations in
git's exact order; unsupported presets refuse honestly.
- show gains --stat (full diff.c show_stats geometry), -s/--no-patch,
--name-only, --no-ext-diff, --format; oracle-verified byte-identical
to real git in both languages.
- commit report counts binary files as files with zero lines (shares
diffstat's NUL sniff); fixes a TS-only miss of mode-only changes.
Himalaya:
- --attach wires up: repeatable path-typed flag (first leaf-level
type="path" option; python reads via new FlagView.as_paths), reads
through inv.ops.dispatch, multipart/mixed with a content-addressed
boundary so both builders emit identical bytes.
- TS serialization is now byte-identical to python's
EmailMessage.as_bytes(policy=SMTP): new mime.ts ports RFC 2047
encoded words (q/b chooser), unstructured/address header folding,
RFC 2231 filename params, and the 7bit/8bit/qp/base64 body
selection; fixes threading-header order, empty-subject rendering,
and multi-line header refusal. 31 shared pins
(integ/fixtures/himalaya/mime_parity.json) asserted by both suites,
plus differential fuzzing against the python oracle.
Docs list the new git capabilities and remaining deliberate limits;
51 new integ cases across bash/git/himalaya targets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: codex round + generalize pattern quoting, multi-line strings, MIME to core
Codex review fixes (all pinned against real git 2.37/2.54, bash 5.2
docker, and CPython 3.12):
- git format: empty entries keep their separators under format: and
their terminators under tformat: (only an empty template is silent);
%xHH emits a raw byte through the shell's byte-escape convention;
bare --format gets git's own fatal while bare --pretty stays medium.
git show's format: header drops its trailing newline the same way.
- ANSI-C \u/\U: values now ride bash's u32toutf8 (UTF-8 locale) -
surrogate halves and values past Unicode become raw UTF-8-shaped
bytes, 0x80000000 and past produce nothing. The previous "verbatim"
rule (and the crash codex flagged) was a C-locale pin artifact.
- himalaya: an ASCII attachment filename holding any line break is
refused with EmailMessage's exact error; the RFC 2231 path
percent-encodes the same characters (new shared parity pin).
Review round (generality follow-ups to the case-pattern fix):
- expand_case_pattern is now expand_pattern and serves all three
pattern-word constructs: case patterns, the [[ == ]] right side
(replacing the whole-node right_literal boolean that broke mixed
quoting), and parameter-expansion operands. Quoted operand nodes
match literally; opaque regex tokens get a lexical scanner honoring
single/double/ANSI-C quotes, backslash binds, and live $-refs.
escape_glob moved beside GLOB_CHARS in utils/glob_walk (both
languages), killing a duplicated constant.
- Multi-line double-quoted strings: the newline bytes belong to no
tree-sitter token. TypeScript dropped them entirely and python
collapsed blank lines; both now re-emit per row step anchored on the
quote tokens (leading, trailing and blank lines included).
- MIME machinery out of himalaya: mime.ts is now the runtime-agnostic
@struktoai/mirage-core utils/mime (TextEncoder + shared base64, no
Buffer), exporting the header guard as assertHeaderValue; the fixed
extension table lives in utils/filetype as MIME_BY_EXTENSION /
mime_type_for beside the FileType maps in both languages. Fixed a
silent TS-only bug found in the same file: jpg guessed IMAGE_PNG.
python's local drain() replaced by the existing io materialize.
Coverage: 40+ new docker/real-git/CPython-pinned unit rows across both
languages, 20 new integ cases (git format band, param/test/quoted bash
categories, surrogate byte-count), one new shared MIME parity pin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(monty): port to pydantic-monty 0.0.19 and stop rewriting files on append
0.0.19 moved execution into subprocess workers: Monty became a pool,
MontyRepl is gone, and run_async moved onto a checked-out session. run()
and eval() now drive the pool, and eval() is what the policy layer runs
config-borne scripts on, so the pin is exact while the API keeps moving.
Cancelling a feed no longer stops a worker, so both paths reclaim the
worker pid on CancelledError. Without it a safeguard timeout wedged the
line forever and pool teardown blocked uninterruptibly.
Also fixes four bridge gaps: mkdir, rename and rmdir never reached the
workspace and failed with a bogus ENOENT on paths that existed; piped
stdin was discarded, and now binds as a global beside argv; and appends
re-sent the whole file every write, which made a write loop quadratic
(200 appends shipped 164 KB to build a 1.7 KB file). Monty hands the
append hook the new text alone, so it routes to the mount's append op.
Documents what a command_safeguard does and does not stop inside a
sandbox, where mirage holds only a client and the process keeps running.
* fix(monty): bump the ts binding to 0.0.19 and route the write ops
Path.mkdir, rmdir, unlink and rename declined in the monty callback, so
they applied inside the sandbox's own tree and never reached the mount.
The bridge already carried the ops; this wires them up, matching the
python runtime.
* fix(monty): address review on append fallback, pool race, and cross-mount rename
- appends fall back to the whole-file flush on mounts without the
optional append op (S3 registers write but not append), remembering
the mount so it costs one failed dispatch rather than one per append
- the worker pool is cached as a task, so two cold runs cannot each
build one and leak the loser's workers past close()
- a cancelled eval session hands its checkout back instead of dropping
a lease the pool can no longer reach
- mkdir forwards parents and answers exist_ok itself
- a rename across two mounts raises EXDEV in both languages: the
dispatcher picks the mount from the source alone, so crossing would
drop the source and write the target into the wrong backend
* docs(monty): correct the EXDEV rationale, monty ships no shutil
* fix(monty): reject mkdir over a file, and type the mutation errors
* fix(find): classify walked entries through stat, drop the is_dir_name heuristics
* fix(test): satisfy noUncheckedIndexedAccess in the gmail find fixture
* fix(dropbox): walk the native find over a scratch index
* test(integ): pin find -type f over attachments and uploads
* test(integ): find -type f matches a pdf upload too
* ci: retrigger after the Actions outage
* fix(email): route the TS find through the generic walk so every flag applies
* chore(deps): bump h2 to 4.4.1 for GHSA-6hr6-w5qg-qmwg
* feat(gws): place creates in the folder scope, and report Google's errors
Three fixes and a mock-server sweep, all on the Google surface.
The gws CLI ignored folder_id, so a create landed in My Drive's root
even when the CLI and a gdrive mount shared one config. The editors'
create methods carry no parents field at all, so those are moved with a
follow-up Drive update; drive files create and copy just default their
parents. An explicit parents still wins.
Google reports why a call failed in the response body, but the python
client raised before reading it, so an agent only ever saw "Bad
Request". Typescript already did this correctly. The API and OAuth
error shapes are both handled.
An account CLI mutates its service by id, which no vfs path can be
derived from, so a newly created file had no cache entry to expire and
the agent's next ls could not see its own work. Write verbs now drop
the listings of the mounts their service backs, declared per spec so a
Slack or S3 mount alongside keeps its cache. This corrected two integ
goldens that had been asserting the stale listing.
The fake Google server grew the Sheets endpoints it never had
(values:batchUpdate, batchGet, batchClear, clear, sheets:copyTo, the
dimension requests), plus the matching gaps in docs, slides and drive.
gws integ coverage is now 35 of 35 leaves.
* fix(gws): address codex review on the folder-scope PR
Three findings, all real.
Cached file bodies survived a CLI write. drop_service_listings cleared
the resource index but not the file cache, and all five Google resources
cache reads, so a cat after a Docs or Sheets edit kept serving pre-write
content without reaching Google. A stale listing hides a create, a stale
body hides an edit, and only the listing was being dropped. Adds a
prefix-scoped evict_prefix to the file cache contract, implemented on the
RAM and Redis stores in both languages, exposed as CacheManager.drop_prefix
and called beside the index clear. The function is drop_service_caches now,
since listings no longer describes what it does. Redis escapes the prefix
before using it as a SCAN MATCH pattern, because a mount path may hold
glob metacharacters.
Scoped placement never sent supportsAllDrives. Every other Drive helper
in the repo sends it, and a folder scope may name a Shared Drive folder,
so both the injected parent and the relocation patch would fail there,
stranding an already created editor file in My Drive. Both paths send it
now, only when mirage is the one injecting.
An explicitly empty parents array was silently replaced. The check was a
truthiness test, so parents: [] read as absent while parents: ["root"]
was honored, which contradicts the documented rule that an explicit
array wins. Presence of the key is the test now.
Adds unit coverage in both languages for each finding, plus two integ
cases pinning the stale body end to end. The integ pair has to span two
command lines: apply_io runs once per line, so a cache warmed earlier on
the same line is never live for a later read, and a single-line case
passes with or without the fix. Documents folder scope and its two
divergences from the upstream passthrough in gws.mdx.
* test(cache): pin drop_prefix on a root mount
A root mount strips to the empty prefix, so the eviction argument is
"/" and matches every key. Correct, but non-obvious enough that a
reviewer reasoned their way to the wrong answer and back, so both
languages now assert it instead of leaving it to be re-derived.
* feat(git): a git CLI over any mount, in both languages
Adds `git` as a builtin CLI (issue #705 item 1): status, log, show,
diff, branch, add, reset, commit, checkout. It takes no config, and the
repository is read entirely through the mount ops, so a repo on RAM, on
disk or on an object store reads the same way. Python goes through
dulwich, TypeScript through isomorphic-git backed by a PromiseFsClient
over the dispatcher.
Status is pinned against the real git binary across 22 repository states
and 6 spellings, the mutation verbs are read back by real git and pass
`git fsck`, and the rename-similarity score matches dulwich digit for
digit. Both run in integ on `git-ram` and `git-disk` in both languages.
Also fixes the grep family this leaned on:
- grep reads a basic regular expression by default, as POSIX says, with
-G for the default and -E for extended. `grep -l` and `grep -rl`
compiled their own pattern and still read extended.
- zgrep does the same. Its -E flag was read and discarded.
- an operand grep could not search exits 2, as GNU does, instead of
being flattened to 1.
- `grep -l` on a directory reports it rather than walking it. The
shared fallback called the recursive walk whenever a failed read
turned out to be a directory, which made -l behave like -rl.
Supporting pieces: ranged reads (`read_range`) with a read-and-slice
fallback and a seeking implementation for disk, PATH-typed group options
resolved against the working directory, and git's three unknown-option
dialects.
* fix(rg): report an unreadable operand as exit 2, like ripgrep
CI caught four goldens the grep change had missed, all outside the JSON
harness targets I had been running, plus a divergence the change itself
introduced.
The divergence: TypeScript routes rg through grepGeneric, so rg moved to
exit 2 there while python's rg, which has its own generic, stayed at 1.
Real ripgrep exits 2 for an operand it could not read, so TypeScript was
right and python is brought up to it rather than the other way round.
Its single-operand path also let the error escape to the shared handler,
which flattens every OSError to exit 1; rg now reports it itself, the
same fix grep needed.
The rule is now stated once in grep_helper (`exit_code_for` /
`exitCodeFor`) and imported by both generics in both languages, instead
of living in the grep generic where rg could not reach it.
Goldens updated: integ/cross_commands.{py,ts}, the shared observability
contract, langfuse and dify. Each pinned exit 1 for a missing operand
and carried a comment calling it a deliberate divergence.
* fix(git): refuse the three mutations that could lose work
Three review findings, each measured against git 2.50.1 before fixing.
checkout only compared tracked paths, so a branch holding a file the
working tree has untracked wrote its blob straight over it. The file is
in no index and no tree, so nothing could see it and nothing could get
it back. The untracked set is now part of the conflict check, which
needs UNTRACKED_ALL rather than the mode status uses: "normal" collapses
a wholly untracked directory to one row, and git names the file inside
it. An ignored file stays overwritable, which is git's own split. When
both kinds of conflict apply git prints both paragraphs and aborts once,
so CheckoutConflictError carries both lists.
branch -d deleted a branch HEAD does not contain, dropping the only name
pointing at those commits. It now refuses, and -D is added, because -d
alone would be a delete with no way to say no. dulwich's can_fast_forward
answers exactly this and cannot be used: it asks the repository for its
grafts and shallow boundary and a bare BaseRepo raises. Walker is what
log already walks with and needs only the object store.
add -u ignored its pathspecs and restaged every tracked file, which is
how an unrelated edit reaches the next commit. git tells two misses
apart and so does this now: a pathspec naming nothing is a fatal about
the pathspec, one naming an untracked file is a fatal about git not
knowing it, both exit 128.
Also two CI failures. The spec dumps were stale for the -G flag added
with the zgrep BRE fix. And the git fixture exported its identity as
environment variables, which reach only its own commits, so a test that
committed into the built repository failed with "Author identity
unknown" on any runner with no global identity. It now records the same
identity in the repository, leaving the object ids unchanged.
Covered by unit tests in both languages and six integ cases across
git-ram and git-disk on both hosts.
* fix(test): the redis missing-file suite still pinned grep at exit 1
Sibling of the disk suite, which was updated with the exit-2 change. This
one needs a live server, so it skips silently without REDIS_URL and the
local run never reached it. Only the grep case moves: cat, head, tail and
wc all still exit 1 for an operand they could not read, which is the
split the comment now records here as it does next door.
* fix(grep): read an operand's type from stat, not from how the read failed
Four integ jobs caught this across seven backends, and it is one mistake
with three faces. Classifying a directory operand *after* a failed read
makes the answer depend on what each backend does about reading one, and
they disagree: s3, gridfs, hf and nextcloud read a directory path without
complaint and hand back nothing, and ssh raises an asyncssh SFTP error
that is not an OSError at all, so it escaped the catch entirely and
exited 1 with an unattributed "grep: Is a directory". So the operand is
now stat-ed before it is read, which is what GNU does and what the -r
branch of the same function already did.
operand_is_directory had its own version of the same error: a readdir
that did not raise counted as a directory, and a prefix store answers
readdir for any path at all, returning nothing for one that is not
there. Every missing file on those backends therefore read as a
directory. The listing must now be non-empty to count, which costs a
genuinely empty directory being invisible there, the same divergence du
already documents and the safer way round.
The catch also widens from three exception types to WALK_ERRORS, the
tuple every other operand-tolerant walk in the repo uses, so an errno
split cannot make grep abort where tree and grep -r keep going.
One test fake resolved stat with `undefined as never` and only worked
while nothing read the value. It resolves with a real FileStat now,
which is what the postgres backend answers there; the assertion the test
exists for is untouched.
Verified on gridfs and gridfs-prefix, which reproduce the prefix-store
half locally, plus ram, disk and redis for regressions: all five targets
on both hosts, 0 failed.
* fix(git): three review findings on -C, checkout -b and reset
`-C` accepted any path that was not missing, so naming a file inside a
repository walked up and ran in the parent instead of failing the way
git's chdir does. For a write verb that means mutating a repository the
caller never named. It now refuses a non-directory with git's own second
wording, "Not a directory".
`checkout -b <new> <start>` forced the new branch to HEAD and dropped the
start point without a word, so every commit after it landed on the wrong
history. The operand is honored, and a start point that is not a commit
gets git's sentence naming both it and the branch rather than the
generic "ambiguous argument" the same lookup failure produces elsewhere.
`reset <operand>` that selected no path unstaged nothing and exited 0,
which a script reads as "the index was reset". Two different mistakes
reach that point and they now get different fatals. A typo is git's
"ambiguous argument". A revision is not: real git resets the index to any
commit named there, measured on 2.50.1, so the review's premise that git
refuses one is wrong. This build resets from HEAD only, and says which
feature is missing instead of claiming a revision it can resolve is
unknown. Recorded as a divergence in both git.mdx files.
Six integ cases across git-ram and git-disk on both hosts, plus unit
tests in both languages.
* chore: merge main and regenerate the grep and zgrep specs
#713 landed a `pair` field on the option spec while this branch was
open. CI builds the PR merged with main, so it regenerated specs
carrying the field and compared them against grep.json and zgrep.json,
the two files this branch had already rewritten for `-G`, which
therefore predate it. Every other spec file came across from main
already carrying it.
The regeneration also needs a rebuilt dist: gen-specs.ts reads the built
package, and against a stale one it dies on an Operand field it does not
know, which silently leaves the typescript half unregenerated and shows
up as a python/typescript parity divergence rather than as a build
error.
* fix(shell): write a hex or octal escape as the byte it names
printf '\xc3\xa9' wrote four bytes where bash writes two: \xHH read as a
code point and was then UTF-8 encoded, so \xff came out as c3 bf rather
than the single byte it names. Octal escapes and echo -e had it too.
A byte above ASCII now rides as its surrogate escape and every place the
shell turns its own text into bytes decodes it back, which covers printf,
echo, %q, heredocs and here strings. \u and \U still name code points.
* fix(gsheets): ask for the cell values the mount documents
.gsheet.json promised cell values at .sheets[0].data[0].rowData[] and
carried none: spreadsheets.get returns no grid data unless the request
asks for it, so the file was tab metadata against real Google too, not
only against the fake server.
The fake server implements the endpoint now, with the shapes taken from
the live API: no startRow or startColumn at zero, {} for an empty cell,
numbers parsed under USER_ENTERED, row and column metadata, tableRange on
append, and the whole SheetProperties in an addSheet reply.
* fix(shell): mask a byte escape and keep a non-BMP character whole
Three octal digits reach past one byte, and bash writes the low byte of
those. The sentinel mapped the whole value instead, so 256..511 landed on
U+DD00..U+DDFF, outside the range surrogateescape can encode: printf
'\400' raised UnicodeEncodeError rather than printing a NUL. Mask to one
byte before building the sentinel.
In TypeScript the sentinel range also swallowed the low half of an
ordinary surrogate pair, so a non-BMP character with a low surrogate in
U+DC80..U+DCFF (U+10080 is D800 DC80) came out as U+FFFD plus a raw 0x80
even with no escape in sight. Only a lone low surrogate is a byte now.
The non-BMP integ case named U+1F600, whose surrogates are D83D DE00 and
never were in the sentinel range, so it passed against the broken
encoder. It names U+10080 instead, with an echo case and a case that
mixes the character with a byte escape.
Pinned against bash 5.2 in a UTF-8 locale.
* test(gws): match live Sheets on grid size and cell typing
The mock reported a fixed 1000x26 grid, so a tab holding more rows than
that claimed to hold 1000 while carrying more, and handed out one
rowMetadata entry for only the first 1000. Live grows the grid to what
was written, so gridProperties, rowMetadata and columnMetadata all read
from one extent now: 1313 written rows report 1313.
Cell typing went through Number(), which is looser than USER_ENTERED: a
whitespace-only cell became numeric zero and 0x10 became 16, where live
keeps both as strings. A plain decimal test replaces it. TRUE is a
boolean rather than a string, and formattedValue is the rendered value
rather than the raw input, so 007 reports "7" and 4.50 reports "4.5". A
number typed with an exponent keeps a scientific format, so 1e3 reports
"1.00E+03".
Verified against the live API: 18 typing cases and 8 exponent cases
match exactly. Currency, percent, thousands-separated and date-shaped
cells stay strings and say so in a comment, since they need locale-aware
number formats.
The fixture seeds a boolean, a hex-looking string and an exponent so the
typing is asserted through the mount.
* feat(jq): support the rest of jq's flag surface
Only -r, -c and -s were accepted, with no long forms at all. Adds
-n/-R/-j/-a/-S/-e/-M/-f/-h, every long spelling, --raw-output0, --tab,
--indent, --unbuffered, --arg, --argjson, --rawfile, --slurpfile,
--args, --jsonargs, --stream and --seq, in Python and TypeScript.
Two spec-parser additions carry it: Option(pair=True) for two-token
options like --arg name value, and Operand(text_when=...) so --args
turns later operands into positional strings instead of input files.
Also fixes -s to slurp across every operand rather than per file, and
an empty input to print nothing instead of erroring.
* fix(jq): read inputs as a call, not as the word
The detector matched `.inputs`, `{inputs}`, `$inputs`, a module member
and any string or comment, and a match switches the run into drain mode,
so `jq -c '.inputs'` over a two-document stream printed one line where
jq prints two. Both languages now blank string bodies, comments and
object-shorthand keys before searching for the token, keeping
interpolations as code. $ARGS is read the same way.
The native fixture also tolerates EPIPE on the child's stdin: `jq -n`
exits without reading, which raced the fixture's write and failed the
whole node package on unhandled errors with every test passing.
Adds integ coverage for the rest of the flag surface, including the long
spellings, the refused options and the usage errors.
* feat(himalaya): align the CLI with the upstream pimalaya grammar
- mailbox is -m/--mailbox, message id is a positional operand
- envelope search with upstream's query DSL, replacing list filter flags
- message compose writes RFC 5322 to stdout unless --send; message send
takes raw MIME, so the two chain through a pipe
- upstream aliases: ls, sr, write, new, fwd
core/email/send.* is deleted: the email mount is read-only, so the
composer and the SMTP transport moved into the himalaya package.
Fixes three node-side bugs found while testing:
- sendMail({raw}) needs an explicit envelope; Bcc is stripped by hand
- parseSearchCriteria silently dropped unknown keys, so NOT SEEN matched
every message
- parseImapDate rolled a bad month or day into a wrong date
* fix(himalaya): satisfy the CI eslint rules and stop the yapf/isort import flip
* fix(himalaya): search the Date header and bound envelope pages
Date conditions emit SENTON/SENTBEFORE/SENTSINCE rather than
ON/BEFORE/SINCE, which match the mailbox internal date, and the missing
upstream `before` condition is now parsed.
Envelope listing asked IMAP for every matching uid and fetched headers
for all of them before slicing one page. It now fetches the newest
page * page_size, capped by the account's max_messages.
* test(himalaya): cover every flag in integ, and fix what it found
Grows the himalaya integ suite from 38 to 64 cases: every search
condition, sorter, flag and alias, the composer flags, both quoting
options, stdin bodies, the send operand form, unknown flags and a
missing mailbox.
Three bugs it turned up, all in the email client:
- Five IMAP selects ignored their response, so a missing mailbox left
the session in AUTH and the next command complained about that
instead ('command SEARCH illegal in state AUTH' in python, imapflow's
bare 'Command failed' in typescript). Both now say no such mailbox.
- A refused SEARCH returned an empty uid list, so criteria the server
cannot answer read as 'matched nothing'. It now raises.
- Found because a mutation emitting a malformed date stayed green.
* feat(shell): discover installed CLIs through man, type and which
man renders an installed CLI from its own spec, type reports it as a cli, and which is new. Precedence now lives in one lazy layers() generator that serves both route (the winner) and route_all (every layer, which type -a prints).
* refactor(shell): share one option scanner across the bash builtins
command, type and which each hand-rolled the same non-permuting letter
scan. They now share scan_options/last_of, which pins bash's grammar in
one place: a long spelling refuses on its second dash, and a mutually
exclusive group resolves to the last letter typed.
type -f is a filter over the layer list instead of a pop-and-restore on
the session function table, and man takes the install it already looked
up rather than fetching it again.
* fix(shell): keep the layers under a reserved word visible
type -a stopped at the keyword, so a function sharing a reserved word's
name never showed. bash prints both lines (function time { :; }; type -a
time), and mirage's parser lets any reserved word be a function name, so
the shadow was reachable and hidden.
time and coproc leave the keyword table with it: mirage implements
neither, so type called them keywords while running one reported command
not found. which now drops the keyword layer before picking a winner
rather than after, so it reports the function underneath instead of
nothing.
* refactor(shell): split lookup into a package the way route and condition are
types.py holds NameKind, constants.py the consumer and description
tables, classify.py the layer walk, handle.py the two builtins. The
keyword pool moves to the name-pool leaf beside SHELL_NAMES, where the
CLI registry can read it: installing a CLI under a reserved word was
allowed and would never have been reachable, and is now refused.
command.py and lookup build their result triples with the shared
helpers instead of by hand, and man renders its missing-description
placeholder in one place.
* fix(shell): keep command -V's diagnostics when another name resolved
The shared-helper refactor routed the any-found case through ok(),
which drops stderr, so command -V ls nope printed the found line and
swallowed 'command: nope: not found'. bash prints both and exits 0
(pinned), and the TypeScript sibling always did.
All three handlers now build one result with a computed exit code, so
the diagnostics can never ride on the status again, and the mixed case
is a test in both languages.
`tac` is GNU coreutils; macOS ships `tail -r` instead. The native fixtures
capture only the child's stdout, so on a host without `tac` the native side
silently yields "" and the comparison fails with a confusing diff rather than
reporting the missing binary.
Guard both sides on the binary's presence rather than the platform, so the
tests still run wherever `tac` exists (Linux CI, or macOS with coreutils
installed) and skip cleanly where it does not:
- typescript/packages/node/src/commands/native_tac.test.ts: `it.skipIf`
keyed off `command -v tac`, mirroring native_fmt.test.ts's skipIf shape.
- python/tests/commands/native/test_tac.py: `pytest.mark.skipif` on
`shutil.which("tac") is None`, mirroring test_fmt.py's named-mark shape.
Verified both directions on macOS: 4 TS / 6 py tests skip with no `tac` on
PATH, and all 10 pass with a GNU-equivalent `tac` shim on PATH.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): slack, discord, ntn and linear as builtin CLI packages
Migrate the remaining service bundles to installed CLIs with the settled
vocabularies (OpenClaw slack/discord actions, the official Notion CLI
grammar as ntn, linear's noun/verb tree), including the new API surface
they need (slack pins/emoji/reactions/history, discord edit/delete/
threads/polls, notion page updates). Configs move resource -> core in
both languages and the mounts go filesystem-only. chroma-query stays a
mount command on purpose: it is path-scoped.
Fixes two CLI dispatcher bugs found by the new integ cases in both
languages: leaf exceptions now become the command's IOResult (so
redirects and ; sequencing work, with the GNU prog: prefix), and argv
words re-enter the walk as typed so quoted glob-looking flag values
survive.
Adds cli-facet integ targets and cases for all four CLIs (108 cases
both runners), extends the fake slack/discord/notion servers, and moves
CLI usage out of the resource docs into new CLI sections on both tabs.
* integ: cover the remaining linear write verbs in the cli facet
* feat(cli): linear add-label and set-project resolve names via the issue's team
* fix(cli): codex review, standalone thread type and bounded notion search pagination
* fix(crossmount): honor wc --total and stop du -h rounding twice
The FANOUT combiners re-parse each native run's rendered text, so a flag
that changes output shape has to be re-implemented in the combiner or it
diverges from the single-mount path. Two did.
wc ignored --total entirely. Worse, it guessed that a run with more than
one row ended in a per-run total and dropped that row, so a glob operand
silently lost a file. Native runs are now forced to --total=never and
combine_wc delegates to the generic's own format_count_rows, which is
what applies --total for one mount.
du -c -h fed each run's already humanized total back through parse_size,
so two 1500 byte operands read as 1536 each and printed 3.0K where one
mount prints 2.9K. Native runs now report exact bytes and du_total
humanizes once.
That removed the last caller of utils/formatting.parse_size in both
languages, so it and the orphaned SIZE_UNITS constant are gone.
* fix(crossmount): diagnose an invalid wc --total before the override
Forcing --total=never on the native runs also hid a bad value from
them, so `wc --total=bogus` across mounts ran clean. Python then raised
out of the combiner and an upstream handler happened to render it,
while TypeScript returned null and exited 0 with no diagnostic at all.
Both now parse the user's flags in run_fanout before overriding, and
return the GNU message with exit 1, which is what one mount does.
* feat(policy): absorb output safeguards into the policy layer as Limit
* fix(policy): loop-based trailing-slash strip in limitOverride (CodeQL polynomial-redos)
* fix(integ): surface policy EACCES through the TS fuse read/write callbacks, skip code-policy cases in the CLI harness
* chore(deps): bump cryptography 50.0.0 and aiohttp 3.14.3 for audit advisories
* fix(policy): stamp builtin producers at the dispatch chokepoint, purge safeguard vocabulary from docs
* fix(google): surface Shared Drives in the g* mounts, and make the Graph service root configurable
The g* mounts sent no corpora flags, so gdocs/gsheets/gslides silently
missed every Shared Drive file while gdrive showed them. They now search
every corpus, and report Drive's incompleteSearch flag so a short listing
is never cached as the directory. gdrive's root had the same bug already:
a failed Shared Drive enumeration was cached as a My-Drive-only root.
GRAPH_API stops being a module constant. The service root now comes from
the mount config, so a mount can address a national cloud, and the integ
fake is reached by configuring a mount instead of rebinding a global in
every module that spells a URL. OneDrive also gains /groups/{id}/drive
and /users/{id}/drive addressing, with a validator rejecting more than
one drive target.
Closes#305, #308, #309.
* fix(msgraph): escape Graph identifiers, and drop the national cloud table
Two Codex findings on #695, plus a scope cut.
Escaping: drive_base interpolated identifiers raw, so a guest user's UPN
(guest_contoso.com#EXT#@fabrikam.com) had its `#` read as a URL fragment
and Graph received a truncated /users path. id_segment now escapes each
identifier as one path segment, applied in drive_base, SharePoint's
item_url and its site drive listing. TypeScript already did this.
The safe set is "!*'()" rather than an empty one so quote matches
encodeURIComponent exactly. Drive ids are b!<base64url>, and the ref
paths built off them go into a JSON body Graph reads literally, so
escaping the `!` on one side only would have traded a guest-user bug for
a copy and rename bug on every drive-id mount.
Docs: the OneDrive example used GraphCloud without importing it, and it
was not re-exported from the documented module. That example is gone
with the enum below, so the broken import goes with it.
National clouds: the GraphCloud enum, GRAPH_CLOUD_HOSTS and the `cloud`
field are removed from both languages. mirage cannot verify those four
hostnames, and a table of unverifiable constants reads as a guarantee
the library cannot make. graph_base_url stays and is now the single way
to name a non-worldwide root, which is what #305 actually asked for: the
service root is read from config rather than a module constant, so two
mounts in one process can address different deployments and the integ
fake is reached by configuring a mount. The docs now point at Microsoft's
own national cloud documentation for the root.
* fix(find): classify the start point once, above every backend
`find` asked the backend to walk a start point that is not a directory,
and each answered differently: a Graph 404, a silently empty listing, or
ENOTDIR. The reported symptom was `find -L <cross-mount link>`, but the
operand rewrite was correct all along: `find /data/a.txt`, with no
symlink anywhere, failed identically.
Every native find op listed `<start>/children` unconditionally and
hardcoded the start as `kind="d"`. Extends the shape #688 built for
symlink start points so the start point is classified once and every
backend inherits the answer: a symlink goes to link_results, a
non-directory is reported and walked no further, a directory walks.
The stat rides the op dispatcher as a new `stat_path` fact rather than a
threaded callable, for the reason #688 recorded when the same threading
left `hf` behind, and because the router resolves `find -L` into another
mount before the command runs. Only a positive non-directory answer
short-circuits: a stat that sees nothing is not proof of absence when a
backend's directories are implicit.
Also fixed, found on the way:
- `find /d \( -name x \)` classified `(` and `)` as PATH operands, giving
find two phantom start points, hidden because the native path reads
paths[0] and drops the rest
- `tree <file>` diverged three ways by backend; GNU prints the marker,
counts the file and exits 0
- Python's missing-operand message printed the path twice instead of
`No such file or directory`
Fixes#684, which this change needed: gdrive was the one native-find
backend that could not be tested at all, so `find` there reached the
real Drive API. Widens the fake's patch targets to resolve/tree/rename,
teaches it the `name=` filter, and makes an unresolvable target a hard
error. The e2e s3 fake needed `list_objects_v2` for the same reason.
Pinned against GNU findutils 4.10.0 and tree 2.2.1 on debian:stable-slim
before changing behavior. Mirrored in Python and TypeScript.
* fix(find): decide absence above the backend, not per backend
`find <missing>` exited 1 only on the three backends that wire a stat
into find, and `tree <missing>` rendered GNU's marker on twelve of
eighteen targets. Both were the same gap: a backend could not tell
"absent" from "a directory that exists only as its children", so
neither command could either.
The rule already existed in the repo. `test -d` has always asked both
channels a backend can answer on, and only both coming back empty means
nothing is there. `resolve_path_stat` / `resolvePathStat` is now that one
implementation, shared by the start-point probe and `path_kind`, so a
traversal and `test -d` cannot disagree about whether a path exists.
Measured on 18 targets before changing anything: an implicit directory
answers, a missing path does not, and an empty `mkdir`'d directory
answers everywhere except hf, whose `mkdir` creates nothing and which
`root_create_mkdir` already excludes for that reason.
Two pre-existing fixture bugs the uniform rule exposed, both of which a
silent `find` had been hiding:
- TypeScript's integration fixture seeded `store.files` without the
ancestor directories, a store no write can produce. RAM keeps
directories explicitly, so `ls /data/logs` printed nothing and
`test -d /data/logs` was false there already. The python mirror
fixture has always `mkdir`'d each ancestor; now both do.
- The nextcloud exclusion was missing from `tree_start_missing`'s
target list because it needs a real server, which is what turned
`integ-data` and `integ-ts` red.
Verified against a local `nextcloud:30-apache`, the image CI uses.
aiohttp <= 3.14.1 accepts compressed WebSocket frames without negotiated
permessage-deflate (CVE-2026-59881, moderate). Fixed in 3.14.2; the lock
moves to 3.14.3, and the pyproject floor is raised so the fix cannot be
resolved away.
Clears the red audit job.
* fix(symlinks): report and follow links the way GNU does
Symlinks live in the namespace, so no backend readdir or stat can see
them and every command was blind to them unless wired by hand. Six GNU
divergences, each pinned against debian:stable-slim first, mirrored in
python and typescript.
- ls -l on a dangling link failed the whole listing with exit 2; GNU
prints the link row and exits 0. ls -l/-d on a live link silently
followed it, and ls -R and -F omitted links entirely.
- readlink -e printed a dangling target and exited 0; GNU requires the
whole resolved path to exist. -f now requires only the parent.
- file followed the link and relabelled the operand as its target; GNU
prints "symbolic link to <target verbatim>", or "broken symbolic link
to ..." when the target is absent.
- du omitted links; GNU lists one line per link under -a and does not
follow a link operand without -L.
- find followed a symlink start point; GNU's default is -P, which
reports the link and stops. -P/-H/-L are leading options, not
predicates, so the expression splitter consumes them first.
- -L has to apply below the operand too, not just to it: ls -L stats
each link child and reports the target under the link's own name, and
du -L stops counting links as entries of their own.
Generalize the seam while it is being touched:
- A command opts in by naming a `links: LinkView` parameter. execute_cmd
offers the fact to every handler and accepts_kwarg reads the signature
to decide delivery, so there is no allowlist to keep in step. This also
removes the hardcoded stat_overlay command list. TypeScript needs no
equivalent: a command that does not read `links` off its context
ignores it.
- LinkView bundles all five link facts, so a new need is a field read
rather than a keyword threaded through four layers.
- Link merging lives in the generic above the native-op/walk fork, so a
mount's symlink behavior cannot depend on whether its backend ships a
native find or du op.
- ls merges links as stat rows instead of appending to its own stdout,
which is what makes -R, -F, -l and sorting work. Deletes inject_links
and its latent dedup bug, where splitting a GNU long row on tabs
returned the whole line.
Also moves data shapes and constant tables to the modules the layout
calls for (ops/types, commands/builtin/constants, utils/constants) in
both languages, and adds 33 integ cases that run in both runners.
du sizes a link at its target length, which is not a divergence: mirage
counts bytes, GNU's --apparent-size mode, and there GNU reports
len(target) too. Two known gaps, both documented rather than papered over: du -L
undercounts a link pointing outside the operand's own subtree, and ls
-RL does not descend a directory link. A version of the latter that only
worked on RAM was removed rather than shipped, because descending needs
the child read re-keyed onto the resolved target and that only resolves
on backends with real directory entries. Symlink behavior that depends
on which backend is mounted is the exact failure this branch adds a rule
against.
A link operand is caught on both readdir shapes. Backends without real
directories (s3, nextcloud) answer readdir on a link path with an empty
list rather than raising, so the link row has to be resolved there too
or the operand renders as an empty directory. Pinned by unit tests
against the generic in both languages, so the case is covered without
needing a remote backend to reproduce it.
Fixes#414
* fix(integ): drop the empty-directory dependency from the symlink fixtures
* fix(symlinks): give bespoke stat wrappers the link table, and make find's link options last-wins
s3 and gridfs ship their own stat command, and neither declared the
links parameter, so signature-based injection silently withheld the
namespace link table and stat on a symlink reported ENOENT on exactly
those two backends. TypeScript is unaffected: its wrappers forward the
whole opts object, so the generic reads opts.links regardless.
find states its link policy as a leading option and GNU takes the last
one, so find -L -P x must not follow while find -P -L x must. The
scanner returned true on seeing any -L or -H anywhere.
* feat(metadata): walk the subtree for chmod, chown and chgrp -R
All three refused -R with exit 2. GNU splits three ways here: chmod -R
skips a traversed link but follows a command-line link to a directory,
while chown -R and chgrp -R change the link node itself because POSIX
gives -R an implicit -P.
Two bugs surfaced while pinning it. chown -h wrote uid onto the node but
link_stat dropped ownership when rendering, so nothing ever showed it;
only mode is meaningless on a link. And TypeScript rebuilt stat rows
field by field where Python uses model_copy, dropping uid/gid on the
link row, so FileStat gains a with() method and the four rename sites
use it.
* fix(find): classify a link by its target under -L
find -L must test a link as what it points at, not as a link: a link to
a file is -type f, a link to a directory is -type d, and only a dangling
link stays -type l. link_results classified every namespace link as l
regardless, so find -L d -type f missed the links and find -L d -type l
wrongly named the live ones.
Both find paths reach link_results, so the classification lands above
the native-op/walk fork rather than in one branch.
* fix(symlinks): stop backends losing links and -L one wrapper at a time
`find -L` classifies a link by statting its target, and that stat was
reached through the callable wired for the -mtime post-filter, which is
gated on ops.local. Post-filtering costs one stat per result, so gating
it is right; classification costs one stat per link, so gating it was
not. Off the local backends nothing was statted and every link stayed
typed `l`: -type f and -type d missed them, -type l still listed them.
The fact belongs on LinkView, not on a threaded argument, so it is now
`target_stat` / `targetStat`. It resolves through the op dispatcher
rather than a backend key, which also fixes a link pointing into another
mount: the old code built the key from the search root's mount prefix
and could never resolve one. Pinned in integ/crossmount/find/deref.json.
The same omission then turned up in ten bespoke commands that shadow a
link-aware generic without declaring `links`, and in the same ten for
`-L`, which lands in the wrapper's opaque flag bag when unnamed. Both
failures run fine and exit 0, so nothing notices until someone makes a
link on that backend. tests/commands/test_links_optin.py asserts it
instead, deriving from the builders which link parameters each family
requires; it caught history/ls on the first run.
TypeScript needs no guard because wrappers forward the whole opts
object. Its one exception, email/find, walks its own tree and never
reaches a generic, so it now calls linkResults directly.
* fix(find): trim slashes without a regex, and derive the link guard from the router
Exporting linkResults made its input library-reachable, which surfaced
the polynomial-ReDoS trim inside it (CodeQL js/polynomial-redos, high).
Both `/^\/+|\/+$/g` trims now use the loop-based stripSlash helper.
The guard's list of required link options was hardcoded. It now reads
DEREFERENCE_FLAGS and LAST_WINS_LINK_OPTIONS, so a new link option
taught to the router starts being required of bespoke wrappers too.
* fix(symlinks): let real backend failures surface while statting a link target
link_target_stat caught OSError around _stat_or_none, which already maps
the missing-target case to None. The extra catch could therefore only
swallow a genuine failure: a permission, connection or I/O error came
back as a dangling link, printed as -type l with exit 0.
Only the two legitimate no-target cases stay None now, a loop from
namespace.follow and a missing target from _stat_or_none. Everything
else propagates. Same in TypeScript, where the bare catch did the same.
* feat(policy): policy package with the pre_command hook, absorbing the command guards
* refactor(policy): GuardSpec is a data shape, move it into types
* docs(policy): trim the seam narration from the Policy base docstring
* fix(policy): fire the hook at the dispatch chokepoint, cover flag paths, harden TS spec handling
* fix(policy): ln refusal wording follows the link kind
* chore(policy): keep hasSymlinkFlag module-private
* feat(spec): merge value_kind and type into one ValueType axis (python)
* feat(spec): merge value_kind and type into one ValueType axis (typescript + dumps)
* test(spec): float coverage, lint fixpoint
* fix(spec): linear-time FLOAT_VALUE regex (CodeQL polynomial-redos)
* fix(spec): numeric refusal before choices in option_error, add FlagView.as_float
* feat(cli): tree walk with git-pinned diagnostics
* fix(cli): narrow instead of assert, CI formatter alignment
* refactor(cli): one help renderer, --help as a registered option
* fix(cli): walk option precedence per flat parser, module layout
* fix(mv): refuse a non-empty destination directory, not only under -T
GNU mv refuses to replace a non-empty directory whether the target was
named outright with -T or mapped under an existing destination
directory. The refusal existed but was gated on no_target_dir, so
`mv src dest/` merged the trees at exit 0 instead (#272).
Dropping the gate also required moving the check after the clobber
gate: GNU lets -n and --update=none skip such a target silently at
exit 0, which the old ordering turned into an error. Pinned against
coreutils 9.7 in debian:stable-slim, all seven arms:
mv src dest/ -> cannot overwrite 'dest/src': Directory not empty
mv -T src dest/src -> same
mv -f src dest/ -> same (-f does not bypass)
mv -n src dest/ -> exit 0, skipped
mv -n -T src dest/src -> exit 0, skipped
mv src dest/ -> exit 0 when dest/src is empty
cp -r src dest/ -> exit 0, still merges
The -b arms keep working through backup_displaces, which renames the
target aside before the refusal can apply.
* fix(crossmount): compare storage identity, not path spelling, in cp/mv
Two mounts can address one store: two disk mounts on a shared root, one
bucket mounted twice, or the same resource object mounted at two
prefixes. Cross-mount routing keys off the mount prefix alone, so such a
pair looked like a genuine cross-mount move and mv did read -> write ->
unlink, writing the object over itself and then deleting it (#154):
$ mv /m1/x.txt /m2/x.txt exit 0
$ cat /m2/x.txt No such file or directory
$ find /m1 /m1
The generic cp/mv already had a same-file guard and a backend_key hook
for it, but nothing injected a key and the default is the mount-relative
path, which carries no mount identity. Resources now answer
storage_id(); make_storage_key pairs it with the mount-relative path and
the executor threads it through route -> relay -> cp/mv.
The default is per-instance identity, so a backend that says nothing
keeps today's behavior. That is the safe direction: a false "different"
only fails to catch an alias, while a false "same" would refuse a real
move. disk (resolved root), s3 (endpoint, bucket, key prefix) and redis
(url, key prefix) override it, so two separately built instances over
one target still compare equal.
Verified: aliased mounts refuse with the coreutils "are the same file"
line and the bytes survive; distinct stores sharing a basename, distinct
paths within one store, and two ram mounts all still move normally.
* fix(crossmount): mirror the storage-identity guard in typescript
Same shape as the python side: Resource gains an optional storageId,
BaseResource defaults it to a per-instance serial (JS has no identity
primitive), and disk, s3 and redis override it with the config that
actually pins their storage. makeStorageKey pairs it with the
mount-relative path and command.ts threads it through
handleCrossMount -> runRelay -> runCp/runMv.
storageId is optional on the interface because some resources implement
Resource without extending BaseResource; a resource that omits it is
keyed by its mount prefix, which reproduces the pre-existing behavior
rather than risking a false "same file".
* fix(s3): guard a same-key copy and rename in the core, py+ts
core.rename was copy_object + an unconditional delete_object, and
core.copy a plain copy_object, neither checking whether the resolved
keys were equal. Safety came from the remote refusing the request
rather than from our code (#150).
Both now follow POSIX rename(2) for the equal-key case: succeed and
perform no other action, with a missing source still raising ENOENT.
What the probing actually showed, which corrects the issue's premise:
- MinIO REJECTS a plain self-copy, versioned or not, exactly like AWS
(InvalidRequest). It is not one of the lenient stores the issue
names, so the data loss is not reachable there today.
- But a self-copy IS accepted once the request carries
MetadataDirective=REPLACE, and the delete then removes the object.
Confirmed against MinIO: on a versioned bucket the old version
survives behind a delete marker, on a plain bucket it is gone. So
the loss is one plausible code change away.
Coverage: the s3 mock's copy_object is deliberately lenient, which is
what makes the guard observable; a real store's rejection would mask
it either way. The mock now counts copy_object/delete_object so the
test asserts no write was issued at all, not merely that the object
survived. Confirmed discriminating: disabling the guard fails four of
the seven, one of them literally on an emptied store.
* test(integ): cover the non-empty-dir refusal and aliased mounts
Two overwrite cases for #272, on the seed that already builds the
shape: /data/ow/t holds two files and /data/ow/s/t is the source, so
`mv /data/ow/s/t /data/ow` maps onto a non-empty target. One asserts
the refusal, one asserts -n skips it at exit 0. Both pinned against
coreutils 9.7 replaying the same state in debian:stable-slim.
For #154 the harness could not express the shape at all: every builder
allocates fresh storage, so no two mounts could address one store. A
mount may now carry alias_of naming an already-built mount, reused in
both runners (spelled snake_case, matching seed_root). The new
ram-alias target mounts one RAMResource at /data and /alias, and
alias_both_prefixes_see_one_store proves the aliasing is real before
the guard cases assert anything.
ram, disk, ram-alias, ram-root: 4074 passed on the python host, 4069
on typescript-node, 0 failed on both.
* test(ts): unit-cover the storage-identity key
Mirrors tests/workspace/mount/test_storage.py: aliased prefixes compare
equal, distinct stores do not, a config-pinned resource matches across
instances, the ancestor prefix boundary survives, and a resource that
declares no storageId falls back to per-mount identity rather than
risking a false same-file refusal.
* chore: apply yapf formatting and satisfy restrict-template-expressions
yapf reflowed the three crossmount relay signatures and the new s3
guard test. The eslint fix is real, not cosmetic: storageId
interpolated a number straight into a template literal, which
@typescript-eslint/restrict-template-expressions rejects.
* fix(crossmount): trim slashes with rstripSlash, not a backtracking regex
CodeQL flagged all three `/\/+$/` trims in makeStorageKey as
js/polynomial-redos (high): the pattern backtracks on a path with many
repeated slashes, and the input is a caller-supplied virtual path.
utils/slash.ts already exports the loop-based rstripSlash for exactly
this reason. Same result, linear time, and it matches the trim idiom
used elsewhere in the codebase.
The python side was never affected: it uses str.rstrip('/').
* fix(crossmount): canonicalize the storage key, keep identity for bare resources
Three review findings, two of them the same data loss the guard was
meant to close.
**Nested backings collapsed onto one key.** The key joined the storage
id and the mount-relative path with a delimiter, so overlapping roots
stayed apart:
/a -> /srv/data, /b -> /srv/data/sub
/a/sub/x.txt -> disk:/srv/data:/sub/x.txt
/b/x.txt -> disk:/srv/data/sub:/x.txt
Same file, different keys, so mv wrote it onto itself and unlinked it.
Reproduced: exit 0 and the file gone from disk. The two parts are now
concatenated into one path-like string, so both render as
disk:/srv/data/sub/x.txt. s3 and redis had the same flaw with nested
key prefixes and now join theirs path-like too. A sibling that merely
shares a name prefix (/srv/data vs /srv/dataX) still stays distinct,
since the relative part always starts with a slash.
**Bare resources lost object identity.** The typescript fallback keyed
a resource with no storageId by its mount prefix, which handed one
object two identities. Browser resources (browser s3, opfs) implement
Resource directly rather than extending BaseResource, so they took
that path and a self-move still relayed a write then an unlink. A
WeakMap now assigns a serial per object, so the fix is not Node-only.
Python gained the matching getattr fallback for a resource that does
not inherit BaseResource.
**Nested function.** storage_key was a closure; the repo prohibits
nested functions. It is module level now, bound with functools.partial.
* feat(cli): recursive CLISpec atop CommandSpec
* fix(cli): CI formatter and lint fixes for the new tests
* fix(cli): whitespace-aware name check, validation into compile module
* style: yapf formatting per CI env
* refactor(workspace): split the Workspace orchestration into a workspace package in both languages
* review: type the workspace dispatch callback as a DispatchFn protocol
* fix(shell): honor line continuations and reject unterminated backticks
Two parser-level divergences from GNU bash, both pre-existing.
A trailing backslash is a line continuation. With nothing to continue
onto, bash drops it and runs the command, while mirage raised a syntax
error and exited 2, so `echo a\` produced nothing. Parity decides: only
an odd-length trailing run ends in a live continuation, since each
earlier pair is an escaped backslash. This also settles the last backtick
divergence, `echo `echo a\\`` now prints `a`.
An unterminated backtick was accepted and run. tree-sitter parses
`echo `echo a` as a complete command, so the region is scanned directly.
Quoting follows the shell reader: single quotes protect a backtick,
double quotes do not, and once inside a substitution only a backslash
escapes, which is why `"`echo '`'`"` is an error in bash rather than a
quoted backtick.
Also completes the stat/read size sweep across the last backends:
lancedb, chroma, qdrant, jaeger and langfuse .jsonl all hold the
invariant. notion page.json and langfuse trace/prompt .json are
size-unknown by design, since sizing them at listing time would cost an
extra API call per entry; both are now pinned as cold/read/warm cases and
carry a comment saying why, which is what was missing when I mistook the
same shape on trello for a bug.
* test(jaeger): scope the size invariant to the seeded services
CI caught one mismatch the local run never hit. Jaeger self-instruments,
so /j/services/jaeger/* is live telemetry that keeps growing: a trace
listed at time T can gain spans before it is read, and stat then
disagrees with the read through no fault of the sizing.
Walk only the four seeded services, named explicitly so self-telemetry
cannot drift back in, and assert an exact 8/8 so an empty glob fails
instead of passing vacuously. 101 of the 109 files in that tree were
self-telemetry.
* feat(spec): declarative count, multiple, choices, required and default option fields
* fix(types): widen fanout depth-flag helper to the int-carrying flag union
* fix(spec): a default on a multiple option lands as a one-element list
* fix(du,opfs,find): pair the du ops, move opfs to the factory, cover find predicates
Three issues that all sat on the generic command layer.
optional fields, but they are not independent. A backend setting only
the cheaper du_size silently degraded du to one operand line with no
directory rows and an inert -a. They are now one value (DuOps in
python, DuOps<A> in typescript) with both halves required, so half a
native du cannot be constructed. That makes compute_entries required
through run_du/runDu, so the degraded branch is deleted rather than
documented.
hand-registering 69 command families across 70 files. It now declares
a CommandIO like every other backend. opfs gains numfmt, od and
truncate, which it never had.
but only the malformed-value half was covered in integ. Adds nine
cases across trello, linear and discord: one precise positive filter
each, plus two unsatisfiable filters that must return empty. Those two
are the regression guard, since an ignored predicate returns the whole
tree.
* chore(spec): regenerate browser specs for the commands opfs gained
The factory gives opfs numfmt, od and truncate, which it did not
register by hand. Regenerated with scripts/gen_specs.py and
typescript/scripts/gen-specs.ts.
* test(sizes): assert the size invariant across linear, github_ci and trello
Extends the stat/read size sweep to the three remaining fake-server
backends. linear walks the whole mount, github_ci uses per-kind globs
because recursive find across runs is deliberately refused, and trello
covers its sized kinds.
trello's comments.jsonl is left size-unknown on purpose (it needs a
per-card actions call), so a cold stat reports 0 while a read returns
the real bytes. That is pinned as its own case rather than treated as a
failure, so the next reader does not chase it.
* fix(shell): keep whitespace between expansions and split adjacent backticks
Two tree-sitter tokenisation quirks made the expander drop or merge text.
Whitespace between two expansions inside double quotes was dropped, so
"$(a) $(b)" printed "ab". tree-sitter does not emit a whitespace-only run
as string content; it folds it into the following node, so the `$(` token
text is literally " $(". `${a}` and `$a` already compensated with their
own prefix handling, `$(...)`, `$((...))`, `$?`/`$#` and "${arr[@]}" did
not. One `_folded_whitespace` helper now covers every expansion branch.
Unquoted words never fold, so word splitting is unchanged.
Adjacent backtick substitutions were merged into one node when the gap
between them was empty or whitespace-only, so `a` `b` expanded to
"a` `echo b", quoted or not. The backtick region is now re-lexed from the
node's own text instead of trusting the grammar. Rewriting backticks to
$(...) before parsing would shift byte offsets, which history recording
and error snippets depend on.
Every expected value is pinned against GNU bash.
* fix(shell): respect backslash escapes when splitting a backtick region
Codex review on #677: a command substitution ending in `\\` before its
closing backtick has an escaped backslash, not an escaped backtick, so
the delimiter was being swallowed and the command text ran past it.
Consume escape pairs whole instead of counting slashes, gated to the
inside of a command and to the three characters POSIX escapes there
(`$`, backtick, backslash). Parity falls out: `\\` is consumed as one
unit, so a backtick straight after it still closes the region.
This also unescapes backtick bodies, which mirage never did, fixing a
divergence that predates the re-lex: `echo 'a\\'` in backticks now
prints a single backslash like GNU bash.
* Add a typed, spec-validated FlagView to TypeScript
Python reads command flags through FlagView, which takes the command's
spec and throws when asked for a name the spec does not declare.
TypeScript had no equivalent: commands reached into the raw dispatcher
bag with `flags.a === true`, so a name that no longer exists reads as
false rather than failing.
That failure mode is silent. Renaming tee's --append to --add in the
spec and running the suite shows it: with the raw reads tee's own tests
pass 7/7 and exit 0 while the flag has quietly become a no-op; with
FlagView the same rename fails 39 tests immediately.
Adds flagKwargName to constants (the dash-stripping rule the parser had
inlined, now shared rather than duplicated), specFlagNames and FlagView
to types, and migrates tee as a first consumer. Placement mirrors
Python: flag_kwarg_name in spec/constants.py, FlagView and
spec_flag_names in spec/types.py.
Deliberately not a sweep. The remaining commands are better migrated
alongside the parser's dest unification, which removes the both-name
reads a migration would otherwise write and then delete.
* Address Codex review: export FlagView, make asInt strict
FlagView was reachable only by deep import, so a node or browser command
could not use the sanctioned accessor. It now ships from both the spec
barrel and the package root alongside FlagValue, specFlagNames and
flagKwargName.
asInt used parseInt, which takes the numeric prefix of '5x' and returns
NaN for 'abc', and NaN still satisfies number, so a malformed value flowed
onward. It now validates the whole string and throws, matching Python's
int(): verified identical on '5x', 'abc', '', '1.5', ' 42 ', '-7', '+7'
and '1_0'.
* Migrate the first generic commands onto FlagView
13 commands now read flags only through the spec-validated view, and 9
more are partly across. Done in verified batches rather than one sweep:
each batch is typechecked and run against the full core suite, which
caught three bad rewrites (an insertion landing mid-expression in
unexpand and expand, one inside an object literal in tree, and reads
rewritten in basename with no declaration in scope because its command
is an arrow function).
Those all failed at the compiler, which is the argument for batching: a
single 53-file commit would have buried them.
* Migrate the bulk of the generic commands onto FlagView
35 of 45 flag-reading commands now go only through the view; 10 have
30 sites left that need hand edits.
Two real defects the batches caught, neither by tests alone:
split -d and -x, and mktemp -p, take an optional value, so they arrive
as boolean true when passed bare. Rewriting them to asStr dropped that
form and silently disabled numeric suffixes; split's own test caught it.
An audit of every asStr call against its option's valueKind found the
other three sites of the same class, now on raw().
stat's chained ternary was mangled by a partial regex match into
'(fl.asStr(c) ?? typeof) opts.flags.f', which only failed because it
would not parse. It is now a plain ?? chain.
Also removed an unreachable instanceof PathSpec branch in shuf: the
parser only ever stores strings, booleans and arrays in the flag bag.
* Finish the FlagView migration and enforce it with a lint rule
Every generic command now reads flags only through the spec-validated
view: zero direct opts.flags reads remain. The last sites were the list
shapes in awk and sed (Array.isArray(x) ? x : typeof x === 'string' ?
[x] : [] is exactly asList), comm's and join's numeric-keyed selectors,
and xxd's local toInt over the raw union.
The lint rule is the point of going to 100%. Partial adoption protects
only the migrated files; banning opts.flags outright makes drift
impossible for commands nobody has written yet. Its selector is scoped
to opts.flags rather than any .flags member, because a first attempt
flagged RegExp.prototype.flags in zgrep. Mutation-checked: putting one
raw read back in tee fails lint.
* Guard the Python side against raw flag reads too
Python needed no migration: its wrappers already bind flags to typed
parameters and use FlagView for the remaining spellings, so the generic
commands never see a bag at all. Auditing the whole command tree found
zero raw reads.
What it lacked was the enforcement TypeScript just gained, so this adds
the mirror of the lint rule as a test. Reads only: crossmount fanout
legitimately assigns into a bag it builds for the sub-commands it
dispatches, which a first cut flagged. Mutation-checked by adding one
flags.get back to tac.
* Apply yapf formatting to the flag-read guard
The hook rewraps the assert message. My earlier local run reported clean
before this file's last edit, so the reformat only surfaced in CI.
* refactor(spec): compile specs once and unify flag dests onto the long spelling
parse_command rebuilt its lookup tables from the spec on every
invocation; they now compile once per spec into a cached CompiledSpec
(spec/compile.py), with attached-value candidates ordered longest-first
so -name can never lose a match to -n by set iteration order.
Both spellings of an option now land on one canonical dest (the long
name when both exist), so cp --update=all -u is last-wins without the
per-spelling mirror, sort -k1 --key=2 -k3 accumulates in true command
line order, and the both-name or-chains in 12 generics collapse to one
read. spec_flag_names returns canonical names only, so a stale short
read raises through FlagView instead of silently reading False; the
sites that raised are fixed here (11 or-chain generics, the wc
formatter helpers, fanout's head/tail header forcing, and the s3 mkdir
wrapper that read flags through signature params).
* refactor(spec): mirror CompiledSpec and dest unification to TypeScript
Same shape as the python change: compile.ts lowers a CommandSpec once
(WeakMap-cached) into the tables parseCommand walks, both spellings of
an option land on the canonical long dest, and specFlagNames returns
canonical names only so stale short reads throw through FlagView.
Sweeps the TS readers the canonical dests surface: the or-chain
generics, sort.ts's per-spelling alias helpers (whose short-then-long
list concatenation lost key order), the shared checksum and base64
helpers, fanout's head/tail header forcing, and the fallback-less
wrappers that read flags through signature params or bare short keys
(s3/gridfs/opfs mkdir, postgres head pushdown, mongodb tail -f,
curl -f).
Also fixes the same wrapper class on the python side, found by codex
review and the integ reds on the previous commit: curl -f exit 22,
gridfs mkdir -p/-v, mongodb tail -f, and head -c provisioning (the
parity mismatch: provision read 'c' while head now delivers 'bytes',
degrading the exact estimate to a range).
* fix(provision): head byte caps arrive as the canonical 'bytes' dest in TS too
Same fix as the python side: the head/tail provision read only 'c',
but head now delivers the canonical 'bytes', so head -c priced as a
0..size range instead of exact (the prov_head_c reds on integ-ts,
integ-shared-ts and integ-shared-parity). tail's -c stays short-only
and still arrives as 'c'.
* Add a typed, spec-validated FlagView to TypeScript
Python reads command flags through FlagView, which takes the command's
spec and throws when asked for a name the spec does not declare.
TypeScript had no equivalent: commands reached into the raw dispatcher
bag with `flags.a === true`, so a name that no longer exists reads as
false rather than failing.
That failure mode is silent. Renaming tee's --append to --add in the
spec and running the suite shows it: with the raw reads tee's own tests
pass 7/7 and exit 0 while the flag has quietly become a no-op; with
FlagView the same rename fails 39 tests immediately.
Adds flagKwargName to constants (the dash-stripping rule the parser had
inlined, now shared rather than duplicated), specFlagNames and FlagView
to types, and migrates tee as a first consumer. Placement mirrors
Python: flag_kwarg_name in spec/constants.py, FlagView and
spec_flag_names in spec/types.py.
Deliberately not a sweep. The remaining commands are better migrated
alongside the parser's dest unification, which removes the both-name
reads a migration would otherwise write and then delete.
* Address Codex review: export FlagView, make asInt strict
FlagView was reachable only by deep import, so a node or browser command
could not use the sanctioned accessor. It now ships from both the spec
barrel and the package root alongside FlagValue, specFlagNames and
flagKwargName.
asInt used parseInt, which takes the numeric prefix of '5x' and returns
NaN for 'abc', and NaN still satisfies number, so a malformed value flowed
onward. It now validates the whole string and throws, matching Python's
int(): verified identical on '5x', 'abc', '', '1.5', ' 42 ', '-7', '+7'
and '1_0'.
* Migrate the first generic commands onto FlagView
13 commands now read flags only through the spec-validated view, and 9
more are partly across. Done in verified batches rather than one sweep:
each batch is typechecked and run against the full core suite, which
caught three bad rewrites (an insertion landing mid-expression in
unexpand and expand, one inside an object literal in tree, and reads
rewritten in basename with no declaration in scope because its command
is an arrow function).
Those all failed at the compiler, which is the argument for batching: a
single 53-file commit would have buried them.
* Migrate the bulk of the generic commands onto FlagView
35 of 45 flag-reading commands now go only through the view; 10 have
30 sites left that need hand edits.
Two real defects the batches caught, neither by tests alone:
split -d and -x, and mktemp -p, take an optional value, so they arrive
as boolean true when passed bare. Rewriting them to asStr dropped that
form and silently disabled numeric suffixes; split's own test caught it.
An audit of every asStr call against its option's valueKind found the
other three sites of the same class, now on raw().
stat's chained ternary was mangled by a partial regex match into
'(fl.asStr(c) ?? typeof) opts.flags.f', which only failed because it
would not parse. It is now a plain ?? chain.
Also removed an unreachable instanceof PathSpec branch in shuf: the
parser only ever stores strings, booleans and arrays in the flag bag.
* Finish the FlagView migration and enforce it with a lint rule
Every generic command now reads flags only through the spec-validated
view: zero direct opts.flags reads remain. The last sites were the list
shapes in awk and sed (Array.isArray(x) ? x : typeof x === 'string' ?
[x] : [] is exactly asList), comm's and join's numeric-keyed selectors,
and xxd's local toInt over the raw union.
The lint rule is the point of going to 100%. Partial adoption protects
only the migrated files; banning opts.flags outright makes drift
impossible for commands nobody has written yet. Its selector is scoped
to opts.flags rather than any .flags member, because a first attempt
flagged RegExp.prototype.flags in zgrep. Mutation-checked: putting one
raw read back in tee fails lint.
* Guard the Python side against raw flag reads too
Python needed no migration: its wrappers already bind flags to typed
parameters and use FlagView for the remaining spellings, so the generic
commands never see a bag at all. Auditing the whole command tree found
zero raw reads.
What it lacked was the enforcement TypeScript just gained, so this adds
the mirror of the lint rule as a test. Reads only: crossmount fanout
legitimately assigns into a bag it builds for the sub-commands it
dispatches, which a first cut flagged. Mutation-checked by adding one
flags.get back to tac.
* Apply yapf formatting to the flag-read guard
The hook rewraps the assert message. My earlier local run reported clean
before this file's last edit, so the reformat only surfaced in CI.
* test(slack): assert the size invariant over every user and channel day
The users/*.json size comes from the users.list member, but read renders
the users.info response, so the advertised size is exact only while those
two endpoints encode identically. Verified live on 2026-08-01: every real
user in a live workspace matched byte for byte, key order included.
Pinning that verification in integ where it can be: two cases compare
stat -c %s against cat | wc -c across all 11 users and all 90 channel
days rather than pinning one constant for one file, and count the matches
so an unexpanded glob fails instead of passing silently.
Both comments now record what integ cannot cover: the fake builds
users.list and users.info from one row through one helper, so they agree
by construction and a Slack-side divergence would stay green.
* test(discord): assert the size invariant over every member, day and file
Same treatment as slack, one commit earlier. Three cases compare
stat -c %s against cat | wc -c across all 3 members, all 60 channel days
and every attachment, counting matches so an unexpanded glob fails
instead of passing silently.
The attachment case is the one that carries weight. members and
chat.jsonl are sized from the same endpoint read fetches, so they can
only drift by staleness, but an attachment's size comes from Discord's
reported attachment.size while read returns the downloaded bytes, which
is the same class of assumption github_ci artifacts needed checking for.
It also covers what absolute pins cannot: the rendered chat.jsonl for a
day holding an attachment embeds the CDN url, which carries the server's
port, so its byte count differs between an in-process ephemeral port and
CI's fixed one. Comparing stat to wc -c is port-independent.
* test(mail): assert the size invariant over every gmail and email message
Same treatment as slack and discord. Four cases compare stat -c %s
against cat | wc -c across all 11 gmail messages and 6 attachments, and
all 5 email messages and 2 attachments, counting matches so an
unexpanded glob fails instead of passing silently.
email is the one with a real divergence risk, and unlike the slack and
discord fakes the integ server can actually expose it. readdir sizes a
message from fetch_headers, which despite the name pulls BODY.PEEK[] and
parses it with _parse_multi_fetch; read pulls the same bytes through
fetch_message and parses them with parse_rfc822. Two parsers, one
renderer, so a dict that differs by one field or one key makes the
advertised size wrong. GreenMail is a real IMAP server, so these cases
exercise both parsers against real RFC822 bytes rather than a fixture
that agrees by construction.
gmail is lower risk: readdir and read both call messages.get with
format=full, so only staleness can drift them.
* Split the TypeScript builtin command specs by family
builtins.ts held all 93 command specs in one 1189 line table while
Python splits the same specs across nine family modules. The TypeScript
side now mirrors that layout: builtin_specs/ holds archive, fs_mutate,
hashing, listing, net, runtime, search, text_proc and viewing, with an
index that aggregates them and raises on a duplicate name the way
Python's __init__ does. builtins.ts keeps specOf and BUILTIN_SPECS, so
no caller changes.
The command to family assignment was taken from Python's modules rather
than reassigned by hand, and the two key sets already agreed exactly at
93 names.
This is a pure move. Regenerating the spec dump from the split code
produces byte identical JSON for all 93 commands in both the node and
browser outputs, which is the gate added in #666 doing the job it was
built for. The gate is not vacuous: dropping a single spec fails
generation with "no builtin spec: wget".
The null prototype freeze on BUILTIN_SPECS is preserved, so a name like
toString still misses rather than resolving an Object.prototype member.
* fix(spec): index the split spec modules without a type assertion
* Consolidate S3 provider configs and make credentials optional
Two cleanup items.
The 14 S3-compatible provider configs were each a hand-written copy of
the same credentials, endpoint rule and S3Config conversion. They now
share a base (s3_alias.py, s3_alias.ts) and declare only their endpoint
rule, which removes about 680 lines. Python had no test coverage for any
of them, so a characterization test went in first and the refactor
happened under it.
Credentials are now optional on every provider in both languages.
S3Config already left them optional, so omitting them falls through to
the usual AWS resolution order: profile, environment, shared credentials
file, instance role. Only r2 exposed that before, and even r2 required
keys alongside the profile it accepted, so the profile could never be
used on its own. session_token and aws_profile moved onto the shared
base, which also deleted the bespoke to_s3_config overrides on r2 and
supabase. Required is now bucket plus whatever the endpoint needs.
Also from the stale-debt pass: a resolved issue doc is deleted, the
TypeScript forwardIndex knob keeps its comment corrected to the real
reason it exists (WorkspaceFS reaches OpsRegistry directly and has no
equivalent of Python's Ops on_write hook), with index-invalidation tests
added in both languages for the dispatch and fs paths, and gen_specs.py
now fails loudly when a command module will not import instead of
silently dumping a spec set missing every optional backend.
* Ignore docs/plan/ so plan docs cannot be committed again
The gitignore covered docs/plans/ but not the singular docs/plan/, which
is how the one tracked plan doc got committed. That doc is deleted in
this branch; this closes the hole behind it.
* Fix eslint errors the local run missed
Two typed-lint errors that only surface once the TypeScript dists are
built: a forbidden non-null assertion in the index invalidation test and
a type parameter used once in the alias optional helper.
* Honor a named profile in the TypeScript S3 client
The alias configs forwarded aws_profile into S3Config but createS3Client
never read it, so a named profile was silently dropped and the default
credential chain signed with whatever it found, which can reach the
wrong account. The SDK takes profile as a per-client AWS_PROFILE, so
forwarding it is enough; explicit keys still win, matching the
precedence Python gets from aioboto3.Session(profile_name=...).
Found by Codex review on this PR. Pre-existing for r2, which already
forwarded the field, and made reachable everywhere by this branch.
* feat(sizes): render-at-readdir sizes for github_ci, trello and notion
Batch 2 of the fskit size push-down (plan: ~/Desktop/fskit.md). Each
listing already carries the payload read renders, so readdir seeds a
sized IndexEntry and stat serves it. Zero added API calls.
github_ci workflow/run/job JSON and trello workspace/board/list/card
JSON use the shared node-dir shape: the parent readdir set_dirs each
child with a sized node.json, and the child readdir consults list_dir
before bootstrapping the parent. Notion differs because a database dir
listing is dynamic, so database.json's size travels in
extra[database_json_size].
Artifacts stay class T: size_in_bytes is the zip's exact byte length,
confirmed against two live samples. Flags stay False for all three:
job logs, annotations, comments.jsonl and page.json are class N.
integ gains a github_ci target backed by a fixed Actions dataset in the
fake server, with 24 stat==read pins.
* feat(sizes): render-at-readdir sizes for jaeger, qdrant and lancedb
Batch 3 of the fskit size push-down. qdrant and lancedb flip
SIZES_ALWAYS_KNOWN; jaeger stays False because operations.json is class
C and traces are class P.
jaeger traces are genuinely P: verified live that the /api/traces search
document renders byte-identically to the by-id fetch for all four seeded
traces, including the one listed under two services. operations.json is
sized by one /api/operations call at the service dir readdir.
lancedb needed table_columns/tableColumns to widen the select beyond
ids; it excludes vector and blob columns in schema order so projected
rows render byte-identically. Blob entries stay unsized on purpose,
since sizing a blob at listing time would let one undecodable value fail
the whole directory listing rather than just that file's read.
A previously pinned integ expectation was wrong: jg_read_28 asserted du
-sh printed 0B, which was an artifact of unknown sizes, and now prints
the real total.
* feat(sizes): sizes for discord, email, langfuse and chroma
Batch 4, the last of the fskit size push-down. discord, email and chroma
flip SIZES_ALWAYS_KNOWN; langfuse stays False because traces and prompts
are class N.
chroma's producer-supplied size was measurably wrong for every file in
our own fixture: 180 declared against 166 rendered, 90 against 52, and
150 against 159, which truncated reads. It describes the source document
rather than the chunk join mirage serves, so it can never be the byte
length. It moved to extra.source_size and the real size is measured by
one batched chunk scan per directory, run lazily from stat and cached.
This also corrected provenance byte accounting and du totals.
langfuse items.jsonl follows the class C shape: the dataset dir readdir
makes the one call and seeds a sized entry.
Carries the discord fake-API arc. The new integ/server/discord_server.py
exposed four real backend bugs: message pagination used the oldest id as
the after cursor when discord answers newest-first, attachments were
classified as directories, unknown paths returned an empty listing
instead of ENOENT/ENOTDIR, and a cold ls of a guild dir ENOENTed in
python only.
* fix(sizes): propagate backend failures from the stat parent-listing fallback
Review follow-up on the stat fallback these three backends use to
populate a cold index: the TypeScript side caught every error from the
parent readdir and fell through to ENOENT, so an auth failure, a rate
limit or a transport error read back as "file not found". Python already
caught only FileNotFoundError. Now only isMissingPath errors fall
through, which is the rule that helper's own contract states.
Reachable through the new size lookups, since workspace.json and the
other sized files resolve through this path on a cold index.
Also pins that discord's base_url config key reaches DiscordResource as
baseUrl through the registry: normalizeFields snake-to-camels by default
so no explicit rename is needed, and a silent miss there would send
configured mounts to the public API.
* feat(spec): cross-language command spec parity check
Add scripts/check_spec_parity.py and wire it into CI next to the existing
spec drift gate. It diffs the generated python and typescript spec trees
command by command: every option (help text, value kind, repeatability,
shorthand), every operand, the _meta flags, and the resource set each
command registers under. Structural divergences live in
spec/parity_exceptions.json with a reason, and a stale entry fails the
check so an exception cannot outlive what it documents.
gen-specs.ts could only see command groups the package index re-exports,
so HISTORY_COMMANDS and GRIDFS_COMMANDS were silently missing from the
dump. It now asserts every builtin *_COMMANDS is reachable, and both
backends are exported.
The check found four real bugs:
shuf required a write op it only needs for -o. TypeScript marked the
builder write:true and threw before doing anything, so shuf did not
register on 21 read-only backends. Python registered it but eagerly
called ops.require(WRITE), so every invocation raised. Both now resolve
the write op lazily, matching sed -i / sort -o / uniq -o.
The TypeScript factory only gated on the generic write op, so truncate
registered on databricks_volume, dropbox and the hf backends and rmdir on
the hf backends, none of which have those ops. Ported Python's
requirements mechanism; the tables are now identical in both languages.
TypeScript search carried no cost provisioning on qdrant, lancedb or
mem0, where Python does.
js and node had no help text in TypeScript, and curl, history and seq
differed. Aligned per option.
* fix(spec): compare spec metadata per resource, not as unions
Addresses two review findings on the parity gate.
gen-specs.ts treated any index.ts read failure as an absent file, so a
permission or IO error silently dropped every command group in that
directory, defeating the reachability assertion exactly when the source
scan was incomplete. Only ENOENT continues now.
The _meta union flags cannot say which resource carries a provision, an
aggregate, the write flag or a filetype, so dropping one backend's
provision while another kept it left every union unchanged and the gate
accepted the regression. Both generators now emit _meta.by_resource and
the checker compares keyed by resource, falling back to the unions only
once the per-resource entries agree.
That found six more divergences the unions had masked:
Python's history cat, grep, head, tail and wc carried no aggregate where
TypeScript did, and the same for github and email grep. Added them.
TypeScript had no github du provision where Python does. Added it.
TypeScript still has no github grep provision. Python's is a bespoke cost
model that walks the index and reaches IndexCacheStore internals, so it
is recorded in parity_exceptions.json for its own change rather than
ported here.
Exemptions are now granular: by_resource names one resource and one key,
so exempting github grep's provision cannot also hide the aggregate
divergences on the same command. An exemption counts as used only when it
suppresses a live divergence, so the stale check no longer misreports an
exemption that fully covers its diff.
* feat(shell): export/readonly -p print mode (py + ts, #608)
Bare export/readonly and the -p form print declare -x/-r lines, matching
bash. Invalid option clusters fail with status 2 and the GNU usage line.
Array readonly names print as declare -ar. Unit tests and integ cases
cover both runtimes.
* fix(shell): match GNU on -- , control chars and type flags in -p print
Three defects in the export/readonly print path, all pinned against
bash 5.2 in debian:stable-slim:
- `export -p --` / `readonly -p --` failed with exit 2. The declaration
word collector folded any `-`-leading word into a char set, so `--`
contributed a literal `-` that the option validator then rejected.
Option words are now kept verbatim and in order, and `--` ends option
parsing. Keeping the order also makes the invalid-option message name
the *first* bad letter, as bash does; the old set was hash-ordered, so
`export -zq` reported `-z` or `-q` at random between processes.
- Only newline and carriage return selected `$'...'` quoting, and even
there only \n \r \t were escaped, so a tab, escape or DEL byte was
emitted raw inside double quotes. Any control character now selects
ANSI-C quoting, with bash's named escapes (\a \b \t \n \v \f \r, \E)
and three-digit octal for the rest. Printable non-ASCII stays literal,
which is what bash emits in a UTF-8 locale.
- Type-selecting flags were parsed and discarded, so `readonly -f`
listed variables and `readonly -a` listed scalars. `-a` now narrows to
indexed arrays; `-f` and `-A` list nothing, mirage carrying no readonly
or export attribute for functions or associative arrays.
Python and TypeScript stay mirrored. Adds 8 pytest cases, 6 vitest
cases, and 5 integ cases whose expectations were verified byte-for-byte
against GNU bash.
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
jq_eval collapsed a program's output stream into a single value: one
output bare, many as a list, zero as a JQ_EMPTY sentinel. A list of
outputs then reads the same as one output that is an array, so the
printer recovered the difference from the program text by scanning for
a `[]` at bracket depth 0 (has_top_level_spread).
Text cannot answer that question. Any multi-output program with no
literal `[]` printed a JSON array instead of one line per value, so
`jq -r '.a, .b'` printed `[1,2]`, and so did `range(3)` and `..`.
jq_eval now returns every output in order, format_jq_output prints one
per line, and the heuristic plus both JQ_EMPTY sentinels are gone: an
empty list means no output. Behavior is pinned against GNU jq 1.7 in
debian:stable-slim.
* chore(filetype): drop the eight provision helpers left with no caller
s3, ssh, redis, gridfs, nextcloud, onedrive, sharepoint and
databricks_volume each carried a _provision.py whose only consumer was
the provision= argument of the filetype command factory removed in
#651. Backends that also reach file_read_provision from their own grep
path keep theirs.
* fix(integ): keep the redis provisions and cover them with probe cases
The redis _provision.py is not an orphan of the filetype factory: it is
the public surface the redis example imports, mirroring the publicly
exported redis/provision.ts on the TypeScript side. Deleting it broke
examples/python/redis_resource/example_redis.py, which the Python
examples CI step runs.
It had no integ coverage, which is why nothing caught that. Adds
integ/resources/redis/probe.json with five provision cases running on
both hosts. Values are measured, not guessed: head -n 2 reports
net=0-24 precision=range, which is the hand-written head_tail_provision
bounding the read rather than claiming the whole file.
* test(integ): pin provision behavior for gdrive, google apps and trello
These four backends hand-write provision modules in one language or the
other, and none of them had provision coverage, so a divergence between
the hosts was invisible. Every value here is measured, not guessed.
The numbers differ per backend in ways worth pinning: redis reads over
the wire (net=24), gdrive serves the same file from its index cache
(cache=24 hits=1), a rendered .gdoc.json cannot predict its byte count
(precision=unknown), and head bounds the read as a range rather than
claiming the whole file.
Cases run on both the python and typescript-node hosts, so the layout
difference between the two languages is now gated on behavior.
* fix(provision): charge ops for remote listings, add search provision to TS
exactZeroProvision documents itself as being for backends whose listing
is already in memory, so metadata commands cost no backend I/O. gdocs,
gsheets, gslides and trello are remote API backends whose readdir hits
the wire, and TypeScript wired all four to it: ls reported ops=0 while
Python reported ops=1 for the same listing. Dropping the override lets
them fall through to the op-charging default, which is both the correct
estimate and the layout Python already had.
TypeScript's lancedb and qdrant search carried no provision at all where
Python's did; both now use exactZeroProvision, which is right there
because the search result is materialized in memory. The regenerated
search spec flipping to has_provision: true is that fix showing up.
The gdrive _provision.py had no consumer but its own test, and TS wires
nothing for gdrive, so both are gone.
Every change is pinned by a probe case measured on both hosts.
* fix(shell): fully general $? semantics, C-style for, and observed mtimes for find
* fix(ci): camel bg launch via $! and eslint require-await in statement test
* feat(ssh): native setattr via SFTP, stat surfaces mode and atime like disk
* fix(lint): narrow ssh2 nullable callback errors in ssh set_attrs
* fix(shell): reject readonly assignments in C-style for slots
* integ: pin the C-style for readonly abort
* fix(policy): fail loud on entry-script verdict shapes, stop fanout swallowing
* fix(monty): raise typed fs exceptions in the ts guest
* feat(quickjs): evaluator capability in both languages, js policy scripts
* feat(local): host python runtime for the node package
* test: align the script loader wording
* style: satisfy the ci linters
* fix(runtime): reclaim the local subprocess on safeguard timeout, tolerate stdin EPIPE
* fix(quickjs): bridge the workspace fs into eval, match the real engine's output surface
* feat(sizes): render-at-readdir sizes for slack and gmail; flip mem0 and opfs size-known
* test(integ): pin slack file, dm and empty-day chat, and gmail attachment sizes
* test(fuse): use notion as the size-unknown specimen for the fskit guard
* fix(slack): skip tombstoned and access-restricted file payloads in listings
* refactor: rename the runtime route to policy
* feat(policy): extensible verdict with a deny arm
* refactor(policy): move the safeguard resolver into the policy package
* fix(policy): rename the route key in the ts server config, fix the integ import
* fix(policy): record denied lines in ts history, type the verdict docstring
* fix(policy): attribute deny and admission errors to the command
* feat(policy): typed verdict arms RouteResult and DenyResult
* docs: type the policy example return value
* integ: move the safeguard scenarios to the JSON runtime harness
* fix(policy): gate syntax errors before the policy in python
* fix(shell): report the full syntax error span like bash echoes the line
* fix stat sizes across github, hf, databricks_volume and sharepoint; mark them size-known
* test(hf): move the metadata-stripping list doubles to module scope
* test(integ): pin the size backfills and the truncated-tree fallback
* feat(fuse): size push-down T flips for onedrive, box, dropbox, ssh and nextcloud
* style: prettier format for the nextcloud readdir test
* fix(box,ssh): follow ssh symlinks in TS stat, hide weblinks from direct box stat
* test(integ): pin ssh symlink sizes and box weblink hiding in the shared battery
* test(ssh): add lstat to the mock SFTP client
* refactor(runtime): evaluator capability, route engine on runtimes, drop PythonRuntime
* refactor(runtime): split base into types/errors/mixin, eval transport belongs to the runtime
* docs: module layout convention (types/errors/config/mixin/base)
* docs: trim module layout section
* review: brand the evaluator, bind session inputs and bytes on pyodide
* fix(runtime): refuse unbound interpreters with GNU command-not-found wording
* fix(integ): survive connection resets while jaeger boots
* docs: evaluator page under learn, language icons for the runtime pages
* docs: reframe the evaluator page as the policy engine
* docs: both entries capture python3 in the policy engine example
* docs: real routing case with the ctx payload and verdict semantics
* docs: drop the engine-only yaml block, one parenthetical instead
* docs: live-captured ctx payload, verified config, function route signature
* docs: trim the policy engine section
* fix(fskit): downgrade the size guard from a refusal to a warning
* feat(linear): push file sizes down to readdir so stat matches read
Every linear file is now sized at its parent directory's readdir from the
payload the listing already fetched: team.json from the teams listing,
issue.json from the issues listing, member/project/cycle/document JSON from
their own listings. comments.jsonl costs one bounded comments call, paid
only when the issue directory is entered. Listing a tree never fetches file
content, and stat reports the exact rendered byte length, which is what
fskit needs at lookup time to serve reads. SIZES_ALWAYS_KNOWN flips on.
Invariant pinned in both languages (stat size equals read length for every
file in the tree) and in integ with paired stat/wc cases against the fake
Linear server.
* fix(fuse): drain buffered writes on release, warn on writable fskit mounts
The macFUSE FSKit shim issues WRITE then RELEASE with no FLUSH in between
(the kext always flushes on close), so MountCore dropped every buffered
kernel write at release. Release now drains the handle's write buffer in
both languages.
Testing the CLI over a live fskit mount also measured a shim bug mirage
cannot fix: pages for regions a file did not already have (a new file, an
empty file, a truncate-then-write) flush as NUL bytes of the right length,
and appended regions arrive intact or zeroed depending on cache state,
with no error surfaced to the writer. A new check_writes guard warns at
mount time for writable fskit mounts (metadata ops are reliable; /dev is
excluded), and the behavior is pinned in integ/truth_fskit.json plus a new
CLI battery, integ/cli_fskit.sh, which drives both daemon-backed CLIs
through a real fskit mount and skips cleanly on hosts that cannot engage
the FSKit module. cli_fuse.sh now also asserts that a backend fskit
workspace is rejected cleanly off macOS.
* refactor(integ): move the fuse truth from txt lines to JSON
* refactor(integ): gather the fuse and fskit harness under integ/fuse/
* docs(fuse): document the fskit write-path limits and the size warning
* docs(fuse): tighten the fskit warnings
* refactor(runtime): one constructor shape for every runtime
* integ(runtime): one JSON suite for all runtimes across SDK and CLI
* integ(runtime): drop the e2b suite, docker stays the sandbox coverage
* integ(runtime): cover the --runtime error paths
* fix(integ): keep battery case discovery out of integ/runtime, satisfy ci yapf
* fix(grep): -r with no path operand searches the cwd like GNU
* fix(rg): bare rg searches the cwd like ripgrep, rename rebase_* to respell_*
* fix(cwd): bare find, tree, du and ls default to the cwd like GNU
* integ: cover routing gaps, fix redis du golden and rg battery stderr
* integ: materialise folder-backed mount roots at seed time
* fix(interpreter): refuse uncaptured python/js lines instead of a hidden fallback
* integ: seed the onedrive /data2 root too
* refactor(fuse): split the mount layer into MountCore plus a libfuse adapter
MirageFS was one class doing two jobs: filesystem semantics and talking to
libfuse. The FUSE-specific part was smeared across every method rather than
sitting at a boundary, which is why it could not be reused by another kernel
interface.
MountCore (fuse/core.py, fuse/core.ts) now owns all semantics and imports
nothing from mfusepy or fuse-native. MirageFS keeps only the callback
signatures and error translation. The attr dicts stay as they were: st_mode
and friends are POSIX stat field names, not FUSE ones.
The load-bearing change is error handling. Python raised FuseOSError inline
in ten methods with hand-rolled except chains while TypeScript already had a
single classifyError. Both languages now share one classification table
(fuse/errors.py, fuse/errors.ts), so the same backend failure reports the
same errno on both sides.
That fixes a real bug. Python's rmdir caught OSError before
FileNotFoundError, and FileNotFoundError is an OSError subclass, so rmdir on
a missing path returned ENOTEMPTY instead of ENOENT. No test covered it;
test_errors.py now pins it.
Behavior is otherwise unchanged: the pre-existing suites (test_fs.py 689
lines, fs.test.ts 43 tests) pass untouched apart from renaming private
attribute reads onto .core.
* feat(fuse): fskit mount backend, kext-free mounts on macOS
Adds backend selection to a FUSE mount: Mount(..., fuse_backend="fskit")
routes through macFUSE 5.x's FSKit shim, so the mount runs with no kernel
extension loaded. Closes#82.
Three rules are enforced at mount time rather than discovered at run time:
- macOS only, raising elsewhere instead of silently dropping the option.
- The mountpoint must live under /Volumes. FSKit refuses anything else,
which is what made the reporter's /tmp attempt in #82 fail. The rule is
owned by FuseManager, so no call site has to know it.
- Every mounted resource must be able to size its files. FSKit has no
direct_io, so reads are driven entirely by the reported size: a resource
that stats as 0 before open would serve an empty file with exit code 0.
Mirage refuses such a mount and names the offending prefixes.
That last rule needs a new capability, SIZES_ALWAYS_KNOWN / sizesAlwaysKnown,
default false. Opted in on the byte stores (ram, disk, redis, s3, gridfs)
plus dev and history. History matters more than it looks: it is mounted into
every workspace at /.bash_history, so leaving it false would block every
root-scoped fskit mount.
There is deliberately no "auto" value. Auto-selecting fskit would silently
break every API-backed mount, and an option whose safe value is always the
default is a trap.
TypeScript cannot reach FSKit at all: @zkochan/fuse-native bundles a
pre-macFUSE-5 dylib, so the option never arrives at a driver that
understands it. checkPlatform throws rather than quietly mounting through
the kext. The capability and guards are still mirrored so the semantics
match. Documented as a known gap.
Examples cover both sides of the limit: ram, redis and s3 mount and read
under fskit; slack and gmail show the guard refusing and the supported
workaround of scoping the mount to a byte-store subtree.
* refactor(mount): one backend field, vfs by default
Replaces the fuse boolean and the fuse_backend string with a single
MountBackend StrEnum: vfs, fuse, fskit. VFS is the default and finally has a
name, instead of being spelled fuse=False.
Mount(resource, backend=MountBackend.FUSE)
Mount(resource, backend=MountBackend.FSKIT, mountpoint="/Volumes/x")
Three things this fixes.
The default was previously defined by negation. Most mounts are not
kernel-mounted at all, and that case is what mirage is mostly used for, so it
deserves a name rather than a falsy value.
The overloaded bool | str union is gone. fuse=True and fuse="/path" packed
two decisions into one field; backend and mountpoint separate them.
The enum no longer claims a name it does not own. fuse_backend="fskit" read
as "use FSKit", but which delivery mechanism reaches FSKit (macFUSE 5's shim
today, a native Swift module later) is mirage's business, not the caller's.
One value covers both.
MountBackend lives in mirage.types next to MountMode rather than in
mirage.fuse, so the mount spec does not have to depend on the fuse package
for a value that also covers the non-fuse case. resolve_backend rejects vfs:
reaching it means a kernel mount was requested, and vfs registers nothing.
Config, server routers, and the TypeScript mirror follow, including the YAML
schema (backend: fuse plus mountpoint: <path>) and fuseMounts becoming
kernelMounts.
* fix(mount): migrate the CLI config path and integ fixtures to backend
The backend rename missed three call-site shapes that a literal search for
fuse=True / fuse: true does not catch:
- YAML mountpoints written from shell variables (integ/cli_fuse.sh wrote
`fuse: $dmnt`, five of them), which is the CLI's own end-to-end path.
- Mount options passed a variable rather than a literal (integ/fuse.py and
integ/fuse.ts both used `fuse=pinned`).
- The TypeScript server config schema, which still declared
`fuse?: boolean | string` and exported fuseMounts. It now mirrors Python:
backend plus mountpoint in, kernelMounts out.
The CLI itself has no fuse flag, so its whole surface is the workspace YAML.
Both config loaders were checked against the exact document cli_fuse.sh
writes and agree: /data and /logs resolve to (fuse, <path>), a backend with
no mountpoint resolves to (fuse, None), and a mount with no backend key stays
on vfs and is absent from the kernel mounts.
CLAUDE.md documents the single backend field and the YAML keys.
* refactor(mount): missing is vfs everywhere, and one guarded entry point
Two related cleanups.
Missing meant two different things. Mount.backend defaulted to VFS and an
absent YAML key resolved to VFS, but resolve_backend(None) returned FUSE,
reinterpreting an absent value as a request for a kernel mount. Now every
absent value lands on VFS, and the mount entry points spell their own
default: backend: str | MountBackend = MountBackend.FUSE. The intent is in
the signature instead of hidden in the resolver.
resolve_backend is pure coercion again; require_kernel_backend is the
separate step that rejects VFS. The unknown-name error lists all three
values rather than just the mountable two.
The fskit guards were three separate calls at each mount path, so a new path
could pick up fskit support and silently skip the macOS assert, the /Volumes
rule, or the size check. prepare_backend now runs all of them, and every
mount path routes through it: mount_background, mount, and FuseManager.setup.
Tests pin that a linux platform raises through prepare_backend, not just
through check_platform directly.
Same shape in TypeScript: resolveBackend, requireKernelBackend, prepareBackend.
* fix(fuse): drop the polynomial regex from unsizedMounts
CodeQL js/polynomial-redos, two high-severity alerts on the same helper.
`prefix.replace(/\/+$/, '')` backtracks on a string of many repeated
slashes, and the input is a mount prefix, so it is library input.
Uses core's loop-based rstripSlash instead, which is the existing sanctioned
trim for exactly this. Python was never affected: str.rstrip is a C loop,
not a regex.
* fix(fskit): match the mount recipe and volume ownership verified in #82
Three corrections to the fskit path, all found by re-reading issue #82
against what we actually shipped.
Mount options. #82's only reported working mount passes backend=fskit AND
volname, and omits direct_io. We passed direct_io unconditionally and no
volname, on my assumption that direct_io is simply inert on this path. That
assumption contradicted the one person who has run it, and direct_io is a
libfuse concept, so the shim needing it removed is not surprising. Now
matched exactly, with a comment telling the next person not to restore it.
Mountpoint ownership, which would have broken every fskit mount. /Volumes is
drwxr-xr-x root:wheel, so tempfile.mkdtemp(dir="/Volumes") raises
PermissionError for any non-root user. The reporter mounted at
/Volumes/mirage-* as a normal user, so macFUSE creates the entry: an FSKit
mount is a volume, not a directory we make. The fskit path now names its
mountpoint and never creates it, skips makedirs for a pinned one, and never
rmdirs a /Volumes entry on unmount.
Readiness. The /Volumes entry does not exist until the volume is live, so
bare existence is a real ready signal there, exactly as for WinFsp, and it
does not depend on os.path.ismount recognizing the mount.
These are the same three deviations WinFsp already needed, for the same
underlying reason: the filesystem driver owns the mountpoint. Tests pin the
option set and all three ownership rules, because nothing in CI can reach
this path (macOS 15.4+, macFUSE 5.x, a GUI-enabled FSKit module).
* fix(tests): pin the platform in the fskit manager tests
These three passed on macOS and failed on the Linux runner: FuseManager
routes through prepare_backend, whose check_platform rejects fskit off
darwin. Patch sys.platform explicitly so the assertion under test is the
mountpoint behavior, not the host, matching what test_backend.py already
does in both directions.
* ci: run the fskit backend for real on a macOS runner
Until now fskit shipped on unit tests plus one user report, because no
job could mount it. integ/fuse.py cannot cover it: its sizeless probe is
refused by the fskit size guard by design, and its two-mount scenario
needs something macOS forbids. So this adds integ/fskit.py, one RAM mount
under /Volumes that reads, writes and stats through the kernel.
Advisory like integ-fuse-windows, since whether a hosted runner enables
the FSKit module without a GUI toggle is what the job measures.
The truth file is JSON checked by value, not a txt file checked by
substring: the old harness passes a result of 166 against a truth of 16.
check_json.py is written to serve the remaining txt probes too. Only what
mirage controls is asserted; the mountpoint, the raw mount row and
whether a kext happens to be loaded are reported and left alone.
* ci(fskit): report the mount state before reading it
The first macOS run failed with a bare FileNotFoundError and told us
nothing: every diagnostic in the probe was printed after the read that
crashed. The volume entry appeared (readiness passed, the mount thread
stayed alive) but served no tree, and there was no evidence of why.
Print the mountpoint, its stat flags, the mount row and a listdir before
touching a file, and add an always-run step dumping the mount table,
/Volumes, the registered FSKit modules and the recent macfuse log.
* fix(fskit): a stray /Volumes directory is not a live mount
The macOS integ job reported exists=True isdir=True ismount=False with no
row in the mount table: macFUSE creates the /Volumes entry while mounting
and leaves the empty directory behind when the FSKit handoff fails. Bare
existence as the ready signal accepted that, so a mount that never came up
was reported live and failed with ENOENT on the first read.
Require os.path.ismount for every POSIX backend and keep the existence
shortcut for WinFsp only, where _prepare_mountpoint removes the directory
first so its reappearance really does mean the filesystem is live. The
backend argument is unused now, so it goes. _await_ready had no test at
all; it has one now.
Also register both macFUSE appexes on the runner: the installer registers
only the -local module there, while a developer Mac has both.
* ci(fskit): capture why the macFUSE mount never comes up
Both modules register and enable now, and the mount still times out with
no libfuse error, while the only module the system launches is msdos. So
the mount request is not reaching macFUSE at all. Capture the kext device
nodes, the system extension list and a macfuse-scoped log to tell an
environment limit from a mirage bug.
* docs(fskit): record where the /Volumes rule comes from
The guard read as an arbitrary house rule. It is measured: issue #82 ran
an fskit mount under /tmp, got 'mount_macfuse: the file system is not
available (1)', and the same mount worked once it moved to /Volumes.
* fskit: pin the real write surface, measured on a Mac
The mount works: /Volumes/mirage-*, tagged fskit, tree served, reads exact
(stat size == bytes read, so the size guard holds). ismount is true on a
live FSKit volume, which is what the readiness fix now depends on.
Writes are a different story. In-place writes and unlink work; create,
mkdir and rename return ENOSYS. A failed create still applies: the syscall
reports ENOSYS and the file exists in the resource and through the mount
anyway. Tracing shows mirage's create succeeds and returns a handle, then
the shim fails the syscall, so we cannot report this more accurately.
Pin the whole matrix in the probe and say plainly in the docs that
anything creating files will fail on an FSKit mount.
* examples: one fskit example per language
Five scattered files (ram, redis, s3, slack, gmail) collapse into
examples/python/fuse/fskit.py, which shows the size guard refusing, a live
mount with reads matching their stat size, and every write op with the
errno it returns. Verified end to end on macFUSE 5.3.3.
examples/typescript/fuse/fskit.ts cannot mount, since fuse-native bundles
a pre-macFUSE-5 dylib, so it shows the refusal and both alternatives:
backend fuse there, or the Python package for a kext-free mount. It
typechecks but is not run here, because its fuse leg is a live kext
mount.
* feat(fskit): TypeScript serves fskit after all
The 'fuse-native bundles a pre-macFUSE-5 dylib' claim was wrong: the
libosxfuse.2.dylib it ships is a stub whose install name is
/usr/local/lib/libfuse.2.dylib, and fuse.node links that absolute path,
so Node loads the same macFUSE 5.x libfuse Python does. Verified with a
live mount: /Volumes tagged fskit, cat/wc exact, and the same write
surface as Python (in-place ok, create/mkdir/rename ENOSYS).
Drop the unconditional throw from checkPlatform (macOS-only stays),
generalize appendDirectIO into appendMountOptions, and give mount() the
fskit branch: /Volumes named-not-created, backend=fskit + volname,
no direct_io, ownsMountpoint false so nothing ever rmdirs a /Volumes
entry. The example now mounts for real; docs flip from known-gap to
supported-with-caveats.
The caveat is earned: one of three example runs wedged on an append, and
a dead FSKit volume blocks mount-table enumeration system-wide until the
macFUSE appex is killed. Documented as read-mostly and experimental.
* docs: example READMEs and a concise fskit story
Both examples/ READMEs explain how to run, the naming convention, and why
fskit exists: Apple has deprecated third-party kexts (reduced-security
boot + approval on Apple Silicon already), FSKit is the supported
userspace replacement, and macFUSE 5.x serves the same libfuse API
through it. One mermaid flow shows the two paths differing only in the
kernel-to-userspace hop. The two fuse.mdx warnings shrink to their
load-bearing facts and the python page gains the same why + diagram.
* ci(fskit): report the known hosted-runner limit as a skip
Everything installs and enables headlessly, but the mount request never
reaches macFUSE's FSKit module on a hosted runner, so every run ends in
the same readiness TimeoutError and a permanently red advisory job is
noise. Treat exactly that signature as a skip and drop continue-on-error:
green now means environment-limited or verified, red means something new
broke (a guard regression, a crash, or the runner starts mounting and
diverges from integ/truth_fskit.json).
* feat(fskit): full write surface via macFUSE's Darwin-only callbacks
The read-mostly limitation was never FSKit's: the shim finalizes every
created item through setattr_x and routes rename through renamex, both
Darwin-only fuse_operations fields that mfusepy leaves as reserved NULL
slots. libfuse answered those requests itself with ENOSYS, after our
CREATE/MKDIR had already applied (wire trace: CREATE success, then
SETATTR -78), which also explains why the failures were dirty and why
rename never reached userspace.
fuse/darwin.py replaces the 13 reserved slots with macFUSE's real Apple
tail (same size, asserted) and marshals setattr_x, fsetattr_x and
renamex; MirageFS decomposes setattr_x (size routes to truncate, other
attributes follow the chmod/chown accept-if-exists semantics) and maps
renamex flags (EXCL honored, SWAP refused as ENOTSUP). Installed once
per process from _run_fuse, no-op off macOS, layout-guarded so an
mfusepy upgrade degrades to the old behavior instead of corrupting the
struct.
touch, mkdir, mv, rm and a new-file write roundtrip now all pass on a
real fskit mount; integ/truth_fskit.json pins the full matrix. Docs flip
Python fskit from read-mostly to full-write; TS keeps the old surface
because fuse-native's compiled op table cannot gain new C callbacks from
JS. Remaining upstream caveats stay referenced in code comments:
macfuse#1181 (exec until first read) and macfuse#1165 (root readdir
cache invalidation).
* test(fskit): skip the struct-extension test off macOS, settle yapf's import wrap
mfusepy builds fuse_operations per platform, so the Darwin reserved tail
the Apple fields replace does not exist in the Linux layout and
install_macfuse_extensions correctly bails there; monkeypatching
sys.platform cannot conjure the struct. yapf also disagreed with the
committed import wrapping; both yapf and isort accept the new form.
* feat(resource): generic remote_mount_spec so cloud backends are sandbox-mountable
Lift the S3-only remote_mount_spec to the base class: any backend that
opts in with remotely_mountable and holds a pydantic config serializes
{resource, config} (credentials unwrapped) so a remote mirage, e.g.
inside a sandbox, can reconstruct and mount it. RAM and disk keep the
None default. Enables s3, gdrive and slack; adds base coverage.
* feat(runtime): sandbox runtimes with FUSE-mounted workspaces
Add the RemoteSandbox base and the Daytona, e2b and Docker runtimes:
whole-line execution against a remote or local container, lazy
provisioning on the first line, reattach by id, and sandbox ownership so
teardown only touches what we created.
The workspace becomes visible by running mirage inside the sandbox and
FUSE-mounting each remotable mount live: the host serializes the mounts
into the public workspace config, writes .mirage-workspace.json, and runs
one command, `mirage workspace create`. The CLI auto-spawns the in-sandbox
daemon and mounts synchronously, so the exit code is the ready signal and
stderr carries the error. Reads and writes flow both ways with no sync;
writes reach the backend on file close. Needs an image with mirage baked
in (e.g. mirage-python-fuse).
* feat(sandbox): translate virtual mount paths onto provider mountpoints
The agent speaks virtual paths (/data/a.py); mirage is the control
plane that rewrites them onto each provider's physical mountpoint
(/home/daytona/workspace/data/a.py on Daytona, /workspace/data/a.py on
Docker). Before, only cwd was rebased, so absolute virtual paths broke.
Rewrite is longest-prefix-first and only touches tokens that start
exactly at a mount (/s3 or /s3/...); siblings (/s3.txt), system paths
(/usr/bin), and relative paths are left alone. Each mount is also
exported as a MIRAGE_<PREFIX> env var for paths built at runtime. A
bare / world mount is skipped (it would capture the sandbox's own /usr).
Verified end to end on a real Daytona sandbox with an absolute
/data/in.txt round trip.
* docs(examples): README for the Daytona FUSE runtime example
Documents the working flow: bake the mirage-fuse snapshot once, then
drive an S3-backed workspace whose python3 lines run in a Daytona
sandbox that FUSE-mounts the bucket live. Explains the control-plane
path translation (virtual /data/x rewritten to the sandbox mountpoint).
Fixes the workspace yaml: drops the removed mount: fuse/copy option
(fuse is now the only sandbox mode) and points the active runtime entry
at the mirage-fuse snapshot so the example actually mounts.
* fix(sandbox): typecheck fallback for env tuple; remove RAM runtime example
The env-var test's ?? fallback widened env to {}, failing tsc
(TS2339 on MIRAGE_DATA). Type the fallback like the sibling cwd test.
Remove sandbox_runtime.py: it mounted RAM, which is not
remotely_mountable, so under the fuse-only sandbox path its first
python3 line is rejected. The Daytona example (S3-backed) is the
supported demo.
* feat(docker): bake all backends into the sandbox image, not just s3
The sandbox image install was pinned to mirage-ai[s3,fuse], so a
workspace mounting any other backend failed in-sandbox for lack of its
deps. Install mirage-ai[all,fuse] via a MIRAGE_EXTRAS build arg
(default all) so one image mounts any backend; narrow the arg or extend
FROM the image for a lean build. Verified: [all,fuse] resolves under
pip and postgres/mongodb/gcs import in the built image (1.74 GB).
* feat(docker): sandbox image installs mountable backends only, no agent deps
Add a curated 'sandbox' extra (every mountable backend + fuse) and
default the Dockerfile to it, instead of 'all'. A sandbox is a
filesystem host: it never builds agents or launches other sandboxes,
so the agent frameworks (anthropic/openai/deepagents/openhands/agno/
claude-agent-sdk/pydantic-ai) and provider SDKs (daytona/e2b) that
'all' pulls do not belong. mem0 is excluded too: mem0ai is the lone
backend that hard-requires the openai client.
Verified in the built image: no openai/anthropic/daytona/e2b/mem0,
backends (s3/postgres/mongodb/chroma) and fuse still import. Image
1.74 GB (all) -> 947 MB. Narrow further with --build-arg MIRAGE_EXTRAS
or extend FROM the image.
* docs: document sandbox runtimes (docker/daytona/e2b)
Add a Sandbox page to the Runtimes section for both the Python and
TypeScript docs, wired into docs.json nav. Covers capture-based
routing, the live FUSE-mounted workspace, control-plane virtual-path
translation, the mirage-python-fuse image + MIRAGE_EXTRAS, per-provider
setup, reattach/lifecycle, and resource limits. The existing runtime
pages document the in-process interpreters (monty/wasi/pyodide/local);
this covers the remote whole-line runtimes.
* refactor(resource): unify remote flag, rename remotely_mountable -> remote
One boolean now answers 'does this backend live remotely', replacing
both the verbose remotely_mountable and qdrant's dead is_remote/isRemote
one-off. remote=True unlocks reconstructing the resource elsewhere (FUSE
mount inside a sandbox) via the generic remote_mount_spec. Set on s3,
gdrive, slack, qdrant.
* docs: add cache-invalidation example to invalidate_all_after_remote
Concrete cat -> sandbox-write -> cat example showing why a sandbox line
forces a full local cache reset.
* feat(sandbox): reconcile mounts imperatively, drop the workspace config file
mirage is the control plane: the host workspace is the desired state,
the sandbox is the actual state, and every captured line reconciles the
two through the provider's own exec API. A new or changed mount runs
'mirage mount add <prefix> --fuse <path>' inside the sandbox with the
spec in the exec environment (never a file, never argv), a dropped
mount runs 'mirage mount remove <prefix>', unchanged mounts cost
nothing. Mounts added or removed after the sandbox booted converge on
the next line.
New in-sandbox CLI 'mirage mount add/remove/list': each prefix becomes
its own single-mount daemon workspace (deterministic id), so mounts
attach and detach independently through the existing create/delete
endpoints. The uploaded .mirage-workspace.json and one-shot
mount_workspace are gone; TS mirrors the reconciler (syncMounts,
serialized per line).
Real docker e2e green: reconciled mount add, absolute-path read, FUSE
write-through to S3, host readback.
* test(qdrant): follow the is_remote -> remote rename
* ci(integ): survive chocolatey outages in the WinFsp install
Retry choco three times, fall back to the official WinFsp GitHub
release MSI, and verify winfsp-x64.dll landed so a bad install fails
at the install step instead of as 'Unable to find libfuse' mid-test.
The advisory integ-fuse-windows job went red on a chocolatey.org 503.
* refactor(sandbox): mount once, run lines verbatim, drop path magic
Remove the line translation, the MIRAGE_<prefix> env injection, and
the per-line reconcile state. The contract is now plain: the sandbox
mounts the workspace's backends once at boot (mirage mount add per
mount, spec in the exec env), mounts appear at
<workspace_root>/<prefix>, the session cwd is rebased, and the line
runs verbatim. Path consistency beyond the rebased cwd is the
caller's job; static rewriting could never be complete (quoted code,
runtime-built paths) and half-working magic is worse than none.
Docs and the daytona example teach the relative-path contract. Real
docker e2e green: relative read and write through the mounted bucket
with rebased cwd.
* refactor(sandbox): one general SandboxConfig, shared constants, provider packages
* refactor(sandbox): provider-owned configs, spec-derived mounts, remote sweep
* refactor(sandbox): connect-only runtimes, one in-sandbox workspace
* refactor(sandbox): drop the lazy provider re-exports
* refactor(sandbox): user-provisioned sandboxes, connect and exec only
* fix(sandbox): safeguard whole lines, per-invocation stdin paths, trim sandbox extra
* refactor(safeguard): one resolve_safeguard entry point, shared guard_output boundary
CLI exit codes / changes (push) Has been cancelled
Test (Install) / changes (push) Has been cancelled
Pre-commit / pre-commit (push) Has been cancelled
Integ / changes (push) Has been cancelled
Test (Python) / changes (push) Has been cancelled
Test (TypeScript) / changes (push) Has been cancelled
Integ / integ-facets (mem0) (push) Has been cancelled
Integ / integ-facets (project) (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Python) / runtime (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / python-install (push) Has been cancelled
Test (Install) / ts-install (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-shared-py (push) Has been cancelled
Integ / integ-shared-ts (push) Has been cancelled
Integ / integ-shared-parity (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Integ / integ-fuse-windows (push) Has been cancelled
Integ / runtime-py (push) Has been cancelled
Integ / runtime-ts (push) Has been cancelled
Integ / integ-observability (push) Has been cancelled
Integ / integ-facets (chat) (push) Has been cancelled
Integ / integ-facets (dify) (push) Has been cancelled
Integ / integ-facets (email) (push) Has been cancelled
* fix(shell): seed $? between commands in subshell and brace-group bodies
Inside ( ) and { }, last_exit_code was never updated between children,
so $? always reflected the pre-block value. Match the program-loop
behavior and seed after each child in both Python and TypeScript.
Fixes#476
* fix(shell): finalize lazy exit codes before seeding $? in subshell and brace-group bodies
* fix(shell): seed $? with a barrier in if/loop/case/function bodies too
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* fix(observe): keep the mount prefix on op records in typescript
Op events named the mount-relative path, so two mounts holding the same
filename produced identical records and the audit trail could not say
which backend was touched. Python already had the machinery for this;
typescript was missing three pieces of it.
- pushMountPrefix returns the previous prefix so callers restore it
instead of clearing to '', which a command dispatching into another
mount needs (mirrors push_mount_prefix).
- withMountPrefix re-pushes the prefix around each pull from a stream,
so a command that defers its backend read to the first chunk still
records the right path. Mount.execute wraps the stream plus
IOResult.reads/writes through it (mirrors with_mount_prefix and
_wrap_cmd_streams), with CachableAsyncIterator.wrapSource standing in
for replace_source.
- Dispatcher.dispatch and Mount.executeOp push the prefix around the
backend op, which is how eager writes get theirs (mirrors Ops._call).
The two s3 sites that passed the virtual path to work around this keep
doing so; record() leaves an already-prefixed path alone. Their comments
no longer describe a bug.
* test(observe): pin op record paths per mount in both languages
Two mounts each holding report.json, driven through an eager write, a
lazy read and a cp that crosses mounts. Both languages now emit the
same six records. The typescript test previously carried a comment
explaining why the path was not asserted; that is gone.
* fix(observe): scope the mount prefix per async branch
pushMountPrefix mutated the one RecordingState every branch shares, so
two mounts consumed concurrently (cat /s3/a & cat /db/b) could see each
other's prefix and their restores could overwrite each other, recording
paths like /db/a. Python avoids this by re-setting a frozen Recorder per
task and sharing only the sink.
runWithMountPrefix replaces it: it derives a state for the branch,
shares the records array, and hands it to the async context, so nothing
is mutated in place. The push/restore pairs in Mount.execute,
Mount.executeOp and Dispatcher.dispatch become plain scopes and their
try/finally blocks go away.
Both new context tests fail against the mutating version.
* fix(observe): require a path boundary before treating a record as prefixed
record() decides whether a backend already named the virtual path by
testing startswith(prefix), which also matches a filename that merely
shares the prefix's leading text. A mount at /s3 holding s3-report.txt
recorded /s3-report.txt, dropping the mount. Compare against prefix + "/"
(or the mount root itself) in both languages.
* integ: move history and find arg-errors to the JSON harness, add curl/wget
The JSON harness has grown to cover almost everything the old script-plus-
truth-file suites did, so this retires the ones that are now redundant or
cheap to migrate, and closes the last command with no integ coverage.
Deleted as dead code:
- cases.py / cases.ts (~1.7k lines each). Only three meta helpers were still
imported, by metadata.py / metadata.ts.
- metadata.py / metadata.ts / truth_metadata.txt. Every case they ran already
exists in integ/unix/meta and integ/unix/meta_overlay under the same ids,
on 22 backends instead of 4.
- s3_probe.ts and runners/tools/migrate_legacy.py, referenced by nothing.
- Seven integ/package.json scripts pointing at files that do not exist.
Migrated to the harness:
- history.py + truth_history.txt become integ/bash/history/*.json under a new
ram-history target. It needs its own target because the shared battery's
~1900 commands on ram would otherwise land in the recording and shift the
history line numbers. The observer-introspection tail is dropped: those are
Python API assertions the shell harness cannot express, and
python/tests/observe covers all five. TypeScript gains history coverage it
never had.
- find_arg_errors.py / .ts + truth_find_arg_errors.txt become four
{mount}-bound cases across sixteen dummy-credential argerr-* targets, wired
through one new arg_error builder per language. This is the first integ
coverage for discord and github_ci.
New coverage:
- curl and wget were the only spec commands with zero integ cases. Their unit
tests mock the HTTP layer, so the untested part was the write landing in the
VFS. integ/server/http_server.py is the fixture; the adapter owns its
lifecycle and exports the endpoint as the {http} token, so --facet http
needs no CI setup.
Four curl/wget cases are deliberately absent because python and typescript
disagree: the 404 path (python leaks httpx's message including an MDN link and
exits 1, typescript exits 22), the unreachable-host path (python reports a
platform-specific errno), and the read-only-mount message. Those are backend
bugs to fix separately, not test scaffolding.
Root truth files drop from seven to four: safeguard, runtime, runtime_py and
fuse remain, each blocked on a harness feature.
* integ: restore the metadata snapshot/GC scenarios, pin the observer event shape
Codex review on #649 caught two places where the previous commit removed
coverage rather than moving it.
metadata.py / metadata.ts held three scenarios beyond the case lists that were
migrated to integ/unix/meta and integ/unix/meta_overlay, and truth_metadata.txt
asserted all three: overlay attributes surviving a snapshot reloaded onto a
fresh S3 resource, ConsistencyPolicy.ALWAYS garbage-collecting an attribute
overlay orphaned by an out-of-band unlink, and the same snapshot roundtrip on
RAM. Verifying that the case ids matched was not enough, main() did more than
call those three functions. Both scripts come back trimmed to just those
scenarios, with the meta_stat_line helper inlined now that cases.py is gone.
Both reproduce the original truth output byte for byte. Retiring them for real
needs snapshot and namespace support in the harness, which the file now says.
The history observer assertions are a narrower case. recorder-not-mounted and
the post-clear session projection are genuinely covered by
python/tests/observe. The event shape is not: exit_code, cwd and the op source
were only ever asserted on entries the tests build by hand, never on the real
execute path, so a regression in the wiring from a command to its event would
pass while the rendered views stayed correct. Three tests per language now pin
that on ws.execute(), which also gives TypeScript coverage it did not have.
Doing so surfaced a divergence: TypeScript records the op path mount-relative
('/test.txt') where Python records the virtual path ('/data/test.txt'), so the
prefix Mount.execute sets is not reaching record(). Python's test pins the
correct form; the TypeScript test asserts a read op exists without freezing the
wrong path in place, and says why.
* curl,wget: align exit codes and streams with the real tools (py+ts)
Pinned against curl 8.14.1 and GNU Wget 1.25.0 in debian:stable-slim before
changing anything, per the repo rule for command semantics. The probe found
both languages wrong, in different ways, on almost every failure path.
The root cause was one design mistake shared by both: the transport layer
called raise_for_status() (python) or threw on !resp.ok (typescript), so an
HTTP error status became an exception before either command could see it. That
is wrong for curl, which treats a 404 as a successful transfer, and it meant
the HTTP library's own text reached stderr, including a documentation URL from
httpx and a bare "fetch failed" from undici. The layer now reports status,
reason and body, and each command owns its own CLI semantics.
curl, matching curl 8.14.1:
- an HTTP error status is no longer a failure. The body goes to stdout (or to
-o) and the exit code stays 0.
- new -f/--fail, the only way to make a status an error: exit 22, and nothing
is written.
- a refused connection is exit 7 with a host/port message, not a
platform-specific errno (61 on macOS, 111 on Linux, so the old message could
not even be asserted in CI).
- a failed -o write is exit 23.
- no URL is exit 2 with curl's two-line usage text.
- -o prints nothing on stdout; it used to print "saved to <path>".
- -s now only suppresses the message and -S restores it; neither changes the
exit code.
- -L is honoured. Python passed follow_redirects=True unconditionally, so -L
was a no-op and redirects were always followed.
wget, matching GNU Wget 1.25.0:
- any 4xx/5xx is exit 8 with "ERROR <code>: <reason>.", and the -O target is
still created empty, the way wget truncates it before reading the status.
- a refused connection is exit 4.
- --spider reports on stderr ("Remote file exists." / "Remote file does not
exist -- broken link!!!") rather than stdout, and inherits the same exit 8.
- the progress report moved to stderr; stdout is empty.
- no URL is exit 1 with the usage block.
Two more bugs found on the way: both commands caught only three exception
types, so a missing parent directory escaped uncaught, now fixed by using the
shared WALK_ERRORS set; and python rendered OSError with str(), leaking
"[Errno 13]" and a python repr onto stderr, now using strerror.
Coverage: 35 integ cases under --facet http, byte-identical across both hosts,
plus 31 new python unit tests (test_curl.py, test_wget.py) and 10 for the
transport layer, and the typescript tests updated to the pinned behavior.
Known remaining divergence, deliberately not asserted: a read-only -o target
produces different wording per host because python rejects the write at the
mount layer and typescript at the dispatcher layer. Those cases pin the exit
code with -s/-q so neither host's wording is frozen as correct.
* fix(ci): yapf formatting, a long line, and the tsc-only stderr type
Three CI failures, all mine from the previous commit:
- yapf reformatted the two new test files (multi-line signatures) after the
commit that added them, so the committed tree was not formatter-stable. The
local pre-commit run that reported success had applied those edits without me
re-staging them.
- flake8 E501 on a 82-char assertion in test_curl.py.
- tsc --noEmit rejected DEC.decode(ioResult.stderr ?? ...) because stderr is
ByteSource, which may be an AsyncIterable. vitest does not typecheck, so the
test run was green while the typecheck job was not. Both harnesses now use
IOResult.stderrStr(), which materializes the stream the way the class already
does internally.
The typescript gate failure was only downstream of the typecheck job.
* fix(ci): regenerate the committed specs for curl -f
Adding -f/--fail to the curl spec makes the checked-in spec/ snapshots stale.
The "Spec drift" check is a separate step in the pre-commit workflow rather than
a pre-commit hook, so a local `pre-commit run --all-files` never covers it:
python scripts/gen_specs.py
node --experimental-strip-types typescript/scripts/gen-specs.ts
git diff --exit-code spec/
Regenerated all three snapshots (python, typescript node, typescript browser);
the generators are idempotent on the result.
* ci: one database job driving both hosts against one service fleet
integ-database and integ-database-ts started an identical fleet (postgres,
chroma and qdrant as services, plus a hand-initiated mongodb replica set) and
then ran the same four targets, differing only in which runner they invoked. So
every database PR paid for four container startups, two chroma health waits, two
qdrant health waits and two replica-set initiations twice over.
They are now one job. This is the shape integ-observability and integ-facets
already use: start the fleet once, run the python battery, then the typescript
one. Database was the only family still split by language.
Two details worth keeping:
- the typescript step carries `if: ${{ !cancelled() }}` so a python-only
regression still reports whether typescript has it too. Splitting by job used
to give that for free.
- the per-backend env (CHROMA_HOST, QDRANT_PORT, ...) moved to the job level so
one step per language can cover all four targets, matching how
integ-shared-py runs 34 targets in a single step.
Trades a little wall clock (the two batteries serialize where they used to run
in parallel) for roughly half the runner minutes and half the containers.
79 fewer lines of workflow.
* fix(jq,gws): evaluate multi-document input per value, paginate gws GETs
jq computed output arity with a naive "[]" substring test while jq_eval used a
depth-and-string-aware predicate, so any program with a nested [] inside a
collector had its single array exploded one element per line. Reuse the careful
predicate in both places.
jq also read .json files with strict orjson, so a file holding several JSON
values failed, and the stdin path collapsed a multi-document stream into one
list. Real jq reads a stream of values from any input and evaluates the program
per document. Both paths now do that; -s still slurps.
gws list methods made exactly one HTTP call and dropped nextPageToken, so
'gws drive files list' silently returned the first 100 of N at exit 0.
Pagination is now the default (a truncated listing is indistinguishable from a
complete one) with --page-limit to opt out. Single-response GETs keep their
exact bytes; only real multi-page streams are newline-delimited NDJSON.
Adds 'gws --help' and 'gws <service> --help' matching googleworkspace/cli, and
teaches the integ mock server to paginate. Two existing goldens encoded the jq
bug and are corrected.
* fix(gws): port pagination and help to TypeScript, regenerate specs
The epilog field added to CommandSpec was missing from the committed spec
dumps, which is what the pre-commit spec-drift step caught. Regenerated
both trees and taught the TypeScript side about the field, so the two
dumps keep the same shape.
The PR description claimed TypeScript had no gws passthroughs. It has all
25, so the unpaginated-GET bug was live there too. Ported pagination,
--page-limit, the per-method --help descriptions, and the
gws --help / gws <service> --help surface. render_services() and
render_service_methods() output is byte-identical between the two.
Two defects surfaced during the port:
- withHelpSupport rebuilds the spec from a hand-listed init object to
inject --help, so it dropped epilog on every command. Python is immune
because it uses dataclasses.replace. Typecheck and unit tests both
passed; only the integ case caught it.
- gws --help was registered for gdrive and gsheets only, so a gdocs-,
gslides- or gmail-only mount had the passthroughs but no help. Now
registered for all five in both languages.
Structural alignment between the two implementations:
- drop the orphaned GWS_API_SPEC, replaced by gws_method_spec
- add _parse_page_limit mirroring parsePageLimit, so a bad --page-limit
reports the same message instead of leaking int()'s ValueError text
- re-export the help commands from the gws package so backends stop
reaching into gws.help, matching gws/index.ts
- make the four help description constants module-private on both sides
- drop the now-dead non-GET guard before invalidate_mount_listing
- give each page its own params dict instead of mutating one across calls
gws integ coverage is no longer blocked: g_help_lists_services,
g_help_lists_service_methods, g_files_list_paginates_by_default and
g_files_list_page_limit_truncates run on both hosts.
Docs: a Pagination section on the Drive page covering the default-on
behavior, the NDJSON output, --page-limit and the deliberate divergence
from googleworkspace/cli; the same note on the Gmail passthrough section;
and a JSON with jq section in the bash docs for the stream-of-values
semantics, verified against the jq binary.
* fix(spec): trim the help epilog without a polynomial regex
CodeQL flagged js/polynomial-redos (high) on the `/\n+$/` I used to strip
the epilog's trailing newlines: it backtracks on a long run of '\n'. The
trim now walks backwards, which is what Python's rstrip('\n') already did,
so that side was never affected. Verified identical output on 'Services:',
a single trailing newline, three trailing newlines, and an all-newline
epilog; a 100k-newline epilog now renders instantly.
Also align --page-limit validation across the two. Python used bare
isdigit(), which accepts non-ASCII digits that TypeScript's /^\d+$/
rejects: '١٢' parsed as 12 on one side and errored on the other, and '²'
passed the check then crashed int(). Python now requires isascii() too,
and both suites reject the same five inputs.
* fix(gws): register help commands per resource, drop the nested handler
Both findings from the codex review on #650.
P2, resource filtering. MountEntry.register keys commands by
(name, filetype) and ignores RegisteredCommand.resource, so registering
the whole GWS_SERVICE_HELP_COMMANDS list on every backend made a
gdocs-only mount answer `gws gmail --help` and `gws drive --help` with a
method listing it cannot execute. Reproduced on a gdocs-only workspace
before the fix. gws_help_commands(resource) / gwsHelpCommands(resource)
now build one registration per reachable service, bound to the single
resource asked for, which is what the TypeScript backends were doing with
an explicit filter. A drive mount still reaches docs, sheets and slides.
P1, nested function. make_service_help_command defined its handler inline
and closed over the body, against the repo's no-nested-functions rule.
The handler is now module-scope run_help, bound to its listing with
functools.partial, the same way run_gws_method is bound to its method.
Both languages end up with the same shape, so the per-backend filter
duplication in TypeScript is gone too. ROOT_DESCRIPTION is module-private
on both sides.
* refactor(filetype): remove the bundled format renderers, keep the extension point
mirage shipped renderers for parquet, ORC, feather/arrow/ipc and hdf5/h5, plus
a PDF module that turned out to be entirely dead code: '.pdf' was never in the
factory registry and nothing imported mirage.core.filetype.pdf, so its test
exercised it directly and kept it looking alive.
Both core/filetype/ trees are removed. The dispatch machinery stays, in both
languages and in both places it lives (commands/builtin/filetype_factory and
ops/generic/factory), with empty registries and a comment marking them as the
extension point. A file with an unregistered extension now reads as raw bytes.
Registering a filetype-scoped command still works and is covered:
tests/commands/custom/test_filetype_fns.py registers a '.parquet' handler with
a fake function and asserts dispatch, and test_unregister_removes_all_filetypes
now registers a '.demo' renderer itself rather than leaning on the bundled ones.
Also removed: the parquet/hdf5/pdf extras (and their references from 'all' and
'deepagents'), the hyparquet, hyparquet-writer, apache-arrow and h5wasm
dependencies, integ/resources/columnar, the columnar FileType enum members, the
per-backend filetypeRead declarations that existed only on the TypeScript side,
and the sentence advertising 'cat on .parquet/.orc/.feather returns a formatted
table' from 33 backend and agent prompts, which would otherwise have been
lying to agents.
grep_helper's skip-list and file's MIME map keep their columnar entries: those
are still correct, since the formats remain binary whether or not mirage can
render them.
core/src/ops/generic/factory.ts was reaching NodeJS.ErrnoException through the
columnar packages' type dependencies. core has to work in both runtimes, so it
now uses a structural { code?: string } instead.
Python suite passes, TypeScript core 5458 pass, pre-commit clean.
* refactor(examples): drop the columnar demos alongside the renderers
Deleted, because their whole subject was the removed rendering:
s3_data.py (181 lines built around four columnar constants),
ram_filetypes.ts, box_parquet.ts, dropbox_parquet.ts, ram_parquet.ts.
Also deleted fuse_hooks.py, which was already broken before this change: it
imports mirage.fuse.filetype.data.local.parquet, a module that does not exist
anywhere in the tree.
Edited rather than deleted, since columnar was one section of a broader demo:
gdrive_complex.py, disk.ts, ram_fuse.py, and the openhands README (which
advertised format-aware reads and piped a .parquet through jq).
Everything still referencing parquet only names it in find/ls patterns, which
keeps working: mirage can still list and locate these files, it just no longer
renders them.
* fix(ci): drop the last filetype wiring the removal missed
The nextcloud ops table still declared filetypeRead for feather/hdf5/
parquet, so makeGenericOps threw at module load and every job that
imported @struktoai/mirage-node died before running anything.
Also: regenerate the specs (the crash hid 89 files of drift), drop the
removed pdf/parquet/hdf5 extras from the install matrix, take the
columnar members out of the TS FileType enum and the box/gdrive/dropbox
type guesses so both languages agree again, and stop passing an inert
filetype_read=True from ten Python backends.
* fix(spec): regenerate with every optional backend importable
The previous regeneration ran under the worktree venv, which has no
optional extras installed, so backends that failed to import were
dropped from each spec's resources list. Regenerated with the full
environment: the only remaining change is the emptied filetypes list.
* fix(agents): stop telling models mirage renders columnar formats
The langchain backend mapped parquet/h5/hdf5/feather to text/plain, so
with no renderer registered it handed the model raw binary decoded as
UTF-8 instead of the binary redirect. Those extensions now fall through
to application/octet-stream.
The system prompt, the execute tool description in both languages, and
the README extensibility example all still advertised columnar rendering
as a shipped feature; they now describe it as the extension point it is.
Also drops commands/optional.py, which existed only to soft-import the
removed format helpers and has no callers left.
* refactor(filetype): drop the filetype command factory, keep mount registration
The factory built nine commands per registered extension, but with no
renderers shipping it produced nothing on every one of the ~19 backends
that called it, and its handlers had no test in either language while
the module contract they expected was written down nowhere.
Removes commands/builtin/filetype_factory/ in both languages along with
the filetype_read / filetypeRead op knobs and the reads and provisions
that existed only to feed them. Registration on a mount survives and is
the whole extension point: a command or op carrying a filetype resolves
as (name, filetype) before (name, resource).
examples/{python,typescript}/filetype/ register a .tally renderer end to
end and are gated in CI against integ/truth/*/filetype.txt, so the path
is now exercised rather than asserted. Both emit byte-identical output.
* fix(mkdir): GNU conformance for mkdir and mkdir -p (py+ts)
mkdir was the one write op #641 left short. Four divergences, pinned
against GNU coreutils in docker before changing any semantics.
- mkdir -p across a plain file turned that file into a directory on the
store backends. The -p loop added every component blind, so the new
directory key shadowed the file key: reading it started reporting
EISDIR while the bytes stayed orphaned in the store. Now "cannot
create directory 'a.txt': Not a directory", and the file is untouched.
- mkdir -p onto a plain file target reported nothing at all. GNU calls
that EEXIST, not ENOTDIR: the walk reaches the target itself rather
than crossing it.
- mkdir without -p silently succeeded on an existing path, on every
backend. GNU refuses with "File exists"; only -p is idempotent. Every
internal caller that mirrors a tree through the op (cp -r) already
probes with is_directory first, so none of them relied on the no-op.
- mkdir -p quoted the whole operand where GNU quotes the component it
tripped on ("cannot create directory 'a.txt'", not 'a.txt/sub').
Shared pieces: check_mkdir_target joins check_dest_parents in each
store backend's dest module, so the two destination questions a write
can ask live side by side. The real-filesystem backends keep the
kernel's errnos and only walk the chain to attribute a failure it
already reported, which is what disk/dest and opfs's local
checkMkdirTarget do. error_path/errorVirtualPath is factored out of
format_fs_error so the command layer can quote whatever path an error
names rather than assuming the operand, and mounted_path moves from
generic/cp to utils/key_prefix now that backends address ancestors too.
Deliberate divergence: operand spelling stays normalized, so a relative
argument is reported absolute (as #641 already established).
* test(crossmount): mkdir -p the redis parent chain in the matrix fixture
The fixture built the chain with repeated plain mkdir calls, which only
worked while an existing directory was a silent no-op. Use -p, like the
disk branch beside it already does.
* fix(mkdir): address Codex review on #647
- kindAt swallowed every OPFS resolution failure, so a SecurityError or
InvalidStateError read as an absent component and mkdir carried on.
Only NotFoundError/TypeMismatchError mean "nothing here"; the rest
re-raise.
- disk's component walk used isfile, which misses a FIFO, socket or
device ancestor: makedirs still refused with ENOTDIR but the walk
found nothing to blame and the whole operand was named. It now stats
once and treats any non-directory as the failing component, matching
the TypeScript side.
- mkdir reported an absolute path for a relative operand, where GNU and
mirage's own read-family commands quote it as typed. operand_spelling
rebases whatever path an error names (the operand, an ancestor of it,
or something under it) onto raw_path, so `cd /data && mkdir -p
f.txt/sub` now says 'f.txt' like GNU.
drop_trailing_segments joins rebase_one in utils/path as the ancestor
half of the same idea, with unit tests in both languages, and two integ
cases cover the relative spellings.
* feat(db): semantic virtualization + JSON-harness integ for all DB backends
Postgres/Mongo DB virtualization work plus a full integ-harness migration.
Core:
- Postgres semantic.json (dimensions/time_dimensions/facts/relationships),
byte-identical py+ts.
- Fix SQL identifier injection via quote_ident/qualified (py; ts already safe).
- Fix grep/rg push-down: honor a literal+no-shaping-flags gate so -v/-c/-l/-n
fall through to the generic scan; rg now mirrors grep per backend in py+ts.
- Fix a hidden py/ts divergence: whole-valued double renders as `5` (Postgres
canonical + node driver) not `5.0`; shared canonicalize_row in
utils/json_canonical, applied to postgres rows and mongodb docs/schema.
- Mongodb schema type inference classifies whole-valued doubles as int
(matches the JS driver) and samples deterministically (sorted _id).
Integ:
- Migrate postgres, mongodb, chroma, qdrant, lancedb, notion off the bespoke
.txt truth-file harness onto the shared JSON case harness (targets.json +
resources/<backend>/*.json), byte-identical py==ts; delete the bespoke
*.py/*.ts scripts and truth_*.txt; CI runs `main.py/ts --target <backend>`.
- notion's TS-only MCP/REST transport parity kept as a standalone
self-asserting notion_mcp_parity.ts (no golden file).
* fix(search): faithful grep/rg push-down (Codex review)
- Postgres push-down was case-INsensitive (ILIKE) and left % / _ unescaped,
so `grep Ada` matched `ada` and `rg -F user_id` matched `userXid`. Now
case-sensitive LIKE by default, ILIKE only under -i, and LIKE wildcards are
escaped. Threads case sensitivity through the row + metadata search chain
(py + ts); metadata scan is case-sensitive unless -i.
- search_pushdown_ok rejects newline-joined patterns (-F with multiple -e are
independent alternatives LIKE cannot express) so they take the generic path.
- has_search_shaping_flags now recognizes rg's -I (no filename) and the
file-filtering --glob/--type, forcing those onto the generic scan.
* feat(du): one backend contract and GNU-aligned output
du had two backend shapes (du_total/du_all plus a flat du_multi list) and
printed only the operand line. Both are replaced.
Backends now expose one pair, core/<backend>/du/{size,entries}, wired as
du_size / du_entries. entries returns (entries, total) with leaf files
only, mount-relative, no summary row. The generic lifts them onto virtual
paths and re-spells them as the operand was typed, which fixes du -a
rendering identical lines for two mounts holding the same filename.
Output now follows GNU:
- a line per directory with its recursive total, post-order, derived from
the leaf list rather than a second walk
- --max-depth prunes directory levels, so it finally does something
- -d N as a spelling of --max-depth, with C strtoul base 0 parsing
- -s with -a, -s with --max-depth, and a bad depth are usage errors (exit 1)
- an unreadable operand is named, the rest still print, exit 1
- no operand measures the working directory instead of erroring
A failed stat is not treated as proof of absence: backends that never
materialise a mount root entry (redis) still report their subtree.
Fallback walks are bounded by CommandIO.max_du_entries; when the cap trips
du prints what it accounted for, warns on stderr and exits 1. Slack sets a
low cap because it exposes a directory per conversation per day.
Verified against debian:stable-slim: paths, exit codes and stderr match GNU
on 23 cases, and python and typescript are byte-identical on all of them.
Deliberate divergences (sibling order, empty directories, byte sizes) are
documented in CLAUDE.md.
* refactor(du): drop the redundant typescript du wrappers
s3, gridfs and hf each shipped a bespoke du command that did what the
generic_bind builder already does. s3 and gridfs even wired duSize and
duEntries on their CommandIO and then overrode du anyway; hf only lacked
the wiring. Python deleted its equivalents when the contract landed, so
this is the typescript half of that cleanup.
Removing them is provision-neutral: default_provision / defaultProvision
map du into the metadata family, which returns the same metadata_provision
the wrappers named explicitly.
opfs keeps its wrapper because opfs registers every command by hand and has
no CommandIO at all, and github keeps one in both languages because it
answers du from the index.
* refactor(du): one entry point so backend wrappers cannot drift
The three remaining du commands (github in both languages, opfs) each
repeated the same preamble: parse the flags, split the operands, render.
Wiring rather than semantics, but three copies of an ordering that has to
stay identical, including the GNU rule that flag validation happens before
any I/O.
run_du / runDu now owns those three steps, so a wrapper supplies only its
backend callables. du_operands and du_has_content lose their exports and
become internal to it.
* fix(du): refresh legacy truth files and decode sftp filenames
The four .txt-truth integ backends (chroma, lancedb, notion, qdrant)
predate directory rows, so they diffed against the new du output. All
four regenerated from live runs and verified byte-identical in Python
and TypeScript.
asyncssh types SFTPName.filename as bytes | str, so the annotated walk
tripped mypy's str-bytes-safe check. Decode it the way find.py does.
* fix(du): restore du on nextcloud, onedrive and sharepoint
Deleting the three bespoke du wrappers left "du" in their generic
command override sets, so the factory skipped it and nothing supplied
it. du was unregistered on all three backends.
The overrides existed because those backends returned a flat list where
the generic wanted a (list, total) tuple. The shared du_entries contract
settles that, so the sets are now empty and the factory registers du
with their native ops.
Regenerate the checked-in specs, which is what caught this. du also
loses hf_datasets, hf_models and hf_spaces: the deleted wrapper
registered for all four HF resource names, but no other command does,
so du now matches the rest.
* fix(du): directory markers and failed content probes
Three failures the shared battery caught on backends with no local
harness.
Keyed backends (S3, GridFS) store a zero-byte marker object for a
directory. It reaches the rollup as a leaf, and under -a it overwrote
that directory's computed total, so 'du -a' printed 0 for a directory
whose file it had just listed at 5. Directory sums now win on a clash.
The content probe that tells an implicit directory from an absent path
runs only after stat has already failed, and it calls the backend
again. On gdrive, onedrive, sharepoint and ssh a genuinely missing
operand made that second call raise a driver error, which escaped and
replaced GNU's "cannot access" line. A failing probe is a negative
probe.
The slack walk-budget case expected exit 0. GNU exits 1 on a partial
traversal (pinned against debian:stable-slim: unreadable subdirectory
prints the partial total, warns on stderr, exits 1), so the budget
warning is the same class of answer.
* fix(du): codex review, cwd mount prefix and -s --max-depth=0
The implicit cwd operand for bare 'du' built its backend key by
stripping slashes off the virtual path, so from a non-root mount
(cwd=/ram) the backend was asked for a 'ram' entry inside itself
instead of its own root. TypeScript reported the cwd missing and
exited 1 where Python printed the totals. The key now comes from
opts.mountPrefix via mountKey, matching checksum.ts.
GNU treats -s and --max-depth=0 as the same request: it warns
'summarizing is the same as using --max-depth=0', prints the total and
exits 0, and only a nonzero depth is a real conflict (pinned against
debian:stable-slim). Both were rejected as usage errors. DuFlags now
carries the non-fatal warning through to stderr.
* fix(cp,mv): destination-parent conformance with GNU (py+ts)
cp and mv never create the destination's parent, and the store-backed
backends never grow a key under a directory they did not record. Pinned
against GNU coreutils in docker before changing any semantics.
Six divergences, all reproduced on both ram and disk (they were generic
command bugs, not store bugs):
- cp into a missing parent exited 0 and created the parent plus an
orphan. disk/copy did makedirs(parent) and ram/redis copy wrote the
key blind. Now "cannot create regular file X: No such file or
directory", exit 1.
- cp under a plain file wrote an unreachable key (ram) or reported
"File exists" against the real host path (disk). Now "cannot stat X:
Not a directory", at any depth.
- mv had the same orphan gap in rename: the phantom directory made both
itself and its real parent unlistable.
- cp/mv with several sources and a missing target said "Not a
directory". GNU says "No such file or directory" and keeps "Not a
directory" for a target that exists but is not one.
- cp -rv printed no directory lines. GNU reports directories too,
including the source root.
- cp -r into a bad parent reported one error per subdirectory and
copied the tree anyway. Now one error and nothing copied.
The disk backends also leaked the real host path into user-facing
stderr (mkdir, touch, redirect, stat, copy). disk_errors/diskError
restamp the errno against PathSpec.virtual, so only the mount path is
ever reported.
Shared pieces: ancestors() in utils/path, a per-backend dest module
used by both rename and copy, and dest_parent_error in the generic
layer so every backend gets GNU's wording through the existing stat
seam. entry_kind now absorbs ENOTDIR as "does not exist"; isMissingPath
stays ENOENT-only so read-family commands keep reporting "Not a
directory" verbatim.
Prefix stores (s3, gridfs, databricks) have implicit directories and
are deliberately exempt. Accepted divergences: operand spelling is
normalized, cp -rv sibling order is sorted rather than readdir, and
cp -r dir dir refuses without leaving GNU's partial copy behind.
* fix(write): destination-parent conformance for the whole write family (py+ts)
A write is not `mkdir -p`. GNU reports ENOENT on a missing parent
component and ENOTDIR when one is a plain file, at any depth; mirage
either created the chain silently or grew an orphan key under a
directory it never recorded. That orphan makes both the phantom
directory and its real parent unlistable.
Every op that places a key at a caller-supplied path now shares one
destination probe per backend (rename, copy, create, write, append,
mkdir). Previously only rename and copy had it.
Backends:
- ram, redis: create had no check at all, write/append checked only the
immediate parent and always reported ENOENT
- disk: write, create and append silently ran `mkdir -p`; append also
leaked the host path into stderr
- opfs: resolveFileHandle passed `create: true` for every path segment,
so write, create, append, copy and rename all built the chain on
demand. Contrary to the note in CLAUDE.md, opfs does not inherit this
from a real filesystem: the OPFS API creates per segment.
Commands:
- touch and mkdir report the GNU line (`cannot touch 'X'`,
`cannot create directory 'X'`) and keep going past a failed operand,
exiting 1, instead of aborting the command
- tee and curl -o render the shared strerror rather than the backend's
own exception text, while keeping the raw message for refusals whose
wording is load-bearing (read-only mount, unsupported op) or that
carry the only description of the cause (transport errors)
- opfs exists returns false for a plain-file component instead of
leaking a DOMException
The redis probe tests one component at a time (SISMEMBER) rather than
pulling the whole directory set: it now runs on every write, so a
membership test per component beats transferring every directory in the
mount.
Fixtures that encoded the old behaviour are updated, not worked around:
the ram readdir orphan test and integ/runtime.{py,ts} built their
orphans by calling rename, which now refuses. readdir stays defensive
about orphans because a restored snapshot or another client can still
seed one.
Pinned against real GNU in docker (debian:stable-slim, bash not dash;
the redirect diagnostics and exit codes differ between the two).
Deliberate divergences kept: mirage names a redirect target without a
`bash: line N:` prefix (#635), and quoting follows GNU's POSIX locale
(`'X'`, not the UTF-8 locale's fancy quotes).
* test(redis): assert errno, not the old parent-does-not-exist wording
These two are gated on REDIS_URL, so a local run without redis skips them
and reports green. Run the node suite with REDIS_URL set to match CI.
* fix(disk): create the mount root at construction, not via the first write
DiskResource pointed at a path that does not exist used to work only
because write_bytes ran `mkdir -p`, which created the mount root as a
side effect. Removing that auto-mkdir broke it.
The mount root is infrastructure, not a path component a caller asked
for, so create it up front and keep writes reporting ENOENT for a
missing parent. This is what TypeScript already does in
DiskResource.open(); Python has no open() hook, so the constructor is
the equivalent seam.
* fix(cp,mv): keep ENOTDIR on source operands, blame the right copy operand
Two review findings.
A source that traverses a plain file is `cannot stat 'X': Not a
directory` in GNU, but both loops hard-coded "No such file or
directory". The backends cannot supply the distinction: stat answers
ENOENT for a path under a plain file just as it does for an absent one,
so only readdir splits the two. source_kind therefore walks the parent
chain the way dest_parent_error walks a destination's, and only on the
failure path. entry_kind keeps absorbing ENOTDIR, which is correct for
destinations, where the wording is re-derived.
copyFile and shutil.copy2 both answer ENOENT for a missing source and
for a missing destination parent, so restamping unconditionally against
the source named the wrong operand. Probe the source on failure to tell
them apart. Python was not restamping at all here, so it also leaked the
host path; it now shares a disk_error() helper with disk_errors,
mirroring the TypeScript diskError.
* style: let yapf own the signature wrap in the new disk copy test
* feat(integ): jaeger backend, real-server observability integ, cross-backend contract
Add a jaeger backend and put both observability backends under a real server
in integ, then extend the shared read-only contract across nine backends.
Jaeger backend (python + typescript):
- service-scoped tree: /services/<name>/{operations.json,traces/<id>.json}.
Jaeger's search API requires a service, so there is no listable /traces.
- always sends an explicit microsecond window: `lookback` is ignored by the
query API, so without start/end the search silently returns nothing.
- an unknown service answers 200 with an empty list, so existence is checked
against the service list rather than inferred from an empty listing.
Langfuse fixes found by running against a real self-hosted instance:
- prompt versions were unlistable: the list endpoint returns a `versions`
array, and reading a scalar `version` collapsed each prompt to one 0.json.
- dataset runs rendered as an indented document under a .jsonl name.
- a 404 leaked the raw SDK error instead of ENOENT; a 500 still propagates.
- stat accepted any plausible path without checking it exists.
- typescript requested dataset runs under /v2/, a hard 404 on a real server.
- typescript applied a hidden 7-day window that hid traces cat could serve.
- an unrecognized path resolved to scope "root", so the grep/rg push-down
answered a missing file with every trace in the mount and exit 0.
GNU alignment:
- du reported a missing operand as size 0 with exit 0; now reports it and
exits 1 while keeping output for the operands that exist.
- tree wrote a malformed line to stderr where GNU writes nothing.
ENOENT alignment across backends:
- trello, linear, slack and email readdir returned [] for an unrecognized
path, so ls and tree reported a bogus path as real but empty.
- email selected an unvalidated IMAP folder, leaking "command SEARCH illegal
in state AUTH" to the caller.
- email and slack hand-rolled an ENOENT whose message carried an "ENOENT: "
prefix, and slack's dropped the mount prefix.
- trello typescript readBytes lacked the virtual path python passes, so the
message reported the mount-relative path.
Integ:
- resources/observability/ holds langfuse (135 cases), jaeger (99) and the
36-case shared contract, which runs on nine read-only backends via a new
{mount} token so one case can cover backends with different mount paths.
- targets gain a facet, and the runners gain --facet, so CI runs one backend
family per job: observability, project, email, chat, dify, mem0, core.
- langfuse runs against a six-container compose stack, jaeger against a
single container seeded over OTLP.
Two contract cases stay scoped away from email and gmail: their grep/rg push
down to a server-side search that reports "no matches" for a path that does
not exist. Six typescript-only ENOENT prefix sites remain in box, discord,
gdocs, gdrive, gsheets and gslides.
Also includes the pre-existing session-mode integ migration that was already
in the tree: session_modes scripts replaced by session/modes.json.
* fix(jaeger,langfuse): CodeQL ReDoS, eslint findings, formatter pass
- stat used a /\/+$/ trailing-slash regex in jaeger and langfuse, which
CodeQL flags as js/polynomial-redos on a path from library input. Both now
use the existing loop-based rstripSlash. Python already used str.rstrip.
- errorMessage stringified an unknown errors[0].msg, which could render as
[object Object]; now requires a non-empty string, mirroring python's
truthiness check.
- readdir tested pattern against undefined, which its type excludes.
- drop an unused fixture constant in read.test.ts.
* fix(integ): shell-attributed redirect golden, regenerate specs for jaeger
The ro_write_refused contract case expected "echo: <path>: Operation not
supported", but #635 (which landed in main after this branch was cut)
attributes a failed redirect target to the shell rather than the command, so
the correct stderr has no command prefix. It is the only contract case using a
redirect; mkdir, rm and tee take file operands and keep their prefixes.
Regenerate spec/ for the new jaeger resource (154 one-line additions across
both generators). CI runs gen_specs.py and gen-specs.ts as a separate drift
step, not through pre-commit, so a local pre-commit run does not catch this.
* fix(ci): build mirage-browser in the observability and facet integ jobs
The typescript runner imports mirage-browser statically for the opfs target,
so its dist must exist even in a job that runs no browser target. Both new
jobs hand-roll their setup and built only core and node, so every typescript
host step died with ERR_MODULE_NOT_FOUND. The pre-existing jobs get the build
from the integ-battery-setup action, which is why only these two were hit.
This was masked in the first run: the typescript step is skipped when the
python step fails, so the stale golden hid it.
* fix(jaeger,du): codex review, service-scoped reads and a real request deadline
read() served a trace fetched by id through any service directory, so
/services/<other>/traces/<id>.json returned content that stat and ls both
report absent. It now asserts the service exists and that the trace's own
process table names it. Membership comes from the trace document, not the
service listing, because that listing is windowed and limited and would hide a
trace that really does belong.
The typescript jaeger transport ignored requestTimeout: the config field, the
snake-case mapping and the transport option all existed, but nothing read them
and fetch ran with no deadline, so a stalled endpoint hung the command forever.
Threaded through as seconds, matching python's httpx timeout and dify's
existing requestTimeout convention.
du's operand stat caught every exception and reported it as
"No such file or directory", so an auth failure or a backend bug printed a
wrong reason and a partial total. Narrowed to isMissingPath, matching python's
(FileNotFoundError, ValueError). The test mock rejected with a bare
Error('ENOENT') rather than a stamped FsError, which is what let this pass.
Moved the jaeger resource's command and op imports to module scope, per the
repo's import rule. Verified no cycle: the registry and workspace still import
cleanly. This also matches the typescript resource, which already imports
JAEGER_COMMANDS at module scope.
Integ gains a cat and a stat through a foreign service, the pair that proves
the two agree.
CLI exit codes / changes (push) Has been cancelled
Test (Install) / changes (push) Has been cancelled
Integ / changes (push) Has been cancelled
Test (Python) / changes (push) Has been cancelled
Test (TypeScript) / changes (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / python-install (push) Has been cancelled
Test (Install) / ts-install (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-shared-py (push) Has been cancelled
Integ / integ-shared-ts (push) Has been cancelled
Integ / integ-shared-parity (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Integ / integ-fuse-windows (push) Has been cancelled
Integ / runtime-py (push) Has been cancelled
Integ / runtime-ts (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Python) / runtime (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
* test(integ): add a fake GitHub API server and a github target
Fakes the four api.github.com routes the github backend calls (repo, git
trees, git blobs, code search) and wires a `github` target for both hosts.
Code search is token-indexed like the real API rather than substring, so
the battery sees the same push-down semantics production does.
Adds `GitHubConfig.base_url` to match the existing TypeScript `baseUrl`
seam; Python had a hardcoded API_BASE and could not be pointed anywhere.
* fix(grep): require -w and a literal pattern for search push-down
grep/rg narrow their scope with the provider's search API, then scan only
the narrowed files. Provider search matches whole words while grep matches
substrings, so for a bare literal the narrowed set is a strict subset of
the real matches and files holding the literal only inside a longer word
were silently dropped. Over the new fake, `grep -rl quokka /repo` returned
3 files while the same pattern under the size threshold returned 4.
Push-down now needs -w, where both sides agree and any tokenizer
disagreement can only over-fetch, which the local scan filters. It also
needs a fully literal pattern: a regex narrows on an extracted literal, so
the searched term is only part of the match (`foo[0-9]` matches `foo1`
under -w, but searching `foo` never returns a file whose only token is
`foo1`). Applies to github, box, dropbox and slack in both languages.
Drops files_only_shortcircuit: it bails on -w, which push-down now
requires, so it was unreachable, and it cannot be re-enabled safely
because over-fetch is fatal to a path that skips the scan.
* fix(grep): extend the -w push-down gate to gmail, discord and mongodb
Same defect as github/box/dropbox/slack: the provider's search matches
whole words while grep matches substrings, so a bare literal under-reports.
gmail and discord return search results verbatim as the grep output, so
their native path now needs -w too.
mongodb needed a different fix. Its $text branch matches whole words and
stems them, so it both missed `foo` inside `foobar` and matched stems the
pattern never had, with no local re-scan to filter either. $regex was
already the fallback and takes the pattern as written, so the $text branch
is gone.
Integ coverage for the narrow-then-scan backends: the box and dropbox fake
servers now match content by whole word like the real providers, since a
substring fake agrees with a full scan and proves nothing. A /search mount
over a 3-file trap fixture pins both paths; removing the gate makes the
non--w cases fail with the exact under-report.
Also updates the slack and gmail integ cases that asserted native-search
output, and adds gmail push-down tests, which did not exist.
* fix(integ): move the fake GitHub server off trello's port
The new fake GitHub server was started on 5095, which trello_server.py
already uses. Trello started second, failed to bind, and its requests
reached the GitHub fake instead, so every trello case failed with
"Trello API error (/members/me/organizations): HTTP 404".
* Fix dependency and CodeQL security alerts
* fix(stat): port linear-time format parser to Python (CodeQL #247)
The TypeScript fix left the identical backtracking regex live in the
Python mirror, so stat -c FORMAT could still hang on a crafted format.
Replace _FORMAT_RE with the same cursor-based directive scanner.
* fix(readdir): ENOENT vs ENOTDIR for a missing path (py+ts, 6 backends)
`ls` on a missing path reported the wrong errno string on several
backends, and Python/TypeScript disagreed on the same backend.
GNU pin (`docker run --rm debian:stable-slim`):
ls /nope -> No such file or directory
ls /data/nope -> No such file or directory
ls /data/nope/deeper -> No such file or directory (however deep)
ls /data/a.txt/x -> Not a directory (component is a file)
ls /data/a.txt/ -> Not a directory
The rule: ENOTDIR only when a path component exists and is not a
directory; a component that does not exist is ENOENT, at any depth.
Every affected site was wrong, in both directions:
| backend | before |
|----------|--------------------------------------------|
| py disk | always ENOTDIR (+ leaked the real fs path) |
| py ram | always ENOENT (missed genuine ENOTDIR) |
| py redis | always ENOENT |
| ts ram | always ENOTDIR |
| ts disk | mapped ENOENT -> enotdir explicitly |
| ts opfs | mapped NotFoundError -> enotdir |
Store-backed backends (ram, redis) have no kernel to draw the line, so
they now ask a new shared helper — `readdir_error` / `readdirError` —
which walks the ancestors and returns ENOTDIR at the first existing
non-directory component, else ENOENT. Kernel/API-backed backends already
had the right signal and just needed to stop squashing it: disk lets
listdir raise and restamps onto the PathSpec (which also fixes the
real-fs-path leak in the Python error), opfs keeps NotFoundError->ENOENT
distinct from TypeMismatchError->ENOTDIR.
Blast radius is stderr bytes only: no caller distinguishes the two codes
after a readdir (glob_walk, condition operators, chroma/dify walk,
find_action_dispatch, wasm/fs, monty all catch them together). Three
tests pinned the old behavior and were corrected.
Coverage: 6 cross-backend cases in integ/unix/nf/ls.json (ram, disk,
redis, opfs — `nf/` had cat/grep/head/stat/tail/wc but no ls, which is
the gap this lived in), 5 in conformance/cases/ls.json, plus helper and
per-backend unit tests on both sides. The new cases pin exit 1, matching
this tree's `ls` exit and the existing convention in
integ/resources/slack/ls.json; they will need the same bump as the other
ls cases when the 0/1/2 exit-code split lands.
Verified: pre-commit clean; python + TS integ 1886/0 (ram) and 1874/0
(disk) on both hosts; full TS suites and targeted Python suites pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(readdir): stop the errno walk at the first missing component
readdir_error / readdirError scanned every ancestor for a file and
reported ENOTDIR if it found one at any depth. A flat store can hold a
key whose parent is not a directory (RAM/Redis rename writes the
destination key without adding its ancestors), so `ls /missing/a.txt/x`
found the orphan `/missing/a.txt` and said "Not a directory" while
`ls /missing` said "No such file or directory" for the same tree.
Both helpers now take an is_dir / isDir probe and stop where the kernel
stops resolving: the first component that is neither a directory nor a
file is ENOENT. A file component is still ENOTDIR.
* fix(tree/grep/rg): keep walking when readdir reports ENOTDIR
Splitting the readdir errno into ENOENT/ENOTDIR left the python tree
and grep/rg walkers behind: they caught only
`(FileNotFoundError, ValueError)`, and NotADirectoryError is a sibling
of FileNotFoundError under OSError, not a subclass. So on RAM/Redis:
- `tree /a.txt/x` escaped the walk and returned exit 1 with no body,
where `tree /nope` still printed GNU's `[error opening dir]` + exit 2
- `grep -rl foo /a.txt/x /real` aborted and dropped /real's results,
where the same line with `/nope` warned and kept them
- `rg -l foo /a.txt/x /real` did the same
GNU pins (debian:stable-slim): `tree /d/a.txt/x` prints the same body
and exits 2 as `tree /d/nope`; `grep -rl foo /d/a.txt/x /d/real` warns
and still prints `/d/real/b.txt`.
Both cases now route through one named `WALK_ERRORS` tuple (FS_ERRORS
plus the ValueError store backends raise for "not a directory"), so a
future errno split cannot desync these catch sites again. TypeScript
already used a broad catch and was unaffected; python now matches it.
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
`ls a b` flattened every operand's entries into one unseparated list
instead of printing GNU's `a:` / `b:` blocks. Pinned the real rule with
docker (debian:stable-slim, coreutils 9.7): headers are keyed on the
*operand count*, not on how many directories were listed —
ls a -> bare listing, no header
ls a b -> `a:` block, blank line, `b:` block
ls /nope a -> `a:` (2 operands, one of them failed)
ls -R a -> `a:` (recursive always heads)
and non-directory operands print first as one unheaded block, with all
operands sorted together before that split.
Both the non-recursive and `-R` paths now share one `probe_operand` /
`probeOperand` that lists an operand and reports whether it turned out
to be a directory, so unifying them fixed three more divergences found
alongside the reported one:
- operands were listed in command-line order, interleaved: `ls b zfile
a mfile` printed `g.txt /zfile f.txt sub /mfile`
- `-d` did not sort its operands (`ls -d b a` -> `b a`)
- `ls -R zfile a` emitted a bogus `zfile:` header with `zfile` as its
own entry
`ls -R` on a directory whose readdir fails no longer prints an empty
header, and the TS path now type-checks a failed-readdir operand before
treating it as a file row (Python already did).
The exit-code rule is unchanged in meaning but no longer keyed on
stdout being empty: it asks whether any operand listed, so a header
line cannot flip the code. That keeps this independent of the in-flight
ls exit 0/1/2 work.
Verified: 41 Python + 16 TS unit cases with byte-exact stdout, and both
integ hosts at 1882 passed / 0 failed on `--target ram` (all 7 pinned ls
surfaces unchanged, including the 24-operand `ls /data/*.txt` glob).
Cross-mount `ls` (operands on different mounts) still prints flat — the
FANOUT strategy runs one native ls per operand, so each run correctly
sees a single operand. Closing that needs operand count threaded
through the executor and still cannot produce GNU's global operand
sort, so it is left out here.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* fix(shell): shell-attribute unopenable redirect targets (py+ts)
An unopenable redirect target diverged three ways. GNU bash 5.2.37
answers both `cat < missing` and `echo x > /nosuchdir/f` with
`bash: line 1: <target>: No such file or directory` and exit 1 —
attributed to the shell, since the command never runs. mirage said:
py `cat: /data/missing: No such file or directory`
ts `/data/missing` (bare path, no reason)
Both hosts now render `<target>: <strerror>` on the `<` read and the
`>` write side alike, through one helper per host.
Deliberate divergence: GNU's `bash: line N:` prefix is dropped, matching
the house style already set by the other shell-attributed error,
`nosuchcmd: command not found` (bash prints `bash: line 1: nosuchcmd:
command not found`). `bash:` is bash's `$0` and mirage is not bash, and
`line N` has no meaning for a one-line `Workspace.execute` call. Both
docstrings record the choice.
The wording bug hid three behavior bugs, all fixed by returning an
IOResult instead of raising:
- The whole line died. `cat < missing; echo next` never ran `echo next`;
GNU prints the error and continues (rc becomes echo's 0). `&&` now
short-circuits, `||` runs, and pipeline peers still run. The `>` side
had no handler at all on python, so it died the same way.
- Unwinding reached the workspace-level OSError handler, which stamps
`command.split()[0]` — the first word of the *whole line* — so
`cd /data && cat < missing` reported `cd:`.
- Backend prose reached the user as the path: `echo x > /nodir/f` said
`echo: parent directory does not exist: /nodir: No such file or
directory`. The label is now the target's own spelling (`raw_path`),
so a relative target reports as typed like GNU.
The ram/redis `writeBytes` guards threw an untyped Error on TS while
python raised FileNotFoundError, so the TS write side could not reach
the GNU line at all. Both now throw a stamped ENOENT that keeps the
human message (new `enoentWithMessage`, mirroring `eaccesReadOnly`) —
the FUSE bridge and executor builtins still sniff that prose off the
exception.
Non-filesystem errors keep propagating on both paths and both hosts.
Coverage: 20 new unit tests (10 py + 10 ts) and 3 integ cases under
integ/bash/redirect/, all GNU-pinned via docker debian:stable-slim.
Read-only refusal pins move to the shell-attributed spelling on both
hosts (GNU: `bash: line 1: /ro/y.txt: Permission denied`), as does the
mem0 write-rejected integ case.
Known divergence left in place, documented in both docstrings: bash
processes redirects strictly left to right, so `> out < missing` empties
`out` before failing, while mirage creates output files in a second pass
and leaves it untouched. `< missing > out` leaves `out` uncreated on
both, matching bash.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(integ): scope the write-failure redirect case to backends that enforce parents
CI showed `redirect_write_target_unwritable` failing on 11 of its 17
targets, all for the same reason: a missing parent directory is not a
write error there. disk, opfs, nextcloud and the object stores (s3,
s3-prefix, dropbox, dropbox-root, onedrive, sharepoint,
sharepoint-prefix) create the key regardless, so stderr was empty and
the redirect simply succeeded. ssh fails a third way — it raises an
error carrying no path, which still unwinds the line (identically on
both hosts, so parity holds; that shape is pre-existing).
Only ram and redis enforce parent existence, and their `writeBytes`
guard is exactly what this PR retypes, so the case now runs there. The
read-side case still pins the shell-attributed rendering on all 17
targets, and the write path keeps its 10 unit tests per host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(shell): stop the redirect write pass at the first failure (py+ts)
Two review findings on #635.
The write pass continued after a failed open, so `echo x > /nodir/f >
/data/out` reported the first error but still created /data/out. GNU
5.2.37 (pinned, debian:stable-slim) stops processing redirects there:
$ echo x > /nodir/f > /data/out
bash: line 1: /nodir/f: No such file or directory # rc=1
$ ls /data/out -> No such file or directory
The mirror case keeps the earlier target, which bash had already opened
and truncated, so `break` is right for both:
$ echo y > /data/out2 > /nodir/g
bash: line 1: /nodir/g: No such file or directory # rc=1
$ ls -l /data/out2 -> 0 bytes
The python append pre-read only caught FileNotFoundError, so a
PermissionError from the mount guard (`echo x >> /b/f` for a session not
granted /b) escaped handle_redirect entirely and unwound to the
workspace-level OSError handler — reintroducing, for `>>` only, the bug
this PR fixes for `>`: the rest of the line died and the message was
stamped with the whole line's first word. It now catches FS_ERRORS and
logs at debug. TypeScript's readExisting swallowed everything instead,
including backend bugs, so it now rethrows non-filesystem errors; both
hosts land on the same rule.
Coverage: 3 new py unit tests, 3 new ts unit tests, and 2 integ cases on
ram + redis (the targets that enforce parents). py + ts integ green on
both targets, pre-commit and `pnpm -r typecheck` clean.
* fix(shell): shell-attribute an ungranted mount on the TS redirect path
The py/ts parity claim in the previous commit only held for mounts the
session had a READ grant on. A mount with no grant at all goes through
`assertMountAllowed`, and TypeScript's `MountNotAllowedError` carried no
POSIX code, so `isFsError` said false and the redirect write pass
rethrew it. Verified before fixing:
echo leaked > /b/leaked.txt; echo next
py exit 0, "next\n", "/b/leaked.txt: Permission denied\n"
ts exit 1, no "next", "session 'agent' not allowed to access mount '/b'"
That is the exact bug this PR set out to fix, still live on TS for both
`>` and `>>`. Python has no equivalent hole because its guard raises
PermissionError, already a member of FS_ERRORS.
MountNotAllowedError is now stamped EACCES + operand, mirroring python's
PermissionError, so the redirect pass renders the same line both hosts.
The `instanceof MountNotAllowedError` checks in command.ts and the FUSE
bridge run earlier and are untouched, so command-level messages
("cat: session '...' not allowed to access mount '/side'") do not move:
truth_session_modes.txt still matches byte-for-byte on both hosts.
Full TS core (5070) and node (1773) suites pass.
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* fix(ls): GNU exit-code seriousness split (0/1/2) in py+ts
`ls` returned 1 for an inaccessible command-line operand where GNU
coreutils returns 2, and — worse — masked the failure entirely when any
other operand succeeded: `ls /data/nope /data` exited 0.
The rule was `1 if warnings and not results else 0`, i.e. nonzero only
when *every* operand failed. GNU instead ratchets a status upward
(`set_exit_status`): 0 ok, 1 minor problem (trouble met below an
operand), 2 serious trouble (a command-line operand could not be
accessed); serious always wins, minor only upgrades a clean run.
Pinned against GNU coreutils 9.7 (debian:stable-slim), including the
non-root cases that reach the minor-problem path:
ls /nope -> 2
ls /nope /ok (either order) -> 2, still lists /ok
unreadable dir as operand -> 2
ls -R, unreadable subdirectory -> 1, keeps parent output
ls -l, unstattable entries -> 1, lists the siblings
minor + serious together -> 2
Warnings now carry a `serious` flag and are collapsed by
`exit_status_for` / `exitStatusFor`. Seriousness comes from a
`command_line_arg` / `commandLineArg` parameter threaded through
`walk`/`walk_grouped` (`walkGrouped`), false on recursion; a per-entry
stat failure inside a listed directory is always minor.
Three fixes were load-bearing for the split:
* Both executors gated `ls` child-mount and symlink injection on exit
code == 0. A minor problem still lists the directory, so exit 1 would
have silently dropped the injected rows; the gate is now
`!= LS_FAILURE`. (Injection already bails for multi-operand and for
-R/-d, and usage errors return early via UsageError, so the affected
surface is a single-operand listing with a minor warning.)
* Python's minor path was unreachable: `walk` caught only
FileNotFoundError/ValueError/NotADirectoryError, so a PermissionError
escaped the command and discarded stdout. Now `(OSError, ValueError)`,
matching TypeScript.
* TypeScript's `listDir` used `Promise.all`, so one failing child stat
killed the whole listing and surfaced as an operand failure — which
would have exited 2 where Python exits 1. Now `Promise.allSettled`
with per-entry warnings, matching Python's tolerance.
Also fixes a Python-only bug where `walk_grouped` appended a group
unconditionally, so `ls -R /nope` printed a bogus `/nope:` header on
stdout (TypeScript was already correct).
Pinned surfaces: integ/resources/slack/ls.json (slack_ls_archived_missing)
1 -> 2, and five hard-pinned unit tests. conformance/cases/ls.json had no
error coverage, so two cases were added; they assert the status via
`2>/dev/null; echo rc=$?` because the stderr text itself diverges for an
unrelated pre-existing reason (Python+disk and TypeScript+RAM report
"Not a directory" where GNU and Python+RAM say "No such file or
directory") — tracked separately.
Verified: 26 Python + 14 TypeScript unit tests; integ 1611 passed / 0
failed on both hosts; conformance green on both; both hosts byte-identical
across all 12 pinned scenarios.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(ls): mem0 golden, -R separator, review nits
- integ/resources/mem0/basic.json: mem0_readdir_of_file_enotdir asserted
exit 1 for an inaccessible command-line operand; GNU returns 2. This
was the second nonzero-ls golden in the repo and the only CI failure.
- The -R blank-line separator was keyed on the operand index, so a first
operand that failed (rendering no group) still made the next one emit a
leading blank. It now keys on what was actually emitted, matching GNU
and the TypeScript path.
- Test denial stubs moved to module scope (no nested functions) and the
new docstrings carry Args types.
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* feat(shell): case ;&/;;& fallthrough, printf -v, source args, unset -f/-v (#608 Tier 2)
Fixes the four Tier 2 items from #608 that silently produced wrong output
rather than a missing feature. All are pure language/session semantics
with no job-table involvement.
- case `;&` runs the next arm's body unconditionally (fallthrough); `;;&`
keeps testing later patterns. The terminator is now captured per arm in
get_case_items/getCaseItems and honored by handle_case/handleCase.
- printf `-v NAME` assigns the formatted text to a variable or array
element `NAME[idx]` instead of writing to stdout, keeping GNU's
exit-1-still-assigns behavior on a bad number.
- source/. positional args set $1..$# for the sourced script and restore
the parent's afterwards; the no-arg form keeps the parent's params.
- unset gains -v/-f/-n: -f unsets a function only, -v a variable only, a
bare name a variable if one exists else a function; plus whole-array
and arr[i] element unset. -n is a documented no-op (no nameref
attribute exists, matching bash on a non-nameref name).
Divergence: a trailing `;&`/`;;&` immediately before `esac` does not
tree-sitter-parse (same on main); it is a no-op spelling and the loud
rejection is acceptable.
GNU bash 5.2 pinned for every case. Python and TS unit tests added (each
verified to fail on the unfixed code), 11 integ cases under
bash/{case,printf,source,builtin} run 1834 passed / 0 failed on both the
Python and TS hosts.
* fix(shell): close the #630 review findings on printf -v, unset, and source
Every codex/claude review finding plus CodeQL alert 259, in both
languages, each behavior pinned against GNU bash 5.2 first.
- unset of an array element keeps the later positions: an interior
element leaves a hole, a trailing one shortens the array. The old
del/splice shifted every later index, so ${a[1]} returned what used to
be ${a[2]}.
- printf -v and unset refuse a readonly target, resolving an arr[i]
operand to its base name (which is what readonly records, and what
bash names in the error).
- printf -v on a bare name whose variable already holds an array writes
element 0 and keeps the tail instead of deleting the array.
- printf -v mutates nothing until the assignment succeeds, so an
out-of-range subscript no longer destroys the existing scalar. It
reports `bash: TARGET: bad array subscript` with status 1.
- printf -v validates the name before the format runs: status 2 with
"not a valid identifier", and no conversion errors.
- unset 'scalar[0]' unsets the scalar; a non-zero subscript on a scalar
reports "not an array variable" with status 1. Both were no-ops.
- TS source/. positional args use wordText, so $1 is the operand as
typed, not the resolved mount path (Python and bash already agreed).
- python seeded name[idx] from an empty scalar as an empty array, so
Z=""; printf -v 'Z[-1]' hi errored instead of assigning element 0.
The readonly guards were unreachable from the shell until execute_node
also learned that a bare `readonly NAME` operand parses as a
variable_name node, and that an array assignment still has to mark its
name readonly. That also makes local/export register bare names.
CodeQL 259 was a real prototype pollution: session.arrays['__proto__']
returns Object.prototype on a plain object, so printf -v '__proto__[0]'
wrote through to it. sessionEntry/setSessionEntry in session.ts guard
every read and write of a session record.
Known divergence, documented in both docstrings: arrays are stored
densely, so an interior hole is an empty string and still counts toward
${#a[@]} and ${a[@]}. That predates this change (a[9]=v pads the same
way); a sparse array model is its own PR.
8 new/updated integ cases under bash/{builtin,printf,source} pass
identically on the Python and TypeScript hosts.
* fix(shell): make arrays sparse so unset arr[i] and arr[9]=v stop lying
Arrays were stored as a dense list of values, so an index with no
element had to be spelled as an empty string. That made two things lie:
`unset arr[i]` left a countable empty element (or, before the last
commit, shifted every later index), and `arr[9]=v` reported
${#arr[@]} as 10.
Arrays are now `list[str | None]` / `(string | null)[]`, where null is a
hole: addressable, but not an element. One shared primitive owns the
rules (mirage/shell/array.py, shell/array.ts) and every consumer goes
through it:
- ${arr[@]} / ${arr[*]} and the slice, strip, replace, and case forms
expand only the assigned values.
- ${#arr[@]} counts assigned elements; ${!arr[@]} lists their indices.
- ${arr[i]} on a hole is empty and counts as unset for ${arr[i]:-x}.
- a negative subscript resolves against the extent (one past the
highest assigned index), so ${arr[-1]} is the last real element.
- arr+=(x) starts at the extent, refilling the slot a trailing unset
freed and skipping interior holes.
- unset arr[i] drops trailing holes, keeping the extent honest.
- declare -a NAME with no value now declares an empty array rather than
an empty scalar, so ${#NAME[@]} is 0 and NAME[3]=x leaves index 0
unassigned.
- an existing scalar seeds element 0 even when empty, matching bash on
Z=""; Z[3]=x.
An empty string stays a real element, so a=("" y) still has two.
14 shell-level cases pinned against GNU bash 5.2 cover holes,
extents, negative subscripts, slicing, case/replace ops, iteration,
append, reassignment, and declare -a; they run in both tests/shell and
shell_arrays.test.ts, plus 10 unit tests per language on the primitive
itself and two integ cases on both hosts.
* fix(shell): count an empty scalar as one element in ${#x[@]}
Expansion seeded the implicit one-element array from a scalar only when
the scalar was truthy, so `Z=""` looked like an empty array: ${#Z[@]}
reported 0 and ${!Z[@]} was empty, where bash reports 1 and 0. An
actually-unset name is the case that has no elements, so the seed now
keys off presence in env rather than emptiness.
Found while auditing the sparse-array consumers; pinned against GNU
bash 5.2 and covered in tests/shell/test_arrays.py and
shell_arrays.test.ts.
* fix(shell): write array elements through splice (CodeQL 260)
arraySet/arrayUnset assigned through a script-derived subscript
(`arr[idx] = value`), which is the prototype-polluting-assignment shape
even though the index is a validated non-negative number and the target
is always a real array. splice cannot name a property at all, and both
call sites already guarantee idx < arr.length at that point, so the
behavior is unchanged.
* fix(test): drop a dead ?? on ExecuteResult.stderr
ExecuteResult.stderr is a Uint8Array, never null, so the fallback in the
readonly test tripped @typescript-eslint/no-unnecessary-condition. Local
`pre-commit run --files <changed>` missed it because the type-aware rule
only reported it against a fresh build; `--all-files` reproduces CI.
* fix(shell): close the six codex findings on the sparse array model
All pinned against GNU bash 5.2 first, both languages.
- ${a[@]:o:l} slices by SUBSCRIPT, not by position among the assigned
values. For a=([1]=b [3]=d [9]=j), ${a[@]:2} is `d j` because it keeps
every index >= 2; length then caps how many are taken. Compacting with
array_values first turned the offset into an ordinal. A negative offset
resolves against the extent and yields nothing when it stays negative,
where the old code clamped to 0 and returned everything.
- `declare -a` / `local -a` inside a function are local again: the
caller's array is recorded in session._local_arrays / localArrays and
restored on return, and the declaration shadows with a fresh empty
array. `f(){ declare -a leak; leak[2]=x; }; f` no longer leaks, and
declaring an existing array local no longer mutates the caller's.
- `x=foo; declare -a x` migrates the scalar to element 0 instead of
installing an empty array that shadowed it, so ${#x[@]} is 1. An
existing array is left alone.
- Array declarations are staged until the readonly guard passes, so
`readonly -a a=(y)` on an already-readonly name fails with the old
value intact instead of overwriting it first.
- `printf -v 'a[]'` is rejected as an invalid identifier with status 2;
the subscript group now requires at least one character. `a[ ]` stays a
valid arithmetic 0.
- `unset 'a[-2]'` on a shorter array reports `unset: [-2]: bad array
subscript` with status 1 and keeps the array, rather than silently
succeeding. bash prints only the bracketed part here, unlike the
assignment and printf -v paths which name the base.
Also fixed in passing: a declaration-context `a+=(y)` stored under the
literal key "a+" instead of appending to `a`.
26 new bash-pinned cases per language in tests/shell/test_arrays.py and
shell_arrays.test.ts (79 each), plus unit tests for array_slice and the
two new error paths.
Three fixes from an audit of the TypeScript side.
1. WorkspaceFS.exists/isDir/isFile caught every exception and returned
false, so an auth failure, timeout, or backend bug read back as "this
path does not exist". Python already had the right answer: it swallows
only (FileNotFoundError, ValueError), where ValueError is the registry's
"no mount matches path". Added noMount() plus an isMissingPath()
predicate and applied it to the three probes, to pathExists/isDirectory
in the copy helpers, and to the same pattern in realpath and df, both of
which had already diverged from their narrow Python twins.
2. LanceDB blob decoding called the Node global Buffer inside the
runtime-agnostic core package, which breaks in browsers. It now uses the
existing decodeBase64 helper, matching its qdrant sibling. The two gmail
base64url helpers moved onto the shared helpers as well.
3. TypeScript conformance only ran against RAM while Python ran RAM, disk
and Redis. The runner now covers all three with fresh backend state per
case, and all 34 cases pass on every backend. Teardown is split so
clearing Redis happens before Workspace.close() destroys the client,
rather than silently opening a second connection per case.
Also wired integ/runners/parity.py into CI as integ-shared-parity and added
it to the gate; it existed but nothing ran it. Locally it compares 5386
case/target pairs with no mismatches. Added conformance/** to the python and
typescript workflow path filters, which did not trigger those suites before.
The test stubs that faked backend errors as new Error("not found") or
new Error("ENOENT"), with the code in the message instead of on the error,
now use enoent() like real backends do.
* fix(stat): apply the namespace attr overlay in the bespoke stat commands (py+ts)
#626 wired the namespace attr overlay (chmod/chown/touch on backends with no
setattr op) into the generic_bind `stat` builder, but the three hand-written
`stat` command overrides bypass that builder entirely and kept reading the raw
backend stat. None of those backends override `ls`, so `ls -l` merged the
overlay while `stat -c` did not, and the two disagreed for the same file:
$ touch -t 202401010000 f && chmod 640 f # on an opfs mount
$ stat -c "%y|%a" f
2026-07-24T23:44:39.946Z|644 # OPFS write time, default mode
$ ls -l f
-rw-r----- 1 user user 6 Jan 1 00:00 f # touched + chmod'd
Fixed in browser opfs, core s3 and node gridfs (ts) plus s3 and gridfs (py).
On the Python side `stat_overlay` was not declared at all, so it fell into
`**_extra` and was silently swallowed.
TS gains an exported `overlaidStat(stat, overlay)` in generic_bind/adapter.ts,
mirroring the Python `overlaid_stat` that already existed, and all six stat/ls
call sites now bind through it instead of repeating the inline ternary.
`history` overrides both `ls` and `stat` with raw backend stats, so it already
self-agrees; it is a read-only view mount and is left alone.
Coverage: the existing meta_overlay cases assert through the harness `check`,
which routes via `ws.dispatch('stat')` (the ops facade) and therefore masked
this command-level gap, so the new integ cases assert on `stat -c` stdout
directly. Unit tests cover opfs (workspace-level) and py s3/gridfs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(stat): concrete mock types in the s3/gridfs stat tests
Codex review on #631: the fake backend helpers annotated the accessor and
index as `object`, which defeats type checking exactly where the mocks are
meant to mirror the production call signatures. AGENTS.md reserves `object`
for opaque `**flags` bags only.
Use S3Accessor/GridFSAccessor and IndexCacheStore, and replace `_render`'s
`**kw: object` pass-through with an explicit `stat_overlay: StatOverlay | None`
parameter — it forwards exactly one kwarg, so it was never a flag bag.
`_fake_stat_core` keeps its parameter literally named `index`: bound_op binds
it by keyword (`partial(fn, accessor, index=index)`), so renaming it to
`_index` would break the no-overlay path. The overlay path calls it
positionally, and both paths are covered by the two tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(shell): single-quoted redirect target silently wrote nowhere
`> 'path'` exited 0 without creating the file. `TARGET_TYPES` — the
allowlist of tree-sitter node types accepted as a redirect operand —
listed STRING (double quotes) but not RAW_STRING (single quotes), so
parseFileRedirect never matched the operand and left target_node None
with target "". Downstream that empty string was used as the path
verbatim. Both hosts had the identical defect.
The empty path is a real key in the store, so every single-quoted
target in a session aliased onto one phantom file. Beyond the reported
`>` data loss this also meant:
- `2> 'f'` swallowed stderr entirely (nonzero exit, empty stderr)
- `< 'f'` returned an unrelated file's bytes
- `>> 'f'` / `&> 'f'` wrote nothing
- `<<< 'text'` inside a redirected statement delivered a bare
newline (parseHerestringRedirect shares the same gate)
Fix is one entry per host. Pinned against GNU bash 5.2.37 in docker
(debian:stable-slim) with a 26-case diff harness: byte-identical after
the change.
Tests: 15 Python, 13 TypeScript, all failing on a reverted fix. The
double-quoted and unquoted parametrizations act as controls. New integ
group integ/bash/redirect/quoted_target.json (13 cases, seq
603200-603212) using the canonical 20-target mutation list copied from
redirect.json — hf/hf-prefix excluded, as they reject mutations. Both
hosts: 3248 passed, 0 failed on ram+disk; reverting the fix fails
exactly the 9 single-quoted cases.
Aliasing tests assert on bytes rather than the error message because
the missing-stdin-source wording still differs between the two hosts
and from GNU; that pre-existing divergence is filed separately and
reproduces with unquoted targets too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(shell): hoist redirect-target test helpers to module scope
Review follow-up on #632. `targetTypeOf` and `statementWith` were
declared inside the `describe` callback, which violates the repo-wide
no-nested-functions rule (AGENTS.md L107-108) and also contradicts the
file's own precedent — `node()` has always been module-scope.
Both now sit beside `node()`. `statementWith` is renamed
`redirectStatement` (a file-global name should say what it builds) and
constructs its own command node rather than capturing the describe-scope
`command`/`commandName` consts, so the move carries no state with it and
those two consts are gone.
Test-only; no production change. 30 tests in the file pass, the full
core suite is 4973 passed / 0 failed, and all 7 packages typecheck.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(cp/mv): GNU update/backup/target-dir/exchange flag semantics (#609 Tier 1)
Implement the cp/mv "semantics flags" slice of #609 Tier 1 with Python/
TypeScript parity, pinned against GNU coreutils 9.7.
cp gains -u/--update[=all|none|none-fail|older], -b/--backup[=CONTROL]
+ -S/--suffix, -t/--target-directory, -T/--no-target-directory, the long
spellings of -r/-a/-f/-n/-v, and -f/-i/--strip-trailing-slashes as
documented no-ops, plus GNU overwrite-type guards and arity errors.
mv additionally gains --exchange (atomic swap via three renames),
--no-copy (cross-mount refusal), and rename-level -n/-u/-b gating, sharing
cp's transfer/backup/policy engine.
Supporting fixes that fell out of end-to-end testing:
- New Option.short_value flag so an optional-value short (cp -b/-u) stays a
clusterable boolean (`-bv`) instead of eating the cluster remainder as a
value; only --backup=/--update= carry values (GNU).
- cp/mv now route path-valued flags (cp -t /other/mount/dir) through
cross-mount detection, and see the touch/chmod stat overlay so -u
freshness matches ls/stat.
- Single per-source entryKind probe + overwrite-gate early-out so API
backends pay no extra stat per entry when no gating flag is set.
New shared backup helper (utils/backup) implements GNU version-control
naming (simple/numbered/existing, ~ and .~N~ suffixes); env-less default
is `existing`, VERSION_CONTROL/SIMPLE_BACKUP_SUFFIX deliberately not read.
Coverage: Python + TS unit tests for every new flag and error path, and
26 new integ cases across integ/unix/{cp,mv}. Full integ battery green on
ram (1849) and disk (1837), 0 failures, on both hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cp/mv): address codex review + green the CI (#609 Tier 1 follow-up)
CI fixes
- Spec drift: the new cp/mv options were never regenerated, and the new
Option.short_value field leaked into spec/python via asdict() while the
explicit TS serializer omitted it, breaking the two trees' key parity.
Emit short_value from gen-specs.ts too and regenerate all three trees.
- integ-shared-{py,ts}: the 26 new cp/mv cases listed hf/hf-prefix, whose
backend rejects these mutations ("Operation not supported"). Every
pre-existing cp/mv case excludes it; drop it to match the canonical set.
- integ-shared-ts opfs: OPFS_CP/OPFS_MV passed the raw backend stat, so -u
freshness could not see touch -d (OPFS has no setattr; touched times live
in the namespace overlay). Read opts.statOverlay like OPFS_LS already
does. Fixed properly rather than by excluding the target — mtime cases
include opfs by convention (the meta_overlay facet exists for it).
Codex findings, each pinned against GNU coreutils 9.7
- P1 --exchange staged through a fixed `<target>.~xchg~`, silently
clobbering a real file of that name, and left the operands half-moved on
a mid-sequence failure. GNU's renameat2(RENAME_EXCHANGE) touches nothing
else, so probe for a free staging name and roll back on failure; report
the leftover path when the rollback itself fails. Docstrings no longer
claim three renames are atomic.
- P1 A files-only find loop dropped every directory holding no files once
any update/backup mode was set, and copied an empty tree to nothing.
Narrow per_entry_native so the no-op modes (--update=all, --backup=none)
keep the whole-tree dir_copy, and recreate directories via a new optional
NativeCopy.mkdir on the genuinely-gating path.
- P1 The backup version scan swallowed readdir failures and read them as
"no numbered backups", which then picks .~1~ and overwrites backup
history. Propagate, and abort the overwrite instead.
- P2 -u and --update are one GNU option, so the last spelling wins; flags
are stored per spelling, which let a fixed read order override
command-line order. Mirror aliases in the parser for optional-value
options only — repeatable ones accumulate (sort -k/--key concatenates
both lists and would double).
- P2 mv -b -T refused a nonempty directory target before the backup could
displace it; GNU renames it aside and installs the source.
- P2 A cross-mount directory backup called read_bytes on a directory.
Walk the tree for the primitive strategies and defer to dir_copy on the
native one. Also fixes the same swallowed-readdir shape in mv's -T
emptiness probe, where "empty" was the clobbering direction.
Coverage
- 18 new integ cases across both hosts: the behaviors above plus gaps the
PR left uncovered — mv --no-copy (cross-mount refusal, source kept,
same-mount no-op), mv --backup=numbered, mv -S, mv --update=none, cp
--backup=none, and the long spellings (--suffix, --target-directory,
--no-target-directory, --recursive, --verbose, --strip-trailing-slashes).
- 20 new unit tests (py+ts) for the staging-name collision, both rollback
paths, dir preservation, no-op-mode dir_copy retention, backup-scan
failure, the -b -T backup, and last-wins aliasing.
Verified: pre-commit clean, full py suite, ts core 4991 + browser 167,
integ ram/disk/opfs green and byte-identical across the py and ts hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(test): satisfy tsc --noEmit in the new cp/mv tests
vitest transpiles without typechecking, so these slipped past a local
`pnpm --filter mirage-core test` and only the CI Typecheck step caught them.
- cp.test.ts: the typed-find helper declared its own `{ type?: string }`
options shape, which is not assignable to FindFn's FindOptions under
exactOptionalPropertyTypes (FindOptions.type is `string | null`). Use
FindOptions itself.
- mv.test.ts: eacces() takes one argument, not two.
Verified with the recursive typecheck CI runs (`pnpm -r typecheck`, all 7
packages clean) rather than the single-package filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: scope two new integ cases to capable backends; mirror attached short values
integ (the 6 remaining integ-shared failures, identical on both hosts)
- mv_b_T_backs_up_nonempty_dir backs the target up by *renaming a
directory*, which the dirless object stores cannot do — a directory there
is a key prefix with no object of its own, so s3 reports "The specified
key does not exist" and gridfs "No such file or directory". Dropped s3,
s3-prefix, gridfs, gridfs-prefix (20 -> 16 targets). No existing case
asserts a successful same-mount directory rename, which is consistent
with this being a backend limitation rather than a regression.
- xm_mv_no_copy_same_mount_ok needs a same-mount mv, i.e. a rename op,
which hf does not register; its sibling cases pass there only because
they refuse before touching the backend. Dropped hf/hf-prefix.
- Both crossmount groups now work inside their own subdirectories instead
of writing to the top level of /data and /data2, so they leave no residue
in the shared battery session.
parser
- The attached-short-value path (`-d10`) was the one write site my alias
mirroring missed, so last-wins held for `--long=` but not for the short
form: `split --numeric-suffixes=3 -d10` resolved to 3 instead of 10.
Python only — the TS regex already covered all 7 call sites. Regression
test on both sides pins the mirror and both orderings.
Verified: full py suite (8782 passed), all 7 packages typecheck, integ
ram/disk/opfs green on both hosts with identical counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: yapf line-wrapping on the alias-mirroring call sites
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(cp/mv): drop the object-typed backup param, share the flag helpers
Follow-up to the review round. Claude's review listed the `object`-typed
backup helper params among what it examined; it ruled them out as a bug, but
CLAUDE.md forbids `object` on a parameter outright ("only acceptable as the
value type of an opaque flag bag"), and chasing it surfaced a py/ts
divergence next to it.
- backup_control's `value` was `object`. FlagView.raw() legitimately returns
object (the bag IS opaque), but that stops at the boundary: for -b/--backup
the real type is `str | bool | None`. Narrow it in a new shared
`backup_raw()` and type the parameter accordingly. TS mirrors it —
backupRaw/backupControl were `unknown`.
- mv.py had reimplemented backup_raw and target_flags inline, 15 lines of
duplicated flag interpretation, while TS already exported backupRaw and
targetFlags and mv.ts used both. Un-privatized the two cp.py helpers and
used them from mv.py, so the two languages now share the same shape and
flag semantics live in one place per CLAUDE.md ("adding or changing a flag
should touch the spec and the generic, not N wrappers").
No behavior change: mypy clean (1555 files), pre-commit clean, full py suite
8782 passed, ts core 4992, integ ram/disk/opfs green on both hosts with
identical counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add TypeScript Graph and Mem0 parity
* Close the #628 review findings across Python and TypeScript
CodeQL flagged 12 polynomial-regex alerts, all the same `/^\/+|\/+$/g` path
trim on library input. They now use the existing loop-based stripSlash /
rstripSlash helpers.
Codex findings:
- `find` on an unscoped SharePoint mount returned nothing, because the
synthetic site and library levels carry no drive id. It now walks those
levels and delegates each library with a depth offset, so -maxdepth and
-mindepth still count from the real start path. Python had the same bug.
- A missing Mem0 memory raised the provider error instead of ENOENT, so
`test -e` and `cat` could not report "No such file or directory".
- `find -empty` treated every Graph folder as non-empty. It now reads the
folder facet's childCount, which Graph returns by default.
- A Graph folder's `size` is aggregate subtree storage, not a rendered
content length, so it moves from FileStat.size to extra.size_bytes.
Claude review findings:
- copyTree did not invalidate nested destination listings when a recursive
copy merged into existing folders, unlike Python's copy_tree.
- OneDrive readdir had a no-index branch that bypassed the 404 to
ENOENT/ENOTDIR translation.
- SharePoint du/duAll did not swallow a stale-cache ENOENT the way OneDrive
and both Python backends do.
The shared find_arg_errors truth file only listed the Python backends, so
the TypeScript job diffed against a stale file. Both lists are now
alphabetical and identical, and Python covers mem0, onedrive and sharepoint
as well.
Adding Mem0 integration cases (7 to 19) surfaced two divergences. A failed
redirect write in TypeScript printed the raw error, leaking the op-registry
wording, and now goes through formatFsError. That exposed the read-only
guard reporting different labels: Python used the guard's own message where
TypeScript reported the operand, so Python now stamps errno and the path and
both print `<cmd>: <path>: Permission denied`.
* Drop the unused @smithy/node-http-handler dependency
These two edits belonged in the merge commit but were never staged, so the
merge kept the dependency while carrying main's lockfile without it, and CI's
frozen-lockfile install refused to run.
Main's per-client httpAgentProvider only needs http-proxy-agent and
https-proxy-agent, which it already declares, so the branch adds no
dependencies and the lockfile matches main. The changeset drops the S3 proxy
and LanceDB lines that #627 ships.
* Route Graph and Mem0 config through the secret-redaction machinery
The three new resources bypassed the machinery that backs the other 66
config files. `Mem0Config.apiKey` and `MsGraphConfig.accessToken` were plain
strings with no zod schema and no `secretStr()`, where Python types both as
`SecretStr`. Each resource then hand-wrote its redacted state literal, so any
field added to the config later would silently leak into snapshot state or
vanish from it.
They now declare schemas and derive state from them:
- `MSGRAPH_CONFIG_SHAPE` marks `accessToken` secret and accepts the provider
callable as well as a literal, and the OneDrive and SharePoint schemas
spread it, mirroring Python's MsGraphConfig base model.
- `Mem0ConfigSchema` marks `apiKey` secret.
- `getState()` calls `redactXConfig`, the analog of Python's
`config_state` to `redacted_config_dump` path.
Registration was an unvalidated double cast on exactly these three of the 46
node factories, and the same three in the browser, so a bad config only
failed later at the first API call. They now call `normalizeXConfig`, which
camelCases the input and validates it against the schema like every other
resource.
Also regenerates spec/, which never picked up the three resources. That is a
separate CI step rather than a pre-commit hook, so a local `pre-commit run
--all-files` did not catch it.
`Mem0-User-ID` is documented rather than changed: it is the official SDK's
own client identifier, md5 of the API key, and the endpoints and pagination
match the SDK too, so the hand-rolled fetch client stays faithful to it.
* Fix TypeScript S3 proxy and blob decoding
* Remove TypeScript changeset
* Keep S3 request handler runtime-only
* Build S3 proxy agents per client instead of sharing one handler
Every S3 op destroys its own client in a finally block, so a
resource-lifetime NodeHttpHandler let one op's cleanup tear down
sockets a concurrent op was still using (ls -l stats entries in
parallel). Core now takes an httpAgentProvider callable, mirroring
presignedUrlProvider, and calls it once per client; node supplies
only the two proxy agents. Empty proxy is disabled, matching Python.
* Redact the S3 proxy in Python state to match TypeScript
Proxy URLs routinely carry user:pass, and TypeScript marks the field
as a secret. Python typed it as a plain str, so the same config showed
the proxy verbatim in serialized state and <REDACTED> in TS. Retype it
as SecretStr across S3 and the 14 aliases, and reveal it at the one
place botocore needs the raw value.
.pnpm-store/ is a 910M local pnpm store at the repo root that showed up as
untracked in every git status. Root-anchored so it only matches the top-level
directory.
* Fully close the od and numfmt review findings
od declared a single positional FILE while the command takes FILE..., so
the spec contract in spec/ misdescribed it and the generic only worked
because the parser falls back to the last positional kind for extra
operands. Declare rest=PATH, matching the eleven sibling FILE... commands.
numfmt scaled output did not match GNU: it printed 1K where GNU prints
1.0k. Implement GNU's actual rule in both languages: round away from zero
keeping one decimal only below 10, re-check the unit afterwards since
rounding can push a value back over the base (999.4 -> 1000 -> 1.0k), and
render through a half-even printf so an unscaled 2.5 prints 2. SI spells
kilo lowercase while IEC stays uppercase, and --from now accepts
lowercase suffixes.
Both implementations were diffed against coreutils 9.8 across 37 cases
covering si, iec, iec-i and none, with zero differences.
* Derive the SI display units from the shared suffix order
The two tables differed only in kilo, so the eleven units were spelled out
twice per language. Deriving the SI one keeps them from drifting apart.
* fix(numfmt): snap FP noise before ceil so 1000.00000001 keeps 1.1k
The TS roundAwayFromZero subtracted a fixed 1e-9 before ceil, which also
erased any genuine offset smaller than the epsilon: numfmt --to=si
1000.00000001 collapsed to 1.0k while GNU 9.8 and the Python Decimal path
both keep it at 1.1k. Snap to 15 significant digits instead, dropping only
sub-ulp binary noise.
* test(integ): warm-read provision cases; drop redundant nextcloud standalone
Add unix/provision/warm.json asserting cache-served re-reads
(net=0 cache=N ops=1 hits=1) across the 18 cache-backed backends,
on both hosts via the existing provision harness.
Remove nextcloud.py + truth_nextcloud.txt (and its CI step): the
declarative battery already runs 1644 shared cases against nextcloud
on py+ts, a strict superset of the standalone's stdout-only surface.
* test(integ): shared consistency + warm-read battery cases (python host)
Add unix/consistency/basic.json: LAZY serves stale, ALWAYS revalidates,
via a per-case consistency policy + generic out-of-band mutate (a shadow
workspace writing to the same service). Verified on s3, s3-prefix,
onedrive, dropbox, sharepoint(+prefix), ssh; hf lazy-only (non-versioned).
Harness: open_consistency() shares one service across read+shadow ws;
run_scenario() executes read/mutate/read steps. TS host mirror next.
* test(integ): skip consistency scenarios on typescript host (py-first)
The consistency cases carry no top-level command; the typescript runner
skips them until the openConsistency/mutate mirror lands. Add the
consistency/scenario optional fields to the TS Case type.
* test(integ): delete redundant onedrive.py standalone
Every onedrive.py section is now covered by the shared battery:
generic commands (battery), streaming/index (unix/provision/basic.json
net=/hits=), warm-read (unix/provision/warm.json), consistency
(unix/consistency/basic.json), exit/timeout (unix/grep/q.json +
safeguard). onedrive runs the full battery on the python host via its
self-starting fake Graph, so the standalone adds nothing.
* test(integ): delete s3.py/s3.ts standalones
s3 command coverage lives in the shared battery (s3, s3-prefix targets
+ provision/warm/consistency). The standalones' remaining unique bits
(cost-estimation, bounded-drain, S3-compat variants) are unit-test
material, dropped per scope. Removes their CI steps + truth files.
* ci(integ): split integ-shared into parallel py + ts jobs
The single integ-shared job ran main.py then main.ts sequentially
(~24 min). Split into integ-shared-py and integ-shared-ts so the two
host batteries run in parallel with isolated fake-server state. The
shared toolchain + fake-server fleet moves into a composite action
(.github/actions/integ-battery-setup) so setup stays defined once.
Gate needs both; path filters re-trigger on the action.
* ci(integ): skip TS package builds in the python battery job
integ-shared-py only needs node to run the tsx fake servers
(gws_server.ts, slack.ts), which import no @struktoai/mirage-* package;
main.py is pure python. Gate the 3 mirage package builds behind a
build-packages input (default true) and pass false from the py job, so
its setup drops ~3 build steps. The ts job keeps the builds.
* refactor(integ): hoist consistency callbacks out of nested closures
Address codex P1: open_consistency/open_target used nested mutate and
cleanup closures, violating the no-nested-functions rule. Replace with
module-level mutate_write/teardown_target bound via functools.partial.
* Fully close the od and numfmt review findings
od declared a single positional FILE while the command takes FILE..., so
the spec contract in spec/ misdescribed it and the generic only worked
because the parser falls back to the last positional kind for extra
operands. Declare rest=PATH, matching the eleven sibling FILE... commands.
numfmt scaled output did not match GNU: it printed 1K where GNU prints
1.0k. Implement GNU's actual rule in both languages: round away from zero
keeping one decimal only below 10, re-check the unit afterwards since
rounding can push a value back over the base (999.4 -> 1000 -> 1.0k), and
render through a half-even printf so an unscaled 2.5 prints 2. SI spells
kilo lowercase while IEC stays uppercase, and --from now accepts
lowercase suffixes.
Both implementations were diffed against coreutils 9.8 across 37 cases
covering si, iec, iec-i and none, with zero differences.
* Derive the SI display units from the shared suffix order
The two tables differed only in kilo, so the eleven units were spelled out
twice per language. Deriving the SI one keeps them from drifting apart.
* feat: coreutils Tier 3 — df capacity command + statfs model (#609)
Add `df` with an honest per-provider capacity model. mirage spans
heterogeneous backends, so `df` reports real numbers only where a backend
can truthfully provide them and a literal `-` everywhere else — never a
fabricated total (the #609 mandate). py ≡ ts.
- CapacityResult + 4-state CapacityState (quota | elastic | na | unknown).
Resource.statfs() defaults to unknown; the disk backend overrides it with
a real statvfs / fs.statfs -> quota.
- `df` is a registry-aware executor handler: it enumerates the target
mounts (all, or the mount containing each FILE), reads each mount's prefix
+ backend kind + statfs, and renders one GNU-byte-exact table. Flags:
-h/-H human sizes, -k/-B block size, -T type, -i inodes, -P POSIX, -a and
--sync accepted no-ops. Use% = ceil(used/(used+avail)*100).
- Honest rendering: quota -> real numbers; elastic/na/unknown -> `-` in
every numeric column. GNU never emits `-` for blocks (it only sees real
filesystems) — the deliberate, more-truthful divergence for a VFS.
Tests: py + ts df unit tests (incl. real-disk statfs), integ df/basic.json
verified byte-identical on both hosts. New df spec JSON (python + ts
browser/node); spec-count tests bumped.
Scoped as PR-A. Deferred to PR-B: promoting statfs to a dispatch op +
wiring the FUSE statfs off the current hardcoded fake (untestable here
without macFUSE), per-provider quota APIs (Drive/Dropbox/OneDrive/Gmail/
Box), and the -l/--output/-t/-x flags.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address df review — missing FILE, -B0, last-wins, symlink (#609)
Closes the four P2 findings from the Codex review of #621, GNU-9.7-pinned,
py ≡ ts:
- Verify each FILE operand exists: df stats a path below its mount root (the
root is the filesystem itself, always present) and errors `df: <op>: No
such file or directory` (exit 1) instead of reporting the mount. Cost is
O(explicit operands), so it stays within df's lightweight design.
- Reject a zero/non-positive -B block size: `df: invalid -B argument '0'`
(exit 1) rather than ZeroDivisionError (py) / Infinity (ts); the message
now matches GNU's short-form.
- Honor the last size-format flag: -h/-H/-k/-B are mutually overriding, so an
ordered scan of the leading option run picks whichever appears last
(df -h -B1M -> 1M-blocks, df -B1M -h -> Size).
- Follow symlinked FILE operands: both dispatchers pass link-resolved
operands to the capacity handler, so df /link reports the target's mount.
Also fixes a TypeScript-only parity bug in the shared splitValueFlags: its
bad-option pre-scan rejected attached valued arguments (-B1M, touch -t<stamp>)
as unknown flags; it now validates inline like the Python helper.
Tests: py test_capacity +4, ts capacity.test +4, integ df +5 (both hosts
3614/3614 byte-identical); metadata suite unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(commands): auto-inject --version for registered commands
Every command registered through command() now accepts --version the
same way it already accepts --help: the flag is injected into the
spec, short-circuits the handler, and prints
"<name> (Mirage) <package version>" to stdout with exit 0.
Covers the Tier 3 --version sweep from #610, including tsort, without
per-command boilerplate. --version-sort on sort stays a separate flag.
* fix(commands): serve --version before mount dispatch and cross-mount routing
The handler-level short-circuit was unreachable on executor paths that
reject or bypass handlers: rm --version on a read-only mount returned the
read-only refusal, and cat --version /ram/a /disk/b took the cross-mount
branch and parsed against the shared spec, which carries no injected
--version, so it failed as an unknown option.
version_request / versionRequest answers from the package before mount
permission checks and cross-mount routing, once the command is known to
be registered and to carry the injected option. It stops at the -- marker
so a literal --version operand stays an operand. The mount-level write
guard also skips its read-only refusal for --help / --version, which
never touch the backend.
Test handlers move to module scope, and integ gains version cases under
unix/ and crossmount/ (including the read-only mount).
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
The audit job fails on every branch: pypdf 6.13.3 picked up four
advisories (CVE-2026-59935/59936/59937/59938), all fixed by 6.14.2.
pypdf is transitive via browser-use. Mirage reads PDFs through
pypdfium2 and never imports pypdf, so nothing in mirage/ or tests/
references it.
uv audit exits 1 on vulnerabilities only, so this alone turns the job
green. The remaining line, socksio is archived, is an adverse project
status that does not affect the exit code; socksio comes from
httpx[socks] via openhands-sdk and cannot be upgraded away.
* Add Tier 2 coreutils support
* Fix Tier 2 CI checks
* Fix PR review findings and CI failures for Tier 2 coreutils
CI:
- regenerate spec/ JSON for the new and changed command specs
- bump the TypeScript BUILTIN_SPECS count to 92
- wire the truncate op for redis, ssh and gridfs on the Python side
(TypeScript already had all three), fixing native truncate on redis
Review findings, applied to both languages:
- checksum: parse BSD/--tag lines under --check, and honor -z for --tag
- join: bounds-check --check-order and --header field reads so blank or
short lines no longer raise IndexError, and honor --nocheck-order
- mktemp: move -q into the generic, narrow it to OSError, and suppress
the creation diagnostic instead of emitting it (GNU); add -q to the
TypeScript generic, which had none
- split: keep the empty record a second trailing separator terminates
- paste: decode -d escapes with a single left-to-right scan so \0 means
the empty delimiter and \\ stays a literal backslash
- executor: merge positional and path-flag scopes instead of dropping
the flag scopes whenever a positional path is present, so a path-flag
target on another mount is routed and rejected rather than misrouted
- mkdir: apply -m/--mode through set_attrs; parse_mode moves to a leaf
util shared by the command layer and the executor builtins
- numfmt: convert only the first field of each stdin record and copy the
rest through verbatim
- od: concatenate every FILE operand as one input
- nl: write an empty line in place of each logical-page delimiter, and
pad a one-character -d with ':' as GNU does
GNU behavior for nl, join, paste and numfmt was pinned against coreutils
9.8. The nl integ expectations encoded the old delimiter handling and are
corrected; six integ cases are added for findings that had no coverage.
* Regenerate truncate spec after wiring redis, ssh and gridfs
* feat(coreutils): env builtin + tr -C/-t + tee --output-error (py+ts)
* fix: address codex review + spec drift for env/tr/tee
- env: lone '-' implies -i; reject -0 with a command (exit 125)
- update direct tr/tee generic callers to flags= (test_phase_m/q)
- regenerate tr/tee spec snapshots
* fix(tee): honor --output-error on write failure (passthrough stdout + exit 1)
Shared write_output helper (py+ts): on a write error, tee still copies
stdin to stdout, prints a GNU-style diagnostic, and exits non-zero.
With a single output sink the four --output-error modes collapse to
this. Used by the generic and the bespoke s3/gridfs tee.
* style: yapf-format merge-resolved command_dispatch imports
* feat: coreutils Tier 2 — chgrp, realpath -e, ln -r, rm safety flags (#609)
Fill the genuine gaps in #609 Tier 2 (medium priority), scoped to what
fits mirage's async control-plane model. Most of Tier 2 (touch -c/-r/-d,
ln -s, readlink -f/-e/-m, chmod, chown) already lives in the
executor-builtins layer; this adds the missing pieces, py ≡ ts:
- chgrp (new): group ownership via the namespace-overlay set_attrs path
(the group half of chown; -h no-deref, rejects -R).
- realpath -e: fix a doubled ENOENT message — raise a plain error rather
than a filesystem error type so format_fs_error emits it verbatim,
matching the TS realpath throw.
- ln -r/--relative: store the target relative to the link's directory;
-n/-T accepted as no-ops (a namespace link name is never dereferenced
nor treated as a directory to descend into).
- rm -i/-I/--preserve-root/--no-preserve-root/--one-file-system: accepted
no-ops — mirage is a non-interactive control plane (no prompt), mount
roots and / are structurally unremovable so the root failsafe is always
on and cannot be disabled, and recursion never crosses a mount boundary
so --one-file-system already matches the default.
Complete integ coverage (integ/unix/{chgrp,ln,realpath,rm}) verified on
ram/disk/opfs across both hosts, plus py/ts unit tests. Regenerated the
rm spec JSON for all three surfaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address PR review — cwd-resolve chmod/chown/chgrp operands, canonicalize ln -r
Two P2 review findings on #617:
- chmod/chown/chgrp had no CommandSpec, so a relative FILE operand with a
non-root cwd (`cd /data/sub && chmod 600 f.txt`) reached the handler as a
bare string and resolved against the mount root, not the session cwd
(ENOENT / wrong path). Add specs for all three (leading MODE/OWNER/GROUP
as TEXT, FILE operands as PATH) so the classifier cwd-resolves them; the
handlers keep self-parsing their flags (declared options are not stripped
from the operand list). Fixes chgrp and the pre-existing chmod/chown bug.
- ln -r computed the relative target lexically, storing an aliased path
instead of the canonical one (`ln -sr /alias/a/f /b/link` -> ../alias/a/f
vs GNU ../real/a/f). Canonicalize the target and link directory through
the symlink table before computing the relative path.
Spec count 86 -> 89: update the Python key-set test and the TS builtins
count test; regenerate the new chmod/chown/chgrp spec JSON. New integ
regression cases (chgrp relative-cwd, ln -r via a symlinked dir), verified
ram + disk on both hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat: getopts shell builtin (py + ts parity)
Implement the POSIX getopts builtin in both runtimes with GNU
bash semantics pinned against a real-bash oracle:
- one option parsed per call; OPTIND/OPTARG tracked in env with a
hidden per-word scan offset on the session
- combined flags (-abc), attached (-afoo) and separate (-a foo) args
- silent mode (leading : in optstring): invalid -> name=?, OPTARG=char;
missing arg -> name=:, OPTARG=char; no stderr
- non-silent mode: illegal option / option requires an argument on
stderr, name=?
- stops on non-option word / - / -- / exhaustion; OPTIND=1 reparses
- positional-args source when no explicit args; usage error exits 2
Wired through the ShellBuiltin enum, builtin dispatch, and provision
node builtin set in both languages.
* test(integ): getopts battery under bash/builtin
Nine getopts cases across the full backend battery: option loop,
combined flags, attached optarg, OPTIND-driven shift remainder,
silent vs non-silent missing-arg and invalid-option, and the
too-few-operands usage error. Each resets OPTIND=1 first, since the
harness reuses one session per target and OPTIND persists across
cases (correct bash semantics). Verified against real mirage output;
all nine green on the ram target.
* fix(getopts): address getopts state-machine review gaps (py+ts)
- reset the hidden scan cursor on any OPTIND assignment/unset (even to
the same value) and guard against a stale offset past the current word
(no more IndexError); treat a nonpositive OPTIND as a restart at arg 1
- scan the active function frame's positional parameters inside a shell
function, mirroring shift
- honor OPTERR=0 to suppress diagnostics in non-silent mode
- validate the destination name (identifier + readonly) at assignment,
matching bash: OPTIND still advances but the write fails with status 1
- isolate the cursor across subshells and propagate it (plus
positional_args in python) through Session.fork()
Adds unit + e2e regression coverage in both languages.
Replace single-field `-k N` with the complete GNU sort key spec,
mirrored across Python and TypeScript:
- Field ranges and character offsets: -k F1[.C1][opts][,F2[.C2][opts]]
- Field-N-to-EOL default (`-k2` spans field 2 to end of line, not just
field 2 -- the largest prior silent divergence)
- Multiple -k keys applied in order (spec `-k` is now repeatable)
- Per-key modifiers n/g/h/V/M/f/r/b; any per-key modifier (incl. -b)
makes a key ignore all global ordering options (GNU key_init rule),
while globals still drive the last-resort whole-line compare
- Last-resort whole-line comparison, disabled by -s
- GNU field model: leading blanks belong to the following field; -b
skips them; -t is a clean split
- Invalid field 0 -> exit 2 with GNU-matching stderr
Pinned against GNU coreutils 9.7. New cross-language
conformance/cases/sort.json validates py/ts parity.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(commands): add md5sum, sha1sum, sha384sum, sha512sum (py+ts)
Closes part of #597 (agent-facing GNU Coreutils coverage gaps): adds four
GNU checksum commands in both the Python and TypeScript implementations,
built on a shared parameterized checksum core that sha256sum now also uses.
- Python: mirage/commands/builtin/generic/checksum.py (shared hashsum core);
thin per-command generics + generic_bind builders; specs in hashing.py;
crossmount Cmd enum + FANOUT_COMMANDS; provision FILE_READ_COMMANDS.
- TypeScript: core/commands/builtin/generic/checksum.ts (checksumGeneric);
sha1Hex/sha384Hex/sha512Hex via WebCrypto in utils/hash.ts (md5Hex reused);
builders + registry + spec + crossmount + provision; browser opfs variants.
- Behavior mirrors sha256sum exactly: `hash file` format, multi-operand,
stdin (`-`), --check (-c) round-trip, GNU missing-operand errors, and
cross-mount fanout.
Scope note: sha224sum, b2sum and cksum are deferred to a follow-up because
WebCrypto lacks SHA-224/BLAKE2b/POSIX-CRC, so the TS side needs vetted
pure-JS implementations that warrant separate review.
Verification: Python unit tests (tests/commands/native/test_checksums.py)
and TS unit tests (native_checksums.test.ts) over ram/disk/s3(/redis);
integ facet cases under integ/unix/{md5sum,sha1sum,sha384sum,sha512sum};
digests oracled against real GNU coreutils. Both integ hosts: 1598/0 on ram.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(commands): add rmdir and unlink (py+ts)
Continues #597 (Wave 1): two GNU filesystem commands in both runtimes.
- rmdir: removes empty directories; -v verbose; GNU errors for
non-empty ("Directory not empty"), missing ("No such file or
directory"), non-dir ("Not a directory"), and missing-operand
(with the Try-'--help' hint). Requires a new Operation.RMDIR
(Python) mirroring MKDIR; TS uses the ops.rmdir undefined-guard
like rm. (-p and --ignore-fail-on-non-empty are a follow-up.)
- unlink: removes a single file; exactly one operand (missing- and
extra-operand usage errors with the hint); EISDIR ("Is a
directory") and ENOENT messages match GNU.
- Wiring per language: generic_bind builders + registry, fs_mutate /
builtins.ts specs, crossmount Cmd enum + FANOUT_COMMANDS,
NO_FOLLOW_COMMANDS (unlink; rmdir already present), browser opfs
variants, regenerated specs. Spec-count guards bumped to 86.
Verification: py + ts unit tests over ram/disk/s3(/redis); integ
facet cases under integ/unix/{rmdir,unlink} (success + every error
shape), oracled against real GNU coreutils. Both integ hosts:
1609/0 on ram. TS core suite 4770/0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(shell): add `:` and `type` builtins (py+ts)
Continues #597 (Wave 1): two Bash builtins in both runtimes.
- `:` (colon): the null command — always exits 0, ignores its
arguments; redirections and ${var:=default} side effects still
apply (mirrors `true`). Added to the ShellBuiltin enum + dispatch.
- `type [-afptP] name ...`: reuses `command -V`'s resolver, so every
mirage-native runnable name reports as a shell builtin (no external
paths — the same documented divergence `command` already makes).
`-t` prints the classification word; `-p`/`-P` print a path (always
empty for pathless builtins); `-f` skips the function table; `-a`
lists all locations (one here). Uses type's all-found exit rule
(0 only when every name resolves); a missing name warns
`type: NAME: not found` on stderr (mirrors `command`'s prefix, a
divergence from bash's `bash: type:` framing) unless a word-only
mode is active. `-t nope` is silent, exit 1.
Verification: py + ts unit tests (handleType in test_command),
oracled against real bash for the shapes that match mirage's model;
integ facet cases under integ/bash/{colon,type}. Both integ hosts:
1622/0 on ram. TS core suite green (the only failures were
Pyodide-init timeouts unrelated to this change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(ssh): bump pinned commands() count 111 -> 117 for Wave 1 additions
The SSH resource provisions the six new Wave 1 commands (md5sum,
sha1sum, sha384sum, sha512sum, rmdir, unlink), so SSHResource.commands()
is now 117. The SSH_COMMANDS-derived assertion tracked automatically;
the hardcoded literal did not and was the sole node-suite failure on CI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(deps): bump pyasn1 0.6.3 -> 0.6.4 to clear DoS advisories
Pre-existing transitive-dependency advisory unrelated to the Wave 1
commands (this repo adds no dependencies). `uv audit` flagged pyasn1
0.6.3 for two DoS CVEs, both fixed in 0.6.4:
- GHSA-hm4w-wwcw-mr6r / CVE-2026-59886 (uncontrolled resource
consumption decoding REAL values)
- GHSA-8ppf-4f7h-5ppj / CVE-2026-59885 (quadratic complexity in
OBJECT IDENTIFIER / RELATIVE-OID processing)
Minimal `uv lock --upgrade-package pyasn1`: only pyasn1 moves, lock
schema unchanged. `uv audit` now exits 0 (verified locally). The
remaining "socksio is archived" line is an adverse-status warning,
not a vulnerability, and does not fail the audit; socksio is a
transitive httpx[socks] dep of an AI SDK, not declared here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(resource): update count + write-tag guards for Wave 1 commands
Four Python resource guards pinned expectations that the Wave 1 command
additions legitimately change; they were the Python `test` job's only
failures (the TS analog was fixed in a082a207):
- discord/slack provider counts: +4 for the new checksum reads
(md5sum/sha1sum/sha384sum/sha512sum), which provision to every
file-read resource. rmdir/unlink do NOT appear here -- they are gated
by Operation.RMDIR/UNLINK, which chat resources lack.
- nextcloud/s3 write-tag sets: add rmdir + unlink, which those real
filesystem backends provision (they have the ops) and correctly tag
write=True.
onedrive/generic guards are non-exhaustive and needed no change.
Verified locally: 1751 passed (remaining failures are the pre-existing
macOS-lacks-tac and mem0-not-installed env gaps).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Restructure the JSON-runner integ corpus from a flat
`category/command.json` layout into a three-level
`category/name/facet.json` tree, so cases are navigable by the flag or
feature they exercise (e.g. `unix/grep/n.json`, `unix/find/name.json`,
`unix/meta/chmod.json`, `resources/dify/search.json`).
Faceting rules:
- unix command files split by the primary flag on the named command,
with reserved `basic` / `error` / `operand` buckets, and populate the
previously-dormant `flags` field on those cases (370 cases).
- feature files plus bash/crossmount/resources split by the case-id's
distinguishing segment (`meta_chmod_*` -> chmod, `col_parquet_*` ->
parquet).
- 9 high-cardinality flat files (provision, printf, gmail, ...) collapse
to a single `basic.json` instead of exploding into 40+ singletons.
- uppercase-only flags get a `.upper` suffix so filenames stay unique on
case-insensitive filesystems (`grep -C` -> `C.upper.json`).
Discovery now walks recursively: `rglob` in the Python harness and the
existing `walkFiles` helper in the TypeScript harness.
177 flat files -> 701 facet files, 1898 cases. The migration preserves
each case's `seq`, so the global execution order and every case body are
byte-identical to before (only the additive `flags` field differs);
verified 0 content mismatches vs the flat tree, unchanged id order, and
the Python runner green (1582/1582 on ram). Adds
runners/tools/refile_facets.py, the deterministic, re-runnable migration
tool (with a case-collision guard).
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
TS `handleBash` called `executeFn(script, { sessionId })` without
forwarding `stdin`, so `echo hi | bash -c 'cat'` dropped the piped input.
Python's `handle_bash` already passes `stdin`, so this was a py/ts parity gap.
The `ExecuteFn` / `ExecuteStringFn` stdin plumbing and the inner `executeFn`
threading already landed upstream with the `command` builtin (#595); the same
path now also carries `bash -c`. This closes the remaining gap in `handleBash`:
- `handleBash` forwards `stdin` to `executeFn`, and nulls it after consuming
stdin as the script in the `-s` path (parity with Python's `stdin = None`)
so `bash -s` does not re-feed the script bytes as the script's own stdin.
- Add integ case `ctl_bash_c_stdin` (`echo hi | bash -c 'cat'` -> `hi`) and a
matching core vitest; Python already had the mirror test.
Verified: core vitest (4752 passed, 0 failures) on top of current upstream/main.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
PR 8 expansion trio (items 1 and 3; item 2 deferred). Both languages,
GNU-pinned via docker bash 5.2.
- ${v:$o}: tree-sitter-bash truncates a $-spelled substring offset
(zero-width `}` + stray siblings), so mirage emitted corrupted text
like `hello2}`. Detect the truncated parse and raise
`bash: ${v}: bad substitution` exit 2. bash accepts the form; the
supported spellings ${v:o} and ${v:$((o))} still work.
- Quoted "${a[@]...}" now word-splits into N words for slices
("${a[@]:1:2}"), per-element ops ("${a[@]/x/y}"), and indices
("${!a[@]}"), matching bash; single-word forms (${a[*]}, ${#a[@]})
unchanged. New expand_array_at/is_multiword_at in the shared
variable module, wired through the parts multi-word splat path.
Item 2 (non-fatal `bash: a: bad array subscript` warning for OOB
negative array reads) is deferred: the empty-expansion behavior is
already GNU-correct and a warning channel across every expansion call
site is disproportionate for a cosmetic line.
11 unit cases each language + 9 integ cases (array.json seq
603000-603005, param.json seq 603100-603102); both hosts pass 1575/1575
on ram.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Restore the single-host api_base override (removed in #584): when set, every Google service host and the OAuth token endpoint derive from it, falling back to the real googleapis.com hosts, so a workspace can point Google backends at a fake or self-hosted server. Mirrored in Python and TypeScript (interface, redacted interface, zod schema, normalize rename) with tests.
Co-authored-by: khj809 <onsealeatang@gmail.com>
* feat(watch): resource change watching + Nextcloud source (#450)
Add a mount-scoped watch API so an external agent service can react to
file changes instead of polling and diffing snapshots.
Core seam (inert without the watcher):
- types: ChangeKind, OverflowPolicy, ResourceChange, Delta
- Workspace.watch() delegation + attach_watch_runtime() slot via a local
WatchDelegate protocol, so core never imports the watch package
mirage/watch package:
- DeltaHook / SupportsChanges / WatchQueue / WatchRuntime protocols
- RAMWatchQueue: per-path coalescing + pluggable overflow policy
- generic ListingDeltaHook (snapshot diff, works on any backend)
- Watcher + enable_watch: per-(mount, root) ref-counted pollers,
bounded mailboxes, invalidate-before-deliver, nudge()
Nextcloud source:
- recursive WebDAV walk (ETag detector, mtime|size fallback)
- NextcloudResource.delta_hook()
Tests: 33 unit tests (queue, poller, watcher, workspace delegation,
nextcloud hook). Integ: integ/watch.py driven by watch_nextcloud.json,
mutating through a separate accessor so the verify read proves
invalidate-before-deliver; wired into test_integ.yml.
* test(watch): self-asserting JSON battery in integ/watch/
Replace the truth-file integ with a self-checking runner. Case files
live in integ/watch/ (one JSON per resource, nextcloud.json today); the
runner asserts expected event kind/path and the post-event verify read,
exiting nonzero on any mismatch. External mutations go through a separate
opendal operator, which generalizes to other backends (s3, gcs) for
future case files. Drops truth_watch.txt and integ/resources/watch_nextcloud.json.
* refactor(watch): package layout + broaden integ checks
- constants.py: DEFAULT_MAX_PENDING, DEFAULT_POLL_INTERVAL, DIR_DETECTOR
(both defaults are ours, not from #450; documented as such)
- queue/ package: WatchQueue protocol + OverflowPolicy + errors in
queue/base.py, RAMWatchQueue in queue/ram.py
- source.py: Source/Subscriber runtime dataclasses out of watcher.py
- drop the default_queue helper; RAMWatchQueue is itself a valid
QueueFactory (root-only ctor), so it is the default directly
- keep the change model (ChangeKind/ResourceChange/Delta) in types.py as
a shared leaf: the producer (Workspace.watch) and the watch machinery
both need it, so a neutral leaf avoids a workspace<->watch cycle
without a TYPE_CHECKING workaround
- integ: nextcloud.json cases now assert cat/head/ls/grep after
create/update/delete, proving reads are fresh post-invalidation (grep
no longer matches a deleted file; old content gone after update)
* feat(watch): precise push via notify() + sample webhook server
Push (the issue's webhook-first ask) without Mirage hosting a server:
the consumer's own service receives the Nextcloud webhook and injects a
precise change.
- Watcher.notify(change): invalidate-before-deliver then deliver to
matching subscribers, no poll; path reframed to the owning mount.
nudge() stays for imprecise doorbells (no path). Pull remains the
reconciliation baseline; duplicates from overlap are coalesced.
- WatchRuntime protocol gains notify().
- integ/watch/webhook_server.py: the reference aiohttp receiver a
consumer copies, mapping NodeCreated/Written/Deleted/Renamed payloads
to ResourceChange and calling notify(). Mirage hosts no HTTP.
- integ runner now runs each case file in BOTH pull (poll+nudge) and
push (webhook->notify) mode, proving both transports yield identical
events and fresh reads.
- 4 notify() unit tests (deliver without active poll, invalidate-before-
deliver ordering, delete->unlink, path reframing).
* refactor(watch): exceptions to errors.py, OverflowPolicy to types.py
- mirage/watch/errors.py holds QueueOverflowError + QueueClosed
(matches the repo's per-package errors.py convention)
- OverflowPolicy StrEnum moves to mirage/types.py next to ChangeKind,
the shared-leaf home for watch enums
- queue/base.py is now just the WatchQueue protocol + QueueFactory
- re-exports in queue/__init__ keep the public import paths stable
* refactor(watch): mirage runs no poller; notify-driven runtime + fingerprint naming
Core decision: mirage ships detection *utilities*, not a detection
*loop*. The consumer owns the loop (their server already exists); the
integ demonstrates it in ~10 lines.
- Watcher is now purely notify-driven: no poll tasks, no Source, no
nudge. notify() = invalidate-before-deliver then fan out to matching
subscribers. watch() works on ANY mount (no capability required to
subscribe); SupportsChanges/delta_hook only power pull detection.
- ListingDeltaHook + NextcloudWalk stay as importable pull utilities;
the consumer's poller is: pull(root, checkpoint) -> notify each
change -> keep checkpoint (ConsumerPoller in integ/watch/run.py).
- Rename detector/version -> fingerprint, aligning with mirage's
existing FileStat.fingerprint concept; shared default_fingerprint()
(native ETag, else mtime|size) moves to the generic poller and the
Nextcloud walk uses it.
- Exceptions live in watch/errors.py; OverflowPolicy in types.py.
- pop(): comment documenting the condition-wait loop (no busy spin).
- integ pull mode = the DIY poller demo, now fully deterministic (pump
once per case, no sleeps); push mode unchanged and now provably
webhook-only (there is no poller to interfere).
- DEFAULT_POLL_INTERVAL deleted (no loop to configure).
* refactor(watch): poller.py -> delta.py, stat_fingerprint in utils, glob + multi-path watch scopes
* feat(watch): push-mode example, scope/warm-cache integ battery, ancestor-chain invalidation
* feat(watch): workspace.watch accepts str/pattern/list at the facade; runtime stays PathSpec-only
* refactor(watch): FileEvent/FileChangeKind/FileMetadata event shape, UTC datetime timestamp
* feat(watch): lazy runtime attach; watch package fully decoupled via WatchMount/WatchRegistry protocols
* test(watch): pin middle-wildcard glob scope; document glob depth vs recursive
* feat(watch): GNU glob depth semantics for watch scopes (slashless = entries, trailing slash = dir subtrees)
* feat(watch): drop recursive flag; root shape defines depth (literal = subtree, /* = shallow, */ = dir subtrees)
* docs(watch): Watch page under Python tab (scopes, event model, push/pull, queues)
* feat(watch): MOVE evicts both sides; integ covers all scope shapes + all FileChangeKinds; watch matrix docs
Implements the bash `command` builtin in Python and TypeScript: runs a
name while bypassing any shadowing shell function, forwards pipe stdin
so `... | command cat` filters upstream output, and supports -v/-V
introspection (bash's any-found exit rule) plus a no-op -p. mirage has
no PATH, so every runnable non-function reports as "shell builtin".
Registered in the ShellBuiltin enum and both command_dispatch ladders;
stdin threaded through ExecuteFn/executeFn. Adds 23 Python + 23 TS unit
tests and integ/bash/command.json battery cases.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* test(integ): PR5 unix write-family + long-tail + archive flag coverage
Adds 70 GNU-pinned integ cases (seq 601000-601351) for previously
untested unix flags, plus the product fixes the coverage surfaced
(Python + TypeScript, with unit tests):
- mv -v: emit GNU "renamed 'src' -> 'dst'" (was cp's arrow form)
- rm -rv/-dv: depth-first per-entry verbose + "removed directory" for
dirs; rm -d on non-empty now GNU "cannot remove ...: Directory not
empty" (shared utils/verbose helper; generic + s3 + gridfs; gridfs
now collects errors instead of aborting)
- mktemp /path/tmpl.XXXX: honor an explicit path template's own
directory (was prepending /tmp)
- split -l: TS read the wrong flag key (AMBIGUOUS_NAMES renames -l to
args_l) so it never segmented by lines
- seq -w: was a value option that ate the start operand; now boolean,
with width = widest formatted value (handles negatives)
Cases: cp/mv/ln/rm/tee/mkdir/touch/mktemp(descoped)/split/csplit (A),
cat/sort/diff/cmp/jq/uniq/nl/cut/join/comm/tail/base64/seq/sha256sum/
realpath/expand/unexpand/date (B), gzip/gunzip/zip/unzip/expr/shuf (C).
Green on py ram/disk + ts ram (1520/1520); PR5-A also py s3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(shell): join -o/-e field mapping + jq -s coverage (py & ts)
join -o FILENUM.FIELD indexed the key-stripped rest, so `1.2` pointed at
field 3 and paired `-o` output came out empty; -e was also ignored in the
-o path. Map against the full 1-based field list (key included) and fill
missing fields with the -e value. Mirrored py & ts; native/unit tests
updated to GNU-correct output.
jq -s already slurps jsonl/stdin correctly; added integ coverage.
7 new integ cases (join -o/-e/-v/-1-2, jq -s); green py ram/disk + ts ram.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(shell): GNU-format tree output + integ coverage (py & ts)
tree emitted unicode box-drawing, no root line, no summary, and -L was
off by one. Rewrite to match GNU (C-locale ASCII):
- root path as the first line
- ASCII branches (|--, `--, | ) matching the docker oracle
- blank line + "N directories, M files" summary (singular when 1; the
file count is omitted under -d); the root counts as a directory once
it has a listed entry (an empty root reports 0)
- -L N shows N levels (was N+1)
- a missing root prints "<path> [error opening dir]" and exits 2
Mirrored py & ts; updated the unicode/format unit tests across generic,
ram, disk, databricks; refreshed the tree.json / nl.json cases and added
5 GNU-pinned integ cases (basic, -L, -d, -a, -I). Green py ram/disk +
ts ram (1530/1530).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(shell): readlink -f/-e/-m canonicalize symlinks (py & ts)
readlink -f/-e/-m fell through to the mount command, which only
normpath'd the operand and never resolved the link. Route every
readlink through the namespace handler and, for -f/-e/-m, resolve the
full symlink chain (namespace.follow) and normalize — GNU realpath-
style — so `readlink -f link` yields the canonical target (chains and
relative targets included). Plain readlink still prints the raw target.
Mirrored py & ts; 6 GNU-pinned integ cases (readlink.json: -f, chain,
relative, -e, -m, -f -n). Green py ram/disk + ts ram (1536/1536).
Documented divergence: -f/-e do not yet fail on a missing intermediate
component (existence checks need async in the sync handler).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(integ): PR6 crossmount redirect/archive/read-routing cases
6 GNU-pinned crossmount cases spanning /data + /data2: redirect > and
>> across mounts, gzip-then-cp-then-zcat archive across mounts, and
jq/tree/xxd routing to the second mount. Green py ram/disk + ts ram
(1542/1542).
Documented divergences (mirage rejects cross-mount, GNU allows; not
added as cases): split/csplit output prefix on the other mount and tar
across mounts ("paths span multiple mounts ... not supported");
find | xargs -I ( -I unsupported by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: update backend tree tests for GNU-format output
The tree rewrite (ee617c6d) adds a root line + summary, so the discord
and slack backend tree tests (exact line count/position) and two
generic-test docstrings needed refreshing to the ASCII format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: yapf-format native/test_join.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(integ): regenerate s3/onedrive/dify truth for GNU tree format
The tree rewrite (ee617c6d) changed the legacy per-backend scripts'
tree output; regenerate the truth fixtures I can run locally
(s3 via moto, onedrive in-process, dify). Only the tree blocks
changed (root line + ASCII branches + summary).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(integ): deprecate legacy tree cases; JSON harness owns tree
The GNU tree rewrite (ee617c6d) changed the tree output pinned by the
legacy per-backend scripts. Rather than maintain those brittle golden
fixtures, drop the redundant `tree` case from the legacy scripts — the
declarative JSON harness now owns tree coverage (integ/unix/tree.json:
basic + -L/-d/-a/-I/-P, plus crossmount routing).
- Removed the tree/tree_table case from chroma, qdrant, lancedb, dify,
notion, s3, onedrive (py + ts where present) and the matching
`=== *tree* ===` sections from their truth files (s3/onedrive/dify
regenerated; chroma/qdrant/lancedb/notion/s3_ts section-pruned).
- Kept each script's unique backend coverage (e.g. s3.py's
gcs/minio/seaweedfs endpoints + columnar rendering) — only the
generic, now-duplicated tree case is dropped.
- Added tree -P to integ/unix/tree.json to complete the flag set.
Green py ram/disk + ts ram (1543/1543).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: resolve CI failures from the tree rewrite + seq spec drift
Full-matrix CI surfaced the tree rewrite's remaining golden/behavior
impact:
- rm -rv on dropbox/box/gdrive printed `removed directory '.../sub/'`
(object stores return dir paths with a trailing slash) — strip it in
removal_lines (py & ts); GNU never prints a trailing slash.
- Regenerate the seq -w spec snapshots (spec/{python,typescript}) —
value_kind text -> none (the "Spec drift" gate).
- Update the tree goldens the rewrite touched: conformance tree_subdir,
slack_tree_day, and the integ/truth redis replay line, to GNU format.
- redis/test_tree.py -L assertion + native_join.test.ts -o assertion
(mirror the fixes already applied to ram/disk / the python native
join test).
- Curate hf/hf-prefix out of the empty-dir tree cases (object stores
don't persist empty directories) and ssh out of cp -R tree cases
(recursive-copy limitation).
Green py ram/disk + ts ram (1543); py/ts unit + conformance green
locally (redis/service targets verified in CI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(opfs): rm -v/-dv verbose + -d empty-check, mkdir -v (browser)
The opfs (browser) rm/mkdir had bespoke implementations that ignored
-v entirely and skipped rm -d's empty-check (giving `rm: Directory not
empty` without the operand). Mirror the fix already applied to the
generic/s3/gridfs rm: per-entry verbose via cpWalk + removalLines,
`removed directory` for dirs, GNU `cannot remove '<path>': Directory
not empty`, and mkdir -v. Green on ts --target opfs (1552/1552).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(shell): GNU control-flow/set-option/builtin semantics + 89 integ cases
set -u/-x/-f enforcement, set -e GNU list semantics, pipeline negation,
break/continue levels, readonly fatal assignment, declare/typeset scoping,
GNU select, base#value arithmetic, subshell background jobs (private job
table), per-line stdin buffer reset (read no longer poisoned after the
first line), GNU read IFS-whitespace trimming. Both languages + mirrored
unit tests. 89 docker-pinned integ cases across bash/control.json,
setopt.json, builtin.json, jobs.json, read.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): run the new control/setopt/builtin/jobs/read cases on the box target
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(shell): complete test/[/[[ condition evaluator + return/fg fixes, 58 integ cases
The old evaluator silently returned false for every operator outside
-z/-n/-f/-d, string =/!=, and the six numeric comparisons: [ -e f ],
[ -s f ], [[ x == pat* ]], [[ x =~ re ]], [[ a && b ]], and -a/-o
combinators all evaluated false with no error, and [ -f dir ] was true.
Same silently-wrong class as brace expansion — core agent idioms
steering control flow the wrong way.
Both languages, GNU-pinned via a 107-probe docker battery (bash 5.2,
106/107 identical) plus 58 exact-command pins:
- test/[: bash arity rules (1-4 args) + recursive-descent -o/-a/!/()
parser beyond four; file ops -e -f -d -s -r -w -x -L -h via a
stat->readdir kind probe and the namespace symlink table; GNU
diagnostics ("integer expression expected", "binary operator
expected", "too many arguments") at exit 2; unquoted expansions
word-split; -1 negative numbers reassembled from the parser's
unary split; ERROR-recovery nodes flattened or truncated at ;.
- [[ ]]: structured tree evaluator — glob ==/!= with quoted-RHS
literal semantics, =~ ERE with BASH_REMATCH capture groups, && ||
! ( ), string < >, arithmetic coercion for -eq family
([[ n -eq 3 ]], [[ 1+1 -eq 2 ]]), no word splitting; a bad operator
is a parse error killing the whole line (exit 2) like bash.
- Operators mirage cannot answer (-p -S -b -c -g -k -u -O -G -N -t,
-nt -ot -ef) fail loudly as capability errors instead of silent
false.
- return: outside function/source now errors-and-continues ($?=2,
GNU message) instead of leaking a stray signal; bare return
propagates $? (function bodies track last_exit_code per statement);
return inside a sourced file stops the source with its status.
- fg: split from wait — "fg: current: no such job" exit 1 with no
jobs; with a job prints the resumed command and adopts its exit.
Coverage: 58 new integ cases (test.json +46, builtin.json +8 incl.
disown/complete/compgen pins and return semantics, setopt.json +2
set +u / bare set, jobs.json +2 fg) across all 19 targets; +89 py and
+82 TS mirrored unit tests. Harness 1364/1364 on python-ram,
python-disk, and typescript-ram with zero curation.
Documented divergences: fg follows interactive bash (non-interactive
GNU always fails "no job control"); [[ error text is
"mirage: conditional binary operator expected"; error messages drop
bash's "line N:" prefix (existing convention). New parser limitations:
[ a \< b ] is an ERROR node (use [[ a < b ]]); a malformed [ a = ]
swallows trailing statements into the test node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: retrigger after transient GitHub API outage in paths-filter
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(shell): test -e/-s truth on prefix stores and size-unknown backends
CI on the cloud-backend targets exposed two condition-evaluator bugs:
- The readdir existence probe treated an empty listing as an existing
directory, but prefix stores (s3, gridfs, hf, nextcloud) list a
missing path as [] instead of raising, so -e/-d/-s answered true for
nonexistent paths. The probe now demands a non-empty listing; those
stores cannot hold an empty directory, so nothing real is lost.
- -s counted a size-unknown stat as non-empty, but dropbox/gdrive/box
stat freshly written empty files as size-unknown, so [ -s empty ]
answered true (also breaking the pre-existing redirect_* cases that
assert via test ! -s). A size-unknown regular file is now read to
answer exactly; the prefetch TTL cache keeps repeat tests cheap.
Adds stub-dispatch regression tests for both backend shapes in Python
and TypeScript.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(shell): -d stub in builtins.test lists an entry (empty listing now means missing)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(shell): split the condition evaluator into a package (py+ts)
condition.py / condition.ts had grown to ~500 lines mixing node types,
operator tables, two evaluators, and the builtin wrapper. Both languages
now mirror the same package layout:
- types: CondNode dataclasses/union, CondError, CondContext
- constants: operator classification sets + INT_COMPARATORS table
(operator.eq/... in Python, bigint lambdas in TS) replacing the
-eq/-ne/-lt if-chains in both evaluators
- operators: operand scoping, path-kind probe, unary/binary application
- flat: test/[ arity rules + the >4-arg recursive-descent parser
- tree: the [[ ]] expression-tree evaluator
- handle: the handle_test/handleTest builtin entry point
Public surface is unchanged: the package __init__ (TS index.ts)
re-exports what importers used before, so call sites are untouched
apart from the TS relative-path bump. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(slack): Prisma-backed fake Slack Web API + integ battery target
Mock the Slack Web API for the integ battery so the Slack grep/rg search
push-down runs live, mirroring the Dropbox search work.
- One shared fake server (integ/server/slack.ts), Prisma + SQLite, seeded
from integ/fixtures/slack/v1.json. Both hosts call it over HTTP (SLACK_URL),
so every API response is byte-identical across hosts.
- search.messages / search.files mocked (in:#channel operators, xoxp- token
gating) so grep -r / rg on a channel push down through the search API.
- New `slack` battery target + 17 declarative cases (resources/slack.json):
structure, content, search push-down, users walk, exit codes.
- Backend endpoint override so it can reach the fake: SlackConfig.base_url
(python) / baseUrl + NodeSlackTransport (ts); browser uses proxyUrl.
- CI: integ-shared starts the fake for both hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(slack): scale the integ fixture to a realistic startup workspace
Replace the hand-authored slack fixture with a deterministic generator
(integ/fixtures/slack/generate.py, SEED=1337) producing the "Kestrel"
startup arc: ~1182 messages across 10 channels + 10 DMs over Nov 2025-Mar
2026, 16 humans (5 departing) + a bot, and 15 attachments including real
tiny pdf/pptx/xlsx/png blobs. Messages carry threads (thread_ts) and
reactions; templated phrase banks and workday-spread timestamps keep it
from reading as synthetic (449 distinct texts, activity 09:00-18:00). A
build-time assert fails loudly if a planted story beat falls outside its
channel's active window.
Extend the Prisma schema and fake server (thread_ts, reactions,
content_path) so chat.jsonl renders threads/reactions and file blobs
download byte-exact, then re-capture resources/slack.json (39 cases,
byte-identical on the Python and TypeScript hosts).
Also format thrown command errors identically on both hosts (GNU
"prog: message"): the Python executor now routes any thrown command error
through format_fs_error like the TS runOnMount, so a throwing case
(slack_add_reaction_missing -> message_not_found) can live in the shared
battery.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(errors): do not double the command-name prefix on self-formatted errors
format_fs_error / formatFsError add the GNU "prog: " prefix to a thrown
command error, but many generic commands (uniq, sed, gzip, gunzip, zcat)
raise a message that already carries it, producing "uniq: uniq: invalid
count". Emit such a message verbatim and only prefix when it is absent
(e.g. the slack API errors that motivated the shared error path). Mirrored
py+ts with regression tests.
Fixes the integ-data battery failures (sed_count_zero, uniq_invalid_count,
gunzip/zcat/gzip_d_no_input); ram battery is 1217/1217 on both hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(printf): full GNU printf builtin with py/ts float parity
The `printf` shell builtin ignored positional arguments — `printf '%s\n' c
a b` printed the literal format instead of cycling the format over the
args. Beyond that first fix, extend it to full GNU coverage, mirrored
byte-for-byte across the Python and TypeScript implementations.
Now supported (pinned against GNU bash's builtin via docker
debian:stable-slim):
- Conversions: %s %c %b %q, %d %i %o %u %x %X, %f %F %e %E %g %G %a %A, %%
- Flags: - + 0 # and space; numeric or * dynamic width/precision
- Format reuse over excess args; missing arg -> "" (%s) / 0 (%d)
- Escapes interpreted once in the same scan, incl. \u/\U unicode
- Integers wrap at 64 bits (%x -1 -> ffffffffffffffff), 0x/octal input,
# alternate form, and the leading-quote code-point form (%d '"A' -> 65)
- Floats: round-half-to-even, correct exponent formatting, inf/nan
- %q full bash shell-quoting incl. $'...' for control/non-ASCII bytes
Parity: bash's builtin formats floats in long double while Python and JS
use IEEE double, so exact bash parity is impossible at high precision and
for %a. The invariant enforced instead is mirage-py == mirage-ts,
byte-identical, verified across ~4900 fuzzed cases plus the curated
corpus; both match bash wherever double suffices. %a and high-precision
output reflect IEEE double (documented in the code).
Tests: comprehensive parametrized unit tests on both sides (75 py, 153
ts) and 53 double-safe integ cases in integ/bash/printf.json, green on
both hosts (1138 passed / 0 failed each).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: re-trigger workflows (paths-filter hit a transient GitHub API 503)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Every gws invocation in the Google examples used a hyphenated name that is
not registered, so each one exited 127. The examples print only stdout and
never check the exit code, so the failures rendered as empty sections.
The registry has two naming families:
generated passthrough gws <service> <resource> <method>
hand-written helpers gws <service> +<verb>
So gws-docs-documents-create becomes gws docs documents create, and
gws-sheets-read becomes gws sheets +read, not gws sheets read. Python and
TypeScript register identical names.
man reads only argv[0], so man now passes the multiword name quoted.
* feat(google): per-service API base-URL overrides for GoogleConfig
A single api_base collapses every Google service onto one host, so docs and slides both derive {api_base}/v1 and collide — a fake server cannot tell them apart. Add optional per-service base overrides (token_url, drive/drive_upload/docs/sheets/slides/gmail_api_base) that take precedence over api_base, falling back to the api_base derivation then the real Google host. token_url is the full token endpoint URL used as-is. Mirrored in Python and TypeScript (interface, redacted interface, zod schema, normalize rename map) with tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* restore unexpected change
* fix(test): build token manager from a real GoogleConfig in test_get_attachment
The SimpleNamespace stub only defined api_base, so it raised AttributeError once gmail_base() started reading the per-service gmail_api_base field (config.gmail_api_base). Use a real GoogleConfig — which carries every per-service field by construction — so the test double stays in sync with the config type (matching how test_client.py builds its token managers).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(google): keep API endpoints fixed
---------
Co-authored-by: khj809 <onsealeatang@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fake Trello (REST) and Linear (GraphQL) servers under integ/, self-seeding
from fixtures, driving trello + linear integ targets on both hosts.
Virtualize the two backends as nested gws-style command families modeled on
the mheap/trello-cli and Finesssee/linear-cli CLIs: linear/trello <noun>
<verb> read commands (list/get/show/members/current/...) emitting normalized
JSON, with the existing write commands folded under the same nested names.
Add Linear documents end to end: a documents/ VFS folder per team plus the
linear document list/get commands (fixture, fake server, queries, client,
normalize, readdir, read, stat), both Python and TypeScript.
py 56 unit + 65 integ, ts 86 unit + 65 integ, parity 0 mismatches.
* feat(box): serve special files raw + grep/rg search push-down
Drop JSON rendering for .boxnote/.boxcanvas/.gdoc/.gsheet/.gslides;
serve every Box item as its raw bytes under its real name. The render
only earns its keep where a structured API can round-trip edits
(Google/gws); Box has no such API, so the projection was one-way and
lossy, and it hid the bytes the search index actually matches on.
Add a grep/rg content-search push-down (mirrors dropbox): a
content_search config flag routes recursive literal scans through Box
/2.0/search scoped by ancestor_folder_ids to narrow candidate files,
then re-scans them locally so output stays byte-identical to a full
walk. Full-walk fallback on API error, truncation, empty result, or
non-folder scope.
* test(box): add command-level search push-down coverage
Port dropbox's narrow/grep/rg search tests to box (both languages) plus
the TS core search test, closing the parity gap: box only had core-level
narrow_paths coverage, so nothing exercised the fire-vs-bypass gate that
decides when grep/rg must skip Box search and read files directly (-v,
-c, regex patterns, --type/--glob, non-folder scope, empty/truncated
results, binary-extension drops). Re-export keepVisible now that rg.test
imports it.
* feat(nextcloud): server-side find via Files Search API with partial predicate push and fallback
- Implement best-effort gather of representable predicates (name, type, size, mtime) so that -name/-type/-mtime/-size queries (and mixes with unsupported like -path) use the indexed SEARCH endpoint.
- Unsupported expressions (brackets, -path only, -o/-not, -empty, non-d/f types) transparently fall back to recursive WebDAV scan.
- Reuse existing search.py pagination/scope/XML; keep exact client post-filter with keep(), depths, start-path semantics.
- Add test for mixed supported+unsupported case.
- Matches proposal: dramatically fewer requests for common cases while preserving behavior.
* fix(nextcloud): include the missing search.py implementation
The previous commit added the find.py wrapper and decision logic but
omitted python/mirage/core/nextcloud/search.py (the actual WebDAV
SEARCH / SearchDAV client that talks to Nextcloud's indexed Files
Search API) and its tests.
Without this file the new server-side path cannot be exercised:
- imports in find.py would fail in some environments
- search_files always effectively unavailable
- all server-side tests and the feature itself broken in CI
This completes the implementation that was in-progress on the branch.
* feat(nextcloud): support Or/Not in server-side find predicates; tests pass
* revert: remove added comments and docstrings per instructions
* fix(nextcloud): make Files Search pushdown safe
* refactor(nextcloud): clean up Files Search
* test(nextcloud): keep search coverage declarative
* refactor(nextcloud): structure find execution
* feat(nextcloud): mirror Files Search in TypeScript
* ci(nextcloud): run TypeScript JSON integration
* chore: apply repository formatting
* refactor(nextcloud): split Files Search modules
* ci(nextcloud): keep TypeScript integ in TS job
* chore(spec): record Nextcloud resource support
* fix(ci): repair TypeScript checks
* perf(nextcloud): avoid zero-depth find scans
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* test(integ): gmail + email targets, attachment coverage, mail backend fixes
* feat(mail): gws gmail official syntax + email commands with himalaya aliases
Gmail follows the official Google Workspace CLI: gws gmail +send/+reply/
+reply-all/+forward/+triage/+read helpers plus a raw gws gmail users ...
Discovery passthrough. Drops the bespoke delete command (use trash).
Email exposes canonical email list/read/send/reply/forward commands, with
himalaya-style aliases dispatching to the same handlers via a shared
add_aliases/withAliases helper. reply-all folds into email reply --all.
Prompts, docs, integ cases, and unit tests updated on both hosts.
* refactor(mail): himalaya email commands (drop email aliases)
Email commands use the himalaya CLI grammar as their canonical names:
himalaya envelope list, himalaya message read/send/reply/forward. The
earlier email* names and the add_aliases/withAliases alias helper are
removed. Docs note that send/reply/forward require a write-mode mount.
* feat(dropbox): grep/rg search push-down via files/search_v2
Recursive grep/rg on a Dropbox mount previously downloaded every file.
With the new content_search / contentSearch config knob (off by default:
full-text search is plan-gated and its index lags recent writes), both
commands now ask /2/files/search_v2 which files contain the pattern's
literal and download only those candidates. Output stays exactly
GNU/ripgrep because the local scan still decides every match:
- Core searchFiles pages search_v2 + search/continue_v2, dedups across
pages, and reports the 10,000-match ceiling; narrowPaths maps
path_lower/path_display back to mount paths under root_path, sorts
narrowed candidates into sorted-readdir walk order, and rebases
raw_path onto the scope spelling so labels match a walk's.
- narrow_scope gates the push-down: literal (or regex-required-literal)
single patterns only, recursive scans only, directory operands only,
and never for output modes that must see every file (grep -v/-c,
rg -v/--type/--glob). Empty/failed/truncated searches fall back to
the full walk; binary-extension candidates are dropped to mirror the
walk's skip; rg prunes hidden candidates segment-wise and forces
walk-style filename labels.
- Both wrappers keep the factory's default_provision so cost estimates
are unchanged.
- fix(grep, python): grep -Rl with a file operand now stats first and
scans the file instead of readdir-walking it (GNU + TS parity);
narrowed candidates exercised this path.
- Fakes gain search_v2 + search/continue_v2 (case-insensitive substring
over names and content — a superset of real token matching, which is
what narrowing needs) with cursor paging; battery adapters enable the
knob, so all dropbox/dropbox-root grep/rg cases now exercise the
push-down live: 988/988 per target on both hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(dropbox): CI fixes — TS grep -Rl file operands, rg -I labels, formatting
The local TS battery ran against a stale mirage-node dist (only core was
rebuilt), so the node DropboxResource never forwarded contentSearch and
search narrowing was silently inactive on the TS host; CI's fresh build
activated it and exposed two latent TS bugs python had already fixed:
- grepFilesOnly walked file operands under -r (readdir on a narrowed
file candidate -> ENOENT warnings, empty output). It now stats first
and takes the single-file scan for file operands (GNU + python
grep_files_only parity); regression tests in both languages.
- rg's plain-line path delegates to grepGeneric, whose single-file body
honors -H over -h, so the wrapper's forced label defeated -I
suppression. Both wrappers now skip forcing H when -I is set;
regression tests in both languages.
Also formats the new files pre-commit never saw locally (they were
untracked when it ran; --all-files only covers git ls-files) and settles
two formatter fights: the provision calls are hoisted onto a shared
dropboxResolveGlob const so Prettier/ESLint agree, and the
test_grep_helper import gets grep_helper via a module import so
yapf/isort converge.
Verified with fresh core+node dists: dropbox/dropbox-root 988/988 on
both hosts (narrowing live), ram/disk 2170/0, core vitest green,
pre-commit converges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(box): python box backend + box integ target on both hosts
Port the TS-only box backend to python (core/box client, api, readdir,
read, stat, du; boxnote/boxcanvas filetype renderers; accessor, commands,
ops, resource, registry). Add a box integ target seeded over the real Box
wire shapes for both hosts, backed by the fake Box server.
Mount a subfolder by folder id (root_folder_id / rootFolderId): Box ids are
stable across renames and there is no path-resolution API.
Fixes surfaced by box, applied to both languages:
- readdir on a file id raises ENOTDIR so ls falls back to the file entry
- root stat fetches the folder's own metadata so find -mtime keeps the root
- walkFind emits the start point for a mount-root operand and supports -empty
- python _find_walk rebases output to the operand spelling
- makeJqProvision threads the index into stat
* feat(box): full write surface + enroll box integ target on write cases
Add the write family to the box backend in both languages: write/tee
(upload + overwrite as a new version), mkdir, unlink, rmdir, rm -r, mv,
cp, touch, truncate. Box is id-native with no path addressing, so write
ops resolve path->id with a fresh folder-listing walk (core/box/resolve)
and keep the read caches coherent through the invalidation context.
Fake Box server gains delete/update/copy endpoints. The box integ target
is now writable and enrolled on the same cases as hf on both hosts.
Notes for the id-native backend (unlike Graph's path addressing):
- stat falls back to the resolver walk when no index is threaded, so the
rm/mv/cp builders and provision estimation resolve ids without one
- mkdir -p invalidates every created level's parent listing, not just the
final target's, so a cached ancestor listing can't hide new folders
- TS cp requires a find op; box gets one backed by walkFind over a scratch
index it populates as it descends
* test(box): expand box integ enrollment to s3 parity; cp -r merge + du provision
Enroll box in the ~170 remaining cases s3 runs (grep/rg/zgrep/exit/errop/
find shell + read coverage), skipping only ORC (unsupported in TS). Both
hosts now run 1101 cases.
Two fixes this surfaced:
- cp -r into an existing directory now merges instead of replacing: box
copy recurses per child into an existing destination folder rather than
clearing and re-copying it server-side, matching GNU cp -r.
- the bespoke box du command now uses metadata_provision so its estimate
is exact (Python; TS already gets it from the factory default).
* fix(box): type box truncate download to satisfy tsc --noEmit
* chore(box): regenerate command specs with box in resources
* docs(box): python resource page, write surface, matrix and nav updates
* chore(spec): add dropbox to TS specs after merge
* feat(gridfs): MongoDB GridFS backend with native revisions and server-side find
Python + TypeScript GridFS resource mirroring the s3 backend: filenames as
slash-separated keys, zero-byte trailing-slash marker docs for directories,
full read/write command surface via the generic factories.
- Writes upload a new revision and keep old ones; reads/listings resolve
latest-per-filename via one fs.files aggregation; snapshot-pinned reads
fetch old revisions by file _id; mv retags filenames server-side so
history moves with the file; rm deletes all revisions.
- find pushes -name/-iname/-type/-size into the fs.files query (anchored
basename regex, marker-shape test, length bounds); every pushed condition
is a superset of GNU semantics and the shared keep() pass stays
authoritative, so behavior matches the other backends exactly.
- Integ: gridfs + gridfs-prefix targets enrolled everywhere s3/s3-prefix
run; 1101/1101 on both hosts; mongo:8 service added to integ-shared.
- Docs: setup + python resource + typescript setup pages, matrix row,
nav entries, README backend lists.
* chore(spec): regenerate command specs with gridfs
* ci: consolidate install matrix and python runtime jobs
* test(integ): enroll gridfs targets in brace expansion cases
* feat(shell): brace expansion in both hosts
Comma lists, numeric/char ranges with step and zero-pad, nesting,
prefix/suffix stitching, and GNU literal fallbacks, expanded at the
word level before classification (bash's brace-before-parameter
ordering via inert atoms for already-expanded children). New
integ/bash/brace.json (25 docker-pinned cases), unit tests mirrored
in Python and TypeScript.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): run dropbox/gdrive targets in the expansion suites
The dropbox, dropbox-root (#558) and gdrive, gdrive-folder (#549)
targets landed while the expansion coverage PRs were in flight; add
them to every case those PRs authored (param, array, var, glob, case,
find additions, brace). Read-only cases run all 16 targets; write
cases drop the read-only hf pair. Verified locally: dropbox and
dropbox-root 1117/1117 (both hosts), gdrive 1040/1040 (both hosts),
gdrive-folder 1009/1009.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The TS generic find walk diverged from Python's walk_find on the search
root and -empty, and both hosts' expression parsers mishandled repeated
-mtime predicates. This made eight battery cases (find_d, find_empty,
find_maxdepth_zero, find_mindepth_zero, find_name_start, find_not_name,
find_size_lt, find_mtime) fail on walk-based backends (dropbox today),
so they were excluded from the dropbox/dropbox-root targets.
- walkFind now emits the search root at depth 0 even when it is the
mount root, mirroring walk_find: `find <mount>` lists the mount,
`-maxdepth 0` prints just the root, and `-name` can match the root's
own basename.
- walkFind computes per-entry emptiness (readdir for dirs, size-0 stat
for files) when the tree contains -empty, via a new treeHasEmpty
mirroring Python's tree_has_empty.
- Both parsers now flatten repeated -mtime predicates to the union of
their windows instead of last-wins, so the GNU tautology
`-mtime +0 -o -mtime -1` imposes no bounds. Last-wins made find_mtime
fail on the Python host too.
- Re-add the eight case ids to the dropbox/dropbox-root battery
targets; both hosts pass 996/996 on each.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): mount a subfolder as the resource root via rootPath
Add an optional rootPath to DropboxResource (node + browser): the
configured folder becomes the mount root, scoping every command and
FUSE/VFS op to that subtree. The root is normalized once on the
accessor ('' for account root, /seg/seg otherwise, '..' rejected) and
prefixed in the two core path builders (readdir's dropboxPathFromKey,
read/stream's dropboxPathFromVirtual), so stat/du/find/glob inherit it
for free. Config dicts accept snake_case root_path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): dropbox mock-server integ covering subfolder mounts
Follow the notion.ts pattern (read-only API backend, in-process fake
server, truth-file diff): integ/dropbox.ts spins up a fake Dropbox API
(oauth2/token, files/list_folder, files/download) and runs the shell
battery against an account-root mount, a rootPath subfolder mount, and
a slash-variant spelling of the same root. The subfolder mount's
request log is asserted to contain no API path outside the configured
root, and sibling/parent-escape reads fail with ENOENT.
Requires a test-only endpoint override in DropboxConfig (one origin
serving oauth + api + content), mirroring the hf fake-hub knob.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): dropbox + dropbox-prefix targets in the declarative battery
Replace the standalone dropbox.ts/truth-file approach with proper
battery targets, mirroring s3/s3-prefix and hf/hf-prefix: 'dropbox'
mounts three isolated fake accounts, 'dropbox-prefix' mounts three
rootPath subfolders of one shared account. The TS adapter self-hosts
a fake Dropbox API per account (oauth2/token, files/list_folder,
files/download via the DropboxConfig endpoint override) and seeds
fixtures into it directly — dropbox is a read-only backend, so the
workspace mkdir/tee seeding path cannot run; Open.seeded lets an
adapter opt out of harness seeding.
The targets join the 632 read-only cases (1264 case-runs, all green,
stable across reruns): write-command cases, cases reading state
written by earlier write cases, and history-reading cases (coupled to
the exact per-target command sequence) stay excluded. Python hosts
skip the target (TS-only backend).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): Python port of the Dropbox backend + battery targets on both hosts
Port the TS dropbox backend to Python, closing the runtime gap:
core/dropbox (_client with token manager + endpoint override, api,
readdir, read/stream, stat), DropboxAccessor with root_path
normalization, read-only CommandIO (du uses the generic readdir+stat
walk), generated ops/commands, DropboxResource + registry entry.
Two behavior fixes shared with TS along the way:
- readdir maps list_folder 409 (path/not_found, path/not_folder) to
ENOENT on both sides, so ls on a file operand falls back to its
stat-the-operand path and missing dirs report No such file or
directory instead of a raw API error.
- the Python find builder's walk fallback now rebases results onto the
operand as typed (rebase_raw), matching generic_find and GNU display
semantics; previously cd /data && find disptree printed absolute
paths on walk-fallback backends.
Integ: dropbox-prefix target renamed to dropbox-root with a root mount
field (dropbox's knob is rootPath, matching ssh/nextcloud's root
convention, not s3's keyPrefix). Both dropbox targets now run on the
python host too: aiohttp fake (integ/server/dropbox_server.py),
DropboxService with out-of-band seeding, and the same seeded opt-out
in the python runner. Case set re-converged empirically across BOTH
hosts: 632 read-only cases per target, 1264 case-runs per host, green
and stable; history-reading and write-dependent cases stay excluded.
Docs: docs/python/resource/dropbox.mdx, resource matrix row gains the
Python link, TS page cross-links, examples/python/dropbox/dropbox.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(integ): move the TS fake Dropbox into integ/server/dropbox.ts
Fake backends live under integ/server/ (hf_server.py, onedrive_server.py,
dropbox_server.py); the TS fake was the odd one out under runners/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: formatter churn from merge + fix stale fake path reference
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(spec): regenerate command specs with the dropbox resource
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): read/write backend — upload, mkdir, rm, mv, cp on both hosts
Wire the Dropbox mutation endpoints (upload, create_folder_v2,
delete_v2, move_v2, copy_v2) into full write support in both
languages: write/create/mkdir/unlink/rmdir/rm_r/rename/copy cores with
cache+ancestor invalidation, EEXIST/ENOENT/EISDIR/ENOTDIR mapping, and
emulated truncate in the ops factory. Key semantics:
- rmdir guards ENOTEMPTY before delete_v2 (which deletes folders
RECURSIVELY — the s3 data-loss lesson); rm -r maps to one call.
- rename/copy replace an existing destination FILE like GNU mv/cp
(delete + retry on to/conflict); folder conflicts propagate. No
dir_copy is wired so cp -r merges into existing dirs file-by-file.
- mkdir owns GNU semantics (EEXIST without -p, ENOENT on missing
parent) since create_folder_v2 auto-creates parents; the mount root
is always-exists (the API rejects the empty path — an unguarded
mkdir -p on the mount root used to plant a corrupt '' folder in the
fakes that listed itself as its own child and looped find forever).
- stat/read gain API-truthful index-less fallbacks (get_metadata /
direct download) so unlink/rmdir classification, the wired TS find
(required by the cp planner), and emulated truncate work.
- single-call uploads cap at ~150 MB (documented; no upload sessions).
TS drops its provisionOverrides (python's defaults match the shared
battery expectations) and both hosts gain the filetype command set
(cat_parquet & co), closing the col_* exclusions.
Battery: the out-of-band seeding opt-out is deleted — dropbox seeds
through the workspace mkdir/tee like every writable backend, which
exercises the write path itself. Fakes gain explicit folder objects,
the write endpoints, real-clock upload stamps (find -mtime), an 8 MiB
aiohttp body cap (example.h5 is ~1.02 MiB), and loud empty-path
guards. dropbox/dropbox-root now run 988 of the 997 s3-covered cases
per target (1976 case-runs per host, green and stable on both): write,
history, and meta chains included; only 8 TS-walkFind predicate gaps
(find_d/find_empty/…, tracked separately) and mtime_dir_not_epoch
(Dropbox folders carry no mtime) stay excluded.
Docs flip to read/write; specs regenerated (51 files gain the dropbox
write/filetype command rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: formatter churn on dropbox write files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(integ): bash parameter-expansion, array, and quoting cases
100 new GNU-pinned cases (debian:stable-slim bash 5.2): bash/param.json
(52), bash/array.json (31), bash/var.json +9, bash/glob.json +8, seq
500600-500907, all 12 targets (read-only).
Product fixes that rode along (Python & TS + unit tests):
- expand_braces refactor: parse-to-struct + injected expand_child
callback, so operand args expand (${x:-$d}, ${x:-$(cmd)}, ${f%$ext});
pattern operands keep glob chars literal via a dollar-ref pass.
- ${var/pat/rep} is now glob-based longest-match, with /# and /%
anchors and case-mod patterns (${v^^[el]}).
- Substring offsets and array subscripts evaluate as arithmetic
(${v:1+1}, ${a[i+1]}) through the shared evaluator.
- Arrays: negative indices, ${a[@]:o:l} slices, ${!a[@]}, per-element
ops, bare $a, a+=()/v+=/a[i]= forms, literals word-split cmdsubs and
resolve globs; bad negative subscript aborts the line like bash.
- Shared fnmatch treats [^...] as negation (bash/glibc): new
mirage/utils/fnmatch.py swept over 10 call sites; TS classBody fix.
- Adjacent ${} expansions in dquotes no longer lose the separating
space (it hides in the next node's leading token).
Deliberate divergences documented in the PR plan: ${v:$o} spelling
(tree-sitter grammar limit; ${v:o} works), quoted "${a[@]:1:2}" word
count, dense arrays, IFS-unaware "$*", no stderr warning on negative
OOB reads.
Both hosts pass all 1081 JSON cases (ram+disk python, ram TS).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): pin case/find caret-class negation and local indirection
Close the probe-only gaps: case patterns and find -name share the
fnmatch [^...] fix, and ${!ref} resolves function locals. 4 cases
(case_caret_class, case_bang_class, find_name_caret_class,
param_indirect_local), GNU-pinned via docker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(gdrive): read/write Drive backend, gws CLI, fake Workspace server, integ targets
Bring Google Drive up to the s3 backend model and add Google Workspace
coverage to the declarative integ harness.
gdrive core (both langs):
- write core: write/mkdir/unlink/rmdir/rm/rename/copy/exists/truncate/create
plus a resolve module that resolves paths by direct Drive queries (Drive is
id-addressed and allows duplicate sibling names, so mutations use server
state, not the read cache). GNU semantics: EEXIST, mv-overwrite with
ENOTEMPTY on non-empty dirs, cp -r merge. Native gdoc/gsheet/gslide files
render as API JSON and reject raw writes.
- folder_id subfolder scoping (the s3 key_prefix analog) and a gdrive-folder
target.
- versions via the Drive Revisions API, wired into snapshot pin/restore.
- find and du backend ops (tree walker mirroring msgraph).
gws commands (both langs):
- passthrough factory generating drive/docs/sheets/slides API commands that
emit raw API-resource JSON.
- a top-level gws dispatcher accepting official CLI syntax
(gws docs documents get, gws sheets +read, ...).
Config seam:
- GoogleConfig gains api_base (point backends at a fake server) and folder_id.
Integ:
- fake Google Workspace server (Drive v3 + Docs v1 + Sheets v4 + Slides v1,
in-memory, /reset, deterministic clock, google-apps MIME auto-linking).
- gdrive and gdrive-folder join the universal battery; a google case group
asserts byte-exact API-resource renders.
- CI starts the fake server before the shared batteries.
* refactor(gws): fold create/batchUpdate into the method table, drop the dispatcher
The hand-written gws create/batchUpdate commands were passthroughs, and the
`gws` dispatcher was a second routing layer on top of the command dispatcher.
- Add docs/sheets/slides create + batchUpdate as rows in the gws method
table; delete the 6 bespoke command files and their now-orphaned
core create/update helpers.
- Delete the gws dispatcher. Every method is a first-class command the
parser resolves directly: `gws-docs-documents-get`, `gws-drive-files-list`,
`gws-docs-write`. No `gws docs documents get` space syntax.
- Tidy run_gws_method with a verb-to-caller map.
- Rewrite the integ google cases to the hyphenated command names.
* feat(gws): restore official space syntax via a thin spec-based gws command
The gws method table stays hyphenated (gws-docs-documents-get), but a
single thin gws command reuses the parser: its spec declares the flags,
its body reconstructs the target name from operands and looks it up in
the method table or helper map. No per-method routing dicts.
* fix(integ): gws fake server returns a generic 500 message
CodeQL flagged js/stack-trace-exposure: the caught exception message
flowed into the HTTP response. Log it server-side and return a generic
'internal error', matching how the real Google API responds.
* feat(gws): resolve nested command names, drop the gws dispatch shim
Command resolution now matches the longest registered command name over
the leading words (git-style), so official Google-CLI syntax like
`gws docs documents get` and `gws docs +write` resolves natively. The
mount registry keeps a first-token index of multi-word names; expand_argv
consumes the matched prefix as the command and the rest as operands, so
route/spec/dispatch key on the joined name unchanged.
This deletes dispatch.py/.ts (the re-parsing shim), their tests, the
resource registration hook, and the hyphenated gws-* command names.
Prompts and the integ google cases already used the space syntax.
* chore(spec): regen command specs, gdrive joins the write-command resource lists
* test(integ): cover files get/copy and sheets +write; fix +write ignoring api_base
The new g_sheet_write case caught a real bug: gws sheets +write built its
URL from the hardcoded SHEETS_API_BASE constant in both languages, so it
bypassed the configured api_base and hit the real Google endpoint. It now
goes through sheets_base(token_manager) like every other gws command.
* feat(gws): unify sheets +write onto the ergonomic helper flags
+write now takes --spreadsheet/--range/--values/--json-values like
+read and +append, instead of the raw API --params/--json shape. The
PUT moved into core gsheets write (update_values) next to append_values
so the command wrapper is wiring only. Prompts and the integ case use
the new surface.
* test(spec): use a nested command name in the help renderer fixture
* feat(gdrive): folder_id scoping works inside Shared Drives, denied writes are EACCES
A folder_id mount root may sit inside a Shared Drive (or be a Shared
Drive id): root_context resolves the root's driveId once via files.get,
memoizes it on the accessor, and threads it through every resolution,
listing, and mkdir walk, so scoped shared-drive mounts list and mutate
correctly (they previously listed empty).
Drive access is per-item, so a write mount can still hold read-only
items: mutations map an API 403 to EACCES on the operand via
eacces_on_denied / eaccesOnDenied instead of leaking a raw HTTP error.
The fake Workspace server gains shared drives (create/list, driveId
inheritance, real files.list visibility rules), the harness gains a
drive field on gws mounts, and a gdrive-shared integ target covers the
scoped shared-drive flow end to end in both languages.
* refactor(gws): gather the declarative tables in methods
GWS_API_SPEC, SERVICE_BASES, and SERVICE_RESOURCES move out of the
factory into methods.py/.ts next to GWS_METHODS, so the methods module
holds all the per-service data and the factory is behavior only.
* chore(gws): drop factory imports left behind by the methods move
69 new JSON cases (seq 500200-500517) covering grep -q/-H/-A/-B/-C/-R/--colour,
find -print/-print0/-delete/-ls and -a/-and/-o/-or with escaped parens,
rg -v/-n/-w/-F/-o/-H/-I/-m/-A/-B/-C/--hidden/--type/--glob/--color, and all
12 previously untested zgrep flags. Expectations pinned against GNU
(debian:stable-slim) and ripgrep 14.1.1 via docker; cases self-provision
/data/{g4,f4,r4,z4} subtrees so the persistent-workspace suites stay isolated.
Product fixes flushed out by the pins (Python + TS + unit tests):
- rg -A/-B/-C parsed but never rendered context; single-file search now
rides the shared grep context renderer (dir walks still skip context,
mirroring grep's documented -H divergence)
- grep -q leaked output on multi-file/-R/-l paths; quiet everywhere with
exit 0/1 by match (incl. -qc all-zero -> 1)
- rg -I kept per-file labels in directory walks; labels dropped (-l keeps paths)
- find -delete could not remove directories emptied by the deepest-first
pass (dispatcher called bare rm); now rm -d
- zgrep -H on stdin now labels "(standard input)"
Still descoped with doc notes: find -prune (loud unknown-predicate error),
grep -I content sniffing, rg dir-walk context.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refactor: rename per-backend op manifests to io.py/io.ts exporting IO
Align the Python and TypeScript manifest naming: each backend's
CommandIO table now lives in commands/builtin/<b>/io.py (exporting IO
and resolve_glob) and commands/builtin/<b>/io.ts (exporting X_IO),
replacing the confusing ops.py/ops.ts (OPS / X_CMD_OPS / RESOLVE_GLOB)
names that collided with the real ops layer. TS op-set exports unify on
the X_OPS suffix (dropping the mixed _VFS_OPS variants), and both
generic ops factories document the two deliberate knob asymmetries
(forwardIndex, filetype_read bool vs filetypeRead list). No behavior
change; golden ops inventories are untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: rename leftover _X_CMD_OPS local aliases to _IO
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(hf): add hf-buckets integ targets + fake Xet hub, fix backend gaps
Add hf and hf-prefix targets to the declarative integ harness (both python
and typescript-node hosts) backed by a new fake Hugging Face hub + Xet CAS
server that real opendal drives end to end.
Backend fixes surfaced by the shared suite (python + typescript mirrored):
- add recursive rm_r and ancestor cache invalidation (buckets have no dir
markers)
- stat dir-probe via list() instead of trailing-slash stat
- wire filetype (parquet/feather) commands into hf
- ts: du file fast-path, add unlink/create/rm_r to hf CommandIO, drop the
bespoke rm override (use the generic factory for provision), make find
-mtime a no-op matching s3/onedrive and thread -empty
- make the shared ts RM_BUILDER resolve rm_r/rmdir lazily so object stores
without rmdir still unlink files
* chore(integ): gather the fake servers under integ/server/
* Remove more dead Python helpers and tests
* chore: drop now-unused sync IO type aliases from utils/types
* fix: restore version api commit, main's restore/state_diff tests use it
* feat(version): cross-plane state diff and surgical restore
* refactor(version): derive mode ranks from the canonical MOUNT_MODE_RANK
* rename(version): restore scoping planes -> categories
* refactor(version): drop grant_widenings, reuse norm and shared version constants
GNU-pinned via docker debian:stable-slim, mirrored in Python and TypeScript:
- ${var:?msg}/${var?msg} now abort like bash: fatal at top level with
status 127 (exact GNU default messages), status 1 when contained by a
subshell or pipeline segment. ${var:=def}/${var=def} assign into the
current scope (function locals included). Multi-word operands
(concatenation nodes) reach the operator.
- $$ expands to the host pid; $! expands to the last background job's
job-table id (deliberate divergence from bash pids so wait $!/kill $!
work), isolated per subshell and reset-safe via Session.last_bg_job_id.
- new exit builtin with bash semantics: stops the current line keeping
earlier output, defaults to $?, wraps mod 256, numeric-argument error
exits 2, too-many-arguments refuses to exit with 1, contained by
subshells/pipeline segments/background jobs, escapes functions. Fatal
expansion errors ride the same ExitSignal unwinding.
- unclassified AST nodes (e.g. C-style for) now return
"mirage: unsupported shell construct: <type>" exit 2 instead of
crashing with TypeError.
- $$ parse fix: simple_expansion special_variable_name is read directly
(rfind previously split $$ into a literal $).
- 22 new integ JSON cases (integ/bash/errop.json, integ/bash/exit.json);
both hosts pass all 902 JSON cases on ram/disk. Unit tests added in
both languages.
Known divergences (documented): error prefix omits bash's "line N:",
eval/source contain exit instead of exiting the outer line, $! is a job
id not a pid.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refactor(runtime): interpreters are handler internals, one Runtime seam
Unify PythonRuntime/JsRuntime into one Runtime contract (RunArgs/RunResult,
captures, attach); derive command bindings from an ordered runtimes list
(first capturer wins, vfs an ordinary entry); one shared interpreter
command core; dispatch injects a single bound runtime only for commands
that have one. Deletes python_runtime/js_runtime/runtime_options kwargs,
yaml keys, selectors, and registry globals.
* fix(integ): cli_runtime battery speaks runtimes: entries, create surfaces entry errors
* refactor(runtime): TS RunResult.stderr null when empty, matching Python
TypeScript mirror of #542. makeGenericOps (packages/core/src/ops/
generic/factory.ts) emits a backend's RegisteredOp set from the same
X_CMD_OPS CommandIO table that feeds makeGenericCommands, so each
backend declares its core surface once. All 28 TS backends migrate to a
one-call ops/<b>/index.ts; ~120 hand-written wrapper files are deleted.
Exported const names (S3_OPS, GMAIL_VFS_OPS, ...) are unchanged, so
resource classes and remapOpsResource aliases are untouched.
Knobs mirror Python: filetypeRead (per-backend extension lists — TS
mixes .h5/.hdf5), mkdirParents (disk/ssh/databricks), overrides
(postgres read limit/offset override, gdocs-family rendered reads stay
hand-written), emulateTruncate (unused by builtins — every TS truncate
is a native core fn — kept for Python/SDK parity), and forwardIndex
(ram/disk/redis/ssh historically call their cores index-less: their
readdir caches listings the mutation ops never invalidate, so
forwarding kwargs.index would serve stale listings after mkdir/rmdir —
caught by fs_monkey.test.ts during migration).
CommandIO gains append/setAttrs; its never-consumed truncate field is
retyped to the real (accessor, path, length) signature. hf's table
gains unlink (the old wrapper dir had one; golden fixture requires it).
The golden ops-inventory snapshots (23 core + 5 node backends) gate the
whole migration — the registered op surface is byte-identical.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
make_generic_ops (mirage/ops/generic/) is the ops-layer analog of
make_generic_commands: given the CommandIO manifest a backend already
declares in commands/builtin/<b>/ops.py, it emits the backend's full
RegisteredOp set. Each mirage/ops/<b>/__init__.py is now a 4-line
factory call; the ~210 hand-written 23-line wrapper files are deleted.
- Five wrapper shapes reproduced exactly (keyword-only index on
read-likes, positional data/length/dst on writes, setattr kwargs), so
the FUSE-observable surface is unchanged.
- Quirks became declarative knobs: emulate_truncate (the read-pad-write
synthesis previously copy-pasted across s3/ssh/ram/redis),
mkdir_parents (disk, databricks_volume), filetype_read
(parquet/feather/orc/hdf5 rendered reads, optional-dep-gated like
try_load_command), and multi-resource fan-out (HF_RESOURCES x4).
- Irregular ops stay hand-written and compose with the factory:
chroma/dify grep+search, gdocs/gsheets/gslides dual-resource
filetype reads (factory read suppressed via overrides={"read"}).
- CommandIO gains append/set_attrs; missing create/truncate core fns
wired into the nextcloud/onedrive/s3/ram/redis/ssh tables so every
golden surface is reproduced bit-for-bit.
- The golden ops-inventory tests (241 ops / 31 backends) pass
unchanged - the whole migration is provably a pure wiring change.
- SDK: GenericResource now auto-derives its VFS/FUSE ops from the same
io= table (auto_ops=True, user ops shadow same-named derived ones),
so one-file custom backends mount over FUSE with zero @op wrappers;
make_generic_ops/OpsTable exported from mirage.sdk.
- BaseResource.register_op accepts bare RegisteredOp instances,
matching the Ops facade.
TypeScript mirror (makeGenericOps) follows in a separate PR; the TS
golden inventory tests are already in place and green.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(types): stdin annotations use the ByteSource alias
* refactor(generics): injected ops are path-only, accessor+index bind at the wrapper
Generic commands no longer take accessor (or a dead index) just to
thread them back into injected callables. Builders and bespoke wrappers
bind both via bound_op (None-passthrough) or partial for write-side
ops, mirroring the TS builders' closures. call_*/resolve_pattern/relay
helpers and the cache read-through wrappers gain path-first forms;
CommandIO-level ops keep the raw (accessor, path, index) shape.
* refactor(index): NULL_INDEX everywhere, no index-is-None branches
index is a required IndexCacheStore on every op and wrapper; callers
with no real index pass NULL_INDEX (the null object built for exactly
this). Dead is-None guards drop out of trello/linear/github_ci/google
readdir+stat, CacheManager, and the github wrappers.
* style(types): parameterize every bare generic, gate with disallow_any_generics
dict -> dict[str, Any] (payloads) / dict[str, object] (flag bags),
Callable -> Callable[..., Any], plus list/tuple/Awaitable/Pattern/
Task/Future/Token/AsyncMongoClient. The mypy gate now carries
disallow_any_generics so bare generics cannot come back.
* style: pre-commit formatting + parameterize main's new bare dicts
* fix: post-merge gate fixes + make security barriers analyzer-recognizable
- resolve_within_root: startswith(root + sep) guard instead of
commonpath (same semantics; the shape CodeQL models as a
path-injection barrier)
- cli table output: trimEnd() instead of the polynomial /\s+$/ trim
- example proxies: re-encode validated endpoint segments before URL
construction (recognized SSRF barrier; no-op for the allowed charset)
- parameterize bare generics arriving from #540 (the new mypy gate
caught them)
* fix(server): plain startswith barrier in resolve_within_root
* chore(integ): retire legacy scripts duplicated by the declarative harness
- migrate assert_real_mtime into the shared battery (unix/mtime.json:
cache-cleared ls -l of a fresh write must not render the epoch
sentinel), all 10 targets
- delete the pure run_cases duplicates: ram/disk/redis/ssh/s3_cases/
onedrive_cases/nextcloud_cases (py) + ram/disk/redis/opfs/ssh/
s3_cases (ts) + truth.txt (no consumers left)
- CI: drop the duplicated steps, remove integ-ssh + integ-ssh-ts jobs
(integ-shared covers ssh on both hosts), update gate + change filters
- cases.py/cases.ts stay: lancedb + qdrant still run run_cases against
their own truth files
* fix(ts): ls -l renders metadata-less entries in the compact placeholder form
Mirrors the python formatter: synthetic API-backend directories (size and
mtime both unknown) render 'drwxr-xr-x<tab>-<tab>-<tab>name' instead of
inventing size 0 and the epoch mtime. Caught by mtime_dir_not_epoch on
the s3 targets in CI (python passed, typescript failed).
* feat(integ): ssh + nextcloud shared-harness targets, wire harness into CI
- ssh = 9th target, first live py<->ts parity network backend: shared
no-auth asyncssh SFTP server (integ/runners/tools/ssh_server.py),
in-process for the python runner, SSH_HOST/SSH_PORT for typescript
- nextcloud = 10th target (python-only), scoped per-run WebDAV subroots,
gated on NEXTCLOUD_URL
- new integ-shared CI job runs both hosts' declarative batteries (moto +
sftp services for the ts host); integ-data runs the nextcloud battery
- fix SFTP rename onto existing file via posix-rename@openssh.com with
plain-rename fallback (py + ts)
- fix ts ssh read: sftp.readFile issues one whole-file READ whose reply
can exceed ssh2's 256KiB packet cap (fatal, strands the call); collect
the chunked read stream instead
- fix nextcloud copy: add the str|PathSpec tolerance shim (cp -r merge
passes plain child paths)
* test(integ): stat mount root on every target
Regression guard for onedrive root stat fetching the Graph root (#528);
runs on all 10 targets.
* Fix latent type and lifecycle bugs
* fix: close ownership for shared resources, gate find -empty, gdrive populate parity, drop mongo DriverInfo
* style: yapf/isort formatting on merged test files
* fix: fan-out grep keeps GNU any-match exit 0, mock github_ci stat populate readdir, current mock s3 mtime
* fix(ts): mirror close ownership, shared stores and copy-shared resources stay open
* refactor: shared op-fn aliases and copy/move strategies in mirage.types, tar package with types and constants
* refactor: single mtime-window primitive, drop msgraph.time re-export, github declares supportsSnapshot
* fix: find -mtime keeps unknown-mtime dirs (s3 integ), mktemp -p PATH-kind spec parity
* fix(find): remote -mtime reverts to TS parity, onedrive root stat carries mtime
- generic_bind find builder: stat filter only for local backends (matches
main + TS; remote backend find ops ignore mtime like the TS s3 core)
- s3/onedrive core find: drop in-core mtime filtering, document the
deliberate no-op in the s3 docstring
- onedrive stat: mount root fetches the real Graph root item so
size/modified are populated (du/find no longer see modified=None)
- integ: fake Graph stamps items at run time like moto; bespoke
onedrive suite freezes the clock for deterministic ls -l; stat_dir
case prints stable fields only
* test: drop remote find -mtime tests reverted to TS-parity semantics
* refactor(msgraph): shared item ops, sharepoint joins the shared harness
drive_ops now carries the whole Graph drive item layer both backends
were duplicating: entry_stat, cached readdir/stat, iter_tree, find,
du walkers, versioned read/stream with observer capture, and
capture_item_metadata. The onedrive and sharepoint op files are
addressing shells (onedrive: config-scoped drive + key_prefix;
sharepoint: resolver levels), so every Graph fix lands in both.
SharePointConfig gains site/drive scoping: a mount can point inside
one document library, making paths drive-relative and the site/drive
namespace levels vanish. Unscoped mounts behave as before.
The fake Graph becomes a small tenant: per-drive FakeGraph states,
/sites discovery and /sites/{id}/drives listing, /drives/{id}/root
routing, drive-aware download and upload-session handling, and
cross-drive copy via parentReference.driveId.
sharepoint is the 8th shared-harness target (three scoped drives in
one fake site) and runs the full 890-case battery. The battery
immediately caught a real divergence: the sharepoint du command was
missing provision=metadata_provision, so du cost plans reported
precision=unknown.
* integ: fake Graph models SharePoint Office enrichment, real version ids
SharePoint (and OneDrive for Business) rewrites Office documents
server-side after an upload: metadata is injected into the file, so
downloaded bytes differ from uploaded bytes and cTag changes without a
user write. The fake now models this as a synchronous, idempotent
marker append on .pptx/.docx/.xlsx uploads (simple PUT and upload
sessions; server-side copies do not re-enrich), keeping shared-case
expectations deterministic.
integ/unix/office.json exercises mirage against the self-modifying
file on both Graph targets: read-after-write returns the enriched
content, sizes stay consistent, repeat reads are stable, server-side
copy preserves bytes, a cross-mount copy re-uploads without
accumulating markers, and plain files stay untouched.
Version ids now follow real Graph numbering (1.0, 2.0) instead of
opaque tags, and the simplified-vs-real edge cases in the fake are
documented inline (range handling, rename numbering, monitor auth,
upload-session status, consumer conflictBehavior).
* feat(store): generation CAS on the workspace meta record
The discovery record gets the same optimistic-concurrency treatment as
session records (#530): cas_set_meta on the state store ABC (RAM
compare-and-bump, Redis reusing the same cas.lua on the workspaces
hash) plus a replace_meta retry helper that merges ours over stored,
preserves created_at, and bumps the generation.
Fixes the attach race: _ensure_meta's create path is now a conditional
create, so two processes attaching a fresh workspace id mint exactly
one discovery record and the loser adopts the winner's default-session
pointer instead of clobbering it. Snapshot restore repoints the record
through replace_meta, serialized on the counter.
* refactor(store): share the CAS retry cap and generation accessor
MAX_FLUSH_RETRIES and MAX_META_RETRIES were the same concept twice;
one CAS_MAX_RETRIES now lives beside the CAS contract in the session
store module, and the repeated 'missing record or field counts as
generation 0' expression becomes generation_of/generationOf used by
both RAM stores, the flush retry, and replace_meta.
OneDriveConfig now subclasses MsGraphConfig and onedrive/_client keeps
only its addressing helpers, re-exporting the shared msgraph transport
(the two clients were byte-identical apart from config annotations).
New mirage/core/msgraph/drive_ops.py holds the drive logic both
backends were duplicating: DriveLoc (an addressing seam mapping
(path, action) to URLs so onedrive's key_prefix drive and sharepoint's
resolved drive ids plug in), copy_once/copy_tree with the conflict
merge recipe, rename_replace, create_child_folder and
upload_session_write. The onedrive and sharepoint copy/rename/mkdir/
write ops are now thin wrappers over it, so future Graph fixes land in
both by construction.
* feat(session): dirty-tracked flush with generation CAS on the session store
Session records gain a generation counter CAS'd at the store (Redis Lua,
RAM compare-and-bump) so two processes flushing the same session table
serialize instead of silently clobbering each other. The manager keeps a
last-flushed baseline per session and skips clean sessions entirely; on
conflict it adopts the stored generation and retries, raising after three
attempts.
* refactor(session): move the CAS Lua script into shipped cas.lua files
Inline script strings become real .lua files next to each redis store,
the pattern BullMQ uses for its Redis scripts. Python loads via
importlib.resources with a package-data entry so wheels ship it; the
node package reads the file beside the module and tsup copies it into
dist beside the bundle.
* style(session): rename flush baseline to persisted, name the retry cap
_flushed read as 'already flushed' when it means 'what the store last
saw from us'; _persisted says that. The retry loop count becomes
MAX_FLUSH_RETRIES and both flush paths gain concise comments on the
clean-skip, the deep-copy baseline, and the conflict-adopt retry.
* integ: exercise the session CAS across languages
The writer seeds a shared session; the reader (the other language)
flush-bumps that foreign record through the Lua compare, gets a stale
cas_set rejected, and proves the conflict-adopt retry lands serialized
after a third writer advances the record behind its back.
* test(session): real-concurrency CAS coverage, unit and cross-process
Unit: five concurrent writers per language read-modify-CAS one record;
every increment must land and the generation must equal the exact write
count. Integ: a python process and a node process hammer the same Redis
record at once (an announce-then-wait barrier makes the loops overlap),
then both languages verify no update was lost.
* fix(onedrive): GNU overwrite semantics against real Graph conflict defaults
Graph copy, move and createUploadSession default to fail on name
conflicts (and conflictBehavior=replace is files-only and unsupported
on OneDrive Consumer), so cp/mv/large-write onto an existing item
worked against the permissive fake but would fail on real OneDrive.
- copy: default-fail attempt, then delete-and-retry for file conflicts
and per-child recursive merge for cp -r into an existing directory,
invalidating each copied node so merged children are visible
- rename: 409 fallback deletes a conflicting file or empty folder and
retries; non-empty folder conflicts keep the error like mv
- write: upload sessions request conflictBehavior replace (the default
409s on the final chunk)
- mkdir: create with fail and tolerate nameAlreadyExists (idempotent,
matching the s3 core)
The fake Graph now enforces the documented conflict semantics
(query-param and item-body conflictBehavior, per-operation monitor
payloads, 409 shapes), lists versions newest-first with distinct
timestamps, bumps eTag but not cTag on rename, and uses fragment-safe
upload URLs (the old '#' token 404'd every upload session).
integ/unix/overwrite.json adds shared overwrite/merge cases for all
targets; new unit tests cover the conflict paths.
* fix(sharepoint): mirror onedrive Graph conflict semantics
Same recipe as the onedrive core, drive-id aware: copy resolves
nameAlreadyExists with delete-and-retry for files and a per-child
merge for folders (with per-node cache invalidation), rename falls
back to deleting a conflicting file or empty folder before retrying,
upload sessions request conflictBehavior replace, and mkdir creates
with fail and tolerates an existing item.
- add the onedrive target (py-only host) driven by the fake Graph server
in-process; mounts partition one drive via prefixes data/xm2/res
- fix onedrive drive_ref_path dropping key_prefix: cp on a prefixed mount
silently landed at the drive root, cross-dir mv had the same latent bug
- raise the fake Graph client_max_size to 8MB (aiohttp's 1MiB default
413'd uploads real Graph accepts)
- add onedrive to every case's targets; full 884-case battery passes
* integ: migrate META/META_OVERLAY/SLEEP to the shared harness, GNU-align ln and columnar ls
- ln multi-source now says "ln: target 'x': Not a directory" (GNU), both langs
- drop the TS bespoke filetype ls handler; GNU ls prints the bare operand,
matching python; remove the dead python filetype ls() helpers
- harness: cases gain check {stat, fields} (dispatch-stat readback compared
and parity-checked) and expect.elapsed {min, max} (timed execute)
- unix/meta.json: 19 ls -l metadata cases on all six targets
- unix/meta_overlay.json: 16 dispatch-stat cases on opfs/s3/s3-prefix
- unix/sleep.json: sleep timing cases with elapsed bounds
- fix opfs ls dropping opts.statOverlay so chmod/chown/touch render in
ls -l like every other backend
* fix CI: regen specs after ls filetype removal, ssh command count 111
Also commit the js/node runtime specs the generators emit but were never
checked in (the drift gate diffs tracked files only, so untracked spec
output passes silently).
* integ: migrate PROVISION cases to the shared harness
- cases gain provision: true (execute as a cost plan, render the
net/write/cache/ops/hits/precision line as observed stdout, parity
cross-checked) and clear_cache: true (cold-cache group start)
- unix/provision.json: 75 cases captured live on ram, byte-identical to
the legacy truth.txt values; the duplicated legacy prov_seq id becomes
prov_seq_2
* feat(ids): mint UUIDv7 workspace and session ids, drop reserved defaults
- delete DEFAULT_WORKSPACE_ID / DEFAULT_SESSION_ID / DEFAULT_AGENT_ID in both languages
- new id helpers: python mirage/utils/ids.py (uuid6 package), TS core/src/utils/ids.ts (uuid package v7); exported from mirage top level and core index
- Workspace ctor mints ids when not passed; explicit ids remain free-form and are the attach surface
- attaching without a session id adopts the discovery record's default_session_id (SessionManager.adopt_default re-keys before hydration)
- snapshot restore adopts the snapshot's default session identity and repoints the discovery record
- agent_id is truly optional: namespace user is None when unclaimed, whoami fails GNU-style (cannot find name for user ID, exit 1), observer and jobs record empty attribution
- server: new_workspace_id moved to core utils, REST id resolution unchanged
- version: mirage/version.py reads the installed distribution (pyproject is the single source); hardcoded 0.0.3 and the unknown fallback removed
- integ: state_store asserts uuid7 pointer + cross-language adoption; history.py pins an explicit default session
* integ: session_modes pins an explicit default session id
Add s3 (per-mount bucket) and s3-prefix (shared bucket, key_prefix
subfolder) targets to the shared declarative harness, running the same
case suite as ram/disk/redis via an S3_ENDPOINT env (moto/minio) or an
in-process moto fallback for standalone python.
Align columnar (parquet/feather/hdf5/orc) rendering across Python and
TypeScript so it can join the shared parity suite:
- canonical type vocabulary int64/float64/string in both languages
- new python table.py mirroring the TS shared table.ts renderer, with a
JS-String()-compatible number formatter (full-precision, ECMA
Number::toString) replacing pandas to_string/to_csv
- unify TS parquet onto the shared table.ts renderer
- fix python hdf5 file() (was crashing: no such attribute)
- fix TS grep -c on columnar (was dumping CSV instead of counting)
- align wc -l columnar output to GNU form <count> <path> in both langs
Add 22 columnar cases on a new /res mount (real parquet/feather/h5
fixtures) across all targets; orc stays python-only. Regenerate
truth_s3.txt and truth_onedrive.txt for the new render.
* refactor(store): unify namespace, observer, and session stores under WorkspaceStateStore
* chore(deps): bump mcp to 1.28.1 for GHSA-hvrp-rf83-w775
* integ: cross-language WorkspaceStateStore interop, both directions
* chore(py): shrink mypy baseline 117->84 (resource accessor typing)
Narrow each resource's accessor attribute to its concrete type so the
backend resolve_glob/stat calls type-check; wrap fingerprint stat() paths
with PathSpec.from_str_path (fixes a latent str-to-stat bug); retype the
shared hf_buckets core to _HfAccessor (common base of the hf siblings);
make DevStore a RAMStore subclass (_DevFiles a dict subclass); narrow the
sync redis get_state() reads (redis-py ResponseT is an async/sync union).
Full Python test suite green (9139 passed; fuse excluded, macOS limit).
* chore(py): shrink mypy baseline 84->74 (workspace/executor/shell/resource typing)
* chore(py): clear non-generic mypy errors, shrink baseline 74->56
* chore(py): clear all generic command-layer mypy errors (127->0)
* chore(py): enable mypy as a hard gate (pre-commit + CI), drop ratchet baseline
* fix(py): sed -i guards write availability; dedupe mypy pre-commit hook post-merge
* fix(py): pyproject trailing newline; run mypy on hosted pre-commit too
* fix(cp,awk): GNU-align target-not-directory and awk -f missing errors
cp/mv with multiple sources to a non-directory now report
`cp: target 'X': Not a directory` in both languages: Python doubled the message
and TypeScript omitted the strerror. awk with a missing -f program file reports
`awk: <path>: No such file or directory` in both; Python used the mount-relative
path and TypeScript emitted only the bare path.
Updates the non-directory assertions (workspace cross-mount and databricks_volume
backend tests) to the GNU-aligned message, adds the missing TypeScript cp/mv unit
coverage (including the not-a-directory guard) and a crossmount cp reads test.
* test(integ): shared py/ts declarative harness with live parity
Declarative integration harness consumed by both Python and TypeScript from one
source of truth, alongside the untouched legacy suite (not wired into CI):
targets.json, 747 cases in unix/bash/crossmount JSON migrated with real
exit/stdout/stderr, byte-exact fixtures/files/v1, py+ts runners, and
runners/parity.py to cross-check both runtimes' live output. Excludes
integ/fixtures from whitespace hooks to preserve byte-exact seed data.
* chore(py): shrink mypy baseline 127->117 (server/executor/agents/misc)
Clear the dulwich version store (wrap bytes shas in ObjectID/Ref NewTypes),
workspace executor command + mount (flag dict dict[str, object], Session
_local_vars field, ByteSource _wrap narrowing, str/bytes stderr rename),
server/agents typing (Workspace.execute overload on provision -> IOResult
vs ProvisionResult clears all agent adapters), types/config registries,
databricks optional-import via TYPE_CHECKING, email-triage via FlagView.
Full Python test suite green (9120 passed; fuse excluded, macOS limit).
* chore(py): use # type: ignore for databricks optional-import shim
Match the codebase convention (workspace.py RedisFileCacheStore,
wasi.py wasmtime) instead of a one-off TYPE_CHECKING split.
* feat(namespace): reconcile orphaned overlays on remote delete
When a backend reports a path gone, the deletion signal now feeds both
consumers: the file cache evicts and the namespace GCs the orphaned
attribute overlay (an authoritative symlink is left intact). This closes
the gap where a chmod'd-then-remotely-deleted file left a dead node forever.
Extract a Reconciler that owns the reconcile concern (the pre-existing
ALWAYS fingerprint-freshness check plus the new deletion GC), so the
dispatcher goes back to routing. Add a NamespaceBlock to the YAML config
so the namespace store can be pointed at Redis (shared across agents).
Both languages.
* style: satisfy pre-commit (drop unused type-ignore, wrap long line, formatters)
* feat(namespace): unify reconcile across all read paths + no-fingerprint fix
Single-mount shell reads (cat/ls/stat) resolve through the mount registry,
not the dispatcher, so orphan-GC never fired for them. Route both paths
through one shared Reconciler: it gains a probe() that re-stats via the ops
registry, a reconcileRead() the registry calls per read command, and keeps
mayServeCached()/onOpMissing() for the dispatcher. A remote delete now GCs
the orphaned overlay whether reached by shell command, cross-mount, or FUSE.
Also fix the no-fingerprint case: under ALWAYS a backend that carries no
fingerprint cannot be cheaply verified, so drop the cached copy and re-read
(catching a remote delete via the fresh read) instead of serving stale; and
propagate a transient re-check error rather than serving stale, matching both
languages.
Adds an s3 overlay_orphan_gc integ case (both langs) covering the shell path.
* refactor(namespace): break registry->reconcile cycle via dependency inversion
The registry only calls reconcile_read, so depend on a local ReadReconciler
interface instead of importing the concrete Reconciler behind TYPE_CHECKING.
The dependency now points down (reconcile imports the mount layer, not the
reverse); the Reconciler satisfies the interface structurally. Both langs.
* Fix path.strip() to strip slashes instead of whitespace in collision error message
In raise_on_collisions(), the error message used path.strip() which
strips whitespace, while all surrounding code correctly uses
path.strip("/") to strip leading/trailing slashes. This caused the
collision error message to display the path with slashes intact but
whitespace stripped, inconsistent with the ancestor display.
* fix: handle empty input correctly in sort, rev, fold, and expand commands
- sort: guard on all_lines instead of output to avoid discarding blank-only files
- rev/fold: return empty bytes for empty input instead of spurious newline
- expand -i: match tabs preceded by spaces in leading whitespace
* fix(ts): fold empty-input guard must not swallow a single blank line
splitLinesNoTrailing collapsed both "" and "\n" to [], so the new
length===0 guard emitted empty output for a lone newline where GNU and
the Python side emit "\n". Distinguish truly-empty from blank-only,
matching splitSortLines and Python split_lines. Also apply yapf/prettier
formatting the PR missed (fold.py, rev.py, fold.ts, sort.ts).
* test(integ): cover fold/rev/sort/expand empty-input edge cases
Add labelled byte-count cases (py + ts) so truth.txt asserts them as
fixed substrings: empty file -> 0 bytes, lone blank line -> 1 byte,
expand -i on a space-then-tab -> 10 bytes. Each expected value differs
from the pre-fix output, so the cases guard the regressions. Verified
green on ram, disk, and redis backends.
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* feat(runtime): wasi/quickjs runs see FUSE-mounted workspace mounts
The wasm runtimes preopen active FUSE mountpoints (pull-model provider
read per run), so sandboxed python3/js code reads and writes mounts
live through the kernel and the namespace, with the same cache and
modes as shell commands. A mount claiming / relocates the interpreter
build to /.mirage-wasi with PYTHONHOME following. Without FUSE the
sandbox keeps today's empty filesystem.
Covered by live runtime tests (plain dirs stand in for mountpoints), a
real-FUSE sandbox probe in the CLI battery (the daemon serves its own
mount while a wasm run blocks on it), and doc updates; TS quickjs has
no preopen equivalent and is documented as a gap.
* refactor(runtime): bridge wasi/quickjs guest file I/O through dispatch via WASI interception
Replace the FUSE-preopen mechanism: the wasm runtimes now shadow the
guest's preview1 filesystem imports with host functions (wasm_fs.py)
that route mount paths through the workspace dispatch and serve the
interpreter build read-only from the host. Mounts are visible with no
FUSE, no fuse: key, and no setup; write modes and per-session mount
narrowing enforce at open(). Also fixes dispatch rename passing dst
unaddressed, and invalidates caches on mkdir/rmdir/rename.
* refactor(runtime): split wasm_fs into a wasm package (abi, bridge, fs, host, runtime)
One module per job: abi.py holds the preview1 wire constants and
record packers (distinct numbering from POSIX, so nothing is reusable
from errno/os), bridge.py the sync dispatch hop, fs.py the path
router, host.py the preview1 host functions, runtime.py the shared
WasmRuntime. Tests mirror the split; test_host.py guards the ABI
table against the method signatures.
* feat: disk attrs read the real inode, setattr overlays only the residual
* fix: disk snapshot preserves file mode; cover clamped-mode mv + state round-trip
* feat: route FUSE through the namespace (links + attr overlay)
* test(integ): cover the full namespace command surface in the cross-mount battery
* refactor(fuse): drop the synthetic /.mirage/whoami surface
* feat: whoami is the workspace user, claimed by agent_id and shared via the namespace store
The accessor methods named after builtins made bare str/list/int/bool
resolve to the methods (not the types) in later annotations within the
class body, which is why FlagView.list needed the ugly builtins.list[...]
escape hatch. Renaming to as_bool/as_int/as_str/as_list removes the
shadowing entirely: annotations are plain builtins again, no builtins.*
qualification needed. Updates all 149 call sites (fl./flags. and the
inline FlagView(...).bool(...) in crossmount fanout/detect) and the
CLAUDE.md FlagView doc. raw() keeps its name (not a builtin).
Genuinely fixable, zero-runtime-risk typing fixes that fully clear
their modules from the ratchet baseline (340 -> 333):
- cli/output.fail is NoReturn (raises typer.Exit); fixes cli/workspace
missing-return.
- FlagView.list/raw annotations qualified with builtins.* (the str/list
methods shadow the builtins in class scope).
- orc/parquet batches and text digits get element-typed annotations.
- sharepoint/readdir drops redundant re-annotations (no-redef).
- resource/base drops an unused type-ignore.
Full pytest + targeted module tests green; annotation-only, no behavior
change. Remaining baseline is genuine per-module typing work (union-attr
None-guards etc.), tracked for incremental follow-up.
The pyodide-backed python3 core tests boot slowly on loaded CI shards and
intermittently trip the 5s vitest default (heredoc, pipe-stdin). Give the
whole suite a 30s ceiling.
Rename the shared wasm runner and sweep the host/guest wording out of
the docstrings, comments, and docs in favor of plain terms (the run, the
sandbox, the module).
The engine has WASI capability isolation (no fs/mount/network), but not a
resource sandbox: CPU and memory are only bounded by command_safeguards.
Rename the section and spell out the distinction so it does not read as a
safe-for-untrusted-code claim.
Both general commands hand-rolled an identical script-path resolver (py
_resolve_script; ts resolveScript plus a local normalizePosix). Move one
copy to the shared command utils (py utils/paths, ts utils/operands),
reusing the canonical resolve_path/posixNormpath, and drop the
duplicates. Unit coverage added on both sides.
The quickjs CI job ran tests/runtime/python/test_quickjs.py, but the js
runtime tests live under tests/runtime/js/ (test_quickjs + test_select);
point the job at the directory. Also bump httplib2 0.31.2 -> 0.32.0 and
setuptools 82.0.1 -> 83.0.0 to clear the two audit advisories.
Download qjs-wasi.wasm in the cli cross job and point
MIRAGE_QUICKJS_HOME at it, so the Python daemon runs the js probe on
quickjs instead of falling back to the 127 build hint. Both languages
now execute 6 * 7 and assert an identical js_out=42.
Adds a JavaScript runtime alongside python3: `node` and `js` (aliases)
run small scripts and pipe transforms on a sandboxed quickjs engine,
mirroring the python3 command family.
- py: QuickJsRuntime runs quickjs-ng (qjs-wasi.wasm) on wasmtime; wasi
and quickjs now share a WasmGuestRuntime helper (compile+cwasm cache,
per-run epoch cancellation). Option-block registry pulled into a
neutral runtime/options.py.
- ts: QuickJsRuntime runs quickjs-emscripten (browser + node) with a
std/console/scriptArgs bootstrap so a script behaves identically to
the py runtime. runtime_options resolution shared via runtime_options.ts.
- Command: -e inline, a mounted script file, or stdin; scriptArgs for
argv, std.in.readAsString() for stdin, .mjs / -m for ES modules.
Sandboxed: no node builtins, no npm, no host fs, no mounts.
- Config: `runtime: {js: quickjs, quickjs: {home: ...}}` both languages;
jsRuntime threads workspace -> registry -> mount -> command like
python_runtime. Graceful default when quickjs is not set up.
- Tests both langs, CLI runtime battery js probe, CI runtime-quickjs
job, docs pages.
Review finding: the clamp's 'evicted straight after the add' rationale
only holds for the RAM store. RedisFileCacheStore documents cache_limit
as advisory (no client-side eviction), so clamping an explicit
max_drain_bytes above the limit silently overrode user configuration
and stopped caching files the Redis store could genuinely hold.
drain_budget now honors an explicit max_drain_bytes as configured and
derives only the None default from cache_limit, which still bounds the
client-side drain buffer on advisory stores. TS drainBudget mirrors.
The integ jobs caught a cache-poisoning regression from the previous
commit: pipes.py close_quietly duck-types on aclose for SIGPIPE-style
teardown, so giving CachableAsyncIterator a public aclose made an
early-exiting pipeline (cat x | head) close the underlying S3 stream.
The background drain then saw immediate exhaustion and cached the first
8192-byte chunk as the whole file; wc/grep/jq/md5 all read the poisoned
entry (truth_s3 diff: 8192 vs 15533 on all four object-store mounts).
Rename the helper to _close_source so teardown keeps no-opping on the
wrapper and the drain still closes an over-budget source. Regression
test drives the exact pipeline sequence: partial consume, close_quietly,
apply_io, and asserts the cache holds the full content.
Review findings on the drain clamp: drain_bounded returned early
without closing the underlying iterator, so streaming backends kept
their HTTP response open until GC, and this path is now the default.
drain_bounded (TS drainBounded) closes the source on budget exceed;
CachableAsyncIterator grows an aclose() delegating to the source.
Also drop the console.info on the TS skip path: budget exceed is a
normal event now and core has no logger, so it polluted stdout. The
Python side keeps logger.info, which is silent unless configured.
max_drain_bytes=None previously meant an unbounded it.drain(): a
partially consumed stream (head of a multi-GB object) kept downloading
and buffering the whole remainder in process memory, then the oversized
add evicted every other entry and finally itself, wiping the cache.
Draining past cache_limit can never produce a cache hit, so the budget
now resolves via FileCacheMixin.drain_budget (TS drainBudget):
min(max_drain_bytes or cache_limit, cache_limit). None keeps meaning
"default", now bounded; the unbounded path is gone. Snapshot state
still round-trips the raw configured value.
The index tightening left the per-backend _provision.py cost helpers
(redis/s3/ssh/onedrive/nextcloud/sharepoint) and the generic
make_file_read_provision with a keyword-only index and no default, but
the provision cost estimator invokes them without threading index,
crashing the redis example (file_read_provision missing index).
Default index to NULL_INDEX: a missing index degrades the estimate to
UNKNOWN precision, exactly as the old index=None guard did.
Caught by the CI Examples step (example_redis.py vs truth), which the
local pytest+integ battery does not cover.
Adds NullIndexCacheStore (NULL_INDEX): a no-op index whose lookups miss
and whose writes are discarded, so behavior is identical to the old
index=None path. Every backend core op now takes a non-optional
index: IndexCacheStore = NULL_INDEX instead of IndexCacheStore | None =
None, and the 'if index is not None' cache guards are deleted (the cache
read/populate run unconditionally against a real store or the no-op).
Why a default and not required: backend ops are also invoked through
untyped Callables (the generic find/grep walk, filetype handlers), which
mypy cannot check; requiring the arg there crashed at runtime (find over
a mount returned nothing). Defaulting to NULL_INDEX is crash-safe and
behavior-preserving, and still lets the typed dispatch/wrapper path
thread the real mount index.
Also fixes a latent bug the change surfaced: the databricks_volume
rename/mkdir/rmdir ops-layer wrappers passed kwargs.get('index')
positionally, sending None into a now-guardless stat; they fall back to
NULL_INDEX. Resource fingerprint() methods and du/exists helpers thread
self._index or NULL_INDEX. Baseline 441 -> 343.
Tests that passed index=None or omitted it now pass NULL_INDEX.
The index parameter is optional-typed everywhere, but for the API
backends that resolve virtual paths to remote IDs (discord, email,
gdocs/gdrive/gmail/gsheets/gslides, github, github_ci, linear, slack,
trello, chroma/dify find, google tree_ops) it is not a cache: without
it they cannot resolve and raise enoent. Ops always threads the mount's
real index, so None never reaches them at runtime.
Make index required (IndexCacheStore, no default) on those signatures
and delete the now-dead 'if index is None: raise' guards. find keeps a
keyword-only index. This turns the PR #379 class of bug (a caller
forgetting to thread the index, failing only on a cold read at runtime)
into a mypy error at the call site.
Baseline shrinks 369 -> 348. Tests: obsolete index-None-raises cases
removed; wrapper/scope tests thread a real index.
* feat(namespace): pluggable NamespaceStore, RAM default with Redis persistence
* refactor(namespace): NamespaceStore abstract base, RAM and Redis in separate files
* refactor(namespace): NodeMetaKey StrEnum in types; group store files into namespace/ package
* feat(metadata): chmod/chown/touch via setattr with namespace overlay fallback
* test(integ): guard cases.py mirage import against the redis.py sys.path shadow
* fix(metadata): review fixes for metadata ops
- bump snapshot format to v3 (symlinks key renamed to nodes)
- disk chmod keeps owner access on the real inode, sidecar reports requested bits
- touch on stat-only mounts reports Read-only file system instead of crashing
- content writes clear overlay mtime/atime so touched times do not freeze
- mv drops the replaced destination's node meta and keys moves into linked dirs under the real path
- glob rm purges node meta via segment-wise pattern match
- symbolic chmod on directories builds on 755
- touch -r resolves relative references against cwd
- extract shared builtins helpers (flag/operand parsing) into shared.py
* fix(metadata): typed setattr kwargs, non-null node stderr for the mypy ratchet
* chore(py): mypy ratchet gate with per-module baseline
mypy runs over mirage/ in pre-commit (and CI via pre-commit --all-files).
The 441 modules with pre-existing errors are baselined with
ignore_errors overrides in pyproject; the other 1,099 modules are
enforced now. The baseline only shrinks: cleaning a module means
deleting its entry so regressions in it fail. Warm runs take ~1s.
* chore(deps): bump json-repair 0.59.5 -> 0.61.4 (GHSA-xf7x-x43h-rpqh)
- mount_grants/mountGrants -> mount_modes/mountModes; the coercion is
parse_mount_mode/parseMountMode; docs, CLI help, and the integ battery
(session_modes.*) follow. Values are just MountMode, so the session
carries mode ceilings, not a second 'role' vocabulary.
- TS core Workspace now takes python-style per-entry [resource, mode,
safeguards?] tuples; the modeOverrides option is gone (its silent-drop
risk showed up immediately: the integ battery still passed it and the
READ override stopped applying).
- integ/session_modes.py|ts wired back into test_integ.yml (the steps
had gone stale against the renamed files).
* test: construct PathSpec at core-call sites instead of raw strings
Sixty-five call sites across eighteen files passed string paths
directly to core functions, leaning on the per-function coercion
shim the next commit deletes.
* refactor(core): PathSpec-only core, drop str coercion shims
Core ops now require PathSpec; the Ops facade remains the only
str-to-PathSpec boundary (mount-aware), and helper-injected fns keep
the to_pathspec adapter in utils/wrap. Deletes 191 isinstance-str
coercion blocks across 166 core modules, the str branch in
resolve_glob_with, and the sharepoint resolver str fork.
Production fixes surfaced by the removal: ram/redis read_stream now
forward the PathSpec to stream() instead of a computed string key;
generic tar keeps the archive operand as a PathSpec and builds specs
for extracted members; generic patch reads targets via PathSpec and
builds specs for parsed patch paths.
Tests that exercised the deleted coercion are removed; remaining test
call sites construct PathSpec explicitly or bind the production
call_readdir/call_stat/call_read_bytes adapters.
* fix(core): forward PathSpec in redis/ram read and realpath/iconv operands
The coercion-shim removal exposed three more production sites that
passed computed strings to spec-only fns, all caught by CI (redis
integ, notion integ, snapshot interop, redis-gated unit jobs):
- redis stream() recorded the op with the PathSpec instead of the
stripped key; redis/ram read() unwrapped to mount_path before
calling read_bytes, which reads path.virtual.
- realpath -e normalized the operand to a string before the injected
stat; it now rebuilds a spec for the normalized path (notion
realpath_dotdot case).
- generic iconv passed output_path.mount_path to write_bytes.
Adds workspace-level regressions for realpath -e with .. on a mount
and iconv -o.
* fix(commands): wrap derived output paths in archive generics as PathSpec
gzip/gunzip/split/csplit/unzip build output paths by string arithmetic
(suffixes, part names, entry names) and passed them raw to the injected
write_bytes/mkdir. The deleted per-backend coercion shims used to absorb
that; backends whose ops read the spec unconditionally (onedrive,
nextcloud in the shared-truth suites) crashed with 'str' object has no
attribute 'virtual'. Wrap each derived path with PathSpec.from_str_path
in mount-relative space, matching the tar/patch fix in this branch.
io.writes keys are unchanged.
* feat(session): per-mount role grants with read/write/exec narrowing
Sessions can now be granted per-mount roles: create_session(id, mounts={'/data': 'read', '/scratch': 'write'}). The mount mode stays the ceiling; a grant only narrows it. List form keeps the old allowlist behavior. Exposed on the SDK, server, and CLI (-m /data:read) in both languages, with integ truth coverage. A user-defined root mount is now governed by grants instead of bypassing them, and the python capability error gained the GNU-style command prefix to match TS. The mirage runtime package is renamed to mirage/context (core/src/context).
* fix(ts): default modeOverrides in grants test for exactOptionalPropertyTypes
* chore(ts): unexport MOUNT_MODE_RANK, knip gate
* test(integ): session grants through the CLI in the parity battery
The new probes caught a real divergence: TS isExecAllowed let the root
mount decide alone, so an exec grant on an EXEC data mount never opened
the gate. Mirror the Python any-mount loop (the root short-circuit and
defaultMode fallback were dead since the scratch root became a real
mount).
* feat(session): filesystem-style r/rw/rwx grant role aliases
The grant ladder is cumulative, so the cumulative chmod spellings map
cleanly: r=read, rw=write, rwx=exec. Accepted everywhere roles parse
(create_session, server, both CLIs); bit-style forms (w, x) are
rejected; serialization stays read/write/exec. Docs section updated
off the removed allowed_mounts API.
* docs(fuse): OS support matrix + Windows WinFsp page, import error points there
New home/setup/fuse.mdx: OS-by-SDK matrix (Linux gated, macOS local,
Windows python experimental via WinFsp with the advisory CI job as
evidence, TS has no Windows path) plus a platform-quirks table.
New home/setup/windows.mdx: WinFsp install (winget/choco/MSI) and the
Windows behaviors (unmount at process exit, mount-level ownership,
stat opens a handle so size-unknown files hydrate on first stat).
Python and TS setup pages link the matrix; the python import error now
names the OS drivers and the matrix URL.
* docs(nav): FUSE is its own sidebar group below Setup
* integ(fuse): deterministic size-unknown probes + advisory WinFsp job
A SizelessOps stat proxy (py) / stat wrapper (ts) simulates API-backed
resources whose size is unknown until fetched: stat 0 pre-open, full
content on read, real size after open. Both runners share the same
truth lines. The new integ-fuse-windows job is advisory
(continue-on-error, not in the gate): it installs WinFsp and runs the
python battery to surface what actually breaks on Windows.
* ci: explain why the advisory windows fuse job is python-only
* ci: windows fuse job dumps raw output before checking truth lines
* fuse(win): WinFsp mountpoint conventions in the python mount path
WinFsp requires a nonexistent mountpoint and creates it itself, so
_prepare_mountpoint removes a pre-created empty directory on win32
(rmdir only: non-empty raises instead of discarding). Readiness polls
bare existence there (os.path.ismount does not recognize WinFsp
directory mounts), and the teardown paths skip fusermount, which does
not exist on Windows (WinFsp unmounts when the serving process exits).
First real signal from the advisory windows job: 'Cannot create
WinFsp-FUSE file system: mount point in use.'
* fuse(win): guard getuid/getgid, map ownership via WinFsp uid=-1
os.getuid/os.getgid do not exist on Windows (round-2 signal from the
advisory job: getattr crashed and the WinFsp service died with
c000000d). MirageFS caches uid/gid once with a hasattr guard, matching
fs.ts. On win32 the mount passes uid=-1,gid=-1: the WinFsp-FUSE
builtin that presents files as owned by the mounting user, per the
WinFsp FAQ; raw POSIX ids would map through the SFU/Cygwin table
(0 = LocalSystem) instead.
* integ(fuse): CRLF-proof check_lines, platform-aware preopen verdict
Round-3 windows signal: the full battery already passes over WinFsp;
every line read as MISSING only because the runner checks the truth
file out with CRLF (and Windows Python writes CRLF), so the substring
probes carried a stray CR. check_lines strips CR on both sides.
api_stat_preopen becomes a verdict line: POSIX stats size-unknown
files as 0 pre-open, while Windows cannot stat without opening a
handle, so hydrate-on-open makes even the pre-open stat real. CLAUDE.md
gets the WinFsp conventions.
* fuse(ts): drop the 100 MiB sentinel, append direct_io at mount time
Size-unknown API files now stat as 0 before open and serve the real
size from fgetattr after open, matching Python. fuse-native's option
serializer has a fixed allowlist without direct_io, so mount.ts wraps
_fuseOptions() at runtime (a pnpm patch would not reach consumers).
Verified live on the macOS kext: cat/wc/head/tail/cp/readFile all
correct, JSON.parse of FUSE-read files works (no more zero padding);
without direct_io cat reads 0 bytes, so the option is load-bearing.
Docs: the limitations entry now documents the real constraint (sync
access to your own mount from the mounting process deadlocks the event
loop) and the TS FUSE setup page gains the per-tool size table.
* docs(ts): size-unknown semantics live in limitations, setup page points there
* chore(rules): no silent exception swallows, spec-drift CI gate, input dedup
Every except-pass site now either logs (the fourteen broad
best-effort populate paths and the stream closer) or carries a loud
one-line reason (forty narrow typed handlers: mkdir -p semantics,
tolerant IMAP parses, Retry-After fallbacks, ENOENT translations,
provision degrades, daemon-already-gone races). Pre-commit CI gains a
spec-drift step regenerating spec/ and failing on diff. linear/trello
share resolve_text_input via utils/stream. The nextcloud/onedrive/s3
du wrappers stay: their core du signatures do not fit the factory du
builder contract (attempting the migration broke s3 du -c/-h/-a and
cross-mount fan-out; unifying those signatures is its own change).
* style: shorten swallow-site comments to line limit
* chore(deps): pillow 12.3.0 for the five published advisories
Gmail sizeEstimate and Drive's storage size for google-apps files are
source-side numbers, not the rendered .gmail.json/.gdoc/.gsheet/.gslide
JSON length. They now live in extra (size_estimate / source_size) and
FileStat.size stays None until read. Raw binary downloads in gdrive keep
Drive's size: it is the real byte length.
* stat: sizes are render-derived or None, never storage or source numbers
postgres rows.jsonl reported table_size_bytes (on-disk pages) as the file
size: live repro showed stat/wc -c at 32768 for 662 rendered bytes. dify
documents reported the uploaded source file's size while read renders
joined segment text. Both now report size None with the storage/source
number kept in extra (size_bytes / source_size).
Unmasked by the fix: chroma and dify find dropped sizeless files entirely
under -size filters instead of counting them as 0 (the documented rule);
fixed in py chroma/dify and TS chroma.
* integ: pin sizeless render semantics for postgres, dify, chroma
postgres/dify truths updated (stat 0, provision unknown instead of the
49x-off exact estimate) plus new -size cases in both runners; chroma
gains a metadata-sizeless doc and a find -size case that regresses the
sizeless-drop bug in both languages.
* feat(runtime): pluggable python runtimes with monty sandbox
- PythonRuntime interface in both languages: name/run/close (plus runRepl in TS), runtimes are pure interpreters with injected workspaceBridge and listMounts capabilities
- Python: new leaf package mirage/runtime/python (base/local/monty/select); monty is the default python3 runtime, bridging pathlib and open() to workspace mounts; pydantic-monty moves to the optional 'monty' extra with graceful 127 degradation
- TypeScript: executor/python/runtimes/ with pyodide (default, moved from runtime.ts) and monty on @pydantic/monty (optional peer dep); pull-model mount visibility via MirageBridge.prefixes(), push machinery removed
- Selection: Workspace python_runtime kwarg / pythonRuntime option and yaml runtime: {python: ...}; names validated at config load with cross-language hints
- Fix: TS server workspaces router dropped pythonRuntime from whitelisted options
- CI: runtime-no-monty bare-venv job, integ runtime-py/runtime-ts jobs over ram/redis/s3/mongodb with shared truth, cli_runtime.sh daemon battery
- Docs: Runtimes group with python runtime pages for both languages
* test(integ): read mongodb collection schema.json under monty python3
* refactor(runtime): runtime name constants as the single source of truth
Python derives DEFAULT_PYTHON_RUNTIME and PYTHON_RUNTIMES from the runtime
classes' name attributes and the config default references the constant.
TypeScript hoists the names into runtimes/interface.ts; the classes, the
selector, and the server config validation all reference them, with
PYTHON_RUNTIMES exported so the server set cannot drift from core.
* test(integ): cli battery covers the default runtime named explicitly in yaml
* fix(examples): browser build externalizes @pydantic/monty
The monty browser entry imports @bjorn3/browser_wasi_shim without
declaring it as a dependency, so bundling the optional runtime breaks
vite builds. External keeps the dynamic import unresolved; selecting
monty in the browser example degrades to the runtime-unavailable path.
* fix(examples): python3 examples under the monty default
ram_python argv demos use the monty argv global instead of sys.argv.
example.py pins the local runtime for its sys.stdin piping demos and,
like python_s3, gains EXEC mode (both bit-rotted under READ). The s3
script uses readlines(): monty file handles are not iterable, noted in
the runtime docs.
* fuse: serve real sizes for size-unknown files after open (closes#83)
Unopened API-backed files still stat as 0 (no fake sizes), but the mount
now sets attr_timeout=0 and getattr(fh) answers with the open-hydrated
content length from a 30s TTL prefetch cache. Fixes wc -c printing 0,
BSD cp copying 0 bytes, and tail -c dumping the whole file over macOS
kext mounts; Linux keeps working and drops redundant refetches.
Cleanup: typed Handle dataclass replaces the stringly handle dicts,
MirageFS takes Ops (duck-typing removed), Ops.mount_prefixes() replaces
private _mounts reach-ins.
* examples(fuse): demonstrate size-unknown semantics on linear/slack/discord
stat-before-open 0, wc real bytes (and message count for chat.jsonl),
stat-after-read real size; discord example skips the virtual /.mirage
entry it previously crashed on. All three verified against live APIs.
* refactor(sed): read-safe generic builder replaces 14 bespoke wrappers (python)
sed joins the generic_bind builders: the stream-to-stdout path works on
every read-capable backend and only -i needs a write op, failing with
'sed: -i not supported on this backend'. Deletes the bespoke wrappers,
make_sed, and the dead databricks _helpers; databricks -i now works
(the backend has a write op, the old refusal was a wrapper limitation).
* refactor(sed): TS mirror, builder replaces 13 bespoke wrappers
SED_BUILDER joins the TS factory (read-safe; -i gated at runtime with
'sed: -i not supported on this backend'), makeSed and the per-backend
wrappers are deleted, and sed lands on every read-capable factory
backend like python. OPFS builds its command list by hand, so it keeps
a thin wrapper over sedGeneric. Specs regenerated; slack/discord
command-count pins follow.
* test(integ): sed read-only probe across the seven bespoke suites
Every read-only bespoke suite (dify, chroma, notion, lancedb, qdrant,
mongodb, postgres) now pins that streaming sed works and sed -i fails
with exit 1, in both languages against the shared truths. The builder's
PermissionError drops its 'sed:' prefix (the dispatcher adds it) and
the TS short-circuit emits the same formatted line so py/ts outputs
stay byte-identical.
* style: noqa the new chroma cases import
* refactor(stream): seven read_stream copies collapse onto stream_from_bytes
* refactor(google): shared Drive-item tree_ops and make_rm for the docs family
gdocs/gsheets/gslides stat and unlink collapse onto
core/google/tree_ops factories (the silent readdir-populate except now
logs); their byte-identical rm wrappers bind
generic/rm_command.make_rm, which exists because the factory rm builder
calls unlink without the index these backends resolve ids through.
* refactor(ts/stream): streamFromBytes helper, nine inline read-stream copies collapse
Mirrors python's utils/wrap.stream_from_bytes: the seven chat/KB ops
files and the lancedb/qdrant core stream pair all synthesized a stream
from the whole read inline.
* style: prune unused type imports after stream collapse
* refactor(provision): shared exact-zero and index-hit helpers, four clusters collapse
generic_bind gains exact_zero_provision (chat/KB metadata: the virtual
tree is already in memory) and index_hit_read_provision (chat reads
cost ops, not sized transfers), both langs. The seven-backend chat
_provision copies, the lancedb/qdrant zero pair, and the google-family
size-summing copies become re-export shims; the google family moves to
make_file_read_provision, dropping an older fork with a silent except,
dead index reassignment, no glob expansion, and no cost floors (two
gdrive expectations updated to the canonical semantics). TS mirrors
with the same shims; dead gdrive/provision.ts deleted per knip.
* chore(provision): drop linear/trello shims orphaned by the find migration
Their only consumers were the bespoke find wrappers PR #482 deleted;
verified against live Linear and Trello that provisioning now flows
through the factory defaults.
* docs(examples): live provision probes for API backends without integ
Walks each cred-configured chat/google backend to a real file and runs
the cat/grep/ls cost-estimate trio, mirroring integ's provision probe.
Verified live: discord, slack, linear, trello, langfuse, email (chat
reads charge index-hit ops, metadata is zero-cost, sizeless rendered
files degrade to UNKNOWN with a read-op floor). github_ci/gdocs/gmail
report their credential failures instead of silently passing.
* style: format provision example
* fix(find): -size treats directories as size 0; -path matches the display path
-size was a files-only imperative filter, so directories (including the
start path) bypassed it and find <dir> -size +N emitted every directory
the traversal listed. Directories now contribute size 0 at every
size-aware site (walk_find, 12 backend cores, emit_start_path), so +N
excludes them and -N keeps them; a deliberate, documented divergence
from GNU, which compares the inode size (closes#318).
-path/-ipath matched the mount-relative key, so any pattern naming the
mount segment (find /data -path '*data/sub*') matched nothing. Path
nodes are now stamped with the mount prefix at the two chokepoints (the
generic find command and walk_find) and evaluated against the display
path exactly as printed, GNU-pinned (closes#396).
Both languages; stale tests updated to the new pins; shared integ gains
find_size_gt_dirs/find_size_lt_dirs/find_path_* cases, truth
regenerated and verified byte-identical on all 8 local runners.
* fix(find): -size follows GNU strict bounds and unit round-up
GNU -size +N keeps ceil(size/unit) > N and -N keeps ceil(size/unit) < N,
so +0c no longer matches directories (size 0) or empty files, and non-c
units round the size up before comparing (-1k keeps only empty files).
Both parsers now emit inclusive byte bounds; new integ boundary cases.
* docs: FUSE size/mount gotchas and find -size semantics in CLAUDE.md
* docs(find): comment the dirs-as-0 -size rule at every enforcement site
* fix(find): TS flags path uses the GNU -size bounds; sizeless files count as size 0
findGeneric kept a private parseSize copy with the old inclusive bounds,
so every factory backend still had the off-by-one on the flags path;
dedupe onto the fixed findParse.parseSize. The mongodb pin follows the
size-0 convention for sizeless rendered files (matches FUSE, which
reports 0 before a file is opened).
* refactor(find): the wrapper SaaS backends honor -size and -path
slack, discord and langfuse drop their find wrappers onto the factory
walk with an is_dir_name hint (matching TS after #482). github_ci keeps
its wrapper for the cross-run guard and email for the folder-level
-name IMAP pushdown (now gated to name-only queries); both route local
walking through walk_find. TS github_ci gains the same guard + walkFind
wiring. Per-backend find tests both langs; example scripts exercise
-path and -size live.
* test(github_ci): find test satisfies tsc, FileStat instances and full CommandOpts
* refactor(find): mongodb postgres lancedb qdrant on the generic find walk
CommandIO gains an optional is_dir_name hint the generic find walk
passes through, which was the only capability keeping these four
byte-identical wrappers alive. Their find provision stays
metadata_provision via the factory default.
* refactor(find): linear trello on the generic find walk
Their hand-rolled walk+filter predates walk_find, stat'ed every entry
anyway, and only honored name/iname/type/depth; the generic find adds
size/mtime and expression support for free.
* refactor(commands): shared default_paths in utils, drop six copies
* fix(find): generic walk fallback honors every start point like GNU
* refactor(ts/find): walk fallback in FIND_BUILDER, 12 backends drop find wiring
Ports the python design: CommandIO gains an isDirName hint and the
factory find builder walks readdir/stat when no backend find op is
wired, instead of throwing. The twelve walkFind-wiring core find
modules and eight command wrappers are deleted; isDirName rules move
next to each backend's readdir. Fixes find on gdocs/gdrive/gsheets/
gslides, which registered the generic find with no find op and threw
at runtime (their core find modules were written but never wired).
Core find tests keep their expectations through a local walk helper.
* refactor(ts/find): discord langfuse slack linear trello on factory find, shared defaultPaths
The five hand-rolled wrapper walks join the factory fallback
(discord/langfuse/slack contribute an extension isDirName hint;
linear/trello classify via stat as before) and gain size/mtime and
expression support. defaultPaths dedupes into utils/operands.
databricks_volume's resource-level find inlines walkFind since the
core module it re-exported is gone; walkFind is now a core export.
* chore(ts): drop provision exports orphaned by the find migration
* test(find): backend wiring coverage for the factory find walk
Per-backend find tests over faked clients / embedded fixtures:
mongodb, postgres, linear, trello (commands/builtin) and lancedb,
qdrant (core, next to their conftest fixtures); TS linear gets a
factory-find test with a teams-only transport. Covers synthetic
metadata files, name/iname/type/depth/size filters, negation,
multi-root operand order, mid-path glob operands, and pins that a
bare word in expression position is a parse error and that sizeless
rendered files drop out of -size bounds.
* style: yapf format for the lancedb/qdrant find tests
* test(net): jina curl asserts the reader envelope, not page content
Jina sometimes serves a cached third-party snapshot of example.com, so
'Example Domain' is not a stable expectation; the URL Source line is.
* test(net): jina assertion matches the exact URL Source line
CodeQL reads a bare 'https://example.com' in body as incomplete URL
substring sanitization; anchoring to the envelope line start is both
scanner-clean and a stricter check.
* chore(net): remove the Jina Reader integration
The --jina curl flag routed GETs through r.jina.ai, whose live
behavior (auth walls, cached third-party snapshots of example.com)
kept breaking CI and tripping CodeQL. Drop the flag, the URL-prefix
helpers, and the live test in both languages; specs regenerated.
rg -e a -e b joined the patterns with a newline and passed the literal
to the native search, returning empty on postgres (all four scope
levels) and email (IMAP text search). The pushdown now requires a
single pattern and multi -e falls through to the generic with globs
still resolved, matching the other backends (#347, closes it).
MountCommandUnsupported now renders as GNU coreutils would:
'<cmd>: <operand>: Operation not supported' (EOPNOTSUPP strerror,
as-typed operand) instead of '<cmd>: not supported on the <backend>
backend', in both languages (closes#394).
- cat/search wrappers declare dispatcher-injected index/cwd explicitly
instead of fishing them out of **_extra with .get()
- dify stat() guards entry=None like stat_light before dereferencing
- ensure_tree drops the redundant visibility re-filter
(list_all_documents already filters); the tree test fake now honors
that contract
- search reuses read.segment_text, dropping the duplicate
segment_content
- find wrappers annotate index as IndexCacheStore | None
- is_mount_root computes the mount prefix once
* refactor(glob): extract resolve_glob_with from make_resolve_glob
* refactor(glob): notion resolve_glob delegates to resolve_glob_with
* refactor(glob): lancedb langfuse mongodb postgres qdrant on resolve_glob_with
* refactor(glob): disk hf_buckets redis s3 ssh on resolve_glob_with
* refactor(glob): email gdocs gdrive gmail gsheets gslides slack on resolve_glob_with
* refactor(glob): discord linear trello chroma dify on resolve_glob_with
Unmatched word-shaped globs now stay literal (GNU nullglob-off), so the
chroma zero-match test asserts the literal passthrough and the dify
directory-pattern test uses a consistent virtual/pattern spec.
* refactor(glob): remaining backends on resolve_glob_with
databricks_volume drops its local SCOPE_ERROR copy (same 10000 value as
the shared constant); onedrive and sharepoint gain the standard match
cap they were missing; github_ci keeps is_cross_run_root in place.
* refactor(ts/glob): discord langfuse slack linear trello notion on resolveGlobWith
The five chat/kb resolvers were still the naive single-directory loop
(no mid-path expansion, no nullglob-off literal fallback); notion
inlined an equivalent copy of the shared loop. linear/trello bind their
readdir filter through a closure. discord/slack zero-match tests assert
the literal passthrough now.
* refactor(glob): date_glob reuses GLOB_CHARS from glob_walk both langs
* fix(read): directory operands refuse with GNU EISDIR, implicit dirs probed via readdir (#457)
* feat(shell): bash arithmetic evaluator, (( )) command and $(( )) assignments (#369)
* feat(crossmount): paste/comm/join relay over the shared generics, awk streams (#445)
* test(integ): arith, EISDIR, and relay-interleave coverage; OPFS dirs raise EISDIR
* feat(spec): commands own operand arity, extra operands refuse with GNU errors (#452)
* fix(read): implicit-dir probe confirms against the parent listing, TS stream stats first
The ENOENT readdir probe trusted the operand's own listing: postgres
fabricates schema children for any name (Is a directory for missing
files) and lancedb raises driver errors that escaped the probe. The
parent listing decides instead, and any probe failure keeps the
original ENOENT. TS streamRefusingDirs now stats before reading like
the Python twin, so sftp directory reads report EISDIR instead of an
opaque Failure. Also updates the stale parser overflow test (#452
pass-through) and the databricks tr file-operand test (GNU tr takes
no file operands).
* refactor(spec,shell): CommandName enum, arith constants/errors modules, ArithParser/ArithEvaluator
Arith tokenizer/limit constants move to shell/constants and ArithError
to shell/errors in both languages; the parser and evaluator classes get
their public names. CommandName (StrEnum / TS string enum) lands in
spec/types next to OperandKind, used by the arity guards and the usage
message shapes; the diff/cmp hint grouping becomes USAGE_HINT_PREFIX in
spec/constants. USAGE_EXIT keys stay plain strings: types.py imports
constants for flag_kwarg_name, so the table cannot import the enum
without a cycle.
* fix(commands): read family keeps partial output past missing operands
cat/head/tail/wc abort on the first missing operand: the error reaches
the executor chokepoint before any output is emitted (or the lazy drain
discards it), so the good operands' output is lost. GNU keeps going:
partial stdout, one stderr line per failed operand, exit 1.
Builders now partition operands eagerly (split_readable / splitReadable)
and stream only the readable ones; wc's format_multi reports failed
operands and still prints the total row (0 total when none resolve).
Cross-mount run_operands catches filesystem errors at materialize so a
lazy drain failure no longer escapes to the route catch-all, which
dropped every operand's output and printed a line with no strerror.
Single-mount and cross-mount are byte-identical in both languages;
integ covers cat/wc/head/tail good+missing on ram, redis, and s3.
* refactor(errors): FS_ERRORS single catch set; document grep exit rule
The recoverable filesystem exception tuple was spelled out at five catch
sites; derive FS_ERRORS from the strerror table in utils/errors so the
catch set and the formatter cannot drift (mirrors TS isFsError). Hoist
runOperands' TextEncoder to module scope. Document at combinedExit that
grep's exit 0 on match-despite-read-error is a deliberate GNU divergence
pinned by integ.
* fix(commands): read family sweep processes every operand, keeps partial output
cut, tac, zcat, and strings processed only paths[0], silently dropping
every other operand (cut even exited 0 with a missing operand). They now
handle each operand independently and concatenate in operand order, and
nl's counter continues across operands instead of resetting per file,
all per GNU.
The remaining multi-file read builders (nl, md5, sha256sum, strings,
tac, rev, cut, expand, unexpand, fold, fmt, zcat) and the stat generic
now continue past a missing operand: partial output, one stderr line per
failed operand, exit 1 (resolve_readable/withReadable wrap the shared
split). sort keeps its GNU-style abort.
Cross-mount STREAM errors respell the internal cat sub-run's prefix to
the real command name, so cross-mount bytes match single-mount. Integ
pins nl (STREAM respell), md5 (FANOUT), and the previously uncovered
cross-mount stat on ram, redis, and s3.
* test(integ): partial-read battery on every backend; sed keeps going, sort aborts
Add PARTIAL_READ_CASES to the shared battery (both langs, truth.txt):
multi-operand correctness for cut/tac/nl/strings/zcat plus good+missing
for the whole read family, so all 13 shared-truth backends pin partial
output, per-operand stderr, and exit codes. cross_commands adds
cut/tac/sed/sort and the missing cross-mount stat coverage.
Pinning exposed two more divergences, fixed both langs: sed aborted
single-mount but GNU keeps going past a missing operand (generic_sed now
skips and reports per operand, including -i, exit 1 where GNU exits 2);
sort kept partial output cross-mount but GNU aborts (run_stream now
drops the body for sort when any operand failed, matching single-mount).
* fix(ts): missing-operand handling moves into the read-family generics
The withReadable builder wrapper only protected factory-built backends;
OPFS's bespoke wrappers call the generics directly, so integ-ts lost the
partial output on every swept command. The per-operand read handling now
lives inside the generics themselves (readOperands/operandsIo in
utils/operands), like cat/head/tail/wc/sed already do, and the factory
builders go back to thin wiring. nl, cut, rev, and sha256sum read their
operands eagerly in paths mode so the IOResult is sealed before the
output stream is handed back (tail/wc already buffer this way).
* refactor(crossmount): respell helper named for the fetch, prefix from Cmd.CAT
The rewrite is an artifact of the STREAM strategy fetching operands via
a native Cmd.CAT sub-run, not anything about the cat command itself, so
the name says what it means and the stripped prefix derives from the
enum instead of a hardcoded literal.
* fix(daemon): make pid file path configurable via MIRAGE_HOME / MIRAGE_PID_FILE
The daemon pid file was hardcoded to Path.home()/.mirage/daemon.pid in two
places (server/app.py and cli/daemon.py) and written unconditionally at
lifespan startup, so daemon start crashed on read-only / unset $HOME
(containers, sandboxed users, immutable rootfs) with no way to override.
Add a shared resolver (server/paths.py) exposing mirage_home(),
pid_file_path(), default_version_root(), default_snapshot_root(). Resolution
order for the pid file: build_app(pid_file=...) > $MIRAGE_PID_FILE >
$MIRAGE_HOME/daemon.pid > ~/.mirage/daemon.pid. MIRAGE_HOME also backs the
repos/ and snapshots/ defaults so the whole tree relocates with one var;
existing MIRAGE_VERSION_ROOT / MIRAGE_SNAPSHOT_ROOT still take precedence.
Both app.py and cli/daemon.py now read the single resolver so they can never
diverge. Mirrored in TypeScript (server env.ts/paths.ts/app.ts/index.ts/
bin/daemon.ts, cli daemon.ts) per repo parity guidance.
Note: auth_token and config.toml still key off Path.home(); left as follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(test): remove unnecessary parentheses in import to satisfy isort
* fix(daemon): route all .mirage paths through mirage_home and absolutize overrides
Complete the MIRAGE_HOME migration: the spawn log dir, auth token file,
and config.toml now resolve under mirage_home() instead of a hardcoded
~/.mirage, so daemon spawn works when $HOME is not writable. Relative
MIRAGE_HOME / MIRAGE_PID_FILE values are absolutized so the daemon and
later CLI invocations agree on one location. TS buildApp gains the
pidFile option (resolved once, mirroring py app.state.pid_file) and the
server barrel exports only the consumed path helpers.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* feat(shell): builtin specs for xargs/timeout/read, GNU echo/shift/return
* style: pre-commit formatting on new spec files, drop non-null assertions
* docs(bash): document builtin option support and GNU exit codes
* test(ts): mirror py parse_duration unit tests, satisfies knip export check
* test(integ): pin numeric shorthand and unknown short flag classification
* refactor(shell): ECHO_OPTION lives with SHELL_SPECS, echo's option surface is spec territory
* fix(ts): make core tsc --noEmit clean and gate typecheck in CI
* fix(ts): node typecheck clean; CI typechecks all packages
* chore(ts): noImplicitOverride + noFallthroughCasesInSwitch; knip config
* chore(ts): dead-code sweep via knip; gate knip in pre-commit; drop mirage-internal example refs
* chore(ts): re-strip dead awk_helper exports after main merge
Truth churn beyond the new cases is deliberate GNU alignment: ls
operands print as given, the glob_pattern_dup_word grep label is now
relative as typed, and prov_glob_unmatched gains one op because cat
now attempts the literal read.
The 22 bespoke core/<backend>/glob.ts copies of the same list-and-match
loop collapse to one-line bindings of their own readdir, matching the
Python factory adapter. This is what gives TS mount commands mid-path
globs, typed spelling, and zero-match literals in one place.
A glob in a non-final segment (s*/x.txt) expands segment by segment
instead of erroring; matches spell as bash implies (typed head +
matched tail); an unmatched glob word stays the literal so the command
errors on it like GNU (cat '*.nope' -> No such file, exit 1). Shell
consumers walk via the backend's own single-level resolve_glob, so no
backend needs mid-path support.
Cover all three cross-mount strategies in the two-mount missing-operand
integ checks: cat (STREAM), grep (FANOUT), diff (RELAY), each running
against ram, redis, and s3 destinations. grep pins single-mount
semantics: exit 0 on a match with the missing operand on stderr.
Move format_fs_error into the leaf utils/errors (both langs) and call it
from the single-mount executor catch and the cross-mount route catch, so
the GNU strerror line is formatted in exactly one place. Cover the live
RELAY path (diff/cmp missing operand) with unit and integ tests; that
path previously crashed on the removed PathSpec.display.
Cross-mount reads (cat/head/tail/wc/grep/rg) route filesystem errors
through handle_cross_mount's catch, which emitted only "<cmd>: <path>",
dropping the GNU strerror suffix the single-mount chokepoint appends.
A missing operand now reports
"cat: /ram/missing: No such file or directory", matching single-mount
cat and GNU parity, in both Python and TypeScript. Adds a focused unit
test plus a two-mount integ case where one path is missing.
Fixes#296
Co-authored-by: rikaqu0223-arch <266202640+rikaqu0223-arch@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- relative_spec: one constructor for typed word + cwd -> PathSpec
- raw_path is total (defaults to virtual); PathSpec.display deleted
- word_text/wordText: single str-or-spelling coercion helper
- rebase_display -> rebase_raw, nullable param dropped
- lazy-stream drain errors respell operands as typed via
ExecutionNode.paths (TS drain now also formats GNU lines)
- integ: relspell battery (walker labels, error spelling, ./ glob,
du trailing slash)
tree-sitter-bash hoists a trailing redirect over the whole &&/|| list,
so 'cd /data && echo hi > BARE' expanded the target before cd ran
(writing /BARE) and 'a && b > f' captured both commands' output. The
executor now re-associates at the redirected-statement branch:
redirected(list(L, op, R), r) runs as list(L, op, redirected(R, r)),
expanding the target only when R runs. Multi-redirect chains compound
correctly since each level re-associates independently; compound and
subshell bodies keep the whole-body redirect (bash group semantics).
The provision planner mirrors the re-association. Replaces the partial
list handling inside handle_redirect, which ran after eager expansion,
dropped the left side's stdout, and never ran the right side of a
semicolon list.
The Argv text view, for/select loop variables, and function positional
args used PathSpec.virtual, so 'echo sub/file.txt' printed the resolved
absolute path where bash hands programs their words unchanged. All
three now render PathSpec.display (raw_path as typed), and resolve_globs
stamps glob matches with the display the typed word implies: sub/*.txt
expands to sub/a.txt, absolute patterns keep absolute matches, and a
zero-match relative glob keeps the typed literal (TS withPrefix now
carries rawPath). Relative values re-resolve on use, so 'for f in
sub/*.txt; do cat $f; done' still reads the right files. New shared
truth pins test/echo/for/function relative-word behavior; diff additive.
'cd /data && test -f plain.txt' returned 1 even when the file exists:
-f and -d scoped string operands without the session cwd. Relative
strings now resolve through the shared resolve_path before the stat or
readdir probe, matching bash; empty operands stay false without a
backend call. Both langs.
The parser's private _resolve duplicated utils resolve_path; both langs
now use the shared primitive (TS resolvePath moves to utils/path with
the py argument order). This fixes the TS source builtin, which passed
its arguments in py order to the old (cwd, path) signature and so
resolved every sourced script to the cwd; py was already correct. New
shared-truth cases pin source. NUMERIC_SHORT joins AMBIGUOUS_NAMES in
spec constants.
A redirect target is a path by definition (the operator is the
context), so expand_redirects now uses classify_bare_path like the TS
side already did; classify_word left bare and extensionless relative
targets as text, sending 'echo hi > BARE' and '> sub/OUT' to the wrong
location. The provision planner gains the TS cmdsub guard so a target
hidden behind a suppressed substitution degrades precision instead of
costing a phantom write. Drops the planned blanket relative-path
heuristic relaxation as obsolete: spec PATH kinds already classify
extensionless relative operands, and relaxing the bare heuristic would
absolutize shell-consumer words, hurting bash parity.
The word rule is now an explicit WordPolicy enum (route/types) derived
from the consumer: SHELL words are always shell-resolved and spec hints
ignored; MOUNT words classify by spec with patterns kept for pushdown.
spec_for_command owns the cwd-mount/shared-SPECS lookup as the single
seam for the default-mount split, asserting that mount_for is total
instead of defensively catching. classify splits into a package
(heuristic/path/parts) so the shape heuristics have their own module.
TS mirrors, and posixNormpath/shlexSplit move from classify to utils.
Shell consumers get bash semantics: globs expand regardless of the command's spec, so spec TEXT kinds no longer suppress echo glob expansion. Mount commands classify by spec, falling back to the shared SPECS when the cwd mount does not know the command.
The parser records each argv position's operand kind, so the same word can be TEXT in one slot and PATH in another (grep '*.txt' *.txt: the pattern stays text, the file glob resolves). Spec-TEXT words never reach resolve_globs as PathSpec anymore, so its text_args guard is gone.
fanout/combine held only exit-code merging after the wc/du split, so it is now fanout/exit; the TS byte-join helpers move next to their only caller in fanout.ts.
Py StrEnum members compare and hash as plain strings, so executor call sites keep passing str; TS narrows once in route.ts and types the strategy layer as Cmd.
Whitelisted commands spanning mounts execute natively per operand on the owning mount (globs expand inside each native run) and combine: STREAM chains flagless cat into one stdin-mode run, FANOUT merges per-operand outputs (wc/du re-total, grep exit 2>0>1), RELAY keeps cp/mv/diff/cmp on dispatcher primitives. Non-whitelisted commands still refuse. Executor factors run_on_mount and injects it as the single-mount runner.
route/ now holds types (the Consumer enum), constants (the command-name
policy sets), and the route function, mirrored in both languages. The
scattered name sets move next to NAMESPACE_COMMANDS: JOB_BUILTINS from
the executor and NO_FOLLOW_COMMANDS from the links builtin. Import
path mirage.workspace.route is unchanged.
Shared-truth cases on every backend runner: zero-match echo keeps the
literal word, test -f sees the resolved match, function positional
args receive matches, unknown names exit 127, and ln with an expanded
multi-match source gets the GNU refusal. Truth regenerated; the diff
is purely additive (no existing case output changed).
A command belongs to the layer whose state it mutates. route(name,
session, registry) names that layer (SESSION, NAMESPACE, FUNCTION,
MOUNT, UNKNOWN) and drives both the dispatch branch and the word
policy: shell consumers get shell-resolved words in both argv views,
mount commands keep pattern PathSpecs for backend pushdown, unknown
names fail 127 before any backend work, and a zero-match glob keeps
the literal word (bash with nullglob off).
Fixed commands:
- cat /s3/logs/*.gz: the backend directory was listed twice per glob,
now once (backends resolve; the shell no longer pre-lists)
- echo /ram/*.nope: printed an empty line, now prints the literal word
- test -f /ram/*.md: stat'd the pattern (rc=1), now stats the match
- f() { echo $1; }; f /ram/*.txt: positional args held the pattern,
now the matches ($1 is the first match, $# the match count)
- ln -s /ram/*.txt /ram/lnk: silently created a link named by the
literal pattern, now the GNU multi-source refusal (target is not a
directory); a single match links to the match
- nosuchcmd /ram/x: exited 1 with "not supported on the
ResourceName.RAM backend" after a wasted readdir, now exits 127
"command not found" with no backend I/O
- rmdir /data (mount root): Python now refuses with the same Device
or resource busy message TS already used
The mount-root guard keeps precedence over the 127 path so protective
refusals keep their specific messages.
Move spec hints and glob resolution into workspace/expand
(classify_argv -> spec_hints with spec_word_kinds, resolve_globs ->
globs) so the word pipeline lives in one package, one stage per file.
expand_argv() builds a frozen Argv(name, args, operands); the dispatch
consumes named views instead of slicing word lists, and xargs/timeout
move to executor/builtins building their inner lines with shlex.join
(py) / shellJoin (ts).
Fixes:
- variable and quoted command names run ($E hi, "echo" hi;
COMMAND_NAME nodes now expand their child)
- xargs no longer drops initial args (xargs wc -l)
- xargs input words stay literal argv tokens (no $() execution, no
whitespace splitting)
- timeout preserves quoting of already-expanded words
- safeguards resolve against the expanded command name
- du, md5, and file with operands spanning mounts now route through the
cross-mount machinery like the other readers: each operand is stat'd
or read via dispatch-relayed primitives on its owning mount and the
shared generic formats the output, matching single-mount lines. du
totals come from the same stat/readdir walk the factory builder uses
for backends without a native du op.
- the planner shares is_cross_mount, so their plans now sum per-mount
estimates instead of flooring to unknown (xm_du_multi and
prov_xmount_md5/du battery pins)
- cp records client-streamed source bytes as reads in its IOResult, so
apply_io populates the file cache: a cold cross-mount cp warms the
source (cachev_xmount_cp_cold_populates then cachev_cat_after_cp
bytes=0 on the s3 suites)
- remove provisionOverrides from box gdrive databricks_volume langfuse
discord github_ci linear email hf: py passes none, so TS factory
commands (grep/rg/ls/du/find) now use the shared family estimators
(glob expansion, recursive walks, floors, py-matching op counts)
- delete the gdrive/databricks/hf provision clone files; databricks and
hf bespoke find/head rewire to shared metadataProvision and
makeHeadTailProvision
- keep the deliberately zero-cost bespoke estimators for rendered/API
backends (langfuse discord github_ci linear email box), mirroring the
py _provision modules
- byte estimators expand glob operands through the backend resolve_glob
(index-accelerated, same machinery the executor uses); unmatched
patterns stay honest UNKNOWN
- recursive grep/rg plans walk directories the way the executor does
(readdir/stat, columnar files skipped, capped at MAX_PLAN_WALK) so
grep -r over a tree prices exactly
- pathless invocations of read families (pipe stages, heredoc stdin)
plan as exact zero backend bytes: cat f | grep x | wc -l is now exact
- suppressed command substitutions degrade explicitly at the planner
(command operands, for-loop values, redirect targets) instead of
leaking a fake exact-zero plan
- TS provision walk rebases specs to the executor's resource view
before readdir/resolve_glob (backends expect mount-relative specs)
- fix TS RedisIndexEntry.file defaulting size to 0: readdir-listed
files poisoned index-first size lookups with a lying zero
- battery: prov_glob, prov_glob_unmatched, prov_grep_r, prov_pathless,
prov_heredoc, prov_for_cmdsub, prov_redirect_cmdsub; pipe pins flip
to exact
The Redis file cache store had no drain-task registry, so apply_io
silently skipped background drains: streamed cold reads populated the
RAM cache but never the Redis cache, and Workspace.close() crashed
with AttributeError touching _cache._drain_tasks on a Redis-cached
workspace.
The store now owns the same registry as the RAM store (py _drain_tasks
dict / ts drainTasks map), remove() cancels the pending drain for that
key, and clear() cancels all. Drained entries carry the record
fingerprint, matching the RAM store since the fingerprint threading in
apply_io. cache_limit gets a comment stating it is advisory on Redis
(no client-side LRU; cap via server maxmemory + allkeys-lru). Exported
applyIo/readFingerprint from the TS core index so node-package tests
can drive the fill path directly.
The add() EXISTS-then-SET race this makes more reachable is filed as
issue #438.
- case arms run every statement in both walkers: get_case_items returns the
full body statement list; executor chains stdout across statements and the
planner sums them per arm (py + TS)
- provision is truly dry: the plan walk gets an inert evaluator so command
substitutions expand to empty and degrade to unknown instead of executing
- sed provisioned as a file read via make_sed_provision (-i floors to
unknown) across 14 py and 12 TS backends; dify cat provisioned
- scaled(n) multiplies estimated_cost_usd so loop costs scale
- s3 example shows a user cost model: wrap cat's provision_fn to price GET
ops + egress and re-register; integ pins prov_cost_cat/for/unpriced_stage
- integ: case_multi_arm cases, prov_sed/prov_sed_inplace, prov_cmdsub now
net=0 unknown
apply_io stored every cache entry with the MD5-of-content default, so
under ConsistencyPolicy.ALWAYS is_fresh only matched simple-PUT S3
ETags; OneDrive (cTag), gdrive (remote_time), Postgres (sha256), and
multipart S3 uploads evicted and refetched on every warm read.
apply_io/applyIo now accept the command's op records and stamp each
cache fill (including the background drain, which looks the fingerprint
up after the drain since streaming backends record it lazily) with the
backend fingerprint. A warm re-apply with no backend read record and
unchanged bytes is skipped so it cannot clobber the cold-read
fingerprint with the MD5 default. Workspace.execute threads
scope.records through Dispatcher.apply_io in both Python and TS.
Tests: unit coverage on apply_io in both languages, plus a Python
end-to-end test that a warm read under ALWAYS with an unchanged
multipart-style (non-MD5) ETag serves from cache without a second
get_object. The shared S3 mock gains etag_suffix, per-method call
counters, and patch_s3_session.
s3_cases.ts was the one shared-truth runner left without a second
mount, so its /data2 paths fell through to the implicit root mount
and the cross-mount section diffed against the shared truth in CI.
Second key prefix on the same bucket, like the python twin.
Every shared-truth runner now mounts a second resource of its own
backend at /data2 (second bucket/tmpdir/subroot/key-prefix), and the
battery gains a cross-mount section (concat, cp, mv, links, grep,
find, pipes, cd across mounts; du's explicit multi-mount rejection is
pinned) plus provision pins for symlinked and mount-spanning commands.
run_cache_verify_cases (s3, gcs, onedrive) proves with backend byte
accounting that a second cat of a cached file pulls zero bytes,
directly or through a symlink (link and target share one cache
entry, cross-mount links included), and that provision reports the
hit.
Three bugs the coverage caught:
- the planner never namespace-followed paths, so provisioning a
symlink statted the link on the backend (unknown) and the cache-hit
check missed the target's entry; handle_command_provision now
follows links like the dispatcher, threaded from the workspace
namespace
- a command spanning mounts was estimated entirely against the first
path's backend, so foreign paths never resolved; paths now group by
their own mount (glob operands and unresolvable args ride with the
primary group) and the per-mount estimates sum, gated on
is_cross_mount so commands the executor rejects across mounts
(md5, du, file) still report unknown instead of costing a run that
errors
- the TS executor passed raw argv as the cross-mount text args, so
grep -c searched for the literal pattern "-c" (all counts zero)
and grep -n lost every match; the cross-mount path now parses
against the mount spec like python
The cross-mount cp with a warm source is pinned per language: it
reads through the cache but does not populate it, and TS records do
not count write bytes (known accounting divergence).
Both the executor and the provision planner walked the tree-sitter
AST with their own hand-maintained node-type ladders, and the planner
copy drifted (if crashed until recently; select was costed as a
bounded for; env prefixes and function calls fell to unknown; eval
was claimed free). This makes drift structural instead of accidental:
- shell/node_kind owns every node-type check, including the lookahead
that splits select from for and until from while. Both walkers
dispatch on NodeKind; unclassified node types are UNSUPPORTED for
both.
- redirect expansion (heredoc vars, target words, attached pipelines)
moves from an inline executor block to workspace/expand/redirects,
used by both walkers; the FOO=1 prefix split moves to
shell/helpers.split_env_prefix, used by command dispatch and the
planner.
- planner drift fixes: env-prefixed commands plan the command;
function calls plan the body (walk-local definitions plus
session.functions, with a recursion guard); eval and source are no
longer free builtins and report unknown; select plans like while
(unbounded); until gets an explicit branch; comments are free.
- redirect costing (closes#429): a < source is planned as a full
read of the file; > and >> to a mount bracket the write 0..inner
read high as a range (unknown when the inner plan has no ceiling);
stderr redirects, fd duplications, /dev targets, and heredocs stay
free.
- drift guard: a snippet-per-NodeKind table test in both languages
pins the planner output for every statement kind and fails if the
enum grows without a decision (tests/shell/test_node_kind.py,
tests/workspace/provision/test_node_coverage.py, and TS twins).
Battery: +8 provision cases (env prefix, function call and recursion,
eval, select, until, redirect in, /dev/null) and prov_redirect_out
now expects the write envelope; all other truth lines are unchanged,
which pins the executor refactor as behavior-identical.
cli.mdx was the only page documenting provision. Add an Estimate
Before You Run section to both quickstarts (execute with
provision=True / { provision: true }, ProvisionResult fields,
aggregation, one-line opt-in helpers) and a provision step to the
new-resource guide (factory commands estimate for free, bespoke
commands pass provision= to the decorator, rendered files must stat
size=None so estimates degrade to floors).
Shared battery gains 52 provision cases (families, combinators, complex
bash aggregation, graceful-default families) run identically on
ram/disk/redis/opfs/ssh/s3/onedrive/nextcloud in both languages.
Cache-flip cases (write-through, cold, read-through, partial) in
s3.py/s3.ts/onedrive.py; provision probes in
chroma/qdrant/lancedb/notion/dify/history/mongodb/postgres.
Fixes found by the coverage:
- if statements crashed provision in both languages (body node-list fed
to the single-node planner); branches now sum their condition ladder
with the body and bracket as alternatives, with a conditions-only
fall-through when there is no else
- TS Dispatcher.applyIo cached reads for every backend (missing the
isCacheablePath gate Python has), so provision reported phantom cache
hits on non-caching backends
- mongodb/postgres per-backend provisions claimed exact zero-byte
reads; both now bind the shared estimators
- Python provision stat errors outside FileNotFoundError blew up into
exit-1 results; estimates now degrade to unknown on any stat error
- stat joined the metadata family (one op, not zero); s3 bespoke
stat/du/rm/mkdir/touch and onedrive/nextcloud du wired to shared
estimators
Graceful defaults for previously unestimated commands: iconv (full
read), file (bounded range), gzip/tar/split/etc (read floor, unknown
output), cp (0..total on read and write), rm/mkdir/touch/ln/mktemp
(zero-byte op counts, recursive rm floors), seq/date/bc/expr (zero
cost) plus a first-mount fallback so pathless commands resolve. mv and
tee stay unknown deliberately.
OPFS and history migrate to shared defaults via withDefaultProvisions;
the old first-path-only provision generics are deleted. Integ provision
lines now include the write column; ram examples gain a PROVISION
section.
- reads (cat/grep/head/ls/cp/...) follow links before dispatch, including
mid-path directory links and targets on other mounts
- rm unlinks the entry (rm -r purges under a dir); mv renames it
- VFS ops follow too; cache keys under the real path
- ls surfaces links like child mounts (-F name@, -l name -> target)
- ELOOP renders as GNU 'Too many levels of symbolic links'
- integ: shared sym cases + truth, cross_commands symlink and cache
checks, parity.sh probes, chroma/qdrant/postgres/mongodb meta-file
links, history records commands as typed
- fold ("/x").strip("/") and stripSlash('/x') literals to their values
- fold empty-prefix mount_key/mountKey literal calls
- drop identity _normalize_path in chroma/dify find
- rename original -> virtual, as_typed -> raw_path
- remove prefix field and strip_prefix/key properties
- add required resource_path stamped by the mount at dispatch
- add strip_mount/mount_key/rekey/mount_prefix_of in utils/key_prefix
- mirror in TypeScript (resourcePath, rawPath, mountKey et al)
Backends hand-rolled the mount-prefix strip inline (an 11-line ritual
re-implementing logic that already exists as PathSpec.strip_prefix /
.key / dir.key). Replace every clean instance with the property.
- python: 44 backend read/stat/stream/readdir files
- typescript: 28 backend files (same families + box/dropbox)
- net ~-300 LOC of source; virtual_key reconstruction left untouched
where reused, so index-key normalization is preserved
Fixes two bugs:
- a PathSpec was left where a stripped str was expected, so
record(...) hit 'startswith' on a prefixed s3 read (reassign to
path.strip_prefix)
- the TS hand-roll omitted the path-boundary guard, so a sibling
like /database mis-stripped under a /data prefix; .key restores the
guarded result and aligns TS with python
Adds PathSpec.stripPrefix/key unit tests in both languages (boundary
guard, mount-root, no-prefix), which had no prior coverage.
Bespoke-shape backends (s3 ternary, github/discord/postgres variants)
are deferred to the prefix-field removal pass.
Port the Phase 2 symlink work to TypeScript so ln/readlink/cd resolve
through the Namespace addressing layer, matching Python.
- utils/path: resolveSymlinks + CycleError + MAX_SYMLINK_HOPS
- Namespace: LinkEntry table + symlink/readlink/unlink/rename/isLink/
symlinkTargets/replaceSymlinks + resolve(follow) following the table
- new executor/builtins/links.ts (handleLn/handleReadlink/linkFlags)
- thread namespace through executeNode/executeCommand deps; ln/readlink
router intercepts (ln gated on -s; readlink falls through for -f/-e/-m)
- cd follows the symlink table with -L/-P and ELOOP on cycles
- snapshot capture/restore of the symlink table
Both languages now produce identical output, so the shared integ
truth.txt is updated: ln -s is a real symlink (not a copy), so it no
longer shows in ls, and the ln_read_back case reads the target via
readlink. cat-follow stays a later phase.
Tests: path/namespace/symlinks + node ln/readlink; integ ram/disk/redis
verified byte-identical py and ts.
Make ln -s / readlink / cd-through-symlink resolve through the
Namespace addressing layer instead of the backend.
- utils/path: resolve_symlinks + CycleError + MAX_SYMLINK_HOPS(40),
longest-prefix substitution with verbatim relative targets
- Namespace: _symlinks table of LinkEntry{target, mtime}, with
symlink/readlink/unlink/rename/is_link/symlink_targets and
resolve(follow=True) that follows links before mapping to a mount
- executor: thread namespace through run_command_tree -> execute_node
-> execute_command -> _dispatch_command_body
- ln/readlink router intercepts (ln gated on -s; readlink falls through
to the native command for -f/-e/-m canonicalization)
- cd follows the symlink table with -L/-P, ELOOP on cycles
- snapshot capture/restore of the symlink table
cat/ls/glob follow-on-read and rm/mv routing through the namespace are
deferred to the next phase.
Make Namespace pure addressing (resolve + mount_for); pull the Dispatcher
back out as a standalone collaborator that calls Namespace for addressing.
Op execution and caching stay entirely in the Dispatcher, out of the
addressing layer. Workspace holds both and routes dispatch/apply_io/
invalidate through the Dispatcher. Behavior-preserving; py and ts.
Introduce a Namespace that owns the mount registry and the op dispatcher
behind one surface (resolve/dispatch/stat/readdir/apply_io/invalidate).
Workspace now routes dispatch, apply_io, and post-write invalidation through
it instead of holding the dispatcher directly. resolve takes a follow flag
(no-op until the symlink table lands). Behavior-preserving; mirrored in
Python and TypeScript.
The TS mongodb integ was missing the multi-pattern grep -r case that the
Python side and shared truth_mongodb.txt already expect, failing
integ-database-ts.
The existing grep_e_multi case greps a file (documents.jsonl), which on
mongodb falls through to the generic grep and was never broken. The actual
bug is at directory scope, where the search pushdown fires. Add a grep -r
-e alpha -e beta case over the collection directory with distinct,
absolute-path-prefixed truth lines; it returns empty without the guard and
the 4 rows with it.
mongodb, email, langfuse, discord, slack and gmail push a single pattern
to a native search / DB query (regex, ILIKE, coarse filter, search API).
With multiple -e patterns, pattern_arg joins them with a newline and the
pushdown treats the whole thing as one literal, matching nothing. A warm
cache masked this (the generic grep handles -e); cold reads exposed it.
Guard the pushdown with a single-pattern check so multi-pattern greps fall
through to the generic grep in both Python and TypeScript. github is already
safe (pattern only narrows paths, falls back to a full glob). Adds a discord
regression test that fails without the guard.
TS du walk fallback: generic du builder now walks stat/readdir when a
backend has no native du op (mirrors Python du_multi), and du dropped from
13 override sets (chroma, databricks_volume, github_ci, gmail, lancedb,
notion, linear, langfuse, mongodb, qdrant, email, postgres, trello).
GNU/coreutils convergence:
- TS S3 ls -l/stat carry mtime on warm (index) reads
- PY diff/cmp usage error exit 2 (UsageError), not 1
- TS diff -u GNU single-line hunk header (@@ -1 +1 @@)
- PY head/tail invalid -n/-c GNU message + exit 1 (number_flag_error)
- TS rg -c omits zero-count files (nonzeroCountStream)
- PY+TS du -h / ls -lh human sizes float, not floored (shared humanSize)
Backend bugs:
- TS discord/slack stat map mime via shared filetypeFromMimetype
- TS hf rm -r <file> unlinks (was wrongly rejected); dir errors mirror PY
- TS linear/trello find honor -type and include the search root
- PY s3 stat on a trailing-slash dir prefers the prefix over a same-name object
- TS ssh registers filetype commands + wires provision; append invalidates cache
Integ: du cases for lancedb/qdrant/chroma (byte-identical py/ts).
Dead code: remove chroma/grep.ts, ram/mkdir_p.ts, provision/builtins.ts,
resolve_first_path, strip_prefix_from_path_kwargs.
Exercise cd across mount boundaries: mount-to-mount hops, cd / above all
mounts, relative .. across the boundary, cd -, // collapse, -P/-- on a
cross-mount target, and $CDPATH spanning two mounts. Runs on every dst leg
(ram, redis, s3).
- collapse leading // in python resolve_path (cd //x -> /x)
- accept -L/-P/-e/-@ options, -- terminator, and clusters; unknown -> exit 2
- treat quoted '~' as a literal directory, not $HOME
- error on too many arguments (exit 1)
- unset $HOME: bare cd errors (HOME not set), ~ stays literal, $HOME expands empty
- support $CDPATH search (empty entry = cwd; non-empty hit prints resolved dir)
- integ cwd cases + truth.txt coverage for all of the above
The cache-as-store refactor adds an empty root anchor at / when no user /
mount is given. summary._userMounts filters only /dev + history (matching
python summary.py), so the root anchor now counts as a user mount. Update
the summary tests to expect it (mount_count=2: /data/ + /), matching
python's make_brief/make_detail behavior.
Mirror the Python cache-as-store architecture (#401 + virtual-root
follow-up). The file cache was simultaneously the resource of the
/_default/ mount and a hidden store; decouple them so the cache is only a
hidden store reached via registry.fileCache, and arg-less commands / root
listing resolve against a plain empty root anchor (the user's / mount if
given, else an auto-added empty RAM mount at /). Removes setDefaultMount,
the defaultMount getter, the /_default/ namespace, and the isFileCache
redirect guard; mountForCommand and ALWAYS-eviction now use the root mount
and the hidden store directly, matching python/mirage/workspace/mount.
The synthetic root anchor is not forwarded into the Pyodide filesystem
(it would hijack Pyodide's own / holding the stdlib); a user-provided /
mount still is. This is the one TS-only detail (Python has no Pyodide).
Behavior is unchanged: core/node/browser suites green and every offline
integ matches truth (ram/disk/redis/s3_cases/cross + s3 native warm-serve).
Also mirror the missing Python cache tests to TS:
- cache_invalidation.test.ts (write-site invalidation + cat/rm eviction),
mirroring test_cache_invalidation.py and test_cache_write_through.py
- lock.test.ts, mirroring test_lock.py
- rewrite cache_mount.test.ts to assert the cache/root decoupling
(mirroring test_cache_mount.py)
The s3 patch override carried its own hunk applier. The python copy was
broken: it spliced additions at the hunk start instead of walking context,
so any hunk with a leading context line corrupted the first line. Drop the
s3 patch override in both languages so s3 (and its aliases) use the shared
generic patch, which walks context correctly.
Add a patch round-trip to the shared integ cases (apply, forward-only -N,
reverse) so every run_cases backend (ram/disk/redis/ssh/s3/onedrive) now
exercises real hunk application, not just the no-input path.
Also guard databricks head -c against negative counts so a negative -c
falls through to the generic head instead of a meaningless prefix range
read, mirroring the typescript guard.
Generic ls fell back to stat only when readdir threw. Backends without
real directories (s3) list the "<file>/" prefix and return empty rather
than ENOTDIR, so 'ls <file>' / 'ls <glob>' printed nothing. Mirror the
Python ls empty-readdir fallback: stat the operand and list it when it is
a non-directory. Add shared ls_file/ls_glob integ cases (py + ts).
du/duAll listed with the directory prefix (trailing slash), so a file
operand's key never matched and du reported 0. List with the stem (no
trailing slash) and count an object only when it equals the operand or
lives under it, mirroring the Python s3 du.
Add the warm-serve cases to s3.ts, mirror assert_real_mtime into the
ram/disk/redis/ssh drivers, and add cross_commands.ts. Fix the TS
cross-mount path it surfaced: thread parsed flags into handleCrossMount
and implement recursive cp/mv (no-clobber, omitting-directory, verbose)
plus the wc total line, matching the Python crossmount behavior.
Mirror python #378. Add CacheManager.cachedBytes + read_through.ts
(cacheAwareReadStream/ReadBytes, path-keyed cacheAwareStream/Eager,
cachedPrefixBytes). The generic_bind factory wraps read ops with
withReadCache and metadata ops with withStatCache (size filled from the
cached blob); the shared consumers (grep/rg/head/tail/wc) wrap their
injected reader at the choke point so bespoke commands warm-serve too.
databricks head gains a cached-prefix / range-read -c fast path. The
resolveMount cache-mount redirect is removed: a read-only command now
stays on its real mount and serves cached bytes in place (keeping the
ALWAYS-consistency stale eviction), so per-mount stat/safeguards no
longer get bypassed on warm reads. Adds read_through + cache_uniform
unit guards.
The four HF resources (buckets/datasets/models/spaces) now build their
filesystem commands from makeGenericCommands, called once per resource over
HF_RESOURCES with the shared HfAccessor ops. du/sed/find/rm stay bespoke
(du contract, no generic sed builder, index-threaded find, rm records
cache-invalidation keys); cp/mv are suppressed since HF has no copy/rename
op. The exact-command-set assertion becomes a superset check; the
all-four-resources invariant test is unchanged.
notion/box/dropbox/email now build their filesystem commands via
makeGenericCommands. Platform write/search commands (notion_*), search
push-downs (find/grep/rg), sed, and the email_* surface stay bespoke.
box/dropbox thread du through duTotal/duAll ops; find stays bespoke
everywhere (threads the cache index); du is suppressed where there is no
native du op. The per-backend bespoke-awk no-input test is removed now
that every awk is factory-generated (covered by the generic awk suite).
github/github_ci/linear/trello/langfuse/gmail now build their filesystem
commands via makeGenericCommands. Search push-downs (find/grep/rg), the
github du (flat-list contract) and sed, and the platform write/search
surface (linear_issue_*, linear_search, trello_card_*, gws_gmail_*) stay
bespoke. find is kept bespoke everywhere (threads the cache index, which
the generic FindOp can't); du is suppressed where there is no native du op.
mongodb/postgres/chroma/qdrant/lancedb/databricks_volume now build their
filesystem commands via makeGenericCommands. Search push-downs (find/grep/
rg/search), head/tail/wc LIMIT push-downs, sed, and the databricks write
commands that record cache-invalidation keys (mkdir/touch/rm) plus its
index-threaded find stay bespoke. du is suppressed on backends with no
native du op (TS generic du/find lack Python's readdir-walk fallback).
Stale exact-command-set tests become superset checks.
gdrive/gdocs/gsheets/gslides/slack/discord now build their filesystem
commands via makeGenericCommands; bespoke API commands (gws_*, slack_*,
discord_*, rm), search push-downs (grep/rg/find[/head]) and sed are kept.
Co-located command tests source the generic command from each backend's
COMMANDS instead of a deleted wrapper.
Subdir scopes and regex patterns (via an extracted required literal) now
narrow through GitHub code search instead of fetching every file. grep/rg -l
short-circuits to the narrowed file list without reading content; rg applies
its hidden/--type/--glob filter to that set. Shared narrow_scope helper used by
both grep and rg. Addresses #404.
Listing a Databricks volume directory cost 1 list call plus one
get_metadata per entry (and a second call per subdirectory), even though
list_directory_contents already returns size, mtime, and is-directory for
every entry. ls, ls -l, tree, find, and glob all paid this N+1.
readdir already filled the per-resource index with each entry's size and
type but dropped the modified time, and stat never read the index back, so
the generic ls/tree/find walkers re-fetched metadata per entry. Adopt the
existing s3/nextcloud idiom:
- readdir now stores each entry's modified time (remote_time), converting
the listing's epoch-ms last_modified to the same ISO string stat emits.
- stat consults index.get() first: a cached entry returns size/mtime/type
with no API call; a listed-but-absent path returns ENOENT without a
probe; SDK fallback on miss. exists/cp/rm/rmdir benefit too.
- Generic walkers are untouched and stay backend-agnostic.
- TS mirrored; the TS find wrapper now threads opts.index into the walker
(Python already did) so the cache is actually used.
Output of ls/ls -l/ls -t/ls -S/tree/find -type/-size/-mtime is unchanged;
a directory listing needing per-entry size/mtime/type now costs one list
call instead of 1+N+D.
Mirror GNU sed -f: read the script from a file, joined with -e, with the
positional reflowing to a file path when -e/-f is present. Adds the -f
option + provided_by/-f to the sed spec in both languages. Shared integ
cases (cases.py/cases.ts) + truth, native -f tests, ram -f unit tests,
and a sharepoint example demo.
Ops.is_mounted backed the open()/os patches' decision of workspace-path vs
real-OS-path via _resolve, which the catch-all virtual root makes succeed
for every absolute path. That routed real filesystem reads (a FUSE
mountpoint, /tmp) into ops and broke the fuse integ. is_mounted now matches
only explicit mounts, skipping the bare / root; ops still route to the root
via _resolve, this gate is only about what the interception leaves alone.
It is the declarative form of grep's if(!pattern_given) getopt check; the
same scenario clap names required_unless_present and docopt expresses as
alternate usage patterns. Lives in the spec because Mirage classifies args
before a backend is chosen, so per-command imperative parsing does not fit.
Promote the root from an out-of-_mounts anchor to an ordinary mount at /.
When the user mounts / (programmatically or via YAML) that resource is the
root; otherwise an empty RAM root is mounted internally. The root is a
normal catch-all entry (last in longest-prefix order), so resolution, ops
routing, ls /, and snapshots treat it uniformly. is_exec_allowed drops its
/ short-circuit (any non-dev EXEC mount enables exec); unmount unregisters
ops by kind so a same-kind unmount cannot strip the root's ops.
Surfaced and fixed latent spec bugs that were masked when cwd=/ resolved to
no mount: csplit -f is a PATH (not TEXT), sed's script positional is
provided_by -e, and single PATH-flag values are recovered to PathSpec like
positionals so a cwd-resolved relative flag path still gets prefix-stripped.
Adds integ/root.py covering ram/disk/yaml root switching.
The hidden file cache no longer needs the /_default placeholder mount.
Rename it to the virtual root anchored at /: set_root_mount / root_mount,
kept out of _mounts so it stays the neutral fallback for root listing
(ls /) and arg-less command resolution without shadowing real mounts in
longest-prefix routing. The session layer already treated / as the cache
root with exact-match allowlisting, so the move is safe. Also drop the
dead cache_mount property and the now-redundant default-mount guard in
the command executor.
The read cache used to BE the /_default mount's resource. Now the default
mount is a plain empty RAMResource and the cache is a hidden store reached
via registry.file_cache: cache-hit accounting and ALWAYS-eviction reference
it directly instead of the default mount's resource. The /_default anchor
stays (empty, cache-free) because root listing (ls /) and arg-less command
resolution need a neutral base mount; only the cache's mount identity is
removed.
Trims test_cache_mount.py to the read-through behaviors plus a decoupling
test.
Removing the cache-mount redirect (#378) exposed a latent file-case bug
the redirect was masking: du/du_all on ssh ran sftp.readdir() on a file,
which raises, so 'du <file>' produced empty output instead of the file's
size. Stat-guard both so a non-directory operand returns its own size,
mirroring the nextcloud/onedrive/sharepoint/hf fix. Covered by integ/ssh.py
(du_multi, nl_pin_du) against the shared truth.txt.
These backends have no mock integ server, so the examples verify caching
live: cold vs warm timing on cat/grep/head/tail/wc over one file. Live
runs show ~450x (github) and ~90x (langfuse) warm speedups.
cross_commands: cross-mount read family (cat/head/tail/wc/grep) and glob
multi-file serve from cache under LAZY via out-of-band mutation.
s3: per-mount warm-serve cases asserting a warm read pulls 0 backend bytes.
Move warm-read serving into generic_grep/generic_rg/head_multi/tail_multi/
wc format_multi so any command (generic or bespoke) serves cached bytes
without each call site remembering to wrap its reader. head_multi/tail_multi
become sync factories that capture the cache manager in-scope (eager capture)
so a lazily-drained stream still serves the warm read after the command's
cache-manager scope is gone. dify cat and databricks head -c inject
cache-aware readers directly; bespoke grep/rg/head pass raw readers.
Add a consumer-level runtime guard, a cross-mount cache-serve test, and
warm-read drain cases to the dify and onedrive integ suites.
Warm-read serving lived only in the factory ops wrapper, so bespoke
read commands (dify cat, databricks/discord head, grep/rg fallbacks on
slack/github/email/gmail/langfuse/discord/github_ci) read raw and hit
the backend even when the file was cached. Extract the cache-aware
reader wrappers into mirage.cache.read_through and have every read path
inject a wrapped reader. Read-then-write commands (sed -i, tee) keep
reading fresh. Adds a uniformity guard test that fails if a read-only
command on a caching backend injects a raw reader.
Removing the redirect exposed latent file-case bugs the cache mount was
masking: du walked a file as a directory (returning 0) and the Graph
backends' readdir 404'd uncatchably on a file. Stat-guard du/du_all in
onedrive/sharepoint/nextcloud/hf_buckets and convert readdir's 404 to
NotADirectory/enoent in onedrive/sharepoint so the generic ls falls back
to stat. Adds file-case unit tests; corrects truth_onedrive.
with_read_cache serves cached content and with_stat_cache fills
render-dependent size on the real mount, so read-only commands keep
their backend handlers instead of being redirected to the cache mount.
Builder gains an explicit read flag.
Read commands get a cache-aware ops adapter from the factory: a warm
read serves from the file cache via CacheManager.cached_bytes and is
declared in the command's IOResult. The cache manager is captured
eagerly when the ops read is called (inside the command's manager
scope), not lazily at stream-drain time when that scope is gone.
- Builder gains a read flag; 36 file-content readers set read=True
- factory wraps read commands with with_read_cache
- the warm/cold mount redirect is kept only for bespoke backends
(FACTORY_READ_RESOURCES gate); factory backends (s3) serve in place,
keeping their real mount's safeguards and custom handlers
- fix s3 du for a file path: it listed children of a trailing-slash
prefix and returned 0; now counts the object itself plus children
Importing Disk/RedisObserverStore in observe/__init__ pulled aiofiles/redis
into the eager 'import mirage' path, breaking minimal install and import
isolation. They stay reachable via direct module import.
TypeScript:
- Add startBasename + emitStartPath to findEval (mirror Python); export
from the core barrel.
- Fix chroma/notion mount-root name (root mapped to / gave an empty
basename) and github/hf root emission (gates suppressed the mount root).
- Route ram/disk/redis/s3/ssh/opfs root emission through emitStartPath.
- Add github/find + notion mount-root tests; update hf find tests.
Integ:
- Add find -maxdepth 0 and find -name <root> cases to chroma/dify/notion
harnesses (py + ts) and regenerate truth, so the mount-root fix is
exercised against the live/mock servers.
- Move FindArgs dataclass into find_eval (leaf) and drop the TYPE_CHECKING
back-import from generic/find, removing the circular dependency.
- Add start_basename(path) as the single source for the start path's name
and replace the 5 divergent per-backend derivations with it.
- github: filter a file start path by -size; drop the redundant base==""
clause; set root is_empty=None (github does not support -empty).
- sharepoint: drop dead split_path call; add find unit tests (mock Graph).
- Add github file-start unit tests.
Route ram/redis through emit_start_path (size param added for file
roots) and fix chroma/dify/notion mount-root name (root mapped to /
gave an empty basename, so -name <root> never matched). Add mount-root
unit tests; update stale hf_buckets tests to the root-emitting output.
#397. github now emits the mount root (was never emitted); hf_buckets and
sharepoint un-gate the mount-root case. All three route through
emit_start_path with the start path's display basename. Update github unit
tests for the now-included root.
WIP for #397. Introduce emit_start_path in find_eval (owns the start-path
keep/depth/-maxdepth 0/-mindepth 0 logic) and route the 5 block-based ops
through it. Byte-identical integ for disk/s3/onedrive/ssh; nextcloud mirrors
the same pattern.
Public per-mount Mount(resource, mode=, fuse=) spec replaces the
workspace-level fuse param. Workspace owns a per-prefix FuseManager
registry (add_fuse_mount/remove_fuse_mount); FuseManager is a passive
mounting primitive; on the TS side FUSE lives entirely in the node
Workspace. mfusepy is an optional import so the base install imports
without the [fuse] extra, and the py import cycle is broken (the FUSE
layer takes Ops, not Workspace). Collisions are rejected before mounting
and a setup failure rolls back cleanly. Adds real-libfuse integ + CLI
end-to-end coverage and a 30m timeout on the interop job.
Closes#312. Aligns find with GNU coreutils across every backend and both
languages:
- B1: emit the search start directory at depth 0 (subject to predicates) in
the generic walker and each per-backend find (ram/disk/redis/s3/onedrive/
ssh/nextcloud in py; ram/disk/opfs/redis/s3/ssh in ts); fix root-key
rebasing in the generic find command.
- B2/B3: -empty and -not negation.
- B4: unknown/unsupported predicates are a hard error (exit 1) instead of
silently matching all.
- Fan-out: drop a descendant mount's own root line; the mount point is
synthesized centrally with its display name and the predicate tree.
Regenerate integ/truth.txt and add unit + integ regression coverage.
Closes#383. When a command's path operand resolves to a concrete mount that
does not register that command, raise MountCommandUnsupported instead of
silently falling back to the /_default/ mount (which reported a misleading
"No such file" for a file that exists). Path-less general commands still fall
back to /_default/. Adds registry regression coverage.
The telegram and paperclip backends were already removed from the codebase;
paperclip had no remaining references but telegram lingered in docs, config and
the TS ResourceName enum. Drop it from the READMEs (all mirrors), .env.example,
the general command spec resource lists, and the TS ResourceName enum + tests
(count 48 -> 47). The architecture SVGs keep telegram as an illustrative label.
Migrate databricks_volume, hf_buckets and sharepoint to make_generic_commands.
sharepoint mirrors onedrive (filetype commands, dir_copy folder copy, du
flat-list override, sed); databricks keeps head (byte-range pushdown for -c) and
sed; hf_buckets wires its partial write ops and skips cp/mv (no server-side
copy/rename op). The cp builder now falls back to a readdir-walk find when a
backend has no native find op, so recursive cp works without one.
Migrate github_ci, langfuse, trello, linear, github and email to
make_generic_commands. Each keeps only its genuine overrides: find/grep/rg
where they push down to a search API or guard recursive cross-scope reads,
platform write commands (trello_card_*, linear_issue_*, email_*), github du
(du_multi flat-list) and sed, plus email's filetype commands for columnar
attachments. All remain read-only (no write op wired).
Generic head/tail/wc read the whole relation, which trips the postgres
too-large-read guard on big tables and loses the server-side count for
wc -l. Restore the bespoke wrappers (LIMIT/OFFSET pushdown for postgres
head/tail, COUNT(*) for postgres/mongodb wc -l).
Migrate the read-only database backends (mongodb, postgres, chroma, qdrant,
lancedb) to make_generic_commands. Bespoke overrides kept per backend:
- mongodb: cat (path-dispatch documents vs .json metadata), find/grep/rg
(query pushdown), tail (-f change streams)
- postgres: find/grep/rg (SQL pushdown)
- chroma: find (path normalisation), search (query API), sed (no builder)
- qdrant/lancedb: find, search (query API); grep/rg are pure generic
qdrant/postgres/lancedb have no native streaming read, so the stream op is
synthesized from the whole-row read.
Migrate the read-only SaaS backends notion and dify to make_generic_commands
(gmail/gdrive pattern). notion keeps only the bespoke notion_* writers; dify
keeps cat/find (avoid an extra document-detail API call), search (retrieval
pushdown), and sed (no generic builder).
Also fix the generic stat builder to thread the cache index into generic_stat,
so index-backed backends resolve directory mtime on a cold call (notion stat
-c '%y' returned empty before).
Replace the ~63 hand-written command wrappers in nextcloud and onedrive with
make_generic_commands, mirroring the ssh migration. Keep sed (no generic
builder) and du (its du_all returns a flat list, the du_multi contract) as
overrides, matching s3.
Add an optional dir_copy op to CommandIO so the generic cp builder preserves
onedrive's server-side folder copy; other backends leave it None (per-file
copy, unchanged).
Index-backed backends (gdrive/gmail/slack/discord) failed cold reads with
'No such file' because generic_bind builders dropped the cache index when
handing read/stat ops to the generics. Wrap read_bytes/read_stream/stat_fn
with with_index where the generic calls them without an index (grep/rg keep
bare read_bytes since call_read_bytes threads it).
Also fix the Google vfs examples (wrong path joins, gmail date descent) and
gmail_dates/sheets examples (real layout, write mount).
The skip is the only essential part; the separate unsupported_commands helper,
gaps set, and info log were ceremony. Inline 'b.write and ops.write is None'
in the generation loop.
- gdrive's native read_stream is a coroutine (Workspace-aware); synthesize the
stream op from the whole-file read like gdocs/gmail
- generic read_stream was called without index, breaking index-backed backends
(gdrive/gmail/...); bind the runtime index via with_index in cat + 14 builders
(None-safe so read_bytes-only callers/tests still work)
- update slack/discord resource command-count asserts for the grown read set
- cat-streaming spy: accept the new index arg
A read-only backend wires no write op, so the factory skips every write=True
builder in one check (ops.write is None) instead of per-command op tuples.
Same skip set (17 write commands) for read-only backends, none for full-FS.
Replace hand-written per-backend command wrappers with make_generic_commands:
- ssh: full read-write op set, only sed kept (no generic builder)
- gdocs/gsheets/gslides: read-only to the factory, bespoke gws_* + rm kept
- gdrive: read factory + filetype + bespoke sed + 10 gws_* writers
- gmail/slack/discord: read factory; bespoke search push-down grep/rg,
channel-aware find, discord history head, and platform commands kept
Factory gains op-requirement auto-skip (unsupported_commands), readdir-walk
fallbacks for find/du when a backend has no native op, and an rm -r/-d guard.
~200 wrapper/obsolete-test files deleted; registration parity verified.
generic_cp/generic_mv gain a primitive mode (gated on no native copy/rename):
they walk via injected readdir/stat and mkdir dirs / write read_bytes for files,
so any primitive-only caller (cross-mount) reuses one tree implementation.
Backends keep their native copy/rename fast path untouched.
Cross-mount ops.py is deleted. The crossmount handlers are now pure wiring:
primitives.py exposes just relay (dispatch->data, the one str->PathSpec coercion
point) and stream (bytes->iterator for the read family); transfer/read/compare
build dispatch-relayed primitives and hand them to the generics. The compound
_copy/_walk/_find/_rename/_is_dir are gone. write takes bytes (not a stream), so
a cross-mount file copy materializes once -- the documented barrier.
Replace _fetch/_read_bytes/_stat/_readdir with a single _op(dispatch, op, ...)
that routes any primitive (read/stat/readdir/write/mkdir/unlink/rmdir) and is
the sole str->PathSpec coercion point. Delete _copy_tree/_remove_tree; mv now
copies via the same find+copy as cp then removes bottom-up, using dir/file
types captured by one _walk so a vanished S3 virtual dir is never re-stat'd.
Replace the bare 'except FileExistsError: pass' with an explicit _is_dir guard
so a real target conflict still raises. Drop the DispatchIO.reads field; cat
collects its own reads.
Relocate the cross-mount layer beside the generics it delegates to. Split the
old 'transfer' (cp/mv/diff/cmp) into transfer.py (cp/mv) and compare.py
(diff/cmp); aggregate -> read.py; adapter -> ops.py; router -> route.py.
Every __init__.py is re-exports only; all implementation lives in named
modules.
Delete the _served_stream/_served_bytes/_served_stat/_served_readdir shims and
the eager-read buffer. They reimplemented what DispatchIO already provides; the
read family now passes io.read_stream/read_bytes/stat/readdir to the generics,
exactly as every backend's read builder passes ops.*. grep over a directory
operand now warns 'Is a directory' and continues (GNU) instead of aborting.
Cross-mount cp/mv/diff/cmp and the multi-file read family (cat/head/
tail/wc/grep) now delegate to the same generic commands every backend
uses, with paths read/written on their owning mount through dispatch.
Deletes the hand-rolled cross_mount.py (split into cross/ package:
detect, adapter, read, router).
Also fixes two GNU divergences the old code had: cross diff emitted
unified output even without -u (now normal format by default), and
generic cp gains the missing 'omitting directory' error for a
non-recursive directory source.
Adds integ/cross_cp.py to the Integ CI: exercises cross cp -r, -rn
no-clobber, mv -r, and the read-family aggregation for ram->ram,
ram->redis, and ram->s3 (moto).
Migrate gzip/gunzip/zcat/tar/zip/unzip/cmp/csplit/diff/mktemp/patch/shuf/
split/tsort to generated builders (ram/redis/disk/s3); sed and s3 patch
stay as overrides. Wrap builder descriptors in a Builder NamedTuple so the
factory reads by attribute instead of tuple position.
- split the grouped builder modules into one file per command
(generic_bind/builders/<cmd>.py, each exporting a BUILDER descriptor)
- rename CommandIO.ready -> is_mounted, scope_cap -> max_glob_matches
- replace **_extra: object sinks with **kwargs
Define the default command set once in commands/builtin/generic_bind and
generate each backend's COMMANDS from a CommandIO adapter instead of
shipping ~63 near-identical wrapper files per backend.
- ram/redis/disk generate 48 commands each from one CommandIO
- s3 generates 42; keeps 6 remote-specific overrides (stat/du/rm/mkdir/
tee/touch) plus provision_overrides for grep/rg/ls/find
- ~186 wrapper files removed, net ~-8.4k LOC
_diff_pair/_diff_dirs took 10 and 13 positional params (i,w,b,e,u,q
threaded individually). Collapse the six booleans into a frozen
_DiffFlags dataclass, matching the TS DiffFlags struct and the repo's
'generics parse flags into a frozen struct' convention. Behavior
unchanged; integ truth.txt identical.
The 'Create MinIO bucket' step curl'd mc without -f, so a transient bad
redirect from dl.min.io wrote a non-binary to /tmp/mc and execution failed
with a shell syntax error. Use curl -f, retry up to 3x, and validate the
binary with --version before use.
Self-review found diff -r <file> <file> raised 'Not a directory' (GNU
treats -r as a no-op for file args). Gate recursion on both top-level
paths being directories; otherwise fall back to a plain pair diff.
Add file-file regression tests (py + ts).
Mirror the TS generic (optional stat); recursion runs only when both
readdir and stat are injected. Unbreaks direct callers of the generic
diff() that pass read_bytes/readdir_fn only (test_phase_z).
Replace the shallow top-level-files-only diff -r with a real recursive
walk matching GNU: descends into common subdirectories, emits
'Only in <dir>: <name>' for entries unique to one side, a
'File <a> is a directory while file <b> is a regular file' line for
type mismatches, and a 'diff -r <a> <b>' header per differing file.
- inject stat into the generic diff (11 py + 12 ts backend wrappers) to
distinguish files from directories across backends
- add recursive/only-in/identical-tree tests (py across ram/s3/disk, ts
across ram/disk)
- expand the integ diff_recursive case (subdir + only-in) and regen the
shared truth.txt; verified byte-identical py/ts on ram+disk
Runs diff -r over two directories on every backend in both languages,
diffed against the shared truth.txt, locking in PY/TS parity for the
recursive-diff fix. (tar -j/-J stays at unit level: the shared battery
also runs on opfs/browser where bzip2/xz are intentionally unsupported.)
compressjs and @napi-rs/lzma are CommonJS; named ESM imports failed in the
built dist under Node's native loader (vitest's transform masked it),
crashing the node package at import. Load them with createRequire instead,
matching workspace.ts.
- diff -r: fix recursive basename comparison in the py generic and
implement recursive diff in the ts generic, wired across all backends
- tar -j/-J: add real bzip2/xz via a core codec registry; node registers
compressjs + @napi-rs/lzma, browser stays unsupported
- (( ... )): surface the arithmetic command as exit 1 instead of crashing,
matching the python workspace error handling
- find -size: align the s3 test with the committed dirs-pass-minSize semantic
- gdrive cp -r / mv -r: invalidate the index in the cross-mount harness
after out-of-band seeds (mirrors mirage write-side invalidation)
- multi-session cwd/env isolation + history: un-skip (already supported)
- patch: drop stale s3/disk skip
- ci: add mongodb service + MONGODB_URI so mongo-gated pytest runs
Extract the duplicated Microsoft Graph transport into a neutral
core/msgraph module (config + client), mirroring core/google. SharePoint
now re-exports the shared transport and keeps only its own drive-id
addressing. OneDrive is left untouched.
Add SharePoint setup and Python resource docs.
Mock now serves a filter-aware search, GET /databases/{id}, and
POST /databases/{id}/query in both the Python REST mock and the TS
REST+MCP mocks. Adds ls/cat/jq cases for /databases, database.json,
and a row page; regenerates truth_notion.txt. py/ts output byte-identical,
MCP parity 27/27.
- Resolve merge with current main (Notion connector refactor: enoent
helper, list_block_tree rename, native scope pushdown removed)
- database.json carries metadata + typed property schema only (no inline
rows); rows are the row-page directories
- Dedup page-subtree traversal so /databases/<db>/<row> reuses the page path
- Fix readdir warm-cache dropping the mount prefix in notion/linear/trello
- Update docs + examples for Notion database support
Importing mirage evicts the integ dir from sys.path, so onedrive.py's
'from onedrive_server import ...' (after 'from cases import ...') failed
with ModuleNotFoundError. Revert cases.py to the 'Jan 1 00:00' literal
(EPOCH_LS_TIME stays defined in formatting.py for the command code).
- add direct tests: epoch_to_iso (test_timeutil.py) and epochToIso +
utcDateFolder (new dates.test.ts).
- both helpers now truncate to whole seconds, so py and ts emit
byte-identical second-precision ISO-Z for any input (previously py kept
microseconds, ts milliseconds); slack call sites drop the now-redundant
int()/Math.floor().
- timeutil.epoch_to_iso (py) + utils/dates.epochToIso (ts) wrap the
shared 'unix epoch seconds -> ISO-Z' conversion.
- use it in slack, discord, disk, ssh stat (py) and slack/discord (ts),
replacing inline to_iso_z(fromtimestamp(...)) / new Date().toISOString().
- snowflake_to_iso stays in discord (it is Discord-specific) but its
final step now goes through the shared helper.
- _slack_modified/slackModified treat a non-positive epoch as unset
(None) so a channel missing 'created' (stored '0') shows no mtime
instead of a 1970 date.
- slack/discord converters reuse to_iso_z instead of hand-rolling the
+00:00->Z replace (py); discord reuses the exported DISCORD_EPOCH
instead of a literal / re-declared const (py + ts).
- snowflake/epoch -> ISO now emit whole-second precision in both
languages, so stat/ls -l strings match across the Python and TS CLIs
(previously py kept microseconds, ts milliseconds).
- extract the 'Jan 1 00:00' epoch ls sentinel to EPOCH_LS_TIME in
formatting.py; the integ mtime guard references it instead of
re-typing the literal.
- integ/s3.py frozen mtime is tz-aware UTC.
truth_notion.txt is shared by integ/notion.py and integ/notion.ts; the
ls_l_pages and stat_dir_a cases were only in the Python harness, so the
TS run diffed short. Add the same cases (same order) to notion.ts.
ssh joins ram/disk/redis in assert_real_mtime. The guard now clears the
write-through read cache before listing so it validates the backend's own
mtime (ssh reads it from SFTP attrs.mtime); the warm read-cache not
carrying mtime is a separate pre-existing concern.
langchain, langchain-anthropic (+anthropic), langsmith, msgpack,
pydantic-settings, pypdf updated to advisory-fixed versions.
uv audit now reports no known vulnerabilities.
- opfs: directories have no native timestamp, so derive modified from the
newest file child's lastModified (null when the dir holds no files).
- integ: assert_real_mtime in cases.py creates a file + dir and asserts
ls -l shows a non-epoch mtime; wired into the ram/disk/redis harnesses
(silent on success, so the shared truth.txt diff is unaffected).
stat returned no modified time (ls -l showed "Jan 1 00:00") because readdir
never stored remote_time on index entries, or stat never surfaced it.
- s3, nextcloud: capture LastModified / getlastmodified in readdir, return
modified in the stat index fast-path (mirrors onedrive #361). s3 folders
stay bare (synthetic common-prefixes).
- notion, linear, github_ci, trello: ISO-native source already in the index,
wire stat -> modified (+ github_ci readdir captures workflow/artifact
updated_at; notion captures child-page last_edited_time).
- slack (epoch seconds), discord (snowflake): convert to ISO before ls -l,
whose formatter only parses ISO.
- local backends (disk/ram/redis/opfs) unaffected: they read mtime directly
with no index fast-path to shadow it.
Python + TypeScript both. Unit tests added/extended; notion integ gains
ls -l + folder stat cases; s3 integ freezes moto time for determinism;
linear/github_ci/trello/slack/discord examples gain ls -l demos.
- onedrive_server fake Graph: emit folder size + lastModifiedDateTime
- integ: add ls -l on a dir-listing and stat on a folder
- regen truth_onedrive.txt (file mtimes Jan 1 -> Mar 31; folder size/mtime)
- readdir/stat: use existing ResourceType enum instead of raw strings
- Delete orphaned discord/files_list.ts (unused since refactor #55)
- Delete empty curl/wget native test stubs in py and ts (no coverage;
curl/wget are covered by test_net.py and curl_persist.test.ts)
touch and mkdir keyed IOResult.writes on path.original (mount-prefixed),
which the workspace re-prefixes again, producing doubled keys like
/databricks/databricks/... Switched to path.stripPrefix to match rm and
the s3/ram backends. Adds command tests asserting mount-relative keys.
The Databricks SDK GetMetadataResponse exposes content_length and
last_modified (an RFC 7231 HTTP-date string), not file_size /
modification_time. stat read the latter, so size and mtime were always
None and ls -l rendered size 0 and an epoch date for every file (ls
calls stat per entry).
Read content_length and last_modified, and parse the HTTP-date into ISO
in _modified so the ls long-format renderer (which needs fromisoformat)
formats it instead of falling back to the epoch. Mirrors the TypeScript
core, which already reads the content-length/last-modified headers and
converts the date.
Realign the databricks_volume test fakes to the real SDK shape; they had
mirrored the buggy attribute names, which hid the bug. DirectoryEntry
listing fakes keep file_size/last_modified (correct for that API).
patch_attr no-opped via hasattr, so a future helper rename would silently
skip installing the fake and let a test reach the real OneDrive accessor.
patch_attr now reports whether it patched, and patch_module asserts at
least one known I/O helper was patched per command module.
mkdir/mv/touch tests now assert the mount-relative io.writes key and that
the doubled /onedrive/onedrive/... prefix is absent, covering the
strip_prefix regression for all five write commands (cp/rm already did).
ls -l and stat size checks anchor 17 to the words.txt row / size= token
instead of a loose substring match.
The OneDrive uniq wrapper pre-parsed -f/-s/-w with int() and passed ints
to generic_uniq, which declares str | None and re-parses via _parse_count.
Pass the raw flag values straight through like every other backend so the
generic owns None/0/int semantics and the 'uniq: invalid count' error
message.
Resolves GHSA-jm82-fx9c-mx94 (missing stream length values ignore
defined limits), the only finding failing the uv audit CI step.
pypdf is a transitive dep (openhands-tools -> browser-use -> pypdf);
this is a lockfile-only bump. The remaining 'socksio is archived'
adverse status is informational and does not affect the audit exit
code.
The integ harness wrote `out + "\n"` when output lacked a trailing newline, so
truth.txt couldn't distinguish "foo" from "foo\n" — masking sed's missing-
final-newline preservation. Emit a `\ No newline at end of output` sentinel
instead (both languages), so the integ now genuinely verifies it.
Regenerated truth.txt: four cases legitimately gain the sentinel — head_c5,
cat_no_nl, base64_stdin_d, and sed_no_final_nl. TS and Python outputs remain
byte-identical (no hidden trailing-newline divergence was being masked).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `sed 's/o/O/0'` to EXIT_CODE_CASES so the zero-occurrence-count rejection
(exit 1) is exercised cross-language in the integ battery, not just in unit
tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rework the engine so the pattern space holds line content WITHOUT the
line-separator newline (GNU sed's model), re-adding the newline on output and
preserving a missing final newline. This fixes the remaining multi-line
divergences in one place:
- hold-space accumulation: `H;${x;p}` no longer gains an extra newline.
- `N` + addresses: the line number / `$` track the last line consumed, so the
join-all idiom `:a;N;$!ba;s/\n/,/g` works.
- a file with no trailing newline keeps it absent on output.
The #326 anchor strip in regexReplace/addrMatches and the per-command newline
juggling (y, G/H) are removed — no longer needed now that the pattern space is
newline-free. Behavior is unchanged for every existing case (full suites +
integ diff clean); only the previously-divergent multi-line cases change.
Also fix two smaller GNU gaps in the `s` parser:
- escaped delimiter: `s/a\/b/c/` now parses (the field scan honors `\<delim>`).
- a zero occurrence count (`s/x/y/0`) is rejected, matching GNU.
- sed_helper.ts / sed_helper.py: content-only pattern space, tail-newline
tracking, lineno updated by N, simplified regexReplace/addrMatches/y
- unit tests (join-all, hold accumulation, missing final newline, N+$,
escaped delimiter, zero-count) in both languages
- native parity vs real sed (missing final newline, escaped delimiter);
join-all is GNU-only (BSD can't parse `:a;N;$!ba`) so it's unit+integ only
- integ: sed_join_all / _hold_accum / _escaped_delim / _no_final_nl in both
languages; truth regenerated, validated against gsed 4.10
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`addr!command` runs the command on every line the address does NOT select —
e.g. `2!d`, `/foo/!d`, `$!p`, `1,3!s/x/y/`, and the `$!b` branch idiom. The
parser consumes an optional `!` (whitespace allowed) after the address and
sets a `negate` flag; the executor inverts the match decision while the range
state is still tracked normally.
- sed_helper.ts / sed_helper.py: parse + apply negation (all command forms)
- unit tests (sed_helper.test.ts, generic/test_sed.py)
- native parity vs real sed (2!d, /b/!d)
- integ: sed_neg_line / _regex / _lastp / _range in both languages; truth
regenerated and validated against gsed 4.10
Note: the `:a;N;$!ba;s/\n/,/g` join-all idiom still diverges — not due to
negation but the pre-existing N/`$` + trailing-newline-in-pattern-space
limitation (same root as hold-space accumulation and #326).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backends with no shared-cases integ runner (hf, box, dropbox, onedrive)
get sed in their examples so sed is demonstrated/verified on each mount:
- hf_models (py + ts): sed read-transform over the read mount — verified by
running both examples against the public sapientinc/HRM-Text-1B repo.
- box, dropbox (ts, read-only): read-transform + the -i PermissionError
rejection.
- onedrive (py, writable): cat | sed read-transform.
box/dropbox/onedrive need live credentials to run, so they're demonstrated
but not executed here; their factory wiring is identical to gdrive
(read-only) / s3 (writable), which are covered by tests + integ.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comprehensive GNU-conformance sweep of the sed surface. Two bugs surfaced and
fixed:
- TS dropped the unescaped `&` (whole match) in replacements — emitted a literal
`&` while Python and GNU substitute the matched text. translateReplacement now
maps bare `&` -> `$&` (`\&` stays literal).
- `G`/`H` skipped the append when the hold space was empty, so `G` never added
the GNU blank line. They now append unconditionally (both languages).
Add 16 integ cases (both languages, each validated byte-for-byte against
gsed 4.10): & whole-match + literal, s///i, alt delimiter, line/last/range
addresses, one-line i/c, c by regex, q, G double-space, N join, {} block,
semicolon multi-command, and -E backreference. truth.txt regenerated.
Note: hold-space *accumulation* (e.g. `H;${x;p}`) still diverges from GNU
because the pattern space retains its trailing newline (same root as #326);
that needs a pattern-space-newline refactor and is out of scope here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GitHub is the one dedup'd backend with a unique factory wiring
(glob_when = index is not None) and a mock test harness. Add a sed test on
the read-only GitHub mount via the existing github_mock fixture (no network):
read-transform to stdout, per-line + global substitution, address delete, and
the -i PermissionError rejection. Exercises the make_sed factory for github.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run the repo's pre-commit hooks over the sed changes: yapf and prettier
reformatting (no behavior change) and wrap an over-length docstring line in
generic/sed_command.py (flake8 E501). All pre-commit hooks now pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github/uniq was the only wrapper passing a raw bytes-returning reader into
a generic command; every other backend already passes an async generator.
Wrap github_read in stream_from_bytes like its siblings, consume read_stream
directly in generic/uniq, and delete the now-unused _open_read_stream helper.
`sed -e E1 -e E2 ...` now applies the expressions in sequence (joined with
newlines, GNU-style); previously only the first was used and the rest were
mis-parsed as file paths. Make -e a repeatable value option in the spec and
assemble the script from its values, falling back to the positional operand.
GNU's "if any -e is given, all bare args are files" rule can't be expressed in
the static arg spec (a bare file lands in the positional script slot), so the
sed factories recover it: when -e is present the positional operands are
resolved as path operands (carrying the mount prefix). This keeps
`sed -e 's/../../' file` working rather than regressing to stdin.
Also fixes a pre-existing parser bug exposed by newline-joined scripts: the
`s`/`y` parser split the whole remaining program on the delimiter, folding a
following command into the flags (e.g. `s/a/b/;d`). It now reads pattern and
replacement up to the delimiter and consumes only the trailing flag chars, so
multi-command scripts with a non-terminal s/y parse correctly.
- spec: -e -> value + repeatable (builtins.ts, search.py, JSON exports)
- generic sed (TS+Py): assemble script from -e values; treat `\n` scripts as
programs
- makeSed / make_sed factories: positional-as-path reinterpretation under -e
- s/y parser bounded at the command separator (TS sed_helper.ts, Py
sed_helper.py)
- native parity (multiple -e, -e + file) and integ (sed_multi_e, sed_e_file)
in both languages; truth.txt regenerated and validated against gsed 4.10
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- python: reset _auto in unmount(); importorskip so the module collects
without the fuse extra; add generated-mountpoint removal test
- integ: add a generated (fuse: true) mount and assert caller-owned dirs
survive while generated temp dirs are removed (py == ts)
Mirror the TS makeSed factory on the Python side. The 13 wrappers shared the
same shape (glob, read, optional write, -i rejection, delegate to the generic
engine) but with real per-backend variation, which the factory captures as
config: `glob_when` predicate, `make_read` builder, `write_bytes`, and an
`inplace_error` (exception type + message). Faithfully preserves each backend's
behavior — including databricks_volume's index+prefix reader and ValueError on
-i, github's index-gated glob, and the root/store glob gates.
Also fixes a copy-paste bug: chroma's read-only -i message said "Dify".
- new python/.../generic/sed_command.py (make_sed)
- rewrite all 13 backend sed.py wrappers to call it
Verified: 183 Python sed tests pass; all 13 modules import and register
(hf_buckets → 4 resources); integ PY ram/disk diff clean against truth.txt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 14 TypeScript sed wrappers were near-identical: resolve glob, reject -i on
read-only mounts (copy-pasted 5x), then delegate to sedGeneric. Extract that
into a single `makeSed({resource, stream, write?, glob?, readOnlyMount?})`
factory in core; each backend is now a small config. No behavior change.
- new core/commands/builtin/generic/sed_command.ts (exported as makeSed)
- rewrite ram/s3/databricks/gdrive/box/chroma/dropbox/github (core),
disk/redis/ssh/hf (node), opfs (browser) to use it
- read-only `-i` message standardized (still contains "read-only <Mount> mount"
so existing assertions hold)
Verified: core 3295, node 1442 suites pass; gdrive -i rejection + native sed
(64) pass; integ unchanged (TS ram/disk diff clean); lint + build green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GNU sed scripts are Basic Regular Expressions by default and Extended only
under -E/-r, but the engine always fed patterns straight to the host regex
(JS RegExp / Python re), which is ERE-like. So BRE scripts were wrong:
`s/\(foo\)/[\1]/` treated `\(` as a literal paren, `s/a\+/X/` as a literal
`+`, while bare `s/a+/X/` wrongly acted as a quantifier. The -E flag was
declared but ignored, and -r was unknown.
Add a BRE->ERE translator (swap the special/literal roles of `( ) { } + ? |`
and their backslashed forms; positional `^`/`$` anchors; leading `*` literal)
applied to both `s` patterns and regex addresses when -E/-r is absent. Thread
the `extended` flag from the generic command through the per-line engine.
Register -r as an alias of -E in the command spec (+ regenerated spec JSON).
- typescript/core sed_helper.ts: breToEre + extended threading; generic/sed.ts
reads opts.flags.E/r
- python sed_helper.py: _bre_to_ere + extended threading; generic/sed.py gains
`extended`; all 13 backend wrappers pass it
- spec: add -r (builtins.ts, search.py, generated JSON exports)
- unit tests (breToEre + BRE/ERE behavior) in both languages
- native parity vs real sed for portable BRE/ERE forms (+ -r alias)
- integ: sed_bre_* / sed_ere_* / sed_r_alias in cases.ts + cases.py;
regenerated truth.txt; every case validated against gsed 4.10
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two missing GNU sed commands to the shared engine (both languages, all
backends):
- `y/src/dst/` transliterates characters by position (errors when the two
sets differ in length), leaving the line-separator newline untouched.
- `c\text` changes lines: for a single address (or none) the text is emitted
on each match; for a range it is emitted once, when the range closes (or at
EOF), matching GNU sed. The matched lines are deleted.
Also fix the `a`/`i`/`c` text parser: it stripped the leading backslash of the
classic `cmd\<newline>text` form but kept the newline, so `a`/`i` emitted a
spurious leading blank line (visible in the integ sed_append case). Strip the
backslash-newline continuation properly.
- typescript/core sed_helper.ts: parse + execute y and c; text-prefix fix
- python sed_helper.py: same (str.translate for y; id(cmd) range tracking)
- unit tests (sed_helper.test.ts, generic/test_sed.py)
- native parity vs real sed (native_sed.test.ts, native/test_sed.py): y, c
single-address and range
- integ: sed_y / sed_c_addr / sed_c_range in cases.ts + cases.py; regenerate
truth.txt (also drops the sed_append blank line)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve cache-consistency conflict: keep the single CacheManager
invalidation path and drop invalidate_index_dirs.
Remove is_remote and cacheable in favor of a single caches_reads flag
declared per resource. Live-insert databases (postgres, mongodb,
chroma) set caches_reads=False so reads always hit the backend;
lancedb caches only for remote-scheme tables. Fix the postgres and
chroma cold grep -e multi-pattern path (route multi-pattern to the
generic grep instead of the single-pattern pushdown).
GNU sed's substitution flags `N` (replace the Nth occurrence) and `Ng`
(replace the Nth and every later occurrence) were silently ignored — the
engine only honored `g`/`i`. The `p` flag (print the pattern space when a
substitution is made) was likewise ignored, so `s///p` and `sed -n 's///p'`
did not print.
Implement both in the shared per-line engine (so every backend benefits):
- parse a numeric count from the s-flags and replace per-match accordingly
(count defaults to 1; `g` means "from the Nth onward", else "the Nth only")
- on a successful substitution with the `p` flag, emit the pattern space
Matched against GNU sed semantics. (Note: `Ng` is GNU-only; BSD/macOS sed
rejects it, so native-parity tests use only BSD∩GNU forms — `Ng` is covered
by unit and golden integ tests.)
- typescript/core sed_helper.ts: regexReplace gains a count param; s-branch
parses count + p
- python sed_helper.py: per-match counter in the s-branch; count + p
- unit tests (sed_helper.test.ts, generic/test_sed.py)
- native parity (native_sed.test.ts, native/test_sed.py): s///2, s///p, -n s///p
- integ: sed_count_nth / sed_count_nth_g / sed_sub_p in cases.ts + cases.py,
seed /data/oooo.txt, regenerate shared truth.txt
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generic sed single-substitution fast-path (used by every backend) ran
`text.replace` / `re.sub` over the whole file buffer. That anchored `^`/`$`
at the buffer ends and only touched the first match in the entire file, so
`sed 's/^#[0-9]*$/.../ ' file` was a no-op and a non-global `s///` missed the
first match on every line but the first. The bug was identical in TS and
Python (the stdin path already used the per-line engine, which is why #326
only surfaced on stdin).
Route the fast-path through the per-line engine (executeProgram /
_execute_program) in both languages, matching GNU sed for file arguments and
in-place edits across all backends.
- typescript/core generic/sed.ts: use executeProgram in the isSimpleSub branch
- python generic/sed.py: use _execute_program in the simple-sub branch
- native_sed.test.ts: parity cases for file-arg anchored sub, first-match-per-
line, and -i anchored sub (vs real GNU sed)
- integ: add file-argument sed_anchor_sub_file / sed_firstmatch_file cases to
cases.ts + cases.py, seed /data/multi.txt, regenerate shared truth.txt
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add dedicated `sed_anchor_*` cases to the shared integ battery (cases.ts +
cases.py), exercised against the common truth.txt by both the Python and
TypeScript harnesses (and every backend) in CI. Covers anchored substitution
(plain, -E, /g) and an anchored regex address, mirroring the issue's stdin
repro. Seeds /data/anchors.txt and regenerates truth.txt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The TS sed kept the line-separator newline in each line (splitLinesKeepEnds),
and JS regex `$` (without the `m` flag) only matches the absolute end of input.
So anchored scripts like `s/^#[0-9]*$/#TS/` were a no-op against "#123\n",
diverging from the Python implementation and GNU sed.
Strip a single trailing newline before substituting / matching a regex
address and re-append it afterwards, so `^`/`$` anchor to line content per
POSIX sed semantics. Matches Python sed and real GNU sed.
- regexReplace: strip/re-append trailing newline around the substitution
- addrMatches: match regex addresses against line content sans trailing newline
- add sed_helper.test.ts unit coverage for anchored subs and addresses
- add native (real-sed) parity cases for anchored subs/addresses
- revert integ bash_history_format workaround to the anchored pattern
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- TS: wire tail -f to the change stream (spec -f/--follow + watchStream
branch); it was previously unwired so tail -f returned stale docs
- decouple caching from is_remote via a cacheable flag (caches_reads /
cachesReads gate); MongoDB sets cacheable=false so reads stay live and
tail -f is not masked by a cached snapshot
- accessor: narrow version lookup to PackageNotFoundError and log it
- integ: migrate seeding to AsyncMongoClient; add live change-stream case
on the warm books collection (py + ts); run MongoDB as a replica set
- docs: update mongodb cursor/cache notes
Add an in-band coherence block to the s3 (moto) integ in both languages:
cache a listing with ls, mutate via cp (core copy) and rm -r (core rm_r),
then list again under LAZY and assert fresh state. Verified to go stale
when the copy/rm hook is removed. ram/disk don't cache listings in the
harness, so s3 is where these hooks are actually exercised; the gzip case
only covers the write/unlink hooks.
Every backend mutation now invalidates at the write site, so the
dispatcher's after-the-fact invalidate_index_dirs is dead weight: it
re-invalidated dirs the hooks already handled, and the native-API
create commands it supposedly covered report an empty IOResult. Remove
it from both dispatchers, leaving a single invalidation path. The
cache-warming half of apply_io is unchanged.
Phase 1 hooked the common mutators (write/unlink/rename/mkdir/truncate)
but left create/copy/rm/rmdir relying on the after-the-fact
invalidate_index_dirs path. Hook these too across s3, redis, ssh,
nextcloud, onedrive and databricks (py+ts), and add the missing
onedrive base mutators. Now every backend mutation invalidates at the
write site.
CodeQL's JS analysis did not treat resolveWithinRoot's startsWith check as a
sanitizer across the helper return, re-flagging the readFileSync/mkdirSync/
statSync sinks as path-injection. Inline the resolve + startsWith(root+sep)
guard directly in the load/snapshot handlers so the barrier dominates the sink.
The shared helper stays for the version backend (workspaceId is also
allowlist-validated) and the Python handlers, which CodeQL accepts.
- resolve_within_root keeps os.path.realpath: the Python CLI sends
Path.resolve() (symlink-resolved) paths, so the daemon must resolve symlinks
too or a snapshot root reached through a symlink (e.g. /tmp) is wrongly
rejected. This mirrors the TS CLI/daemon pair, which both use path.resolve.
- integ/cross.sh exports MIRAGE_SNAPSHOT_ROOT and writes its tars there so the
cross-language interop check works with the confined snapshot endpoints.
- resolve_within_root uses os.path.abspath (pure normalization), mirroring TS
path.resolve, instead of realpath which resolved symlinks and diverged.
- load_workspace confines the path before the id-conflict check, matching the
TS handler's error precedence.
CodeQL request-forgery / stack-trace-exposure / xss in TS examples:
- Pin the discord/slack token proxies to a fixed upstream origin and reject
anything that escapes it; stop returning raw error details to the client.
- HTML-escape untrusted OAuth error values before writing them into the DOM in
the box/dropbox/gdocs/gdrive PKCE pages.
CodeQL polynomial-redos / double-escaping / insecure-randomness in core:
- Replace super-linear trim regexes with loop-based helpers (stripUnderscores
for title sanitizing, rstripNewlines for aggregators/boxcanvas, char loops
for mktemp); anchor the awk accumulator regex.
- Decode the S3 XML entity '&' last so '&lt;' no longer double-unescapes.
- Generate browser session ids with crypto.getRandomValues instead of Math.random.
Fixes CodeQL path-injection and missing-rate-limiting on the daemon (py+ts):
- Add resolveWithinRoot/validatePathSegment helpers; confine /v1/workspaces
snapshot + load paths to a configured snapshot root (MIRAGE_SNAPSHOT_ROOT,
default ~/.mirage/snapshots) and validate workspaceId as a safe path segment
in the version backend.
- Reject non-object request config so a string body cannot trigger an
arbitrary file read.
- Register @fastify/rate-limit globally with a tighter per-route cap on the
snapshot/load endpoints.
pyjwt >=2.13.0 (clears auth alerts incl. HS256 forgery #99, PyJWKClient
SSRF, alg bypass). nodemailer >=8.0.9, form-data >=4.0.6, tar >=7.5.16,
vite >=7.3.5, @opentelemetry/core >=2.8.0 via pnpm overrides, each
capped within its current major. Clears 13 of 15 open alerts.
5.7.0/5.7.1 added an EntityReplacer regression that rejects
addEntity('#xD'), which the AWS SDK uses to parse S3 XML error
responses, breaking integ-ts. 5.7.2+ keeps the GHSA-gh4j-gqv2-49f6
fix without the regression.
aiohttp 3.14 added a required stream_writer kwarg to ClientResponse;
aioresponses 0.7.8 does not pass it, breaking the onedrive test mocks.
Add a version-guarded conftest shim that injects a default stream_writer
into aioresponses' ClientResponse construction, then move aiohttp to
>=3.14.0. This clears the last two aiohttp alerts (#35, #36).
aiohttp >=3.13.4,<3.14 (capped below 3.14 until aioresponses supports
its new ClientResponse API), pygments 2.20.0, and pnpm overrides for
uuid >=11.1.1 and fast-xml-parser >=5.7.0. Clears 12 of 16 alerts.
Add direct coverage for the two riskiest, previously-untested parts of
the Motor->PyMongo Async migration:
- iter_inserts: exercises `async with await col.watch(...)`, the change
stream path (was only ever patched out wholesale in stream tests)
- accessor: verifies the client is constructed via AsyncMongoClient with
driver=_DRIVER_INFO, plus per-event-loop caching and listing-cache
invalidation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Motor is deprecated (May 14, 2026); migrate to the PyMongo Async API
(AsyncMongoClient), which ships inside pymongo.
- pyproject: mongodb extra motor>=3.7.1 -> pymongo>=4.9
- swap AsyncIOMotorClient -> AsyncMongoClient imports and annotations
- await aggregate()/list_indexes()/watch(), which are coroutines in
PyMongo Async (returned cursors directly in Motor)
- update test fakes to AsyncMock and assert_awaited_* accordingly
- docs: uv add motor -> uv add pymongo
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- emit the start path at depth 0; implement -empty
- real expression grammar parser (-not/-o/-a, parentheses) replacing
the dead _extract_not_name path; shared predicate-tree evaluator
- unknown predicates and invalid -size/-maxdepth/-mtime args exit 1
- validate -type letters; bound parser recursion depth
- migrate onedrive find to the generic; remove dead find_predicate_error
and _extract_* helpers
- fan-out mount synthesis and child-mount -maxdepth honor the parsed
expression tree
- chroma/dify: derive entry kind so -type works under the tree (was
hardcoded to file)
- emit the start path for an empty directory across the object/API
backends (s3 marker, nextcloud/hf scan base, onedrive stat); ssh/
nextcloud/hf compute is_empty for -empty; fix nextcloud/hf metadata
shadowing in the size/mtime path
- align integ truth for the now-correct -name OR and -not output
- mirror all changes across Python and TypeScript
Directory handling under -size is tracked separately in #318.
Option A: the bespoke SaaS backends (slack, discord, langfuse, github_ci,
email, trello, linear) now validate -size/-mtime via a shared
_validate_size_mtime helper (py) / findSizeMtimeError (ts), so a malformed
value exits 1 with the GNU message instead of being ignored and surfacing a
later network error. (Valid-but-unsupported size/mtime filtering is tracked
in #320.)
Expands integ/find_arg_errors.{py,ts} to 13 backends x 4 cases
(maxdepth/mindepth/size/mtime), byte-identical py/ts. Adds the Google
Workspace backends + databricks. github (networks at construct), notion (TS
OAuth provider), hf_buckets (bucket-id validation) and onedrive (py-only) are
documented exclusions from the cred-free cross-language suite.
Adds integ/find_arg_errors.{py,ts}: mounts every SaaS find backend
(slack, discord, gmail, trello, linear, langfuse, github_ci, email) with
dummy creds and asserts invalid -maxdepth/-mindepth exit 1 with the
GNU-style message. The error is raised during flag parsing before any
network call, so the suite needs no credentials and runs in CI. py and ts
diff against the same truth_find_arg_errors.txt (byte-identical).
Also reorders trello/linear find to parse depth before the API walk so the
error is reported without a wasted network round-trip.
Adds find_bad_maxdepth/mindepth/size/mtime, find_empty_size, and
echo_empty_squote/dquote to the shared integ suite (cases.py + cases.ts).
These run across every shared-suite backend (ram, disk, redis, s3, ssh,
nextcloud, opfs) and assert identical GNU-style exit-1 messages and the
preserved empty-argument behavior. truth.txt regenerated.
The earlier fix only covered the generic find path. Eight backends with
bespoke find (email, discord, langfuse, github_ci, onedrive, slack, trello,
linear) parsed -maxdepth/-mindepth inline, so an invalid value still raised
a raw ValueError (py) or silently became NaN (ts).
Python now routes those through _parse_depth (raises FindParseError); the
TypeScript bespoke backends share an invalidFindArg helper (exit 1, clean
stderr) reused by findGeneric. Adds bad-arg tests for github_ci (py) and
discord (ts).
find now exits 1 with a GNU-style message for invalid -maxdepth, -mindepth,
-size, and -mtime values (py raises FindParseError, ts returns IOResult);
the fan-out traversal no longer swallows it.
The shell expander now keeps a quoted word that expands to "" (e.g. "" or
"$EMPTY") as a real argument, matching bash, while still dropping unquoted
empty expansions and empty "$@". This also fixes a latent _parse_flags bug
where an empty flag value matched the root path's stripped scope key.
The TS workspace-YAML parser read camelCase keys (commandSafeguards/maxLines/
on_exceed...) while Python uses snake_case, so a Python-style config silently
applied no safeguard. Normalize snake_case keys to camelCase at the config
boundary (in-memory option stays camelCase, like the snapshot format split).
- config.ts: snakeToCamel/camelizeKeys for the safeguard block; YAML key is
command_safeguards with max_bytes/max_lines/timeout_seconds/on_exceed
- TypeScript CLI CI job: per-mount safeguard block mirroring the Python CLI
- cross job: /guard mount in cross.yaml + assert both CLIs cap cat identically
- relocate snapshot logic out of Workspace into workspace/snapshot/{api,state}.ts
free functions; Workspace keeps thin lifecycle wrappers (mirrors python layout)
- remove Workspace.toStateDict()/restore() methods; all callers use free functions
- applyStateDict now restores sessions (cwd/env), finished jobs, and the default
session/agent id, matching python apply_state_dict
- buildMountArgs aggregates every redacted mount missing an override into one error
- mirage_version reports the real package version (py dist-name fix + ts package.json)
- copy() reconstructs via _fromState directly (no tar round-trip)
New 'cross' job installs both Python and Node, builds both CLIs, starts
redis + MinIO, and runs integ/cross.sh (bidirectional 4-mount snapshot
round-trip). Wired into the cli-gate.
fromState now uses a provided override for any mount (not just redacted
ones), mirroring Python build_mount_args, so disk/redis content loads into
the real backend instead of being RAM-ized (which crashed on missing
dirs). RAM loadState tolerates absent dirs/modified. Add integ/cross.sh +
integ/cross.yaml bidirectional driver; verified locally both directions.
Python reconstructed mounts by importing resource_class as a dotted path;
TypeScript writes resource_class as the resource kind ('ram'). Resolve the
class from resource_state.type via REGISTRY first (the key both languages
share), falling back to the dotted path. Add a TS-written-tar golden test
so a TS snapshot loads in Python.
Dedupe norm/parent/basename/dirname across backends to the canonical
helpers in utils/path (norm, parent, gnuBasename, gnuDirname):
- python: ram+redis drop 36 local _norm and 4 _parent; chroma/dify use
shared parent + gnu_basename; add norm/parent to utils/path.
- typescript: ram/redis/disk/opfs utils re-export shared norm/parent and
gnuBasename; ssh norm, s3/chroma/gmail/email dirname, glob basenameOf,
rg_helper/zip_cmd basename, and agent parentOf all converge to the
canonical helpers (exported from the core barrel).
agents: converging parentOf to gnuDirname surfaced a recursion bug in
ensureParent for a root-relative path (gnuDirname returns '.'); guard '.'
and add a regression test.
Net -224 lines, behavior-preserving (verified: py 8048, ts core/node/
browser/agents green, integ ram output byte-identical to truth).
Snapshot/load now read/write the same tar+manifest.json+blob format as
Python (snake_case keys, __file blob refs). Wire Workspace.snapshot/load/
copy to manifest split/resolve + tar_io; delete the JSON persist path.
Add fingerprints/live_only_mounts to the manifest and a redis-aware
version check. Update server versioning (stateTree, clone) to the
snake_case state dict. PY-written RAM snapshot now loads in TS.
CodeQL js/polynomial-redos (the 2 new alerts on this PR): gmail sanitize
used /^_+|_+$/g and the read builtin used /\n+$/. Both are polynomial on
repeated chars. Switched to loop-based trims (gmail mirrors py .strip('_')).
CodeQL js/polynomial-redos: chroma used .replace(/\/+$/,'') and
/^\/+|\/+$/g which are polynomial on inputs with many '/'. Switched to
the loop-based rstripSlash/lstripSlash/stripSlash helpers (behavior
identical; chroma unit tests + py/TS integ unchanged).
The integ not-found block (grep/stat on a missing path) leaked driver
errors or fabricated directories instead of ENOENT:
- mongodb grep: stat the scope target before pushdown so a missing path
raises ENOENT instead of a Mongo 'database names cannot contain .' error.
- postgres stat: validate schema/entity existence (was the gmail synthetic
-directory antipattern, returning a fake dir for any path).
- postgres grep: stat-guard the scope pushdown like mongodb.
- postgres read (TS): raise enoent with the full virtual path, not the
prefix-stripped key (tail/wc showed /__nf_missing__.txt).
Mirrored py + TS; verified all four DB integ scenarios (mongodb, postgres,
chroma, lancedb) pass in both languages against live docker services.
Bump transitive npm deps to patched versions (hono/qs/ws/uuid via in-range
lockfile update; @anthropic-ai/sdk, ip-address, protobufjs via pnpm.overrides):
- @anthropic-ai/sdk 0.90.0 -> 0.104.1 (GHSA-p7fg-763f-g4gf)
- hono 4.12.15 -> 4.12.25: 8 advisories (routing, Set-Cookie injection, JSX
HTML/CSS injection, bodyLimit bypass, JWT scheme, cache Vary leak, IPv6 deny)
- protobufjs 8.0.2 -> 8.6.3 (GHSA-jggg-4jg4-v7c6)
- qs 6.15.1 -> 6.15.2 (GHSA-q8mj-m7cp-5q26)
- ws -> 8.21.0 (GHSA-58qx-3vcg-4xpx)
- ip-address 10.1.0 -> 10.2.0 (GHSA-v2v4-37r5-5v8g)
- uuid 11.1.0 -> 11.1.1, 13.0.0 -> 13.0.2 (GHSA-w5hq-g745-h8pq)
Two alerts intentionally not patched, to be dismissed as not-in-path:
- fast-xml-parser (GHSA-gh4j-gqv2-49f6): vulnerable 5.5.8 is pinned by
@aws-sdk/xml-builder@3.972.18 and incompatible with the 5.7.x fix; forcing it
broke S3 XML handling. The advisory is an XMLBuilder injection; the SDK only
serializes its own protocol XML, not untrusted input.
- uuid 10.0.0 (via @langchain/langgraph-checkpoint): only fixable by a cross-
major force; the vulnerable path (v3/v5/v6 with a buf arg) is not used.
Bump 8 Python deps to patched versions (asyncssh floor raised in
pyproject.toml; rest transitive via lock):
- asyncssh 2.22.0 -> 2.23.1: AuthorizedKeysFile %u traversal (GHSA-g794-3fmp-753h)
- authlib 1.6.11 -> 1.7.2: OIDC open redirect (GHSA-r95x-qfjj-fjj2)
- cryptography 46.0.5 -> 49.0.0: buffer overflow (GHSA-p423-j2cm-9vmq)
- idna 3.11 -> 3.18: idna.encode bypass (GHSA-65pc-fj4g-8rjx)
- pypdf 6.10.2 -> 6.13.2: parse DoS (GHSA-248m-82v9-q6g6, GHSA-cj93-chg6-vgv8)
- pytest 9.0.2 -> 9.0.3 [dev]: tmpdir handling (GHSA-6w46-j5rx-g56g)
- requests 2.32.5 -> 2.34.2: temp-file reuse (GHSA-gc5v-m9x4-r6x2)
- starlette 0.52.1 -> 1.3.1: Host header path poisoning (GHSA-86qp-5c8j-p5mr)
aiohttp intentionally not bumped: all 6 of its advisories are unreachable
in mirage, which uses aiohttp as a client only (no aiohttp.web server, no
CookieJar.load, no per-request cookies). 4 are server-side; the 2 client-side
cookie ones need apis mirage never calls. Bumping to 3.14 (required for 2 of
them) also breaks the aioresponses test mock. To be dismissed as not-in-path.
Backends stripped the mount prefix (and sometimes used the key/raw path)
in not-found and other fs-error messages, so errors showed a partial path
instead of the full virtual path (mount prefix + typed path). enoent/enotdir
now carry the original PathSpec and the command chokepoints append the GNU
strerror once, byte-identically in both languages.
Includes the databricks_volume sweep (every fs-error site now uses the full
virtual path), example not-found demos, and regenerated integ truth files.
integ/history.py exercises the history builtin (session projection,
-c tombstones), /.bash_history via generic commands, dotfile
visibility, and the hidden recorder's event stream; exact-diffed
against truth_history.txt in CI. Writing it surfaced a real bug:
events tied on the same millisecond rendered in file order, not
execution order, so the recorder now stamps a monotonic seq on every
event and queries sort by (timestamp, seq).
Delete ExecutionHistory and the history= threading through the executor.
The observer log (now unmounted, purely hidden) records all command and
clear events; /.bash_history is a view mount rendering the GNU histfile
fresh per read, and history is a shell builtin (GNU lookup order) routed
to a bespoke command on the view resource. /.sessions is removed from
the agent namespace; snapshot StateKey.HISTORY carries raw events.
@@ -24,7 +24,434 @@ Run Python commands from `python/`, TypeScript commands from `typescript/`.
- Keep Python and TypeScript layout, architecture, and semantics mirrored as much as practical.
- When changing one implementation, check the other for the matching pattern or feature. If one side is more correct, use it to improve the weaker side instead of copying a bad design.
- For major Python or TypeScript changes, consider adding or updating integration coverage under `integ/`.
-Known gap: TypeScript does not support ORC files. Python registers `.orc` in its filetype factory (`mirage/core/filetype/orc.py` plus per-backend `read_orc` ops); the TypeScript filetype factory only covers parquet, feather/arrow/ipc, and hdf5/h5. Do not assume `.orc` commands work in TypeScript.
-**The layout half of that rule is gated.** `scripts/check_layout_parity.py` diffs the module-name sets of every `mirage/<pkg>/` against its TypeScript counterpart (core/node/browser unioned onto one namespace, plus cli/server/agents by prefix), folding camelCase, hyphens and a leading underscore so a rename reads as a rename rather than a missing module; `__init__.py` and `index.ts` are skipped because only Python needs one per directory. Intentional differences live in `spec/layout_exceptions.json` with a reason, and a stale entry fails as loudly as a new gap. `--strict` (what CI runs) does not demand zero: it fails when the count moves off the committed `baseline` in *either* direction, so new drift is blocked and closing a divergence has to be locked in by lowering the number. Run it without `--strict` for the full advisory report; that report is how layout work gets scoped.
- **mirage ships no filetype renderers, and no factory for them.** Parquet, ORC, feather/arrow/ipc and hdf5/h5 rendering are gone, along with the `parquet`/`hdf5`/`pdf` extras, the `hyparquet`/`apache-arrow`/`h5wasm` dependencies, and the whole `commands/builtin/filetype_factory/` package in both languages (with its `filetype_read` / `filetypeRead` op knobs). A file with an unregistered extension is read as raw bytes. The one surviving extension point is registration on a mount: a command or op carrying a `filetype` resolves as `(name, filetype)` before `(name, resource)` before `(name,)`. `examples/{python,typescript}/filetype/` register a `.tally` renderer end to end and are gated in CI against `integ/truth/*/filetype.txt`; `tests/commands/custom/test_filetype_fns.py` and `test_unregister_removes_all_filetypes` cover the unit path.
## Module Layout
Packages split by role, one module per concern, the same way in both languages:
- **`types.py`** — data shapes only: frozen dataclasses, type aliases, Literal unions (e.g. `runtime/types.py` holds `RunArgs`/`RunResult`/`EvalValue`/`EvalResult`/`ScriptSource`). No logic.
- **`config.py`** — configuration knobs and their coercion (e.g. `runtime/config.py` holds `RuntimeConfig`, which fails loud on unknown fields).
- **`mixin.py`** — opt-in capability mixins: stateless, no constructor, abstract methods only (e.g. `runtime/mixin.py` holds `EvaluatorMixin`). Capability is detected by type (`isinstance`), never by probing for a method.
- **`base.py`** — the package's core ABC and nothing else (e.g. `runtime/base.py` is just `Runtime`).
## History
Command history is a recording, not a command log. A hidden `Observer` records every top-level command as timestamp-ordered events (`COMMAND`, `CLEAR`, `DELETE`, op events); the user-facing surfaces are just views of those events.
- **Observer + ObserverStore.** The `Observer` owns a storage-agnostic `ObserverStore` (`append`/`write`/`readAll`/`readMatching`/`clear`/`close`), not a mount. Stores: `RAMObserverStore` (core, default), `DiskObserverStore` and `RedisObserverStore` (node). RAM is just the default, history can persist to disk or Redis.
- **Two views over the same events.** `/.bash_history` is a read-only view mount (`HistoryViewResource`) rendered in GNU bash histfile format (`#<epoch>` line then the command), so `cat`/`grep`/`tail`/`find` work on it for free. The `history` shell builtin (GNU `-c -d -a -n -r -w -s -p` + count) routes through the same mount, so file and builtin never disagree.
- **Recording scope.** Top-level lines record; nested evals (`$()`, `eval`, `source`, `xargs`) run with `record: false`, so their inner ops bubble to the parent and no spurious command is logged (mirrors GNU's line reader).
- **Snapshots.** History is captured as events into snapshot state and restored on load.
- **Format is GNU bash, not zsh** (`#<epoch>`, not `: <ts>:<dur>;<cmd>`).
## CLIs
An installed CLI is a typed program tree (`CLISpec`) bound to a head word on the
workspace. It is **dispatched by name, never by operand path**: the VFS is how an
agent discovers state, the CLI is how it acts.
- **An account CLI consults no mount; `git` is the one that does.** The two are
different tiers, told apart by `config_model`. An account CLI
(`slack`, `linear`, `gws`, …) declares one, initializes from it, and reaches a
service, so a mount would be a second source of truth for the same account.
`git` declares none, because there is nothing to authenticate to, and its
subject is a repository that lives on a mount, so it reads that repository
through the op dispatcher like any command. That is what makes `git` work on a
RAM mount, a disk mount or an object store without knowing which. It reaches
the dispatcher through `CLIDoors`, one door per state plane: `dispatch` and
`stat_path` for data, `ns` for the name plane (mount boundaries, links, attr
overlay), and `session_view` for session state. The mount prefix is
`ns.mounts.root_of`, not a fourth field. `handle_cli`/`handleCli` puts the
record on `inv.doors`; the field is None/absent outside a workspace, and a
verb that never reads it cannot touch a mount, so this stays opt-in per verb
rather than ambient. The door is one field read rather than a parameter list
the dispatcher inspects, because every leaf takes exactly one `CLIInvocation`
and nothing is threaded through keyword injection. Every field is spelled the
way `CommandOpts` (`commands/config.py`) spells it, so one fact reached from a
CLI leaf and reached from a command handler has one name; meta-tests pin that
(`tests/commands/cli/test_doors_parity.py`, and a compile-time mapped type in
`commands/cli/types.test.ts`).
Do not give an account CLI a mount, and do not give `git` a `config_model`.
Nuance: an account CLI verb MAY read an unrelated workspace file the user
named on the line through `inv.doors.dispatch` (himalaya's `--attach` reads the
attachment path this way); what it must not do is treat a mount as a second
view of its own account's data.
- **The lifecycle is host-side only.** `register_cli`/`unregister_cli`
(`workspace.py`, `workspace.ts`) are called by the embedding program, never by
a line the agent types, and there is no `install`/`uninstall` shell builtin.
Keep it that way: an agent must not be able to take away the tools it was
given. Shadowing is the one thing it can do (define a shell function with the
same name), which is bash's own rule, reversible with `unset -f`, bypassable
with `command <name>`, and visible in `type -a`. A deployment that needs a head
word pinned enforces that in the policy layer's `pre_execute`, not in the CLI
registry.
- **Precedence is written down once**, in `_layers`/`layers`
from the long spelling when absent, which covers most options and is why only
the four upstream overrides declare one). `Option.env` is a fourth and is
**not** a synonym for `default`: an env-sourced value counts as *supplied*,
so clap echoes it in a usage line where a defaulted one is invisible, and it
is read from the session rather than frozen into the spec. The executor fills
it, so a leaf reads one flag instead of a flag and a fallback.
- **Imitating a Rust CLI means imitating serde_json, so the engine parser does
not get to decide.** `ntn api` echoes its JSON parse failures verbatim, and
those words are serde_json's (`EOF while parsing an object at line 1 column 1`), which neither `json.loads` nor `JSON.parse` can produce and which the
two of them word differently from each other anyway. So
`commands/cli/builtin/ntn/serde.py` (`serde.ts`) is a scanner whose only job
is that message, and it is the **authority on validity too**: the engine
parser runs only on input the scanner already accepted, because the two
disagree about what JSON is (python accepts `NaN`, serde refuses it, and
silently sending `NaN` is worse than either). Three rules that are easy to
get wrong and are pinned by a probed table in `test_serde.py` /
`serde.test.ts`: columns count **bytes**, not characters (`["é"x` fails at
column 6); a newline advances the line and zeroes the column; and the
recursion limit is 128, refused at the opening bracket of the 128th
container. Do not "simplify" this to a try/except around the engine parser.
- **`ntn api` has three body sources, and both the order and the exit codes are
observable.** stdin, `--data` and inline `path=value` / `path:=json` inputs
are each validated in that order, and only then is "more than one source"
reported, so a malformed pipe outranks a malformed `--data` and both outrank
the conflict. The exit codes are two families, neither of them argparse's 2:
a body that arrived malformed is **1** (`error: Invalid JSON from stdin`),
while a line the CLI will not interpret at all is **5** (the conflict, an
unparseable inline input, an empty `--data`), and those carry a second
` hint:` line. There is **no object check anywhere**: `--data '[]'` posts
the array, and any body source at all makes the call a POST even when what
it carries is empty, so `--data '{}'` posts rather than falling back to GET
on falsiness. `name==value` stays a query parameter whatever the method is.
- **Discoverability is part of shipping a CLI**, and it comes from the spec, so
it works for a user's own registered CLI exactly as for a builtin one. `man <cli>` and `man <cli> <verb>...` render through `node_help`/`nodeHelp`, the
same renderer `--help` uses, so a manual cannot drift from the program; bare
`man` lists installs under `# clis`. `type` reports an installed CLI as its own
kind (`type -t` prints `cli`, a sixth word beside bash's five, because reusing
`file` would promise `type -p` a path that does not exist). `which` prints the
bare name, never a fabricated path.
## Notion data sources
The notion backend speaks **`Notion-Version: 2025-09-03`**, the generation that
split a database into a container plus one or more *data sources*. The split is
not cosmetic and it is not optional:
- **The database object no longer carries `properties`.** The column schema
lives only on the data source, so `GET /v1/databases/{id}` answers with
`data_sources: [{id, name}]` and nothing to render a schema from. Do not
synthesize one back onto the container; the fake deliberately omits it so
anything that still reads a column list off a database fails loudly.
- **The mount nests accordingly**, because that is where the data is:
`databases/<Title>__<db-id>/database.json` is the container, and
`databases/<Title>__<db-id>/<Name>__<ds-id>/data_source.json` is the schema,
with the row pages under the data source. A row page therefore sits at depth
4 under `databases/`, not 3; `readdir`/`read`/`stat` all key on that in both
languages. The name stutters for a single-source database because Notion
names the auto-created data source after its database, which is honest and
disappears the moment a database holds two.
- **A row's parent is its data source**: `{type: "data_source_id", data_source_id, database_id}`. The database id rides along because Notion
kept emitting it, and the fake stores rows keyed by database, which is the
same fact one derivation away.
- **`/search` rejects `filter.value = "database"`** at this version; the
searchable schema-bearing object is `data_source`. Listing databases is
therefore "search data sources, take the distinct parents, retrieve each",
which costs one extra call per database and is the only way to get the
title and url the directory is named and rendered from.
- ids are **not interchangeable**: a data source id is not its database id.
The fake derives one from the other (`d5000000` + the database's tail) rather
than reusing it, precisely so an id mix-up cannot pass.
## Mount boundaries
A mount root is not an ordinary directory, and a mount nested inside
another mount's tree is invisible to the backend that owns the parent:
the child's keys live in a different resource, so the parent's `readdir`
never lists it. Two mechanisms follow from that, and they are separate.
- **`MountView` is how a command sees the boundaries** (`ops/types.py`,
`ops/types.ts`), and it is offered the way `LinkView` is: a command
opts in by naming a `mounts` parameter, `execute_cmd`/`executeCmd`
delivers it only to handlers that do, and there is no list of
boundary-aware commands anywhere. It carries `descendants` (mount
roots strictly under a path), `is_root`, and `root_of`.
A traversal command that renders **lines** does not need it: the
by absolute path (the bundled `libosxfuse.2.dylib` is a stub with that
install name), so `backend=fskit` + `volname` reach macFUSE 5.x's own
libfuse via `appendMountOptions` in `mount.ts`. Verified live. Caveat: a TS
fskit mount intermittently wedged on a write op in testing (probed from
child processes), and a dead FSKit volume blocks mount-table enumeration
system-wide until the macFUSE appex process is killed; treat fskit from TS
as read-mostly and experimental.
A mount is ready only when `os.path.ismount` says so, never when the
`/Volumes` entry merely exists: macFUSE creates that directory while
mounting and leaves it behind if the FSKit handoff fails, which reads as a
live mount and then fails with ENOENT on the first read.
**Python fskit mounts have the full write surface, and only because of
`fuse/darwin.py`.** The FSKit shim finalizes every created item through
macFUSE's Darwin-only `setattr_x` and routes rename through `renamex`;
mfusepy leaves those `fuse_operations` slots as reserved NULLs, so without
the extension module create/mkdir fail with ENOSYS *after* the op already
applied and rename never reaches userspace (verified by libfuse wire
trace: CREATE success, then SETATTR -78). Do not remove the
`install_macfuse_extensions()` call in `mount.py`, and keep the struct
tail in sync with macFUSE's fuse.h if mfusepy changes layout. Pinned in
`integ/fuse/truth_fskit.json` on macFUSE 5.3.3 / macOS 26.
TS fskit stays read-mostly: fuse-native's compiled op table cannot gain
new C callbacks from JS. Upstream shim caveats that remain for both:
exec-until-first-read fails (macfuse#1181) and root readdir cache cannot
be invalidated (macfuse#1165).
`integ/fuse/fskit.py` is the only coverage of a real FSKit mount, and it only
runs on a Mac with macFUSE 5.x: on the `integ-fskit-macos` job a hosted
runner installs and enables everything but the mount request never reaches
macFUSE, so the job reports that one known timeout signature as a skip and
stays green; any other failure there is real.
- **Directory and unknown sizes.** `getattr` reports `st_size` 0 for directories and for API-backed size-unknown files that have not been opened recently. Reads stay correct because Python mounts with `direct_io` (kernel reads to EOF regardless of st_size) AND `attr_timeout=0` (post-open fstat routes to `getattr(path, fh)`, which serves the real size of the open-hydrated content); prefetched bytes live in a 30s TTL cache (`PREFETCH_TTL`) so release-then-stat does not refetch. All three pieces are load-bearing: without `attr_timeout=0`, `wc -c` prints 0, BSD `cp` copies 0 bytes, and `tail -c` dumps the whole file; without `direct_io`, `cat` reads 0 bytes on macOS. Do not "fix" getattr to report real sizes eagerly (one API fetch per `ls -l` entry), and do not report fake sizes: stat-only tools (`tar`, `rsync`, `test -s`) seeing 0 matches procfs precedent. TypeScript uses the same recipe: `@zkochan/fuse-native` doesn't serialize a `direct_io` option, so `mount.ts` appends it to the option string at runtime (`appendDirectIO`; a pnpm patch would not reach consumers), plus `attrTimeout: '0'` + fgetattr. The old 100 MiB sentinel is gone; do not reintroduce it.
- **TS FUSE mounts are served by the mounting process's event loop.** Never touch your own mountpoint synchronously (`readFileSync`, `statSync`, `execFileSync`) from the process that created the mount: the call deadlocks the loop that must answer it, the kernel times out, and every later op fails with `Device not configured`/`ENOTCONN` — which looks exactly like a broken mount. Probe from a child process or use async APIs (see `examples/typescript/fuse/helper.ts`). Python is immune (FUSE loop runs on a thread).
- **`FileStat.size` must be the rendered content's byte length or `None`, never a storage-side or source-side number.** A confidently wrong size is worse than an unknown one: over FUSE it makes `wc -c`/`ls -l` lie and risks truncated copies, while `None` rides the unknown-size machinery above. Postgres `rows.jsonl` (on-disk `table_size_bytes` vs rendered JSONL), Dify documents (uploaded source size vs rendered segment text), Gmail messages (`sizeEstimate` vs rendered `.gmail.json`), Drive-rendered google-apps files (Drive storage size vs rendered `.gdoc/.gsheet/.gslide` JSON; raw binary downloads keep Drive's size), and Microsoft Graph folders (OneDrive/SharePoint report an aggregate subtree `size` on the folder facet, not any content length) made this mistake; their storage/source numbers now live in `extra` (`size_bytes` / `source_size` / `size_estimate`). Do not reintroduce it in new backends. Graph's `folder.childCount` rides along in `extra` too, and is what `find -empty` reads for a directory.
- **macOS allows only one FUSE mount per process.** The second mount dies with `fuse: cannot register signal source` (mfusepy registers libfuse signal handlers, which only the first mount in a process can claim). Multi-mount scenarios (`integ/fuse/fuse.py` mounts two) pass only on Linux; do not debug them as regressions on macOS. A failed run leaks the first mount: list with `mount | grep MirageFS`, clean with `umount <mountpoint>`.
- **Windows (WinFsp) conventions differ in three ways**, all handled in the python mount path (`_prepare_mountpoint`, `_await_ready`, the teardown branches): the mountpoint must NOT exist (WinFsp creates it; an existing dir fails with "mount point in use"), `os.path.ismount` never sees WinFsp directory mounts (readiness = bare existence), and there is no `fusermount` (WinFsp unmounts when the serving process exits). Ownership: mount with `uid=-1,gid=-1` (WinFsp builtin: files owned by the mounting user); never report raw POSIX ids into the SFU/Cygwin SID mapping, and `os.getuid` does not exist there (MirageFS caches a guarded uid/gid once). Behavior quirk: Windows cannot stat without opening a handle, so size-unknown files hydrate on first stat and report their real size even "pre-open" (multi-mount per process works). The `integ-fuse-windows` job is advisory (not in the gate).
## Development Setup
@@ -80,13 +507,119 @@ Invoke the venv's `pre-commit` binary directly (not via `uv --directory python r
## Rules
- **Shell-style commands** (cat, grep, du, find, head, tail, wc, ls, etc.) follow POSIX / Unix coreutils semantics as much as possible; match BSD/GNU behavior and document any deliberate divergence.
- **Shell-style commands** (cat, grep, du, find, head, tail, wc, ls, etc.) follow POSIX / Unix coreutils semantics as much as possible; match BSD/GNU behavior and document any deliberate divergence. Pin exact GNU behavior with docker (`debian:stable-slim`) before changing command semantics.
- **`find -size` is strict and rounds up.** GNU `+N` keeps `ceil(size/unit) > N`, `-N` keeps `ceil(size/unit) < N`, bare `N` keeps `ceil(size/unit) == N` (so `-size -1k` matches only empty files and `+0c` excludes empty ones). The parsers (`_parse_size` / `parseSize`) translate this once into inclusive byte bounds; backend cores just keep `min_size <= size <= max_size` and must not re-interpret the spec. Deliberate divergence: directories count as size 0 (GNU compares the inode size, e.g. 4096 on ext4), which matches what `find` sees over a mirage FUSE mount.
- **`find`'s row for the start point is the generic's, not the backend's.** GNU lists a start point before descending into it, and every native find op used to decide that row from its own listing, which is only a proxy for existence: an object store holding no keys under the prefix and no directory marker reported nothing at all for a directory `test -d` and `tree` both saw, and ssh called every directory non-empty so `-empty` never matched one. The generic stats the start point through the dispatcher (`resolve_start`, which asks both channels a backend can answer on via `resolve_path_stat`, since on a prefix store a directory is the set of keys under it rather than an object), takes one readdir for `-empty` (`dir_empty`, wired by the builder), and then replaces whatever row the backend produced for it (`with_root_row` / `withRootRow`). A native find op may still emit the start path and they all do, because it is the answer when no dispatcher probe is wired (a command constructed outside a workspace); with one wired it is discarded. A new backend only has to report descendants.
- **`find` classifies walked entries through `stat`, never by name.** The
walk's one in-band proof is a trailing slash on a cold listing (box, gdrive
and dropbox mark folders that way, and no backend renders a file with one);
every other entry is classified by the index-backed `stat` that the same
readdir just populated, so the lookup is RAM, not another API call. There is
no per-backend `is_dir_name`/`isDirName` hint: those heuristics guessed
wrong as soon as a child's name was user-controlled (email and gmail
attachments and slack uploads carry whatever name the sender gave them, and
were reported as directories, so `find -type f` missed them). Do not
reintroduce name-based classification in a backend; if stat misclassifies an
entry, fix that backend's stat.
- **An archiver walks a directory operand; it does not read it.** `tar`
and `zip` decide every member first (`plan_create` / `planCreate` in
`generic/tar/create.*`, `plan_zip` / `planZip` in
`generic/zip_cmd.*`) and only then write, which is what lets an
exclusion prune a whole subtree and keeps the ordering stable. Both
plans are built on **one traversal**, `scan_operand` / `scanOperand`
(`generic/archive/walk.*`), which merges three sources no single one
can see: the backend walk (reusing find's `walk_find` / `walkFind`, so
an archiver classifies an entry through `stat` exactly as find does,
never by name), the namespace's symlinks, and the mount table. It
reports paths, never names, because naming is exactly where the two
formats disagree; the two things they disagree about in the traversal
itself are parameters (`dereference`, `recurse`), so **a third
archiver adds a caller, not a second walk**. Members are named from
`PathSpec.raw_path`, so `tar -C d x` stores `x`, not `d/x`.
**A directory is its own member**, with GNU's trailing slash and no
content, which is the only record an empty directory leaves and the
reason extraction has to `mkdir` for one. **A symlink is a symlink
member** (`SYMTYPE`, target in `linkname`), never a file of its
target's bytes, unless `-h` says to follow it.
**Two links to one target are not a loop**, and both are archived; the
only loop is one `resolve` refuses to resolve, since the namespace
already walks the chain under a hop limit and raises `CycleError` at
the end of it. That arrives as a fatal `Problem` carrying GNU's
`Too many levels of symbolic links`, reported per member with the
directory entry kept, rather than as an exception that aborts the
plan. **Every `-C` is checked, not just the last**: GNU chdirs at each
one and fails at the first it cannot enter, so the option accumulates
(`multiple=True`) and the planner walks the list.
Two deliberate divergences from GNU, both documented in place:
siblings are sorted rather than emitted in readdir order (the same
choice `du` makes, for the same reason), and a descendant mount is
never crossed (see "Mount boundaries"). Everything else is pinned
against GNU tar 1.35 on `debian:stable-slim`: the leading-slash
warning, `Cowardly refusing to create an empty archive` (exit 2), a
per-operand `Cannot stat` plus one trailer (exit 2, and the other
operands still archive), a `-C` it cannot enter (exit 2, no archive
written), and `archive cannot contain itself; not dumped` (exit 0).
A backend error must never reach the user as itself: an unreadable
operand is reported in virtual path space with tar's wording, because
the raw `IsADirectoryError` leaked the host path behind a disk mount.
- **`zip` is Info-ZIP, which inverts tar's two defaults.** A directory
operand contributes only its own entry unless `-r` says to descend,
and a symlink is *followed* unless `-y` says to store the link, where
tar always descends and always stores unless `-h`. Both are just the
`recurse` / `dereference` arguments to the shared scan. The rest is
pinned against Info-ZIP 3.0 on `debian:stable-slim`: a leading slash
is stripped **in silence** (tar warns, zip does not), `-j` junks to
the basename and drops directory entries entirely, `-x` is
**anchored** on the whole stored name (`d/sub/*` matches, `sub/*` does
not) where tar's `--exclude` is unanchored, an unreachable operand is
`\tzip warning: name not matched: <name>` and does not stop the run,
and a run that matched nothing prints `zip error: Nothing to do!`,
exits **12**, and writes no archive. `-q` silences the warnings but
never that error. Two deliberate divergences: `-x` takes one pattern
per occurrence (mirage's spec has no variadic option value, and
`-x a -x b` says the same thing), and the `adding:` line carries no
`(deflated N%)` suffix, since the ratio depends on the compressor and
would differ between the two languages.
- **Tar formats come from a library, not from hand-rolled block code, and
bzip2 is read-only in TypeScript.** Python builds archives with stdlib
`tarfile`; TypeScript uses `modern-tar` behind `tar_helper.ts`, which
keeps the `TarEntry` shape (`name`/`data`/`isFile`/`isDir`/`linkname`)
the two call sites already speak and is the only place the dependency
is named. Do not go back to writing ustar blocks by hand: the version
that did truncated any name past 100 bytes instead of using the ustar
`prefix` field or a PAX header (so a deep member extracted to the wrong
path), and read a PAX/GNU extension block as if it were a member (so
`tar -t` on any archive GNU or Python wrote listed a phantom
`././@PaxHeader` row). `writeTar`/`readTar` are async for this reason.
Compression is a registry (`registerCompressionCodec`): gzip is built
in via `CompressionStream`, and a codec may be **decompress-only**,
which is the one deliberate py/ts divergence here. Python reads and
writes `.tar.bz2` because `bz2` is stdlib; TypeScript only reads one,
because every JavaScript bzip2 *compressor* is GPL (`compressjs`,
`archive-wasm`) and an Apache-2.0 package cannot ship that, while
`seek-bzip` (MIT) decodes. `tar -cj` therefore exits 1 with
`tar: bzip2 not supported`, the same answer browser core already gives
for an unregistered codec. If a permissively licensed bzip2 compressor
appears, adding `compress` to that one codec closes the gap with no
other change.
- **The TypeScript `walkFind` answers in mount-relative keys; the Python
`walk_find` answers in virtual paths.** TS's stands in for a backend's
native find op, so a caller that needs virtual paths (tar does, to
name members and compare against mount prefixes) lifts them with
`mountPrefixOf` the way `findGeneric` does. That lift lives once, in
`generic_bind/archive_io.*`, which is where both archivers get their
walk. This asymmetry is real and has bitten once: a unit test on an
unprefixed mount cannot see it, so cover a prefixed mount too.
- **`du` has one backend contract: `size` and `entries`.** Each backend exposes `core/<backend>/du/size.py` (recursive byte total for one path) and `core/<backend>/du/entries.py` (per-file breakdown), wired as `du_size` / `du_entries` on the adapter. `entries` returns `(entries, total)` where entries are **leaf files only, in mount-relative path space, with no summary row**; the generic lifts them onto virtual paths (`to_virtual`, via `mount_prefix_of`) and re-spells them as the operand was typed (`respell_raw`). A backend that returns backend-key paths, or appends its own roll-up row, makes two mounts holding the same filename render identical lines. Do not reintroduce a second shape; the old flat-list `du_multi` contract is gone.
- **`du` prints a line per directory, derived not walked.** GNU prints one line per directory with its recursive total, post-order (children before parents), plus one per file under `-a`. Backends only ever report leaf files, so the generic derives the directory rows by summing each leaf into every ancestor (`rollup`, same name both languages), then emits post-order with siblings sorted. Two deliberate divergences: GNU orders siblings by `readdir` (filesystem-dependent), mirage sorts them; and an empty directory is invisible to mirage because no leaf points at it. Sizes are bytes, not GNU's 1 KiB blocks, since an object store has no block size. `--max-depth` prunes only what is printed, never the walk, because every printed total still covers the whole subtree. Verify changes with the differential harness against `debian:stable-slim`: paths, exit codes and stderr must match GNU exactly.
- **`du` usage errors exit 1, not 2.** `du` is absent from `USAGE_EXIT`, which is correct: GNU du exits 1 for `-s` with `-a` ("cannot both summarize and show all entries"), `-s` with `--max-depth` ("warning: summarizing conflicts with --max-depth=N"), and a bad depth ("invalid maximum depth 'x'"). All three are raised by `parse_flags` / `parseDuFlags`*before* any I/O, mirroring GNU's option-parse order: the depth is parsed as the option is read, so a bad depth wins over the conflict checks. An unreadable operand is not a usage error: GNU names it (`du: cannot access 'x': No such file or directory`), prints every other operand, and exits 1, and still prints `0 total` under `-c` when every operand failed. With no operand at all, du measures the working directory; it never says "missing operand".
- **`du` walks are bounded.** Backends with no native du op are walked one `readdir` at a time, which on an API tree is one request per directory. `CommandIO.max_du_entries` caps that walk; when it trips, `du` prints what it accounted for, writes a notice to stderr and exits 1 (GNU's behavior for a tree it could not fully read), rather than hanging or silently reporting a wrong number. Slack sets a low cap (`DU_MAX_ENTRIES`) because it exposes a directory per conversation per day against a ~50/minute rate limit.
- **Async-native by default.** I/O uses `aiofiles` / `redis.asyncio` / `aioboto3`, and command pipelines are async generators.
- **Python unit tests mirror src 1:1 where reasonable.** Try to have a matching `tests/<path>/test_a.py` for each source file `mirage/<path>/a.py`. `__init__.py`, pure type-stub modules, and trivial re-exports are fine to skip; modules with real logic should have one.
- **Do not add `__init__.py` files under `tests/`.** Tests are namespace packages and pytest discovers them without `__init__.py`. Don't create one when adding a new test directory.
- **Monkeypatching a backend command module in tests:** the command imports its helpers by value (`from mirage.core.<backend>.read import read_bytes`), so to intercept them you must rebind the name inside the command module, not the core source module. But the command module is hard to reach: the backend package re-exports the command function in `__init__.py` (`from .cat import cat`), which shadows the submodule of the same name, so `import mirage.commands.builtin.<backend>.cat as mod`, `from ...<backend> import cat as mod`, and even pytest's string target `monkeypatch.setattr("mirage.commands.builtin.<backend>.cat.read_bytes", fake)` all resolve to the function, not the module (`AttributeError`). The command is also wrapped by `@command`, so `cat.__globals__` is the decorator's module. Reach the real command-module namespace through the unwrapped function and patch the dict: `monkeypatch.setitem(cat.__wrapped__.__globals__, "read_bytes", fake)`.
- Avoid add any comments or docstrings on the top of the file.
-Do not create nested functions.
-**A nested function must capture the scope around it.** Nesting is for closures: the def reads a name from the enclosing function, or binds one through a parameter default (`def f(m, _n=n)`, the loop-variable idiom). Everything the architecture nests is that shape — op factories (`ops/generic/factory.py`), provision builders, the read-through cache, and every decorator's wrapper — and those stay. What is banned is the nesting that buys nothing: a helper written inside a function although it reads only its own arguments, which rebuilds a function object per call and hides a testable unit where no test can reach it. Put that one at module level. Enforced for Python by `tests/test_nested_functions_are_closures.py`.
- Add type to Args for docstring.
- Do not add comment after each line of code in the format of "# 10MB - trigger segmentation for files larger than this". The most you can add is "# 10MB".
- For all imports you need to put to the top of the file. Don't have imports within each function.
@@ -97,6 +630,31 @@ Invoke the venv's `pre-commit` binary directly (not via `uv --directory python r
- Don't add too many printings or comments in the code.
- Don't add README.md unless I ask you to do so.
- Use uv add to install new dependencies.
- **Command wrappers and flags.** The dispatcher passes parsed command-line flags as keyword arguments. Wrappers must declare dispatcher-injected parameters (`stdin`, `index`, `prefix`) explicitly in their signature — never fish them out of `**flags` with `.get()`. Treat `**flags: object` as an opaque bag of true command-line flags and forward it wholesale to the generic command. When a wrapper genuinely needs a flag value itself (e.g. a search push-down), read it through `FlagView` (`fl = FlagView(flags)` then `fl.bool("F")`, `fl.int("m")`, `fl.str("type")`, `fl.list("e")`) or a shared domain accessor like `pattern_arg` — never raw `flags.get(...)` / isinstance chains.
- **Generic commands own flag interpretation.** Backend wrappers are wiring only (glob resolution, backend I/O injection, pass-through of `texts` and `flags`); all flag semantics live in the generic command for that family, mirroring the TS generics. Adding or changing a flag should touch the spec and the generic, not N wrappers.
- **Command handlers take `(accessor, paths, texts, opts)` — the same four positionals in both languages.** The dispatcher (`Mount.execute_cmd` / `Mount.executeCmd`) constructs one `CommandOpts` per invocation carrying stdin, the flag bag, cwd, the mount prefix, the index, and every namespace fact; the provision path builds the same bag with `command`/`spec` set (`ProvisionFn`has the same four-positional shape). Handlers never declare a flag or an injected fact as a parameter — `tests/commands/test_no_dead_flag_params.py` pins the exact signature. When a wrapper genuinely needs a flag value itself (e.g. a search push-down), read it through a spec-bound `FlagView` (`fl = FlagView(opts.flags, spec=SPECS["grep"])` then `fl.as_bool("F")`, `fl.as_int("m")`, `fl.as_str("type")`, `fl.as_list("e")`) or a shared domain accessor like `pattern_arg` — never raw `flags.get(...)` / isinstance chains, and never a raw `kwargs`/`_extra` read either (`tests/commands/test_no_raw_flag_reads.py`). To override a flag before delegating, pass `dataclasses.replace(opts, flags=bag)` down, never a hand-built `CommandOpts`. **A PATH-typed flag reaches a python command as a `PathSpec`, not a string** — the executor promotes it (`workspace/executor/command/flags.py`), so read it with `fl.as_paths(name)`; `as_str` reads it as absent and the operand is silently never used. TypeScript's bag carries the resolved virtual-path string instead, so its twin is `fl.asStr(name)`.
- **Generic commands own flag interpretation.** Backend wrappers are wiring only (glob resolution, backend I/O injection, pass-through of `texts` and `opts`); all flag semantics live in the generic command for that family, mirroring the TS generics. Adding or changing a flag should touch the spec and the generic, not N wrappers. Every literal `FlagView` query name must be a dest of a spec bound in the same module — `tests/commands/test_flag_query_names.py` and `commands/flag_query_names.test.ts` fail on a typo'd spelling without needing the code path to run.
- **Generics parse flags once into a frozen struct.** Each generic defines a `@dataclass(frozen=True, slots=True)` flag struct plus a module-level `parse_flags(fl, ...)` (mirroring the TS `parseFlags` struct); the function body reads only struct attributes, never string keys. Construct the FlagView with the command's spec (`FlagView(flags, spec=SPECS["grep"])`) so a typo in a flag name raises KeyError instead of silently reading as False/None.
- **Never annotate anything as `object`** — not a parameter, not a return, not a type argument (`dict[str, object]`, `Callable[..., object]`). `object` reads as "we did not decide": it accepts bytes where JSON was meant and a PathSpec where a flag value was meant, so every use site pays for it with an isinstance chain back to the set the author had in mind. Name the real type instead:
- a parsed command-line flag is `FlagValue` (`mirage.commands.spec.types`) — `**flags: FlagValue`, `Mapping[str, FlagValue]`, `FlagValue | None` for a single raw read. The TypeScript side has always called it `FlagValue` too (`commands/spec/types.ts`); keep the two spellings identical.
- a decoded JSON payload or an API field is `JsonValue` (`mirage.types`). It is recursive, so it is spelled as a forward-reference string until the floor is 3.12 — a union with it must be quoted: `Awaitable["JsonValue | X"]`.
- a path is `str | PathSpec`, a backend handle is `accessor: Accessor` (`mirage.accessor.base`), an index is `index: IndexCacheStore | None` (`mirage.cache.index`), a stat function is `StatFn` (`mirage.types`). Ignored variadics are still typed (`*texts: str`).
- a sentinel is a one-member `Enum`, never `object()`; that keeps it distinguishable from the real values sharing the variable.
`tests/commands/test_no_object_annotations.py` enforces this and carries the only exemptions: four Python protocol methods (`__setattr__`, `__contains__`, `Mapping.pop`) whose signatures the language fixes, and one guard whose whole job is to catch a value the annotations already claim cannot arrive. Adding to that allowlist needs the same kind of reason.
- **Every session write goes through `SessionView.set`.** A deployment refuses a
name (`AWS_*`, a credential) with a `pre_session` rule, and a writer that
reaches `session.env` directly makes that rule advisory. `export` and a plain
assignment always cleared the gate; the expansion-time writers did not, so
`${X:=d}`, `$((X=5))`, `(( ))`, `printf -v` and `for ((X=0; ...))` each wrote
past every policy. The view is threaded into expansion explicitly rather than
read from ambient state, and defaults to `None` (correct outside a workspace,
where the env is the only state there is). Reads (`$X`) stay sync; only the
writers await. `tests/workspace/expand/test_session_gate.py` and
`integ/session/writers.json` hold the line in both languages.
- **The record client is a substrate, not a session detail.** Sessions, the
namespace node table and workspace metadata are three tables that persist the
same way, so the keyed-record clients live in `workspace/record/`
(`DiskRecordClient` with an `O_CREAT|O_EXCL` lockfile and `rename(2)`,
`S3RecordClient` with If-Match CAS) and import none of the three. They used to
live inside the session package, which made the other two import upward into
it; a layering test in each language fails if that returns. The two clients
deliberately do not merge: different concurrency contract, different blob
Mirage is **a Unified Virtual File System for AI Agents**: it mounts services and data sources like S3, Google Drive, Slack, Gmail, and Redis side-by-side as one filesystem. Any LLM that already knows bash can read, grep, and pipe across every backend out of the box, with zero new vocabulary.
awaitws.execute('slack send-message --channel general --text "report is up"')
```
## About
- **One interface instead of N SDKs and M MCPs.** Every service speaks the same filesystem semantics, and pipelines compose across services as naturally as on a local disk.
- **Portable workspaces:** clone, snapshot, and version a workspace; agent runs move between machines without restarting or reconfiguring the system.
- **Embeddable:** the Python and TypeScript SDKs run in-process inside FastAPI, Express, browser apps, or any async runtime; no separate process required.
- **Agent integrations:** OpenAI Agents SDK, Vercel AI SDK, LangChain, Pydantic AI, CAMEL, and OpenHands via the SDKs; coding agents like Claude Code and Codex via the lightweight CLI + daemon.
- **Agent integrations:** OpenAI Agents SDK, Vercel AI SDK, LangChain, Pydantic AI, CAMEL, and OpenHands via the SDKs; coding agents through native adapters, installable plugins, MCP, or FUSE.
Mirage plugs into agent frameworks as a sandbox or tool layer. POSIX operations such as `read` can also be customized per resource and filetype, e.g. reading a PDF returns parsed pages instead of raw bytes.
Mirage plugs into agent frameworks as a sandbox or tool layer. POSIX operations such as `read` can also be customized per resource and filetype: Mirage ships no filetype renderers, so a format renders however you register it, and a command registered for one resource and extension wins over the generic one.
@@ -4,7 +4,7 @@ description: Run bash commands with `execute()`, per-call `cwd`/`env` overrides,
icon: terminal
---
Mirage Bash is how agents act on the workspace. `execute()` parses a bash-style command, looks up the target session, resolves mounts, runs the executor, applies I/O side effects, and records history.
Mirage Bash is how agents act on the workspace. `execute()` parses a bash-style command, looks up the target session, resolves mounts, runs the executor, applies I/O side effects, and records history through the [Observer](/home/observer).
## Per-call overrides: `cwd`, `env`
@@ -96,7 +96,7 @@ Both bindings support cooperative cancellation observed at recursion boundaries
Per-call timeout is not a CLI flag yet. Use `--bg` to get a job id and `mirage job cancel` to terminate.
Per-call timeout is not a CLI flag yet. Use `--bg` to get a job id and `mirage job cancel` to terminate, or wrap the command in the `timeout` builtin: `mirage execute -w demo -c "timeout 30 <cmd>"` exits `124` on overrun.
</Tab>
</Tabs>
@@ -108,6 +108,110 @@ Both bindings support cooperative cancellation observed at recursion boundaries
| Many isolated commands sharing scoped state | `session_id=...` (Py) / `sessionId` (TS) | a separate terminal |
| Persistent shell mutations | run without options | `cd /data; cmd` |
## JSON with `jq`
`jq` reads a **stream of JSON values** and runs the program once per
value, matching the real `jq` binary. The filename is irrelevant: a
`.json` file holding several concatenated or pretty-printed values is a
stream just like a `.jsonl` file, and so is multi-document input arriving
on stdin.
```bash
# Two documents in one file -> the program runs twice, one line each
cat /data/events.json
{"id": 1}
{"id": 2}
jq -c '.id' /data/events.json
1
2
# -s slurps the whole stream into a single array first
jq -c -s 'map(.id)' /data/events.json
[1,2]
```
Because evaluation is per document, commands that emit newline-delimited
JSON (such as [paginated `gws` list calls](/python/resource/gdrive#pagination))
pipe straight into `jq` with no reshaping.
Output arity follows the program, not the input. A jq program emits a
stream of values and each one prints on its own line, so `.a[]`, `.a, .b`
and `range(3)` all print several lines, while a program that collects
into an array (`[.a[] | .t]`) emits one value and prints one line.
```bash
jq -c '.name, .age' /data/user.json
"alice"
30
jq -c '[.name, .age]' /data/user.json
["alice",30]
```
### Flags
| Reading input | |
| --- | --- |
| `-n`, `--null-input` | run once against `null`; `inputs` still reads the operands |
| `-R`, `--raw-input` | each line is a string, not a JSON document |
| `-s`, `--slurp` | one value for the whole stream, spanning every operand |
| `--stream` | read each document as its `[path, leaf]` events |
| `--seq` | read and write RFC 7464 sequences (RS before each value) |
| `-f`, `--from-file` | read the program from a file |
| `--arg name value` | bind `$name` to a string |
| `--argjson name value` | bind `$name` to a JSON value |
| `--rawfile name file` | bind `$name` to a file's text |
| `--slurpfile name file` | bind `$name` to a file's documents, as an array |
| `--args`, `--jsonargs` | read the remaining operands into `$ARGS.positional` |
Build JSON with a binding rather than by hand: the value arrives as a
value, so quotes and newlines in it need no escaping.
```bash
# text -> JSON array, no string surgery
printf 'alpha\nbeta\n' > /data/lines.txt
jq -Rn -c '[inputs]' /data/lines.txt
["alpha","beta"]
# a shell value that contains quotes
jq -n -c --arg v 'a"b' '{msg: $v}'
{"msg":"a\"b"}
# a whole file as one JSON string
jq -n -c --rawfile body /data/lines.txt '{text: $body}'
{"text":"alpha\nbeta\n"}
```
`$ARGS` is always defined, carrying `named` (the `--arg` family) and
`positional` (`--args` / `--jsonargs`).
Three limits worth knowing. `inputs` is bound to whatever is still
unread, so a program that drains it (`[., inputs]`, `reduce inputs as
$x`) runs once and sees everything, but the stateful single `input` and a
partial drain (`first(inputs)`) are not modeled. `--stream` reads whole
documents and expands them, which matches jq except that jq's incremental
parser splits the closing event of an input with no trailing newline into
its own `-s` group. And `--seq` reads and writes the separator, but drops
text before the first one silently where jq names it on stderr.
Not implemented, and reported as an unknown option rather than quietly
ignored: `-C` (colorized output, which an agent would only have to strip
again), `-L` (no module system, so `include` has nothing to search),
`--stream-errors` (it reports truncated-parse errors, which whole-value
reads never produce), and `--build-configuration`.
## Supported bash syntax
Mirage Bash is a tree-sitter-bash parser plus a custom executor. It implements the constructs LLMs reach for most often. What is not supported returns a clear, parseable error so an agent can self-correct on its next turn.
@@ -119,7 +223,9 @@ Mirage Bash is a tree-sitter-bash parser plus a custom executor. It implements t
- **Builtin options (GNU semantics):** `echo -n/-e/-E` (leading-word option rule: `echo hi -n` prints `hi -n`), `read -r`, `xargs -n/-0/-d/-r/--` (batching, GNU exit codes: `123` when an invocation fails, `126`/`127` stop the run), `timeout DURATION` with `s`/`m`/`h`/`d` suffixes (kills at the deadline with exit `124`, usage errors exit `125`). `shift` and `return` report bash's `numeric argument required` errors.
- **Name lookup:** `type name` reports what a name resolves to (`type -t` prints one of `keyword`, `function`, `cli`, `builtin`; `type -a` lists every layer holding the name), `which name` prints the name of anything runnable (there is no PATH, so there is no path to print) and reports a miss through exit `1` alone, and `man name` renders a page: a command's spec, or an installed CLI's own `--help` tree (`man linear issue create`).
- **Globs:** `*`, `?`, `[...]` classes and `[!...]` negation (Python `fnmatch` semantics in both implementations), resolved by the shell or pushed down to the resource.
- **Comments:** `#`.
@@ -128,8 +234,9 @@ Mirage Bash is a tree-sitter-bash parser plus a custom executor. It implements t
- **Job control:** `bg`, `disown`. (`fg`, `jobs`, `wait`, `kill`, `ps` work; use the `--background` flag and `mirage job` CLI for long-running work.)
- **Output process substitution:** `>(cmd)` (the `<(cmd)` direction works).
- **Builtin options with no process backing:** `xargs -I`/`-P` (exit `1`) and `timeout -s`/`-k`/`--preserve-status` (exit `125`) return an `unsupported option` error: commands run as coroutines inside the workspace, so there is no process to signal or parallelize.
Each returns `exit_code 2` with stderr `mirage: unsupported builtin: <name>` or `mirage: unsupported: process substitution >(...)`.
Each returns `exit_code 2` with stderr `mirage: unsupported builtin: <name>` or `mirage: unsupported: process substitution >(...)`, except the builtin options above, which use the listed GNU-shaped exit codes.
### Syntax errors
@@ -139,13 +246,15 @@ Commands the parser cannot make sense of return `exit_code 2` with stderr `mirag
The daemon's `--background` flag detaches a job and returns a job id. It is not the same as the bash `&` operator, which the shell does support inline (`sleep 30 &`). Use `&` for in-shell job parallelism, `--background` (or `mirage job`) for long-lived work that should outlive the request.
## Per-session mount capability
## Per-session mount modes
A session can be created with an explicit allowlist of mount prefixes. Any command whose path resolves to a mount outside that list is rejected with `mirage: session 'agent' not allowed to access mount '/X'` and exit code 1. Default sessions (no allowlist) keep their current unrestricted behavior, so existing code is unaffected.
A session can be created with its own per-mount modes, like a container that mounts the same volume `ro` while another mounts it `rw`. Each listed prefix carries a mode ceiling on the `read < write < exec` ladder, written as the words `read`/`write`/`exec` or the cumulative filesystem aliases `r`/`rw`/`rwx` (exec implies write implies read, so bit-style forms like a bare `w` are rejected). A command touching a mount the session was not given is rejected with `mirage: session 'agent' not allowed to access mount '/X'` and exit code 1; a command exceeding the session's mode fails exactly like it would on a read-only mount.
This is a soft boundary, enforced inside the daemon process, not an OS or process-level isolation. Use it to shrink the blast radius of prompt-injection in multi-agent workspaces: a Slack-only agent cannot pivot to read `/linear`, `/github`, or any other mount it was not given.
If a session is created without `mounts`, it is unrestricted: every mount behaves per its own configured mode. A list of prefixes (instead of a mapping) restricts the session to those mounts but keeps each at its own mode, and a bare `-m /data` on the CLI does the same for one mount. A session's mode can only narrow, never widen: the effective permission is the weaker of the mount's own mode and the session's mode, so `rw` on a `READ` mount is still read-only.
The check fires for every code path that reaches a mount: shell commands (`cat`, `ls`, ...), redirects (`>`, `<`), cross-mount `cp`/`mv`, `wget -O`, `curl -o`, command substitution `$(...)`, subshells `(...)`, pipes, `&&`/`||` chains, background jobs, and the programmatic `ws.ops.read/write/...` API. Two infrastructure prefixes are always allowed regardless of the allowlist: the observer prefix (`/.sessions`, where command history is recorded) and the cache mount (`/_default`, wherestateless text-processing commands like `wc` live).
This is a soft boundary, enforced inside the daemon process, not an OS or process-level isolation. Use it to shrink the blast radius of prompt-injection in multi-agent workspaces: a Slack-only agent cannot pivot to read `/linear`, `/github`, or any other mount it was not given. Note that FUSE mounts are not scoped by session modes: a FUSE mount is a host-level surface served under the default unrestricted view (mount modes still apply, session narrowing does not).
The check fires for every code path that reaches a mount: shell commands (`cat`, `ls`, ...), redirects (`>`, `<`), cross-mount `cp`/`mv`, `wget -O`, `curl -o`, command substitution `$(...)`, subshells `(...)`, pipes, `&&`/`||` chains, background jobs, and the programmatic `ws.ops.read/write/...` API. Infrastructure prefixes are always accessible: the history view (`/.bash_history`, which the `history` builtin and the GNU histfile render from) and the implicit scratch root (`/`, where stateless text-processing commands like `wc` resolve when given no path). A user-defined `/` mount is not infrastructure; sessions must be given `/` explicitly to touch it.
# mirage: session 'slack-agent' not allowed to access mount '/linear'
@@ -178,7 +288,7 @@ The check fires for every code path that reaches a mount: shell commands (`cat`,
</Tab>
</Tabs>
The allowlist is a property of the session, so it covers every command issued under that `session_id`, including subshells, pipelines, and recursive `bash -c '...'`. It does not change `MountMode`: a write to a mount in the allowlist is still rejected if the mount is `READ`. The two checks compose.
The modes are a property of the session, so they cover every command issued under that `session_id`, including subshells, pipelines, and recursive `bash -c '...'`. They do not change the mount's own `MountMode`: a write to a session-writable mount is still rejected if the mount itself is `READ`. The two checks compose.
| `mirage session create WS [--id NAME]` | Add a named session (own cwd + env). |
| `mirage session create WS [--id NAME] [-m /prefix[:mode]]...` | Add a named session (own cwd + env), optionally restricted to mounts with a mode ceiling (`read`/`write`/`exec` or `r`/`rw`/`rwx`). |
| `mirage session list WS` | List sessions for a workspace. |
| `mirage session delete WS SESSION` | Close a session. |
| `mirage execute --workspace_id WS [--session_id S] [--background] --command "..."` | Run a command. Pipes stdin automatically when stdout is not a TTY. |
description: Compare Mirage resources by mount mode and setup path, organized by category.
description: Compare Mirage resources by access, runtime, and setup path.
icon: grid-2
---
@@ -8,7 +8,7 @@ icon: grid-2
Use this page to pick the first resource to try. If you want the fastest path, start with RAM or disk. If you need real integrations, jump from the setup guide to the resource docs.
The **Mount Mode** column shows which `MountMode` values the resource supports (`read`, `write`, and `exec`, the last makes a mount executable so commands can launch binaries from it; typically used with Disk or RAM). The **Docs** column shows where each resource is available, with one icon per supported runtime: <Icon icon="python" /> Python, <Icon icon="node-js" /> TypeScript (Node), and <Icon icon="globe" /> TypeScript (Browser). Click an icon to jump to that runtime's docs.
The **Access** column separates filesystem access from API actions. `read`, `write`, and `exec` are `MountMode` values; `exec` allows commands to launch binaries from the mount and is typically used with Disk or RAM. `actions` means the resource also exposes mutating commands such as sending a message or creating an issue; run those commands on a `WRITE` mount. The **Docs** column shows where each resource is available, with one icon per supported runtime: <Icon icon="python" /> Python, <Icon icon="node-js" /> TypeScript (Node), and <Icon icon="globe" /> TypeScript (Browser). Click an icon to jump to that runtime's docs.
Sections below mirror the **Setup** sidebar so you can pick a category and follow the same path through credentials → resource docs.
@@ -16,7 +16,7 @@ Sections below mirror the **Setup** sidebar so you can pick a category and follo
No external setup, these run locally or against a connection string you already have.
| OneDrive | read, write | [OneDrive](/home/setup/onedrive) | [<Icon icon="python" />](/python/resource/onedrive) [<Icon icon="node-js" />](/typescript/setup/onedrive) [<Icon icon="globe" />](/typescript/setup/onedrive) | Files and folders through Microsoft Graph |
| SharePoint | read, write | [SharePoint](/home/setup/sharepoint) | [<Icon icon="python" />](/python/resource/sharepoint) [<Icon icon="node-js" />](/typescript/setup/sharepoint) [<Icon icon="globe" />](/typescript/setup/sharepoint) | Document libraries through Microsoft Graph; `site`, `drive`, and `key_prefix` can scope a mount to one folder subtree |
| OpenHands | [<Icon icon="python" />](/python/agents/openhands) | For the OpenHands agent SDK. |
| Claude Code (CLI) | [<Icon icon="python" />](/python/agents/claude-code) [<Icon icon="node-js" />](/typescript/agents/claude-code) | Mount via FUSE; run `claude` against the mountpoint. |
| Codex (CLI) | [<Icon icon="python" />](/python/agents/codex) [<Icon icon="node-js" />](/typescript/agents/codex) | Mount via FUSE; run `codex` against the mountpoint. |
| Codex (CLI and app) | [<Icon icon="python" />](/python/agents/codex) [<Icon icon="node-js" />](/typescript/agents/codex) | Python uses FUSE; TypeScript provides the plugin with Mirage tools and stale-write protection, plus FUSE. |
| Grok Build | [<Icon icon="python" />](/python/agents/grok-build) [<Icon icon="node-js" />](/typescript/agents/grok-build) | Python uses FUSE; TypeScript provides the Grok plugin with Mirage tools, plus FUSE. |
description: Set up a MongoDB connection for the GridFS file-storage resource.
---
GridFS stores files inside MongoDB (metadata in `fs.files`, content chunks in `fs.chunks`). The GridFS resource only needs a MongoDB connection URI plus a database name; the connection is set up exactly like the [MongoDB resource](/home/setup/mongodb).
## Credentials
### 1. Get Your Connection URI
#### Local MongoDB
```bash
# Default local instance
MONGODB_URI=mongodb://localhost:27017
```
#### MongoDB Atlas (Cloud)
1. Go to https://cloud.mongodb.com
1. Select your cluster -> **Connect** -> **Drivers**
# in-process duplicate (shares remote resources, restores local content fresh)
cp = await ws.copy()
```
```typescript TypeScript
// TODO(snapshot-ts): API not implemented yet.
// Tracking parity with Python's Workspace.snapshot / Workspace.load.
import { DriftPolicy, Workspace } from '@struktoai/mirage-node'
// capture to a tar file
await ws.snapshot('run.tar')
// replay with strict drift detection (default)
const restored = await Workspace.load('run.tar')
// or restore the structure while reading current remote content
const current = await Workspace.load('run.tar', {
driftPolicy: DriftPolicy.OFF,
})
// in-process duplicate
const copy = await ws.copy()
```
</CodeGroup>
Both `snapshot` and `copy` are `async` because fingerprint capture stats each touched path on a `SUPPORTS_SNAPSHOT` mount.
TypeScript file-path snapshot I/O is available in Node. In the browser, `Workspace.load(snapshotBytes)` accepts a `Uint8Array`, and `Workspace.fromState(...)` restores a state object; writing the tar to a file or download target is the application's responsibility.
Both `snapshot` and `copy` are async because they serialize workspace state and collect recorded fingerprints.
## Versioning: commit, checkout, clone
@@ -113,7 +128,7 @@ Paths that carry a revision pin are skipped — the pinned read serves the exact
| `STRICT` *(default)* | Raise `ContentDriftError` on first mismatch. | Reproducing an agent run; you want to know the world moved. |
| `OFF` | Skip drift checks entirely. Evict snapshot cache for fingerprinted paths so reads serve current. | You only wanted the workspace skeleton, not the bytes. |
Pass via `Workspace.load(..., drift_policy=DriftPolicy.OFF)`.
Pass via `await Workspace.load(..., drift_policy=DriftPolicy.OFF)`.
## How It Composes With Caching
@@ -146,29 +161,25 @@ The cache is the optimization, the fingerprint is the verifier, and the pin is t
## Resource Support Matrix
Snapshot support is **opt-in per resource** via `SUPPORTS_SNAPSHOT = True`. Resources without it surface in a load-time warning and serve current state with no drift detection.
Remote drift detection is opt-in per resource through `SUPPORTS_SNAPSHOT` in Python and `supportsSnapshot` in TypeScript. A working adapter must also attach a fingerprint to each read record; a revision is optional and enables pinned replay.
Legend: ✅ = supported · 🟡 = adapter needed (no work in flight) · 📝 = planned · ❌ = not applicable.
Legend: ✅ = implemented · 🟡 = resource opts in, but recorded reads do not yet carry the fingerprint · ❌ = live-only · — = unavailable in that runtime.
| Supabase | ✅ | ❌ | 📝 | ❌ | `ETag` only | Inherits `S3Resource`. Supabase's S3-compat endpoint does not surface object `VersionId`, so drift detection works but pinning is not available. |
| S3 and S3-compatible object stores | ✅ | ✅ | ✅ | When `VersionId` is available | Uses `ETag`; compatible providers without object versions still get drift detection. |
| OneDrive | ✅ | — | — | ✅ | Uses Microsoft Graph `cTag` plus the current version id. |
| SharePoint | ✅ | — | — | ✅ | Same Microsoft Graph version flow as OneDrive. |
| Hugging Face resources | 🟡 | 🟡 | — | ❌ | Resources opt in, but read records do not yet include the stat fingerprint. |
| Nextcloud | 🟡 | — | — | ❌ | The resource opts in, but read records do not yet include the WebDAV ETag. |
| GitHub and Google Drive | ❌ | ❌ | ❌ | ❌ | Current reads are live-only; revision-aware replay is not wired yet. |
| Other remote resources | ❌ | ❌ | ❌ | ❌ | Snapshot restores their config/state, then reads current remote content. |
RAM, Disk, and Redis serialize their resource state into the snapshot. They do not need a remote drift check or revision pin: replay restores the captured state directly. Credentials and connection details remain redacted and may require a resource override at load time.
d="M 119.18,86.64 98.02,57.3 c 0,0 0,0 0,0 L 63.77,9.8 c -1.74,-2.4 -5.76,-2.4 -7.49,0 l -34.24,47.49 c 0,0 0,0 0,0 L 0.87,86.64 c -0.86,1.2 -1.1,2.73 -0.65,4.13 0.46,1.4 1.55,2.5 2.95,2.96 l 55.41,18.14 c 0,0 0,0 0.01,9e-4 0.46,0.15 0.94,0.23 1.43,0.23 0.49,0 0.97,-0.08 1.43,-0.23 0,0 0,0 0.01,0 L 116.87,93.73 c 1.4,-0.46 2.5,-1.55 2.95,-2.96 0.46,-1.4 0.22,-2.93 -0.65,-4.13 z m -59.15,-66.25 22.21,30.8 -20.77,-6.8 c -0.16,-0.05 -0.33,-0.04 -0.49,-0.08 -0.16,-0.04 -0.32,-0.06 -0.48,-0.08 -0.16,-0.02 -0.31,-0.08 -0.47,-0.08 -0.16,0 -0.31,0.06 -0.47,0.08 -0.17,0.02 -0.32,0.04 -0.48,0.08 -0.16,0.03 -0.33,0.03 -0.48,0.08 h 0 l -20.64,6.76 -0.13,0.04 22.21,-30.8 z m -31.38,43.52 24.18,-7.92 2.58,-0.84 V 101.12 L 12.06,86.92 Z m 36,37.2 V 55.15 l 26.76,8.76 16.59,23 z"/>
Same for the `_vfs` examples (`patchNodeFs`): Python reports `4 ops, 20760 bytes`, TS reports `0 ops, 0 bytes`.
## Cause
In TypeScript, `ws.fs.readFile` / `ws.fs.readdir` (used by `patchNodeFs` and the FUSE layer in `packages/node/src/fuse/fs.ts`) bypass the record-keeping dispatch, so nothing lands in `ws.records`. In Python, the equivalent VFS and FUSE paths go through `apply_io`, which appends `OpRecord`s, so `ws.ops.records` counts every read.
## Scope
Cross-backend, not notion-specific: Linear's TS `_vfs`/`_fuse` examples have the same gap, and any resource read through `ws.fs` or a FUSE mount is unrecorded. Command execution (`ws.execute`) records normally in both languages.
## Fix direction
Route `ws.fs` reads (or at least the FUSE/`patchNodeFs` entry points) through the same dispatch/recording layer the executor uses, mirroring Python's `apply_io` accounting. Then the example `Stats:` lines become comparable across languages.
## Status
Open. Found 2026-06-09 while verifying notion example parity (PR #226); deferred because the fix belongs in the workspace fs op layer, not per-backend.
This pulls in `camel-ai>=0.2.40,<0.3` and `markitdown>=0.1.5`. Note: `mirage-ai[camel]` is mutually exclusive with `[openai]`, `[openhands]`, and `[pydantic-ai]`, camel pins `pydantic<=2.12.0` while the other agent SDKs require `>=2.12.2`. Pick one stack per environment.
This pulls in `camel-ai>=0.2.90,<0.3` and `markitdown>=0.1.5`. Note: `mirage-ai[camel]` is mutually exclusive with `[openai]`, `[openhands]`, and `[pydantic-ai]`: CAMEL pins `pydantic<=2.12.0` while the other agent SDKs require a newer release. Pick one stack per environment.
description: Run Anthropic's Claude Agent SDK against a Mirage workspace via an in-process MCP server exposing execute, read, write, edit, ls, and grep tools.
icon: /images/claude-logo.svg
---
The [Claude Agent SDK](https://code.claude.com/docs/en/agent-sdk/) builds agents on Claude. Mirage exposes any `Workspace` to the SDK as an in-process MCP server, so every file and shell operation the agent runs is routed through Mirage instead of the host filesystem.
This is distinct from [Claude Code](/python/agents/claude-code), which points the `claude` CLI at a [FUSE](/python/setup/fuse) mountpoint. Use this SDK integration when you build your own agent with `claude_agent_sdk.query()` and want Mirage tools rather than the built-in file tools.
## Install
```bash
uv add 'mirage-ai[claude-agent-sdk]'
```
## Usage
`build_options` wires a workspace into a ready-to-use `ClaudeAgentOptions`: it registers the Mirage MCP server, restricts the agent to Mirage's tools, and injects a system prompt describing the mounted paths.
```python
from claude_agent_sdk import query
from mirage import Workspace
from mirage.agents.claude_agent_sdk import build_options
from mirage.resource.s3 import S3Config, S3Resource
@@ -4,7 +4,7 @@ description: Run OpenAI's Codex CLI against any Mirage workspace by mounting it
icon: /images/openai-logo.svg
---
[OpenAI Codex](https://github.com/openai/codex) is OpenAI's coding-agent CLI. Like [Claude Code](/python/agents/claude-code), it operates on a real filesystem and doesn't expose a pluggable backend, so Mirage integrates by [FUSE-mounting](/python/setup/fuse) a workspace and letting you point `codex` at the mountpoint.
[OpenAI Codex](https://github.com/openai/codex) can use Mirage through either an installable [TypeScript plugin](/typescript/agents/codex) or a FUSE mount. The Python integration [FUSE-mounts](/python/setup/fuse) a workspace as a normal host directory, so Codex's built-in filesystem and shell tools operate on Mirage without a separate tool server.
## Install
@@ -17,18 +17,19 @@ Then install [OpenAI Codex](https://github.com/openai/codex) separately.
## Usage
```python
from mirage import MountMode, Workspace
from mirage import Mount, MountBackend, MountMode, Workspace
from mirage.resource.ram import RAMResource
with Workspace({"/": RAMResource()}, mode=MountMode.WRITE, fuse=True) as ws:
with Workspace(
{"/": Mount(RAMResource(), mode=MountMode.WRITE, backend=MountBackend.FUSE)}) as ws:
print(f"cd {ws.fuse_mountpoint} && codex")
input("Press Enter when done...")
```
The mountpoint behaves like a regular directory. Codex's `read_file`, `write_file`, and `run_shell` tools all dispatch through Mirage's ops layer.
The mountpoint behaves like a regular directory. Codex's built-in file and shell tools all dispatch through Mirage's ops layer. For the Codex app, open the mountpoint as the project folder instead of starting `codex` from it.
## Why FUSE instead of an SDK integration?
## FUSE or plugin?
The Codex CLI's tool surface isn't pluggable, it expects host file operations. FUSE makes those host operations *be* Mirage operations.
Use Python FUSE when you want Codex's built-in filesystem tools to see Mirage as an ordinary directory. This works in the CLI and app, but requires an OS FUSE driver and does not add Mirage's agent-level stale-write check.
You lose per-tool customization and Mirage's op-record telemetry; you gain zero integration effort and compatibility with every Codex feature. Same trade-off as [Claude Code](/python/agents/claude-code).
Use the [TypeScript plugin](/typescript/agents/codex) when you want named Mirage tools, Pi-style stale-write protection, and plugin installation through Codex. The plugin uses Mirage's standard MCP adapter internally.
description: Run Grok Build against any Python Mirage workspace by mounting it as a real filesystem via FUSE.
icon: terminal
---
[Grok Build](https://x.ai/cli) can use Mirage through either an installable [TypeScript plugin](/typescript/agents/grok-build) or a FUSE mount. The Python integration [FUSE-mounts](/python/setup/fuse) a workspace as a normal host directory, so Grok's built-in filesystem and shell tools operate on Mirage without a separate tool server.
## Install
```bash
uv add 'mirage-ai[fuse]'
```
Install Grok Build separately.
## Usage
```python
from mirage import Mount, MountBackend, MountMode, Workspace
from mirage.resource.ram import RAMResource
with Workspace(
{"/": Mount(RAMResource(), mode=MountMode.WRITE, backend=MountBackend.FUSE)}) as ws:
print(f"cd {ws.fuse_mountpoint} && grok")
input("Press Enter when done...")
```
The mountpoint behaves like a regular directory. Start `grok` there and its built-in file and shell tools dispatch through Mirage's ops layer.
## FUSE or plugin?
Use Python FUSE when you want Grok's built-in tools to see Mirage as an ordinary directory. This requires an OS FUSE driver and does not add Mirage's agent-level stale-write check.
Use the [TypeScript plugin](/typescript/agents/grok-build) when you want named Mirage tools, Pi-style stale-write protection, and Grok plugin installation. The plugin uses Mirage's standard MCP adapter internally.
description: Give a Haystack Agent a bash tool over a Mirage workspace via MirageShellTool.
icon: /images/haystack-logo.svg
---
[Haystack](https://haystack.deepset.ai) is deepset's framework for building LLM applications and agents. The integration is maintained by deepset and lives in their [haystack-core-integrations](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mirage) repository, so it installs as its own package rather than as a `mirage-ai` extra.
## Install
```bash
uv add mirage-haystack
```
<Note>
`mirage-haystack` pins an exact `mirage-ai` version rather than tracking the latest, because Mirage is pre-1.0. Check the [integration's `pyproject.toml`](https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/mirage/pyproject.toml) for which one, since installing it may move an existing `mirage-ai` in your environment.
</Note>
## Usage
Describe the mounts, wrap the workspace in a tool, hand the tool to an `Agent`.
```python
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
ChatMessage.from_user("How many lines in /s3/log.txt mention 'alert'?"),
])
print(result["messages"][-1].text)
```
The tool exposes a single `command` parameter, so the model writes ordinary bash and pipes across mounts. Its description is generated from the mount tree, so the model is told which paths exist without you writing a prompt for it.
## Guarding what the agent can do
Two controls, and they are not interchangeable.
`read_only=True` on a mount is the write boundary. Mirage refuses every write to that mount whatever command is used, so this is what prevents modification and deletion.
`allowed_commands` restricts which command names may run. It is checked against every command Mirage would execute, including ones nested in `$(...)`, backticks and subshells, so `ls "$(rm x)"` is rejected unless `rm` is allowed too. Treat it as steering, not a sandbox: allowing a command that runs other commands (`eval`, `bash`, `sh`, `source`, `xargs`) effectively allows anything.
Commands never reach the host shell either way. Mirage interprets them itself, so the blast radius is the mounts you attached.
## Exports
| Symbol | Purpose |
| --- | --- |
| `MirageWorkspace` | Declares the mount tree, and runs commands against it with `run` / `run_async`. Serializable with `to_dict` / `from_dict`. |
| `MirageMount` | One mount: `path`, `resource`, `config`, `read_only`. |
| `MirageShellTool` | The Haystack `Tool` that gives an `Agent` the bash surface. |
| `MirageError` | Base error, with `MirageConfigError` and `MirageCommandNotAllowedError`. |
## Links
- [Integration page](https://haystack.deepset.ai/integrations/mirage) on haystack.deepset.ai.
- [Source and README](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mirage) in `haystack-core-integrations`.
- [`mirage-haystack`](https://pypi.org/project/mirage-haystack) on PyPI.
description: Back deepagents and other LangGraph-style agents with a Mirage workspace via LangchainWorkspace.
title: LangChain (Deep Agents)
description: Back Deep Agents with a Mirage workspace via LangchainWorkspace.
icon: /images/langchain-logo.svg
---
[deepagents](https://github.com/langchain-ai/deepagents) is LangChain's framework for long-horizon coding agents. It accepts a pluggable `backend` for filesystem and shell operations, Mirage ships one.
[Deep Agents](https://github.com/langchain-ai/deepagents) is LangChain's framework for long-horizon coding agents. It accepts a pluggable `backend` for filesystem and shell operations, and Mirage ships one.
This pulls in `deepagents>=0.4.12`. Bring your own LangChain chat model (`langchain-anthropic`, `langchain-openai`, etc.).
This pulls in `deepagents>=0.6.12`. Bring your own LangChain chat model (`langchain-anthropic`, `langchain-openai`, etc.).
## Usage
@@ -41,19 +41,24 @@ agent = create_deep_agent(
result = agent.invoke({
"messages": [{"role": "user", "content": "Create /report.md and summarize."}],
})
for text in extract_text(result["messages"]):
for text in extract_text(result["messages"][-1:]):
print(text)
```
## Multimodal files
`read_file` can pass images, PDFs, audio, video, PPT, and PPTX files from a Mirage mount to a model as multimodal content. The selected model and provider must support the corresponding input type. Text files continue to use line-based pagination.
## Exports
| Symbol | Purpose |
| --- | --- |
| `LangchainWorkspace` | `Backend` implementation for deepagents, wires reads, writes, edits, and shell. |
| `LangchainWorkspace` | `SandboxBackendProtocol` implementation for Deep Agents, wires reads, writes, edits, search, and shell. |
| `extract_text` | Pulls the text content out of LangChain messages. |
| `build_system_prompt` | Generates a system prompt that describes mounted paths to the model. |
## Examples
- [`examples/python/agents/langchain/ram_pdf_deepagent.py`](https://github.com/strukto-ai/mirage/blob/main/examples/python/agents/langchain/ram_pdf_deepagent.py), RAM-backed PDF reading with no external storage credentials.
description: Implements pydantic-ai-backend's SandboxProtocol so pydantic-deepagents and any pydantic-ai agent can run inside a Mirage workspace.
title: Pydantic AI
description: Implement pydantic-ai-backend's SandboxProtocol so Pydantic AI agents can run inside a Mirage workspace.
icon: /images/pydantic-logo.svg
---
[pydantic-deepagents](https://github.com/vstorm-co/pydantic-deepagents) is a Claude Code–style deep agent harness built on [pydantic-ai](https://github.com/pydantic/pydantic-ai). It uses [`pydantic-ai-backend`](https://pypi.org/project/pydantic-ai-backend/)'s `SandboxProtocol` for filesystem, shell, grep, and edit operations, Mirage's `PydanticAIWorkspace` is a drop-in implementation of that protocol.
[Pydantic AI](https://github.com/pydantic/pydantic-ai) agents can use [`pydantic-ai-backend`](https://pypi.org/project/pydantic-ai-backend/)'s `SandboxProtocol` for filesystem, shell, grep, and edit operations. Mirage's `PydanticAIWorkspace` is a drop-in implementation of that protocol and also works with higher-level harnesses such as [pydantic-deepagents](https://github.com/vstorm-co/pydantic-deepagents).
## Install
@@ -12,7 +12,7 @@ icon: /images/pydantic-logo.svg
uv add 'mirage-ai[pydantic-ai]'
```
This pulls in `pydantic-ai>=1.35` and `pydantic-ai-backend>=0.1.0`. To use it with [pydantic-deepagents](https://github.com/vstorm-co/pydantic-deepagents):
This pulls in `pydantic-ai-slim[anthropic,openai]>=2.13.0` and `pydantic-ai-backend>=0.2.16`. To use it with [pydantic-deepagents](https://github.com/vstorm-co/pydantic-deepagents):
| `PydanticAIWorkspace` | `SandboxProtocol` implementation backed by a Mirage workspace. |
| `build_system_prompt` | Generates a system prompt that describes mounted paths to the model. |
`PydanticAIWorkspace` routes file operations through the Ops layer directly and shell operations through `Workspace.execute()` for full pipe and flag support. PDF reads are converted to images via `pages_to_images` so the agent can pass them as `BinaryContent`.
`PydanticAIWorkspace` routes file operations through the Ops layer directly and shell operations through `Workspace.execute()` for full pipe and flag support. Enable `image_support` and `document_support` on the console toolset to pass images and PDFs from any Mirage mount to multimodal models as native `BinaryContent`.
- [`examples/python/agents/pydantic_ai/s3_pdf_agent.py`](https://github.com/strukto-ai/mirage/blob/main/examples/python/agents/pydantic_ai/s3_pdf_agent.py), PDF page-to-image pipeline.
- [`examples/python/agents/pydantic_ai/s3_pdf_agent.py`](https://github.com/strukto-ai/mirage/blob/main/examples/python/agents/pydantic_ai/s3_pdf_agent.py), native PDF input from S3.
- [`examples/python/agents/pydantic_ai/slack_pdf_agent.py`](https://github.com/strukto-ai/mirage/blob/main/examples/python/agents/pydantic_ai/slack_pdf_agent.py), native image and PDF reads from Slack.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Zecheng Zhang
