gws, notion and github each take a --port and then hardcode 127.0.0.1.
That is right for every direct invocation and wrong inside a container:
the published port accepts the connection and then answers nothing, which
reaches the caller as an empty reply with no error anywhere explaining it.
--host leaves the default alone, so nothing that runs these on a host
changes. github also takes --advertise, because the base URL it writes
into html_url and clone_url is not always where it bound -- a container
binds 0.0.0.0, and 0.0.0.0 in a URL sends the caller nowhere.
The remaining integ servers carry the same hardcoded bind and are left
alone here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Also share the file-open umask rule between the plain and exec redirect
paths, and route the four new builtins through the existing option
scanner, refusal helper and single-quote renderer instead of their own.
integ-runtime sets INTEG_RUNTIME_STRICT=1, which turns an unmet suite
requirement into a failure unless the suite declares itself optional.
Every existing suite is satisfiable on a GitHub runner -- the workflow
even starts the docker container the docker suite needs -- so these two
are the first that genuinely are not: smolvm needs a hypervisor
(/dev/kvm) the hosted runners do not have, and sandlock needs its CLI
plus Landlock ABI v6 (Linux 6.12+), with no apt package to add cheaply.
Each carries an optional_reason naming what a host would need and when
the flag should come off, and the runner still prints the skip, so the
dropped coverage stays visible rather than reading as a pass.
Two codex P2 findings, both verified against GNU before acting.
`humanScaled` computed `n * 10` as a double. That leaves the safe-integer
range a little under a petabyte, and the product rounds down before the
ceiling, landing on the wrong tenth: 1914029841632461 bytes rendered
`1.7P` in TypeScript where GNU (`numfmt --to=iec --round=up`) and the
integer-exact Python say `1.8P`. A real py/ts divergence reachable from
`ls -h`, `du -h` and `df -h` on a large enough tree. The scaling now runs
in BigInt, which is what Python's arbitrary-precision ints were doing all
along, so the two sides compute the same way rather than agreeing by
luck. Both GNU tables gain the row; reverting to the float product fails
it.
The `divergence` exemption tested only for the key's presence, so
`"divergence": null` or `""` bought an asymmetric matrix while explaining
nothing -- the one thing the key exists to supply, and it would read as a
considered decision in review. Both runners now require a non-empty
string, with the six falsy/ill-typed shapes pinned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex P1: a python-only module under runtime/python left
check_layout_parity.py --strict red (297 against a 296 baseline). Making
sandlock a package the way monty already is moves the divergence from a
module to a directory, which the exceptions file excuses as a subtree --
the honest shape here, since there is no TypeScript counterpart to
mirror and growth inside it is expected rather than drift.
Constants and config move out of the runtime module to match monty's and
smolvm's layout: constants.py carries the env var, the CLI hint, and the
system read set; config.py carries SandlockConfig.
Docs: smolvm joins the sandbox provider table and gets its own section on
both language pages, covering the refused-state probe and the fact that
one machine is one guest, so concurrent lines share a filesystem and a
process table. sandlock joins the python runtime table with a section on
what "process" reach does and does not buy, and on the environment it
deliberately does not inherit. Vendor marks added under the existing
docs/images/<name>-logo.svg convention.
The job was copied from the `test` job beside it, which pins 22. It
runs no project code -- `pnpm audit` resolves the lockfile and queries
the advisory API -- so it is free to track the newer line `test_integ`
is already on, and there is no reason for it to trail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the priority items of #608 in both languages.
- let: (( )) as a builtin, ordered writes, last-value status
- umask: session mask applied to touch/mkdir/redirect creates
- shopt: full option table, plus nullglob/failglob/dotglob/globstar;
extglob is refused rather than stored, since the parser has no mode
- alias/unalias: definition, expansion (expand_aliases, same-line rule,
trailing-blank chaining), type/command -v reporting
- mapfile/readarray with -d -n -O -s -t -u -C -c
- read -a -d -n -N -t, plus the non-tty no-ops -p -s -e -i
- declare -g and declare -n: references resolve on every read and write
- disown, and wait -f/-n/-p
- exec: redirect-only form installs on the shell; the command form is
refused, having no OS process to replace
Two sandboxed execution options, both using the host CLI as transport so
neither adds an SDK dependency.
smolvm joins the RemoteSandbox family as a fourth provider alongside
docker/e2b/daytona, mirroring DockerRuntime's shape: `machine status
--json` is the liveness probe and each line is one `smolvm machine exec`
with the merged env, rebased cwd, real stdin, and separated stderr.
`machine exec` would start a stopped machine itself, but this family
never manages sandbox lifecycle, so connect() refuses anything but
"running" and names why — "unreachable" (VMM alive, guest agent dead)
and "frozen" (a fork base, paused by design) get their own recovery
hints rather than a bare not-running.
sandlock is a new python-tier engine: the same real CPython `local`
spawns, wrapped in `sandlock run -r ... -w ...` so Landlock and seccomp
bound where its I/O can land. It sits between the bridged engines and
`local` in the preference order — reach stays "process", because
confinement narrows which host paths the code touches while the
workspace gate still never sees the I/O, but a "process" runtime need no
longer be an unbounded one. Unlike `local` it does not inherit
os.environ, which would hand every live backend credential to confined
code.
Both stay out of DEFAULT_ENTRIES: sandlock is Linux-only, and binding it
ahead of monty would silently reroute python3 by host OS.
A node_modules that predates the lockfile does not announce itself. It
surfaces as a type error inside whichever package the drifted dependency
types, in a file nobody touched, which reads as a source bug.
The worked example: `pnpm -r build` failing at
packages/dsh/src/fs.ts:88 with
TS2345: Argument of type '"FS_TOO_LARGE"' is not assignable to
parameter of type 'FsErrorCode'
on a checkout whose install predated 3bbb79121, the commit that bumped
@deepseek-ai/dsh-fs 0.0.1-rc.1 -> 0.1.0-rc.6 and with it the FsErrorCode
union that 88698fa08 then used. Source and lockfile were both correct;
only the install was behind, and the message pointed at neither.
pnpm can say that itself, so let it. `pnpm install` is the whole fix.
The setting has to live here rather than in .npmrc: inside a workspace
pnpm overwrites .npmrc settings from this file, so a
verify-deps-before-run= line there resolves in `pnpm config get` and is
still never consulted.
No effect on CI, which installs from the lockfile before it builds, and
none on a pull: verifyDepsBeforeRun is absent from pnpm's workspace-state
settings snapshot, so taking this commit does not itself invalidate an
up-to-date node_modules.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`df -H` is powers of 1000, and GNU prints that kilo as `k` -- mirage
printed `K`, which is the 1024 unit. Confirmed against a real `df -H` on
a 100 KiB tmpfs, which GNU renders `103k` while `df -h` on the same
mount renders `100K`; `numfmt --to=si` and `--to=iec` agree. Only kilo
differs in case, M and up are capitals in both tables.
Nothing pinned `df -H` at all, in either language, which is why the
letter could be wrong since the flag was written. Both sides now assert
the quota fixture's three numbers under `-h` and `-H` against GNU, and
reverting the unit table fails the `-H` assertion.
This is the SI half of the same `human_readable` engine the previous
commit fixed for `-h`; the rounding was already shared and already
correct here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the two remaining small items of Block E. Both were largely
recon: much of what they filed had already been fixed, and the part
that had not turned up a real GNU divergence.
Item 31 (T3-7), rule/doc reconciliation
---------------------------------------
The `object`-parameter half is already done: #825's mypy inversion took
the package from the filed 28 down to 7, of which 3 are docstring prose
and the remaining 4 are protocol methods (`__contains__`, `pop`) whose
signatures typeshed dictates, all already listed in the no-object gate.
`utils/errors.py` reads `str | PathSpec` today, and `builders/sed.py` no
longer holds flag values at all -- item 22 moved that to the generic.
That left the nested-function rule, which the plan asked to decide
before gating. Measured: 39 nested defs, and **every one of them closes
over its enclosing scope** -- 38 by free variable, and `sed_helper._repl`
by binding through parameter defaults, the loop-variable idiom, with a
comment saying so. A flat "do not nest" is a rule the architecture
cannot keep (op factories, provision builders, the read-through cache,
every decorator's wrapper), so gating it as written would have meant 39
standing violations.
So the rule now says what it was reaching for: a nested def must capture
the scope around it, or it belongs at module level. That is mechanical
and enforceable -- `tests/test_nested_functions_are_closures.py` reads
free variables from `symtable` and parameter defaults from the AST. It
passes on the tree as-is and fires on a helper that reads only its own
arguments.
Layout: the eight test-only TS directories are flattened
(`ram/{cat,cut,grep,head,ls,tail,wc}/`, `ssh/ls/`), `awk_helper.ts` moves
up beside its nine siblings, and the `provision.ts` / `_provision.ts`
split is settled on Python's convention -- `_provision` for a backend's
own, bare `provision` for the shared ones (cli, generic_bind). That last
one was not cosmetic: Python already had `_provision.py` for github,
gmail, redis and email, so renaming **closed 4 real parity divergences**
(layout baseline 296 -> 292). The `findEval`/`findParse` line was
already done in #827.
Item 32 (T3-8), test/CI tidiness
--------------------------------
The two asymmetric `ls` conformance matrices are raised to symmetry, and
they pass: python `[ram,disk,redis]` vs typescript `[ram]` was stale
caution, not a divergence. Both runners now reject an asymmetric matrix
at load time unless the case carries a `divergence` key explaining why --
a case is a parity claim, and narrowing one side reads as coverage while
the side that still lists the backend goes green. Loading the corpus
under the new assertion proves no other case was asymmetric. The README
documented the override as "not yet needed"; it exists now.
Adds the `ts-audit` job, mirroring `test_python.yml`'s exactly (same
`continue-on-error`, same out-of-gate placement). Nothing ran `pnpm
audit` for the TS tree while `typescript/package.json` carried 27
hand-written CVE overrides that only a person remembering to check kept
current.
Integ facets for `ls`: `-A`, `-d`, `-r`, `-S`, `-h`, all pinned against
GNU coreutils 9.7 in docker. `-S` uses regular files only, because GNU
sorts a directory by its inode size while mirage counts it as 0 -- a
divergence CLAUDE.md documents deliberately.
The `-h` facet found a real bug
-------------------------------
`ls -h` disagreed with GNU three ways, and because the flag had zero
cross-backend coverage nothing caught it. GNU prints a count below one
unit with no suffix (`24`); mirage printed `24B`. GNU rounds *up* to the
precision shown (1025 bytes is `1.1K`); mirage gave `1.0K`. GNU drops
the decimal once the value reaches ten (`10K`); mirage gave `10.0K`.
One shared engine replaces both formatters in both languages -- GNU runs
`-h` and `-H` through one `human_readable`, and so do we now. Rounding
up can carry past the base (1048575 ceils to 1024K, which GNU shows as
`1.0M`), so the unit is re-chosen after rounding rather than once up
front. 21 GNU-read points are pinned as a table in each language.
Blast radius, all re-pinned: five integ cases that had recorded the `B`
suffix (du, discord, langfuse, email, gmail), and the du fan-out tests.
Those last ones needed care rather than a new number: they exist to
prove the total is humanized once instead of twice, and their 1500+1500
stops discriminating under correct rounding, since 3000 bytes and two
`1.5K` readings both render `3.0K`. They now use 1025+1025, where
single-rounding gives `2.1K` and double-rounding would give `2.2K`.
Verified: pre-commit clean, integ 8054/0 on ram+disk+redis in both
languages, conformance green in both, and the layout, spec, barrel,
docs, case-target and PathSpec gates all pass.
Not covered, still open on item 32: integ facets for `ls -t` (mtime
order is not stable across backends without a seeded fixture), mktemp,
gzip, the checksum `--check` companions, `df -H/-k/-a`, and `cmp
-n/-b/-i`; and the T2-7 helper unit-suite mirroring. Item 31 leaves the
provision *presence* sets unreconciled (Python has lancedb/qdrant, TS has
trello) -- that is a question about which backends should provision at
all, not about naming.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Conflicts, both from ratchets/config the two branches moved independently:
- python/pyproject.toml: #825 inverted the mypy config (global strict +
an opt-out list of not-yet-annotated modules) and deleted the old
opt-in allowlist my last commit had edited. Took upstream's file
wholesale -- github.pushdown now gets strict from the global default
and passes it, so the retarget commit is subsumed. Retargeted the one
stale entry the rename left in the NEW list:
generic.rm_command -> generic.rm_cmd (the module still needs its
exemption: one no-any-return at rm_cmd.py:89), keeping its exempt
status exactly as it was under the old name.
- spec/layout_exceptions.json: both sides lowered the baseline from 299
(mine to 293, upstream's to 296). Neither value is right after the
merge; ran the gate, which reports 290 -- the two closure sets are
disjoint (299 - 6 - 3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The core/github/scope.py -> pushdown.py rename left the pyproject
per-module mypy override pointing at the old name, silently dropping
disallow_untyped_defs/warn_return_any coverage for the module.
(codex P2 on #827)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four review findings, three of which the TypeScript side shared, so both
sides are fixed and tested together.
- file_version stamped the bytes handed to `write` while every later
check reads them back. On a mount whose read op renders, the agent's
own next write was refused as somebody else's change. Both sides now
stamp what a later read returns.
- version stamps were keyed by the caller's spelling, but read and write
follow the namespace symlink table, so `/alias` and `/target` got
separate stamps and an edit through the other name skipped the
staleness check. Keyed through `namespace.follow` now.
- the grep tool reported every exit code as success. Exit 1 is "no
match" and stays a success; >1 (an unreadable path) is now an error.
- `Server[object, object]` discarded the SDK's context types. It is
`Server[dict[str, Any], Any]`: the default lifespan yields an empty
dict and nothing here reads a request payload. The no-object gate only
swept function signatures, so it could not see a type argument in a
variable annotation; extended to annotated assignments, which found
one more (`sort_helper.seen`, now `set[DedupKey]`).
Also narrows `Resource.resolve_glob` from `list[str | PathSpec]` to
`list[PathSpec]`. The union is the argv type -- a command line really
does mix text words and paths -- and it had leaked a layer past the
split into a function that only ever takes paths. TypeScript's twin is
already `readonly PathSpec[]`, all 39 implementations take PathSpec, and
every call site passes one. Nothing checked that, because
`make_resolve_glob` returned `Callable[..., Any]`; it now returns a
`ResolveGlobFn` protocol, and `ResolveGlobOp` (the consumer side) drops
the same union. Re-widening any override now fails two ways.
`tests/cli/test_mcp.py` asserted on rendered `--help` text, which is laid
out to the terminal's width; it passed locally and wrapped away on CI.
It reads the declarations instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- coerce -i/-l/-u before the pre_session gate, so a rule judges the value that lands
- ${m[k]:=v} and ${a[3]:=v} land on the named element; [@]/[*] and a bad index refuse
- arithmetic writes land in evaluation order across bare and subscripted targets, and a bare array name reads as element 0
- -i resolves element references (resolver moved from session/elements into session/state, no cycle)
- test -v unquotes an associative subscript
3bbb79121 rewrote agents/langchain/backend.py to follow deepagents' new grep
cap -- GrepResult.truncated and the keyword-only max_count on grep/agrep -- and
bumped python/uv.lock and typescript/packages/agents/package.json to match. It
left python/pyproject.toml at deepagents>=0.6.12.
0.6.12 is the last release before that API. Its GrepResult is a two-field
dataclass, so constructing one with truncated=True raises TypeError, and
BackendProtocol.grep has no max_count for the override to match. Both landed in
0.7.0, not 0.7.6 as that commit's message says -- checked by reading
backends/protocol.py out of every published 0.6.x and 0.7.x wheel.
Nothing in CI resolves the floor, which is why it stayed green: uv.lock pins
0.7.6, and Test (Install) only exercises a base install with no extras. The gap
is reachable from a downstream pip install mirage-ai[deepagents].
Raised to >=0.7.6, matching the lock and what the TypeScript side already got.
The floor is the version we test against, which is 3bbb79121's own rationale.
No behavior change: backend.py is untouched and already conforms to the 0.7.x
contract on every clause it specifies -- a total cap rather than a per-file one,
truncated only when matches were actually dropped, and exactly max_count with
none dropped reported complete.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two cleanup-plan items, both Python-side.
Item 29 -- `mirage mcp`. TypeScript shipped a six-tool stdio MCP server;
Python had none, so a pip-install user could not point Cursor or Claude
Desktop at a workspace and `mirage --help` differed by distribution.
Adding the entry point alone would have duplicated the tools, because
this side kept them private inside the Claude Agent SDK integration, so
the shared layer comes first:
- agents/tool_descriptions.py -- the six strings, one copy.
- agents/tool_operations.py -- MirageToolOperations, lifted out of the
SDK server's private _MirageTools.
- agents/file_version.py -- stale-write protection, which this side
lacked entirely. TS stamps stored bytes; here the stamp covers the
rendered bytes, because this read tool has always rendered and an
edit must search what the agent was actually shown.
- agents/mcp/server.py + cli/mcp.py -- the server and `mirage mcp`.
- server/workspace_config.py -- config discovery (candidates, env
names, walk up from cwd), which Python had nowhere, so every entry
point had to be handed an explicit path.
The server is the low-level MCP Server rather than FastMCP: FastMCP does
not forward a version, and TS advertises one. Handlers are bound methods,
not decorated closures, so nothing nests.
Item 28 -- the mypy allowlist. 54 modules opted *in* to annotation
checking against 1826, so the default was unchecked and every new file
joined the unchecked side. The default is now strict, with a list of
what is not yet annotated that only shrinks. 166 annotations cleared
along the way; the remainder is named module by module.
Two real defects surfaced by the annotations, neither of them typing:
- Workspace._original_open / _original_os were invented by assignment
in lifecycle.patch_process, so unpatch without a patch raised
AttributeError. Declared, and the restore is guarded.
- sed_generic declared a non-optional writer while its own docstring
and its `write_bytes is None` branch said otherwise; the builder
passes None whenever the backend cannot write.
Tests keep the PathSpec rule instead of full strict: measured, full
strict on python/tests is 2374 errors, of which 634 are `str` where a
pydantic field declares SecretStr -- which pydantic coerces at runtime --
and most of the rest is the monkeypatched-fake pattern CLAUDE.md
sanctions. The rule that is violated for real is PathSpec, 19 times, and
scripts/check_test_pathspec.py now holds that line. One of the 19 was a
latent AttributeError: tests/e2e passes a str to s3 write_bytes, which
reads .mount_path, and the test skips without a live versioned bucket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
readonly -f freezes a function: either definition syntax and unset (-f
or auto) refuse in GNU's voice, exit 1, and the old body stays; a
non-function operand is 'not a function'; the bare form lists the
frozen set. The set is a session field beside functions, inherited by
forks and child shells. declare -rf freezes the same way, -F prints
names, and -f prints a name row where GNU pretty-prints the body, which
mirage does not carry.
jobs gains -l -n -p -r -s, a jobspec filter (%N or N), 'no such job'
exit 1, and the usage refusal exit 2, all applied to mirage's own row
shape: -p prints the table id where GNU prints a pid, and -s lists
nothing since mirage jobs never stop.
Pinned against bash 5.2.37; 21 unit cases per language, 3 battery
cases on 22 targets, both hosts byte-identical.
Phase 1 of the core/ restructure plan: introduce mirage/core/render/json.py
and core/render/json.ts (json_text/json_bytes, compact_json_text/bytes,
jsonl_bytes) with a cross-language byte-parity fixture, and migrate every
core/ backend render site onto the kit: langfuse + jaeger (generic render
modules deleted), mem0, discord, slack (history/search/users), gcal, gdocs,
gsheets, gslides, gmail, qdrant, email, and postgres-TS. linear/notion/trello
normalize helpers become thin delegates (their created_at sort stays local;
the ~90 CLI call sites keep their names for a follow-up audit). jq/format
and py-postgres (orjson) deliberately stay local.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
awk splits statements at brace depth zero and outside quotes, and a
`{ ... }` compound contributes its inner statements in place, so
`{{print $1}}` prints the first word the way gawk does instead of being
refused; the chroma golden that had pinned the old empty output is
corrected.
find -printf renders %m/%M from a reported mode (chmod overlay) with the
per-kind constants as the fallback, and %Y on a symlink row reports the
target's kind through the namespace (N when dangling).
TS parseDateExpr truncates fractional seconds instead of rounding into
the next second, matching Date parsing and Python.
The Notion fake validates every child of a block append before shifting
siblings or inserting, so a refused request mutates nothing.
The value-shaping attributes apply in the session door on every write,
which is where bash applies them: -i evaluates the incoming text as
arithmetic (a bad expression refuses in the evaluator's voice, fatal for
a plain assignment and exit 1 for a builtin), -l/-u fold case, and each
applies per element on an array. A declaration pre-marks the name so its
own value coerces too, after the local snapshot so a function scope is
not leaked. -l and -u are exclusive; a cluster naming both sets neither.
n+=3 on an integer name adds rather than concatenates.
+attr turns an attribute off through the gated mark door; +r on a
readonly name and +a/+A on an array refuse as GNU does, and an unknown
letter refuses with the usage line, exit 2. -t and -n are stored and
printed; nameref aliasing is deliberately not wired, since it is a seam
through every expansion site and a partial alias is worse than none.
Pinned against bash 5.2.37; 51 unit cases per language, 7 battery cases
on 22 targets, both hosts byte-identical.
The loader read a script before it stat'd it, and only a real
filesystem answers a directory read with EISDIR: WebDAV serves the
collection's HTML as bytes (nextcloud ran it as a script), ssh's raw
error carries no errno, and a keyed store answers ENOENT. The stat
probe now runs first in both languages. The directory cases give hf a
key to see (hf holds no empty directory), and the multi-line grep -r
cases sort, since box lists folders first.
sed -i writes for every command, not only s and d; multi-file output is
concatenated raw; an escaped delimiter inside an address regex no longer
cuts the address short, and \%re% custom delimiters parse.
awk refuses constructs it cannot run (exit 2, "unsupported construct")
instead of echoing their source; simple assignment and OFS now execute
and unset names render empty.
date -d parses ISO, @epoch and the gnulib relative grammar in both
languages (TS printed NaN with exit 0), invalid input is exit 1 with GNU
wording; TS strftime learns %F %c %C %g %G %V %U %W %h %k %l %n %P %q %r
%R %t %x %X, and Python expands %q ahead of strftime.
find -printf is implemented (%p %P %f %h %d %s %y %Y %m %M %T escapes),
registered on the CommandSpec so the format is not read as a path, and
stats through the dispatcher when no overlay stat is wired.
Fakes: the GitHub server strips the internal files list from list-commits,
accepts refs/heads/x on /contents and /commits, and reports files as
objects on one commit; the GWS Drive server evaluates fullText contains
and answers 400 for an unknown query field; the Notion server honors
after on block append with the live API's response and validation shape.
Integ: new unix cases for each command fix, awk_ofs golden corrected,
and cli-gh/cli-gws/cli-ntn cases that drive every fake fix through the
CLIs; ntn conformance re-pinned against the real binary.
declare -A / local -A / readonly -A build a real map: keyed and pair
literals, element and += writes, unset of one key, declare -p rendering
with GNU's trailing space and key quoting, kind conversion refusals,
[[ -v ]] on elements, and read/printf -v into an element. Arithmetic
gains element lvalues (a[i]++, m[k]+=2) through an ElementOps seam the
session fills, so $((a[i])) and ((m[k]=5)) resolve without the
evaluator learning what a session is. Two pinned divergences: keys walk
in sorted order, and a bare declare -A prints =().
Pinned against bash 5.2.37; 49 unit cases per language, 15 battery
cases on 22 targets, both hosts byte-identical.
grep --include/--exclude resolve by command-line order (last matching
rule decides, no-match default follows the first rule), pinned against
GNU grep 3.11; the bag's insertion order carries it through a new
FlagView.typed_order. is_create_mode stops at --, so a member named
-c after the terminator no longer reads as create mode. env -S and
--split-string are consumed on a shebang line instead of naming the
interpreter. A slash-carrying head word joins the admission context
so a path-pattern guard sees the executed file.
The revert missed integ/, so two CI jobs failed on code the unit suite
never imports: integ/root.py awaited to_workspace_kwargs twice (once
split across lines, which is why the line-based grep missed it), and the
github watch builder and the shared-battery adapter still called
GitHubResource.build.
Swept with an AST walk over every .py in the repo rather than a grep this
time, matching an Await whose call resolves to one of the reverted names;
it now reports none. Verified by rerunning the two failing steps:
integ/root.py exits 0, and the github battery is 23 passed 0 failed
against the fake.
Push mode showed only the hand-written FileEvent path. It now leads
with the three mappers that exist, says why you import one rather than
ask the mount for it, and keeps the hand-written version for a backend
without one.
Slack gets the two traps written down: the day is bucketed in UTC while
the client shows local time, and a thread reply belongs to the parent's
day because chat.jsonl renders conversations.history, which returns
parents only. The matrix drops Slack from planned and gives redis its
own row.
Hosts the endpoint Slack posts to, unwraps the event_callback envelope,
maps it with SlackEventHook and notifies the workspace, with a live
watch stream printing what comes out.
Verified against a real workspace: a message maps to
channels/incident__C0B0DB9K11T/2026-08-13/chat.jsonl, file_shared to
that day's files directory, and the mount serves both.
The type and the method bought one line: reaching a hook from a mount
instead of from an import. Nothing needs that, because a push payload
has no vendor-neutral shape, so the caller always knows the backend it
is decoding for. The integ battery proves it: the event mode has to
build a watchdog payload before it can call anything, so the call site
was disk-specific with or without a generic accessor.
Consumers construct the hook they want:
from mirage.core.slack.watch import SlackEventHook
hook = SlackEventHook(resource.accessor)
build_event_hook goes with them, since the resource method was its only
caller. What stays is to_events, which is the whole feature: the path
arithmetic that has to live beside the backend.
delta_hook stays as it is. pull(root, checkpoint) is the same call for
every backend, so one case set drives ten of them in integ; that is the
generic reach event_hook never had.
Both from review on #820.
ensure_tree decided hydration by whether accessor.tree held anything,
but an empty repository (or one holding only excluded gitlinks) hydrates
to {}. That read as never-hydrated, so every direct-tree command
refetched it, and twice per call once an index was wired: ensure_live_index
seeds an empty root, then the fallback fetch runs because the tree still
looks empty. Measured 2 fetches on the first ensure_tree and 2 more on
each later one; now 1 in every combination of empty/non-empty and
index/no-index. Tracked by accessor.tree_loaded, set by the constructor
when a tree is supplied and by every site that reseats one (refill_index,
the watch delta hook).
is_default_branch compared ref against a default_branch that is None
until something hydrates it, so it answered False for a mount pinned to
the default branch. A bare read hydrates only the tree, so that wrong
answer could last the life of the mount. It now returns None for not
known yet, and callers needing a definite answer await
ensure_default_branch first. TypeScript keeps a plain bool because
construction there fetches the fact; noted in the docstring.
The pull and push batteries both hand the watcher a FileEvent the
harness built, so the mapping half of the feature had no integ coverage
at all: nothing exercised to_events. Event mode reuses the same cases
and the same freshness checks, but triggers by handing the backend the
watchdog notification a real filesystem watcher would emit, so a
delivered change can only have come from the hook.
The per-case modes default was the pair [pull, push], which silently
skipped every case of the new mode and still reported OK. It is now
every mode, since a per-case list is how a case opts out.
chat.jsonl renders conversations.history, which returns parents only,
so a thread reply appears in no day file at all. What changes is the
parent's row, whose reply_count and latest_reply the same listing
carries. Bucketing by the reply's own ts refreshed today's directory
and left the parent's day stale, silently, because notifying a path
that exists but did not change looks like success.
affected_ts now returns the parent's ts for a reply, both for a
broadcast reply (Slack puts those in the channel history too), and the
message's own otherwise. Deletes and edits read thread_ts from the
message they describe, not the envelope.
Also records why file_change and file_deleted stay unmapped: neither
carries a conversation or the share day, and files.info on a deleted
file errors, but the index already stores each blob's Slack id, so a
reverse lookup would name the path once the hook is handed the index.
SupportsEvents had exactly one caller in the repo and it was a test
asserting the protocol itself; SupportsChanges had none at all, not
even a test. TypeScript never had either one and its Resource just
declares deltaHook?() / eventHook?(), which is the same feature working
without them.
BaseResource now answers delta_hook() / event_hook() with None, so a
consumer checks the return value instead of the resource's type. None
is the common case, not an edge: 52 of 55 resources have no event hook
and 26 have no delta hook.
`.some(isMountBlock)` is not a narrowing TypeScript can follow, so the
else branch still types `config.mounts` as holding a `MirageMountBlock`
and the assertion is what carries the fact. It was removed by an
eslint --fix run whose type information was momentarily unresolvable, so
the rule read the assertion as redundant; the build then failed in a
package the change had no business touching.
Our mypy gate reads the source tree, so it proves nothing to anyone who
installs us: without the PEP 561 marker a type checker refuses to read an
installed package's annotations and every mirage import resolves to Any.
Verified against the haystack integration, whose pyproject carries an
ignore_missing_imports override for exactly this reason; with the marker
its mypy run typechecks against the real signatures and passes.
A prefix assignment is a session write like any other and the form
exports the name for the command, but the pre-check only ran the hidden
half and the seeding goes through seed_var, the ungated door. A
deployment refusing SECRET_* still saw `SECRET_K=leak printenv SECRET_K`
print the secret. The loop now asks pre_session with the value.
`declare -x NAME` on a name that already exists writes no value, so the
export mark was the only session write there was and it was stamped
directly, which let an agent export a host-seeded credential the
deployment had refused. It goes through view.mark now.
GNU keeps the valid operands of a declaration and reports the invalid
one, so `declare -x GOOD=1 1BAD=x` exits 1 and still answers
`declare -x GOOD="1"`. The stamp was gated on the aggregate status, so
GOOD came back unexported; it is driven by what actually stored.
Session payloads always carry var_attrs, since its presence is what
tells a session record from a bare process environment.
Adds a meta-test in each language that enumerates every ungated
set_attr/setAttr call site and requires each to name the gated write
that covers it, so the next one fails a test rather than a review.
GoogleConfig demanded client_id + refresh_token and minted its own
tokens, while MsGraphConfig already took access_token as either a
SecretStr or a provider callable. Same OAuth problem, two answers in one
codebase, and the Google half locked out anything that cannot produce a
refresh token: a service account, or a host application that already
owns the OAuth dance. A consumer worked around it by monkeypatching
refresh_access_token, a private function.
GoogleConfig now takes the same access_token field. A supplied token
short-circuits the grant in TokenManager.get_token, and a provider is
read every call rather than cached, since the provider is the caller's
own cache. client_id and refresh_token become optional, with a validator
refusing a config that names no credential at all.
The second half is a bug that predates this and is already reachable on
shipped code: pydantic cannot serialize a function, so redacted_config_dump
raised PydanticSerializationError on any config holding a provider
callable, and took the whole snapshot with it. Verified against a
OneDrive mount, which is exactly the shape the haystack integration
builds. Secret fields are now excluded from the pydantic dump and written
back by the walk, and a callable redacts in both modes rather than being
revealed, which would freeze one expiring token into a snapshot that
outlives it. Redaction is already the contract that makes
requires_resource_override demand a fresh resource at load.
0.0.5 made build_resource a coroutine so GitHubResource could fetch the
repo tree before returning. That fixed a real defect (the two fetches ran
in __init__ over a blocking urlopen and froze the daemon's loop) but paid
for it with the one function every caller who describes a mount as data
comes through: the YAML loader, the daemon's create/load routes, clone,
and every embedder reaching it through mirage.sdk. The haystack
integration, our first outside consumer, broke on it.
Hydrating lazily removes a round trip instead of adding one. Nothing
seeded the index at build time, so the first readdir ran
ensure_live_index and refetched the whole tree, discarding the one the
constructor had just paid for; measured two git/trees calls where one
does. GitHubResource now names the repository and contacts nothing, and
the tree and default branch arrive through ensure_tree and
ensure_default_branch on first use, each behind a lock so concurrent
first reads cost one request. readdir and read already hydrated through
ensure_live_index; only find, du and grep's narrow read accessor.tree
directly and needed wiring.
normalize_resources now refuses a non-resource and names the mount. The
old failure was 'coroutine' object has no attribute 'set_index', raised
two frames away in install_mounts, naming a method the caller never
called and no mount.
TypeScript stays async: its factory type is uniformly
(config) => Promise<Resource> and two of its backends need it. Recorded
as a deliberate divergence rather than mirrored.
A message, reaction, pin or shared file names a channel by id and a
message by ts; the mount names a directory <name>__<C-id> and buckets
the day in UTC. That arithmetic is the backend's, so the hook lives
beside it rather than in the consumer, which would otherwise reimplement
it and get the UTC rule wrong for a fifth of every day with no error to
notice.
core/slack/watch/ constants types payload hook
Resolution is memoized: an event carries the id, the directory carries
the name, so conversations.info (plus users.info for a DM) runs once per
conversation instead of once per message against a ~50/minute tier. A
rename drops the entry, since the subtree moved.
Coarser answers where the notification cannot say more: a listing change
is UNKNOWN on channels/dms/users, and a shared file is UNKNOWN on that
day's files directory, because the rendered name needs metadata the
event does not carry.
Each was one file holding constants, helpers and a class, and disk also
held both halves of the feature: the delta-hook walk and the event hook
shared a file with nothing in common but the word watch.
core/disk/watch/ constants walk hook
core/redis/watch/ constants hook
Same module names in both languages. The node barrels export only the
build hooks the resources import; tests reach the classes directly.
_virtual returned the mount-absolute key both caches are keyed by, but
read like it returned PathSpec.virtual verbatim, so it hid that its job
is re-deriving that key against the manager's own prefix. Renamed to
_cache_key / cacheKey, with the locals following.
Four Args docstrings said the parameter was a resource-relative path.
That was already stale and became wrong when the method started reading
only virtual.
Each open is visible to the next, so `set -C; echo x > a > a` creates
`a` on the first redirect and refuses the second. Probing every target
against one pre-command snapshot passed both and wrote the output. The
opens a refused statement already performed are applied, since bash
leaves the file existing and empty.
`declare -rx X=1` carries both attributes. Readonly answers first, so
the export stamp moved into a helper both branches call.
`export -n ARR=(b)` takes the attribute off an array. The store keeps
whatever the name carried, so an unapplied mark left it exported.
`set -v` echoes comments and blank lines, which the reader consumes
like any other line.
- disk: read watchdog's `src_path`, not `path`. The docstring already
claimed watchdog's field names and `dest_path` already matched, so
every forwarded watchdog event mapped to nothing.
- disk: a move whose source is outside the mount is a CREATE at the
destination. Discarding the event left a file in the mount that no
listing knew about.
- disk: the mount root itself normalizes to `/`, not `/.`, matching
what the TypeScript side already did.
- redis: an external mkdir/rmdir touches the single `<prefix>dir` set,
and the message names the key, never the member. An empty directory
has no `file:` key at all, so those changes were invisible. They now
map to UNKNOWN at the mount root.
Adds the push half of change detection, alongside the existing delta_hook
pull half.
- EventHook / SupportsEvents: a resource can map one service notification
to FileEvents. Mirage owns no transport; the consumer runs the socket or
webhook receiver and passes what arrived, so only the path arithmetic
lives beside the backend.
- invalidate_subtree on CacheManager plus invalidate_prefix on every index
store, wired to FileChangeKind.UNKNOWN in the watcher. A notification
that names only a scope now drops everything below it, which is what
UNKNOWN already promised in its docstring.
- disk and redis event hooks, both languages, with workspace-level tests.
Also fixes a pre-existing bug found on the way. CacheManager._virtual
inferred whether a caller's path was already mount-absolute by comparing
strings, so a mount-relative /day under a /d mount was read as absolute
and every eviction targeted the wrong key. An eviction that hits no key is
silent, which is why it survived.
The fix is to stop inferring: read only path.virtual and rebuild the key
against the manager's own prefix, exactly as Mount.execute_op already
does. That also explains why the roughly fifty from_str_path sites that
fabricate a resource_path are harmless: the dispatcher passes
path.virtual and re-derives, so a fabricated key only matters to a
consumer that reads the pair without going through a mount.
New adversarial suites in both languages cover mount/child pairs whose
names collide by prefix (/d holding day, /d holding d, /data holding
database), which is how the second case was found.
The refused-open case ends its line with ls on a file that correctly does
not exist, and GNU ls exits 2 for a missing operand, so the case expected
0 where both hosts rightly answer 2. Pinned against bash 5.2.37.
Excuse the two cases that cannot run on hf, which keys no directory
marker, so a case that makes a directory and then looks it up finds
nothing there. That puts the coverage gate back on its baseline.
Persist variable attributes instead of promoting every scalar to
exported on reload, and keep an attribute-free payload readable as a
process environment.
Route `readonly NAME` and `local`/`declare` through the gated mark door,
so a pre_session rule can refuse the name and a bare declaration no
longer invents an empty value.
Refuse a noclobber redirect before the command runs, not after, and stop
noexec at execute_node plus each loop driver so it holds one construct
deep without spinning.
Restore the process view for prefix assignments and SessionProfile env,
which narrowing env_snapshot to the exported set had broken.
43 GNU-pinned cases: `assign/export_attr.json`, plus noclobber, noexec,
verbose and braceexpand under `setopt/`, which had errexit, noglob,
nounset, pipefail and xtrace but none of stage 1. The noexec cases are
what caught the subshell bug.
Both runners and `state_store` wrote into `ws.env`, which the read-only
projection refuses, so the battery could not start on either host. A
meta-test now scans for that write shape across both trees: mypy runs on
`mirage/` alone, so `integ/` and `tests/` are exactly where it recurs
unseen.
`export`, `export -n`, `declare -x`, `declare -p` and `set -a` now agree
on one attribute stored on the variable record.
`env` and `export -p` carry exported names only, which is what GNU does.
`$X`, arithmetic and `[[ ]]` still resolve against every variable, so the
process view and the shell view are separate functions in both languages;
TypeScript had collapsed them into one. Adds bash's third variable state,
declared and exported but unset, which `declare -x Z` reports and `env`
omits. Attribute writes with no value go through a gated door so the
policy layer still sees them.
`export`, `readonly`, `local` and `declare` refuse a word that is not a
valid identifier, each in its own voice, instead of exiting 0 and storing
a name that `$1BAD` can never spell back. Quoting decides whether an
empty operand survives to be refused.
Aligns TypeScript to Python where Python was right, including `env -i
FOO=bar`, which built the child environment without the attribute. Python
was wrong about PWD across `fork`, and both were wrong about OLDPWD.
`session.env` is a read-only projection of the variable records, so
`Object.assign(ws.env, ...)` throws instead of landing. dsh and the two
version tests still wrote through it.
Mirrors the Python side of tier 2 stage 1: noclobber with `>|` as the
override, noexec, verbose, the `set -o`/`+o` listing and the remaining
option letters.
Also fixes `set -n` inside a subshell in both languages. `handle_subshell`
runs a second statement loop beside `execute_program`'s, so the check had
to be stated there too. Without it `set -n` worked at top level and did
nothing one paren deep.
Same design, same names, same semantics as the Python commit before it:
Session.vars holds one readonly ShellVar per name, env/arrays/readonlyVars
become projections, and the two local save/restore stacks collapse into
one that shadows the whole record.
The projections are frozen for the same reason Python's are mappingproxy:
an assignment into a plain object lands in a throwaway and vanishes, which
is the failure the store exists to remove. TypeScript catches those at
compile time rather than at runtime, which is how the seedVar(ws, ...)
mis-targeting that Python only surfaced under test was found here before
running anything.
attrLetters is pinned against the same bash 5.2.37 clusters in
variable.test.ts, so the two languages cannot drift on print order.
Full core suite 8266 passed, tsc/eslint/knip/prettier clean, layout parity
at baseline.
A variable's facts were spread across session.env, session.arrays and
session.readonly_vars, with "scalar or array" encoded as which dict the
name lived in and mutual exclusion maintained by hand at every write.
The remaining #608 attributes would each have added another container
(-x -i -l -u -n -t, plus associative values and a third local stack):
thirteen name-keyed containers, any two of which could disagree about
one name. That has already cost bugs here twice -- printf -v wiping an
array, readonly_vars sitting permanently empty.
Session.vars now holds one frozen ShellVar per name (value plus
attributes) and env/arrays/readonly_vars are read-only projections of
it. Frozen because every writer already computed on a copy and handed
the result to the session door; the type now enforces what was
convention, so the door is the only way to change a variable and an
attribute write can finally be gated the way a value write is.
Two things the migration surfaced:
- The projections return mappingproxy, not a plain dict. A dict made
session.env[k] = v write into a throwaway and vanish silently, and
the suite went green on it -- the exact failure the record exists to
remove. Failing loud is what located all 125 write sites.
- bash has a third state, declared-but-unset: readonly ONLY prints
declare -r ONLY and ${ONLY-d} still expands to d, while readonly
EMPTY= prints declare -r EMPTY="" and does not. ShellVar.value is
therefore optional; None is that state, and export FOO and declare -i
n need it too.
Attribute print order is pinned exhaustively against bash 5.2.37 over
all 72 ordered pairs: a/A, then i n r t x, then l/u. -a and -A are
derived from the value's type rather than stored, so they cannot
contradict the value they describe.
Also adds the Stage 1 tests that were missing: a leftover reference
broke set -C outright and the whole suite stayed green.
set -C was unreachable by its own letter: only five letters were mapped to
option names, so -C fell into the ignore-unknown-letter branch while
set -o noclobber worked. Maps all nineteen bash letters, then implements
the three options that silently did nothing:
- -C refuses a truncating open onto an existing target (empty counts),
leaves the file intact, exit 1. >| is a new operator that overrides it
for one redirect; >> is unaffected. Existence is asked of the op
dispatcher, and only when the option is on, so the ordinary redirect
path costs no extra round trip.
- -n reads without executing, checked in the top-level statement loop,
which gives bash's one-way trip for free: a later set +n never runs.
- -v echoes input as the reader consumes it. The unit is a line, not a
statement, so set -v; echo a echoes nothing while set -v\necho a does.
set -o and set +o with no name print the option table in both spellings,
and +B is honoured rather than merely stored so the listing does not
report a state nothing acts on.
All pinned against GNU bash 5.2.37 on debian:stable-slim.
With the barrel down to its public surface, knip can finally see core's
unused exports. 233 findings: 74 are the surviving index.ts lines, whose
consumers live in examples/, docs/ and integ/, outside knip's project
root, and are gated by scripts/check_barrel_surface.py instead.
Of the remaining 159, six turned out to be false as well. knip strips a
barrel line and then cascades into whatever that line alone kept alive,
so CommandContext, ExecuteResultContext, RAMWatchQueue and Watcher are
still reachable through index.ts, and S3BrowserOperation is re-exported
by packages/browser.
The other 153 are gone: 61 dead re-export lines in the sub-barrels, 52
symbols that lost the export keyword but are still used in their own
file, and 40 deleted outright.
knip.json is unchanged. `pnpm exec knip` is a pre-commit gate, and
includeEntryExports would fail it on the 74 barrel lines.
The Files API answers a honored Range with no Content-Range and the SDK
surfaces no status, so taking that header as the proof sliced an already
windowed body a second time. A body longer than the window is the only
proof available and the only case that matters.
Past the end huggingface echoes `bytes 99-2/3` and OpenDAL refuses to
parse it, which reached the caller as a crash rather than an empty read.
#805 shrank core's barrel to its public surface and moved every in-repo
import onto a module path, which collided with this branch in five files.
The barrel line this branch added for `listingError` is dropped: the new
gate refuses a name that only package source imports, and the four node
readdirs that wanted it now name `@struktoai/mirage-core/utils/errors`
like their siblings. `enoent` goes with it in hf and ssh, and
`enotdir`/`readdirError` in gridfs -- each was left over from the private
`listingError` wrapper this branch deleted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hf was left out of #809, which fixed this for s3, gridfs and nextcloud.
readdir answered a missing path with an empty listing, so ls exited 0
where stat and every other backend report ENOENT.
What proves existence differs per store and was probed rather than
assumed. hf is a third rule: the tree API lists children only, with no
self-entry, and the service refuses a directory marker client-side
(create_dir=false), so a bucket directory exists exactly while it holds
a key and any entry at all proves it. The Hub answers a missing subpath
with 200 and [], which the lister reports as an empty result rather than
raising, so the existing NotFound arm never fired.
readdir now tracks whether the listing saw anything and refuses when it
saw nothing, with the mount root exempt and the errno coming from the
shared readdir_error / readdirError helper.
The TypeScript fake rejected with NotFound where the service answers [],
which would have made the new tests vacuous; it now matches. That fake
was also why the existing "raises ENOENT for a missing directory" test
passed without reaching the code under it.
integ: hf and hf-prefix join the seven ls/missing cases and their
exceptions are deleted. The baseline stays at 2297 because it counts
unexcused gaps only, and those pairs were excused.
Three cases only passed before because this bug masked them, all from hf
having no empty directory: rm_interactive_noops and rm_root_and_fs_flags
end with the directory emptied, and xm_mv_no_copy_source_kept lists a
directory that mkdir -p never keyed. hf is dropped from those three, with
a reason recorded for the one whose directory reports it as a gap.
Sending a Range is a request, not an instruction: a server may answer
200 with the whole representation, and box, dropbox, gdrive, onedrive,
sharepoint and databricks all handed that back whole for what the
caller asked to be a window. They now carry the window as one value
and check the answer, the way slack and discord already did.
The node mongo driver validates a window against the file and refuses
every POSIX-shaped answer: a start past the end, an end past the end,
and an omitted end, which it defaults to 0 and then reads as ending
before the start. That last one broke every read-to-EOF on gridfs.
Python is unaffected because its driver seeks through a file object.
Three P1s, each of which was present in both languages while the review
saw only one half.
github: the walk fetched a complete tree, diffed it and threw it away.
find, du and grep's scope counter read accessor.tree directly since the
re-key, so they answered from the tree the mount was built with until an
unrelated read refilled the index, and a pull that reported a CREATE was
followed by a find that could not see the file. A truncated tree is
still refused rather than adopted.
dropbox: path_display carries the server's casing and root_path the
user's, and dropbox paths are case-insensitive, so a configured /team
whose displayed path is /Team matched nothing, kept the root on the
front of every virtual path, and put every event outside the watch
scope. The comparison now folds case; the slice keeps the server's
casing below the root.
disk: an unreadable directory is not an empty one. The typescript walk
caught every readdir error and python's os.walk swallowed them by
default, so EACCES or EIO diffed into a DELETE for every child and a
CREATE for each once access returned. Only ENOENT is swallowed now.
Tests pin all three in both languages.
Rebasing onto #812 broke the build, and the break was one my own gate
could not see. #812 rewrote the redis example to take (accessor, paths,
texts, opts) and added a typecheck script for examples/typescript, so
tsc now says:
redis/redis.ts(18,42): error TS2305: Module
'"@struktoai/mirage-core"' has no exported member 'CommandOpts'.
check_barrel_surface.py only ever asked whether every line in index.ts
has a consumer. The other direction -- a consumer importing a name no
line carries -- it never asked, so the shrunken barrel could go stale
under any consumer change and only tsc would notice. tsc is not enough
on its own either: it reads examples and integ, but a name in a docs
page is prose to it. The gate now owns both halves.
Adding the second rule needed the first one's bookkeeping fixed.
own_exports() counted a runtime package's own names only when they came
from a relative module, so a name the index re-exports from a core
subpath was credited to core's barrel instead. That mattered because
node and browser both name DiscordConfig, LinearConfig, NotionConfig and
SlackConfig from their core subpaths -- every TypeScript consumer gets
them from the runtime package, and the four lines in index.ts were dead.
The rule is now "any module but the bare barrel supplies the name
itself", which drops those four.
Net: 80 names -> 77. Verified the new rule fires rather than trusting a
green run: deleting the CommandOpts line reproduces the failure with the
name reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shortcut folded into readdir_error assumed a path that exists proves
its ancestors are directories. A flat store breaks that: ram and redis
rename without creating the destination's ancestors, so they can hold
/missing/a.txt with /missing absent, and readdir of the orphan itself
came back ENOTDIR where resolution stops at the gap and the answer is
ENOENT. One level below (/missing/a.txt/x) was already right, which is
why the existing orphan test did not catch it.
readdir_error goes back to walking every component, the listed path
included. The shortcut moves to `listing_error` / `listingError` beside
it, whose docstring states the premise it needs -- a store that cannot
hold an orphan -- and which delegates to the walk for everything it does
not settle. The eight backends that qualify (s3, gridfs, nextcloud,
dropbox, databricks, msgraph, ssh, hf) call it, so the fast path is
still one shared implementation rather than one copy each; ram and redis
call readdir_error directly.
Both new behaviors are pinned in both languages: the orphan case against
readdir_error, and the one-probe count against listing_error.
Two OpenDAL behaviours the python binding never exposes, both fatal to the
typescript integ run and neither visible from a python-host run.
Its node binding takes an exact byte range and refuses to return fewer
bytes than asked for ("reader got too little data", expect 100 / actual 3),
where a POSIX read comes back short. Python reads through a file object, so
f.read(n) stops at EOF like any file. Both node backends now retry the read
unbounded and take the window themselves.
Past that, an offset beyond EOF answers 416, and OpenDAL reports the
store's whole response as message text rather than fields: the status is
only readable inside the string, the code is OpenDAL's own, and SabreDAV's
RequestedRangeNotSatisfiable has no spaces for the spaced spelling to
match. Widened isUnsatisfiableRange / is_unsatisfiable_range to that shape
in both languages, so the read comes back empty as POSIX expects.
Two codex findings on #805.
P2: node and browser lost Mem0Config, OneDriveConfig and SharePointConfig
from their public API. Stage 1 deleted that block from both barrels as a
duplicate of `export * from '@struktoai/mirage-core'`, which was true when
core exported 1457 names; stage 4 shrank core to 80 and left the premise
false. The three resources still arrive through the star -- they are among
the 80 -- so only the config types went missing. Both barrels name them
module-by-module now. Measured: node and browser each shed ~1356 further
names through that star, which is the intended core shrink, and every one
is still reachable by module path; these three were the only names their
own barrels had ever listed.
P1: `with { type: 'json' }` in version.ts parses from Node 20.10, and the
tsup build used to inline package.json so any Node could load it. Nothing
declared a floor -- no package had an `engines` field at all -- so this
would have surfaced as a SyntaxError at import instead of an install
warning. All eight published packages now declare `node: ">=20.10.0"` and
the three docs lines saying "Node.js 20" say 20.10. Hard-coding the
version instead would break a deliberate parity: mirage/version.py reads
importlib.metadata for the same reason this reads the shipped
package.json.
index.ts goes from 1589 lines / 1457 names to 102 / 80. What survives is
what the repo's own consumers -- examples, docs, integ, the spec
generator -- actually import; everything else is reached by module path,
as `mirage.resource.s3` is in Python.
gen-specs.ts stops enumerating `Object.keys(Core)` for command groups and
reads them from the modules that declare them, so core no longer needs
the reachability assertion: a directory scan cannot miss a group. node
and browser keep it, being bundled behind one entry. Spec output is
byte-identical.
The gate is scripts/check_barrel_surface.py, not knip's
includeEntryExports. knip's project root is typescript/, so the barrel's
only remaining consumers sit outside its graph -- it calls 74 of the 80
survivors unused -- and core's `./*` exports map makes every module an
entry, turning the flag into a repo-wide dead-export check (233 findings,
159 of them real and left for their own sweep). The script enforces the
two rules that matter instead: no package source imports the barrel, and
every name in it has a consumer. Both were checked to fail when violated.
Also fixes two examples that imported a type that does not exist
(HfDatasetsConfig, HfSpacesConfig; the real name is HfRepoConfig).
Nothing caught it because examples/typescript is never type-checked.
Every cross-package import of `@struktoai/mirage-core` now names the
module it wants, the way `mirage.resource.s3` does in Python. The barrel
only existed because the package had one door; stage 2 gave it a subpath
map, so the door count no longer forces the name list.
Two things the barrel was hiding come out with it:
- The seven bundled CLI specs registered themselves at import time, so
which CLIs a caller could resolve depended on what it happened to
import. The registry now seeds them, and specs.test.ts pins that they
resolve from the registry module alone.
- browser and node reached zod through core's re-export. They keep doing
so, now through `resource/secrets` -- the module whose helpers are
typed against that instance. Declaring their own zod resolved a
second copy (4.4.3 next to core's 4.3.6), which type-checks but takes
134s on one config file.
`core/src/index.ts` is a 1589-line hand-written list because the package
had exactly one door: tsup bundled `src/index.ts` and nothing else was
reachable. This gives it the door Python already has — `./*` maps onto the
built tree, so a consumer can import one module the way Python does
(`from mirage.resource.s3 import S3Resource`) instead of going through a
barrel that has to name it first.
The map is a single wildcard, so a new module needs no build config and no
barrel edit. The `.` entry is unchanged, so nothing has to migrate yet.
tsup cannot do this: `bundle: false` leaves `from "../x.ts"` in the output
rather than rewriting to `.js`, and its dts step fails outright. `tsc`
already rewrites correctly (`rewriteRelativeImportExtensions` is on in
tsconfig.base), so core builds with `tsc -p tsconfig.build.json` plus a
clean step, since tsc overwrites but never deletes. Tests and their
helpers are excluded from the build but still checked by `tsc --noEmit`.
Unbundling surfaced three defects the bundler was hiding, each invisible
while esbuild stood between the source and the runtime:
- Core imports `node:module`, `node:path` and `node:worker_threads` but
never declared `@types/node`; it typechecked only because core's own
test files dragged the types in, so CLAUDE.md's rule that core work in
both runtimes was unenforced. Declaring those three modules locally with
only the members used makes the boundary compiler-enforced — adding
`@types/node` would have let any module here reach for `node:fs` and
still pass.
- `version.ts` imported `../package.json` with no `with { type: 'json' }`.
tsup inlined the JSON; Node refuses it. Every CLI end-to-end test failed
on this.
- One extensionless relative import (`./constants`), legal under
`moduleResolution: Bundler` and unresolvable by Node.
tsup is no longer a core devDependency, which knip confirms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for giving core a subpath `exports` map, but each of these is
a defect on its own: a module exported from two places in one barrel is
two lines to keep in sync, and the second is easy to miss.
Six modules in `core/src/index.ts` were exported from two blocks each
(io/types, generic/cat, generic/head, grep_helper, runtime/table,
cache/index/config) — merged into one block apiece, no name added or
removed. `sed_helper` exported three symbols under both their own names
and a `sed`-prefixed alias; the aliases have zero consumers anywhere in
the monorepo, so they are gone.
`utils/key_prefix.ts` looked like a seventh, but its flat block and
`export * as keyPrefix` are both live — `node/src/core/gridfs/_client.ts`
imports the namespace. Left alone.
The runtime barrels each re-exported names their own
`export * from '@struktoai/mirage-core'` already covers: `MountMode` one
line below the star in browser, and the Mem0/OneDrive/SharePoint block in
both. Nothing shadows those names locally, so the star is the only
source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The spec is regenerated by a CI step of its own, not by a pre-commit hook,
so adding the slot to those two command_io tables left the committed
inventory a commit behind with nothing local to say so.
Two ways a windowed read was quietly answered with the whole file.
A backend that registers its own read op does not go through
makeGenericOps, so neither the native-range dispatch nor the read-and-slice
fallback reaches it. opfs declared readRange on its CommandIO and still
returned every byte through the op path, which is what the integ battery
caught; gcal, gdocs, gsheets and gslides had the same hole in both
languages and no coverage pointing at it. Each now reads the window off
kwargs: opfs forwards it (File is a Blob, so it slices natively), the four
rendered ones slice after building their bytes.
Sending a Range is a request, not an instruction. RFC 9110 lets a server
answer 200 with the whole representation, and slack and discord fetch
attachments from a CDN, so the header alone was never proof. downloadFile
now takes the window rather than a prepared header and trims anything that
did not come back 206 (window_if_unranged / windowIfUnranged). Reported by
codex.
The slack and discord fakes returned full bodies whatever the request, so
they could not have caught either half. Both now serve 206 with
Content-Range, and 416 past the end.
Every agent SDK was resolving to the floor of its own range, so the adapters
were being tested against builds a month older than the pins implied. Two were
outside their caret entirely (@openai/agents 0.13.5 with 0.16.0 published,
pi-coding-agent 0.80.10 with 0.84.2). The examples workspace carries the same
pin set and was bumped with it.
deepagents 0.7.6 added max_count to BackendProtocol.grep/agrep. mypy caught the
incompatible override; TypeScript accepted the shorter signature structurally
and typechecked clean, so both sides are fixed. max_count is a total cap across
files, which is not what grep -m means, so it is applied to the collected
matches rather than pushed onto the command line, and truncated is only set
when matches were actually dropped.
dsh-shell moves to the 0.1.0 line to match dsh-fs. DeepSeek relicensed between
the two (BSD-3-Clause and restricted, to MIT and public), and the published
types are identical. That family tags its 0.1.0 line as next while latest still
points at 0.0.x, so the siblings are pinned in pnpm.overrides.
Clears a low advisory on @ai-sdk/provider-utils that reached examples through
mastra 1.51's @ai-sdk/ui-utils, which 1.59 drops. Both audits are clean.
Note: @opencode-ai/plugin 1.18.18 pulls ini@7, which needs node >= 24.15.0.
CI already resolves above that floor.
Adds an optional read_range slot to the op table. Backends that can fetch a
window do; the generic read op falls back to read-and-slice for the rest, so
no backend has to implement one. Wired in both languages for box, databricks,
dify, discord, disk, dropbox, gdrive, gridfs, hf, nextcloud, onedrive, opfs,
ram, redis, s3, sharepoint, slack and ssh.
Shared helpers live in utils/ranges (range_header for a raw push-down,
slice_window for rendered content, is_unsatisfiable_range to normalise a 416).
Backends that render their bytes take the window right after building them,
which is why dify and ram are on the native list too: a windowed read is
answered the same way everywhere, whatever is behind the mount.
Fixes a crash on hf and nextcloud: OpenDAL's reader seeks rather than sending
a header, so a window past EOF raised from the seek instead of surfacing a
416. Both now read as empty. Caught by the integ battery, not by unit tests.
Removes the github_ci resource, its commands, ops, docs and examples.
Integ: a shared ranges/read.json across 21 targets, plus per-backend cases for
slack, discord and dify, reached through ws.dispatch since no shell command
asks for a window.
The re-key made the index key mount-absolute, but TypeScript's stat kept
prepending the mount prefix when it derived the parent to list, so a
lookup that missed asked for /repo/repo and never populated the entry.
stat then reported ENOENT for a file that exists, and the read family's
implicit-directory probe found the name in the parent listing and
answered EISDIR, so cat on a freshly written file said 'Is a directory'.
Python computed the parent from the key directly and was already right.
`ls <file>/x` reported "No such file or directory" on every API-backed
backend, where GNU (and mirage's own ram/disk/redis/opfs/s3/gridfs)
report "Not a directory". Each of those readdirs raised a flat
`enoent(path)` when its API 404'd, instead of deciding the errno the way
the store-backed backends do.
The API cannot make that call for them: Dropbox answers `path/not_found`
for a path under a file exactly as it does for a name that is simply
absent, Graph 404s both children listings alike, SFTP 3 has one code for
every unresolvable name, and the Databricks Files API answers 404 to
both. So the errno has to come from walking the ancestors, which is what
`readdir_error` / `readdirError` already does for ram, redis, disk, s3,
gridfs and nextcloud. Wired onto that helper: dropbox, databricks,
onedrive, sharepoint (through the shared msgraph `readdir_items`), ssh
and hf. The probes stay on the failure path, at one request per
component.
gdrive and box do not use the helper, and should not: both resolve a
path by walking it through the index cache, which IS the ancestor walk,
already done and already paid for. box already refused a non-folder
entry there; gdrive did not, so listing a file's id returned an empty
child set and the recursion above reported ENOENT. It now makes the same
check box makes, at no extra request.
Two side effects worth naming:
- The one-probe shortcut the three keyed stores had each copied into
their own `_listing_error` moves into `readdir_error` itself, so all
nine backends get it from one place and the three copies are deleted.
It is behavior-preserving (a path that exists implies every ancestor
of it is a directory, so the walk cannot reach any answer but ENOTDIR)
and strictly cheaper: a `readdir` on a plain file is now one probe
rather than one per component, and a missing path is one fewer.
- hf gets the ENOTDIR half only. Unlike s3 it can store no directory
marker at all (see core/hf_buckets/mkdir), so "no keys under this
prefix" covers both a path the repo does not have and a directory
whose last file was removed. A stored key above the operand proves
ENOTDIR outright; ENOENT would be a guess, and guessing it would turn
`mkdir d; touch d/a; rm d/a; ls d` into an error. Dropped rather than
guessed, so an emptied directory still lists as empty.
`integ/unix/ls/missing.json::ls_operand_under_a_file` regains the 13
targets it was missing. Verified on both hosts: every target runs its
whole battery with 0 failures. Adding those targets made the case-target
gate surface the siblings' gaps; box turned out to answer all six
sibling cases and both `operands.json` cases exactly as GNU does and was
simply absent from them, so it joins those too (baseline 2297 -> 2295).
hf's six sibling gaps are recorded in target_exceptions.json with the
reason above rather than closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The case file #809 added drops 31 (case, target) pairs its ls siblings
still exercise, which fails check_case_targets.py --strict on main.
- box passes all 7 cases on both hosts, so it was an omission; add it
- hf/hf-prefix report a missing path as an empty listing, so ls exits 0
instead of ENOENT: the defect #809 fixed for s3/gridfs/nextcloud,
recorded as an exception naming the bug
- ls_operand_under_a_file wants ENOTDIR, which the path-addressed
backends cannot distinguish from ENOENT; excused with a reason
Back to the 2297 baseline with no baseline change.
Add a typecheck script so `pnpm -r typecheck` covers the examples
workspace, and fix the 32 errors it reports.
- 21 FUSE examples used MountBackend without importing it, so they
failed at runtime with ReferenceError
- hf_datasets/hf_spaces imported config types that do not exist; both
resources take HfRepoConfig
- redis provision helpers take (accessor, paths, texts, opts), matching
the Python example
- postgres_fuse/postgres_vfs read a module-level env const inside main,
where the undefined guard does not narrow
- s3_browser/presigner passed Record<string, unknown> as ListObjectsV2 input
- lancedb: registry.get may return undefined, and TextEmbeddingFunction
.sourceField takes no argument
Adds pull change detection to s3, github, dropbox, disk, gridfs, the hf
family, ssh, gdrive, graph and box, so eleven resource families now
answer "what changed under this root since my checkpoint".
Two shared helpers keep a new backend from writing its own loop:
synth_dirs builds the directory rows a prefix store implies, and
ReaddirWalk descends through a backend's own readdir with a fresh
private index per pull.
Adding github to the integ battery surfaced two real bugs. Its index was
keyed repo-relative while the other eleven backends and CacheManager key
mount-absolute, so cache invalidation never reached a github mount at
all, silently, since evicting an absent key succeeds. And invalidate_dir
drops a row rather than expiring it, so github's readers took a dropped
index for real absence. Github now keys mount-absolute like everyone
else, its repo-relative path logic reads the git tree on the accessor
the way TypeScript's always has, and ensure_live_index refetches when
the index holds no listing.
Box fingerprinted on modified_at while the same listing already carried
sha1, so two writes in one second were indistinguishable. It now prefers
sha1.
The integ watch battery covers eleven targets at 238 cases, and the disk
example is gated in both languages against one shared truth file.
* fix(s3,gridfs,nextcloud): a prefix with no keys is ENOENT, not an empty directory
`ls` on a path that does not exist printed nothing and exited 0 on the
keyed stores and on nextcloud, where every other backend exits 2 with
GNU's diagnostic. `stat` has always been right here; `readdir` was not.
It lists what is under a prefix and returns whatever it finds, and for
a prefix holding nothing that is an empty list rather than an error,
which `ls`'s probe_operand reads as an empty directory. So `ls` and
`stat` disagreed about whether the same path existed, and `ls` is the
one command in the family that reads the listing directly.
Seven lines diverged from ram/disk/GNU, all from that one hole: a
missing operand, a missing nested operand, a path below a file, the
same under -l, a second operand beside a good one (which also printed
a phantom header for the missing path), an unmatched glob, and a
directory listed after rm -r.
Each readdir now tracks whether the listing saw anything at all and
refuses when it saw nothing. What proves existence differs by store
and both were probed against a live server rather than assumed:
s3, gridfs an empty directory is a zero-byte marker keyed at the
prefix itself, which the entry loop skips as a
non-child, so the flag is set from the raw page before
that skip
nextcloud PROPFIND lists the collection itself, so an empty
directory yields one entry and only a missing path
yields none. OpenDAL's WebDAV lister reports that as an
empty result rather than raising, which is why the
existing `except NotFound` never fired
The mount root is exempt everywhere: it exists because it is mounted.
The errno comes from the shared readdir_error / readdirError helper
that ram, redis and disk already use, so ENOTDIR-vs-ENOENT stays
decided in one place. One probe runs ahead of the walk: a path that is
itself a file is ENOTDIR whatever the ancestors say, because every
ancestor of a stored key is a prefix by construction. No probe runs on
a successful listing.
is_not_found moves from a private copy in s3/stat.py into s3/_client.py
so readdir can share it, mirroring TypeScript's isNotFoundError, which
has always lived in _client.ts.
Both nextcloud test fakes were lower fidelity than the server and would
have made the new tests vacuous: the python one omitted the collection
self-entry, and the TypeScript one rejected with NotFound where the
server answers []. Both now match what op.list actually returns.
integ/unix/ls/missing.json pins all seven, each oracled against GNU
coreutils on debian:stable-slim first. `ls <file>/x` carries a narrower
target list: databricks, dropbox, onedrive, sharepoint, ssh and gdrive
answer ENOENT there where GNU says ENOTDIR, a separate pre-existing gap
in backends this change does not touch.
test_ls_does_not_show_removed_file was pinning the bug. It wrote
/data/arch/c.txt with no mkdir, so no marker for arch/ ever existed and
removing the last key removes the directory too -- which is what stat
and ram have always reported. It now mkdirs explicitly and keeps a
sibling, so it pins invalidation rather than directory lifetime, and a
companion test pins the emptied-implicit-directory answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(errors): a path component resolves as the directory when both exist
readdir_error walked the ancestors asking is_file before is_dir, which
is not a distinction ram, redis or disk can make -- a path there is one
or the other. A keyed store can hold both an object `a` and a prefix
`a/`, which is a coexistence s3's stat already handles deliberately,
and traversal only ever reaches an intermediate component through the
directory. With an object `a` and a key `a/x.txt`, `ls /a/never` has to
report ENOENT because `never` is absent, not ENOTDIR because `a` is
also an object.
Fixed in the shared helper rather than the three keyed-store callers,
so s3, gridfs and nextcloud all get it. The order is immaterial where
the two are mutually exclusive, so ram, redis and disk are unaffected;
it is also one probe cheaper for a directory ancestor, which now stops
the walk without a second lookup.
Pinned in both languages at the helper (the coexistence case and its
mirror, where nothing coexists and ENOTDIR must survive) and at the
s3 and gridfs readdirs.
Reported by codex on #809.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(rename): directories move whole, on every backend that fakes them (py+ts)
The reported bug was dropbox refusing `mv <srcdir> <dstdir>` when
`<dstdir>/<srcname>` existed as an EMPTY directory: move_v2 rejects any
existing destination, and the conflict handler delete-and-retried only
for a destination file. rename(2) replaces an empty directory, and the
generic mv layer has already proved it empty before calling the backend,
so dropbox now mirrors msgraph's rename_replace -- list the folder one
entry deep, and keep the original error only when a child is there, so a
non-empty destination still reports GNU's "Directory not empty".
Sweeping every backend for the same question turned up three more, and
they are exactly the synthetic-directory keyed stores, where a directory
is not an object but a fact derived from keys and someone has to walk
the subtree by hand. In all four that walk handled files only:
s3 no directory rename at all; raw botocore NoSuchKey reached the
user. Now a prefix walk carries the mkdir marker and the whole
subtree, and a source that is neither object nor prefix is
ENOENT rather than backend text.
gridfs no directory rename at all; ENOENT for a directory ls had just
listed. Now a prefix retag across every revision.
ram moved files, orphaned nested subdirectories. The orphans imply
a phantom source tree, so the old name reappeared in its
parent's listing and then stat as missing -- the shape
check_dest_parents already refuses on the way in.
redis same code shape, same bug.
ram and redis get one _move_subtree that re-keys dirs, files, modified
and attrs uniformly; that also picked up nested files' mtimes, which
were being dropped across a directory rename.
Backends with real directories were already correct and are untouched:
disk (rename(2)), ssh (posix_rename), nextcloud (WebDAV MOVE), box,
gdrive, onedrive, sharepoint and dropbox (server-side folder move),
databricks (recursive copy + rm), opfs (recursive copy).
s3 classifies the source with HeadObject before copying anything rather
than falling back off a failed CopyObject. Stores disagree about a
missing source and a lenient S3-compatible one accepts the copy while
writing nothing -- the repo's own mock does exactly that -- so an
error-driven fallback would have deleted a source whose copy never
landed.
None of this was visible because a case file's `targets` is an opt-in
allowlist: a backend that cannot pass is deleted from the list and the
divergence becomes an omission nobody can see. dropbox was parked that
way, and s3/gridfs directory rename was never tested at all.
check_case_targets.py compares each case file against the modal target
set of its own directory and reports what it drops, on the same ratchet
as check_layout_parity.py: advisory by default, --strict in CI fails
when the count moves off the baseline in either direction, and a stale
excuse fails as loudly as a new gap.
empty_dir.json regains all six dropped targets, and dir_rename.json is
new -- nothing in the battery renamed a directory before. The ram and
redis unit tests already had a test_rename_dir_moves_children, but it
nested only one level, which is how the orphan survived; the new cases
add a subdirectory and fail without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(s3): a refused DeleteObjects key is not a successful move; gate per case
Two codex findings on #808, both real.
DeleteObjects answers 200 with per-key results and reports a key it
refused under "Errors" rather than raising, so reading only the absence
of an exception called the move a success while the whole source tree
survived beside the fresh copy -- a duplicated directory nobody is told
about. Both languages now collect those keys and fail. PermissionError /
EACCES because a refused delete is a lock or a policy in practice, and
because it is an fs error, so mv reports the operand and keeps going
rather than aborting the command line. Both trees are left in place,
which is what GNU mv leaves behind when the unlink half fails after the
copy half landed.
The mock's delete_objects returned None, which is why the gap was
invisible to a unit test; it now answers with Deleted/Errors the way the
real API does, and `undeletable` is how a test asks for the refused half.
The coverage gate keyed on the union of a file's cases, which hides the
exact omission it exists to catch: a sequence-style file states one
scenario across prep/act/verify cases, so dropping a backend from only
the verifying case leaves the union unchanged while the runner silently
skips the assertion. The unit is now one (case, target) pair, and the
baseline moves 836 -> 2297 with it. A "files" excuse still covers every
case in that file, since the usual reason is a property of the file.
Verified against the reported scenario: removing s3 from only
mv_dir_rename_subtree_followed now fails the gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Seventeen pages still documented parquet, ORC, HDF5 and feather
rendering, which neither language ships any more: the ops are absent
from the source, the extras are gone from pyproject, and the two example
files the pages linked to do not exist. Ten carried a whole Data Format
Support section, the rest advertised cat_parquet and friends inline.
Filenames and find examples stay. A .parquet object still sits on these
backends and still reads as raw bytes, so only the rendering claims are
untrue.
Also: home/python.mdx was a lone redirect stub with no nav entry and no
inbound links, four images had no reference anywhere in the repo, and
four pages carried no description.
The sidebar rendered six databases, three cloud stores and every CLI as
the same generic glyph, so the icon carried no information. Thirteen
brand marks now stand in: notion, linear, mongodb (also gridfs, which is
mongodb's), postgres, qdrant, chroma, redis, supabase, databricks, box,
cloudflare for r2, nextcloud and langfuse, applied across the home,
python and typescript trees.
Twelve come from simple-icons at the pinned 16.28.0 with the brand hex
filled in, which is the convention the existing logos already follow;
regenerating minio-logo.svg that way reproduces the committed file byte
for byte. Notion's brand hex is #000000, so it takes the neutral gray
the OpenAI mark already uses rather than vanishing in dark mode. Chroma
and langfuse are not in simple-icons and come from each project's own
mark instead.
Oracle, LanceDB and QingStor keep their generic icons: oracle was
removed from simple-icons upstream and the other two publish no usable
svg.
github_ci is gone from the docs, with its nav entries, its resource
matrix row and the resource index bullet.
PR-0e from the restructure audit (bugs 8+9+G5). The new mv-onto-empty-dir
integ pin caught the rename write-flavor staleness in disk, ssh, onedrive,
and sharepoint; every backend rename now takes the unlink flavor on dst.
Dropbox folder-conflict mv and s3/gridfs directory rename are pre-existing
divergences excluded from the pin and filed as follow-ups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gws was left as the only quoted description, which is the shape that
invites the next unquoted colon. Quoting the rest makes the frontmatter
uniform and immune to it. No parsed value changes.
This also re-targets the CLI pages for the docs build. The icons from
#801 never went live: that deploy aborted on the gws parse error and
dropped all sixteen files with it, and #803 only changed the two gws
pages, so the incremental build republished those alone.
The description held an unquoted ": ", which YAML reads as a nested
mapping key, so Mintlify failed the whole deploy on both gws pages.
Nothing checked frontmatter before CI, so a one-character mistake only
surfaced as a failed deploy. check_docs_frontmatter.py parses every page
the way the docs build does and runs beside the other repo-root checks.
Both from the codex review of #800, both verified against Python first.
P2 — a null secret was being masked. `redactValueWithSchema` returns
null *before* it asks whether a field is secret, mirroring Python's
`if value is None: continue` in `_walk_config_dump`. The hand-written
qdrant and lancedb redactors masked unconditionally, and both keys are
`string | null`, so a keyless local Qdrant or an on-disk LanceDB got a
`<REDACTED>` marker for a credential it never had — making
`Workspace.load` demand a fresh config for a self-contained snapshot.
`RedactedConfig<T, K>` now keeps null in the redacted twin when the
source allows it, so the type says this too.
P1 — chroma has no credential, so its state carried no marker, so
`buildMountArgs` handed back a `RAMResource` and a /chroma mount loaded
as an empty local directory. The cause is broader than chroma:
TypeScript rebuilds *nothing* from state, because core cannot import
`buildResource` (it lives in node/browser), while Python reconstructs the
class from `resource_state["type"]` via its registry
(`_resource_class_for`). Until core gets a resource factory,
`resourceStateRequiresOverride` also honors an explicit
`needs_override`, and the six config-backed resources set it — so the
mount refuses to load rather than coming back empty. Python already
writes that field on four resources and reads it nowhere; TypeScript now
reads it, which also protects Python-written snapshots loaded in TS.
Tests: keyless lancedb/qdrant cases on both sides (Python passes as-is,
which is the proof the null rule was already its behavior), and the
registry sweep now asserts every config-backed backend demands a
resource at load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(readme): fix the hero snippet and resync every mirror
The hero snippet called ws.command(...), which exists in neither
language: registration is the standalone command() plus mount.register.
It is replaced with a Python example that mounts ram, redis and slack
side by side, captures python with monty, and installs a CLI, all of it
run against the published 0.0.5 packages first.
Two more corrections. The filetype sentence promised parsed PDF pages,
which the filetype removal took away, so it now says a format renders
however you register it. DeepSeek Harness joins the coding agents row.
The eleven mirrors are regenerated from the root rather than patched,
which also closes drift they had accumulated: a stale backend list, the
old CLI + daemon integrations line, a missing Grok Build entry and a
Codex link pointing at the wrong docs path.
* docs(cli): give each CLI page its own icon
Every CLI page shared icon: terminal, so the sidebar was nine identical
rows. Each now takes the icon its service already uses elsewhere in the
docs: slack, discord, github for gh, google for gws, envelope for
himalaya, book for ntn and chart-gantt for linear (matching the notion
and linear setup pages, since Font Awesome carries no brand mark for
either), and git-alt for git. gws and himalaya also get their names
spelled GWS and Himalaya; the rest stay lowercase because that is the
head word you type.
* examples(filetype): register through the public mount accessor
The example reached into ws._registry.mount_for, but ws.mount is public
and returns the same MountEntry. Output is unchanged, so the CI truth
file still matches.
The interface I added to `resource/base.ts` was a byte-identical copy of
a private one that already sat in `workspace/snapshot/types.ts` feeding
`ResourceState`. Snapshot types now import the exported one, so the
Resource contract and the snapshot format name one shape.
The doc comment now says what the two keys are for rather than
paraphrasing: `type` is the registry name Python's `_resource_class_for`
looks up before falling back to the mount's `resource_class` import path,
and `config` is what `resourceStateRequiresOverride` scans. It also
records why the literal twin of Python's `dict[str, Any]` does not work
here — a TS interface has no implicit index signature, so
`Record<string, unknown>` would reject every named `XResourceState`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`snapshot/state.ts` cast every mount's resource into `{ getState }` and
called it unconditionally, so a resource without one was a runtime crash
at save time rather than a compile error. postgres and mongodb — both
registry-mountable, in node and browser — had none, and snapshotting such
a mount died with `resource.getState is not a function`.
Structure follows Python's, which already had all of this:
- `BaseResource` gains `getState()` returning `{type: this.kind}` and a
no-op `loadState()`, the twins of `BaseResource.get_state` /
`load_state`. `kind` moves onto the base as abstract so the default can
spell itself, mirroring Python's `name`.
- `Resource` declares both non-optionally, so the two casts in
`snapshot/state.ts` go away.
- The eight config-backed resources that had no state — chroma, dify,
qdrant (core), postgres, mongodb (node + browser), lancedb — now carry
their config, redacted. Inheriting the bare default would be worse than
crashing: with no `<REDACTED>` marker,`resourceStateRequiresOverride`
returns false and `buildMountArgs` substitutes an empty RAMResource,
turning a live database mount into an empty directory on load.
- Redacted fields were taken from Python's own `secret_field_names`, not
guessed: qdrant/lancedb `apiKey`, postgres `dsn`, mongodb `uri`, dify
`apiKey` (Python masks it in `get_state` rather than on the model),
chroma none.
- `HfResource` re-narrows `getState` to abstract so the bare default
cannot reach a Hub resource. Python has no shared Hub base; its four
resources each spell `get_state`.
- Test fakes that stood up a `Resource` by hand now extend `BaseResource`
like real resources do, so they inherit the default instead of carrying
a stub. The three that cannot (two object literals, and the bare class
that exists to have no `storageId`) spell the pair.
- mem0/onedrive/sharepoint swap `Record<string, unknown>` for named state
types, joining the ~35 resources that already had them.
Tests: a registry-derived sweep over the six database backends asserts
getState/loadState exist and the credential is masked, plus an end-to-end
`toStateDict` over a postgres mount. Python gains the twin sweep in
`tests/resource/test_state_round_trip.py` — it passes as-is, which is the
point: this closes a TypeScript-only gap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(discord): parity batch — dirname-parsed scope, tombstoned attachments, date-less grep paths, smart head revival (py+ts)
Phase 0 PR-0d from the restructure plan (bugs 5+6+7 plus one found
during the scope investigation).
1. Scope detection contract (bug 6). Python's detect_scope was async
and resolved guild/channel ids through the index cache; the TypeScript
twin parses them out of the `name__id` dirnames the tree itself mints.
The index store is an exact-key lookup, so the async machinery could
never resolve anything the dirname doesn't already carry — but it
COULD miss (cold cache), and then grep/rg raised "cannot resolve guild
ID" internally and fell back to a per-file scan with a spurious
stderr warning, on paths whose ids were sitting right in the operand.
Python now mirrors slack-py/discord-ts: sync, dirname-parsed, with the
same field set (level, use_native, names, container). Consequences
picked up on purpose:
- members-container paths no longer push down to guild message search
(use_native false, matching TS) — message search cannot answer a
grep over member JSONs.
- the "root-level search not yet supported" special case is gone; a
root operand falls through to the generic scan like TS and errors
GNU-shaped.
- bare names (no `__id`) now skip the push-down silently instead of
depending on cache temperature.
2. coalesce_scopes ported to TypeScript. Python (like slack) coalesces
concrete same-channel chat.jsonl operands into one channel-wide native
search under -w; TS discord never had it, so the same command produced
formatted search lines on python and raw JSONL scan lines on TS.
TS gains coalesceScopes with python's exact semantics.
3. Tombstoned attachments filtered (bug 5). Discord listed any
attachment with an id; slack requires id + download URL + size because
tombstoned/access-restricted payloads carry an id but nothing to read
— they showed as phantom files that ENOENT on read, with unknown
sizes. Both discord twins now carry slack's three-part guard, and the
integ fake gained a `tombstoned` seed knob to prove it end to end.
4. Date-less grep result paths (bug 7). A search hit without a
timestamp rendered `.../channel//chat.jsonl`. Both twins now derive
the day from the message snowflake (the same day readdir buckets it
under); a hit with neither timestamp nor parseable id points at the
channel dir, slack-style.
5. Smart head was dead code (found during 6). head.py gated its
native first-N-messages fetch on scope.level == "file" — a level no
detect_scope in either language has ever produced — so every head ran
the full-day fetch. Revived on level "messages", plus the end-of-day
bound the branch always needed: with `after`, a short day spills into
the next one until `limit` is met, so unbounded revival would print
lines chat.jsonl does not contain. Output is byte-identical to the
generic path by construction (single renderer). TS has no head
override; its scan produces the same bytes.
Also removed the vestigial 'file' member from the TS DiscordLevel
union.
Tests
- py unit: test_scope.py rewritten against the sync API (mirrors
scope.test.ts case-for-case), test_grep_pushdown.py reworked to
`name__id` paths with new cold-cache + bare-name pins, test_head.py
new (smart gate, day-boundary spill, generic fallbacks), readdir +
formatters guards. Full suite green.
- ts unit: scope.test.ts +4 (coalesce), readdir.test.ts +1
(tombstone), search.test.ts +2 (snowflake fallback, date-less),
grep.test.ts single-file scan repointed + coalesce-native case.
Core suite 8205 passed.
- integ: fixture gains a tombstoned attachment and a second releases
day; new scope.json cases pin channel-native grep -w with empty
stderr, coalesced single-file grep -w, and head day boundaries.
Post-fix: discord + cli-discord = 62/62 on BOTH hosts. Pre-fix
(sources stashed, pins kept): TS fails 3 (ghost listed in ls/find,
raw-line grep), python fails 2 (ghost in ls/find) — python's grep
pins pass pre-fix only because the battery's shared workspace
pre-warms the index; the cold-cache symptom is pinned at unit level.
- pnpm -r typecheck clean, pre-commit clean (mypy: no issues in 1819
files).
Known family-wide follow-ups deliberately not taken here: native
search push-down ignores -r on directory operands (slack too), and a
coalesced/date-level push-down can return hits outside the operand
date set (slack too). Both predate this change and need one decision
across the chat family.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(discord): codex round — coalesce only message files, smart-head guards (py+ts)
Review round on #797, all four findings taken:
- P1: coalescing keyed on `not use_native` swallowed file_blob operands
— grep/rg -w on an attachment became a channel-wide MESSAGE search
that says nothing about the requested bytes (pre-existing python
behavior the port had copied into TS). coalesce_scopes/coalesceScopes
now require every operand to be a chat.jsonl messages scope, and the
call sites gate on level == "messages".
- P2: smart head forwarded any count as one Discord page; the endpoint
caps limit at 100, and zero/negative (all-but-last-N) forms cannot be
a single page. Counts outside 1..100 keep the generic path.
- P2: an empty day rendered "\n" instead of b"" — the branch now reuses
history_jsonl_bytes, the same renderer read() uses, so empty is empty
and the bytes match the generic path by construction.
- P2: head -v bypassed the ==> path <== header; verbose keeps the
generic path.
Pins: py unit +5 (file-blob scan, -v, >100/negative counts, empty-day
b""), ts unit +3 (coalesce blob/mixed null, blob-scan no-search),
integ +3 (attachment grep -w scans bytes, head -v header, empty-day
head). Pre-fix: TS fails the attachment pin; python additionally fails
the -v and empty-day pins (got the predicted lone "\n"). Post-fix:
discord + cli-discord = 65/65 on both hosts; typecheck + pre-commit
clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Release Python 0.0.5a1
* Release TypeScript 0.0.5-alpha.1
* Release TypeScript dsh 0.0.1-alpha.1
* test(version): let the --version assertions accept a prerelease
The five TS assertions pinned the output to \d+\.\d+\.\d+, which cannot
express a prerelease, so every one of them failed the moment the version
became 0.0.5-alpha.1. Python asserts the same output with startswith and
was unaffected. The shape is still asserted, now with an optional
prerelease suffix.
* Release 0.0.5 (mirage-dsh 0.0.5 line at 0.0.1)
* feat(ts): install a CLI from a file reference in yaml
`clis: {tally: {cli: ./tally.mjs:TALLY}}` now resolves in TypeScript, so
a deployment can ship its own program tree without a host program
calling registerCli. Mirrors the `":" in name` branch of Python's
`cli_spec_for`, one layer up: `cliSpecFor` lives in core, which has no
filesystem and is synchronous, so the ref resolves in the config layer.
The referenced file is imported by Node, so .mjs, .js and .ts all work;
a .ts spec rides Node's strip-only type stripping and is refused with a
hint when it uses a construct that needs codegen.
* fix(ts): node's specifier rule for cli refs, and a real spec check
Both from review.
A bare specifier carries slashes in Node (`@scope/pkg`, `pkg/subpath`),
so copying Python's `"/" in source` test read every scoped or subpath
package as a file, rebased it under the config directory and reported it
missing. A path is now what Node calls a path: relative or absolute,
plus a bare filename carrying a module suffix.
A class name is not proof a tree is usable. Dispatch reads subcommands,
aliases and options at every level, so a value that only answers to the
name crashed on the first line an agent typed instead of failing the
create. The shape is checked recursively instead.
* refactor(runtime): replace confined with a three-value reach marker
A runtime now declares reach: vfs | process | remote, stating whether
the workspace dispatch gate is its only door or the code can act
around it (host process doors, another machine). The default is
process, the no-promise claim, so a custom runtime must narrow its
reach explicitly. The dsh sandbox claim reads the aggregate: every
runtime at vfs means workspace-write, anything wider means no claim.
Mirrored in Python (RuntimeReach in runtime/types.py) with reach
declared on monty, quickjs, wasi, vfs, local, and RemoteSandbox.
* fix(pyodide): seal the js module so the guest has no host door
Pyodide's default exposes the host globalThis as the `js` module,
which under Node handed guest code js.process (host env, confirmed
reading HOME) and js.fetch (network) — doors around the workspace
bridge that made the runtime's reach='vfs' claim false and
contradicted its own 'no network' docstring. Pass a null-prototype
jsglobals to loadPyodide: `import js` still resolves but the host
globals are unreachable through it, while pyodide's internals (which
capture their globals at load time) and the FS bridge are unaffected.
Pinned by jsglobals.test.ts.
* feat(dsh): stamp ShellSandboxInfo on run results and process handles
When the world is fully workspace-bound (vfsOnly), run() and start()
now fill dsh's optional sandbox field: mode workspace-write,
enforcement 'full' (the VFS gate is unbypassable, so unlike an OS
sandbox on an old kernel there is no promised effect it fails to
govern), denied false (mirage has no out-of-band denial channel; a
refused write fails in-band as an ordinary command error), and
runnerFailed false (the executor is the runner). The process handle
stamps it on settle. Omitted when any runtime reaches beyond the
workspace.
* feat(dsh): stream background command output through a JobConsole
Adds a public ExecuteOptions.sink: pass a JobConsole and the line's
output streams into it as each statement finishes, instead of being
returned whole (the result then carries only the exit code). This
reuses the executor's existing internal sink mechanism, so a compound
line flushes per statement and stdout/stderr keep their channels.
MirageShellProcess is rewritten over that seam: start() runs the
command with a console as its sink and a follow loop drains it into
the read buffer, so readOutput() delivers output incrementally and
stdout/stderr interleave in order (stderr opened by a marker) rather
than stderr being concatenated at the end. The unread backlog is
bounded to stdoutMaxBytes (tail kept, lossy flagged) so a reader that
never drains cannot grow it without limit.
* feat(dsh): spill the full stream to a workspace file on overrun
Adds an opt-in spillDir config. When a background command's streamed
output overruns its delta budget, the full stdout and stderr are
written to files under that workspace directory and readOutput()
points at them (stdoutSpillPath/stderrSpillPath), so a reader can
recover what the delta dropped by reading the spill through the same
VFS. Memory stays bounded: each channel buffers only until the first
overrun, then flushes to its file and appends from there. A write
failure (no writable mount at the path) disables the sink and leaves
the paths undefined, the honest 'no safe path' answer. Default unset,
so nothing spills unless a deployment asks for it.
* docs(dsh): custom backends, background streaming, and the reach model
Corrects the sandbox-claim wording to the reach model (workspace-write
when every runtime reaches only the vfs, dropped when one reaches the
host), and adds a Custom backends section (registerResourceFactory,
host-side before the workspace builds) and a Background commands
section (per-statement streaming, bounded backlog, spillDir).
* test(dsh,core): satisfy lint on the streaming tests
Narrow spill paths with an explicit guard instead of a non-null
assertion (forbidden in the dsh package), and drop the now-unnecessary
ExecuteResult casts the sink overload already implies.
* style: prettier formatting on the streaming changes
* fix(core): drain a buffered line into the sink
A sink only saw output the command-tree walk emitted, so a whole-line
runtime, the syntax gate, a policy denial and a failed line all answered
with bytes in hand that a streaming caller never read. executeLine now
moves any buffered result into the console on every path, in one place
rather than five, and the result stays empty as it already did when the
line streamed.
* fix(dsh): bound the console store, make the spill dir idempotent
Capping the delta did not bound memory: reading a chunk advances a
cursor but frees nothing, so an uncapped store held every chunk of a
noisy background command for the life of the process. The store now
carries a retention budget, and the drain reports a trimmed chunk as
lossy and stops the spill, since a file missing the middle of a stream
is worse than no file.
The spill directory is created through ensureDirPath, which walks the
ancestors and accepts a refusal for a directory that now exists, so two
commands overrunning at once do not cost the loser its spill.
* feat(shell): redis console store with cross-language streaming integ
* feat(shell): console config block, shared console battery, redis store package
* refactor(shell): job_table package, and codex fixes on the redis console
Split job_table into types/constants/table in both languages, mirroring
the console package.
Redis console review fixes: the minted key prefix is public (JobConsole
.store plus RedisConsoleStore.key_prefix) so an external reader can be
handed a console's address; keys expire ttl_seconds after the last
append (default one day) instead of accumulating; the ending chunk is
terminal in the store itself, so an emit racing a kill past the local
guard is dropped server-side; and the TS loader validates the console
block's value types the way Pydantic already did.
* refactor(ts): four shared factories mirroring the Python originals (#609 item 24 / T2-9)
TypeScript had grown four copy-drift sites where Python already has one
shared module. Each is extracted onto the Python shape, so the two trees
line up module-for-module.
builtins/shared.ts (twin of workspace/executor/builtins/shared.py). Three
independent errorResult()s with three different signatures — capacity.ts
hardcoded the 'df' command, links.ts hardcoded exit 1, only metadata.ts
took both — so a GNU exit-code fix could not be shared. They collapse onto
result/ok/fail/finish, plus operandText/absPath/splitFlags/splitValueFlags/
expandOperands. Drops links.ts's re-implemented abs+splitFlags and
capacity.ts's import of splitValueFlags from metadata.ts (df reaching into
chmod/touch). `Result` moves here from scope.ts, where Python keeps it in
shared.py; the other 12 builtins change only their import line.
The chmod/chown/chgrp/touch handlers each tracked an exitCode variable
that was only ever set to 1 next to an errors.push, so finish() derives it
from `errors` exactly as Python does and the variable is gone.
workspace/executor/interpreter.ts. python/handle.ts and js/handle.ts were
the same 130 lines apart from the label, the unavailable-error class, the
payload flag, Python's -m guard and -x, and js's .mjs module flag.
makeInterpreterHandler takes those as a spec; both handle.ts keep their
path, their exported name and their signature, so callers and
runtime.test.ts's dynamic import are untouched. The .mjs check moves into
the js wrapper, which is where Python's js.py keeps it. Kept in
workspace/executor rather than beside commands/builtin/general/
interpreter.ts because no commands/ module imports workspace/ today and
this should not be the first.
makeRm. generic/rm_command.ts already existed and gcal already used it;
the gdocs/gsheets/gslides bodies were verified character-identical to it
modulo indentation, so the three 79-line files become 20-line bindings.
core/google/tree_ops.ts (twin of core/google/tree_ops.py). makeStat and
makeUnlink over a backend's readdir; the six modules stay as real files
exporting `stat`/`unlink`, so the barrel and the provision/io importers do
not move. The three local eisdir() copies give way to the shared one in
utils/errors.ts: isFsError keys on `code`, which is unchanged, and its
message (the bare virtual path) is what Python's IsADirectoryError(raw)
already produced.
No behavior change intended. -995/+147 across the 26 edited files.
* fix(js): honor the explicit `-` stdin operand, like python3 and node
`js - a b` read a file literally named `-` and failed with
"js: /-: No such file", while `python3 - a b` on the same tree read the
program from stdin. Python has no such split: resolve_source's
STDIN_OPERAND branch (interpreter.py) is unconditional, so both
interpreters honor the operand, and js.py inherits it by taking the
default Argv0Rules. Real `node -` agrees — it reads the program from
stdin and keeps the following words as argv.
Verified against Python by calling resolve_source directly for the js
call shape: `js - a b` yields mode=stdin, args=['a', 'b'], no script
path. Both quickjs hosts then build scriptArgs the same way (an absent
prog leaves the args alone), so the operand lands identically.
Found while extracting the shared interpreter handler in the previous
commit; kept separate because it is a behavior change, not a move.
The unit test fails without this fix (exit 1, not 0). The integ case
pins py=ts.
* test(ts): mirror the Python suites for the two new shared modules
Python has tests/workspace/executor/builtins/test_shared.py and
tests/core/google/test_tree_ops.py; TypeScript had neither twin, and both
modules are now shared chokepoints rather than per-command code.
shared.test.ts ports test_shared.py case for case: the result triple
(ok/fail/finish, including finish keeping a carried IOResult's writes and
deriving the exit code from `errors`), operandText/absPath, the three
splitFlags cases, the three splitValueFlags cases, and the expandOperands
glob expansion over a real RAM mount.
tree_ops.test.ts covers makeStat over a stub readdir: the synthetic
owned/shared roots answer as directories without a readdir round trip, an
indexed document is served straight from the index, a miss warms the
parent and then raises ENOENT, and a call with no index raises ENOENT
without reaching the backend.
It deliberately does not port test_tree_ops.py's one case — that a readdir
raising a non-fs error propagates out of stat. TypeScript swallows it in a
bare catch and reports ENOENT instead, which is a real pre-existing
divergence in the T1-G "honest errors" family; pinning current TS behavior
here would entrench it. Filed separately.
* refactor(ts): keep the interpreter opt interfaces module-private
Nothing outside interpreter.ts imports them — both handle.ts wrappers pass
object literals — so knip reported them as unused exports.
* fix(google): only an absent parent may collapse into ENOENT (py+ts)
Every Drive-family stat/read/unlink warms a cold index by listing the
parent, then retries. Python narrows that call to FileNotFoundError, so
an auth failure or a dropped connection reaches the caller; TypeScript
swallowed it in a bare `catch {}` and reported "No such file" instead.
Five TS sites did this (google/tree_ops stat+unlink, gdrive stat, gdrive
read, gdocs/gsheets/gslides read) — seven catch blocks in all.
tree_ops.unlink was wrong in the other direction: Python called readdir
completely unguarded, so removing a path under a missing directory
reported the parent rather than the operand. GNU names the operand:
$ rm nodir/file.txt
rm: cannot remove 'nodir/file.txt': No such file or directory
(pinned against coreutils in debian:stable-slim). Both sides now use the
shape Python's make_stat already had: swallow a not-found parent, let
everything else through. TypeScript spells it with the existing
isEnoent(), the twin of `except FileNotFoundError`.
gdrive/readdir warms a parent unguarded in both languages — already
aligned, left alone.
Tests fail without the fix. Two of them mirror Python suites that had
pinned this since before the TS port (test_stat_propagates_parent_
refresh_failure, test_read_propagates_parent_refresh_failure); the
tree_ops pair is new on both sides.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(cache): one entry_or_warm for every cold-index lookup (py+ts)
Twelve call sites across eight backends had grown their own copy of
"ask the index, list the parent if cold, retry, else ENOENT", each with
its own idea of what a failed listing means. They now share one function
in cache/index/warm, so exactly one catch in the tree decides that.
The previous commit narrowed seven of those catches by hand; this makes
the narrowing structural instead, and sweeps the five that commit did
not reach:
box/read (x2) bare catch, py narrowed -> divergence
gmail/read bare catch, py narrowed -> divergence
dropbox/read (x2) bare catch, py caught (FileNotFoundError,
DropboxApiError) -- but readdir already turns the
API's 409 for a missing path into ENOENT, so that
second arm only ever swallowed live API failures.
Python had the same bug here, from the other side.
gdrive/readdir warms a parent unguarded in BOTH languages; already
aligned, deliberately left alone.
Also folds eight private eisdir() copies into the shared helper. The
copies built `EISDIR: ${p}` and never stamped virtualPath, and
execute.ts renders errorVirtualPath(err) straight into stderr, so a
directory read printed
cat: EISDIR: /docs/owned: Is a directory
where Python printed the GNU form. Two TS tests asserted on that
message with /EISDIR/ and so had been holding the bug in place; they now
assert the stamped code and virtualPath, which is what Python's
IsADirectoryError(virtual) carries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): pin the Drive-family cold-lookup error paths on both hosts
Five cases over the gapps and gdrive targets, all curated by running the
battery and reading the real output rather than guessing the strings:
gapp_cat_dir_is_gnu cat /docs/owned -> Is a directory
gapp_cat_missing_parent cat .../nope/x.gdoc.json -> names the operand
gapp_ls_missing_parent ls .../nope -> ENOENT, exit 2
gapp_rm_missing_parent rm .../nope/x.gdoc.json -> names the operand
gdrive_cat_dir_is_gnu cat /data/sub -> Is a directory
clear_cache forces the cold index, which is the branch entry_or_warm
now owns. Both wordings match GNU pinned in debian:stable-slim
(`cat: <path>: Is a directory`, exit 1).
Two things these cases do NOT prove, so nobody reads more into them:
- rm builds its own diagnostic from p.rawPath, so it reports the
operand whatever the backend error carries. The unlink
operand-naming fix is pinned by test_tree_ops.py, not here.
- `cat` on a directory is answered by the read generic off stat, so
the eisdir consolidation is pinned by the gdrive/dropbox unit tests
(which assert virtualPath, the field errorVirtualPath reads), not by
these.
They sit at seq 930030+ deliberately: at 520100 their clear_cache wiped
the index that gdocs_prov_cat (930023) expects warm, and that case
started failing on hits=1 vs hits=0.
Verified: gapps + gdrive, 2193 passed / 0 failed, on the typescript-node
host and again on the python host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* style: formatters over the warm-helper sweep, + the py eisdir message pin
isort regroups the hand-inserted `partial` / `entry_or_warm` imports and
prettier rewraps one dropbox thunk; no behavior change.
Also completes the eisdir work on the Python side: the gdrive read test
asserted only the exception type, so it would not have caught the "EISDIR: "
prefix the deleted TS copies were adding. It now asserts the message is the
bare operand, which is the string the TS twin pins through virtualPath.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(gate): lower the layout-parity baseline to 301
The ratchet fails on a drop as well as a rise, so a closed divergence has
to be locked in. Net -1 from three real movements:
-1 core/google: python-only ['tree_ops'] tree_ops.ts now exists
-1 workspace/executor/builtins: python-only ['shared'] shared.ts now exists
+1 workspace/executor: typescript-only [... 'interpreter']
The +1 is the deliberate layering call from this PR: the shared interpreter
handler lives under workspace/ rather than beside Python's twin in
commands/builtin/general/, because no commands/ module imports workspace/
today and this should not be the first. Recording it here so the net -1 does
not hide that one improvement paid for one new divergence.
cache/index/warm does not appear: it landed on both sides, so it is neutral.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(spec): js/node end option parsing at the first operand, like python
`node - -e x` ran `x` instead of the piped program, and `node s.js -m`
flipped the interpreter into module mode instead of handing s.js its own
-m. Both real interpreters disagree, pinned locally:
$ echo 'console.log(process.argv.slice(1))' | node - -e 'nope'
[ '-', '-e', 'nope' ]
$ echo 'import sys; print(sys.argv)' | python3 - -c 'nope'
['-', '-c', 'nope']
(node 22.8.0, CPython 3.13)
`Operand.remainder` (argparse nargs=REMAINDER) already exists and both
parsers already honor it in one line — the first operand ends option
parsing, so every word after it is an operand verbatim. `python`/`python3`
have carried it all along; js/node simply never got it, in EITHER
language, so this is a symmetric py+ts spec fix rather than a TS-only one.
Verified by calling parse_command directly: js/node/python/python3 now
agree on `- -e PROG` and on `s.js -m --module`, while a flag BEFORE the
first operand is still the interpreter's (`js -m -e CODE a`).
One consequence beyond the reported case: remainder also turns off
lenient_dash_operands, so `js -Z foo` now reports an invalid option
instead of silently treating `-Z` as the script path (which produced
"js: /-Z: No such file"). Real node agrees — `node -Z foo` prints
"node: bad option: -Z". Nothing in integ or the suites relied on the old
leniency.
Reported by codex on #792. Its premise was wrong in one respect — it said
to mark the rest operand "as the Python specs do", but Python's js/node
spec is Operand(type="str") with no remainder, so Python had the same bug.
Fixing TS alone would have created a divergence.
Tests: 4 parametrized parser cases per language over the real js/node/
python/python3 specs, plus a quickjs end-to-end case. All 4 py and all 5
TS fail without the spec change (python/python3 pass either way, which is
what makes them the control). integ/runtime/quickjs.json `stdin_operand`
gains the -e and -m steps.
Verified: pytest 14159 passed / 0 failed; pnpm -r typecheck clean;
core suite 8129 passed (the one python_repl timeout is the known
Pyodide-boot-under-load flake, 10/10 alone); runtime/run.ts quickjs
11 passed / 0 failed on the typescript host; spec drift, spec parity
(93 match), layout parity (301 = baseline) and width table all clean.
The python host's quickjs suite needs the MIRAGE_QUICKJS_HOME WASI blob
that CI downloads, so that leg is covered by test_integ.yml rather than
locally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(quickjs): end qjs's own option parsing before the program's argv
The python host's quickjs runtime appends the program's arguments
straight onto the qjs command line:
argv += ["-e", args.code, *named, *args.args]
Unlike CPython's -c, qjs's -e does NOT end option parsing, so any
program argument that spells a qjs switch was read as one. Pinned
against the build CI uses (quickjs-ng v0.15.1 qjs.c): the option loop
runs while `*argv[optind] == '-'`, breaks only on a bare `--`, and
scriptArgs start at `argv + optind`.
That is what made integ-runtime red on the previous commit, and only on
the python host:
node - -e 'console.log("flag")' -> flag (second -e wins)
node - -m -> (empty) (module mode flipped)
Both are now `--`-separated, so they land in scriptArgs where the parser
already put them. qjs consumes the `--` itself, and the -e branch still
wins over any filename, so a named program keeps scriptArgs[0].
Surfaced by the remainder fix in the previous commit rather than caused
by it: before that, mirage's own parser swallowed those words, so they
never reached the runtime. Any program argument spelling a qjs switch
had this hole -- `node script.js -m` included.
TypeScript needs no twin. Its quickjs runtime is in-process and assigns
scriptArgs as a global (`quickjs.ts:387`), never building a qjs command
line, which is why the TS host passed the same integ steps. Checked the
other python runtimes too: wasi.py and local.py both go through CPython
`-c`, which DOES end option parsing (pinned: `python3 -c prog -e FLAG -m`
puts `-e FLAG -m` in sys.argv), and monty passes argv as a data global.
quickjs was the only one.
Verified with the real engine, after downloading the pinned
qjs-wasi.wasm: `runtime/run.py quickjs` on the python host is 11 passed
/ 0 failed with this change and reproduces CI's exact two-line failure
without it; the whole python runtime battery is 99 passed / 0 failed
(bridge/docker/wasi skip on unmet env). pytest 14192 passed / 0 failed
-- 33 more than the previous run, because the @live quickjs tests now
execute instead of skipping.
The three new unit tests pin the argv shape without needing the build,
and all three fail without this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(find): s3+gridfs synthesize implicit dirs, honest -empty start (py+ts)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cache,find): refresh ancestor listings, keep negated -name exact, dedup shadowed paths
A deep write on a keyed store materializes every missing level of the key at
once, but only the immediate parent's listing was invalidated, so `ls` served
the pre-write view until the 600s index TTL expired. That needed both
invalidation surfaces: `cache.context.invalidate_ancestors` only fires when a
cache manager is pushed (the command path: tee, cp, touch), while a shell
redirect, FUSE, and the ops facade go through the dispatcher's own post-write
bookkeeping. s3/gridfs `write` gets the first, `CacheManager.invalidate_ancestors`
the second. databricks_volume is left alone deliberately: its write requires
the parent to exist, so it cannot materialize a level.
Nextcloud compiles `-name` to a SQL LIKE, a deliberate superset that the
client-side keep() re-filters. Negation has nothing to re-filter with:
NOT(superset) withholds rows the true predicate keeps, so `! -name '*_b.txt'`
dropped `axb.txt`, which GNU keeps. Only an exact comparison is negatable now,
and a `[` or `\` pattern is not pushed down in either direction.
find on s3/gridfs could print one path twice when a file key and a synthesized
directory name it (`data/a` beside `data/a/b.txt`), which these stores allow
and a filesystem does not; all four implementations dedup like hf already did.
The TS integ runner grows the generic consistency adapter python already had:
adapters expose a shadow workspace over the same store, so scenario cases stop
silently skipping for s3/gridfs/dropbox/ssh/hf, and a target that cannot build
one says so instead of dropping its cases in silence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ssh): fingerprint the TS stat by mtime so ALWAYS revalidates
The python ssh stat carries `fingerprint=mod_str or None` (the remote
mtime); the TypeScript twin hard-coded `fingerprint: null`. Under
ConsistencyPolicy.ALWAYS, reconcile.probe reads that fingerprint, so
TypeScript returned Verdict.UNKNOWN, mayServeCached said yes, and the
cached copy kept being served after the server had replaced it.
Surfaced by the generic consistency adapter in this PR: with only
onedrive/sharepoint registered, `consistency_always_revalidates` had
never run against ssh on the TypeScript host. It now does, and failed
with "v1\nv1\n" against python's "v1\nv2\n".
The remote mtime is the only cheap change token SFTP offers, which is
why python uses it; directories get it too, matching python's single
FileStat. Audited the rest of the class: the other hard-coded
`fingerprint: null` sites (core/dify/stat.ts, node/nextcloud/watch.ts)
both match python's `fingerprint=None`.
Not fixed here: the same function formats `modified`/`atime` with
`new Date().toISOString()` (`.000Z`) where python emits second-precision
`Z` via epoch_to_iso. Aligning that means exporting epochToIso from
core's public index, which is a wider surface change than this fix
needs.
integ `consistency_always_revalidates` on ssh is the regression test —
it fails on the old code and passes on the new; ran the ssh target 3x
(2250 passed, 0 failed each) since the fingerprint is a 1-second-
resolution mtime and I wanted the granularity risk ruled out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(shell): job console for streaming background output
Background jobs stored their output in two byte fields that were only
readable once the job ended, so nothing could watch a running job and a
reader had no way to say where it had got to.
Replace them with a JobConsole: an append-only log of timestamped chunks
addressed by sequence number, where a reader's whole state is one
integer. Readers can join late, read from any position, and disappear
without the console noticing. The job's ending is an in-band CONTROL
chunk, so "is it done" and "what did it print" are the same question.
- mirage/shell/console/: config, store protocol, RAM store, JobConsole.
The RAM store keeps a waiter registry keyed by event loop, so a
reader parked on another thread and loop is woken safely.
- Job drops stdout/stderr for a console. All writes to a job's status,
exit code, and ending happen in one place, so the table has a single
writer and the polling _refresh is gone.
- kill and kill_all are async and join on the console, so a killed job
is settled before kill returns.
- Thread the console down the executor as an output sink. Sequencing
constructs (loops, lists, groups, subshells, conditionals) pass it to
their children, so each statement lands as it finishes instead of the
whole construct arriving at the end. Capture sites (command
substitution, pipe stages, redirects) do not inherit it and keep
receiving their output as a value.
- TypeScript: connect the job's AbortController to the executor, which
already checked the signal but was never given one. kill now stops a
running job, so it joins like Python instead of settling the job
itself.
- Drop the unreachable "no job table" branch in handle_background: the
workspace always has one, so the console is never optional.
Snapshots keep the existing stdout/stderr byte format; a restored job
rebuilds a finished console from it.
* feat(shell): bare `wait` adopts job output, plus compound-job integ coverage
The battery had no case that backgrounded a compound construct: all
eight existing job cases background a simple command, so the path this
branch rewrote was untested. Adding those cases surfaced a real bug.
`wait` with no operand waited for every job and then discarded what they
printed, while `wait <id>` returned it. A real shell has nothing to
adopt because its jobs share the terminal and have already printed;
mirage jobs print to their console, so the shell has to surface it or
the output is stranded. This was pre-existing, not introduced here.
- Bare `wait` now concatenates every unreaped job's console in job-id
order, then reaps. Not just the running ones: a job that finished
before the line was reached still has output nobody has read, and
whether it finished in time is a scheduling accident. Reaping keeps a
second `wait` from reprinting, and matches GNU, where `jobs` prints
nothing once `wait` has returned.
- 18 integ cases in bash/jobs/compound.json covering backgrounded for,
while, if, &&, subshell and brace groups, plus the capture sites that
must not leak (command substitution, pipe stages, redirects) and
stderr routing. Both hosts: 1629 passed, 0 failed.
- Unit coverage for bare `wait` in both languages.
The existing "completed jobs cleared after listing" test now uses
`wait %1`, which waits without reaping, so it still exercises what it
was written for; bare `wait` gets its own case.
* fix(shell): reparse `((` as nested subshells when it is not arithmetic
`((echo a); echo b)` failed with a syntax error. tree-sitter-bash lexes
`((` as the arithmetic opener and the lexer cannot back out, so a
subshell that immediately opens another subshell never parses. Bash
resolves the same ambiguity by trying the arithmetic command first and
reparsing as nested subshells when that fails.
Do the same: on a tree that already has an error, split the `((` openers
sitting inside the error and keep the retry only if it parses cleanly.
Commands that parse today are untouched, so no working command's offsets
move, and the original error is preserved when no reparse helps.
The guard is the subtle part. Splitting only openers inside an ERROR
subtree is NOT sufficient: tree-sitter's error region swallows
neighbouring tokens, so a valid `((i++))` next to a bad opener also
reports as errored, and splitting it silently turns arithmetic into a
subshell running `i++` as a command. That is a wrong parse rather than a
rejected one, which is worse. An arithmetic construct has to close with
`))`, so a line containing no `))` anywhere cannot hold a real
arithmetic opener and splitting it is provably safe; lines that mix both
keep their error.
Covered by unit tests in both languages, including the mixed line that
must stay rejected, plus four integ cases in bash/subshell/nested.json.
Both hosts: 1633 passed, 0 failed.
* fix(shell): judge each `((` opener on its own span, not on the error region
The first cut bailed out whenever the line contained `))` anywhere,
which left `i=1; ((i++)); ((echo x); echo $i)` rejected even though bash
runs it. That guard was sound but far too broad.
Judge each opener separately instead, by parsing its balanced span on
its own: `((i++))` stands alone cleanly and is left untouched, while
`((echo x); echo $i)` does not and gets split. This is what makes a
valid arithmetic command safe when it shares a line with a broken
opener, which scope alone cannot do because tree-sitter's error region
covers both. The span scan skips parens inside quotes and backslash
escapes, so a literal `")"` cannot throw off the depth, and an
unbalanced span is assumed arithmetic and left alone.
Newly working, all matching GNU:
i=1; ((i++)); ((echo x); echo $i) -> x 2
((echo ")"); echo b) -> ) b
((echo a); echo b); ((echo c); echo d) -> a b c d
((echo a) && (echo b)) -> a b
`(((echo d)))` still errors, which matches GNU: bash rejects it too.
Four more integ cases in bash/subshell/nested.json, both hosts 1637
passed 0 failed, plus unit tests in both languages.
* fix: address codex review — wait reaping, control-chunk retention, parse byte offsets
- Targeted `wait %N`/`fg` now reap the job after adopting its output, so a
later bare `wait` cannot replay the same console (GNU bash deletes a job
waited on by id; pinned with docker bash 5.2). Job numbering restarts at 1
once the table empties, matching GNU, so repeated `cmd & wait %1` keeps
working after reaps.
- RAM console stores never trim the terminal CONTROL chunk: evicting it left
wait_finished()/follow() blocked forever under byte budgets smaller than
the outcome payload.
- Python parse: _balanced_end/_is_arithmetic now scan the encoded bytes, so
tree-sitter's byte offsets index correctly when multibyte text precedes an
ambiguous `((` opener (e.g. `echo é; ((echo a); echo b)`).
- TS parse: hoist isArithmetic to module scope (no nested functions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: rename misnamed left_stderr to right_stderr in handle_background
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(shell): kill settles the job instead of joining a runner that may never notice the abort
`kill` aborted the job then joined on the console's finish. But the abort
is only observed where something checks it (the executor between nodes,
and commands that take the signal), so a job sitting inside one long
command never sees it and `kill` waits forever on exactly the runaway job
it is trying to stop. Every existing kill test used a runner that does
observe the abort, so the join looked safe.
kill now settles the job itself — abort, mark KILLED/137, emit `Killed`,
finish the console — and does not join. The console drops emits after the
ending chunk, so a runner still unwinding cannot append past its own
death. Python also gains the `status != RUNNING` guards in `_settle` that
TypeScript already had, so a runner completing in the window after `kill`
(cancel is deferred via call_soon_threadsafe) cannot relabel a killed job.
Regression test added in both languages with a runner that ignores the
abort; it times out on the pre-fix code and passes after.
* fix(shell): settle jobs at teardown, and stop close() stranding readers
Two of the four codex findings on the merge were real.
Teardown settled jobs on main, where kill was sync. This branch made
kill async and teardown dropped to a bare cancel, so a job stayed
RUNNING with no ending chunk and anyone parked on wait_finished waited
forever on a workspace that was already gone. Both close paths are
async, and kill no longer joins the runner, so they await kill_all
instead. It runs before any resource closes, so a job cannot keep
touching one that is already gone. The sync half keeps the bare cancel
as a last resort for a caller with no loop at all.
ConsoleStore.close promised to wake blocked readers, but follow() and
wait_finished() loop: woken, they re-read, find no CONTROL chunk, and
wait again on a console nobody will write to. Stores now carry closed
state, wait() returns immediately once set, and both loops end on it.
Regression tests in both languages, each verified to fail without its
fix. They use an abort-ignoring runner rather than sleep, which is the
one command that consumes the signal and would pass either way, and
they release that runner in a finally so a failure cannot turn into a
hang at loop teardown.
* refactor(io): exit-code reads delegate to the stream source
A streaming command's status can depend on its content (grep's
exit_on_empty settles the origin only at drain time), so merge() no
longer copies the number: the merged result links to its right-hand
original and reads through the link, always fresh. An explicit write
stores locally and severs the link, so aggregated or overridden
statuses still win (issue #43). Deletes sync_exit_code/syncExitCode
and every call site in both languages; the old one-shot sync also
detached on first use, so a sync before the drain froze a stale 0
forever, a hazard class the delegating read removes.
* refactor(console): split config into constants, types and utils
KILLED_OUTCOME moves to constants, Channel/ConsoleChunk/ReadResult to
types (ReadResult joins from store.py, mirroring where TS already kept
it), exit_outcome to utils, per the module-layout convention; config.py
held no configuration. Same split in TypeScript.
* fix(jobs): wait joins the console's ending chunk, not the status field
kill and settle both flip the status before their final appends, so a
status-based return let a waiter snapshot and reap a killed job before
the Killed marker or ending chunk was persisted: a waiter on another
loop today, any store that suspends tomorrow, and in TypeScript any
concurrent waiter, since every await yields a microtask. wait now joins
wait_finished for live jobs (restored jobs keep the no-task fast path),
and wait_all joins every job rather than only the running ones, since
bare wait snapshots killed jobs too. Gated-store regression tests in
both languages, verified failing against the old wait.
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The LLM-facing Notion and Trello mount prompts described trees that no
backend serves, identically in both languages.
Notion omitted the data source level entirely. Since Notion-Version
2025-09-03 a database is a container plus one or more data sources, and
both readdirs have served three levels under `databases/` for a while:
`database.json` plus one dir per data source at depth 2, `data_source.json`
plus row pages at depth 3, row page dirs from depth 4. The prompt still
showed row pages hanging directly off the database.
The prose was wrong the same way, and in a way that would send an agent
looking for a key that is not there: it claimed `database.json` carries
"its typed property schema". It does not. `normalize_database` emits no
`properties` at all -- the schema lives on `normalize_data_source` -- and
`ls` on a database dir lists data sources, not rows.
Trello dropped two directories that readdir has always seeded on every
board: `members/` and `labels/`, each holding one `<name>__<id>.json` per
entry. Only `board.json` and `lists/` were shown.
Both prompts' Notion ntn guidance had also drifted from the CLI, which is
itself gated against real npm ntn@0.21.9 by integ/ntn_conformance.ts. It
advertised `ntn blocks append`, a verb that does not exist, told the agent
to pass ids as `--page`/`--block`/`--datasource`, which are not options,
and used `--json` as a request body when it is a boolean output flag. The
docs already said all of this (docs/*/cli/ntn.mdx: "Ids are positional,
not flags"); only the prompts were stale. They now teach the five real
verbs, positional ids, `pages create --parent/--content`, and blocks,
comments and search through `ntn api`.
The Notion prompts render byte-identical across the two languages; the
ids sentence was previously Python-only.
Docs needed no change -- they were already correct on every count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(cli): one door per state plane for CLI verbs
CLIVerbOpts becomes CLIDoors on inv.doors, carrying dispatch and
stat_path for the data plane, ns for the name plane and session_view
for session state. mount_root is gone: it was the same callable as
ns.mounts.root_of, wired identically at both call sites. Field names
now match CommandOpts, pinned by a meta-test in each language.
Growing the doors closed four gaps in git:
- git add stored a symlink as a regular file holding its target's
bytes; it now writes mode 120000 with the target as the blob.
- a broken symlink was invisible, because the walk followed links
through stat_path; it lstats through ns.links first.
- dispatch("unlink") could not remove a namespace link, so checkout
to a branch without one left it behind.
- commit ignored GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL and EMAIL.
* refactor(workspace): move the record client out of the session package
Sessions, the namespace node table and workspace metadata are three
tables that persist the same way, so the keyed-record clients belong
under workspace/record/ rather than inside one of their consumers.
Before this the namespace and the state store imported upward into the
session package for a client that knows nothing about sessions.
A layering test in each language fails if the record tier grows an
import back into a tier that persists through it.
* fix(shell): route every session write through the policy gate
export and a plain assignment cleared pre_session; the expansion-time
writers did not, so ${X:=d}, $((X=5)), (( )), printf -v and
for ((X=0; ...)) wrote straight to the session env past every policy.
A deployment refusing AWS_* was one ${AWS_PROFILE:=x} away from being
advisory.
All five now write through SessionView.set. The view is threaded into
expansion explicitly and defaults to None, which is correct outside a
workspace. Reads stay sync.
* docs: CLIDoors, the record tier and the session write gate
CLAUDE.md still described CLIVerbOpts with mount_root on inv.ops.
* fix(shell): gate a subscripted printf target, and keep the array
`printf -v 'name[i]'` wrote the session's array table directly, so a
pre_session rule refusing the name never saw it. The door speaks in
whole variables and has since it was written, so the element write now
builds the array on a copy and hands that through view.set, which is
what set_var already asks writers with subscripts to do.
An expansion-time write to a name holding an array was storing a
scalar, which discarded every element after the first. bash writes
element 0 and keeps the tail: `a=(1 2 3)` then `$((a=5))` leaves
`5 2 3`, and `c=("" 9)` then `${c:=x}` leaves `x 9`. Both pinned
against bash 5.2.
* fix(git): replace a link and a file across a checkout
A path that is a symlink on one branch and a regular file on the other
was restored by writing the blob through the link. The dispatcher
follows it, so the content landed in the file the link pointed at,
which no branch touched, and the link stayed in the working tree while
HEAD and the index recorded a file. Linking over a regular file left
that file behind the link the same way.
git replaces the entry in both directions, pinned against git 2.47.
The kind that is there now is a namespace lookup, so a file restored
over a file still costs one write.
* fix(glob): a match named like the glob word is not the nullglob literal
Pathname expansion dropped a real match whose name equalled the word
that globbed for it. With `*a.txt` and `xa.txt` in a directory,
`echo /data/*a.txt` printed only `/data/xa.txt`; GNU bash 5.2.37
(debian:stable-slim) prints both, because the live `*` matches the
literal `*` in the first name.
The merge that unions a backend's matches with the namespace's (nested
mount roots and symlinks, which no backend can see) has to discard the
backend's nullglob-off fallback: a zero-match backend answers with the
literal word, and that is "no match", not an entry to merge against.
It identified the fallback by comparing the returned path to the word.
That comparison cannot work. After quote removal a zero-match glob and
a file genuinely named `*a.txt` are the same string, so the returned
value carries no evidence of which happened, and the real match was
thrown away. A `?` pattern was unaffected only because its word never
equals the name it matches.
resolve_glob_with already answers two different questions depending on
the spec it is handed: a word-shaped spec gets bash's own answer with
the literal reinstated, a directory-shaped one (PathSpec.dir) gets
matches alone. The expander now asks the dir-shaped question, so an
empty list means nothing matched, the comparison is gone, and
resolve_globs stays the only place that reinstates the literal -- which
it has to be, since only it can see whether the backend and the
namespace between them matched anything. That shape contract was an
incidental comment and is now stated in resolve_glob_with /
resolveGlobWith, because the fix rests on it.
TypeScript issued its backend glob call before branching, so a mid-path
word paid for a request whose result was then discarded; python's
if/elif/else never did. The call now lives in the branch that uses it.
_echo_resolve_glob in test_execute_node.py echoed back whatever spec it
was given, commented as matching real resolve_glob behavior. That was
only ever true on the word-shaped zero-match branch, so three tests
asserting the zero-match literal were passing through the unsound
filter. It now answers as a backend holding nothing the pattern
matches. The same latent echo in test_globs.py's default mock is fixed
too; no test exercises it today, but it would mislead the next one.
Pinned against GNU bash 5.2.37 on debian:stable-slim:
echo /data/*a.txt -> /data/*a.txt /data/xa.txt
echo /data/*d/f -> /data/*d/f /data/xd/f
cd /data; echo *a.txt -> *a.txt xa.txt
integ `glob_matched_name_is_the_word` covers it on the backends that
allow `*` in a name; both hosts report 4999 passed, 0 failed on
ram+disk, and the case goes red without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(glob): a match is a child of the directory it was globbed in
Codex review on #790: `resolve_glob` is a public extension point, so the
spec shape the expander sends is not something it can assume a resource
honors. Every in-repo resource funnels through `resolve_glob_with`,
which reinstates a literal only for a word-shaped spec, but a resource
that implements nullglob-off on its own answers a no-match ask with the
spec it was handed -- now the directory. With the word comparison gone,
`/mount/sub/*.nope` could have expanded to `/mount/sub/` and handed a
command the directory.
The merge now keeps only specs strictly under the globbed directory,
which is the guard `_level_matches` already applies for the same reason
one level down. The distinction from the comparison it replaces is the
point: that one tested against the WORD, which a real match can be
spelled exactly like, and was unsound for it. This one tests against the
DIRECTORY, which no match can equal, because a match is strictly longer
than the directory holding it. So the public hook is defended without
reintroducing the ambiguity.
The directory it tests against is the unmarked one #783 already derives
for `_namespace_children`. A match is a real path a backend listed, so a
glob character quoted in the directory's own name is a character of it;
comparing against the marked spelling would reject every match under a
directory like `'/data/*d'/`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`mountForPrefix` throws on a miss and returns a non-nullable MountEntry;
`tryMountForPrefix` is the one that returns null (the #781 mount-lookup
contract). The `?.` here predates that split, so it is unreachable, and
`no-unnecessary-condition` started failing the lint gate when #788
landed -- on main as well as on this branch, so it is not this PR's
change. Unblocking CI here rather than leaving both red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A word-level flag cannot say which metacharacter was quoted, and bash
decides that one occurrence at a time: `'/data/*'?.txt` still globs, on
the `?` alone, against a literal star. With files `*a.txt` and `xa.txt`,
GNU `rm '/data/*'?.txt` removes exactly `*a.txt`; reading the word as a
whole activated both metacharacters and removed both files.
Quoting now rides with the characters. `expand_words` returns the word
with every quoted glob character replaced by its own mark, and the
marks are produced by the expander itself, per node: a quoted node marks
everything it encloses (so `"$p"?.txt` differs from `$p?.txt`), a bare
word marks only what a backslash quoted, and a concatenation just joins
its children. At the matcher, `glob_pattern` hands a marked character
over as its own one-character class, which is what `escape_glob`
already does for quoted text in pattern contexts.
Three properties make this cheap rather than invasive. No mark is a
glob character, so `has_glob` already answers whether a word still
globs -- which retires the separate pattern-stripping pass and the `--`
index alignment it needed, and makes `set -f` just "mark every word".
A mark is one character wide, so every length relation a spec's
virtual/directory/resource_path/raw_path already depended on keeps
holding. And the marks come off at two places only: when resolution
finishes, and at the no-match fallback, where bash's own answer is the
word after quote removal.
Pinned against GNU bash 5.2.37 / coreutils 9.7 on debian:stable-slim,
including the `'*'*.txt` case previously documented as a deliberate
divergence -- bash matches only `*a.txt` there, and so does mirage now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bash applies pathname expansion only to unquoted metacharacters, but
quote removal happened before classification, so classify_word saw the
bare text of '/data/*.txt' and stamped a pattern: chmod 644 '/data/*.txt'
chmodded a.txt and exited 0 where GNU fails with 'cannot access', and
touch '/data/*.txt' touched a.txt instead of creating the literal name.
rm '/data/*.txt' could silently delete every match.
Expansion is the last layer that still sees quoting, so expand_parts
gains a metadata twin expand_words returning ExpandedWord(text,
globbable): a word is globbable only when it carries a glob character
from an unquoted region. A plain word is scanned raw (has_unescaped_glob
honors backslash quoting), a quoted node never contributes, an unquoted
expansion's words are live the way bash treats them, a concatenation is
live when any child is, and a brace template counts an atom only when
its child was an unquoted expansion holding glob characters
(template_globbable). expand_argv and expand_and_classify then reuse the
noglob shape: a non-globbable pattern PathSpec drops its pattern, so
every consumer (shell resolve, backend pushdown, per-builtin re-glob,
for/select word lists) reads the literal name.
Pinned against GNU bash 5.2.37 / coreutils 9.7 on debian:stable-slim:
'...'/"..."/backslash forms all literal; "/data/"*.txt still globs;
unquoted $p globs while "$p" does not; {'*',x} stays literal;
touch/rm address the literal name; for f in '...' iterates once.
Both languages, kept in lockstep: unit tests mirror across
test_parts.py / parts.test.ts, test_brace.py / brace.test.ts,
test_glob_walk.py / glob_walk.test.ts, quoting coverage suites, plus an
integ battery (integ/bash/glob/quoted.json) asserting both hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(shell): ambient session for nested evals, whole-body cmdsub
A re-entrant execute ($(), backticks, eval, source, xargs, timeout,
command, env, process substitution) now continues in the live ambient
session instead of re-resolving an id through the session manager, so
per-call cwd/env forks and restricted named sessions confine every
nested line. Background jobs rebind the ambient session around the
job task, so a nested eval inside '... &' runs in the job's fork and
can no longer read or move the parent session. Command substitution
executes its whole body: the old node filter ran only the first
command node, so $(a; b) ran only a, and declarations, assignments
and control flow ran nothing at all.
Both languages, red-first tests, plus six shared integ cases
(integ/bash/jobs/bg.json, integ/bash/cmdsub/body.json) pinned
against real bash.
* fix(shell): scope the ambient session to its workspace
Codex review on #788.
The ambient session is now owned by the workspace that published it,
so a callback fired mid-line that reaches a SECOND workspace resolves
that workspace's own session instead of adopting this one: a session
carries one workspace's cwd, env and mount grants, and an unrestricted
session must never stand in for a restricted one. A nested bind (a
background job's fork) inherits the owner, so it stays attributed to
the workspace running it.
The TypeScript policy router now decides with the session the line
actually runs, matching Python: it re-resolved the registered session
for cwd and env, so a policy could authorize a relative command
against one directory and have it execute in another. The router no
longer holds a session manager at all.
A background job binds its fork ambiently only where the async context
isolates tasks. The fallback storage a browser gets is one global slot
restored when the scope settles, which would show the job's fork to
the foreground for the job's whole life; there the job's inner evals
resolve by id as they did before.
* fix(links): resolve an operand's link prefix, and honor a trailing slash
POSIX resolves a path one component at a time, and only the last one is
exempt for a command with lstat semantics. mirage resolved either all of
an operand or none of it, so a no-follow command never resolved the
prefix either: `stat /data/dlink/f2` answered "No such file or
directory" and `du` reported 0.
Always resolve the directory prefix; resolve the last component when the
command follows or the operand was typed with a trailing slash, which
POSIX reads as `dlink/.`. A slash then requires a directory, so
`cat reg/` is "Not a directory" rather than the file's contents.
tar is the one command a slash does not reach (GNU strips it and
archives the link); zip is the counter-example and honors it. rm, rmdir,
mv and unlink refuse instead of following, so a slash no longer deletes
the link it was protecting. mkdir now lstats the name it creates, so
`mkdir -p dangle` collides with the link instead of creating its missing
target.
Every expectation is probed against GNU coreutils 9.4 / tar 1.35 on
debian:stable-slim. Python and TypeScript answer identically on all 73
probed cases.
* fix(links): honor a trailing slash on keyed and synthetic backends
A listing never reaches the stat guard, and on a keyed store it cannot
tell "not a directory" from "no keys under this prefix" on its own, so
`ls flink/` answered with an empty listing and exit 0. Guard readdir
too, but only refuse on a stat that actually answers: on a prefix or
synthetic store a directory is the set of keys under it rather than an
object, so a miss is not evidence of a non-directory.
s3 and gridfs ship their own rm and mkdir, so the generic builders'
link handling never ran for them. The refusals move to one shared
helper both tiers call, rather than a second copy per backend.
Drop the anchored-repetition trims CodeQL flags as polynomial (ReDoS)
for the loop-based rstripSlash helper the repo already has.
hf has no rmdir op at all, so the command is not registered on that
mount and answers ENOTSUP before any operand rule applies; its two
rmdir cases are scoped away from it. A keyed store cannot hold an
empty directory either, so the battery no longer asserts one.
* fix(links): validate a removal line before it drops a link, and refuse a slashed mv source
The dispatcher removes a symlink operand from the namespace before the
command layer parses, so a line that layer rejects has to leave the link
alone: GNU answers `unlink a b` with "extra operand" and `rm --bogus x`
with "unrecognized option", both with everything still in place.
mv gets the same treatment for a link typed with a trailing slash.
rename(2) never follows, so POSIX's `dlink/` == `dlink/.` asks for a
directory the call will not resolve, and GNU refuses in four wordings
depending on what the link leads to and what the destination is.
Also pass the index through the slash guard's readdir probe: a synthetic
backend resolves a path through it and cannot stat without one, so chroma
answered "missing index" for `ls /knowledge/`.
All wordings pinned against GNU coreutils 9.7 on debian:stable-slim.
* feat(dsh): ship the package as a dsh bundle with declarative mounts
* fix(dsh): claim workspace-write only for a confined runtime world, recheck aborts after the ready wait
* fix(runtime): declare confined on the monty test double
* refactor: one mount-lookup contract — mount_for raises, try_mount_for returns None (py+ts)
Cleanup-plan item 23 (T1-K). Python's mount_for raised bare ValueError
while TypeScript's mountFor returned null, so every call site coped its
own way and the two languages drifted. Both registries now expose the
same explicit pair:
- mount_for / mountFor: the path must be mounted; a miss raises the
typed NoMountError (py) / noMount-stamped Error (ts). Never caught —
callers with a fallback use the try variant instead.
- try_mount_for / tryMountFor (+ _prefix twins): None/null on a miss.
All ~70 call sites swept with per-site intent: expected-miss sites
(classify, globs, storage keys, drift, provision, cross-mount scopes,
history, df, man, find actions, cache gates) take the try variant with
their existing fallbacks; post-routing invariant sites (metadata
builtins, expand_operands, spec_hints, watch) keep the raising lookup.
Blanket `except ValueError` catches around lookups are gone, so backend
ValueErrors can no longer be swallowed as "no mount".
Cross-language divergences fixed along the contract:
- TS groupByMount silently dropped unmounted paths; it now propagates
like Python.
- Python mount_for_command could route a command to the /dev/ mount;
it now skips DEV_PREFIX like TypeScript.
- TS applyStateDict resolved snapshot prefixes by longest-prefix match,
so a stale nested-mount prefix would load resource state into the
ancestor mount; it now uses the exact-prefix lookup like Python.
- TS specForCommand masked a broken-registry miss with ?? null; it now
propagates like Python's documented behavior.
- TS Workspace.mount() returned null where Python raised; both raise.
- TS watch layer's private throwing wrapper is gone; the WatchRegistry
interface carries the raising contract directly.
- Deleted the duplicated hasGlob(r) method probes (metadata.ts,
expand/globs.ts) in favor of the plain resource.glob !== undefined
narrowing mount.ts already used.
Verified no user-visible behavior change: an 18-case unmounted-path
probe (chmod/touch/chown/chgrp/cat/ls/stat/cd/rm/cp/du/mkdir/redirect)
is byte-identical before/after on both languages and matches GNU
(docker debian:stable-slim). New integ file integ/unix/meta/
unmounted.json pins the GNU-exact errors on 22 targets; ram battery
2438/2438 on both hosts.
Related #74 (the /dev/ skip removes one wrong-pick class; the
first-wins ambiguity remains) and #30 (contract groundwork; cross-mount
command alignment still open).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: normalize the prefix argument in py try_mount_for_prefix (codex P2)
The new try variant inherited the old exact-string compare, so the
registration spelling ("/data", "data/") missed the stored "/data/"
entry only in Python; the TS twin normalizes. Normalize with the same
idiom mount/unmount use, delegate is_mount_root through it like the TS
registry does, and pin the spelling set in both languages' tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* fix(shell): don't unescape an already-expanded word in classification
classify_word ran a second POSIX quote-removal pass over words the
expansion layer had already unquoted, so a backslash that survived
expansion (a literal name character) was stripped again. printf's own
format arg is path-shaped, so `printf '/data/x\ty'` lost the tab before
printf ran, and `ln -s "$(...)"` stored a target with no tab.
Delete _unescape_path / unescapePath and both call sites in each
language. The relative-branch call site was unreachable: _RELATIVE_PATH
admits no backslash.
* test(integ): give the symv directory a real file so hf can remove it
hf's mkdir is a no-op, so a directory holding only symlinks (namespace
state, never backend keys) has no backend existence and rm -r reports
ENOENT. Write a file at setup the way sym/setup.json does; rm -r takes
it along with the directory.
* docs(hf): say why hf mkdir cannot write a directory marker
The comment claimed object stores have no real directories, but s3 and
gridfs are object stores and both write a zero-byte `key/` marker. The
real reason is the client: OpenDAL's hf service reports create_dir
unsupported and rejects a write to a slash-terminated path, both
client-side, in the python and node bindings alike.
* feat(session): per-session hidden paths and vars via SessionProfile
* fix(session): close the hidden-var write doors, HOME channel, hydration, and native-find bypass
* fix(session): filter hidden arrays from expansion and gate the raw env writers
GNU spells find's negation two ways and mirage only understood one.
`find /data -name '*.txt' ! -empty` exited 1 with `unknown predicate
'/!'`, because find's spec declares `rest=Operand(type="path")`, so
every word without a leading dash is classified as a start point and
`!` was cwd-prefixed into the path `/!`. `-not` worked, and so did
`\( ... \)`, because the parens were already exempted through
`ignore_tokens`; `!` was simply missing from that set. Nothing
downstream needed teaching -- `!` has always been in the expression
parser's operator table.
Fixing the classification made a second bug reachable: a line that
ends on an operator. `find /data !` used to print the whole tree and
then a bogus "No such file or directory" for `/!`; with the operand
gone it would have negated nothing in silence. Both parsers now refuse
it in GNU's words, and the check runs where the operator is consumed
rather than in the primary, because by then the token that needed a
right-hand side is no longer on the line:
find: expected an expression after '!'
find: expected an expression between '!' and ')'
Two py/ts divergences surfaced on the way and are closed here:
- The TypeScript parser skipped an ignore token silently, leaving its
`wordKinds` slot null, which means "guess from the shape" -- and the
shape of a grammar token says nothing about it. Python stated TEXT.
TypeScript states it now too.
- `spec_hints.ts` carried a by-value override that re-nulled ignore
tokens after the fact. It was redundant once the parser states the
kind, and it was position-blind: it matched an option's value as
readily as a grammar token, so `find /d -name '!'` lost the TEXT the
parser had correctly given the pattern. Deleted.
The new cases also caught a live bug in the Nextcloud backend, fixed
here. Its find pushes the predicate tree down as a WebDAV SEARCH, and
Nextcloud inverts a <d:not> by swapping the comparison inside it for
its opposite, so a <d:not> around anything but a single comparison
raises server-side:
Binary operators inside "not" is not supported (HTTP 500)
Probed against nextcloud:30-apache (30.0.17), the image CI runs:
not(like) and not(is-collection) answer 207, while not(or(..)),
not(and(..)) and not(not(..)) all 500. The compiler now tracks whether
a compiled condition is a bare comparison and refuses to negate one
that is not, which falls that find back to the scan walk, which has no
such limit. `-not` reached this first and has been answering 500 all
along: `find /data -not -type f` compiles to not(not(is-collection)),
unnoticed because find_not_name was the battery's only negation case
and negates a bare -name. find_bang_type_f now pins it.
Pinned against GNU findutils 4.10.0 on debian:stable-slim as a
three-way byte-identical differential (GNU vs python vs typescript)
over 25 output cases and 15 error cases on one shared seed tree. One
known divergence is normalized and commented: GNU quotes as `x' where
mirage quotes 'x' repo-wide, which is pre-existing and already pinned
in integ/unix/find/error.json. Also left alone as pre-existing: GNU's
wording for a binary operator with nothing before it, the
paren-specific messages, and "paths must precede expression".
integ gains five output cases in find/not.json and three error cases
in find/error.json. The output cases carry seq 295, the same seq as
find_not_name whose expected output they derive from: the battery runs
one shared session ordered by seq, and every one of the eight fixtures
that would otherwise intrude (/data/fs at 302, /data/dr at 317,
/data/rgm at 459, /data/arch at 467, /data/xm2ctree at 755, /data/g4
at 500200, /data/f4 at 500300, /data/roots1 at 785) is created after
295 and never cleaned up. The one case already at 295,
find_empty_dir_holding_only_a_link, removes its own tree in the same
line, which is what leaves the slot usable.
chroma and github each gain the `!` twin of the `-not` case added in
#772, placed immediately after it in both file order and seq
(chroma_find_empty_composes_with_name -> 630056,
github_find_empty_composes_with_name -> 530014). Both verified against
a locally run fake on both hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(stat): render the symlink arrow for %N
stat -c '%N' on a symlink printed only the quoted name. GNU prints
'name' -> 'target'. The target was already on FileStat.extra under
LINK_TARGET_KEY, which is what ls -l and file read, so %N now reads
the same fact.
Pinned against GNU coreutils 9.7 on debian:stable-slim, which showed
two more rules: any format modifier drops the quoting (only a bare
%N quotes), and a width or precision applies to the name and the
target separately rather than to the joined line. That second rule
had to be implemented either way to render %20N on a link, and it
also fixes %30N on a plain file, which used to keep the quotes.
Tests in both languages plus six integ cases across all 22 targets.
* fix(stat): quote %N the way gnulib shell-escape does
The helper picked double quotes for any name holding an apostrophe, so
a'b$c rendered as "a'b$c" and replaying the advertised shell-safe output
expanded $c. It also passed control characters through raw.
Probed GNU 9.7 across ASCII 32-126 and the C0 range: single quotes are
the default with '\'' escaping, double quotes are an optimization for a
name whose only awkward character is an apostrophe, and a control
character becomes a $'..' segment spliced between single-quoted runs.
Now applies to a symlink target too, since %N quotes each field on its
own.
* fix(glob): let glob expansion see nested mount roots and symlinks
Glob expansion enumerated candidates from a single backend's readdir.
A nested mount's keys live in another resource and no resource stores a
symlink, so neither was ever offered as a candidate, even though both
already show up in a plain listing via merge_readdir.
Both glob tiers now merge the same namespace_names union a listing uses,
session-filtered, and matches sort like bash. A glob whose directory
holds a child mount is expanded before routing so its matches route per
mount, and that happens before the follow policy so a matched symlink is
followed exactly as a typed one is.
Pinned against GNU coreutils 9.7 on debian:stable-slim with a tmpfs at
base/inner and the same symlink.
* fix(glob): expand boundary globs before admission, follow linked dirs
Codex review on #775.
Run the boundary expansion before pre_command, not just before the
follow policy: a word left unexpanded reaches MountRootPolicy as the
literal pattern, so `tar -cf out.tar /base/*` archived a mount root the
same operand typed by hand is refused for. cp -r and zip too.
Compare operands as words rather than by count, so a glob matching
exactly one name still installs. `du /base/i*` failed while
`du /base/inner` worked.
Resolve a glob's parent directory through the namespace link table
before a backend lists it, spelling matches back under the typed name.
Pinned against GNU bash on debian:stable-slim: `echo base/d*/f2` ->
`base/dlink/f2`, `echo base/*/f2` -> `base/dlink/f2 base/sub/f2`.
Drop a walk match that is the listed directory itself. A backend asked
to list a path that is really a file answers with that file, which
walked back out as a doubled segment (`/base/f*/f1` -> `/base/base/f1`)
where bash keeps the literal.
* test(integ): cover glob expansion across mount and link boundaries
Seven cases on ram-nested for the boundary glob rules, and one on
ram/disk for a glob whose parent is a symlinked directory. Five of the
eight fail against the previous commit, one per fix.
Expectations are pinned against GNU on debian:stable-slim with a tmpfs
at data/inner, except the mount-root refusal, which is mirage's own
documented divergence.
* feat(dsh): DeepSeek Harness fs and shell providers over a mirage workspace
* docs(dsh): monty python run and multi-source mounts in the dsh example and page
* docs(dsh): tidy the example into sectioned helpers
* docs(dsh): trim the page to install, compose, python, one world
* docs(dsh): use mirage as the example report owner
* docs(dsh): slack-hosted python script example
* docs(dsh): rename the slack-hosted script to example.py
* fix(dsh): match Slack upload names and newest-first date dirs in the example
* fix(dsh): address codex review on lock retention, pre-aborted signals, cyclic links, and byte limits
* docs(dsh): generalize the page, compose sample on redis + ram
* docs(dsh): keep slack in the compose sample
* docs(dsh): state the python3 capture forms
* docs(dsh): show the runtimes capture spec with an inline python sample
* docs(dsh): monty is the workspace python, capturing python and python3
* docs(dsh): one continuous example, no second workspace
* docs(dsh): show monty explicitly capturing python and python3
* docs(dsh): one example end to end — ram + redis + slack, monty capturing python
* docs(dsh): drop the one execution world section
* docs(dsh): state the in-memory property with measured capacity
* docs(dsh): drop the capacity numbers
* docs(dsh): drop the in-memory paragraph
* docs(dsh): add a harness comparison table
* docs(dsh): add concurrency column on a 2 vcpu / 8 gb server
* docs(dsh): correct the live python engine footprint to 27 MB
* docs(dsh): no em dashes, dsh row name, serving scenario before the table
* docs(dsh): tmp mount, redis-bound report, numeric concurrency table with startup
* docs(dsh): note the sandbox provisioning cost the cli sessions would add
* docs(dsh): sandbox startup cost in the cli startup cells
`du /empty` reported `cannot access '/empty': No such file or directory`
and exited 1 for the implied parent of a nested mount, even though it
printed the right rows. `/empty` is a real directory, but no backend
holds an entry for it: the content lives in the descendant mount's own
resource.
du ran bound to one mount and decided existence from that mount's stat,
so it could not see either fact that makes such a path a directory: a
mount nested below it, or a symlink below it. Both are namespace state.
The dispatcher already answers correctly for them (namespace_stat behind
the backend's miss), and that answer is offered to every handler as
`stat_path`; `find` was the only consumer, which is why find was the one
sibling that got this right.
du_operands now asks that probe when the backend stat fails, ahead of the
has_content walk, which is both cheaper and authoritative. has_content
stays behind it for a backend with no directory entry for its own mount
root.
stat_path rather than registry.descendant_mounts on purpose: the registry
lookup is not session-filtered, so proving presence from the mount table
would answer `0 /empty` for a session that may not see /empty/hole and
confirm a walled-off mount's parent. It also misses the link-implied
case, which has no descendant mount at all. Both are pinned by tests.
Pinned against GNU coreutils 9.7 on debian:stable-slim with a tmpfs
mounted at /empty/hole: both rows, no diagnostic, exit 0.
* feat(du): honor -S/--separate-dirs (py + ts)
GNU -S makes each directory total count only files that sit directly in
it, leaving subdirectory sizes out of the parent. Shared rollup and
summarize paths now match coreutils 9.7, with unit and integ coverage.
* fix(du): apply -S across nested mounts, keep the -c total recursive
The traversal fan-out re-derives du's tree centrally from the per-mount
blocks, so -S was silently dropped there: du -S over a mount printed the
recursive total. Teach the merge about it, and split the operand's own
row from what it contributes to -c, which GNU keeps recursive.
Add a guard that fails when a new du flag is neither applied centrally
nor classified as per-run, which is how -S went missing.
* chore(spec): regenerate du spec json for -S
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* feat(workspace): state doors and per-plane views
* fix(workspace): one general session door for every variable write
Codex round: shell readlink dispatches through the op door; array-shaped
assignments and subscripted unsets can no longer bypass pre_session.
SessionView.set is now general over variable shapes (str | ShellArray),
pre_session_gate is private to the door and takes a SessionContext that
carries session_id, staged declaration arrays store through the builtins,
for/select loop variables write through the view, and readonly pre-checks
are separated from policy-denial catches at every shell site. Also fixes
export ARR=(x) printing the export listing and CI formatter drift.
* test(integ): pin the state-door fixes; readonly declaration arrays abandon the line
* ci(integ): key the WinFsp install on the DLL, not choco's exit code
* wip(seam): py dispatcher builds CommandOpts; handlers take (accessor, paths, texts, opts); adapter op protocols; DispatchFn typing
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(seam): dispatcher-built CommandOpts, typed adapter ops, named ParsedCommand, one DispatchFn home (#609 items 26+23)
Item 26 (T2-8 seam typing), all four pieces, plus task 23's static
FlagView query-name gate:
- The python dispatcher (Mount.execute_cmd) now constructs one
CommandOpts per invocation and calls every handler as
fn(accessor, paths, texts, opts) — the TS convention. Flags stop
sharing a bag with injected context, accepts_kwarg opt-in dies, and
all 72 builders, 75 bespoke wrappers and ~30 provision functions
become pure wiring; the provision path builds the same bag with
command/spec set. Generics that took trailing fact params (ls, find,
tree, zip, tar, du, file, stat) now read opts.links/opts.mounts/...
like their TS twins.
- adapter.py gains per-slot op protocols (ReaddirOp/StatOp/WriteOp/...)
mirroring adapter.ts, so wiring readdir where stat belongs no longer
type-checks; Builder.fn/provision and CommandIO fields are typed.
Surfaced and fixed real drift: hf/databricks mkdir had index before
parents (a positional parents call would land in index), wget -O
dropped its PathSpec value under as_str.
- TS parseFlags returns a named ParsedCommand (twin of the py
NamedTuple) instead of a 15-slot positional tuple; optionError takes
(cmdName, parsed) like python's option_error.
- DispatchFn moves to runtime/types.ts (python's home for the same
protocol); the crossmount re-export inversion and the CommandDispatch
duplicate are gone; 38 python files stop spelling it
Callable[..., Any]. The dead CommandOpts.resource field (zero readers)
is deleted along with the CrossResourceStub that existed to fill it.
- Task 23: tests/commands/test_flag_query_names.py and
commands/flag_query_names.test.ts fail on a FlagView query naming a
dest no spec bound in the module declares, without needing the code
path to run (both verified to fire on a planted typo).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(examples): redis example calls provisions with the 4-positional shape
file_read_provision/head_tail_provision/metadata_provision are
(accessor, paths, texts, opts) now; the direct demo calls still passed
the old command= kwarg, crashing the example before the persistence
section (CI examples gate).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): pin the #772 fixes where the harness can reach them
- find -empty on chroma + github (the wrappers used to drop the flag,
which would flood every path; -empty now rides find_generic), plus a
-not -empty composition case on each so the pin has positive output
- mkdir -p gains databricks/databricks-prefix (zero prior mkdir
coverage on that backend; the param-order bug itself is only
reachable positionally, so MkdirOp+mypy is its real gate)
- wget -O / curl -o were already pinned by integ/resources/http
Both hosts green: py chroma+github 83 ok, ts all five targets 4070 ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(ts): derive config types from their zod schemas
Every resource and account config declared its shape twice, once as an
interface and once as the zod schema that actually validates it, plus a
redacted twin that re-listed every field by hand. Two type helpers in
resource/secrets.ts replace both: ConfigOf reads the shape off the schema,
and RedactedConfig builds the twin from the config plus the names of its
secret fields.
ConfigOf is not a bare z.infer: that renders an optional field as
`k?: T | undefined`, which under exactOptionalPropertyTypes is wider than
the `k?: T` these configs were written with and stops an S3 alias's state
from matching its base's.
The twin is reached through an `as unknown as` cast, so nothing checked it
and it had drifted. gcs, oci, r2, supabase, wasabi, hf_buckets, node's s3
and gridfs all typed redacted credentials as plain strings, and box dropped
a field the schema carries.
Ten files keep a hand-written config type because their schema is not the
config's own shape, each saying so in place: s3 has a profile the schema
lacks, google/box/dropbox omit callbacks no snapshot can carry, and the
S3-alias family and hf_buckets describe the resolved config the redactor
fills in.
Browser trello and linear declared readonly arrays their schemas did not,
which is the only runtime change here.
* fix(ts): keep readonly arrays on core linear and node trello configs
The old interfaces declared teamIds/boardIds as readonly string[], and
deriving from the schema narrowed both to string[], so an as const value
a consumer used to pass no longer typechecks. Mirror the .readonly() the
browser twins already carry.
The whole sweep removed exactly four readonly array fields, in browser
linear/trello and these two; the browser pair was already fixed.
Two CI-only gate failures the local pre-commit hooks never run:
- check_layout_parity --strict is a two-way ratchet, and pairing
commands/builtin/utils/operands.py with its TS twin closed one
divergence (305 -> 304); lower the committed baseline to lock the
improvement in.
- The directory-list checksum pin assumed `mkdir -p` leaves a statable
empty directory, which huggingface's keyed backend does not have (no
marker object, nothing in the parent listing), so hf/hf-prefix
reported ENOENT instead of GNU's "read error". Narrow the case's
targets, matching the existing unlink_dir precedent; the
missing-list pin stays on hf, where it passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(integ/github): repo, contents, issues and compare routes
The fake served the read path the `github` resource mounts -- repo
metadata, git/trees, git/blobs -- plus the Actions dataset `github_ci`
needs. A task that *acts* on a repository had nothing to call.
Adds the routes such a task uses: GET /user; POST /user/repos; DELETE
/repos/{owner}/{repo}; POST .../forks; GET .../branches and
.../branches/{branch}; GET .../commits; GET and PUT .../contents/{path};
GET and POST .../issues; GET .../compare/{basehead}.
Writes are visible to the next read, which is the point -- an issue
filed shows up in the listing, a committed file reads back with its new
bytes and a new blob sha, and the commit is recorded so `compare`
can answer which files moved. `FakeRepo` grows `issues` and `commits`
for that; a repository with no writes yet still reports one synthetic
root commit, so "the latest commit" is answerable before the agent has
done anything.
Three behaviours worth pinning rather than leaving to chance:
- A fork deep-copies. A task that forks an archive repo per run and then
commits to the fork must not write through to the source.
- PUT /contents enforces GitHub's sha rule in both directions: replacing
an existing file without the current blob sha is 409, and supplying
one for a file that does not exist is 422. A task that reads before
writing is doing so for this reason.
- A compare against a base the repository has never seen is 404, not an
empty file list. Answering "nothing changed" to a question about an
unrelated commit is the shape of wrongness that reads as success.
`_commit_list` is newest-first, so `compare` collects the commits
*before* it reaches the base; walking past the base instead would report
the commits the base already contains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(integ/github): git data API, rename, raw content and GHES paths
The write half a task that builds a repository needs, plus the two shapes
a real client insists on:
- PATCH /repos/{owner}/{repo} renames, carrying the content with it, and
POST /forks takes a name so a fork can be named in one step.
- The git data API -- get commit, create tree, create commit, move ref --
so a multi-file push lands as one commit. A staged tree is invisible
until a ref points at it, which is what git does and what keeps an
abandoned tree from showing up in a read.
- GET /git/ref/{ref}, and trees by commit sha, because a client resolves
a ref to a commit before reading anything.
- /raw/{owner}/{repo}/{ref}/{path}, with a text content type for text, so
a reader does not base64 the whole repository.
- Every route is served under /api/v3 as well as at the root: a client
pointed at a host that is not github.com talks Enterprise.
- GET /search/repositories, and an unrouted fallback that names the
method and path on stderr. Both exist for the same reason: a 404 is an
answer to a real client, so an unimplemented endpoint reads as a
negative result rather than as an error.
- Seeding takes a per-repo default branch, since a template whose branch
is master is graded at that ref by name.
* feat(gh): a GitHub CLI, and the write routes the fake needs behind it
The `github` mount is the read half -- a repository is a tree, so listing
and reading it is `ls` and `cat`. There was no write half at all:
GITHUB_IO carries readdir/read/stat where GDRIVE_IO carries the full set,
so nothing could commit a file, and forking and renaming are account
operations a filesystem has no shape for in any case.
`gh` is that half, spelled as cli.github.com spells it:
gh repo view [OWNER/REPO]
gh repo fork OWNER/REPO [--fork-name NAME]
gh repo rename NEW-NAME -R OWNER/REPO
gh api ENDPOINT [-X METHOD] [-f key=value] [-F key=value]
`rename` takes the new name as the operand and the repository as -R, which
is the reverse of what the shape of the line suggests and is upstream's.
`-f` sends a string and `-F` reads true/false/null/integers as their JSON
types, which is gh's own --raw-field / --field split; a call with no fields
is a GET, one with fields a POST unless -X says otherwise. Committing a
file is `gh api ... -X PUT -f content="$(base64 -w0 f)"`, which is what
real gh and real GitHub require of each other -- there is no file verb to
mimic.
The transport was GET-only, so it grows a `request`, and an empty body (204,
an empty 202) decodes to null rather than throwing on a call that worked.
Fake, in support: DELETE /contents with the same sha rule as the replace,
and GET /commits/{ref}, which is not /git/commits/{sha} -- it takes a branch
name and reports the file list. Both were found by the unrouted logger
during a run: an agent issued 35 deletes, every one 404'd, and it reported
the files as removed, because to a real client a 404 is an answer.
* feat(integ/github): repository metadata, star-sorted search and readme
A task that picks *between* repositories reads their descriptions and star
counts and never opens one, so seeding gains a second mode: --metadata takes
a JSON file keyed by owner/name and creates repositories that are metadata
only. The tree and contents endpoints answer 404 for them, which is what
GitHub says about a repository with no files.
search/repositories honours sort=stars and matches the description as well
as the name, since a repository is found by what it says it does at least as
often as by what it is called. Terms OR rather than AND, which is looser
than GitHub and errs towards showing a caller the row it wants.
GET /repos/{owner}/{repo}/readme, found by the unrouted logger during a run.
* feat(integ/github): branches
The fake kept exactly one branch, which rules out every task whose premise
is a difference between two of them. FakeRepo now holds a file map and a
commit list per branch:
- `files` and `commits` stay bound to the default branch, so every route
that does not name a ref reads what it always read.
- `branch_for` resolves a ref -- a branch name, HEAD, or a commit sha
belonging to one branch's history -- and `tree_of` returns that
branch's files, so a bad ref is a 404 rather than a silent read of the
default branch.
- contents, tree, raw, readme, branches, branch, commits, commit and
git/ref are ref-aware; PUT and DELETE /contents write to the branch
their body names; PATCH /git/refs moves the branch it names.
- Each branch has its own synthetic root commit, so two branches of a
fixture do not share a head.
- A fork copies every branch, not just the default one.
- Seeding takes `into=<branch>`, so a two-branch fixture is two
directories and two --repo flags naming one repository.
The default-branch binding is what keeps this small: 18 call sites read
`repo.files` and none of them had to change.
* feat(integ/github): GET /git/refs/{prefix}
The plural is a different endpoint from the singular: git/ref/<full-ref>
returns one object, git/refs/<prefix> a list of everything beneath it. A
caller picks whichever it expects, so serving only the singular made the
plural read as 'no such ref'. Found by the unrouted logger during a run.
* feat(integ/github): account repo listing, bare /contents, mirror-stable shas
Three things a task that reasons across several repositories needs.
GET /users/{login}/repos and GET /user/repos. A task that asks "is there a
repository for this on my GitHub" answers it here, and a 404 read as "the
account has none" -- a wrong answer rather than an error, which is the
failure mode the unrouted logger exists for.
GET /repos/{owner}/{repo}/contents, with no trailing slash. GitHub serves
both spellings of the root and a caller picks either; only one was routed.
A branch's root commit sha is now derived from its content rather than from
the repository's name. That is what `git clone --mirror` followed by a push
actually gives you -- the same shas -- and a grader that reads an initial
sha from an upstream repository and the latest from a local copy, then asks
what changed between them, depends on it. Two branches still differ exactly
when their trees differ.
* feat(integ/github): POST /git/refs, so a branch can be created
The last write a task can ask for that the fake had no answer to. A new
branch starts as a copy of whatever the base sha resolves to -- which is
what a branch is, another name for one commit and everything reachable from
it -- with its own empty history, so the two do not share future commits.
422 on a ref that is not under refs/heads/, on a name that already exists,
and on a base sha that resolves to nothing.
* feat(integ/github): GET /, the API root
github.com answers the root with a map of endpoint URL templates. Serving
only the endpoints beneath it meant a client probing the root got a 404,
which reads as the host not being there at all rather than as an unfamiliar
API. Found by the unrouted logger during a run.
* feat(integ/github): seed a branch's commit history, not just its tree
Seeding a directory filled a tree and nothing else, so every seeded
branch answered GET /commits with one synthetic root. That is enough for
a task that reads files and wrong for one that asks when something
arrived. `task-tracker` wants the tasks added by the *most recent*
commit on each of fifteen developer branches, and each branch has five
or six -- so a history of one does not make the task hard, it makes it
unanswerable while looking answerable.
--commits owner/name=<file> reads a manifest keyed by branch, each an
array oldest first, and is applied after every tree: a manifest names
branches, and a branch no directory was seeded into would otherwise be
created here as an empty tree with a history, which reads as a branch
whose files were all deleted.
Shas are derived from full_name, branch, date and message rather than
carried, for the same reason the root's is derived from its tree: a
fixture has to answer the same way twice.
Commits now carry commit.author/committer (name, email, date) and
author.login, which is what a client sorting or filtering by date reads,
and the synthetic root carries them too against a fixed epoch -- it
stands for everything before the fixture rather than for a moment, so
"now" would make it sort ahead of the commits it precedes.
GET /commits/{ref} now renders `files` as {filename, status} objects the
way GitHub does. The list endpoint keeps paths, which is also GitHub's
split: `files` is absent from a commit in a list and detailed when one
commit is asked for. _commit_files is shared with the compare route,
which was building the same objects inline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(github): back the transport with octokit
Wraps @octokit/core behind the unchanged GitHubTransport, so the github
mount and the gh CLI both gain retry and throttling. Handles three
octokit behaviours: {} in a url is a route template that silently eats
the segment, the "METHOD /path" form splits on whitespace, and 204
decodes to '' rather than null. The write throttle is github.com's own
secondary rate limit, so it is off for any other host.
* fix(gh): name the mount its writes invalidate, and parse the host segment
serves was empty, so the executor's post-write cache drop did nothing and
a committed file still read back as its pre-write bytes. parseRepo took
the first two segments of [HOST/]OWNER/REPO, so github.com/acme/tools
resolved to github.com/acme -- a different repository, reported as
success.
* fix(cache): invalidate the index instead of clearing it
A cleared index reads exactly like one that was never filled, so github --
whose index is the whole listing rather than a cache in front of one --
could not tell a cache drop from an empty repository, and ls reported the
mount root missing. IndexCacheStore grows invalidate(), which expires
entries in place, and github refetches once on an EXPIRED lookup. Also
fixes a latent bug with no CLI involved: after the 24h index TTL lapsed a
github mount answered ls with exit 0 and no output. Redis cannot express
stale (an expired key is gone), so it still clears; the comment says why.
* feat(gh): a python gh CLI, mirroring the typescript one
Same four verbs and the same grammar. The python client was GET-only, so
it grows github_request alongside github_get. Layout divergence 305 -> 303.
* test(integ): a gh CLI battery on both hosts
15 cases on a new cli-gh target, covering the write-then-read path through
the mount, the refusals, and the reversed rename grammar. The fake grows
POST /reset, because the cli facet runs both hosts against one process and
writes would otherwise leak between them; it also stops rejecting wrapped
base64, which real GitHub accepts. The unrouted-request logger is now a CI
failure rather than a log line.
* fix(github): probe the parent listing, so an update is not read back stale
The index tracks freshness per directory, never per entry, so get()
never answers EXPIRED and the refill it guarded was unreachable: after a
write invalidated the index the blob's row survived carrying the
pre-write sha, and the read served the old bytes. Create and delete went
through readdir and worked; an update did not.
Readdir on the TypeScript side refilled on any miss, which spent a full
recursive-tree call on every ENOENT and diverged from python. Both now
refill only on EXPIRED.
* test(cache): pin what invalidate keeps that clear discards
A cleared store reads exactly like one that was never filled, which is
what made a github mount report an empty repository. Assert the two
answer differently.
* fix(spec): render an operand's declared name outside the clap dialect
A named slot printed as <text>, so 'man gh api' said <text> where gh
says <endpoint>. argparse prints the dest too. gh is the only spec this
changes; ntn is clap and already rendered it.
* test(integ): cover the gh surface the battery missed
Update-then-read is the case that found the stale-read bug. Also the
help and discovery surface, a leading-slash endpoint, typed fields on a
GET, and a missing endpoint.
* docs(gh): a page per host for the gh CLI
Install, the mount-reads/CLI-acts split, the api rules, and a table of
what differs from gh 2.85.
* fix(github): keep an api field from steering the request
Octokit reads loose parameters off the same object that carries url,
method and headers, so `gh api X -f url=...` retargeted the call
instead of sending the field. The query is spelled into the url and the
body travels as `data`. The python client already separated them.
* fix(cli): let a handler say it did not write
A leaf declares `write` statically because for almost every verb it is
static, but `gh api` carries its method on the line, so a read like
`gh api /user` expired every github mount the install serves. A result
may now report what the spec cannot know.
* feat(gh): expand the owner, repo and branch placeholders
gh's own examples are written with them, so `gh api
repos/{owner}/{repo}/releases` asked for literal braces and 404'd. An
install's repo and branch stand in for the current checkout. Any other
brace pair still reaches the wire, which is gh's behavior too.
* feat(gh): render repo view the way gh renders it
A name line, a description line, then the README, with the separator
omitted when there is none. Probed against gh 2.85. The REST object is
still one `gh api repos/OWNER/REPO` away.
* test(integ): cover placeholders and the new repo view
Also carries a branch on the gh install, which is what {branch}
expands from.
* fix(gh): declare branch on the GhConfig type, and type the test trees
tsc runs in CI but not in pre-commit: GhConfig's zod schema grew
`branch` while its interface did not, and the new github tests built
tree items with a `mode` key the type has no room for.
* refactor(gh): derive GhConfig from its schema instead of declaring it twice
The zod schema is the one doing real work: it validates an install's
config and carries the secretStr marker redaction reads. A hand-written
interface beside it only adds a shape that can drift, which is how
branch reached the schema and not the type.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GNU 9.7 pinned on debian:stable-slim: each operand is its own checksum
list, verified in order; an unreadable one reports its strerror and the
run continues with exit 1, a directory operand reports the literal
"read error" (fopen succeeds, the read fails), and --status keeps the
strerror lines while silencing OK/summaries. The defect predated this
PR in the TS generic (paths[0] since #731), so both sides are fixed
together: py hashsum loops with WALK_ERRORS translation, TS
checksumGeneric loops over checkFile with the new shared isEisdir
(twin of isEnoent), and the five py checksum builders bind
dir_aware_stream like their TS twins so implicit keyed-backend
directories classify as EISDIR on the check path. Mirrored unit tests
plus two unix/sha256sum integ pins (seq 960093/960094), verified on
both hosts over ram+disk (4823/4823 each).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cat/head/tail/wc x postgres+mongodb mixed-operand pins, history tail
cross-mount continue, and a postgres directory-operand EISDIR pin that
exercises the implicit-dir parent-listing probe end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A lazy async-generator wrapper deferred the factory reader call to
drain time, past the command's cache-manager contextvar scope, so a
warm S3 LAZY cat re-read the backend instead of serving the cache.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
postgres/mongodb mixed good+missing operand facets, history cross-mount
continue, test_builders_declare_only_dispatcher_params ratchet; history
ls/tree/stat/find and dify find/cat wrappers ride the generics.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
26 builders forward **flags wholesale; each generic parses once into a
frozen struct via a spec-bound FlagView (ls/du/find/sed/tar/tree + 20
mechanical). rm/rmdir/ln/touch are builder-is-implementation and read
the bag through FlagView inline, mirroring the TS builders.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
operands.py + dir-aware stat/stream seam; cat/head/tail/wc/checksum5/
cut/expand/unexpand/fmt/fold/nl/rev/tac/strings/md5/zcat generics own
split_readable + flags; 16 builders and 12 bespoke wrappers become wiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(runtime): name the default python engine (#609 item 19 / T1-N)
The default world was an opaque positional triple on both sides
(`("monty", "quickjs", vfs)` in Python, `['pyodide','quickjs','vfs']`
in TypeScript), so the single fact the two implementations disagree on
— which python engine a default world registers — was readable only by
diffing two literals in two languages.
Lift it to a named registration, `DEFAULT_PYTHON`, on each side, and
derive `DEFAULT_ENTRIES` from it. Python stays on monty, TypeScript
stays on pyodide: no behavior change, but the divergence is now one
greppable constant carrying twin comments that name the reason
(`@pydantic/monty` cannot answer builtin `open()` calls yet, while
`pydantic-monty` can) and point at each other.
Also drops the second hardcoded copy of the same fact in the TS
`Runtimes` constructor, where `options.python` was forwarded on a
literal `name === 'pyodide'` test.
Recorded as an executable pin rather than prose: the new
`integ/runtime/defaults.json` reads the split back out of a live
default world on each host — `python3 -m nosuchmodule` is refused by
name on Python (`-m is not supported by the 'monty' runtime`) and
answered by CPython's own "No module named" on TypeScript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): pin that the -m refusal is byte-identical on both hosts
The defaults case shows Python's default world refusing `python3 -m`
and TypeScript's answering it, which reads like a message divergence.
It is not: it is purely which engine each default world registers.
Pin the actual parity claim ungated, so it is checked on both hosts:
run monty explicitly on each and assert the same bytes, including the
label normalization that makes the `python` alias report as `python3`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(runtime): one verb casing on the TS bridge
* feat(runtime): create and truncate join the TS bridge, quickjs open dispatches the real op
* refactor(runtime): one fopen-mode parser, chmod parser renamed parse_chmod
* feat(runtime): python readdir answers resolved entries like the TS bridge
* style: formatter output and lint fixes
* refactor(ops): WorkspaceFS becomes Ops in ops/ops.ts with the ledger on it
* feat(ops): union the facade surface across languages
* style: prettier reformat in fuse core
* test(integ): guest op-mode battery + ledger pins; record create/truncate in direct cores, rebind the recorder across the hop
* fix(ops): exclusivity outranks truncate for wx, and only a confirmed absence establishes in the quickjs shim
* fix(integ): cli.sh seeds nested parents and skips facade steps as sdk-only
CI caught this on hf and hf-prefix, identically on both language hosts:
expected "/data/cdp/lnk\n/data/cdp/t\n", got "/\n"
stderr: "cd: /lnk: No such file or directory"
The case created `/data/cdp/t` with `mkdir -p` and then looked inside it.
hf's `stat` short-circuits on a cached parent listing and raises ENOENT
without consulting the backend, and creating a nested directory does not
invalidate the ancestor that gained a child -- so the `$CDPATH` candidate
was rejected and `cd` fell through to the bare `/lnk`.
Reuses the symlink the setup case already makes, the way cdmode.json was
rebuilt onto `disptree/d` for the same reason. `CDPATH=/data` with the
operand `lkq` exercises the identical path -- the announcement is the
selected spelling, the cwd is the target -- without creating anything.
The hf caching bug itself is untouched and deserves its own issue; this
is the second fixture it has forced a rewrite of.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two behaviours from the `set -P` commit that its own integ cases missed.
A `$CDPATH` hit under `-P` announces the spelling it selected and lands
on the target, so the printed path and the resulting cwd disagree --
the same split as `cd -P -`, reached by a different route.
`set -o` applying everything named before the name it rejects is pinned
through pipefail's own effect rather than by reading the option back:
`set -o pipefail -o bogus` exits 2, and the pipeline after it still
reports its failing left-hand side.
Kept out of the `$PWD` commit so dropping that one leaves this in place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`$PWD` was answered by the expander before it reached the environment,
so the three things bash lets you do to it all silently did nothing:
PWD=/clobber; echo "$PWD" GNU /clobber mirage the cwd
unset PWD; echo "$PWD" GNU (empty) mirage the cwd
env | grep ^PWD GNU PWD=... mirage nothing
`$OLDPWD` was already a real variable written by `cd`, and `$HOME` reads
the env, which left `$PWD` the only one faked. It is now seeded at
session construction (bash exports it from startup) and re-stated by
every `cd`, and the special case in the expander is gone -- the fix is a
deletion, not a branch.
Two couplings are load-bearing and pinned:
* `$OLDPWD` is a straight copy of `$PWD` as it stands, not of the
shell's own record. After `PWD=/clobber; cd /data` a following
`cd -` tries /clobber and fails; after `unset PWD; cd /data` it is
empty. `cd` then re-states `$PWD`, so it always repairs itself.
* The shell does NOT read `$PWD` back when deciding where to go.
Clobbering it and running `cd ..` from /data/lk still lands on
/data, so `logical_cwd` stays the shell's own record.
`fork` carries `$PWD` onto a caller-supplied cwd for the same reason it
drops the logical name there.
Consequences worth knowing: `export -p`, `env` and bare `set` now list
`PWD`, and a CLI subprocess inherits it -- both of which is what bash
does, and both of which reach past `cd`, so this is kept as its own
commit.
Pinned against bash 5.2 on debian:stable-slim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four fixes to the logical-cwd work, two from review and two from a sweep
for what the design did not reach.
`fork` now drops the logical name when a caller supplies `cwd`. An
`execute(cwd=...)` call forks with the cwd overridden but inherited the
persistent session's logical spelling, so `pwd`, `$PWD` and a logical
`cd ..` inside that call all described the directory the session used to
be in while operands resolved against the requested one. Deciding it
inside `fork` rather than at the call site is what keeps the next caller
from forgetting; TypeScript needed it spelled out because `??` cannot
choose `undefined`.
`cd -P` announces the path as selected, not the one it resolves to. GNU
prints the spelling and lands on the target, and the two deliberately
disagree:
cd /tmp/lk; cd /tmp/other; cd -P - -> prints /tmp/lk, pwd /tmp/deep/real
CDPATH=/opt/c cd -P lnk -> prints /opt/c/lnk, pwd /opt/c/t
Overwriting `logical` under -P had collapsed both into the target.
`set -P` / `set -o physical` now work. They are the session-wide version
of the flag this branch added, and GNU applies them to `cd` and `pwd`
alike -- but `P` named no option, so it was dropped as an unknown cluster
letter, and while `set -o physical` did store the option nothing read it.
Both were silent, exit 0. `pwd -L` under `set -P` reports the physical
path for free: no logical name is ever recorded, so the pair collapses.
`set -o` rejects a name bash does not have, with GNU's `set: NAME:
invalid option name` and exit 2, keeping the settings named before it and
dropping the rest of the line. Accepting anything is what let
`set -o physical` look supported while doing nothing.
Pinned against bash 5.2 on debian:stable-slim, including the full 27-name
`set -o` table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shell kept one name for the working directory. bash keeps two, and
every surface that reports a cwd picks one of them:
cd /data/lk pwd = /data/lk pwd -P = /data/deep/real
cd /data/lk; cd .. pwd = /data (logical parent)
cd /data/lk; ls .. real (physical parent)
Not just a spelling difference: after following a symlink, `cd ..` and
`ls ..` land in different directories, and bash is internally
inconsistent on purpose -- its `cd` uses the name it remembers, its `ls`
asks the kernel. mirage was self-consistently physical, which is tidier
and not what bash does. It is also why `pwd -L`/`-P` were declared in the
spec and silently ignored: with one path there is no second answer.
`Session.cwd` keeps meaning physical -- it is what every operand resolves
against, so the ~58 readers outside the shell are untouched -- and gains
a `logical_cwd`, None whenever it would be equal. That collapsed
representation is the whole trick: a session that never walks a symlink
carries no new state, and `-P` collapses the pair rather than tracking a
second path forever.
One writer, four readers. `change_dir` maintains the pair (and stores the
logical name in $OLDPWD, which is what `cd -` returns to); `pwd`, `$PWD`
and `cd`'s own join read it. Under -L a relative operand joins the
logical name, under -P the physical one -- that is the `cd -P ..` row.
`set_cwd` is the seam for the four callers that move a session from
outside the shell (snapshot restore, session-store handoff, the
`workspace.cwd` setter). They have no typed spelling, so they drop the
logical name rather than leave it describing where the session used to
be, and they leave $OLDPWD alone because no `cd` ran.
pwd also grows the flag handling it always claimed: -L/-P last-wins
(`pwd -L -P` is physical) sharing cd's splitter, `pwd -x` exiting 2 with
GNU's usage line, and operands ignored. bash never re-validates the
logical name -- removing the link it was spelled through leaves `pwd`
still printing it -- so nothing here checks it either.
Every row pinned in GNU bash 5.2 (debian:stable-slim) and asserted in
both languages: 16 e2e rows each, plus 9 integ cases across 22 targets.
Not persisted across a session-store round trip: `to_dict`/`toJSON` is
the cross-language wire and a restored session collapses to physical,
which is the honest answer for a cwd nobody typed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings, both confirmed against GNU tar 1.35 on
debian:stable-slim.
An operand that cannot be read still owes its notice. `scan.missing`
short-circuited before the prefix was collected, so
`tar -cf out.tar sub/../missing` reported only the stat failure. GNU
announces the prefix first and then fails:
tar: Removing leading `sub/../' from member names
tar: sub/../missing: Cannot stat: No such file or directory
The prefix now comes from the operand's own spelling, before the scan's
problems are reported, so it survives an operand with no entries.
And the notices belong in operand order, not at the front. Collecting
them and prepending put a later operand's notice ahead of an earlier
operand's error; GNU emits diagnostics as it walks:
missing ../file -> Cannot stat missing then notice `../'
../file missing -> notice `../' then Cannot stat missing
`_announce_prefix` / `announcePrefix` now emits in place on first sight
of each distinct prefix, so per-name deduplication is kept without the
reordering.
The TypeScript side kept the old global prepend alongside the new
in-place emission for a moment, which doubled every notice -- caught by
the existing "warns once" assertion before it left the branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keeping an absolute operand's typed spelling (the commit before this)
reached further than cd: tar builds member names from `raw_path`, and
`member_name` only removed a leading slash, so `tar -cf out.tar
/data/sub/../file` began storing `data/sub/../file` -- a name that
extracts somewhere other than where it says.
GNU refuses to store any name that could climb out of the extraction
directory. It drops everything through the *last* `..` segment and falls
back to the leading slash only when there is no `..`:
/data/sub/../file -> file notice `/data/sub/../'
x/../y/f3 -> y/f3 notice `x/../'
../../file -> file notice `../../'
/data/file -> data/file notice `/'
./file -> ./file no notice
A `.` climbs nowhere, so it survives. Stripping is per name rather than
per operand, which is why `tar -cf a.tar ..` owes two notices: `..` for
the directory itself and `../` for everything under it. An operand that
is all traversal leaves nothing to name and GNU stores it as `./`.
The notice now names the prefix it removed instead of always saying `/',
and each distinct prefix is reported once.
Info-ZIP makes the opposite choice -- `zip sub/../g.txt` stores
`sub/../g.txt` verbatim -- so zip_cmd deliberately does not share this,
and an integ case pins that the two stay different.
All rows pinned against GNU tar 1.35 and Info-ZIP on debian:stable-slim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mkdir -p /data/lkd/real` created a directory whose only child was another
directory, and hf could not see it. Its stat short-circuits on a cached
parent listing:
parent_listing = await index.list_dir(parent)
if parent_listing.entries is not None:
raise enoent(path) # never consults the backend
/data's listing was already cached without `lkd`, and writing a file two
levels down did not invalidate the ancestor that gained a new child, so
`cd -P /data/lkp/..` got ENOENT on hf while `-L` passed by landing on the
mount root. Adding a file to the fixture did not help, for the same
reason -- the stale listing is consulted before any of it matters.
Point the link at `disptree/d` instead. It is already in the `files/v1`
fixture that all 22 targets mount at /data, so the case now creates
nothing but the symlink: no mkdir, no backend asked to make an empty
directory real, no cached listing asked to learn a new child.
The ancestor-invalidation gap is a real hf bug and is untouched here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GNU echoes a path operand back exactly as the user spelled it. Pinned in
debian:stable-slim:
du -b /k/ -> 7 /k/
find /k/guides/ -> /k/guides/
ls /k/nope/ -> ls: cannot access '/k/nope/': ...
md5sum /k/guides/../guides/a -> <hash> /k/guides/../guides/a
tree /k/guides/ -> /k/guides/
tree -f /k/guides/ -> /k/guides (-f is the exception)
Nine cases captured the pre-fix spelling, where the classifier normalized
an absolute operand before any command saw it and the trailing slash was
gone by the time du/find/ls/tree echoed it. Every one of these commands
takes a trailing-slash operand, so all nine move to the GNU spelling.
Also give the cd-mode fixture a file. `mkdir -p /data/lkd/real` leaves an
empty directory, which backends with no empty-directory concept (hf) do
not materialize, so `cd -P /data/lkp/..` could not reach /data/lkd there
while `-L` passed by landing on /data. Writing f.txt makes both parents
real on every target, which keeps all 22 rather than dropping hf.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`seed` waited for the query API on 16686 and then pushed spans to the
OTLP receiver on 4318. Those are separate ports on the same container,
bound at different times, so the push could meet a docker-proxy socket
that accepts the connection and resets it -- exactly the failure `query`
already documents and guards against for its own port. It killed
integ-shared-py on PR #763's first run, during setup, before a single
case executed:
ConnectionResetError: [Errno 104] Connection reset by peer
jaeger_seed.py:186 in post_spans
Retry the push itself, on the same POLL_ATTEMPTS/POLL_DELAY cadence, so
the readiness check is the endpoint that has to be ready. Catch OSError
for the reason `query` gives -- the reset arrives raw, past urllib -- and
raise with the last error when the receiver never accepts, rather than
failing confusingly further along.
Verified against a real `jaegertracing/jaeger:latest` container from a
cold start, and against a dead port (fails loudly after the full window).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`cd -P` was unreachable. `_resolve_target`/`resolveTarget` implement it
correctly -- given `/link/..` with `/link -> /deep/real` they return
`/deep` for -P and `/` for -L -- but nothing could reach the physical
branch, so the two modes returned the same answer for every input.
Two layers each normalized the `..` away before `cd` saw it:
- `_cd_candidates` built every candidate with `resolve_path`, which
normpaths. Join without normalizing instead and let `_resolve_target`
normalize, which it already does for both modes; when the workspace has
no symlink table at all, `handle_cd` normalizes the candidate itself.
- the operand arrived already collapsed. `heuristic.py`'s absolute branch
built its PathSpec with `virtual=normpath(word)` and no `raw_path`, so
`raw_path` defaulted to the normalized form and the typed spelling was
gone before any command ran. Pass `raw_path=word`, which is what the
field is documented to hold and what `relative_spec` has always done for
the relative case; `cd` reads it through `_typed_path`.
GNU bash 5.2, pinned in debian:stable-slim with /link -> /deep/real:
cd -L /link/.. PWD=/ cd -P /link/.. PWD=/deep
cd -L /link/sub/.. PWD=/link cd -P /link/sub/.. PWD=/deep/real
All four now match, and `cd` lands in the directory bash lands in for
every pinned case. One divergence is left, and it is a separate design
question: bash's -L keeps the *logical* name in `$PWD` (`cd -L /link`
leaves PWD=/link), while mirage resolves and stores the physical path.
That needs a logical cwd on the session, which would change what every
command resolves against -- and is why `pwd -L`/`-P` are declared in the
spec and ignored today.
Verified: full battery on ram and disk, 4755 cases, 0 failures on each
side, and `parity.py` diffs 4755 rows with 0 divergences. New
`integ/unix/sym/cdmode.json` is what caught that the first cut of this
fix worked in a unit test and not through the real pipeline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(errors): one condition vocabulary, one classifier per language
A condition is named once (FsCondition) and every boundary keeps only
a table from that name to its own number: posix, wasi preview1, and
the monty guest view. CycleError gets its ELOOP seat (was EIO at every
kernel and guest boundary), the wasi wire's EXDEV is 75 per wasi-libc
(18 is EDOM there), and the cross-mount-rename-is-ENOENT decision
moves from an adapter arm into the wasi table row.
* fix(stat): one stat view per language, offset-less stamps read as UTC
mtime, dir-ness, content size and the mode bits live once per language
(utils/stat_view), delegating to the existing naive-stamp-is-UTC
parsers. Three of the four translators (node FUSE, the runtime bridge,
py wasm) read an offset-less backend stamp as local time, so python
FUSE and node FUSE disagreed by the host's UTC offset for the same
stamp; all four now answer the same epoch, pinned by tests.
* refactor(errors): move the wasi and cpython tables into their runtimes
* refactor(errors): classifier arm tables into constants modules
* fix(errors): address review on the ValueError arm and epoch-zero mtime
* fix(himalaya): file the sender's own copy of a sent message
`--send` opened one SMTP conversation and stopped, so mail was delivered
but the sender's Sent mailbox stayed empty. A Sent folder is not produced
by sending: SMTP keeps no record of itself, and the copy is a second,
separate IMAP APPEND that every mail client makes on its own. Four call
sites, not one: `message send` plus the route compose, reply and forward
share.
Upstream v2.0.0 spells that APPEND as `--save <MAILBOX>` on all four
verbs, having replaced 1.x's automatic `message.send.save-copy` config.
Both are here, because they answer different needs:
- `--save <MAILBOX>` is v2 parity. On its own, with no `--send`, it files
the message without sending it, which is how a draft is written.
- `save_copy` is an account-level default, on, and is mirage's own
divergence: an agent that never learned the flag still leaves the
record a human sender would. `sent_folder` pins the mailbox.
The mailbox is asked for rather than guessed. A server implementing RFC
6154 tags one mailbox \Sent in its LIST reply, which is `[Gmail]/Sent
Mail` on Gmail and `Sent Items` on Exchange; failing that the configured
name, failing that `Sent`. This is ahead of upstream, whose v2 wizard
notes IMAP pins the reserved INBOX alone while it waits on LIST RETURN
(SPECIAL-USE) in io-imap.
Failure splits on whether a send already happened. A copy that fails
after a successful send is a warning on stderr with exit 0, because the
message is already gone and a non-zero exit invites a retry that sends it
twice. A `--save` that sends nothing fails loudly, because nothing
happened yet and retrying is safe. Upstream propagates in both cases.
Two bugs found on the way:
- aioimaplib joins command arguments with spaces verbatim and quotes
nothing, so an unquoted mailbox holding a space arrives as two
arguments and the server reads only the first word. That breaks on
exactly the two providers above. `quote_mailbox` now covers the APPEND
and the pre-existing `select_folder`, which had the same latent bug.
imapflow needs none of it, building commands from typed attributes.
- The TypeScript test CONFIG was an untyped literal, so a new required
field did not fail tsc and those tests passed vacuously through the
`!saveCopy` early return. It is annotated now, and a global beforeEach
stubs the accessor so no --send test reaches for a real connection.
GreenMail hands a new account nothing but an INBOX, where a real provider
ships a sent mailbox already made, so both integ seeders create one.
Verified against a live GreenMail on both hosts: the email and
cli-himalaya targets pass 86 and 74 cases including six new ones, and
compose, reply, forward and send each land in Sent with \Seen. Python
suite and TypeScript node suite green, pre-commit clean.
* fix(email): parse a LIST mailbox as an astring, not as a quoted string
Codex review on #761. A LIST mailbox is an astring, so a name needing
no quoting may legally arrive bare, and some servers emit it that way.
Splitting the line on quotes then reads the hierarchy delimiter as the
name: `(\HasNoChildren \Sent) "/" Sent` answered `/`.
That splitter predates this branch and already mis-listed every folder
on such a server, but this branch is what made it consequential: the
name now feeds an APPEND, so a sent copy would have been filed into a
mailbox called `/`.
The three tokens are walked in order instead. The atom form and the NIL
delimiter both parse, and a quoted name is now unescaped rather than cut
at the first inner quote, which the splitter also got wrong.
Verified against a live GreenMail on both hosts, whose own wire format
is the empty-attribute `() "." "INBOX"` shape.
* fix(integ/gws): resolve quoted worksheet names and implement updateCells
Two fidelity gaps in the fake Google Workspace server, both found by
driving a real client at it.
`parseA1` stripped the quotes around a sheet name only on the `!` branch,
so `'Jun-Jul_2025'!A1:G9` resolved but a bare `'Jun-Jul_2025'` did not.
That is not a corner: gspread quotes unconditionally
(`utils.quote_sheet_name`) and sends the bare quoted name to read a whole
worksheet, so `Worksheet.get_all_records()` answered
`Unable to parse range` against any sheet at all. Quoting is now handled
before the `!` split, in `splitRange`, with doubled apostrophes unescaped
the way the real grammar spells an embedded quote -- and, because the
name is consumed first, a `!` inside a quoted name no longer confuses
`lastIndexOf`.
`updateCells` was unimplemented and answered `Unsupported request`. It is
how a caller shortens a sheet it previously wrote longer -- the real
request clears whatever `fields` names across the range that `rows` does
not cover -- and there is no other way to do that through batchUpdate.
Both forms are served: `range` (clear, then write what is supplied) and
`start` (write at a coordinate). Only `userEnteredValue` is kept, since
the fake stores strings and formatting has nowhere to go; a number
renders as its decimal, a bool as TRUE/FALSE, and a formula as its text,
which is consistent with the fake not evaluating formulas.
Three cases in integ/cli/gws.json cover them, each self-contained on a
spreadsheet it creates. All three fail on the unfixed server -- the read
with `Unable to parse range`, the two writes with
`Unsupported request: updateCells` -- and the target goes 65 to 68 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(integ/gws): read a whole-column range like A:Z as cells, not a tab name
`parseA1` decided an unquoted range with no `!` was a tab name unless it
matched `^[A-Z]+\d`, which wants a digit right after the column letters.
`A:Z` has none, so it resolved as a sheet called "A:Z", found nothing, and
came back `Unable to parse range: A:Z`.
That is the range gspread asks for when a caller wants every row of the
first worksheet -- `spreadsheets().values().get(range='A:Z')` is the
ordinary way to read a sheet whose length you do not know -- so a whole
family of readers could not read the fake at all.
The test is now a single pattern covering every A1 form the real API takes
with a half open side: `A1`, `A1:G9`, a whole column span `A:Z`, and a
whole row span `1:5`. A name a tab actually has still wins first, as
before, and the old `includes(' ')` special case is subsumed -- a name with
a space cannot match. `parseCell` already answered `{row: null}` for a
bare column and the range builder already read that as unbounded, so
nothing downstream changed.
One more case in integ/cli/gws.json; the target goes 68 to 69 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(integ/gws): honor the updateCells field mask before clearing values
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the two grep_helper suites produced a ten-name import, and the
two formatters disagree about how to wrap one: yapf wants a hanging
indent, isort wants the names aligned under the paren. Each undoes the
other on every run, so `pre-commit run --all-files` reported both hooks
as "files were modified" forever while the tree kept returning to the
same bytes.
Pin yapf's form and take the import out of isort's hands, which is what
the tree already does at `tests/commands/builtin/generic/test_split.py`
and eight other sites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`control.py` (537 lines) and `jobs.py` (233) had no test on the Python
side while `control.test.ts` and `jobs.test.ts` sit beside their sources;
the only suite references were two `ReturnSignal` imports. Port both,
then cover the three `builtins/` modules neither tree tested at all.
- `test_control.py` ports the TypeScript assertions for if/for/while/
until/case, and adds what only Python has a twin for: the `for ((;;))`
handler, a multi-level `break` decrementing on its way out, and the
`_MAX_WHILE` cap emitting its stderr warning.
- `test_jobs.py` ports wait/kill/jobs/ps and adds `handle_fg`, which has
no TypeScript twin: the no-job message, the unknown id echoed as typed,
and the command-line header prepended to the adopted stdout.
- `test_history.py` pins `_parse_args`, whose bash-getopt corners were
documented in a docstring and asserted nowhere: `-d3` attached, `-dc`
reading "c" as the offset rather than another option, `history -1` as
an invalid option, `--` and the first operand both ending option
parsing, and a bare `-` as an operand.
- `test_dirs.py` covers `handle_cd`: CDPATH search order and the
announce-the-destination rule, the mount-root fallback for a path the
backend cannot stat, symlink following through a prefix and a chain,
and ELOOP.
- `test_scope.py` covers the two PathSpec helpers.
Also move `test_find_action_dispatch.py` beside its source, which lives
at `executor/find_action_dispatch.py`, not `workspace/`.
Not covered on purpose: `cd -P`. `_resolve_target` implements it
correctly -- called directly, `/link/..` gives `/deep` under `-P` and `/`
under `-L` -- but nothing can reach that branch, because every candidate
comes from `_cd_candidates` -> `resolve_path`, which normpaths `..` away
first, so `-L` and `-P` return the same answer for every input. The
TypeScript twin is structurally identical (`resolvePath` then
`resolveTarget`), so this is a shared divergence from bash rather than a
parity gap. Fixing it is a behavior change to `cd`, out of scope here;
asserting today's answer would pin the bug.
Mirror baseline 211 -> 207.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
isort folds the two same-module imports the merge left behind into one
and drops the duplicated `compile_pattern`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit's cleanup only holds if something refuses the next
drift, and the project already enforces comparable rules with
source-walking meta-tests (`test_no_raw_flag_reads.py`,
`test_no_dead_flag_params.py`). The absence of one here is why the
`__init__.py` count reached 87 and why a 24-file directory could sit
beside 19 packages it did not name.
Three hard checks and one ratchet:
- no `__init__.py` anywhere under `tests/`;
- every test directory mirrors a `mirage/` package, or is one of eight
entries in `UNMIRRORED_DIRS` with a written reason -- harnesses and
fixture trees (`commands/native`, `e2e`, `fixtures`, ...) that mirror
nothing on purpose;
- no stale exemption: an entry whose directory is gone, or whose source
twin now exists, fails instead of lingering;
- a ratchet on the number of source modules no test file is named for,
moving in either direction only deliberately. It counts the weaker of
the two readings -- `test_<name>.py` anywhere rather than beside its
twin -- because CLAUDE.md asks for a mirror only "where reasonable"
and several suites cover a family in one file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tests/` is a namespace tree -- pyproject already sets
`--import-mode=importlib`, which resolves the same-basename collisions
(34 different `test_find.py`) an `__init__.py` would otherwise be needed
for -- so CLAUDE.md forbids them. Without a check the count reached 87,
all of them content-free: 81 carried only the license header, 6 were
empty. Delete them; `tests/commands/postgres/` held nothing else and
goes with them.
Then move the three directories that mirrored no source package, because
that is what hid two coverage gaps:
- `tests/resource/object_storage/` collected 24 suites for 19 sibling
packages, so `mirage/resource/s3/s3.py` looked untested. Split it one
directory per source package, the five s3 suites into `tests/resource/s3/`,
and the cross-alias prefix suite up to `tests/resource/` beside
`test_s3_aliases.py` -- renamed to name its subject, since `test_prefix.py`
says nothing at that level.
- `tests/commands/builtin/helpers/` shadowed the flat `*_helper.py` sources.
Its conftest was a byte-identical copy of its parent's, so it just goes.
Three of its five suites collided with a same-named flat suite and were
merged: they test disjoint halves of one helper (`number_flag_error` vs
`parse_counts`; the `sort` command vs `parse_keydef`/`build_config`;
`grep_lines`/`grep_recursive` vs `compile_pattern`/`search_query`). The
sort merge renames a local `sort_lines` wrapper to `_run_sort`, which
otherwise shadows the `sort_helper` import of the same name.
- `tests/commands/ssh/` sat beside `commands/builtin/ssh/`'s twin instead
of under it.
Also flatten the ten single-file `{ram,redis}/{cat,head,ls,tail,wc}/`
directories onto their already-flat siblings, and move `test_man.py` under
`executor/builtins/` where `man.py` lives.
Collection is unchanged: 13863 tests before, 13863 after the moves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A directory present in only one implementation was recorded as a single
finding and its module set discarded, so the 21 one-sided directories
counted 21 rather than the 74 modules they hold. That made the ratchet
blind exactly where drift is easiest: a new module under a directory
already in the baseline -- `agents/mcp`, `ops/postgres`,
`core/onedrive` -- left the count untouched and `--strict` passed.
A one-sided directory now contributes every module inside it. Adding a
module under `agents/mcp` moves the count 310 -> 311 where it used to
sit at 257 forever. Directories become presentational: they are printed
once with their module count and excluded from `total()`, so they are
not double-counted against the modules they now contribute.
An excused directory stays a deliberate blind spot, and keeps excusing
its whole subtree. The excuse is precisely that there is no counterpart
to mirror -- agno is python-only, opfs is browser-only -- so a new
module inside one is expected growth, not drift. Verified both ways: a
probe under `agents/mcp` (unexcused) moves the count, a probe under
`core/opfs` (excused) does not, and stale-entry detection still fires.
Baseline 257 -> 310. The number is larger because the unit is now
consistently one module; no divergence was added.
Reported by codex on #759.
The #370 sweep added `compareCodePoints` to cli/src/config.ts and
cli/src/settings.ts, the first time anything under packages/cli/src
imported @struktoai/mirage-core. The package never declared that
dependency, and tsup externalizes only what package.json declares -- so
rather than leave it an import, tsup inlined the whole core barrel into
the CLI bundle, dragging isomorphic-git's CJS entry along with it.
esbuild rewrites a CJS `require()` it cannot resolve into a shim that
throws, so every `mirage` invocation died at module load with
`Dynamic require of "buffer" is not supported`.
Nothing local caught it: tsc resolves the import through the pnpm
workspace link and the build succeeds. Only the bundled binary breaks,
which is why it surfaced as 89 integ-runtime failures, the cli e2e
suite, and a cross-language snapshot job whose python client could not
reach a daemon that never started.
Declaring the dependency restores externalization: the CLI dist now
carries a plain `from "@struktoai/mirage-core"` and no isomorphic-git.
The unrelated follow-redirects hunk in the lockfile is pnpm's own
peer-dep dedup, emitted by the same install.
CLAUDE.md asks for the two trees to keep a mirrored layout and nothing
checked it, so seven helper modules went missing and two module homes
swapped without anything noticing. scripts/check_layout_parity.py diffs
the module-name sets directory by directory: core/node/browser union onto
one python namespace because CLAUDE.md puts runtime-specific code in
node/browser, while cli, server and agents are separate npm packages
whose python twins are subpackages, so they map by prefix.
Two normalizations keep the report honest. __init__.py and index.ts are
skipped, since only python requires a package file per directory and
comparing them reports a gap across most of the tree that nobody intends
to close. Module and directory names fold camelCase, hyphens and a
leading underscore, so findEval.ts against find_eval.py reads as a
rename rather than two missing modules -- 15 renames separated out from
207 genuinely absent modules that way.
--strict does not demand zero. 257 divergences predate the gate and each
needs its own decision, so it fails when the count moves off the
committed baseline in *either* direction: new drift is blocked, and
closing a divergence has to be locked in by lowering the number rather
than being silently spent. A stale exception fails just as loudly, which
is the failure mode every hand-maintained allowlist in this repo has
already hit.
The report reproduces five findings the audit had filed separately:
findEval/findParse as the only camelCase modules, _provision.ts vs
provision.ts, workspace/snapshot missing keys and types on the python
side, no builtins/shared.ts, and node's cache/redis/file.ts sitting where
python has cache/file/redis.py. That is what it is for -- items 31 to 37
get scoped from its output instead of from taste.
Also gates scripts/gen_width_table.py against its committed output, the
same way the spec trees are gated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent bugs, one shared table.
`-L` is the length of the longest line in *columns*, not characters, and
neither tree measured columns. Pinned against coreutils 9.7:
`printf 'a\tb' | wc -L` is 9, because a tab jumps to the next multiple of
8 -- both trees said 3. `printf 'a\rb' | wc -L` is 1, because carriage
return rewinds the cursor without ending the line: Python counted CR as a
column and said 3, TypeScript split on /\r?\n/ and reached 1 by treating
CR as a line break, which is a different answer that happens to match on
this input. A CJK character is 2 columns, a combining mark is 0, and a
control character is 0 (that is how wc accounts for wcwidth's -1).
`-w` splits on glibc's iswspace, which is Unicode White_Space *minus*
U+0085. Both trees approximated it with a builtin and both were wrong, in
opposite directions: Python's str.isspace() also reports U+001C-U+001F
and U+0085, so `printf 'a\x1cb' | wc -w` counted 2 words; JavaScript's
\s also matches U+FEFF, so `printf 'ab' | wc -w` counted 2. The
plan named Python correct on this row; it is not.
The width data now comes from scripts/gen_width_table.py, which walks
unicodedata and emits both trees plus integ/fixtures/wc/width.json. The
fixture is the contract, the same way integ/fixtures/filetype/tables.json
is: regenerating one tree without the other fails both suites. The
generated tables live under utils/generated/, exempted from yapf and
isort here and already covered by typescript/.prettierignore, because a
formatter that repacks 349 ranges fights the generator on every run.
Cf is not uniformly zero-width, which is the one place raw Unicode
categories would have been wrong: U+200B and U+FEFF measure 0 while
U+00AD, U+0600-U+0605 and U+110BD measure 1. Documented divergence: an
unassigned code point measures 1 here and 0 in glibc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JavaScript orders strings by UTF-16 code unit. An astral character
(U+10000 and up) is a surrogate pair in 0xD800-0xDFFF, so it sorts before
every BMP name from U+E000 up, while Python's `sorted` puts it after.
`ls` on a directory holding a U+E000 name and an emoji listed them in
opposite orders across the two trees. GNU agrees with Python here, in
both C and C.UTF-8: UTF-8 byte order *is* code-point order, so
TypeScript was the only one of the three disagreeing.
Adds utils/sort.ts (compareCodePoints, sortedByCodePoints) and routes
every string sort through it. Three classes of site, only the first of
which was obvious:
- 131 bare `.sort()` calls across 94 files.
- ~40 explicit `a < b ? -1 : a > b ? 1 : 0` comparators, which are the
same code-unit comparison wearing a comparator's clothes.
core/gdrive/readdir.ts even carried the comment "Codepoint compare, not
localeCompare: python sorts by codepoint" directly above one.
- 3 localeCompare calls, whose Python twins are all plain code-point
sorts. ICU collation reorders ASCII too, so those diverged on far more
than astral names.
An integ pin on `ls` and `find` is what caught generic/ls.ts, which the
bare-`.sort()` sweep could not see because it already passed a
comparator. eslint now rejects a zero-argument `.sort()` outright (it is
also wrong for numbers and tuples, which it compares as strings) so the
default cannot come back; typecheck caught the four sites where the
sweep had reached numbers and bigints.
Two latent bugs fixed on the way: git/reset.ts sorted [path, letter]
tuples by their `toString()`, ordering `a+b` before `a` because ',' has
a higher code point than '+'; sort_helper.ts's compareLines is the
`sort` command itself, so the C-locale byte order it should have had is
now what it computes.
Closes#370
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(ops): one door for VFS/FUSE ops in both languages
Ops (python) and WorkspaceFS (typescript) now delegate every op to
Dispatcher.dispatch, so FUSE and programmatic access walk the same
pipeline as a shell command: link follow, session grants, admission
policies, cache read-through, namespace structure and write
invalidation each fire once. The facade keeps only the typed surface
and op recording.
- PolicyDenied.completed distinguishes a pre-deny from a post-deny, so
a completed-then-denied op is still recorded before it propagates.
- A cross-mount rename is refused at the facade with EXDEV, so mv falls
back to copy+unlink. TypeScript classifies FUSE errnos by code
string, so EXDEV had to join the table or it degraded to EIO.
- TypeScript's MountCore was a split surface (reads via ws.fs, writes
via ws.dispatch), so drainOps reported no writes at all. Every
mutation now goes through the facade.
- Retired the forwardIndex knob so TypeScript forwards the index to
every backend the way python always has.
- Deleted the workspace-less Ops pipeline: dispatch is now required.
It was a second door with different behavior (no cache, no namespace
structure, gates only when a caller passed policies), and only tests
reached it.
- Ops.read gains raw, mirroring readFile(path, {raw: true}). FUSE's
read-modify-write hands its merged buffer to write, which stores, so
the read that feeds it must be the stored bytes. Raw reads also skip
the file cache, which is keyed on the path alone and holds whatever a
command rendered.
python 13185 passed, typescript core 7232 and node 2282.
* fix(ops): slice a warm cache for ranged reads, record denied read bytes
Two review findings from codex on the collapse.
A ranged read served from the file cache returned the whole object
instead of the window: the cache holds the full file, and the warm
path handed it back untouched. The dispatcher now slices it with the
same helper the cold path falls back to, so warm and cold agree. This
was already reachable on main through git, which reads pack indexes as
a few bytes at a known offset, and routing Ops.read(offset, size)
through the same door widened it.
A post_ops deny on a read recorded zero bytes, because the suppressed
result is the only place a read's byte count lived (a write's is still
in its own arguments). PolicyDenied now carries completed_bytes,
stamped where it is raised, so records and network_bytes stop
under-reporting traffic that actually happened.
* fix(ops): keep transfer bytes and cache origin through the op door
Two more review findings from codex, both cases where the deny and
limit paths lose a fact the success path has.
A post_ops Limit truncates what the caller receives, but the transfer
already happened, so recording the capped length under-reported
network_bytes by whatever the cap removed. IOResult gains op_bytes,
which the door sets only when a limit truncated a read, and the facade
prefers it over the delivered length.
A post_ops deny on a warm cache hit was recorded against the backend,
so a read served entirely from the file cache counted as network
traffic that never happened. PolicyDenied gains from_cache, stamped
where the warm branch raises, and the facade reads it exactly as the
success path reads a cache-served result.
* fix(ops): a namespace answer is not the parent backend's op
A directory that exists only because a mount or a link sits below it is
served from the namespace without contacting anything, but the facade
attributed it to the mount that lexically owns the path. On a remote
parent that invented a network record for every such lookup, which a
session granted only a nested mount hits on every walk down to its
grant.
The door now names the server explicitly through IOResult.op_source
rather than leaving the facade to infer it from a non-empty reads map:
"ram" for a warm cache hit and for a synthetic namespace answer, unset
when the owning mount served it. That replaces the inference, so the
two cases the door already knew about are now stated rather than
guessed.
The backend keeps the record when it was actually reached and answered
ENOENT, which is a real round trip on a remote mount.
* fix(ops): errors raised after an op ran carry its source and bytes
* style: match CI yapf on the door-error test
* fix(ops): stamp an OpReport at completion, drain drift at the door
Both trees resolved tr's SET operands with a reader built for `echo -e`.
Python's was a regex over eight named escapes, so `tr '\0' -` and every
octal spelling silently matched nothing. TypeScript's was echo's scanner
outright: it read `\x41` as `A`, stopped the whole scan at `\c`, and only
took octal behind a leading zero.
TypeScript did not drift there by accident. workspace/executor/escapes.test.ts
is a port of Python's *echo* test that imported tr's reader, so every
echo-only rule was pinned against the wrong command — the test was holding
the bug in place. It now imports interpretEchoEscapes, which text.ts
exports for it the way builtins/__init__.py already re-exports
_interpret_escapes.
tr's grammar, pinned against coreutils 9.7 (debian:stable-slim): octal is
\NNN with no leading zero required, greedy to three digits, so \0141 is
\014 then a literal 1; a three-digit value over 255 backs off to two
digits (GNU also warns, which a pure reader has no channel for); an
unknown escape drops its backslash, so `tr '\x41' -` deletes x, 4 and 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: GNU byte pins across tail, split, numfmt, sed, cp and find (#609 T1-F)
Six of the eight T1-F rows, each re-pinned against debian:stable-slim
(coreutils 9.7, sed 4.9) before it was touched. Four of them were wrong
in both languages, not one, and two were larger than the audit recorded.
tail -c takes the same sign grammar as -n. TypeScript did
`raw.slice(-bytesMode)`, so `-c -3` dropped the FIRST three bytes;
Python's `abs(c)` turned `-c +3` into the last three. Both now route -c
through one shared parse_counts/parseCounts, which is what the three
hand-written call sites had drifted apart from.
split -t is one byte or the two-character spelling \0. TypeScript took
`ENC.encode(sep)[0]` and Python kept the whole byte string, so `-t XY`
split on 'X' in one tree and on 'XY' in the other where GNU refuses to
run. An empty value is now a report, not a silent newline.
numfmt: --to=none renders in fixed notation from an exact value, so
`--from=si 1Y` prints twenty-five digits instead of TypeScript's `1e+24`
-- and `--from=iec 1Q` at all, which the 28-digit default decimal
context could not hold. The suffix grammar is now GNU's: one unit
letter, lowercase only for kilo, with the trailing i required under
iec-i, optional under auto and refused elsewhere. `1KiB` used to read as
a kilobyte in both trees. Errors carry GNU's four message shapes and
exit 2 instead of raising decimal.InvalidOperation.
sed's missing-script and no-input paths had four spellings across the
two languages, and Python raised where TypeScript returned exit 1. Both
now share two constants, return an IOResult, and use GNU's `no input
files` with its exit 4. GNU's usage block is deliberately not
reproduced.
cp -rv ordered sibling directories by length alone, which is
PYTHONHASHSEED-dependent for equal-length names; both sides key on
(len, path) now.
find -mtime parsed timestamps inline on both sides instead of using the
helper each tree already had: Python stamped UTC over a real reported
offset and let a malformed value raise out of the walk, while
TypeScript's bare Date.parse produced a NaN that slipped past the null
guard and KEPT the entry Python dropped.
Deliberate divergences, documented in place: GNU's quotearg escaping in
the split message (matching the existing truncate choice), and the long
double GNU reads numfmt input into, which prints 1.10 as 1.11 while 1.20
and 1.30 round-trip.
Verified: 4695 integ cases pass on ram and disk in both hosts, py+ts
unit suites green, pnpm -r typecheck clean, spec parity 93/93.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(tail): read a TailCounts field through ?? null, not !== null (codex)
Codex flagged the tailBytes signature change as a public API break.
Backward compatibility is explicitly out of scope here (CLAUDE.md), and
the described failure was wrong -- `tailBytes(data, 2)` does not fall
back to ten lines, it returns the whole input, because `(2).fromByte` is
undefined, `undefined !== null` is true, and `slice(NaN)` is `slice(0)`.
But that is a real hole and it is worth closing on its own: TailCounts
is a plain object, so any field a caller omits arrives as undefined and
a bare `!== null` guard waves it into the wrong branch. Python's twin is
a dataclass whose four fields all default to None and cannot do this.
`normalizeCounts` gives the TypeScript side the same floor, and both
`tailBytes` and the cache gate in the generic read through it.
Four tests pin it, including the exact shape codex described.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(shell): run `sh FILE` / `bash FILE` as a script, not as the program text
handle_bash treated the first non-flag operand as the program text, so
`sh /work/run.sh` executed the literal string `/work/run.sh` and reported
`/work/run.sh: command not found`.
- read the operand through the op dispatcher and run its contents
- positional parameters from the operands, and from `-c 'prog' name a b`
- `$0` is the script (new Session.script_name, read by `$0` and `${@:0}`)
- `-e -u -x -f` were silent no-ops; map them through SET_FLAG_TO_OPTION
and restore afterwards
- GNU diagnostics: missing 127, unreadable 126, directory 126, prefixed
with the head word so `sh` no longer says `bash:`
* refactor(shell): state the bash/sh startup rules once instead of case by case
The first commit fixed the reported case; this one replaces the case
lists it left behind with the rule each one was standing in for. Every
change below closes a divergence pinned against GNU bash 5.2 on
debian:stable-slim.
- The startup option grammar is `set`'s grammar, so both now read it from
one place (`shell/options`). `-o name` was parsed only to be thrown
away, which silently dropped `bash -o pipefail` even though pipefail is
implemented; `+x` / `+o` were not option words at all, so `bash +x f`
looked for a file called `+x`; and the options were a list of names to
enable, which cannot express `bash -e +e` (last one wins). `set -eo
pipefail` was broken by the same gap and is fixed by the same change.
- bash's long options carry arity now, not just names: `--rcfile FILE`
consumes FILE, and an unknown `--long` is a usage error rather than a
script path (`bash --version` used to report a missing file).
- The program comes from stdin whenever no operand names one. `-s` is one
case of that rule, not the rule, so `echo prog | bash` ran nothing and
`bash -s A B` ran the file `A`.
- One reader for a script file, shared by `source` and a nested shell,
refining a keyed backend's ENOENT for a directory into EISDIR. `source`
reported a directory as missing, spelled the path resolved rather than
as typed, and had a third copy of the payload drain in TypeScript.
Bare `source` now gives GNU's usage error instead of statting the cwd.
- The exit code comes from the errno: 127 only for ENOENT, 126 for
everything else. It was `PermissionError ? 126 : 127` in Python and
`EACCES` in TypeScript, so ENOTDIR was reported as 127 in both.
- One child-shell snapshot, shared by `( … )` and a nested `bash`/`sh`.
The nested shell isolated three hand-picked fields, so a `cd`, an
`export` or a function definition inside `bash -c` leaked into the
caller; the subshell's own list had never been told about `script_name`
and neither had Python's `Session.fork`, which is a hand-written
literal whose docstring promised it could not happen. The field list is
declared once and a test fails until every dataclass field is
classified. `source` deliberately does not snapshot: a sourced `set -x`
does leak in bash.
- `$0` reads one `Session.argv0` accessor; Python's `or` treated an empty
`-c` name as absent where TypeScript's `??` did not.
* fix(integ): keep the cwd case self-contained, and pin symlinked script operands
Two follow-ups from review.
`ctl_sh_script_file_cwd_isolated` changed the session's working directory
and never changed it back. The battery shares one session per target, so
that re-rooted every relative operand in every later case. It runs in a
subshell now, which restores the directory and exercises the other half
of the same snapshot.
Review also asked whether a script operand that is a namespace symlink
reaches the backend unresolved, since the backend cannot see a link the
namespace owns. It does not: the operand resolves before dispatch, and
absolute, relative and cross-mount links all run their target while a
broken one reports the link at exit 127. Pinned in both languages and in
the battery, since nothing covered it.
* fix(shell): start a child shell outside every source its caller is inside
`session.source_depth` was transient, so the child-shell snapshot neither
reset nor restored it. Running `bash FILE` from inside a sourced script
therefore left the depth positive, and a top-level `return` in the child
was absorbed as if the child were itself sourced: GNU prints "can only
`return' from a function or sourced script", keeps going and reports 0,
where mirage ended the child early with the returned status. The field
joins CHILD_SHELL_FIELDS so the same snapshot that already restores it
carries it, and the nested shell sets it to 0 the way it sets `$0`.
`OptionWord` also moves to `shell/types.py` / `shell/types.ts`, next to
the SET_FLAG_TO_OPTION table it is the shape of, per the module layout.
* fix(traversal): cross a nested mount correctly in du, find, ls -R and tree
A mount nested inside another mount's tree is invisible to the parent
backend's readdir, so the parent's keys under that path are shadowed.
Six bugs followed from that, all fixed here in both languages.
- du counted the shadowed keys. The generic now opts into MountView and
prunes leaves under a descendant mount after the link merge, then
recomputes the total from the survivors.
- du -c printed one total row per mount. The bytes-level re-total is
extracted as merge_du_totals so the executor fan-out reuses the same
code, with -h humanizing once at the merge instead of per block.
- Fan-out sub-runs never received LinkView, so symlinks vanished from
find/du/tree/grep -r exactly when a nested mount existed.
- A line whose operands span mounts never fanned out inside an operand,
because the cross-mount runner executes one single-mount native run
per operand. run_with_fanout wraps that seam; it forwards resolve_hint
as well as stdin, which the STREAM strategy needs.
- ls -R was filtered a line at a time. Its format is PATH: plus bare
names, so the path-first filter dropped the shadowed header and kept
its entries, and ate the blank separator. It is filtered a group at a
time now.
- tree is no longer fanned out at all: its output is one document, so a
per-mount block printed a second root, drawing and summary. It crosses
inside the generic through a new injected fact, readdir_path, a
dispatcher-backed listing offered beside stat_path.
Pinned against GNU coreutils 9.7 and tree 2.2.1 on debian:stable-slim.
* fix(traversal): fold nested du into ancestors, and stop tree disclosing an ungranted mount
Three review findings, all reproduced before fixing.
tree crossed a boundary from the mount table alone: a crossing entry's
row is synthesized as a directory without asking any backend, so the
dispatcher never saw it and could not refuse it. `tree /base` drew and
counted an ungranted `/base/private` while ls, find and du all hid it.
The child-mount lookup is now session-filtered, the way ls filters the
same fact. du keeps the raw list on purpose: an ungranted mount still
shadows the parent's keys.
du concatenated per-mount blocks, so every ancestor row carried only the
parent backend's own bytes. Pinned on coreutils 9.7 over a real tmpfs
mount, GNU folds: `du base` prints `7 base/inner` then `17 base`, and
`du -s base` prints one row. The blocks are now merged at the leaf level
and the tree derived by the same rollup a single-mount run uses, so
folding, post-order, --max-depth and -a come from one implementation.
rollup grows a `dirs` hint so an empty mount keeps GNU's `0` row, and
the mount roots are stat'ed rather than assumed to be directories,
because /.bash_history is a mount serving a single file.
tree's cross-mount stat threw a bare Error, which isWalkError does not
recognize, so a vanished entry rejected the whole run instead of being
skipped. It throws a stamped enoent now, matching python's
FileNotFoundError.
The three nested integ truths encoded the pre-fix output: ls -R had no
blank line between groups (GNU 9.7 prints one) and tree showed two roots
and two summaries (real tree 2.2.1 over a mount prints one document).
* feat(gws): add calendar and forms API passthrough, and mock routes
Adds `gws calendar` (calendarList, calendars, events, freebusy) and
`gws forms` (forms, responses) in both languages, plus routeCalendar
and routeForms in the integ GWS mock.
* feat(gcal): mount Google Calendar as day directories
Adds the gcal resource: one directory per calendar, one per local day,
one JSON file per event, named <eventId>__<HHMM-HHMM>_<Title>.gcal.json.
rm deletes the event; everything else goes through `gws calendar`.
Day bucketing uses one mount-wide zone, defaulting to the primary
calendar's, with bounds computed as consecutive local midnights so a DST
day is 23 or 25 hours rather than a fixed 24.
TypeScript mirror still to come, so spec parity is red until it lands.
* test(integ): cover gws calendar and forms, fix the service count
Adds a calendar/v1 fixture seeded through events.insert on both hosts,
and 12 cases: overlap querying, all-day end-exclusivity, q search,
freebusy, delete, form create/batchUpdate, and that a form's id is
reachable through Drive (the Forms API has no list method, so that is
the only path an agent has to an existing form).
gws --help now lists 8 services, not 6.
* fix(gcal): address codex review
- percent-encode Discovery path params, so a holiday calendar id's "#"
no longer truncates the URL at /calendars/en.usa (py + ts)
- resolve a zone-less dateTime through its declared timeZone instead of
leaving it naive, which raised TypeError when bucketed (py + mock)
- register commands and ops at module scope; there is no cycle
- validate a day is a real date, not merely date-shaped, so 2026-02-30
is ENOENT rather than a directory readdir then fails on
- centre the default window in the bucket zone, not the host's
* test(integ): cover calendar and forms mutations, fix the stale service count
* feat(gcal): mirror the calendar mount in typescript
* fix(gcal): drop the redundant close override and the polynomial regexes
* fix(notion): add GET /v1/comments, version the database object, align ntn API errors
The fake had no GET /v1/comments, only POST, so nothing could read a
comment back. Comments were also seeded with one global position while
createComment numbers by the parent's count, so a new comment could sort
ahead of seeded ones; seeding is now per parent.
databaseJson was version-blind and always answered 2025-09-03. A caller
sending Notion-Version: 2022-06-28 now gets the column schema under
properties. The MCP arm keeps the default: a tool call carries no version
header and must render identically to the REST arm.
ntn reported every API failure through the executor's generic fallback
(ntn <verb>: <message>, exit 1), dropping the status, reason phrase and
Notion's code. failure.py / failure.ts render upstream's wording, probed
against ntn 0.21.9, and every leaf is registered wrapped so a new verb
cannot miss it. datasources query now reports the data source miss with
upstream's hint instead of the database fallback's own 404.
* fix(ntn): narrow the guarded result before destructuring it
CommandFnResult is a nullable union, so annotating the awaited value as a
pair failed tsc --noEmit. The test now narrows instead: guarded returning
null would mean it swallowed the failure, which is worth failing on.
Three review findings, all of them cases where TypeScript accepted a
config Python refuses — or built something other than what was asked.
**The s3 store group was exempt from key validation.** The exemption
generalized `mounts.*.config`, which Python types as a bare dict, to
every credential-carrying block. That does not hold: an s3 group is
`S3StoreBlock(S3Config)` with extra="forbid", a strict model, so
`endpoint_urll` is a load error in Python and silently ignored here.
It now has a key table like every other block; only the camelizing is
skipped, which is all the backend's own spellings ever needed.
**`store.namespace`/`store.observer` accepted `type: s3`.** Both
languages took the config and failed later, at plane construction, with
`S3WorkspaceStateStore` raising for the planes it does not host. Both
now refuse it at the loader, with the message the store used to raise.
**A built cache store passed as `options.cache` became a RAM cache.**
Every `CacheConfig` field is optional, so a store is structurally
assignable to it; the workspace then built a cache of its own and the
supplied store never saw a read, a write, or a close. Structural typing
cannot refuse this, so `buildFileCache` says so instead. The test that
covered this path only asserted construction succeeded, which is why it
kept passing — it now asserts the rejection.
The two loader rules are pinned in the shared fixture, so both languages
must agree; each was proven red before the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ownership was spelled two different ways, and TypeScript was missing
the half Python has.
Python names ownership: `Workspace(store=..., owns_store=True)`, set by
the config loader and the daemon route beside the store they build.
TypeScript had no such knob — `owned = options.store === undefined` —
so a store built from a `store:` block, or the daemon's own disk
default, was always treated as borrowed and never closed. For redis or
s3 that is a client that is never quit, once per workspace the daemon
creates. Adds `ownsStore` and sets it at both build sites.
The cache had the mirror-image problem: `options.cache` accepted a
built store *or* a config and inferred ownership by duck-typing on
`.get`, which is neither Python's model nor a rule anything else here
follows. It now takes config only, as Python does, so the workspace
always builds the cache and always closes it — `resolveFileCache` and
`ownsCache` go away with it. A store core cannot build still reaches
`buildFileCache` through `registerFileCacheStore`, the seam node's
redis store already uses.
`WorkspaceArgs.options` was a hand-copied subset of `WorkspaceOptions`;
it is now the type itself. That list is what once dropped `clis` and
`guards` between the config and the daemon.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Self-review pass over the PR, for special cases that should have been
general rules and for shapes that stopped earning their keep:
- The s3 exemption was spelled `block.type === 's3'` in two places
(skip-camelize, skip-key-check) that encode one rule: a group whose
type names a backend carries the backend's config, not ours. Named
once as BACKEND_CONFIG_TYPES; a second such backend now joins a set
instead of needing both sites found again.
- `buildCache` had decayed to an identity cast once the workspace took
over building the store, and RamCacheBlock/RedisCacheBlock duplicated
core's CacheConfig/RedisCacheConfig field for field. Both deleted;
WorkspaceArgs.options.cache narrows to what it now actually carries.
- buildCliEntries kept its own copy of the cli key list this PR had just
added as CLI_KEYS; the store group names were spelled three times.
- absolutizeScripts: isPlainObject guards for the casts it grew, and a
comment about running before or after normalization that only one of
the two is now true.
The key tables are copied by hand from Python's pydantic models, so a
field added there would be refused here until someone hit it. Pinned the
way rejected.json pins the other direction: integ/fixtures/config/
accepted.json exercises every key of every block and both suites read
it. Proved it bites by dropping `root` from STORE_KEYS — the store case
fails with `unknown store key \`root\``.
Docs showed TypeScript constructing a RedisFileCacheStore beside a
declarative `index:`, which is the divergence this PR removed; the
snippet is now the config form, matching its own index line and Python.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two P1 review findings, both real, both introduced by this PR's own new
surfaces.
**Close declaratively constructed cache stores.** `closeWorkspace`
cleared the cache but never closed it — harmless while the only store it
could build was RAM, wrong the moment `cache: {type: redis}` became
expressible, since `clear()` itself connects and nothing then quits the
client. Python's `close_async` pairs them in a try/finally; TS now does
the same, behind an `ownsCache` flag so a store the caller built stays
theirs to close (the split `ownsStateStore` already draws).
That exposed a pre-existing leak one layer up: `configToWorkspaceArgs`
*built* the store and handed it over as `options.cache`, leaving the
daemon's redis client with no owner at shutdown. The loader now passes
the block through as a `CacheConfig` — which is what it already was —
so the workspace builds, owns and closes it.
**Normalize s3 store configuration.** An s3 store group IS an S3Config,
whose snake_case spellings do not camelize into the TS field names:
`aws_access_key_id` is `accessKeyId`, not `awsAccessKeyId`, and
`timeout` is seconds where `timeoutMs` is milliseconds. Casting the
camelized block silently dropped every credential and the endpoint, so a
configured store would have authenticated against the wrong service.
Fixed on both sides of the seam: `normalizeConfigKeys` now leaves an s3
group alone (as it already does for `mounts.*.config`, for exactly this
reason), and `buildStoreGroup` runs it through `normalizeS3Config` —
the same translation the s3 mount uses.
Both regressions are pinned by tests proven to fail without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(runtime): make the embedded python literals raw so TS stops eating their escapes
A template literal processes backslash escapes before Python sees the
text, so a lone \n inside an embedded Python string became a real
newline and truncated the literal. The symptom was remote from the
cause: every pyodide run died with pyodide_fatal_error, print(42)
included.
String.raw makes the escapes mean what they say; the emitted programs
are byte-identical, which is how the conversion was checked. The new
test compiles each wrapper on the interpreter that runs it, so a future
break reports a named SyntaxError with a line number instead. Reading
the .ts and un-escaping it by hand is not a substitute; that models the
escaping wrong and reports a false pass.
* fix(shell): splice a quoted "$@" into its word instead of replacing it
A quoted string holding a bare "$@" was replaced wholesale by the
positional parameters, so every literal in the word was discarded:
`f() { echo "saw: $@"; }; f a b` printed `a b`. Both languages.
The braced `${a[@]}` splat already spliced correctly, so `$@` and
`${@}` now take that path rather than carrying rules of their own,
which also fixes `${@}` not word-splitting at all and `${@/x/y}` acting
on the joined value. Three consequences worth naming:
- an empty splat that is the whole word yields no word, where it used
to yield an empty one (`x "${empty[@]}" y` is two words, not three)
- a slice numbers the parameters from 1, so `${@:0}` yields $0 first
- `${#@}` is untouched and still wrong (it measures the joined string);
it is a different code path and stays for its own change
Pinned against bash 5.2.37 in debian:stable-slim, not the macOS bash
3.2, which drops $0 from `${@:0}` and would have pinned the wrong rule.
* fix(shell): keep a splat whose single element is empty
`fragments == [""]` decided "no word" from the rendered text, but zero
elements and one empty element render the same, so `set -- ""` passed no
argument where bash passes one empty one. `${@}` and `${arr[@]}` too.
Track whether a splat yielded elements instead of inferring it.
The word survives iff the joined text is non-empty OR some splat yielded
at least one element. An empty expansion beside it does not count: with
no parameters, "$u$@" is no word at all, even though "$u" alone is an
empty word. All 16 combinations pinned on bash 5.2.37 in
debian:stable-slim.
The TypeScript CLI loaded the YAML, camelized it, and POSTed the
result; the daemon then ran `loadWorkspaceConfig` on that. With the
loader now refusing keys Python refuses, its own output no longer
round-tripped — `default_session_id` arrived as `defaultSessionId` and
was rejected.
Camelizing before sending was the actual defect: it made the wire
format differ from Python's, whose CLI sends `model_dump()` (snake_case)
and whose daemon validates the same spelling. Splits the loader into
`checkWorkspaceConfig{,File}` — interpolate, validate, resolve script
paths, and stop — from the camelizing `loadWorkspaceConfig{,File}`. The
CLI sends the checked mapping; the daemon does the single authoritative
load. Env interpolation stays client-side so a missing var still fails
before the round trip.
Caught by the cli-gate and cross-language interop jobs, which are the
only callers that cross that boundary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two Tier 1 items, both cases of a contract Python enforces and
TypeScript does not.
T1-M — core lifecycle
- `RedisIndexCacheStore.close()` had zero callers, so a mount
configured `index: {type: redis}` kept its client open past
`closeWorkspace` and the Node process never exited. `BaseResource`
now owns a `close()` that releases the index behind a `#closed`
guard, mirroring `resource/base.py`. The 29 hand-written
`close() { return Promise.resolve() }` overrides are deleted (they
existed only to satisfy the interface, and each one silently opted
its backend out of the teardown); the seven with real work call
`super.close()`. `noImplicitOverride` makes forgetting it a compile
error.
- `installDriftState` fired `void cache.remove(path)` per entry from a
sync function, so under `DriftPolicy.OFF` a read issued right after
`fromState` could still be served the snapshot bytes OFF exists to
bypass — and a rejected remove was an unhandled rejection. Adds the
sync `FileCache.evictPaths` seam Python has had (`evict_paths`):
implemented in the RAM store, a documented no-op in redis.
- `DiskObserverStore.walk` swallowed every readdir failure, so an
EACCES mid-tree yielded a silently partial `/.bash_history` and a
`clear()` that reported success while leaving files. Errors now
propagate; only a missing root reads as "nothing recorded yet".
Python's root probe used `os.path.isdir`, which has the same blind
spot, so it moves to the same explicit check. Adds
`ObserverStoreBase` (twin of Python's) so `readAll`/`close` stop
being copy-pasted into three stores.
T1-L — TS workspace-config validation and cache-as-config
- Python sets `extra="forbid"` on 13 config blocks; TS validated
unknown keys in two places and copied everything else through, so
`mount_point:`, `consistancy:` and `limitt:` were hard errors in one
language and silently ignored in the other. Adds per-block
allowed-key tables for the top level, mounts, cache, index, store
(+ groups), clis and guards, checked before normalization and
against Python's canonical snake_case spellings — a camelCase alias
TS would accept for free is just as unportable as a typo. Guards now
fail at load, as in Python. The local `snakeToCamel` (which diverged
from core's on uppercase-after-underscore and trailing underscore)
is replaced by the exported one.
- `store: {type: disk}` silently built a RAM state store, so state a
user believed was persisted was not; disk and s3 groups are now
wired, matching `_build_state_store`.
- `WorkspaceOptions.cache` took a built store where `index` takes a
config, so a programmatic TS consumer could not ask for a redis file
cache declaratively. Adds `core/workspace/workspace/cache.ts`
(`buildFileCache`, twin of `workspace/workspace/cache.py`) plus a
`registerFileCacheStore` seam — core cannot import node, so the
redis store registers itself the way runtimes already do — and
widens the option to `FileCache | CacheConfig`.
Verified with a shared `integ/fixtures/config/rejected.json` both
suites assert against, so a config accepted by one loader and refused
by the other fails a test until they agree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two call sites outside python/ that the first pass missed, both required
CI jobs:
- `runners/python/adapters.py` built its github mount with
`GitHubResource(GitHubConfig(...))`, which now raises TypeError against
the tree-taking constructor before a single case runs — the whole python
github battery was unusable. `GitHubService.resource` and `build_github`
are async; `build_mounts` awaits whatever the builder table returns, so
the other forty builders stay plain sync functions.
- `integ/root.py` indexed and splatted the coroutine from
`to_workspace_kwargs`, failing the virtual-root job with
`TypeError: 'coroutine' object is not subscriptable`.
Also corrects two comments (py adapters + the ts twin) that justified the
out-of-process github fake by "GitHubResource fetches the repo tree with a
blocking urlopen from its constructor". That is no longer true; it stays
external because both hosts share the one fake.
Verified locally against the CI fake (`--port 5098`, same --repo args):
github 20/20, github_ci 11/11, ram 2332/2332, disk 2297/2297,
ram-alias 8/8 (the alias_of branch), and `integ/root.py` all checks OK.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`GitHubResource.__init__` fetched the repo's default branch and its
recursive git tree before returning, over `urllib.request.urlopen` — the
only blocking HTTP call in the package. A constructor cannot await, which
is the entire reason that sync client existed.
It is reachable on a live event loop: `server/routers/workspaces.py`'s
`async def load_workspace` calls `build_resource` synchronously, so
mounting a github repo froze the daemon's loop for two GitHub round trips
— every other mount's in-flight I/O and the FUSE queue stalled with it.
Measured against a deliberately slow local GitHub, a 1.24s build let a
50ms heartbeat coroutine tick once; it now ticks 24 times over the same
1.24s.
TypeScript already had the answer, but not the one the audit recorded:
`GitHubResource.open()` there is a no-op, and the work lives in
`static async create` behind a private constructor, with the whole
`ResourceFactory` type declared `(config) => Promise<Resource>`. Mirror
that rather than the `open()` hook — a lazy hook would move the failure
from build time to first path resolution, which is a new divergence, not
a fix.
- `BaseResource.build`: async classmethod factory, default just calls the
constructor. Named `build`, not TypeScript's `create`, because `create`
is already an op name (make an empty file, what `touch` calls) and ops
are served by `__getattr__` — a real `create` on the class shadows every
backend's create op. Caught by the databricks_volume suite.
- `build_resource` is now async and awaits that factory through
`_instantiate`, which falls back to the plain constructor: registered
and entry-point resources need not subclass `BaseResource`.
- `WorkspaceConfig.to_workspace_kwargs` follows, matching TypeScript's
already-async `configToWorkspaceArgs`. `Workspace(**kwargs)` stays sync.
- `GitHubResource.__init__` takes the fetched tree and touches no network;
`GitHubResource.build` does the two fetches over the existing async
`github_get`. `github_get_sync`, `fetch_tree_sync` and
`fetch_default_branch_sync` are deleted.
- New `tests/resource/test_no_blocking_http.py` fails on any `urlopen`,
`urlretrieve` or `requests` import under `mirage/`, including
function-local ones.
- Docs updated on both sides; the TypeScript github pages showed
`new GitHubResource({...})`, which has never compiled against its
private constructor.
Snapshot load is untouched: github's state redacts its token, so
`requires_resource_override` always sends it down the live-override path
and never reaches `_construct_resource`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DELETE /v1/blocks/{id} is the only delete verb the public API has and the
only one the MCP tool surface exposes, but the fake had no route for it
and `ntn api` had no DELETE method, so nothing could remove anything.
`archived` is upstream's deprecated alias for `in_trash` and always
returns the same value; the fake stored two columns, so a PATCH of
`archived` left the row in database queries while `ntn pages trash`
(in_trash) worked. Now one stored bit, both spellings on the wire.
A database row's cells now ride in its page.json under `properties`.
Also: trashing a child page moved only the page row, leaving the
child_page block in the parent listing; `datasources query` took its
TSV columns from the schema where upstream takes them from the returned
rows; and two MCP-parity cases pointed at pre-data-source paths, so they
compared two identical errors and asserted nothing.
Docs: both ntn.mdx files documented the pre-#740 grammar (--page flags,
ntn blocks/comments/search). notion.mdx, the TS setup doc and both
examples predated the data-source split; the examples were broken.
* fix(ids): stamp uuid7 from the clock, not from the previous id
uuid6's uuid7 orders ids inside one millisecond by bumping the
timestamp past the previous one. The borrowed value becomes the next
call's floor, so the error compounds: a burst of 5000 ids stamps every
later one 4991 ms into the future, and it stays there until the clock
catches up. These are workspace and session ids, used as time-ordered
database keys.
Mint the id here instead, with RFC 9562's dedicated counter in rand_a:
48-bit timestamp, 12-bit intra-millisecond counter, 62 random bits. The
timestamp is only ever a real clock reading, and an exhausted
millisecond waits for the next rather than borrowing.
Drops the uuid6 dependency, which nothing else used. TypeScript already
had this right, via the uuid package's counter.
* feat(python3): parse CPython's option table, and seed sys.path on pyodide
python3's spec carried one option, -c, and a free-text rest, so every
other switch fell through to the operand slot: 'python3 -u s.py' ran
'/-u' as the script and failed, 'python3 -zz -c ...' exited 0 with the
flag swallowed, and -V and -h were read as paths.
Two spec knobs fix the class rather than the command. stop_at_operand
turns off the parser's dash-operand leniency and ends option parsing at
the first operand, which is CPython's own rule; ends_options marks the
options that carry a program (-c, -m), so the words after them are the
program's argv rather than python3's. Both mirror in TypeScript.
On top of that: -m runs through runpy, with a find_spec probe so a
missing module is CPython's one line instead of a traceback; the four
source doors each report the argv[0] CPython gives them; and the
interpreter-init switches ride in RunArgs.flags, honored where the
engine can and reported on stderr where it cannot.
argv[0] needed one more piece. The local and wasi tiers hand CPython
the program through -c, so it hardcoded argv[0] to '-c' and named every
frame '<string>'. bootstrap() re-compiles under the real name, which
fixes argv[0] and the traceback filename together.
Pyodide gains sysPath, which glob-expands and prepends mount paths once
the mounts are in place, so a vendored wheel imports without the
sys.path.append incantation. A glob that matches nothing is reported on
the run's stderr; the seed cannot report it itself, since it runs
before stderr is captured. packages, packageBaseUrl and lockFileURL
reach a prebuilt distribution, and the wrapper honors -O, -B, -W, -X,
-E and -I.
Verified against CPython 3.13.7: all 14 probed lines match on stdout,
stderr and exit code.
* refactor(spec): borrow argparse's REMAINDER instead of two bespoke parser knobs
python3's line needs opposite treatment of an unknown dash word on
either side of the script: `-zz` before it is python3's usage error,
`--foo` after it is data the script must receive. The first pass added
CommandSpec.stop_at_operand and Option.ends_options to say that, which
put two per-command dialect fields on dataclasses every command shares.
The first half is not ours to invent: it is argparse's
nargs=argparse.REMAINDER, and POSIX's own option order (GNU's permuting
default is the extension, which `POSIXLY_CORRECT=1 ls a -1` shows).
argparse spells it on the positional slot, so it goes on Operand under
argparse's name, and CommandSpec is untouched.
The second half argparse cannot express at all: `add_argument('-c')`
beside nargs=REMAINDER answers `unrecognized arguments: -u`, and CPython
parses its own command line in C for that reason. So it stays out of the
grammar. POSIX already spells the handoff `--`, and the parser already
consumes one, so a table in workspace/route inserts it after -c/-m's
value, beside the other command-name-keyed line rules. The marker is
added unconditionally: CPython passes a typed `--` through as data
(`python3 -c p -- -u` gives the program ['-c','--','-u']), so the parser
eats exactly the one added here.
Net: Option and CommandSpec return to what they were, Operand gains one
borrowed field. CLAUDE.md now states the rule as a literal test, since a
CLISpec is a CommandSpec and cannot be used to escape it.
* docs: require POSIX and argparse, not either, for a shared grammar field
POSIX alone specifies an option whose argument is a program (sh -c, and
python3's own synopsis), so an either-or rule would license the Option
field the section exists to refuse.
* fix(python3): address review on init switches, and declare the four missing ones
- pyodide restores warnings.filters after a run, so -W error no longer
follows every later line on the warm interpreter
- -OOO saturates at compile()'s max of 2 rather than raising ValueError
- -E/-I stop deleting PYTHON* from os.environ: CPython's -E stops those
variables configuring startup, it does not hide them from the program
- pyodide reports -E/-I/-s/-S through unhonoredNotice instead of faking
them; unhonored() grew a honored subset, and now reports -OO too
- declare -b, -P, -x and --check-hash-based-pycs, which CPython accepts
and mirage was refusing. -x is answered by the source resolver, since
handing it to an engine that runs code via -c would honor it nowhere
- end_options_after_program walks a short cluster letter by letter, so
-uc 'p' -v hands -v to the program, and steps over long value options
* fix(integ): argv[0] for a payload is -c on every engine, monty included
Monty's DEFAULT_PROG placeholder only applied because nothing ever told
it a name. The python3 command now supplies CPython's own answer, so
these four policy-routing steps read -c. They still distinguish monty
from local, since argv is a monty global and CPython would NameError.
Also picks up the formatting of the new flags test: it was untracked
when pre-commit ran, and --all-files goes through git ls-files.
* fix(python3): do not rewrite a shadowed name, and stop over-claiming -W and -X
- the `--` handoff is skipped when a shell function shadows python3.
bash's rule gives the function the line, and it has no CPython option
table to read a marker with. `command python3` masks the function for
its inner run, so the rewrite applies there again. A CLI cannot reach
this at all: register_cli refuses a shell builtin's name.
- an invalid -W filter is reported as CPython reports it and the program
still runs, instead of _OptionError escaping the wrapper and killing a
line every other runtime completes.
- a known -X name is reported as unhonored, since populating
sys._xoptions is all a warm interpreter can do for one. An arbitrary
name stays silent, which is all CPython does with it either.
- -O's reach is written down: the payload is compiled at the requested
level, a module imported from sys.path is not, and sys.flags stays 0.
`write_output` caught `OSError`, so a write that failed with anything
else aborted the whole command and the remaining operands were never
attempted — even in the default warn mode, where GNU keeps going. That
is every remote backend: `core/s3/write.py` forwards botocore's
`ClientError` out of `put_object` and `core/gridfs/write.py` pymongo's
`PyMongoError` out of `upload_from_stream`, and neither is an `OSError`.
The TypeScript twin catches whatever is thrown and classifies it, which
is why only python had the hole.
`error_line` already told the two apart (shared strerror for a
recognized filesystem refusal, the exception's own message otherwise),
so only the catch had to widen. `Exception`, not `BaseException`, so
cancellation still propagates. Not a swallow: every caught error is
named on stderr and the command exits 1.
The test sink now raises a non-OSError, mirroring the TypeScript sink's
plain `Error` — that alone turns four existing cases red on the old
catch — plus a case pinning the multi-operand diagnostic for an SDK
error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finishes T1-I, the `CommandIO` slots that are wired on one side only.
du (python gains it): `core/box/du/` was written and unreachable —
imported by its own `__init__` and its tests and nothing else — and
dropbox had no python du at all, so both took `DU_BUILDER`'s walk bounded
by `max_du_entries`. Past 10 000 entries that prints a partial total, a
stderr notice and exit 1. Ports dropbox's du beside box's and wires both,
matching the typescript tables.
find (typescript loses it): `core/box/find.ts` and `core/dropbox/find.ts`
were not pushdowns. Neither API has a search call to push to, so both
were the *same* readdir/stat walk the find and cp builders already run —
minus three things the builders pass and an op cannot:
- the mount's own index (they allocated a scratch `RAMIndexCacheStore`,
so every walk started cold and warmed nothing for the next command),
- the namespace stat overlay, because the native branch only applies it
when `ops.local === true` and both backends are remote,
- the symlink table `-empty` counts entries from.
The overlay half is a live divergence: `touch -d 2020-01-01 f` then
`find -mtime +5 f` finds it in python and does not in typescript. Pinned
by extending find/mtime.json's four mtprobe cases to box and dropbox;
they fail on pre-fix typescript on both targets and pass after.
The stale comment that justified the ops ("the cp builder may call find
without threading an index, unlike python") is refuted by cp.ts:47, which
reads `opts.index` and passes it to its own fallback walk.
Leaves box/dropbox with one command_io exception each (python range-reads,
typescript's core read takes no window) instead of three.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wiring `append` into python's ssh CommandIO makes the ops factory emit
an `append` op for ssh, which is what typescript has emitted all along
(`node/.../ssh/io.ts:55`). The expected table is a hand-kept literal, so
the new row has to be added by hand; the other three backends that wire
append (disk, ram, redis) already carry theirs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tee`'s spec declares `rest: {type: "path"}`, so the parser accepts N file
operands — and both generics wrote only `paths[0]` and silently dropped the
rest. `tee a b` put nothing in `b`. Identical in both languages, which is
exactly why no parity gate saw it: a shared bug is not a divergence.
GNU 9.7, pinned in docker: `printf x | tee a b c` writes all three; a failing
operand is diagnosed and skipped while the others still get the data (exit 1);
only `--output-error=exit` stops at the first failure. That last mode had never
been implemented because with one sink it was unobservable — the old comment
said as much — so it was parsed and ignored.
Deliberate divergence, documented at the call site: GNU opens every operand up
front, so under `exit` an *open* failure aborts before anything is written. A
mount has no open/write split, so operands before the failure are already
written. The two agree whenever the failure is at write time, which is what a
remote backend reports.
The `append` CommandIO slot is now wired into the generic, per the decision to
wire rather than delete it. It was declared on both adapters, wired by five
backends, and read by no builder, so `tee -a` was read-modify-write everywhere
— on ssh that meant downloading and re-uploading the whole file plus a
lost-update window. A backend with the slot now appends; the rest keep the
fallback. Where the native path runs, the resulting content is not in hand, so
the path is reported in `writes` but not in `cache`: that is how apply_io is
told to drop the stale entry instead of caching a wrong one. The slot was typed
`(...args: any[]) => unknown`, which is what let five backends wire something no
builder could consume; it is typed like `write` now.
Found while wiring it: python's `core/ssh/append.py` existed with no importer
outside the resource, so python ssh took the read-modify-write path its
typescript twin did not. Wired.
Two more real bugs on the way through:
- TS `tee -a missing-file` threw instead of creating the file. The not-found
test was `/not found/i.test(err.message)`, but `enoent()` puts the *path* in
the message, so it never matched — verified by running it. Python's
`except FileNotFoundError` was always correct. That broken test is why s3 and
gridfs had bespoke tee wrappers carrying an `exists()` pre-check (with a
`catch {}` that swallowed everything) to route around the generic. All four
wrappers — s3 and gridfs in both languages — are now wiring only, which fixes
their multi-operand bug in the same edit.
- `isEnoent` existed in three copies (`core/generic/find.ts`,
`workspace/reconcile.ts`, `executor/builtins/metadata.ts`) and in none of them
was it next to `isFsError`. One copy now, in `utils/errors.ts`.
Verified: three integ cases across 17 targets, each proven red on pre-fix code
and green after, on both hosts — the multi-operand pair fails as `DUP|` (first
operand only) and `tee -a` on a missing file fails TS-only. `xm_tee_multi`
passed all along because it is cross-mount: one operand per mount, so each
backend tee wrote its single `paths[0]`. Full core/node/browser vitest and the
python suite green; specs regenerate with no drift and the parity gate passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(notion): normalize written properties against the data source schema
The fake echoed the client's property object back, so a PATCH that omitted
`type` (which the API allows) stored an untyped value that its own reader
rendered blank. Fill id and type from the column and resolve a select name to
the whole option.
* fix(notion): persist a minted select option into the schema
The answer's option id was only usable once. A later write naming that id
alone missed the schema lookup and came back with an empty name, which is the
blank cell this PR is about. Add the option to its column when it is minted.
* fix(notion): queue a workspace's writes at the door
Two writes to one workspace interleave at any await, so both could read the
same schema and the later write would drop the other's minted option while the
page that minted it kept the id. Serialize every mutating request per
workspace; GET stays concurrent.
Resolves both codex findings on #742, plus the regression they framed and a
divergence found alongside it.
The presigned-fetch shim (core/s3/_client_browser.ts) reads only `Key` off a
GetObject input and drops `Range`, so #742's `readRange: rangeOf(s3Read)`
returned the *whole object* as the requested window — on all fifteen browser
S3-family resources (s3, r2, gcs, minio, ceph, backblaze, wasabi,
digitalocean, scaleway, supabase, aliyun, tencent, oci, qingstor, seaweedfs).
Before #742 the ops factory took the slice fallback and answered correctly.
`core/s3/stream.ts` now owns a `readRange` that branches: with the AWS SDK it
asks for one ranged GET as before, and on the presigned path it streams and
stops at `offset + size`. Adding a real Range header there is not the fix —
the header makes the request non-simple and trips a CORS preflight presigned
deployments generally do not allow — so the shim now *refuses* a Range it
cannot serve instead of answering wrongly. Stopping early only saves the bytes
if the transfer stops too: `bodyFromResponse` released the reader's lock
without cancelling, so the rest of the object arrived anyway.
Codex C2, unsatisfiable ranges: a window at or past EOF is an empty read under
POSIX and a 416 over HTTP, so wiring a native range changed the answer for the
same call. Normalized once per language in the ops factory rather than in each
reader, keyed on a shared `isUnsatisfiableRange` / `is_unsatisfiable_range`
that reads the botocore, aiohttp, httpx and aws-sdk-js error shapes. The node
S3 mock sliced silently instead of refusing, which would have let the fix look
correct without being tested; it now answers 416 like the real thing.
Codex C1, node vs browser: `merge_variants` resolves the two runtimes by
preferring node, which turned a divergence into a silently discarded value.
`check_variant_facts` now diffs the two variants for both resource-fact tables
*before* that merge, with its own stale-checked `variant_resource_facts`
exemption table. It found thirty real divergences: the browser S3 resource
declared neither `sizesAlwaysKnown` nor `storageId` where its node twin
declared both, so a browser `mv` between two mounts of one bucket saw two
separate storages. Fixed rather than exempted — the browser class gains both,
and the identity now comes from a shared `s3StorageId` so the runtimes cannot
compute different answers for one bucket. Zero exemptions needed.
Found alongside: TS `rangeRead`'s fourth argument is end-exclusive on ssh, hf,
nextcloud and databricks_volume, and in every python twin, but s3 and gridfs
read it as a length — `range_read(p, 10, 20)` returned twenty bytes from
offset ten where python returned ten. Both corrected. s3's inline Range
builder also now goes through the shared `rangeHeader`.
Verified: core/node/browser vitest and the full python suite green, both spec
trees regenerate with no drift, and the parity gate proven red on a planted
node/browser divergence and on a stale variant exemption.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(workspace): merge mount structure into the door's readdir and stat (R1)
* fix(structure): gate synthetic door answers, tighten bridge LIST errors, drop the flagged regexes
* test(integ): root ls truths carry the trailing newline the door renderer emits
* fix(structure): synthesize link ancestors like mount prefixes, feed one union to the door and the ls fact
* fix(structure): filter link names by owning-mount grant, serve granted structure under an ungranted parent
* fix(structure): ls lists structure-only dirs, nested-mount preload fails loud, fs.ts fallback gates with the synthetic prefix
* fix: fan-out gates on raw descendants, nested-mount rename raises EXDEV in pyodide
* refactor: rename ops/structure to ops/namespace_view, promote norm_dir into path utils
* fix: ls serves namespace-only directories under -R and -d, fan-out threads the child-mounts fact
* feat(notion): rebuild the fake on Prisma, migrate to data sources, align ntn to upstream
The fake Notion server is rebuilt on Prisma and its writes now land, so a
create is readable back through the mount and through the CLI. Slack's
writes never landed either; fixed the same way.
The client moves to Notion-Version 2025-09-03, the generation that split a
database into a container plus one or more data sources. The database object
no longer carries `properties`, so the mount nests to match: the schema and
the rows live under `databases/<Title>__<id>/<Name>__<ds-id>/`.
`ntn` is realigned to the real CLI verb for verb. `blocks`, `comments` and
`search` were mirage inventions and are gone; they are `ntn api v1/...`,
which subsumes the whole REST surface offline. Added `api`, `whoami`,
`auth token` and `datasources resolve`.
Every case in integ/cli/ntn.json is now asserted twice: once inside a mirage
workspace and once as the same shell line run by the real npm ntn binary
pointed at the same fake. A golden both agree on is by construction what the
official CLI prints. It caught five wrong goldens and one real fake bug
before they could be committed as correct.
* chore(spec): regenerate the spec snapshots for the four new fields
Operand gained `name` and `required`, Option gained `metavar` and `env`, and
all four serialize into every command's spec JSON. The TypeScript generator
had to learn to emit them too, or python and typescript specs diverge on all
93 commands.
* fix(ntn): address the codex review, and correct the fix it suggested
Four P2s, all real.
- TypeScript declared --filter-file and never read it, so a query with one
silently returned unfiltered rows. It now reads the path through the op
dispatcher, the way Python does and the way himalaya reads --attach.
- Both languages recognised `Header:Value` inputs and then discarded them.
Probed on the wire against the real binary: it sends them. Threaded through
as per-request headers in both.
- Malformed --data. Codex asked Python to converge on TypeScript's exit 2
usage error; probing the real binary shows neither was right, so both now
answer exit 1 with `error: Invalid JSON from --data`, and a conformance
case pins it.
- Env-backed options were filled after validation, so an option declaring
both `env` and `required` could not be satisfied by a populated variable.
The fill moved ahead of validation in both languages.
ntn api had an empty test file, which is why the first two survived. It now
covers classification precedence, headers, the body sources and the output
serializer.
* fix(ntn): satisfy CI's flake8 and the test-inclusive typecheck
E129 on a wrapped return annotation, and a ByteSource|null decode in the new
api test. Both are CI-only: the repo's flake8 is stricter than the local hook,
and `pnpm -r typecheck` covers test files that a bare -p tsconfig run skips.
* fix(ntn): match the real CLI on api body sources and JSON parse errors
`ntn api` diverged from the upstream binary in ways the conformance
harness could not see, because no case covered them.
- inline `path:=json` failures now carry serde_json's own message, from
a new scanner that also decides validity (python's json accepts NaN
and serde does not). Columns count bytes, newlines reset them, and
the recursion limit is 128.
- no object check on a body: `--data '[]'` is sent, not refused, and
any body source makes the call a POST, so `--data '{}'` no longer
falls back to GET on falsiness.
- stdin is a body source, and the three sources are validated in
upstream's order before the conflict is reported.
- refusals exit 1 for a malformed body and 5 for a line the CLI will
not interpret, both with upstream's hint lines.
- `name==value` survives a non-GET instead of being dropped.
- typescript gains PUT, which python already had.
Every byte is probed against ntn 0.21.9 and gated by 7 new conformance
cases, which the real binary is run through too.
* test(ntn): pin that the scanner never hands the engine parser a failure
`api` calls json.loads / JSON.parse only on input the serde scanner
accepted, so a document the scanner passes has to parse. A gap there
would put a raw parse error back on the command's error path, which is
what the scanner was introduced to remove. Checked over 8,145 generated
documents; the list here is the readable subset.
* fix(cli): answer group refusals in the CLI's own dialect, and let the environment through the parse
Three findings from the codex review.
- A group-level option refusal always exited 129 with git's wording,
even for a CLAP program whose own leaves exit 2. The style now picks
both: clap answers `error: unexpected argument '--x' found`, one usage
line and a footer, at every level of the tree. Probed against ntn
0.21.9 and pinned by three conformance cases.
- An option's declared environment variable was filled after the parse,
so an env-backed int went unvalidated, an env-backed choice untested
and an env-backed path stayed a raw string that FlagView.as_paths()
could not see. It now lands inside the parse, exactly where a default
does and just ahead of one, so it gets the same coercion and the same
required credit. apply_env is gone.
- `ntn pages edit --json` threw the PATCH response away and printed the
page id, on the one flag that asks for the response.
The clap renderer moved to the spec layer: the walk needs it and cannot
import a module that reaches the workspace.
Registry membership only ever said a backend could be *built*. What it
does once mounted — how stale a listing may be, whether reads are cached,
whether a `du` is pushed down to the API — was a second hand-maintained
surface with no gate at all, which is how python served up-to-ten-minute
stale listings of a live postgres schema while typescript pinned 0.
`spec/*/resources.json` now carries two more tables, diffed by
`check_spec_parity.py` with the same one-fact-per-exemption rule the
command checks use:
- `capabilities`: per registry name, `index_ttl` / `caches_reads` /
`supports_snapshot` / `sizes_always_known`, plus whether the class
overrides `storage_id` and `statfs`.
- `command_io`: per backend command package, the wired `CommandIO` slots
plus `local` / `max_glob_matches` / `max_du_entries`.
Python reads both off its classes and dataclasses. TypeScript reads them
from the source declarations (`scripts/resource_facts.ts`), because the
twins are instance fields and observing them at runtime would mean
constructing the resource — and construction is not inert:
`buildResource('github', {})` issues an HTTP request and `postgres` opens
a connection. A value the extractor cannot read as a literal is dumped as
`<expr:…>` rather than guessed.
The gate found 15 live divergences on its first run. Fixed here:
- `readRange` was wired on `disk` alone in typescript while python pushed
the window down on twelve backends, so `head -c 100` on a large object
downloaded the whole thing and sliced. Wired on the five whose read
already takes an `{offset, size}` window (s3, gridfs, nextcloud, hf,
databricks_volume) via a new `rangeOf` adapter, and `gen-specs` now
refuses to emit when a reader takes a window with no slot to hand it —
the twin of python's `test_read_range_optin.py`, which is why python
was fully wired and typescript was not. The check keys on `offset` *and*
`size`: postgres pairs `offset` with `limit` to mean SQL rows.
- python github ignored `SCOPE_ERROR` and refused globs at 10001 matches
instead of 5001; the constant had no importers.
The remaining ten are documented in `parity_exceptions.json`, each naming
one resource and one key: box/dropbox `du`+`find` and the seven readers
with no window argument (T1-I), github `supports_snapshot` (snapshot
redesign), lancedb `caches_reads` (computed per URI), notion/hf `find`
(slot on one side, bespoke command on the other), ssh `append` (wired in
typescript, read by no builder on either side).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(tar): drop the GPL bzip2 dependency, and parse tar with modern-tar
compressjs is GPLv2-or-later and sat in dependencies of the Apache-2.0
mirage-node package. Replace it with seek-bzip (MIT), which only decodes,
so a CompressionCodec may now be decompress-only and tar -cj exits 1 with
"tar: bzip2 not supported" the way browser core already answers for an
unregistered codec. Python keeps both directions through stdlib bz2.
Replace the hand-rolled ustar reader and writer with modern-tar (MIT,
zero dependency, web streams) behind the existing tar_helper seam. The
hand-rolled version truncated any name past 100 bytes instead of using
the ustar prefix field or a PAX header, so a deep member extracted to the
wrong path, and it read a PAX extension block as a member, so tar -t on
any archive GNU or Python wrote listed a phantom ././@PaxHeader row.
writeTar and readTar are async as a result.
* test(tar): cover -h, and round-trip bodies through the xz codec
-h had no test at all: the name a link member carries is the same whether
tar stored the link or followed it, so the existing symlink test could not
tell them apart. Read the archive back instead and assert the member kind,
in both directions, plus the unreachable-target case.
The xz test created and listed, which only proves headers survived. Extract
too, so the decompressed bodies are checked byte for byte.
* fix(archive): tar and zip archive a directory, and -C re-bases the operands after it
tar -cf d.tar d exited 1 with "tar: <hostpath>/d: Is a directory": every
operand went straight to read_bytes, with no isdir check and no
recursion, so the most common tar invocation there is could not work.
zip -r <dir> had the identical defect. Both now walk.
Create is a two-phase pass in each: decide every member, then write.
Both plans are built on one traversal, scan_operand / scanOperand
(generic/archive/walk), which merges three sources no single one can
see: the backend walk (find's walk_find, so an entry is classified
through stat and never by name), the namespace's symlinks, and the
mount table. It reports paths, never names, because naming is where the
two formats part company; the two things they disagree about in the
traversal itself are parameters, so tar passes recurse=True and
dereference=-h while zip passes recurse=-r and dereference=not -y.
A directory is its own member, so an empty one survives a round trip,
and both extractors now mkdir for one. A symlink is a symlink member
under tar (SYMTYPE, target in linkname) and under zip -y (mode 0120777).
An unreadable operand is reported in virtual path space with the
archiver's own wording, because the raw IsADirectoryError was leaking
the host path behind a disk mount.
tar -C was ignored: `tar -czf /work/out.tgz -C /work/check my_paper`
failed as "paths span multiple mounts", the same defect class as unzip
-p in #725, because the operand resolved against the session cwd and
the router then saw a phantom mount span. -C is not a flag the command
reads once, it is a chdir for the operands typed after it, so it is now
declared in the spec (CommandSpec.operand_base) and resolved by the one
component that walks the line positionally: the parser reports a base
per word and the classifier resolves each operand against it.
MountView is how a command sees mount boundaries, offered the way
LinkView is (name a `mounts` parameter, nothing else). A traversal that
renders lines gets this free from the executor's fan-out; one that
emits a single binary object cannot, which is why the archivers read
the table themselves. A descendant mount is not crossed: the mountpoint
stays an entry and its contents are dropped with GNU's
--one-file-system wording, since descending would archive by accident
what MountRootPolicy now refuses on purpose for tar, zip and cp in a
source slot.
Semantics pinned against GNU tar 1.35 and Info-ZIP 3.0 on
debian:stable-slim, including Info-ZIP's inverted defaults, its
anchored -x, its silent leading-slash strip, and "Nothing to do!"
exiting 12 with no archive written.
* chore(spec): the dumps state what a command declares, not what it defaulted to
Both generators emitted every field of every dataclass, so `truncate`,
which declares one thing (`-s`/`--size` takes a string), spent 27 lines
of spec body restating 21 defaults that read identically in all 93
files. `zip` was 198 lines for five flags, 11 of the 13 keys on each
option being defaults.
Anything equal to its default is now dropped on both sides. `type`
survives even at its default, because what a token is is the first
thing a reader looks for, and `"rest": {}` says less than
`"rest": {"type": "path"}`.
The defaults come from the dataclass fields in python and from a
default-constructed instance in typescript, rather than from a table
either side could let drift. The two must drop exactly the same keys,
and the parity gate reports every command if they do not.
This narrows that gate's blast radius rather than widening it. When
`operand_base` was missing from gen-specs.ts, python emitted the key in
all 93 files and typescript in none, so parity reported ~95 divergences
to sift. The same bug now reports one, on tar, which is the only
command that sets it.
check_spec_parity's option-diff renderer keyed options by `o["long"] or
o["short"]`, which raises once an option carries only the spelling it
declared.
279 files, 22113 lines of restated defaults gone.
* fix(archive): strip the mount prefix by scan, not by a backtracking regex
childSpec measured the backend key with `.replace(/^\/+|\/+$/g, '')`.
The trailing alternative backtracks on a run of slashes, which CodeQL
flags as js/polynomial-redos, and the input is a resource path a mount
supplies.
utils/slash.ts already has stripSlash, which walks the two ends by
charCode and cannot backtrack. Use it. slash.test.ts pins it against
the regex it replaces.
* fix(archive): the five symlink and mode bugs the codex review found
All five reproduced against a live workspace first, then pinned against
GNU tar 1.35 and Info-ZIP 3.0 on debian:stable-slim.
A symlink operand never reached the planner as a link. tar and zip were
absent from NO_FOLLOW_COMMANDS, so the router rewrote the operand
through the link table and `tar -cf o.tar link` stored a regular file
holding the target's bytes where GNU stores a symlink member of size 0.
It also skipped the planner's cross-mount refusal, since by then there
was no link left to refuse. Both archivers now lstat, and carry no
DEREFERENCE_FLAGS entry on purpose: -h and -y are the planner's to read.
Only the last -C was checked. `tar -cf m.tar -C missing x -C good y`
reported `x: Cannot stat` and still wrote an archive holding y, where
GNU chdirs at each -C and dies at the first it cannot enter. The option
accumulates now and the planner walks the list, so the first bad one is
fatal and no members are written.
Two links to one target were called a loop, and a real loop was not
caught at all. Both were the same mistake: an operand-wide `seen` set
doing detection the namespace already does properly under a hop limit.
Deleting it archives both names, as GNU and Info-ZIP do, and catching
the CycleError that resolve raises turns a genuine cycle into one fatal
problem per member with GNU's "Too many levels of symbolic links",
keeping the directory entry and exiting 2 instead of throwing out of
the planner for a bare exit 1.
The mount-root refusal denied member selectors. Under -t and -x an
operand names something inside the archive, so `tar -tf a.tar data`
was refused as busy when `data` happened to spell a mount. Gated on
create mode now, dashless first word included.
A test asserted the loop bug rather than catching it; it is replaced.
Five integ cases cover the lot end to end in both languages.
* test(archive): build the test LinkView's stat through FileStat, not a cast
integ-shared-py passes build-packages: false — the action's own note says
the python host needs node only for the tsx fake servers, which import no
mirage package. The typescript *runner* does import them, so the self-test
died on ERR_MODULE_NOT_FOUND there. It moves to integ-shared-ts, the job
that builds the packages; the setup action installs the python venv
unconditionally, so that job has both halves.
Reproducing it locally exposed a false pass: `strict (ts)` asserts a
non-zero exit, and a crash-on-import is also non-zero, so it went green
for the wrong reason. A smoke check now proves the runner starts (an
unknown facet exits 2 without running a case) before any exit code is read
as a verdict, and run_typescript keeps the first stderr line mentioning an
error so a failure names its cause instead of printing a bare exit code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
P1 — the database battery ran without --strict. My sweep that added the
flag matched single-line invocations, and integ-database's two commands
use a folded YAML scalar, so they were missed: losing mongodb, postgres,
chroma or qdrant to a renamed variable would still leave both jobs green.
Both now pass --strict; a re-scan that handles multi-line invocations
confirms every runner call in the workflow now carries it.
P2 — the gate self-test silently skipped its typescript half. The `integ`
job installs python but neither node nor integ/node_modules, so the
typescript assertions never ran while the step reported success — exactly
the report-green-having-tested-nothing failure this file exists to catch.
Adds --require-ts, which turns an absent tsx into a failure, and moves the
step to integ-shared-py, the job that installs both hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dify/find, mongodb/rg and postgres/rg had no provision while their generic
siblings on the same backend did; routing the hand-written command tails
through the family catalog gives them one, so `has_provision` flips to true
for those three in all three spec trees.
Spec parity: 93 commands match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught two things the local run could not.
pre-commit: `pre-commit run --all-files` only checks TRACKED files, so
gate_selftest.py was never linted locally while it was untracked. autoflake
drops an unused import and yapf reformats it.
--strict: the core facet is deliberately split across jobs (postgres and
mongodb run in integ-database, chroma/qdrant/lancedb in integ-data, notion
and nextcloud in their own steps), so integ-shared-{py,ts} skip six
services by design and --strict had no way to hear that. Adds --allow-skip,
which tolerates a declared skip, still fails an undeclared one, and rejects
a service name that does not exist so the list cannot rot into silently
widening what --strict accepts. The two --facet core invocations name the
six; every other --strict invocation needs none. gate_selftest covers both
directions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Items 11, 12 and 14 of the parity plan, plus the backend half of item 11.
Item 14: both runners' floor guard is facet-only and all-or-nothing, so a
facet that loses SOME targets still exits 0 — measured at 161 -> 71 passing
with TRELLO_ENDPOINT unset, still green. Adds a `services` table to
targets.json replacing both hand-rolled service->env if-chains (13 py / 21
ts, already drifted), `--strict` on all 16 CI invocations, load-time gates
for duplicate case ids and unknown target refs, a zero-pairs floor in
parity.py, and gate_selftest.py asserting all of it on both hosts in CI.
`check` becomes additive (expect.check) so a case can pin stdout AND the
post-condition; the 15 existing cases are migrated.
The duplicate-id gate found a live bug: unix/lookup/type.json and
bash/type/* shared six ids, and parity.py keys rows by (target, id), so one
of each pair was silently dropped from the py/ts diff.
Item 11: TS Limit.aggr gets a mapped-type field table (a fifth bound is now
a compile error); with_default_provisions lands in python and is applied to
history and then to every backend. Of 39 hand-named provisions per
language, 21 were pure catalog duplication and now come from the catalog
(verified 21/21 land on the identical estimator); 14 are genuinely bespoke
and stay. mongodb/rg, postgres/rg and dify/find had no provision at all
while their generic siblings did — now pinned by new integ cases.
Item 12: cut's legacy_flags shim, cp's find_type knob, grep's unreachable
files_only arm, OpRecord.rel_path + mount_prefix, the duplicate
chainStreams, the unreachable root vitest.config.ts (which also carried
passWithNoTests: true), and the dead mongodb/postgres _provision modules in
both languages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(flags): every command reads the bag through a spec-bound FlagView (#292)
The 152 raw `opts.flags.x` reads under `commands/builtin/` now go through
`new FlagView(opts.flags, spec)`, which throws on a name the spec does not
declare instead of reading it as false. The eslint rule that already
enforced this for `generic/*.ts` now covers `commands/builtin/**`, so the
next one fails CI rather than shipping.
Three things the guard caught on the way in:
- `bc -l`, `date -I`, `wget -O` and `email/rg -l` each read a key the
parser can never emit -- those shorts are disambiguated to `args_*`
(`AMBIGUOUS_NAMES`) -- and three test files pinned the same dead shape.
- `generic_bind/provision.ts` carried the TypeScript twin of the Python
`touch -r` bug fixed in #733: the shared write-metadata estimator read
`r`/`R` blindly, so `touch -r REF` (a reference file, not recursion)
reported UNKNOWN precision. Provisions now receive the invoked command's
spec (`CommandOpts.spec`, mirroring Python's `spec=` keyword) and resolve
by spelling, so head's `bytes` and tail's `c` need no hardcoded guess.
- `seq -w`'s string branch was unreachable: the spec declares it boolean.
cp/mv stop declaring their own flag-bag type. `parseCpFlags`/`parseMvFlags`
and the four helpers take a `FlagView` the way `parse_cp_flags` does in
Python, the local `export type Flags` is gone, and the builders and the
crossmount relays build the view with `specOf('cp'|'mv')`. The bag's value
type is the already-exported `FlagValue` everywhere -- 70 inline copies of
`string | boolean | number | string[]` across 43 files, plus the private
alias in `cross_mount.ts`, now name it.
Deliberately not widened to include PathSpec: Python's executor promotes
PATH-typed flag values, TypeScript's `command/flags.ts:91` keeps the
resolved virtual-path string, and no site in the tree writes a PathSpec
into the bag. cp's PathSpec-tolerant read stays, with a comment pointing
at the Python promotion it is reserved for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(flags): three Python bugs the raw bag reads were hiding (#292)
Converting the TypeScript wrappers to a spec-bound FlagView surfaced the
same rule violations on the Python side, and each one hid a live defect.
- **`--desc_file`/`--text_file` never worked on trello.** A PATH-typed
flag reaches a Python command as a PathSpec (the executor promotes it),
so `fl.as_str("desc_file")` read it as absent while the guard beside it
used the raw `_extra.get` and saw the value -- the branch was entered
and the file was never read. The eight trello wrappers now build the
view and read path flags through a shared `file_operand` helper.
- **`gzip -1` compressed at zlib's default.** `extract_level` looked the
bag up by the bare digit, but `-1` is the one digit the parser
disambiguates to `args_1`. `gzip -2`..`-9` worked, `-1` silently did
not. The native test passed because it only asserted `len(-9) <= len(-1)`.
- **`cp --backup --suffix=` used an empty suffix**, naming the original as
its own backup, where GNU 9.7 falls back to `~` (pinned with docker).
TypeScript already had this right, so the fix goes to Python and both
sides now pin it.
`test_no_raw_flag_reads.py` only matched `flags.get(` -- the `kwargs` and
`_extra` names a wrapper collects the same bag into slipped past, which is
how all of the above survived. It now matches all three (and exempts
`config.py`, which answers --help/--version off the bag as spec layer).
qdrant/lancedb stop fishing `cwd` out of `_extra`, curl's `-o` and unzip's
`-d` stop claiming to be `str`, and CLAUDE.md records the PathSpec rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): pin gzip -1, cp/mv empty --suffix=, touch -r provision (#292)
The three flag fixes earlier in this branch were covered by unit tests
only, so nothing proved them across the 20 backends the battery sweeps.
- `unix/gzip/1.json` mirrors `9.json` for the one digit the parser
disambiguates to `args_1`, which is the one that was broken.
- `cp`/`mv` grow an empty `--suffix=` case pinning the GNU 9.7 fallback
to `~` (docker-pinned), next to the existing `.bak` cases.
- `unix/provision/basic.json` grows `prov_touch_r`, which sits between
`prov_rm_r` (unknown) and `prov_mkdir` (exact): before the fix the
reference file read as recursion and the estimate degraded to
UNKNOWN. Its op count is 2 -- the provision layer scopes every
PathSpec in the command line, reference file included -- and both
hosts agree on that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(integ): serve trello from a shared Prisma fake, the way slack is (#292)
Trello was the last write-capable service whose fake ran per-host: the
Python runner started `server/trello_server.py` in-process while CI
started a second copy of the same file for the TypeScript runner. Two
processes seeded from one fixture agree only as long as nobody edits
one of them, and neither host could observe what the other wrote.
`server/trello.ts` replaces it, built the way `server/slack.ts` is:
Prisma + SQLite, `db push` into a fresh file per instance, seeded from
`fixtures/trello/v1.json`, one shared process both hosts call over
TRELLO_ENDPOINT with POST /reset rolling back to the fixture. Writes now
persist in a store rather than a dict, so a card created by one host is
visible to the next command instead of dying with the process.
Every collection the API returns has a caller-visible order, so each
table carries an explicit `seq` and is read back with `orderBy: seq`;
membership (card members, card labels) is rows rather than JSON arrays
so an idempotent add is a `findFirst`, matching the endpoints. Writes
draw ids from one counter and stamp one date, exactly as the fake they
replace did, so `crd_new_1`/`cmt_new_2` still number the same way.
Verified by diffing both servers endpoint for endpoint -- 27 requests
covering every route, the 404 bodies, a create/update/assign/label/
comment/unlabel write sequence, an idempotent re-add, a unicode card
name, and the reads after those writes -- all byte-identical.
The datasource env var is now INTEG_DB_URL (it backs two services, not
one) and `slack:setup` is `prisma:setup`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(trello): --desc_file/--text_file address the mount, not the URL (#292)
`resources/trello/cmd.json` only ever ran reads, so no trello write was
covered on either host. Adding that coverage found the other half of the
`--desc_file` bug this branch already fixed once: reading the flag now
works, but the path handed to the backend was still the virtual one.
`core/trello/read` keys off the mount-relative path and quotes the
virtual one in an ENOENT. Both hosts were passing the virtual path as
the key, so `/board/workspaces/...` looked for a `board` collection that
does not exist and every read missed -- an in-mount operand died with
ENOENT, an off-mount one with "Operation not supported". Python now
carries the whole PathSpec through `file_operand` and reads
`resource_path`; TypeScript, whose bag holds only a string, strips the
prefix with `mountKey(filePath, opts.mountPrefix)`. The shared
`resolve_text_input` is generic over the path shape so the reader can
take whichever one its backend is addressed by.
The same new cases caught a divergence in the API error text: TypeScript
named the failing endpoint, Python did not. Python now names it too --
an agent reading a 404 needs to know which call made it.
`resources/trello/write.json` covers create (inline desc), create into a
list, update by `--desc_file`, update by inline desc, comment by
`--text_file`, comment by stdin, comment append, assign/label/move,
unlabel, the off-mount refusal, and the 404. They run after every read
pin (seq 940010+) because they mutate the board the size pins measure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(tar): accept GNU's old option style
`tar xzf a.tgz` failed with "must specify -c, -x, or -t" because a first
word with no leading dash was read as an operand. It is a cluster of
option letters whose arguments follow as separate words, expanded now by
expand_old_style / expandOldStyle before anything else reads the line.
Per letter, not one -xzf token: a cluster may hold several
argument-taking letters (tar xfC a.tgz out) and may continue past one
(tar cfz a.tgz f gzips), neither of which getopt clustering can express.
* chore(spec): regenerate spec dumps for old_option_style
* refactor(pyodide): serve mounts from an Emscripten filesystem, not a python shim
Replace the 425-line python source string the runtime injected with a
filesystem mounted through pyodide.FS.mount, so interception moves below
the interpreter instead of rebinding names inside it.
This fixes two bugs the shim shipped. os.open/os.write/os.fdopen and a
bare os.truncate recorded nothing, because the shim rebound 13 names and
those were not among them, so a low-level write applied to guest memory
and was dropped at exit 0. And a cross-mount rename raised errno 18,
which is EDOM under pyodide's musl numbering, so a guest comparing
against errno.EXDEV (75) never matched; the old test asserted the
message string python rendered from that same literal, so it could not
fail.
Every prefix is re-seeded on every run. The conformance rows covering
post-boot seeding only passed before because the shim's lazy backfill
used run_sync, and vitest enables JSPI while production does not.
A file the mount lists but will not serve now becomes a node that
refuses to open with EIO, rather than a hole an append would fill by
replacing the file.
* refactor(pyodide): name the filesystem package vfs, not fs
Mirrors the python side, where the module holding the guest filesystem
is vfs.py beside runtime/vfs.py. Leaving the package as fs/ while its
entry module became vfs.ts would have kept the mixed spelling the
rename is removing.
* refactor(pyodide): name the sentinel and the whence values
Number.MAX_SAFE_INTEGER appeared five times as the has-not-written-yet
sentinel, and llseek compared whence against bare 1 and 2. Both are now
in constants.ts beside the mode bits, matching the python side where
READONLY_HINT moved to wasm/constants.py.
* refactor(runtime): one mount op vocabulary for the typescript sandbox runtimes
New runtime/vfs.ts holds RuntimeVFS, planFlush and the mount routing that
pyodide, quickjs and monty each had their own copy of, mirroring the python
core. BridgeDispatchFn gains APPEND, so a handle that only extended a file
ships its tail: eight 3-byte appends now cost 24 bytes rather than 536, and
the two amplification rows in the conformance suite flip from it.fails to
it. runtime/config.ts takes coerceRuntimeConfig out of base.ts so a typed
config has the same home in both languages.
monty.ts becomes a monty/ package (binding, constants, errors, osaccess,
runtime, vfs) matching the python split, and MontyVFS gains the negative
cache python already had: monty asks whether a path exists on nearly every
guest expression, and each miss was costing a fresh listing.
mirage_bridge.ts splits into vfs/journal.ts and vfs/preload.ts, and
js/mirage_fs.ts becomes js/vfs.ts, so every module holding a guest
filesystem is named vfs. Dead MontyVFS.mountOf dropped in both languages.
* fix(pyodide): refuse a root mount, mount nested prefixes parent-first
Three review findings.
A `/` prefix stripped to the empty string, which Emscripten takes as a
detached pseudo-mount no path reaches: the guest kept reading and writing
MEMFS, and a write reported success the resource never saw. `/` is already
MEMFS's own mount root holding the stdlib, so there is nowhere to put it;
the prefix is now skipped with one warning rather than silently dropped.
prefixes() is longest-first, which is what routing wants and the reverse of
what the mount table wants: with /data/ and /data/inner/ the child mounted
first and the parent then mounted over it, orphaning the child. Unmount
deepest-first, mount shallowest-first.
MontyVFS lives as long as the runtime while python rebuilds its
MirageOSAccess per run, so the negative cache added here outlived the
command it belonged to and hid a file another writer created between two
monty runs. Reset it at the top of run and eval.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
pytest only honors pytestmark in test modules, so the redis suite
crashed on RedisStore's URL parse instead of skipping on machines
without REDIS_URL. The gate moves into the shared store fixture, and
the registry raw-kwargs test gets the same skipif plus the env URL
instead of a hardcoded localhost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two edge cases pinned against coreutils 9.7 and fixed in both
languages: 'no properly formatted checksum lines found' keys on the
parsed-line count (a malformed line next to an --ignore-missing skip
misread as it) and still prints under --status; 'no file was verified'
fires whenever --ignore-missing leaves zero OK lines — mismatches
included — follows the WARNING block, and --status silences the text
while its exit 1 stands. Also hoists the dev workspace-test helper to
module scope per the no-nested-functions rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dict.pop override gains the supertype's overloads (mypy), the dify
zulu formatter narrows to its number/string domain instead of
String(object) (eslint no-base-to-string), and the awk propagation
test throws from the stream factory instead of a yield-less generator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TS dify had ported only python's stat_light: no document-detail fetch,
so document_id, refreshed tokens/indexing_status, detail-derived
source_size, and the updated_at mtime were missing. The heavy stat now
mirrors python field-for-field (second-precision zulu mtime included),
with the same cost containment: a new opsOverrides on the generic
command factory (the TS twin of python's ops_overrides) pins ls to the
light stat, and find upgrades to the heavy stat only under -mtime.
df_prov_ls's net=0 pin stays green, and a new df_stat_mtime case pins
'stat -c %y' on both hosts.
rm /dev/null raised an uncaught KeyError('/null') in python and
silently no-opped in TS while rm -v printed a false removed claim.
Both stores are now a synthetic null/zero overlay with tombstones over
a real backing map — root-on-linux semantics, pinned in docker: the
removal succeeds and the path is genuinely gone, a later redirect
recreates it as a regular file, and writes to a live /dev/null still
vanish. /dev stays excluded from snapshot capture on both sides.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TS ls/tree/rg/checksum/awk caught every error class where python
narrows to (OSError, ValueError)/FileNotFoundError, so auth failures
and transport errors on those paths vanished from listings or were
laundered into GNU-shaped 'cannot access' lines. A new isWalkError
(isFsError + the noMount stamp, python's ValueError twin) narrows all
seven ls sites, tree's walk, rg's dir probe, and checksum -c; awk -f
mirrors python's absence-only catch. Python's own loose sites tighten
the other way: grep/rg per-entry warnings catch WALK_ERRORS instead of
Exception, and the existence probes in builders/common catch
MISS_ERRORS.
checksum -c also becomes GNU-complete (coreutils 9.7 pins): recorded
names resolve against the cwd as PathSpecs — a relative name used to
reach backend readers as a bare str and die on .virtual — and stderr
now carries the per-file strerror lines (kept under --status), the
prefixed WARNING block in GNU's order, per-line --warn diagnostics,
and the no-properly-formatted-lines / no-file-was-verified fatals.
awk -f resolves relative program files against the cwd too instead of
the mount root. The two tier2 integ pins move to the GNU behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python's generic find read paths[0] and dropped the rest, so
`find /a /b -name x` returned only /a's matches on all 12 backends
that wire a native find op (the readdir walk already looped). The
generic now owns the per-operand loop — GNU semantics pinned on
findutils 4.10.0: operand order, duplicates walk twice, a missing
operand gets its diagnostic and exit 1 while every other operand's
rows still print (the old path discarded them with the stdout).
The per-root -path tree is stamped into a local, so the shared
FindArgs stays unprefixed, and bespoke wrappers (github, chroma,
dify, history) inherit the loop through generic_find.
TypeScript looped already but re-sorted across roots at the end,
interleaving operands; the cross-root sort is gone.
integ/unix/find/roots.json pins operand order, duplicate roots, and
partial output past a missing middle operand on both hosts (22
targets; the order case provisions its own subtree since shared
directories accumulate per-target state).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five private TS guessType copies (box/gdrive/dropbox/gmail/email) drift
from the shared table py already uses: 8 typed extensions read as BINARY
(jsonl/tsv/py/js/ts/yaml/yml/toml), so the same file stats differently
per language. Py gmail/email guess through stdlib mimetypes, whose
platform tables make .ts video/mp2t and .gz unknown. All seven sites now
import the shared table; log->TEXT and gzip->GZIP (the private copies'
two extra keys) join it on both sides. The qdrant/lancedb 4-entry image
maps fold into a shared image_type_for_extension, and FILE_MIME_MAP
drops the parquet/orc/feather/hdf5 keys unreachable since #651.
integ/fixtures/filetype/tables.json now pins every table; both unit
suites assert equality so one side cannot drift alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex review on #728.
A relative operand after os.chdir into a mount missed every prefix check,
so the mutation (and, already on main, the write) stayed in MEMFS and the
run exited 0. Paths now resolve through abspath before the mount test.
An append whose base read failed for any reason treated the base as empty,
so a file that exists but is momentarily unreadable would be replaced by
the tail alone. Only a confirmed absence starts empty now, via the
repo's own isMissingPath.
The replay continued past a failed mutation, which could apply a later
entry against a prerequisite that never landed (a rename moving a stale
temp file onto its destination). It stops at the first failure and reports
how many entries it dropped.
The shim patched only open, listdir, stat and scandir, so every mutation
spelling mutated MEMFS and never reached the mount, and an append-mode
open on a file MEMFS had not seen flushed its whole buffer over content
the run never read.
Patch the os primitives instead of a spelling list (makedirs, pathlib and
shutil all reach the mutation through os), record an append as its tail
rather than the whole file, and backfill on non-truncating writable opens.
Guest mutations now go into an ordered journal that the runtime replays
after the script, so a write-then-rename lands the way it ran. None of it
consults the bridge inline, so mutations need no JSPI.
RFC 3501 compares SENTON/SENTBEFORE/SENTSINCE (and ON/BEFORE/SINCE)
"disregarding time and timezone", so a message written 05 Jan 23:30 -0500
answers a search for the 5th. Converting to UTC filed it under the 6th, and
`himalaya envelope search date 2026-01-05` then selected mail its own
directory did not contain.
Python drops the UTC conversion, restoring what it did before this branch.
TypeScript was always UTC and is the side that was wrong: `Date` keeps only
the instant, so the stated offset is parsed back out of the string (numeric
plus RFC 5322's obsolete zone names) and added before reading the fields. A
header with no zone at all reads its local fields, which hands back the wall
clock exactly as written.
Verified identical output between the two languages across nine timestamps
under TZ=UTC, America/New_York and Asia/Tokyo.
A message with no parseable Date: header fell straight to 1970-01-01, so
every undated message collapsed into one directory and /mail/<FOLDER>/<date>/
carried no information. The header still wins (it is what himalaya's SENT*
date conditions search on); INTERNALDATE fills the hole, epoch only if
neither parses.
Metadata is now requested ahead of the body in the FETCH item list: a server
may answer items in the requested order, and anything after BODY[] lands on
the line behind the literal where the response parsers never look.
Python normalizes offsets to UTC, matching TypeScript and both gmail
backends; it previously kept the sender's offset, so 23:30 -0500 bucketed a
day apart between hosts.
INTERNALDATE is transport metadata, not part of the message, so it is
dropped in the single renderer. The five other places that serialized a
fetched message by hand (himalaya read/list/search, email rg, searchAndFormat)
now route through that renderer too, so `himalaya message read` cannot drift
from `cat` of the same file.
* refactor(runtime): declare the default captures once per language tier
* test(runtime): conformance suite for runtime mount access
* chore(deps): bump pypdf to 6.15.0 for two DoS advisories
uv audit rejects pypdf 6.14.2 over GHSA-fp3f-mc75-235c (large
/ToUnicode streams) and GHSA-fwg2-594c-jp42 (large CID font width
ranges), both fixed in 6.15.0. pypdf is transitive only —
openhands-tools -> browser-use -> pypdf, nothing in the tree imports
it — so this is a lock-only bump that leaves the rest of the
resolution untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* style: wrap the monty rmdir command under the column limit
flake8 caught the one-liner at 80 columns; yapf cannot split a string
literal, so it passed the formatter and failed Check PEP8. Split it at
the same point as the neighbouring rename rows, which leaves the
concatenated command byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(runtime): assert the verification command itself succeeded
An absence want was read off the stdout of a check that was never
required to succeed, so a mutation that damaged the mount could certify
itself: `ls /nope` answers exit 2 and empty stdout, and "gone.txt is
absent" holds for every x in an empty string. Verified against the
suite before the fix — a row whose verifier cannot succeed passed.
Both helpers now require exit 0 from each check before inspecting its
output. Positive wants were already safe, since empty stdout fails the
substring, but they get the clearer message too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(unzip): select archive members, exit 11 on unmatched filespecs
* fix(unzip): match filespecs bytewise, read each selected ZipInfo
Pinned against Info-ZIP 6.00 on debian:stable-slim:
- `?` stands for one byte, not one code point, so `?.txt` misses
`é.txt` and `??.txt` hits it. Python matched code points and
TypeScript matched UTF-16 units; both now match encoded bytes.
- A name lookup resolves every duplicate member to the last one, so
`unzip -p a.zip dup.txt` served the same payload twice. Read the
selected ZipInfo instead.
Test callbacks move to module scope.
* feat: $'...' quoting, git log/show formats, himalaya --attach + byte-exact MIME parity (py+ts)
Shell:
- $'...' ANSI-C strings decode per bash 5.2 (docker-pinned): full escape
table, octal/hex/unicode/control forms, segment-only NUL truncation;
$"..." keeps plain double-quote semantics. New shell/escapes decoder,
expansion branches, allowlist sites, dollar-quote-aware backtick scan.
- Quoted case patterns match like bash: patterns stay nodes and a new
expand_case_pattern renders quoting into the fnmatch dialect (quoted
segments literal, unquoted globs live, backslash escapes, expansion
results live unless double-quoted, no word-splitting).
- export/local/declare/readonly accept quoted assignment operands; a
bare $ word survives as a literal argument (echo $).
Git:
- log gains --all (multi-root walk with annotated-tag peeling) and
--format/--pretty: oneline/short/medium/full/fuller presets,
format:/tformat: templates, ~20 placeholders incl %d decorations in
git's exact order; unsupported presets refuse honestly.
- show gains --stat (full diff.c show_stats geometry), -s/--no-patch,
--name-only, --no-ext-diff, --format; oracle-verified byte-identical
to real git in both languages.
- commit report counts binary files as files with zero lines (shares
diffstat's NUL sniff); fixes a TS-only miss of mode-only changes.
Himalaya:
- --attach wires up: repeatable path-typed flag (first leaf-level
type="path" option; python reads via new FlagView.as_paths), reads
through inv.ops.dispatch, multipart/mixed with a content-addressed
boundary so both builders emit identical bytes.
- TS serialization is now byte-identical to python's
EmailMessage.as_bytes(policy=SMTP): new mime.ts ports RFC 2047
encoded words (q/b chooser), unstructured/address header folding,
RFC 2231 filename params, and the 7bit/8bit/qp/base64 body
selection; fixes threading-header order, empty-subject rendering,
and multi-line header refusal. 31 shared pins
(integ/fixtures/himalaya/mime_parity.json) asserted by both suites,
plus differential fuzzing against the python oracle.
Docs list the new git capabilities and remaining deliberate limits;
51 new integ cases across bash/git/himalaya targets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: codex round + generalize pattern quoting, multi-line strings, MIME to core
Codex review fixes (all pinned against real git 2.37/2.54, bash 5.2
docker, and CPython 3.12):
- git format: empty entries keep their separators under format: and
their terminators under tformat: (only an empty template is silent);
%xHH emits a raw byte through the shell's byte-escape convention;
bare --format gets git's own fatal while bare --pretty stays medium.
git show's format: header drops its trailing newline the same way.
- ANSI-C \u/\U: values now ride bash's u32toutf8 (UTF-8 locale) -
surrogate halves and values past Unicode become raw UTF-8-shaped
bytes, 0x80000000 and past produce nothing. The previous "verbatim"
rule (and the crash codex flagged) was a C-locale pin artifact.
- himalaya: an ASCII attachment filename holding any line break is
refused with EmailMessage's exact error; the RFC 2231 path
percent-encodes the same characters (new shared parity pin).
Review round (generality follow-ups to the case-pattern fix):
- expand_case_pattern is now expand_pattern and serves all three
pattern-word constructs: case patterns, the [[ == ]] right side
(replacing the whole-node right_literal boolean that broke mixed
quoting), and parameter-expansion operands. Quoted operand nodes
match literally; opaque regex tokens get a lexical scanner honoring
single/double/ANSI-C quotes, backslash binds, and live $-refs.
escape_glob moved beside GLOB_CHARS in utils/glob_walk (both
languages), killing a duplicated constant.
- Multi-line double-quoted strings: the newline bytes belong to no
tree-sitter token. TypeScript dropped them entirely and python
collapsed blank lines; both now re-emit per row step anchored on the
quote tokens (leading, trailing and blank lines included).
- MIME machinery out of himalaya: mime.ts is now the runtime-agnostic
@struktoai/mirage-core utils/mime (TextEncoder + shared base64, no
Buffer), exporting the header guard as assertHeaderValue; the fixed
extension table lives in utils/filetype as MIME_BY_EXTENSION /
mime_type_for beside the FileType maps in both languages. Fixed a
silent TS-only bug found in the same file: jpg guessed IMAGE_PNG.
python's local drain() replaced by the existing io materialize.
Coverage: 40+ new docker/real-git/CPython-pinned unit rows across both
languages, 20 new integ cases (git format band, param/test/quoted bash
categories, surrogate byte-count), one new shared MIME parity pin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(monty): port to pydantic-monty 0.0.19 and stop rewriting files on append
0.0.19 moved execution into subprocess workers: Monty became a pool,
MontyRepl is gone, and run_async moved onto a checked-out session. run()
and eval() now drive the pool, and eval() is what the policy layer runs
config-borne scripts on, so the pin is exact while the API keeps moving.
Cancelling a feed no longer stops a worker, so both paths reclaim the
worker pid on CancelledError. Without it a safeguard timeout wedged the
line forever and pool teardown blocked uninterruptibly.
Also fixes four bridge gaps: mkdir, rename and rmdir never reached the
workspace and failed with a bogus ENOENT on paths that existed; piped
stdin was discarded, and now binds as a global beside argv; and appends
re-sent the whole file every write, which made a write loop quadratic
(200 appends shipped 164 KB to build a 1.7 KB file). Monty hands the
append hook the new text alone, so it routes to the mount's append op.
Documents what a command_safeguard does and does not stop inside a
sandbox, where mirage holds only a client and the process keeps running.
* fix(monty): bump the ts binding to 0.0.19 and route the write ops
Path.mkdir, rmdir, unlink and rename declined in the monty callback, so
they applied inside the sandbox's own tree and never reached the mount.
The bridge already carried the ops; this wires them up, matching the
python runtime.
* fix(monty): address review on append fallback, pool race, and cross-mount rename
- appends fall back to the whole-file flush on mounts without the
optional append op (S3 registers write but not append), remembering
the mount so it costs one failed dispatch rather than one per append
- the worker pool is cached as a task, so two cold runs cannot each
build one and leak the loser's workers past close()
- a cancelled eval session hands its checkout back instead of dropping
a lease the pool can no longer reach
- mkdir forwards parents and answers exist_ok itself
- a rename across two mounts raises EXDEV in both languages: the
dispatcher picks the mount from the source alone, so crossing would
drop the source and write the target into the wrong backend
* docs(monty): correct the EXDEV rationale, monty ships no shutil
* fix(monty): reject mkdir over a file, and type the mutation errors
* fix(find): classify walked entries through stat, drop the is_dir_name heuristics
* fix(test): satisfy noUncheckedIndexedAccess in the gmail find fixture
* fix(dropbox): walk the native find over a scratch index
* test(integ): pin find -type f over attachments and uploads
* test(integ): find -type f matches a pdf upload too
* ci: retrigger after the Actions outage
* fix(email): route the TS find through the generic walk so every flag applies
* chore(deps): bump h2 to 4.4.1 for GHSA-6hr6-w5qg-qmwg
* feat(gws): place creates in the folder scope, and report Google's errors
Three fixes and a mock-server sweep, all on the Google surface.
The gws CLI ignored folder_id, so a create landed in My Drive's root
even when the CLI and a gdrive mount shared one config. The editors'
create methods carry no parents field at all, so those are moved with a
follow-up Drive update; drive files create and copy just default their
parents. An explicit parents still wins.
Google reports why a call failed in the response body, but the python
client raised before reading it, so an agent only ever saw "Bad
Request". Typescript already did this correctly. The API and OAuth
error shapes are both handled.
An account CLI mutates its service by id, which no vfs path can be
derived from, so a newly created file had no cache entry to expire and
the agent's next ls could not see its own work. Write verbs now drop
the listings of the mounts their service backs, declared per spec so a
Slack or S3 mount alongside keeps its cache. This corrected two integ
goldens that had been asserting the stale listing.
The fake Google server grew the Sheets endpoints it never had
(values:batchUpdate, batchGet, batchClear, clear, sheets:copyTo, the
dimension requests), plus the matching gaps in docs, slides and drive.
gws integ coverage is now 35 of 35 leaves.
* fix(gws): address codex review on the folder-scope PR
Three findings, all real.
Cached file bodies survived a CLI write. drop_service_listings cleared
the resource index but not the file cache, and all five Google resources
cache reads, so a cat after a Docs or Sheets edit kept serving pre-write
content without reaching Google. A stale listing hides a create, a stale
body hides an edit, and only the listing was being dropped. Adds a
prefix-scoped evict_prefix to the file cache contract, implemented on the
RAM and Redis stores in both languages, exposed as CacheManager.drop_prefix
and called beside the index clear. The function is drop_service_caches now,
since listings no longer describes what it does. Redis escapes the prefix
before using it as a SCAN MATCH pattern, because a mount path may hold
glob metacharacters.
Scoped placement never sent supportsAllDrives. Every other Drive helper
in the repo sends it, and a folder scope may name a Shared Drive folder,
so both the injected parent and the relocation patch would fail there,
stranding an already created editor file in My Drive. Both paths send it
now, only when mirage is the one injecting.
An explicitly empty parents array was silently replaced. The check was a
truthiness test, so parents: [] read as absent while parents: ["root"]
was honored, which contradicts the documented rule that an explicit
array wins. Presence of the key is the test now.
Adds unit coverage in both languages for each finding, plus two integ
cases pinning the stale body end to end. The integ pair has to span two
command lines: apply_io runs once per line, so a cache warmed earlier on
the same line is never live for a later read, and a single-line case
passes with or without the fix. Documents folder scope and its two
divergences from the upstream passthrough in gws.mdx.
* test(cache): pin drop_prefix on a root mount
A root mount strips to the empty prefix, so the eviction argument is
"/" and matches every key. Correct, but non-obvious enough that a
reviewer reasoned their way to the wrong answer and back, so both
languages now assert it instead of leaving it to be re-derived.
* feat(git): a git CLI over any mount, in both languages
Adds `git` as a builtin CLI (issue #705 item 1): status, log, show,
diff, branch, add, reset, commit, checkout. It takes no config, and the
repository is read entirely through the mount ops, so a repo on RAM, on
disk or on an object store reads the same way. Python goes through
dulwich, TypeScript through isomorphic-git backed by a PromiseFsClient
over the dispatcher.
Status is pinned against the real git binary across 22 repository states
and 6 spellings, the mutation verbs are read back by real git and pass
`git fsck`, and the rename-similarity score matches dulwich digit for
digit. Both run in integ on `git-ram` and `git-disk` in both languages.
Also fixes the grep family this leaned on:
- grep reads a basic regular expression by default, as POSIX says, with
-G for the default and -E for extended. `grep -l` and `grep -rl`
compiled their own pattern and still read extended.
- zgrep does the same. Its -E flag was read and discarded.
- an operand grep could not search exits 2, as GNU does, instead of
being flattened to 1.
- `grep -l` on a directory reports it rather than walking it. The
shared fallback called the recursive walk whenever a failed read
turned out to be a directory, which made -l behave like -rl.
Supporting pieces: ranged reads (`read_range`) with a read-and-slice
fallback and a seeking implementation for disk, PATH-typed group options
resolved against the working directory, and git's three unknown-option
dialects.
* fix(rg): report an unreadable operand as exit 2, like ripgrep
CI caught four goldens the grep change had missed, all outside the JSON
harness targets I had been running, plus a divergence the change itself
introduced.
The divergence: TypeScript routes rg through grepGeneric, so rg moved to
exit 2 there while python's rg, which has its own generic, stayed at 1.
Real ripgrep exits 2 for an operand it could not read, so TypeScript was
right and python is brought up to it rather than the other way round.
Its single-operand path also let the error escape to the shared handler,
which flattens every OSError to exit 1; rg now reports it itself, the
same fix grep needed.
The rule is now stated once in grep_helper (`exit_code_for` /
`exitCodeFor`) and imported by both generics in both languages, instead
of living in the grep generic where rg could not reach it.
Goldens updated: integ/cross_commands.{py,ts}, the shared observability
contract, langfuse and dify. Each pinned exit 1 for a missing operand
and carried a comment calling it a deliberate divergence.
* fix(git): refuse the three mutations that could lose work
Three review findings, each measured against git 2.50.1 before fixing.
checkout only compared tracked paths, so a branch holding a file the
working tree has untracked wrote its blob straight over it. The file is
in no index and no tree, so nothing could see it and nothing could get
it back. The untracked set is now part of the conflict check, which
needs UNTRACKED_ALL rather than the mode status uses: "normal" collapses
a wholly untracked directory to one row, and git names the file inside
it. An ignored file stays overwritable, which is git's own split. When
both kinds of conflict apply git prints both paragraphs and aborts once,
so CheckoutConflictError carries both lists.
branch -d deleted a branch HEAD does not contain, dropping the only name
pointing at those commits. It now refuses, and -D is added, because -d
alone would be a delete with no way to say no. dulwich's can_fast_forward
answers exactly this and cannot be used: it asks the repository for its
grafts and shallow boundary and a bare BaseRepo raises. Walker is what
log already walks with and needs only the object store.
add -u ignored its pathspecs and restaged every tracked file, which is
how an unrelated edit reaches the next commit. git tells two misses
apart and so does this now: a pathspec naming nothing is a fatal about
the pathspec, one naming an untracked file is a fatal about git not
knowing it, both exit 128.
Also two CI failures. The spec dumps were stale for the -G flag added
with the zgrep BRE fix. And the git fixture exported its identity as
environment variables, which reach only its own commits, so a test that
committed into the built repository failed with "Author identity
unknown" on any runner with no global identity. It now records the same
identity in the repository, leaving the object ids unchanged.
Covered by unit tests in both languages and six integ cases across
git-ram and git-disk on both hosts.
* fix(test): the redis missing-file suite still pinned grep at exit 1
Sibling of the disk suite, which was updated with the exit-2 change. This
one needs a live server, so it skips silently without REDIS_URL and the
local run never reached it. Only the grep case moves: cat, head, tail and
wc all still exit 1 for an operand they could not read, which is the
split the comment now records here as it does next door.
* fix(grep): read an operand's type from stat, not from how the read failed
Four integ jobs caught this across seven backends, and it is one mistake
with three faces. Classifying a directory operand *after* a failed read
makes the answer depend on what each backend does about reading one, and
they disagree: s3, gridfs, hf and nextcloud read a directory path without
complaint and hand back nothing, and ssh raises an asyncssh SFTP error
that is not an OSError at all, so it escaped the catch entirely and
exited 1 with an unattributed "grep: Is a directory". So the operand is
now stat-ed before it is read, which is what GNU does and what the -r
branch of the same function already did.
operand_is_directory had its own version of the same error: a readdir
that did not raise counted as a directory, and a prefix store answers
readdir for any path at all, returning nothing for one that is not
there. Every missing file on those backends therefore read as a
directory. The listing must now be non-empty to count, which costs a
genuinely empty directory being invisible there, the same divergence du
already documents and the safer way round.
The catch also widens from three exception types to WALK_ERRORS, the
tuple every other operand-tolerant walk in the repo uses, so an errno
split cannot make grep abort where tree and grep -r keep going.
One test fake resolved stat with `undefined as never` and only worked
while nothing read the value. It resolves with a real FileStat now,
which is what the postgres backend answers there; the assertion the test
exists for is untouched.
Verified on gridfs and gridfs-prefix, which reproduce the prefix-store
half locally, plus ram, disk and redis for regressions: all five targets
on both hosts, 0 failed.
* fix(git): three review findings on -C, checkout -b and reset
`-C` accepted any path that was not missing, so naming a file inside a
repository walked up and ran in the parent instead of failing the way
git's chdir does. For a write verb that means mutating a repository the
caller never named. It now refuses a non-directory with git's own second
wording, "Not a directory".
`checkout -b <new> <start>` forced the new branch to HEAD and dropped the
start point without a word, so every commit after it landed on the wrong
history. The operand is honored, and a start point that is not a commit
gets git's sentence naming both it and the branch rather than the
generic "ambiguous argument" the same lookup failure produces elsewhere.
`reset <operand>` that selected no path unstaged nothing and exited 0,
which a script reads as "the index was reset". Two different mistakes
reach that point and they now get different fatals. A typo is git's
"ambiguous argument". A revision is not: real git resets the index to any
commit named there, measured on 2.50.1, so the review's premise that git
refuses one is wrong. This build resets from HEAD only, and says which
feature is missing instead of claiming a revision it can resolve is
unknown. Recorded as a divergence in both git.mdx files.
Six integ cases across git-ram and git-disk on both hosts, plus unit
tests in both languages.
* chore: merge main and regenerate the grep and zgrep specs
#713 landed a `pair` field on the option spec while this branch was
open. CI builds the PR merged with main, so it regenerated specs
carrying the field and compared them against grep.json and zgrep.json,
the two files this branch had already rewritten for `-G`, which
therefore predate it. Every other spec file came across from main
already carrying it.
The regeneration also needs a rebuilt dist: gen-specs.ts reads the built
package, and against a stale one it dies on an Operand field it does not
know, which silently leaves the typescript half unregenerated and shows
up as a python/typescript parity divergence rather than as a build
error.
* fix(shell): write a hex or octal escape as the byte it names
printf '\xc3\xa9' wrote four bytes where bash writes two: \xHH read as a
code point and was then UTF-8 encoded, so \xff came out as c3 bf rather
than the single byte it names. Octal escapes and echo -e had it too.
A byte above ASCII now rides as its surrogate escape and every place the
shell turns its own text into bytes decodes it back, which covers printf,
echo, %q, heredocs and here strings. \u and \U still name code points.
* fix(gsheets): ask for the cell values the mount documents
.gsheet.json promised cell values at .sheets[0].data[0].rowData[] and
carried none: spreadsheets.get returns no grid data unless the request
asks for it, so the file was tab metadata against real Google too, not
only against the fake server.
The fake server implements the endpoint now, with the shapes taken from
the live API: no startRow or startColumn at zero, {} for an empty cell,
numbers parsed under USER_ENTERED, row and column metadata, tableRange on
append, and the whole SheetProperties in an addSheet reply.
* fix(shell): mask a byte escape and keep a non-BMP character whole
Three octal digits reach past one byte, and bash writes the low byte of
those. The sentinel mapped the whole value instead, so 256..511 landed on
U+DD00..U+DDFF, outside the range surrogateescape can encode: printf
'\400' raised UnicodeEncodeError rather than printing a NUL. Mask to one
byte before building the sentinel.
In TypeScript the sentinel range also swallowed the low half of an
ordinary surrogate pair, so a non-BMP character with a low surrogate in
U+DC80..U+DCFF (U+10080 is D800 DC80) came out as U+FFFD plus a raw 0x80
even with no escape in sight. Only a lone low surrogate is a byte now.
The non-BMP integ case named U+1F600, whose surrogates are D83D DE00 and
never were in the sentinel range, so it passed against the broken
encoder. It names U+10080 instead, with an echo case and a case that
mixes the character with a byte escape.
Pinned against bash 5.2 in a UTF-8 locale.
* test(gws): match live Sheets on grid size and cell typing
The mock reported a fixed 1000x26 grid, so a tab holding more rows than
that claimed to hold 1000 while carrying more, and handed out one
rowMetadata entry for only the first 1000. Live grows the grid to what
was written, so gridProperties, rowMetadata and columnMetadata all read
from one extent now: 1313 written rows report 1313.
Cell typing went through Number(), which is looser than USER_ENTERED: a
whitespace-only cell became numeric zero and 0x10 became 16, where live
keeps both as strings. A plain decimal test replaces it. TRUE is a
boolean rather than a string, and formattedValue is the rendered value
rather than the raw input, so 007 reports "7" and 4.50 reports "4.5". A
number typed with an exponent keeps a scientific format, so 1e3 reports
"1.00E+03".
Verified against the live API: 18 typing cases and 8 exponent cases
match exactly. Currency, percent, thousands-separated and date-shaped
cells stay strings and say so in a comment, since they need locale-aware
number formats.
The fixture seeds a boolean, a hex-looking string and an exponent so the
typing is asserted through the mount.
* feat(jq): support the rest of jq's flag surface
Only -r, -c and -s were accepted, with no long forms at all. Adds
-n/-R/-j/-a/-S/-e/-M/-f/-h, every long spelling, --raw-output0, --tab,
--indent, --unbuffered, --arg, --argjson, --rawfile, --slurpfile,
--args, --jsonargs, --stream and --seq, in Python and TypeScript.
Two spec-parser additions carry it: Option(pair=True) for two-token
options like --arg name value, and Operand(text_when=...) so --args
turns later operands into positional strings instead of input files.
Also fixes -s to slurp across every operand rather than per file, and
an empty input to print nothing instead of erroring.
* fix(jq): read inputs as a call, not as the word
The detector matched `.inputs`, `{inputs}`, `$inputs`, a module member
and any string or comment, and a match switches the run into drain mode,
so `jq -c '.inputs'` over a two-document stream printed one line where
jq prints two. Both languages now blank string bodies, comments and
object-shorthand keys before searching for the token, keeping
interpolations as code. $ARGS is read the same way.
The native fixture also tolerates EPIPE on the child's stdin: `jq -n`
exits without reading, which raced the fixture's write and failed the
whole node package on unhandled errors with every test passing.
Adds integ coverage for the rest of the flag surface, including the long
spellings, the refused options and the usage errors.
* feat(himalaya): align the CLI with the upstream pimalaya grammar
- mailbox is -m/--mailbox, message id is a positional operand
- envelope search with upstream's query DSL, replacing list filter flags
- message compose writes RFC 5322 to stdout unless --send; message send
takes raw MIME, so the two chain through a pipe
- upstream aliases: ls, sr, write, new, fwd
core/email/send.* is deleted: the email mount is read-only, so the
composer and the SMTP transport moved into the himalaya package.
Fixes three node-side bugs found while testing:
- sendMail({raw}) needs an explicit envelope; Bcc is stripped by hand
- parseSearchCriteria silently dropped unknown keys, so NOT SEEN matched
every message
- parseImapDate rolled a bad month or day into a wrong date
* fix(himalaya): satisfy the CI eslint rules and stop the yapf/isort import flip
* fix(himalaya): search the Date header and bound envelope pages
Date conditions emit SENTON/SENTBEFORE/SENTSINCE rather than
ON/BEFORE/SINCE, which match the mailbox internal date, and the missing
upstream `before` condition is now parsed.
Envelope listing asked IMAP for every matching uid and fetched headers
for all of them before slicing one page. It now fetches the newest
page * page_size, capped by the account's max_messages.
* test(himalaya): cover every flag in integ, and fix what it found
Grows the himalaya integ suite from 38 to 64 cases: every search
condition, sorter, flag and alias, the composer flags, both quoting
options, stdin bodies, the send operand form, unknown flags and a
missing mailbox.
Three bugs it turned up, all in the email client:
- Five IMAP selects ignored their response, so a missing mailbox left
the session in AUTH and the next command complained about that
instead ('command SEARCH illegal in state AUTH' in python, imapflow's
bare 'Command failed' in typescript). Both now say no such mailbox.
- A refused SEARCH returned an empty uid list, so criteria the server
cannot answer read as 'matched nothing'. It now raises.
- Found because a mutation emitting a malformed date stayed green.
* feat(shell): discover installed CLIs through man, type and which
man renders an installed CLI from its own spec, type reports it as a cli, and which is new. Precedence now lives in one lazy layers() generator that serves both route (the winner) and route_all (every layer, which type -a prints).
* refactor(shell): share one option scanner across the bash builtins
command, type and which each hand-rolled the same non-permuting letter
scan. They now share scan_options/last_of, which pins bash's grammar in
one place: a long spelling refuses on its second dash, and a mutually
exclusive group resolves to the last letter typed.
type -f is a filter over the layer list instead of a pop-and-restore on
the session function table, and man takes the install it already looked
up rather than fetching it again.
* fix(shell): keep the layers under a reserved word visible
type -a stopped at the keyword, so a function sharing a reserved word's
name never showed. bash prints both lines (function time { :; }; type -a
time), and mirage's parser lets any reserved word be a function name, so
the shadow was reachable and hidden.
time and coproc leave the keyword table with it: mirage implements
neither, so type called them keywords while running one reported command
not found. which now drops the keyword layer before picking a winner
rather than after, so it reports the function underneath instead of
nothing.
* refactor(shell): split lookup into a package the way route and condition are
types.py holds NameKind, constants.py the consumer and description
tables, classify.py the layer walk, handle.py the two builtins. The
keyword pool moves to the name-pool leaf beside SHELL_NAMES, where the
CLI registry can read it: installing a CLI under a reserved word was
allowed and would never have been reachable, and is now refused.
command.py and lookup build their result triples with the shared
helpers instead of by hand, and man renders its missing-description
placeholder in one place.
* fix(shell): keep command -V's diagnostics when another name resolved
The shared-helper refactor routed the any-found case through ok(),
which drops stderr, so command -V ls nope printed the found line and
swallowed 'command: nope: not found'. bash prints both and exits 0
(pinned), and the TypeScript sibling always did.
All three handlers now build one result with a computed exit code, so
the diagnostics can never ride on the status again, and the mixed case
is a test in both languages.
`tac` is GNU coreutils; macOS ships `tail -r` instead. The native fixtures
capture only the child's stdout, so on a host without `tac` the native side
silently yields "" and the comparison fails with a confusing diff rather than
reporting the missing binary.
Guard both sides on the binary's presence rather than the platform, so the
tests still run wherever `tac` exists (Linux CI, or macOS with coreutils
installed) and skip cleanly where it does not:
- typescript/packages/node/src/commands/native_tac.test.ts: `it.skipIf`
keyed off `command -v tac`, mirroring native_fmt.test.ts's skipIf shape.
- python/tests/commands/native/test_tac.py: `pytest.mark.skipif` on
`shutil.which("tac") is None`, mirroring test_fmt.py's named-mark shape.
Verified both directions on macOS: 4 TS / 6 py tests skip with no `tac` on
PATH, and all 10 pass with a GNU-equivalent `tac` shim on PATH.
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): slack, discord, ntn and linear as builtin CLI packages
Migrate the remaining service bundles to installed CLIs with the settled
vocabularies (OpenClaw slack/discord actions, the official Notion CLI
grammar as ntn, linear's noun/verb tree), including the new API surface
they need (slack pins/emoji/reactions/history, discord edit/delete/
threads/polls, notion page updates). Configs move resource -> core in
both languages and the mounts go filesystem-only. chroma-query stays a
mount command on purpose: it is path-scoped.
Fixes two CLI dispatcher bugs found by the new integ cases in both
languages: leaf exceptions now become the command's IOResult (so
redirects and ; sequencing work, with the GNU prog: prefix), and argv
words re-enter the walk as typed so quoted glob-looking flag values
survive.
Adds cli-facet integ targets and cases for all four CLIs (108 cases
both runners), extends the fake slack/discord/notion servers, and moves
CLI usage out of the resource docs into new CLI sections on both tabs.
* integ: cover the remaining linear write verbs in the cli facet
* feat(cli): linear add-label and set-project resolve names via the issue's team
* fix(cli): codex review, standalone thread type and bounded notion search pagination
* fix(crossmount): honor wc --total and stop du -h rounding twice
The FANOUT combiners re-parse each native run's rendered text, so a flag
that changes output shape has to be re-implemented in the combiner or it
diverges from the single-mount path. Two did.
wc ignored --total entirely. Worse, it guessed that a run with more than
one row ended in a per-run total and dropped that row, so a glob operand
silently lost a file. Native runs are now forced to --total=never and
combine_wc delegates to the generic's own format_count_rows, which is
what applies --total for one mount.
du -c -h fed each run's already humanized total back through parse_size,
so two 1500 byte operands read as 1536 each and printed 3.0K where one
mount prints 2.9K. Native runs now report exact bytes and du_total
humanizes once.
That removed the last caller of utils/formatting.parse_size in both
languages, so it and the orphaned SIZE_UNITS constant are gone.
* fix(crossmount): diagnose an invalid wc --total before the override
Forcing --total=never on the native runs also hid a bad value from
them, so `wc --total=bogus` across mounts ran clean. Python then raised
out of the combiner and an upstream handler happened to render it,
while TypeScript returned null and exited 0 with no diagnostic at all.
Both now parse the user's flags in run_fanout before overriding, and
return the GNU message with exit 1, which is what one mount does.
* feat(policy): absorb output safeguards into the policy layer as Limit
* fix(policy): loop-based trailing-slash strip in limitOverride (CodeQL polynomial-redos)
* fix(integ): surface policy EACCES through the TS fuse read/write callbacks, skip code-policy cases in the CLI harness
* chore(deps): bump cryptography 50.0.0 and aiohttp 3.14.3 for audit advisories
* fix(policy): stamp builtin producers at the dispatch chokepoint, purge safeguard vocabulary from docs
* fix(google): surface Shared Drives in the g* mounts, and make the Graph service root configurable
The g* mounts sent no corpora flags, so gdocs/gsheets/gslides silently
missed every Shared Drive file while gdrive showed them. They now search
every corpus, and report Drive's incompleteSearch flag so a short listing
is never cached as the directory. gdrive's root had the same bug already:
a failed Shared Drive enumeration was cached as a My-Drive-only root.
GRAPH_API stops being a module constant. The service root now comes from
the mount config, so a mount can address a national cloud, and the integ
fake is reached by configuring a mount instead of rebinding a global in
every module that spells a URL. OneDrive also gains /groups/{id}/drive
and /users/{id}/drive addressing, with a validator rejecting more than
one drive target.
Closes#305, #308, #309.
* fix(msgraph): escape Graph identifiers, and drop the national cloud table
Two Codex findings on #695, plus a scope cut.
Escaping: drive_base interpolated identifiers raw, so a guest user's UPN
(guest_contoso.com#EXT#@fabrikam.com) had its `#` read as a URL fragment
and Graph received a truncated /users path. id_segment now escapes each
identifier as one path segment, applied in drive_base, SharePoint's
item_url and its site drive listing. TypeScript already did this.
The safe set is "!*'()" rather than an empty one so quote matches
encodeURIComponent exactly. Drive ids are b!<base64url>, and the ref
paths built off them go into a JSON body Graph reads literally, so
escaping the `!` on one side only would have traded a guest-user bug for
a copy and rename bug on every drive-id mount.
Docs: the OneDrive example used GraphCloud without importing it, and it
was not re-exported from the documented module. That example is gone
with the enum below, so the broken import goes with it.
National clouds: the GraphCloud enum, GRAPH_CLOUD_HOSTS and the `cloud`
field are removed from both languages. mirage cannot verify those four
hostnames, and a table of unverifiable constants reads as a guarantee
the library cannot make. graph_base_url stays and is now the single way
to name a non-worldwide root, which is what #305 actually asked for: the
service root is read from config rather than a module constant, so two
mounts in one process can address different deployments and the integ
fake is reached by configuring a mount. The docs now point at Microsoft's
own national cloud documentation for the root.
* fix(find): classify the start point once, above every backend
`find` asked the backend to walk a start point that is not a directory,
and each answered differently: a Graph 404, a silently empty listing, or
ENOTDIR. The reported symptom was `find -L <cross-mount link>`, but the
operand rewrite was correct all along: `find /data/a.txt`, with no
symlink anywhere, failed identically.
Every native find op listed `<start>/children` unconditionally and
hardcoded the start as `kind="d"`. Extends the shape #688 built for
symlink start points so the start point is classified once and every
backend inherits the answer: a symlink goes to link_results, a
non-directory is reported and walked no further, a directory walks.
The stat rides the op dispatcher as a new `stat_path` fact rather than a
threaded callable, for the reason #688 recorded when the same threading
left `hf` behind, and because the router resolves `find -L` into another
mount before the command runs. Only a positive non-directory answer
short-circuits: a stat that sees nothing is not proof of absence when a
backend's directories are implicit.
Also fixed, found on the way:
- `find /d \( -name x \)` classified `(` and `)` as PATH operands, giving
find two phantom start points, hidden because the native path reads
paths[0] and drops the rest
- `tree <file>` diverged three ways by backend; GNU prints the marker,
counts the file and exits 0
- Python's missing-operand message printed the path twice instead of
`No such file or directory`
Fixes#684, which this change needed: gdrive was the one native-find
backend that could not be tested at all, so `find` there reached the
real Drive API. Widens the fake's patch targets to resolve/tree/rename,
teaches it the `name=` filter, and makes an unresolvable target a hard
error. The e2e s3 fake needed `list_objects_v2` for the same reason.
Pinned against GNU findutils 4.10.0 and tree 2.2.1 on debian:stable-slim
before changing behavior. Mirrored in Python and TypeScript.
* fix(find): decide absence above the backend, not per backend
`find <missing>` exited 1 only on the three backends that wire a stat
into find, and `tree <missing>` rendered GNU's marker on twelve of
eighteen targets. Both were the same gap: a backend could not tell
"absent" from "a directory that exists only as its children", so
neither command could either.
The rule already existed in the repo. `test -d` has always asked both
channels a backend can answer on, and only both coming back empty means
nothing is there. `resolve_path_stat` / `resolvePathStat` is now that one
implementation, shared by the start-point probe and `path_kind`, so a
traversal and `test -d` cannot disagree about whether a path exists.
Measured on 18 targets before changing anything: an implicit directory
answers, a missing path does not, and an empty `mkdir`'d directory
answers everywhere except hf, whose `mkdir` creates nothing and which
`root_create_mkdir` already excludes for that reason.
Two pre-existing fixture bugs the uniform rule exposed, both of which a
silent `find` had been hiding:
- TypeScript's integration fixture seeded `store.files` without the
ancestor directories, a store no write can produce. RAM keeps
directories explicitly, so `ls /data/logs` printed nothing and
`test -d /data/logs` was false there already. The python mirror
fixture has always `mkdir`'d each ancestor; now both do.
- The nextcloud exclusion was missing from `tree_start_missing`'s
target list because it needs a real server, which is what turned
`integ-data` and `integ-ts` red.
Verified against a local `nextcloud:30-apache`, the image CI uses.
aiohttp <= 3.14.1 accepts compressed WebSocket frames without negotiated
permessage-deflate (CVE-2026-59881, moderate). Fixed in 3.14.2; the lock
moves to 3.14.3, and the pyproject floor is raised so the fix cannot be
resolved away.
Clears the red audit job.
* fix(symlinks): report and follow links the way GNU does
Symlinks live in the namespace, so no backend readdir or stat can see
them and every command was blind to them unless wired by hand. Six GNU
divergences, each pinned against debian:stable-slim first, mirrored in
python and typescript.
- ls -l on a dangling link failed the whole listing with exit 2; GNU
prints the link row and exits 0. ls -l/-d on a live link silently
followed it, and ls -R and -F omitted links entirely.
- readlink -e printed a dangling target and exited 0; GNU requires the
whole resolved path to exist. -f now requires only the parent.
- file followed the link and relabelled the operand as its target; GNU
prints "symbolic link to <target verbatim>", or "broken symbolic link
to ..." when the target is absent.
- du omitted links; GNU lists one line per link under -a and does not
follow a link operand without -L.
- find followed a symlink start point; GNU's default is -P, which
reports the link and stops. -P/-H/-L are leading options, not
predicates, so the expression splitter consumes them first.
- -L has to apply below the operand too, not just to it: ls -L stats
each link child and reports the target under the link's own name, and
du -L stops counting links as entries of their own.
Generalize the seam while it is being touched:
- A command opts in by naming a `links: LinkView` parameter. execute_cmd
offers the fact to every handler and accepts_kwarg reads the signature
to decide delivery, so there is no allowlist to keep in step. This also
removes the hardcoded stat_overlay command list. TypeScript needs no
equivalent: a command that does not read `links` off its context
ignores it.
- LinkView bundles all five link facts, so a new need is a field read
rather than a keyword threaded through four layers.
- Link merging lives in the generic above the native-op/walk fork, so a
mount's symlink behavior cannot depend on whether its backend ships a
native find or du op.
- ls merges links as stat rows instead of appending to its own stdout,
which is what makes -R, -F, -l and sorting work. Deletes inject_links
and its latent dedup bug, where splitting a GNU long row on tabs
returned the whole line.
Also moves data shapes and constant tables to the modules the layout
calls for (ops/types, commands/builtin/constants, utils/constants) in
both languages, and adds 33 integ cases that run in both runners.
du sizes a link at its target length, which is not a divergence: mirage
counts bytes, GNU's --apparent-size mode, and there GNU reports
len(target) too. Two known gaps, both documented rather than papered over: du -L
undercounts a link pointing outside the operand's own subtree, and ls
-RL does not descend a directory link. A version of the latter that only
worked on RAM was removed rather than shipped, because descending needs
the child read re-keyed onto the resolved target and that only resolves
on backends with real directory entries. Symlink behavior that depends
on which backend is mounted is the exact failure this branch adds a rule
against.
A link operand is caught on both readdir shapes. Backends without real
directories (s3, nextcloud) answer readdir on a link path with an empty
list rather than raising, so the link row has to be resolved there too
or the operand renders as an empty directory. Pinned by unit tests
against the generic in both languages, so the case is covered without
needing a remote backend to reproduce it.
Fixes#414
* fix(integ): drop the empty-directory dependency from the symlink fixtures
* fix(symlinks): give bespoke stat wrappers the link table, and make find's link options last-wins
s3 and gridfs ship their own stat command, and neither declared the
links parameter, so signature-based injection silently withheld the
namespace link table and stat on a symlink reported ENOENT on exactly
those two backends. TypeScript is unaffected: its wrappers forward the
whole opts object, so the generic reads opts.links regardless.
find states its link policy as a leading option and GNU takes the last
one, so find -L -P x must not follow while find -P -L x must. The
scanner returned true on seeing any -L or -H anywhere.
* feat(metadata): walk the subtree for chmod, chown and chgrp -R
All three refused -R with exit 2. GNU splits three ways here: chmod -R
skips a traversed link but follows a command-line link to a directory,
while chown -R and chgrp -R change the link node itself because POSIX
gives -R an implicit -P.
Two bugs surfaced while pinning it. chown -h wrote uid onto the node but
link_stat dropped ownership when rendering, so nothing ever showed it;
only mode is meaningless on a link. And TypeScript rebuilt stat rows
field by field where Python uses model_copy, dropping uid/gid on the
link row, so FileStat gains a with() method and the four rename sites
use it.
* fix(find): classify a link by its target under -L
find -L must test a link as what it points at, not as a link: a link to
a file is -type f, a link to a directory is -type d, and only a dangling
link stays -type l. link_results classified every namespace link as l
regardless, so find -L d -type f missed the links and find -L d -type l
wrongly named the live ones.
Both find paths reach link_results, so the classification lands above
the native-op/walk fork rather than in one branch.
* fix(symlinks): stop backends losing links and -L one wrapper at a time
`find -L` classifies a link by statting its target, and that stat was
reached through the callable wired for the -mtime post-filter, which is
gated on ops.local. Post-filtering costs one stat per result, so gating
it is right; classification costs one stat per link, so gating it was
not. Off the local backends nothing was statted and every link stayed
typed `l`: -type f and -type d missed them, -type l still listed them.
The fact belongs on LinkView, not on a threaded argument, so it is now
`target_stat` / `targetStat`. It resolves through the op dispatcher
rather than a backend key, which also fixes a link pointing into another
mount: the old code built the key from the search root's mount prefix
and could never resolve one. Pinned in integ/crossmount/find/deref.json.
The same omission then turned up in ten bespoke commands that shadow a
link-aware generic without declaring `links`, and in the same ten for
`-L`, which lands in the wrapper's opaque flag bag when unnamed. Both
failures run fine and exit 0, so nothing notices until someone makes a
link on that backend. tests/commands/test_links_optin.py asserts it
instead, deriving from the builders which link parameters each family
requires; it caught history/ls on the first run.
TypeScript needs no guard because wrappers forward the whole opts
object. Its one exception, email/find, walks its own tree and never
reaches a generic, so it now calls linkResults directly.
* fix(find): trim slashes without a regex, and derive the link guard from the router
Exporting linkResults made its input library-reachable, which surfaced
the polynomial-ReDoS trim inside it (CodeQL js/polynomial-redos, high).
Both `/^\/+|\/+$/g` trims now use the loop-based stripSlash helper.
The guard's list of required link options was hardcoded. It now reads
DEREFERENCE_FLAGS and LAST_WINS_LINK_OPTIONS, so a new link option
taught to the router starts being required of bespoke wrappers too.
* fix(symlinks): let real backend failures surface while statting a link target
link_target_stat caught OSError around _stat_or_none, which already maps
the missing-target case to None. The extra catch could therefore only
swallow a genuine failure: a permission, connection or I/O error came
back as a dangling link, printed as -type l with exit 0.
Only the two legitimate no-target cases stay None now, a loop from
namespace.follow and a missing target from _stat_or_none. Everything
else propagates. Same in TypeScript, where the bare catch did the same.
* feat(policy): policy package with the pre_command hook, absorbing the command guards
* refactor(policy): GuardSpec is a data shape, move it into types
* docs(policy): trim the seam narration from the Policy base docstring
* fix(policy): fire the hook at the dispatch chokepoint, cover flag paths, harden TS spec handling
* fix(policy): ln refusal wording follows the link kind
* chore(policy): keep hasSymlinkFlag module-private
* feat(spec): merge value_kind and type into one ValueType axis (python)
* feat(spec): merge value_kind and type into one ValueType axis (typescript + dumps)
* test(spec): float coverage, lint fixpoint
* fix(spec): linear-time FLOAT_VALUE regex (CodeQL polynomial-redos)
* fix(spec): numeric refusal before choices in option_error, add FlagView.as_float
* feat(cli): tree walk with git-pinned diagnostics
* fix(cli): narrow instead of assert, CI formatter alignment
* refactor(cli): one help renderer, --help as a registered option
* fix(cli): walk option precedence per flat parser, module layout
* fix(mv): refuse a non-empty destination directory, not only under -T
GNU mv refuses to replace a non-empty directory whether the target was
named outright with -T or mapped under an existing destination
directory. The refusal existed but was gated on no_target_dir, so
`mv src dest/` merged the trees at exit 0 instead (#272).
Dropping the gate also required moving the check after the clobber
gate: GNU lets -n and --update=none skip such a target silently at
exit 0, which the old ordering turned into an error. Pinned against
coreutils 9.7 in debian:stable-slim, all seven arms:
mv src dest/ -> cannot overwrite 'dest/src': Directory not empty
mv -T src dest/src -> same
mv -f src dest/ -> same (-f does not bypass)
mv -n src dest/ -> exit 0, skipped
mv -n -T src dest/src -> exit 0, skipped
mv src dest/ -> exit 0 when dest/src is empty
cp -r src dest/ -> exit 0, still merges
The -b arms keep working through backup_displaces, which renames the
target aside before the refusal can apply.
* fix(crossmount): compare storage identity, not path spelling, in cp/mv
Two mounts can address one store: two disk mounts on a shared root, one
bucket mounted twice, or the same resource object mounted at two
prefixes. Cross-mount routing keys off the mount prefix alone, so such a
pair looked like a genuine cross-mount move and mv did read -> write ->
unlink, writing the object over itself and then deleting it (#154):
$ mv /m1/x.txt /m2/x.txt exit 0
$ cat /m2/x.txt No such file or directory
$ find /m1 /m1
The generic cp/mv already had a same-file guard and a backend_key hook
for it, but nothing injected a key and the default is the mount-relative
path, which carries no mount identity. Resources now answer
storage_id(); make_storage_key pairs it with the mount-relative path and
the executor threads it through route -> relay -> cp/mv.
The default is per-instance identity, so a backend that says nothing
keeps today's behavior. That is the safe direction: a false "different"
only fails to catch an alias, while a false "same" would refuse a real
move. disk (resolved root), s3 (endpoint, bucket, key prefix) and redis
(url, key prefix) override it, so two separately built instances over
one target still compare equal.
Verified: aliased mounts refuse with the coreutils "are the same file"
line and the bytes survive; distinct stores sharing a basename, distinct
paths within one store, and two ram mounts all still move normally.
* fix(crossmount): mirror the storage-identity guard in typescript
Same shape as the python side: Resource gains an optional storageId,
BaseResource defaults it to a per-instance serial (JS has no identity
primitive), and disk, s3 and redis override it with the config that
actually pins their storage. makeStorageKey pairs it with the
mount-relative path and command.ts threads it through
handleCrossMount -> runRelay -> runCp/runMv.
storageId is optional on the interface because some resources implement
Resource without extending BaseResource; a resource that omits it is
keyed by its mount prefix, which reproduces the pre-existing behavior
rather than risking a false "same file".
* fix(s3): guard a same-key copy and rename in the core, py+ts
core.rename was copy_object + an unconditional delete_object, and
core.copy a plain copy_object, neither checking whether the resolved
keys were equal. Safety came from the remote refusing the request
rather than from our code (#150).
Both now follow POSIX rename(2) for the equal-key case: succeed and
perform no other action, with a missing source still raising ENOENT.
What the probing actually showed, which corrects the issue's premise:
- MinIO REJECTS a plain self-copy, versioned or not, exactly like AWS
(InvalidRequest). It is not one of the lenient stores the issue
names, so the data loss is not reachable there today.
- But a self-copy IS accepted once the request carries
MetadataDirective=REPLACE, and the delete then removes the object.
Confirmed against MinIO: on a versioned bucket the old version
survives behind a delete marker, on a plain bucket it is gone. So
the loss is one plausible code change away.
Coverage: the s3 mock's copy_object is deliberately lenient, which is
what makes the guard observable; a real store's rejection would mask
it either way. The mock now counts copy_object/delete_object so the
test asserts no write was issued at all, not merely that the object
survived. Confirmed discriminating: disabling the guard fails four of
the seven, one of them literally on an emptied store.
* test(integ): cover the non-empty-dir refusal and aliased mounts
Two overwrite cases for #272, on the seed that already builds the
shape: /data/ow/t holds two files and /data/ow/s/t is the source, so
`mv /data/ow/s/t /data/ow` maps onto a non-empty target. One asserts
the refusal, one asserts -n skips it at exit 0. Both pinned against
coreutils 9.7 replaying the same state in debian:stable-slim.
For #154 the harness could not express the shape at all: every builder
allocates fresh storage, so no two mounts could address one store. A
mount may now carry alias_of naming an already-built mount, reused in
both runners (spelled snake_case, matching seed_root). The new
ram-alias target mounts one RAMResource at /data and /alias, and
alias_both_prefixes_see_one_store proves the aliasing is real before
the guard cases assert anything.
ram, disk, ram-alias, ram-root: 4074 passed on the python host, 4069
on typescript-node, 0 failed on both.
* test(ts): unit-cover the storage-identity key
Mirrors tests/workspace/mount/test_storage.py: aliased prefixes compare
equal, distinct stores do not, a config-pinned resource matches across
instances, the ancestor prefix boundary survives, and a resource that
declares no storageId falls back to per-mount identity rather than
risking a false same-file refusal.
* chore: apply yapf formatting and satisfy restrict-template-expressions
yapf reflowed the three crossmount relay signatures and the new s3
guard test. The eslint fix is real, not cosmetic: storageId
interpolated a number straight into a template literal, which
@typescript-eslint/restrict-template-expressions rejects.
* fix(crossmount): trim slashes with rstripSlash, not a backtracking regex
CodeQL flagged all three `/\/+$/` trims in makeStorageKey as
js/polynomial-redos (high): the pattern backtracks on a path with many
repeated slashes, and the input is a caller-supplied virtual path.
utils/slash.ts already exports the loop-based rstripSlash for exactly
this reason. Same result, linear time, and it matches the trim idiom
used elsewhere in the codebase.
The python side was never affected: it uses str.rstrip('/').
* fix(crossmount): canonicalize the storage key, keep identity for bare resources
Three review findings, two of them the same data loss the guard was
meant to close.
**Nested backings collapsed onto one key.** The key joined the storage
id and the mount-relative path with a delimiter, so overlapping roots
stayed apart:
/a -> /srv/data, /b -> /srv/data/sub
/a/sub/x.txt -> disk:/srv/data:/sub/x.txt
/b/x.txt -> disk:/srv/data/sub:/x.txt
Same file, different keys, so mv wrote it onto itself and unlinked it.
Reproduced: exit 0 and the file gone from disk. The two parts are now
concatenated into one path-like string, so both render as
disk:/srv/data/sub/x.txt. s3 and redis had the same flaw with nested
key prefixes and now join theirs path-like too. A sibling that merely
shares a name prefix (/srv/data vs /srv/dataX) still stays distinct,
since the relative part always starts with a slash.
**Bare resources lost object identity.** The typescript fallback keyed
a resource with no storageId by its mount prefix, which handed one
object two identities. Browser resources (browser s3, opfs) implement
Resource directly rather than extending BaseResource, so they took
that path and a self-move still relayed a write then an unlink. A
WeakMap now assigns a serial per object, so the fix is not Node-only.
Python gained the matching getattr fallback for a resource that does
not inherit BaseResource.
**Nested function.** storage_key was a closure; the repo prohibits
nested functions. It is module level now, bound with functools.partial.
* feat(cli): recursive CLISpec atop CommandSpec
* fix(cli): CI formatter and lint fixes for the new tests
* fix(cli): whitespace-aware name check, validation into compile module
* style: yapf formatting per CI env
* refactor(workspace): split the Workspace orchestration into a workspace package in both languages
* review: type the workspace dispatch callback as a DispatchFn protocol
* fix(shell): honor line continuations and reject unterminated backticks
Two parser-level divergences from GNU bash, both pre-existing.
A trailing backslash is a line continuation. With nothing to continue
onto, bash drops it and runs the command, while mirage raised a syntax
error and exited 2, so `echo a\` produced nothing. Parity decides: only
an odd-length trailing run ends in a live continuation, since each
earlier pair is an escaped backslash. This also settles the last backtick
divergence, `echo `echo a\\`` now prints `a`.
An unterminated backtick was accepted and run. tree-sitter parses
`echo `echo a` as a complete command, so the region is scanned directly.
Quoting follows the shell reader: single quotes protect a backtick,
double quotes do not, and once inside a substitution only a backslash
escapes, which is why `"`echo '`'`"` is an error in bash rather than a
quoted backtick.
Also completes the stat/read size sweep across the last backends:
lancedb, chroma, qdrant, jaeger and langfuse .jsonl all hold the
invariant. notion page.json and langfuse trace/prompt .json are
size-unknown by design, since sizing them at listing time would cost an
extra API call per entry; both are now pinned as cold/read/warm cases and
carry a comment saying why, which is what was missing when I mistook the
same shape on trello for a bug.
* test(jaeger): scope the size invariant to the seeded services
CI caught one mismatch the local run never hit. Jaeger self-instruments,
so /j/services/jaeger/* is live telemetry that keeps growing: a trace
listed at time T can gain spans before it is read, and stat then
disagrees with the read through no fault of the sizing.
Walk only the four seeded services, named explicitly so self-telemetry
cannot drift back in, and assert an exact 8/8 so an empty glob fails
instead of passing vacuously. 101 of the 109 files in that tree were
self-telemetry.
* feat(spec): declarative count, multiple, choices, required and default option fields
* fix(types): widen fanout depth-flag helper to the int-carrying flag union
* fix(spec): a default on a multiple option lands as a one-element list
* fix(du,opfs,find): pair the du ops, move opfs to the factory, cover find predicates
Three issues that all sat on the generic command layer.
optional fields, but they are not independent. A backend setting only
the cheaper du_size silently degraded du to one operand line with no
directory rows and an inert -a. They are now one value (DuOps in
python, DuOps<A> in typescript) with both halves required, so half a
native du cannot be constructed. That makes compute_entries required
through run_du/runDu, so the degraded branch is deleted rather than
documented.
hand-registering 69 command families across 70 files. It now declares
a CommandIO like every other backend. opfs gains numfmt, od and
truncate, which it never had.
but only the malformed-value half was covered in integ. Adds nine
cases across trello, linear and discord: one precise positive filter
each, plus two unsatisfiable filters that must return empty. Those two
are the regression guard, since an ignored predicate returns the whole
tree.
* chore(spec): regenerate browser specs for the commands opfs gained
The factory gives opfs numfmt, od and truncate, which it did not
register by hand. Regenerated with scripts/gen_specs.py and
typescript/scripts/gen-specs.ts.
* test(sizes): assert the size invariant across linear, github_ci and trello
Extends the stat/read size sweep to the three remaining fake-server
backends. linear walks the whole mount, github_ci uses per-kind globs
because recursive find across runs is deliberately refused, and trello
covers its sized kinds.
trello's comments.jsonl is left size-unknown on purpose (it needs a
per-card actions call), so a cold stat reports 0 while a read returns
the real bytes. That is pinned as its own case rather than treated as a
failure, so the next reader does not chase it.
* fix(shell): keep whitespace between expansions and split adjacent backticks
Two tree-sitter tokenisation quirks made the expander drop or merge text.
Whitespace between two expansions inside double quotes was dropped, so
"$(a) $(b)" printed "ab". tree-sitter does not emit a whitespace-only run
as string content; it folds it into the following node, so the `$(` token
text is literally " $(". `${a}` and `$a` already compensated with their
own prefix handling, `$(...)`, `$((...))`, `$?`/`$#` and "${arr[@]}" did
not. One `_folded_whitespace` helper now covers every expansion branch.
Unquoted words never fold, so word splitting is unchanged.
Adjacent backtick substitutions were merged into one node when the gap
between them was empty or whitespace-only, so `a` `b` expanded to
"a` `echo b", quoted or not. The backtick region is now re-lexed from the
node's own text instead of trusting the grammar. Rewriting backticks to
$(...) before parsing would shift byte offsets, which history recording
and error snippets depend on.
Every expected value is pinned against GNU bash.
* fix(shell): respect backslash escapes when splitting a backtick region
Codex review on #677: a command substitution ending in `\\` before its
closing backtick has an escaped backslash, not an escaped backtick, so
the delimiter was being swallowed and the command text ran past it.
Consume escape pairs whole instead of counting slashes, gated to the
inside of a command and to the three characters POSIX escapes there
(`$`, backtick, backslash). Parity falls out: `\\` is consumed as one
unit, so a backtick straight after it still closes the region.
This also unescapes backtick bodies, which mirage never did, fixing a
divergence that predates the re-lex: `echo 'a\\'` in backticks now
prints a single backslash like GNU bash.
* Add a typed, spec-validated FlagView to TypeScript
Python reads command flags through FlagView, which takes the command's
spec and throws when asked for a name the spec does not declare.
TypeScript had no equivalent: commands reached into the raw dispatcher
bag with `flags.a === true`, so a name that no longer exists reads as
false rather than failing.
That failure mode is silent. Renaming tee's --append to --add in the
spec and running the suite shows it: with the raw reads tee's own tests
pass 7/7 and exit 0 while the flag has quietly become a no-op; with
FlagView the same rename fails 39 tests immediately.
Adds flagKwargName to constants (the dash-stripping rule the parser had
inlined, now shared rather than duplicated), specFlagNames and FlagView
to types, and migrates tee as a first consumer. Placement mirrors
Python: flag_kwarg_name in spec/constants.py, FlagView and
spec_flag_names in spec/types.py.
Deliberately not a sweep. The remaining commands are better migrated
alongside the parser's dest unification, which removes the both-name
reads a migration would otherwise write and then delete.
* Address Codex review: export FlagView, make asInt strict
FlagView was reachable only by deep import, so a node or browser command
could not use the sanctioned accessor. It now ships from both the spec
barrel and the package root alongside FlagValue, specFlagNames and
flagKwargName.
asInt used parseInt, which takes the numeric prefix of '5x' and returns
NaN for 'abc', and NaN still satisfies number, so a malformed value flowed
onward. It now validates the whole string and throws, matching Python's
int(): verified identical on '5x', 'abc', '', '1.5', ' 42 ', '-7', '+7'
and '1_0'.
* Migrate the first generic commands onto FlagView
13 commands now read flags only through the spec-validated view, and 9
more are partly across. Done in verified batches rather than one sweep:
each batch is typechecked and run against the full core suite, which
caught three bad rewrites (an insertion landing mid-expression in
unexpand and expand, one inside an object literal in tree, and reads
rewritten in basename with no declaration in scope because its command
is an arrow function).
Those all failed at the compiler, which is the argument for batching: a
single 53-file commit would have buried them.
* Migrate the bulk of the generic commands onto FlagView
35 of 45 flag-reading commands now go only through the view; 10 have
30 sites left that need hand edits.
Two real defects the batches caught, neither by tests alone:
split -d and -x, and mktemp -p, take an optional value, so they arrive
as boolean true when passed bare. Rewriting them to asStr dropped that
form and silently disabled numeric suffixes; split's own test caught it.
An audit of every asStr call against its option's valueKind found the
other three sites of the same class, now on raw().
stat's chained ternary was mangled by a partial regex match into
'(fl.asStr(c) ?? typeof) opts.flags.f', which only failed because it
would not parse. It is now a plain ?? chain.
Also removed an unreachable instanceof PathSpec branch in shuf: the
parser only ever stores strings, booleans and arrays in the flag bag.
* Finish the FlagView migration and enforce it with a lint rule
Every generic command now reads flags only through the spec-validated
view: zero direct opts.flags reads remain. The last sites were the list
shapes in awk and sed (Array.isArray(x) ? x : typeof x === 'string' ?
[x] : [] is exactly asList), comm's and join's numeric-keyed selectors,
and xxd's local toInt over the raw union.
The lint rule is the point of going to 100%. Partial adoption protects
only the migrated files; banning opts.flags outright makes drift
impossible for commands nobody has written yet. Its selector is scoped
to opts.flags rather than any .flags member, because a first attempt
flagged RegExp.prototype.flags in zgrep. Mutation-checked: putting one
raw read back in tee fails lint.
* Guard the Python side against raw flag reads too
Python needed no migration: its wrappers already bind flags to typed
parameters and use FlagView for the remaining spellings, so the generic
commands never see a bag at all. Auditing the whole command tree found
zero raw reads.
What it lacked was the enforcement TypeScript just gained, so this adds
the mirror of the lint rule as a test. Reads only: crossmount fanout
legitimately assigns into a bag it builds for the sub-commands it
dispatches, which a first cut flagged. Mutation-checked by adding one
flags.get back to tac.
* Apply yapf formatting to the flag-read guard
The hook rewraps the assert message. My earlier local run reported clean
before this file's last edit, so the reformat only surfaced in CI.
* refactor(spec): compile specs once and unify flag dests onto the long spelling
parse_command rebuilt its lookup tables from the spec on every
invocation; they now compile once per spec into a cached CompiledSpec
(spec/compile.py), with attached-value candidates ordered longest-first
so -name can never lose a match to -n by set iteration order.
Both spellings of an option now land on one canonical dest (the long
name when both exist), so cp --update=all -u is last-wins without the
per-spelling mirror, sort -k1 --key=2 -k3 accumulates in true command
line order, and the both-name or-chains in 12 generics collapse to one
read. spec_flag_names returns canonical names only, so a stale short
read raises through FlagView instead of silently reading False; the
sites that raised are fixed here (11 or-chain generics, the wc
formatter helpers, fanout's head/tail header forcing, and the s3 mkdir
wrapper that read flags through signature params).
* refactor(spec): mirror CompiledSpec and dest unification to TypeScript
Same shape as the python change: compile.ts lowers a CommandSpec once
(WeakMap-cached) into the tables parseCommand walks, both spellings of
an option land on the canonical long dest, and specFlagNames returns
canonical names only so stale short reads throw through FlagView.
Sweeps the TS readers the canonical dests surface: the or-chain
generics, sort.ts's per-spelling alias helpers (whose short-then-long
list concatenation lost key order), the shared checksum and base64
helpers, fanout's head/tail header forcing, and the fallback-less
wrappers that read flags through signature params or bare short keys
(s3/gridfs/opfs mkdir, postgres head pushdown, mongodb tail -f,
curl -f).
Also fixes the same wrapper class on the python side, found by codex
review and the integ reds on the previous commit: curl -f exit 22,
gridfs mkdir -p/-v, mongodb tail -f, and head -c provisioning (the
parity mismatch: provision read 'c' while head now delivers 'bytes',
degrading the exact estimate to a range).
* fix(provision): head byte caps arrive as the canonical 'bytes' dest in TS too
Same fix as the python side: the head/tail provision read only 'c',
but head now delivers the canonical 'bytes', so head -c priced as a
0..size range instead of exact (the prov_head_c reds on integ-ts,
integ-shared-ts and integ-shared-parity). tail's -c stays short-only
and still arrives as 'c'.
* Add a typed, spec-validated FlagView to TypeScript
Python reads command flags through FlagView, which takes the command's
spec and throws when asked for a name the spec does not declare.
TypeScript had no equivalent: commands reached into the raw dispatcher
bag with `flags.a === true`, so a name that no longer exists reads as
false rather than failing.
That failure mode is silent. Renaming tee's --append to --add in the
spec and running the suite shows it: with the raw reads tee's own tests
pass 7/7 and exit 0 while the flag has quietly become a no-op; with
FlagView the same rename fails 39 tests immediately.
Adds flagKwargName to constants (the dash-stripping rule the parser had
inlined, now shared rather than duplicated), specFlagNames and FlagView
to types, and migrates tee as a first consumer. Placement mirrors
Python: flag_kwarg_name in spec/constants.py, FlagView and
spec_flag_names in spec/types.py.
Deliberately not a sweep. The remaining commands are better migrated
alongside the parser's dest unification, which removes the both-name
reads a migration would otherwise write and then delete.
* Address Codex review: export FlagView, make asInt strict
FlagView was reachable only by deep import, so a node or browser command
could not use the sanctioned accessor. It now ships from both the spec
barrel and the package root alongside FlagValue, specFlagNames and
flagKwargName.
asInt used parseInt, which takes the numeric prefix of '5x' and returns
NaN for 'abc', and NaN still satisfies number, so a malformed value flowed
onward. It now validates the whole string and throws, matching Python's
int(): verified identical on '5x', 'abc', '', '1.5', ' 42 ', '-7', '+7'
and '1_0'.
* Migrate the first generic commands onto FlagView
13 commands now read flags only through the spec-validated view, and 9
more are partly across. Done in verified batches rather than one sweep:
each batch is typechecked and run against the full core suite, which
caught three bad rewrites (an insertion landing mid-expression in
unexpand and expand, one inside an object literal in tree, and reads
rewritten in basename with no declaration in scope because its command
is an arrow function).
Those all failed at the compiler, which is the argument for batching: a
single 53-file commit would have buried them.
* Migrate the bulk of the generic commands onto FlagView
35 of 45 flag-reading commands now go only through the view; 10 have
30 sites left that need hand edits.
Two real defects the batches caught, neither by tests alone:
split -d and -x, and mktemp -p, take an optional value, so they arrive
as boolean true when passed bare. Rewriting them to asStr dropped that
form and silently disabled numeric suffixes; split's own test caught it.
An audit of every asStr call against its option's valueKind found the
other three sites of the same class, now on raw().
stat's chained ternary was mangled by a partial regex match into
'(fl.asStr(c) ?? typeof) opts.flags.f', which only failed because it
would not parse. It is now a plain ?? chain.
Also removed an unreachable instanceof PathSpec branch in shuf: the
parser only ever stores strings, booleans and arrays in the flag bag.
* Finish the FlagView migration and enforce it with a lint rule
Every generic command now reads flags only through the spec-validated
view: zero direct opts.flags reads remain. The last sites were the list
shapes in awk and sed (Array.isArray(x) ? x : typeof x === 'string' ?
[x] : [] is exactly asList), comm's and join's numeric-keyed selectors,
and xxd's local toInt over the raw union.
The lint rule is the point of going to 100%. Partial adoption protects
only the migrated files; banning opts.flags outright makes drift
impossible for commands nobody has written yet. Its selector is scoped
to opts.flags rather than any .flags member, because a first attempt
flagged RegExp.prototype.flags in zgrep. Mutation-checked: putting one
raw read back in tee fails lint.
* Guard the Python side against raw flag reads too
Python needed no migration: its wrappers already bind flags to typed
parameters and use FlagView for the remaining spellings, so the generic
commands never see a bag at all. Auditing the whole command tree found
zero raw reads.
What it lacked was the enforcement TypeScript just gained, so this adds
the mirror of the lint rule as a test. Reads only: crossmount fanout
legitimately assigns into a bag it builds for the sub-commands it
dispatches, which a first cut flagged. Mutation-checked by adding one
flags.get back to tac.
* Apply yapf formatting to the flag-read guard
The hook rewraps the assert message. My earlier local run reported clean
before this file's last edit, so the reformat only surfaced in CI.
* test(slack): assert the size invariant over every user and channel day
The users/*.json size comes from the users.list member, but read renders
the users.info response, so the advertised size is exact only while those
two endpoints encode identically. Verified live on 2026-08-01: every real
user in a live workspace matched byte for byte, key order included.
Pinning that verification in integ where it can be: two cases compare
stat -c %s against cat | wc -c across all 11 users and all 90 channel
days rather than pinning one constant for one file, and count the matches
so an unexpanded glob fails instead of passing silently.
Both comments now record what integ cannot cover: the fake builds
users.list and users.info from one row through one helper, so they agree
by construction and a Slack-side divergence would stay green.
* test(discord): assert the size invariant over every member, day and file
Same treatment as slack, one commit earlier. Three cases compare
stat -c %s against cat | wc -c across all 3 members, all 60 channel days
and every attachment, counting matches so an unexpanded glob fails
instead of passing silently.
The attachment case is the one that carries weight. members and
chat.jsonl are sized from the same endpoint read fetches, so they can
only drift by staleness, but an attachment's size comes from Discord's
reported attachment.size while read returns the downloaded bytes, which
is the same class of assumption github_ci artifacts needed checking for.
It also covers what absolute pins cannot: the rendered chat.jsonl for a
day holding an attachment embeds the CDN url, which carries the server's
port, so its byte count differs between an in-process ephemeral port and
CI's fixed one. Comparing stat to wc -c is port-independent.
* test(mail): assert the size invariant over every gmail and email message
Same treatment as slack and discord. Four cases compare stat -c %s
against cat | wc -c across all 11 gmail messages and 6 attachments, and
all 5 email messages and 2 attachments, counting matches so an
unexpanded glob fails instead of passing silently.
email is the one with a real divergence risk, and unlike the slack and
discord fakes the integ server can actually expose it. readdir sizes a
message from fetch_headers, which despite the name pulls BODY.PEEK[] and
parses it with _parse_multi_fetch; read pulls the same bytes through
fetch_message and parses them with parse_rfc822. Two parsers, one
renderer, so a dict that differs by one field or one key makes the
advertised size wrong. GreenMail is a real IMAP server, so these cases
exercise both parsers against real RFC822 bytes rather than a fixture
that agrees by construction.
gmail is lower risk: readdir and read both call messages.get with
format=full, so only staleness can drift them.
* Split the TypeScript builtin command specs by family
builtins.ts held all 93 command specs in one 1189 line table while
Python splits the same specs across nine family modules. The TypeScript
side now mirrors that layout: builtin_specs/ holds archive, fs_mutate,
hashing, listing, net, runtime, search, text_proc and viewing, with an
index that aggregates them and raises on a duplicate name the way
Python's __init__ does. builtins.ts keeps specOf and BUILTIN_SPECS, so
no caller changes.
The command to family assignment was taken from Python's modules rather
than reassigned by hand, and the two key sets already agreed exactly at
93 names.
This is a pure move. Regenerating the spec dump from the split code
produces byte identical JSON for all 93 commands in both the node and
browser outputs, which is the gate added in #666 doing the job it was
built for. The gate is not vacuous: dropping a single spec fails
generation with "no builtin spec: wget".
The null prototype freeze on BUILTIN_SPECS is preserved, so a name like
toString still misses rather than resolving an Object.prototype member.
* fix(spec): index the split spec modules without a type assertion
* Consolidate S3 provider configs and make credentials optional
Two cleanup items.
The 14 S3-compatible provider configs were each a hand-written copy of
the same credentials, endpoint rule and S3Config conversion. They now
share a base (s3_alias.py, s3_alias.ts) and declare only their endpoint
rule, which removes about 680 lines. Python had no test coverage for any
of them, so a characterization test went in first and the refactor
happened under it.
Credentials are now optional on every provider in both languages.
S3Config already left them optional, so omitting them falls through to
the usual AWS resolution order: profile, environment, shared credentials
file, instance role. Only r2 exposed that before, and even r2 required
keys alongside the profile it accepted, so the profile could never be
used on its own. session_token and aws_profile moved onto the shared
base, which also deleted the bespoke to_s3_config overrides on r2 and
supabase. Required is now bucket plus whatever the endpoint needs.
Also from the stale-debt pass: a resolved issue doc is deleted, the
TypeScript forwardIndex knob keeps its comment corrected to the real
reason it exists (WorkspaceFS reaches OpsRegistry directly and has no
equivalent of Python's Ops on_write hook), with index-invalidation tests
added in both languages for the dispatch and fs paths, and gen_specs.py
now fails loudly when a command module will not import instead of
silently dumping a spec set missing every optional backend.
* Ignore docs/plan/ so plan docs cannot be committed again
The gitignore covered docs/plans/ but not the singular docs/plan/, which
is how the one tracked plan doc got committed. That doc is deleted in
this branch; this closes the hole behind it.
* Fix eslint errors the local run missed
Two typed-lint errors that only surface once the TypeScript dists are
built: a forbidden non-null assertion in the index invalidation test and
a type parameter used once in the alias optional helper.
* Honor a named profile in the TypeScript S3 client
The alias configs forwarded aws_profile into S3Config but createS3Client
never read it, so a named profile was silently dropped and the default
credential chain signed with whatever it found, which can reach the
wrong account. The SDK takes profile as a per-client AWS_PROFILE, so
forwarding it is enough; explicit keys still win, matching the
precedence Python gets from aioboto3.Session(profile_name=...).
Found by Codex review on this PR. Pre-existing for r2, which already
forwarded the field, and made reachable everywhere by this branch.
* feat(sizes): render-at-readdir sizes for github_ci, trello and notion
Batch 2 of the fskit size push-down (plan: ~/Desktop/fskit.md). Each
listing already carries the payload read renders, so readdir seeds a
sized IndexEntry and stat serves it. Zero added API calls.
github_ci workflow/run/job JSON and trello workspace/board/list/card
JSON use the shared node-dir shape: the parent readdir set_dirs each
child with a sized node.json, and the child readdir consults list_dir
before bootstrapping the parent. Notion differs because a database dir
listing is dynamic, so database.json's size travels in
extra[database_json_size].
Artifacts stay class T: size_in_bytes is the zip's exact byte length,
confirmed against two live samples. Flags stay False for all three:
job logs, annotations, comments.jsonl and page.json are class N.
integ gains a github_ci target backed by a fixed Actions dataset in the
fake server, with 24 stat==read pins.
* feat(sizes): render-at-readdir sizes for jaeger, qdrant and lancedb
Batch 3 of the fskit size push-down. qdrant and lancedb flip
SIZES_ALWAYS_KNOWN; jaeger stays False because operations.json is class
C and traces are class P.
jaeger traces are genuinely P: verified live that the /api/traces search
document renders byte-identically to the by-id fetch for all four seeded
traces, including the one listed under two services. operations.json is
sized by one /api/operations call at the service dir readdir.
lancedb needed table_columns/tableColumns to widen the select beyond
ids; it excludes vector and blob columns in schema order so projected
rows render byte-identically. Blob entries stay unsized on purpose,
since sizing a blob at listing time would let one undecodable value fail
the whole directory listing rather than just that file's read.
A previously pinned integ expectation was wrong: jg_read_28 asserted du
-sh printed 0B, which was an artifact of unknown sizes, and now prints
the real total.
* feat(sizes): sizes for discord, email, langfuse and chroma
Batch 4, the last of the fskit size push-down. discord, email and chroma
flip SIZES_ALWAYS_KNOWN; langfuse stays False because traces and prompts
are class N.
chroma's producer-supplied size was measurably wrong for every file in
our own fixture: 180 declared against 166 rendered, 90 against 52, and
150 against 159, which truncated reads. It describes the source document
rather than the chunk join mirage serves, so it can never be the byte
length. It moved to extra.source_size and the real size is measured by
one batched chunk scan per directory, run lazily from stat and cached.
This also corrected provenance byte accounting and du totals.
langfuse items.jsonl follows the class C shape: the dataset dir readdir
makes the one call and seeds a sized entry.
Carries the discord fake-API arc. The new integ/server/discord_server.py
exposed four real backend bugs: message pagination used the oldest id as
the after cursor when discord answers newest-first, attachments were
classified as directories, unknown paths returned an empty listing
instead of ENOENT/ENOTDIR, and a cold ls of a guild dir ENOENTed in
python only.
* fix(sizes): propagate backend failures from the stat parent-listing fallback
Review follow-up on the stat fallback these three backends use to
populate a cold index: the TypeScript side caught every error from the
parent readdir and fell through to ENOENT, so an auth failure, a rate
limit or a transport error read back as "file not found". Python already
caught only FileNotFoundError. Now only isMissingPath errors fall
through, which is the rule that helper's own contract states.
Reachable through the new size lookups, since workspace.json and the
other sized files resolve through this path on a cold index.
Also pins that discord's base_url config key reaches DiscordResource as
baseUrl through the registry: normalizeFields snake-to-camels by default
so no explicit rename is needed, and a silent miss there would send
configured mounts to the public API.
* feat(spec): cross-language command spec parity check
Add scripts/check_spec_parity.py and wire it into CI next to the existing
spec drift gate. It diffs the generated python and typescript spec trees
command by command: every option (help text, value kind, repeatability,
shorthand), every operand, the _meta flags, and the resource set each
command registers under. Structural divergences live in
spec/parity_exceptions.json with a reason, and a stale entry fails the
check so an exception cannot outlive what it documents.
gen-specs.ts could only see command groups the package index re-exports,
so HISTORY_COMMANDS and GRIDFS_COMMANDS were silently missing from the
dump. It now asserts every builtin *_COMMANDS is reachable, and both
backends are exported.
The check found four real bugs:
shuf required a write op it only needs for -o. TypeScript marked the
builder write:true and threw before doing anything, so shuf did not
register on 21 read-only backends. Python registered it but eagerly
called ops.require(WRITE), so every invocation raised. Both now resolve
the write op lazily, matching sed -i / sort -o / uniq -o.
The TypeScript factory only gated on the generic write op, so truncate
registered on databricks_volume, dropbox and the hf backends and rmdir on
the hf backends, none of which have those ops. Ported Python's
requirements mechanism; the tables are now identical in both languages.
TypeScript search carried no cost provisioning on qdrant, lancedb or
mem0, where Python does.
js and node had no help text in TypeScript, and curl, history and seq
differed. Aligned per option.
* fix(spec): compare spec metadata per resource, not as unions
Addresses two review findings on the parity gate.
gen-specs.ts treated any index.ts read failure as an absent file, so a
permission or IO error silently dropped every command group in that
directory, defeating the reachability assertion exactly when the source
scan was incomplete. Only ENOENT continues now.
The _meta union flags cannot say which resource carries a provision, an
aggregate, the write flag or a filetype, so dropping one backend's
provision while another kept it left every union unchanged and the gate
accepted the regression. Both generators now emit _meta.by_resource and
the checker compares keyed by resource, falling back to the unions only
once the per-resource entries agree.
That found six more divergences the unions had masked:
Python's history cat, grep, head, tail and wc carried no aggregate where
TypeScript did, and the same for github and email grep. Added them.
TypeScript had no github du provision where Python does. Added it.
TypeScript still has no github grep provision. Python's is a bespoke cost
model that walks the index and reaches IndexCacheStore internals, so it
is recorded in parity_exceptions.json for its own change rather than
ported here.
Exemptions are now granular: by_resource names one resource and one key,
so exempting github grep's provision cannot also hide the aggregate
divergences on the same command. An exemption counts as used only when it
suppresses a live divergence, so the stale check no longer misreports an
exemption that fully covers its diff.
* feat(shell): export/readonly -p print mode (py + ts, #608)
Bare export/readonly and the -p form print declare -x/-r lines, matching
bash. Invalid option clusters fail with status 2 and the GNU usage line.
Array readonly names print as declare -ar. Unit tests and integ cases
cover both runtimes.
* fix(shell): match GNU on -- , control chars and type flags in -p print
Three defects in the export/readonly print path, all pinned against
bash 5.2 in debian:stable-slim:
- `export -p --` / `readonly -p --` failed with exit 2. The declaration
word collector folded any `-`-leading word into a char set, so `--`
contributed a literal `-` that the option validator then rejected.
Option words are now kept verbatim and in order, and `--` ends option
parsing. Keeping the order also makes the invalid-option message name
the *first* bad letter, as bash does; the old set was hash-ordered, so
`export -zq` reported `-z` or `-q` at random between processes.
- Only newline and carriage return selected `$'...'` quoting, and even
there only \n \r \t were escaped, so a tab, escape or DEL byte was
emitted raw inside double quotes. Any control character now selects
ANSI-C quoting, with bash's named escapes (\a \b \t \n \v \f \r, \E)
and three-digit octal for the rest. Printable non-ASCII stays literal,
which is what bash emits in a UTF-8 locale.
- Type-selecting flags were parsed and discarded, so `readonly -f`
listed variables and `readonly -a` listed scalars. `-a` now narrows to
indexed arrays; `-f` and `-A` list nothing, mirage carrying no readonly
or export attribute for functions or associative arrays.
Python and TypeScript stay mirrored. Adds 8 pytest cases, 6 vitest
cases, and 5 integ cases whose expectations were verified byte-for-byte
against GNU bash.
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
jq_eval collapsed a program's output stream into a single value: one
output bare, many as a list, zero as a JQ_EMPTY sentinel. A list of
outputs then reads the same as one output that is an array, so the
printer recovered the difference from the program text by scanning for
a `[]` at bracket depth 0 (has_top_level_spread).
Text cannot answer that question. Any multi-output program with no
literal `[]` printed a JSON array instead of one line per value, so
`jq -r '.a, .b'` printed `[1,2]`, and so did `range(3)` and `..`.
jq_eval now returns every output in order, format_jq_output prints one
per line, and the heuristic plus both JQ_EMPTY sentinels are gone: an
empty list means no output. Behavior is pinned against GNU jq 1.7 in
debian:stable-slim.
* chore(filetype): drop the eight provision helpers left with no caller
s3, ssh, redis, gridfs, nextcloud, onedrive, sharepoint and
databricks_volume each carried a _provision.py whose only consumer was
the provision= argument of the filetype command factory removed in
#651. Backends that also reach file_read_provision from their own grep
path keep theirs.
* fix(integ): keep the redis provisions and cover them with probe cases
The redis _provision.py is not an orphan of the filetype factory: it is
the public surface the redis example imports, mirroring the publicly
exported redis/provision.ts on the TypeScript side. Deleting it broke
examples/python/redis_resource/example_redis.py, which the Python
examples CI step runs.
It had no integ coverage, which is why nothing caught that. Adds
integ/resources/redis/probe.json with five provision cases running on
both hosts. Values are measured, not guessed: head -n 2 reports
net=0-24 precision=range, which is the hand-written head_tail_provision
bounding the read rather than claiming the whole file.
* test(integ): pin provision behavior for gdrive, google apps and trello
These four backends hand-write provision modules in one language or the
other, and none of them had provision coverage, so a divergence between
the hosts was invisible. Every value here is measured, not guessed.
The numbers differ per backend in ways worth pinning: redis reads over
the wire (net=24), gdrive serves the same file from its index cache
(cache=24 hits=1), a rendered .gdoc.json cannot predict its byte count
(precision=unknown), and head bounds the read as a range rather than
claiming the whole file.
Cases run on both the python and typescript-node hosts, so the layout
difference between the two languages is now gated on behavior.
* fix(provision): charge ops for remote listings, add search provision to TS
exactZeroProvision documents itself as being for backends whose listing
is already in memory, so metadata commands cost no backend I/O. gdocs,
gsheets, gslides and trello are remote API backends whose readdir hits
the wire, and TypeScript wired all four to it: ls reported ops=0 while
Python reported ops=1 for the same listing. Dropping the override lets
them fall through to the op-charging default, which is both the correct
estimate and the layout Python already had.
TypeScript's lancedb and qdrant search carried no provision at all where
Python's did; both now use exactZeroProvision, which is right there
because the search result is materialized in memory. The regenerated
search spec flipping to has_provision: true is that fix showing up.
The gdrive _provision.py had no consumer but its own test, and TS wires
nothing for gdrive, so both are gone.
Every change is pinned by a probe case measured on both hosts.
* fix(shell): fully general $? semantics, C-style for, and observed mtimes for find
* fix(ci): camel bg launch via $! and eslint require-await in statement test
* feat(ssh): native setattr via SFTP, stat surfaces mode and atime like disk
* fix(lint): narrow ssh2 nullable callback errors in ssh set_attrs
* fix(shell): reject readonly assignments in C-style for slots
* integ: pin the C-style for readonly abort
* fix(policy): fail loud on entry-script verdict shapes, stop fanout swallowing
* fix(monty): raise typed fs exceptions in the ts guest
* feat(quickjs): evaluator capability in both languages, js policy scripts
* feat(local): host python runtime for the node package
* test: align the script loader wording
* style: satisfy the ci linters
* fix(runtime): reclaim the local subprocess on safeguard timeout, tolerate stdin EPIPE
* fix(quickjs): bridge the workspace fs into eval, match the real engine's output surface
* feat(sizes): render-at-readdir sizes for slack and gmail; flip mem0 and opfs size-known
* test(integ): pin slack file, dm and empty-day chat, and gmail attachment sizes
* test(fuse): use notion as the size-unknown specimen for the fskit guard
* fix(slack): skip tombstoned and access-restricted file payloads in listings
* refactor: rename the runtime route to policy
* feat(policy): extensible verdict with a deny arm
* refactor(policy): move the safeguard resolver into the policy package
* fix(policy): rename the route key in the ts server config, fix the integ import
* fix(policy): record denied lines in ts history, type the verdict docstring
* fix(policy): attribute deny and admission errors to the command
* feat(policy): typed verdict arms RouteResult and DenyResult
* docs: type the policy example return value
* integ: move the safeguard scenarios to the JSON runtime harness
* fix(policy): gate syntax errors before the policy in python
* fix(shell): report the full syntax error span like bash echoes the line
* fix stat sizes across github, hf, databricks_volume and sharepoint; mark them size-known
* test(hf): move the metadata-stripping list doubles to module scope
* test(integ): pin the size backfills and the truncated-tree fallback
* feat(fuse): size push-down T flips for onedrive, box, dropbox, ssh and nextcloud
* style: prettier format for the nextcloud readdir test
* fix(box,ssh): follow ssh symlinks in TS stat, hide weblinks from direct box stat
* test(integ): pin ssh symlink sizes and box weblink hiding in the shared battery
* test(ssh): add lstat to the mock SFTP client
* refactor(runtime): evaluator capability, route engine on runtimes, drop PythonRuntime
* refactor(runtime): split base into types/errors/mixin, eval transport belongs to the runtime
* docs: module layout convention (types/errors/config/mixin/base)
* docs: trim module layout section
* review: brand the evaluator, bind session inputs and bytes on pyodide
* fix(runtime): refuse unbound interpreters with GNU command-not-found wording
* fix(integ): survive connection resets while jaeger boots
* docs: evaluator page under learn, language icons for the runtime pages
* docs: reframe the evaluator page as the policy engine
* docs: both entries capture python3 in the policy engine example
* docs: real routing case with the ctx payload and verdict semantics
* docs: drop the engine-only yaml block, one parenthetical instead
* docs: live-captured ctx payload, verified config, function route signature
* docs: trim the policy engine section
* fix(fskit): downgrade the size guard from a refusal to a warning
* feat(linear): push file sizes down to readdir so stat matches read
Every linear file is now sized at its parent directory's readdir from the
payload the listing already fetched: team.json from the teams listing,
issue.json from the issues listing, member/project/cycle/document JSON from
their own listings. comments.jsonl costs one bounded comments call, paid
only when the issue directory is entered. Listing a tree never fetches file
content, and stat reports the exact rendered byte length, which is what
fskit needs at lookup time to serve reads. SIZES_ALWAYS_KNOWN flips on.
Invariant pinned in both languages (stat size equals read length for every
file in the tree) and in integ with paired stat/wc cases against the fake
Linear server.
* fix(fuse): drain buffered writes on release, warn on writable fskit mounts
The macFUSE FSKit shim issues WRITE then RELEASE with no FLUSH in between
(the kext always flushes on close), so MountCore dropped every buffered
kernel write at release. Release now drains the handle's write buffer in
both languages.
Testing the CLI over a live fskit mount also measured a shim bug mirage
cannot fix: pages for regions a file did not already have (a new file, an
empty file, a truncate-then-write) flush as NUL bytes of the right length,
and appended regions arrive intact or zeroed depending on cache state,
with no error surfaced to the writer. A new check_writes guard warns at
mount time for writable fskit mounts (metadata ops are reliable; /dev is
excluded), and the behavior is pinned in integ/truth_fskit.json plus a new
CLI battery, integ/cli_fskit.sh, which drives both daemon-backed CLIs
through a real fskit mount and skips cleanly on hosts that cannot engage
the FSKit module. cli_fuse.sh now also asserts that a backend fskit
workspace is rejected cleanly off macOS.
* refactor(integ): move the fuse truth from txt lines to JSON
* refactor(integ): gather the fuse and fskit harness under integ/fuse/
* docs(fuse): document the fskit write-path limits and the size warning
* docs(fuse): tighten the fskit warnings
* refactor(runtime): one constructor shape for every runtime
* integ(runtime): one JSON suite for all runtimes across SDK and CLI
* integ(runtime): drop the e2b suite, docker stays the sandbox coverage
* integ(runtime): cover the --runtime error paths
* fix(integ): keep battery case discovery out of integ/runtime, satisfy ci yapf
* fix(grep): -r with no path operand searches the cwd like GNU
* fix(rg): bare rg searches the cwd like ripgrep, rename rebase_* to respell_*
* fix(cwd): bare find, tree, du and ls default to the cwd like GNU
* integ: cover routing gaps, fix redis du golden and rg battery stderr
* integ: materialise folder-backed mount roots at seed time
* fix(interpreter): refuse uncaptured python/js lines instead of a hidden fallback
* integ: seed the onedrive /data2 root too
* refactor(fuse): split the mount layer into MountCore plus a libfuse adapter
MirageFS was one class doing two jobs: filesystem semantics and talking to
libfuse. The FUSE-specific part was smeared across every method rather than
sitting at a boundary, which is why it could not be reused by another kernel
interface.
MountCore (fuse/core.py, fuse/core.ts) now owns all semantics and imports
nothing from mfusepy or fuse-native. MirageFS keeps only the callback
signatures and error translation. The attr dicts stay as they were: st_mode
and friends are POSIX stat field names, not FUSE ones.
The load-bearing change is error handling. Python raised FuseOSError inline
in ten methods with hand-rolled except chains while TypeScript already had a
single classifyError. Both languages now share one classification table
(fuse/errors.py, fuse/errors.ts), so the same backend failure reports the
same errno on both sides.
That fixes a real bug. Python's rmdir caught OSError before
FileNotFoundError, and FileNotFoundError is an OSError subclass, so rmdir on
a missing path returned ENOTEMPTY instead of ENOENT. No test covered it;
test_errors.py now pins it.
Behavior is otherwise unchanged: the pre-existing suites (test_fs.py 689
lines, fs.test.ts 43 tests) pass untouched apart from renaming private
attribute reads onto .core.
* feat(fuse): fskit mount backend, kext-free mounts on macOS
Adds backend selection to a FUSE mount: Mount(..., fuse_backend="fskit")
routes through macFUSE 5.x's FSKit shim, so the mount runs with no kernel
extension loaded. Closes#82.
Three rules are enforced at mount time rather than discovered at run time:
- macOS only, raising elsewhere instead of silently dropping the option.
- The mountpoint must live under /Volumes. FSKit refuses anything else,
which is what made the reporter's /tmp attempt in #82 fail. The rule is
owned by FuseManager, so no call site has to know it.
- Every mounted resource must be able to size its files. FSKit has no
direct_io, so reads are driven entirely by the reported size: a resource
that stats as 0 before open would serve an empty file with exit code 0.
Mirage refuses such a mount and names the offending prefixes.
That last rule needs a new capability, SIZES_ALWAYS_KNOWN / sizesAlwaysKnown,
default false. Opted in on the byte stores (ram, disk, redis, s3, gridfs)
plus dev and history. History matters more than it looks: it is mounted into
every workspace at /.bash_history, so leaving it false would block every
root-scoped fskit mount.
There is deliberately no "auto" value. Auto-selecting fskit would silently
break every API-backed mount, and an option whose safe value is always the
default is a trap.
TypeScript cannot reach FSKit at all: @zkochan/fuse-native bundles a
pre-macFUSE-5 dylib, so the option never arrives at a driver that
understands it. checkPlatform throws rather than quietly mounting through
the kext. The capability and guards are still mirrored so the semantics
match. Documented as a known gap.
Examples cover both sides of the limit: ram, redis and s3 mount and read
under fskit; slack and gmail show the guard refusing and the supported
workaround of scoping the mount to a byte-store subtree.
* refactor(mount): one backend field, vfs by default
Replaces the fuse boolean and the fuse_backend string with a single
MountBackend StrEnum: vfs, fuse, fskit. VFS is the default and finally has a
name, instead of being spelled fuse=False.
Mount(resource, backend=MountBackend.FUSE)
Mount(resource, backend=MountBackend.FSKIT, mountpoint="/Volumes/x")
Three things this fixes.
The default was previously defined by negation. Most mounts are not
kernel-mounted at all, and that case is what mirage is mostly used for, so it
deserves a name rather than a falsy value.
The overloaded bool | str union is gone. fuse=True and fuse="/path" packed
two decisions into one field; backend and mountpoint separate them.
The enum no longer claims a name it does not own. fuse_backend="fskit" read
as "use FSKit", but which delivery mechanism reaches FSKit (macFUSE 5's shim
today, a native Swift module later) is mirage's business, not the caller's.
One value covers both.
MountBackend lives in mirage.types next to MountMode rather than in
mirage.fuse, so the mount spec does not have to depend on the fuse package
for a value that also covers the non-fuse case. resolve_backend rejects vfs:
reaching it means a kernel mount was requested, and vfs registers nothing.
Config, server routers, and the TypeScript mirror follow, including the YAML
schema (backend: fuse plus mountpoint: <path>) and fuseMounts becoming
kernelMounts.
* fix(mount): migrate the CLI config path and integ fixtures to backend
The backend rename missed three call-site shapes that a literal search for
fuse=True / fuse: true does not catch:
- YAML mountpoints written from shell variables (integ/cli_fuse.sh wrote
`fuse: $dmnt`, five of them), which is the CLI's own end-to-end path.
- Mount options passed a variable rather than a literal (integ/fuse.py and
integ/fuse.ts both used `fuse=pinned`).
- The TypeScript server config schema, which still declared
`fuse?: boolean | string` and exported fuseMounts. It now mirrors Python:
backend plus mountpoint in, kernelMounts out.
The CLI itself has no fuse flag, so its whole surface is the workspace YAML.
Both config loaders were checked against the exact document cli_fuse.sh
writes and agree: /data and /logs resolve to (fuse, <path>), a backend with
no mountpoint resolves to (fuse, None), and a mount with no backend key stays
on vfs and is absent from the kernel mounts.
CLAUDE.md documents the single backend field and the YAML keys.
* refactor(mount): missing is vfs everywhere, and one guarded entry point
Two related cleanups.
Missing meant two different things. Mount.backend defaulted to VFS and an
absent YAML key resolved to VFS, but resolve_backend(None) returned FUSE,
reinterpreting an absent value as a request for a kernel mount. Now every
absent value lands on VFS, and the mount entry points spell their own
default: backend: str | MountBackend = MountBackend.FUSE. The intent is in
the signature instead of hidden in the resolver.
resolve_backend is pure coercion again; require_kernel_backend is the
separate step that rejects VFS. The unknown-name error lists all three
values rather than just the mountable two.
The fskit guards were three separate calls at each mount path, so a new path
could pick up fskit support and silently skip the macOS assert, the /Volumes
rule, or the size check. prepare_backend now runs all of them, and every
mount path routes through it: mount_background, mount, and FuseManager.setup.
Tests pin that a linux platform raises through prepare_backend, not just
through check_platform directly.
Same shape in TypeScript: resolveBackend, requireKernelBackend, prepareBackend.
* fix(fuse): drop the polynomial regex from unsizedMounts
CodeQL js/polynomial-redos, two high-severity alerts on the same helper.
`prefix.replace(/\/+$/, '')` backtracks on a string of many repeated
slashes, and the input is a mount prefix, so it is library input.
Uses core's loop-based rstripSlash instead, which is the existing sanctioned
trim for exactly this. Python was never affected: str.rstrip is a C loop,
not a regex.
* fix(fskit): match the mount recipe and volume ownership verified in #82
Three corrections to the fskit path, all found by re-reading issue #82
against what we actually shipped.
Mount options. #82's only reported working mount passes backend=fskit AND
volname, and omits direct_io. We passed direct_io unconditionally and no
volname, on my assumption that direct_io is simply inert on this path. That
assumption contradicted the one person who has run it, and direct_io is a
libfuse concept, so the shim needing it removed is not surprising. Now
matched exactly, with a comment telling the next person not to restore it.
Mountpoint ownership, which would have broken every fskit mount. /Volumes is
drwxr-xr-x root:wheel, so tempfile.mkdtemp(dir="/Volumes") raises
PermissionError for any non-root user. The reporter mounted at
/Volumes/mirage-* as a normal user, so macFUSE creates the entry: an FSKit
mount is a volume, not a directory we make. The fskit path now names its
mountpoint and never creates it, skips makedirs for a pinned one, and never
rmdirs a /Volumes entry on unmount.
Readiness. The /Volumes entry does not exist until the volume is live, so
bare existence is a real ready signal there, exactly as for WinFsp, and it
does not depend on os.path.ismount recognizing the mount.
These are the same three deviations WinFsp already needed, for the same
underlying reason: the filesystem driver owns the mountpoint. Tests pin the
option set and all three ownership rules, because nothing in CI can reach
this path (macOS 15.4+, macFUSE 5.x, a GUI-enabled FSKit module).
* fix(tests): pin the platform in the fskit manager tests
These three passed on macOS and failed on the Linux runner: FuseManager
routes through prepare_backend, whose check_platform rejects fskit off
darwin. Patch sys.platform explicitly so the assertion under test is the
mountpoint behavior, not the host, matching what test_backend.py already
does in both directions.
* ci: run the fskit backend for real on a macOS runner
Until now fskit shipped on unit tests plus one user report, because no
job could mount it. integ/fuse.py cannot cover it: its sizeless probe is
refused by the fskit size guard by design, and its two-mount scenario
needs something macOS forbids. So this adds integ/fskit.py, one RAM mount
under /Volumes that reads, writes and stats through the kernel.
Advisory like integ-fuse-windows, since whether a hosted runner enables
the FSKit module without a GUI toggle is what the job measures.
The truth file is JSON checked by value, not a txt file checked by
substring: the old harness passes a result of 166 against a truth of 16.
check_json.py is written to serve the remaining txt probes too. Only what
mirage controls is asserted; the mountpoint, the raw mount row and
whether a kext happens to be loaded are reported and left alone.
* ci(fskit): report the mount state before reading it
The first macOS run failed with a bare FileNotFoundError and told us
nothing: every diagnostic in the probe was printed after the read that
crashed. The volume entry appeared (readiness passed, the mount thread
stayed alive) but served no tree, and there was no evidence of why.
Print the mountpoint, its stat flags, the mount row and a listdir before
touching a file, and add an always-run step dumping the mount table,
/Volumes, the registered FSKit modules and the recent macfuse log.
* fix(fskit): a stray /Volumes directory is not a live mount
The macOS integ job reported exists=True isdir=True ismount=False with no
row in the mount table: macFUSE creates the /Volumes entry while mounting
and leaves the empty directory behind when the FSKit handoff fails. Bare
existence as the ready signal accepted that, so a mount that never came up
was reported live and failed with ENOENT on the first read.
Require os.path.ismount for every POSIX backend and keep the existence
shortcut for WinFsp only, where _prepare_mountpoint removes the directory
first so its reappearance really does mean the filesystem is live. The
backend argument is unused now, so it goes. _await_ready had no test at
all; it has one now.
Also register both macFUSE appexes on the runner: the installer registers
only the -local module there, while a developer Mac has both.
* ci(fskit): capture why the macFUSE mount never comes up
Both modules register and enable now, and the mount still times out with
no libfuse error, while the only module the system launches is msdos. So
the mount request is not reaching macFUSE at all. Capture the kext device
nodes, the system extension list and a macfuse-scoped log to tell an
environment limit from a mirage bug.
* docs(fskit): record where the /Volumes rule comes from
The guard read as an arbitrary house rule. It is measured: issue #82 ran
an fskit mount under /tmp, got 'mount_macfuse: the file system is not
available (1)', and the same mount worked once it moved to /Volumes.
* fskit: pin the real write surface, measured on a Mac
The mount works: /Volumes/mirage-*, tagged fskit, tree served, reads exact
(stat size == bytes read, so the size guard holds). ismount is true on a
live FSKit volume, which is what the readiness fix now depends on.
Writes are a different story. In-place writes and unlink work; create,
mkdir and rename return ENOSYS. A failed create still applies: the syscall
reports ENOSYS and the file exists in the resource and through the mount
anyway. Tracing shows mirage's create succeeds and returns a handle, then
the shim fails the syscall, so we cannot report this more accurately.
Pin the whole matrix in the probe and say plainly in the docs that
anything creating files will fail on an FSKit mount.
* examples: one fskit example per language
Five scattered files (ram, redis, s3, slack, gmail) collapse into
examples/python/fuse/fskit.py, which shows the size guard refusing, a live
mount with reads matching their stat size, and every write op with the
errno it returns. Verified end to end on macFUSE 5.3.3.
examples/typescript/fuse/fskit.ts cannot mount, since fuse-native bundles
a pre-macFUSE-5 dylib, so it shows the refusal and both alternatives:
backend fuse there, or the Python package for a kext-free mount. It
typechecks but is not run here, because its fuse leg is a live kext
mount.
* feat(fskit): TypeScript serves fskit after all
The 'fuse-native bundles a pre-macFUSE-5 dylib' claim was wrong: the
libosxfuse.2.dylib it ships is a stub whose install name is
/usr/local/lib/libfuse.2.dylib, and fuse.node links that absolute path,
so Node loads the same macFUSE 5.x libfuse Python does. Verified with a
live mount: /Volumes tagged fskit, cat/wc exact, and the same write
surface as Python (in-place ok, create/mkdir/rename ENOSYS).
Drop the unconditional throw from checkPlatform (macOS-only stays),
generalize appendDirectIO into appendMountOptions, and give mount() the
fskit branch: /Volumes named-not-created, backend=fskit + volname,
no direct_io, ownsMountpoint false so nothing ever rmdirs a /Volumes
entry. The example now mounts for real; docs flip from known-gap to
supported-with-caveats.
The caveat is earned: one of three example runs wedged on an append, and
a dead FSKit volume blocks mount-table enumeration system-wide until the
macFUSE appex is killed. Documented as read-mostly and experimental.
* docs: example READMEs and a concise fskit story
Both examples/ READMEs explain how to run, the naming convention, and why
fskit exists: Apple has deprecated third-party kexts (reduced-security
boot + approval on Apple Silicon already), FSKit is the supported
userspace replacement, and macFUSE 5.x serves the same libfuse API
through it. One mermaid flow shows the two paths differing only in the
kernel-to-userspace hop. The two fuse.mdx warnings shrink to their
load-bearing facts and the python page gains the same why + diagram.
* ci(fskit): report the known hosted-runner limit as a skip
Everything installs and enables headlessly, but the mount request never
reaches macFUSE's FSKit module on a hosted runner, so every run ends in
the same readiness TimeoutError and a permanently red advisory job is
noise. Treat exactly that signature as a skip and drop continue-on-error:
green now means environment-limited or verified, red means something new
broke (a guard regression, a crash, or the runner starts mounting and
diverges from integ/truth_fskit.json).
* feat(fskit): full write surface via macFUSE's Darwin-only callbacks
The read-mostly limitation was never FSKit's: the shim finalizes every
created item through setattr_x and routes rename through renamex, both
Darwin-only fuse_operations fields that mfusepy leaves as reserved NULL
slots. libfuse answered those requests itself with ENOSYS, after our
CREATE/MKDIR had already applied (wire trace: CREATE success, then
SETATTR -78), which also explains why the failures were dirty and why
rename never reached userspace.
fuse/darwin.py replaces the 13 reserved slots with macFUSE's real Apple
tail (same size, asserted) and marshals setattr_x, fsetattr_x and
renamex; MirageFS decomposes setattr_x (size routes to truncate, other
attributes follow the chmod/chown accept-if-exists semantics) and maps
renamex flags (EXCL honored, SWAP refused as ENOTSUP). Installed once
per process from _run_fuse, no-op off macOS, layout-guarded so an
mfusepy upgrade degrades to the old behavior instead of corrupting the
struct.
touch, mkdir, mv, rm and a new-file write roundtrip now all pass on a
real fskit mount; integ/truth_fskit.json pins the full matrix. Docs flip
Python fskit from read-mostly to full-write; TS keeps the old surface
because fuse-native's compiled op table cannot gain new C callbacks from
JS. Remaining upstream caveats stay referenced in code comments:
macfuse#1181 (exec until first read) and macfuse#1165 (root readdir
cache invalidation).
* test(fskit): skip the struct-extension test off macOS, settle yapf's import wrap
mfusepy builds fuse_operations per platform, so the Darwin reserved tail
the Apple fields replace does not exist in the Linux layout and
install_macfuse_extensions correctly bails there; monkeypatching
sys.platform cannot conjure the struct. yapf also disagreed with the
committed import wrapping; both yapf and isort accept the new form.
* feat(resource): generic remote_mount_spec so cloud backends are sandbox-mountable
Lift the S3-only remote_mount_spec to the base class: any backend that
opts in with remotely_mountable and holds a pydantic config serializes
{resource, config} (credentials unwrapped) so a remote mirage, e.g.
inside a sandbox, can reconstruct and mount it. RAM and disk keep the
None default. Enables s3, gdrive and slack; adds base coverage.
* feat(runtime): sandbox runtimes with FUSE-mounted workspaces
Add the RemoteSandbox base and the Daytona, e2b and Docker runtimes:
whole-line execution against a remote or local container, lazy
provisioning on the first line, reattach by id, and sandbox ownership so
teardown only touches what we created.
The workspace becomes visible by running mirage inside the sandbox and
FUSE-mounting each remotable mount live: the host serializes the mounts
into the public workspace config, writes .mirage-workspace.json, and runs
one command, `mirage workspace create`. The CLI auto-spawns the in-sandbox
daemon and mounts synchronously, so the exit code is the ready signal and
stderr carries the error. Reads and writes flow both ways with no sync;
writes reach the backend on file close. Needs an image with mirage baked
in (e.g. mirage-python-fuse).
* feat(sandbox): translate virtual mount paths onto provider mountpoints
The agent speaks virtual paths (/data/a.py); mirage is the control
plane that rewrites them onto each provider's physical mountpoint
(/home/daytona/workspace/data/a.py on Daytona, /workspace/data/a.py on
Docker). Before, only cwd was rebased, so absolute virtual paths broke.
Rewrite is longest-prefix-first and only touches tokens that start
exactly at a mount (/s3 or /s3/...); siblings (/s3.txt), system paths
(/usr/bin), and relative paths are left alone. Each mount is also
exported as a MIRAGE_<PREFIX> env var for paths built at runtime. A
bare / world mount is skipped (it would capture the sandbox's own /usr).
Verified end to end on a real Daytona sandbox with an absolute
/data/in.txt round trip.
* docs(examples): README for the Daytona FUSE runtime example
Documents the working flow: bake the mirage-fuse snapshot once, then
drive an S3-backed workspace whose python3 lines run in a Daytona
sandbox that FUSE-mounts the bucket live. Explains the control-plane
path translation (virtual /data/x rewritten to the sandbox mountpoint).
Fixes the workspace yaml: drops the removed mount: fuse/copy option
(fuse is now the only sandbox mode) and points the active runtime entry
at the mirage-fuse snapshot so the example actually mounts.
* fix(sandbox): typecheck fallback for env tuple; remove RAM runtime example
The env-var test's ?? fallback widened env to {}, failing tsc
(TS2339 on MIRAGE_DATA). Type the fallback like the sibling cwd test.
Remove sandbox_runtime.py: it mounted RAM, which is not
remotely_mountable, so under the fuse-only sandbox path its first
python3 line is rejected. The Daytona example (S3-backed) is the
supported demo.
* feat(docker): bake all backends into the sandbox image, not just s3
The sandbox image install was pinned to mirage-ai[s3,fuse], so a
workspace mounting any other backend failed in-sandbox for lack of its
deps. Install mirage-ai[all,fuse] via a MIRAGE_EXTRAS build arg
(default all) so one image mounts any backend; narrow the arg or extend
FROM the image for a lean build. Verified: [all,fuse] resolves under
pip and postgres/mongodb/gcs import in the built image (1.74 GB).
* feat(docker): sandbox image installs mountable backends only, no agent deps
Add a curated 'sandbox' extra (every mountable backend + fuse) and
default the Dockerfile to it, instead of 'all'. A sandbox is a
filesystem host: it never builds agents or launches other sandboxes,
so the agent frameworks (anthropic/openai/deepagents/openhands/agno/
claude-agent-sdk/pydantic-ai) and provider SDKs (daytona/e2b) that
'all' pulls do not belong. mem0 is excluded too: mem0ai is the lone
backend that hard-requires the openai client.
Verified in the built image: no openai/anthropic/daytona/e2b/mem0,
backends (s3/postgres/mongodb/chroma) and fuse still import. Image
1.74 GB (all) -> 947 MB. Narrow further with --build-arg MIRAGE_EXTRAS
or extend FROM the image.
* docs: document sandbox runtimes (docker/daytona/e2b)
Add a Sandbox page to the Runtimes section for both the Python and
TypeScript docs, wired into docs.json nav. Covers capture-based
routing, the live FUSE-mounted workspace, control-plane virtual-path
translation, the mirage-python-fuse image + MIRAGE_EXTRAS, per-provider
setup, reattach/lifecycle, and resource limits. The existing runtime
pages document the in-process interpreters (monty/wasi/pyodide/local);
this covers the remote whole-line runtimes.
* refactor(resource): unify remote flag, rename remotely_mountable -> remote
One boolean now answers 'does this backend live remotely', replacing
both the verbose remotely_mountable and qdrant's dead is_remote/isRemote
one-off. remote=True unlocks reconstructing the resource elsewhere (FUSE
mount inside a sandbox) via the generic remote_mount_spec. Set on s3,
gdrive, slack, qdrant.
* docs: add cache-invalidation example to invalidate_all_after_remote
Concrete cat -> sandbox-write -> cat example showing why a sandbox line
forces a full local cache reset.
* feat(sandbox): reconcile mounts imperatively, drop the workspace config file
mirage is the control plane: the host workspace is the desired state,
the sandbox is the actual state, and every captured line reconciles the
two through the provider's own exec API. A new or changed mount runs
'mirage mount add <prefix> --fuse <path>' inside the sandbox with the
spec in the exec environment (never a file, never argv), a dropped
mount runs 'mirage mount remove <prefix>', unchanged mounts cost
nothing. Mounts added or removed after the sandbox booted converge on
the next line.
New in-sandbox CLI 'mirage mount add/remove/list': each prefix becomes
its own single-mount daemon workspace (deterministic id), so mounts
attach and detach independently through the existing create/delete
endpoints. The uploaded .mirage-workspace.json and one-shot
mount_workspace are gone; TS mirrors the reconciler (syncMounts,
serialized per line).
Real docker e2e green: reconciled mount add, absolute-path read, FUSE
write-through to S3, host readback.
* test(qdrant): follow the is_remote -> remote rename
* ci(integ): survive chocolatey outages in the WinFsp install
Retry choco three times, fall back to the official WinFsp GitHub
release MSI, and verify winfsp-x64.dll landed so a bad install fails
at the install step instead of as 'Unable to find libfuse' mid-test.
The advisory integ-fuse-windows job went red on a chocolatey.org 503.
* refactor(sandbox): mount once, run lines verbatim, drop path magic
Remove the line translation, the MIRAGE_<prefix> env injection, and
the per-line reconcile state. The contract is now plain: the sandbox
mounts the workspace's backends once at boot (mirage mount add per
mount, spec in the exec env), mounts appear at
<workspace_root>/<prefix>, the session cwd is rebased, and the line
runs verbatim. Path consistency beyond the rebased cwd is the
caller's job; static rewriting could never be complete (quoted code,
runtime-built paths) and half-working magic is worse than none.
Docs and the daytona example teach the relative-path contract. Real
docker e2e green: relative read and write through the mounted bucket
with rebased cwd.
* refactor(sandbox): one general SandboxConfig, shared constants, provider packages
* refactor(sandbox): provider-owned configs, spec-derived mounts, remote sweep
* refactor(sandbox): connect-only runtimes, one in-sandbox workspace
* refactor(sandbox): drop the lazy provider re-exports
* refactor(sandbox): user-provisioned sandboxes, connect and exec only
* fix(sandbox): safeguard whole lines, per-invocation stdin paths, trim sandbox extra
* refactor(safeguard): one resolve_safeguard entry point, shared guard_output boundary
CLI exit codes / changes (push) Has been cancelled
Test (Install) / changes (push) Has been cancelled
Pre-commit / pre-commit (push) Has been cancelled
Integ / changes (push) Has been cancelled
Test (Python) / changes (push) Has been cancelled
Test (TypeScript) / changes (push) Has been cancelled
Integ / integ-facets (mem0) (push) Has been cancelled
Integ / integ-facets (project) (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Python) / runtime (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / python-install (push) Has been cancelled
Test (Install) / ts-install (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-shared-py (push) Has been cancelled
Integ / integ-shared-ts (push) Has been cancelled
Integ / integ-shared-parity (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Integ / integ-fuse-windows (push) Has been cancelled
Integ / runtime-py (push) Has been cancelled
Integ / runtime-ts (push) Has been cancelled
Integ / integ-observability (push) Has been cancelled
Integ / integ-facets (chat) (push) Has been cancelled
Integ / integ-facets (dify) (push) Has been cancelled
Integ / integ-facets (email) (push) Has been cancelled
* fix(shell): seed $? between commands in subshell and brace-group bodies
Inside ( ) and { }, last_exit_code was never updated between children,
so $? always reflected the pre-block value. Match the program-loop
behavior and seed after each child in both Python and TypeScript.
Fixes#476
* fix(shell): finalize lazy exit codes before seeding $? in subshell and brace-group bodies
* fix(shell): seed $? with a barrier in if/loop/case/function bodies too
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* fix(observe): keep the mount prefix on op records in typescript
Op events named the mount-relative path, so two mounts holding the same
filename produced identical records and the audit trail could not say
which backend was touched. Python already had the machinery for this;
typescript was missing three pieces of it.
- pushMountPrefix returns the previous prefix so callers restore it
instead of clearing to '', which a command dispatching into another
mount needs (mirrors push_mount_prefix).
- withMountPrefix re-pushes the prefix around each pull from a stream,
so a command that defers its backend read to the first chunk still
records the right path. Mount.execute wraps the stream plus
IOResult.reads/writes through it (mirrors with_mount_prefix and
_wrap_cmd_streams), with CachableAsyncIterator.wrapSource standing in
for replace_source.
- Dispatcher.dispatch and Mount.executeOp push the prefix around the
backend op, which is how eager writes get theirs (mirrors Ops._call).
The two s3 sites that passed the virtual path to work around this keep
doing so; record() leaves an already-prefixed path alone. Their comments
no longer describe a bug.
* test(observe): pin op record paths per mount in both languages
Two mounts each holding report.json, driven through an eager write, a
lazy read and a cp that crosses mounts. Both languages now emit the
same six records. The typescript test previously carried a comment
explaining why the path was not asserted; that is gone.
* fix(observe): scope the mount prefix per async branch
pushMountPrefix mutated the one RecordingState every branch shares, so
two mounts consumed concurrently (cat /s3/a & cat /db/b) could see each
other's prefix and their restores could overwrite each other, recording
paths like /db/a. Python avoids this by re-setting a frozen Recorder per
task and sharing only the sink.
runWithMountPrefix replaces it: it derives a state for the branch,
shares the records array, and hands it to the async context, so nothing
is mutated in place. The push/restore pairs in Mount.execute,
Mount.executeOp and Dispatcher.dispatch become plain scopes and their
try/finally blocks go away.
Both new context tests fail against the mutating version.
* fix(observe): require a path boundary before treating a record as prefixed
record() decides whether a backend already named the virtual path by
testing startswith(prefix), which also matches a filename that merely
shares the prefix's leading text. A mount at /s3 holding s3-report.txt
recorded /s3-report.txt, dropping the mount. Compare against prefix + "/"
(or the mount root itself) in both languages.
* integ: move history and find arg-errors to the JSON harness, add curl/wget
The JSON harness has grown to cover almost everything the old script-plus-
truth-file suites did, so this retires the ones that are now redundant or
cheap to migrate, and closes the last command with no integ coverage.
Deleted as dead code:
- cases.py / cases.ts (~1.7k lines each). Only three meta helpers were still
imported, by metadata.py / metadata.ts.
- metadata.py / metadata.ts / truth_metadata.txt. Every case they ran already
exists in integ/unix/meta and integ/unix/meta_overlay under the same ids,
on 22 backends instead of 4.
- s3_probe.ts and runners/tools/migrate_legacy.py, referenced by nothing.
- Seven integ/package.json scripts pointing at files that do not exist.
Migrated to the harness:
- history.py + truth_history.txt become integ/bash/history/*.json under a new
ram-history target. It needs its own target because the shared battery's
~1900 commands on ram would otherwise land in the recording and shift the
history line numbers. The observer-introspection tail is dropped: those are
Python API assertions the shell harness cannot express, and
python/tests/observe covers all five. TypeScript gains history coverage it
never had.
- find_arg_errors.py / .ts + truth_find_arg_errors.txt become four
{mount}-bound cases across sixteen dummy-credential argerr-* targets, wired
through one new arg_error builder per language. This is the first integ
coverage for discord and github_ci.
New coverage:
- curl and wget were the only spec commands with zero integ cases. Their unit
tests mock the HTTP layer, so the untested part was the write landing in the
VFS. integ/server/http_server.py is the fixture; the adapter owns its
lifecycle and exports the endpoint as the {http} token, so --facet http
needs no CI setup.
Four curl/wget cases are deliberately absent because python and typescript
disagree: the 404 path (python leaks httpx's message including an MDN link and
exits 1, typescript exits 22), the unreachable-host path (python reports a
platform-specific errno), and the read-only-mount message. Those are backend
bugs to fix separately, not test scaffolding.
Root truth files drop from seven to four: safeguard, runtime, runtime_py and
fuse remain, each blocked on a harness feature.
* integ: restore the metadata snapshot/GC scenarios, pin the observer event shape
Codex review on #649 caught two places where the previous commit removed
coverage rather than moving it.
metadata.py / metadata.ts held three scenarios beyond the case lists that were
migrated to integ/unix/meta and integ/unix/meta_overlay, and truth_metadata.txt
asserted all three: overlay attributes surviving a snapshot reloaded onto a
fresh S3 resource, ConsistencyPolicy.ALWAYS garbage-collecting an attribute
overlay orphaned by an out-of-band unlink, and the same snapshot roundtrip on
RAM. Verifying that the case ids matched was not enough, main() did more than
call those three functions. Both scripts come back trimmed to just those
scenarios, with the meta_stat_line helper inlined now that cases.py is gone.
Both reproduce the original truth output byte for byte. Retiring them for real
needs snapshot and namespace support in the harness, which the file now says.
The history observer assertions are a narrower case. recorder-not-mounted and
the post-clear session projection are genuinely covered by
python/tests/observe. The event shape is not: exit_code, cwd and the op source
were only ever asserted on entries the tests build by hand, never on the real
execute path, so a regression in the wiring from a command to its event would
pass while the rendered views stayed correct. Three tests per language now pin
that on ws.execute(), which also gives TypeScript coverage it did not have.
Doing so surfaced a divergence: TypeScript records the op path mount-relative
('/test.txt') where Python records the virtual path ('/data/test.txt'), so the
prefix Mount.execute sets is not reaching record(). Python's test pins the
correct form; the TypeScript test asserts a read op exists without freezing the
wrong path in place, and says why.
* curl,wget: align exit codes and streams with the real tools (py+ts)
Pinned against curl 8.14.1 and GNU Wget 1.25.0 in debian:stable-slim before
changing anything, per the repo rule for command semantics. The probe found
both languages wrong, in different ways, on almost every failure path.
The root cause was one design mistake shared by both: the transport layer
called raise_for_status() (python) or threw on !resp.ok (typescript), so an
HTTP error status became an exception before either command could see it. That
is wrong for curl, which treats a 404 as a successful transfer, and it meant
the HTTP library's own text reached stderr, including a documentation URL from
httpx and a bare "fetch failed" from undici. The layer now reports status,
reason and body, and each command owns its own CLI semantics.
curl, matching curl 8.14.1:
- an HTTP error status is no longer a failure. The body goes to stdout (or to
-o) and the exit code stays 0.
- new -f/--fail, the only way to make a status an error: exit 22, and nothing
is written.
- a refused connection is exit 7 with a host/port message, not a
platform-specific errno (61 on macOS, 111 on Linux, so the old message could
not even be asserted in CI).
- a failed -o write is exit 23.
- no URL is exit 2 with curl's two-line usage text.
- -o prints nothing on stdout; it used to print "saved to <path>".
- -s now only suppresses the message and -S restores it; neither changes the
exit code.
- -L is honoured. Python passed follow_redirects=True unconditionally, so -L
was a no-op and redirects were always followed.
wget, matching GNU Wget 1.25.0:
- any 4xx/5xx is exit 8 with "ERROR <code>: <reason>.", and the -O target is
still created empty, the way wget truncates it before reading the status.
- a refused connection is exit 4.
- --spider reports on stderr ("Remote file exists." / "Remote file does not
exist -- broken link!!!") rather than stdout, and inherits the same exit 8.
- the progress report moved to stderr; stdout is empty.
- no URL is exit 1 with the usage block.
Two more bugs found on the way: both commands caught only three exception
types, so a missing parent directory escaped uncaught, now fixed by using the
shared WALK_ERRORS set; and python rendered OSError with str(), leaking
"[Errno 13]" and a python repr onto stderr, now using strerror.
Coverage: 35 integ cases under --facet http, byte-identical across both hosts,
plus 31 new python unit tests (test_curl.py, test_wget.py) and 10 for the
transport layer, and the typescript tests updated to the pinned behavior.
Known remaining divergence, deliberately not asserted: a read-only -o target
produces different wording per host because python rejects the write at the
mount layer and typescript at the dispatcher layer. Those cases pin the exit
code with -s/-q so neither host's wording is frozen as correct.
* fix(ci): yapf formatting, a long line, and the tsc-only stderr type
Three CI failures, all mine from the previous commit:
- yapf reformatted the two new test files (multi-line signatures) after the
commit that added them, so the committed tree was not formatter-stable. The
local pre-commit run that reported success had applied those edits without me
re-staging them.
- flake8 E501 on a 82-char assertion in test_curl.py.
- tsc --noEmit rejected DEC.decode(ioResult.stderr ?? ...) because stderr is
ByteSource, which may be an AsyncIterable. vitest does not typecheck, so the
test run was green while the typecheck job was not. Both harnesses now use
IOResult.stderrStr(), which materializes the stream the way the class already
does internally.
The typescript gate failure was only downstream of the typecheck job.
* fix(ci): regenerate the committed specs for curl -f
Adding -f/--fail to the curl spec makes the checked-in spec/ snapshots stale.
The "Spec drift" check is a separate step in the pre-commit workflow rather than
a pre-commit hook, so a local `pre-commit run --all-files` never covers it:
python scripts/gen_specs.py
node --experimental-strip-types typescript/scripts/gen-specs.ts
git diff --exit-code spec/
Regenerated all three snapshots (python, typescript node, typescript browser);
the generators are idempotent on the result.
* ci: one database job driving both hosts against one service fleet
integ-database and integ-database-ts started an identical fleet (postgres,
chroma and qdrant as services, plus a hand-initiated mongodb replica set) and
then ran the same four targets, differing only in which runner they invoked. So
every database PR paid for four container startups, two chroma health waits, two
qdrant health waits and two replica-set initiations twice over.
They are now one job. This is the shape integ-observability and integ-facets
already use: start the fleet once, run the python battery, then the typescript
one. Database was the only family still split by language.
Two details worth keeping:
- the typescript step carries `if: ${{ !cancelled() }}` so a python-only
regression still reports whether typescript has it too. Splitting by job used
to give that for free.
- the per-backend env (CHROMA_HOST, QDRANT_PORT, ...) moved to the job level so
one step per language can cover all four targets, matching how
integ-shared-py runs 34 targets in a single step.
Trades a little wall clock (the two batteries serialize where they used to run
in parallel) for roughly half the runner minutes and half the containers.
79 fewer lines of workflow.
* fix(jq,gws): evaluate multi-document input per value, paginate gws GETs
jq computed output arity with a naive "[]" substring test while jq_eval used a
depth-and-string-aware predicate, so any program with a nested [] inside a
collector had its single array exploded one element per line. Reuse the careful
predicate in both places.
jq also read .json files with strict orjson, so a file holding several JSON
values failed, and the stdin path collapsed a multi-document stream into one
list. Real jq reads a stream of values from any input and evaluates the program
per document. Both paths now do that; -s still slurps.
gws list methods made exactly one HTTP call and dropped nextPageToken, so
'gws drive files list' silently returned the first 100 of N at exit 0.
Pagination is now the default (a truncated listing is indistinguishable from a
complete one) with --page-limit to opt out. Single-response GETs keep their
exact bytes; only real multi-page streams are newline-delimited NDJSON.
Adds 'gws --help' and 'gws <service> --help' matching googleworkspace/cli, and
teaches the integ mock server to paginate. Two existing goldens encoded the jq
bug and are corrected.
* fix(gws): port pagination and help to TypeScript, regenerate specs
The epilog field added to CommandSpec was missing from the committed spec
dumps, which is what the pre-commit spec-drift step caught. Regenerated
both trees and taught the TypeScript side about the field, so the two
dumps keep the same shape.
The PR description claimed TypeScript had no gws passthroughs. It has all
25, so the unpaginated-GET bug was live there too. Ported pagination,
--page-limit, the per-method --help descriptions, and the
gws --help / gws <service> --help surface. render_services() and
render_service_methods() output is byte-identical between the two.
Two defects surfaced during the port:
- withHelpSupport rebuilds the spec from a hand-listed init object to
inject --help, so it dropped epilog on every command. Python is immune
because it uses dataclasses.replace. Typecheck and unit tests both
passed; only the integ case caught it.
- gws --help was registered for gdrive and gsheets only, so a gdocs-,
gslides- or gmail-only mount had the passthroughs but no help. Now
registered for all five in both languages.
Structural alignment between the two implementations:
- drop the orphaned GWS_API_SPEC, replaced by gws_method_spec
- add _parse_page_limit mirroring parsePageLimit, so a bad --page-limit
reports the same message instead of leaking int()'s ValueError text
- re-export the help commands from the gws package so backends stop
reaching into gws.help, matching gws/index.ts
- make the four help description constants module-private on both sides
- drop the now-dead non-GET guard before invalidate_mount_listing
- give each page its own params dict instead of mutating one across calls
gws integ coverage is no longer blocked: g_help_lists_services,
g_help_lists_service_methods, g_files_list_paginates_by_default and
g_files_list_page_limit_truncates run on both hosts.
Docs: a Pagination section on the Drive page covering the default-on
behavior, the NDJSON output, --page-limit and the deliberate divergence
from googleworkspace/cli; the same note on the Gmail passthrough section;
and a JSON with jq section in the bash docs for the stream-of-values
semantics, verified against the jq binary.
* fix(spec): trim the help epilog without a polynomial regex
CodeQL flagged js/polynomial-redos (high) on the `/\n+$/` I used to strip
the epilog's trailing newlines: it backtracks on a long run of '\n'. The
trim now walks backwards, which is what Python's rstrip('\n') already did,
so that side was never affected. Verified identical output on 'Services:',
a single trailing newline, three trailing newlines, and an all-newline
epilog; a 100k-newline epilog now renders instantly.
Also align --page-limit validation across the two. Python used bare
isdigit(), which accepts non-ASCII digits that TypeScript's /^\d+$/
rejects: '١٢' parsed as 12 on one side and errored on the other, and '²'
passed the check then crashed int(). Python now requires isascii() too,
and both suites reject the same five inputs.
* fix(gws): register help commands per resource, drop the nested handler
Both findings from the codex review on #650.
P2, resource filtering. MountEntry.register keys commands by
(name, filetype) and ignores RegisteredCommand.resource, so registering
the whole GWS_SERVICE_HELP_COMMANDS list on every backend made a
gdocs-only mount answer `gws gmail --help` and `gws drive --help` with a
method listing it cannot execute. Reproduced on a gdocs-only workspace
before the fix. gws_help_commands(resource) / gwsHelpCommands(resource)
now build one registration per reachable service, bound to the single
resource asked for, which is what the TypeScript backends were doing with
an explicit filter. A drive mount still reaches docs, sheets and slides.
P1, nested function. make_service_help_command defined its handler inline
and closed over the body, against the repo's no-nested-functions rule.
The handler is now module-scope run_help, bound to its listing with
functools.partial, the same way run_gws_method is bound to its method.
Both languages end up with the same shape, so the per-backend filter
duplication in TypeScript is gone too. ROOT_DESCRIPTION is module-private
on both sides.
* refactor(filetype): remove the bundled format renderers, keep the extension point
mirage shipped renderers for parquet, ORC, feather/arrow/ipc and hdf5/h5, plus
a PDF module that turned out to be entirely dead code: '.pdf' was never in the
factory registry and nothing imported mirage.core.filetype.pdf, so its test
exercised it directly and kept it looking alive.
Both core/filetype/ trees are removed. The dispatch machinery stays, in both
languages and in both places it lives (commands/builtin/filetype_factory and
ops/generic/factory), with empty registries and a comment marking them as the
extension point. A file with an unregistered extension now reads as raw bytes.
Registering a filetype-scoped command still works and is covered:
tests/commands/custom/test_filetype_fns.py registers a '.parquet' handler with
a fake function and asserts dispatch, and test_unregister_removes_all_filetypes
now registers a '.demo' renderer itself rather than leaning on the bundled ones.
Also removed: the parquet/hdf5/pdf extras (and their references from 'all' and
'deepagents'), the hyparquet, hyparquet-writer, apache-arrow and h5wasm
dependencies, integ/resources/columnar, the columnar FileType enum members, the
per-backend filetypeRead declarations that existed only on the TypeScript side,
and the sentence advertising 'cat on .parquet/.orc/.feather returns a formatted
table' from 33 backend and agent prompts, which would otherwise have been
lying to agents.
grep_helper's skip-list and file's MIME map keep their columnar entries: those
are still correct, since the formats remain binary whether or not mirage can
render them.
core/src/ops/generic/factory.ts was reaching NodeJS.ErrnoException through the
columnar packages' type dependencies. core has to work in both runtimes, so it
now uses a structural { code?: string } instead.
Python suite passes, TypeScript core 5458 pass, pre-commit clean.
* refactor(examples): drop the columnar demos alongside the renderers
Deleted, because their whole subject was the removed rendering:
s3_data.py (181 lines built around four columnar constants),
ram_filetypes.ts, box_parquet.ts, dropbox_parquet.ts, ram_parquet.ts.
Also deleted fuse_hooks.py, which was already broken before this change: it
imports mirage.fuse.filetype.data.local.parquet, a module that does not exist
anywhere in the tree.
Edited rather than deleted, since columnar was one section of a broader demo:
gdrive_complex.py, disk.ts, ram_fuse.py, and the openhands README (which
advertised format-aware reads and piped a .parquet through jq).
Everything still referencing parquet only names it in find/ls patterns, which
keeps working: mirage can still list and locate these files, it just no longer
renders them.
* fix(ci): drop the last filetype wiring the removal missed
The nextcloud ops table still declared filetypeRead for feather/hdf5/
parquet, so makeGenericOps threw at module load and every job that
imported @struktoai/mirage-node died before running anything.
Also: regenerate the specs (the crash hid 89 files of drift), drop the
removed pdf/parquet/hdf5 extras from the install matrix, take the
columnar members out of the TS FileType enum and the box/gdrive/dropbox
type guesses so both languages agree again, and stop passing an inert
filetype_read=True from ten Python backends.
* fix(spec): regenerate with every optional backend importable
The previous regeneration ran under the worktree venv, which has no
optional extras installed, so backends that failed to import were
dropped from each spec's resources list. Regenerated with the full
environment: the only remaining change is the emptied filetypes list.
* fix(agents): stop telling models mirage renders columnar formats
The langchain backend mapped parquet/h5/hdf5/feather to text/plain, so
with no renderer registered it handed the model raw binary decoded as
UTF-8 instead of the binary redirect. Those extensions now fall through
to application/octet-stream.
The system prompt, the execute tool description in both languages, and
the README extensibility example all still advertised columnar rendering
as a shipped feature; they now describe it as the extension point it is.
Also drops commands/optional.py, which existed only to soft-import the
removed format helpers and has no callers left.
* refactor(filetype): drop the filetype command factory, keep mount registration
The factory built nine commands per registered extension, but with no
renderers shipping it produced nothing on every one of the ~19 backends
that called it, and its handlers had no test in either language while
the module contract they expected was written down nowhere.
Removes commands/builtin/filetype_factory/ in both languages along with
the filetype_read / filetypeRead op knobs and the reads and provisions
that existed only to feed them. Registration on a mount survives and is
the whole extension point: a command or op carrying a filetype resolves
as (name, filetype) before (name, resource).
examples/{python,typescript}/filetype/ register a .tally renderer end to
end and are gated in CI against integ/truth/*/filetype.txt, so the path
is now exercised rather than asserted. Both emit byte-identical output.
* fix(mkdir): GNU conformance for mkdir and mkdir -p (py+ts)
mkdir was the one write op #641 left short. Four divergences, pinned
against GNU coreutils in docker before changing any semantics.
- mkdir -p across a plain file turned that file into a directory on the
store backends. The -p loop added every component blind, so the new
directory key shadowed the file key: reading it started reporting
EISDIR while the bytes stayed orphaned in the store. Now "cannot
create directory 'a.txt': Not a directory", and the file is untouched.
- mkdir -p onto a plain file target reported nothing at all. GNU calls
that EEXIST, not ENOTDIR: the walk reaches the target itself rather
than crossing it.
- mkdir without -p silently succeeded on an existing path, on every
backend. GNU refuses with "File exists"; only -p is idempotent. Every
internal caller that mirrors a tree through the op (cp -r) already
probes with is_directory first, so none of them relied on the no-op.
- mkdir -p quoted the whole operand where GNU quotes the component it
tripped on ("cannot create directory 'a.txt'", not 'a.txt/sub').
Shared pieces: check_mkdir_target joins check_dest_parents in each
store backend's dest module, so the two destination questions a write
can ask live side by side. The real-filesystem backends keep the
kernel's errnos and only walk the chain to attribute a failure it
already reported, which is what disk/dest and opfs's local
checkMkdirTarget do. error_path/errorVirtualPath is factored out of
format_fs_error so the command layer can quote whatever path an error
names rather than assuming the operand, and mounted_path moves from
generic/cp to utils/key_prefix now that backends address ancestors too.
Deliberate divergence: operand spelling stays normalized, so a relative
argument is reported absolute (as #641 already established).
* test(crossmount): mkdir -p the redis parent chain in the matrix fixture
The fixture built the chain with repeated plain mkdir calls, which only
worked while an existing directory was a silent no-op. Use -p, like the
disk branch beside it already does.
* fix(mkdir): address Codex review on #647
- kindAt swallowed every OPFS resolution failure, so a SecurityError or
InvalidStateError read as an absent component and mkdir carried on.
Only NotFoundError/TypeMismatchError mean "nothing here"; the rest
re-raise.
- disk's component walk used isfile, which misses a FIFO, socket or
device ancestor: makedirs still refused with ENOTDIR but the walk
found nothing to blame and the whole operand was named. It now stats
once and treats any non-directory as the failing component, matching
the TypeScript side.
- mkdir reported an absolute path for a relative operand, where GNU and
mirage's own read-family commands quote it as typed. operand_spelling
rebases whatever path an error names (the operand, an ancestor of it,
or something under it) onto raw_path, so `cd /data && mkdir -p
f.txt/sub` now says 'f.txt' like GNU.
drop_trailing_segments joins rebase_one in utils/path as the ancestor
half of the same idea, with unit tests in both languages, and two integ
cases cover the relative spellings.
2026-07-27 00:08:30 -07:00
5258 changed files with 369395 additions and 94779 deletions
@@ -24,7 +24,18 @@ Run Python commands from `python/`, TypeScript commands from `typescript/`.
- Keep Python and TypeScript layout, architecture, and semantics mirrored as much as practical.
- When changing one implementation, check the other for the matching pattern or feature. If one side is more correct, use it to improve the weaker side instead of copying a bad design.
- For major Python or TypeScript changes, consider adding or updating integration coverage under `integ/`.
-Known gap: TypeScript does not support ORC files. Python registers `.orc` in its filetype factory (`mirage/core/filetype/orc.py` plus per-backend `read_orc` ops); the TypeScript filetype factory only covers parquet, feather/arrow/ipc, and hdf5/h5. Do not assume `.orc` commands work in TypeScript.
-**The layout half of that rule is gated.** `scripts/check_layout_parity.py` diffs the module-name sets of every `mirage/<pkg>/` against its TypeScript counterpart (core/node/browser unioned onto one namespace, plus cli/server/agents by prefix), folding camelCase, hyphens and a leading underscore so a rename reads as a rename rather than a missing module; `__init__.py` and `index.ts` are skipped because only Python needs one per directory. Intentional differences live in `spec/layout_exceptions.json` with a reason, and a stale entry fails as loudly as a new gap. `--strict` (what CI runs) does not demand zero: it fails when the count moves off the committed `baseline` in *either* direction, so new drift is blocked and closing a divergence has to be locked in by lowering the number. Run it without `--strict` for the full advisory report; that report is how layout work gets scoped.
- **mirage ships no filetype renderers, and no factory for them.** Parquet, ORC, feather/arrow/ipc and hdf5/h5 rendering are gone, along with the `parquet`/`hdf5`/`pdf` extras, the `hyparquet`/`apache-arrow`/`h5wasm` dependencies, and the whole `commands/builtin/filetype_factory/` package in both languages (with its `filetype_read` / `filetypeRead` op knobs). A file with an unregistered extension is read as raw bytes. The one surviving extension point is registration on a mount: a command or op carrying a `filetype` resolves as `(name, filetype)` before `(name, resource)` before `(name,)`. `examples/{python,typescript}/filetype/` register a `.tally` renderer end to end and are gated in CI against `integ/truth/*/filetype.txt`; `tests/commands/custom/test_filetype_fns.py` and `test_unregister_removes_all_filetypes` cover the unit path.
## Module Layout
Packages split by role, one module per concern, the same way in both languages:
- **`types.py`** — data shapes only: frozen dataclasses, type aliases, Literal unions (e.g. `runtime/types.py` holds `RunArgs`/`RunResult`/`EvalValue`/`EvalResult`/`ScriptSource`). No logic.
- **`config.py`** — configuration knobs and their coercion (e.g. `runtime/config.py` holds `RuntimeConfig`, which fails loud on unknown fields).
- **`mixin.py`** — opt-in capability mixins: stateless, no constructor, abstract methods only (e.g. `runtime/mixin.py` holds `EvaluatorMixin`). Capability is detected by type (`isinstance`), never by probing for a method.
- **`base.py`** — the package's core ABC and nothing else (e.g. `runtime/base.py` is just `Runtime`).
## History
@@ -36,12 +47,410 @@ Command history is a recording, not a command log. A hidden `Observer` records e
- **Snapshots.** History is captured as events into snapshot state and restored on load.
- **Format is GNU bash, not zsh** (`#<epoch>`, not `: <ts>:<dur>;<cmd>`).
## CLIs
An installed CLI is a typed program tree (`CLISpec`) bound to a head word on the
workspace. It is **dispatched by name, never by operand path**: the VFS is how an
agent discovers state, the CLI is how it acts.
- **An account CLI consults no mount; `git` is the one that does.** The two are
different tiers, told apart by `config_model`. An account CLI
(`slack`, `linear`, `gws`, …) declares one, initializes from it, and reaches a
service, so a mount would be a second source of truth for the same account.
`git` declares none, because there is nothing to authenticate to, and its
subject is a repository that lives on a mount, so it reads that repository
through the op dispatcher like any command. That is what makes `git` work on a
RAM mount, a disk mount or an object store without knowing which. It reaches
the dispatcher through `CLIDoors`, one door per state plane: `dispatch` and
`stat_path` for data, `ns` for the name plane (mount boundaries, links, attr
overlay), and `session_view` for session state. The mount prefix is
`ns.mounts.root_of`, not a fourth field. `handle_cli`/`handleCli` puts the
record on `inv.doors`; the field is None/absent outside a workspace, and a
verb that never reads it cannot touch a mount, so this stays opt-in per verb
rather than ambient. The door is one field read rather than a parameter list
the dispatcher inspects, because every leaf takes exactly one `CLIInvocation`
and nothing is threaded through keyword injection. Every field is spelled the
way `CommandOpts` (`commands/config.py`) spells it, so one fact reached from a
CLI leaf and reached from a command handler has one name; meta-tests pin that
(`tests/commands/cli/test_doors_parity.py`, and a compile-time mapped type in
`commands/cli/types.test.ts`).
Do not give an account CLI a mount, and do not give `git` a `config_model`.
Nuance: an account CLI verb MAY read an unrelated workspace file the user
named on the line through `inv.doors.dispatch` (himalaya's `--attach` reads the
attachment path this way); what it must not do is treat a mount as a second
view of its own account's data.
- **The lifecycle is host-side only.** `register_cli`/`unregister_cli`
(`workspace.py`, `workspace.ts`) are called by the embedding program, never by
a line the agent types, and there is no `install`/`uninstall` shell builtin.
Keep it that way: an agent must not be able to take away the tools it was
given. Shadowing is the one thing it can do (define a shell function with the
same name), which is bash's own rule, reversible with `unset -f`, bypassable
with `command <name>`, and visible in `type -a`. A deployment that needs a head
word pinned enforces that in the policy layer's `pre_execute`, not in the CLI
registry.
- **Precedence is written down once**, in `_layers`/`layers`
from the long spelling when absent, which covers most options and is why only
the four upstream overrides declare one). `Option.env` is a fourth and is
**not** a synonym for `default`: an env-sourced value counts as *supplied*,
so clap echoes it in a usage line where a defaulted one is invisible, and it
is read from the session rather than frozen into the spec. The executor fills
it, so a leaf reads one flag instead of a flag and a fallback.
- **Imitating a Rust CLI means imitating serde_json, so the engine parser does
not get to decide.** `ntn api` echoes its JSON parse failures verbatim, and
those words are serde_json's (`EOF while parsing an object at line 1 column 1`), which neither `json.loads` nor `JSON.parse` can produce and which the
two of them word differently from each other anyway. So
`commands/cli/builtin/ntn/serde.py` (`serde.ts`) is a scanner whose only job
is that message, and it is the **authority on validity too**: the engine
parser runs only on input the scanner already accepted, because the two
disagree about what JSON is (python accepts `NaN`, serde refuses it, and
silently sending `NaN` is worse than either). Three rules that are easy to
get wrong and are pinned by a probed table in `test_serde.py` /
`serde.test.ts`: columns count **bytes**, not characters (`["é"x` fails at
column 6); a newline advances the line and zeroes the column; and the
recursion limit is 128, refused at the opening bracket of the 128th
container. Do not "simplify" this to a try/except around the engine parser.
- **`ntn api` has three body sources, and both the order and the exit codes are
observable.** stdin, `--data` and inline `path=value` / `path:=json` inputs
are each validated in that order, and only then is "more than one source"
reported, so a malformed pipe outranks a malformed `--data` and both outrank
the conflict. The exit codes are two families, neither of them argparse's 2:
a body that arrived malformed is **1** (`error: Invalid JSON from stdin`),
while a line the CLI will not interpret at all is **5** (the conflict, an
unparseable inline input, an empty `--data`), and those carry a second
` hint:` line. There is **no object check anywhere**: `--data '[]'` posts
the array, and any body source at all makes the call a POST even when what
it carries is empty, so `--data '{}'` posts rather than falling back to GET
on falsiness. `name==value` stays a query parameter whatever the method is.
- **Discoverability is part of shipping a CLI**, and it comes from the spec, so
it works for a user's own registered CLI exactly as for a builtin one. `man <cli>` and `man <cli> <verb>...` render through `node_help`/`nodeHelp`, the
same renderer `--help` uses, so a manual cannot drift from the program; bare
`man` lists installs under `# clis`. `type` reports an installed CLI as its own
kind (`type -t` prints `cli`, a sixth word beside bash's five, because reusing
`file` would promise `type -p` a path that does not exist). `which` prints the
bare name, never a fabricated path.
## Notion data sources
The notion backend speaks **`Notion-Version: 2025-09-03`**, the generation that
split a database into a container plus one or more *data sources*. The split is
not cosmetic and it is not optional:
- **The database object no longer carries `properties`.** The column schema
lives only on the data source, so `GET /v1/databases/{id}` answers with
`data_sources: [{id, name}]` and nothing to render a schema from. Do not
synthesize one back onto the container; the fake deliberately omits it so
anything that still reads a column list off a database fails loudly.
- **The mount nests accordingly**, because that is where the data is:
`databases/<Title>__<db-id>/database.json` is the container, and
`databases/<Title>__<db-id>/<Name>__<ds-id>/data_source.json` is the schema,
with the row pages under the data source. A row page therefore sits at depth
4 under `databases/`, not 3; `readdir`/`read`/`stat` all key on that in both
languages. The name stutters for a single-source database because Notion
names the auto-created data source after its database, which is honest and
disappears the moment a database holds two.
- **A row's parent is its data source**: `{type: "data_source_id", data_source_id, database_id}`. The database id rides along because Notion
kept emitting it, and the fake stores rows keyed by database, which is the
same fact one derivation away.
- **`/search` rejects `filter.value = "database"`** at this version; the
searchable schema-bearing object is `data_source`. Listing databases is
therefore "search data sources, take the distinct parents, retrieve each",
which costs one extra call per database and is the only way to get the
title and url the directory is named and rendered from.
- ids are **not interchangeable**: a data source id is not its database id.
The fake derives one from the other (`d5000000` + the database's tail) rather
than reusing it, precisely so an id mix-up cannot pass.
## Mount boundaries
A mount root is not an ordinary directory, and a mount nested inside
another mount's tree is invisible to the backend that owns the parent:
the child's keys live in a different resource, so the parent's `readdir`
never lists it. Two mechanisms follow from that, and they are separate.
- **`MountView` is how a command sees the boundaries** (`ops/types.py`,
`ops/types.ts`), and it is offered the way `LinkView` is: a command
opts in by naming a `mounts` parameter, `execute_cmd`/`executeCmd`
delivers it only to handlers that do, and there is no list of
boundary-aware commands anywhere. It carries `descendants` (mount
roots strictly under a path), `is_root`, and `root_of`.
A traversal command that renders **lines** does not need it: the
by absolute path (the bundled `libosxfuse.2.dylib` is a stub with that
install name), so `backend=fskit` + `volname` reach macFUSE 5.x's own
libfuse via `appendMountOptions` in `mount.ts`. Verified live. Caveat: a TS
fskit mount intermittently wedged on a write op in testing (probed from
child processes), and a dead FSKit volume blocks mount-table enumeration
system-wide until the macFUSE appex process is killed; treat fskit from TS
as read-mostly and experimental.
A mount is ready only when `os.path.ismount` says so, never when the
`/Volumes` entry merely exists: macFUSE creates that directory while
mounting and leaves it behind if the FSKit handoff fails, which reads as a
live mount and then fails with ENOENT on the first read.
**Python fskit mounts have the full write surface, and only because of
`fuse/darwin.py`.** The FSKit shim finalizes every created item through
macFUSE's Darwin-only `setattr_x` and routes rename through `renamex`;
mfusepy leaves those `fuse_operations` slots as reserved NULLs, so without
the extension module create/mkdir fail with ENOSYS *after* the op already
applied and rename never reaches userspace (verified by libfuse wire
trace: CREATE success, then SETATTR -78). Do not remove the
`install_macfuse_extensions()` call in `mount.py`, and keep the struct
tail in sync with macFUSE's fuse.h if mfusepy changes layout. Pinned in
`integ/fuse/truth_fskit.json` on macFUSE 5.3.3 / macOS 26.
TS fskit stays read-mostly: fuse-native's compiled op table cannot gain
new C callbacks from JS. Upstream shim caveats that remain for both:
exec-until-first-read fails (macfuse#1181) and root readdir cache cannot
be invalidated (macfuse#1165).
`integ/fuse/fskit.py` is the only coverage of a real FSKit mount, and it only
runs on a Mac with macFUSE 5.x: on the `integ-fskit-macos` job a hosted
runner installs and enables everything but the mount request never reaches
macFUSE, so the job reports that one known timeout signature as a skip and
stays green; any other failure there is real.
- **Directory and unknown sizes.** `getattr` reports `st_size` 0 for directories and for API-backed size-unknown files that have not been opened recently. Reads stay correct because Python mounts with `direct_io` (kernel reads to EOF regardless of st_size) AND `attr_timeout=0` (post-open fstat routes to `getattr(path, fh)`, which serves the real size of the open-hydrated content); prefetched bytes live in a 30s TTL cache (`PREFETCH_TTL`) so release-then-stat does not refetch. All three pieces are load-bearing: without `attr_timeout=0`, `wc -c` prints 0, BSD `cp` copies 0 bytes, and `tail -c` dumps the whole file; without `direct_io`, `cat` reads 0 bytes on macOS. Do not "fix" getattr to report real sizes eagerly (one API fetch per `ls -l` entry), and do not report fake sizes: stat-only tools (`tar`, `rsync`, `test -s`) seeing 0 matches procfs precedent. TypeScript uses the same recipe: `@zkochan/fuse-native` doesn't serialize a `direct_io` option, so `mount.ts` appends it to the option string at runtime (`appendDirectIO`; a pnpm patch would not reach consumers), plus `attrTimeout: '0'` + fgetattr. The old 100 MiB sentinel is gone; do not reintroduce it.
- **TS FUSE mounts are served by the mounting process's event loop.** Never touch your own mountpoint synchronously (`readFileSync`, `statSync`, `execFileSync`) from the process that created the mount: the call deadlocks the loop that must answer it, the kernel times out, and every later op fails with `Device not configured`/`ENOTCONN` — which looks exactly like a broken mount. Probe from a child process or use async APIs (see `examples/typescript/fuse/helper.ts`). Python is immune (FUSE loop runs on a thread).
- **`FileStat.size` must be the rendered content's byte length or `None`, never a storage-side or source-side number.** A confidently wrong size is worse than an unknown one: over FUSE it makes `wc -c`/`ls -l` lie and risks truncated copies, while `None` rides the unknown-size machinery above. Postgres `rows.jsonl` (on-disk `table_size_bytes` vs rendered JSONL), Dify documents (uploaded source size vs rendered segment text), Gmail messages (`sizeEstimate` vs rendered `.gmail.json`), Drive-rendered google-apps files (Drive storage size vs rendered `.gdoc/.gsheet/.gslide` JSON; raw binary downloads keep Drive's size), and Microsoft Graph folders (OneDrive/SharePoint report an aggregate subtree `size` on the folder facet, not any content length) made this mistake; their storage/source numbers now live in `extra` (`size_bytes` / `source_size` / `size_estimate`). Do not reintroduce it in new backends. Graph's `folder.childCount` rides along in `extra` too, and is what `find -empty` reads for a directory.
- **macOS allows only one FUSE mount per process.** The second mount dies with `fuse: cannot register signal source` (mfusepy registers libfuse signal handlers, which only the first mount in a process can claim). Multi-mount scenarios (`integ/fuse.py` mounts two) pass only on Linux; do not debug them as regressions on macOS. A failed run leaks the first mount: list with `mount | grep MirageFS`, clean with `umount <mountpoint>`.
- **macOS allows only one FUSE mount per process.** The second mount dies with `fuse: cannot register signal source` (mfusepy registers libfuse signal handlers, which only the first mount in a process can claim). Multi-mount scenarios (`integ/fuse/fuse.py` mounts two) pass only on Linux; do not debug them as regressions on macOS. A failed run leaks the first mount: list with `mount | grep MirageFS`, clean with `umount <mountpoint>`.
- **Windows (WinFsp) conventions differ in three ways**, all handled in the python mount path (`_prepare_mountpoint`, `_await_ready`, the teardown branches): the mountpoint must NOT exist (WinFsp creates it; an existing dir fails with "mount point in use"), `os.path.ismount` never sees WinFsp directory mounts (readiness = bare existence), and there is no `fusermount` (WinFsp unmounts when the serving process exits). Ownership: mount with `uid=-1,gid=-1` (WinFsp builtin: files owned by the mounting user); never report raw POSIX ids into the SFU/Cygwin SID mapping, and `os.getuid` does not exist there (MirageFS caches a guarded uid/gid once). Behavior quirk: Windows cannot stat without opening a handle, so size-unknown files hydrate on first stat and report their real size even "pre-open" (multi-mount per process works). The `integ-fuse-windows` job is advisory (not in the gate).
## Development Setup
@@ -100,7 +509,108 @@ Invoke the venv's `pre-commit` binary directly (not via `uv --directory python r
- **Shell-style commands** (cat, grep, du, find, head, tail, wc, ls, etc.) follow POSIX / Unix coreutils semantics as much as possible; match BSD/GNU behavior and document any deliberate divergence. Pin exact GNU behavior with docker (`debian:stable-slim`) before changing command semantics.
- **`find -size` is strict and rounds up.** GNU `+N` keeps `ceil(size/unit) > N`, `-N` keeps `ceil(size/unit) < N`, bare `N` keeps `ceil(size/unit) == N` (so `-size -1k` matches only empty files and `+0c` excludes empty ones). The parsers (`_parse_size` / `parseSize`) translate this once into inclusive byte bounds; backend cores just keep `min_size <= size <= max_size` and must not re-interpret the spec. Deliberate divergence: directories count as size 0 (GNU compares the inode size, e.g. 4096 on ext4), which matches what `find` sees over a mirage FUSE mount.
- **`du` has one backend contract: `size` and `entries`.** Each backend exposes `core/<backend>/du/size.py` (recursive byte total for one path) and `core/<backend>/du/entries.py` (per-file breakdown), wired as `du_size` / `du_entries` on the adapter. `entries` returns `(entries, total)` where entries are **leaf files only, in mount-relativepath space, with no summary row**; the generic lifts them onto virtual paths (`to_virtual`, via `mount_prefix_of`) and re-spells them as the operand was typed (`rebase_raw`). A backend that returns backend-key paths, or appends its own roll-up row, makes two mounts holding the same filename render identical lines. Do not reintroduce a second shape; the old flat-list `du_multi` contract is gone.
- **`find`'s row for the start point is the generic's, not the backend's.** GNU lists a start point before descending into it, and every native find op used to decide that row from its own listing, which is only a proxy for existence: an object store holding no keys under the prefix and no directory marker reported nothing at all for a directory `test -d` and `tree` both saw, and ssh called every directory non-empty so `-empty` never matched one. The generic stats the start point through the dispatcher (`resolve_start`, which asks both channels a backend can answer on via `resolve_path_stat`, since on a prefix store a directory is the set of keys under it rather than an object), takes one readdir for `-empty` (`dir_empty`, wired by the builder), and then replaces whatever row the backend produced for it (`with_root_row` / `withRootRow`). A native find op may still emit the start path and they all do, because it is the answer when no dispatcher probe is wired (a command constructed outside a workspace); with one wired it is discarded. A new backend only has to report descendants.
- **`find` classifies walked entries through `stat`, never by name.** The
walk's one in-band proof is a trailing slash on a cold listing (box, gdrive
and dropbox mark folders that way, and no backend renders a file with one);
every other entry is classified by the index-backed `stat` that the same
readdir just populated, so the lookup is RAM, not another API call. There is
no per-backend `is_dir_name`/`isDirName` hint: those heuristics guessed
wrong as soon as a child's name was user-controlled (email and gmail
attachments and slack uploads carry whatever name the sender gave them, and
were reported as directories, so `find -type f` missed them). Do not
reintroduce name-based classification in a backend; if stat misclassifies an
entry, fix that backend's stat.
- **An archiver walks a directory operand; it does not read it.** `tar`
and `zip` decide every member first (`plan_create` / `planCreate` in
`generic/tar/create.*`, `plan_zip` / `planZip` in
`generic/zip_cmd.*`) and only then write, which is what lets an
exclusion prune a whole subtree and keeps the ordering stable. Both
plans are built on **one traversal**, `scan_operand` / `scanOperand`
(`generic/archive/walk.*`), which merges three sources no single one
can see: the backend walk (reusing find's `walk_find` / `walkFind`, so
an archiver classifies an entry through `stat` exactly as find does,
never by name), the namespace's symlinks, and the mount table. It
reports paths, never names, because naming is exactly where the two
formats disagree; the two things they disagree about in the traversal
itself are parameters (`dereference`, `recurse`), so **a third
archiver adds a caller, not a second walk**. Members are named from
`PathSpec.raw_path`, so `tar -C d x` stores `x`, not `d/x`.
**A directory is its own member**, with GNU's trailing slash and no
content, which is the only record an empty directory leaves and the
reason extraction has to `mkdir` for one. **A symlink is a symlink
member** (`SYMTYPE`, target in `linkname`), never a file of its
target's bytes, unless `-h` says to follow it.
**Two links to one target are not a loop**, and both are archived; the
only loop is one `resolve` refuses to resolve, since the namespace
already walks the chain under a hop limit and raises `CycleError` at
the end of it. That arrives as a fatal `Problem` carrying GNU's
`Too many levels of symbolic links`, reported per member with the
directory entry kept, rather than as an exception that aborts the
plan. **Every `-C` is checked, not just the last**: GNU chdirs at each
one and fails at the first it cannot enter, so the option accumulates
(`multiple=True`) and the planner walks the list.
Two deliberate divergences from GNU, both documented in place:
siblings are sorted rather than emitted in readdir order (the same
choice `du` makes, for the same reason), and a descendant mount is
never crossed (see "Mount boundaries"). Everything else is pinned
against GNU tar 1.35 on `debian:stable-slim`: the leading-slash
warning, `Cowardly refusing to create an empty archive` (exit 2), a
per-operand `Cannot stat` plus one trailer (exit 2, and the other
operands still archive), a `-C` it cannot enter (exit 2, no archive
written), and `archive cannot contain itself; not dumped` (exit 0).
A backend error must never reach the user as itself: an unreadable
operand is reported in virtual path space with tar's wording, because
the raw `IsADirectoryError` leaked the host path behind a disk mount.
- **`zip` is Info-ZIP, which inverts tar's two defaults.** A directory
operand contributes only its own entry unless `-r` says to descend,
and a symlink is *followed* unless `-y` says to store the link, where
tar always descends and always stores unless `-h`. Both are just the
`recurse` / `dereference` arguments to the shared scan. The rest is
pinned against Info-ZIP 3.0 on `debian:stable-slim`: a leading slash
is stripped **in silence** (tar warns, zip does not), `-j` junks to
the basename and drops directory entries entirely, `-x` is
**anchored** on the whole stored name (`d/sub/*` matches, `sub/*` does
not) where tar's `--exclude` is unanchored, an unreachable operand is
`\tzip warning: name not matched: <name>` and does not stop the run,
and a run that matched nothing prints `zip error: Nothing to do!`,
exits **12**, and writes no archive. `-q` silences the warnings but
never that error. Two deliberate divergences: `-x` takes one pattern
per occurrence (mirage's spec has no variadic option value, and
`-x a -x b` says the same thing), and the `adding:` line carries no
`(deflated N%)` suffix, since the ratio depends on the compressor and
would differ between the two languages.
- **Tar formats come from a library, not from hand-rolled block code, and
bzip2 is read-only in TypeScript.** Python builds archives with stdlib
`tarfile`; TypeScript uses `modern-tar` behind `tar_helper.ts`, which
keeps the `TarEntry` shape (`name`/`data`/`isFile`/`isDir`/`linkname`)
the two call sites already speak and is the only place the dependency
is named. Do not go back to writing ustar blocks by hand: the version
that did truncated any name past 100 bytes instead of using the ustar
`prefix` field or a PAX header (so a deep member extracted to the wrong
path), and read a PAX/GNU extension block as if it were a member (so
`tar -t` on any archive GNU or Python wrote listed a phantom
`././@PaxHeader` row). `writeTar`/`readTar` are async for this reason.
Compression is a registry (`registerCompressionCodec`): gzip is built
in via `CompressionStream`, and a codec may be **decompress-only**,
which is the one deliberate py/ts divergence here. Python reads and
writes `.tar.bz2` because `bz2` is stdlib; TypeScript only reads one,
because every JavaScript bzip2 *compressor* is GPL (`compressjs`,
`archive-wasm`) and an Apache-2.0 package cannot ship that, while
`seek-bzip` (MIT) decodes. `tar -cj` therefore exits 1 with
`tar: bzip2 not supported`, the same answer browser core already gives
for an unregistered codec. If a permissively licensed bzip2 compressor
appears, adding `compress` to that one codec closes the gap with no
other change.
- **The TypeScript `walkFind` answers in mount-relative keys; the Python
`walk_find` answers in virtual paths.** TS's stands in for a backend's
native find op, so a caller that needs virtual paths (tar does, to
name members and compare against mount prefixes) lifts them with
`mountPrefixOf` the way `findGeneric` does. That lift lives once, in
`generic_bind/archive_io.*`, which is where both archivers get their
walk. This asymmetry is real and has bitten once: a unit test on an
unprefixed mount cannot see it, so cover a prefixed mount too.
- **`du` has one backend contract: `size` and `entries`.** Each backend exposes `core/<backend>/du/size.py` (recursive byte total for one path) and `core/<backend>/du/entries.py` (per-file breakdown), wired as `du_size` / `du_entries` on the adapter. `entries` returns `(entries, total)` where entries are **leaf files only, in mount-relative path space, with no summary row**; the generic lifts them onto virtual paths (`to_virtual`, via `mount_prefix_of`) and re-spells them as the operand was typed (`respell_raw`). A backend that returns backend-key paths, or appends its own roll-up row, makes two mounts holding the same filename render identical lines. Do not reintroduce a second shape; the old flat-list `du_multi` contract is gone.
- **`du` prints a line per directory, derived not walked.** GNU prints one line per directory with its recursive total, post-order (children before parents), plus one per file under `-a`. Backends only ever report leaf files, so the generic derives the directory rows by summing each leaf into every ancestor (`rollup`, same name both languages), then emits post-order with siblings sorted. Two deliberate divergences: GNU orders siblings by `readdir` (filesystem-dependent), mirage sorts them; and an empty directory is invisible to mirage because no leaf points at it. Sizes are bytes, not GNU's 1 KiB blocks, since an object store has no block size. `--max-depth` prunes only what is printed, never the walk, because every printed total still covers the whole subtree. Verify changes with the differential harness against `debian:stable-slim`: paths, exit codes and stderr must match GNU exactly.
- **`du` usage errors exit 1, not 2.** `du` is absent from `USAGE_EXIT`, which is correct: GNU du exits 1 for `-s` with `-a` ("cannot both summarize and show all entries"), `-s` with `--max-depth` ("warning: summarizing conflicts with --max-depth=N"), and a bad depth ("invalid maximum depth 'x'"). All three are raised by `parse_flags` / `parseDuFlags`*before* any I/O, mirroring GNU's option-parse order: the depth is parsed as the option is read, so a bad depth wins over the conflict checks. An unreadable operand is not a usage error: GNU names it (`du: cannot access 'x': No such file or directory`), prints every other operand, and exits 1, and still prints `0 total` under `-c` when every operand failed. With no operand at all, du measures the working directory; it never says "missing operand".
- **`du` walks are bounded.** Backends with no native du op are walked one `readdir` at a time, which on an API tree is one request per directory. `CommandIO.max_du_entries` caps that walk; when it trips, `du` prints what it accounted for, writes a notice to stderr and exits 1 (GNU's behavior for a tree it could not fully read), rather than hanging or silently reporting a wrong number. Slack sets a low cap (`DU_MAX_ENTRIES`) because it exposes a directory per conversation per day against a ~50/minute rate limit.
@@ -109,7 +619,7 @@ Invoke the venv's `pre-commit` binary directly (not via `uv --directory python r
- **Do not add `__init__.py` files under `tests/`.** Tests are namespace packages and pytest discovers them without `__init__.py`. Don't create one when adding a new test directory.
- **Monkeypatching a backend command module in tests:** the command imports its helpers by value (`from mirage.core.<backend>.read import read_bytes`), so to intercept them you must rebind the name inside the command module, not the core source module. But the command module is hard to reach: the backend package re-exports the command function in `__init__.py` (`from .cat import cat`), which shadows the submodule of the same name, so `import mirage.commands.builtin.<backend>.cat as mod`, `from ...<backend> import cat as mod`, and even pytest's string target `monkeypatch.setattr("mirage.commands.builtin.<backend>.cat.read_bytes", fake)` all resolve to the function, not the module (`AttributeError`). The command is also wrapped by `@command`, so `cat.__globals__` is the decorator's module. Reach the real command-module namespace through the unwrapped function and patch the dict: `monkeypatch.setitem(cat.__wrapped__.__globals__, "read_bytes", fake)`.
- Avoid add any comments or docstrings on the top of the file.
-Do not create nested functions.
-**A nested function must capture the scope around it.** Nesting is for closures: the def reads a name from the enclosing function, or binds one through a parameter default (`def f(m, _n=n)`, the loop-variable idiom). Everything the architecture nests is that shape — op factories (`ops/generic/factory.py`), provision builders, the read-through cache, and every decorator's wrapper — and those stay. What is banned is the nesting that buys nothing: a helper written inside a function although it reads only its own arguments, which rebuilds a function object per call and hides a testable unit where no test can reach it. Put that one at module level. Enforced for Python by `tests/test_nested_functions_are_closures.py`.
- Add type to Args for docstring.
- Do not add comment after each line of code in the format of "# 10MB - trigger segmentation for files larger than this". The most you can add is "# 10MB".
- For all imports you need to put to the top of the file. Don't have imports within each function.
@@ -120,7 +630,31 @@ Invoke the venv's `pre-commit` binary directly (not via `uv --directory python r
- Don't add too many printings or comments in the code.
- Don't add README.md unless I ask you to do so.
- Use uv add to install new dependencies.
- **Command wrappers and flags.** The dispatcher passes parsed command-line flags as keyword arguments. Wrappers must declare dispatcher-injected parameters (`stdin`, `index`, `prefix`) explicitly in their signature — never fish them out of `**flags` with `.get()`. Treat `**flags: object` as an opaque bag of true command-line flags and forward it wholesale to the generic command. When a wrapper genuinely needs a flag value itself (e.g. a search push-down), read it through `FlagView` (`fl = FlagView(flags)` then `fl.as_bool("F")`, `fl.as_int("m")`, `fl.as_str("type")`, `fl.as_list("e")`) or a shared domain accessor like `pattern_arg` — never raw `flags.get(...)` / isinstance chains.
- **Generic commands own flag interpretation.** Backend wrappers are wiring only (glob resolution, backend I/O injection, pass-through of `texts` and `flags`); all flag semantics live in the generic command for that family, mirroring the TS generics. Adding or changing a flag should touch the spec and the generic, not N wrappers.
- **Command handlers take `(accessor, paths, texts, opts)` — the same four positionals in both languages.** The dispatcher (`Mount.execute_cmd` / `Mount.executeCmd`) constructs one `CommandOpts` per invocation carrying stdin, the flag bag, cwd, the mount prefix, the index, and every namespace fact; the provision path builds the same bag with `command`/`spec` set (`ProvisionFn`has the same four-positional shape). Handlers never declare a flag or an injected fact as a parameter — `tests/commands/test_no_dead_flag_params.py` pins the exact signature. When a wrapper genuinely needs a flag value itself (e.g. a search push-down), read it through a spec-bound `FlagView` (`fl = FlagView(opts.flags, spec=SPECS["grep"])` then `fl.as_bool("F")`, `fl.as_int("m")`, `fl.as_str("type")`, `fl.as_list("e")`) or a shared domain accessor like `pattern_arg` — never raw `flags.get(...)` / isinstance chains, and never a raw `kwargs`/`_extra` read either (`tests/commands/test_no_raw_flag_reads.py`). To override a flag before delegating, pass `dataclasses.replace(opts, flags=bag)` down, never a hand-built `CommandOpts`. **A PATH-typed flag reaches a python command as a `PathSpec`, not a string** — the executor promotes it (`workspace/executor/command/flags.py`), so read it with `fl.as_paths(name)`; `as_str` reads it as absent and the operand is silently never used. TypeScript's bag carries the resolved virtual-path string instead, so its twin is `fl.asStr(name)`.
- **Generic commands own flag interpretation.** Backend wrappers are wiring only (glob resolution, backend I/O injection, pass-through of `texts` and `opts`); all flag semantics live in the generic command for that family, mirroring the TS generics. Adding or changing a flag should touch the spec and the generic, not N wrappers. Every literal `FlagView` query name must be a dest of a spec bound in the same module — `tests/commands/test_flag_query_names.py` and `commands/flag_query_names.test.ts` fail on a typo'd spelling without needing the code path to run.
- **Generics parse flags once into a frozen struct.** Each generic defines a `@dataclass(frozen=True, slots=True)` flag struct plus a module-level `parse_flags(fl, ...)` (mirroring the TS `parseFlags` struct); the function body reads only struct attributes, never string keys. Construct the FlagView with the command's spec (`FlagView(flags, spec=SPECS["grep"])`) so a typo in a flag name raises KeyError instead of silently reading as False/None.
- **Never annotate a parameter as `object`.** Use the real type: a backend handle is `accessor: Accessor` (`mirage.accessor.base`), an index is `index: IndexCacheStore | None` (`mirage.cache.index`). Ignored variadics are still typed (`*texts: str`). `object` is only acceptable as the value type of an opaque flag bag (`**flags: object`).
- **Never annotate anything as `object`** — not a parameter, not a return, not a type argument (`dict[str, object]`, `Callable[..., object]`). `object` reads as "we did not decide": it accepts bytes where JSON was meant and a PathSpec where a flag value was meant, so every use site pays for it with an isinstance chain back to the set the author had in mind. Name the real type instead:
- a parsed command-line flag is `FlagValue` (`mirage.commands.spec.types`) — `**flags: FlagValue`, `Mapping[str, FlagValue]`, `FlagValue | None` for a single raw read. The TypeScript side has always called it `FlagValue` too (`commands/spec/types.ts`); keep the two spellings identical.
- a decoded JSON payload or an API field is `JsonValue` (`mirage.types`). It is recursive, so it is spelled as a forward-reference string until the floor is 3.12 — a union with it must be quoted: `Awaitable["JsonValue | X"]`.
- a path is `str | PathSpec`, a backend handle is `accessor: Accessor` (`mirage.accessor.base`), an index is `index: IndexCacheStore | None` (`mirage.cache.index`), a stat function is `StatFn` (`mirage.types`). Ignored variadics are still typed (`*texts: str`).
- a sentinel is a one-member `Enum`, never `object()`; that keeps it distinguishable from the real values sharing the variable.
`tests/commands/test_no_object_annotations.py` enforces this and carries the only exemptions: four Python protocol methods (`__setattr__`, `__contains__`, `Mapping.pop`) whose signatures the language fixes, and one guard whose whole job is to catch a value the annotations already claim cannot arrive. Adding to that allowlist needs the same kind of reason.
- **Every session write goes through `SessionView.set`.** A deployment refuses a
name (`AWS_*`, a credential) with a `pre_session` rule, and a writer that
reaches `session.env` directly makes that rule advisory. `export` and a plain
assignment always cleared the gate; the expansion-time writers did not, so
`${X:=d}`, `$((X=5))`, `(( ))`, `printf -v` and `for ((X=0; ...))` each wrote
past every policy. The view is threaded into expansion explicitly rather than
read from ambient state, and defaults to `None` (correct outside a workspace,
where the env is the only state there is). Reads (`$X`) stay sync; only the
writers await. `tests/workspace/expand/test_session_gate.py` and
`integ/session/writers.json` hold the line in both languages.
- **The record client is a substrate, not a session detail.** Sessions, the
namespace node table and workspace metadata are three tables that persist the
same way, so the keyed-record clients live in `workspace/record/`
(`DiskRecordClient` with an `O_CREAT|O_EXCL` lockfile and `rename(2)`,
`S3RecordClient` with If-Match CAS) and import none of the three. They used to
live inside the session package, which made the other two import upward into
it; a layering test in each language fails if that returns. The two clients
deliberately do not merge: different concurrency contract, different blob
Mirage is **a Unified Virtual File System for AI Agents**: it mounts services and data sources like S3, Google Drive, Slack, Gmail, and Redis side-by-side as one filesystem. Any LLM that already knows bash can read, grep, and pipe across every backend out of the box, with zero new vocabulary.
Mirage plugs into agent frameworks as a sandbox or tool layer. POSIX operations such as `read` can also be customized per resource and filetype, e.g. reading a PDF returns parsed pages instead of raw bytes.
Mirage plugs into agent frameworks as a sandbox or tool layer. POSIX operations such as `read` can also be customized per resource and filetype: Mirage ships no filetype renderers, so a format renders however you register it, and a command registered for one resource and extension wins over the generic one.
Build JSON with a binding rather than by hand: the value arrives as a
value, so quotes and newlines in it need no escaping.
```bash
# text -> JSON array, no string surgery
printf 'alpha\nbeta\n' > /data/lines.txt
jq -Rn -c '[inputs]' /data/lines.txt
["alpha","beta"]
# a shell value that contains quotes
jq -n -c --arg v 'a"b' '{msg: $v}'
{"msg":"a\"b"}
# a whole file as one JSON string
jq -n -c --rawfile body /data/lines.txt '{text: $body}'
{"text":"alpha\nbeta\n"}
```
`$ARGS` is always defined, carrying `named` (the `--arg` family) and
`positional` (`--args` / `--jsonargs`).
Three limits worth knowing. `inputs` is bound to whatever is still
unread, so a program that drains it (`[., inputs]`, `reduce inputs as
$x`) runs once and sees everything, but the stateful single `input` and a
partial drain (`first(inputs)`) are not modeled. `--stream` reads whole
documents and expands them, which matches jq except that jq's incremental
parser splits the closing event of an input with no trailing newline into
its own `-s` group. And `--seq` reads and writes the separator, but drops
text before the first one silently where jq names it on stderr.
Not implemented, and reported as an unknown option rather than quietly
ignored: `-C` (colorized output, which an agent would only have to strip
again), `-L` (no module system, so `include` has nothing to search),
`--stream-errors` (it reports truncated-parse errors, which whole-value
reads never produce), and `--build-configuration`.
## Supported bash syntax
Mirage Bash is a tree-sitter-bash parser plus a custom executor. It implements the constructs LLMs reach for most often. What is not supported returns a clear, parseable error so an agent can self-correct on its next turn.
@@ -119,8 +223,9 @@ Mirage Bash is a tree-sitter-bash parser plus a custom executor. It implements t
- **Builtin options (GNU semantics):** `echo -n/-e/-E` (leading-word option rule: `echo hi -n` prints `hi -n`), `read -r`, `xargs -n/-0/-d/-r/--` (batching, GNU exit codes: `123` when an invocation fails, `126`/`127` stop the run), `timeout DURATION` with `s`/`m`/`h`/`d` suffixes (kills at the deadline with exit `124`, usage errors exit `125`). `shift` and `return` report bash's `numeric argument required` errors.
- **Name lookup:** `type name` reports what a name resolves to (`type -t` prints one of `keyword`, `function`, `cli`, `builtin`; `type -a` lists every layer holding the name), `which name` prints the name of anything runnable (there is no PATH, so there is no path to print) and reports a miss through exit `1` alone, and `man name` renders a page: a command's spec, or an installed CLI's own `--help` tree (`man linear issue create`).
- **Globs:** `*`, `?`, `[...]` classes and `[!...]` negation (Python `fnmatch` semantics in both implementations), resolved by the shell or pushed down to the resource.
Same for the `_vfs` examples (`patchNodeFs`): Python reports `4 ops, 20760 bytes`, TS reports `0 ops, 0 bytes`.
## Cause
In TypeScript, `ws.fs.readFile` / `ws.fs.readdir` (used by `patchNodeFs` and the FUSE layer in `packages/node/src/fuse/fs.ts`) bypass the record-keeping dispatch, so nothing lands in `ws.records`. In Python, the equivalent VFS and FUSE paths go through `apply_io`, which appends `OpRecord`s, so `ws.ops.records` counts every read.
## Scope
Cross-backend, not notion-specific: Linear's TS `_vfs`/`_fuse` examples have the same gap, and any resource read through `ws.fs` or a FUSE mount is unrecorded. Command execution (`ws.execute`) records normally in both languages.
## Fix direction
Route `ws.fs` reads (or at least the FUSE/`patchNodeFs` entry points) through the same dispatch/recording layer the executor uses, mirroring Python's `apply_io` accounting. Then the example `Stats:` lines become comparable across languages.
## Status
Open. Found 2026-06-09 while verifying notion example parity (PR #226); deferred because the fix belongs in the workspace fs op layer, not per-backend.
description: Give a Haystack Agent a bash tool over a Mirage workspace via MirageShellTool.
icon: /images/haystack-logo.svg
---
[Haystack](https://haystack.deepset.ai) is deepset's framework for building LLM applications and agents. The integration is maintained by deepset and lives in their [haystack-core-integrations](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mirage) repository, so it installs as its own package rather than as a `mirage-ai` extra.
## Install
```bash
uv add mirage-haystack
```
<Note>
`mirage-haystack` pins an exact `mirage-ai` version rather than tracking the latest, because Mirage is pre-1.0. Check the [integration's `pyproject.toml`](https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/mirage/pyproject.toml) for which one, since installing it may move an existing `mirage-ai` in your environment.
</Note>
## Usage
Describe the mounts, wrap the workspace in a tool, hand the tool to an `Agent`.
```python
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
ChatMessage.from_user("How many lines in /s3/log.txt mention 'alert'?"),
])
print(result["messages"][-1].text)
```
The tool exposes a single `command` parameter, so the model writes ordinary bash and pipes across mounts. Its description is generated from the mount tree, so the model is told which paths exist without you writing a prompt for it.
## Guarding what the agent can do
Two controls, and they are not interchangeable.
`read_only=True` on a mount is the write boundary. Mirage refuses every write to that mount whatever command is used, so this is what prevents modification and deletion.
`allowed_commands` restricts which command names may run. It is checked against every command Mirage would execute, including ones nested in `$(...)`, backticks and subshells, so `ls "$(rm x)"` is rejected unless `rm` is allowed too. Treat it as steering, not a sandbox: allowing a command that runs other commands (`eval`, `bash`, `sh`, `source`, `xargs`) effectively allows anything.
Commands never reach the host shell either way. Mirage interprets them itself, so the blast radius is the mounts you attached.
## Exports
| Symbol | Purpose |
| --- | --- |
| `MirageWorkspace` | Declares the mount tree, and runs commands against it with `run` / `run_async`. Serializable with `to_dict` / `from_dict`. |
| `MirageMount` | One mount: `path`, `resource`, `config`, `read_only`. |
| `MirageShellTool` | The Haystack `Tool` that gives an `Agent` the bash surface. |
| `MirageError` | Base error, with `MirageConfigError` and `MirageCommandNotAllowedError`. |
## Links
- [Integration page](https://haystack.deepset.ai/integrations/mirage) on haystack.deepset.ai.
- [Source and README](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mirage) in `haystack-core-integrations`.
- [`mirage-haystack`](https://pypi.org/project/mirage-haystack) on PyPI.
| `gh api` pretty-prints the response; real gh prints the body verbatim | the reader is usually an agent or `jq`, and an indented body is what a human reads |
| `gh repo view` has no `--json`/`-q`/`-t`; the default text view matches | upstream's field set is GraphQL's, with 75 names; `gh api repos/O/R` is the JSON route |
| `--paginate`, `-q/--jq`, `-H`, `--input`, `--silent`, `-i` are not built | not built yet; pipe to `jq` for selection, and paginate with `-f page=2` |
| `-F` reads no `@file` values, and `key[sub]=` / `key[]=` nesting is flat | not built yet |
| `repo rename` has no `-y/--yes` | there is no prompt to skip: nothing in a workspace is interactive |
| `repo fork` has no `--clone`, `--remote`, `--org` | those act on a host checkout and a git remote, which a workspace has neither of |
| a `HOST/` prefix is accepted but never routes | one install is one account on one host; a second host is a second install |
The interactive and host-side verbs (`auth`, `gist`, `codespace`,
`browse`, `repo clone`) are out of scope for a virtualized CLI, the same
Mount it like any builtin: `Workspace({"/jira/": make_jira_resource(cfg)})`.
The escape hatches mirror what builtins use: optional `CommandIO` fields unlock more surface (`write` enables the byte-mutation family; `find`/`du_total`/`du_all` become native fast paths; `is_dir_name` hints virtual directories), `overrides=` suppresses a generic command you replace, and `commands=[...]` adds bespoke `@command` verbs.
The escape hatches mirror what builtins use: optional `CommandIO` fields unlock more surface (`write` enables the byte-mutation family; `find`/`du_total`/`du_all` become native fast paths), `overrides=` suppresses a generic command you replace, and `commands=[...]` adds bespoke `@command` verbs.
VFS/FUSE ops are derived from the same table automatically (`make_generic_ops` under the hood): read/readdir/stat plus whatever mutations the table carries. Pass `ops=[...]` only for irregular handlers (they shadow same-named derived ops), or `auto_ops=False` to opt out.
@@ -79,6 +79,7 @@ python/mirage/
resource/<name>/
__init__.py
config.py
prompt.py
<name>.py
accessor/<name>.py
core/<name>/
@@ -86,12 +87,10 @@ python/mirage/
readdir.py
stat.py
ops/<name>/
__init__.py
read.py
readdir.py
stat.py
__init__.py # OPS derived from the CommandIO table
commands/builtin/<name>/
__init__.py
io.py # the CommandIO table
<resource-specific commands>.py
```
@@ -143,28 +142,35 @@ Add write, append, create, unlink, rename, or directory operations only when the
## 3. Ops Layer
Ops are thin typed adapters from the workspace dispatcher to core functions. Declare dispatcher-injected arguments explicitly and keep `**flags: object` opaque.
Ops are derived, not hand-written. `ops/<name>/__init__.py` builds the whole VFS/FUSE op family from the same `CommandIO` table the commands use:
```python
from mirage.accessor.base import Accessor
from mirage.cache.index import IndexCacheStore
from mirage.core.my_resource.read import read_bytes
from mirage.commands.builtin.my_resource.io import IO
from mirage.ops.generic import make_generic_ops
OPS = make_generic_ops("my_resource", IO)
```
`make_generic_ops` emits read/readdir/stat plus whatever mutations the table carries — a `CommandIO` slot updates commands and ops together, and ops whose table field is `None` are omitted. Knobs mirror backend semantics, e.g. `make_generic_ops("databricks_volume", IO, mkdir_parents=True)`.
Write a dedicated op module only for an irregular handler with no generic equivalent (a native `grep` push-down, a semantic `search`), and append it to the derived list:
```python
from mirage.core.my_resource.grep import grep_bytes
Mark mutation ops with `write=True`. Export the decorated functions as `OPS` from `ops/<name>/__init__.py`.
Mark a hand-written mutation op with `write=True` so `MountMode.READ` remains a real boundary (derived ops carry this from the table).
## 4. Commands
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Zecheng Zhang
