main
191 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zecheng Zhang
|
98fcdcb2e2 |
feat(resource): assemble a TypeScript backend from one CommandIO table
python's GenericResource wires the whole generic command set, the glob resolver and the VFS/FUSE ops from one table, and TypeScript had every ingredient but no class that assembled them, so a custom backend there was still written out by hand. Adds GenericResource<A>, generic over the accessor so the table is checked against the core functions it holds rather than against Accessor. It keeps TypeScript's own wiring style: commands() and ops() return arrays instead of python's register() loop mutating state in the constructor. And there is no sdk.ts, because core's barrel is 99 lines and gated in both directions while the ./* exports map already makes every module importable, so the five names the new example and doc reach for are the only additions to it. Both classes gained sizes_always_known and supports_snapshot. Without the first, no user-written backend could be mounted on FSKit at all, since resolve_backend refuses a resource that cannot size its files. The two one-file examples answer one shared truth file, so the two SDKs cannot drift without a red build. docs/typescript/resource/new.mdx is the twin of the python page, whose stale du_total/du_all is corrected to du. The layout baseline drops to 255: resource/generic was one of the counted divergences. |
||
|
Zecheng Zhang
|
43e8a67180 |
Merge pull request #877 from strukto-ai/fix/os-verb-routing
fix(os): route every path-taking os verb through the workspace |
||
|
Zecheng Zhang
|
0e59a6c659 |
fix(os): lstat a mounted link from its own node row
os.lstat rebuilt a link's stat_result from the target string, so uid and gid came back as the process defaults and a chown -h (os.lchown, or os.chown with follow_symlinks=False) could not be observed afterwards. It reads the namespace link row now, which is where every other no-follow surface already looks, and ls -l and os.lstat agree again. The readlink probe stays in front of the table read and is the gate, since the node table has no session: a hidden link is still absent. posix_mode answers LINK_MODE for a symlink row, because the bits on a link are never consulted. chmod -h still stores them and still shows nothing. |
||
|
Zecheng Zhang
|
d1eea898b8 |
fix(os): keep the process patch off while a backend serves an op
A disk mount whose root sits at or under its own virtual prefix hands the host a path is_mounted answers True for, so the patched os module routed the backend's own physical path back into the same backend and the process wedged instead of raising. ops/host_io.py is the bypass: the two patched doors read it, and the two places a backend actually runs (execute_op, execute_cmd) plus the watch delta walk set it. Streams are wrapped, since a backend opens its file on the first __anext__. It is a process-global depth rather than a ContextVar because aiofiles reaches the host through loop.run_in_executor, which drops the context. os.readlink off a mount hands back the host's answer untouched, so a bytes path answers bytes rather than a str of them. One truth file now runs twice, the second time with -X utf8=0, the mode where pathlib passes io.open's "locale" sentinel. |
||
|
Zecheng Zhang
|
da2c910fd0 |
fix(permissions): answer every operand's ask, and scope discovery to reachable verbs
A line whose operands were asked about by different rules presented only the deepest, so one nod ran the whole line. Every ask that wins a subject of its own is now reported and the door requires all of them, one at a time, spending a once grant only when the line is fully answered. The dsh read-only twin carried mount modes and hidden paths but not the source's command rules or hidden variables, so a role-denied CLI verb ran there. A mode bounds a mount; an account CLI reaches a service. man narrowed a CLI tree to the verbs the allow list reaches, through the same visibility question the bare listing already asked, generalized from a head word to a verb path. |
||
|
Zecheng Zhang
|
5639b62790 |
fix(os): route every path-taking os verb through the workspace
make_os_module patched 10 os names, so every other os call inside a Workspace block reached the host filesystem with a virtual path in hand. runtime/verbs.py and verbs.ts now classify every guest filesystem verb once: 23 route through an op door, 14 refuse with a condition, 30 pass through to the host, and an unclassified name defaults to ENOTSUP. patch_process installs the routing as attributes on the real os module instead of swapping sys.modules["os"], so a plain module-level import os routes and pathlib, shutil and glob follow. os.stat answers a real os.stat_result, which is what shutil.copystat and os.path.samefile need. |
||
|
Zecheng Zhang
|
5a44983251 |
fix(permissions): judge a line subject by subject, and keep a mount section inside its mount
Three codex findings, each reproduced before the fix and after.
A line is now judged one subject at a time. One global best match let a
deeper ask on the destination answer for a deny on the source, so
`cp /protected/secret /review/deep/out` came back as an approval request
and a nod meant for the destination carried the protected file out. The
matcher gained `subjects`, `rule_reach` and `rule_applies`; `decide`
resolves each subject by depth as before and then combines across them by
verb, since every path a line names has to survive it. The carve-out law
is unchanged on the operand it was written for.
A mount section's name pattern is anchored to the mount when the role
compiles. Both places those entries are read from have lost the section by
then: the hidden set is one list for the whole session, and the op door
matches a rule's paths without consulting rule.mount. So
`mounts./repo.paths.hide: ["*.pem"]` hid `/other/key.pem`, and a path-only
deny under `/repo` refused a read of it.
Also in this commit:
- `Outcome` moves to `policy/types`, `Decision` to a new `policy/config`,
and the verb ordering to `policy/constants`, in both languages.
- `integ/fixtures/config/accepted.json` becomes `blocks.json` and the
permission verbs get a file each (`allow`, `ask`, `deny`), with the
whole rule grammar per verb where there had been one case.
- integ gains the two combinations the audit found missing: a
multi-operand line, and a mount section's pattern stopping at its mount.
- `examples/{python,typescript}/permissions/` is a worked role document:
two roles over three mounts, top-level and mount rules at different
anchor depths, and one mount denied for one role and hidden for the
other. Output is identical in both languages and gated against
`integ/truth/permissions.json`.
- `parseSessionProfile` is exported, since a TypeScript embedder could not
declare a role without it.
|
||
|
Zecheng Zhang
|
7d40d85e35 | examples: load S3 data on a remote box through the ssh runtime | ||
|
Zecheng Zhang
|
cc7d537026 |
Merge pull request #839 from strukto-ai/chore/integ-txt-to-json
Retire the integ truth txt files for JSON |
||
|
Zecheng Zhang
|
f368743310 |
fix(integ): make the redis examples self-contained, recapture their truth
integ/truth/python/redis.json asserted a `ls /data/` of five entries when the example creates three. The two extras were leftovers in my local redis db 0: example_redis.py hardcoded that URL and ignored $REDIS_URL, db 0 is the polluted one here, and the example never cleaned up, so both --emit runs saw the same garbage and captured it as truth. On CI's fresh redis the listing is three entries and the check failed. integ/truth/python/redis_vfs.json had the same five entries for a different reason: it only ever passed because example_redis.py runs earlier in the same job and left its files in the shared mirage:fs: namespace. That is an ordering dependency nothing declared, and making example_redis.py tidy up is what exposed it. So both examples now read $REDIS_URL like their cache and index siblings already did, name the key prefix they use, and delete it on the way out. Running them in either order against a dirty db now gives the same output as a fresh one, and both truth files are recaptured from a clean db: redis.json 152 -> 139 matchers with one volatile, redis_vfs.json 20 matchers with none, which matches its TypeScript twin. example_redis_fuse.py gets the $REDIS_URL read too; it has no truth file but it carried the same hardcode. TypeScript needs no change: redis.ts already clears the namespace and the TS truths are order independent. |
||
|
bytecii
|
9c1916878a |
fix(integ): follow the _client rename into integ/ and the examples
The rename sweep in
|
||
|
Zecheng Zhang
|
283da2c8fb |
Merge pull request #820 from strukto-ai/fix/build-resource-sync
fix: make build_resource sync again, hydrate github lazily |
||
|
Zecheng Zhang
|
99b4cb4f00 |
docs(slack): add a Slack watcher example
Hosts the endpoint Slack posts to, unwraps the event_callback envelope, maps it with SlackEventHook and notifies the workspace, with a live watch stream printing what comes out. Verified against a real workspace: a message maps to channels/incident__C0B0DB9K11T/2026-08-13/chat.jsonl, file_shared to that day's files directory, and the mount serves both. |
||
|
Zecheng Zhang
|
8575b9b2f4 |
fix(github): hydrate the tree lazily, make build_resource sync again
0.0.5 made build_resource a coroutine so GitHubResource could fetch the repo tree before returning. That fixed a real defect (the two fetches ran in __init__ over a blocking urlopen and froze the daemon's loop) but paid for it with the one function every caller who describes a mount as data comes through: the YAML loader, the daemon's create/load routes, clone, and every embedder reaching it through mirage.sdk. The haystack integration, our first outside consumer, broke on it. Hydrating lazily removes a round trip instead of adding one. Nothing seeded the index at build time, so the first readdir ran ensure_live_index and refetched the whole tree, discarding the one the constructor had just paid for; measured two git/trees calls where one does. GitHubResource now names the repository and contacts nothing, and the tree and default branch arrive through ensure_tree and ensure_default_branch on first use, each behind a lock so concurrent first reads cost one request. readdir and read already hydrated through ensure_live_index; only find, du and grep's narrow read accessor.tree directly and needed wiring. normalize_resources now refuses a non-resource and names the mount. The old failure was 'coroutine' object has no attribute 'set_index', raised two frames away in install_mounts, naming a method the caller never called and no mount. TypeScript stays async: its factory type is uniformly (config) => Promise<Resource> and two of its backends need it. Recorded as a deliberate divergence rather than mirrored. |
||
|
Zecheng Zhang
|
bcf170afd9 |
Merge pull request #811 from strukto-ai/feat/watch-delta-hooks
feat(watch): ship delta_hook for ten more backends |
||
|
Zecheng Zhang
|
3bbb791217 |
chore(agents): upgrade the agent SDKs, and follow deepagents' new grep cap
Every agent SDK was resolving to the floor of its own range, so the adapters were being tested against builds a month older than the pins implied. Two were outside their caret entirely (@openai/agents 0.13.5 with 0.16.0 published, pi-coding-agent 0.80.10 with 0.84.2). The examples workspace carries the same pin set and was bumped with it. deepagents 0.7.6 added max_count to BackendProtocol.grep/agrep. mypy caught the incompatible override; TypeScript accepted the shorter signature structurally and typechecked clean, so both sides are fixed. max_count is a total cap across files, which is not what grep -m means, so it is applied to the collected matches rather than pushed onto the command line, and truncated is only set when matches were actually dropped. dsh-shell moves to the 0.1.0 line to match dsh-fs. DeepSeek relicensed between the two (BSD-3-Clause and restricted, to MIT and public), and the published types are identical. That family tags its 0.1.0 line as next while latest still points at 0.0.x, so the siblings are pinned in pnpm.overrides. Clears a low advisory on @ai-sdk/provider-utils that reached examples through mastra 1.51's @ai-sdk/ui-utils, which 1.59 drops. Both audits are clean. Note: @opencode-ai/plugin 1.18.18 pulls ini@7, which needs node >= 24.15.0. CI already resolves above that floor. |
||
|
Zecheng Zhang
|
88698fa08b |
feat(ops): optional byte-range reads across backends, and drop github_ci
Adds an optional read_range slot to the op table. Backends that can fetch a window do; the generic read op falls back to read-and-slice for the rest, so no backend has to implement one. Wired in both languages for box, databricks, dify, discord, disk, dropbox, gdrive, gridfs, hf, nextcloud, onedrive, opfs, ram, redis, s3, sharepoint, slack and ssh. Shared helpers live in utils/ranges (range_header for a raw push-down, slice_window for rendered content, is_unsatisfiable_range to normalise a 416). Backends that render their bytes take the window right after building them, which is why dify and ram are on the native list too: a windowed read is answered the same way everywhere, whatever is behind the mount. Fixes a crash on hf and nextcloud: OpenDAL's reader seeks rather than sending a header, so a window past EOF raised from the seek instead of surfacing a 416. Both now read as empty. Caught by the integ battery, not by unit tests. Removes the github_ci resource, its commands, ops, docs and examples. Integ: a shared ranges/read.json across 21 targets, plus per-backend cases for slack, discord and dify, reached through ws.dispatch since no shell command asks for a window. |
||
|
Zecheng Zhang
|
d312ce978f | Merge branch 'main' into feat/watch-delta-hooks | ||
|
Zecheng Zhang
|
1146305c24 |
chore(examples): typecheck examples/typescript in CI
Add a typecheck script so `pnpm -r typecheck` covers the examples workspace, and fix the 32 errors it reports. - 21 FUSE examples used MountBackend without importing it, so they failed at runtime with ReferenceError - hf_datasets/hf_spaces imported config types that do not exist; both resources take HfRepoConfig - redis provision helpers take (accessor, paths, texts, opts), matching the Python example - postgres_fuse/postgres_vfs read a module-level env const inside main, where the undefined guard does not narrow - s3_browser/presigner passed Record<string, unknown> as ListObjectsV2 input - lancedb: registry.get may return undefined, and TextEmbeddingFunction .sourceField takes no argument |
||
|
Zecheng Zhang
|
4eef9413f9 |
feat(watch): ship delta_hook for ten more backends
Adds pull change detection to s3, github, dropbox, disk, gridfs, the hf family, ssh, gdrive, graph and box, so eleven resource families now answer "what changed under this root since my checkpoint". Two shared helpers keep a new backend from writing its own loop: synth_dirs builds the directory rows a prefix store implies, and ReaddirWalk descends through a backend's own readdir with a fresh private index per pull. Adding github to the integ battery surfaced two real bugs. Its index was keyed repo-relative while the other eleven backends and CacheManager key mount-absolute, so cache invalidation never reached a github mount at all, silently, since evicting an absent key succeeds. And invalidate_dir drops a row rather than expiring it, so github's readers took a dropped index for real absence. Github now keys mount-absolute like everyone else, its repo-relative path logic reads the git tree on the accessor the way TypeScript's always has, and ensure_live_index refetches when the index holds no listing. Box fingerprinted on modified_at while the same listing already carried sha1, so two writes in one second were indistinguishable. It now prefers sha1. The integ watch battery covers eleven targets at 238 cases, and the disk example is gated in both languages against one shared truth file. |
||
|
Zecheng Zhang
|
a1a769848e |
docs: fix the README hero snippet, resync the mirrors, and give each CLI page its own icon (#801)
* docs(readme): fix the hero snippet and resync every mirror The hero snippet called ws.command(...), which exists in neither language: registration is the standalone command() plus mount.register. It is replaced with a Python example that mounts ram, redis and slack side by side, captures python with monty, and installs a CLI, all of it run against the published 0.0.5 packages first. Two more corrections. The filetype sentence promised parsed PDF pages, which the filetype removal took away, so it now says a format renders however you register it. DeepSeek Harness joins the coding agents row. The eleven mirrors are regenerated from the root rather than patched, which also closes drift they had accumulated: a stale backend list, the old CLI + daemon integrations line, a missing Grok Build entry and a Codex link pointing at the wrong docs path. * docs(cli): give each CLI page its own icon Every CLI page shared icon: terminal, so the sidebar was nine identical rows. Each now takes the icon its service already uses elsewhere in the docs: slack, discord, github for gh, google for gws, envelope for himalaya, book for ntn and chart-gantt for linear (matching the notion and linear setup pages, since Font Awesome carries no brand mark for either), and git-alt for git. gws and himalaya also get their names spelled GWS and Himalaya; the rest stay lowercase because that is the head word you type. * examples(filetype): register through the public mount accessor The example reached into ws._registry.mount_for, but ws.mount is public and returns the same MountEntry. Output is unchanged, so the CI truth file still matches. |
||
|
Zecheng Zhang
|
f44c6e5c25 | docs(dsh): drop the redundant workspace mode option (#777) | ||
|
Zecheng Zhang
|
829a7e5c29 |
feat(dsh): DeepSeek Harness fs and shell providers over a mirage workspace (#774)
* feat(dsh): DeepSeek Harness fs and shell providers over a mirage workspace * docs(dsh): monty python run and multi-source mounts in the dsh example and page * docs(dsh): tidy the example into sectioned helpers * docs(dsh): trim the page to install, compose, python, one world * docs(dsh): use mirage as the example report owner * docs(dsh): slack-hosted python script example * docs(dsh): rename the slack-hosted script to example.py * fix(dsh): match Slack upload names and newest-first date dirs in the example * fix(dsh): address codex review on lock retention, pre-aborted signals, cyclic links, and byte limits * docs(dsh): generalize the page, compose sample on redis + ram * docs(dsh): keep slack in the compose sample * docs(dsh): state the python3 capture forms * docs(dsh): show the runtimes capture spec with an inline python sample * docs(dsh): monty is the workspace python, capturing python and python3 * docs(dsh): one continuous example, no second workspace * docs(dsh): show monty explicitly capturing python and python3 * docs(dsh): one example end to end — ram + redis + slack, monty capturing python * docs(dsh): drop the one execution world section * docs(dsh): state the in-memory property with measured capacity * docs(dsh): drop the capacity numbers * docs(dsh): drop the in-memory paragraph * docs(dsh): add a harness comparison table * docs(dsh): add concurrency column on a 2 vcpu / 8 gb server * docs(dsh): correct the live python engine footprint to 27 MB * docs(dsh): no em dashes, dsh row name, serving scenario before the table * docs(dsh): tmp mount, redis-bound report, numeric concurrency table with startup * docs(dsh): note the sandbox provisioning cost the cli sessions would add * docs(dsh): sandbox startup cost in the cli startup cells |
||
|
bytecii
|
ef99849b94 |
refactor(seam): dispatcher-built CommandOpts, typed adapter ops, named ParsedCommand, one DispatchFn home (#609 items 26+23) (#772)
* wip(seam): py dispatcher builds CommandOpts; handlers take (accessor, paths, texts, opts); adapter op protocols; DispatchFn typing Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(seam): dispatcher-built CommandOpts, typed adapter ops, named ParsedCommand, one DispatchFn home (#609 items 26+23) Item 26 (T2-8 seam typing), all four pieces, plus task 23's static FlagView query-name gate: - The python dispatcher (Mount.execute_cmd) now constructs one CommandOpts per invocation and calls every handler as fn(accessor, paths, texts, opts) — the TS convention. Flags stop sharing a bag with injected context, accepts_kwarg opt-in dies, and all 72 builders, 75 bespoke wrappers and ~30 provision functions become pure wiring; the provision path builds the same bag with command/spec set. Generics that took trailing fact params (ls, find, tree, zip, tar, du, file, stat) now read opts.links/opts.mounts/... like their TS twins. - adapter.py gains per-slot op protocols (ReaddirOp/StatOp/WriteOp/...) mirroring adapter.ts, so wiring readdir where stat belongs no longer type-checks; Builder.fn/provision and CommandIO fields are typed. Surfaced and fixed real drift: hf/databricks mkdir had index before parents (a positional parents call would land in index), wget -O dropped its PathSpec value under as_str. - TS parseFlags returns a named ParsedCommand (twin of the py NamedTuple) instead of a 15-slot positional tuple; optionError takes (cmdName, parsed) like python's option_error. - DispatchFn moves to runtime/types.ts (python's home for the same protocol); the crossmount re-export inversion and the CommandDispatch duplicate are gone; 38 python files stop spelling it Callable[..., Any]. The dead CommandOpts.resource field (zero readers) is deleted along with the CrossResourceStub that existed to fill it. - Task 23: tests/commands/test_flag_query_names.py and commands/flag_query_names.test.ts fail on a FlagView query naming a dest no spec bound in the module declares, without needing the code path to run (both verified to fire on a planted typo). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(examples): redis example calls provisions with the 4-positional shape file_read_provision/head_tail_provision/metadata_provision are (accessor, paths, texts, opts) now; the direct demo calls still passed the old command= kwarg, crashing the example before the persistence section (CI examples gate). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(integ): pin the #772 fixes where the harness can reach them - find -empty on chroma + github (the wrappers used to drop the flag, which would flood every path; -empty now rides find_generic), plus a -not -empty composition case on each so the pin has positive output - mkdir -p gains databricks/databricks-prefix (zero prior mkdir coverage on that backend; the param-order bug itself is only reachable positionally, so MkdirOp+mypy is its real gate) - wget -O / curl -o were already pinned by integ/resources/http Both hosts green: py chroma+github 83 ok, ts all five targets 4070 ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: bytecii <bytecii@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
90e564273d |
fix(github): build resources asynchronously, delete the sync HTTP path (#609 T1-J)
`GitHubResource.__init__` fetched the repo's default branch and its
recursive git tree before returning, over `urllib.request.urlopen` — the
only blocking HTTP call in the package. A constructor cannot await, which
is the entire reason that sync client existed.
It is reachable on a live event loop: `server/routers/workspaces.py`'s
`async def load_workspace` calls `build_resource` synchronously, so
mounting a github repo froze the daemon's loop for two GitHub round trips
— every other mount's in-flight I/O and the FUSE queue stalled with it.
Measured against a deliberately slow local GitHub, a 1.24s build let a
50ms heartbeat coroutine tick once; it now ticks 24 times over the same
1.24s.
TypeScript already had the answer, but not the one the audit recorded:
`GitHubResource.open()` there is a no-op, and the work lives in
`static async create` behind a private constructor, with the whole
`ResourceFactory` type declared `(config) => Promise<Resource>`. Mirror
that rather than the `open()` hook — a lazy hook would move the failure
from build time to first path resolution, which is a new divergence, not
a fix.
- `BaseResource.build`: async classmethod factory, default just calls the
constructor. Named `build`, not TypeScript's `create`, because `create`
is already an op name (make an empty file, what `touch` calls) and ops
are served by `__getattr__` — a real `create` on the class shadows every
backend's create op. Caught by the databricks_volume suite.
- `build_resource` is now async and awaits that factory through
`_instantiate`, which falls back to the plain constructor: registered
and entry-point resources need not subclass `BaseResource`.
- `WorkspaceConfig.to_workspace_kwargs` follows, matching TypeScript's
already-async `configToWorkspaceArgs`. `Workspace(**kwargs)` stays sync.
- `GitHubResource.__init__` takes the fetched tree and touches no network;
`GitHubResource.build` does the two fetches over the existing async
`github_get`. `github_get_sync`, `fetch_tree_sync` and
`fetch_default_branch_sync` are deleted.
- New `tests/resource/test_no_blocking_http.py` fails on any `urlopen`,
`urlretrieve` or `requests` import under `mirage/`, including
function-local ones.
- Docs updated on both sides; the TypeScript github pages showed
`new GitHubResource({...})`, which has never compiled against its
private constructor.
Snapshot load is untouched: github's state redacts its token, so
`requires_resource_override` always sends it down the live-override path
and never reaches `_construct_resource`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Zecheng Zhang
|
7fbe3aedb8 |
fix(notion): delete verb, one trash bit, row cells, and stale docs (#747)
DELETE /v1/blocks/{id} is the only delete verb the public API has and the
only one the MCP tool surface exposes, but the fake had no route for it
and `ntn api` had no DELETE method, so nothing could remove anything.
`archived` is upstream's deprecated alias for `in_trash` and always
returns the same value; the fake stored two columns, so a PATCH of
`archived` left the row in database queries while `ntn pages trash`
(in_trash) worked. Now one stored bit, both spellings on the wire.
A database row's cells now ride in its page.json under `properties`.
Also: trashing a child page moved only the page row, leaving the
child_page block in the parent listing; `datasources query` took its
TSV columns from the schema where upstream takes them from the returned
rows; and two MCP-parity cases pointed at pre-data-source paths, so they
compared two identical errors and asserted nothing.
Docs: both ntn.mdx files documented the pre-#740 grammar (--page flags,
ntn blocks/comments/search). notion.mdx, the TS setup doc and both
examples predated the data-source split; the examples were broken.
|
||
|
Zecheng Zhang
|
619be25d32 |
feat(himalaya): align the CLI with the upstream pimalaya grammar (#709)
* feat(himalaya): align the CLI with the upstream pimalaya grammar
- mailbox is -m/--mailbox, message id is a positional operand
- envelope search with upstream's query DSL, replacing list filter flags
- message compose writes RFC 5322 to stdout unless --send; message send
takes raw MIME, so the two chain through a pipe
- upstream aliases: ls, sr, write, new, fwd
core/email/send.* is deleted: the email mount is read-only, so the
composer and the SMTP transport moved into the himalaya package.
Fixes three node-side bugs found while testing:
- sendMail({raw}) needs an explicit envelope; Bcc is stripped by hand
- parseSearchCriteria silently dropped unknown keys, so NOT SEEN matched
every message
- parseImapDate rolled a bad month or day into a wrong date
* fix(himalaya): satisfy the CI eslint rules and stop the yapf/isort import flip
* fix(himalaya): search the Date header and bound envelope pages
Date conditions emit SENTON/SENTBEFORE/SENTSINCE rather than
ON/BEFORE/SINCE, which match the mailbox internal date, and the missing
upstream `before` condition is now parsed.
Envelope listing asked IMAP for every matching uid and fetched headers
for all of them before slicing one page. It now fetches the newest
page * page_size, capped by the account's max_messages.
* test(himalaya): cover every flag in integ, and fix what it found
Grows the himalaya integ suite from 38 to 64 cases: every search
condition, sorter, flag and alias, the composer flags, both quoting
options, stdin bodies, the send operand form, unknown flags and a
missing mailbox.
Three bugs it turned up, all in the email client:
- Five IMAP selects ignored their response, so a missing mailbox left
the session in AUTH and the next command complained about that
instead ('command SEARCH illegal in state AUTH' in python, imapflow's
bare 'Command failed' in typescript). Both now say no such mailbox.
- A refused SEARCH returned an empty uid list, so criteria the server
cannot answer read as 'matched nothing'. It now raises.
- Found because a mutation emitting a malformed date stayed green.
|
||
|
Zecheng Zhang
|
fc524e1d90 |
feat(cli): slack, discord, ntn and linear as builtin CLI packages (#703)
* feat(cli): slack, discord, ntn and linear as builtin CLI packages Migrate the remaining service bundles to installed CLIs with the settled vocabularies (OpenClaw slack/discord actions, the official Notion CLI grammar as ntn, linear's noun/verb tree), including the new API surface they need (slack pins/emoji/reactions/history, discord edit/delete/ threads/polls, notion page updates). Configs move resource -> core in both languages and the mounts go filesystem-only. chroma-query stays a mount command on purpose: it is path-scoped. Fixes two CLI dispatcher bugs found by the new integ cases in both languages: leaf exceptions now become the command's IOResult (so redirects and ; sequencing work, with the GNU prog: prefix), and argv words re-enter the walk as typed so quoted glob-looking flag values survive. Adds cli-facet integ targets and cases for all four CLIs (108 cases both runners), extends the fake slack/discord/notion servers, and moves CLI usage out of the resource docs into new CLI sections on both tabs. * integ: cover the remaining linear write verbs in the cli facet * feat(cli): linear add-label and set-project resolve names via the issue's team * fix(cli): codex review, standalone thread type and bounded notion search pagination |
||
|
Zecheng Zhang
|
a39755b3b1 |
feat(policy): absorb output safeguards into the policy layer as Limit (#694)
* feat(policy): absorb output safeguards into the policy layer as Limit * fix(policy): loop-based trailing-slash strip in limitOverride (CodeQL polynomial-redos) * fix(integ): surface policy EACCES through the TS fuse read/write callbacks, skip code-policy cases in the CLI harness * chore(deps): bump cryptography 50.0.0 and aiohttp 3.14.3 for audit advisories * fix(policy): stamp builtin producers at the dispatch chokepoint, purge safeguard vocabulary from docs |
||
|
Zecheng Zhang
|
d13ec69ba4 |
feat(cli): himalaya and gws as builtin CLI packages plus integ cli facet (#699)
* feat(cli): himalaya and gws as builtin CLI packages plus integ cli facet * fix(cli): flake8 re-exports, gws refreshFn config, docs register_cli, EmailConfig to core |
||
|
Zecheng Zhang
|
3d1dc0dfe9 |
Runtime and policy sweep: parity fixes, quickjs evaluator, node local runtime (#663)
* fix(policy): fail loud on entry-script verdict shapes, stop fanout swallowing * fix(monty): raise typed fs exceptions in the ts guest * feat(quickjs): evaluator capability in both languages, js policy scripts * feat(local): host python runtime for the node package * test: align the script loader wording * style: satisfy the ci linters * fix(runtime): reclaim the local subprocess on safeguard timeout, tolerate stdin EPIPE * fix(quickjs): bridge the workspace fs into eval, match the real engine's output surface |
||
|
Zecheng Zhang
|
45dd79d292 |
refactor(runtime): evaluator capability, route engine on runtimes, drop PythonRuntime (#657)
* refactor(runtime): evaluator capability, route engine on runtimes, drop PythonRuntime * refactor(runtime): split base into types/errors/mixin, eval transport belongs to the runtime * docs: module layout convention (types/errors/config/mixin/base) * docs: trim module layout section * review: brand the evaluator, bind session inputs and bytes on pyodide * fix(runtime): refuse unbound interpreters with GNU command-not-found wording * fix(integ): survive connection resets while jaeger boots * docs: evaluator page under learn, language icons for the runtime pages * docs: reframe the evaluator page as the policy engine * docs: both entries capture python3 in the policy engine example * docs: real routing case with the ctx payload and verdict semantics * docs: drop the engine-only yaml block, one parenthetical instead * docs: live-captured ctx payload, verified config, function route signature * docs: trim the policy engine section |
||
|
Zecheng Zhang
|
3a5eb5c260 |
fskit size guard warns, Linear gets size push-down (#656)
* fix(fskit): downgrade the size guard from a refusal to a warning * feat(linear): push file sizes down to readdir so stat matches read Every linear file is now sized at its parent directory's readdir from the payload the listing already fetched: team.json from the teams listing, issue.json from the issues listing, member/project/cycle/document JSON from their own listings. comments.jsonl costs one bounded comments call, paid only when the issue directory is entered. Listing a tree never fetches file content, and stat reports the exact rendered byte length, which is what fskit needs at lookup time to serve reads. SIZES_ALWAYS_KNOWN flips on. Invariant pinned in both languages (stat size equals read length for every file in the tree) and in integ with paired stat/wc cases against the fake Linear server. * fix(fuse): drain buffered writes on release, warn on writable fskit mounts The macFUSE FSKit shim issues WRITE then RELEASE with no FLUSH in between (the kext always flushes on close), so MountCore dropped every buffered kernel write at release. Release now drains the handle's write buffer in both languages. Testing the CLI over a live fskit mount also measured a shim bug mirage cannot fix: pages for regions a file did not already have (a new file, an empty file, a truncate-then-write) flush as NUL bytes of the right length, and appended regions arrive intact or zeroed depending on cache state, with no error surfaced to the writer. A new check_writes guard warns at mount time for writable fskit mounts (metadata ops are reliable; /dev is excluded), and the behavior is pinned in integ/truth_fskit.json plus a new CLI battery, integ/cli_fskit.sh, which drives both daemon-backed CLIs through a real fskit mount and skips cleanly on hosts that cannot engage the FSKit module. cli_fuse.sh now also asserts that a backend fskit workspace is rejected cleanly off macOS. * refactor(integ): move the fuse truth from txt lines to JSON * refactor(integ): gather the fuse and fskit harness under integ/fuse/ * docs(fuse): document the fskit write-path limits and the size warning * docs(fuse): tighten the fskit warnings |
||
|
Zecheng Zhang
|
0c3c827ac9 |
feat(fuse): fskit mount backend and a MountCore/adapter split (#648)
* refactor(fuse): split the mount layer into MountCore plus a libfuse adapter MirageFS was one class doing two jobs: filesystem semantics and talking to libfuse. The FUSE-specific part was smeared across every method rather than sitting at a boundary, which is why it could not be reused by another kernel interface. MountCore (fuse/core.py, fuse/core.ts) now owns all semantics and imports nothing from mfusepy or fuse-native. MirageFS keeps only the callback signatures and error translation. The attr dicts stay as they were: st_mode and friends are POSIX stat field names, not FUSE ones. The load-bearing change is error handling. Python raised FuseOSError inline in ten methods with hand-rolled except chains while TypeScript already had a single classifyError. Both languages now share one classification table (fuse/errors.py, fuse/errors.ts), so the same backend failure reports the same errno on both sides. That fixes a real bug. Python's rmdir caught OSError before FileNotFoundError, and FileNotFoundError is an OSError subclass, so rmdir on a missing path returned ENOTEMPTY instead of ENOENT. No test covered it; test_errors.py now pins it. Behavior is otherwise unchanged: the pre-existing suites (test_fs.py 689 lines, fs.test.ts 43 tests) pass untouched apart from renaming private attribute reads onto .core. * feat(fuse): fskit mount backend, kext-free mounts on macOS Adds backend selection to a FUSE mount: Mount(..., fuse_backend="fskit") routes through macFUSE 5.x's FSKit shim, so the mount runs with no kernel extension loaded. Closes #82. Three rules are enforced at mount time rather than discovered at run time: - macOS only, raising elsewhere instead of silently dropping the option. - The mountpoint must live under /Volumes. FSKit refuses anything else, which is what made the reporter's /tmp attempt in #82 fail. The rule is owned by FuseManager, so no call site has to know it. - Every mounted resource must be able to size its files. FSKit has no direct_io, so reads are driven entirely by the reported size: a resource that stats as 0 before open would serve an empty file with exit code 0. Mirage refuses such a mount and names the offending prefixes. That last rule needs a new capability, SIZES_ALWAYS_KNOWN / sizesAlwaysKnown, default false. Opted in on the byte stores (ram, disk, redis, s3, gridfs) plus dev and history. History matters more than it looks: it is mounted into every workspace at /.bash_history, so leaving it false would block every root-scoped fskit mount. There is deliberately no "auto" value. Auto-selecting fskit would silently break every API-backed mount, and an option whose safe value is always the default is a trap. TypeScript cannot reach FSKit at all: @zkochan/fuse-native bundles a pre-macFUSE-5 dylib, so the option never arrives at a driver that understands it. checkPlatform throws rather than quietly mounting through the kext. The capability and guards are still mirrored so the semantics match. Documented as a known gap. Examples cover both sides of the limit: ram, redis and s3 mount and read under fskit; slack and gmail show the guard refusing and the supported workaround of scoping the mount to a byte-store subtree. * refactor(mount): one backend field, vfs by default Replaces the fuse boolean and the fuse_backend string with a single MountBackend StrEnum: vfs, fuse, fskit. VFS is the default and finally has a name, instead of being spelled fuse=False. Mount(resource, backend=MountBackend.FUSE) Mount(resource, backend=MountBackend.FSKIT, mountpoint="/Volumes/x") Three things this fixes. The default was previously defined by negation. Most mounts are not kernel-mounted at all, and that case is what mirage is mostly used for, so it deserves a name rather than a falsy value. The overloaded bool | str union is gone. fuse=True and fuse="/path" packed two decisions into one field; backend and mountpoint separate them. The enum no longer claims a name it does not own. fuse_backend="fskit" read as "use FSKit", but which delivery mechanism reaches FSKit (macFUSE 5's shim today, a native Swift module later) is mirage's business, not the caller's. One value covers both. MountBackend lives in mirage.types next to MountMode rather than in mirage.fuse, so the mount spec does not have to depend on the fuse package for a value that also covers the non-fuse case. resolve_backend rejects vfs: reaching it means a kernel mount was requested, and vfs registers nothing. Config, server routers, and the TypeScript mirror follow, including the YAML schema (backend: fuse plus mountpoint: <path>) and fuseMounts becoming kernelMounts. * fix(mount): migrate the CLI config path and integ fixtures to backend The backend rename missed three call-site shapes that a literal search for fuse=True / fuse: true does not catch: - YAML mountpoints written from shell variables (integ/cli_fuse.sh wrote `fuse: $dmnt`, five of them), which is the CLI's own end-to-end path. - Mount options passed a variable rather than a literal (integ/fuse.py and integ/fuse.ts both used `fuse=pinned`). - The TypeScript server config schema, which still declared `fuse?: boolean | string` and exported fuseMounts. It now mirrors Python: backend plus mountpoint in, kernelMounts out. The CLI itself has no fuse flag, so its whole surface is the workspace YAML. Both config loaders were checked against the exact document cli_fuse.sh writes and agree: /data and /logs resolve to (fuse, <path>), a backend with no mountpoint resolves to (fuse, None), and a mount with no backend key stays on vfs and is absent from the kernel mounts. CLAUDE.md documents the single backend field and the YAML keys. * refactor(mount): missing is vfs everywhere, and one guarded entry point Two related cleanups. Missing meant two different things. Mount.backend defaulted to VFS and an absent YAML key resolved to VFS, but resolve_backend(None) returned FUSE, reinterpreting an absent value as a request for a kernel mount. Now every absent value lands on VFS, and the mount entry points spell their own default: backend: str | MountBackend = MountBackend.FUSE. The intent is in the signature instead of hidden in the resolver. resolve_backend is pure coercion again; require_kernel_backend is the separate step that rejects VFS. The unknown-name error lists all three values rather than just the mountable two. The fskit guards were three separate calls at each mount path, so a new path could pick up fskit support and silently skip the macOS assert, the /Volumes rule, or the size check. prepare_backend now runs all of them, and every mount path routes through it: mount_background, mount, and FuseManager.setup. Tests pin that a linux platform raises through prepare_backend, not just through check_platform directly. Same shape in TypeScript: resolveBackend, requireKernelBackend, prepareBackend. * fix(fuse): drop the polynomial regex from unsizedMounts CodeQL js/polynomial-redos, two high-severity alerts on the same helper. `prefix.replace(/\/+$/, '')` backtracks on a string of many repeated slashes, and the input is a mount prefix, so it is library input. Uses core's loop-based rstripSlash instead, which is the existing sanctioned trim for exactly this. Python was never affected: str.rstrip is a C loop, not a regex. * fix(fskit): match the mount recipe and volume ownership verified in #82 Three corrections to the fskit path, all found by re-reading issue #82 against what we actually shipped. Mount options. #82's only reported working mount passes backend=fskit AND volname, and omits direct_io. We passed direct_io unconditionally and no volname, on my assumption that direct_io is simply inert on this path. That assumption contradicted the one person who has run it, and direct_io is a libfuse concept, so the shim needing it removed is not surprising. Now matched exactly, with a comment telling the next person not to restore it. Mountpoint ownership, which would have broken every fskit mount. /Volumes is drwxr-xr-x root:wheel, so tempfile.mkdtemp(dir="/Volumes") raises PermissionError for any non-root user. The reporter mounted at /Volumes/mirage-* as a normal user, so macFUSE creates the entry: an FSKit mount is a volume, not a directory we make. The fskit path now names its mountpoint and never creates it, skips makedirs for a pinned one, and never rmdirs a /Volumes entry on unmount. Readiness. The /Volumes entry does not exist until the volume is live, so bare existence is a real ready signal there, exactly as for WinFsp, and it does not depend on os.path.ismount recognizing the mount. These are the same three deviations WinFsp already needed, for the same underlying reason: the filesystem driver owns the mountpoint. Tests pin the option set and all three ownership rules, because nothing in CI can reach this path (macOS 15.4+, macFUSE 5.x, a GUI-enabled FSKit module). * fix(tests): pin the platform in the fskit manager tests These three passed on macOS and failed on the Linux runner: FuseManager routes through prepare_backend, whose check_platform rejects fskit off darwin. Patch sys.platform explicitly so the assertion under test is the mountpoint behavior, not the host, matching what test_backend.py already does in both directions. * ci: run the fskit backend for real on a macOS runner Until now fskit shipped on unit tests plus one user report, because no job could mount it. integ/fuse.py cannot cover it: its sizeless probe is refused by the fskit size guard by design, and its two-mount scenario needs something macOS forbids. So this adds integ/fskit.py, one RAM mount under /Volumes that reads, writes and stats through the kernel. Advisory like integ-fuse-windows, since whether a hosted runner enables the FSKit module without a GUI toggle is what the job measures. The truth file is JSON checked by value, not a txt file checked by substring: the old harness passes a result of 166 against a truth of 16. check_json.py is written to serve the remaining txt probes too. Only what mirage controls is asserted; the mountpoint, the raw mount row and whether a kext happens to be loaded are reported and left alone. * ci(fskit): report the mount state before reading it The first macOS run failed with a bare FileNotFoundError and told us nothing: every diagnostic in the probe was printed after the read that crashed. The volume entry appeared (readiness passed, the mount thread stayed alive) but served no tree, and there was no evidence of why. Print the mountpoint, its stat flags, the mount row and a listdir before touching a file, and add an always-run step dumping the mount table, /Volumes, the registered FSKit modules and the recent macfuse log. * fix(fskit): a stray /Volumes directory is not a live mount The macOS integ job reported exists=True isdir=True ismount=False with no row in the mount table: macFUSE creates the /Volumes entry while mounting and leaves the empty directory behind when the FSKit handoff fails. Bare existence as the ready signal accepted that, so a mount that never came up was reported live and failed with ENOENT on the first read. Require os.path.ismount for every POSIX backend and keep the existence shortcut for WinFsp only, where _prepare_mountpoint removes the directory first so its reappearance really does mean the filesystem is live. The backend argument is unused now, so it goes. _await_ready had no test at all; it has one now. Also register both macFUSE appexes on the runner: the installer registers only the -local module there, while a developer Mac has both. * ci(fskit): capture why the macFUSE mount never comes up Both modules register and enable now, and the mount still times out with no libfuse error, while the only module the system launches is msdos. So the mount request is not reaching macFUSE at all. Capture the kext device nodes, the system extension list and a macfuse-scoped log to tell an environment limit from a mirage bug. * docs(fskit): record where the /Volumes rule comes from The guard read as an arbitrary house rule. It is measured: issue #82 ran an fskit mount under /tmp, got 'mount_macfuse: the file system is not available (1)', and the same mount worked once it moved to /Volumes. * fskit: pin the real write surface, measured on a Mac The mount works: /Volumes/mirage-*, tagged fskit, tree served, reads exact (stat size == bytes read, so the size guard holds). ismount is true on a live FSKit volume, which is what the readiness fix now depends on. Writes are a different story. In-place writes and unlink work; create, mkdir and rename return ENOSYS. A failed create still applies: the syscall reports ENOSYS and the file exists in the resource and through the mount anyway. Tracing shows mirage's create succeeds and returns a handle, then the shim fails the syscall, so we cannot report this more accurately. Pin the whole matrix in the probe and say plainly in the docs that anything creating files will fail on an FSKit mount. * examples: one fskit example per language Five scattered files (ram, redis, s3, slack, gmail) collapse into examples/python/fuse/fskit.py, which shows the size guard refusing, a live mount with reads matching their stat size, and every write op with the errno it returns. Verified end to end on macFUSE 5.3.3. examples/typescript/fuse/fskit.ts cannot mount, since fuse-native bundles a pre-macFUSE-5 dylib, so it shows the refusal and both alternatives: backend fuse there, or the Python package for a kext-free mount. It typechecks but is not run here, because its fuse leg is a live kext mount. * feat(fskit): TypeScript serves fskit after all The 'fuse-native bundles a pre-macFUSE-5 dylib' claim was wrong: the libosxfuse.2.dylib it ships is a stub whose install name is /usr/local/lib/libfuse.2.dylib, and fuse.node links that absolute path, so Node loads the same macFUSE 5.x libfuse Python does. Verified with a live mount: /Volumes tagged fskit, cat/wc exact, and the same write surface as Python (in-place ok, create/mkdir/rename ENOSYS). Drop the unconditional throw from checkPlatform (macOS-only stays), generalize appendDirectIO into appendMountOptions, and give mount() the fskit branch: /Volumes named-not-created, backend=fskit + volname, no direct_io, ownsMountpoint false so nothing ever rmdirs a /Volumes entry. The example now mounts for real; docs flip from known-gap to supported-with-caveats. The caveat is earned: one of three example runs wedged on an append, and a dead FSKit volume blocks mount-table enumeration system-wide until the macFUSE appex is killed. Documented as read-mostly and experimental. * docs: example READMEs and a concise fskit story Both examples/ READMEs explain how to run, the naming convention, and why fskit exists: Apple has deprecated third-party kexts (reduced-security boot + approval on Apple Silicon already), FSKit is the supported userspace replacement, and macFUSE 5.x serves the same libfuse API through it. One mermaid flow shows the two paths differing only in the kernel-to-userspace hop. The two fuse.mdx warnings shrink to their load-bearing facts and the python page gains the same why + diagram. * ci(fskit): report the known hosted-runner limit as a skip Everything installs and enables headlessly, but the mount request never reaches macFUSE's FSKit module on a hosted runner, so every run ends in the same readiness TimeoutError and a permanently red advisory job is noise. Treat exactly that signature as a skip and drop continue-on-error: green now means environment-limited or verified, red means something new broke (a guard regression, a crash, or the runner starts mounting and diverges from integ/truth_fskit.json). * feat(fskit): full write surface via macFUSE's Darwin-only callbacks The read-mostly limitation was never FSKit's: the shim finalizes every created item through setattr_x and routes rename through renamex, both Darwin-only fuse_operations fields that mfusepy leaves as reserved NULL slots. libfuse answered those requests itself with ENOSYS, after our CREATE/MKDIR had already applied (wire trace: CREATE success, then SETATTR -78), which also explains why the failures were dirty and why rename never reached userspace. fuse/darwin.py replaces the 13 reserved slots with macFUSE's real Apple tail (same size, asserted) and marshals setattr_x, fsetattr_x and renamex; MirageFS decomposes setattr_x (size routes to truncate, other attributes follow the chmod/chown accept-if-exists semantics) and maps renamex flags (EXCL honored, SWAP refused as ENOTSUP). Installed once per process from _run_fuse, no-op off macOS, layout-guarded so an mfusepy upgrade degrades to the old behavior instead of corrupting the struct. touch, mkdir, mv, rm and a new-file write roundtrip now all pass on a real fskit mount; integ/truth_fskit.json pins the full matrix. Docs flip Python fskit from read-mostly to full-write; TS keeps the old surface because fuse-native's compiled op table cannot gain new C callbacks from JS. Remaining upstream caveats stay referenced in code comments: macfuse#1181 (exec until first read) and macfuse#1165 (root readdir cache invalidation). * test(fskit): skip the struct-extension test off macOS, settle yapf's import wrap mfusepy builds fuse_operations per platform, so the Darwin reserved tail the Apple fields replace does not exist in the Linux layout and install_macfuse_extensions correctly bails there; monkeypatching sys.platform cannot conjure the struct. yapf also disagreed with the committed import wrapping; both yapf and isort accept the new form. |
||
|
Zecheng Zhang
|
d5d08eb541 |
feat(runtime): sandbox runtimes with FUSE-mounted workspaces (#590)
* feat(resource): generic remote_mount_spec so cloud backends are sandbox-mountable
Lift the S3-only remote_mount_spec to the base class: any backend that
opts in with remotely_mountable and holds a pydantic config serializes
{resource, config} (credentials unwrapped) so a remote mirage, e.g.
inside a sandbox, can reconstruct and mount it. RAM and disk keep the
None default. Enables s3, gdrive and slack; adds base coverage.
* feat(runtime): sandbox runtimes with FUSE-mounted workspaces
Add the RemoteSandbox base and the Daytona, e2b and Docker runtimes:
whole-line execution against a remote or local container, lazy
provisioning on the first line, reattach by id, and sandbox ownership so
teardown only touches what we created.
The workspace becomes visible by running mirage inside the sandbox and
FUSE-mounting each remotable mount live: the host serializes the mounts
into the public workspace config, writes .mirage-workspace.json, and runs
one command, `mirage workspace create`. The CLI auto-spawns the in-sandbox
daemon and mounts synchronously, so the exit code is the ready signal and
stderr carries the error. Reads and writes flow both ways with no sync;
writes reach the backend on file close. Needs an image with mirage baked
in (e.g. mirage-python-fuse).
* feat(sandbox): translate virtual mount paths onto provider mountpoints
The agent speaks virtual paths (/data/a.py); mirage is the control
plane that rewrites them onto each provider's physical mountpoint
(/home/daytona/workspace/data/a.py on Daytona, /workspace/data/a.py on
Docker). Before, only cwd was rebased, so absolute virtual paths broke.
Rewrite is longest-prefix-first and only touches tokens that start
exactly at a mount (/s3 or /s3/...); siblings (/s3.txt), system paths
(/usr/bin), and relative paths are left alone. Each mount is also
exported as a MIRAGE_<PREFIX> env var for paths built at runtime. A
bare / world mount is skipped (it would capture the sandbox's own /usr).
Verified end to end on a real Daytona sandbox with an absolute
/data/in.txt round trip.
* docs(examples): README for the Daytona FUSE runtime example
Documents the working flow: bake the mirage-fuse snapshot once, then
drive an S3-backed workspace whose python3 lines run in a Daytona
sandbox that FUSE-mounts the bucket live. Explains the control-plane
path translation (virtual /data/x rewritten to the sandbox mountpoint).
Fixes the workspace yaml: drops the removed mount: fuse/copy option
(fuse is now the only sandbox mode) and points the active runtime entry
at the mirage-fuse snapshot so the example actually mounts.
* fix(sandbox): typecheck fallback for env tuple; remove RAM runtime example
The env-var test's ?? fallback widened env to {}, failing tsc
(TS2339 on MIRAGE_DATA). Type the fallback like the sibling cwd test.
Remove sandbox_runtime.py: it mounted RAM, which is not
remotely_mountable, so under the fuse-only sandbox path its first
python3 line is rejected. The Daytona example (S3-backed) is the
supported demo.
* feat(docker): bake all backends into the sandbox image, not just s3
The sandbox image install was pinned to mirage-ai[s3,fuse], so a
workspace mounting any other backend failed in-sandbox for lack of its
deps. Install mirage-ai[all,fuse] via a MIRAGE_EXTRAS build arg
(default all) so one image mounts any backend; narrow the arg or extend
FROM the image for a lean build. Verified: [all,fuse] resolves under
pip and postgres/mongodb/gcs import in the built image (1.74 GB).
* feat(docker): sandbox image installs mountable backends only, no agent deps
Add a curated 'sandbox' extra (every mountable backend + fuse) and
default the Dockerfile to it, instead of 'all'. A sandbox is a
filesystem host: it never builds agents or launches other sandboxes,
so the agent frameworks (anthropic/openai/deepagents/openhands/agno/
claude-agent-sdk/pydantic-ai) and provider SDKs (daytona/e2b) that
'all' pulls do not belong. mem0 is excluded too: mem0ai is the lone
backend that hard-requires the openai client.
Verified in the built image: no openai/anthropic/daytona/e2b/mem0,
backends (s3/postgres/mongodb/chroma) and fuse still import. Image
1.74 GB (all) -> 947 MB. Narrow further with --build-arg MIRAGE_EXTRAS
or extend FROM the image.
* docs: document sandbox runtimes (docker/daytona/e2b)
Add a Sandbox page to the Runtimes section for both the Python and
TypeScript docs, wired into docs.json nav. Covers capture-based
routing, the live FUSE-mounted workspace, control-plane virtual-path
translation, the mirage-python-fuse image + MIRAGE_EXTRAS, per-provider
setup, reattach/lifecycle, and resource limits. The existing runtime
pages document the in-process interpreters (monty/wasi/pyodide/local);
this covers the remote whole-line runtimes.
* refactor(resource): unify remote flag, rename remotely_mountable -> remote
One boolean now answers 'does this backend live remotely', replacing
both the verbose remotely_mountable and qdrant's dead is_remote/isRemote
one-off. remote=True unlocks reconstructing the resource elsewhere (FUSE
mount inside a sandbox) via the generic remote_mount_spec. Set on s3,
gdrive, slack, qdrant.
* docs: add cache-invalidation example to invalidate_all_after_remote
Concrete cat -> sandbox-write -> cat example showing why a sandbox line
forces a full local cache reset.
* feat(sandbox): reconcile mounts imperatively, drop the workspace config file
mirage is the control plane: the host workspace is the desired state,
the sandbox is the actual state, and every captured line reconciles the
two through the provider's own exec API. A new or changed mount runs
'mirage mount add <prefix> --fuse <path>' inside the sandbox with the
spec in the exec environment (never a file, never argv), a dropped
mount runs 'mirage mount remove <prefix>', unchanged mounts cost
nothing. Mounts added or removed after the sandbox booted converge on
the next line.
New in-sandbox CLI 'mirage mount add/remove/list': each prefix becomes
its own single-mount daemon workspace (deterministic id), so mounts
attach and detach independently through the existing create/delete
endpoints. The uploaded .mirage-workspace.json and one-shot
mount_workspace are gone; TS mirrors the reconciler (syncMounts,
serialized per line).
Real docker e2e green: reconciled mount add, absolute-path read, FUSE
write-through to S3, host readback.
* test(qdrant): follow the is_remote -> remote rename
* ci(integ): survive chocolatey outages in the WinFsp install
Retry choco three times, fall back to the official WinFsp GitHub
release MSI, and verify winfsp-x64.dll landed so a bad install fails
at the install step instead of as 'Unable to find libfuse' mid-test.
The advisory integ-fuse-windows job went red on a chocolatey.org 503.
* refactor(sandbox): mount once, run lines verbatim, drop path magic
Remove the line translation, the MIRAGE_<prefix> env injection, and
the per-line reconcile state. The contract is now plain: the sandbox
mounts the workspace's backends once at boot (mirage mount add per
mount, spec in the exec env), mounts appear at
<workspace_root>/<prefix>, the session cwd is rebased, and the line
runs verbatim. Path consistency beyond the rebased cwd is the
caller's job; static rewriting could never be complete (quoted code,
runtime-built paths) and half-working magic is worse than none.
Docs and the daytona example teach the relative-path contract. Real
docker e2e green: relative read and write through the mounted bucket
with rebased cwd.
* refactor(sandbox): one general SandboxConfig, shared constants, provider packages
* refactor(sandbox): provider-owned configs, spec-derived mounts, remote sweep
* refactor(sandbox): connect-only runtimes, one in-sandbox workspace
* refactor(sandbox): drop the lazy provider re-exports
* refactor(sandbox): user-provisioned sandboxes, connect and exec only
* fix(sandbox): safeguard whole lines, per-invocation stdin paths, trim sandbox extra
* refactor(safeguard): one resolve_safeguard entry point, shared guard_output boundary
|
||
|
Zecheng Zhang
|
b117125495 |
refactor(filetype): remove the bundled format renderers, keep the extension point (#651)
* refactor(filetype): remove the bundled format renderers, keep the extension point
mirage shipped renderers for parquet, ORC, feather/arrow/ipc and hdf5/h5, plus
a PDF module that turned out to be entirely dead code: '.pdf' was never in the
factory registry and nothing imported mirage.core.filetype.pdf, so its test
exercised it directly and kept it looking alive.
Both core/filetype/ trees are removed. The dispatch machinery stays, in both
languages and in both places it lives (commands/builtin/filetype_factory and
ops/generic/factory), with empty registries and a comment marking them as the
extension point. A file with an unregistered extension now reads as raw bytes.
Registering a filetype-scoped command still works and is covered:
tests/commands/custom/test_filetype_fns.py registers a '.parquet' handler with
a fake function and asserts dispatch, and test_unregister_removes_all_filetypes
now registers a '.demo' renderer itself rather than leaning on the bundled ones.
Also removed: the parquet/hdf5/pdf extras (and their references from 'all' and
'deepagents'), the hyparquet, hyparquet-writer, apache-arrow and h5wasm
dependencies, integ/resources/columnar, the columnar FileType enum members, the
per-backend filetypeRead declarations that existed only on the TypeScript side,
and the sentence advertising 'cat on .parquet/.orc/.feather returns a formatted
table' from 33 backend and agent prompts, which would otherwise have been
lying to agents.
grep_helper's skip-list and file's MIME map keep their columnar entries: those
are still correct, since the formats remain binary whether or not mirage can
render them.
core/src/ops/generic/factory.ts was reaching NodeJS.ErrnoException through the
columnar packages' type dependencies. core has to work in both runtimes, so it
now uses a structural { code?: string } instead.
Python suite passes, TypeScript core 5458 pass, pre-commit clean.
* refactor(examples): drop the columnar demos alongside the renderers
Deleted, because their whole subject was the removed rendering:
s3_data.py (181 lines built around four columnar constants),
ram_filetypes.ts, box_parquet.ts, dropbox_parquet.ts, ram_parquet.ts.
Also deleted fuse_hooks.py, which was already broken before this change: it
imports mirage.fuse.filetype.data.local.parquet, a module that does not exist
anywhere in the tree.
Edited rather than deleted, since columnar was one section of a broader demo:
gdrive_complex.py, disk.ts, ram_fuse.py, and the openhands README (which
advertised format-aware reads and piped a .parquet through jq).
Everything still referencing parquet only names it in find/ls patterns, which
keeps working: mirage can still list and locate these files, it just no longer
renders them.
* fix(ci): drop the last filetype wiring the removal missed
The nextcloud ops table still declared filetypeRead for feather/hdf5/
parquet, so makeGenericOps threw at module load and every job that
imported @struktoai/mirage-node died before running anything.
Also: regenerate the specs (the crash hid 89 files of drift), drop the
removed pdf/parquet/hdf5 extras from the install matrix, take the
columnar members out of the TS FileType enum and the box/gdrive/dropbox
type guesses so both languages agree again, and stop passing an inert
filetype_read=True from ten Python backends.
* fix(spec): regenerate with every optional backend importable
The previous regeneration ran under the worktree venv, which has no
optional extras installed, so backends that failed to import were
dropped from each spec's resources list. Regenerated with the full
environment: the only remaining change is the emptied filetypes list.
* fix(agents): stop telling models mirage renders columnar formats
The langchain backend mapped parquet/h5/hdf5/feather to text/plain, so
with no renderer registered it handed the model raw binary decoded as
UTF-8 instead of the binary redirect. Those extensions now fall through
to application/octet-stream.
The system prompt, the execute tool description in both languages, and
the README extensibility example all still advertised columnar rendering
as a shipped feature; they now describe it as the extension point it is.
Also drops commands/optional.py, which existed only to soft-import the
removed format helpers and has no callers left.
* refactor(filetype): drop the filetype command factory, keep mount registration
The factory built nine commands per registered extension, but with no
renderers shipping it produced nothing on every one of the ~19 backends
that called it, and its handlers had no test in either language while
the module contract they expected was written down nowhere.
Removes commands/builtin/filetype_factory/ in both languages along with
the filetype_read / filetypeRead op knobs and the reads and provisions
that existed only to feed them. Registration on a mount survives and is
the whole extension point: a command or op carrying a filetype resolves
as (name, filetype) before (name, resource).
examples/{python,typescript}/filetype/ register a .tally renderer end to
end and are gated in CI against integ/truth/*/filetype.txt, so the path
is now exercised rather than asserted. Both emit byte-identical output.
|
||
|
Zecheng Zhang
|
37d734b67e |
feat(integ): jaeger backend, real-server observability integ, cross-backend contract (#643)
* feat(integ): jaeger backend, real-server observability integ, cross-backend contract
Add a jaeger backend and put both observability backends under a real server
in integ, then extend the shared read-only contract across nine backends.
Jaeger backend (python + typescript):
- service-scoped tree: /services/<name>/{operations.json,traces/<id>.json}.
Jaeger's search API requires a service, so there is no listable /traces.
- always sends an explicit microsecond window: `lookback` is ignored by the
query API, so without start/end the search silently returns nothing.
- an unknown service answers 200 with an empty list, so existence is checked
against the service list rather than inferred from an empty listing.
Langfuse fixes found by running against a real self-hosted instance:
- prompt versions were unlistable: the list endpoint returns a `versions`
array, and reading a scalar `version` collapsed each prompt to one 0.json.
- dataset runs rendered as an indented document under a .jsonl name.
- a 404 leaked the raw SDK error instead of ENOENT; a 500 still propagates.
- stat accepted any plausible path without checking it exists.
- typescript requested dataset runs under /v2/, a hard 404 on a real server.
- typescript applied a hidden 7-day window that hid traces cat could serve.
- an unrecognized path resolved to scope "root", so the grep/rg push-down
answered a missing file with every trace in the mount and exit 0.
GNU alignment:
- du reported a missing operand as size 0 with exit 0; now reports it and
exits 1 while keeping output for the operands that exist.
- tree wrote a malformed line to stderr where GNU writes nothing.
ENOENT alignment across backends:
- trello, linear, slack and email readdir returned [] for an unrecognized
path, so ls and tree reported a bogus path as real but empty.
- email selected an unvalidated IMAP folder, leaking "command SEARCH illegal
in state AUTH" to the caller.
- email and slack hand-rolled an ENOENT whose message carried an "ENOENT: "
prefix, and slack's dropped the mount prefix.
- trello typescript readBytes lacked the virtual path python passes, so the
message reported the mount-relative path.
Integ:
- resources/observability/ holds langfuse (135 cases), jaeger (99) and the
36-case shared contract, which runs on nine read-only backends via a new
{mount} token so one case can cover backends with different mount paths.
- targets gain a facet, and the runners gain --facet, so CI runs one backend
family per job: observability, project, email, chat, dify, mem0, core.
- langfuse runs against a six-container compose stack, jaeger against a
single container seeded over OTLP.
Two contract cases stay scoped away from email and gmail: their grep/rg push
down to a server-side search that reports "no matches" for a path that does
not exist. Six typescript-only ENOENT prefix sites remain in box, discord,
gdocs, gdrive, gsheets and gslides.
Also includes the pre-existing session-mode integ migration that was already
in the tree: session_modes scripts replaced by session/modes.json.
* fix(jaeger,langfuse): CodeQL ReDoS, eslint findings, formatter pass
- stat used a /\/+$/ trailing-slash regex in jaeger and langfuse, which
CodeQL flags as js/polynomial-redos on a path from library input. Both now
use the existing loop-based rstripSlash. Python already used str.rstrip.
- errorMessage stringified an unknown errors[0].msg, which could render as
[object Object]; now requires a non-empty string, mirroring python's
truthiness check.
- readdir tested pattern against undefined, which its type excludes.
- drop an unused fixture constant in read.test.ts.
* fix(integ): shell-attributed redirect golden, regenerate specs for jaeger
The ro_write_refused contract case expected "echo: <path>: Operation not
supported", but #635 (which landed in main after this branch was cut)
attributes a failed redirect target to the shell rather than the command, so
the correct stderr has no command prefix. It is the only contract case using a
redirect; mkdir, rm and tee take file operands and keep their prefixes.
Regenerate spec/ for the new jaeger resource (154 one-line additions across
both generators). CI runs gen_specs.py and gen-specs.ts as a separate drift
step, not through pre-commit, so a local pre-commit run does not catch this.
* fix(ci): build mirage-browser in the observability and facet integ jobs
The typescript runner imports mirage-browser statically for the opfs target,
so its dist must exist even in a job that runs no browser target. Both new
jobs hand-roll their setup and built only core and node, so every typescript
host step died with ERR_MODULE_NOT_FOUND. The pre-existing jobs get the build
from the integ-battery-setup action, which is why only these two were hit.
This was masked in the first run: the typescript step is skipped when the
python step fails, so the stale golden hid it.
* fix(jaeger,du): codex review, service-scoped reads and a real request deadline
read() served a trace fetched by id through any service directory, so
/services/<other>/traces/<id>.json returned content that stat and ls both
report absent. It now asserts the service exists and that the trace's own
process table names it. Membership comes from the trace document, not the
service listing, because that listing is windowed and limited and would hide a
trace that really does belong.
The typescript jaeger transport ignored requestTimeout: the config field, the
snake-case mapping and the transport option all existed, but nothing read them
and fetch ran with no deadline, so a stalled endpoint hung the command forever.
Threaded through as seconds, matching python's httpx timeout and dify's
existing requestTimeout convention.
du's operand stat caught every exception and reported it as
"No such file or directory", so an auth failure or a backend bug printed a
wrong reason and a partial total. Narrowed to isMissingPath, matching python's
(FileNotFoundError, ValueError). The test mock rejected with a bare
Error('ENOENT') rather than a stamped FsError, which is what let this pass.
Moved the jaeger resource's command and op imports to module scope, per the
repo's import rule. Verified no cycle: the registry and workspace still import
cleanly. This also matches the typescript resource, which already imports
JAEGER_COMMANDS at module scope.
Integ gains a cat and a stat through a foreign service, the pair that proves
the two agree.
|
||
|
Zecheng Zhang
|
9e0b6ef49d | feat(watch): attach/detach runtime surface, per-root overflow collapse, nested-mount coverage (#598) | ||
|
Zecheng Zhang
|
4a3347b75c |
feat(watch): resource change watching + Nextcloud source (#450) (#594)
* feat(watch): resource change watching + Nextcloud source (#450) Add a mount-scoped watch API so an external agent service can react to file changes instead of polling and diffing snapshots. Core seam (inert without the watcher): - types: ChangeKind, OverflowPolicy, ResourceChange, Delta - Workspace.watch() delegation + attach_watch_runtime() slot via a local WatchDelegate protocol, so core never imports the watch package mirage/watch package: - DeltaHook / SupportsChanges / WatchQueue / WatchRuntime protocols - RAMWatchQueue: per-path coalescing + pluggable overflow policy - generic ListingDeltaHook (snapshot diff, works on any backend) - Watcher + enable_watch: per-(mount, root) ref-counted pollers, bounded mailboxes, invalidate-before-deliver, nudge() Nextcloud source: - recursive WebDAV walk (ETag detector, mtime|size fallback) - NextcloudResource.delta_hook() Tests: 33 unit tests (queue, poller, watcher, workspace delegation, nextcloud hook). Integ: integ/watch.py driven by watch_nextcloud.json, mutating through a separate accessor so the verify read proves invalidate-before-deliver; wired into test_integ.yml. * test(watch): self-asserting JSON battery in integ/watch/ Replace the truth-file integ with a self-checking runner. Case files live in integ/watch/ (one JSON per resource, nextcloud.json today); the runner asserts expected event kind/path and the post-event verify read, exiting nonzero on any mismatch. External mutations go through a separate opendal operator, which generalizes to other backends (s3, gcs) for future case files. Drops truth_watch.txt and integ/resources/watch_nextcloud.json. * refactor(watch): package layout + broaden integ checks - constants.py: DEFAULT_MAX_PENDING, DEFAULT_POLL_INTERVAL, DIR_DETECTOR (both defaults are ours, not from #450; documented as such) - queue/ package: WatchQueue protocol + OverflowPolicy + errors in queue/base.py, RAMWatchQueue in queue/ram.py - source.py: Source/Subscriber runtime dataclasses out of watcher.py - drop the default_queue helper; RAMWatchQueue is itself a valid QueueFactory (root-only ctor), so it is the default directly - keep the change model (ChangeKind/ResourceChange/Delta) in types.py as a shared leaf: the producer (Workspace.watch) and the watch machinery both need it, so a neutral leaf avoids a workspace<->watch cycle without a TYPE_CHECKING workaround - integ: nextcloud.json cases now assert cat/head/ls/grep after create/update/delete, proving reads are fresh post-invalidation (grep no longer matches a deleted file; old content gone after update) * feat(watch): precise push via notify() + sample webhook server Push (the issue's webhook-first ask) without Mirage hosting a server: the consumer's own service receives the Nextcloud webhook and injects a precise change. - Watcher.notify(change): invalidate-before-deliver then deliver to matching subscribers, no poll; path reframed to the owning mount. nudge() stays for imprecise doorbells (no path). Pull remains the reconciliation baseline; duplicates from overlap are coalesced. - WatchRuntime protocol gains notify(). - integ/watch/webhook_server.py: the reference aiohttp receiver a consumer copies, mapping NodeCreated/Written/Deleted/Renamed payloads to ResourceChange and calling notify(). Mirage hosts no HTTP. - integ runner now runs each case file in BOTH pull (poll+nudge) and push (webhook->notify) mode, proving both transports yield identical events and fresh reads. - 4 notify() unit tests (deliver without active poll, invalidate-before- deliver ordering, delete->unlink, path reframing). * refactor(watch): exceptions to errors.py, OverflowPolicy to types.py - mirage/watch/errors.py holds QueueOverflowError + QueueClosed (matches the repo's per-package errors.py convention) - OverflowPolicy StrEnum moves to mirage/types.py next to ChangeKind, the shared-leaf home for watch enums - queue/base.py is now just the WatchQueue protocol + QueueFactory - re-exports in queue/__init__ keep the public import paths stable * refactor(watch): mirage runs no poller; notify-driven runtime + fingerprint naming Core decision: mirage ships detection *utilities*, not a detection *loop*. The consumer owns the loop (their server already exists); the integ demonstrates it in ~10 lines. - Watcher is now purely notify-driven: no poll tasks, no Source, no nudge. notify() = invalidate-before-deliver then fan out to matching subscribers. watch() works on ANY mount (no capability required to subscribe); SupportsChanges/delta_hook only power pull detection. - ListingDeltaHook + NextcloudWalk stay as importable pull utilities; the consumer's poller is: pull(root, checkpoint) -> notify each change -> keep checkpoint (ConsumerPoller in integ/watch/run.py). - Rename detector/version -> fingerprint, aligning with mirage's existing FileStat.fingerprint concept; shared default_fingerprint() (native ETag, else mtime|size) moves to the generic poller and the Nextcloud walk uses it. - Exceptions live in watch/errors.py; OverflowPolicy in types.py. - pop(): comment documenting the condition-wait loop (no busy spin). - integ pull mode = the DIY poller demo, now fully deterministic (pump once per case, no sleeps); push mode unchanged and now provably webhook-only (there is no poller to interfere). - DEFAULT_POLL_INTERVAL deleted (no loop to configure). * refactor(watch): poller.py -> delta.py, stat_fingerprint in utils, glob + multi-path watch scopes * feat(watch): push-mode example, scope/warm-cache integ battery, ancestor-chain invalidation * feat(watch): workspace.watch accepts str/pattern/list at the facade; runtime stays PathSpec-only * refactor(watch): FileEvent/FileChangeKind/FileMetadata event shape, UTC datetime timestamp * feat(watch): lazy runtime attach; watch package fully decoupled via WatchMount/WatchRegistry protocols * test(watch): pin middle-wildcard glob scope; document glob depth vs recursive * feat(watch): GNU glob depth semantics for watch scopes (slashless = entries, trailing slash = dir subtrees) * feat(watch): drop recursive flag; root shape defines depth (literal = subtree, /* = shallow, */ = dir subtrees) * docs(watch): Watch page under Python tab (scopes, event model, push/pull, queues) * feat(watch): MOVE evicts both sides; integ covers all scope shapes + all FileChangeKinds; watch matrix docs |
||
|
Zecheng Zhang
|
09b38deef8 | Fix example command registrations (#588) | ||
|
Zecheng Zhang
|
19aae81a4a |
fix(examples): use real gws command names (#585)
Every gws invocation in the Google examples used a hyphenated name that is not registered, so each one exited 127. The examples print only stdout and never check the exit code, so the failures rendered as empty sections. The registry has two naming families: generated passthrough gws <service> <resource> <method> hand-written helpers gws <service> +<verb> So gws-docs-documents-create becomes gws docs documents create, and gws-sheets-read becomes gws sheets +read, not gws sheets read. Python and TypeScript register identical names. man reads only argv[0], so man now passes the multiword name quoted. |
||
|
Zecheng Zhang
|
fa4a5af14d |
feat(box): serve special files raw + grep/rg search push-down (#579)
* feat(box): serve special files raw + grep/rg search push-down Drop JSON rendering for .boxnote/.boxcanvas/.gdoc/.gsheet/.gslides; serve every Box item as its raw bytes under its real name. The render only earns its keep where a structured API can round-trip edits (Google/gws); Box has no such API, so the projection was one-way and lossy, and it hid the bytes the search index actually matches on. Add a grep/rg content-search push-down (mirrors dropbox): a content_search config flag routes recursive literal scans through Box /2.0/search scoped by ancestor_folder_ids to narrow candidate files, then re-scans them locally so output stays byte-identical to a full walk. Full-walk fallback on API error, truncation, empty result, or non-folder scope. * test(box): add command-level search push-down coverage Port dropbox's narrow/grep/rg search tests to box (both languages) plus the TS core search test, closing the parity gap: box only had core-level narrow_paths coverage, so nothing exercised the fire-vs-bypass gate that decides when grep/rg must skip Box search and read files directly (-v, -c, regex patterns, --type/--glob, non-folder scope, empty/truncated results, binary-extension drops). Re-export keepVisible now that rg.test imports it. |
||
|
bytecii
|
db35b4888f |
feat(dropbox): grep/rg search push-down via files/search_v2 (#568)
* feat(dropbox): grep/rg search push-down via files/search_v2 Recursive grep/rg on a Dropbox mount previously downloaded every file. With the new content_search / contentSearch config knob (off by default: full-text search is plan-gated and its index lags recent writes), both commands now ask /2/files/search_v2 which files contain the pattern's literal and download only those candidates. Output stays exactly GNU/ripgrep because the local scan still decides every match: - Core searchFiles pages search_v2 + search/continue_v2, dedups across pages, and reports the 10,000-match ceiling; narrowPaths maps path_lower/path_display back to mount paths under root_path, sorts narrowed candidates into sorted-readdir walk order, and rebases raw_path onto the scope spelling so labels match a walk's. - narrow_scope gates the push-down: literal (or regex-required-literal) single patterns only, recursive scans only, directory operands only, and never for output modes that must see every file (grep -v/-c, rg -v/--type/--glob). Empty/failed/truncated searches fall back to the full walk; binary-extension candidates are dropped to mirror the walk's skip; rg prunes hidden candidates segment-wise and forces walk-style filename labels. - Both wrappers keep the factory's default_provision so cost estimates are unchanged. - fix(grep, python): grep -Rl with a file operand now stats first and scans the file instead of readdir-walking it (GNU + TS parity); narrowed candidates exercised this path. - Fakes gain search_v2 + search/continue_v2 (case-insensitive substring over names and content — a superset of real token matching, which is what narrowing needs) with cursor paging; battery adapters enable the knob, so all dropbox/dropbox-root grep/rg cases now exercise the push-down live: 988/988 per target on both hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(dropbox): CI fixes — TS grep -Rl file operands, rg -I labels, formatting The local TS battery ran against a stale mirage-node dist (only core was rebuilt), so the node DropboxResource never forwarded contentSearch and search narrowing was silently inactive on the TS host; CI's fresh build activated it and exposed two latent TS bugs python had already fixed: - grepFilesOnly walked file operands under -r (readdir on a narrowed file candidate -> ENOENT warnings, empty output). It now stats first and takes the single-file scan for file operands (GNU + python grep_files_only parity); regression tests in both languages. - rg's plain-line path delegates to grepGeneric, whose single-file body honors -H over -h, so the wrapper's forced label defeated -I suppression. Both wrappers now skip forcing H when -I is set; regression tests in both languages. Also formats the new files pre-commit never saw locally (they were untracked when it ran; --all-files only covers git ls-files) and settles two formatter fights: the provision calls are hoisted onto a shared dropboxResolveGlob const so Prettier/ESLint agree, and the test_grep_helper import gets grep_helper via a module import so yapf/isort converge. Verified with fresh core+node dists: dropbox/dropbox-root 988/988 on both hosts (narrowing live), ram/disk 2170/0, core vitest green, pre-commit converges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: bytecii <bytecii@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
607d057d6c | Upgrade agent framework integrations (#573) | ||
|
Zecheng Zhang
|
f6d38c0699 | Improve Pi agent integration (#571) | ||
|
Zecheng Zhang
|
56af010779 |
Upgrade agent SDK integrations and add OpenAI file reading (#569)
* Upgrade agent SDK integrations * Refactor agent file type constants * Support compatible agent API providers |
||
|
bytecii
|
a51aa7e375 |
feat(dropbox): subfolder mounts via rootPath + Python port + battery targets (#558)
* feat(dropbox): mount a subfolder as the resource root via rootPath
Add an optional rootPath to DropboxResource (node + browser): the
configured folder becomes the mount root, scoping every command and
FUSE/VFS op to that subtree. The root is normalized once on the
accessor ('' for account root, /seg/seg otherwise, '..' rejected) and
prefixed in the two core path builders (readdir's dropboxPathFromKey,
read/stream's dropboxPathFromVirtual), so stat/du/find/glob inherit it
for free. Config dicts accept snake_case root_path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): dropbox mock-server integ covering subfolder mounts
Follow the notion.ts pattern (read-only API backend, in-process fake
server, truth-file diff): integ/dropbox.ts spins up a fake Dropbox API
(oauth2/token, files/list_folder, files/download) and runs the shell
battery against an account-root mount, a rootPath subfolder mount, and
a slash-variant spelling of the same root. The subfolder mount's
request log is asserted to contain no API path outside the configured
root, and sibling/parent-escape reads fail with ENOENT.
Requires a test-only endpoint override in DropboxConfig (one origin
serving oauth + api + content), mirroring the hf fake-hub knob.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): dropbox + dropbox-prefix targets in the declarative battery
Replace the standalone dropbox.ts/truth-file approach with proper
battery targets, mirroring s3/s3-prefix and hf/hf-prefix: 'dropbox'
mounts three isolated fake accounts, 'dropbox-prefix' mounts three
rootPath subfolders of one shared account. The TS adapter self-hosts
a fake Dropbox API per account (oauth2/token, files/list_folder,
files/download via the DropboxConfig endpoint override) and seeds
fixtures into it directly — dropbox is a read-only backend, so the
workspace mkdir/tee seeding path cannot run; Open.seeded lets an
adapter opt out of harness seeding.
The targets join the 632 read-only cases (1264 case-runs, all green,
stable across reruns): write-command cases, cases reading state
written by earlier write cases, and history-reading cases (coupled to
the exact per-target command sequence) stay excluded. Python hosts
skip the target (TS-only backend).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): Python port of the Dropbox backend + battery targets on both hosts
Port the TS dropbox backend to Python, closing the runtime gap:
core/dropbox (_client with token manager + endpoint override, api,
readdir, read/stream, stat), DropboxAccessor with root_path
normalization, read-only CommandIO (du uses the generic readdir+stat
walk), generated ops/commands, DropboxResource + registry entry.
Two behavior fixes shared with TS along the way:
- readdir maps list_folder 409 (path/not_found, path/not_folder) to
ENOENT on both sides, so ls on a file operand falls back to its
stat-the-operand path and missing dirs report No such file or
directory instead of a raw API error.
- the Python find builder's walk fallback now rebases results onto the
operand as typed (rebase_raw), matching generic_find and GNU display
semantics; previously cd /data && find disptree printed absolute
paths on walk-fallback backends.
Integ: dropbox-prefix target renamed to dropbox-root with a root mount
field (dropbox's knob is rootPath, matching ssh/nextcloud's root
convention, not s3's keyPrefix). Both dropbox targets now run on the
python host too: aiohttp fake (integ/server/dropbox_server.py),
DropboxService with out-of-band seeding, and the same seeded opt-out
in the python runner. Case set re-converged empirically across BOTH
hosts: 632 read-only cases per target, 1264 case-runs per host, green
and stable; history-reading and write-dependent cases stay excluded.
Docs: docs/python/resource/dropbox.mdx, resource matrix row gains the
Python link, TS page cross-links, examples/python/dropbox/dropbox.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(integ): move the TS fake Dropbox into integ/server/dropbox.ts
Fake backends live under integ/server/ (hf_server.py, onedrive_server.py,
dropbox_server.py); the TS fake was the odd one out under runners/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: formatter churn from merge + fix stale fake path reference
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(spec): regenerate command specs with the dropbox resource
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): read/write backend — upload, mkdir, rm, mv, cp on both hosts
Wire the Dropbox mutation endpoints (upload, create_folder_v2,
delete_v2, move_v2, copy_v2) into full write support in both
languages: write/create/mkdir/unlink/rmdir/rm_r/rename/copy cores with
cache+ancestor invalidation, EEXIST/ENOENT/EISDIR/ENOTDIR mapping, and
emulated truncate in the ops factory. Key semantics:
- rmdir guards ENOTEMPTY before delete_v2 (which deletes folders
RECURSIVELY — the s3 data-loss lesson); rm -r maps to one call.
- rename/copy replace an existing destination FILE like GNU mv/cp
(delete + retry on to/conflict); folder conflicts propagate. No
dir_copy is wired so cp -r merges into existing dirs file-by-file.
- mkdir owns GNU semantics (EEXIST without -p, ENOENT on missing
parent) since create_folder_v2 auto-creates parents; the mount root
is always-exists (the API rejects the empty path — an unguarded
mkdir -p on the mount root used to plant a corrupt '' folder in the
fakes that listed itself as its own child and looped find forever).
- stat/read gain API-truthful index-less fallbacks (get_metadata /
direct download) so unlink/rmdir classification, the wired TS find
(required by the cp planner), and emulated truncate work.
- single-call uploads cap at ~150 MB (documented; no upload sessions).
TS drops its provisionOverrides (python's defaults match the shared
battery expectations) and both hosts gain the filetype command set
(cat_parquet & co), closing the col_* exclusions.
Battery: the out-of-band seeding opt-out is deleted — dropbox seeds
through the workspace mkdir/tee like every writable backend, which
exercises the write path itself. Fakes gain explicit folder objects,
the write endpoints, real-clock upload stamps (find -mtime), an 8 MiB
aiohttp body cap (example.h5 is ~1.02 MiB), and loud empty-path
guards. dropbox/dropbox-root now run 988 of the 997 s3-covered cases
per target (1976 case-runs per host, green and stable on both): write,
history, and meta chains included; only 8 TS-walkFind predicate gaps
(find_d/find_empty/…, tracked separately) and mtime_dir_not_epoch
(Dropbox folders carry no mtime) stay excluded.
Docs flip to read/write; specs regenerated (51 files gain the dropbox
write/filetype command rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: formatter churn on dropbox write files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Zecheng Zhang
|
186804d5f7 | chore(daemon): MIRAGE_HOME is the single root; mirror the disk-store default in TS (#551) | ||
|
Zecheng Zhang
|
a9aa94e723 |
refactor(runtime): interpreters are handler internals, one Runtime seam (#543)
* refactor(runtime): interpreters are handler internals, one Runtime seam Unify PythonRuntime/JsRuntime into one Runtime contract (RunArgs/RunResult, captures, attach); derive command bindings from an ordered runtimes list (first capturer wins, vfs an ordinary entry); one shared interpreter command core; dispatch injects a single bound runtime only for commands that have one. Deletes python_runtime/js_runtime/runtime_options kwargs, yaml keys, selectors, and registry globals. * fix(integ): cli_runtime battery speaks runtimes: entries, create surfaces entry errors * refactor(runtime): TS RunResult.stderr null when empty, matching Python |
||
|
Zecheng Zhang
|
1c193bc4e8 |
Optional-typing sweep: path-only generic ops, NULL_INDEX, no bare generics (#541)
* refactor(types): stdin annotations use the ByteSource alias * refactor(generics): injected ops are path-only, accessor+index bind at the wrapper Generic commands no longer take accessor (or a dead index) just to thread them back into injected callables. Builders and bespoke wrappers bind both via bound_op (None-passthrough) or partial for write-side ops, mirroring the TS builders' closures. call_*/resolve_pattern/relay helpers and the cache read-through wrappers gain path-first forms; CommandIO-level ops keep the raw (accessor, path, index) shape. * refactor(index): NULL_INDEX everywhere, no index-is-None branches index is a required IndexCacheStore on every op and wrapper; callers with no real index pass NULL_INDEX (the null object built for exactly this). Dead is-None guards drop out of trello/linear/github_ci/google readdir+stat, CacheManager, and the github wrappers. * style(types): parameterize every bare generic, gate with disallow_any_generics dict -> dict[str, Any] (payloads) / dict[str, object] (flag bags), Callable -> Callable[..., Any], plus list/tuple/Awaitable/Pattern/ Task/Future/Token/AsyncMongoClient. The mypy gate now carries disallow_any_generics so bare generics cannot come back. * style: pre-commit formatting + parameterize main's new bare dicts * fix: post-merge gate fixes + make security barriers analyzer-recognizable - resolve_within_root: startswith(root + sep) guard instead of commonpath (same semantics; the shape CodeQL models as a path-injection barrier) - cli table output: trimEnd() instead of the polynomial /\s+$/ trim - example proxies: re-encode validated endpoint segments before URL construction (recognized SSRF barrier; no-op for the allowed charset) - parameterize bare generics arriving from #540 (the new mypy gate caught them) * fix(server): plain startswith barrier in resolve_within_root |