* feat(dropbox): grep/rg search push-down via files/search_v2
Recursive grep/rg on a Dropbox mount previously downloaded every file.
With the new content_search / contentSearch config knob (off by default:
full-text search is plan-gated and its index lags recent writes), both
commands now ask /2/files/search_v2 which files contain the pattern's
literal and download only those candidates. Output stays exactly
GNU/ripgrep because the local scan still decides every match:
- Core searchFiles pages search_v2 + search/continue_v2, dedups across
pages, and reports the 10,000-match ceiling; narrowPaths maps
path_lower/path_display back to mount paths under root_path, sorts
narrowed candidates into sorted-readdir walk order, and rebases
raw_path onto the scope spelling so labels match a walk's.
- narrow_scope gates the push-down: literal (or regex-required-literal)
single patterns only, recursive scans only, directory operands only,
and never for output modes that must see every file (grep -v/-c,
rg -v/--type/--glob). Empty/failed/truncated searches fall back to
the full walk; binary-extension candidates are dropped to mirror the
walk's skip; rg prunes hidden candidates segment-wise and forces
walk-style filename labels.
- Both wrappers keep the factory's default_provision so cost estimates
are unchanged.
- fix(grep, python): grep -Rl with a file operand now stats first and
scans the file instead of readdir-walking it (GNU + TS parity);
narrowed candidates exercised this path.
- Fakes gain search_v2 + search/continue_v2 (case-insensitive substring
over names and content — a superset of real token matching, which is
what narrowing needs) with cursor paging; battery adapters enable the
knob, so all dropbox/dropbox-root grep/rg cases now exercise the
push-down live: 988/988 per target on both hosts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(dropbox): CI fixes — TS grep -Rl file operands, rg -I labels, formatting
The local TS battery ran against a stale mirage-node dist (only core was
rebuilt), so the node DropboxResource never forwarded contentSearch and
search narrowing was silently inactive on the TS host; CI's fresh build
activated it and exposed two latent TS bugs python had already fixed:
- grepFilesOnly walked file operands under -r (readdir on a narrowed
file candidate -> ENOENT warnings, empty output). It now stats first
and takes the single-file scan for file operands (GNU + python
grep_files_only parity); regression tests in both languages.
- rg's plain-line path delegates to grepGeneric, whose single-file body
honors -H over -h, so the wrapper's forced label defeated -I
suppression. Both wrappers now skip forcing H when -I is set;
regression tests in both languages.
Also formats the new files pre-commit never saw locally (they were
untracked when it ran; --all-files only covers git ls-files) and settles
two formatter fights: the provision calls are hoisted onto a shared
dropboxResolveGlob const so Prettier/ESLint agree, and the
test_grep_helper import gets grep_helper via a module import so
yapf/isort converge.
Verified with fresh core+node dists: dropbox/dropbox-root 988/988 on
both hosts (narrowing live), ram/disk 2170/0, core vitest green,
pre-commit converges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(box): python box backend + box integ target on both hosts
Port the TS-only box backend to python (core/box client, api, readdir,
read, stat, du; boxnote/boxcanvas filetype renderers; accessor, commands,
ops, resource, registry). Add a box integ target seeded over the real Box
wire shapes for both hosts, backed by the fake Box server.
Mount a subfolder by folder id (root_folder_id / rootFolderId): Box ids are
stable across renames and there is no path-resolution API.
Fixes surfaced by box, applied to both languages:
- readdir on a file id raises ENOTDIR so ls falls back to the file entry
- root stat fetches the folder's own metadata so find -mtime keeps the root
- walkFind emits the start point for a mount-root operand and supports -empty
- python _find_walk rebases output to the operand spelling
- makeJqProvision threads the index into stat
* feat(box): full write surface + enroll box integ target on write cases
Add the write family to the box backend in both languages: write/tee
(upload + overwrite as a new version), mkdir, unlink, rmdir, rm -r, mv,
cp, touch, truncate. Box is id-native with no path addressing, so write
ops resolve path->id with a fresh folder-listing walk (core/box/resolve)
and keep the read caches coherent through the invalidation context.
Fake Box server gains delete/update/copy endpoints. The box integ target
is now writable and enrolled on the same cases as hf on both hosts.
Notes for the id-native backend (unlike Graph's path addressing):
- stat falls back to the resolver walk when no index is threaded, so the
rm/mv/cp builders and provision estimation resolve ids without one
- mkdir -p invalidates every created level's parent listing, not just the
final target's, so a cached ancestor listing can't hide new folders
- TS cp requires a find op; box gets one backed by walkFind over a scratch
index it populates as it descends
* test(box): expand box integ enrollment to s3 parity; cp -r merge + du provision
Enroll box in the ~170 remaining cases s3 runs (grep/rg/zgrep/exit/errop/
find shell + read coverage), skipping only ORC (unsupported in TS). Both
hosts now run 1101 cases.
Two fixes this surfaced:
- cp -r into an existing directory now merges instead of replacing: box
copy recurses per child into an existing destination folder rather than
clearing and re-copying it server-side, matching GNU cp -r.
- the bespoke box du command now uses metadata_provision so its estimate
is exact (Python; TS already gets it from the factory default).
* fix(box): type box truncate download to satisfy tsc --noEmit
* chore(box): regenerate command specs with box in resources
* docs(box): python resource page, write surface, matrix and nav updates
* chore(spec): add dropbox to TS specs after merge
* feat(gridfs): MongoDB GridFS backend with native revisions and server-side find
Python + TypeScript GridFS resource mirroring the s3 backend: filenames as
slash-separated keys, zero-byte trailing-slash marker docs for directories,
full read/write command surface via the generic factories.
- Writes upload a new revision and keep old ones; reads/listings resolve
latest-per-filename via one fs.files aggregation; snapshot-pinned reads
fetch old revisions by file _id; mv retags filenames server-side so
history moves with the file; rm deletes all revisions.
- find pushes -name/-iname/-type/-size into the fs.files query (anchored
basename regex, marker-shape test, length bounds); every pushed condition
is a superset of GNU semantics and the shared keep() pass stays
authoritative, so behavior matches the other backends exactly.
- Integ: gridfs + gridfs-prefix targets enrolled everywhere s3/s3-prefix
run; 1101/1101 on both hosts; mongo:8 service added to integ-shared.
- Docs: setup + python resource + typescript setup pages, matrix row,
nav entries, README backend lists.
* chore(spec): regenerate command specs with gridfs
* ci: consolidate install matrix and python runtime jobs
* test(integ): enroll gridfs targets in brace expansion cases
* feat(dropbox): mount a subfolder as the resource root via rootPath
Add an optional rootPath to DropboxResource (node + browser): the
configured folder becomes the mount root, scoping every command and
FUSE/VFS op to that subtree. The root is normalized once on the
accessor ('' for account root, /seg/seg otherwise, '..' rejected) and
prefixed in the two core path builders (readdir's dropboxPathFromKey,
read/stream's dropboxPathFromVirtual), so stat/du/find/glob inherit it
for free. Config dicts accept snake_case root_path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): dropbox mock-server integ covering subfolder mounts
Follow the notion.ts pattern (read-only API backend, in-process fake
server, truth-file diff): integ/dropbox.ts spins up a fake Dropbox API
(oauth2/token, files/list_folder, files/download) and runs the shell
battery against an account-root mount, a rootPath subfolder mount, and
a slash-variant spelling of the same root. The subfolder mount's
request log is asserted to contain no API path outside the configured
root, and sibling/parent-escape reads fail with ENOENT.
Requires a test-only endpoint override in DropboxConfig (one origin
serving oauth + api + content), mirroring the hf fake-hub knob.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integ): dropbox + dropbox-prefix targets in the declarative battery
Replace the standalone dropbox.ts/truth-file approach with proper
battery targets, mirroring s3/s3-prefix and hf/hf-prefix: 'dropbox'
mounts three isolated fake accounts, 'dropbox-prefix' mounts three
rootPath subfolders of one shared account. The TS adapter self-hosts
a fake Dropbox API per account (oauth2/token, files/list_folder,
files/download via the DropboxConfig endpoint override) and seeds
fixtures into it directly — dropbox is a read-only backend, so the
workspace mkdir/tee seeding path cannot run; Open.seeded lets an
adapter opt out of harness seeding.
The targets join the 632 read-only cases (1264 case-runs, all green,
stable across reruns): write-command cases, cases reading state
written by earlier write cases, and history-reading cases (coupled to
the exact per-target command sequence) stay excluded. Python hosts
skip the target (TS-only backend).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): Python port of the Dropbox backend + battery targets on both hosts
Port the TS dropbox backend to Python, closing the runtime gap:
core/dropbox (_client with token manager + endpoint override, api,
readdir, read/stream, stat), DropboxAccessor with root_path
normalization, read-only CommandIO (du uses the generic readdir+stat
walk), generated ops/commands, DropboxResource + registry entry.
Two behavior fixes shared with TS along the way:
- readdir maps list_folder 409 (path/not_found, path/not_folder) to
ENOENT on both sides, so ls on a file operand falls back to its
stat-the-operand path and missing dirs report No such file or
directory instead of a raw API error.
- the Python find builder's walk fallback now rebases results onto the
operand as typed (rebase_raw), matching generic_find and GNU display
semantics; previously cd /data && find disptree printed absolute
paths on walk-fallback backends.
Integ: dropbox-prefix target renamed to dropbox-root with a root mount
field (dropbox's knob is rootPath, matching ssh/nextcloud's root
convention, not s3's keyPrefix). Both dropbox targets now run on the
python host too: aiohttp fake (integ/server/dropbox_server.py),
DropboxService with out-of-band seeding, and the same seeded opt-out
in the python runner. Case set re-converged empirically across BOTH
hosts: 632 read-only cases per target, 1264 case-runs per host, green
and stable; history-reading and write-dependent cases stay excluded.
Docs: docs/python/resource/dropbox.mdx, resource matrix row gains the
Python link, TS page cross-links, examples/python/dropbox/dropbox.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(integ): move the TS fake Dropbox into integ/server/dropbox.ts
Fake backends live under integ/server/ (hf_server.py, onedrive_server.py,
dropbox_server.py); the TS fake was the odd one out under runners/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: formatter churn from merge + fix stale fake path reference
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(spec): regenerate command specs with the dropbox resource
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(dropbox): read/write backend — upload, mkdir, rm, mv, cp on both hosts
Wire the Dropbox mutation endpoints (upload, create_folder_v2,
delete_v2, move_v2, copy_v2) into full write support in both
languages: write/create/mkdir/unlink/rmdir/rm_r/rename/copy cores with
cache+ancestor invalidation, EEXIST/ENOENT/EISDIR/ENOTDIR mapping, and
emulated truncate in the ops factory. Key semantics:
- rmdir guards ENOTEMPTY before delete_v2 (which deletes folders
RECURSIVELY — the s3 data-loss lesson); rm -r maps to one call.
- rename/copy replace an existing destination FILE like GNU mv/cp
(delete + retry on to/conflict); folder conflicts propagate. No
dir_copy is wired so cp -r merges into existing dirs file-by-file.
- mkdir owns GNU semantics (EEXIST without -p, ENOENT on missing
parent) since create_folder_v2 auto-creates parents; the mount root
is always-exists (the API rejects the empty path — an unguarded
mkdir -p on the mount root used to plant a corrupt '' folder in the
fakes that listed itself as its own child and looped find forever).
- stat/read gain API-truthful index-less fallbacks (get_metadata /
direct download) so unlink/rmdir classification, the wired TS find
(required by the cp planner), and emulated truncate work.
- single-call uploads cap at ~150 MB (documented; no upload sessions).
TS drops its provisionOverrides (python's defaults match the shared
battery expectations) and both hosts gain the filetype command set
(cat_parquet & co), closing the col_* exclusions.
Battery: the out-of-band seeding opt-out is deleted — dropbox seeds
through the workspace mkdir/tee like every writable backend, which
exercises the write path itself. Fakes gain explicit folder objects,
the write endpoints, real-clock upload stamps (find -mtime), an 8 MiB
aiohttp body cap (example.h5 is ~1.02 MiB), and loud empty-path
guards. dropbox/dropbox-root now run 988 of the 997 s3-covered cases
per target (1976 case-runs per host, green and stable on both): write,
history, and meta chains included; only 8 TS-walkFind predicate gaps
(find_d/find_empty/…, tracked separately) and mtime_dir_not_epoch
(Dropbox folders carry no mtime) stay excluded.
Docs flip to read/write; specs regenerated (51 files gain the dropbox
write/filetype command rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: formatter churn on dropbox write files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: bytecii <bytecii@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(gdrive): read/write Drive backend, gws CLI, fake Workspace server, integ targets
Bring Google Drive up to the s3 backend model and add Google Workspace
coverage to the declarative integ harness.
gdrive core (both langs):
- write core: write/mkdir/unlink/rmdir/rm/rename/copy/exists/truncate/create
plus a resolve module that resolves paths by direct Drive queries (Drive is
id-addressed and allows duplicate sibling names, so mutations use server
state, not the read cache). GNU semantics: EEXIST, mv-overwrite with
ENOTEMPTY on non-empty dirs, cp -r merge. Native gdoc/gsheet/gslide files
render as API JSON and reject raw writes.
- folder_id subfolder scoping (the s3 key_prefix analog) and a gdrive-folder
target.
- versions via the Drive Revisions API, wired into snapshot pin/restore.
- find and du backend ops (tree walker mirroring msgraph).
gws commands (both langs):
- passthrough factory generating drive/docs/sheets/slides API commands that
emit raw API-resource JSON.
- a top-level gws dispatcher accepting official CLI syntax
(gws docs documents get, gws sheets +read, ...).
Config seam:
- GoogleConfig gains api_base (point backends at a fake server) and folder_id.
Integ:
- fake Google Workspace server (Drive v3 + Docs v1 + Sheets v4 + Slides v1,
in-memory, /reset, deterministic clock, google-apps MIME auto-linking).
- gdrive and gdrive-folder join the universal battery; a google case group
asserts byte-exact API-resource renders.
- CI starts the fake server before the shared batteries.
* refactor(gws): fold create/batchUpdate into the method table, drop the dispatcher
The hand-written gws create/batchUpdate commands were passthroughs, and the
`gws` dispatcher was a second routing layer on top of the command dispatcher.
- Add docs/sheets/slides create + batchUpdate as rows in the gws method
table; delete the 6 bespoke command files and their now-orphaned
core create/update helpers.
- Delete the gws dispatcher. Every method is a first-class command the
parser resolves directly: `gws-docs-documents-get`, `gws-drive-files-list`,
`gws-docs-write`. No `gws docs documents get` space syntax.
- Tidy run_gws_method with a verb-to-caller map.
- Rewrite the integ google cases to the hyphenated command names.
* feat(gws): restore official space syntax via a thin spec-based gws command
The gws method table stays hyphenated (gws-docs-documents-get), but a
single thin gws command reuses the parser: its spec declares the flags,
its body reconstructs the target name from operands and looks it up in
the method table or helper map. No per-method routing dicts.
* fix(integ): gws fake server returns a generic 500 message
CodeQL flagged js/stack-trace-exposure: the caught exception message
flowed into the HTTP response. Log it server-side and return a generic
'internal error', matching how the real Google API responds.
* feat(gws): resolve nested command names, drop the gws dispatch shim
Command resolution now matches the longest registered command name over
the leading words (git-style), so official Google-CLI syntax like
`gws docs documents get` and `gws docs +write` resolves natively. The
mount registry keeps a first-token index of multi-word names; expand_argv
consumes the matched prefix as the command and the rest as operands, so
route/spec/dispatch key on the joined name unchanged.
This deletes dispatch.py/.ts (the re-parsing shim), their tests, the
resource registration hook, and the hyphenated gws-* command names.
Prompts and the integ google cases already used the space syntax.
* chore(spec): regen command specs, gdrive joins the write-command resource lists
* test(integ): cover files get/copy and sheets +write; fix +write ignoring api_base
The new g_sheet_write case caught a real bug: gws sheets +write built its
URL from the hardcoded SHEETS_API_BASE constant in both languages, so it
bypassed the configured api_base and hit the real Google endpoint. It now
goes through sheets_base(token_manager) like every other gws command.
* feat(gws): unify sheets +write onto the ergonomic helper flags
+write now takes --spreadsheet/--range/--values/--json-values like
+read and +append, instead of the raw API --params/--json shape. The
PUT moved into core gsheets write (update_values) next to append_values
so the command wrapper is wiring only. Prompts and the integ case use
the new surface.
* test(spec): use a nested command name in the help renderer fixture
* feat(gdrive): folder_id scoping works inside Shared Drives, denied writes are EACCES
A folder_id mount root may sit inside a Shared Drive (or be a Shared
Drive id): root_context resolves the root's driveId once via files.get,
memoizes it on the accessor, and threads it through every resolution,
listing, and mkdir walk, so scoped shared-drive mounts list and mutate
correctly (they previously listed empty).
Drive access is per-item, so a write mount can still hold read-only
items: mutations map an API 403 to EACCES on the operand via
eacces_on_denied / eaccesOnDenied instead of leaking a raw HTTP error.
The fake Workspace server gains shared drives (create/list, driveId
inheritance, real files.list visibility rules), the harness gains a
drive field on gws mounts, and a gdrive-shared integ target covers the
scoped shared-drive flow end to end in both languages.
* refactor(gws): gather the declarative tables in methods
GWS_API_SPEC, SERVICE_BASES, and SERVICE_RESOURCES move out of the
factory into methods.py/.ts next to GWS_METHODS, so the methods module
holds all the per-service data and the factory is behavior only.
* chore(gws): drop factory imports left behind by the methods move
Semantic search is now a 'search <query> <path>' command returning ranked
rows as canonical file paths plus score, instead of a virtual _search/<query>/
directory. grep/rg stay lexical. Mirrors dify's search command; identical
output across Python and TypeScript (score 0.2679).
* feat(dify): add Dify Knowledge resource
* fix(dify): stream read commands and tighten find behavior
* refactor(dify): refactor command implementations
* feat(dify): add search and document assembly
* refactor(dify): normalize path specs
* test(dify): split tests by layer and command
* feat(examples): add Dify knowledge example
* docs(dify): add resource documentation
* fix(dify): configure slug metadata name
* fix(dify): align resolve_glob with cross-resource convention
core/dify/glob.py was diverging from disk/s3/notion: it expanded
directories without a pattern and treated `**` as a special recursive
walk. Rewrite to follow the standard `resolved → pattern → else`
flow so non-glob paths pass through and only basename fnmatch is
applied. Add the missing resolve_glob call in the dify find and ls
commands.
* fix(dify): match OSError format on IsADirectoryError
* Add read-only Dify search commands
* fix(dify): repair ls/rg/stat/tree/grep after main merge
- ls: remove dropped `accessor`/`filetype_fns` kwargs from generic_ls;
fix stat partial to not pre-bake index (generic_ls passes it)
- rg: remove bound_* nested helpers and filetype_fns kwarg; pass
read_bytes/readdir/stat_light directly; pre-bind index only for
read_stream (generic_rg calls it without index)
- stat: remove redundant partial(stat_core, index=index); pass stat_fn
and index separately so generic_stat can forward index correctly
- tree: fix stat partial to not pre-bake index
- grep: remove nested _readdir/_stat/_read_bytes/_read_stream helpers;
use direct refs + partial(read_stream, index=index)
- test_rg: add get_segments mock for read_bytes path (directory scan);
remove redundant iter_segment_pages mock from directory test case
* fix(resource): drop stale paperclip entry, register dify in registry test
* test(dify): add mock-server integration test mirroring s3
* docs(dify): use Dify logo instead of brain icon
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* refactor(notion): delegate read commands to generic; remove telegram resource
- remove telegram resource fully (commands, core, ops, resource, accessor,
tests, registry/types entries, docs, examples)
- notion: cat/head/tail/ls/stat/tree delegate to generic (mirror trello);
grep/rg keep native Notion-search branch, delegate byte-scan fallback
* refactor(ssh): delegate commands to generic; align du_all core contract
- migrate ssh read/text/archive commands to generic (mirror disk) via
injected core.ssh callables; keep write ops (cp/mv/rm/touch/mkdir/ln) custom
- SSHAccessor.root property so generic guards work
- core/ssh du_all returns (entries, total) to match disk/generic_du contract
- cat keeps file_read_provision (cache-hit provision); ls defaults empty
paths to cwd (matches prior ssh behavior)
* feat(notion): wire jq + find to generic via new core scaffolding
- add core/notion/stream.py (read_stream) + core/notion/find.py (find_core
walk+filter matching generic_find contract)
- notion jq delegates to generic_jq; notion find delegates to generic_find
- offline test for notion find_core (type/name/depth/size filters)
* chore(ssh): drop unused paramiko dep; normalize touched docs
paramiko was declared in the ssh extra but never imported (ssh uses asyncssh).
* chore(ssh): actually drop paramiko from pyproject + uv.lock
The previous commit advertised the drop but only touched docs/code. This
removes paramiko from the ssh extra (asyncssh is the only ssh dep) and
syncs the lock (removes paramiko + bcrypt/invoke/pynacl transitives).
* fix(cache): keep write commands on the real mount, not the cache
resolve_mount redirected any command on a fully-cached remote path to
the cache mount. Read commands want that, but write commands (rm, mv,
truncate, redirects) then mutated only the in-memory cache and returned
exit 0 while the backend was untouched: e.g. `cat /ssh/x` followed by
`rm /ssh/x` left x on the server. Gate the redirect on read-only
commands; writes go write-through to the backend and the existing
apply_io step evicts the stale cache entry.
* test(integ): s3 rm-after-cat hits backend and evicts cache
End-to-end regression for the resolve_mount cache-routing fix: caching a
key then removing it must delete on the backend (not just the cache) and
the same-session re-read must miss. Fails if write commands are routed to
the cache mount.
* test(notion): offline coverage for generic command wiring
* feat(resource): add 9 S3-compatible provider wrappers
Thin Config + Resource(S3Resource) wrappers reusing core/s3 via
to_s3_config(), matching the r2/gcs/oci/supabase pattern. Each reports
ResourceName.S3 so all S3 commands/ops apply unchanged.
- Self-hosted (endpoint required, path-style): MinIO, Ceph
- Region-derived endpoints (virtual-host): Wasabi, Backblaze B2,
DigitalOcean Spaces, Tencent COS, Alibaba OSS, Scaleway, QingStor
Registered in resource registry; combined docs page + resource-matrix
rows. Stays on aioboto3 (no OpenDAL).
* feat(examples): add S3-compatible provider examples; fix Aliyun OSS endpoint
- Add one example per provider (minio/ceph/wasabi/backblaze/digitalocean/
tencent/aliyun/scaleway/qingstor), matching the r2/oci example shape.
- MinIO example is self-contained (WRITE mode, seeds + cleans up) and was
verified end-to-end against a local MinIO server.
- Fix Aliyun OSS endpoint: S3-compatible host is s3.oss-<region>.aliyuncs.com
(the AWS SDK cannot use the native oss-<region>.aliyuncs.com endpoint).
- QingStor endpoint flagged as override-if-needed (no clean region-derived
S3 host).
* test(integ): add MinIO mount to the moto-based S3 integration harness
Mounts MinIO alongside s3/gcs against the moto mock server and runs the
identical command battery, so matching output proves the MinIOResource
wrapper routes through the S3 backend with full parity (same grep counts,
streaming byte budgets, and index fast-path: ListObjectsV2=2 HeadObject=0).
Regenerated truth_s3.txt; verified deterministic.
* style: yapf-format minio example
* docs: split S3-compatible providers into per-provider pages with logos
Replace combined s3_compatible page with 9 provider pages (minio, ceph,
wasabi, backblaze, digitalocean, tencent, aliyun, scaleway, qingstor),
each with a brand logo and a dedicated setup page. Move Hugging Face
resources into their own nav section with the HF logo.
* feat(py): add NextcloudResource for WebDAV/Nextcloud support
Implements a new resource type for Nextcloud, ownCloud, Hetzner Storage
Share, and generic WebDAV servers. Agents can now mount WebDAV endpoints
as virtual filesystems via NextcloudResource.
- Core WebDAV client using aiohttp (existing dep) + xml.etree for PROPFIND
- Full VFS ops: read, readdir, stat, write, mkdir, unlink, rmdir, rename, copy, create, truncate
- 14 builtin commands: cat, ls, grep, find, head, tail, wc, stat, tree, mkdir, rm, cp, mv, touch
- ETag-based fingerprinting for snapshot/drift detection
- Registered in resource registry as 'nextcloud'
- 13 tests covering config, resource, and WebDAV client parsing
* refactor(nextcloud): reuse generic command set + docs
* fix(cache): evict child entries on index invalidate_dir
invalidate_dir cleared the children listing and expiry but left the
per-child entry rows, so a stat fast-path get() after a recursive
delete returned a stale entry. Evict the child entries too. Covers
both RAM and Redis index stores.
* feat(nextcloud): migrate to OpenDAL + add integ and tests
Rewrite the accessor and core ops on the OpenDAL webdav operator,
mirroring the hf_buckets migration. find/du now walk via op.scan
instead of the index, fixing a crash from the generic wrappers not
threading an index. Add unit tests (read/stat/readdir/write/find/du)
and an s3-mirrored integ run with CI workflow.
* fix(server): restore cache cast in summary buildInternals
A merge dropped the `as typeof ws.cache & { snapshotEntries?... }`
cast, making cache.snapshotEntries?.() error-typed and failing the
ts-eslint no-unsafe-* rules. Restore it to match main.
---------
Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com>
* refactor(hf-buckets): migrate to Apache OpenDAL backend
Replace aiohttp + huggingface_hub + hf_xet with opendal's Hf service.
opendal becomes a required dependency; hf_buckets extra removed.
Net: -730 lines, single shared accessor base for future HF resource types
(datasets, models, spaces).
* feat(resource): add hf_datasets, hf_models, hf_spaces (OpenDAL)
Three new HF resources sharing the OpenDAL Hf backend via _HfAccessor.
Each is a thin Config + Accessor (REPO_TYPE) + Resource wrapper that
reuses core/hf_buckets ops and the 57 hf_buckets shell commands
(expanded via resource=HF_RESOURCES list).
- ResourceName.HF_DATASETS / HF_MODELS / HF_SPACES
- Examples target the user-supplied repos:
AlienKevin/SWE-ZERO-12M-trajectories,
sapientinc/HRM-Text-1B,
HuggingFaceBio/carbon-demo
- Docs: setup + resource pages + matrix + docs.json entries
All reads are lazy: ls is one HTTP call, weights stream only on cat.
* feat(examples): add vfs + fuse examples for hf_datasets/models/spaces
Match the hf_buckets layout (basic + _vfs + _fuse) for each new resource.
Default repo_ids: AlienKevin/SWE-ZERO-12M-trajectories, sapientinc/HRM-Text-1B,
HuggingFaceBio/carbon-demo.
* fix(hf): restore find filter kwargs (name, type, maxdepth, size, mtime)
The OpenDAL migration in a3040fd dropped find's filter kwargs, so the
command layer's calls (name=, type=, maxdepth=, ...) raised
'unexpected keyword argument'. Restore the full filter set, applied
during op.scan() iteration. Synthesize directory entries from file
paths so 'find' matches POSIX semantics (was returning files only).
Affects all 4 HF resources (buckets/datasets/models/spaces) since
they share the same core implementation.
* feat(examples): expand hf_datasets/models/spaces to match hf_buckets depth
Each new resource example now exercises the full read-only command
surface: discovery (ls/tree/stat/find variants), read (cat/head/tail/
wc), search (grep/jq), pipelines, cd/pwd, barriers (&&, ||, ;),
quoting + $(), background jobs, PROVISION dry-run cost plans, and
streaming-chain backpressure. Verified end-to-end against the
user-supplied repos.
Highlights: model example provisions cat /m/model.safetensors as
~2.2GB but never reads it; a 128-byte head only transferred 8KB.
* types: add ResourceName.HF_BUCKETS
* accessor: add HfBucketsConfig + HfBucketsAccessor
* deps: add aioresponses dev dep for hf_buckets tests
* core/hf_buckets: aiohttp client + URL helpers
* core/hf_buckets: stat (HEAD + tree probe)
* core/hf_buckets: read_bytes, range_read, read_stream (aiohttp)
* core/hf_buckets: readdir + exists
* core/hf_buckets: find + glob
* core/hf_buckets: du + du_all
* ops/hf_buckets: read/readdir/stat
* commands/hf_buckets: phase 1 read-only commands + provisioning
* resource: HfBucketsResource (read-only) + registry entry
* commands/hf_buckets: phase 2 extended read-only commands
* examples: hf_buckets vfs + fuse
* examples: rename hf_buckets_{vfs,fuse}.py and add hf.py + style fixups
* examples(hf_buckets): add PROVISION section to hf.py
* deps: add huggingface_hub for hf_buckets writes
* core/hf_buckets: write_bytes, unlink, create (huggingface_hub backend)
* examples/hf_buckets: add write+remove demo
* core/hf_buckets: try/except optional huggingface_hub import at top of file
* examples/hf_buckets: rename hf*.py to hf_buckets*.py; fix flake8 line-length
* examples/hf_buckets: drop redundant env var checks
* core/hf_buckets: stat fast-path via index cache; don't follow resolve redirect
ls/getattr was hanging because each entry's stat hit /resolve/{key} with
allow_redirects=True, costing a roundtrip to cas-bridge.xethub.hf.co per
file (and 500ing on zero-byte files whose Xet hash is all-zeros).
Two changes mirror the S3 stat pattern:
- Consult the IndexCacheStore populated by readdir first; on hit, return
immediately with no network. On miss but parent listed, raise
FileNotFoundError without probing.
- On true cache miss, HEAD with allow_redirects=False and read size from
X-Linked-Size (and fingerprint from X-Xet-Hash) on the 302 itself.
* hf_buckets: register parquet/feather/orc/hdf5 read ops; route grep/rg/tail through generic
- ops/hf_buckets: split read.py into read/ package; add read_{parquet,feather,orc,hdf5}
loaded via try_load_command so FUSE/VFS cat of these files pretty-prints (matches s3 layout).
- commands/hf_buckets/grep, rg, tail: delegate to generic helpers introduced in #102.
Net -210 lines across the three wrappers.
* ci: verify hf_buckets extra is optional; docs: register HF Buckets
CI: new hf-buckets-optional-deps job installs base mirage without the
hf_buckets extra and asserts:
- huggingface_hub is absent
- mirage.resource.hf_buckets + core read modules import cleanly
- write/unlink module imports succeed (try/except guarded) but the sync
helpers raise ImportError with the install hint when called
Then installs the extra and asserts the top-level HfApi binding wires through.
Docs: new credential setup page (home/setup/hf_buckets), Python install
page (python/setup/hf_buckets), and resource usage page
(python/resource/hf_buckets); registered all three in docs.json and
added an HF Buckets row to the resource matrix.
* docs: drop python/setup/{r2,gcs,oci,hf_buckets}; align resource pages with s3
The python/setup pages duplicated the python/resource pages and were
linked via 'see Python X Setup' redirects that landed on a slim copy of
the same content. Removed all four files; merged the install detail
into each resource page (which already had the config table) and
retargeted back-links from home/setup and typescript/setup to the
canonical python/resource pages.
Restructured docs/python/resource/{r2,gcs,oci,supabase,hf_buckets}.mdx
to follow the s3.mdx layout: Config + Filesystem Layout + Cache +
Example + Shell Commands + Use Cases. For S3-compatible backends the
Shell Commands section links to s3 instead of duplicating the full
command tables. hf_buckets keeps a Scoping section since its config
exposes key_prefix; the others don't and were not previously
documented as supporting it.
* docs: inline full s3-style Shell Commands section into r2/gcs/oci/supabase/hf_buckets
Previously these pages linked to /python/resource/s3#shell-commands.
The user wanted each page to be self-contained, matching s3's depth.
R2/GCS/OCI/Supabase extend S3Resource so they share the full s3 command
surface (read, text processing, file ops, path utils, compression,
encoding, data formats, audio). hf_buckets gets the same set minus
Audio Support (no local_audio/hf_buckets registration).
* fix(commands): thread index into resolve_glob for SaaS backends
discord/github_ci/langfuse/mongodb/paperclip/postgres/telegram/linear/
trello wrappers called resolve_glob(accessor, paths) without the index,
so glob expansion bypassed the index cache. Pass index through (43 sites).
* refactor(resource): remove paperclip resource
A unified virtual filesystem for AI agents. Mount S3, Google Drive,
Slack, Gmail, GitHub, Linear, Notion, Postgres, MongoDB, SSH, and
more behind one filesystem so agents read, write, and pipe across
services with familiar shell commands.
Ships Python (mirage-ai) and TypeScript (@struktoai/mirage-*) SDKs,
a CLI, FUSE mounts, and adapters for OpenAI Agents SDK, Vercel AI
SDK, LangChain deepagents, Pydantic AI, CAMEL, OpenHands, Mastra,
Pi Coding Agent, plus FUSE-based integration with Claude Code and
Codex.
Apache 2.0 licensed.
Zecheng Zhang