main
99 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zecheng Zhang
|
98fcdcb2e2 |
feat(resource): assemble a TypeScript backend from one CommandIO table
python's GenericResource wires the whole generic command set, the glob resolver and the VFS/FUSE ops from one table, and TypeScript had every ingredient but no class that assembled them, so a custom backend there was still written out by hand. Adds GenericResource<A>, generic over the accessor so the table is checked against the core functions it holds rather than against Accessor. It keeps TypeScript's own wiring style: commands() and ops() return arrays instead of python's register() loop mutating state in the constructor. And there is no sdk.ts, because core's barrel is 99 lines and gated in both directions while the ./* exports map already makes every module importable, so the five names the new example and doc reach for are the only additions to it. Both classes gained sizes_always_known and supports_snapshot. Without the first, no user-written backend could be mounted on FSKit at all, since resolve_backend refuses a resource that cannot size its files. The two one-file examples answer one shared truth file, so the two SDKs cannot drift without a red build. docs/typescript/resource/new.mdx is the twin of the python page, whose stale du_total/du_all is corrected to du. The layout baseline drops to 255: resource/generic was one of the counted divergences. |
||
|
Zecheng Zhang
|
43e8a67180 |
Merge pull request #877 from strukto-ai/fix/os-verb-routing
fix(os): route every path-taking os verb through the workspace |
||
|
Zecheng Zhang
|
d1eea898b8 |
fix(os): keep the process patch off while a backend serves an op
A disk mount whose root sits at or under its own virtual prefix hands the host a path is_mounted answers True for, so the patched os module routed the backend's own physical path back into the same backend and the process wedged instead of raising. ops/host_io.py is the bypass: the two patched doors read it, and the two places a backend actually runs (execute_op, execute_cmd) plus the watch delta walk set it. Streams are wrapped, since a backend opens its file on the first __anext__. It is a process-global depth rather than a ContextVar because aiofiles reaches the host through loop.run_in_executor, which drops the context. os.readlink off a mount hands back the host's answer untouched, so a bytes path answers bytes rather than a str of them. One truth file now runs twice, the second time with -X utf8=0, the mode where pathlib passes io.open's "locale" sentinel. |
||
|
bytecii
|
a9c1aba03c |
refactor(browser): the resource layer inherits BaseResource instead of restating it
Every resource in `packages/node` (28/28) and `packages/core` (8/8) extends
`BaseResource`; in `packages/browser` none of the 19 did. They declared
`implements Resource` and hand-rolled the base's own members, which cost
behavior rather than style:
- `Workspace` hands a mount's index config to `resource.setIndex?.()`
(`workspace/workspace/workspace.ts`). `setIndex` exists only on
`BaseResource`, so that optional call silently no-opped: mounting a browser
backend with `index: {...}` ignored it -- no custom ttl, no Redis index --
where node and python honored it.
- each hand-rolled `close()` returned `Promise.resolve()` without closing the
index it had built, the exact leak `BaseResource.close` documents ("a mount
configured `index: {type: redis}` holds a client that nothing else closes").
- the eager `new RAMIndexCacheStore(...)` in every constructor built a store
even for resources that never read one; the base builds it lazily.
- `workspace/mount/storage.ts` carries a WeakMap serial-number fallback
written for this gap. It stays for third-party resources, but no
first-party browser resource depends on it now.
Two smaller browser-only gaps close alongside:
- `deltaHook` reached box, dropbox, gdrive, github and s3 in node but not in
browser, so watch/delta was silently absent there. The five core
`buildDeltaHook`s take only an accessor and import nothing runtime-specific.
Verified before wiring s3: the presigned-URL shim implements
`ListObjectsV2Command`, honors `ContinuationToken`, and returns
`IsTruncated`/`NextContinuationToken`, so the listing walk paginates.
- browser kept a `linear/config.ts` duplicating `core/linear/config.ts` field
for field, and inlined in `registry.ts` the rename maps core already exports
as `normalizeLinearConfig` / `normalizeGitHubConfig`. Both now come from
core, as node takes them, which also retires the restated
`LinearBrowserCtorConfig`. Python keeps linear's config in
`core/linear/config.py` with no `resource/linear/config.py`, so this matches
the python layout too.
Deliberately unchanged: discord, slack and notion configs are not drift.
Browser discord/slack take `{proxyUrl, getHeaders}` against core's `{token}`
because a browser cannot hold a bot token, and browser notion takes an MCP
`authProvider` where core takes an `apiKey`; browser s3's config is the
presigned variant. Repointing any of them at core would break browser auth.
`base_contract.test.ts` pins both halves: a sweep asserting every exported
`*Resource` inherits `BaseResource`, and a behavioral pair on the mount index
config. Against the pre-change tree it fails with `TypeError: r.setIndex is
not a function`.
Python needed no change -- all its resources already extend `BaseResource`
(the 14 s3 aliases inherit through `S3Resource`).
Layout baseline 257 -> 256: dropping browser's duplicate linear config closed
a divergence, and the gate fails on improvement too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
bytecii
|
1f4b1f5bfd |
refactor: collapse duplicated config, sanitizer, render and drive-op bodies
Three cleanup items from the round-2 duplication survey. No behavior
change except one py/TS divergence fix, called out below.
Runtime-agnostic config modules move into core. The seven
`resource/<name>/config.ts` modules under browser and node were
byte-identical to each other -- `diff` reported zero difference -- and
hold nothing runtime-specific, so CLAUDE.md puts them in core. The six
google ones land at `core/src/resource/<name>/config.ts`, where python
already keeps them and nine other backends already keep theirs;
`GitHubConfig` lands in `core/src/core/github/config.ts` next to
`GhConfig`, mirroring `python/mirage/core/github/config.py`, which
closes a layout divergence.
One sanitizer replaces five copies. `sanitize_title` (gdocs, gsheets,
gslides) and `_sanitize` (gmail, email) were the same function differing
only in the empty-input fallback and the length budget, so both become
arguments of `sanitize_label` in `utils/sanitize`, which already owned
the regexes and `sanitize_name`. The gzip stream helpers move to a new
`mirage/utils/compress.py` -- the twin of `utils/compress.ts`, another
closed layout divergence -- so gunzip stops carrying its own copy of
the decompressor. `jsonl_bytes_by_created_at` replaces linear's and
trello's `to_jsonl_bytes` on both sides, and databricks' two spellings
of `_is_directory` become one in `_helpers`.
onedrive and sharepoint were true name-only twins in exactly two files,
`exists` and `truncate`; both now come from `make_exists` /
`make_truncate` in msgraph, the module that already owns everything
else these two share. Their other sixteen core modules genuinely
differ in how they address items and are left alone.
Divergence fixed: the four typescript sanitizer copies used `/[^\w\s\-.]/g`,
and javascript's `\w` is ascii-only, so a CJK title rendered as a row of
underscores where python -- whose `\w` is unicode-aware -- kept it.
`utils/sanitize.ts` already spelled the class as `\p{L}\p{N}_`, so
adopting it moves typescript onto python's behavior. No test or integ
case pinned the old output; both sides now pin the new one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Zecheng Zhang
|
ca283c5c89 |
Merge pull request #860 from bytecii/fix/dropbox-list-folder-walk-cap
fix(dropbox): bound the emptiness probe's walk, not just its page |
||
|
bytecii
|
1a209f8e6c |
fix(dropbox): bound the emptiness probe's walk, not just its page
`list_folder` took a `limit` that it passed to /files/list_folder as that endpoint's page-size hint and then looped on `has_more`, accumulating every entry. So `limit` capped the page, not the walk. Four callers used it as a bounded emptiness probe and got the opposite of what they wanted -- one request per child of the folder to answer a yes/no: dropbox rmdir and rename, in both languages. Same defect and same fix as Google Drive in #854: the page hint keeps its own name (`page_size` / `pageSize`, still spelled `limit` on Dropbox's wire) and the new `limit` is the walk cap, breaking the continuation loop once that many entries are in hand. The four probe call sites are unchanged -- `limit=1` now denotes the bound they always meant. readdir and watch pass no cap, so they still walk to completion. The pin has to run the real `list_folder`, because the bug lives inside it: a fake substituted for it answers in one call whatever it is handed, so it cannot tell a bounded probe from an unbounded one, and asserting only on the recorded cap passes against the broken form. The fakes stand in for the transport instead, page the way the live API does, and record both the cap each caller asked for and the request count -- the count is what fails (3 vs 1) before the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
68c93c29b7 |
refactor(kits): K1 wave 2 — hf onto object_store, opfs onto the ops factory
Finishes the K1 wave the object_store/hierarchy kits opened in #848. - object_store driver gains `markers_supported` and makes `move_file`, `move_prefix` and `copy_file` optional. A store that cannot hold a directory marker now says so once instead of every factory guessing, and `make_rename`/`make_copy` refuse at build time rather than emitting an op that dies on the first call. - hf moves onto the kit: one driver (`resource: "hf"`) replaces 11 hand-written cores per language. The Hub refuses `create_dir` and has no server-side move or copy, so it declares `markers_supported=False` and leaves move/copy unset — mkdir becomes a no-op and rename/cp stay unwired (ENOTSUP), which is what they already did. - The write factories translate a driver's not-found into ENOENT on the PathSpec. Driver primitives speak keys, so the store's own error names a backend key; only the factory holds the path the user typed, and that is the only spelling a message may carry. - opfs ops move to `makeGenericOps`, deleting 22 hand-written files. `mkdirParents` is set because OPFS resolves one directory handle per segment. - Five python ops tables passed `emulate_truncate=True` to a table that already has a native truncate, so the flag never fired; TypeScript never had them. Dropped (dropbox's is real and stays). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
5d6f49bd06 | feat(runtime): add the ssh sandbox provider | ||
|
bytecii
|
dd14575d3e |
feat(ts): K1 object_store + K2 hierarchy kits — s3/gridfs + jaeger/langfuse/mem0 exemplars
Mirrors the Python kits: core/object_store (ObjectStoreDriver + 14 op factories, s3/gridfs drivers, shared object_store command overrides via requireOp) and core/hierarchy (Codec/Route/match_route, readdir/stat/read factories) with jaeger, langfuse and mem0 migrated. mem0 reshaped to python stems (scope/readdir/stat/read/client/search); gridfs du/walk.ts absorbed into the driver; langfuse grep/rg share the route table via SEARCH_KINDS, giving TS rg the search pushdown py already had. Layout parity baseline ratchets 269 -> 260. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
bytecii
|
9164339fb3 |
chore: lower the layout-parity baseline to 268
core/generic/find.py now mirrors core/generic/find.ts, so the gate's divergence count fell by one. The ratchet fails on a drop as well as a rise, so the improvement has to be locked in rather than spent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
0e749345e5 |
refactor(layout): items 33+35 — module homes and workspace/config placement
Block E items 33 (T3-3 module homes) and 35 (T3-2 workspace/config
placement) from the cleanup plan. Item 34 (T3-1 runtime layout) was in
scope but is already closed on main; the verification is in the PR body.
Layout parity gate: 286 -> 269 unexcused divergences, baseline lowered.
Item 33 — module homes:
- py utils/fingerprint.py -> watch/fingerprint.py (all six consumers
are watch machinery; TS is already at watch/fingerprint.ts)
- py workspace/provision/rollup.py -> provision/rollup.py (no workspace
imports; TS is at the leaf provision/rollup.ts)
- py workspace/provision/builtins.py deleted (one function, one caller;
TS keeps it module-level in provision_node.ts)
- py 9 snapshot key enums -> new workspace/snapshot/keys.py
- py IndexType -> cache/index/config.py, CacheType -> cache/file/config.py
- ts node/cache/redis/file.ts -> node/cache/file/redis.ts, taking
add.lua with it (read relatively by its only consumer and copied by
an explicit tsup path, so tsup.config.ts moves with them)
- ts new core/src/concurrency/limiter.ts, replacing the private
Semaphore in accessor/dify.ts
NodeMetaKey deliberately stays beside NodeMeta: it names NodeMeta's own
fields, and importing the snapshot package from mount/namespace closes a
real cycle (namespace -> snapshot.keys -> snapshot/__init__ -> api ->
state -> namespace).
Item 35 — workspace/config placement:
- ts core/workspace/workspace.ts -> core/workspace/workspace/workspace.ts
- ts node/workspace/mount_spec.ts -> core/workspace/mount/spec.ts
- ts new core/workspace/workspace/guard.ts (three call sites converged
onto it from their own typeof === 'string' tests)
- ts new node/workspace/workspace/kernel_mounts.ts (NodeWorkspace
211 -> 169 lines, now delegations)
- ts server/src/config.ts -> node/src/config.ts, exported from the node
barrel so a mirage-node consumer can load a YAML workspace config
Three bugs fixed in the dify semaphore on the way, each pinned by a test
verified to fail against the old code: release() returned the permit to
the pool before waking the waiter (two holders at capacity 1), double
release inflated the count, and maxConcurrency < 1 was silently clamped
where python raises.
Also drops `cache: pnpm` from the ts-audit job added in #831. That job
runs `pnpm audit` and never installs, so the pnpm store does not exist;
setup-node's post-run save then fails path validation on every cache
miss and silently skips on a hit, which is why it was green on main and
red here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Zecheng Zhang
|
dcbc668c15 |
Merge pull request #834 from bytecii/feat/sandbox-runtimes
feat(runtime): smolvm microVM provider (py+ts) and confined sandlock python runtime (py) |
||
|
bytecii
|
15a8dbe043 |
refactor(sandlock): split into a package, excuse the layout divergence, document both runtimes
Codex P1: a python-only module under runtime/python left check_layout_parity.py --strict red (297 against a 296 baseline). Making sandlock a package the way monty already is moves the divergence from a module to a directory, which the exceptions file excuses as a subtree -- the honest shape here, since there is no TypeScript counterpart to mirror and growth inside it is expected rather than drift. Constants and config move out of the runtime module to match monty's and smolvm's layout: constants.py carries the env var, the CLI hint, and the system read set; config.py carries SandlockConfig. Docs: smolvm joins the sandbox provider table and gets its own section on both language pages, covering the refused-state probe and the fact that one machine is one guest, so concurrent lines share a filesystem and a process table. sandlock joins the python runtime table with a section on what "process" reach does and does not buy, and on the environment it deliberately does not inherit. Vendor marks added under the existing docs/images/<name>-logo.svg convention. |
||
|
bytecii
|
44b430ef35 |
Merge remote-tracking branch 'upstream/main' into cleanup/items-31-32
# Conflicts: # spec/layout_exceptions.json |
||
|
bytecii
|
73f5bd7c21 |
chore: cleanup items 31+32 — nested-function rule, layout, CI audit, ls facets
Closes the two remaining small items of Block E. Both were largely recon: much of what they filed had already been fixed, and the part that had not turned up a real GNU divergence. Item 31 (T3-7), rule/doc reconciliation --------------------------------------- The `object`-parameter half is already done: #825's mypy inversion took the package from the filed 28 down to 7, of which 3 are docstring prose and the remaining 4 are protocol methods (`__contains__`, `pop`) whose signatures typeshed dictates, all already listed in the no-object gate. `utils/errors.py` reads `str | PathSpec` today, and `builders/sed.py` no longer holds flag values at all -- item 22 moved that to the generic. That left the nested-function rule, which the plan asked to decide before gating. Measured: 39 nested defs, and **every one of them closes over its enclosing scope** -- 38 by free variable, and `sed_helper._repl` by binding through parameter defaults, the loop-variable idiom, with a comment saying so. A flat "do not nest" is a rule the architecture cannot keep (op factories, provision builders, the read-through cache, every decorator's wrapper), so gating it as written would have meant 39 standing violations. So the rule now says what it was reaching for: a nested def must capture the scope around it, or it belongs at module level. That is mechanical and enforceable -- `tests/test_nested_functions_are_closures.py` reads free variables from `symtable` and parameter defaults from the AST. It passes on the tree as-is and fires on a helper that reads only its own arguments. Layout: the eight test-only TS directories are flattened (`ram/{cat,cut,grep,head,ls,tail,wc}/`, `ssh/ls/`), `awk_helper.ts` moves up beside its nine siblings, and the `provision.ts` / `_provision.ts` split is settled on Python's convention -- `_provision` for a backend's own, bare `provision` for the shared ones (cli, generic_bind). That last one was not cosmetic: Python already had `_provision.py` for github, gmail, redis and email, so renaming **closed 4 real parity divergences** (layout baseline 296 -> 292). The `findEval`/`findParse` line was already done in #827. Item 32 (T3-8), test/CI tidiness -------------------------------- The two asymmetric `ls` conformance matrices are raised to symmetry, and they pass: python `[ram,disk,redis]` vs typescript `[ram]` was stale caution, not a divergence. Both runners now reject an asymmetric matrix at load time unless the case carries a `divergence` key explaining why -- a case is a parity claim, and narrowing one side reads as coverage while the side that still lists the backend goes green. Loading the corpus under the new assertion proves no other case was asymmetric. The README documented the override as "not yet needed"; it exists now. Adds the `ts-audit` job, mirroring `test_python.yml`'s exactly (same `continue-on-error`, same out-of-gate placement). Nothing ran `pnpm audit` for the TS tree while `typescript/package.json` carried 27 hand-written CVE overrides that only a person remembering to check kept current. Integ facets for `ls`: `-A`, `-d`, `-r`, `-S`, `-h`, all pinned against GNU coreutils 9.7 in docker. `-S` uses regular files only, because GNU sorts a directory by its inode size while mirage counts it as 0 -- a divergence CLAUDE.md documents deliberately. The `-h` facet found a real bug ------------------------------- `ls -h` disagreed with GNU three ways, and because the flag had zero cross-backend coverage nothing caught it. GNU prints a count below one unit with no suffix (`24`); mirage printed `24B`. GNU rounds *up* to the precision shown (1025 bytes is `1.1K`); mirage gave `1.0K`. GNU drops the decimal once the value reaches ten (`10K`); mirage gave `10.0K`. One shared engine replaces both formatters in both languages -- GNU runs `-h` and `-H` through one `human_readable`, and so do we now. Rounding up can carry past the base (1048575 ceils to 1024K, which GNU shows as `1.0M`), so the unit is re-chosen after rounding rather than once up front. 21 GNU-read points are pinned as a table in each language. Blast radius, all re-pinned: five integ cases that had recorded the `B` suffix (du, discord, langfuse, email, gmail), and the du fan-out tests. Those last ones needed care rather than a new number: they exist to prove the total is humanized once instead of twice, and their 1500+1500 stops discriminating under correct rounding, since 3000 bytes and two `1.5K` readings both render `3.0K`. They now use 1025+1025, where single-rounding gives `2.1K` and double-rounding would give `2.2K`. Verified: pre-commit clean, integ 8054/0 on ram+disk+redis in both languages, conformance green in both, and the layout, spec, barrel, docs, case-target and PathSpec gates all pass. Not covered, still open on item 32: integ facets for `ls -t` (mtime order is not stable across backends without a seeded fixture), mktemp, gzip, the checksum `--check` companions, `df -H/-k/-a`, and `cmp -n/-b/-i`; and the T2-7 helper unit-suite mirroring. Item 31 leaves the provision *presence* sets unreconciled (Python has lancedb/qdrant, TS has trello) -- that is a question about which backends should provision at all, not about naming. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
0a5cc8dbfc |
Merge upstream/main into refactor/survivor-renames
Conflicts, both from ratchets/config the two branches moved independently: - python/pyproject.toml: #825 inverted the mypy config (global strict + an opt-out list of not-yet-annotated modules) and deleted the old opt-in allowlist my last commit had edited. Took upstream's file wholesale -- github.pushdown now gets strict from the global default and passes it, so the retarget commit is subsumed. Retargeted the one stale entry the rename left in the NEW list: generic.rm_command -> generic.rm_cmd (the module still needs its exemption: one no-any-return at rm_cmd.py:89), keeping its exempt status exactly as it was under the old name. - spec/layout_exceptions.json: both sides lowered the baseline from 299 (mine to 293, upstream's to 296). Neither value is right after the merge; ran the gate, which reports 290 -- the two closure sets are disjoint (299 - 6 - 3). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
8e5d9a9c5b |
refactor(naming): command-layer consistency — narrow→pushdown, findEval/findParse→snake_case, history_cmd→history, rm_command→rm_cmd
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
bytecii
|
31b91ac885 |
Merge remote-tracking branch 'upstream/main' into feat/mypy-strict-and-py-mcp
# Conflicts: # python/mirage/commands/builtin/generic/tar/tar.py # python/mirage/commands/builtin/generic/unzip.py |
||
|
bytecii
|
1f59264232 |
refactor(naming): survivor renames — github pushdown, gcal/msgraph _client, sharepoint resolve, mem0 detect_scope (plan §3)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
6b6c8e51bc |
Merge pull request #821 from strukto-ai/fix/tar-grep-exec-gaps
fix(commands): tar/unzip member selectors and relay extraction, grep file filters, direct path execution |
||
|
bytecii
|
3293cb5ff4 |
feat(py): mirage mcp over stdio, and invert the mypy allowlist
Two cleanup-plan items, both Python-side.
Item 29 -- `mirage mcp`. TypeScript shipped a six-tool stdio MCP server;
Python had none, so a pip-install user could not point Cursor or Claude
Desktop at a workspace and `mirage --help` differed by distribution.
Adding the entry point alone would have duplicated the tools, because
this side kept them private inside the Claude Agent SDK integration, so
the shared layer comes first:
- agents/tool_descriptions.py -- the six strings, one copy.
- agents/tool_operations.py -- MirageToolOperations, lifted out of the
SDK server's private _MirageTools.
- agents/file_version.py -- stale-write protection, which this side
lacked entirely. TS stamps stored bytes; here the stamp covers the
rendered bytes, because this read tool has always rendered and an
edit must search what the agent was actually shown.
- agents/mcp/server.py + cli/mcp.py -- the server and `mirage mcp`.
- server/workspace_config.py -- config discovery (candidates, env
names, walk up from cwd), which Python had nowhere, so every entry
point had to be handed an explicit path.
The server is the low-level MCP Server rather than FastMCP: FastMCP does
not forward a version, and TS advertises one. Handlers are bound methods,
not decorated closures, so nothing nests.
Item 28 -- the mypy allowlist. 54 modules opted *in* to annotation
checking against 1826, so the default was unchecked and every new file
joined the unchecked side. The default is now strict, with a list of
what is not yet annotated that only shrinks. 166 annotations cleared
along the way; the remainder is named module by module.
Two real defects surfaced by the annotations, neither of them typing:
- Workspace._original_open / _original_os were invented by assignment
in lifecycle.patch_process, so unpatch without a patch raised
AttributeError. Declared, and the restore is guarded.
- sed_generic declared a non-optional writer while its own docstring
and its `write_bytes is None` branch said otherwise; the builder
passes None whenever the backend cannot write.
Tests keep the PathSpec rule instead of full strict: measured, full
strict on python/tests is 2374 errors, of which 634 are `str` where a
pydantic field declares SecretStr -- which pydantic coerces at runtime --
and most of the rest is the monkeypatched-fake pattern CLAUDE.md
sanctions. The rule that is violated for real is PathSpec, 19 times, and
scripts/check_test_pathspec.py now holds that line. One of the 19 was a
latent AttributeError: tests/e2e passes a str to s3 write_bytes, which
reads .mount_path, and the test skips without a live versioned bucket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Zecheng Zhang
|
2ba312572b |
fix: close ten agent-corpus defects in sed, awk, date, find and the fakes
sed -i writes for every command, not only s and d; multi-file output is concatenated raw; an escaped delimiter inside an address regex no longer cuts the address short, and \%re% custom delimiters parse. awk refuses constructs it cannot run (exit 2, "unsupported construct") instead of echoing their source; simple assignment and OFS now execute and unset names render empty. date -d parses ISO, @epoch and the gnulib relative grammar in both languages (TS printed NaN with exit 0), invalid input is exit 1 with GNU wording; TS strftime learns %F %c %C %g %G %V %U %W %h %k %l %n %P %q %r %R %t %x %X, and Python expands %q ahead of strftime. find -printf is implemented (%p %P %f %h %d %s %y %Y %m %M %T escapes), registered on the CommandSpec so the format is not read as a path, and stats through the dispatcher when no overlay stat is wired. Fakes: the GitHub server strips the internal files list from list-commits, accepts refs/heads/x on /contents and /commits, and reports files as objects on one commit; the GWS Drive server evaluates fullText contains and answers 400 for an unknown query field; the Notion server honors after on block append with the live API's response and validation shape. Integ: new unix cases for each command fix, awk_ofs golden corrected, and cli-gh/cli-gws/cli-ntn cases that drive every fake fix through the CLIs; ntn conformance re-pinned against the real binary. |
||
|
Zecheng Zhang
|
3e614781a6 | fix(commands): tar/unzip member selectors and relay extraction, grep file filters, direct path execution | ||
|
Zecheng Zhang
|
bc10dba0ca |
fix(spec): record the read_range slot dify and ram gained
The spec is regenerated by a CI step of its own, not by a pre-commit hook, so adding the slot to those two command_io tables left the committed inventory a commit behind with nothing local to say so. |
||
|
Zecheng Zhang
|
88698fa08b |
feat(ops): optional byte-range reads across backends, and drop github_ci
Adds an optional read_range slot to the op table. Backends that can fetch a window do; the generic read op falls back to read-and-slice for the rest, so no backend has to implement one. Wired in both languages for box, databricks, dify, discord, disk, dropbox, gdrive, gridfs, hf, nextcloud, onedrive, opfs, ram, redis, s3, sharepoint, slack and ssh. Shared helpers live in utils/ranges (range_header for a raw push-down, slice_window for rendered content, is_unsatisfiable_range to normalise a 416). Backends that render their bytes take the window right after building them, which is why dify and ram are on the native list too: a windowed read is answered the same way everywhere, whatever is behind the mount. Fixes a crash on hf and nextcloud: OpenDAL's reader seeks rather than sending a header, so a window past EOF raised from the seek instead of surfacing a 416. Both now read as empty. Caught by the integ battery, not by unit tests. Removes the github_ci resource, its commands, ops, docs and examples. Integ: a shared ranges/read.json across 21 targets, plus per-backend cases for slack, discord and dify, reached through ws.dispatch since no shell command asks for a window. |
||
|
Zecheng Zhang
|
5cfb4598b8 |
feat(ts): install a CLI from a file reference in yaml (#798)
* feat(ts): install a CLI from a file reference in yaml
`clis: {tally: {cli: ./tally.mjs:TALLY}}` now resolves in TypeScript, so
a deployment can ship its own program tree without a host program
calling registerCli. Mirrors the `":" in name` branch of Python's
`cli_spec_for`, one layer up: `cliSpecFor` lives in core, which has no
filesystem and is synchronous, so the ref resolves in the config layer.
The referenced file is imported by Node, so .mjs, .js and .ts all work;
a .ts spec rides Node's strip-only type stripping and is refused with a
hint when it uses a construct that needs codegen.
* fix(ts): node's specifier rule for cli refs, and a real spec check
Both from review.
A bare specifier carries slashes in Node (`@scope/pkg`, `pkg/subpath`),
so copying Python's `"/" in source` test read every scoped or subpath
package as a file, rebased it under the config directory and reported it
missing. A path is now what Node calls a path: relative or absolute,
plus a bare filename carrying a module suffix.
A class name is not proof a tree is usable. Dispatch reads subcommands,
aliases and options at every level, so a value that only answers to the
name crashed on the first line an agent typed instead of failing the
create. The shape is checked recursively instead.
|
||
|
bytecii
|
ebec2d6926 |
refactor(ts): four shared factories mirroring the Python originals (#609 item 24 / T2-9) (#792)
* refactor(ts): four shared factories mirroring the Python originals (#609 item 24 / T2-9) TypeScript had grown four copy-drift sites where Python already has one shared module. Each is extracted onto the Python shape, so the two trees line up module-for-module. builtins/shared.ts (twin of workspace/executor/builtins/shared.py). Three independent errorResult()s with three different signatures — capacity.ts hardcoded the 'df' command, links.ts hardcoded exit 1, only metadata.ts took both — so a GNU exit-code fix could not be shared. They collapse onto result/ok/fail/finish, plus operandText/absPath/splitFlags/splitValueFlags/ expandOperands. Drops links.ts's re-implemented abs+splitFlags and capacity.ts's import of splitValueFlags from metadata.ts (df reaching into chmod/touch). `Result` moves here from scope.ts, where Python keeps it in shared.py; the other 12 builtins change only their import line. The chmod/chown/chgrp/touch handlers each tracked an exitCode variable that was only ever set to 1 next to an errors.push, so finish() derives it from `errors` exactly as Python does and the variable is gone. workspace/executor/interpreter.ts. python/handle.ts and js/handle.ts were the same 130 lines apart from the label, the unavailable-error class, the payload flag, Python's -m guard and -x, and js's .mjs module flag. makeInterpreterHandler takes those as a spec; both handle.ts keep their path, their exported name and their signature, so callers and runtime.test.ts's dynamic import are untouched. The .mjs check moves into the js wrapper, which is where Python's js.py keeps it. Kept in workspace/executor rather than beside commands/builtin/general/ interpreter.ts because no commands/ module imports workspace/ today and this should not be the first. makeRm. generic/rm_command.ts already existed and gcal already used it; the gdocs/gsheets/gslides bodies were verified character-identical to it modulo indentation, so the three 79-line files become 20-line bindings. core/google/tree_ops.ts (twin of core/google/tree_ops.py). makeStat and makeUnlink over a backend's readdir; the six modules stay as real files exporting `stat`/`unlink`, so the barrel and the provision/io importers do not move. The three local eisdir() copies give way to the shared one in utils/errors.ts: isFsError keys on `code`, which is unchanged, and its message (the bare virtual path) is what Python's IsADirectoryError(raw) already produced. No behavior change intended. -995/+147 across the 26 edited files. * fix(js): honor the explicit `-` stdin operand, like python3 and node `js - a b` read a file literally named `-` and failed with "js: /-: No such file", while `python3 - a b` on the same tree read the program from stdin. Python has no such split: resolve_source's STDIN_OPERAND branch (interpreter.py) is unconditional, so both interpreters honor the operand, and js.py inherits it by taking the default Argv0Rules. Real `node -` agrees — it reads the program from stdin and keeps the following words as argv. Verified against Python by calling resolve_source directly for the js call shape: `js - a b` yields mode=stdin, args=['a', 'b'], no script path. Both quickjs hosts then build scriptArgs the same way (an absent prog leaves the args alone), so the operand lands identically. Found while extracting the shared interpreter handler in the previous commit; kept separate because it is a behavior change, not a move. The unit test fails without this fix (exit 1, not 0). The integ case pins py=ts. * test(ts): mirror the Python suites for the two new shared modules Python has tests/workspace/executor/builtins/test_shared.py and tests/core/google/test_tree_ops.py; TypeScript had neither twin, and both modules are now shared chokepoints rather than per-command code. shared.test.ts ports test_shared.py case for case: the result triple (ok/fail/finish, including finish keeping a carried IOResult's writes and deriving the exit code from `errors`), operandText/absPath, the three splitFlags cases, the three splitValueFlags cases, and the expandOperands glob expansion over a real RAM mount. tree_ops.test.ts covers makeStat over a stub readdir: the synthetic owned/shared roots answer as directories without a readdir round trip, an indexed document is served straight from the index, a miss warms the parent and then raises ENOENT, and a call with no index raises ENOENT without reaching the backend. It deliberately does not port test_tree_ops.py's one case — that a readdir raising a non-fs error propagates out of stat. TypeScript swallows it in a bare catch and reports ENOENT instead, which is a real pre-existing divergence in the T1-G "honest errors" family; pinning current TS behavior here would entrench it. Filed separately. * refactor(ts): keep the interpreter opt interfaces module-private Nothing outside interpreter.ts imports them — both handle.ts wrappers pass object literals — so knip reported them as unused exports. * fix(google): only an absent parent may collapse into ENOENT (py+ts) Every Drive-family stat/read/unlink warms a cold index by listing the parent, then retries. Python narrows that call to FileNotFoundError, so an auth failure or a dropped connection reaches the caller; TypeScript swallowed it in a bare `catch {}` and reported "No such file" instead. Five TS sites did this (google/tree_ops stat+unlink, gdrive stat, gdrive read, gdocs/gsheets/gslides read) — seven catch blocks in all. tree_ops.unlink was wrong in the other direction: Python called readdir completely unguarded, so removing a path under a missing directory reported the parent rather than the operand. GNU names the operand: $ rm nodir/file.txt rm: cannot remove 'nodir/file.txt': No such file or directory (pinned against coreutils in debian:stable-slim). Both sides now use the shape Python's make_stat already had: swallow a not-found parent, let everything else through. TypeScript spells it with the existing isEnoent(), the twin of `except FileNotFoundError`. gdrive/readdir warms a parent unguarded in both languages — already aligned, left alone. Tests fail without the fix. Two of them mirror Python suites that had pinned this since before the TS port (test_stat_propagates_parent_ refresh_failure, test_read_propagates_parent_refresh_failure); the tree_ops pair is new on both sides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(cache): one entry_or_warm for every cold-index lookup (py+ts) Twelve call sites across eight backends had grown their own copy of "ask the index, list the parent if cold, retry, else ENOENT", each with its own idea of what a failed listing means. They now share one function in cache/index/warm, so exactly one catch in the tree decides that. The previous commit narrowed seven of those catches by hand; this makes the narrowing structural instead, and sweeps the five that commit did not reach: box/read (x2) bare catch, py narrowed -> divergence gmail/read bare catch, py narrowed -> divergence dropbox/read (x2) bare catch, py caught (FileNotFoundError, DropboxApiError) -- but readdir already turns the API's 409 for a missing path into ENOENT, so that second arm only ever swallowed live API failures. Python had the same bug here, from the other side. gdrive/readdir warms a parent unguarded in BOTH languages; already aligned, deliberately left alone. Also folds eight private eisdir() copies into the shared helper. The copies built `EISDIR: ${p}` and never stamped virtualPath, and execute.ts renders errorVirtualPath(err) straight into stderr, so a directory read printed cat: EISDIR: /docs/owned: Is a directory where Python printed the GNU form. Two TS tests asserted on that message with /EISDIR/ and so had been holding the bug in place; they now assert the stamped code and virtualPath, which is what Python's IsADirectoryError(virtual) carries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(integ): pin the Drive-family cold-lookup error paths on both hosts Five cases over the gapps and gdrive targets, all curated by running the battery and reading the real output rather than guessing the strings: gapp_cat_dir_is_gnu cat /docs/owned -> Is a directory gapp_cat_missing_parent cat .../nope/x.gdoc.json -> names the operand gapp_ls_missing_parent ls .../nope -> ENOENT, exit 2 gapp_rm_missing_parent rm .../nope/x.gdoc.json -> names the operand gdrive_cat_dir_is_gnu cat /data/sub -> Is a directory clear_cache forces the cold index, which is the branch entry_or_warm now owns. Both wordings match GNU pinned in debian:stable-slim (`cat: <path>: Is a directory`, exit 1). Two things these cases do NOT prove, so nobody reads more into them: - rm builds its own diagnostic from p.rawPath, so it reports the operand whatever the backend error carries. The unlink operand-naming fix is pinned by test_tree_ops.py, not here. - `cat` on a directory is answered by the read generic off stat, so the eisdir consolidation is pinned by the gdrive/dropbox unit tests (which assert virtualPath, the field errorVirtualPath reads), not by these. They sit at seq 930030+ deliberately: at 520100 their clear_cache wiped the index that gdocs_prov_cat (930023) expects warm, and that case started failing on hits=1 vs hits=0. Verified: gapps + gdrive, 2193 passed / 0 failed, on the typescript-node host and again on the python host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * style: formatters over the warm-helper sweep, + the py eisdir message pin isort regroups the hand-inserted `partial` / `entry_or_warm` imports and prettier rewraps one dropbox thunk; no behavior change. Also completes the eisdir work on the Python side: the gdrive read test asserted only the exception type, so it would not have caught the "EISDIR: " prefix the deleted TS copies were adding. It now asserts the message is the bare operand, which is the string the TS twin pins through virtualPath. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(gate): lower the layout-parity baseline to 301 The ratchet fails on a drop as well as a rise, so a closed divergence has to be locked in. Net -1 from three real movements: -1 core/google: python-only ['tree_ops'] tree_ops.ts now exists -1 workspace/executor/builtins: python-only ['shared'] shared.ts now exists +1 workspace/executor: typescript-only [... 'interpreter'] The +1 is the deliberate layering call from this PR: the shared interpreter handler lives under workspace/ rather than beside Python's twin in commands/builtin/general/, because no commands/ module imports workspace/ today and this should not be the first. Recording it here so the net -1 does not hide that one improvement paid for one new divergence. cache/index/warm does not appear: it landed on both sides, so it is neutral. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(spec): js/node end option parsing at the first operand, like python `node - -e x` ran `x` instead of the piped program, and `node s.js -m` flipped the interpreter into module mode instead of handing s.js its own -m. Both real interpreters disagree, pinned locally: $ echo 'console.log(process.argv.slice(1))' | node - -e 'nope' [ '-', '-e', 'nope' ] $ echo 'import sys; print(sys.argv)' | python3 - -c 'nope' ['-', '-c', 'nope'] (node 22.8.0, CPython 3.13) `Operand.remainder` (argparse nargs=REMAINDER) already exists and both parsers already honor it in one line — the first operand ends option parsing, so every word after it is an operand verbatim. `python`/`python3` have carried it all along; js/node simply never got it, in EITHER language, so this is a symmetric py+ts spec fix rather than a TS-only one. Verified by calling parse_command directly: js/node/python/python3 now agree on `- -e PROG` and on `s.js -m --module`, while a flag BEFORE the first operand is still the interpreter's (`js -m -e CODE a`). One consequence beyond the reported case: remainder also turns off lenient_dash_operands, so `js -Z foo` now reports an invalid option instead of silently treating `-Z` as the script path (which produced "js: /-Z: No such file"). Real node agrees — `node -Z foo` prints "node: bad option: -Z". Nothing in integ or the suites relied on the old leniency. Reported by codex on #792. Its premise was wrong in one respect — it said to mark the rest operand "as the Python specs do", but Python's js/node spec is Operand(type="str") with no remainder, so Python had the same bug. Fixing TS alone would have created a divergence. Tests: 4 parametrized parser cases per language over the real js/node/ python/python3 specs, plus a quickjs end-to-end case. All 4 py and all 5 TS fail without the spec change (python/python3 pass either way, which is what makes them the control). integ/runtime/quickjs.json `stdin_operand` gains the -e and -m steps. Verified: pytest 14159 passed / 0 failed; pnpm -r typecheck clean; core suite 8129 passed (the one python_repl timeout is the known Pyodide-boot-under-load flake, 10/10 alone); runtime/run.ts quickjs 11 passed / 0 failed on the typescript host; spec drift, spec parity (93 match), layout parity (301 = baseline) and width table all clean. The python host's quickjs suite needs the MIRAGE_QUICKJS_HOME WASI blob that CI downloads, so that leg is covered by test_integ.yml rather than locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(quickjs): end qjs's own option parsing before the program's argv The python host's quickjs runtime appends the program's arguments straight onto the qjs command line: argv += ["-e", args.code, *named, *args.args] Unlike CPython's -c, qjs's -e does NOT end option parsing, so any program argument that spells a qjs switch was read as one. Pinned against the build CI uses (quickjs-ng v0.15.1 qjs.c): the option loop runs while `*argv[optind] == '-'`, breaks only on a bare `--`, and scriptArgs start at `argv + optind`. That is what made integ-runtime red on the previous commit, and only on the python host: node - -e 'console.log("flag")' -> flag (second -e wins) node - -m -> (empty) (module mode flipped) Both are now `--`-separated, so they land in scriptArgs where the parser already put them. qjs consumes the `--` itself, and the -e branch still wins over any filename, so a named program keeps scriptArgs[0]. Surfaced by the remainder fix in the previous commit rather than caused by it: before that, mirage's own parser swallowed those words, so they never reached the runtime. Any program argument spelling a qjs switch had this hole -- `node script.js -m` included. TypeScript needs no twin. Its quickjs runtime is in-process and assigns scriptArgs as a global (`quickjs.ts:387`), never building a qjs command line, which is why the TS host passed the same integ steps. Checked the other python runtimes too: wasi.py and local.py both go through CPython `-c`, which DOES end option parsing (pinned: `python3 -c prog -e FLAG -m` puts `-e FLAG -m` in sys.argv), and monty passes argv as a data global. quickjs was the only one. Verified with the real engine, after downloading the pinned qjs-wasi.wasm: `runtime/run.py quickjs` on the python host is 11 passed / 0 failed with this change and reproduces CI's exact two-line failure without it; the whole python runtime battery is 99 passed / 0 failed (bridge/docker/wasi skip on unmet env). pytest 14192 passed / 0 failed -- 33 more than the previous run, because the @live quickjs tests now execute instead of skipping. The three new unit tests pin the argv shape without needing the build, and all three fail without this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: bytecii <bytecii@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
4c421247ea |
fix(find): ! is negation, not a start point
GNU spells find's negation two ways and mirage only understood one.
`find /data -name '*.txt' ! -empty` exited 1 with `unknown predicate
'/!'`, because find's spec declares `rest=Operand(type="path")`, so
every word without a leading dash is classified as a start point and
`!` was cwd-prefixed into the path `/!`. `-not` worked, and so did
`\( ... \)`, because the parens were already exempted through
`ignore_tokens`; `!` was simply missing from that set. Nothing
downstream needed teaching -- `!` has always been in the expression
parser's operator table.
Fixing the classification made a second bug reachable: a line that
ends on an operator. `find /data !` used to print the whole tree and
then a bogus "No such file or directory" for `/!`; with the operand
gone it would have negated nothing in silence. Both parsers now refuse
it in GNU's words, and the check runs where the operator is consumed
rather than in the primary, because by then the token that needed a
right-hand side is no longer on the line:
find: expected an expression after '!'
find: expected an expression between '!' and ')'
Two py/ts divergences surfaced on the way and are closed here:
- The TypeScript parser skipped an ignore token silently, leaving its
`wordKinds` slot null, which means "guess from the shape" -- and the
shape of a grammar token says nothing about it. Python stated TEXT.
TypeScript states it now too.
- `spec_hints.ts` carried a by-value override that re-nulled ignore
tokens after the fact. It was redundant once the parser states the
kind, and it was position-blind: it matched an option's value as
readily as a grammar token, so `find /d -name '!'` lost the TEXT the
parser had correctly given the pattern. Deleted.
The new cases also caught a live bug in the Nextcloud backend, fixed
here. Its find pushes the predicate tree down as a WebDAV SEARCH, and
Nextcloud inverts a <d:not> by swapping the comparison inside it for
its opposite, so a <d:not> around anything but a single comparison
raises server-side:
Binary operators inside "not" is not supported (HTTP 500)
Probed against nextcloud:30-apache (30.0.17), the image CI runs:
not(like) and not(is-collection) answer 207, while not(or(..)),
not(and(..)) and not(not(..)) all 500. The compiler now tracks whether
a compiled condition is a bare comparison and refuses to negate one
that is not, which falls that find back to the scan walk, which has no
such limit. `-not` reached this first and has been answering 500 all
along: `find /data -not -type f` compiles to not(not(is-collection)),
unnoticed because find_not_name was the battery's only negation case
and negates a bare -name. find_bang_type_f now pins it.
Pinned against GNU findutils 4.10.0 on debian:stable-slim as a
three-way byte-identical differential (GNU vs python vs typescript)
over 25 output cases and 15 error cases on one shared seed tree. One
known divergence is normalized and commented: GNU quotes as `x' where
mirage quotes 'x' repo-wide, which is pre-existing and already pinned
in integ/unix/find/error.json. Also left alone as pre-existing: GNU's
wording for a binary operator with nothing before it, the
paren-specific messages, and "paths must precede expression".
integ gains five output cases in find/not.json and three error cases
in find/error.json. The output cases carry seq 295, the same seq as
find_not_name whose expected output they derive from: the battery runs
one shared session ordered by seq, and every one of the eight fixtures
that would otherwise intrude (/data/fs at 302, /data/dr at 317,
/data/rgm at 459, /data/arch at 467, /data/xm2ctree at 755, /data/g4
at 500200, /data/f4 at 500300, /data/roots1 at 785) is created after
295 and never cleaned up. The one case already at 295,
find_empty_dir_holding_only_a_link, removes its own tree in the same
line, which is what leaves the slot usable.
chroma and github each gain the `!` twin of the `-not` case added in
#772, placed immediately after it in both file order and seq
(chroma_find_empty_composes_with_name -> 630056,
github_find_empty_composes_with_name -> 530014). Both verified against
a locally run fake on both hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Thomas Hart
|
20543fb031 |
feat(du): Honor -S/--separate-dirs (py + ts) (#722)
* feat(du): honor -S/--separate-dirs (py + ts) GNU -S makes each directory total count only files that sit directly in it, leaving subdirectory sizes out of the parent. Shared rollup and summarize paths now match coreutils 9.7, with unit and integ coverage. * fix(du): apply -S across nested mounts, keep the -c total recursive The traversal fan-out re-derives du's tree centrally from the per-mount blocks, so -S was silently dropped there: du -S over a mount printed the recursive total. Teach the merge about it, and split the operand's own row from what it contributes to -c, which GNU keeps recursive. Add a guard that fails when a new du flag is neither applied centrally nor classified as per-run, which is how -S went missing. * chore(spec): regenerate du spec json for -S --------- Co-authored-by: Zecheng Zhang <zechengzhang97@gmail.com> |
||
|
bytecii
|
31e83de9d3 |
Merge remote-tracking branch 'upstream/main' into feat/generic-owns-operands-and-flags
# Conflicts: # spec/layout_exceptions.json |
||
|
bytecii
|
6f49389aac |
ci(gates): lock in the operands.py layout win; hf keeps no empty dirs
Two CI-only gate failures the local pre-commit hooks never run: - check_layout_parity --strict is a two-way ratchet, and pairing commands/builtin/utils/operands.py with its TS twin closed one divergence (305 -> 304); lower the committed baseline to lock the improvement in. - The directory-list checksum pin assumed `mkdir -p` leaves a statable empty directory, which huggingface's keyed backend does not have (no marker object, nothing in the parent listing), so hf/hf-prefix reported ENOENT instead of GNU's "read error". Narrow the case's targets, matching the existing unlink_dir precedent; the missing-list pin stays on hf, where it passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
41e43274db |
feat(gh): a GitHub CLI, and the write half of the integ GitHub fake (#768)
* feat(integ/github): repo, contents, issues and compare routes
The fake served the read path the `github` resource mounts -- repo
metadata, git/trees, git/blobs -- plus the Actions dataset `github_ci`
needs. A task that *acts* on a repository had nothing to call.
Adds the routes such a task uses: GET /user; POST /user/repos; DELETE
/repos/{owner}/{repo}; POST .../forks; GET .../branches and
.../branches/{branch}; GET .../commits; GET and PUT .../contents/{path};
GET and POST .../issues; GET .../compare/{basehead}.
Writes are visible to the next read, which is the point -- an issue
filed shows up in the listing, a committed file reads back with its new
bytes and a new blob sha, and the commit is recorded so `compare`
can answer which files moved. `FakeRepo` grows `issues` and `commits`
for that; a repository with no writes yet still reports one synthetic
root commit, so "the latest commit" is answerable before the agent has
done anything.
Three behaviours worth pinning rather than leaving to chance:
- A fork deep-copies. A task that forks an archive repo per run and then
commits to the fork must not write through to the source.
- PUT /contents enforces GitHub's sha rule in both directions: replacing
an existing file without the current blob sha is 409, and supplying
one for a file that does not exist is 422. A task that reads before
writing is doing so for this reason.
- A compare against a base the repository has never seen is 404, not an
empty file list. Answering "nothing changed" to a question about an
unrelated commit is the shape of wrongness that reads as success.
`_commit_list` is newest-first, so `compare` collects the commits
*before* it reaches the base; walking past the base instead would report
the commits the base already contains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(integ/github): git data API, rename, raw content and GHES paths
The write half a task that builds a repository needs, plus the two shapes
a real client insists on:
- PATCH /repos/{owner}/{repo} renames, carrying the content with it, and
POST /forks takes a name so a fork can be named in one step.
- The git data API -- get commit, create tree, create commit, move ref --
so a multi-file push lands as one commit. A staged tree is invisible
until a ref points at it, which is what git does and what keeps an
abandoned tree from showing up in a read.
- GET /git/ref/{ref}, and trees by commit sha, because a client resolves
a ref to a commit before reading anything.
- /raw/{owner}/{repo}/{ref}/{path}, with a text content type for text, so
a reader does not base64 the whole repository.
- Every route is served under /api/v3 as well as at the root: a client
pointed at a host that is not github.com talks Enterprise.
- GET /search/repositories, and an unrouted fallback that names the
method and path on stderr. Both exist for the same reason: a 404 is an
answer to a real client, so an unimplemented endpoint reads as a
negative result rather than as an error.
- Seeding takes a per-repo default branch, since a template whose branch
is master is graded at that ref by name.
* feat(gh): a GitHub CLI, and the write routes the fake needs behind it
The `github` mount is the read half -- a repository is a tree, so listing
and reading it is `ls` and `cat`. There was no write half at all:
GITHUB_IO carries readdir/read/stat where GDRIVE_IO carries the full set,
so nothing could commit a file, and forking and renaming are account
operations a filesystem has no shape for in any case.
`gh` is that half, spelled as cli.github.com spells it:
gh repo view [OWNER/REPO]
gh repo fork OWNER/REPO [--fork-name NAME]
gh repo rename NEW-NAME -R OWNER/REPO
gh api ENDPOINT [-X METHOD] [-f key=value] [-F key=value]
`rename` takes the new name as the operand and the repository as -R, which
is the reverse of what the shape of the line suggests and is upstream's.
`-f` sends a string and `-F` reads true/false/null/integers as their JSON
types, which is gh's own --raw-field / --field split; a call with no fields
is a GET, one with fields a POST unless -X says otherwise. Committing a
file is `gh api ... -X PUT -f content="$(base64 -w0 f)"`, which is what
real gh and real GitHub require of each other -- there is no file verb to
mimic.
The transport was GET-only, so it grows a `request`, and an empty body (204,
an empty 202) decodes to null rather than throwing on a call that worked.
Fake, in support: DELETE /contents with the same sha rule as the replace,
and GET /commits/{ref}, which is not /git/commits/{sha} -- it takes a branch
name and reports the file list. Both were found by the unrouted logger
during a run: an agent issued 35 deletes, every one 404'd, and it reported
the files as removed, because to a real client a 404 is an answer.
* feat(integ/github): repository metadata, star-sorted search and readme
A task that picks *between* repositories reads their descriptions and star
counts and never opens one, so seeding gains a second mode: --metadata takes
a JSON file keyed by owner/name and creates repositories that are metadata
only. The tree and contents endpoints answer 404 for them, which is what
GitHub says about a repository with no files.
search/repositories honours sort=stars and matches the description as well
as the name, since a repository is found by what it says it does at least as
often as by what it is called. Terms OR rather than AND, which is looser
than GitHub and errs towards showing a caller the row it wants.
GET /repos/{owner}/{repo}/readme, found by the unrouted logger during a run.
* feat(integ/github): branches
The fake kept exactly one branch, which rules out every task whose premise
is a difference between two of them. FakeRepo now holds a file map and a
commit list per branch:
- `files` and `commits` stay bound to the default branch, so every route
that does not name a ref reads what it always read.
- `branch_for` resolves a ref -- a branch name, HEAD, or a commit sha
belonging to one branch's history -- and `tree_of` returns that
branch's files, so a bad ref is a 404 rather than a silent read of the
default branch.
- contents, tree, raw, readme, branches, branch, commits, commit and
git/ref are ref-aware; PUT and DELETE /contents write to the branch
their body names; PATCH /git/refs moves the branch it names.
- Each branch has its own synthetic root commit, so two branches of a
fixture do not share a head.
- A fork copies every branch, not just the default one.
- Seeding takes `into=<branch>`, so a two-branch fixture is two
directories and two --repo flags naming one repository.
The default-branch binding is what keeps this small: 18 call sites read
`repo.files` and none of them had to change.
* feat(integ/github): GET /git/refs/{prefix}
The plural is a different endpoint from the singular: git/ref/<full-ref>
returns one object, git/refs/<prefix> a list of everything beneath it. A
caller picks whichever it expects, so serving only the singular made the
plural read as 'no such ref'. Found by the unrouted logger during a run.
* feat(integ/github): account repo listing, bare /contents, mirror-stable shas
Three things a task that reasons across several repositories needs.
GET /users/{login}/repos and GET /user/repos. A task that asks "is there a
repository for this on my GitHub" answers it here, and a 404 read as "the
account has none" -- a wrong answer rather than an error, which is the
failure mode the unrouted logger exists for.
GET /repos/{owner}/{repo}/contents, with no trailing slash. GitHub serves
both spellings of the root and a caller picks either; only one was routed.
A branch's root commit sha is now derived from its content rather than from
the repository's name. That is what `git clone --mirror` followed by a push
actually gives you -- the same shas -- and a grader that reads an initial
sha from an upstream repository and the latest from a local copy, then asks
what changed between them, depends on it. Two branches still differ exactly
when their trees differ.
* feat(integ/github): POST /git/refs, so a branch can be created
The last write a task can ask for that the fake had no answer to. A new
branch starts as a copy of whatever the base sha resolves to -- which is
what a branch is, another name for one commit and everything reachable from
it -- with its own empty history, so the two do not share future commits.
422 on a ref that is not under refs/heads/, on a name that already exists,
and on a base sha that resolves to nothing.
* feat(integ/github): GET /, the API root
github.com answers the root with a map of endpoint URL templates. Serving
only the endpoints beneath it meant a client probing the root got a 404,
which reads as the host not being there at all rather than as an unfamiliar
API. Found by the unrouted logger during a run.
* feat(integ/github): seed a branch's commit history, not just its tree
Seeding a directory filled a tree and nothing else, so every seeded
branch answered GET /commits with one synthetic root. That is enough for
a task that reads files and wrong for one that asks when something
arrived. `task-tracker` wants the tasks added by the *most recent*
commit on each of fifteen developer branches, and each branch has five
or six -- so a history of one does not make the task hard, it makes it
unanswerable while looking answerable.
--commits owner/name=<file> reads a manifest keyed by branch, each an
array oldest first, and is applied after every tree: a manifest names
branches, and a branch no directory was seeded into would otherwise be
created here as an empty tree with a history, which reads as a branch
whose files were all deleted.
Shas are derived from full_name, branch, date and message rather than
carried, for the same reason the root's is derived from its tree: a
fixture has to answer the same way twice.
Commits now carry commit.author/committer (name, email, date) and
author.login, which is what a client sorting or filtering by date reads,
and the synthetic root carries them too against a fixed epoch -- it
stands for everything before the fixture rather than for a moment, so
"now" would make it sort ahead of the commits it precedes.
GET /commits/{ref} now renders `files` as {filename, status} objects the
way GitHub does. The list endpoint keeps paths, which is also GitHub's
split: `files` is absent from a commit in a list and detailed when one
commit is asked for. _commit_files is shared with the compare route,
which was building the same objects inline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(github): back the transport with octokit
Wraps @octokit/core behind the unchanged GitHubTransport, so the github
mount and the gh CLI both gain retry and throttling. Handles three
octokit behaviours: {} in a url is a route template that silently eats
the segment, the "METHOD /path" form splits on whitespace, and 204
decodes to '' rather than null. The write throttle is github.com's own
secondary rate limit, so it is off for any other host.
* fix(gh): name the mount its writes invalidate, and parse the host segment
serves was empty, so the executor's post-write cache drop did nothing and
a committed file still read back as its pre-write bytes. parseRepo took
the first two segments of [HOST/]OWNER/REPO, so github.com/acme/tools
resolved to github.com/acme -- a different repository, reported as
success.
* fix(cache): invalidate the index instead of clearing it
A cleared index reads exactly like one that was never filled, so github --
whose index is the whole listing rather than a cache in front of one --
could not tell a cache drop from an empty repository, and ls reported the
mount root missing. IndexCacheStore grows invalidate(), which expires
entries in place, and github refetches once on an EXPIRED lookup. Also
fixes a latent bug with no CLI involved: after the 24h index TTL lapsed a
github mount answered ls with exit 0 and no output. Redis cannot express
stale (an expired key is gone), so it still clears; the comment says why.
* feat(gh): a python gh CLI, mirroring the typescript one
Same four verbs and the same grammar. The python client was GET-only, so
it grows github_request alongside github_get. Layout divergence 305 -> 303.
* test(integ): a gh CLI battery on both hosts
15 cases on a new cli-gh target, covering the write-then-read path through
the mount, the refusals, and the reversed rename grammar. The fake grows
POST /reset, because the cli facet runs both hosts against one process and
writes would otherwise leak between them; it also stops rejecting wrapped
base64, which real GitHub accepts. The unrouted-request logger is now a CI
failure rather than a log line.
* fix(github): probe the parent listing, so an update is not read back stale
The index tracks freshness per directory, never per entry, so get()
never answers EXPIRED and the refill it guarded was unreachable: after a
write invalidated the index the blob's row survived carrying the
pre-write sha, and the read served the old bytes. Create and delete went
through readdir and worked; an update did not.
Readdir on the TypeScript side refilled on any miss, which spent a full
recursive-tree call on every ENOENT and diverged from python. Both now
refill only on EXPIRED.
* test(cache): pin what invalidate keeps that clear discards
A cleared store reads exactly like one that was never filled, which is
what made a github mount report an empty repository. Assert the two
answer differently.
* fix(spec): render an operand's declared name outside the clap dialect
A named slot printed as <text>, so 'man gh api' said <text> where gh
says <endpoint>. argparse prints the dest too. gh is the only spec this
changes; ntn is clap and already rendered it.
* test(integ): cover the gh surface the battery missed
Update-then-read is the case that found the stale-read bug. Also the
help and discovery surface, a leading-slash endpoint, typed fields on a
GET, and a missing endpoint.
* docs(gh): a page per host for the gh CLI
Install, the mount-reads/CLI-acts split, the api rules, and a table of
what differs from gh 2.85.
* fix(github): keep an api field from steering the request
Octokit reads loose parameters off the same object that carries url,
method and headers, so `gh api X -f url=...` retargeted the call
instead of sending the field. The query is spelled into the url and the
body travels as `data`. The python client already separated them.
* fix(cli): let a handler say it did not write
A leaf declares `write` statically because for almost every verb it is
static, but `gh api` carries its method on the line, so a read like
`gh api /user` expired every github mount the install serves. A result
may now report what the spec cannot know.
* feat(gh): expand the owner, repo and branch placeholders
gh's own examples are written with them, so `gh api
repos/{owner}/{repo}/releases` asked for literal braces and 404'd. An
install's repo and branch stand in for the current checkout. Any other
brace pair still reaches the wire, which is gh's behavior too.
* feat(gh): render repo view the way gh renders it
A name line, a description line, then the README, with the separator
omitted when there is none. Probed against gh 2.85. The REST object is
still one `gh api repos/OWNER/REPO` away.
* test(integ): cover placeholders and the new repo view
Also carries a branch on the gh install, which is what {branch}
expands from.
* fix(gh): declare branch on the GhConfig type, and type the test trees
tsc runs in CI but not in pre-commit: GhConfig's zod schema grew
`branch` while its interface did not, and the new github tests built
tree items with a `mode` key the type has no room for.
* refactor(gh): derive GhConfig from its schema instead of declaring it twice
The zod schema is the one doing real work: it validates an install's
config and carries the secretStr marker redaction reads. A hand-written
interface beside it only adds a shape that can drift, which is how
branch reached the schema and not the type.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Zecheng Zhang
|
e28d7e51d7 |
refactor(runtime): guest op surface parity and one Ops facade (#767)
* refactor(runtime): one verb casing on the TS bridge * feat(runtime): create and truncate join the TS bridge, quickjs open dispatches the real op * refactor(runtime): one fopen-mode parser, chmod parser renamed parse_chmod * feat(runtime): python readdir answers resolved entries like the TS bridge * style: formatter output and lint fixes * refactor(ops): WorkspaceFS becomes Ops in ops/ops.ts with the ledger on it * feat(ops): union the facade surface across languages * style: prettier reformat in fuse core * test(integ): guest op-mode battery + ledger pins; record create/truncate in direct cores, rebind the recorder across the hop * fix(ops): exclusivity outranks truncate for wx, and only a confirmed absence establishes in the quickjs shim * fix(integ): cli.sh seeds nested parents and skips facade steps as sdk-only |
||
|
bytecii
|
372e02970b |
fix(gate): count modules inside one-sided directories
A directory present in only one implementation was recorded as a single finding and its module set discarded, so the 21 one-sided directories counted 21 rather than the 74 modules they hold. That made the ratchet blind exactly where drift is easiest: a new module under a directory already in the baseline -- `agents/mcp`, `ops/postgres`, `core/onedrive` -- left the count untouched and `--strict` passed. A one-sided directory now contributes every module inside it. Adding a module under `agents/mcp` moves the count 310 -> 311 where it used to sit at 257 forever. Directories become presentational: they are printed once with their module count and excluded from `total()`, so they are not double-counted against the modules they now contribute. An excused directory stays a deliberate blind spot, and keeps excusing its whole subtree. The excuse is precisely that there is no counterpart to mirror -- agno is python-only, opfs is browser-only -- so a new module inside one is expected growth, not drift. Verified both ways: a probe under `agents/mcp` (unexcused) moves the count, a probe under `core/opfs` (excused) does not, and stale-entry detection still fires. Baseline 257 -> 310. The number is larger because the unit is now consistently one module; no divergence was added. Reported by codex on #759. |
||
|
bytecii
|
a6b59533c2 |
feat(gate): diff the python and typescript module layouts (#609 item 30)
CLAUDE.md asks for the two trees to keep a mirrored layout and nothing checked it, so seven helper modules went missing and two module homes swapped without anything noticing. scripts/check_layout_parity.py diffs the module-name sets directory by directory: core/node/browser union onto one python namespace because CLAUDE.md puts runtime-specific code in node/browser, while cli, server and agents are separate npm packages whose python twins are subpackages, so they map by prefix. Two normalizations keep the report honest. __init__.py and index.ts are skipped, since only python requires a package file per directory and comparing them reports a gap across most of the tree that nobody intends to close. Module and directory names fold camelCase, hyphens and a leading underscore, so findEval.ts against find_eval.py reads as a rename rather than two missing modules -- 15 renames separated out from 207 genuinely absent modules that way. --strict does not demand zero. 257 divergences predate the gate and each needs its own decision, so it fails when the count moves off the committed baseline in *either* direction: new drift is blocked, and closing a divergence has to be locked in by lowering the number rather than being silently spent. A stale exception fails just as loudly, which is the failure mode every hand-maintained allowlist in this repo has already hit. The report reproduces five findings the audit had filed separately: findEval/findParse as the only camelCase modules, _provision.ts vs provision.ts, workspace/snapshot missing keys and types on the python side, no builtins/shared.ts, and node's cache/redis/file.ts sitting where python has cache/file/redis.py. That is what it is for -- items 31 to 37 get scoped from its output instead of from taste. Also gates scripts/gen_width_table.py against its committed output, the same way the spec trees are gated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
cfbe51b1fc |
feat(gws): calendar and forms API passthrough (#754)
* feat(gws): add calendar and forms API passthrough, and mock routes Adds `gws calendar` (calendarList, calendars, events, freebusy) and `gws forms` (forms, responses) in both languages, plus routeCalendar and routeForms in the integ GWS mock. * feat(gcal): mount Google Calendar as day directories Adds the gcal resource: one directory per calendar, one per local day, one JSON file per event, named <eventId>__<HHMM-HHMM>_<Title>.gcal.json. rm deletes the event; everything else goes through `gws calendar`. Day bucketing uses one mount-wide zone, defaulting to the primary calendar's, with bounds computed as consecutive local midnights so a DST day is 23 or 25 hours rather than a fixed 24. TypeScript mirror still to come, so spec parity is red until it lands. * test(integ): cover gws calendar and forms, fix the service count Adds a calendar/v1 fixture seeded through events.insert on both hosts, and 12 cases: overlap querying, all-day end-exclusivity, q search, freebusy, delete, form create/batchUpdate, and that a form's id is reachable through Drive (the Forms API has no list method, so that is the only path an agent has to an existing form). gws --help now lists 8 services, not 6. * fix(gcal): address codex review - percent-encode Discovery path params, so a holiday calendar id's "#" no longer truncates the URL at /calendars/en.usa (py + ts) - resolve a zone-less dateTime through its declared timeZone instead of leaving it naive, which raised TypeError when bucketed (py + mock) - register commands and ops at module scope; there is no cycle - validate a day is a real date, not merely date-shaped, so 2026-02-30 is ENOENT rather than a directory readdir then fails on - centre the default window in the bucket zone, not the host's * test(integ): cover calendar and forms mutations, fix the stale service count * feat(gcal): mirror the calendar mount in typescript * fix(gcal): drop the redundant close override and the polynomial regexes |
||
|
Zecheng Zhang
|
003753c21a |
feat(python3): parse CPython's option table, and seed sys.path on pyodide (#743)
* fix(ids): stamp uuid7 from the clock, not from the previous id
uuid6's uuid7 orders ids inside one millisecond by bumping the
timestamp past the previous one. The borrowed value becomes the next
call's floor, so the error compounds: a burst of 5000 ids stamps every
later one 4991 ms into the future, and it stays there until the clock
catches up. These are workspace and session ids, used as time-ordered
database keys.
Mint the id here instead, with RFC 9562's dedicated counter in rand_a:
48-bit timestamp, 12-bit intra-millisecond counter, 62 random bits. The
timestamp is only ever a real clock reading, and an exhausted
millisecond waits for the next rather than borrowing.
Drops the uuid6 dependency, which nothing else used. TypeScript already
had this right, via the uuid package's counter.
* feat(python3): parse CPython's option table, and seed sys.path on pyodide
python3's spec carried one option, -c, and a free-text rest, so every
other switch fell through to the operand slot: 'python3 -u s.py' ran
'/-u' as the script and failed, 'python3 -zz -c ...' exited 0 with the
flag swallowed, and -V and -h were read as paths.
Two spec knobs fix the class rather than the command. stop_at_operand
turns off the parser's dash-operand leniency and ends option parsing at
the first operand, which is CPython's own rule; ends_options marks the
options that carry a program (-c, -m), so the words after them are the
program's argv rather than python3's. Both mirror in TypeScript.
On top of that: -m runs through runpy, with a find_spec probe so a
missing module is CPython's one line instead of a traceback; the four
source doors each report the argv[0] CPython gives them; and the
interpreter-init switches ride in RunArgs.flags, honored where the
engine can and reported on stderr where it cannot.
argv[0] needed one more piece. The local and wasi tiers hand CPython
the program through -c, so it hardcoded argv[0] to '-c' and named every
frame '<string>'. bootstrap() re-compiles under the real name, which
fixes argv[0] and the traceback filename together.
Pyodide gains sysPath, which glob-expands and prepends mount paths once
the mounts are in place, so a vendored wheel imports without the
sys.path.append incantation. A glob that matches nothing is reported on
the run's stderr; the seed cannot report it itself, since it runs
before stderr is captured. packages, packageBaseUrl and lockFileURL
reach a prebuilt distribution, and the wrapper honors -O, -B, -W, -X,
-E and -I.
Verified against CPython 3.13.7: all 14 probed lines match on stdout,
stderr and exit code.
* refactor(spec): borrow argparse's REMAINDER instead of two bespoke parser knobs
python3's line needs opposite treatment of an unknown dash word on
either side of the script: `-zz` before it is python3's usage error,
`--foo` after it is data the script must receive. The first pass added
CommandSpec.stop_at_operand and Option.ends_options to say that, which
put two per-command dialect fields on dataclasses every command shares.
The first half is not ours to invent: it is argparse's
nargs=argparse.REMAINDER, and POSIX's own option order (GNU's permuting
default is the extension, which `POSIXLY_CORRECT=1 ls a -1` shows).
argparse spells it on the positional slot, so it goes on Operand under
argparse's name, and CommandSpec is untouched.
The second half argparse cannot express at all: `add_argument('-c')`
beside nargs=REMAINDER answers `unrecognized arguments: -u`, and CPython
parses its own command line in C for that reason. So it stays out of the
grammar. POSIX already spells the handoff `--`, and the parser already
consumes one, so a table in workspace/route inserts it after -c/-m's
value, beside the other command-name-keyed line rules. The marker is
added unconditionally: CPython passes a typed `--` through as data
(`python3 -c p -- -u` gives the program ['-c','--','-u']), so the parser
eats exactly the one added here.
Net: Option and CommandSpec return to what they were, Operand gains one
borrowed field. CLAUDE.md now states the rule as a literal test, since a
CLISpec is a CommandSpec and cannot be used to escape it.
* docs: require POSIX and argparse, not either, for a shared grammar field
POSIX alone specifies an option whose argument is a program (sh -c, and
python3's own synopsis), so an either-or rule would license the Option
field the section exists to refuse.
* fix(python3): address review on init switches, and declare the four missing ones
- pyodide restores warnings.filters after a run, so -W error no longer
follows every later line on the warm interpreter
- -OOO saturates at compile()'s max of 2 rather than raising ValueError
- -E/-I stop deleting PYTHON* from os.environ: CPython's -E stops those
variables configuring startup, it does not hide them from the program
- pyodide reports -E/-I/-s/-S through unhonoredNotice instead of faking
them; unhonored() grew a honored subset, and now reports -OO too
- declare -b, -P, -x and --check-hash-based-pycs, which CPython accepts
and mirage was refusing. -x is answered by the source resolver, since
handing it to an engine that runs code via -c would honor it nowhere
- end_options_after_program walks a short cluster letter by letter, so
-uc 'p' -v hands -v to the program, and steps over long value options
* fix(integ): argv[0] for a payload is -c on every engine, monty included
Monty's DEFAULT_PROG placeholder only applied because nothing ever told
it a name. The python3 command now supplies CPython's own answer, so
these four policy-routing steps read -c. They still distinguish monty
from local, since argv is a monty global and CPython would NameError.
Also picks up the formatting of the new flags test: it was untracked
when pre-commit ran, and --all-files goes through git ls-files.
* fix(python3): do not rewrite a shadowed name, and stop over-claiming -W and -X
- the `--` handoff is skipped when a shell function shadows python3.
bash's rule gives the function the line, and it has no CPython option
table to read a marker with. `command python3` masks the function for
its inner run, so the rewrite applies there again. A CLI cannot reach
this at all: register_cli refuses a shell builtin's name.
- an invalid -W filter is reported as CPython reports it and the program
still runs, instead of _OptionError escaping the wrapper and killing a
line every other runtime completes.
- a known -X name is reported as unhonored, since populating
sys._xoptions is all a warm interpreter can do for one. An arbitrary
name stays silent, which is all CPython does with it either.
- -O's reach is written down: the payload is compiled at the requested
level, a module imported from sys.path is not, and sys.flags stays 0.
|
||
|
bytecii
|
9a86bf72ff |
Merge pull request #746 from bytecii/feat/box-dropbox-io-slots
fix(box,dropbox): wire the du slot in python, drop the fake find slot in ts (#609 T1-I) |
||
|
bytecii
|
1d8fb1cb53 |
fix(box,dropbox): wire the du slot in python, drop the fake find slot in ts (#609 T1-I)
Finishes T1-I, the `CommandIO` slots that are wired on one side only.
du (python gains it): `core/box/du/` was written and unreachable —
imported by its own `__init__` and its tests and nothing else — and
dropbox had no python du at all, so both took `DU_BUILDER`'s walk bounded
by `max_du_entries`. Past 10 000 entries that prints a partial total, a
stderr notice and exit 1. Ports dropbox's du beside box's and wires both,
matching the typescript tables.
find (typescript loses it): `core/box/find.ts` and `core/dropbox/find.ts`
were not pushdowns. Neither API has a search call to push to, so both
were the *same* readdir/stat walk the find and cp builders already run —
minus three things the builders pass and an op cannot:
- the mount's own index (they allocated a scratch `RAMIndexCacheStore`,
so every walk started cold and warmed nothing for the next command),
- the namespace stat overlay, because the native branch only applies it
when `ops.local === true` and both backends are remote,
- the symlink table `-empty` counts entries from.
The overlay half is a live divergence: `touch -d 2020-01-01 f` then
`find -mtime +5 f` finds it in python and does not in typescript. Pinned
by extending find/mtime.json's four mtprobe cases to box and dropbox;
they fail on pre-fix typescript on both targets and pass after.
The stale comment that justified the ops ("the cp builder may call find
without threading an index, unlike python") is refuted by cp.ts:47, which
reads `opts.index` and passes it to its own fallback walk.
Leaves box/dropbox with one command_io exception each (python range-reads,
typescript's core read takes no window) instead of three.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
bytecii
|
80f096a9f4 |
fix(tee): write every operand, and use the append slot (#609 T1-I)
`tee`'s spec declares `rest: {type: "path"}`, so the parser accepts N file
operands — and both generics wrote only `paths[0]` and silently dropped the
rest. `tee a b` put nothing in `b`. Identical in both languages, which is
exactly why no parity gate saw it: a shared bug is not a divergence.
GNU 9.7, pinned in docker: `printf x | tee a b c` writes all three; a failing
operand is diagnosed and skipped while the others still get the data (exit 1);
only `--output-error=exit` stops at the first failure. That last mode had never
been implemented because with one sink it was unobservable — the old comment
said as much — so it was parsed and ignored.
Deliberate divergence, documented at the call site: GNU opens every operand up
front, so under `exit` an *open* failure aborts before anything is written. A
mount has no open/write split, so operands before the failure are already
written. The two agree whenever the failure is at write time, which is what a
remote backend reports.
The `append` CommandIO slot is now wired into the generic, per the decision to
wire rather than delete it. It was declared on both adapters, wired by five
backends, and read by no builder, so `tee -a` was read-modify-write everywhere
— on ssh that meant downloading and re-uploading the whole file plus a
lost-update window. A backend with the slot now appends; the rest keep the
fallback. Where the native path runs, the resulting content is not in hand, so
the path is reported in `writes` but not in `cache`: that is how apply_io is
told to drop the stale entry instead of caching a wrong one. The slot was typed
`(...args: any[]) => unknown`, which is what let five backends wire something no
builder could consume; it is typed like `write` now.
Found while wiring it: python's `core/ssh/append.py` existed with no importer
outside the resource, so python ssh took the read-modify-write path its
typescript twin did not. Wired.
Two more real bugs on the way through:
- TS `tee -a missing-file` threw instead of creating the file. The not-found
test was `/not found/i.test(err.message)`, but `enoent()` puts the *path* in
the message, so it never matched — verified by running it. Python's
`except FileNotFoundError` was always correct. That broken test is why s3 and
gridfs had bespoke tee wrappers carrying an `exists()` pre-check (with a
`catch {}` that swallowed everything) to route around the generic. All four
wrappers — s3 and gridfs in both languages — are now wiring only, which fixes
their multi-operand bug in the same edit.
- `isEnoent` existed in three copies (`core/generic/find.ts`,
`workspace/reconcile.ts`, `executor/builtins/metadata.ts`) and in none of them
was it next to `isFsError`. One copy now, in `utils/errors.ts`.
Verified: three integ cases across 17 targets, each proven red on pre-fix code
and green after, on both hosts — the multi-operand pair fails as `DUP|` (first
operand only) and `tee -a` on a missing file fails TS-only. `xm_tee_multi`
passed all along because it is cross-mount: one operand per mount, so each
backend tee wrote its single `paths[0]`. Full core/node/browser vitest and the
python suite green; specs regenerate with no drift and the parity gate passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
bytecii
|
cc91155480 |
fix(s3): browser ranged reads, EOF windows, and a node/browser gate
Resolves both codex findings on #742, plus the regression they framed and a divergence found alongside it. The presigned-fetch shim (core/s3/_client_browser.ts) reads only `Key` off a GetObject input and drops `Range`, so #742's `readRange: rangeOf(s3Read)` returned the *whole object* as the requested window — on all fifteen browser S3-family resources (s3, r2, gcs, minio, ceph, backblaze, wasabi, digitalocean, scaleway, supabase, aliyun, tencent, oci, qingstor, seaweedfs). Before #742 the ops factory took the slice fallback and answered correctly. `core/s3/stream.ts` now owns a `readRange` that branches: with the AWS SDK it asks for one ranged GET as before, and on the presigned path it streams and stops at `offset + size`. Adding a real Range header there is not the fix — the header makes the request non-simple and trips a CORS preflight presigned deployments generally do not allow — so the shim now *refuses* a Range it cannot serve instead of answering wrongly. Stopping early only saves the bytes if the transfer stops too: `bodyFromResponse` released the reader's lock without cancelling, so the rest of the object arrived anyway. Codex C2, unsatisfiable ranges: a window at or past EOF is an empty read under POSIX and a 416 over HTTP, so wiring a native range changed the answer for the same call. Normalized once per language in the ops factory rather than in each reader, keyed on a shared `isUnsatisfiableRange` / `is_unsatisfiable_range` that reads the botocore, aiohttp, httpx and aws-sdk-js error shapes. The node S3 mock sliced silently instead of refusing, which would have let the fix look correct without being tested; it now answers 416 like the real thing. Codex C1, node vs browser: `merge_variants` resolves the two runtimes by preferring node, which turned a divergence into a silently discarded value. `check_variant_facts` now diffs the two variants for both resource-fact tables *before* that merge, with its own stale-checked `variant_resource_facts` exemption table. It found thirty real divergences: the browser S3 resource declared neither `sizesAlwaysKnown` nor `storageId` where its node twin declared both, so a browser `mv` between two mounts of one bucket saw two separate storages. Fixed rather than exempted — the browser class gains both, and the identity now comes from a shared `s3StorageId` so the runtimes cannot compute different answers for one bucket. Zero exemptions needed. Found alongside: TS `rangeRead`'s fourth argument is end-exclusive on ssh, hf, nextcloud and databricks_volume, and in every python twin, but s3 and gridfs read it as a length — `range_read(p, 10, 20)` returned twenty bytes from offset ten where python returned ten. Both corrected. s3's inline Range builder also now goes through the shared `rangeHeader`. Verified: core/node/browser vitest and the full python suite green, both spec trees regenerate with no drift, and the parity gate proven red on a planted node/browser divergence and on a stale variant exemption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
bytecii
|
92a9196ac9 |
feat(spec): gate resource capabilities and CommandIO slots (#609 item 4)
Registry membership only ever said a backend could be *built*. What it
does once mounted — how stale a listing may be, whether reads are cached,
whether a `du` is pushed down to the API — was a second hand-maintained
surface with no gate at all, which is how python served up-to-ten-minute
stale listings of a live postgres schema while typescript pinned 0.
`spec/*/resources.json` now carries two more tables, diffed by
`check_spec_parity.py` with the same one-fact-per-exemption rule the
command checks use:
- `capabilities`: per registry name, `index_ttl` / `caches_reads` /
`supports_snapshot` / `sizes_always_known`, plus whether the class
overrides `storage_id` and `statfs`.
- `command_io`: per backend command package, the wired `CommandIO` slots
plus `local` / `max_glob_matches` / `max_du_entries`.
Python reads both off its classes and dataclasses. TypeScript reads them
from the source declarations (`scripts/resource_facts.ts`), because the
twins are instance fields and observing them at runtime would mean
constructing the resource — and construction is not inert:
`buildResource('github', {})` issues an HTTP request and `postgres` opens
a connection. A value the extractor cannot read as a literal is dumped as
`<expr:…>` rather than guessed.
The gate found 15 live divergences on its first run. Fixed here:
- `readRange` was wired on `disk` alone in typescript while python pushed
the window down on twelve backends, so `head -c 100` on a large object
downloaded the whole thing and sliced. Wired on the five whose read
already takes an `{offset, size}` window (s3, gridfs, nextcloud, hf,
databricks_volume) via a new `rangeOf` adapter, and `gen-specs` now
refuses to emit when a reader takes a window with no slot to hand it —
the twin of python's `test_read_range_optin.py`, which is why python
was fully wired and typescript was not. The check keys on `offset` *and*
`size`: postgres pairs `offset` with `limit` to mean SQL rows.
- python github ignored `SCOPE_ERROR` and refused globs at 10001 matches
instead of 5001; the constant had no importers.
The remaining ten are documented in `parity_exceptions.json`, each naming
one resource and one key: box/dropbox `du`+`find` and the seven readers
with no window argument (T1-I), github `supports_snapshot` (snapshot
redesign), lancedb `caches_reads` (computed per URI), notion/hf `find`
(slot on one side, bespoke command on the other), ssh `append` (wired in
typescript, read by no builder on either side).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
bytecii
|
0851e54bc5 |
Merge pull request #737 from bytecii/chore/integ-floor-and-dead-code
refactor(integ,provisions): make the integ runner fail on zero work, and single-source the tables that existed twice (#292) |
||
|
Zecheng Zhang
|
07bca1b503 |
fix(archive): tar and zip archive a directory, and -C re-bases the operands after it (#738)
* fix(archive): tar and zip archive a directory, and -C re-bases the operands after it tar -cf d.tar d exited 1 with "tar: <hostpath>/d: Is a directory": every operand went straight to read_bytes, with no isdir check and no recursion, so the most common tar invocation there is could not work. zip -r <dir> had the identical defect. Both now walk. Create is a two-phase pass in each: decide every member, then write. Both plans are built on one traversal, scan_operand / scanOperand (generic/archive/walk), which merges three sources no single one can see: the backend walk (find's walk_find, so an entry is classified through stat and never by name), the namespace's symlinks, and the mount table. It reports paths, never names, because naming is where the two formats part company; the two things they disagree about in the traversal itself are parameters, so tar passes recurse=True and dereference=-h while zip passes recurse=-r and dereference=not -y. A directory is its own member, so an empty one survives a round trip, and both extractors now mkdir for one. A symlink is a symlink member under tar (SYMTYPE, target in linkname) and under zip -y (mode 0120777). An unreadable operand is reported in virtual path space with the archiver's own wording, because the raw IsADirectoryError was leaking the host path behind a disk mount. tar -C was ignored: `tar -czf /work/out.tgz -C /work/check my_paper` failed as "paths span multiple mounts", the same defect class as unzip -p in #725, because the operand resolved against the session cwd and the router then saw a phantom mount span. -C is not a flag the command reads once, it is a chdir for the operands typed after it, so it is now declared in the spec (CommandSpec.operand_base) and resolved by the one component that walks the line positionally: the parser reports a base per word and the classifier resolves each operand against it. MountView is how a command sees mount boundaries, offered the way LinkView is (name a `mounts` parameter, nothing else). A traversal that renders lines gets this free from the executor's fan-out; one that emits a single binary object cannot, which is why the archivers read the table themselves. A descendant mount is not crossed: the mountpoint stays an entry and its contents are dropped with GNU's --one-file-system wording, since descending would archive by accident what MountRootPolicy now refuses on purpose for tar, zip and cp in a source slot. Semantics pinned against GNU tar 1.35 and Info-ZIP 3.0 on debian:stable-slim, including Info-ZIP's inverted defaults, its anchored -x, its silent leading-slash strip, and "Nothing to do!" exiting 12 with no archive written. * chore(spec): the dumps state what a command declares, not what it defaulted to Both generators emitted every field of every dataclass, so `truncate`, which declares one thing (`-s`/`--size` takes a string), spent 27 lines of spec body restating 21 defaults that read identically in all 93 files. `zip` was 198 lines for five flags, 11 of the 13 keys on each option being defaults. Anything equal to its default is now dropped on both sides. `type` survives even at its default, because what a token is is the first thing a reader looks for, and `"rest": {}` says less than `"rest": {"type": "path"}`. The defaults come from the dataclass fields in python and from a default-constructed instance in typescript, rather than from a table either side could let drift. The two must drop exactly the same keys, and the parity gate reports every command if they do not. This narrows that gate's blast radius rather than widening it. When `operand_base` was missing from gen-specs.ts, python emitted the key in all 93 files and typescript in none, so parity reported ~95 divergences to sift. The same bug now reports one, on tar, which is the only command that sets it. check_spec_parity's option-diff renderer keyed options by `o["long"] or o["short"]`, which raises once an option carries only the spelling it declared. 279 files, 22113 lines of restated defaults gone. * fix(archive): strip the mount prefix by scan, not by a backtracking regex childSpec measured the backend key with `.replace(/^\/+|\/+$/g, '')`. The trailing alternative backtracks on a run of slashes, which CodeQL flags as js/polynomial-redos, and the input is a resource path a mount supplies. utils/slash.ts already has stripSlash, which walks the two ends by charCode and cannot backtrack. Use it. slash.test.ts pins it against the regex it replaces. * fix(archive): the five symlink and mode bugs the codex review found All five reproduced against a live workspace first, then pinned against GNU tar 1.35 and Info-ZIP 3.0 on debian:stable-slim. A symlink operand never reached the planner as a link. tar and zip were absent from NO_FOLLOW_COMMANDS, so the router rewrote the operand through the link table and `tar -cf o.tar link` stored a regular file holding the target's bytes where GNU stores a symlink member of size 0. It also skipped the planner's cross-mount refusal, since by then there was no link left to refuse. Both archivers now lstat, and carry no DEREFERENCE_FLAGS entry on purpose: -h and -y are the planner's to read. Only the last -C was checked. `tar -cf m.tar -C missing x -C good y` reported `x: Cannot stat` and still wrote an archive holding y, where GNU chdirs at each -C and dies at the first it cannot enter. The option accumulates now and the planner walks the list, so the first bad one is fatal and no members are written. Two links to one target were called a loop, and a real loop was not caught at all. Both were the same mistake: an operand-wide `seen` set doing detection the namespace already does properly under a hop limit. Deleting it archives both names, as GNU and Info-ZIP do, and catching the CycleError that resolve raises turns a genuine cycle into one fatal problem per member with GNU's "Too many levels of symbolic links", keeping the directory entry and exiting 2 instead of throwing out of the planner for a bare exit 1. The mount-root refusal denied member selectors. Under -t and -x an operand names something inside the archive, so `tar -tf a.tar data` was refused as busy when `data` happened to spell a mount. Gated on create mode now, dashless first word included. A test asserted the loop bug rather than catching it; it is replaced. Five integ cases cover the lot end to end in both languages. * test(archive): build the test LinkView's stat through FileStat, not a cast |
||
|
bytecii
|
fda274d78e |
chore(spec): regenerate the spec trees for the three commands that gained a provision
dify/find, mongodb/rg and postgres/rg had no provision while their generic siblings on the same backend did; routing the hand-written command tails through the family catalog gives them one, so `has_provision` flips to true for those three in all three spec trees. Spec parity: 93 commands match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Zecheng Zhang
|
fdfb8a075b |
fix(tar): accept GNU's old option style (#734)
* fix(tar): accept GNU's old option style `tar xzf a.tgz` failed with "must specify -c, -x, or -t" because a first word with no leading dash was read as an operand. It is a cluster of option letters whose arguments follow as separate words, expanded now by expand_old_style / expandOldStyle before anything else reads the line. Per letter, not one -xzf token: a cluster may hold several argument-taking letters (tar xfC a.tgz out) and may continue past one (tar cfz a.tgz f gzips), neither of which getopt clustering can express. * chore(spec): regenerate spec dumps for old_option_style |
||
|
bytecii
|
f328bc0314 | fix(parity): truncate/split data-loss flag values, od radix, registry membership, and the gate that missed them (#609 Block A) (#706) | ||
|
Zecheng Zhang
|
e9d69f319b |
fix(unzip): select archive members, exit 11 on unmatched filespecs (#725)
* fix(unzip): select archive members, exit 11 on unmatched filespecs * fix(unzip): match filespecs bytewise, read each selected ZipInfo Pinned against Info-ZIP 6.00 on debian:stable-slim: - `?` stands for one byte, not one code point, so `?.txt` misses `é.txt` and `??.txt` hits it. Python matched code points and TypeScript matched UTF-16 units; both now match encoded bytes. - A name lookup resolves every duplicate member to the last one, so `unzip -p a.zip dup.txt` served the same payload twice. Read the selected ZipInfo instead. Test callbacks move to module scope. |
||
|
Zecheng Zhang
|
67945abf63 |
feat(git): a git CLI over any mount, in both languages (#710)
* feat(git): a git CLI over any mount, in both languages Adds `git` as a builtin CLI (issue #705 item 1): status, log, show, diff, branch, add, reset, commit, checkout. It takes no config, and the repository is read entirely through the mount ops, so a repo on RAM, on disk or on an object store reads the same way. Python goes through dulwich, TypeScript through isomorphic-git backed by a PromiseFsClient over the dispatcher. Status is pinned against the real git binary across 22 repository states and 6 spellings, the mutation verbs are read back by real git and pass `git fsck`, and the rename-similarity score matches dulwich digit for digit. Both run in integ on `git-ram` and `git-disk` in both languages. Also fixes the grep family this leaned on: - grep reads a basic regular expression by default, as POSIX says, with -G for the default and -E for extended. `grep -l` and `grep -rl` compiled their own pattern and still read extended. - zgrep does the same. Its -E flag was read and discarded. - an operand grep could not search exits 2, as GNU does, instead of being flattened to 1. - `grep -l` on a directory reports it rather than walking it. The shared fallback called the recursive walk whenever a failed read turned out to be a directory, which made -l behave like -rl. Supporting pieces: ranged reads (`read_range`) with a read-and-slice fallback and a seeking implementation for disk, PATH-typed group options resolved against the working directory, and git's three unknown-option dialects. * fix(rg): report an unreadable operand as exit 2, like ripgrep CI caught four goldens the grep change had missed, all outside the JSON harness targets I had been running, plus a divergence the change itself introduced. The divergence: TypeScript routes rg through grepGeneric, so rg moved to exit 2 there while python's rg, which has its own generic, stayed at 1. Real ripgrep exits 2 for an operand it could not read, so TypeScript was right and python is brought up to it rather than the other way round. Its single-operand path also let the error escape to the shared handler, which flattens every OSError to exit 1; rg now reports it itself, the same fix grep needed. The rule is now stated once in grep_helper (`exit_code_for` / `exitCodeFor`) and imported by both generics in both languages, instead of living in the grep generic where rg could not reach it. Goldens updated: integ/cross_commands.{py,ts}, the shared observability contract, langfuse and dify. Each pinned exit 1 for a missing operand and carried a comment calling it a deliberate divergence. * fix(git): refuse the three mutations that could lose work Three review findings, each measured against git 2.50.1 before fixing. checkout only compared tracked paths, so a branch holding a file the working tree has untracked wrote its blob straight over it. The file is in no index and no tree, so nothing could see it and nothing could get it back. The untracked set is now part of the conflict check, which needs UNTRACKED_ALL rather than the mode status uses: "normal" collapses a wholly untracked directory to one row, and git names the file inside it. An ignored file stays overwritable, which is git's own split. When both kinds of conflict apply git prints both paragraphs and aborts once, so CheckoutConflictError carries both lists. branch -d deleted a branch HEAD does not contain, dropping the only name pointing at those commits. It now refuses, and -D is added, because -d alone would be a delete with no way to say no. dulwich's can_fast_forward answers exactly this and cannot be used: it asks the repository for its grafts and shallow boundary and a bare BaseRepo raises. Walker is what log already walks with and needs only the object store. add -u ignored its pathspecs and restaged every tracked file, which is how an unrelated edit reaches the next commit. git tells two misses apart and so does this now: a pathspec naming nothing is a fatal about the pathspec, one naming an untracked file is a fatal about git not knowing it, both exit 128. Also two CI failures. The spec dumps were stale for the -G flag added with the zgrep BRE fix. And the git fixture exported its identity as environment variables, which reach only its own commits, so a test that committed into the built repository failed with "Author identity unknown" on any runner with no global identity. It now records the same identity in the repository, leaving the object ids unchanged. Covered by unit tests in both languages and six integ cases across git-ram and git-disk on both hosts. * fix(test): the redis missing-file suite still pinned grep at exit 1 Sibling of the disk suite, which was updated with the exit-2 change. This one needs a live server, so it skips silently without REDIS_URL and the local run never reached it. Only the grep case moves: cat, head, tail and wc all still exit 1 for an operand they could not read, which is the split the comment now records here as it does next door. * fix(grep): read an operand's type from stat, not from how the read failed Four integ jobs caught this across seven backends, and it is one mistake with three faces. Classifying a directory operand *after* a failed read makes the answer depend on what each backend does about reading one, and they disagree: s3, gridfs, hf and nextcloud read a directory path without complaint and hand back nothing, and ssh raises an asyncssh SFTP error that is not an OSError at all, so it escaped the catch entirely and exited 1 with an unattributed "grep: Is a directory". So the operand is now stat-ed before it is read, which is what GNU does and what the -r branch of the same function already did. operand_is_directory had its own version of the same error: a readdir that did not raise counted as a directory, and a prefix store answers readdir for any path at all, returning nothing for one that is not there. Every missing file on those backends therefore read as a directory. The listing must now be non-empty to count, which costs a genuinely empty directory being invisible there, the same divergence du already documents and the safer way round. The catch also widens from three exception types to WALK_ERRORS, the tuple every other operand-tolerant walk in the repo uses, so an errno split cannot make grep abort where tree and grep -r keep going. One test fake resolved stat with `undefined as never` and only worked while nothing read the value. It resolves with a real FileStat now, which is what the postgres backend answers there; the assertion the test exists for is untouched. Verified on gridfs and gridfs-prefix, which reproduce the prefix-store half locally, plus ram, disk and redis for regressions: all five targets on both hosts, 0 failed. * fix(git): three review findings on -C, checkout -b and reset `-C` accepted any path that was not missing, so naming a file inside a repository walked up and ran in the parent instead of failing the way git's chdir does. For a write verb that means mutating a repository the caller never named. It now refuses a non-directory with git's own second wording, "Not a directory". `checkout -b <new> <start>` forced the new branch to HEAD and dropped the start point without a word, so every commit after it landed on the wrong history. The operand is honored, and a start point that is not a commit gets git's sentence naming both it and the branch rather than the generic "ambiguous argument" the same lookup failure produces elsewhere. `reset <operand>` that selected no path unstaged nothing and exited 0, which a script reads as "the index was reset". Two different mistakes reach that point and they now get different fatals. A typo is git's "ambiguous argument". A revision is not: real git resets the index to any commit named there, measured on 2.50.1, so the review's premise that git refuses one is wrong. This build resets from HEAD only, and says which feature is missing instead of claiming a revision it can resolve is unknown. Recorded as a divergence in both git.mdx files. Six integ cases across git-ram and git-disk on both hosts, plus unit tests in both languages. * chore: merge main and regenerate the grep and zgrep specs #713 landed a `pair` field on the option spec while this branch was open. CI builds the PR merged with main, so it regenerated specs carrying the field and compared them against grep.json and zgrep.json, the two files this branch had already rewritten for `-G`, which therefore predate it. Every other spec file came across from main already carrying it. The regeneration also needs a rebuilt dist: gen-specs.ts reads the built package, and against a stale one it dies on an Operand field it does not know, which silently leaves the typescript half unregenerated and shows up as a python/typescript parity divergence rather than as a build error. |