Files
Zecheng Zhang 41e43274db feat(gh): a GitHub CLI, and the write half of the integ GitHub fake (#768)
* feat(integ/github): repo, contents, issues and compare routes

The fake served the read path the `github` resource mounts -- repo
metadata, git/trees, git/blobs -- plus the Actions dataset `github_ci`
needs. A task that *acts* on a repository had nothing to call.

Adds the routes such a task uses: GET /user; POST /user/repos; DELETE
/repos/{owner}/{repo}; POST .../forks; GET .../branches and
.../branches/{branch}; GET .../commits; GET and PUT .../contents/{path};
GET and POST .../issues; GET .../compare/{basehead}.

Writes are visible to the next read, which is the point -- an issue
filed shows up in the listing, a committed file reads back with its new
bytes and a new blob sha, and the commit is recorded so `compare`
can answer which files moved. `FakeRepo` grows `issues` and `commits`
for that; a repository with no writes yet still reports one synthetic
root commit, so "the latest commit" is answerable before the agent has
done anything.

Three behaviours worth pinning rather than leaving to chance:

- A fork deep-copies. A task that forks an archive repo per run and then
  commits to the fork must not write through to the source.
- PUT /contents enforces GitHub's sha rule in both directions: replacing
  an existing file without the current blob sha is 409, and supplying
  one for a file that does not exist is 422. A task that reads before
  writing is doing so for this reason.
- A compare against a base the repository has never seen is 404, not an
  empty file list. Answering "nothing changed" to a question about an
  unrelated commit is the shape of wrongness that reads as success.

`_commit_list` is newest-first, so `compare` collects the commits
*before* it reaches the base; walking past the base instead would report
the commits the base already contains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(integ/github): git data API, rename, raw content and GHES paths

The write half a task that builds a repository needs, plus the two shapes
a real client insists on:

  - PATCH /repos/{owner}/{repo} renames, carrying the content with it, and
    POST /forks takes a name so a fork can be named in one step.
  - The git data API -- get commit, create tree, create commit, move ref --
    so a multi-file push lands as one commit. A staged tree is invisible
    until a ref points at it, which is what git does and what keeps an
    abandoned tree from showing up in a read.
  - GET /git/ref/{ref}, and trees by commit sha, because a client resolves
    a ref to a commit before reading anything.
  - /raw/{owner}/{repo}/{ref}/{path}, with a text content type for text, so
    a reader does not base64 the whole repository.
  - Every route is served under /api/v3 as well as at the root: a client
    pointed at a host that is not github.com talks Enterprise.
  - GET /search/repositories, and an unrouted fallback that names the
    method and path on stderr. Both exist for the same reason: a 404 is an
    answer to a real client, so an unimplemented endpoint reads as a
    negative result rather than as an error.
  - Seeding takes a per-repo default branch, since a template whose branch
    is master is graded at that ref by name.

* feat(gh): a GitHub CLI, and the write routes the fake needs behind it

The `github` mount is the read half -- a repository is a tree, so listing
and reading it is `ls` and `cat`. There was no write half at all:
GITHUB_IO carries readdir/read/stat where GDRIVE_IO carries the full set,
so nothing could commit a file, and forking and renaming are account
operations a filesystem has no shape for in any case.

`gh` is that half, spelled as cli.github.com spells it:

    gh repo view [OWNER/REPO]
    gh repo fork OWNER/REPO [--fork-name NAME]
    gh repo rename NEW-NAME -R OWNER/REPO
    gh api ENDPOINT [-X METHOD] [-f key=value] [-F key=value]

`rename` takes the new name as the operand and the repository as -R, which
is the reverse of what the shape of the line suggests and is upstream's.
`-f` sends a string and `-F` reads true/false/null/integers as their JSON
types, which is gh's own --raw-field / --field split; a call with no fields
is a GET, one with fields a POST unless -X says otherwise. Committing a
file is `gh api ... -X PUT -f content="$(base64 -w0 f)"`, which is what
real gh and real GitHub require of each other -- there is no file verb to
mimic.

The transport was GET-only, so it grows a `request`, and an empty body (204,
an empty 202) decodes to null rather than throwing on a call that worked.

Fake, in support: DELETE /contents with the same sha rule as the replace,
and GET /commits/{ref}, which is not /git/commits/{sha} -- it takes a branch
name and reports the file list. Both were found by the unrouted logger
during a run: an agent issued 35 deletes, every one 404'd, and it reported
the files as removed, because to a real client a 404 is an answer.

* feat(integ/github): repository metadata, star-sorted search and readme

A task that picks *between* repositories reads their descriptions and star
counts and never opens one, so seeding gains a second mode: --metadata takes
a JSON file keyed by owner/name and creates repositories that are metadata
only. The tree and contents endpoints answer 404 for them, which is what
GitHub says about a repository with no files.

search/repositories honours sort=stars and matches the description as well
as the name, since a repository is found by what it says it does at least as
often as by what it is called. Terms OR rather than AND, which is looser
than GitHub and errs towards showing a caller the row it wants.

GET /repos/{owner}/{repo}/readme, found by the unrouted logger during a run.

* feat(integ/github): branches

The fake kept exactly one branch, which rules out every task whose premise
is a difference between two of them. FakeRepo now holds a file map and a
commit list per branch:

  - `files` and `commits` stay bound to the default branch, so every route
    that does not name a ref reads what it always read.
  - `branch_for` resolves a ref -- a branch name, HEAD, or a commit sha
    belonging to one branch's history -- and `tree_of` returns that
    branch's files, so a bad ref is a 404 rather than a silent read of the
    default branch.
  - contents, tree, raw, readme, branches, branch, commits, commit and
    git/ref are ref-aware; PUT and DELETE /contents write to the branch
    their body names; PATCH /git/refs moves the branch it names.
  - Each branch has its own synthetic root commit, so two branches of a
    fixture do not share a head.
  - A fork copies every branch, not just the default one.
  - Seeding takes `into=<branch>`, so a two-branch fixture is two
    directories and two --repo flags naming one repository.

The default-branch binding is what keeps this small: 18 call sites read
`repo.files` and none of them had to change.

* feat(integ/github): GET /git/refs/{prefix}

The plural is a different endpoint from the singular: git/ref/<full-ref>
returns one object, git/refs/<prefix> a list of everything beneath it. A
caller picks whichever it expects, so serving only the singular made the
plural read as 'no such ref'. Found by the unrouted logger during a run.

* feat(integ/github): account repo listing, bare /contents, mirror-stable shas

Three things a task that reasons across several repositories needs.

GET /users/{login}/repos and GET /user/repos. A task that asks "is there a
repository for this on my GitHub" answers it here, and a 404 read as "the
account has none" -- a wrong answer rather than an error, which is the
failure mode the unrouted logger exists for.

GET /repos/{owner}/{repo}/contents, with no trailing slash. GitHub serves
both spellings of the root and a caller picks either; only one was routed.

A branch's root commit sha is now derived from its content rather than from
the repository's name. That is what `git clone --mirror` followed by a push
actually gives you -- the same shas -- and a grader that reads an initial
sha from an upstream repository and the latest from a local copy, then asks
what changed between them, depends on it. Two branches still differ exactly
when their trees differ.

* feat(integ/github): POST /git/refs, so a branch can be created

The last write a task can ask for that the fake had no answer to. A new
branch starts as a copy of whatever the base sha resolves to -- which is
what a branch is, another name for one commit and everything reachable from
it -- with its own empty history, so the two do not share future commits.

422 on a ref that is not under refs/heads/, on a name that already exists,
and on a base sha that resolves to nothing.

* feat(integ/github): GET /, the API root

github.com answers the root with a map of endpoint URL templates. Serving
only the endpoints beneath it meant a client probing the root got a 404,
which reads as the host not being there at all rather than as an unfamiliar
API. Found by the unrouted logger during a run.

* feat(integ/github): seed a branch's commit history, not just its tree

Seeding a directory filled a tree and nothing else, so every seeded
branch answered GET /commits with one synthetic root. That is enough for
a task that reads files and wrong for one that asks when something
arrived. `task-tracker` wants the tasks added by the *most recent*
commit on each of fifteen developer branches, and each branch has five
or six -- so a history of one does not make the task hard, it makes it
unanswerable while looking answerable.

--commits owner/name=<file> reads a manifest keyed by branch, each an
array oldest first, and is applied after every tree: a manifest names
branches, and a branch no directory was seeded into would otherwise be
created here as an empty tree with a history, which reads as a branch
whose files were all deleted.

Shas are derived from full_name, branch, date and message rather than
carried, for the same reason the root's is derived from its tree: a
fixture has to answer the same way twice.

Commits now carry commit.author/committer (name, email, date) and
author.login, which is what a client sorting or filtering by date reads,
and the synthetic root carries them too against a fixed epoch -- it
stands for everything before the fixture rather than for a moment, so
"now" would make it sort ahead of the commits it precedes.

GET /commits/{ref} now renders `files` as {filename, status} objects the
way GitHub does. The list endpoint keeps paths, which is also GitHub's
split: `files` is absent from a commit in a list and detailed when one
commit is asked for. _commit_files is shared with the compare route,
which was building the same objects inline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(github): back the transport with octokit

Wraps @octokit/core behind the unchanged GitHubTransport, so the github
mount and the gh CLI both gain retry and throttling. Handles three
octokit behaviours: {} in a url is a route template that silently eats
the segment, the "METHOD /path" form splits on whitespace, and 204
decodes to '' rather than null. The write throttle is github.com's own
secondary rate limit, so it is off for any other host.

* fix(gh): name the mount its writes invalidate, and parse the host segment

serves was empty, so the executor's post-write cache drop did nothing and
a committed file still read back as its pre-write bytes. parseRepo took
the first two segments of [HOST/]OWNER/REPO, so github.com/acme/tools
resolved to github.com/acme -- a different repository, reported as
success.

* fix(cache): invalidate the index instead of clearing it

A cleared index reads exactly like one that was never filled, so github --
whose index is the whole listing rather than a cache in front of one --
could not tell a cache drop from an empty repository, and ls reported the
mount root missing. IndexCacheStore grows invalidate(), which expires
entries in place, and github refetches once on an EXPIRED lookup. Also
fixes a latent bug with no CLI involved: after the 24h index TTL lapsed a
github mount answered ls with exit 0 and no output. Redis cannot express
stale (an expired key is gone), so it still clears; the comment says why.

* feat(gh): a python gh CLI, mirroring the typescript one

Same four verbs and the same grammar. The python client was GET-only, so
it grows github_request alongside github_get. Layout divergence 305 -> 303.

* test(integ): a gh CLI battery on both hosts

15 cases on a new cli-gh target, covering the write-then-read path through
the mount, the refusals, and the reversed rename grammar. The fake grows
POST /reset, because the cli facet runs both hosts against one process and
writes would otherwise leak between them; it also stops rejecting wrapped
base64, which real GitHub accepts. The unrouted-request logger is now a CI
failure rather than a log line.

* fix(github): probe the parent listing, so an update is not read back stale

The index tracks freshness per directory, never per entry, so get()
never answers EXPIRED and the refill it guarded was unreachable: after a
write invalidated the index the blob's row survived carrying the
pre-write sha, and the read served the old bytes. Create and delete went
through readdir and worked; an update did not.

Readdir on the TypeScript side refilled on any miss, which spent a full
recursive-tree call on every ENOENT and diverged from python. Both now
refill only on EXPIRED.

* test(cache): pin what invalidate keeps that clear discards

A cleared store reads exactly like one that was never filled, which is
what made a github mount report an empty repository. Assert the two
answer differently.

* fix(spec): render an operand's declared name outside the clap dialect

A named slot printed as <text>, so 'man gh api' said <text> where gh
says <endpoint>. argparse prints the dest too. gh is the only spec this
changes; ntn is clap and already rendered it.

* test(integ): cover the gh surface the battery missed

Update-then-read is the case that found the stale-read bug. Also the
help and discovery surface, a leading-slash endpoint, typed fields on a
GET, and a missing endpoint.

* docs(gh): a page per host for the gh CLI

Install, the mount-reads/CLI-acts split, the api rules, and a table of
what differs from gh 2.85.

* fix(github): keep an api field from steering the request

Octokit reads loose parameters off the same object that carries url,
method and headers, so `gh api X -f url=...` retargeted the call
instead of sending the field. The query is spelled into the url and the
body travels as `data`. The python client already separated them.

* fix(cli): let a handler say it did not write

A leaf declares `write` statically because for almost every verb it is
static, but `gh api` carries its method on the line, so a read like
`gh api /user` expired every github mount the install serves. A result
may now report what the spec cannot know.

* feat(gh): expand the owner, repo and branch placeholders

gh's own examples are written with them, so `gh api
repos/{owner}/{repo}/releases` asked for literal braces and 404'd. An
install's repo and branch stand in for the current checkout. Any other
brace pair still reaches the wire, which is gh's behavior too.

* feat(gh): render repo view the way gh renders it

A name line, a description line, then the README, with the separator
omitted when there is none. Probed against gh 2.85. The REST object is
still one `gh api repos/OWNER/REPO` away.

* test(integ): cover placeholders and the new repo view

Also carries a branch on the gh install, which is what {branch}
expands from.

* fix(gh): declare branch on the GhConfig type, and type the test trees

tsc runs in CI but not in pre-commit: GhConfig's zod schema grew
`branch` while its interface did not, and the new github tests built
tree items with a `mode` key the type has no room for.

* refactor(gh): derive GhConfig from its schema instead of declaring it twice

The zod schema is the one doing real work: it validates an install's
config and carries the secretStr marker redaction reads. A hand-written
interface beside it only adds a shape that can drift, which is how
branch reached the schema and not the type.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 21:03:53 -07:00

166 lines
5.7 KiB
Python

# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
from collections import defaultdict
from datetime import datetime, timedelta, timezone
from unittest.mock import MagicMock
import pytest
import mirage.core.github.tree
from mirage.cache.index import IndexEntry
from mirage.cache.index.ram import RAMIndexCacheStore
from mirage.core.github.readdir import readdir
from mirage.core.github.tree_entry import TreeEntry
from mirage.types import PathSpec
def _index_from_tree(tree: dict[str, TreeEntry]) -> RAMIndexCacheStore:
index = RAMIndexCacheStore()
dirs: dict[str, list[tuple[str, IndexEntry]]] = defaultdict(list)
for path, entry in tree.items():
parts = path.rsplit("/", 1)
if len(parts) == 2:
parent, name = "/" + parts[0], parts[1]
else:
parent, name = "/", parts[0]
resource_type = "folder" if entry.type == "tree" else "file"
idx_entry = IndexEntry(
id=entry.sha,
name=name,
resource_type=resource_type,
size=entry.size,
)
dirs[parent].append((name, idx_entry))
for parent, entries in dirs.items():
index._entries.update({
("/" + parent.strip("/") + "/" + name).replace("//", "/"):
e
for name, e in entries
})
child_keys = sorted(
("/" + parent.strip("/") + "/" + name).replace("//", "/")
for name, _ in entries)
index._children[parent] = child_keys
index._expiry[parent] = datetime.now(
timezone.utc) + timedelta(days=365)
return index
@pytest.fixture
def tree():
return {
"src":
TreeEntry(path="src", type="tree", sha="aaa", size=None),
"src/main.py":
TreeEntry(path="src/main.py", type="blob", sha="bbb", size=120),
"src/utils":
TreeEntry(path="src/utils", type="tree", sha="ccc", size=None),
"src/utils/helpers.py":
TreeEntry(path="src/utils/helpers.py", type="blob", sha="ddd",
size=80),
"README.md":
TreeEntry(path="README.md", type="blob", sha="eee", size=50),
}
@pytest.mark.asyncio
async def test_readdir_root(tree):
index = _index_from_tree(tree)
result = await readdir(
None, PathSpec(resource_path="", virtual="/", directory="/"), index)
assert result == ["/README.md", "/src"]
@pytest.mark.asyncio
async def test_readdir_subdirectory(tree):
index = _index_from_tree(tree)
result = await readdir(
None, PathSpec(resource_path="src", virtual="/src", directory="/src"),
index)
assert result == ["/src/main.py", "/src/utils"]
@pytest.mark.asyncio
async def test_readdir_nested(tree):
index = _index_from_tree(tree)
result = await readdir(
None,
PathSpec(resource_path="src/utils",
virtual="/src/utils",
directory="/src/utils"), index)
assert result == ["/src/utils/helpers.py"]
@pytest.mark.asyncio
async def test_readdir_missing_directory(tree):
index = _index_from_tree(tree)
accessor = MagicMock()
accessor.truncated = False
with pytest.raises(FileNotFoundError):
await readdir(
accessor,
PathSpec(resource_path="nonexistent",
virtual="/nonexistent",
directory="/nonexistent"), index)
# The index *is* the listing here, seeded once from the recursive tree, so
# an expired one is a tree that aged out rather than a repository that
# emptied. Before the refill, `ls /repo` exited 0 with no output once the
# day-long TTL lapsed, and reported the mount root missing after a `gh`
# write invalidated it.
@pytest.mark.asyncio
async def test_readdir_refills_an_expired_index(tree, monkeypatch):
index = _index_from_tree(tree)
await index.invalidate()
calls = []
async def fake_fetch_tree(config, owner, repo, ref):
calls.append((owner, repo, ref))
return tree, False
monkeypatch.setattr(mirage.core.github.tree, "fetch_tree", fake_fetch_tree)
accessor = MagicMock()
accessor.truncated = False
result = await readdir(
accessor, PathSpec(resource_path="", virtual="/", directory="/"),
index)
assert result == ["/README.md", "/src"]
assert len(calls) == 1
# A miss against a live index is a real absence, so it must stay one call
# of nothing: refilling here would spend a full recursive-tree fetch on
# every ENOENT.
@pytest.mark.asyncio
async def test_readdir_does_not_refill_on_a_real_miss(tree, monkeypatch):
index = _index_from_tree(tree)
calls = []
async def fake_fetch_tree(config, owner, repo, ref):
calls.append((owner, repo, ref))
return tree, False
monkeypatch.setattr(mirage.core.github.tree, "fetch_tree", fake_fetch_tree)
accessor = MagicMock()
accessor.truncated = False
with pytest.raises(FileNotFoundError):
await readdir(
accessor,
PathSpec(resource_path="nonexistent",
virtual="/nonexistent",
directory="/nonexistent"), index)
assert calls == []