Files
strukto-ai--mirage/python/mirage/core/google/_client.py
T
Zecheng Zhang 67945abf63 feat(git): a git CLI over any mount, in both languages (#710)
* feat(git): a git CLI over any mount, in both languages

Adds `git` as a builtin CLI (issue #705 item 1): status, log, show,
diff, branch, add, reset, commit, checkout. It takes no config, and the
repository is read entirely through the mount ops, so a repo on RAM, on
disk or on an object store reads the same way. Python goes through
dulwich, TypeScript through isomorphic-git backed by a PromiseFsClient
over the dispatcher.

Status is pinned against the real git binary across 22 repository states
and 6 spellings, the mutation verbs are read back by real git and pass
`git fsck`, and the rename-similarity score matches dulwich digit for
digit. Both run in integ on `git-ram` and `git-disk` in both languages.

Also fixes the grep family this leaned on:

- grep reads a basic regular expression by default, as POSIX says, with
  -G for the default and -E for extended. `grep -l` and `grep -rl`
  compiled their own pattern and still read extended.
- zgrep does the same. Its -E flag was read and discarded.
- an operand grep could not search exits 2, as GNU does, instead of
  being flattened to 1.
- `grep -l` on a directory reports it rather than walking it. The
  shared fallback called the recursive walk whenever a failed read
  turned out to be a directory, which made -l behave like -rl.

Supporting pieces: ranged reads (`read_range`) with a read-and-slice
fallback and a seeking implementation for disk, PATH-typed group options
resolved against the working directory, and git's three unknown-option
dialects.

* fix(rg): report an unreadable operand as exit 2, like ripgrep

CI caught four goldens the grep change had missed, all outside the JSON
harness targets I had been running, plus a divergence the change itself
introduced.

The divergence: TypeScript routes rg through grepGeneric, so rg moved to
exit 2 there while python's rg, which has its own generic, stayed at 1.
Real ripgrep exits 2 for an operand it could not read, so TypeScript was
right and python is brought up to it rather than the other way round.
Its single-operand path also let the error escape to the shared handler,
which flattens every OSError to exit 1; rg now reports it itself, the
same fix grep needed.

The rule is now stated once in grep_helper (`exit_code_for` /
`exitCodeFor`) and imported by both generics in both languages, instead
of living in the grep generic where rg could not reach it.

Goldens updated: integ/cross_commands.{py,ts}, the shared observability
contract, langfuse and dify. Each pinned exit 1 for a missing operand
and carried a comment calling it a deliberate divergence.

* fix(git): refuse the three mutations that could lose work

Three review findings, each measured against git 2.50.1 before fixing.

checkout only compared tracked paths, so a branch holding a file the
working tree has untracked wrote its blob straight over it. The file is
in no index and no tree, so nothing could see it and nothing could get
it back. The untracked set is now part of the conflict check, which
needs UNTRACKED_ALL rather than the mode status uses: "normal" collapses
a wholly untracked directory to one row, and git names the file inside
it. An ignored file stays overwritable, which is git's own split. When
both kinds of conflict apply git prints both paragraphs and aborts once,
so CheckoutConflictError carries both lists.

branch -d deleted a branch HEAD does not contain, dropping the only name
pointing at those commits. It now refuses, and -D is added, because -d
alone would be a delete with no way to say no. dulwich's can_fast_forward
answers exactly this and cannot be used: it asks the repository for its
grafts and shallow boundary and a bare BaseRepo raises. Walker is what
log already walks with and needs only the object store.

add -u ignored its pathspecs and restaged every tracked file, which is
how an unrelated edit reaches the next commit. git tells two misses
apart and so does this now: a pathspec naming nothing is a fatal about
the pathspec, one naming an untracked file is a fatal about git not
knowing it, both exit 128.

Also two CI failures. The spec dumps were stale for the -G flag added
with the zgrep BRE fix. And the git fixture exported its identity as
environment variables, which reach only its own commits, so a test that
committed into the built repository failed with "Author identity
unknown" on any runner with no global identity. It now records the same
identity in the repository, leaving the object ids unchanged.

Covered by unit tests in both languages and six integ cases across
git-ram and git-disk on both hosts.

* fix(test): the redis missing-file suite still pinned grep at exit 1

Sibling of the disk suite, which was updated with the exit-2 change. This
one needs a live server, so it skips silently without REDIS_URL and the
local run never reached it. Only the grep case moves: cat, head, tail and
wc all still exit 1 for an operand they could not read, which is the
split the comment now records here as it does next door.

* fix(grep): read an operand's type from stat, not from how the read failed

Four integ jobs caught this across seven backends, and it is one mistake
with three faces. Classifying a directory operand *after* a failed read
makes the answer depend on what each backend does about reading one, and
they disagree: s3, gridfs, hf and nextcloud read a directory path without
complaint and hand back nothing, and ssh raises an asyncssh SFTP error
that is not an OSError at all, so it escaped the catch entirely and
exited 1 with an unattributed "grep: Is a directory". So the operand is
now stat-ed before it is read, which is what GNU does and what the -r
branch of the same function already did.

operand_is_directory had its own version of the same error: a readdir
that did not raise counted as a directory, and a prefix store answers
readdir for any path at all, returning nothing for one that is not
there. Every missing file on those backends therefore read as a
directory. The listing must now be non-empty to count, which costs a
genuinely empty directory being invisible there, the same divergence du
already documents and the safer way round.

The catch also widens from three exception types to WALK_ERRORS, the
tuple every other operand-tolerant walk in the repo uses, so an errno
split cannot make grep abort where tree and grep -r keep going.

One test fake resolved stat with `undefined as never` and only worked
while nothing read the value. It resolves with a real FileStat now,
which is what the postgres backend answers there; the assertion the test
exists for is untouched.

Verified on gridfs and gridfs-prefix, which reproduce the prefix-store
half locally, plus ram, disk and redis for regressions: all five targets
on both hosts, 0 failed.

* fix(git): three review findings on -C, checkout -b and reset

`-C` accepted any path that was not missing, so naming a file inside a
repository walked up and ran in the parent instead of failing the way
git's chdir does. For a write verb that means mutating a repository the
caller never named. It now refuses a non-directory with git's own second
wording, "Not a directory".

`checkout -b <new> <start>` forced the new branch to HEAD and dropped the
start point without a word, so every commit after it landed on the wrong
history. The operand is honored, and a start point that is not a commit
gets git's sentence naming both it and the branch rather than the
generic "ambiguous argument" the same lookup failure produces elsewhere.

`reset <operand>` that selected no path unstaged nothing and exited 0,
which a script reads as "the index was reset". Two different mistakes
reach that point and they now get different fatals. A typo is git's
"ambiguous argument". A revision is not: real git resets the index to any
commit named there, measured on 2.50.1, so the review's premise that git
refuses one is wrong. This build resets from HEAD only, and says which
feature is missing instead of claiming a revision it can resolve is
unknown. Recorded as a divergence in both git.mdx files.

Six integ cases across git-ram and git-disk on both hosts, plus unit
tests in both languages.

* chore: merge main and regenerate the grep and zgrep specs

#713 landed a `pair` field on the option spec while this branch was
open. CI builds the PR merged with main, so it regenerated specs
carrying the field and compared them against grep.json and zgrep.json,
the two files this branch had already rewritten for `-G`, which
therefore predate it. Every other spec file came across from main
already carrying it.

The regeneration also needs a rebuilt dist: gen-specs.ts reads the built
package, and against a stale one it dies on an Operand field it does not
know, which silently leaves the typescript half unregenerated and shows
up as a python/typescript parity divergence rather than as a build
error.
2026-08-05 13:38:03 -07:00

229 lines
7.5 KiB
Python

# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import asyncio
import time
from typing import Any
import aiohttp
from mirage.core.google.config import GoogleConfig
from mirage.resource.secrets import reveal_secret
TOKEN_URL = "https://oauth2.googleapis.com/token"
DRIVE_API_BASE = "https://www.googleapis.com/drive/v3"
DOCS_API_BASE = "https://docs.googleapis.com/v1"
SLIDES_API_BASE = "https://slides.googleapis.com/v1"
SHEETS_API_BASE = "https://sheets.googleapis.com/v4"
GMAIL_API_BASE = "https://gmail.googleapis.com/gmail/v1"
DRIVE_UPLOAD_BASE = "https://www.googleapis.com/upload/drive/v3"
TOKEN_BUFFER_SECONDS = 300
def token_url(config: GoogleConfig) -> str:
return f"{config.api_base}/token" if config.api_base else TOKEN_URL
def drive_base(token_manager: "TokenManager") -> str:
base = token_manager.config.api_base
return f"{base}/drive/v3" if base else DRIVE_API_BASE
def drive_upload_base(token_manager: "TokenManager") -> str:
base = token_manager.config.api_base
return f"{base}/upload/drive/v3" if base else DRIVE_UPLOAD_BASE
def docs_base(token_manager: "TokenManager") -> str:
base = token_manager.config.api_base
return f"{base}/v1" if base else DOCS_API_BASE
def slides_base(token_manager: "TokenManager") -> str:
base = token_manager.config.api_base
return f"{base}/v1" if base else SLIDES_API_BASE
def sheets_base(token_manager: "TokenManager") -> str:
base = token_manager.config.api_base
return f"{base}/v4" if base else SHEETS_API_BASE
def gmail_base(token_manager: "TokenManager") -> str:
base = token_manager.config.api_base
return f"{base}/gmail/v1" if base else GMAIL_API_BASE
async def refresh_access_token(config: GoogleConfig, ) -> tuple[str, int]:
"""Exchange refresh token for a new access token.
Args:
config (GoogleConfig): OAuth2 credentials.
Returns:
tuple[str, int]: (access_token, expires_in_seconds)
"""
data = {
"client_id": config.client_id,
"refresh_token": reveal_secret(config.refresh_token),
"grant_type": "refresh_token",
}
client_secret = reveal_secret(config.client_secret)
if client_secret:
data["client_secret"] = client_secret
async with aiohttp.ClientSession() as session:
async with session.post(token_url(config), data=data) as resp:
resp.raise_for_status()
body = await resp.json()
return body["access_token"], body["expires_in"]
class TokenManager:
"""Manages OAuth2 access token lifecycle."""
def __init__(self, config: GoogleConfig) -> None:
self.config = config
self._access_token: str | None = None
self._expires_at: float = 0
self._lock = asyncio.Lock()
async def get_token(self) -> str:
async with self._lock:
if self._access_token and time.time() < self._expires_at:
return self._access_token
token, expires_in = await refresh_access_token(self.config)
self._access_token = token
self._expires_at = (time.time() + expires_in -
TOKEN_BUFFER_SECONDS)
return self._access_token
async def google_headers(token_manager: TokenManager, ) -> dict[str, str]:
token = await token_manager.get_token()
return {"Authorization": f"Bearer {token}"}
async def google_get(
token_manager: TokenManager,
url: str,
params: dict[str, Any] | None = None,
) -> dict[str, Any]:
headers = await google_headers(token_manager)
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, params=params) as resp:
resp.raise_for_status()
return await resp.json()
async def google_post(
token_manager: TokenManager,
url: str,
json: dict[str, Any],
) -> dict[str, Any]:
headers = await google_headers(token_manager)
async with aiohttp.ClientSession() as session:
async with session.post(url, headers=headers, json=json) as resp:
resp.raise_for_status()
return await resp.json()
async def google_put(
token_manager: TokenManager,
url: str,
json: dict[str, Any],
) -> dict[str, Any]:
headers = await google_headers(token_manager)
async with aiohttp.ClientSession() as session:
async with session.put(url, headers=headers, json=json) as resp:
resp.raise_for_status()
return await resp.json()
async def google_patch(
token_manager: TokenManager,
url: str,
json: dict[str, Any],
params: dict[str, str] | None = None,
) -> dict[str, Any]:
headers = await google_headers(token_manager)
async with aiohttp.ClientSession() as session:
async with session.patch(url,
headers=headers,
json=json,
params=params) as resp:
resp.raise_for_status()
return await resp.json()
async def google_send_bytes(
token_manager: TokenManager,
method: str,
url: str,
data: bytes,
content_type: str,
params: dict[str, str] | None = None,
) -> dict[str, Any]:
"""Send a raw byte payload (upload endpoints) and return the JSON reply.
Args:
token_manager (TokenManager): OAuth2 token manager.
method (str): HTTP method ("POST" or "PATCH").
url (str): API URL.
data (bytes): request body.
content_type (str): Content-Type header for the body.
params (dict | None): query parameters.
"""
headers = await google_headers(token_manager)
headers["Content-Type"] = content_type
async with aiohttp.ClientSession() as session:
async with session.request(method,
url,
headers=headers,
data=data,
params=params) as resp:
resp.raise_for_status()
return await resp.json()
async def google_delete(
token_manager: TokenManager,
url: str,
) -> None:
headers = await google_headers(token_manager)
async with aiohttp.ClientSession() as session:
async with session.delete(url, headers=headers) as resp:
resp.raise_for_status()
async def google_get_bytes(
token_manager: TokenManager,
url: str,
range_header: str | None = None,
) -> bytes:
"""GET a URL as raw bytes, optionally only a byte range of it.
Args:
token_manager (TokenManager): OAuth2 token manager.
url (str): API URL.
range_header (str | None): an HTTP ``Range`` value, or None for
the whole body.
"""
headers = await google_headers(token_manager)
if range_header:
headers["Range"] = range_header
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers) as resp:
resp.raise_for_status()
return await resp.read()