feat(resource): a mount can name its resource class instead of a registry key

resource: ./wiki.py:WikiResource now resolves the way a clis entry's cli
value already did, in both languages, with a relative path rebased on the
config file's directory. A registry name still wins, so a name cannot be
reread as code. TypeScript honors a static create ahead of the
constructor, which is how a backend needing IO at setup is spelled there.
This commit is contained in:
Zecheng Zhang
2026-08-22 01:07:30 -07:00
parent b7000945e6
commit e4d6866336
10 changed files with 372 additions and 51 deletions
+13 -1
View File
@@ -64,7 +64,19 @@ or ship it as a normal package with an entry point — discovered automatically
jira = "mypackage.backends:JiraResource"
```
The entry point resolves to the resource class; declare a `CONFIG_CLS` class attribute when the constructor takes a typed config. See `examples/python/other/custom_resource.py` for a complete runnable backend in one file.
The entry point resolves to the resource class; declare a `CONFIG_CLS` class attribute when the constructor takes a typed config.
Neither step is needed to mount from YAML. A `resource` value carrying a colon names the class directly, the same way a `clis` entry's `cli` value names a spec tree, so a deployment can point at a file next to the config or at a class inside an installed package:
```yaml
mounts:
/jira:
resource: ./jira.py:JiraResource
/wiki:
resource: mypackage.backends:WikiResource
```
A relative path resolves against the config file's directory, not the server's working directory. A registry name always wins over a reference, so a name can never be reread as code. See `examples/python/other/custom_resource.py` for a complete runnable backend in one file.
## Contribute a Builtin Resource
+12
View File
@@ -74,6 +74,18 @@ registerResourceFactory('jira', (config) => Promise.resolve(new JiraResource(con
The registry takes a factory rather than a class because a browser backend is often reached through a dynamic import; `buildResource('jira', config)` then works exactly as it does for a builtin.
Registering is not needed to mount from config. A `resource` value carrying a colon names the class directly, the same way a `clis` entry's `cli` value names a spec tree, so a deployment can point at a file next to the config or at a class inside an installed package:
```yaml
mounts:
/jira:
resource: ./jira.mjs:JiraResource
/wiki:
resource: my-backends:WikiResource
```
A relative path resolves against the config file's directory, not the server's working directory; a bare specifier is Node's to resolve, so a package name is left alone. A registry name always wins over a reference, so a name can never be reread as code. A `static async create` is honored ahead of the constructor, which is how a backend whose setup needs I/O is spelled here.
Snapshots are the one place the registry does not reach: `Workspace.load` builds its mounts before any factory is consulted, so a saved custom mount has to be handed back explicitly, as `Workspace.load(state, { resources: { '/jira/': new JiraResource(cfg) } })`. `GenericResource` says so in its own state, which is what turns a forgotten override into a refusal to load rather than a mount that comes back empty. `Workspace.copy` needs nothing, since it passes the live resource through.
See `examples/typescript/other/custom_resource.ts` for a complete runnable backend in one file, and `examples/python/other/custom_resource.py` for its Python twin. Both are asserted against the same truth file, so the two SDKs cannot drift.
+21 -14
View File
@@ -394,7 +394,12 @@ def _absolutize_scripts(raw: dict[str, Any], base: Path) -> None:
for block in clis.values():
if isinstance(block, dict):
_absolutize_script_key(block, base)
_absolutize_cli_ref(block, base)
_absolutize_code_ref(block, "cli", base)
mounts = raw.get("mounts")
if isinstance(mounts, dict):
for block in mounts.values():
if isinstance(block, dict):
_absolutize_code_ref(block, "resource", base)
def _absolutize_script_key(entry: dict[str, Any], base: Path) -> None:
@@ -411,23 +416,25 @@ def _absolutize_script_key(entry: dict[str, Any], base: Path) -> None:
entry["script"] = str(base / script.strip())
def _absolutize_cli_ref(entry: dict[str, Any], base: Path) -> None:
"""Rebase one ``clis`` entry's path-form ``cli`` reference.
def _absolutize_code_ref(entry: dict[str, Any], key: str, base: Path) -> None:
"""Rebase a path-form colon reference under ``key``.
``cli: ./tool.py:TREE`` means "next to the config file", the same
build-context rule ``script:`` follows; without this the pointer
reaches ``load_attr`` relative and resolves against the server
process's cwd. A module dotpath (``pkg.mod:TREE``) is left alone:
importlib resolves it, not the filesystem. The split matches
``load_attr``'s own test, so the two cannot disagree about what a
path is.
``cli: ./tool.py:TREE`` and ``resource: ./wiki.py:WikiResource`` both
mean "next to the config file", the same build-context rule
``script:`` follows; without this the pointer reaches ``load_attr``
relative and resolves against the server process's cwd. A module
dotpath (``pkg.mod:TREE``) is left alone: importlib resolves it, not
the filesystem. The split matches ``load_attr``'s own test, so the
two cannot disagree about what a path is.
Args:
entry (dict[str, Any]): a ``clis`` mapping entry, mutated in
place.
entry (dict[str, Any]): a ``clis`` or ``mounts`` mapping entry,
mutated in place.
key (str): the field holding the reference, ``cli`` or
``resource``.
base (Path): directory containing the config file.
"""
ref = entry.get("cli")
ref = entry.get(key)
if not isinstance(ref, str) or ":" not in ref:
return
source, attr = ref.rsplit(":", 1)
@@ -435,7 +442,7 @@ def _absolutize_cli_ref(entry: dict[str, Any], base: Path) -> None:
return
if Path(source).is_absolute():
return
entry["cli"] = f"{base / source}:{attr}"
entry[key] = f"{base / source}:{attr}"
def _build_runtime_entries(
+37 -8
View File
@@ -258,6 +258,34 @@ def resolve_class(ref: str | type) -> type:
return ref if isinstance(ref, type) else load_backend_class(ref)
def _resolve_entry(name: str) -> ResourceEntry | None:
"""Find the entry a mount's ``resource`` value names, or None.
Four rungs, in the order ``commands.cli.specs.cli_spec_for`` uses for
a ``cli`` value, because the two are the same question asked of two
tiers: builtin, explicitly registered, a colon reference naming code
directly, then ``mirage.resources`` entry points.
The colon rung needs no loader of its own. ``resolve_class`` already
reads a ``"source:ClassName"`` string, so the reference becomes an
ordinary entry with no config class, which means an out-of-tree class
carrying ``CONFIG_CLS`` gets its typed config built exactly as a
builtin's does. It is tried before the entry points because a colon
is unambiguous: the value names code, so there is nothing to discover
and no reason to pay for a scan of every installed package.
Args:
name (str): the mount's ``resource`` value.
"""
entry = REGISTRY.get(name) or _CUSTOM.get(name)
if entry is not None:
return entry
if ":" in name:
return ResourceEntry(name, None)
_load_entry_point_resources()
return _CUSTOM.get(name)
def build_resource(name: str,
config: dict[str, Any] | None = None) -> "BaseResource":
"""Construct a resource instance by its registry name.
@@ -266,7 +294,10 @@ def build_resource(name: str,
importing this module does not pull in every resource's
dependencies. Only the resources actually used get loaded. Lookup
order: builtin ``REGISTRY``, then :func:`register_resource` names,
then ``mirage.resources`` entry points from installed packages.
then a colon reference naming a class directly
(``./wiki.py:WikiResource`` or ``mypkg.backends:WikiResource``), then
``mirage.resources`` entry points from installed packages. See
:func:`_resolve_entry`.
**Synchronous on purpose. Do not make this async.** It is the door
every caller who describes a mount as data comes through: the YAML
@@ -289,7 +320,8 @@ def build_resource(name: str,
``static async create``.
Args:
name (str): registry key such as ``"s3"`` or ``"ram"``.
name (str): registry key such as ``"s3"`` or ``"ram"``, or a
colon reference such as ``"./wiki.py:WikiResource"``.
config (dict | None): kwargs for the resource's ``Config``
class when one exists; otherwise raw resource kwargs
(e.g. ``{"root": "/tmp"}`` for ``"disk"``).
@@ -298,13 +330,10 @@ def build_resource(name: str,
BaseResource: a fresh resource instance.
Raises:
KeyError: ``name`` is neither builtin, registered, nor
installed.
KeyError: ``name`` is neither builtin, registered, a colon
reference, nor installed.
"""
entry = REGISTRY.get(name)
if entry is None:
_load_entry_point_resources()
entry = _CUSTOM.get(name)
entry = _resolve_entry(name)
if entry is None:
raise KeyError(
f"unknown resource {name!r}; known: {known_resources()}")
+50
View File
@@ -684,6 +684,56 @@ clis:
assert ref == f"{tmp_path / 'tool.py'}:TREE"
@pytest.mark.asyncio
async def test_mounts_path_form_resource_rebases_on_the_config_dir(
tmp_path, monkeypatch):
# `resource: ./wiki.py:WikiResource` reads the same way `cli:` does,
# so it follows the same build-context rule.
(tmp_path / "wiki.py").write_text("""\
from mirage.resource.ram.ram import RAMResource
class WikiResource(RAMResource):
pass
""")
cfg_file = tmp_path / "ws.yaml"
cfg_file.write_text("""\
mounts:
/wiki:
resource: ./wiki.py:WikiResource
""")
monkeypatch.chdir(tmp_path.parent)
cfg = load_config(cfg_file)
assert cfg.mounts[
"/wiki"].resource == f"{tmp_path / 'wiki.py'}:WikiResource"
mount = cfg.to_workspace_kwargs()["resources"]["/wiki"]
assert type(mount.resource).__name__ == "WikiResource"
@pytest.mark.asyncio
async def test_mounts_module_dotpath_resource_is_left_alone(tmp_path):
cfg_file = tmp_path / "ws.yaml"
cfg_file.write_text("""\
mounts:
/wiki:
resource: mypkg.backends:WikiResource
""")
cfg = load_config(cfg_file)
assert cfg.mounts["/wiki"].resource == "mypkg.backends:WikiResource"
@pytest.mark.asyncio
async def test_mounts_registry_name_is_left_alone(tmp_path):
cfg_file = tmp_path / "ws.yaml"
cfg_file.write_text("""\
mounts:
/data:
resource: ram
""")
cfg = load_config(cfg_file)
assert cfg.mounts["/data"].resource == "ram"
@pytest.mark.asyncio
async def test_clis_module_dotpath_reference_is_left_alone(tmp_path):
# importlib resolves a dotpath, not the filesystem, so rebasing it
+26
View File
@@ -206,6 +206,32 @@ def test_register_resource_spec_string(clean_registry):
assert built.root == "/spec"
def test_colon_reference_builds_without_a_registry(clean_registry):
# A colon means the value names code, so it resolves with nothing
# registered and no entry points scanned.
built = build_resource("tests.resource.test_registry:FakeKwargsResource",
{"root": "/ref"})
assert isinstance(built, FakeKwargsResource)
assert built.root == "/ref"
def test_colon_reference_uses_the_config_cls_attribute(clean_registry):
built = build_resource(
"tests.resource.test_registry:FakeConfigClsResource",
{"url": "http://ref"})
assert built.config.url == "http://ref"
def test_colon_reference_does_not_shadow_a_builtin_name(clean_registry):
# A registry name always wins, so a name can never be reread as code.
assert type(build_resource("ram")).__name__ == "RAMResource"
def test_colon_reference_to_a_missing_attribute_raises(clean_registry):
with pytest.raises(ValueError):
build_resource("tests.resource.test_registry:NoSuchResource")
def test_known_resources_includes_custom(clean_registry):
register_resource("fake_custom", FakeCustomResource, FakeCustomConfig)
names = known_resources()
@@ -724,6 +724,63 @@ describe('clis section', () => {
})
})
// Mirrors python/tests/config/test_loader.py's mounts `resource:` cases.
// A `resource` value carrying a colon names a class the same way `cli:`
// names a spec, which is what lets a deployment mount its own backend
// from YAML without registering a factory in a host program.
describe('mounts resource: reference', () => {
const CORE_RES = pathToFileURL(
resolve(fileURLToPath(import.meta.url), '../../../core/dist/index.js'),
).href
const BACKEND =
`import {RAMResource} from ${JSON.stringify(CORE_RES)}\n` +
'export class WikiResource extends RAMResource {}\n'
it('builds a resource out of a file next to the config', async () => {
const dir = mkdtempSync(join(tmpdir(), 'mirage-res-'))
writeFileSync(join(dir, 'wiki.mjs'), BACKEND)
writeFileSync(
join(dir, 'ws.yaml'),
'mounts:\n /wiki:\n resource: ./wiki.mjs:WikiResource\n',
)
const cfg = loadWorkspaceConfigFile(join(dir, 'ws.yaml'))
const args = await configToWorkspaceArgs(cfg)
expect(args.resources['/wiki']?.[0]?.constructor.name).toBe('WikiResource')
rmSync(dir, { recursive: true, force: true })
})
it('rebases a relative ref onto the config file directory', () => {
const dir = mkdtempSync(join(tmpdir(), 'mirage-res-'))
writeFileSync(join(dir, 'wiki.mjs'), BACKEND)
writeFileSync(
join(dir, 'ws.yaml'),
'mounts:\n /wiki:\n resource: ./wiki.mjs:WikiResource\n',
)
const cfg = loadWorkspaceConfigFile(join(dir, 'ws.yaml'))
expect(cfg.mounts['/wiki']?.resource).toBe(`${join(dir, 'wiki.mjs')}:WikiResource`)
rmSync(dir, { recursive: true, force: true })
})
it('leaves a package specifier alone', () => {
const dir = mkdtempSync(join(tmpdir(), 'mirage-res-'))
writeFileSync(
join(dir, 'ws.yaml'),
'mounts:\n /wiki:\n resource: my-backends:WikiResource\n',
)
const cfg = loadWorkspaceConfigFile(join(dir, 'ws.yaml'))
expect(cfg.mounts['/wiki']?.resource).toBe('my-backends:WikiResource')
rmSync(dir, { recursive: true, force: true })
})
it('leaves a registry name alone', () => {
const dir = mkdtempSync(join(tmpdir(), 'mirage-res-'))
writeFileSync(join(dir, 'ws.yaml'), 'mounts:\n /data:\n resource: ram\n')
const cfg = loadWorkspaceConfigFile(join(dir, 'ws.yaml'))
expect(cfg.mounts['/data']?.resource).toBe('ram')
rmSync(dir, { recursive: true, force: true })
})
})
// Mirrors python/tests/config/test_loader.py's `cli: ./tool.py:TREE`
// cases. A `cli` value carrying a colon points at code rather than
// naming a registered spec, which is what lets a deployment install its
+25 -17
View File
@@ -671,7 +671,12 @@ function absolutizeScripts(raw: Record<string, unknown>, base: string): void {
for (const block of Object.values(raw.clis)) {
if (!isPlainObject(block)) continue
absolutizeScriptKey(block, base)
absolutizeCliRef(block, base)
absolutizeCodeRef(block, 'cli', base)
}
}
if (isPlainObject(raw.mounts)) {
for (const block of Object.values(raw.mounts)) {
if (isPlainObject(block)) absolutizeCodeRef(block, 'resource', base)
}
}
}
@@ -685,21 +690,22 @@ function absolutizeScriptKey(entry: Record<string, unknown>, base: string): void
}
/**
* Rebase one `clis` entry's path-form `cli` reference onto `base`.
* Rebase a path-form colon reference under `key` onto `base`.
*
* `cli: ./tool.mjs:TREE` means "next to the config file", the same
* build-context rule `script:` follows; without this the pointer reaches
* `loadAttr` relative and resolves against the server process's cwd. A
* package specifier (`my-clis:JIRA`) is left alone: Node resolves it,
* not the filesystem. The split is `splitRef`/`isModulePath`, the same
* pair `loadAttr` uses, so the two cannot disagree about what a path is.
* `cli: ./tool.mjs:TREE` and `resource: ./wiki.mjs:WikiResource` both
* mean "next to the config file", the same build-context rule `script:`
* follows; without this the pointer reaches `loadAttr` relative and
* resolves against the server process's cwd. A package specifier
* (`my-clis:JIRA`) is left alone: Node resolves it, not the filesystem.
* The split is `splitRef`/`isModulePath`, the same pair `loadAttr` uses,
* so the two cannot disagree about what a path is.
*/
function absolutizeCliRef(entry: Record<string, unknown>, base: string): void {
const ref = entry.cli
function absolutizeCodeRef(entry: Record<string, unknown>, key: string, base: string): void {
const ref = entry[key]
if (typeof ref !== 'string' || !ref.includes(':')) return
const [source, attr] = splitRef(ref)
if (!isModulePath(source) || isAbsolute(source)) return
entry.cli = `${join(base, source)}:${attr}`
entry[key] = `${join(base, source)}:${attr}`
}
/**
@@ -966,12 +972,14 @@ async function buildCliEntries(
if (block.config !== undefined && !isPlainObject(block.config)) {
throw new Error(`clis entry '${name}': config must be a mapping`)
}
// A `cli` value carrying a colon points at code the way `resource:`
// never does: `./tool.mjs:TALLY` (a file) or `my-clis:JIRA` (a
// package specifier). A bare name stays a name for the workspace to
// resolve against the registered specs. Mirrors the `":" in name`
// branch of Python's `cli_spec_for`, one layer up: `cliSpecFor`
// lives in core, which has no filesystem and is synchronous.
// A `cli` value carrying a colon points at code: `./tool.mjs:TALLY`
// (a file) or `my-clis:JIRA` (a package specifier). A bare name stays
// a name for the workspace to resolve against the registered specs.
// Mirrors the `":" in name` branch of Python's `cli_spec_for`, one
// layer up: `cliSpecFor` lives in core, which has no filesystem and
// is synchronous. A `resource:` value reads the same way, resolved by
// `buildResource` rather than here, because a mount block reaches the
// registry and a `clis` block does not.
const entry = hasScript
? new CLISpec({
name,
@@ -12,9 +12,10 @@
// limitations under the License.
// ========= Copyright 2026 @ Strukto.AI All Rights Reserved. =========
import { readFileSync } from 'node:fs'
import { resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join, resolve } from 'node:path'
import { fileURLToPath, pathToFileURL } from 'node:url'
import { describe, expect, it } from 'vitest'
import { normalizeOneDriveConfig } from '@struktoai/mirage-core/accessor/onedrive'
import { tokenUrl } from '@struktoai/mirage-core/core/google/client'
@@ -315,3 +316,76 @@ describe('ResourceName coverage', () => {
expect(BUILTIN_RESOURCES.filter((n) => !names.has(n))).toEqual([])
})
})
// A colon reference names a class rather than a registry key, so it is
// exercised through a real file on disk: that is the path a deployment
// takes, and it is the only way the loader's own module resolution is
// covered. The fixture is `.mjs` because Node strips types without
// compiling them, so a `.ts` fixture using a parameter property would be
// refused at load.
describe('buildResource colon reference', () => {
const CORE = pathToFileURL(
resolve(fileURLToPath(import.meta.url), '../../../../core/dist/index.js'),
).href
const BACKEND =
`import {RAMResource} from ${JSON.stringify(CORE)}\n` +
'export class WikiResource extends RAMResource {\n' +
' constructor(config) { super(); this.config = config }\n' +
'}\n' +
'export class LateResource extends RAMResource {\n' +
' static async create(config) { const r = new LateResource(); r.config = config; return r }\n' +
'}\n' +
'export class NotAResource {}\n' +
'export const NOT_A_CLASS = {name: "wiki"}\n'
function fixture(): string {
const dir = mkdtempSync(join(tmpdir(), 'mirage-resref-'))
writeFileSync(join(dir, 'wiki.mjs'), BACKEND)
return dir
}
it('builds from the class the ref names', async () => {
const dir = fixture()
const built = await buildResource(`${join(dir, 'wiki.mjs')}:WikiResource`, { root: '/ref' })
expect(built.constructor.name).toBe('WikiResource')
expect((built as unknown as { config: unknown }).config).toEqual({ root: '/ref' })
await built.close()
rmSync(dir, { recursive: true, force: true })
})
it('prefers a static create over the constructor', async () => {
const dir = fixture()
const built = await buildResource(`${join(dir, 'wiki.mjs')}:LateResource`, { root: '/late' })
expect(built.constructor.name).toBe('LateResource')
expect((built as unknown as { config: unknown }).config).toEqual({ root: '/late' })
await built.close()
rmSync(dir, { recursive: true, force: true })
})
it('never lets a ref shadow a registry name', async () => {
// A registry name always wins, so a name cannot be reread as code.
const built = await buildResource('ram')
expect(built.constructor.name).toBe('RAMResource')
await built.close()
})
it('refuses a ref that names something other than a class', async () => {
const dir = fixture()
await expect(buildResource(`${join(dir, 'wiki.mjs')}:NOT_A_CLASS`)).rejects.toThrow(
'must name a class',
)
rmSync(dir, { recursive: true, force: true })
})
it('refuses a class that does not build a resource', async () => {
const dir = fixture()
await expect(buildResource(`${join(dir, 'wiki.mjs')}:NotAResource`)).rejects.toThrow(
'did not build a resource',
)
rmSync(dir, { recursive: true, force: true })
})
it('still reports an unknown bare name as unknown', async () => {
await expect(buildResource('nope')).rejects.toThrow('unknown resource')
})
})
@@ -19,6 +19,7 @@ import type { LanceDBConfig } from '@struktoai/mirage-core/resource/lancedb/conf
import type { QdrantConfig } from '@struktoai/mirage-core/resource/qdrant/config'
import { normalizeFields } from '@struktoai/mirage-core/utils/normalize'
import { compareCodePoints } from '@struktoai/mirage-core/utils/sort'
import { loadAttr } from './loader.ts'
/**
* Construct a resource by registry name. Mirrors Python's
@@ -322,18 +323,63 @@ export function register(name: string, factory: ResourceFactory): void {
}
/**
* Build a resource instance by registry name. Builtins win over custom
* registrations. Throws if the name is unknown.
* True when a loaded export answers the calls every mount makes on a
* resource.
*
* The check is `open`/`close`, the two the workspace calls on every mount
* whatever the backend is, so a value that passes cannot fail on the
* lifecycle. It exists because a colon reference loads whatever the file
* exports: without it a typo'd export name reaches `installMounts` and
* fails there, naming a frame the author never wrote.
*/
function looksLikeResource(value: unknown): boolean {
if (value === null || typeof value !== 'object') return false
const node = value as Record<string, unknown>
return typeof node.open === 'function' && typeof node.close === 'function'
}
/**
* Build a resource from a colon reference naming a class directly.
*
* `static create` is honored ahead of the constructor because that is
* how a backend whose setup needs I/O is spelled here (github and
* databricks_volume both do), and it is the one thing this tier has that
* Python's does not: `build_resource` is synchronous there, so an
* out-of-tree Python class hydrates lazily instead.
*/
async function buildFromRef(ref: string, config: Record<string, unknown>): Promise<Resource> {
const exported = await loadAttr(ref)
if (typeof exported !== 'function') {
throw new Error(`resource ref ${JSON.stringify(ref)} must name a class, got ${typeof exported}`)
}
const cls = exported as {
create?: (config: Record<string, unknown>) => unknown
new (config: Record<string, unknown>): unknown
}
const built = await (typeof cls.create === 'function' ? cls.create(config) : new cls(config))
if (!looksLikeResource(built)) {
throw new Error(`resource ref ${JSON.stringify(ref)} did not build a resource`)
}
return built as Resource
}
/**
* Build a resource instance by registry name, or from a colon reference
* naming a class directly (`./wiki.mjs:WikiResource`, or the package
* specifier `my-pkg/backends:WikiResource`).
*
* Builtins win over custom registrations, and both win over a reference,
* so a name can never be reinterpreted as code. Throws if the name is
* neither. Mirrors the ladder in Python's `_resolve_entry`, minus its
* entry-point rung: Node has no equivalent of `importlib.metadata`, so a
* package ships a resource here by exporting it and being named.
*/
export async function buildResource(
name: string,
config: Record<string, unknown> = {},
): Promise<Resource> {
const factory = REGISTRY[name] ?? CUSTOM[name]
if (factory === undefined) {
throw new Error(
`unknown resource ${JSON.stringify(name)}; known: ${knownResources().join(', ')}`,
)
}
return factory(config)
if (factory !== undefined) return factory(config)
if (name.includes(':')) return buildFromRef(name, config)
throw new Error(`unknown resource ${JSON.stringify(name)}; known: ${knownResources().join(', ')}`)
}