Files
github--spec-kit/tests/test_setup_plan_python_parity.py
Manfred Riem 16cfab7724 feat(presets): resolve constitution templates at command time (#3984)
* feat(presets): resolve constitutions at command time

Gate install-time constitution materialization behind the constitution-sync preset while preserving one-time init seeding and authored-file safeguards.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84

* fix(presets): emit composed template content

Add a machine-readable preset resolve mode backed by PresetResolver.resolve_content and require the constitution command to consume it.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84

* fix(presets): unify runtime template composition

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84

* fix(presets): secure runtime template resolution

Align runtime resolution across script variants, validate registry path components, and honor canonical extension ordering and convention paths.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): align runtime priority semantics

Normalize and tie-break preset priorities consistently across script variants, and preserve template bytes when Python materializes generated files.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): stop at effective template base

Avoid parsing irrelevant lower layers once resolution reaches a replace base, and decode raw bytes so Python preserves source line endings.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): align extension template resolution

Support root-level extension templates across runtime resolvers, fail safely when Bash cannot parse an extension registry, and validate requested templates in every prerequisite output mode.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): resolve dotted command identifiers

Route safe dotted names through command resolution, correct traversal coverage, and make Windows CI text decoding explicit.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Avoid orphan feature directories on template errors

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Align malformed preset manifest handling

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Complete runtime resolver parity

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Fail closed on resolver input errors

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Force UTF-8 and full manifest validation

Force UTF-8 decoding for registry and manifest reads in the Bash and
PowerShell embedded-Python parsers so resolution no longer depends on the
process locale, and validate every manifest template entry's required
fields, type, and strategy consistent with the canonical PresetManifest.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Fail closed on empty manifests and corrupt registries

Reject manifests missing the provides/templates sections or declaring an
empty template list in all three runtime resolvers, matching the canonical
PresetManifest which treats those as invalid instead of silently degrading a
composing layer to a convention `replace` lookup.

Make a corrupt or unreadable extension registry fail closed in Bash,
PowerShell, and Python instead of swallowing the error and treating every
on-disk extension directory as unregistered-and-enabled, which could activate
a disabled extension.

Read the preset and extension registries as explicit UTF-8 in the PowerShell
resolver so priority/enabled-state decoding no longer depends on the process
code page under Windows PowerShell 5.1.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* fix(presets): fail closed when extension registry is not a regular file

The Bash and Python resolvers used is_file()/`-f` to gate reading the
extension `.registry`, which returns false for a directory or a broken
symlink at that path. In those cases the resolvers treated the registry
as absent and scanned every on-disk extension directory as unregistered
and enabled — a fail-open path. Detect any filesystem entry at the
registry path (including broken symlinks) and reject unless it is a
readable regular file. PowerShell now rejects a non-leaf entry explicitly
for parity. Adds directory- and broken-symlink parity regressions.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* fix(presets): fail closed on corrupt registry in canonical resolver and PowerShell

Two remaining fail-open paths for an invalid extension registry:

- The canonical PresetResolver enumerated extensions through
  ExtensionRegistry, whose _load() normalizes a corrupt or unreadable
  registry to an empty mapping. The directory scan then admitted every
  on-disk extension directory as unregistered-and-enabled, so a corrupt
  registry could still supply constitution content at init and through
  constitution-sync materialization. Add a non-invasive is_corrupt()
  probe (recovery behavior for install/enable/disable is unchanged) and
  raise from _get_all_extensions_by_priority() when the registry exists
  but is invalid. _load() now also recovers from OSError/UnicodeDecodeError
  so a directory or unreadable registry no longer crashes construction.

- The PowerShell resolver gated the registry read with Test-Path, which
  returns false for a dangling symlink on Windows, letting a broken
  .registry symlink bypass the guard and enable every on-disk extension.
  Detect the entry via directory enumeration (which observes a broken
  symlink) and reject it unless it is a readable regular file.

Adds canonical corrupt/directory-registry regressions and extends the
broken-symlink parity test to PowerShell.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* fix(presets): detect dangling registry symlink in ExtensionRegistry.is_corrupt

is_corrupt() gated on Path.exists(), which follows symlinks and returns
False for a dangling .registry symlink — so the canonical PresetResolver
treated it as an absent registry and fell back to scanning every on-disk
extension directory as unregistered-and-enabled, reopening the fail-open
path this guard closes. Detect lexical existence with os.path.lexists and
require a regular file before parsing, so a broken symlink (or directory)
is reported corrupt and resolution fails closed. Adds a canonical
broken-symlink regression alongside the directory case.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c
2026-08-10 10:22:31 -05:00

375 lines
12 KiB
Python

"""Parity tests for the Python setup-plan port."""
from __future__ import annotations
from pathlib import Path
import pytest
from tests.conftest import requires_bash
from tests.parity_helpers import (
HAS_POWERSHELL,
POWERSHELL_EXE,
bash_cmd,
break_wrap_layer,
clean_env,
install_composition_stack,
install_scripts,
json_stdout,
make_repo,
normalize_repo_paths,
ps_cmd,
py_cmd,
run,
write_feature_json,
)
SCRIPT = "setup-plan"
TEMPLATE_BODY = "# Plan Template\n\nBody.\n"
def _setup_repo(tmp_path: Path, name: str = "proj", template: bool = True) -> Path:
repo = make_repo(tmp_path, name)
install_scripts(repo, SCRIPT)
write_feature_json(repo)
(repo / "specs" / "001-my-feature").mkdir(parents=True)
if template:
templates = repo / ".specify" / "templates"
templates.mkdir(parents=True)
(templates / "plan-template.md").write_text(TEMPLATE_BODY, encoding="utf-8")
return repo
@pytest.fixture
def repo(tmp_path: Path) -> Path:
return _setup_repo(tmp_path)
@requires_bash
def test_python_fresh_copy_matches_bash(tmp_path: Path) -> None:
repo_a = _setup_repo(tmp_path, "proj-a")
repo_b = _setup_repo(tmp_path, "proj-b")
bash = run(bash_cmd(repo_a, SCRIPT, "--json"), repo_a)
py = run(py_cmd(repo_b, SCRIPT, "--json"), repo_b)
assert py.returncode == bash.returncode == 0
assert normalize_repo_paths(py.stdout, repo_b) == normalize_repo_paths(
bash.stdout, repo_a
)
assert normalize_repo_paths(py.stderr, repo_b) == normalize_repo_paths(
bash.stderr, repo_a
)
for repo in (repo_a, repo_b):
plan = repo / "specs" / "001-my-feature" / "plan.md"
assert plan.read_bytes() == TEMPLATE_BODY.encode("utf-8")
@requires_bash
def test_all_variants_materialize_composed_plan_template(tmp_path: Path) -> None:
repos = [
_setup_repo(tmp_path, "bash"),
_setup_repo(tmp_path, "powershell"),
_setup_repo(tmp_path, "python"),
]
expected = ""
for current in repos:
expected = install_composition_stack(
current, "plan-template", TEMPLATE_BODY
)
results = [
run(bash_cmd(repos[0], SCRIPT, "--json"), repos[0]),
run(py_cmd(repos[2], SCRIPT, "--json"), repos[2]),
]
checked_repos = [repos[0], repos[2]]
if HAS_POWERSHELL:
results.insert(1, run(ps_cmd(repos[1], SCRIPT, "-Json"), repos[1]))
checked_repos.insert(1, repos[1])
assert all(result.returncode == 0 for result in results)
for current in checked_repos:
assert (
current / "specs" / "001-my-feature" / "plan.md"
).read_text(encoding="utf-8") == expected
@requires_bash
def test_all_variants_fail_for_broken_plan_composition(tmp_path: Path) -> None:
repos = [
_setup_repo(tmp_path, "bash"),
_setup_repo(tmp_path, "powershell"),
_setup_repo(tmp_path, "python"),
]
for current in repos:
install_composition_stack(current, "plan-template", TEMPLATE_BODY)
break_wrap_layer(current, "plan-template")
results = [
run(bash_cmd(repos[0], SCRIPT, "--json"), repos[0]),
run(py_cmd(repos[2], SCRIPT, "--json"), repos[2]),
]
if HAS_POWERSHELL:
results.append(run(ps_cmd(repos[1], SCRIPT, "-Json"), repos[1]))
assert all(result.returncode != 0 for result in results)
@requires_bash
@pytest.mark.parametrize("args", [("--json",), ()], ids=["json", "text"])
def test_python_existing_plan_matches_bash(repo: Path, args: tuple[str, ...]) -> None:
plan = repo / "specs" / "001-my-feature" / "plan.md"
plan.write_text("# existing\n", encoding="utf-8")
bash = run(bash_cmd(repo, SCRIPT, *args), repo)
py = run(py_cmd(repo, SCRIPT, *args), repo)
assert py.returncode == bash.returncode == 0
assert py.stdout == bash.stdout
assert py.stderr == bash.stderr
assert plan.read_text(encoding="utf-8") == "# existing\n"
@requires_bash
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_all_variants_ignore_extra_arguments(tmp_path: Path) -> None:
repos = [
_setup_repo(tmp_path, "bash"),
_setup_repo(tmp_path, "powershell"),
_setup_repo(tmp_path, "python"),
]
bash = run(bash_cmd(repos[0], SCRIPT, "--json", "--bogus"), repos[0])
ps = run(ps_cmd(repos[1], SCRIPT, "-Json", "--bogus"), repos[1])
py = run(py_cmd(repos[2], SCRIPT, "--json", "--bogus"), repos[2])
assert bash.returncode == ps.returncode == py.returncode == 0
assert normalize_repo_paths(bash.stdout, repos[0]) == normalize_repo_paths(
ps.stdout, repos[1]
) == normalize_repo_paths(py.stdout, repos[2])
assert normalize_repo_paths(bash.stderr, repos[0]) == normalize_repo_paths(
ps.stderr, repos[1]
) == normalize_repo_paths(py.stderr, repos[2])
@requires_bash
def test_python_missing_template_matches_bash(tmp_path: Path) -> None:
repo_a = _setup_repo(tmp_path, "proj-a", template=False)
repo_b = _setup_repo(tmp_path, "proj-b", template=False)
bash = run(bash_cmd(repo_a, SCRIPT, "--json"), repo_a)
py = run(py_cmd(repo_b, SCRIPT, "--json"), repo_b)
assert py.returncode == bash.returncode == 0
assert normalize_repo_paths(py.stderr, repo_b) == normalize_repo_paths(
bash.stderr, repo_a
)
for repo in (repo_a, repo_b):
plan = repo / "specs" / "001-my-feature" / "plan.md"
assert plan.read_text(encoding="utf-8") == ""
@requires_bash
@pytest.mark.parametrize(
("registry", "expected"),
[
(
'{"presets": {"alpha": {"priority": "high"}, "beta": {"priority": 1}}}',
"# beta plan\n",
),
(
'{"presets": {"alpha": {"priority": 2}, "beta": {"priority": 1}, "gamma": {"priority": null}}}',
"# beta plan\n",
),
("[]", "# alpha plan\n"),
('{"presets":[]}', "# alpha plan\n"),
('{"presets":null}', "# alpha plan\n"),
],
ids=[
"mixed_priorities",
"null_priority",
"list_root",
"list_presets",
"null_presets",
],
)
def test_all_variants_normalize_or_fallback_for_registry(
tmp_path: Path, registry: str, expected: str
) -> None:
"""Priorities normalize canonically; malformed shapes fall back to directories."""
repos = [
_setup_repo(tmp_path, "bash", template=False),
_setup_repo(tmp_path, "powershell", template=False),
_setup_repo(tmp_path, "python", template=False),
]
for repo in repos:
presets = repo / ".specify" / "presets"
for name, body in (
(".hidden", "# hidden\n"),
("beta", "# beta plan\n"),
("alpha", "# alpha plan\n"),
):
(presets / name / "templates").mkdir(parents=True)
(presets / name / "templates" / "plan-template.md").write_text(
body, encoding="utf-8"
)
(presets / ".registry").write_text(
registry, encoding="utf-8"
)
bash = run(bash_cmd(repos[0], SCRIPT, "--json"), repos[0])
py = run(py_cmd(repos[2], SCRIPT, "--json"), repos[2])
results = [(bash, repos[0]), (py, repos[2])]
if HAS_POWERSHELL:
results.insert(
1,
(run(ps_cmd(repos[1], SCRIPT, "-Json"), repos[1]), repos[1]),
)
assert all(result.returncode == 0 for result, _ in results)
assert len(
{
normalize_repo_paths(result.stdout, repo)
for result, repo in results
}
) == 1
assert len(
{
normalize_repo_paths(result.stderr, repo)
for result, repo in results
}
) == 1
for _, repo in results:
plan = repo / "specs" / "001-my-feature" / "plan.md"
assert plan.read_text(encoding="utf-8") == expected
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_powershell_broken_registry_fallback_sorts_directories(
tmp_path: Path,
) -> None:
repo = _setup_repo(tmp_path, "powershell", template=False)
presets = repo / ".specify" / "presets"
for name in ("alpha", "beta"):
templates = presets / name / "templates"
templates.mkdir(parents=True)
(templates / "plan-template.md").write_text(
f"# {name} plan\n", encoding="utf-8"
)
(presets / ".registry").write_text("{broken", encoding="utf-8")
common = repo / ".specify" / "scripts" / "powershell" / "common.ps1"
common_ps = str(common).replace("'", "''")
alpha_ps = str(presets / "alpha").replace("'", "''")
beta_ps = str(presets / "beta").replace("'", "''")
repo_ps = str(repo).replace("'", "''")
command = f"""
. '{common_ps}'
function Get-ChildItem {{
@(
[PSCustomObject]@{{ Name = 'beta'; FullName = '{beta_ps}' }}
[PSCustomObject]@{{ Name = 'alpha'; FullName = '{alpha_ps}' }}
)
}}
Resolve-Template -TemplateName 'plan-template' -RepoRoot '{repo_ps}'
"""
result = run(
[POWERSHELL_EXE, "-NoProfile", "-Command", command],
repo,
)
assert result.returncode == 0
assert result.stderr == ""
assert Path(result.stdout.strip()).read_text(encoding="utf-8") == (
"# alpha plan\n"
)
@requires_bash
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
@pytest.mark.parametrize(
"context", ["missing", "invalid_json", "invalid_utf8", "invalid_init_dir"]
)
def test_all_variants_feature_context_error_matches(
tmp_path: Path, context: str
) -> None:
repo = make_repo(tmp_path)
install_scripts(repo, SCRIPT)
env = None
if context == "invalid_json":
(repo / ".specify" / "feature.json").write_text(
"{not json", encoding="utf-8"
)
elif context == "invalid_utf8":
(repo / ".specify" / "feature.json").write_bytes(b"\xff")
elif context == "invalid_init_dir":
env = clean_env()
env["SPECIFY_INIT_DIR"] = str(tmp_path / "missing")
bash = run(bash_cmd(repo, SCRIPT, "--json"), repo, env)
ps = run(ps_cmd(repo, SCRIPT, "-Json"), repo, env)
py = run(py_cmd(repo, SCRIPT, "--json"), repo, env)
assert bash.returncode == ps.returncode == py.returncode == 1
assert bash.stdout == ps.stdout == py.stdout == ""
assert bash.stderr == ps.stderr == py.stderr
@requires_bash
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
@pytest.mark.parametrize(
"registry",
[
'{"presets":{"alpha":{"enabled":false,"priority":1}}}',
'{"presets":{"alpha":"invalid"}}',
],
ids=["disabled", "invalid_metadata"],
)
def test_all_variants_ignore_inactive_preset_template(
tmp_path: Path, registry: str
) -> None:
repos = [
_setup_repo(tmp_path, "bash"),
_setup_repo(tmp_path, "powershell"),
_setup_repo(tmp_path, "python"),
]
for current in repos:
preset_templates = (
current / ".specify" / "presets" / "alpha" / "templates"
)
preset_templates.mkdir(parents=True)
(preset_templates / "plan-template.md").write_text(
"# Disabled preset\n", encoding="utf-8"
)
(current / ".specify" / "presets" / ".registry").write_text(
registry, encoding="utf-8"
)
bash = run(bash_cmd(repos[0], SCRIPT, "--json"), repos[0])
ps = run(ps_cmd(repos[1], SCRIPT, "-Json"), repos[1])
py = run(py_cmd(repos[2], SCRIPT, "--json"), repos[2])
assert bash.returncode == ps.returncode == py.returncode == 0
assert normalize_repo_paths(bash.stdout, repos[0]) == normalize_repo_paths(
ps.stdout, repos[1]
) == normalize_repo_paths(py.stdout, repos[2])
assert normalize_repo_paths(bash.stderr, repos[0]) == normalize_repo_paths(
ps.stderr, repos[1]
) == normalize_repo_paths(py.stderr, repos[2])
for current in repos:
assert (
current / "specs" / "001-my-feature" / "plan.md"
).read_text(encoding="utf-8") == TEMPLATE_BODY
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_python_json_output_matches_powershell(repo: Path) -> None:
plan = repo / "specs" / "001-my-feature" / "plan.md"
plan.write_text("# existing\n", encoding="utf-8")
ps = run(ps_cmd(repo, SCRIPT, "-Json"), repo)
py = run(py_cmd(repo, SCRIPT, "--json"), repo)
assert py.returncode == ps.returncode == 0
assert json_stdout(py) == json_stdout(ps)