Files
github--spec-kit/tests/test_setup_tasks_python_parity.py
T
Manfred Riem 16cfab7724 feat(presets): resolve constitution templates at command time (#3984)
* feat(presets): resolve constitutions at command time

Gate install-time constitution materialization behind the constitution-sync preset while preserving one-time init seeding and authored-file safeguards.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84

* fix(presets): emit composed template content

Add a machine-readable preset resolve mode backed by PresetResolver.resolve_content and require the constitution command to consume it.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84

* fix(presets): unify runtime template composition

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84

* fix(presets): secure runtime template resolution

Align runtime resolution across script variants, validate registry path components, and honor canonical extension ordering and convention paths.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): align runtime priority semantics

Normalize and tie-break preset priorities consistently across script variants, and preserve template bytes when Python materializes generated files.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): stop at effective template base

Avoid parsing irrelevant lower layers once resolution reaches a replace base, and decode raw bytes so Python preserves source line endings.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): align extension template resolution

Support root-level extension templates across runtime resolvers, fail safely when Bash cannot parse an extension registry, and validate requested templates in every prerequisite output mode.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(presets): resolve dotted command identifiers

Route safe dotted names through command resolution, correct traversal coverage, and make Windows CI text decoding explicit.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Avoid orphan feature directories on template errors

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Align malformed preset manifest handling

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Complete runtime resolver parity

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Fail closed on resolver input errors

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Force UTF-8 and full manifest validation

Force UTF-8 decoding for registry and manifest reads in the Bash and
PowerShell embedded-Python parsers so resolution no longer depends on the
process locale, and validate every manifest template entry's required
fields, type, and strategy consistent with the canonical PresetManifest.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* Fail closed on empty manifests and corrupt registries

Reject manifests missing the provides/templates sections or declaring an
empty template list in all three runtime resolvers, matching the canonical
PresetManifest which treats those as invalid instead of silently degrading a
composing layer to a convention `replace` lookup.

Make a corrupt or unreadable extension registry fail closed in Bash,
PowerShell, and Python instead of swallowing the error and treating every
on-disk extension directory as unregistered-and-enabled, which could activate
a disabled extension.

Read the preset and extension registries as explicit UTF-8 in the PowerShell
resolver so priority/enabled-state decoding no longer depends on the process
code page under Windows PowerShell 5.1.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* fix(presets): fail closed when extension registry is not a regular file

The Bash and Python resolvers used is_file()/`-f` to gate reading the
extension `.registry`, which returns false for a directory or a broken
symlink at that path. In those cases the resolvers treated the registry
as absent and scanned every on-disk extension directory as unregistered
and enabled — a fail-open path. Detect any filesystem entry at the
registry path (including broken symlinks) and reject unless it is a
readable regular file. PowerShell now rejects a non-leaf entry explicitly
for parity. Adds directory- and broken-symlink parity regressions.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* fix(presets): fail closed on corrupt registry in canonical resolver and PowerShell

Two remaining fail-open paths for an invalid extension registry:

- The canonical PresetResolver enumerated extensions through
  ExtensionRegistry, whose _load() normalizes a corrupt or unreadable
  registry to an empty mapping. The directory scan then admitted every
  on-disk extension directory as unregistered-and-enabled, so a corrupt
  registry could still supply constitution content at init and through
  constitution-sync materialization. Add a non-invasive is_corrupt()
  probe (recovery behavior for install/enable/disable is unchanged) and
  raise from _get_all_extensions_by_priority() when the registry exists
  but is invalid. _load() now also recovers from OSError/UnicodeDecodeError
  so a directory or unreadable registry no longer crashes construction.

- The PowerShell resolver gated the registry read with Test-Path, which
  returns false for a dangling symlink on Windows, letting a broken
  .registry symlink bypass the guard and enable every on-disk extension.
  Detect the entry via directory enumeration (which observes a broken
  symlink) and reject it unless it is a readable regular file.

Adds canonical corrupt/directory-registry regressions and extends the
broken-symlink parity test to PowerShell.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

* fix(presets): detect dangling registry symlink in ExtensionRegistry.is_corrupt

is_corrupt() gated on Path.exists(), which follows symlinks and returns
False for a dangling .registry symlink — so the canonical PresetResolver
treated it as an absent registry and fell back to scanning every on-disk
extension directory as unregistered-and-enabled, reopening the fail-open
path this guard closes. Detect lexical existence with os.path.lexists and
require a regular file before parsing, so a broken symlink (or directory)
is reported corrupt and resolution fails closed. Adds a canonical
broken-symlink regression alongside the directory case.

Assisted-by: GitHub Copilot (model: claude-opus-4.8, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7dbce70f-80c6-4e14-a30d-78cb358bcb84
Copilot-Session: 3158e06f-95df-4e3a-843f-f159a35aa30c
2026-08-10 10:22:31 -05:00

271 lines
9.4 KiB
Python

"""Parity tests for the Python setup-tasks port."""
from __future__ import annotations
from pathlib import Path
import pytest
from tests.conftest import requires_bash
from tests.parity_helpers import (
HAS_POWERSHELL,
bash_cmd,
break_wrap_layer,
clean_env,
install_composition_stack,
install_scripts,
json_stdout,
make_repo,
normalize_status_text,
ps_cmd,
py_cmd,
run,
write_feature_json,
)
SCRIPT = "setup-tasks"
def _setup_repo(tmp_path: Path) -> Path:
repo = make_repo(tmp_path)
install_scripts(repo, SCRIPT)
write_feature_json(repo)
feature = repo / "specs" / "001-my-feature"
feature.mkdir(parents=True)
(feature / "plan.md").write_text("# plan\n", encoding="utf-8")
(feature / "spec.md").write_text("# spec\n", encoding="utf-8")
templates = repo / ".specify" / "templates"
templates.mkdir(parents=True)
(templates / "tasks-template.md").write_text("# Tasks Template\n", encoding="utf-8")
return repo
@pytest.fixture
def repo(tmp_path: Path) -> Path:
return _setup_repo(tmp_path)
@requires_bash
def test_python_json_output_matches_bash(repo: Path) -> None:
feature = repo / "specs" / "001-my-feature"
(feature / "research.md").write_text("# research\n", encoding="utf-8")
(feature / "data-model.md").write_text("# model\n", encoding="utf-8")
(feature / "quickstart.md").write_text("# quickstart\n", encoding="utf-8")
(feature / "contracts" / "v1").mkdir(parents=True)
bash = run(bash_cmd(repo, SCRIPT, "--json"), repo)
py = run(py_cmd(repo, SCRIPT, "--json"), repo)
assert py.returncode == bash.returncode == 0
assert py.stderr == bash.stderr == ""
assert json_stdout(py) == json_stdout(bash)
@requires_bash
def test_python_text_output_matches_bash(repo: Path) -> None:
feature = repo / "specs" / "001-my-feature"
(feature / "research.md").write_text("# research\n", encoding="utf-8")
(feature / "contracts").mkdir() # present but empty -> reported missing
bash = run(bash_cmd(repo, SCRIPT), repo)
py = run(py_cmd(repo, SCRIPT), repo)
assert py.returncode == bash.returncode == 0
assert py.stderr == bash.stderr == ""
assert normalize_status_text(py.stdout) == normalize_status_text(bash.stdout)
@requires_bash
def test_python_override_template_wins_matches_bash(repo: Path) -> None:
overrides = repo / ".specify" / "templates" / "overrides"
overrides.mkdir(parents=True)
(overrides / "tasks-template.md").write_text("# Override\n", encoding="utf-8")
bash = run(bash_cmd(repo, SCRIPT, "--json"), repo)
py = run(py_cmd(repo, SCRIPT, "--json"), repo)
assert py.returncode == bash.returncode == 0
assert json_stdout(py) == json_stdout(bash)
assert json_stdout(py)["TASKS_TEMPLATE"].endswith("overrides/tasks-template.md")
@requires_bash
def test_all_variants_return_composed_tasks_template(repo: Path) -> None:
expected = install_composition_stack(
repo, "tasks-template", "# Tasks Template\n"
)
results = [
run(bash_cmd(repo, SCRIPT, "--json"), repo),
run(py_cmd(repo, SCRIPT, "--json"), repo),
]
if HAS_POWERSHELL:
results.append(run(ps_cmd(repo, SCRIPT, "-Json"), repo))
assert all(result.returncode == 0 for result in results)
assert all(
json_stdout(result)["TASKS_TEMPLATE_CONTENT"] == expected
for result in results
)
@requires_bash
def test_all_variants_fail_for_broken_tasks_composition(repo: Path) -> None:
install_composition_stack(repo, "tasks-template", "# Tasks Template\n")
break_wrap_layer(repo, "tasks-template")
results = [
run(bash_cmd(repo, SCRIPT, "--json"), repo),
run(py_cmd(repo, SCRIPT, "--json"), repo),
]
if HAS_POWERSHELL:
results.append(run(ps_cmd(repo, SCRIPT, "-Json"), repo))
assert all(result.returncode != 0 for result in results)
assert all(result.stdout == "" for result in results)
@requires_bash
@pytest.mark.parametrize(
"missing",
["plan.md", "spec.md", "tasks-template"],
ids=["missing_plan", "missing_spec", "missing_tasks_template"],
)
def test_python_error_output_matches_bash(repo: Path, missing: str) -> None:
if missing == "tasks-template":
(repo / ".specify" / "templates" / "tasks-template.md").unlink()
else:
(repo / "specs" / "001-my-feature" / missing).unlink()
bash = run(bash_cmd(repo, SCRIPT, "--json"), repo)
py = run(py_cmd(repo, SCRIPT, "--json"), repo)
assert py.returncode == bash.returncode == 1
assert py.stdout == bash.stdout == ""
assert py.stderr == bash.stderr
@requires_bash
def test_python_unknown_option_matches_bash(repo: Path) -> None:
bash = run(bash_cmd(repo, SCRIPT, "--bogus"), repo)
py = run(py_cmd(repo, SCRIPT, "--bogus"), repo)
assert py.returncode == bash.returncode == 1
assert py.stdout == bash.stdout == ""
assert py.stderr == bash.stderr
@requires_bash
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_powershell_unknown_option_matches_siblings(repo: Path) -> None:
bash = run(bash_cmd(repo, SCRIPT, "--bogus"), repo)
ps = run(ps_cmd(repo, SCRIPT, "--bogus"), repo)
py = run(py_cmd(repo, SCRIPT, "--bogus"), repo)
assert ps.returncode == bash.returncode == py.returncode == 1
assert ps.stdout == bash.stdout == py.stdout == ""
assert ps.stderr == bash.stderr == py.stderr
@requires_bash
def test_help_beats_unknown_option_matches_bash(repo: Path) -> None:
"""--help must win over a later unknown option and exit 0."""
bash = run(bash_cmd(repo, SCRIPT, "--help", "--bogus"), repo)
py = run(py_cmd(repo, SCRIPT, "--help", "--bogus"), repo)
assert py.returncode == bash.returncode == 0
assert py.stderr == bash.stderr == ""
assert "Usage" in py.stdout and "Usage" in bash.stdout
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_powershell_help_beats_unknown_option(repo: Path) -> None:
"""-Help must win over unknown-argument validation like the siblings."""
ps = run(ps_cmd(repo, SCRIPT, "-Help", "--bogus"), repo)
assert ps.returncode == 0, ps.stderr
assert ps.stderr == ""
assert "Usage" in ps.stdout
@requires_bash
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
@pytest.mark.parametrize(
"context", ["missing", "invalid_json", "invalid_utf8", "invalid_init_dir"]
)
def test_all_variants_feature_context_error_matches(
tmp_path: Path, context: str
) -> None:
repo = make_repo(tmp_path)
install_scripts(repo, SCRIPT)
env = None
if context == "invalid_json":
(repo / ".specify" / "feature.json").write_text(
"{not json", encoding="utf-8"
)
elif context == "invalid_utf8":
(repo / ".specify" / "feature.json").write_bytes(b"\xff")
elif context == "invalid_init_dir":
env = clean_env()
env["SPECIFY_INIT_DIR"] = str(tmp_path / "missing")
bash = run(bash_cmd(repo, SCRIPT, "--json"), repo, env)
ps = run(ps_cmd(repo, SCRIPT, "-Json"), repo, env)
py = run(py_cmd(repo, SCRIPT, "--json"), repo, env)
assert bash.returncode == ps.returncode == py.returncode == 1
assert bash.stdout == ps.stdout == py.stdout == ""
assert bash.stderr == ps.stderr == py.stderr
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_python_json_output_matches_powershell(repo: Path) -> None:
feature = repo / "specs" / "001-my-feature"
(feature / "research.md").write_text("# research\n", encoding="utf-8")
(feature / "contracts" / "v1").mkdir(parents=True)
ps = run(ps_cmd(repo, SCRIPT, "-Json"), repo)
py = run(py_cmd(repo, SCRIPT, "--json"), repo)
assert py.returncode == ps.returncode == 0
assert json_stdout(py) == json_stdout(ps)
@requires_bash
@pytest.mark.skipif(not HAS_POWERSHELL, reason="no PowerShell available")
def test_missing_template_error_matches_all_variants(repo: Path) -> None:
(repo / ".specify" / "templates" / "tasks-template.md").unlink()
bash = run(bash_cmd(repo, SCRIPT, "--json"), repo)
ps = run(ps_cmd(repo, SCRIPT, "-Json"), repo)
py = run(py_cmd(repo, SCRIPT, "--json"), repo)
assert bash.returncode == ps.returncode == py.returncode == 1
assert bash.stdout == ps.stdout == py.stdout == ""
assert bash.stderr == ps.stderr == py.stderr
def test_python_text_output_survives_a_legacy_stdout_code_page(repo: Path) -> None:
"""Text mode must not crash when stdout cannot encode the status glyphs.
On Windows sys.stdout falls back to the ANSI code page whenever it is not a
console — which is every time an agent or a workflow step captures the
output. U+2713 is unencodable in cp1252, so printing it raised
UnicodeEncodeError and truncated the document listing. The ASCII fallback is
the rendering these markers already have in-tree (Test-FileExists in
scripts/powershell/common.ps1, and normalize_status_text).
"""
feature = repo / "specs" / "001-my-feature"
(feature / "research.md").write_text("# research\n", encoding="utf-8")
(feature / "contracts").mkdir()
env = clean_env()
env["PYTHONIOENCODING"] = "cp1252"
result = run(py_cmd(repo, SCRIPT), repo, env=env)
assert result.returncode == 0, result.stderr
assert "UnicodeEncodeError" not in result.stderr
for doc in ("research.md", "data-model.md", "contracts/", "quickstart.md"):
assert doc in result.stdout, (doc, result.stdout)
assert "[OK] research.md" in normalize_status_text(result.stdout), result.stdout