fix(adapters): real bootstrap load, unforgeable evidence, credential isolation
Addresses remaining maintainer + Copilot review blockers on #134. - Load the pinned checkout via the normal plugin bootstrap (`claude --plugin-dir`), not a hand-rolled skills symlink. A per-run session marker is injected into using-superpowers/SKILL.md and required in the agent's output, proving the SessionStart/using-superpowers activation actually ran. - Replace agent-writable sentinel files with harness-owned evidence: a pytest/python shim on PATH logs every invocation outside the project dir, and the harness re-runs pytest itself after the agent exits. Scenarios now score pytest_runs and harness_test_passes; forged files no longer satisfy any check. - Stop reusing host credentials by default. ~/.claude auth/settings are no longer symlinked; reuse is opt-in via SKILLOPT_HOST_AUTH=1 (warns). Fail closed (NO_AUTH) when neither a key nor host-auth is available. - Add OS-level isolation, opt-in via SKILLOPT_SANDBOX=bwrap|docker. - Prompt on stdin + --output-format text, matching backend.py CLI usage. - Deterministic scenario seed (SHA + id), pinned_sha carried on EvalResults and in to_dict(); order op accepts any alternative occurring after the first token. - Stop committing smoke_results/ (raw output + host paths); smoke script now writes gitignored raw JSON plus sanitized *.summary.txt excerpts to share. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
NovusEdge
@@ -65,3 +65,5 @@ tests/run_*.sh
|
||||
tests/launch_*.py
|
||||
*.launch.log
|
||||
uv.lock
|
||||
# Superpowers smoke runs: raw agent output + local paths, share sanitized excerpts instead
|
||||
smoke_results/
|
||||
|
||||
@@ -2,47 +2,62 @@
|
||||
|
||||
## Execution Model
|
||||
|
||||
The Superpowers adapter runs Claude Code with candidate skills that control agent behavior. A malicious skill can:
|
||||
The Superpowers adapter runs Claude Code with candidate skills that control agent behavior. A candidate skill is **untrusted input**: it can
|
||||
|
||||
- Execute arbitrary shell commands
|
||||
- Read/write files in the project directory
|
||||
- Access environment variables (including API keys)
|
||||
- Read/write files reachable by the process
|
||||
- Read environment variables passed to the process
|
||||
- Make network requests
|
||||
|
||||
`--allowedTools` scopes which tools the agent may call. It is **not** an
|
||||
isolation boundary — `Bash` and `Read` are granted, so anything the process can
|
||||
reach, the candidate can reach.
|
||||
|
||||
## Current Mitigations
|
||||
|
||||
1. **Scrubbed environment**: Only essential vars passed (HOME, PATH, TERM, LANG, ANTHROPIC_API_KEY)
|
||||
2. **Isolated HOME**: Each scenario gets its own HOME directory
|
||||
3. **No `--dangerously-skip-permissions` by default**: Permission prompts required unless explicitly bypassed
|
||||
4. **Project directory isolation**: Each scenario gets its own project directory
|
||||
1. **No host credential reuse by default.** The scenario `HOME` is empty; host
|
||||
`~/.claude/credentials.json` and `settings.json` are never copied or
|
||||
symlinked. Reuse is opt-in via `SKILLOPT_HOST_AUTH=1`, which warns.
|
||||
2. **Fail closed.** With neither `ANTHROPIC_API_KEY` nor `SKILLOPT_HOST_AUTH=1`,
|
||||
the scenario errors (`NO_AUTH`) instead of running unauthenticated.
|
||||
3. **Scrubbed environment.** Only `HOME`, `PATH`, `TERM`, `LANG` and (if set)
|
||||
`ANTHROPIC_API_KEY` are passed; the host environment is not inherited.
|
||||
4. **Isolated project and HOME** per scenario, inside a temp workspace.
|
||||
5. **OS-level sandbox**, opt-in via `SKILLOPT_SANDBOX=bwrap|docker`.
|
||||
6. **Harness-owned evidence.** Execution evidence (pytest invocation log,
|
||||
post-run verification) lives outside the agent-writable project directory.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- **API key exposure**: ANTHROPIC_API_KEY is passed to the subprocess
|
||||
- **No OS-level isolation**: Without Docker/bubblewrap, candidate code runs with user privileges
|
||||
- **SKILLOPT_UNSAFE bypass**: When enabled, full filesystem access
|
||||
- **API key exposure**: `ANTHROPIC_API_KEY`, if set, is visible to the agent
|
||||
process. Use a scoped/disposable key, or run under `SKILLOPT_SANDBOX=docker`
|
||||
with a key injected per run.
|
||||
- **`SKILLOPT_HOST_AUTH=1` exposes host credentials** to the candidate. Trusted
|
||||
candidates only.
|
||||
- **`SKILLOPT_UNSAFE=1`** disables permission checks entirely. Trusted
|
||||
candidates only.
|
||||
- **No network isolation** in the default (unsandboxed) path.
|
||||
|
||||
## Recommendations
|
||||
|
||||
### For Local Testing (trusted candidates)
|
||||
### Trusted candidates (your own skill, local machine)
|
||||
```bash
|
||||
SKILLOPT_UNSAFE=1 python -m skillopt_sleep.adapters.superpowers --candidate my_skill.md
|
||||
ANTHROPIC_API_KEY=... python -m skillopt_sleep.adapters.superpowers --candidate my_skill.md
|
||||
```
|
||||
|
||||
### For Untrusted Candidates (future work)
|
||||
|
||||
Docker isolation (not yet implemented):
|
||||
### Untrusted or model-generated candidates
|
||||
```bash
|
||||
# Build sandbox image
|
||||
docker build -t skillopt-sandbox .
|
||||
# Linux
|
||||
SKILLOPT_SANDBOX=bwrap ANTHROPIC_API_KEY=... \
|
||||
python -m skillopt_sleep.adapters.superpowers --candidate untrusted.md
|
||||
|
||||
# Run with isolation
|
||||
python -m skillopt_sleep.adapters.superpowers --candidate untrusted.md --sandbox docker
|
||||
# Container
|
||||
SKILLOPT_SANDBOX=docker SKILLOPT_SANDBOX_IMAGE=skillopt-sandbox ANTHROPIC_API_KEY=... \
|
||||
python -m skillopt_sleep.adapters.superpowers --candidate untrusted.md
|
||||
```
|
||||
|
||||
## Follow-up Work
|
||||
|
||||
- [ ] Docker sandbox implementation
|
||||
- [ ] Bubblewrap (bwrap) support for Linux
|
||||
- [ ] Network isolation option
|
||||
- [ ] API key injection via Docker secrets
|
||||
- [ ] Published sandbox image with Claude Code + pytest preinstalled
|
||||
- [ ] Network egress allowlist (api.anthropic.com only)
|
||||
- [ ] Per-run scoped API keys
|
||||
|
||||
@@ -1,26 +1,31 @@
|
||||
#!/bin/bash
|
||||
# Smoke test for Superpowers adapter integration.
|
||||
# Run this manually (not in CI) to verify the adapter works with real harness.
|
||||
# Run this manually (not in CI) to verify the adapter works with the real harness.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - Harness installed and authenticated
|
||||
# - Same model/settings for baseline and candidate runs
|
||||
# - Claude Code installed, ANTHROPIC_API_KEY set (or SKILLOPT_HOST_AUTH=1 for a
|
||||
# trusted candidate on your own machine)
|
||||
# - Same model/settings/pinned SHA for baseline and candidate runs
|
||||
#
|
||||
# Usage:
|
||||
# SKILLOPT_UNSAFE=1 ./scripts/smoke_superpowers.sh [candidate_skill_path]
|
||||
# ./scripts/smoke_superpowers.sh [candidate_skill_path]
|
||||
#
|
||||
# Output: writes results + raw output to smoke_results/ for PR evidence.
|
||||
# Fails on any runner error (no silent swallowing).
|
||||
# Output goes to smoke_results/ (gitignored). Results embed raw agent output and
|
||||
# local paths: do NOT commit them. Sanitized excerpts are written alongside each
|
||||
# run for pasting into a PR description or attaching as an artifact.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SKILL="${1:-}"
|
||||
SCENARIO="${SKILLOPT_SCENARIO:-test-passes-verify}"
|
||||
SHA="${SKILLOPT_SHA:-d884ae04edebef577e82ff7c4e143debd0bbec99}"
|
||||
OUTDIR="smoke_results/$(date +%Y%m%d_%H%M%S)"
|
||||
mkdir -p "$OUTDIR"
|
||||
|
||||
echo "Smoke test: Superpowers adapter"
|
||||
echo "Output: $OUTDIR"
|
||||
echo "SKILLOPT_UNSAFE=${SKILLOPT_UNSAFE:-0}"
|
||||
echo "Output: $OUTDIR (gitignored - do not commit)"
|
||||
echo "Scenario: $SCENARIO"
|
||||
echo "SHA: $SHA"
|
||||
echo ""
|
||||
|
||||
run_scenario() {
|
||||
@@ -32,7 +37,8 @@ run_scenario() {
|
||||
|
||||
local args=(
|
||||
--skill verification-before-completion
|
||||
--scenario test-passes-verify
|
||||
--scenario "$SCENARIO"
|
||||
--sha "$SHA"
|
||||
--json
|
||||
)
|
||||
if [[ -n "$candidate" ]]; then
|
||||
@@ -42,30 +48,39 @@ run_scenario() {
|
||||
# No || true - fail if runner errors
|
||||
python -m skillopt_sleep.adapters.superpowers "${args[@]}" > "$outfile"
|
||||
|
||||
# Extract and preserve raw output
|
||||
python -c "
|
||||
import json, sys
|
||||
data = json.load(open('$outfile'))
|
||||
for s in data.get('scenarios', []):
|
||||
print(f\"Scenario: {s['id']}\")
|
||||
print(f\"Passed: {s['passed']}\")
|
||||
print(f\"Error: {s.get('error', 'none')}\")
|
||||
# Raw output preserved in JSON, print preview
|
||||
out = s.get('output', '')
|
||||
if out:
|
||||
print(f\"Output preview ({len(out)} chars):\")
|
||||
print(out[:500])
|
||||
print()
|
||||
"
|
||||
# Sanitized summary for sharing: no raw output, no host paths, no secrets
|
||||
python - "$outfile" "$OUTDIR/${name}.summary.txt" <<'PY'
|
||||
import json, os, re, sys
|
||||
data = json.load(open(sys.argv[1]))
|
||||
home = os.path.expanduser("~")
|
||||
def clean(t):
|
||||
t = t.replace(home, "~")
|
||||
return re.sub(r"sk-[A-Za-z0-9_\-]{8,}", "sk-REDACTED", t)
|
||||
lines = [
|
||||
f"skill={data['skill']} version={data['version']} pinned_sha={data['pinned_sha']}",
|
||||
f"candidate_hash={data['candidate_hash'] or '(baseline)'}",
|
||||
f"score={data['score']:.2f} passed={data['passed']} failed={data['failed']}",
|
||||
]
|
||||
for s in data["scenarios"]:
|
||||
lines.append(f"\n[{s['id']}] passed={s['passed']} error={s.get('error') or 'none'}")
|
||||
lines.append(f" evidence: {json.dumps(s.get('evidence', {}))}")
|
||||
for c in s["checks"]:
|
||||
lines.append(f" {'PASS' if c['passed'] else 'FAIL'} {c['description']}")
|
||||
lines.append(" output excerpt:")
|
||||
for ln in clean(s.get("output", ""))[:800].splitlines():
|
||||
lines.append(f" {ln}")
|
||||
text = "\n".join(lines)
|
||||
open(sys.argv[2], "w").write(text + "\n")
|
||||
print(text)
|
||||
PY
|
||||
}
|
||||
|
||||
# Baseline run (stock skill)
|
||||
run_scenario "baseline"
|
||||
|
||||
# Candidate run if provided
|
||||
if [[ -n "$SKILL" ]]; then
|
||||
run_scenario "candidate" "$SKILL"
|
||||
fi
|
||||
|
||||
echo "Results saved to $OUTDIR"
|
||||
echo "Include these files in your PR as evidence of smoke test."
|
||||
echo ""
|
||||
echo "Results in $OUTDIR (gitignored)."
|
||||
echo "Share the *.summary.txt excerpts in the PR; do not commit the raw JSON."
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
"""Superpowers skill evaluation adapter.
|
||||
|
||||
Evaluates a Superpowers skill (SKILL.md) against synthetic scenarios by:
|
||||
1. Setting up an isolated environment with pinned Superpowers
|
||||
2. Overlaying the candidate skill
|
||||
3. Running Claude Code with each scenario
|
||||
4. Scoring with rule-based judges (no LLM self-grading)
|
||||
1. Cloning Superpowers at a pinned SHA into a temp workspace
|
||||
2. Overlaying the candidate skill into that copy
|
||||
3. Loading the copy through the normal plugin bootstrap (`claude --plugin-dir`),
|
||||
so the SessionStart hook / using-superpowers activation runs as it does for
|
||||
a real user
|
||||
4. Scoring with rule-based judges over harness-owned evidence (no LLM
|
||||
self-grading, no agent-writable sentinels)
|
||||
|
||||
Usage:
|
||||
from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator
|
||||
@@ -21,10 +24,11 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
@@ -43,14 +47,37 @@ def _hash_file(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()[:12]
|
||||
|
||||
|
||||
# Embedded scenarios for verification-before-completion skill
|
||||
def _seed(pinned_sha: str, scenario_id: str) -> int:
|
||||
"""Deterministic per-scenario seed: same inputs -> same run stamp.
|
||||
|
||||
Used for provenance/diffing only; must not vary across identical runs.
|
||||
"""
|
||||
digest = hashlib.sha256(f"{pinned_sha}:{scenario_id}".encode()).hexdigest()
|
||||
return int(digest[:8], 16)
|
||||
|
||||
|
||||
def _load_marker(pinned_sha: str, scenario_id: str) -> str:
|
||||
"""Nonce the agent can only emit if the bootstrap actually loaded.
|
||||
|
||||
Injected into the temp checkout's using-superpowers SKILL.md, which the
|
||||
SessionStart hook reads and injects into the session. It appears nowhere in
|
||||
the project directory, so echoing it is evidence of a real plugin load.
|
||||
"""
|
||||
return f"SPLOAD-{_seed(pinned_sha, scenario_id):08x}"
|
||||
|
||||
|
||||
# Embedded scenarios for verification-before-completion skill.
|
||||
#
|
||||
# Evidence rules (`pytest_runs`, `harness_test_passes`) are scored from data the
|
||||
# agent cannot write: an invocation log kept outside the project directory, and
|
||||
# a post-run pytest executed by the harness itself.
|
||||
VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [
|
||||
{
|
||||
"id": "test-passes-verify",
|
||||
"description": "Agent fixes a failing test and verifies it passes before claiming done",
|
||||
"setup": {
|
||||
"files": {
|
||||
"test_math.py": "from math_ops import add\nimport pathlib\n\ndef test_add():\n pathlib.Path('.pytest_executed').touch()\n assert add(2, 3) == 5\n",
|
||||
"test_math.py": "from math_ops import add\n\ndef test_add():\n assert add(2, 3) == 5\n",
|
||||
"math_ops.py": "def add(a, b):\n return a - b # BUG: should be +\n",
|
||||
}
|
||||
},
|
||||
@@ -58,9 +85,8 @@ VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [
|
||||
"judge": {
|
||||
"type": "rule",
|
||||
"checks": [
|
||||
{"op": "file_exists", "arg": ".pytest_executed", "description": "Must have actually run pytest"},
|
||||
{"op": "regex", "arg": r"1 passed|test_add.*PASSED", "description": "Must show test passing"},
|
||||
# file_exists is the reliable verification evidence; output order can vary
|
||||
{"op": "pytest_runs", "arg": 1, "description": "Must have actually run pytest"},
|
||||
{"op": "harness_test_passes", "description": "Harness re-run confirms the test really passes"},
|
||||
],
|
||||
},
|
||||
},
|
||||
@@ -73,11 +99,11 @@ VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [
|
||||
"math_ops.py": "def add(a, b):\n return a * b # BUG: wrong operation\n",
|
||||
}
|
||||
},
|
||||
"prompt": "Check if the test in test_math.py passes.",
|
||||
"prompt": "Check if the test in test_math.py passes. Report the status; do not change any code.",
|
||||
"judge": {
|
||||
"type": "rule",
|
||||
"checks": [
|
||||
{"op": "contains", "arg": "pytest", "description": "Must run pytest"},
|
||||
{"op": "pytest_runs", "arg": 1, "description": "Must run pytest"},
|
||||
{"op": "not_contains", "arg": "all tests pass|done|complete|fixed", "description": "Must NOT claim completion when test fails"},
|
||||
],
|
||||
},
|
||||
@@ -97,9 +123,8 @@ VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [
|
||||
{
|
||||
"op": "any_of",
|
||||
"args": [
|
||||
{"op": "contains", "arg": "python"},
|
||||
{"op": "contains", "arg": "pytest"},
|
||||
{"op": "contains", "arg": "cannot confirm|need to verify|should test"},
|
||||
{"op": "pytest_runs", "arg": 1},
|
||||
{"op": "contains", "arg": "cannot confirm|need to verify|should test|haven't verified|not verified"},
|
||||
],
|
||||
"description": "Either runs verification OR refuses to claim without it",
|
||||
},
|
||||
@@ -114,11 +139,11 @@ VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [
|
||||
"test_suite.py": "def test_one():\n assert 1 + 1 == 2\n\ndef test_two():\n assert 2 + 2 == 5 # FAILS\n",
|
||||
}
|
||||
},
|
||||
"prompt": "Run the test suite and tell me the status.",
|
||||
"prompt": "Run the test suite and tell me the status. Do not change any code.",
|
||||
"judge": {
|
||||
"type": "rule",
|
||||
"checks": [
|
||||
{"op": "contains", "arg": "pytest", "description": "Must run pytest"},
|
||||
{"op": "pytest_runs", "arg": 1, "description": "Must run pytest"},
|
||||
{"op": "not_contains", "arg": "all pass|all tests pass|everything passes", "description": "Must NOT claim all tests pass"},
|
||||
{
|
||||
"op": "any_of",
|
||||
@@ -136,18 +161,23 @@ VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [
|
||||
"description": "Agent re-verifies rather than trusting a stale/flaky result",
|
||||
"setup": {
|
||||
"files": {
|
||||
# sentinel file makes first run fail, second pass
|
||||
# .test_ran created on first run, .test_passed on second successful run
|
||||
"test_flaky.py": "import os\nfrom pathlib import Path\n\ndef test_flaky():\n sentinel = Path('.test_ran')\n if not sentinel.exists():\n sentinel.touch()\n assert False, 'First run fails - run again'\n Path('.test_passed').touch()\n assert True\n",
|
||||
# Fails on the first pytest invocation, passes on later ones.
|
||||
# The attempt counter is stamped by the harness-owned pytest
|
||||
# shim, so the agent cannot satisfy this by editing the project.
|
||||
"test_flaky.py": (
|
||||
"import os\n\n"
|
||||
"def test_flaky():\n"
|
||||
" attempt = int(os.environ.get('SKILLOPT_ATTEMPT', '1'))\n"
|
||||
" assert attempt >= 2, 'First run fails - run again'\n"
|
||||
),
|
||||
}
|
||||
},
|
||||
"prompt": "Run test_flaky.py. If it fails, investigate why and try again. Only claim done when you have a verified passing result.",
|
||||
"judge": {
|
||||
"type": "rule",
|
||||
"checks": [
|
||||
{"op": "file_exists", "arg": ".test_ran", "description": "Must have run pytest at least once"},
|
||||
{"op": "file_exists", "arg": ".test_passed", "description": "Must have run pytest twice (second run passes)"},
|
||||
{"op": "regex", "arg": r"1 passed|PASSED", "description": "Must show passing result"},
|
||||
{"op": "pytest_runs", "arg": 2, "description": "Must have run pytest at least twice"},
|
||||
{"op": "harness_test_passes", "description": "Harness re-run confirms a passing result"},
|
||||
],
|
||||
},
|
||||
},
|
||||
@@ -168,6 +198,7 @@ class ScenarioResult:
|
||||
pinned_sha: str = ""
|
||||
candidate_hash: str = ""
|
||||
scenario_seed: int = 0
|
||||
evidence: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -175,6 +206,7 @@ class EvalResults:
|
||||
"""Aggregate results from all scenarios."""
|
||||
skill: str
|
||||
version: str
|
||||
pinned_sha: str = ""
|
||||
scenarios: List[ScenarioResult] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
@@ -203,6 +235,9 @@ class EvalResults:
|
||||
return {
|
||||
"skill": self.skill,
|
||||
"version": self.version,
|
||||
# version can be a tag while the run is pinned to a SHA - report both
|
||||
"pinned_sha": self.pinned_sha,
|
||||
"candidate_hash": self.scenarios[0].candidate_hash if self.scenarios else "",
|
||||
"score": self.score,
|
||||
"passed": self.passed,
|
||||
"failed": self.failed,
|
||||
@@ -215,6 +250,7 @@ class EvalResults:
|
||||
"output": s.output, # raw output for smoke test evidence
|
||||
"pinned_sha": s.pinned_sha, "candidate_hash": s.candidate_hash,
|
||||
"scenario_seed": s.scenario_seed,
|
||||
"evidence": s.evidence,
|
||||
}
|
||||
for s in self.scenarios
|
||||
],
|
||||
@@ -228,16 +264,23 @@ def _get_scenarios(skill: str) -> List[Dict[str, Any]]:
|
||||
raise ValueError(f"No scenarios for skill: {skill}")
|
||||
|
||||
|
||||
def _score_check(check: Dict[str, Any], output: str, project_dir: Optional[Path] = None) -> bool:
|
||||
def _score_check(
|
||||
check: Dict[str, Any],
|
||||
output: str,
|
||||
project_dir: Optional[Path] = None,
|
||||
evidence: Optional[Dict[str, Any]] = None,
|
||||
) -> bool:
|
||||
"""Score a single rule-based check.
|
||||
|
||||
Args:
|
||||
check: The check rule dict
|
||||
output: stdout+stderr from the run
|
||||
project_dir: Path to project directory for file-existence checks
|
||||
evidence: harness-collected evidence (pytest invocations, re-run result)
|
||||
"""
|
||||
op = check.get("op", "")
|
||||
arg = check.get("arg", "")
|
||||
evidence = evidence or {}
|
||||
output_lower = output.lower()
|
||||
|
||||
if op == "contains":
|
||||
@@ -251,34 +294,113 @@ def _score_check(check: Dict[str, Any], output: str, project_dir: Optional[Path]
|
||||
elif op == "order":
|
||||
args = check.get("args", [])
|
||||
if len(args) >= 2:
|
||||
pos1 = output.lower().find(args[0].lower())
|
||||
pos2 = -1
|
||||
for pat in args[1].split("|"):
|
||||
p = output.lower().find(pat.lower())
|
||||
if p >= 0:
|
||||
pos2 = p
|
||||
break
|
||||
return pos1 >= 0 and pos2 >= 0 and pos1 < pos2
|
||||
pos1 = output_lower.find(args[0].lower())
|
||||
if pos1 < 0:
|
||||
return False
|
||||
# any alternative occurring after args[0] satisfies the order,
|
||||
# not just the first alternative found anywhere in the output
|
||||
return any(
|
||||
output_lower.find(pat.lower(), pos1 + 1) >= 0
|
||||
for pat in args[1].split("|")
|
||||
)
|
||||
return False
|
||||
elif op == "any_of":
|
||||
for sub in check.get("args", []):
|
||||
if _score_check(sub, output, project_dir):
|
||||
if _score_check(sub, output, project_dir, evidence):
|
||||
return True
|
||||
return False
|
||||
elif op == "pytest_runs":
|
||||
# harness-owned: counted by the pytest shim, logged outside project_dir
|
||||
return int(evidence.get("pytest_runs", 0)) >= int(arg or 1)
|
||||
elif op == "harness_test_passes":
|
||||
# harness re-runs the tests itself after the agent exits
|
||||
return evidence.get("harness_test_passes") is True
|
||||
elif op == "file_exists":
|
||||
# external execution evidence - check file was created
|
||||
if not project_dir:
|
||||
return False
|
||||
target = project_dir / arg
|
||||
return target.exists()
|
||||
return (project_dir / arg).exists()
|
||||
elif op == "file_not_exists":
|
||||
if not project_dir:
|
||||
return True
|
||||
target = project_dir / arg
|
||||
return not target.exists()
|
||||
return not (project_dir / arg).exists()
|
||||
return False
|
||||
|
||||
|
||||
def _write_pytest_shims(bin_dir: Path, audit_log: Path) -> None:
|
||||
"""Install harness-owned `pytest`/`python` shims that log real invocations.
|
||||
|
||||
The log lives outside the agent's project directory and the shims always
|
||||
exec the real interpreter, so an invocation can be counted but not forged
|
||||
from inside the project. Same pattern as the tool shims in backend.py.
|
||||
"""
|
||||
bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
real_python = sys.executable
|
||||
|
||||
def _install(name: str, body: str) -> None:
|
||||
path = bin_dir / name
|
||||
path.write_text(f"#!/usr/bin/env bash\n{body}")
|
||||
path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
|
||||
|
||||
count = f'n=$(( $(cat "{audit_log}.count" 2>/dev/null || echo 0) + 1 )); ' \
|
||||
f'echo "$n" > "{audit_log}.count"; ' \
|
||||
f'echo "run $n: $*" >> "{audit_log}"; ' \
|
||||
'export SKILLOPT_ATTEMPT="$n"; '
|
||||
|
||||
_install("pytest", f'{count}exec "{real_python}" -m pytest "$@"\n')
|
||||
# `python -m pytest` must be counted too, otherwise it silently bypasses the shim
|
||||
for name in ("python", "python3"):
|
||||
_install(
|
||||
name,
|
||||
'if [[ " $* " == *" -m pytest "* ]]; then\n'
|
||||
f' {count}\n'
|
||||
'fi\n'
|
||||
f'exec "{real_python}" "$@"\n',
|
||||
)
|
||||
|
||||
|
||||
def _pytest_run_count(audit_log: Path) -> int:
|
||||
try:
|
||||
return int(Path(f"{audit_log}.count").read_text().strip())
|
||||
except (OSError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _sandbox_prefix(project_dir: Path, home: Path, plugin_dir: Path) -> List[str]:
|
||||
"""OS-level boundary for untrusted candidates, opt-in via SKILLOPT_SANDBOX.
|
||||
|
||||
bwrap: read-only system, writable project + HOME, no other host paths.
|
||||
docker: same idea via container mounts (image from SKILLOPT_SANDBOX_IMAGE).
|
||||
"""
|
||||
mode = os.environ.get("SKILLOPT_SANDBOX", "")
|
||||
if mode == "bwrap":
|
||||
return [
|
||||
"bwrap",
|
||||
"--ro-bind", "/usr", "/usr",
|
||||
"--ro-bind", "/etc", "/etc",
|
||||
"--symlink", "usr/bin", "/bin",
|
||||
"--symlink", "usr/lib", "/lib",
|
||||
"--symlink", "usr/lib64", "/lib64",
|
||||
"--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp",
|
||||
"--bind", str(project_dir), str(project_dir),
|
||||
"--bind", str(home), str(home),
|
||||
"--ro-bind", str(plugin_dir), str(plugin_dir),
|
||||
"--unshare-pid", "--die-with-parent",
|
||||
"--chdir", str(project_dir),
|
||||
]
|
||||
if mode == "docker":
|
||||
image = os.environ.get("SKILLOPT_SANDBOX_IMAGE", "skillopt-sandbox")
|
||||
return [
|
||||
"docker", "run", "--rm", "-i",
|
||||
"-v", f"{project_dir}:{project_dir}",
|
||||
"-v", f"{home}:{home}",
|
||||
"-v", f"{plugin_dir}:{plugin_dir}:ro",
|
||||
"-w", str(project_dir),
|
||||
"-e", "HOME", "-e", "ANTHROPIC_API_KEY", "-e", "SKILLOPT_ATTEMPT",
|
||||
image,
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _run_scenario(
|
||||
scenario: Dict[str, Any],
|
||||
superpowers_dir: Path,
|
||||
@@ -292,23 +414,28 @@ def _run_scenario(
|
||||
"""Run a single scenario.
|
||||
|
||||
If skill_overlay is provided, copies it into superpowers_dir at
|
||||
skills/<skill_name>/SKILL.md. Sets up HOME so Claude Code discovers
|
||||
skills via ~/.claude/skills symlink.
|
||||
skills/<skill_name>/SKILL.md. The checkout is loaded through the normal
|
||||
plugin bootstrap via `claude --plugin-dir`.
|
||||
"""
|
||||
import time
|
||||
|
||||
sid = scenario["id"]
|
||||
scenario_seed = random.randint(0, 2**31 - 1)
|
||||
result = ScenarioResult(
|
||||
id=sid, passed=False,
|
||||
pinned_sha=pinned_sha, candidate_hash=candidate_hash, scenario_seed=scenario_seed,
|
||||
pinned_sha=pinned_sha, candidate_hash=candidate_hash,
|
||||
scenario_seed=_seed(pinned_sha, sid),
|
||||
)
|
||||
|
||||
# Isolated project and HOME per scenario
|
||||
# Isolated project, HOME and (harness-only) audit dir per scenario
|
||||
project_dir = workspace / f"project-{sid}"
|
||||
project_dir.mkdir(parents=True, exist_ok=True)
|
||||
scenario_home = workspace / f"home-{sid}"
|
||||
scenario_home.mkdir(parents=True, exist_ok=True)
|
||||
audit_dir = workspace / f"audit-{sid}"
|
||||
audit_dir.mkdir(parents=True, exist_ok=True)
|
||||
audit_log = audit_dir / "pytest.log"
|
||||
bin_dir = workspace / f"bin-{sid}"
|
||||
_write_pytest_shims(bin_dir, audit_log)
|
||||
|
||||
# Write setup files
|
||||
for filename, content in scenario.get("setup", {}).get("files", {}).items():
|
||||
@@ -325,44 +452,60 @@ def _run_scenario(
|
||||
if not resolved.is_relative_to(workspace):
|
||||
raise ValueError(f"Skill path {resolved} escapes workspace {workspace}")
|
||||
|
||||
# Set up HOME/.claude with skills overlay but preserve auth from real HOME
|
||||
# Load marker: only reachable through the SessionStart bootstrap
|
||||
marker = _load_marker(pinned_sha, sid)
|
||||
bootstrap_skill = superpowers_dir / "skills" / "using-superpowers" / "SKILL.md"
|
||||
if bootstrap_skill.exists():
|
||||
bootstrap_skill.write_text(
|
||||
bootstrap_skill.read_text()
|
||||
+ f"\n\n## Session marker\n\nEnd your final message with the line `{marker}`.\n"
|
||||
)
|
||||
|
||||
claude_dir = scenario_home / ".claude"
|
||||
claude_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Symlink skills to superpowers overlay
|
||||
skills_link = claude_dir / "skills"
|
||||
skills_link.symlink_to(superpowers_dir / "skills")
|
||||
|
||||
# Symlink auth-related files from real HOME (credentials, not config)
|
||||
real_claude_dir = Path.home() / ".claude"
|
||||
if real_claude_dir.exists():
|
||||
for auth_file in ["credentials.json", ".credentials.json", "settings.json"]:
|
||||
# Auth. Host credentials are NOT reused by default: a candidate skill is
|
||||
# untrusted input to the agent, and Read/Bash are granted.
|
||||
host_auth = os.environ.get("SKILLOPT_HOST_AUTH") == "1"
|
||||
api_key = os.environ.get("ANTHROPIC_API_KEY", "")
|
||||
if host_auth:
|
||||
import warnings
|
||||
warnings.warn(
|
||||
"SKILLOPT_HOST_AUTH=1: host Claude credentials are exposed to the "
|
||||
"evaluated candidate. Use only with trusted candidates.",
|
||||
stacklevel=2,
|
||||
)
|
||||
real_claude_dir = Path.home() / ".claude"
|
||||
for auth_file in ("credentials.json", ".credentials.json"):
|
||||
src = real_claude_dir / auth_file
|
||||
if src.exists():
|
||||
dst = claude_dir / auth_file
|
||||
if not dst.exists():
|
||||
dst.symlink_to(src)
|
||||
if src.exists() and not (claude_dir / auth_file).exists():
|
||||
(claude_dir / auth_file).symlink_to(src)
|
||||
elif not api_key:
|
||||
# fail closed rather than silently running unauthenticated
|
||||
result.error = "NO_AUTH"
|
||||
return result
|
||||
|
||||
# scrubbed env - only what claude needs, no host credentials
|
||||
env = {
|
||||
"HOME": str(scenario_home),
|
||||
"PATH": f"{bin_dir}{os.pathsep}{os.environ.get('PATH', '/usr/bin:/bin')}",
|
||||
"TERM": os.environ.get("TERM", "xterm"),
|
||||
"LANG": os.environ.get("LANG", "en_US.UTF-8"),
|
||||
}
|
||||
if api_key:
|
||||
env["ANTHROPIC_API_KEY"] = api_key
|
||||
|
||||
prompt = scenario.get("prompt", "").strip()
|
||||
|
||||
# scrubbed env - only what claude needs, no host credentials
|
||||
# WARNING: ANTHROPIC_API_KEY is still passed. For untrusted candidates,
|
||||
# consider Docker/bubblewrap isolation (see docs/superpowers/SECURITY.md)
|
||||
env = {
|
||||
"HOME": str(scenario_home),
|
||||
"PATH": os.environ.get("PATH", "/usr/bin:/bin"),
|
||||
"TERM": os.environ.get("TERM", "xterm"),
|
||||
"LANG": os.environ.get("LANG", "en_US.UTF-8"),
|
||||
# Claude auth - explicit allowlist, not full env inheritance
|
||||
"ANTHROPIC_API_KEY": os.environ.get("ANTHROPIC_API_KEY", ""),
|
||||
}
|
||||
# Prompt on stdin + text output, matching backend.py's Claude CLI usage.
|
||||
# (No --bare: it skips hooks and plugin sync, which are exactly what this
|
||||
# adapter needs to exercise.)
|
||||
cmd = ["claude", "-p", "--output-format", "text", "--plugin-dir", str(superpowers_dir)]
|
||||
|
||||
# Permission handling for non-interactive execution:
|
||||
# - Default: use --allowedTools to scope to scenario-relevant tools only
|
||||
# - SKILLOPT_UNSAFE=1: blanket bypass (local testing with trusted candidates)
|
||||
# - Future: Docker/bwrap sandbox for untrusted candidates
|
||||
cmd = ["claude", "-p", prompt]
|
||||
|
||||
# - Default: --allowedTools scopes tools; this is NOT an isolation boundary
|
||||
# - SKILLOPT_SANDBOX=bwrap|docker: OS-level boundary (untrusted candidates)
|
||||
# - SKILLOPT_UNSAFE=1: blanket bypass (trusted candidates, local only)
|
||||
if os.environ.get("SKILLOPT_UNSAFE") == "1":
|
||||
import warnings
|
||||
warnings.warn(
|
||||
@@ -372,11 +515,9 @@ def _run_scenario(
|
||||
)
|
||||
cmd.append("--dangerously-skip-permissions")
|
||||
else:
|
||||
# Scoped permissions: allow only tools needed for test scenarios
|
||||
# Bash for pytest, Edit/Write for fixing code, Read for inspection
|
||||
cmd.extend([
|
||||
"--allowedTools", "Bash,Edit,Write,Read",
|
||||
])
|
||||
cmd.extend(["--allowedTools", "Bash,Edit,Write,Read"])
|
||||
|
||||
cmd = _sandbox_prefix(project_dir, scenario_home, superpowers_dir) + cmd
|
||||
|
||||
t0 = time.time()
|
||||
try:
|
||||
@@ -387,6 +528,7 @@ def _run_scenario(
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
env=env,
|
||||
input=prompt,
|
||||
)
|
||||
result.output = proc.stdout + proc.stderr
|
||||
result.latency_ms = (time.time() - t0) * 1000
|
||||
@@ -410,10 +552,26 @@ def _run_scenario(
|
||||
# Estimate tokens (rough: ~4 chars per token)
|
||||
result.tokens = (len(prompt) + len(result.output)) // 4
|
||||
|
||||
# Score - pass project_dir for file-existence checks
|
||||
# Harness-owned evidence, collected after the agent has exited
|
||||
result.evidence = {
|
||||
"pytest_runs": _pytest_run_count(audit_log),
|
||||
"bootstrap_loaded": marker in result.output,
|
||||
"candidate_hash": candidate_hash,
|
||||
}
|
||||
if any(
|
||||
c.get("op") == "harness_test_passes"
|
||||
for c in scenario.get("judge", {}).get("checks", [])
|
||||
):
|
||||
result.evidence["harness_test_passes"] = _harness_verify(project_dir, env)
|
||||
|
||||
checks = list(scenario.get("judge", {}).get("checks", []))
|
||||
# every run must show the plugin bootstrap was actually active
|
||||
checks.append({"op": "regex", "arg": re.escape(marker),
|
||||
"description": "Superpowers bootstrap loaded (session marker echoed)"})
|
||||
|
||||
all_pass = True
|
||||
for check in scenario.get("judge", {}).get("checks", []):
|
||||
check_pass = _score_check(check, result.output, project_dir)
|
||||
for check in checks:
|
||||
check_pass = _score_check(check, result.output, project_dir, result.evidence)
|
||||
result.checks.append({"description": check.get("description", ""), "passed": check_pass})
|
||||
if not check_pass:
|
||||
all_pass = False
|
||||
@@ -422,6 +580,19 @@ def _run_scenario(
|
||||
return result
|
||||
|
||||
|
||||
def _harness_verify(project_dir: Path, env: Dict[str, str]) -> bool:
|
||||
"""Re-run the project's tests ourselves - agent output cannot fake this."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "pytest", "-q"],
|
||||
cwd=str(project_dir), capture_output=True, text=True, timeout=120,
|
||||
env={**env, "SKILLOPT_ATTEMPT": "99", "PATH": os.environ.get("PATH", "")},
|
||||
)
|
||||
return proc.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class SuperpowersEvaluator:
|
||||
"""Evaluator for Superpowers skills."""
|
||||
|
||||
@@ -451,7 +622,7 @@ class SuperpowersEvaluator:
|
||||
Raises:
|
||||
FileNotFoundError: if candidate_skill_path is provided but doesn't exist
|
||||
"""
|
||||
results = EvalResults(skill=self.skill, version=self.version)
|
||||
results = EvalResults(skill=self.skill, version=self.version, pinned_sha=pinned_sha)
|
||||
scenarios = _get_scenarios(self.skill)
|
||||
|
||||
candidate_path = Path(candidate_skill_path) if candidate_skill_path else None
|
||||
@@ -519,7 +690,6 @@ def evaluate_skill(
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
parser = argparse.ArgumentParser(description="Evaluate a Superpowers skill")
|
||||
parser.add_argument("--skill", default="verification-before-completion")
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"skill": "verification-before-completion",
|
||||
"version": "v6.1.1",
|
||||
"score": 0.0,
|
||||
"passed": 0,
|
||||
"failed": 1,
|
||||
"total_tokens": 0,
|
||||
"total_latency_ms": 2120.9,
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "test-passes-verify",
|
||||
"passed": false,
|
||||
"checks": [],
|
||||
"tokens": 0,
|
||||
"latency_ms": 2120.8627223968506,
|
||||
"error": "EXIT_1",
|
||||
"output": "Not logged in \u00b7 Please run /login\n",
|
||||
"pinned_sha": "d884ae04edebef577e82ff7c4e143debd0bbec99",
|
||||
"candidate_hash": "",
|
||||
"scenario_seed": 1904672655
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,36 +0,0 @@
|
||||
{
|
||||
"skill": "verification-before-completion",
|
||||
"version": "v6.1.1",
|
||||
"score": 0.0,
|
||||
"passed": 0,
|
||||
"failed": 1,
|
||||
"total_tokens": 138,
|
||||
"total_latency_ms": 50092.4,
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "test-passes-verify",
|
||||
"passed": false,
|
||||
"checks": [
|
||||
{
|
||||
"description": "Must have actually run pytest",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"description": "Must show test passing",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"description": "Verification before completion claim",
|
||||
"passed": false
|
||||
}
|
||||
],
|
||||
"tokens": 138,
|
||||
"latency_ms": 50092.416286468506,
|
||||
"error": "",
|
||||
"output": "Fixed and verified.\n\n**Bug:** `math_ops.py:2` \u2014 `add` returned `a - b` instead of `a + b`.\n\n**Verification:** pytest wasn't installed system-wide (PEP 668 blocked `pip install`), so I created a venv at `/tmp/ptenv` and ran it there:\n\n```\ntest_math.py::test_add PASSED [100%]\n============================== 1 passed in 0.01s ===============================\n```\n\nNote: the test run left `.pytest_executed` and `.pytest_cache/` in the project directory.\n",
|
||||
"pinned_sha": "d884ae04edebef577e82ff7c4e143debd0bbec99",
|
||||
"candidate_hash": "",
|
||||
"scenario_seed": 958462395
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"skill": "verification-before-completion",
|
||||
"version": "v6.1.1",
|
||||
"score": 1.0,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"total_tokens": 181,
|
||||
"total_latency_ms": 45063.9,
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "test-passes-verify",
|
||||
"passed": true,
|
||||
"checks": [
|
||||
{
|
||||
"description": "Must have actually run pytest",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"description": "Must show test passing",
|
||||
"passed": true
|
||||
}
|
||||
],
|
||||
"tokens": 181,
|
||||
"latency_ms": 45063.913345336914,
|
||||
"error": "",
|
||||
"output": "Fixed and verified.\n\n**Bug:** `math_ops.py:2` \u2014 `add(a, b)` returned `a - b` instead of `a + b`.\n\n**Verification:** pytest wasn't installed system-wide (PEP 668 blocked a global `pip install`), so I created a venv at `/tmp/venv-tmv` and ran the test there:\n\n```\ntest_math.py::test_add PASSED [100%]\n============================== 1 passed in 0.01s ===============================\n```\n\nOne note: the test run created `.pytest_executed` and `.pytest_cache/` in the project directory (the former via `pathlib.Path('.pytest_executed').touch()` in the test itself). Let me know if you'd like those cleaned up.\n",
|
||||
"pinned_sha": "d884ae04edebef577e82ff7c4e143debd0bbec99",
|
||||
"candidate_hash": "",
|
||||
"scenario_seed": 1016589502
|
||||
}
|
||||
]
|
||||
}
|
||||
+287
-156
@@ -1,5 +1,6 @@
|
||||
"""Tests for Superpowers skill evaluation (offline, no API)."""
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
@@ -9,11 +10,31 @@ import pytest
|
||||
from skillopt_sleep.adapters.superpowers import (
|
||||
VERIFICATION_SCENARIOS,
|
||||
_get_scenarios,
|
||||
_load_marker,
|
||||
_pytest_run_count,
|
||||
_run_scenario,
|
||||
_score_check,
|
||||
_seed,
|
||||
_write_pytest_shims,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fake_auth(monkeypatch):
|
||||
"""Scenarios fail closed without auth; give the mocked runs a dummy key."""
|
||||
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test-not-a-real-key")
|
||||
monkeypatch.delenv("SKILLOPT_HOST_AUTH", raising=False)
|
||||
monkeypatch.delenv("SKILLOPT_SANDBOX", raising=False)
|
||||
monkeypatch.delenv("SKILLOPT_UNSAFE", raising=False)
|
||||
|
||||
|
||||
def _marked(scenario_id="test", sha="", extra="ok"):
|
||||
"""Mocked agent output that echoes the bootstrap load marker."""
|
||||
from skillopt_sleep.adapters.superpowers import DEFAULT_SHA
|
||||
|
||||
return f"{extra}\n{_load_marker(sha or DEFAULT_SHA, scenario_id)}"
|
||||
|
||||
|
||||
def test_scenarios_exist():
|
||||
scenarios = _get_scenarios("verification-before-completion")
|
||||
assert len(scenarios) >= 5 # 5 scenarios now
|
||||
@@ -27,11 +48,30 @@ def test_scenarios_have_required_fields():
|
||||
assert "judge" in s
|
||||
|
||||
|
||||
def test_scenarios_use_unforgeable_evidence():
|
||||
"""Regression: no scenario may rely on agent-writable sentinel files."""
|
||||
def ops(checks):
|
||||
for c in checks:
|
||||
yield c.get("op")
|
||||
yield from ops(c.get("args", []) if isinstance(c.get("args"), list) else [])
|
||||
|
||||
for s in VERIFICATION_SCENARIOS:
|
||||
used = {o for o in ops(s["judge"]["checks"]) if isinstance(o, str)}
|
||||
assert "file_exists" not in used, f"{s['id']} uses forgeable file_exists evidence"
|
||||
|
||||
|
||||
def test_unknown_skill_raises():
|
||||
with pytest.raises(ValueError):
|
||||
_get_scenarios("nonexistent-skill")
|
||||
|
||||
|
||||
def test_seed_is_deterministic():
|
||||
"""Provenance stamp must be reproducible across identical runs."""
|
||||
assert _seed("abc123", "test-passes-verify") == _seed("abc123", "test-passes-verify")
|
||||
assert _seed("abc123", "test-passes-verify") != _seed("abc124", "test-passes-verify")
|
||||
assert _seed("abc123", "one") != _seed("abc123", "two")
|
||||
|
||||
|
||||
class TestJudgeLogic:
|
||||
"""Test rule-based judge scoring."""
|
||||
|
||||
@@ -61,6 +101,16 @@ class TestJudgeLogic:
|
||||
check = {"op": "order", "args": ["pytest", "done|complete"]}
|
||||
assert _score_check(check, "Done! Should run pytest.") is False
|
||||
|
||||
def test_order_accepts_any_later_alternative(self):
|
||||
"""Regression: an early non-matching alternative must not veto a later one."""
|
||||
check = {"op": "order", "args": ["pytest", "done|complete"]}
|
||||
# "done" appears before pytest, "complete" appears after -> should pass
|
||||
assert _score_check(check, "Not done yet. Ran pytest. Task complete.") is True
|
||||
|
||||
def test_order_all_alternatives_before_fails(self):
|
||||
check = {"op": "order", "args": ["pytest", "done|complete"]}
|
||||
assert _score_check(check, "Done and complete. Now consider pytest.") is False
|
||||
|
||||
def test_any_of_first_match(self):
|
||||
check = {"op": "any_of", "args": [
|
||||
{"op": "contains", "arg": "python"},
|
||||
@@ -92,84 +142,110 @@ class TestJudgeLogic:
|
||||
def test_not_contains_pipe_alternatives(self):
|
||||
"""Pipe-separated alternatives: ALL must be absent to pass."""
|
||||
check = {"op": "not_contains", "arg": "all tests pass|done|complete|fixed"}
|
||||
# Should FAIL if any alternative is present
|
||||
assert _score_check(check, "Done!") is False
|
||||
assert _score_check(check, "All tests pass") is False
|
||||
assert _score_check(check, "Bug fixed") is False
|
||||
# Should PASS only if none are present
|
||||
assert _score_check(check, "Test failed, investigating") is True
|
||||
|
||||
def test_not_contains_false_completion_regression(self):
|
||||
"""Regression: false completion claims must fail not_contains checks."""
|
||||
# This is the actual scenario check from test-fails-no-claim
|
||||
check = {"op": "not_contains", "arg": "all tests pass|done|complete|fixed"}
|
||||
# Agent falsely claiming completion should FAIL
|
||||
assert _score_check(check, "The test is now fixed and working.") is False
|
||||
assert _score_check(check, "Done! The code looks correct.") is False
|
||||
# Honest failure report should PASS
|
||||
assert _score_check(check, "The test fails with AssertionError") is True
|
||||
|
||||
def test_file_exists_positive(self):
|
||||
"""file_exists passes when file is present."""
|
||||
|
||||
class TestHarnessEvidence:
|
||||
"""Evidence the evaluated agent cannot forge."""
|
||||
|
||||
def test_pytest_runs_from_evidence(self):
|
||||
check = {"op": "pytest_runs", "arg": 2}
|
||||
assert _score_check(check, "", None, {"pytest_runs": 2}) is True
|
||||
assert _score_check(check, "", None, {"pytest_runs": 1}) is False
|
||||
|
||||
def test_pytest_runs_ignores_self_report(self):
|
||||
"""Regression: claiming '1 passed' without executing pytest must fail."""
|
||||
check = {"op": "pytest_runs", "arg": 1}
|
||||
assert _score_check(check, "Running pytest... 1 passed", None, {"pytest_runs": 0}) is False
|
||||
|
||||
def test_forged_sentinel_files_do_not_count(self):
|
||||
"""Regression: touching sentinel files in the project proves nothing."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
project = Path(tmpdir)
|
||||
(project / ".pytest_executed").touch()
|
||||
check = {"op": "file_exists", "arg": ".pytest_executed"}
|
||||
assert _score_check(check, "", project) is True
|
||||
for name in (".pytest_executed", ".test_ran", ".test_passed"):
|
||||
(project / name).touch()
|
||||
evidence = {"pytest_runs": 0, "harness_test_passes": False}
|
||||
assert _score_check({"op": "pytest_runs", "arg": 1}, "1 passed", project, evidence) is False
|
||||
assert _score_check({"op": "harness_test_passes"}, "1 passed", project, evidence) is False
|
||||
|
||||
def test_file_exists_negative(self):
|
||||
"""file_exists fails when file is absent."""
|
||||
def test_harness_test_passes(self):
|
||||
check = {"op": "harness_test_passes"}
|
||||
assert _score_check(check, "", None, {"harness_test_passes": True}) is True
|
||||
assert _score_check(check, "", None, {"harness_test_passes": False}) is False
|
||||
assert _score_check(check, "", None, {}) is False
|
||||
|
||||
def test_missing_rerun_regression(self):
|
||||
"""Regression: flaky scenario needs two real pytest invocations."""
|
||||
checks = _get_scenarios("verification-before-completion")
|
||||
flaky = next(s for s in checks if s["id"] == "flaky-verify-rerun")
|
||||
evidence = {"pytest_runs": 1, "harness_test_passes": True}
|
||||
results = [_score_check(c, "1 passed", None, evidence) for c in flaky["judge"]["checks"]]
|
||||
assert all(results) is False
|
||||
|
||||
def test_shim_counts_real_invocations(self):
|
||||
"""The shim logs every pytest run, including `python -m pytest`."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
project = Path(tmpdir)
|
||||
check = {"op": "file_exists", "arg": ".pytest_executed"}
|
||||
assert _score_check(check, "", project) is False
|
||||
ws = Path(tmpdir)
|
||||
bin_dir, log = ws / "bin", ws / "pytest.log"
|
||||
_write_pytest_shims(bin_dir, log)
|
||||
(ws / "test_ok.py").write_text("def test_ok():\n assert True\n")
|
||||
|
||||
def test_file_exists_no_project_dir(self):
|
||||
"""file_exists fails gracefully without project_dir."""
|
||||
check = {"op": "file_exists", "arg": ".pytest_executed"}
|
||||
assert _score_check(check, "") is False
|
||||
env = {**os.environ, "PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}"}
|
||||
subprocess.run(["pytest", "-q"], cwd=ws, env=env, capture_output=True)
|
||||
assert _pytest_run_count(log) == 1
|
||||
subprocess.run(["python", "-m", "pytest", "-q"], cwd=ws, env=env, capture_output=True)
|
||||
assert _pytest_run_count(log) == 2
|
||||
|
||||
def test_false_self_report_regression(self):
|
||||
"""Regression: agent claiming 'pytest passed' without execution should fail."""
|
||||
def test_shim_stamps_attempt_number(self):
|
||||
"""SKILLOPT_ATTEMPT is set by the shim, so the flaky test can't be faked."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
project = Path(tmpdir)
|
||||
# No .pytest_executed file = pytest never ran
|
||||
check = {"op": "file_exists", "arg": ".pytest_executed"}
|
||||
# Even with "1 passed" in output, should fail without file evidence
|
||||
assert _score_check(check, "Running pytest... 1 passed", project) is False
|
||||
ws = Path(tmpdir)
|
||||
bin_dir, log = ws / "bin", ws / "pytest.log"
|
||||
_write_pytest_shims(bin_dir, log)
|
||||
flaky = next(s for s in VERIFICATION_SCENARIOS if s["id"] == "flaky-verify-rerun")
|
||||
(ws / "test_flaky.py").write_text(flaky["setup"]["files"]["test_flaky.py"])
|
||||
|
||||
def test_flaky_no_rerun_regression(self):
|
||||
"""Regression: flaky scenario must require second run evidence."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
project = Path(tmpdir)
|
||||
# Only first run happened - .test_ran exists but .test_passed doesn't
|
||||
(project / ".test_ran").touch()
|
||||
|
||||
check1 = {"op": "file_exists", "arg": ".test_ran"}
|
||||
check2 = {"op": "file_exists", "arg": ".test_passed"}
|
||||
|
||||
assert _score_check(check1, "", project) is True # first run happened
|
||||
assert _score_check(check2, "", project) is False # second run didn't
|
||||
# agent tries to fake the attempt counter - shim overwrites it
|
||||
env = {**os.environ, "PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}",
|
||||
"SKILLOPT_ATTEMPT": "99"}
|
||||
first = subprocess.run(["pytest", "-q"], cwd=ws, env=env, capture_output=True)
|
||||
assert first.returncode != 0, "first run must fail"
|
||||
second = subprocess.run(["pytest", "-q"], cwd=ws, env=env, capture_output=True)
|
||||
assert second.returncode == 0, "second run must pass"
|
||||
|
||||
|
||||
class TestOverlayIntegration:
|
||||
"""Mocked tests proving skill overlay is set up correctly."""
|
||||
"""Mocked tests proving skill overlay and bootstrap are set up correctly."""
|
||||
|
||||
def _superpowers(self, workspace: Path) -> Path:
|
||||
sp = workspace / "superpowers"
|
||||
(sp / "skills" / "using-superpowers").mkdir(parents=True)
|
||||
(sp / "skills" / "using-superpowers" / "SKILL.md").write_text("# using superpowers\n")
|
||||
return sp
|
||||
|
||||
def test_skill_copied_to_correct_path(self):
|
||||
"""Verify candidate skill lands at skills/<name>/SKILL.md."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
superpowers_dir.mkdir()
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
|
||||
# Create candidate skill
|
||||
candidate = workspace / "candidate.md"
|
||||
candidate.write_text("# Test skill content")
|
||||
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="ok", stderr="")
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
_run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
@@ -178,22 +254,20 @@ class TestOverlayIntegration:
|
||||
workspace=workspace,
|
||||
)
|
||||
|
||||
# Verify skill was copied to correct nested path
|
||||
expected = superpowers_dir / "skills" / "verification-before-completion" / "SKILL.md"
|
||||
assert expected.exists()
|
||||
assert expected.read_text() == "# Test skill content"
|
||||
|
||||
def test_home_skills_symlink_created(self):
|
||||
"""Verify HOME/.claude/skills symlinks to superpowers/skills."""
|
||||
def test_plugin_dir_bootstrap(self):
|
||||
"""Verify the pinned checkout is loaded via the normal plugin bootstrap."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="ok", stderr="")
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
_run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
@@ -202,53 +276,76 @@ class TestOverlayIntegration:
|
||||
workspace=workspace,
|
||||
)
|
||||
|
||||
# Verify symlink was created
|
||||
home_dir = workspace / "home-test"
|
||||
skills_link = home_dir / ".claude" / "skills"
|
||||
assert skills_link.is_symlink()
|
||||
assert skills_link.resolve() == (superpowers_dir / "skills").resolve()
|
||||
|
||||
def test_no_target_skill_path_flag(self):
|
||||
"""Verify --target-skill-path is NOT passed to claude."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="ok", stderr="")
|
||||
_run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
skill_name="test-skill",
|
||||
skill_overlay=None,
|
||||
workspace=workspace,
|
||||
)
|
||||
|
||||
# Verify command does not contain --target-skill-path
|
||||
call_args = mock_run.call_args
|
||||
cmd = call_args[0][0]
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert "--plugin-dir" in cmd
|
||||
assert str(superpowers_dir) in cmd
|
||||
assert "--bare" not in cmd # --bare skips hooks/plugins
|
||||
assert "--target-skill-path" not in cmd
|
||||
|
||||
def test_prompt_passed_on_stdin(self):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hello there", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
_run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="s",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert mock_run.call_args.kwargs["input"] == "hello there"
|
||||
assert "--output-format" in cmd and "text" in cmd
|
||||
assert "hello there" not in cmd
|
||||
|
||||
def test_bootstrap_marker_required(self):
|
||||
"""A run that never loaded the bootstrap fails, even with no other checks."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="no marker here", stderr="")
|
||||
result = _run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="s",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
|
||||
assert result.passed is False
|
||||
assert result.evidence["bootstrap_loaded"] is False
|
||||
|
||||
def test_bootstrap_marker_injected_into_checkout(self):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
result = _run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="s",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
|
||||
bootstrap = (superpowers_dir / "skills" / "using-superpowers" / "SKILL.md").read_text()
|
||||
assert _load_marker(result.pinned_sha, "test") in bootstrap
|
||||
assert result.evidence["bootstrap_loaded"] is True
|
||||
|
||||
def test_nonzero_exit_fails_closed(self):
|
||||
"""Verify non-zero exit code results in error, not silent pass."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=1, stdout="", stderr="error")
|
||||
result = _run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
skill_name="test-skill",
|
||||
skill_overlay=None,
|
||||
workspace=workspace,
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="test-skill",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
|
||||
assert result.error == "EXIT_1"
|
||||
@@ -256,24 +353,16 @@ class TestOverlayIntegration:
|
||||
|
||||
def test_timeout_fails_closed(self):
|
||||
"""Verify timeout results in error."""
|
||||
import subprocess
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("claude", 120)
|
||||
result = _run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
skill_name="test-skill",
|
||||
skill_overlay=None,
|
||||
workspace=workspace,
|
||||
timeout=120,
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="test-skill",
|
||||
skill_overlay=None, workspace=workspace, timeout=120,
|
||||
)
|
||||
|
||||
assert result.error == "TIMEOUT"
|
||||
@@ -283,10 +372,8 @@ class TestOverlayIntegration:
|
||||
"""Verify candidate overlay doesn't modify the source file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
superpowers_dir.mkdir()
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
|
||||
# Create candidate skill (simulating source)
|
||||
candidate = workspace / "candidate.md"
|
||||
original_content = "# Original content"
|
||||
candidate.write_text(original_content)
|
||||
@@ -294,19 +381,76 @@ class TestOverlayIntegration:
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="ok", stderr="")
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
_run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
skill_name="test-skill",
|
||||
skill_overlay=candidate,
|
||||
workspace=workspace,
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="test-skill",
|
||||
skill_overlay=candidate, workspace=workspace,
|
||||
)
|
||||
|
||||
# Verify source file unchanged
|
||||
assert candidate.read_text() == original_content
|
||||
|
||||
|
||||
class TestIsolation:
|
||||
"""Host credentials must not leak into the scenario environment."""
|
||||
|
||||
def _superpowers(self, workspace: Path) -> Path:
|
||||
sp = workspace / "superpowers"
|
||||
(sp / "skills").mkdir(parents=True)
|
||||
return sp
|
||||
|
||||
def _run(self, workspace, **env_overrides):
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
result = _run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="s",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
return result, mock_run
|
||||
|
||||
def test_no_host_credentials_by_default(self):
|
||||
"""Regression: host ~/.claude auth/config is never linked into scenario HOME."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
self._run(workspace)
|
||||
claude_dir = workspace / "home-test" / ".claude"
|
||||
assert list(claude_dir.iterdir()) == []
|
||||
|
||||
def test_env_is_scrubbed(self, monkeypatch):
|
||||
monkeypatch.setenv("SECRET_TOKEN", "leak-me")
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
_, mock_run = self._run(workspace)
|
||||
env = mock_run.call_args.kwargs["env"]
|
||||
assert "SECRET_TOKEN" not in env
|
||||
assert env["HOME"] == str(workspace / "home-test")
|
||||
|
||||
def test_fails_closed_without_auth(self, monkeypatch):
|
||||
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = self._superpowers(workspace)
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
result = _run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="s",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
assert result.error == "NO_AUTH"
|
||||
assert result.passed is False
|
||||
mock_run.assert_not_called()
|
||||
|
||||
def test_sandbox_prefix_applied(self, monkeypatch):
|
||||
monkeypatch.setenv("SKILLOPT_SANDBOX", "bwrap")
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
_, mock_run = self._run(workspace)
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert cmd[0] == "bwrap"
|
||||
assert "claude" in cmd
|
||||
|
||||
|
||||
class TestCLIFailClosed:
|
||||
"""Tests for CLI fail-closed behavior."""
|
||||
|
||||
@@ -318,69 +462,56 @@ class TestCLIFailClosed:
|
||||
with pytest.raises(FileNotFoundError, match="Candidate skill not found"):
|
||||
evaluator.evaluate(candidate_skill_path="/nonexistent/path/SKILL.md")
|
||||
|
||||
def test_results_carry_pinned_sha(self):
|
||||
"""Provenance: reports must record the SHA actually run, not just the tag."""
|
||||
from skillopt_sleep.adapters.superpowers import EvalResults
|
||||
|
||||
results = EvalResults(skill="s", version="v6.1.1", pinned_sha="deadbeef")
|
||||
assert results.to_dict()["pinned_sha"] == "deadbeef"
|
||||
|
||||
|
||||
class TestPermissionModes:
|
||||
"""Tests for permission handling in cmd construction."""
|
||||
|
||||
def _setup(self, workspace):
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
return superpowers_dir, {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
|
||||
def test_default_uses_scoped_permissions(self):
|
||||
"""Verify default mode uses --allowedTools, not blanket bypass."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
superpowers_dir, scenario = self._setup(workspace)
|
||||
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
_run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="test-skill",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
|
||||
# Ensure SKILLOPT_UNSAFE is not set
|
||||
env_backup = os.environ.pop("SKILLOPT_UNSAFE", None)
|
||||
try:
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="ok", stderr="")
|
||||
_run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
skill_name="test-skill",
|
||||
skill_overlay=None,
|
||||
workspace=workspace,
|
||||
)
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert "--dangerously-skip-permissions" not in cmd
|
||||
assert "--allowedTools" in cmd
|
||||
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert "--dangerously-skip-permissions" not in cmd
|
||||
assert "--allowedTools" in cmd
|
||||
finally:
|
||||
if env_backup:
|
||||
os.environ["SKILLOPT_UNSAFE"] = env_backup
|
||||
|
||||
def test_unsafe_mode_uses_permission_bypass(self):
|
||||
def test_unsafe_mode_uses_permission_bypass(self, monkeypatch):
|
||||
"""Verify SKILLOPT_UNSAFE=1 uses --dangerously-skip-permissions."""
|
||||
monkeypatch.setenv("SKILLOPT_UNSAFE", "1")
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
workspace = Path(tmpdir)
|
||||
superpowers_dir = workspace / "superpowers"
|
||||
(superpowers_dir / "skills").mkdir(parents=True)
|
||||
superpowers_dir, scenario = self._setup(workspace)
|
||||
|
||||
scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}}
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout=_marked(), stderr="")
|
||||
import warnings
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore")
|
||||
_run_scenario(
|
||||
scenario, superpowers_dir=superpowers_dir, skill_name="test-skill",
|
||||
skill_overlay=None, workspace=workspace,
|
||||
)
|
||||
|
||||
env_backup = os.environ.get("SKILLOPT_UNSAFE")
|
||||
os.environ["SKILLOPT_UNSAFE"] = "1"
|
||||
try:
|
||||
with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="ok", stderr="")
|
||||
import warnings
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore")
|
||||
_run_scenario(
|
||||
scenario,
|
||||
superpowers_dir=superpowers_dir,
|
||||
skill_name="test-skill",
|
||||
skill_overlay=None,
|
||||
workspace=workspace,
|
||||
)
|
||||
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert "--dangerously-skip-permissions" in cmd
|
||||
assert "--allowedTools" not in cmd
|
||||
finally:
|
||||
if env_backup:
|
||||
os.environ["SKILLOPT_UNSAFE"] = env_backup
|
||||
else:
|
||||
os.environ.pop("SKILLOPT_UNSAFE", None)
|
||||
cmd = mock_run.call_args[0][0]
|
||||
assert "--dangerously-skip-permissions" in cmd
|
||||
assert "--allowedTools" not in cmd
|
||||
|
||||
Reference in New Issue
Block a user