AGENTS.md has always said that a skill shipping scripts/ puts its tests in
tests/<name>/, but nothing checked it: 54 of the 100 such skills had no suite
at all, including docx, pptx, xlsx, pdf and scanpy. All 100 now do.
tests/_meta is the guard. It runs the shared structural contract across every
skill in a single process -- safe because it parses scripts with ast and never
imports them -- and fails when a skill ships scripts/ without a suite or
without a [skills.<name>] entry in skill-requirements.toml. It needs no
scientific packages and finishes in seconds, so skill-tests.yml blocks every
pull request on it, plus the packages=[] suites.
tests/_contract holds what the per-skill suites were each reimplementing:
frontmatter conformance, the 500-line limit, no tests or bytecode under
skills/, local links resolving, scripts parsing, no eval/exec/os.system, no
standard-library shadowing, no hardcoded local paths, valid shell scripts. Also
the --help contract, which skips when a skill's packages are absent and runs
for real under --isolated, and shared behaviour for the files docx/pptx/xlsx
and five schematic-shipping skills carry byte-identical copies of, with drift
detection so they cannot diverge silently.
67 new suites, 3325 test functions. The existing suites were retrofitted: 18
no longer pin an exact skill version, so a version bump no longer breaks a
test; 11 duplicated structural methods removed; 19 wired to the --help
contract; and 5 that failed collection without their packages now skip
cleanly. run_all.py in the bare project environment goes from 6 failures to 0.
Writing the tests surfaced 19 defects in the skills, fixed here with version
bumps. The ones that changed scientific output:
- openpiv reported vorticity 0 for a rotating flow, from a sign error in
openpiv's y-up coordinate relabelling; solid-body rotation now gives 2w
exactly, on grids of either orientation
- deepchem returned solubility predictions in z-scored space while labelling
them log(mol/L), because it transformed a y-less dataset instead of
untransforming the output
- neuropixels-analysis had the Allen and IBL ISI thresholds swapped,
contradicting its own references/QUALITY_METRICS.md and inverting the two
standards' relative strictness
- scanpy's summarize() raised TypeError on every AnnData under anndata 0.13,
which reports an unnamed None key on .layers; scanpy convert was broken
- experimental-design's Latin hypercube was never reproducible: pyDOE3 draws
from its own default_rng and ignores numpy's global seed
- primekg shipped a hardcoded path naming a person, which is why
no_personal_paths is now a contract rule
The remainder is upstream API drift, each verified against the installed
package: retired symbols in bioservices 1.16, gget helpers that returned lists
where a string was written, ArviZ 1.x kwargs in pymc, a positional-only
factory in pymoo, ReduceLROnPlateau(verbose=) in torch 2.13, a removed scvelo
parameter, and PyPDF2 in scientific-slides.
Two manifest environments could not build and are pinned: gget, where an
unpinned scanpy walked back to 1.9.8 and pulled llvmlite 0.36 which does not
compile on 3.13, and pymatgen, pinned to the snapshot its own _common.py
enforces rather than loosening that check. deepchem gains torch, without which
no model class exists.
python tests/run_all.py --isolated: 101 passed, 0 failed.
Timothy Kassis
4fb7e0bc29