The field parser in format_bibtex.py and validate_citations.py matched
`\{([^}]*)\}`, which stops at the first closing brace. Any title with a
protected term -- `{Highly accurate prediction with {AlphaFold}}` -- was
truncated mid-value, and rewriting it produced a .bib with unbalanced braces
that no BibTeX engine reads. extract_metadata.py emits that protection
deliberately, so the documented Phase 2 -> Phase 3 pipeline corrupted its own
output, and format_bibtex.py defaulted to overwriting the input in place.
Replace both parsers with a brace-depth scanner in scripts/_common.py, which
also owns entry rendering, page normalisation, and one citation-key scheme
shared by every producer -- entries from Crossref, PubMed, OpenAlex, and
Scholar now collide when they are the same paper, which is what makes
deduplication work at all. Writing is opt-in: without --output or --in-place
the result goes to stdout and the input is untouched.
Also fixed:
- pages: `replace('-', '--')` turned `583--589` into `583----589`; PubMed's
abbreviated `1123-30` now expands to `1123--1130` rather than staying a
range that resolves to nothing.
- publisher was read from Crossref and dropped by the writer, so every @book
and @techreport failed the required-field check.
- an arXiv preprint with a DataCite DOI was typed @article with no journal --
guaranteed to fail validation. It is @misc unless journal_ref names a venue.
- --url and PMCID were classified by identify_type but had no handler, so a
publisher URL without a DOI in its path errored out. URLs now resolve via
the citation_doi meta tag; PMCIDs via the NCBI ID converter.
- citation keys were unsanitised in search_pubmed.py: `Müller2021pmid123`
and `O'Brien2021pmid124` are not valid BibTeX keys.
- DOIs were interpolated into API URLs unquoted.
- verify_doi used HEAD on doi.org, which follows to the publisher; several
answer HEAD with 403 behind a bot check, so good DOIs looked unresolvable.
Ask Crossref, then DataCite, and treat transport failures as unknown.
- valid_entries subtracted the error count from the entry count and reported
`Valid entries: -2` for a one-entry file.
- structured abstracts kept only their first section; titles lost inline
markup and carried PubMed's trailing full stop.
- two `note` fields could be emitted in one entry.
- --auto-fix was advertised in --help as "not implemented yet" and did nothing.
- venue_standards was assigned inside validate_file, so a fresh validator did
not have it.
Add search_openalex.py: ~250M works, keyless REST API, cursor pagination,
abstracts rebuilt from the inverted index. The skill's own "single source
bias" pitfall previously had only two sources to offer, one of them scraped.
Drop the bundled schematic generators (1,148 lines) in favour of the
scientific-schematics skill, and update the shared-copy contract accordingly.
Reference-list length is editorial judgement, so a venue shortfall is now a
warning; only an explicit --min-count is enforced as an error.
Dependencies drop from eight packages to two: bibtexparser, biopython,
crossref-commons, pylatexenc, and selenium were declared but never imported,
and the parsing layer is now standard library.
Timothy Kassis
d39264b47c