github-featured/wanshuiyin--auto-claude-code-research-in-sleep

Files

T

Ruofeng Yang b20577e56b feat(audit): deterministic evidence pre-check before the claim jury (Hermes borrow #3 )

Third Hermes-borrow increment — the reconcile pattern (a model's self-report
cross-checked against mechanical ground-truth). Before spending a cross-model
codex call to judge whether a claim is supported, mechanically verify the cited
evidence EXISTS: does the source file exist, and does the cited number/string
actually appear in it? Catches hallucinated evidence for free.

Two-stage, with ARIS's boundary: stage 1 (this, deterministic) catches
HALLUCINATION (cited path missing / value absent); stage 2 (the cross-model
jury) catches WRONG-BUT-REAL (number is there but doesn't support the claim). A
`verified` means ONLY "the evidence exists" — existence is execution-completeness
(safe same-model / deterministic); support stays the jury's verdict
(acceptance-gate: drive, not acquit).

- tools/evidence_check.py — check_claim / check_batch + CLI. The numeric matcher
  is SAFETY-CRITICAL (must never false-`verify`): allow-list token boundaries
  (only string start/end, Unicode whitespace, and a small ASCII set are safe —
  everything else fails closed), exact Decimal equality (no float tolerance) with
  percent-flag consistency, and a post-match whitespace-grouping guard. Fails
  closed on dates/times/versions/fractions/locale grouping/Unicode delimiters →
  those go to the jury; real numbers (73.2≡73.20, 1,000≡1000, .5≡0.5, sci
  notation, JSON, normal sentences) still match.
- skills/shared-references/evidence-precheck.md — the two-stage convention,
  conservative-by-design (false-negative over false-positive), verified≠correct.
- skills/result-to-claim/SKILL.md — Step 1.5: pre-check cited (value, source),
  downgrade hallucinated evidence to unsupported before the codex call, and pass
  the per-claim status into the Step-2 jury prompt.
- tests/test_evidence_check.py — 12 tests incl. adversarial suites (rounds 1–5:
  word-embedding, sci-notation, percent, float precision, comma/space/apostrophe
  grouping, .5/1., fractions, dates, times, versions, locale decimals, Unicode
  minus/dash/fullwidth/thin-space, multi-whitespace) — all fail closed.

Cross-model reviewed by codex GPT-5.5 xhigh over 5 fix rounds (+ Gemini
independent final check): DO-NOT-SHIP → each round hardened the matcher against a
new false-`verified` class → SHIP. The no-false-verify property is now
allow-list-based (fail-closed by construction), not a deny-list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-05-30 16:42:41 +08:00

3.2 KiB

Raw Permalink Blame History

Evidence Pre-check

ARIS's claim audits (/result-to-claim, /experiment-audit, /paper-claim-audit) spend a cross-model (codex/gemini) call to judge whether a claim is supported. The cheapest, most common integrity failure is hallucinated evidence: a claim cites a number + a source file, and the file doesn't exist or the number isn't in it. You should not need a model call to catch that.

Two stages — and `verified` ≠ `correct`

stage 1  tools/evidence_check.py   deterministic · no model · fail-closed
         catches HALLUCINATION — cited path missing, or cited value not in source.
stage 2  the cross-model jury      codex/gemini
         catches WRONG-BUT-REAL — the number IS in the file, but it doesn't
         support the claim.

A verified from stage 1 means only that the cited evidence exists — never that the claim holds. Existence is execution-completeness (deterministic / safe same-model); support is a quality verdict that stays with the cross-model jury (acceptance-gate.md: the pre-check DRIVES a gate, it cannot ACQUIT a claim). This is the reconcile pattern — a model's self-report cross-checked against mechanical ground-truth (adapted from Hermes's curator reconcile-classifier), made into a cheap pre-gate that catches hallucination before the jury runs and spares the codex call on fabricated evidence.

Conservative by design

The pre-check favors false-negative over false-positive: when in doubt it returns not-verified and lets the jury decide — it must never emit a false verified. A pure number is matched by numeric-token equality (so 73.2 matches 73.20 but 73 does NOT match 73.5); a non-numeric value by normalized substring.

Where ARIS uses it

/result-to-claim Step 1.5: parse each claim's cited (value, source), run the batch pre-check, and before the codex judgment mark any claim whose evidence is path_missing / value_not_found as unsupported — evidence not found, and pass the per-claim pre-check status into the codex prompt so the jury sees which claims have verified vs hallucinated evidence.
To extend: /experiment-audit (the "phantom results" check is exactly this) and /paper-claim-audit (every reported number → its result file).

API / CLI

from evidence_check import check_claim, check_batch
check_claim(value, source, root=".")   # -> {status: verified|path_missing|value_not_found, ...}
check_batch([{value, source, id?}, ...], root)  # -> {results:[...], summary:{status: n}}

python3 tools/evidence_check.py <root> --value 73.2 --source results/eval.json   # exit 0 verified
python3 tools/evidence_check.py <root> --batch claims.json   # exit 1 if any claim hallucinated

Cross-references

acceptance-gate.md — the pre-check is the deterministic DRIVE; the jury is the ACQUIT. verified is existence (execution-completeness), not correctness.
reviewer-independence.md — the jury still reads the artifacts itself; the pre-check only flags which claims have evidence to read, never pre-digests the verdict.
experiment-integrity.md — fabricated/phantom results are exactly what stage 1 catches deterministically before stage 2.

3.2 KiB Raw Permalink Blame History