* fix(report): flag a partial LLM failure as degraded, not only a total one
_llm_runtime_status() only set degraded when every LLM call failed
(succeeded == 0). A rate-limited provider that drops a single batch
(e.g. semantic_security_discovery hits a 429) still has succeeded > 0,
so the scan reported a normal risk_assessment even though the
security-critical analyzer never ran. Widen the condition to
succeeded < attempted, so any dropped batch degrades the scan and
the existing fail-closed floor (CAUTION instead of SAFE) applies to a
partial pass too. Updated the two degraded-scan messages to say how
many of the calls failed instead of assuming all of them did.
Covers request 3 of #303 (surface incompleteness in the verdict).
Request 1 (configurable concurrency) shipped in #305; request 2
(retry with backoff) is left to the already-open #29.
Refs #303
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
* fix(report): mark a batch record failed on any dropped batch, and stop meta-analysis fields inheriting other analyzers' failures
Two gaps from review on #362:
1. llm_call_log records were built with
ok=bool(outcome.successful) or not outcome.failures, so an analyzer with
one succeeded batch and one dropped/429'd batch still recorded ok=True.
In that exact case succeeded == attempted at the report layer and the
scan stayed SAFE, defeating the partial-coverage fix. Now the record is
ok=not outcome.failures: any dropped batch marks the whole record failed.
Applied identically in the three semantic analyzers and meta_analyzer,
the four call sites that build this record.
2. meta_analysis_applied and the llm_available field were derived from the
aggregate `degraded` flag, which pools every LLM-backed node together.
That let a different analyzer's dropped batch force
meta_analysis_applied=False, filtering_mode="heuristic" and
llm_available=False even when meta_analyzer's own call fully succeeded,
misstating two independent contracts (meta-analysis ran vs. some
coverage was lost) as one boolean. Both fields now derive from
is_llm_available() plus meta_analyzer's own llm_call_log record only;
the coverage loss from other analyzers still surfaces through
llm_degraded / llm_calls_attempted / llm_calls_succeeded, unchanged.
Verified: test_partial_batch_failure_records_llm_failure (renamed from
..._records_llm_success, now pins ok=False) and three new report-level
tests, run red against the pre-fix code (3 of 4 failed) and green after.
tests/nodes/test_report.py: 66 passed. Full suite in Docker
(python:3.12-slim): 1947 passed, 13 skipped, 4 xfailed, 0 failed. ruff
lint and format-check both pass.
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
* fix(report): require an actual meta_analyzer record for meta_analysis_applied
all([]) is True on an empty list, so an empty meta_analyzer_records left
meta_analysis_applied True even when meta_analyzer made no call at all (the
no-findings path, where it short-circuits to not_applicable). That still
violated the "did meta-analysis actually run" contract from the prior
review.
meta_analysis_applied now requires at least one meta_analyzer record and
all of them ok. llm_available is unchanged: provider availability is a
separate contract from whether meta_analyzer had anything to do, and it
stays vacuously true when meta_analyzer never ran.
Adds a regression covering the no-findings/no-record case, asserting
meta_analysis_applied is False while llm_available stays True.
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
---------
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Amir Fathi
698e2bf29c