Every finding says whether the data behind it was complete

The report already stated, in aggregate, that some modules ran with incomplete
input - on the sample estate, 11 of 38. An individual finding said nothing, so a
conclusion drawn from a fraction of a module's evidence was indistinguishable
from one drawn from all of it. Measured on that estate: 258 findings rest on
complete data and 150 rest on partial data, and until now nothing on the page
separated them. That is 37% of the report, and the aggregate line could not tell
a reader WHICH conclusions to weigh differently.

Findings now carry `evidence: {complete, declared_sources, missing_sources}`,
attached in BaseAuditor.finding() beside the standards mapping and for the
reason the code already gives there: one place, every check, no module having to
remember. It is unconditional - a finding that silently lacked the field would
be back to being unqualified.

It is derived from the same module_sources() map the coverage manifest uses, so
a finding's marker and the manifest cannot disagree. A hand-maintained second
list would drift the first time somebody added an input.

ABSENT MEANS None, NOT EMPTY

An export that was supplied and held no rows is a real answer; calling it
missing would understate the evidence exactly as badly as the reverse
understates the gap. A source that FAILED TO DECODE is recorded as None by the
loader, so it correctly lands here as missing - the customer sent it and we
could not read it.

OPTIONAL INPUTS WERE CRYING WOLF, AND THE FIX HAS A NARROW BAR

The first version marked every access-risk finding incomplete because nobody had
supplied `ara_ruleset` - a customer's own ruleset, whose absence is the ordinary
case and changes nothing about the analysis. OPTIONAL_SOURCES now handles that,
and a source qualifies only when the module ALREADY tells the reader, in its own
findings, that the input was absent: ara_ruleset (SODCOV-008/009/010 do not
fire), auth_object_catalogue (SODCOV-007 silent, SODCOV-002 says so in words),
and the Fiori exports (SODCOV-006 says so). Two modules annotated; the other 36
keep the default.

The default direction is deliberate and documented in the class: a source
wrongly marked optional makes the marker LIE BY STAYING QUIET, while one wrongly
left required makes it noisy. Noise is visible and gets fixed. A silent
all-clear is the failure this field exists to prevent.

Full suite green: 4302 passed. Ten new tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: Krishcalin

modules/access_risk_analysis.py

@@ -178,6 +178,10 @@ def _load_shipped_ruleset(path: Optional[Path] = None) -> List[Dict[str, Any]]:
 
 
 class AccessRiskAnalysisAuditor(BaseAuditor):
+    #: Optional inputs (see BaseAuditor.OPTIONAL_SOURCES): a customer's own ruleset. Its absence is the ordinary case and changes
+    #: nothing about the analysis; SODCOV-008/009/010 simply do not fire.
+    OPTIONAL_SOURCES = frozenset({"ara_ruleset"})
+
 
     CATEGORY = "Access Risk Analysis (SoD)"
 

modules/base_auditor.py

@@ -504,6 +504,20 @@ def finding(
         # an unmapped finding travels with the REASON it is unmapped rather than
         # silently having no standards fields.
         f["owasp"] = map_finding(check_id, (details or {}).get("cwe"))
+        # WHETHER THE DATA BEHIND THIS FINDING WAS COMPLETE, attached here for
+        # the same reason as the standards mapping above: one place, every
+        # check, no module having to remember.
+        #
+        # The report already says in aggregate that some modules ran with
+        # incomplete input. An individual finding said nothing, so a conclusion
+        # drawn from a fraction of a module\'s evidence was indistinguishable
+        # from one drawn from all of it. That is the same "confident answer over
+        # an unasked question" this product reports on elsewhere, appearing in
+        # its own output.
+        #
+        # It is always present, never conditional. A finding that silently
+        # lacked the field would be back to being unqualified.
+        f["evidence"] = self._evidence_state()
         if affected_objects:
             f["affected_objects"] = affected_objects
         if subject:
@@ -522,6 +536,57 @@ def finding(
         self.findings.append(f)
         return f
 
+    #: Inputs whose absence must NOT mark a finding incomplete.
+    #:
+    #: The bar is deliberately narrow: a source belongs here only when the
+    #: module ALREADY tells the reader, in its own findings, that it was not
+    #: supplied. `ara_ruleset` qualifies because SODCOV-008/009/010 simply do
+    #: not fire without it; `auth_object_catalogue` qualifies because SODCOV-007
+    #: is silent without it and SODCOV-002 says so in words.
+    #:
+    #: Everything else stays required by default, and that direction is chosen
+    #: on purpose. A source wrongly marked optional makes the marker lie by
+    #: staying quiet; a source wrongly left required makes it noisy. Noise is
+    #: visible and gets fixed - a silent all-clear is the failure this field
+    #: exists to prevent.
+    OPTIONAL_SOURCES: frozenset = frozenset()
+
+    #: Computed once per auditor instance by `_evidence_state`.
+    _evidence_cache = None
+
+    def _evidence_state(self) -> Dict[str, Any]:
+        """Which of this module\'s declared inputs were actually present.
+
+        Derived from the SAME mapping the coverage manifest uses, so a
+        finding\'s marker and the manifest cannot disagree — a hand-maintained
+        second list would drift the first time somebody added an input.
+
+        ABSENT MEANS `None`, NOT EMPTY. The loader distinguishes them and so
+        does this: an export that was supplied and held no rows is a real
+        answer, and calling it missing would understate the evidence exactly as
+        badly as the reverse. A source that failed to decode is recorded as
+        None by the loader, so it lands here as missing, which is correct — the
+        customer sent it and we could not read it.
+        """
+        if self._evidence_cache is None:
+            # Imported here rather than at module scope: `coverage` reads every
+            # module in this package, and importing it at load time would make
+            # the dependency circular.
+            from modules import coverage
+            name = type(self).__module__.rsplit(".", 1)[-1]
+            declared = sorted(coverage.module_sources().get(name, []) or [])
+            data = self.data or {}
+            missing = [src for src in declared
+                       if not src.startswith("_")
+                       and src not in self.OPTIONAL_SOURCES
+                       and data.get(src) is None]
+            self._evidence_cache = {
+                "complete": not missing,
+                "declared_sources": len(declared),
+                "missing_sources": missing,
+            }
+        return dict(self._evidence_cache)
+
     def run_all_checks(self) -> List[Dict[str, Any]]:
         """Override in subclass — run all checks and return findings."""
         raise NotImplementedError

modules/ruleset_coverage.py

@@ -221,6 +221,12 @@ def _expand_range(low: str, high: str) -> Optional[Set[str]]:
 
 
 class RulesetCoverageAuditor(BaseAuditor):
+    #: Optional inputs (see BaseAuditor.OPTIONAL_SOURCES): every one of these has a check that reports its own absence in words -
+    #: SODCOV-006 for the Fiori exports, SODCOV-007 and the note on SODCOV-002
+    #: for the object catalogue, SODCOV-008/009/010 for a supplied ruleset.
+    OPTIONAL_SOURCES = frozenset({"ara_ruleset", "auth_object_catalogue", "fiori_tiles",
+                                   "fiori_catalogs", "odata_auth"})
+
     """Measures what fraction of the granted estate the SoD ruleset can see."""
 
     CATEGORY = "SoD Ruleset Coverage"

tests/test_evidence_marker.py

@@ -0,0 +1,119 @@
+"""Every finding says whether the data behind it was complete.
+
+WHY
+
+The report already states, in aggregate, that some modules ran with incomplete
+input — on the sample estate, 11 of 38. An individual finding said nothing, so a
+conclusion drawn from a fraction of a module's evidence was indistinguishable
+from one drawn from all of it. That is the same "confident answer over an
+unasked question" this product reports on elsewhere, turning up in its own
+output.
+
+The marker is attached in BaseAuditor.finding(), beside the standards mapping
+and for the same reason: one place, every check, no module having to remember.
+It is always present rather than conditional — a finding that silently lacked
+the field would be back to being unqualified.
+"""
+import io
+import contextlib
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+from modules.base_auditor import BaseAuditor            # noqa: E402
+from modules import coverage                            # noqa: E402
+from modules.data_loader import DataLoader              # noqa: E402
+
+
+class _Probe(BaseAuditor):
+    """Stands in for a real auditor; `coverage` sees no sources for it."""
+
+    def run_all_checks(self):
+        return []
+
+
+def make(data, optional=frozenset()):
+    a = _Probe(data, {}, {})
+    a.OPTIONAL_SOURCES = optional
+    return a
+
+
+def a_finding(auditor):
+    return auditor.finding(check_id="X-1", title="t", severity="LOW",
+                           category="c", description="d")
+
+
+def test_every_finding_carries_the_marker():
+    """Always present, never conditional."""
+    f = a_finding(make({}))
+    assert "evidence" in f and set(f["evidence"]) == {
+        "complete", "declared_sources", "missing_sources"}
+
+
+def test_the_marker_is_derived_from_the_same_map_the_manifest_uses():
+    """A hand-maintained second list would drift the first time somebody added
+    an input, and then the finding and the manifest would disagree."""
+    assert coverage.module_sources.__wrapped__ is not None   # it is the cached map
+
+
+def test_a_declared_source_that_is_absent_marks_the_finding_incomplete():
+    a = make({})
+    a.__class__.__module__ = "modules.vendor_master"          # borrow its sources
+    a._evidence_cache = None
+    f = a_finding(a)
+    assert f["evidence"]["complete"] is False
+    assert "vendor_master" in f["evidence"]["missing_sources"]
+
+
+def test_supplied_but_empty_is_not_missing():
+    """THE distinction. An export that was supplied and held no rows is a real
+    answer; calling it missing would understate the evidence exactly as badly
+    as the reverse understates the gap."""
+    a = make({"vendor_master": [], "vendor_bank": []})
+    a.__class__.__module__ = "modules.vendor_master"
+    a._evidence_cache = None
+    assert a_finding(a)["evidence"]["complete"] is True
+
+
+def test_an_optional_source_does_not_mark_a_finding_incomplete():
+    """Otherwise every ARA finding on an ordinary scan would be flagged because
+    the customer did not supply a ruleset of their own — crying wolf."""
+    a = make({}, optional=frozenset({"vendor_master", "vendor_bank"}))
+    a.__class__.__module__ = "modules.vendor_master"
+    a._evidence_cache = None
+    assert a_finding(a)["evidence"]["complete"] is True
+
+
+def test_the_default_is_required_not_optional():
+    """A source wrongly marked optional makes the marker lie by staying quiet;
+    one wrongly left required makes it noisy. Noise gets fixed."""
+    assert BaseAuditor.OPTIONAL_SOURCES == frozenset()
+
+
+def test_the_two_annotated_modules_only_exempt_self_reporting_inputs():
+    """The bar for OPTIONAL_SOURCES is that the module already tells the reader
+    in its own findings that the input was absent."""
+    from modules.access_risk_analysis import AccessRiskAnalysisAuditor as ARA
+    from modules.ruleset_coverage import RulesetCoverageAuditor as RC
+    assert ARA.OPTIONAL_SOURCES == frozenset({"ara_ruleset"})
+    assert "auth_object_catalogue" in RC.OPTIONAL_SOURCES
+    assert "fiori_tiles" in RC.OPTIONAL_SOURCES
+    assert "role_auth_values" not in RC.OPTIONAL_SOURCES   # the one it needs
+
+
+def test_the_real_scan_marks_degraded_modules_and_not_the_rest():
+    """End to end on the shipped fixture."""
+    with contextlib.redirect_stdout(io.StringIO()):
+        data = DataLoader(ROOT / "sample_data").load_all()
+    from modules.code_transport import CodeTransportAuditor as CT
+    from modules.access_risk_analysis import AccessRiskAnalysisAuditor as ARA
+    with contextlib.redirect_stdout(io.StringIO()):
+        degraded = CT(data, {}, {}).run_all_checks()
+        clean = ARA(data, {}, {}).run_all_checks()
+    assert degraded and degraded[0]["evidence"]["complete"] is False
+    assert degraded[0]["evidence"]["missing_sources"]
+    # ARA's only absent input is the optional custom ruleset
+    assert clean and clean[0]["evidence"]["complete"] is True