Skip to content

Commit 9fd113f

Browse files
Krishcalinclaude
andcommitted
Add up the evidence gaps, and rank what to send next
Every module already recorded its own: a finding assessed on partial input carries `evidence.complete = false` beside `declared_sources` and `missing_sources`, and modules/domains.py keeps `not_supplied` apart from `clear` so no screen reports "we looked and found nothing" about a domain nobody exported. All of it per finding. Nothing added it up, so the one question a customer can act on had no answer: of everything we did not send, what is worth sending first? On the demo estate `auth_objects` alone leaves 350 findings undecided across ten systems, and a reader saw sixty separate caveats with no way to tell they shared four causes. IT RANKS INPUT, NOT RISK. A source at the top would let many checks REACH A VERDICT; it is not a source hiding many problems, because what those checks will conclude is exactly what is unknown. The field is `findings_undecided`, the verb is "decide", and there is deliberately nothing estimating what an export would turn up -- the scanner has no connection to SAP and would be inventing it. Two tests hold that line, one on the payload's field names and one on the words the page prints. `ext_os_commands_sap` is ranked and counted but marked `obtainable_in_rise: false`. Five sources come from the layer SAP operates, and telling a RISE customer to run an OS-level export is an item they can only fail; dropping the row instead would understate the estate for the on-premise readers who can close it. A source named by a module but matching no loader slot is surfaced too, as our defect rather than their homework -- no file ever closes it. THE LATEST OBSERVATION IS THE LATEST RUN, matching how `list_findings` picks `latest_evidence`. This screen is the sum of the sentence FindingDetail prints on each finding, so the two must not be able to disagree about which observation is current. `o.id DESC` gives the same answer until an older run is re-imported -- high row id, low run id -- and a total contradicting the findings it sums is the artefacts-disagreeing failure this codebase keeps warning about. `count(DISTINCT f.id)`, not `count(*)`: the lateral expansion emits a row per (finding, named source), so a repeated name would count a finding twice and lift a source above one with more real findings behind it. Mutation testing found three defects in these tests before they could defend nothing -- a ranking assertion that could not tell ranked order from alphabetical (the same mistake in both suites), a newest-observation test covering only one of the two queries, and the count above. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent b511ef9 commit 9fd113f

9 files changed

Lines changed: 1014 additions & 1 deletion

File tree

frontend/src/App.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import { Paths } from './routes/Paths'
2323
import { Risk } from './routes/Risk'
2424
import { Trend } from './routes/Trend'
2525
import { Dashboard } from './routes/Dashboard'
26+
import { EvidenceGaps } from './routes/EvidenceGaps'
2627
import { Findings } from './routes/Findings'
2728
import { FindingDetail } from './routes/FindingDetail'
2829

@@ -68,6 +69,7 @@ export default function App() {
6869
<Route path="/" element={<Dashboard />} />
6970
<Route path="/account" element={<Account />} />
7071
<Route path="/coverage" element={<Coverage />} />
72+
<Route path="/evidence-gaps" element={<EvidenceGaps />} />
7173
<Route path="/checks/:id" element={<CheckDetail />} />
7274
<Route path="/chokepoints" element={<Chokepoints />} />
7375
<Route path="/requirements/:id" element={<RequirementDetail />} />

frontend/src/api/client.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ import type {
3232
CheckIndexEntry, ChokepointsView,
3333
ServiceRequest, SeveringSet, Coverage, CrqControlsView,
3434
CrqParametersView, CrqQuantifyResult, CrqTrendPoint, CsfFunctionView,
35+
EvidenceGapsView,
3536
ExportValue,
3637
RemediationPlan,
3738
ComplianceView,
@@ -383,6 +384,11 @@ export function coverage(): Promise<Coverage> {
383384
return get<Coverage>('/coverage')
384385
}
385386

387+
/** Which unsupplied export would let the most open findings reach a verdict. */
388+
export function evidenceGaps(): Promise<EvidenceGapsView> {
389+
return get<EvidenceGapsView>('/evidence-gaps')
390+
}
391+
386392
// ══ runs and upload ═════════════════════════════════════════════════════════
387393

388394
export function run(id: number): Promise<ScanRun> {

frontend/src/api/types.ts

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1026,6 +1026,39 @@ export interface Journey {
10261026
* our checks that map to no SAP requirement, which is not a failure — SoD, GRC,
10271027
* financial controls and the attack-path content have no Baseline equivalent,
10281028
* and that is precisely where the product goes beyond it. */
1029+
/** One unsupplied export, and how much of the estate is waiting on it.
1030+
*
1031+
* `findings_undecided` COUNTS FINDINGS THAT COULD NOT REACH A VERDICT, not
1032+
* problems waiting to be discovered. Supplying the source lets those checks
1033+
* answer; it does not say what they will answer. There is deliberately no field
1034+
* estimating the outcome — the scanner has no connection to SAP and would be
1035+
* inventing it. The screen must not word it as "would fix". */
1036+
export interface EvidenceGap {
1037+
source: string
1038+
findings_undecided: number
1039+
checks: number
1040+
systems: number
1041+
/** Filenames the loader accepts for this source, from its own table. Empty
1042+
* when the name is one the loader does not know. */
1043+
files_accepted: string[]
1044+
feeds: string[]
1045+
/** False means the name appears in a module's `missing_sources` but in no
1046+
* loader slot — a defect, not an export anyone can send. */
1047+
known_to_loader: boolean
1048+
/** False for the five sources SAP operates under RISE. Still counted, because
1049+
* an on-premise reader of the same estate can close it — but a RISE customer
1050+
* cannot, and telling them to is advice they cannot take. */
1051+
obtainable_in_rise: boolean
1052+
}
1053+
1054+
export interface EvidenceGapsView {
1055+
gaps: EvidenceGap[]
1056+
/** Findings, not source mentions: one finding blocked on two exports is one
1057+
* undecided finding in two rows, so this is NOT the column's sum. */
1058+
findings_undecided: number
1059+
unknown_sources: string[]
1060+
}
1061+
10291062
export interface Coverage {
10301063
baseline_version: string | null
10311064
requirements_published: number

frontend/src/lib/nav.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import {
2-
Calculator, CircleAlert, ListOrdered, ScrollText, CircleDollarSign, Landmark, LayoutDashboard, LayoutGrid, Scissors, ShieldCheck, TrendingUp, Upload, UserCog, Waypoints, type LucideIcon,
2+
Calculator, CircleAlert, FileQuestion, ListOrdered, ScrollText, CircleDollarSign, Landmark, LayoutDashboard, LayoutGrid, Scissors, ShieldCheck, TrendingUp, Upload, UserCog, Waypoints, type LucideIcon,
33
} from 'lucide-react'
44
import type { Role } from '../api/types'
55

@@ -41,6 +41,11 @@ export const NAV_MAIN: NavItem[] = [
4141
// sits in the main list rather than under a reports submenu.
4242
{ to: '/risk', label: 'Risk ($)', icon: CircleDollarSign },
4343
{ to: '/coverage', label: 'Coverage', icon: ShieldCheck },
44+
// Directly under Coverage, because it is the same question from the other
45+
// side: Coverage says what the catalogue can ever see, this says what THIS
46+
// estate did not send. A reader who accepts the first immediately asks the
47+
// second.
48+
{ to: '/evidence-gaps', label: 'Evidence Gaps', icon: FileQuestion },
4449
{ to: '/domains', label: 'Domains', icon: LayoutGrid },
4550
// Directly under Domains, because it is the same twelve buckets asked a
4651
// narrower question: not "how much is in each" but "what is worst in
Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
import { render, screen, waitFor, within } from '@testing-library/react'
2+
import { beforeEach, describe, expect, it, vi } from 'vitest'
3+
4+
import { EvidenceGaps } from './EvidenceGaps'
5+
import type { EvidenceGap, EvidenceGapsView } from '../api/types'
6+
7+
/**
8+
* The screen that ranks what to send next.
9+
*
10+
* Everything that can go wrong here is a WORDING failure rather than a rendering
11+
* one, which is why these tests read the sentences and not just the numbers.
12+
*
13+
* IT MUST NOT PROMISE AN OUTCOME. "Would fix 35 findings" is the sentence this
14+
* page must never say. The scanner has no connection to SAP and does not know
15+
* what those checks will conclude — that is the entire reason the export is
16+
* being requested. `it_never_promises_what_the_answers_will_be` pins the verb.
17+
*
18+
* THE HEADLINE IS NOT THE COLUMN'S SUM. A finding waiting on two exports is
19+
* one undecided finding in two rows. A reader who adds the column up and gets
20+
* a larger number than the headline will trust neither, so the page says which
21+
* is which in words.
22+
*
23+
* A GAP THE CUSTOMER CANNOT CLOSE MUST SAY SO. Ranking an OS-level export
24+
* first for a RISE customer hands them an item they can only fail.
25+
*
26+
* AN EMPTY LIST IS GOOD NEWS HERE, uniquely in this product. Everywhere else
27+
* an empty list is the ambiguity to be resolved; on this page it means every
28+
* open finding reached its verdict on complete input, and it must not render
29+
* as the same shrug.
30+
*/
31+
32+
vi.mock('../api/client', () => ({
33+
evidenceGaps: vi.fn(),
34+
ApiError: class ApiError extends Error {
35+
status: number
36+
constructor(status: number, message: string) {
37+
super(message)
38+
this.status = status
39+
}
40+
},
41+
}))
42+
43+
vi.mock('../lib/title', () => ({ useTitle: () => {} }))
44+
45+
import { evidenceGaps as fetchEvidenceGaps } from '../api/client'
46+
47+
function gap(over: Partial<EvidenceGap> = {}): EvidenceGap {
48+
return {
49+
source: 'auth_objects',
50+
findings_undecided: 35,
51+
checks: 35,
52+
systems: 2,
53+
files_accepted: ['auth_objects.csv'],
54+
feeds: ['security_params'],
55+
known_to_loader: true,
56+
obtainable_in_rise: true,
57+
...over,
58+
}
59+
}
60+
61+
function view(over: Partial<EvidenceGapsView> = {}): EvidenceGapsView {
62+
return {
63+
gaps: [gap()],
64+
findings_undecided: 35,
65+
unknown_sources: [],
66+
...over,
67+
}
68+
}
69+
70+
const mocked = vi.mocked(fetchEvidenceGaps)
71+
72+
/** The ranked table, scoped. The source name and the undecided count each appear
73+
* twice on a rendered page — once in the summary tile, once in the row — so an
74+
* unscoped getByText matches two elements and throws. Scoping also makes the
75+
* assertions mean what they say: "the row shows this", not "the page mentions
76+
* it somewhere". */
77+
const table = () => within(screen.getByRole('table'))
78+
79+
describe('EvidenceGaps', () => {
80+
beforeEach(() => { vi.clearAllMocks() })
81+
82+
it('ranks the sources in the order the API returned them', async () => {
83+
// THE HEAVY SOURCE IS THE LATER ONE ALPHABETICALLY, deliberately. With the
84+
// counts the other way round a page that re-sorted by name would produce the
85+
// identical list, and this test would pass while defending nothing — the
86+
// mutation that re-sorts survived until these two were swapped.
87+
mocked.mockResolvedValue(view({
88+
gaps: [
89+
gap({ source: 'user_groups', findings_undecided: 35 }),
90+
gap({ source: 'auth_objects', findings_undecided: 3 }),
91+
],
92+
findings_undecided: 38,
93+
}))
94+
render(<EvidenceGaps />)
95+
await waitFor(() => expect(screen.getByRole('table')).toBeInTheDocument())
96+
const cells = screen.getAllByText(/^(auth_objects|user_groups)$/)
97+
// Three matches: the "send this first" tile, then the two rows. The tile and
98+
// the first row must name the same source, or the page contradicts itself.
99+
expect(cells.map((c) => c.textContent)).toEqual(
100+
['user_groups', 'user_groups', 'auth_objects'])
101+
})
102+
103+
it('names the heaviest source as the one to send first', async () => {
104+
mocked.mockResolvedValue(view({
105+
gaps: [gap({ source: 'auth_objects', findings_undecided: 35, checks: 35 })],
106+
}))
107+
render(<EvidenceGaps />)
108+
await waitFor(() => expect(screen.getByText('Send this first')).toBeInTheDocument())
109+
expect(screen.getByText(/35 findings across 35 checks reach a verdict/))
110+
.toBeInTheDocument()
111+
})
112+
113+
it('never promises what the answers will be', async () => {
114+
mocked.mockResolvedValue(view())
115+
render(<EvidenceGaps />)
116+
await waitFor(() => expect(screen.getByRole('table')).toBeInTheDocument())
117+
const page = document.body.textContent ?? ''
118+
// "resolve" is absent too: a finding here may well stay open once decided.
119+
expect(page).not.toMatch(/would fix|will fix|would resolve|fixes \d/i)
120+
expect(page).toMatch(/reach a verdict/i)
121+
expect(page).toMatch(/it does not say what they will answer/i)
122+
})
123+
124+
it('says the headline is not the sum of the column', async () => {
125+
mocked.mockResolvedValue(view({
126+
gaps: [gap({ source: 'auth_objects', findings_undecided: 2 }),
127+
gap({ source: 'user_groups', findings_undecided: 2 })],
128+
// One finding waiting on both: three undecided findings, four row-mentions.
129+
findings_undecided: 3,
130+
}))
131+
render(<EvidenceGaps />)
132+
await waitFor(() => expect(screen.getByRole('table')).toBeInTheDocument())
133+
// The tile, not a row: the rows here read 2 and 2, and the headline is 3.
134+
expect(screen.getByText('Findings undecided').parentElement)
135+
.toHaveTextContent('3')
136+
expect(screen.getByText(/counted once here and appears in both rows/))
137+
.toBeInTheDocument()
138+
})
139+
140+
it('marks a source SAP operates rather than telling the customer to fetch it',
141+
async () => {
142+
mocked.mockResolvedValue(view({
143+
gaps: [gap({ source: 'ext_os_commands_sap', obtainable_in_rise: false })],
144+
}))
145+
render(<EvidenceGaps />)
146+
await waitFor(() => expect(screen.getByRole('table')).toBeInTheDocument())
147+
expect(table().getByText('ext_os_commands_sap')).toBeInTheDocument()
148+
expect(table().getByText(/SAP operates this layer under RISE/))
149+
.toBeInTheDocument()
150+
})
151+
152+
it('leaves an obtainable source unmarked', async () => {
153+
// The negative control: without it the assertion above passes on a page that
154+
// prints that sentence on every row.
155+
mocked.mockResolvedValue(view())
156+
render(<EvidenceGaps />)
157+
await waitFor(() => expect(screen.getByRole('table')).toBeInTheDocument())
158+
expect(screen.queryByText(/SAP operates this layer under RISE/)).toBeNull()
159+
})
160+
161+
it('calls out a source no export can satisfy as our defect', async () => {
162+
mocked.mockResolvedValue(view({
163+
gaps: [gap({ source: 'auth_objekts', known_to_loader: false,
164+
files_accepted: [] })],
165+
unknown_sources: ['auth_objekts'],
166+
}))
167+
render(<EvidenceGaps />)
168+
await waitFor(() =>
169+
expect(screen.getByText(/is not one the loader accepts/)).toBeInTheDocument())
170+
expect(screen.getByText(/defect in the check, not something to collect/))
171+
.toBeInTheDocument()
172+
})
173+
174+
it('shows no defect banner when every source is one we accept', async () => {
175+
mocked.mockResolvedValue(view())
176+
render(<EvidenceGaps />)
177+
await waitFor(() => expect(screen.getByRole('table')).toBeInTheDocument())
178+
expect(screen.queryByText(/is not one the loader accepts/)).toBeNull()
179+
})
180+
181+
it('offers the filenames the loader will accept', async () => {
182+
mocked.mockResolvedValue(view({
183+
gaps: [gap({ files_accepted: ['auth_objects.csv', 'tobj.csv'] })],
184+
}))
185+
render(<EvidenceGaps />)
186+
await waitFor(() =>
187+
expect(screen.getByText('auth_objects.csv · tobj.csv')).toBeInTheDocument())
188+
})
189+
190+
it('reads an empty list as the good news it is', async () => {
191+
mocked.mockResolvedValue(view({ gaps: [], findings_undecided: 0 }))
192+
render(<EvidenceGaps />)
193+
await waitFor(() =>
194+
expect(screen.getByText('Nothing outstanding')).toBeInTheDocument())
195+
expect(screen.getByText(/reached its verdict on complete input/))
196+
.toBeInTheDocument()
197+
// Not the ambiguous empty state this product spends its time telling apart.
198+
expect(screen.queryByText(/Ranked by findings waiting/)).toBeNull()
199+
})
200+
201+
it('explains a refusal rather than rendering an empty page', async () => {
202+
const { ApiError } = await import('../api/client')
203+
mocked.mockRejectedValue(new (ApiError as new (s: number, m: string) => Error)(
204+
403, 'forbidden'))
205+
render(<EvidenceGaps />)
206+
await waitFor(() =>
207+
expect(screen.getByText(/not permitted to see the estate/)).toBeInTheDocument())
208+
})
209+
})

0 commit comments

Comments
 (0)