Skip to content

Commit f051613

Browse files
Krishcalinclaude
andcommitted
Finish the sentence: when was this measured
53980c1 gave every system a date and left two things. This closes both. THE ROLL-UPS. `latest_coverage` selected the newest complete run per system, ordered by `started_at`, and threw the timestamp away — so Domains, CSF and Trend reported a category CLEAR on the strength of an export of any age and could not say which. A control fed by a scan from March read exactly like one fed this morning. The figure quoted is `oldest_days`, NOT `newest_days`. That manifest is a UNION across systems: a module counts as having run if it ran for any of them, so the weakest evidence behind a CLEAR verdict is the oldest run in the union. Dating the answer by the newest would describe it by its best input rather than its worst, which is the mistake SODCOV-000 exists to stop a summary making. A Postgres test pins it: one system scanned yesterday must not make a year-old estate read as current. I was wrong about the cost of this last time. The commit that deferred it said "six call sites and their tests"; `roll_up` reads `coverage["modules"]` and nothing else, so carrying a date alongside is purely additive and nothing else needed changing. THE OFFLINE REPORT, which is the copy that leaves the building. Every export's timestamp was already recorded — in a `modified` column inside a collapsed <details> that on a full bundle runs to a hundred and thirty rows. Nobody was going to open it and scan the column, which is the failure report_generator.py names twice in its own comments: a qualification nobody reaches is not a qualification. It now states the age before the finding count, beside the coverage block. THE ASYMMETRY THAT SHAPED THE OFFLINE HALF. A file's mtime is when it was last WRITTEN on the machine that produced the bundle; copying, unzipping or exporting through a share resets it. So mtime can only ever be LATER than the moment the data left SAP, and the derived age is a FLOOR: "at least 240 days old" is sound and can be acted on, "0 days old" says nothing whatever. The report states the first and is SILENT on the second — a reassuring sentence built on a number that cannot reassure is worse than no sentence — and the sentence says it is a floor, so nobody argues the data is fine because it reads 239. Measured by backdating a copy of sample_data 238 days: "last written on or before 2026-01-06 — at least 238 days ago". On the real sample_data, whose files are fresh, it renders nothing, which is the correct answer rather than a missing one. STALE_AFTER_DAYS now lives once, in modules/coverage.py, and server.queries imports it. Two copies would drift the day somebody tuned one, leaving the console and the PDF a customer sends an auditor disagreeing about the same estate; a test fails if the console redefines it. 18 new Python tests, 5 new console tests; 9 targeted mutations, all caught — including the two that matter most: reporting a fresh-looking bundle as fresh, and dating a roll-up by its best input. 4779 Python tests pass with the database attached, 124 frontend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 3e2a010 commit f051613

14 files changed

Lines changed: 599 additions & 4 deletions

File tree

CLAUDE.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -903,7 +903,9 @@ a page.** Everything below exists so they do not.
903903
| `docs/CHECK_FIRING.md` | How many of our own checks are proven to fire. CI fails if it drifts |
904904
| `queries.list_systems` | Per system: when a completed run last assessed it. `null` = never |
905905
| `queries.estate_freshness` | Current / stale / never, the oldest age, and which systems — built from the same rows the table renders, so headline and table cannot drift |
906-
| `STALE_AFTER_DAYS` | 35: one SAP Security Patch Day cycle plus slack. Governs EMPHASIS only — the measured date is always returned, so no threshold can hide an age |
906+
| `STALE_AFTER_DAYS` | 35: one SAP Security Patch Day cycle plus slack. Governs EMPHASIS only — the measured date is always returned, so no threshold can hide an age. Defined ONCE in `modules/coverage.py`; `server.queries` imports it, so the console and the PDF an auditor reads cannot disagree |
907+
| `queries.latest_coverage` | The manifest now carries `measured`. `oldest_days` is the figure to quote — the manifest is a UNION across systems, so the weakest evidence behind a CLEAR verdict is the oldest run in it. Reaches Domains, CSF and Trend through `roll_up` |
908+
| `coverage.evidence_age` | How old an UPLOAD is, from the file timestamps. A LOWER BOUND, never a measurement: copying or unzipping resets mtime, so it can only make old evidence look fresher. "At least 240 days" is sound; "0 days" says nothing, and `evidence_age_sentence` is silent there rather than reassuring |
907909

908910
### If you touch this
909911

frontend/src/api/types.ts

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -824,6 +824,7 @@ export interface DomainScore {
824824

825825
/** server/analytics.py `journey_summary` — one call for the trend screen. */
826826
export interface Journey {
827+
measured: Measured | null
827828
sla: SlaStatus
828829
aging: AgingBucket[]
829830
mttr: Mttr
@@ -1108,7 +1109,25 @@ export interface CsfFunctionView {
11081109
}
11091110

11101111
/** The whole Core rolled up. server/app.py `api_csf`. */
1112+
/** server/queries.py `latest_coverage` — when the answer above it was
1113+
* measured. Null when no manifest backed the roll-up, which is NOT the same
1114+
* as "today": an answer nobody dated must not be stamped with now.
1115+
*
1116+
* `oldest_days` is the one to quote. The manifest is a union across systems —
1117+
* a module counts as having run if it ran for ANY of them — so the weakest
1118+
* evidence behind a CLEAR verdict is the oldest run in that union. */
1119+
export interface Measured {
1120+
systems: number
1121+
oldest: string
1122+
newest: string
1123+
oldest_days: number
1124+
newest_days: number
1125+
stale_after_days: number
1126+
stale: boolean
1127+
}
1128+
11111129
export interface CsfView {
1130+
measured: Measured | null
11121131
framework: string
11131132
reference: string
11141133
doi: string
@@ -1280,6 +1299,7 @@ export interface SecurityDomain {
12801299
}
12811300

12821301
export interface DomainsView {
1302+
measured: Measured | null
12831303
domains: SecurityDomain[]
12841304
unplaced: {
12851305
counts: Record<string, number>
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
/**
2+
* When was this answer measured?
3+
*
4+
* THE DEFECT. `queries.latest_coverage` selected the newest complete run per
5+
* system, ordered by `started_at`, and dropped the timestamp. So Domains, CSF
6+
* and Trend reported a category CLEAR on the strength of an export of any age
7+
* and could not say which — a control fed by a scan from March read exactly
8+
* like one fed this morning.
9+
*/
10+
import { render, screen } from '@testing-library/react'
11+
import { describe, expect, it } from 'vitest'
12+
13+
import { MeasuredWhen } from './MeasuredWhen'
14+
import type { Measured } from '../api/types'
15+
16+
function measured(over: Partial<Measured> = {}): Measured {
17+
return {
18+
systems: 1, oldest: '2026-08-30T00:00:00Z', newest: '2026-08-30T00:00:00Z',
19+
oldest_days: 2, newest_days: 2, stale_after_days: 35, stale: false,
20+
...over,
21+
}
22+
}
23+
24+
describe('the date under a claim', () => {
25+
it('says nothing at all when the answer was never dated', () => {
26+
// Null is not "today". Stamping an undated answer with now is how a
27+
// manifest nobody supplied becomes a measurement nobody took.
28+
const { container } = render(
29+
<MeasuredWhen measured={null} subject="posture" />)
30+
expect(container).toBeEmptyDOMElement()
31+
})
32+
33+
it('dates a current answer without making a fuss of it', () => {
34+
render(<MeasuredWhen measured={measured()} subject="posture" />)
35+
expect(screen.getByText(/last measured 2 days ago/)).toBeInTheDocument()
36+
expect(screen.queryByText(/Patch Day/)).not.toBeInTheDocument()
37+
})
38+
39+
it('calls out an answer older than one patch cycle', () => {
40+
render(<MeasuredWhen
41+
measured={measured({ oldest_days: 240, stale: true })}
42+
subject="posture" />)
43+
expect(screen.getByText(/last measured 240 days ago/)).toBeInTheDocument()
44+
expect(screen.getByText(/a patch day has passed since/))
45+
.toBeInTheDocument()
46+
expect(screen.getByText(/as it was then/)).toBeInTheDocument()
47+
})
48+
49+
it('quotes the OLDEST system, not the newest', () => {
50+
// THE RULE THAT MATTERS. The manifest is a union: a module counts as having
51+
// run if it ran for any system in scope. So the weakest evidence behind a
52+
// CLEAR verdict is the oldest run in that union, and quoting the newest
53+
// would date the answer by its best input rather than its worst.
54+
render(<MeasuredWhen
55+
measured={measured({ systems: 4, oldest_days: 300, newest_days: 1,
56+
stale: true })}
57+
subject="posture" />)
58+
expect(screen.getByText(/Oldest of 4 systems/)).toBeInTheDocument()
59+
expect(screen.getByText(/300 days ago/)).toBeInTheDocument()
60+
expect(screen.queryByText(/1 day ago/)).not.toBeInTheDocument()
61+
})
62+
63+
it('reads naturally on the day and the day after', () => {
64+
const { rerender } = render(
65+
<MeasuredWhen measured={measured({ oldest_days: 0 })} subject="trend" />)
66+
expect(screen.getByText(/last measured today/)).toBeInTheDocument()
67+
rerender(
68+
<MeasuredWhen measured={measured({ oldest_days: 1 })} subject="trend" />)
69+
expect(screen.getByText(/last measured yesterday/)).toBeInTheDocument()
70+
})
71+
})
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
/**
2+
* When the answer on this screen was measured.
3+
*
4+
* THE DEFECT. `queries.latest_coverage` selected the newest complete run per
5+
* system, ordered by `started_at`, and then threw the timestamp away. So a
6+
* domain reported CLEAR on the strength of an export of any age and the screen
7+
* could not say which — a control fed by a scan from March read exactly like
8+
* one fed this morning. That is the same failure the estate-freshness work
9+
* fixed for the systems table, one layer up.
10+
*
11+
* `oldest_days` IS THE FIGURE, NOT `newest_days`. The manifest is a union
12+
* across systems: a module counts as having run if it ran for any of them. So
13+
* the weakest evidence behind a CLEAR verdict is the oldest run in the union,
14+
* and quoting the newest would date the answer by its best input rather than
15+
* its worst — which is the mistake `SODCOV-000` exists to stop a summary
16+
* making.
17+
*
18+
* One component rather than three copies, so the three screens cannot end up
19+
* disagreeing about the same estate.
20+
*/
21+
import type { Measured } from '../api/types'
22+
23+
export function MeasuredWhen({ measured, subject }: {
24+
measured: Measured | null
25+
/** What was measured, for the sentence: "posture", "coverage", "trend". */
26+
subject: string
27+
}) {
28+
// Null is not "today". An answer nobody dated must not be stamped with now,
29+
// so the line is absent rather than reassuring.
30+
if (!measured) return null
31+
32+
const { oldest_days: days, stale, stale_after_days: threshold } = measured
33+
const when = days === 0 ? 'today'
34+
: days === 1 ? 'yesterday'
35+
: `${days} days ago`
36+
37+
return (
38+
<p className={`text-[12px] mt-1 ${stale ? 'text-high' : 'text-ink3'}`}>
39+
{measured.systems === 1
40+
? `This ${subject} was last measured ${when}`
41+
: `Oldest of ${measured.systems} systems: last measured ${when}`}
42+
{stale && (
43+
<>
44+
{' '}— more than the {threshold}-day SAP Security Patch Day cycle, so
45+
a patch day has passed since. It describes the estate as it was then.
46+
</>
47+
)}
48+
.
49+
</p>
50+
)
51+
}

frontend/src/routes/Csf.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import { useTitle } from '../lib/title'
2222
import { Landmark } from 'lucide-react'
2323
import { CARD_TITLE as CARD_H3, KPI, KPI_NOTE } from '../lib/ui'
2424
import { Donut } from '../components/Donut'
25+
import { MeasuredWhen } from '../components/MeasuredWhen'
2526

2627
const CARD = 'rounded-lg border border-cardline bg-panel p-4'
2728

@@ -83,6 +84,7 @@ export function Csf() {
8384
<Landmark size={22} className="text-accent shrink-0" />
8485
NIST Cybersecurity Framework 2.0
8586
</h1>
87+
<MeasuredWhen measured={view?.measured ?? null} subject="assessment" />
8688
<p className="text-ink2 mb-5">
8789
Your open findings, arranged by the outcome each one is evidence against.
8890
</p>

frontend/src/routes/Domains.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ import type { DomainReach, DomainsView, SecurityDomain } from '../api/types'
2929
import { useTitle } from '../lib/title'
3030
import { LayoutGrid } from 'lucide-react'
3131
import { KPI } from '../lib/ui'
32+
import { MeasuredWhen } from '../components/MeasuredWhen'
3233

3334
const CARD = 'rounded-lg border border-cardline bg-panel p-4'
3435
const G3 = 'grid gap-3.5 [grid-template-columns:repeat(auto-fit,minmax(300px,1fr))]'
@@ -75,6 +76,7 @@ export function Domains() {
7576
<LayoutGrid size={22} className="text-accent shrink-0" />
7677
Security Domains
7778
</h1>
79+
<MeasuredWhen measured={view?.measured ?? null} subject="posture" />
7880
<p className="text-ink2 mb-5 max-w-[80ch]">
7981
Your open findings in the twelve domains this market talks in. Each tile
8082
says two things that are easy to confuse and are not the same:{' '}

frontend/src/routes/Trend.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import type { BurndownPoint, Journey } from '../api/types'
55
import { useTitle } from '../lib/title'
66
import { TrendingUp } from 'lucide-react'
77
import { CARD_TITLE as CARD_H3, KPI, KPI_NOTE } from '../lib/ui'
8+
import { MeasuredWhen } from '../components/MeasuredWhen'
89

910
/**
1011
* The mitigation journey — ported from server/templates/trend.html. It answers
@@ -85,6 +86,7 @@ export function Trend() {
8586
<TrendingUp size={22} className="text-accent shrink-0" />
8687
Mitigation Journey
8788
</h1>
89+
<MeasuredWhen measured={journey?.measured ?? null} subject="trend" />
8890
<p className="text-ink2 mb-4">
8991
What changed, who owns it, and whether it is getting better — over the last{' '}
9092
{window} days.

modules/coverage.py

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
from __future__ import annotations
2828

2929
import ast
30+
import datetime as _dt
3031
import re
3132
from functools import lru_cache
3233
from pathlib import Path
@@ -1498,3 +1499,98 @@ def summarize(counts: Dict[str, int], deployment_mode: str = "on_prem") -> str:
14981499
and not counts.get("modules_not_run")):
14991500
parts.append("Coverage is complete.")
15001501
return " ".join(parts)
1502+
1503+
1504+
# ── how old the evidence is ────────────────────────────────────────────────
1505+
#
1506+
#: How old an export may be before its answers are called stale.
1507+
#:
1508+
#: NOT AN INVENTED NUMBER. SAP publishes Security Notes on Security Patch Day,
1509+
#: the second Tuesday of each month. An export older than one full cycle cannot
1510+
#: account for a patch day that has since passed, so verdicts drawn from it are
1511+
#: answers about a system that no longer exists in that form. 35 days is one
1512+
#: cycle plus the slack between two second-Tuesdays, which fall 28 to 35 days
1513+
#: apart.
1514+
#:
1515+
#: Defined here rather than in `server/` so the offline report and the console
1516+
#: cannot drift apart about what "old" means — `server.queries` imports it.
1517+
STALE_AFTER_DAYS = 35
1518+
1519+
1520+
def evidence_age(manifest: Optional[Iterable[Dict[str, Any]]],
1521+
now: Optional[_dt.datetime] = None) -> Optional[Dict[str, Any]]:
1522+
"""When the exports behind a report were last written.
1523+
1524+
THE NUMBER IS A LOWER BOUND, AND THAT IS THE WHOLE POINT OF THIS FUNCTION.
1525+
It is derived from each file's modification time, which is when the file was
1526+
last WRITTEN on the machine that produced the bundle. Copying a directory,
1527+
unzipping an archive or exporting through a share usually resets that to the
1528+
moment of the copy — so a file's mtime can only ever be LATER than the
1529+
moment the data was really taken out of SAP, never earlier.
1530+
1531+
Which makes the reading asymmetric, and the report has to say so:
1532+
1533+
* "at least 240 days old" is sound. Nothing can make evidence look older
1534+
than it is, so a large figure is a floor and can be acted on.
1535+
* "0 days old" says nothing at all. The files may have been copied this
1536+
morning out of an export taken last year.
1537+
1538+
So this reports the floor and never reassures. `stale` is True only when the
1539+
FLOOR exceeds the threshold; it is never False in a way that means "fresh",
1540+
only in a way that means "this cannot tell you".
1541+
1542+
Returns None when no entry carries a usable timestamp, which is a different
1543+
state from "the evidence is new" and must not render as one.
1544+
"""
1545+
entries = list(manifest or [])
1546+
now = now or _dt.datetime.now()
1547+
stamps: List[_dt.datetime] = []
1548+
for entry in entries:
1549+
raw = str((entry or {}).get("modified") or "").strip()
1550+
if not raw:
1551+
continue
1552+
for shape in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d"):
1553+
try:
1554+
stamps.append(_dt.datetime.strptime(raw[:len(shape) + 4], shape))
1555+
break
1556+
except ValueError:
1557+
continue
1558+
if not stamps:
1559+
return None
1560+
1561+
oldest, newest = min(stamps), max(stamps)
1562+
# The NEWEST file is what dates the bundle: an export directory is written
1563+
# in one sitting, and one stale leftover among a hundred current files does
1564+
# not make the assessment old. The oldest is reported beside it so a bundle
1565+
# assembled over eight months is visible as one.
1566+
floor_days = max(0, (now - newest).days)
1567+
return {
1568+
"files": len(stamps),
1569+
"oldest": oldest.strftime("%Y-%m-%d"),
1570+
"newest": newest.strftime("%Y-%m-%d"),
1571+
"span_days": max(0, (newest - oldest).days),
1572+
"at_least_days": floor_days,
1573+
"stale_after_days": STALE_AFTER_DAYS,
1574+
"stale": floor_days > STALE_AFTER_DAYS,
1575+
}
1576+
1577+
1578+
def evidence_age_sentence(age: Optional[Dict[str, Any]]) -> str:
1579+
"""The one line a reader should see, or nothing.
1580+
1581+
Silent when the floor is small, because a small floor is not evidence of
1582+
freshness — see `evidence_age`. A reassuring sentence built on a number
1583+
that cannot reassure is worse than no sentence.
1584+
"""
1585+
if not age:
1586+
return ""
1587+
if not age["stale"]:
1588+
return ""
1589+
return (
1590+
"The exports behind this report were last written on or before "
1591+
"%s — at least %d days ago, more than the %d-day SAP Security Patch "
1592+
"Day cycle. Findings here describe the system as it was then. A file's "
1593+
"timestamp can only ever be later than the moment the data left SAP, "
1594+
"so this is a floor: the evidence may be older still."
1595+
% (age["newest"], age["at_least_days"], age["stale_after_days"])
1596+
)

modules/domains.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,12 @@ def roll_up(findings: Sequence[Dict[str, Any]],
431431

432432
return {
433433
"domains": out,
434+
# WHEN THE ANSWER WAS MEASURED, carried straight through from the
435+
# manifest. A CLEAR verdict on this screen rests on the newest complete
436+
# run of each system in scope, and until now the screen could not say
437+
# how old that was — see server/queries.latest_coverage. None when the
438+
# caller supplied no manifest, which is a different state from "today".
439+
"measured": (coverage or {}).get("measured"),
434440
"unplaced": {
435441
"counts": dict(sorted(unplaced.items(), key=lambda kv: (-kv[1], kv[0]))),
436442
"total": sum(unplaced.values()),

modules/nist_csf.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -495,6 +495,12 @@ def roll_up(findings: Sequence[Dict[str, Any]],
495495
"reference": "NIST CSWP 29, February 26, 2024",
496496
"doi": "https://doi.org/10.6028/NIST.CSWP.29",
497497
"functions": functions,
498+
# WHEN THE ANSWER WAS MEASURED, carried straight through from the
499+
# manifest. A CLEAR verdict on this screen rests on the newest complete
500+
# run of each system in scope, and until now the screen could not say
501+
# how old that was — see server/queries.latest_coverage. None when the
502+
# caller supplied no manifest, which is a different state from "today".
503+
"measured": (coverage or {}).get("measured"),
498504
"totals": {
499505
"functions": len(FUNCTIONS),
500506
"categories": len(CATEGORIES),

0 commit comments

Comments
 (0)