tests

"9 systems" while naming eight #215

Workflow file for this run

	name: tests

	on:
	push:
	branches: [ main ]
	pull_request:
	workflow_dispatch:

	concurrency:
	group: tests-${{ github.ref }}
	cancel-in-progress: true

	jobs:
	# ------------------------------------------------------------------ #
	# CLI — the standard-library half of the product #
	# ------------------------------------------------------------------ #
	# The charter changed for `server/`, NOT for `modules/` and
	# `sap_scanner.py`. This job installs nothing but pytest, so if anyone
	# imports a third-party package into the scanner core it fails here
	# rather than at a customer who pip-installed nothing.
	cli:
	name: cli (stdlib only, Python ${{ matrix.python-version }})
	runs-on: ubuntu-22.04
	strategy:
	fail-fast: false
	matrix:
	python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
	steps:
	- uses: actions/checkout@v4

	- name: Set up Python ${{ matrix.python-version }}
	uses: actions/setup-python@v5
	with:
	python-version: ${{ matrix.python-version }}

	- name: Install test dependencies (pytest ONLY — this is the point of the job)
	run: python -m pip install --upgrade pip pytest

	- name: Byte-compile the scanner core
	run: python -m compileall modules sap_scanner.py -q

	# The DB-backed suites skip without DB_DSN, which is correct here — this
	# job has no database. They are covered by the `server` job below, and
	# that job FAILS if they skip, so the coverage cannot quietly vanish.
	#
	# EVERY SERVER-TIER SUITE MUST BE LISTED, AND AN INCOMPLETE LIST MADE THIS JOB RED.
	# This job installs pytest and NOTHING ELSE — that is its whole point. Two
	# consequences follow, and only the first was handled before:
	#
	# 1. A test file's imports run at COLLECTION, before any skip marker is
	# consulted. A module-level `import psycopg` / `starlette` therefore
	# aborts the WHOLE run with "Interrupted: N errors during collection",
	# taking every scanner test with it. tests/test_graph_paths.py
	# (-> server.graph -> psycopg) had done this since the attack-path work
	# and tests/test_spa_mount.py (-> starlette) since the SPA migration.
	# 2. A test that imports the server tier INSIDE the test body fails rather
	# than aborting — quieter, equally red. test_api_auth, test_branding,
	# test_enrich, test_sapcontent and the two deferral suites all do this.
	#
	# Excluding them costs no coverage: the `server` job runs every one of them
	# with the real dependencies, and its skip guard means they cannot quietly
	# vanish there. A module-level third-party import in a genuine SCANNER test
	# still fails this job, which is the boundary it exists to defend — and the
	# `purity` job enforces that boundary structurally over modules/ regardless.
	#
	# THE LIST IS DERIVED, NOT WRITTEN DOWN, and that is the whole point.
	#
	# It used to be eighteen `--ignore=` lines maintained by hand, under a
	# comment reading "WHEN YOU ADD A TEST THAT TOUCHES server/, ADD IT HERE.
	# This list rotted once already." By the time it was replaced it had rotted
	# twice: ELEVEN test files imported `server` and were not listed, passing
	# only because the modules they happened to import were dependency-free.
	# That is not a job that passes, it is a job that has not failed yet.
	#
	# tools/stdlib_only_ignores.py answers the real question — "does running
	# this test require a package this job does not install?" — by following
	# each test's imports transitively through server/. It is deliberately
	# conservative about WHERE it looks: an import inside a fixture does not
	# break collection but does break the test, which cost two rounds of
	# measurement to establish.
	#
	# It also preserves the carve-out a naive rule would have destroyed:
	# server/totp.py and server/qr.py import only the standard library, so
	# tests/test_totp.py and tests/test_qr.py still run HERE as well as in the
	# server job — 86 tests of the RFC 6238 core and the ISO/IEC 18004 encoder
	# against published vectors, on every Python in the matrix.
	#
	# Verified in a pytest-only virtualenv before this replaced the list:
	# 2174 passed, 11 skipped, nothing failed.
	- name: Run scanner tests
	run: pytest -q $(python -m tools.stdlib_only_ignores)

	- name: Smoke-run the scanner (all modules over sample_data)
	env:
	PYTHONIOENCODING: utf-8
	run: python sap_scanner.py --data-dir ./sample_data --output /tmp/sap_report.html

	# ------------------------------------------------------------------ #
	# The image itself — ten jobs tested the code and none looked here #
	# ------------------------------------------------------------------ #
	# A customer's security review asks two questions about a container: what is in
	# it, and what does it run as. Nothing answered either.
	#
	# FAILS ON CRITICAL AND HIGH ONLY, and `ignore-unfixed` is set. A build that
	# goes red for a MEDIUM in a base-image package with no available patch teaches
	# everyone to ignore the job, and an ignored gate is worse than no gate — it
	# looks like assurance. Findings at every severity are still uploaded to the
	# Security tab, so nothing is hidden; only the build-breaking threshold moves.
	image:
	name: the built image scans clean and runs unprivileged
	runs-on: ubuntu-22.04
	steps:
	- uses: actions/checkout@v4

	- name: Build the runtime image
	run: docker build -t monitorrisk:ci .

	- name: Scan for known vulnerabilities
	# A REAL TAG, CHECKED AGAINST THE GITHUB API BEFORE BEING WRITTEN HERE.
	# The first version of this line said @0.28.0, which does not exist. The
	# job then failed in "Set up job" — before a single step ran — because the
	# runner could not resolve the action, and the failure said nothing about
	# a version number. Invented dependency versions fail in the least
	# informative place available.
	uses: aquasecurity/trivy-action@v0.36.0
	with:
	image-ref: monitorrisk:ci
	format: table
	exit-code: "1"
	severity: CRITICAL,HIGH
	ignore-unfixed: true
	vuln-type: os,library

	# The image must not run as root even if docker-compose.yml is ignored —
	# a customer running `docker run` directly gets the same guarantees as one
	# using compose, because the USER is baked in rather than configured.
	- name: The image runs as a non-root user
	run: \|
	set -euo pipefail
	uid="$(docker run --rm --entrypoint id monitorrisk:ci -u)"
	echo "image runs as uid $uid"
	test "$uid" != "0"

	# Proves the read_only claim at the image level: with the root filesystem
	# mounted read-only and every capability dropped, the app must still start
	# and answer /health.
	#
	# The uploads VOLUME is not optional under --read-only, and this step is how
	# that was discovered: without it the app died with FileNotFoundError on a
	# path inside the image. The secret is 40 characters because config.validate
	# rejects anything under 32 — the first version used a short one and the
	# container refused to start, correctly. If a future change starts writing to the image, this
	# goes red here rather than at a customer.
	- name: It starts under the compose hardening
	run: \|
	set -euo pipefail
	docker network create ci-net
	docker run -d --name ci-db --network ci-net -e POSTGRES_USER=sapsec -e POSTGRES_PASSWORD=citest -e POSTGRES_DB=sapsec postgres:16
	for i in $(seq 1 30); do
	docker exec ci-db pg_isready -U sapsec -d sapsec && break \|\| sleep 2
	done
	docker run -d --name ci-app --network ci-net --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m --cap-drop ALL --security-opt no-new-privileges:true -e DB_DSN=postgresql://sapsec:citest@ci-db:5432/sapsec -e SESSION_SECRET=ci-only-not-a-real-secret-0123456789abcdef -v ci-uploads:/var/lib/sapsec/uploads -p 127.0.0.1:8000:8000 monitorrisk:ci
	ok=""
	for i in $(seq 1 30); do
	if curl -fsS http://127.0.0.1:8000/health > /tmp/health.json; then ok=1; break; fi
	sleep 3
	done
	docker logs ci-app \| tail -30
	test -n "$ok"
	cat /tmp/health.json

	# ------------------------------------------------------------------ #
	# Stdlib purity — enforce the charter boundary mechanically #
	# ------------------------------------------------------------------ #
	# "modules/ stays stdlib-only" is written in CLAUDE.md, and a rule that
	# lives only in a document is a rule that erodes. This makes it fail a build.
	purity:
	name: modules/ and collect/ import nothing third-party
	runs-on: ubuntu-22.04
	steps:
	- uses: actions/checkout@v4
	- uses: actions/setup-python@v5
	with:
	python-version: "3.12"

	- name: Check every import in modules/, collect/ and sap_scanner.py
	run: \|
	python - <<'PY'
	import ast, sys
	from pathlib import Path

	# `collect` is here because decision D4 ALLOWS that package its own
	# requirements and it deliberately spends none: urllib, ssl and
	# xml.etree cover SOAP, OData, SCIM and REST, so a connector tier with
	# NO dependencies is a stronger position than one with different ones —
	# the customer running it against their own SAP system installs nothing.
	# An allowance that is not taken is only real if something checks.
	allowed = set(sys.stdlib_module_names) \| {"modules", "server", "data",
	"collect"}
	offenders = []

	for path in (sorted(Path("modules").glob("*.py"))
	+ sorted(Path("collect").glob("*.py"))
	+ [Path("sap_scanner.py")]):
	tree = ast.parse(path.read_text(encoding="utf-8"), str(path))
	for node in ast.walk(tree):
	if isinstance(node, ast.Import):
	names = [a.name for a in node.names]
	elif isinstance(node, ast.ImportFrom):
	# level > 0 is a relative import, always local
	names = [node.module or ""] if not node.level else []
	else:
	continue
	for name in names:
	root = name.split(".")[0]
	if root and root not in allowed:
	offenders.append(f"{path}:{node.lineno} {name}")

	if offenders:
	print("::error::modules/ and sap_scanner.py must import only the "
	"standard library. The charter change applies to server/ ONLY.")
	for o in offenders:
	print(" ", o)
	sys.exit(1)
	print("clean: the scanner core imports only the standard library")
	PY

	# ------------------------------------------------------------------ #
	# SAP content drift — the catalogue must stay current #
	# ------------------------------------------------------------------ #
	# data/sap_baseline_requirements.json is DERIVED from SAP's Apache-2.0
	# policy repository. Vendoring it keeps the tool working offline and
	# air-gapped, which is the whole product premise — but a vendored copy
	# of someone else's content goes stale silently, and a coverage page
	# measured against a stale denominator is worse than none.
	#
	# So CI re-derives it from source and fails on any difference. That is
	# also free version currency: SAP updates these policies on its own
	# schedule and the build tells us the day it happens.
	#
	# `continue-on-error` is deliberately NOT set. If the fetch breaks, the
	# check must fail loudly rather than quietly stop verifying.

	- name: docs/CHECKS_REFERENCE.md matches the code
	run: python -m tools.build_checks_reference --check

	# The release gate's coverage table drifted twice while it was maintained
	# by hand, under a note claiming it was derived. It is derived now, and
	# this is what keeps that true: a coverage check added without
	# regenerating fails the build rather than going quietly missing from the
	# document somebody reads before wiring the gate into CI.
	- name: docs/RELEASE_GATE.md matches the modules
	run: python -m tools.build_gate_reference --check

	sap-content:
	name: SAP baseline and notes catalogues are current
	runs-on: ubuntu-22.04
	steps:
	- uses: actions/checkout@v4
	- uses: actions/setup-python@v5
	with:
	python-version: "3.12"

	- name: SAP's Baseline Template archive has not moved
	# The job below covers the POLICY repository. This covers the Baseline
	# Template ZIP, a different artefact on a different cadence — the policy
	# set we derive from is v2.4 and the archive ships V2.6. A new version
	# here means a new change_marker.pdf, which is what makes "the
	# requirement changed" distinguishable from "the system changed" in a
	# repeat scan. HEAD plus a 300 KB ranged read of the archive's own
	# central directory; it never downloads the 99 MB.
	#
	# continue-on-error: SAP's support host is outside our control and a
	# transient 5xx must not fail an unrelated pull request. The step still
	# goes red, which is the signal; it does not block the merge.
	continue-on-error: true
	run: python -m tools.check_baseline_zip

	- name: Fetch SAP's published policy repository
	run: \|
	set -euo pipefail
	git clone --depth 1 --filter=blob:none --sparse \
	https://github.com/SAP-samples/frun-csa-policies-best-practices.git /tmp/sappol
	cd /tmp/sappol
	git sparse-checkout set BaselinePolicies NotesPolicies Schema
	echo "policy files: $(find . -name '*.xml' \| wc -l)"

	- name: Re-derive the catalogue and compare
	run: \|
	python - <<'PY'
	import json, sys
	from pathlib import Path
	sys.path.insert(0, ".")
	from server.sapcontent import build_catalogue, load_catalogue

	fresh = build_catalogue(Path("/tmp/sappol"))
	stored = load_catalogue()

	def norm(cat):
	# Compare the CONTENT, not the run-to-run metadata.
	return {"requirements": cat.get("requirements", []),
	"policies": cat.get("policies", []),
	"counts": cat.get("_meta", {}).get("counts", {})}

	if norm(fresh) != norm(stored):
	fr = {r["requirement"] for r in fresh["requirements"]}
	st = {r["requirement"] for r in stored["requirements"]}
	print("::error::data/sap_baseline_requirements.json is out of date with "
	"SAP's published policies. The coverage page is measured against "
	"this catalogue, so a stale copy misreports coverage.")
	if fr - st:
	print(f" SAP added: {sorted(fr - st)}")
	if st - fr:
	print(f" SAP removed: {sorted(st - fr)}")
	print(f" counts stored={stored.get('_meta',{}).get('counts')} "
	f"fresh={fresh['_meta']['counts']}")
	print(" Regenerate with: python -m server.cli rebuild-sap-catalogue /tmp/sappol")
	raise SystemExit(1)
	print(f"catalogue current: {fresh['_meta']['counts']}")
	PY

	# The notes catalogue is derived from the OTHER half of the same upstream
	# repository, by a different generator, for a different consumer. --strict
	# fails on a header line that looks like a note and did not parse, so a new
	# patch-day header format is a build failure here rather than a silent gap
	# in the note list a customer is working from.
	- name: Re-derive the notes catalogue and compare
	run: python -m tools.build_sap_notes_catalogue --source /tmp/sappol --strict --check

	# ------------------------------------------------------------------ #
	# Brand asset drift — the served PNGs must match the master #
	# ------------------------------------------------------------------ #
	# server/static/* is DERIVED from docs/brand/monitorrisk-master.png. The
	# derived files are committed so a deployment — and every other CI job —
	# needs nothing but the repo. But a committed binary with no provenance is
	# exactly the thing nobody notices going stale, so the same rule as the SAP
	# catalogue above applies: if it is derived, CI re-derives it.
	#
	# Pillow is installed HERE and nowhere else. It is a build-time tool, not a
	# runtime dependency, and requirements.txt stays at four packages.
	brand-assets:
	name: brand assets match the master
	runs-on: ubuntu-22.04
	steps:
	- uses: actions/checkout@v4
	- uses: actions/setup-python@v5
	with:
	python-version: "3.12"
	- run: pip install --quiet Pillow
	- name: Re-derive the brand assets and compare
	run: python tools/build_brand_assets.py --check

	# ------------------------------------------------------------------ #
	# Schema upgrade — the documented upgrade path, actually exercised #
	# ------------------------------------------------------------------ #
	# The `server` job applies schema.sql TWICE, which proves IDEMPOTENCY and not
	# UPGRADE. Those are different properties and the gap between them hid a real
	# defect: `CREATE TABLE IF NOT EXISTS` is a no-op on an existing table, so a
	# changed CHECK list altered a fresh database, left every deployed one on the
	# old rule, and passed CI green because CI always built from empty.
	#
	# This job builds a database from the PREVIOUS commit's schema.sql — a real
	# "already deployed" starting point — then applies the current one and asserts
	# the constraint moved. It is the only job that can fail on a migration that
	# was written but does nothing.
	schema-upgrade:
	name: schema upgrades an existing database
	runs-on: ubuntu-22.04
	services:
	postgres:
	image: postgres:16
	env:
	POSTGRES_USER: sapsec
	POSTGRES_PASSWORD: sapsec
	POSTGRES_DB: sapsec
	ports:
	- 5432:5432
	options: >-
	--health-cmd "pg_isready -U sapsec -d sapsec"
	--health-interval 5s --health-timeout 5s --health-retries 12
	# Not a secret, and a scanner will say otherwise. This credential belongs to
	# a service container that exists only for the life of this job, is reachable
	# only from it, holds nothing but a throwaway schema, and is destroyed with
	# the runner. It matches the `server` job below deliberately — two spellings
	# of a disposable CI password is worse than one. A real secret would come
	# from `secrets.`, and none of this repository's jobs needs one.
	env:
	PGPASSWORD: sapsec
	DSN: postgresql://sapsec:sapsec@localhost:5432/sapsec

	steps:
	# Depth 2 so HEAD~1 exists. A shallow clone would make the previous
	# schema unreachable and the job would silently degrade to a fresh install.
	- uses: actions/checkout@v4
	with: { fetch-depth: 2 }

	- name: Recover the PREVIOUS schema.sql
	run: \|
	set -euo pipefail
	if git rev-parse HEAD~1 >/dev/null 2>&1 \
	&& git cat-file -e HEAD~1:server/schema.sql 2>/dev/null; then
	git show HEAD~1:server/schema.sql > /tmp/previous.sql
	echo "using HEAD~1's schema as the deployed starting point"
	else
	cp server/schema.sql /tmp/previous.sql
	echo "::notice::no previous schema reachable; upgrading from the current one"
	fi

	- name: Build a database from the PREVIOUS schema
	run: psql "$DSN" -v ON_ERROR_STOP=1 -q -f /tmp/previous.sql

	# Data first: a migration that only works on an empty table is not a
	# migration. If widening a constraint ever has to rewrite rows, this is
	# where the job starts failing and asking somebody to decide.
	# SEED EVERY TABLE THE MIGRATION TOUCHES, NOT JUST THE ONE IT USED TO.
	# This job seeded `landscape` alone, so for the whole run `sap_system` was
	# empty — which meant a discriminator CHECK validated against zero rows and
	# always passed, a nullability change was never exercised by an insert, and a
	# unique index built over an empty table could never collide. The comment
	# above promised data coverage the job did not deliver.
	- name: Put rows in it
	run: \|
	set -euo pipefail
	psql "$DSN" -v ON_ERROR_STOP=1 -c \
	"INSERT INTO landscape (name, deployment_mode) VALUES ('upgrade-probe','on_prem');"
	psql "$DSN" -v ON_ERROR_STOP=1 -c \
	"INSERT INTO sap_system (landscape_id, sid, client, tier)
	SELECT id, 'PRD', '100', 'prod' FROM landscape WHERE name = 'upgrade-probe';"

	- name: Apply the CURRENT schema over it, then twice more
	run: \|
	set -euo pipefail
	for i in 1 2 3; do
	psql "$DSN" -v ON_ERROR_STOP=1 -q -f server/schema.sql
	echo " application $i ok"
	done

	- name: The rows survived, unaltered
	run: \|
	set -euo pipefail
	psql "$DSN" -v ON_ERROR_STOP=1 -tAc \
	"SELECT count(*) FROM landscape WHERE name = 'upgrade-probe';" \| grep -qx 1 \
	\|\| { echo "::error::the upgrade destroyed existing data"; exit 1; }
	# The pre-existing ABAP system must come through untouched AND be given
	# the new column's default. A migration that silently rewrote sid or
	# client would re-identify every finding attached to it.
	psql "$DSN" -v ON_ERROR_STOP=1 -tAc \
	"SELECT sid \|\| '\|' \|\| client \|\| '\|' \|\| platform \|\| '\|' \|\|
	coalesce(external_key,'<null>') FROM sap_system;" \
	\| grep -qx 'PRD\|100\|abap\|<null>' \
	\|\| { echo "::error::the upgrade altered the existing sap_system row"; exit 1; }

	# THE SHAPE OF THE TABLE, NOT ONLY ITS CHECK CONSTRAINTS. A nullability
	# change and an index definition are both invisible to the constraint
	# assertion below, and both are load-bearing for decision D8.
	- name: Nullability and the tenant index actually moved
	run: \|
	set -euo pipefail
	nullable=$(psql "$DSN" -v ON_ERROR_STOP=1 -tAc \
	"SELECT string_agg(column_name \|\| '=' \|\| is_nullable, ',' ORDER BY column_name)
	FROM information_schema.columns
	WHERE table_name = 'sap_system'
	AND column_name IN ('sid','client','platform','external_key');")
	echo " $nullable"
	[ "$nullable" = "client=YES,external_key=YES,platform=NO,sid=YES" ] \|\| {
	echo "::error::sap_system column nullability is not what schema.sql declares."
	echo "::error::Relaxing a column in the CREATE TABLE alone is a no-op on an"
	echo "::error::existing database — it needs an ALTER in the migration section."
	exit 1; }

	# CREATE UNIQUE INDEX IF NOT EXISTS is idempotent by NAME only; PostgreSQL
	# never compares the definition. So an index whose predicate or column list
	# changed is a silent no-op on every deployed database, and this is the
	# only place that would notice.
	idx=$(psql "$DSN" -v ON_ERROR_STOP=1 -tAc \
	"SELECT indexdef FROM pg_indexes
	WHERE tablename = 'sap_system' AND indexname = 'sap_system_tenant_key';")
	echo " $idx"
	[ -n "$idx" ] \|\| { echo "::error::sap_system_tenant_key does not exist after the upgrade"; exit 1; }
	case "$idx" in
	"landscape_id, platform, external_key") ;;
	*) echo "::error::tenant index columns are not what schema.sql declares"; exit 1;;
	esac
	case "$idx" in
	"NULLS NOT DISTINCT") ;;
	*) echo "::error::tenant index lost NULLS NOT DISTINCT — duplicate tenants become insertable"; exit 1;;
	esac

	# BEHAVIOUR, NOT ONLY SHAPE. The constraints exist to refuse specific rows;
	# asserting their text proves they are declared, not that they work.
	- name: The migrated table refuses what it must
	run: \|
	set -euo pipefail
	refuses() { # $1 = description, $2 = SQL that must fail
	if psql "$DSN" -v ON_ERROR_STOP=1 -q -c "$2" >/dev/null 2>&1; then
	echo "::error::ACCEPTED but should be refused: $1"; exit 1
	fi
	echo " refused: $1"
	}
	L="(SELECT id FROM landscape WHERE name='upgrade-probe')"

	# An empty-string sid satisfies IS NOT NULL and normalises to the same
	# fingerprint as every other empty sid. This is the exact row decision D8
	# exists to make impossible, and the old schema accepted it.
	refuses "an ABAP row with an empty sid" \
	"INSERT INTO sap_system (landscape_id, platform, sid, client) VALUES ($L,'abap','','100');"
	refuses "an ABAP row with no sid at all" \
	"INSERT INTO sap_system (landscape_id, platform, client) VALUES ($L,'abap','100');"
	refuses "a tenant with no external key" \
	"INSERT INTO sap_system (landscape_id, platform) VALUES ($L,'concur');"
	refuses "an unknown platform" \
	"INSERT INTO sap_system (landscape_id, platform, external_key) VALUES ($L,'myspace','x');"

	# And the rows it must ACCEPT, or the migration has bought safety by
	# making the feature impossible.
	psql "$DSN" -v ON_ERROR_STOP=1 -q -c \
	"INSERT INTO sap_system (landscape_id, platform, external_key, tier)
	VALUES ($L,'successfactors','acme-sf-prod','prod'),
	($L,'successfactors','acme-sf-test','dev');" \
	\|\| { echo "::error::two distinct SaaS tenants were rejected"; exit 1; }
	echo " accepted: two distinct SuccessFactors tenants"

	refuses "the SAME tenant twice" \
	"INSERT INTO sap_system (landscape_id, platform, external_key) VALUES ($L,'successfactors','acme-sf-prod');"

	# add-system names UNIQUE (landscape_id, sid, client) as an ON CONFLICT
	# arbiter. PostgreSQL cannot infer an arbiter from a PARTIAL index, so if
	# that constraint is ever replaced by one this raises 42P10 — at runtime,
	# on a path no pytest executes.
	psql "$DSN" -v ON_ERROR_STOP=1 -q -c \
	"INSERT INTO sap_system (landscape_id, sid, client, tier) VALUES ($L,'PRD','100','qa')
	ON CONFLICT (landscape_id, sid, client) DO UPDATE SET tier = EXCLUDED.tier;" \
	\|\| { echo "::error::ON CONFLICT (landscape_id, sid, client) no longer resolves (42P10)."
	echo "::error::server/cli.py add-system depends on this constraint being non-partial."
	exit 1; }
	echo " ON CONFLICT arbiter still resolves"

	# THE ASSERTION THAT MAKES THIS JOB WORTH HAVING. Compare the constraint
	# LIVE IN THE DATABASE against the one this commit's schema.sql declares.
	# A CHECK edited only inside a CREATE TABLE IF NOT EXISTS fails here.
	# ⚠️ THIS STEP FAILS CLOSED, AND THE FIRST DRAFT DID NOT.
	# It was two `grep -oP` pipelines compared with `[ "$a" = "$b" ]`. When the
	# extraction returned nothing — a locale without PCRE, a renamed constraint,
	# a reworded CHECK — both sides were empty, empty equalled empty, and the
	# step reported success having compared nothing. That is precisely the
	# defect this job exists to catch, reproduced inside the job itself. Both
	# sides are now asserted non-empty before they are compared, and the
	# extraction is Python rather than a locale-dependent grep.
	- name: The constraints actually moved
	run: \|
	python - <<'PY'
	import os, re, subprocess, sys

	# EVERY growable value list, not one. This checked `deployment_mode`
	# alone, which was all there was; decision D8 added `platform`, and a
	# single-constraint assertion would have let it be widened in the
	# CREATE TABLE only — the exact mistake, on the exact next constraint.
	# Adding a row here is what registering a new growable CHECK costs.
	CONSTRAINTS = [
	("deployment_mode", "landscape_deployment_mode_check"),
	("platform", "sap_system_platform_check"),
	]

	sql = open("server/schema.sql", encoding="utf-8").read()
	failed = False

	# PostgreSQL does NOT store `IN (...)`. It normalises it to
	# CHECK ((col = ANY (ARRAY['a'::text, 'b'::text])))
	# so parsing the live side for "IN (" finds nothing and — before the
	# non-empty assertions below — compared nothing against nothing.
	# Pull the quoted literals from either shape instead.
	def values(text):
	return sorted(re.findall(r"'([^']*)'", text))

	for column, conname in CONSTRAINTS:
	print(f"\n--- {column} / {conname} ---")
	declared = re.findall(
	rf"{column}\s+IN\s$([^)])$", sql, re.IGNORECASE)
	if len(declared) < 2:
	sys.exit(f"::error::expected the {column} CHECK in BOTH the "
	f"CREATE TABLE and the constraint-migrations section, found "
	f"{len(declared)}. This assertion has stopped checking.")

	live_raw = subprocess.run(
	["psql", os.environ["DSN"], "-tAc",
	"SELECT pg_get_constraintdef(oid) FROM pg_constraint "
	f"WHERE conname = '{conname}'"],
	capture_output=True, text=True, check=True).stdout.strip()
	if not live_raw:
	sys.exit(f"::error::no constraint named {conname} exists in the "
	"upgraded database - the migration in the "
	"constraint-migrations section did not run")

	# THREE-WAY, and it has to be. Comparing only the LAST declaration —
	# the migration block — against the database proves the migration ran
	# and says nothing about whether it agrees with the CREATE TABLE above
	# it. Editing only the CREATE is exactly the mistake this job exists to
	# catch, and a two-way check passes it: the migration and the database
	# still agree with each other, both on the old list. Every declaration
	# in the file and the live constraint must be the same set.
	lists = [values(d) for d in declared]
	have = values(live_raw)

	for i, v in enumerate(lists):
	print(f"declared #{i + 1} in schema.sql : {v}")
	print(f"live in the database : {have}")

	assert all(lists), f"a declared {column} value list parsed as empty"
	assert have, f"the live {column} value list parsed as empty"

	if len({tuple(v) for v in lists}) != 1:
	print(f"::error::the {column} CHECK is declared more than one "
	"way inside schema.sql. The copy in CREATE TABLE governs fresh "
	"installs and the copy in the constraint-migrations section "
	"governs upgrades - an installation gets whichever one it was "
	"born under. Change both.")
	failed = True
	continue

	if lists[0] != have:
	print(f"::error::the {column} CHECK in the database does not "
	"match schema.sql. A CHECK edited inside CREATE TABLE IF NOT "
	"EXISTS is a no-op on an existing database - add a DROP/ADD "
	"pair to the constraint-migrations section at the end of "
	"schema.sql.")
	failed = True
	continue
	print(f"{column}: matches what this commit declares, on both sites")

	if failed:
	sys.exit(1)
	print("\nevery growable constraint matches on every site")
	PY

	# ------------------------------------------------------------------ #
	# Server — needs a real PostgreSQL #
	# ------------------------------------------------------------------ #
	server:
	name: server (PostgreSQL 16)
	runs-on: ubuntu-22.04
	services:
	postgres:
	image: postgres:16
	env:
	POSTGRES_USER: sapsec
	POSTGRES_PASSWORD: sapsec
	POSTGRES_DB: sapsec
	ports:
	- 5432:5432
	options: >-
	--health-cmd "pg_isready -U sapsec -d sapsec"
	--health-interval 5s
	--health-timeout 5s
	--health-retries 12

	env:
	DB_DSN: postgresql://sapsec:sapsec@localhost:5432/sapsec
	SESSION_SECRET: ci-session-secret-not-a-real-deployment-value-0123456789
	PYTHONIOENCODING: utf-8

	steps:
	- uses: actions/checkout@v4
	- uses: actions/setup-python@v5
	with:
	python-version: "3.12"
	cache: pip
	cache-dependency-path: requirements.txt

	# The console is a build artefact and server/spa is gitignored, so without
	# this step it does not exist in CI: every `built` assertion in
	# tests/test_spa_mount.py skips, and the skip guard below correctly counts
	# that as drift. It matters more since the SPA took "/" — an unbuilt bundle
	# used to mean /ui answered 503 while the server-rendered console carried
	# on, and now means every screen answers 503.
	#
	# Node appears in this job and in the Dockerfile's first stage. It is a
	# build-time dependency in both; nothing JavaScript reaches the runtime
	# image or requirements.txt.
	- uses: actions/setup-node@v4
	with:
	node-version: "22"
	cache: npm
	cache-dependency-path: frontend/package-lock.json

	- name: Build the console (type-checks, then emits server/spa)
	working-directory: frontend
	run: \|
	npm ci
	npm run build
	test -f ../server/spa/index.html \
	\|\| { echo "::error::the bundle was not emitted where the server serves it"; exit 1; }

	# THE TYPE-CHECKER ABOVE PROVES THE CODE COMPILES, NOT WHAT THE SCREEN SAYS.
	#
	# The console reached ~5,000 lines of TSX with `tsc --noEmit` as its only
	# check, and two defects shipped through that gap in a single day: a CSF
	# Category filed under "Assessed here" while it displayed the "Export not
	# supplied" chip an inch above, and a null pass rate coerced to 0% by
	# `?? 0`, so a category nobody had scanned rendered as the customer's
	# worst-performing area and sorted to the top of the table. Both
	# type-check perfectly; both invert the product's central discipline on
	# screen. This step is what makes that class fail a pull request.
	- name: Test the console
	working-directory: frontend
	run: npm test

	# `httpx` is TEST tooling, not a runtime dependency, which is why it is
	# here and not in requirements.txt: starlette's TestClient refuses to
	# construct without it, and tests/test_http_console.py goes through it.
	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -r requirements.txt -r requirements-dev.txt httpx

	- name: Byte-compile the server (catches syntax errors in files tests skip)
	run: python -m compileall server -q

	- name: Verify PostgreSQL is actually reachable
	run: \|
	pg_isready -d "$DB_DSN" \
	\|\| { echo "::error::PostgreSQL unreachable — every DB-backed test would SKIP and the job would still pass"; exit 1; }
	psql "$DB_DSN" -v ON_ERROR_STOP=1 -c 'SELECT version();'

	# The schema is the documented upgrade path: an existing deployment
	# upgrades by re-running this file. It has already broken once (a
	# CREATE OR REPLACE VIEW over SELECT f.* could not survive an ALTER
	# TABLE), so applying it TWICE is the regression test.
	- name: Apply the schema twice — idempotency is the upgrade path
	run: \|
	python -m server.cli init-db
	python -m server.cli init-db

	# SEED BEFORE TESTING, not after. Four tests self-skip with "no findings in
	# this database" — they are DATA-dependent, not DB-dependent, and the
	# end-to-end step below used to be the only thing that created findings, so
	# they never ran and the skip guard below counted them as drift. Seeding first
	# buys real coverage of the finding-detail and code-console surfaces instead of
	# relaxing the guard, which would only move the goalposts.
	# SEED BEFORE TESTING, not after. Several tests are DATA-dependent rather
	# than DB-dependent -- they need a scan to have happened, a second system
	# to scope a viewer to, a second landscape to tell a union from a
	# narrowing -- and the end-to-end step below used to be the only thing
	# that created findings, so they never ran and the guard counted them as
	# drift. Seeding first buys real coverage instead of relaxing the guard.
	#
	# The seed itself lives in tools/seed_test_db.py rather than inline here,
	# so `python -m tools.db_test` can reproduce this run locally against the
	# SAME fixture. It used to be forty lines of Python in this file, which
	# meant a developer could not run the database half the way CI does
	# without copying them -- and a copy is a second thing to keep in step.
	# tests/test_ci_seed_is_not_inlined.py fails if it grows back.
	- name: Seed a scan so data-dependent suites actually execute
	run: python -m tools.seed_test_db

	- name: Run the full test suite
	run: \|
	set -o pipefail
	pytest -q -W ignore::DeprecationWarning \| tee /tmp/pytest.out

	# THE GUARD. Before this existed, `pytest -q` ran the DB-backed suites,
	# they skipped for want of DB_DSN, and the job went green having verified
	# none of the journey, none of the analytics and none of the HTTP layer.
	# That is how a bug that broke EVERY page in the console reached main.
	- name: Fail if the database-backed tests skipped
	run: \|
	for suite in test_integration_ingest test_integration_journey test_http_console; do
	n=$(pytest --collect-only -q "tests/$suite.py" 2>/dev/null \| tail -1)
	echo "$suite: $n"
	done
	if grep -qE '[0-9]+ skipped' /tmp/pytest.out; then
	skipped=$(grep -oE '[0-9]+ skipped' /tmp/pytest.out \| head -1)
	# The seed above now creates a second system and a second landscape,
	# so the scoping and narrowing suites RUN rather than skip: the
	# expected number here is zero. One is tolerated for a scanner test
	# that needs an optional sibling engine, which is absent on some
	# runners. More than that means a DB-backed suite is silently not
	# running -- and a suite that skips is worse than one that does not
	# exist, because it LOOKS verified.
	count=${skipped%% *}
	if [ "$count" -gt 1 ]; then
	echo "::error::$skipped — the DB-backed suites are not executing. A suite that silently skips is worse than one that does not exist, because it LOOKS verified."
	exit 1
	fi
	fi

	# docs/CHECK_FIRING.md publishes how many of the product's checks are
	# PROVEN to produce a finding somewhere in this suite. Nothing regenerated
	# it, so the figure would have drifted the first time anybody added a
	# check -- a published number nobody re-derives is a claim, not a
	# measurement, which is exactly what this repository reports on in
	# customers' systems.
	#
	# It runs HERE, in the only job that executes the full suite: the
	# stdlib-only job runs a subset, and conftest deliberately refuses to
	# write a recording from a partial run rather than publish "these checks
	# are unproven" on the strength of tests that never ran.
	#
	# Any difference fails, in both directions. A figure that rose matters as
	# much as one that fell.
	- name: docs/CHECK_FIRING.md matches what the suite proves
	run: python -m tools.build_firing_reference --check

	- name: Prove the console actually answers end to end
	run: \|
	python - <<'PY'
	import sys; sys.path.insert(0, ".")
	from pathlib import Path
	from fastapi.testclient import TestClient
	from server import app as appmod, auth, db, ingest

	# WHAT THIS STEP IS FOR, AFTER THE JINJA RETIREMENT.
	# It existed because a bug broke EVERY page in the console and three
	# layers of testing said it was fine. The pages are gone; the failure
	# did not go away, it moved. A screen is now a JSON endpoint plus a
	# compiled bundle, so both halves are exercised here: every endpoint a
	# screen reads answers 200 with a real database and a real scan behind
	# it, and every screen ADDRESS serves the console that was built in the
	# step above. Either half alone passes while the product is unusable.

	db.init_schema()
	auth.create_user("ci", "ci-password-1234", "admin")

	land = db.one("INSERT INTO landscape (name, deployment_mode) "
	"VALUES ('ci','rise_pce') RETURNING id")["id"]
	sysid = db.one("INSERT INTO sap_system (landscape_id, sid, client, tier) "
	"VALUES (%s,'PRD','100','prod') RETURNING id", (land,))["id"]
	run = db.one("INSERT INTO scan_run (landscape_id, system_id, status) "
	"VALUES (%s,%s,'pending') RETURNING id", (land, sysid))["id"]
	res = ingest.scan_directory(Path("sample_data"), land, sysid, run,
	deployment_mode="rise_pce",
	default_sid="PRD", default_client="100")
	print(f"scanned: {res['findings']} findings, {res['nodes']} graph nodes")
	assert res["findings"] > 0, "the CI scan produced no findings"

	c = TestClient(appmod.app)
	# The JSON sign-in. The form POST that answered 303 was deleted with the
	# templates it rendered; there is one sign-in surface now.
	r = c.post("/api/auth/login",
	json={"username": "ci", "password": "ci-password-1234"})
	assert r.status_code == 200, r.text[:300]
	assert r.json()["username"] == "ci"

	failed = []

	def check(path, expect=200, note=""):
	resp = c.get(path)
	print(f" {resp.status_code} {path} {note}")
	if resp.status_code != expect:
	failed.append((path, resp.status_code, expect, resp.text[:300]))
	return resp

	print("API — what every screen reads:")
	for path in ("/api/dashboard", "/api/findings", "/api/trend", "/api/risk",
	"/api/paths", "/api/coverage", "/api/account", "/api/systems",
	"/api/landscapes", "/api/views", "/api/findings?tier=P1",
	f"/api/runs/{run}", f"/api/runs/{run}/diff", "/health"):
	check(path)

	fid = db.one("SELECT id FROM finding LIMIT 1")["id"]
	detail = check(f"/api/findings/{fid}")
	if detail.status_code == 200:
	for key in ("check_id", "remediation", "risk_narrative", "latest_details"):
	if key not in detail.json():
	failed.append((f"/api/findings/{fid}", f"missing {key}", "present", ""))
	check(f"/api/findings/{fid}/history")

	print("Console — the addresses people type and bookmark:")
	for path in ("/", "/findings", f"/findings/{fid}", "/trend", "/risk", "/paths",
	"/coverage", "/upload", "/account", "/login", f"/runs/{run}",
	"/v/some-slug"):
	resp = check(path)
	# 503 is the "bundle was never built" answer. The step above builds
	# it, so a 503 here means the artefact did not reach the server and
	# the whole console is down.
	if resp.status_code == 200 and '<div id="root">' not in resp.text:
	failed.append((path, "served something other than the console", "index.html", ""))

	print("Retired URLs — bookmarks from before the cutover:")
	nofollow = TestClient(appmod.app, follow_redirects=False)
	nofollow.cookies.update(c.cookies)
	for old_url, expected in (("/ui", "/"),
	("/ui/findings", "/findings"),
	(f"/ui/findings/{fid}", f"/findings/{fid}")):
	resp = nofollow.get(old_url)
	print(f" {resp.status_code} {old_url} -> {resp.headers.get('location')}")
	if resp.status_code != 301 or resp.headers.get("location") != expected:
	failed.append((old_url, resp.status_code, f"301 -> {expected}", ""))

	# A mistyped endpoint must not be answered with the console: the mount is
	# at the root and is the last route, so this is the guard that keeps an
	# integrator's typo a 404 instead of a page of HTML.
	check("/api/findigs", expect=404, note="(a typo must 404, not render)")

	if failed:
	print("::error::the console did not answer end to end")
	for f in failed:
	print(" ", f)
	raise SystemExit(1)
	print("console answers")
	PY

Navigation Menu

"9 systems" while naming eight #215

Workflow file

"9 systems" while naming eight #215

Uh oh!

Workflow file for this run