AI-assisted fork master: build reproducibility and build-step fixes (removes some upstream CI -- see description) - #7
Open
assisted-by-ai wants to merge 2272 commits into
Open
Conversation
adrelanos
pushed a commit
that referenced
this pull request
Sep 8, 2026
- advance submodule gitlinks to upstream's current pointers so Kicksecure sees zero gitlink delta when pulling our master (fixes PR #7 gitlink changes) - absorbs Kicksecure's own arraybolt3/trixie merge and the kicksecure-debug -> kicksecure-ci-tiny-do-not-use flavor rename - keep our newer frozen pin 20260821T022305Z; our SMBIOS-reader design retained
The build lane has ended three times out of four with the build step never concluding, no 'ERROR detected in script' marker anywhere, and the log stopping mid-trace at 3500_install-packages after ~38-40 minutes. That shape means the process was KILLED, not that the build failed -- but nothing recorded why, so each occurrence cost a full re-run to learn nothing new. Collects the evidence while the runner still exists: - docker inspect OOMKilled / ExitCode on every container -- the decisive field - kernel OOM records, from both dmesg and the journal - memory (including swap, which hosted runners may not have), disk, load Runs on failure only and cannot fail the job: an evidence collector that itself fails replaces one unexplained failure with two. Every command is guarded and the script ends in an explicit success. Verified end to end: prints its sections and exits 0 on a healthy host. Deliberately diagnosis, not a fix. The obvious guess is memory pressure -- the failures began with the 24-gitlink bump, which pulled in newer submodule code and made 3500 heavier -- but guessing at a remedy without the OOMKilled flag would be another 40-minute round trip per attempt. Co-Authored-By: Claude <noreply@anthropic.com>
Literal text and newlines in the format string; the data belongs in the argument. Surfaced by the gate now enforcing R-030. Co-Authored-By: Claude <noreply@anthropic.com>
Pinning the build's umask does not reach files the CHECKOUT created, because the checkout happened first. git records only 100644 / 100755 and applies the umask on top: 0644 on a umask-022 host, 0660 on a umask-007 one. Any build step that then copies a source file with a plain 'cp' carries that mode into the image. Measured on commit 41e408b: /etc/debootstrap/packages came out 0644 in CI and 0640 locally, and it was one of only two differences left between the CI image and a local build of the same commit. grml-debootstrap:2273 copies it with a plain cp, so the mode is inherited straight from the checkout -- the same defect class already fixed for the five derivative-maker chroot placements, but in a submodule and therefore untouched by that fix. Normalising the tree to the mode git actually records fixes the whole class at once, wherever the copy happens, instead of chasing each cp site. Verified on a fixture with a parent and a submodule, all four files chmod'ed to the umask-007 form beforehand: regular files become 0644 and executables 0755 in both trees. Co-Authored-By: Claude <noreply@anthropic.com>
The failure-handler approach added earlier cannot work for this failure mode, and the run that proved it is unambiguous: build (b) died and EVERY step from the build onward -- the 'if: failure()' diagnostic and github's own post-steps included -- is recorded as never having run. The runner itself dies, so nothing downstream of the build can capture anything. The only evidence that survives is whatever already reached the job log. So the sampler runs INSIDE the build step, backgrounded, and its readings interleave into that step's own output right up to the moment of death: MemAvailable, SwapFree, MemTotal and the largest RSS consumer by name, once every 30s. One line per interval, so the sampler cannot itself be what fills the log. The build step's trap kills it when the build returns. ci/diagnose-build-kill is kept but its comment now states its real scope: it helps when the build fails and the RUNNER SURVIVES -- a container OOM-killed inside an otherwise healthy job, where docker's OOMKilled flag is decisive. That case is real and validated against a genuine OOM kill; it is simply not this one. Co-Authored-By: Claude <noreply@anthropic.com>
Proven by synthetic probe rather than reasoned about. Two kill shapes were run
against a throwaway workflow in ~1 minute each instead of ~40:
process-oom a memory-limited child killed -- step FAILS, runner survives:
the step-level collector runs, and so does a separate job.
runner-oom the job does not fail cleanly: the step-level collector is
SKIPPED, while a separate job still runs and reports
'victim: cancelled | 1=success,2=cancelled,3=skipped,4=success'.
The second shape is what the build lane hits: the build step and everything
after it, including github's own post-steps, are recorded as never having run.
No step in that job can report anything, which is why the earlier 'if: failure()'
collector produced nothing.
The postmortem job runs on a FRESH runner, so it survives. It cannot see the
dead runner's memory or dmesg -- the in-step MEMSAMPLE lines cover that, being
written into the log while the build is still alive -- but it makes the shape of
the failure explicit instead of leaving a silent mystery.
Co-Authored-By: Claude <noreply@anthropic.com>
Workaround, deliberately loud rather than silent: the target now warns and sets dist_build_windows_installer=false, so a build that asked for it cannot mistake the result for a Windows installer having been produced. The target downloads the VirtualBox Windows installer and verifies its Authenticode signature with osslsigncode, which fetches every CRL distribution point over the network at verify time. That makes an otherwise offline '--freshness frozen' build depend on a live fetch to crl3.digicert.com, and it failed a CI build outright: PEM routines:get_name:no start line Use the "-CRLfile" option to add one or more CRLs in PEM format. Signature verification: failed while the identical installer verified fine in the sandbox minutes earlier. A real fix already landed in developer-meta-files 9ac5d857 (fetch the CRLs with curl, convert DER to PEM, retry with -CRLfile, revocation checking preserved). This disables the target anyway so the lane stops depending on it while CI reliability is the priority. Rationale, evidence and re-enable steps recorded in ~/TODO.md. Co-Authored-By: Claude <noreply@anthropic.com>
- Both call sites commented out: the per-flavor sanity-tests loop and the prepare-build-machine call that actually triggers the VirtualBox Windows installer download. - WHY: osslsigncode verifies the installer's Authenticode signature by fetching every CRL distribution point over the network at verify time, so an otherwise offline --freshness frozen build depends on a live fetch to crl3.digicert.com. It failed CI outright. - Disabled at the caller, not neutralised in parse-cmd: --target windows keeps meaning what it says for anyone who passes it deliberately. - Details and the re-enable steps: ~/TODO.md, "osslsigncode / Windows installer target". Co-Authored-By: Claude <noreply@anthropic.com>
Call dm-debconf-normalize-owners on the rootfs config.dat before packing (4350 raw/qcow2, 3600 ISO), so debconf question ownership is byte-identical regardless of the build environment's configure sequence. Last local-vs-CI difference. Root cause + fix detail in the developer-meta-files commit. Co-Authored-By: Claude <noreply@anthropic.com>
…cibility - Call dm-normalize-fat-partition on the vfat ESP (found on the same kpartx mapping mke2fs just wrote) so the FAT is byte-identical across build hosts -- the 3.44 MB local-vs-CI difference. mke2fs -d already handles the ext4 root. - Pin SYMLINK mtimes in the reimage tree to SOURCE_DATE_EPOCH (touch -h): the build's file-mtime pinning uses touch WITHOUT --no-dereference, so it follows update-alternatives links (e.g. /usr/sbin/policy-rc.d) and pins the target, leaving the link's own wall-clock mtime -- the last local-vs-CI byte in the root partition. - Add dosfstools + mtools to the build deps (mkfs.fat + mcopy/mmd); they are only Recommends of grml-debootstrap. Co-Authored-By: Claude <noreply@anthropic.com>
Adds check-sufficient-disk: a full ISO+qcow2 build writes a ~1.3 GB sparse raw, a live-build chroot, the qcow2 and working copies, and silently runs out deep in a later step (observed: 3600 live-build bootstrap_cache restore, ~40 min in, 'No space left on device'). Check free space in the build output folder up front and fail with a clear message + how much is needed. Floor 20 GiB, override via dist_build_min_free_gib. Co-Authored-By: Claude <noreply@anthropic.com>
A reproducibility check needs only the tiny '.sha512sums' to compare hashes, but today those sidecars are bundled inside the ~1.3 GB 'repro-image' artifact, so a verifier downloads the whole image for one hash line (learned the hard way). - local-build.yml: new 'repro-checksums-<copy>' artifact (~100 KB) with every '.sha512sums' (+ '.asc'/'.sig'), image + torrent signatures, and the signing PUBLIC keys. repro-image stays as-is for the compare job. - 5200_prepare-release: export the auto-generated signing PUBLIC keys next to the build output (signify '.pub' verifies '.sig'; OpenPGP cert verifies '.asc'). Public keys only; best-effort so it can never fail an otherwise-good build. Bundle assembly verified synthetically against a real local build's output: 22 files, ~110 KB, both pubkeys exported cleanly (sq cert export + keyname.pub). Co-Authored-By: Claude <noreply@anthropic.com>
… too) Image filenames are '<Product>-<flavor>-<version>...', so 'Kicksecure-*' globs silently drop Whonix-Gateway/-Workstation images. Match the distributable images and sidecars by extension (.qcow2.libvirt.xz / .iso / .ova and .sha512sums / .asc / .sig / .torrent / .pub) so both products are covered. Verified against a real build's output: repro-image catches images+sidecars (no .raw), repro-checksums catches only the small sidecars+pubkeys, neither catches the build-cache dirs. Co-Authored-By: Claude <noreply@anthropic.com>
Same generalization as local-build.yml: 'Kicksecure-*.qcow2*/.iso/.ova' globs miss Whonix images; match by extension so both products work. The variables 'Kicksecure-*' VMNAME defaults are per-flavor config (kicksecure-cli etc.), correct as-is; Whonix flavors set their own. Co-Authored-By: Claude <noreply@anthropic.com>
Boot-test image discovery hardcoded 'Kicksecure-*.{iso,ova,qcow2.libvirt.xz}',
which misses Whonix images. Match by extension only. (Also fixed a pre-existing
R-030 printf in the touched file.)
Co-Authored-By: Claude <noreply@anthropic.com>
The disk path (4350) already pins symlink mtimes to SOURCE_DATE_EPOCH; the ISO squashfs chroot had the same update-alternatives symlink wall-clock-mtime leak (the build's mtime-pinning follows links, pinning targets not the links). Add the same 'find -type l -exec touch -h @sde' before mksquashfs. The ISO EFI FAT (efi.img) is already reproducible via live-build (SOURCE_DATE_EPOCH volume-id + pinned mtimes + zeroed fresh image), so no FAT normalization is needed on the ISO path -- confirmed by upstream live-build code review. Co-Authored-By: Claude <noreply@anthropic.com>
- dist_build_sign_and_tag now defaults false (variables): signing rewrites submodule HEADs + re-commits gitlinks, which local/dev/AI builds do not want. An explicit value (--sign-and-tag, env, dispatch site) still wins. - dist_build_ignore_unsigned now defaults to the INVERSE of sign_and_tag, so a non-signing build skips signature verification in lockstep instead of failing git_sanity_test on its own unsigned tree. This is the behaviour git_sanity_test already documented; the code now matches. - Redistributable dispatch opts in explicitly (a redistributable image requires signatures; git_sanity_test rejects unsigned+redistributable): local-build.yml, local-boot-test.yml, ci/reproducible-build-twice, ci/dry-run.d/200_sign-and-tag. CI build behaviour is unchanged -- it signs exactly as before, now by explicit opt-in rather than the default. - parse-cmd help text updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dry-run runs BOTH 200_sign-and-tag and the build (300) via run-as-user, whose 'sudo --preserve-env=PATH' strips every var but PATH. With dist_build_sign_and_tag defaulting off, the earlier fix set it only for step 200, so the build's git_sanity_test saw sign_and_tag=false -> ignore_unsigned=true, colliding with the forced dist_build_redistributable=true (mutually exclusive) and failing the lane. - 300_run-derivative-maker: add dist_build_sign_and_tag=true to the env prefix that already carries CI/arch past run-as-user, so git_sanity_test sees it. - 200_sign-and-tag: keep the env-prefix opt-in so sign-and-tag actually signs (300 asserts the buildinfo carries recorded provenance, not 'unrecorded'). - ci/dry-run: drop the export attempt (run-as-user strips it) and document that the opt-in must ride each run-as-user hand-off's env prefix. The full build lanes (local-build.yml, local-boot-test.yml, ci/reproducible-build-twice) run dm-build-official directly with no run-as-user layer, so their existing export reaches git_sanity_test -- confirmed by the full build passing its signature verification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vbox-ova-reproducible-normalize is present at the pinned helper-scripts gitlink (63608f3d), so the .ova export no longer aborts dm-prepare-release. Offer 'virtualbox' as a target choice; REPRO_MULTI_TARGET_LIST already yields 'virtualbox' for a non-iso target, the upload globs already collect *.ova, and the compare job already maps virtualbox -> *.ova. So a virtualbox dispatch builds two runners' .ova and diffoscopes them byte-for-byte. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-in only
The reader turns a firmware-supplied string (SMBIOS Type 1 Serial Number, after a
'dm-cmdline=' sentinel) into trailing kernel command line, so whoever can write
that field gets 'init=/bin/sh'. The kernel command line is the part of the boot
chain Secure Boot does not authenticate, and vm-config-dist is a dependency of
dist-general-cli, so the reader shipped on bare-metal installs too. It exists only
for the QEMU boot-test harness.
Realistic writers without OS root: Supermicro SUM EditDmiInfo/ChangeDmiInfo over
IPMI with no OS on the box, HPE via RBSU through iLO KVM, and any OEM or
refurbisher DMI editor -- set once in the supply chain it survives disk wipe and
reinstall. Stock Redfish is narrower: SerialNumber is readonly, only AssetTag is
writable.
- help-steps/parse-cmd, help-steps/variables: --smbios-reader [true|false],
default false, exported to the build steps.
- 3500_install-packages: write /etc/default/grub.d/50_dm-smbios-reader.cfg into
the chroot before grub-mkconfig; remove it when disabled, so it cannot survive
from an earlier build in the same slot.
- 3600_convert-raw-to-iso: gate both halves of the ISO path -- the
${dm_smbios_extra} placeholder on --bootappend-live, and inlining the reader
into config.cfg.
- live-build-data/grub-config: the reader lives in smbios-reader.cfg so it can be
omitted; config.cfg no longer carries it.
- local-boot-test.yml: pass --smbios-reader true. Without it every boot leg would
time out with a 0-byte serial log rather than fail loudly.
- Bump the vm-config-dist gitlink for the matching disk-path guard.
Not a Secure Boot bypass: Secure Boot never covered the kernel command line. This
is a hardening gap -- every attacker who can reach it can already do worse -- but
Kicksecure appears to be the only distro reading SMBIOS into the cmdline in
grub.cfg, and under SEV-SNP/TDX it would be a real boundary crossing, which is why
systemd stopped trusting SMBIOS there.
Verified without a rebuild: placeholder escaping is byte-identical to the previous
inline form and empty when disabled; the opt-in was exercised through a built
image's own update-grub in a chroot (absent 0 reader lines / 0 placeholders,
enabled 5 / 11); the QEMU gate was exercised by booting a built image five ways,
canary-checked. The build-option plumbing itself is checked statically only and
needs one real build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
actionlint rejects 'if: false' as a constant condition. The step is already inert, so commenting it out is behaviour-identical and matches how the other disabled vbox pieces are expressed. Re-enable instructions kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ISO copy of the reader, kept identical to the disk copy in vm-config-dist, plus the matching gitlink bump (00dd919..52d2087, reachable on org-ai-assisted). Type 1 offset 7 holds a string INDEX, and 0 means "no string" (DSP0134 6.1.3), so an unguarded --get-string 7 makes GRUB print 'error: failed to retrieve the structure field.' on every ordinary boot of an image built with the reader -- the QEMU gate passes on any QEMU guest, and the serial is set only when a tester injects. Probe the index byte first; read the string only when it is non-zero. Found by coderabbit on org-ai-assisted/vm-config-dist#8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The header described resolvability only, while the job also fails a gitlink that deviates from the merge base with Kicksecure/derivative-maker. - Both invariants stated, with why 1 does not imply 2: a pointer merely BEHIND the submodule's upstream tip is resolvable, yet an upstream merge still conflicts on the submodule for anyone without it checked out. - 'submodules: false' and the default depth 1 are now explained as deliberate: the check must reach its verdict WITHOUT submodule objects, since that is the case that conflicts, and it deepens from origin itself when it needs history. - GITHUB_TOKEN is documented as required, not a rate-limit nicety: a sweep that verified nothing exits 2 rather than reporting a green pass. - Step names say what the steps now do. No behaviour change; the tool needed no workflow change to start enforcing invariant 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Configuration files are not auto-generated. The drop-in was printf-ed inline, so its content lived only inside this build step -- unreadable and undiffable where it belongs, and dependent on this script's quoting. It now ships as vm-config-dist usr/share/vm-config-dist/50_dm-smbios-reader.cfg and is copied with the same 'install --mode' used for 20_dist-base-files.cfg above, which is also what keeps the installed mode independent of the builder's umask. The unconditional safe-rm stays outside the branch on purpose: a disabled build must clear a file left by an earlier build in the same slot. Also bumps the vm-config-dist gitlink to carry the new file (52d2087..0a7b353, reachable on org-ai-assisted). Verified through the image's own update-grub in a chroot, no rebuild: drop-in absent 0 reader lines, 0 cmdline placeholders static file copied 5 reader lines, 11 placeholders value not "true" 0 reader lines, 0 placeholders The first run of that check reported "absent" as enabled -- the image is reused across runs and an earlier run had deliberately left it opted in under the previous filename, which the rename orphaned. The check now clears every 50_dm-*.cfg before the absent case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match dm-reproducible-compare-artifacts' dedicated 'artifacts differ' code (4, was 1). These consumers forward the comparator's exit verbatim, so only the documented contract changes: 0 reproducible, 4 differ, 2 setup, 3 build failed (build-twice), 1 internal error.
…ss.bsh Follow the developer-meta-files rename of reprepro-freshness.bsh (the library also manages package-build-hash manifests). Source-path + comment update only.
Follow package-build-freshness dropping its reprepro path: dm_package_needs_rebuild now takes (source_dir, apt_codename) and decides from the source-content hash, so the call no longer passes the repository name and the comment describes content detection, not a published-version check.
Document the tested behaviour of dm-reproducible-compare-artifacts on real multi-gigabyte iso/ova images: the sha256 verdict is authoritative, but the best-effort diffoscope EXPLANATION can OOM (measured: a 1.4 GB differing .iso -> diffoscope exit 2 'Out of memory'), and the tool degrades gracefully to the sha256 verdict. Note that extending the qcow2 mounted route (attach read-only, diffoscope the mount points) to iso/virtualbox would give them a memory-safe explanation -- a future improvement, not required for a correct verdict.
- libpam-tmpdir (security-misc) points TMPDIR at a path absent in the chroot; debootstrap inheriting it aborts opaquely (Debian #823651) - must clear at the invocation: sudo env_reset + PAM re-set it after the pbuilder configfile is sourced, mirroring the adjacent LD_PRELOAD unset - add R-010 export LC_ALL=C to the strict block (inert here; variables re-sets C.UTF-8 on source) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mount-cleanup refuses a non-root EUID; these two callers invoked it bare
(all 7 others use ${SUDO_TO_ROOT}), so it died 'MUST be run as root'
during 3500 teardown once the build reached it. Also add the missing
'export LC_ALL=C' both files' strict blocks lacked (R-010).
- matches xtrace.bsh output_cmd becoming an array (printf %s\n over echo) - back-compatible with a scalar output_cmd during the rollout window
- Default true: 4x3/BIOS is the reproducible-build default (primary use case is BIOS VMs); grub-mkconfig in the chroot must not read build-host firmware. On env_vars_keep_list so it crosses into the chroot.
The underscore path usr/libexec/helper_scripts/strings.bsh does not exist (the dir is helper-scripts); help-steps/variables is sourced by every build step, so every build aborted. Matches line 185's shellcheck directive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- dist_build_image_upload_supported now STICKS true across multiple --target: dm-upload-images gates the whole upload on it, so token order must not let a later unsupported target flip it back - multiple-flavor guard fires even when one --flavor is source: source_run no longer exempts it, so '--flavor source --flavor X' is refused - note --target/--type compatibility as an accepted, unvalidated limitation
- vm_names_to_be_exported=none (set for --flavor source) no longer derives a bogus '.../none-<ver>.<arch>.raw' path; the vars stay unset, matching the kicksecure single-build case - move set_default_variable + the derivation into source-able variables-lib.bsh so a unit test can source and call them directly
- kicksecure --type host has dist_build_type_long=kicksecure-host, not a substring of Kicksecure-CLI, so the skip does NOT fire and the vars are populated; only workstation/unified builds consume them, so it is unread
- substring match is not path-containment; dist_build_version with '../' escapes it. Documented as a known, out-of-scope limitation (builder-set override, self-inflicted threat model)
- new grml-debootstrap bails when SOURCE_DATE_EPOCH is set (frozen builds always set it) unless --defaultinterfaces is passed
Prints declare -p and exits before any build step; consumed by dist-ai dm-help-steps-tests to baseline variables ahead of a variables refactor. Default unset -> no build impact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tracks the helper-scripts has.sh->has.bsh rename.
- Converts a raw disk image into a highly boot-compatible hybrid ISO (legacy BIOS + UEFI + UEFI Secure Boot via shim->signed grub + loopback.cfg + USB/DVD hybrid) using only packages.debian.org tools; dracut-based. Reimplements what live-build's ISO packaging does, without live-build. Prep for porting 3600_convert-raw-to-iso off live-build. - Static GRUB config templates in dm-raw-to-iso.d/ (no auto-generated config). - All live-build architectures: amd64, i386, arm64, armhf. - dracut --add dmsquash-live (hardened images suppress auto-inclusion); neutralize the input's /etc/fstab + /etc/crypttab so a disk rootfs boots live, not into emergency mode. - Verified: builds an authentic Kicksecure-CLI raw into an ISO that boots to a login prompt under bios and efi-secureboot. Tests + docs live in dist-ai. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Feature-preserving: the ISO keeps the custom GRUB theme, the three live boot roles (USER/SYSMAINT/UNRESTRICTED) and the Utilities + keyboard-layout submenus, now supplied to the generic tool via new --grub-overlay/--live-overlay. - Dead debian-installer scaffolding dropped (already disabled, never reached the ISO; install-to-disk is Calamares in the live session). - The live-build submodule gitlink is intentionally NOT removed (human-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ai-review (grok/claude) on the port: - The ISO squashfs missed initializer-dist's chroot-scripts-post.d (approx apt sources removal, random-seed wipe, autologin, setup wizard, MOK leak check): 4300 skips them for dist_build_iso and runs after 3600, and the old live-build path ran them itself. 3600's prepare_raw now runs them, mirroring 4300. - The ISO grub/shim/memtest build deps are host-arch-specific; installing them unconditionally in 30_dependencies.conf broke arm64 hosts and non-ISO builds. They already live per-arch in 1200 under dist_build_iso; only arch-independent librsvg2-bin stays in 30. Added memtest86+ (amd64) and grub-efi-arm64-signed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ai-review (claude) on the fix commit: - prepare_raw armed the unchroot_unmount cleanup trap but tore the chroot/mount down itself before returning; a later failure (overlay build, tool run) would re-run unchroot-raw/unmount-raw against an already-clean state and mask the real error. Reset the trap to exception_handler_general after teardown. - memtest is an optional diagnostic (its GRUB entry self-hides when absent), so a missing memtest86+ binary now warns and skips instead of failing the whole ISO (matters for a cross-arch / minimal build host); a native amd64 build still installs and stages it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- 'cp' without -p creates dest as source-mode & ~umask, so the ISO's boot/grub/kb_layouts modes depended on the builder's umask; chmod them to fixed 0644/0755 for a byte-identical rebuild. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- the bootstrap apt-get-noninteractive wrapper sources helper-scripts libraries (strings.bsh -> wc-test.sh -> check_runtime.bsh); staging only the wrapper aborted the first in-image apt-get with "strings.bsh: No such file". Stage the transitive closure alongside it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- 4300 no longer skips dist_build_iso: the ISO runs the same chroot-scripts-post.d as raw/qcow2/vbox (all five are live-safe). - rename 3600 -> 4310_convert-raw-to-iso so it runs after 4300; drop the embedded post-scripts + duplicate set-grub-keymap (3500 builds them). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- the boot-test job checks out submodules:false, but dm-boot-test sources helper-scripts' package_installed_check.sh; every leg aborted before qemu. Init that one submodule before the fork-branch switch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- live ISO booted UNHARDENED: the live cmdline carried none of security-misc's kernel hardening (mitigations, slab_nomerge, rd.shell=0, rd.emergency=halt); the old live-build path scraped it from grub-mkconfig, so restore that in 4310 - cross-arch real ISO builds need target-arch loaders the host lacks; fail early and clearly (native-only) instead of deep in dm-raw-to-iso - exotic ISO --arch (e.g. ppc64el) passed CLI validation then aborted in the converter after a ~40 min build; gate it in parse-cmd to the converter's set Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Debian Trixie ships no signed ARM32 GRUB and no ARMHF shim, so dm-raw-to-iso
hard-errors on the missing gcdarm/shimarm for armhf's only EFI platform; every
real armhf ISO build fails. Gate ISO --arch to {amd64,arm64} (the archs with
signed GRUB+shim), same spirit as the ppc64el rejection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- live-build built the ISO bootloader inside the target chroot, so a cross-arch ISO worked; this port sourced the signed EFI loaders + grub modules from the HOST, so cross-build failed on the host's wrong-arch loaders - dm-raw-to-iso now takes the signed loaders, grub modules and unicode.pf2 from the mounted TARGET rootfs (grml installs grub-efi-<arch>-signed + shim-signed per target arch), so a cross-arch build gets the correct-arch loaders - revert the native-only gate in 4310 (it was the wrong fix for the host/target mismatch); cross-build now works, native unchanged Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- 4310: extract the rootfs kernel hardening by token content (strip linux/kernel-path, then root= and a standalone ro) instead of a positional field cut, which silently drops a hardening param -- or leaks root= and breaks the live boot -- if grub's token layout shifts - 1200: reject a non-username $USER (the sudoers word ALL, or a metacharacter) before writing the passwordless-sudo rule; USER=ALL would otherwise grant every account root and still pass visudo --check - dm-raw-to-iso: mksquashfs -one-file-system + proc/*,sys/* excludes, so a silently-failed API-bind unmount cannot pack the build host's /proc,/sys into the public live ISO Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- EFI-signing set (efivar, sbsigntool, keyutils, shim-signed-common, fwupd, fwupd-signed, efibootmgr) is installed by 3500's availability-checked efi_weak_recommended list; a hard ISO entry instead breaks a build on an arch that lacks the package. - base / boot packages (ca-certificates, less, mokutil, dracut, grub-live-dracut, sudo, tirdad, legacy-dist, kicksecure-packages-dependencies-pre, mate-polkit) already come from the flavor metapackage graph or 3500 itself.
- the tool ships no GRUB config: --grub-overlay is required and must carry config.cfg/grub.cfg/loopback.cfg/esp-redirect.cfg; @APPEND_LIVE@ and @timeout@ are substituted in staged cfgs and --serial-console injects the serial terminal. - iso-build-data/grub-config is now the single ISO GRUB config source (loopback.cfg and esp-redirect.cfg moved in from the deleted dm-raw-to-iso.d/); 4310 stages them. - boot-verified in the sandbox: BIOS + EFI + EFI-secureboot reach a login prompt.
- variables: the SUDO_TO_ROOT re-entrance guard flag was exported, so a child process (build steps, recursive derivative-maker/genmkfile) inherited the flag but not SUDO_TO_ROOT, rebuilt the bare default, and skipped the --preserve-env append -- dropping DIST_APTGETOPT_SERIALIZED (and the other passed vars) from sudo. Keep the guard same-shell only: do not export it. - mmdebstrap: define a local exit_with_error; the wrapper cannot source pre, so a failed sanity check aborted with 'error: command not found' instead of the real cause. Also add the missing 'export LC_ALL=C'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the accumulated work on the org-ai-assisted fork's master, offered
upstream. It is AI-assisted work, reviewed by a human before this PR was
opened. Please read the removals below before the diff.
Read this first: what this PR REMOVES from upstream
Four removals are upstream-affecting and were made for the fork's needs, not
yours. They are called out here so they are not discovered inside a 100-file
diff. I am happy to restore any or all of them on request:
.github/workflows/local-test-build.yml-- thebuild:job is deleted.That is the automated builder (ansible,
ANSIBLE_VAULT_PASSWORD, artifacthandling, 240 minute timeout), gated
if: github.repository == 'Whonix/derivative-maker'. The fork cannot run it(no secret, and the gate never matches), so it was dropped; upstream can.
The rest of the file is retained and rescoped to the dry-run gate.
ci/builddeleted (229 lines).ci/github-builddeleted (207 lines).ci/dry-run.d/400_reproducible-buildinfodeleted (62 lines).The fork's lanes drive
help-steps/dm-build-officialdirectly instead of goingthrough
ci/build, which is why those entry points went away here. If upstreamstill uses them, they should be restored before merging.
What it adds
228 commits, 100 files, +5126 -2619. By area:
help-steps/(35 files) --variables,pre,parse-cmd,unmount-treeand the mount/unmount helpers.
build-steps.d/(19 files) -- notably1100_sanity-tests,1200_prepare-build-machine,2100_create-debian-packages,3500_install-packages,3600_convert-raw-to-iso, and a new4350_reimage-raw-reproducible.docker/(8 files) -- the run/setup scripts, including pinning apt to thebuild's frozen snapshot so a container rebuild does not silently change the
package set.
ci/(15 files) and.github/workflows/(7 files) -- see the caveat below.A large share of the work targets build reproducibility: normalising the EFI
FAT ESP,
update-alternativessymlink mtimes, debconf owner spellings, and ucftemp paths, plus a reproducible re-image step.
Caveat: the fork's CI lanes are included
The
local-*.ymlworkflows are the fork's own lanes. Five of the six are gatedif: vars.CI_ENABLED_ORG_AI_ASSISTED == 'true', so in a repository that neversets that variable they resolve false and consume no runner.
local-test-build.ymlis not gated that way, but it is pre-existing upstream and this PR only rescopes
it (and fixes its tag filter:
'*'does not match a slash, so a slash-delimitedtag silently skipped the gate; it is
'**'now).They are included because this PR is the fork's master as-is. Say the word and I
will strip them.
Submodules
No submodule gitlink is touched. Verified: merging this branch changes zero
mode-160000 entries, and the merge is clean in a clone with no submodules
checked out, which is the case that otherwise conflicts.
Testing status, honestly
locally versus CI on the fork's lanes.
(
11269ee1), with no gitlink changes and no conflicts.upstream's own CI, which has secrets and runners the fork does not.
so it can be reviewed or taken piecemeal.
Happy to split this into smaller PRs by area if that is easier to review.