Skip to content

POC: split rootfs into bootstrap + pkgs images - #6062

Draft
eriknordmark wants to merge 56 commits into
lf-edge:masterfrom
eriknordmark:split-rootfs2
Draft

POC: split rootfs into bootstrap + pkgs images#6062
eriknordmark wants to merge 56 commits into
lf-edge:masterfrom
eriknordmark:split-rootfs2

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Proof of concept: split the monolithic rootfs into bootstrap + pkgs

EVE's rootfs partition has a fixed size cap for backward compatibility, but the
platform keeps growing with new services and firmware. This POC splits the
single rootfs into two squashfs images: a small bootstrap image with
everything needed to boot to a stable, minimal system (kernel, init, firmware,
hypervisor, storage/networking, security hardening, and core orchestration
including pillar), and a separate pkgs image holding non-critical services
(observability and ancillary tooling) that are mounted and started once the
system is up. The device boots only the bootstrap image; the pkgs image is
staged and consumed at runtime, keeping the boot path under the legacy size
limit while leaving room to grow.

A new pillar agent, extsloader, discovers and activates the services carried
by the pkgs image, and rescans on base-OS updates. The branch also adds the
LinuxKit templates and make targets to build/run the two images, an erofs/verity
tooling path, size enforcement for the Core and OCI portions, and Eden test
scripts covering split installer boot (kvm/k/k-zfs/xen) and OTA upgrade from an
old monolithic image.

Design and rationale are documented in docs/SPLIT-ROOTFS.md,
docs/SPLIT-ROOTFS-DESIGN.md, and the implementation roadmap.

Status: draft / WIP — opened for visibility and review of the approach. The
series still contains WIP checkpoints and is not yet squashed into logical
units.

OhmSpectator and others added 30 commits June 17, 2026 11:09
@OhmSpectator @eriknordmark
build: add split rootfs artifacts
5b95544 
Add dedicated bootstrap/pkgs LinuxKit templates, glue them into the
build graph, and expose make targets for producing each squashfs so the
new architecture can be built independently of the legacy rootfs.

Signed-off-by: Nikolay Martyanov <nikolay@zededa.com>
@OhmSpectator @eriknordmark
build: add bootstrap/pkgs run workflows
263245c 
Wire the split rootfs into developer workflows by adding live/run
targets, and documenting the new entry points in the help output.

Signed-off-by: Nikolay Martyanov <nikolay@zededa.com>
@OhmSpectator @eriknordmark
pillar: add extsloader agent
b3d32d4 
Introduce the external-services loader that discovers pkgs.img, mounts
it, and starts services through containerd, and register the agent
inside the boot sequence.

Signed-off-by: Nikolay Martyanov <nikolay@zededa.com>
@OhmSpectator @eriknordmark
docs: describe split rootfs POC
22ee7ea 
Add SPLIT-ROOTFS.md outlining the motivation, build pipeline, and Pillar
integration for the bootstrap/pkgs experiment.

Signed-off-by: Nikolay Martyanov <nikolay@zededa.com>
@OhmSpectator @eriknordmark
WIP: universal installer + CONFIG-based HV selection
53abe81 
- Makefile: installer-split target, eve-hv-supported metadata injection
- installer.yml.in: include rootfs-ext.img in installer
- pkg/installer/install: read HV from CONFIG, copy ext to persist
- pkg/grub: read eve-hv-type from CONFIG, write /run/eve-hv-type
- storage-init: select ext4/ZFS based on CONFIG eve-hv-type
- onboot.sh: propagate CONFIG eve-hv-type to runtime
- extsloader: skip kube service when HV != k
- rootfs_core/ext/universal.yml.in: split rootfs templates
- docs/SPLIT-ROOTFS-DESIGN.md: v1.6 with POC implementation status

Signed-off-by: Nikolay Martyanov <ohmspectator@gmail.com>
@OhmSpectator @eriknordmark
WIP: make pillar-k universal for k/kvm flows
54be24a 
- gate zedkube startup in device-steps by eve-hv-type (/run/eve-hv-type, fallback /etc/eve-hv-type).
- subscribe to ENClusterAppStatus in zedmanager only for kube HV.
- update rootfs_ext service template to match the split/universal service set for this WIP.

Signed-off-by: Nikolay Martyanov <ohmspectator@gmail.com>
@OhmSpectator @eriknordmark
WIP: exclude split service dirs from large-file disk metrics
94d9797 
Exclude /persist/pkgs and /persist/eve-services from volumemgr large-file scan to avoid oversized DiskMetric payloads in split rootfs runs. Also tighten FindLargeFiles exclude matching to directory boundaries and add a regression test.

Signed-off-by: Nikolay Martyanov <ohmspectator@gmail.com>
@OhmSpectator @eriknordmark
wip: move kube-only behavior to runtime guards
8c774bb 
Switch kube-only behavior from build-time assumptions to runtime checks for pillar-k running on non-k hypervisor types.

- gate PVC disk metrics and PVC format bootstrap scan by runtime HV type
- add shared kube runtime guard and apply it across exported kubeapi functions in //go:build k files
- add runtime fail-fast checks in zedkube and kubevirt hypervisor exported entry points
- reject kubevirt hypervisor selection when runtime HV type is not k
- make CSI handler constructor safely fall back when kube runtime is disabled
- keep node-drain behavior runtime-aware (NOTSUPPORTED on non-k)
- handle DeleteNAD call-site errors explicitly

This keeps split/universal images runtime-driven instead of relying on compile-time -k behavior.

Signed-off-by: Nikolay Martyanov <ohmspectator@gmail.com>
@OhmSpectator @eriknordmark
WIP
2b69157
@OhmSpectator @eriknordmark
mkrootfs-erofs: add LinuxKit package files
162419a
@OhmSpectator @eriknordmark
WIP: checkpoint split-rootfs before master rebase
966276f
@OhmSpectator @claude @eriknordmark
WIP: split-rootfs OTA with CAS self-heal (monolith->split tested)
d802b34 
Split-rootfs OTA update mechanism:
- Makefile: eve-split target rewritten to avoid recursive make (fixes
  dirty-timestamp and monolithic rootfs rebuild issues). Uses cp instead
  of symlinks so Docker COPY gets real files, not dangling symlinks.
- Dockerfile.in: conditional #SPLIT_ROOTFS_LABEL# for Extension disk label
- baseosmgr/worker.go: WriteExtensionToPersist after WriteToPartition
- baseosmgr/handleextension.go: CAS helper for Extension extraction
- cas/extension.go: FindAdditionalDiskBlob for OCI manifest walking
- extsloader: CAS self-heal via Puller.Pull with FilesTarget{Disks},
  pubsub ProcessChange for BaseOsStatus/ContentTreeStatus subscriptions
- types/locationconsts.go: shared Extension A/B path constants
- Roadmap doc updated with HV-agnostic Core limitation note

Eden test infrastructure:
- tests/eden/run.sh: split-rootfs test entry (test #8)
- tests/eden/prepare-split-rootfs-test.sh: build/push/setup/test script

Tested: monolith(15.11.0) -> split OTA, CAS self-heal extracts Extension,
dm-verity mount, debug/SSH container starts from Extension.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: pubsub ExtsloaderStatus + nodeagent testing window check
0b8d4e1 
- types/extsloadertypes.go: new ExtsloaderStatus type with State enum
  (Starting/Ready/Failed), published via pubsub by extsloader
- extsloader: publish ExtsloaderStatus via pubsub instead of only writing
  /run/extsloader-state.json (kept as debug aid, marked for removal)
- nodeagent: subscribe to ExtsloaderStatus from extsloader, check Extension
  ready state before marking update TestComplete in testing window
  (handletimers.go:checkExtsloaderReady). If /etc/ext-verity-roothash exists
  but Extension not ready, testing window keeps waiting → eventual reboot →
  automatic rollback
- base/logobjecttypes.go: register ExtsloaderStatusLogType

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add cryptsetup to Core, remove legacy loop mount fallback
b903581 
- pkg/pillar/Dockerfile: add cryptsetup package (provides veritysetup)
  to the runtime PKGS. Required for dm-verity verification of Extension.
- extsloader: remove legacy read-only loop mount fallback. dm-verity is
  now mandatory for Extension mounting. If veritysetup is missing or
  roothash not found, mount fails → extsloader reports failed →
  nodeagent testing window holds → rollback.
- Remove "no-verity" PCR12 measurement path since verity is always used.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add broken image builder and verity tooling
d50340d 
- tests/eden/prepare-broken-split-image.sh: builds a split-rootfs EVE
  image with corrupted ext-verity-roothash for rollback testing
- tools/make-ext-verity.sh: dm-verity hash tree generation for Extension

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: cleanup unused Extension images on partition state tran…
6f752cc 
…sitions

Add CleanupUnusedExtension() to baseosmgr/handleextension.go. Called when
a partition transitions to "unused" state:
- After successful update (handleZbootTestComplete): other slot's ext cleaned
- After uninstall (doBaseOsUninstall): other partition marked unused, ext cleaned
- After failed version check (doBaseOsActivate): partition marked unused, ext cleaned

This prevents stale Extension images from accumulating on /persist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@OhmSpectator @eriknordmark
extsloader: rescan extensions on base OS updates
73894a5
@OhmSpectator @eriknordmark
WIP split-rootfs workflow prep and extension cleanup
726990c
@OhmSpectator @claude @eriknordmark
split-rootfs: universal image (HV=uni), EFI HV persistence, extension…
dcae3c2 
… extraction rewrite

- Add HV=uni build variant and eve-uni Make target for universal images
- GRUB: add set_uni_boot with HV selection menu, EFI variable persistence,
  and cross-partition HV discovery for OTA from monolithic images
- Installer: resolve universal HV from GRUB cmdline, write to CONFIG
- onboot.sh: multi-source HV resolution (CONFIG > cmdline > /run > rootfs)
  with "uni" safety guard; device-steps.sh writes EFI var for next boot
- baseosmgr: accept -uni- OTA updates on both kvm and k devices
- Extension extraction: rewrite FindAdditionalDiskBlob → HasExtensionDisk +
  ExtractExtensionDisk using registry.Puller (handles Docker labels correctly)
- vaultmgr: replace file-polling extsloader wait with pubsub subscription
- zedagent: gate attestation on extsloader terminal state (PCR12 stability)
- nodeagent: add diagnostic logging for vault/config timer states
- manage-device-identity.sh: auto-detect swtpm in dist/<arch>/current/swtpm
- extsloader: improved logging for extension search and CAS self-heal
- types/memory.go: add ReadCgroupMemoryLimit/WriteCgroupMemoryLimit helpers
- Eden tests: fix HV-dependent paths, improve cleanup between runs
- Update bpftrace/recovertpm image hashes, update roadmap docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
WIP split-rootfs tests: HV=uni default, TEST_ONLY mode, PCR cleanup
9682f54 
- Default EVE_HV to uni (not kvm) in prepare scripts so rootfs
  identifies as universal image matching test expectations
- Add TEST_ONLY env var for running individual tests with full
  Eden redeploy (clean disk, volumes, TPM state)
- Clear stale Adam PCR enforcement before each test run
- Remove Docker volumes during redeploy to prevent stale registrations
- Wait for split v1 activation via eden.lim.test before running test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: fix HV-type detection for universal (uni) installer
6b3cc76 
Five bugs found and fixed in the HV-type resolution chain for
universal images built with HV=uni:

1. runme.sh: prepare_for_hv() bailed on "uni" — Docker image
   extraction (`docker run lfedge/eve installer_raw`) failed.
   Fix: accept "uni" with k-sized media (uni carries pre-compressed
   kube payloads that don't shrink further in the outer squashfs).

2. installer/install: cmdline check for resolved HV missed "xen" —
   selecting Xen from GRUB menu silently fell back to KVM.
   Fix: add `elif cmdline eve_hv_type=xen`.

3. rootfs.cfg: GRUB menu had no Xen entry. Fix: add Xen menuentry
   that restores the Xen boot chain (multiboot2/xen.gz on x86,
   xen_hypervisor/xen.efi on arm64) undoing set_kvm_boot overrides.

4. storage-init.sh: no guard against "uni" leaking to runtime.
   Fix: explicit fallback to kvm with warning log.

5. make-raw: read /root/etc/eve-hv-type which is "uni" in rootfs,
   so Kubevirt installs got wrong partition sizes (512MB rootfs
   instead of 4GB, 36MB EFI instead of 2GB). Fix: accept EVE_HV
   env var from installer, which passes the resolved eve_flavor.

Additionally, split the _eve_uni variable into two explicit signals:
- _eve_installer: "we are booting from installer media"
  (set only by grub_installer.cfg)
- _eve_uni: "show the HV selection menu"
  (set by set_uni_boot based on context)

This makes the installer-vs-OTA distinction explicit instead of
relying on _eve_uni as a heuristic proxy. During installer boot,
set_uni_boot always shows the menu with detected HV as default
entry. During OTA boot, it resolves silently from the other
partition.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @eriknordmark
pkgs: bump eve-alpine to 32c9b42 for all downstream
eb2ca73 
Auto-generated by ./tools/update-component-sha.sh --pkg ./pkg/alpine
after the preceding pkg/alpine Dockerfile fix produced a new clean
content hash 32c9b42 (no dirty suffix).

Covers 40 pkg/*/Dockerfile, build-tools/ver-update/Makefile, and
eve-tools/bpftrace-compiler/root/Dockerfile. Excludes conf/server,
which has an unrelated pre-existing WIP modification.
@OhmSpectator @claude @eriknordmark
split-rootfs: force HV=uni for all split targets, add run-installer-s…
dae599f 
…plit

Split rootfs is always universal — make all split-related targets
automatically delegate to HV=uni so callers don't need to remember.

Affected targets: eve-split, installer-split, split_rootfs,
core_rootfs, ext_rootfs, live-split, live-split-k, run-split.
eve-uni is now a simple alias for eve-split.

Also add run-installer-split target for QEMU testing of the split
installer (boots installer-split.raw, installs to target.img).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add Eden installer test script
e3f59fd 
Test the full USB installer flow for split-rootfs universal images:
  1. Builds installer-split.raw (auto HV=uni)
  2. Optionally injects pre-defined HV into CONFIG (simulates ZFlash)
  3. Configures Eden with custom-installer.path
  4. QEMU boots installer, installs to target disk, reboots
  5. Verifies: CONFIG HV, Extension mounted, dm-verity, PCR12, EFI var

Usage:
  ./tests/eden/test-split-installer.sh              # default KVM (60s timeout)
  INSTALL_HV=k ./tests/eden/test-split-installer.sh # pre-parametrized Kubevirt
  INSTALL_HV=xen ./tests/eden/test-split-installer.sh # pre-parametrized Xen

Covers test cases I1-I3 from the installer test plan:
  I1: Fresh install KVM (GRUB timeout default)
  I2: Fresh install Kubevirt (partition sizing, ZFS)
  I3: Fresh install Xen (boot chain)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: fix installer test script path for installer-split.raw
cbfb16c 
installer-split.raw lives at dist/<arch>/<ver>/installer-split.raw,
not inside the installer/ subdirectory. Also handle the case where
HV=uni delegation creates a different dist dir than current points to.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: fix installer test swtpm socket path too long
257ee0c 
Symlink installer-split.raw to a short path under eden/dist/ before
passing to Eden. The EVE dist directory name includes version, commit,
kernel tag, and author — far exceeding the 108-byte Unix socket limit.
Eden's StartSWTPM places the socket alongside the image file.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: disable TPM in installer test (GRUB --MORE-- pager)
7396348 
GRUB's measurefs generates TPM errors when efi.mod is not found in the
installer squashfs. The errors fill the GRUB console and trigger the
--MORE-- pager, which blocks unattended boot indefinitely.

Disable TPM for installer tests. PCR12 measurement is already covered
by the OTA test suite (update_split_pcr_enforcement_rollback).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: fix GRUB --MORE-- pager blocking installer boot
deb8c34 
rootfs.cfg sets `set pager=1` which pauses output when the console
fills up. During installer boot, TPM measurement errors (efi.mod not
found in installer squashfs) generate enough output to trigger the
--MORE-- prompt, blocking unattended installation.

Fix: set pager=0 in grub_installer.cfg. The pager is never useful
during installer boot (no one reads paged output on serial console
during automated install).

Re-enable TPM in the installer test script — PCR12 measurement is
part of the verification suite.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: use EVE-built OVMF firmware for installer test (TPM2)
893981e 
Eden's default OVMF download does not include TPM2 drivers. This causes
"Unknown TPM error" in GRUB and broken PCR measurements. The EVE-built
OVMF at dist/<ver>/installer/firmware/ includes TPM2 support.

Configure Eden to use the EVE-built OVMF_CODE.fd + OVMF_VARS.fd via
eve.firmware config key.

Also keep the grub_installer.cfg pager=0 fix as defense-in-depth —
installer boot should never block on paged output regardless of TPM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: fix installer-split missing rootfs-ext.img in squashfs
0ed171f 
The ln -sf rootfs-ext.img command created a self-referencing symlink
that overwrote the real rootfs-ext.img file produced by the ext_rootfs
build. The subsequent rm -f deleted it entirely. The installer squashfs
was missing rootfs-ext.img, causing containerd to fail with "Error
creating 001-installer: exit status 1" (bind mount source not found).

Fix: don't symlink rootfs-ext.img — it already exists as a real file.
Only rootfs.img needs a symlink (to rootfs-core.img, by linuxkit
convention). Only clean up the rootfs.img symlink, not rootfs-ext.img.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OhmSpectator and others added 18 commits June 17, 2026 15:21
@OhmSpectator @claude @eriknordmark
split-rootfs: fix Xen installer menu entry (xen.gz not in installer)
07213b8 
The installer squashfs does not include xen.gz — it's only in the
installed Core rootfs. The Xen menu entry tried to load xen.gz via
multiboot2 and failed with "file /boot/xen.gz not found".

Fix: use KVM boot chain (direct linux) for the installer, but pass
eve_hv_type=xen on cmdline. The installer writes "xen" to CONFIG.
The installed system boots with Xen on first real boot from Core
rootfs which has xen.gz.

This only affects the installer GRUB menu. Normal Xen boot (from
installed system) still uses set_xen_boot with the platform Xen
defaults.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add live-split-xen target, bump dom0_mem for universal …
3fd33cf 
…binary

Add live-split-xen and run-split-xen Make targets for testing the
Xen variant without going through the installer. Same pattern as
live-split-k: inject eve-hv-type=xen into CONFIG partition.

Bump non-k memory settings from 640M to 1024M dom0_mem. The universal
pillar binary (~143MB on disk, ~190MB RSS with THP=madvise) plus
containerd plus extension services cannot fit in 640MB. This caused
the system to thrash on Xen (where dom0_mem is enforced by the
hypervisor) and is also used as the cgroup limit on KVM.

New values: dom0=1024M/1536M, eve=800M/1200M, ctrd=400M/600M.
Previous: dom0=640M/800M, eve=520M/650M, ctrd=320M/400M.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
Revert dom0_mem bump — keep production defaults for backward compat
86da1d4 
The 640M/800M dom0_mem values are production defaults that customers
depend on. Changing them affects cgroup limits for ALL existing KVM
deployments. The Xen memory issue with the universal binary needs a
different solution (e.g., Xen-specific override or conf/grub.cfg).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add Xen-specific dom0_mem settings (1024M)
1d8878d 
On Xen, dom0_mem is enforced by the hypervisor as a physical RAM
limit for dom0. The universal pillar binary (~143MB on disk, ~190MB
RSS) plus containerd plus extension services cannot fit in the
default 640MB. The system thrashes and agents fail to start.

KVM is unaffected — dom0_mem is only a cgroup limit there, and Linux
has full physical RAM available for paging.

Xen memory tier: dom0=1024M/1536M, eve=800M/1200M, ctrd=400M/600M.
KVM unchanged:   dom0=640M/800M,   eve=520M/650M,  ctrd=320M/400M.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add run-split-k target
cc30b34 
Complete the set: run-split (KVM/menu), run-split-k, run-split-xen.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: rename run-split to run-split-kvm, add live-split-kvm
cc975d5 
All split run/live targets now explicitly name the HV variant:

  make live-split-kvm / run-split-kvm   # CONFIG=kvm, no menu wait
  make live-split-k   / run-split-k     # CONFIG=k
  make live-split-xen / run-split-xen   # CONFIG=xen
  make live-split                       # CONFIG=uni (GRUB menu, 60s)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: add run-installer-image target for custom installer images
a2d39cd 
Usage: make run-installer-image INSTALLER_IMAGE=/tmp/usb-test.img

Boots any raw installer image in QEMU (e.g. ZFlash-written loop device
or custom image). Resets OVMF_VARS and creates fresh target disk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: rewrite installer test as wrapper + escript
ce5312e 
Split the installer test into two reusable pieces:

1. Shell wrapper (test-split-installer.sh):
   - Builds installer-split.raw
   - Injects HV into CONFIG (simulates ZFlash)
   - Configures Eden with custom-installer, TPM, EVE OVMF
   - Waits for device to come online

2. Escript (testdata/verify_split_install.txt):
   - Parameterized by EXPECTED_HV env var
   - Verifies CONFIG, runtime HV, Extension, dm-verity, extsloader,
     services, EFI variable, "uni" leak check
   - Kubevirt-specific: zedkube, kube service, 2GB EFI partition

Usage:
  ./tests/eden/test-split-installer.sh              # KVM default
  INSTALL_HV=k ./tests/eden/test-split-installer.sh # Kubevirt
  INSTALL_HV=xen ./tests/eden/test-split-installer.sh # Xen

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: unify installer.img name across split and monolith
d20e357 
The distinct INSTALLER_SPLIT_{TAR,IMG} variables and the temporary
'ln -sf installer-split.img installer.img' hack existed purely to keep
the two build paths distinguishable in the same INSTALLER dir. Nothing
downstream (pkg/installer, runme.sh, OCI layout) referred to the -split
suffix.

Collapse into canonical INSTALLER_{TAR,IMG} and HV-switch the build
recipe: HV=uni uses the core+ext split path; otherwise monolithic.
Add \$(INSTALLER_IMG) to EVE_SPLIT_ARTIFACTS so the eve-split OCI ships
/bits/installer.img natively. Preserves the 'make installer-split'
target name and the 'installer-split.raw' output filename.

Side effect: 'docker run lfedge/eve:<tag>-uni installer_raw > out.raw'
now works for split builds, enabling the standard '-v confdir:/in'
pre-parametrization path (e.g., dropping eve-hv-type into CONFIG).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@OhmSpectator @claude @eriknordmark
split-rootfs: test-split-installer covers kvm/k/k-zfs/xen cases
7df70b4 
Rewrite tests/eden/test-split-installer.sh:
- Loop over 4 pre-parametrized HV cases (kvm, k, k-zfs, xen) using
  docker-run installer_raw with a per-case conf dir mounted at /in,
  writing eve-hv-type into CONFIG via the supported production path
- eve.disks=2 for k-zfs so the installer auto-selects ZFS on multi-disk
- Optional GRUB-menu case (INCLUDE_MENU_CHECK=1) with empty CONFIG,
  inverted assertion that the menu IS shown
- Post-reboot re-verification per case: ssh reboot, wait for EVE to
  come back, re-run verify escript (SKIP_REBOOT=1 to opt out)
- Session-scoped GRUB-menu log check sliced from last 'swtpm is
  starting' marker so prior-case entries don't false-positive
- Waits for ~/.eden/certs/root-certificate.pem after eden setup and
  for dist/swtpm/swtpm-sock after eden start, replacing the manual
  pkill+cert-wipe that raced with eden's own lifecycle

The verify escript lives in the eden repo at
tests/update_eve_image/testdata/verify_split_install.txt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@eriknordmark @claude
docs: reconcile split-rootfs design with code
a1b11e5 
Update SPLIT-ROOTFS-DESIGN.md to match the implemented design. The Extension is stored on the plaintext PERSIST root (not the vault) and trusted via dm-verity; extsloader runs before vault unlock and vaultmgr/zedagent wait for it so PCR 12 is finalized before seal/unseal. Flip the POC-gap table and Phase 1/2 checklists to the as-built state (erofs, dm-verity, PCR 12 implemented), and correct the boot/update flows and threat model to match. Note the one real remaining gap: the stock kernel still lacks CONFIG_EROFS_FS and CONFIG_DM_VERITY.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
docs: soften GRUB Extension-measurement claims
80baeb7 
The future-work section and the EXTA/EXTB option note called a GRUB-based Extension measurement "stronger" / the "cleanest security model" and treated dm-verity verification as "redundant" once GRUB measures. That overstates it: dm-verity already anchors the Extension to the Core's measured root hash (PCR 13) and verifies every block at runtime, which a one-time GRUB measurement does not. Reframe GRUB measurement consistently as cleaner (firmware-rooted, removes the seal/attest ordering dependency on extsloader) rather than stronger, and note dm-verity remains the runtime integrity mechanism.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
split-rootfs: move wwan from Extension to Core
e90a9c9 
Cellular connectivity should not hinge on the Extension image having
been loaded and dm-verity verified. Moving the wwan service into the
bootable Core makes the modem usable as soon as the Core boots,
restoring the original split-rootfs design intent and matching how
wwan behaves in the monolithic image.

The Core has ample room for it: a measured build puts the universal
Core at 224 MB and adds ~21 MB for wwan, still ~55 MB under the 300 MB
Core partition budget. wwan is removed from the Extension so only one
instance manages the modem.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
split-rootfs: enforce Core and OCI size limits
f6cf6bd 
The bootable Core squashfs must fit the legacy 300 MB IMGA/IMGB
partition, but the existing size guard compared it against
ROOTFS_MAXSIZE_MB, which is 4096 MB for k and universal builds. So a
Core that overran the real partition limit would have passed unnoticed,
and nothing bounded the combined deliverable.

Add a 300 MB cap keyed on the Core image and a 4096 MB cap on the
universal/split OCI image (Core plus the disk-0 Extension layer), both
overridable for platform variants. The current build is well within
both.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
docs: incorporate split-rootfs session findings
0a138f5 
Correct the kernel-config prerequisites: DM_VERITY is verified absent from
eve-core_defconfig (the device-mapper base is present, but DM_VERITY itself
is not enabled), not "likely present", and the EROFS gap is now confirmed at
runtime on a booted split image rather than inferred. Record that wwan is
implemented in the Core templates with its measured cost, and document that
the build now enforces the 300MB Core and 4096MB OCI-image size limits.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
docs: record kernel-config fix verified end-to-end
98c9fe3 
Add the CONFIG_EROFS_FS_ZIP=y prerequisite (required because the Extension
is built with mkfs.erofs -zlz4hc,9, so base EROFS_FS cannot mount it), and
mark the EROFS/DM_VERITY kernel gap as enabled and verified end-to-end: with
the three options set in a custom eve-kernel, a booted split image mounts the
dm-verity erofs Extension, onboards, and runs an app instance. The configs are
not yet upstreamed, so stock builds still lack them.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
split-rootfs: add OTA-from-old-image test
21b4310 
Add a draft Eden test that updates a device originally installed by a
pre-10.2.0 EVE (300 MB IMGA/IMGB) to the universal/split image, asserting
the legacy partition layout up front and that the uni Core fits and boots
in the 300 MB slot after OTA. Document the upstream prerequisite in the
roadmap: doBaseOsActivate compared the whole content-tree size (Core plus
the disk-0 Extension layer) against the rootfs partition and wrongly
rejected split images; lf-edge#6044 removes that pre-check and bounds
the bytes actually written instead.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
split-rootfs: fix Dockerfile FROM hashes after rebase
d9c60e7 
Repoint eve-* base-image FROM hashes that the split-rootfs commits
carried from their pre-rebase base to the values the current master
tree actually builds, restoring make check-docker-hashes-consistency.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark @claude
docs: add split-rootfs remaining-work tracker
17f0485 
Add a branch-scoped tracker that consolidates the open items for the
split-rootfs feature: the two hard blockers for a stock build (kernel
EROFS/DM_VERITY config and the OTA partition-size check), the testing
gaps (no unit tests, eden scripts not in CI), cleanup loose ends, and
the larger productionization work from the roadmap.

Also correct stale claims in the design and roadmap docs to match the
settled decisions: eve-wwan lives in Core (not the Extension) so
cellular-only devices stay manageable if the Extension fails, and the
Extension lives on plain PERSIST protected by dm-verity, so the threat
model no longer references vault encryption for it.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewd-zededa andrewd-zededa Awaiting requested review from andrewd-zededa
@christoph-zededa christoph-zededa Awaiting requested review from christoph-zededa
@europaul europaul Awaiting requested review from europaul
@jsfakian jsfakian Awaiting requested review from jsfakian
@milan-zededa milan-zededa Awaiting requested review from milan-zededa milan-zededa is a code owner
@naiming-zededa naiming-zededa Awaiting requested review from naiming-zededa
@OhmSpectator OhmSpectator Awaiting requested review from OhmSpectator
@rene rene Awaiting requested review from rene rene is a code owner
@rucoder rucoder Awaiting requested review from rucoder rucoder is a code owner
@shjala shjala Awaiting requested review from shjala shjala is a code owner
@zedi-pramodh zedi-pramodh Awaiting requested review from zedi-pramodh zedi-pramodh is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@eriknordmark @OhmSpectator