Skip to content

[STACKED on #1441] mm: per-NUMA-node L2 page pools#1445

Draft
gburd wants to merge 6 commits into
cloudius-systems:masterfrom
gburd:pr/numa-per-node-pools
Draft

[STACKED on #1441] mm: per-NUMA-node L2 page pools#1445
gburd wants to merge 6 commits into
cloudius-systems:masterfrom
gburd:pr/numa-per-node-pools

Conversation

@gburd

@gburd gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Note

Stacked on #1441 (node-preferring page allocation). This branch builds on
alloc_page_on_node/alloc_page_from_node from #1441, so the diff is
cumulative until #1441 merges; then it reduces to the single core/mempool.cc
commit. Review #1441 first.

Extends the node-preferring allocator into the hot page-pool path. Previously a
single global L2 pool fed every CPU's per-CPU L1 cache and refilled from the
global free_page_ranges with no node awareness, so on a multi-socket machine
a CPU routinely ran on pages from a remote NUMA node.

Now, when the system has more than one NUMA node, each node gets its own L2 pool
whose refill() prefers pages physically local to that node
(alloc_page_from_node), and each CPU draws from its own node's L2 via
get_l2() (keyed on numa::node_of_cpu). A drained node falls back to any page
so allocation always makes progress. Single-node / non-NUMA systems keep the
original single global_l2 with zero behavior change.

Validation

  • Single-node boot: unchanged (fallback path), tst-numa-alloc passes.
  • Real 2-socket NUMA metal (AWS m5.metal, node0 193 GB / node1 193 GB, 96
    cores)
    , OSv under KVM with two NUMA nodes bound to the real host nodes:
    NUMA: 2 node(s), 8 CPU(s) mapped, 4 memory range(s)
    node 0: 32 on-node, 0 fell back
    node 1: 32 on-node, 0 fell back
    
    100% node-local allocation on both nodes.
  • QEMU-emulated NUMA (no host-node binding) shows a higher node falling back,
    because emulated memory is concentrated on node 0 — an emulation artifact;
    real hardware with genuinely node-distributed memory allocates locally.

tst-numa-alloc reports per-node on-node/fell-back counts as a regression
signal.

Known follow-up (separate): OSv's early memory init can still concentrate
free_page_ranges on lower nodes on some configs; this PR is the allocator-side
half and is correct on real hardware. Boot-time physical-range balancing across
nodes would be a further step if a workload needs it.

@gburd

gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up commit 425c77c on this branch closes the emulated-NUMA node-fallback that the earlier version had (real 2-socket hardware was already 32/32 on both nodes; this fixes the emulated case and any config where the boot allocator coalesces across node boundaries).

Root cause: arch_setup_free_memory() frees all E820 ranges into free_page_ranges at .init priority mempool, before acpi::init()/numa::init() run — so NUMA topology is unknown when memory is registered. The page-range allocator then coalesces physically-adjacent free memory, and since node N and N+1 are adjacent in physical space the whole machine's RAM collapses into essentially one giant free range whose page_range header sits at its low base. alloc_page_from_node() classified a range solely by node_of_phys(base), so a higher node's memory living inside a low-node-based range was invisible and every allocation for it fell back. It was never a draining problem (L2 caps at ~8 MB/node) — it was range classification.

Fix: alloc_page_from_node is now boundary-aware — for each free range it finds the first page-aligned offset intersecting the requested node's SRAT range and carves there, splitting into up-to-three pieces (prefix / carved page / suffix), mirroring the existing alloc_aligned 3-way split. Base-on-node reduces to the old front-carve. Only mutates after the scan completes (no iterator invalidation); single-node is unaffected (alloc_page_from_node is unreachable when nr_nodes<=1).

Emulated QEMU NUMA: 2-node node-1 went 0/32→32/32; 4-node all four nodes 32/32. Single-node regression + tst-mmap still pass.

gburd added 6 commits July 15, 2026 10:10
OSv had no notion of NUMA: the scheduler and allocator treat memory as flat,
which leaves performance on the table on multi-socket bare-metal (the large
EC2/GCP metal instances that are a target for Postgres-over-ZFS).

This is the first, discovery-only step. It adds a numa:: module that parses the
ACPI SRAT (System Resource Affinity Table) and SLIT (System Locality Distance
Information Table) at boot, right after acpi::init(), and exposes the topology:

- numa::nr_nodes() / numa::available()
- numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the
  APIC ids the MADT parse recorded on each sched::cpu
- numa::distance(from, to) -- from SLIT (10 == local per ACPI convention),
  defaulting to 10 local / 20 remote when no SLIT is present
- numa::memory_ranges() -- physical ranges tagged with their node

It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity
subtables, and only trusts SLIT if its locality count matches the node count
SRAT reported. On a machine with no SRAT (the common single-node VM) it reports
one flat node and available() == false; nothing changes behavior.

This intentionally does NOT yet change allocation or scheduling; it only makes
the topology available so a node-aware allocator, scheduler affinity, and
mbind/get_mempolicy can build on it.

Add tests/tst-numa.cc validating the invariants (>= 1 node, every CPU maps in
range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid
nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and
with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges, available).
With NUMA topology now discovered (numa:: module), make the topology-query
syscalls report it instead of a hardcoded single node:

- sys_getcpu() now fills the node-out argument with numa::node_of_cpu() for the
  calling CPU rather than always 0.
- get_mempolicy():
  - MPOL_F_NODE returns the calling CPU's node.
  - MPOL_F_NODE | MPOL_F_ADDR returns the node backing the given address,
    resolved via a page-table walk (virt_to_phys_pt) and numa::node_of_phys().
  - the allowed-nodes mask now sets a bit for every discovered node (not just
    node 0), and rejects a maxnode smaller than the node count with EINVAL.

set_mempolicy() stays a no-op: the topology is known but the physical allocator
is not yet node-aware, so a placement policy cannot be enforced. Its comment is
updated to say so; enforcement will come with the node-aware allocator.

Adds numa::node_of_phys() to map a physical address to its node.

On a machine with no SRAT everything degrades to the previous single-node-0
behavior.

Add tests/tst-numa-mempolicy.cc checking getcpu's node is in range and matches
numa::node_of_cpu, get_mempolicy's MPOL_F_NODE and allowed-mask (bit count ==
node count, maxnode-too-small EINVAL), and the MPOL_F_ADDR path. Verified on OSv
under KVM single-node and with a QEMU 2-node -numa config.

Depends on the "numa: discover NUMA topology from ACPI SRAT/SLIT" change.
First incremental step of a node-aware allocator, on top of the NUMA topology
discovery from the earlier commits.  Adds memory::alloc_page_on_node(node),
which prefers a page whose physical memory lies on the requested NUMA node,
falling back to a normal allocation when that node has no free memory in the
global page-range allocator (or when NUMA is not available / the node is out of
range).

Implementation layers on top of the existing allocator without restructuring
it: page_range_allocator::alloc_page_from_node() walks the free lists, finds a
range whose physical address resolves (via numa::node_of_phys()) to the
requested node, and carves one page from it the same way alloc() does; if none
is found it returns nullptr and alloc_page_on_node() falls back.

This is deliberately a best-effort hint, not full per-node pools: OSv drains
much of physical memory into per-CPU L1/L2 pools that are not yet
node-partitioned, so once a node's global ranges are exhausted the allocator
falls back.  It gives correct placement while node-local global memory is
available and never fails or misplaces a page.  Full per-node L1/L2 pools are
future work (roadmap B2.2-full); the scheduler affinity and mbind/set_mempolicy
enforcement build on this hint.

Add tests/tst-numa-alloc.cc: basic allocation and writability, out-of-range and
negative node fall back cleanly, and (with QEMU -numa) node-0 allocations land
on node 0 while other nodes fall back cleanly with no crash or misplacement.
Passes on OSv under KVM single-node and with a 2-node -numa config; tst-mmap
(the shared allocator path) unaffected.

Depends on the NUMA topology-discovery and getcpu/get_mempolicy changes.
Extends the node-preferring allocator (alloc_page_on_node) into the hot page
pool path. Previously a single global L2 pool fed every CPU's per-CPU L1 cache
and refilled from the global free_page_ranges with no node awareness, so on a
multi-socket machine a CPU routinely ran on pages from a remote node.

Now, when the system has more than one NUMA node, each node gets its own L2
pool whose refill() prefers pages physically local to that node
(alloc_page_from_node), and each CPU draws from its own node's L2 pool via
get_l2() (keyed on numa::node_of_cpu). A drained node falls back to any page so
allocation always makes progress. Single-node / non-NUMA systems keep the
original single global_l2 with zero behavior change.

Verified: single-node boot unchanged (fallback path); 2-node QEMU NUMA boot
brings up per-node pools, node-local allocations stay local and a drained node
falls back cleanly.

Known limitation (separate follow-up): OSv's early memory init concentrates
free_page_ranges on the lower node(s), so higher nodes can fall back until
boot-time physical range distribution across nodes is addressed. The per-node
pools are the necessary allocator-side half of that work.
Under NUMA, OSv's boot allocator coalesces physically adjacent free
memory into one range whose page_range header sits at its base. Because
node N and node N+1 are typically adjacent in physical space, a single
free range can begin on a low node and extend across the boundary into a
higher node. alloc_page_from_node() classified a range only by its base
address, so a higher node's memory - living inside a range based on a
lower node - was invisible, and every allocation for that node fell back
to node-agnostic memory.

Observed under QEMU emulated 2-node NUMA (512M+512M): at test time the
whole free pool was one range phys=0x160f000 size~1004M based on node 0;
node 1 allocations fell back 32/32.

Fix: for each free range, find the first page-aligned offset that lies on
the requested node (intersecting the SRAT memory ranges directly), then
carve there, splitting the range into up-to-three free pieces (prefix /
carved page / suffix). When the base is already on-node this reduces to
the previous front-carve. No behavior change for single-node systems,
where alloc_page_from_node is never reached (nr_nodes<=1 keeps the single
global_l2 pool).

Emulated NUMA after: node 0 and node 1 both 32/32 on-node; 4-node config
places 32/32 on all four nodes. Single-node regression still passes;
tst-mmap still passes.
@gburd
gburd force-pushed the pr/numa-per-node-pools branch from 4429039 to 3d6687e Compare July 15, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant