[STACKED on #1441] mm: per-NUMA-node L2 page pools#1445
Conversation
|
Follow-up commit 425c77c on this branch closes the emulated-NUMA node-fallback that the earlier version had (real 2-socket hardware was already 32/32 on both nodes; this fixes the emulated case and any config where the boot allocator coalesces across node boundaries). Root cause: Fix: Emulated QEMU NUMA: 2-node node-1 went 0/32→32/32; 4-node all four nodes 32/32. Single-node regression + tst-mmap still pass. |
OSv had no notion of NUMA: the scheduler and allocator treat memory as flat, which leaves performance on the table on multi-socket bare-metal (the large EC2/GCP metal instances that are a target for Postgres-over-ZFS). This is the first, discovery-only step. It adds a numa:: module that parses the ACPI SRAT (System Resource Affinity Table) and SLIT (System Locality Distance Information Table) at boot, right after acpi::init(), and exposes the topology: - numa::nr_nodes() / numa::available() - numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the APIC ids the MADT parse recorded on each sched::cpu - numa::distance(from, to) -- from SLIT (10 == local per ACPI convention), defaulting to 10 local / 20 remote when no SLIT is present - numa::memory_ranges() -- physical ranges tagged with their node It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity subtables, and only trusts SLIT if its locality count matches the node count SRAT reported. On a machine with no SRAT (the common single-node VM) it reports one flat node and available() == false; nothing changes behavior. This intentionally does NOT yet change allocation or scheduling; it only makes the topology available so a node-aware allocator, scheduler affinity, and mbind/get_mempolicy can build on it. Add tests/tst-numa.cc validating the invariants (>= 1 node, every CPU maps in range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges, available).
With NUMA topology now discovered (numa:: module), make the topology-query
syscalls report it instead of a hardcoded single node:
- sys_getcpu() now fills the node-out argument with numa::node_of_cpu() for the
calling CPU rather than always 0.
- get_mempolicy():
- MPOL_F_NODE returns the calling CPU's node.
- MPOL_F_NODE | MPOL_F_ADDR returns the node backing the given address,
resolved via a page-table walk (virt_to_phys_pt) and numa::node_of_phys().
- the allowed-nodes mask now sets a bit for every discovered node (not just
node 0), and rejects a maxnode smaller than the node count with EINVAL.
set_mempolicy() stays a no-op: the topology is known but the physical allocator
is not yet node-aware, so a placement policy cannot be enforced. Its comment is
updated to say so; enforcement will come with the node-aware allocator.
Adds numa::node_of_phys() to map a physical address to its node.
On a machine with no SRAT everything degrades to the previous single-node-0
behavior.
Add tests/tst-numa-mempolicy.cc checking getcpu's node is in range and matches
numa::node_of_cpu, get_mempolicy's MPOL_F_NODE and allowed-mask (bit count ==
node count, maxnode-too-small EINVAL), and the MPOL_F_ADDR path. Verified on OSv
under KVM single-node and with a QEMU 2-node -numa config.
Depends on the "numa: discover NUMA topology from ACPI SRAT/SLIT" change.
First incremental step of a node-aware allocator, on top of the NUMA topology discovery from the earlier commits. Adds memory::alloc_page_on_node(node), which prefers a page whose physical memory lies on the requested NUMA node, falling back to a normal allocation when that node has no free memory in the global page-range allocator (or when NUMA is not available / the node is out of range). Implementation layers on top of the existing allocator without restructuring it: page_range_allocator::alloc_page_from_node() walks the free lists, finds a range whose physical address resolves (via numa::node_of_phys()) to the requested node, and carves one page from it the same way alloc() does; if none is found it returns nullptr and alloc_page_on_node() falls back. This is deliberately a best-effort hint, not full per-node pools: OSv drains much of physical memory into per-CPU L1/L2 pools that are not yet node-partitioned, so once a node's global ranges are exhausted the allocator falls back. It gives correct placement while node-local global memory is available and never fails or misplaces a page. Full per-node L1/L2 pools are future work (roadmap B2.2-full); the scheduler affinity and mbind/set_mempolicy enforcement build on this hint. Add tests/tst-numa-alloc.cc: basic allocation and writability, out-of-range and negative node fall back cleanly, and (with QEMU -numa) node-0 allocations land on node 0 while other nodes fall back cleanly with no crash or misplacement. Passes on OSv under KVM single-node and with a 2-node -numa config; tst-mmap (the shared allocator path) unaffected. Depends on the NUMA topology-discovery and getcpu/get_mempolicy changes.
Extends the node-preferring allocator (alloc_page_on_node) into the hot page pool path. Previously a single global L2 pool fed every CPU's per-CPU L1 cache and refilled from the global free_page_ranges with no node awareness, so on a multi-socket machine a CPU routinely ran on pages from a remote node. Now, when the system has more than one NUMA node, each node gets its own L2 pool whose refill() prefers pages physically local to that node (alloc_page_from_node), and each CPU draws from its own node's L2 pool via get_l2() (keyed on numa::node_of_cpu). A drained node falls back to any page so allocation always makes progress. Single-node / non-NUMA systems keep the original single global_l2 with zero behavior change. Verified: single-node boot unchanged (fallback path); 2-node QEMU NUMA boot brings up per-node pools, node-local allocations stay local and a drained node falls back cleanly. Known limitation (separate follow-up): OSv's early memory init concentrates free_page_ranges on the lower node(s), so higher nodes can fall back until boot-time physical range distribution across nodes is addressed. The per-node pools are the necessary allocator-side half of that work.
Under NUMA, OSv's boot allocator coalesces physically adjacent free memory into one range whose page_range header sits at its base. Because node N and node N+1 are typically adjacent in physical space, a single free range can begin on a low node and extend across the boundary into a higher node. alloc_page_from_node() classified a range only by its base address, so a higher node's memory - living inside a range based on a lower node - was invisible, and every allocation for that node fell back to node-agnostic memory. Observed under QEMU emulated 2-node NUMA (512M+512M): at test time the whole free pool was one range phys=0x160f000 size~1004M based on node 0; node 1 allocations fell back 32/32. Fix: for each free range, find the first page-aligned offset that lies on the requested node (intersecting the SRAT memory ranges directly), then carve there, splitting the range into up-to-three free pieces (prefix / carved page / suffix). When the base is already on-node this reduces to the previous front-carve. No behavior change for single-node systems, where alloc_page_from_node is never reached (nr_nodes<=1 keeps the single global_l2 pool). Emulated NUMA after: node 0 and node 1 both 32/32 on-node; 4-node config places 32/32 on all four nodes. Single-node regression still passes; tst-mmap still passes.
4429039 to
3d6687e
Compare
Note
Stacked on #1441 (node-preferring page allocation). This branch builds on
alloc_page_on_node/alloc_page_from_nodefrom #1441, so the diff iscumulative until #1441 merges; then it reduces to the single
core/mempool.cccommit. Review #1441 first.
Extends the node-preferring allocator into the hot page-pool path. Previously a
single global L2 pool fed every CPU's per-CPU L1 cache and refilled from the
global
free_page_rangeswith no node awareness, so on a multi-socket machinea CPU routinely ran on pages from a remote NUMA node.
Now, when the system has more than one NUMA node, each node gets its own L2 pool
whose
refill()prefers pages physically local to that node(
alloc_page_from_node), and each CPU draws from its own node's L2 viaget_l2()(keyed onnuma::node_of_cpu). A drained node falls back to any pageso allocation always makes progress. Single-node / non-NUMA systems keep the
original single
global_l2with zero behavior change.Validation
tst-numa-allocpasses.cores), OSv under KVM with two NUMA nodes bound to the real host nodes:
because emulated memory is concentrated on node 0 — an emulation artifact;
real hardware with genuinely node-distributed memory allocates locally.
tst-numa-allocreports per-node on-node/fell-back counts as a regressionsignal.
Known follow-up (separate): OSv's early memory init can still concentrate
free_page_rangeson lower nodes on some configs; this PR is the allocator-sidehalf and is correct on real hardware. Boot-time physical-range balancing across
nodes would be a further step if a workload needs it.