mm: add node-preferring page allocation (alloc_page_on_node)#1441
Open
gburd wants to merge 3 commits into
Open
mm: add node-preferring page allocation (alloc_page_on_node)#1441gburd wants to merge 3 commits into
gburd wants to merge 3 commits into
Conversation
This was referenced Jul 13, 2026
OSv had no notion of NUMA: the scheduler and allocator treat memory as flat, which leaves performance on the table on multi-socket bare-metal (the large EC2/GCP metal instances that are a target for Postgres-over-ZFS). This is the first, discovery-only step. It adds a numa:: module that parses the ACPI SRAT (System Resource Affinity Table) and SLIT (System Locality Distance Information Table) at boot, right after acpi::init(), and exposes the topology: - numa::nr_nodes() / numa::available() - numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the APIC ids the MADT parse recorded on each sched::cpu - numa::distance(from, to) -- from SLIT (10 == local per ACPI convention), defaulting to 10 local / 20 remote when no SLIT is present - numa::memory_ranges() -- physical ranges tagged with their node It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity subtables, and only trusts SLIT if its locality count matches the node count SRAT reported. On a machine with no SRAT (the common single-node VM) it reports one flat node and available() == false; nothing changes behavior. This intentionally does NOT yet change allocation or scheduling; it only makes the topology available so a node-aware allocator, scheduler affinity, and mbind/get_mempolicy can build on it. Add tests/tst-numa.cc validating the invariants (>= 1 node, every CPU maps in range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges, available).
With NUMA topology now discovered (numa:: module), make the topology-query
syscalls report it instead of a hardcoded single node:
- sys_getcpu() now fills the node-out argument with numa::node_of_cpu() for the
calling CPU rather than always 0.
- get_mempolicy():
- MPOL_F_NODE returns the calling CPU's node.
- MPOL_F_NODE | MPOL_F_ADDR returns the node backing the given address,
resolved via a page-table walk (virt_to_phys_pt) and numa::node_of_phys().
- the allowed-nodes mask now sets a bit for every discovered node (not just
node 0), and rejects a maxnode smaller than the node count with EINVAL.
set_mempolicy() stays a no-op: the topology is known but the physical allocator
is not yet node-aware, so a placement policy cannot be enforced. Its comment is
updated to say so; enforcement will come with the node-aware allocator.
Adds numa::node_of_phys() to map a physical address to its node.
On a machine with no SRAT everything degrades to the previous single-node-0
behavior.
Add tests/tst-numa-mempolicy.cc checking getcpu's node is in range and matches
numa::node_of_cpu, get_mempolicy's MPOL_F_NODE and allowed-mask (bit count ==
node count, maxnode-too-small EINVAL), and the MPOL_F_ADDR path. Verified on OSv
under KVM single-node and with a QEMU 2-node -numa config.
Depends on the "numa: discover NUMA topology from ACPI SRAT/SLIT" change.
First incremental step of a node-aware allocator, on top of the NUMA topology discovery from the earlier commits. Adds memory::alloc_page_on_node(node), which prefers a page whose physical memory lies on the requested NUMA node, falling back to a normal allocation when that node has no free memory in the global page-range allocator (or when NUMA is not available / the node is out of range). Implementation layers on top of the existing allocator without restructuring it: page_range_allocator::alloc_page_from_node() walks the free lists, finds a range whose physical address resolves (via numa::node_of_phys()) to the requested node, and carves one page from it the same way alloc() does; if none is found it returns nullptr and alloc_page_on_node() falls back. This is deliberately a best-effort hint, not full per-node pools: OSv drains much of physical memory into per-CPU L1/L2 pools that are not yet node-partitioned, so once a node's global ranges are exhausted the allocator falls back. It gives correct placement while node-local global memory is available and never fails or misplaces a page. Full per-node L1/L2 pools are future work (roadmap B2.2-full); the scheduler affinity and mbind/set_mempolicy enforcement build on this hint. Add tests/tst-numa-alloc.cc: basic allocation and writability, out-of-range and negative node fall back cleanly, and (with QEMU -numa) node-0 allocations land on node 0 while other nodes fall back cleanly with no crash or misplacement. Passes on OSv under KVM single-node and with a 2-node -numa config; tst-mmap (the shared allocator path) unaffected. Depends on the NUMA topology-discovery and getcpu/get_mempolicy changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First incremental step of a node-aware allocator, on top of the NUMA topology
discovery (#1418) and getcpu/get_mempolicy (#1419) work. Adds
memory::alloc_page_on_node(node), which prefers a page whose physical memorylies on the requested NUMA node, falling back to a normal allocation when that
node has no free memory in the global page-range allocator (or when NUMA is
unavailable / the node is out of range).
How
Layers on top of the existing allocator without restructuring it:
page_range_allocator::alloc_page_from_node()walks the free lists, finds arange whose physical address resolves (via
numa::node_of_phys()) to therequested node, and carves one page the same way
alloc()does; if none isfound it returns
nullptrandalloc_page_on_node()falls back.This is deliberately a best-effort hint, not full per-node pools: OSv drains
much of physical memory into per-CPU L1/L2 pools that are not yet
node-partitioned, so once a node's global ranges are exhausted the allocator
falls back. It gives correct placement while node-local global memory is
available and never fails or misplaces a page. Full per-node L1/L2 pools are
future work; the scheduler affinity and
mbind/set_mempolicyenforcementbuild on this hint.
Testing
tests/tst-numa-alloc.cc: basic allocation and writability, out-of-range andnegative node fall back cleanly, and (with QEMU
-numa) node-0 allocations landon node 0 while other nodes fall back cleanly with no crash or misplacement.
Passes on OSv under KVM single-node and with a 2-node
-numaconfig;tst-mmap(the shared allocator path) unaffected.
Note
Depends on #1418 and #1419 (this branch stacks on them); until they merge the
diff shows their commits too.
(Recreated from #1430, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)