On Tue, 12 Oct 2021 12:12:35 -0500 Alex Sierra alex.sierra@amd.com wrote:
This patch series introduces MEMORY_DEVICE_COHERENT, a type of memory owned by a device that can be mapped into CPU page tables like MEMORY_DEVICE_GENERIC and can also be migrated like MEMORY_DEVICE_PRIVATE. With MEMORY_DEVICE_COHERENT, we isolate the new memory type from other subsystems as far as possible, though there are some small changes to other subsystems such as filesystem DAX, to handle the new memory type appropriately.
We use ZONE_DEVICE for this instead of NUMA so that the amdgpu allocator can manage it without conflicting with core mm for non-unified memory use cases.
How it works: The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver registers the memory with devmap as MEMORY_DEVICE_COHERENT using devm_memremap_pages.
The initial user for this hardware page migration capability will be the Frontier supercomputer project.
To what other uses will this infrastructure be put?
Because I must ask: if this feature is for one single computer which presumably has a custom kernel, why add it to mainline Linux?
Our nodes in the lab have .5 TB of system memory plus 256 GB of device memory split across 4 GPUs, all in the same coherent address space. Page migration is expected to improve application efficiency significantly. We will report empirical results as they become available.
This includes patches originally by Ralph Campbell to change ZONE_DEVICE reference counting as requested in previous reviews of this patch series (see https://patchwork.freedesktop.org/series/90706/). We extended hmm_test to cover migration of MEMORY_DEVICE_COHERENT. This patch set builds on HMM and our SVM memory manager already merged in 5.14. We would like to complete review and merge this migration patchset for 5.16.