Am 2021-09-02 um 4:18 a.m. schrieb Christoph Hellwig:
On Wed, Sep 01, 2021 at 11:40:43AM -0400, Felix Kuehling wrote:
It looks like I'm totally misunderstanding what you are adding here then. Why do we need any special treatment at all for memory that has normal struct pages and is part of the direct kernel map?
The pages are like normal memory for purposes of mapping them in CPU page tables and for coherent access from the CPU.
That's the user page tables. What about the kernel direct map? If there is a normal kernel struct page backing there really should be no need for the pgmap.
I'm not sure. The physical address ranges are in the UEFI system address map as special-purpose memory. Does Linux create the struct pages and kernel direct map for that without a pgmap call? I didn't see that last time I went digging through that code.
So doing some googling finds a patch from Dan that claims to hand EFI special purpose memory to the device dax driver. But when I try to follow the version that got merged it looks it is treated simply as an MMIO region to be claimed by drivers, which would not get a struct page.
Dan, did I misunderstand how E820_TYPE_SOFT_RESERVED works?
From an application perspective, we want file-backed and anonymous mappings to be able to use DEVICE_PUBLIC pages with coherent CPU access. The goal is to optimize performance for GPU heavy workloads while minimizing the need to migrate data back-and-forth between system memory and device memory.
I don't really understand that part. file backed pages are always allocated by the file system using the pagecache helpers, that is using the page allocator. Anonymouns memory also always comes from the page allocator.
I'm coming at this from my experience with DEVICE_PRIVATE. Both anonymous and file-backed pages should be migrateable to DEVICE_PRIVATE memory by the migrate_vma_* helpers for more efficient access by our GPU. (*) It's part of the basic premise of HMM as I understand it. I would expect the same thing to work for DEVICE_PUBLIC memory.
Ok, so you want to migrate to and from them. Not use DEVICE_PUBLIC for the actual page cache pages. That maks a lot more sense.
I see DEVICE_PUBLIC as an improved version of DEVICE_PRIVATE that allows the CPU to map the device memory coherently to minimize the need for migrations when CPU and GPU access the same memory concurrently or alternatingly. But we're not going as far as putting that memory entirely under the management of the Linux memory manager and VM subsystem. Our (and HPE's) system architects decided that this memory is not suitable to be used like regular NUMA system memory by the Linux memory manager.
So yes. It is a Memory Mapped I/O region, which unlike the PCIe BARs that people typically deal with is fully cache coherent. I think this does make more sense as a description.
But to go back to what start this discussion: If these are memory mapped I/O pfn_valid should generally not return true for them.
As I understand it, pfn_valid should be true for any pfn that's part of the kernel's physical memory map, i.e. is returned by page_to_pfn or works with pfn_to_page. Both the hmm_range_fault and the migrate_vma_* APIs use pfns to refer to regular system memory and ZONE_DEVICE pages (even DEVICE_PRIVATE). Therefore I believe pfn_valid should be true for ZONE_DEVICE pages as well.
Regards, Felix
And as you already pointed out in reply to Alex we need to tighten the selection criteria one way or another.