On 2022-02-15 16:47, Jason Gunthorpe wrote:
On Tue, Feb 15, 2022 at 04:35:56PM -0500, Felix Kuehling wrote:
On 2022-02-15 14:41, Jason Gunthorpe wrote:
On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote:
On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote:
Do you know if DEVICE_GENERIC pages would end up as PageAnon()? My assumption was that they would be part of a special mapping.
We need to stop using the special PTEs and VMAs for things that have a struct page. This is a mistake DAX created that must be undone.
Yes, we'll get to it. Maybe we can do it for the non-DAX devmap ptes first given that DAX is more complicated.
Probably, I think we can check the page->pgmap type to tell the difference.
I'm not sure how the DEVICE_GENERIC can work without this, as DAX was made safe by using the unmap_mapping_range(), which won't work here. Is there some other trick being used to keep track of references inside the AMD driver?
Not sure I'm following all the discussion about VMAs and DAX. So I may be answering the wrong question: We treat each ZONE_DEVICE page as a reference to the BO (buffer object) that backs the page. We increment the BO refcount for each page we migrate into it. In the dev_pagemap_ops.page_free callback we drop that reference. Once all pages backed by a BO are freed, the BO refcount reaches 0 [*] and we can free the BO allocation.
Userspace does
- mmap(MAP_PRIVATE) to allocate anon memory
- something to trigger migration to install a ZONE_DEVICE page
- munmap()
Who decrements the refcout on the munmap?
When a ZONE_DEVICE page is installed in the PTE is supposed to be marked as pte_devmap and that disables all the normal page refcounting during munmap().
fsdax makes this work by working the refcounts backwards, the page is refcounted while it exists in the driver, when the driver decides to remove it then unmap_mapping_range() is called to purge it from all PTEs and then refcount is decrd. munmap/fork/etc don't change the refcount.
Hmm, that just means, whether or not there are PTEs doesn't really matter. It should work the same as it does for DEVICE_PRIVATE pages. I'm not sure where DEVICE_PRIVATE page's refcounts are decremented on unmap, TBH. But I can't find it in our driver, or in the test_hmm driver for that matter.
Regards, Felix
Jason