On Tue, Jan 29, 2019 at 07:08:06PM -0500, Jerome Glisse wrote:
On Tue, Jan 29, 2019 at 11:02:25PM +0000, Jason Gunthorpe wrote:
On Tue, Jan 29, 2019 at 03:44:00PM -0500, Jerome Glisse wrote:
But this API doesn't seem to offer any control - I thought that control was all coming from the mm/hmm notifiers triggering p2p_unmaps?
The control is within the driver implementation of those callbacks.
Seems like what you mean by control is 'the exporter gets to choose the physical address at the instant of map' - which seems reasonable for GPU.
will only allow p2p map to succeed for objects that have been tagged by the userspace in some way ie the userspace application is in control of what can be map to peer device.
I would have thought this means the VMA for the object is created without the map/unmap ops? Or are GPU objects and VMAs unrelated?
GPU object and VMA are unrelated in all open source GPU driver i am somewhat familiar with (AMD, Intel, NVidia). You can create a GPU object and never map it (and thus never have it associated with a vma) and in fact this is very common. For graphic you usualy only have hand full of the hundreds of GPU object your application have mapped.
I mean the other way does every VMA with a p2p_map/unmap point to exactly one GPU object?
ie I'm surprised you say that p2p_map needs to have policy, I would have though the policy is applied when the VMA is created (ie objects that are not for p2p do not have p2p_map set), and even for GPU p2p_map should really only have to do with window allocation and pure 'can I even do p2p' type functionality.
Idea is that we can only ask exporter to be predictable and still allow them to fail if things are really going bad.
I think hot unplug / PCI error recovery is one of the 'really going bad' cases..
I think i put it in the comment above the ops but in any cases i should write something in documentation with example and thorough guideline. Note that there won't be any mmu notifier to mmap of a device file unless the device driver calls for it or there is a syscall like munmap or mremap or mprotect well any syscall that work on vma.
This is something we might need to explore, does calling zap_vma_ptes() invoke enough notifiers that a MMU notifiers or HMM mirror consumer will release any p2p maps on that VMA?
If we ever want to support full pin then we might have to add a flag so that GPU driver can refuse an importer that wants things pin forever.
This would become interesting for VFIO and RDMA at least - I don't think VFIO has anything like SVA so it would want to import a p2p_map and indicate that it will not respond to MMU notifiers.
GPU can refuse, but maybe RDMA would allow it...
Jason