On Tue, Oct 6, 2020 at 6:34 PM Daniel Vetter daniel@ffwll.ch wrote:
On Tue, Oct 06, 2020 at 12:49:56PM -0300, Jason Gunthorpe wrote:
On Tue, Oct 06, 2020 at 11:22:14AM +0200, Daniel Vetter wrote:
For reinstanting the pages you need:
- dma_resv_lock, this prevents anyone else from issuing new moves or anything like that
- dma_resv_get_excl + dma_fence_wait to wait for any pending moves to finish. gpus generally don't wait on the cpu, but block the dependent dma operations from being scheduled until that fence fired. But for rdma odp I think you need the cpu wait in your worker here.
Reinstating is not really any different that the first insertion, so then all this should be needed in every case?
Yes. Without move_notify we pin the dma-buf into system memory, so it can't move, and hence you also don't have to chase it. But with move_notify this all becomes possible.
I just realized I got it wrong compared to gpus. I needs to be: 1. dma_resv_lock 2. dma_buf_map_attachment, which might have to move the buffer around again if you're unlucky 3. wait for the exclusive fence 4. put sgt into your rdma ptes 5 dma_resv_unlock
Maybe also something we should document somewhere for dynamic buffers. Assuming I got it right this time around ... Christian? -Daniel