On Tue, Oct 06, 2020 at 08:17:05PM +0200, Daniel Vetter wrote:
So on the gpu we pipeline this all. So step 4 doesn't happen on the cpu, but instead we queue up a bunch of command buffers so that the gpu writes these pagetables (and the flushes tlbs and then does the actual stuff userspace wants it to do).
mlx5 HW does basically this as well.
We just apply scheduling for this work on the device, not in the CPU.
just queue it all up and let the gpu scheduler sort out the mess. End result is that you get a sgt that points at stuff which very well might have nothing even remotely resembling your buffer in there at the moment. But all the copy operations are queued up, so rsn the data will also be there.
The explanation make sense, thanks
But rdma doesn't work like that, so it looks all a bit funny.
Well, I guess it could, but how would it make anything better? I can overlap building the SGL and the device PTEs with something else doing 'move', but is that a workload that needs such agressive optimization?
This is also why the precise semantics of move_notify for gpu<->gpu sharing took forever to discuss and are still a bit wip, because you have the inverse problem: The dma api mapping might still be there
Seems like this all makes a graph of operations, can't start the next one until all deps are finished. Actually sounds a lot like futures.
Would be clearer if this attach API provided some indication that the SGL is actually a future valid SGL..
Jason