The KFD API is quite inflexible in that it allows only mapping entire BOs at the same virtual address on all GPUs. This is incompatible with newer CUDA memory management APIs. (see https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VA.html)
Instead of inventing more KFD APIs to fix this, the goal of this patch series is to enable ROCr to use the DRM render node API for more flexible VA mappings. It builds on the DMABuf export API that is being proposed for RDMA. DMABuf handles exported by KFD can be imported by libdrm-amdgpu and then used with amdgpu_bo_va_op.
This is a first proof of concept and request for comment.
A complete solution will likely need minor tweaks to the KFD API to allow memory allocation and import without a pre-determined virtual address.
Other options that were considered but rejected: - Using GEM API to create BO in KFD VMs. This would require significant plumbing to get those BOs registered with the KFD eviction fence mechanism and "no overcommitment" memory limits - Variation of the above using AMDGPU_GEM_CREATE_VM_ALWAYS_VALID to simplify validation. Doesn't work because it doesn't allow DMABuf exports - Creating KFD BOs with GEM handles. Doesn't help because there is no way to import GEM handles into libdrm-amdgpu
Felix Kuehling (4): drm/amdkfd: Improve amdgpu_vm_handle_moved drm/amdgpu: Attach eviction fence on alloc drm/amdgpu: update mappings not managed by KFD drm/amdgpu: Do bo_va ref counting for KFD BOs
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 99 ++++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 +- drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 18 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 3 +- 5 files changed, 86 insertions(+), 42 deletions(-)