On Mon, Jul 6, 2015 at 8:42 PM, Andrew Chew achew@nvidia.com wrote:
Hello,
I am currently looking into ways to support fixed virtual address allocations and sparse mappings in nouveau, as a step towards supporting CUDA.
CUDA requires that the GPU virtual address for a given buffer match the CPU virtual address. Therefore, when mapping a CUDA buffer, we have to have a way of specifying a particular virtual address to map to (we would ask that the CPU virtual address be used). Currently, as I understand it, the allocator implemented in nvkm/core/mm.c, used to provision virtual addresses, doesn't allow for this (but it's very easy to modify the allocator slightly to allow for this, which I have done locally in my experiments).
In addition, the CUDA use case typically involves allocating a big chunk of address space ahead of time as a way to reserve that chunk for future CUDA use. It then maps individual buffers into that address space as needed. Currently, the virtual address allocation is done during buffer mapping, so in order to support these sparse mappings, it seems to me that the virtual address allocation and buffer mapping need to be decoupled into separate operations.
My current strawman proposal for supporting this is to introduce two new ioctls DRM_IOCTL_NOUVEAU_AS_ALLOC and DRM_IOCTL_NOUVEAU_AS_FREE, that look roughly like this:
#define NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET 0x1 struct drm_nouveau_as_alloc { uint64_t pages; /* in, pages */ uint32_t page_size; /* in, bytes */ uint32_t flags; /* in */ uint64_t offset; /* in/out, byte address */ };
struct drm_nouveau_as_free { uint64_t offset; /* in, byte address */ };
These ioctls just call into the allocator to allocate a range of addresses, resulting in a struct nvkm_vma that tracks that allocation (or releases the struct nvkm_vma back into the virtual address pool in the case of the free ioctl). If NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET is set, offset specifies the requested virtual address. Otherwise, an arbitrary address will be allocated.
Well, this can't just be an address space. You still need bo's, if this is to work with nouveau -- it has to know when to swap things in and out, when they're used, etc. (and/or move between VRAM and GART and system/swap). I suspect that your target here are the GK20A and GM20B chips which don't have dedicated VRAM, but the ioctl's need to work for everything.
Would it be sufficient to extend NOUVEAU_GEM_NEW or create a NOUVEAU_GEM_NEW_FIXED or something? IOW, why do have to separate the concept of a GEM object and a VM allocation?
In addition to this, a way to map/unmap buffers is needed. Ordinarily, one would just use DRM_IOCTL_PRIME_FD_TO_HANDLE to import and map a dmabuf into gem. However, this ioctl will try to grab the virtual address range for this buffer, which will fail in the CUDA case since the virtual address range has been reserved ahead of time. So we perhaps introduce a set of ioctls to map/unmap buffers on top of an already existing virtual address allocation.
My suggestion above is an alternative to this, right? I think dmabufs tend to be used for sharing between devices. I suspect there's more going on here that I don't understand though -- I assume the CUDA use-case is similar to the HSA use-case -- being able to build up data structures that point to one another on the CPU and then process them on the GPU? Can you detail a specific use-case perhaps, including the interactions with the GPU and its address space?
Jérôme, I believe you were doing the HSA kernel implementation. Perhaps you'd have some feedback on this proposal?
Cheers,
-ilia
On Tue, Jul 07, 2015 at 11:29:38AM -0400, Ilia Mirkin wrote:
On Mon, Jul 6, 2015 at 8:42 PM, Andrew Chew achew@nvidia.com wrote:
Hello,
I am currently looking into ways to support fixed virtual address allocations and sparse mappings in nouveau, as a step towards supporting CUDA.
CUDA requires that the GPU virtual address for a given buffer match the CPU virtual address. Therefore, when mapping a CUDA buffer, we have to have a way of specifying a particular virtual address to map to (we would ask that the CPU virtual address be used). Currently, as I understand it, the allocator implemented in nvkm/core/mm.c, used to provision virtual addresses, doesn't allow for this (but it's very easy to modify the allocator slightly to allow for this, which I have done locally in my experiments).
In addition, the CUDA use case typically involves allocating a big chunk of address space ahead of time as a way to reserve that chunk for future CUDA use. It then maps individual buffers into that address space as needed. Currently, the virtual address allocation is done during buffer mapping, so in order to support these sparse mappings, it seems to me that the virtual address allocation and buffer mapping need to be decoupled into separate operations.
My current strawman proposal for supporting this is to introduce two new ioctls DRM_IOCTL_NOUVEAU_AS_ALLOC and DRM_IOCTL_NOUVEAU_AS_FREE, that look roughly like this:
#define NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET 0x1 struct drm_nouveau_as_alloc { uint64_t pages; /* in, pages */ uint32_t page_size; /* in, bytes */ uint32_t flags; /* in */ uint64_t offset; /* in/out, byte address */ };
struct drm_nouveau_as_free { uint64_t offset; /* in, byte address */ };
These ioctls just call into the allocator to allocate a range of addresses, resulting in a struct nvkm_vma that tracks that allocation (or releases the struct nvkm_vma back into the virtual address pool in the case of the free ioctl). If NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET is set, offset specifies the requested virtual address. Otherwise, an arbitrary address will be allocated.
Well, this can't just be an address space. You still need bo's, if this is to work with nouveau -- it has to know when to swap things in and out, when they're used, etc. (and/or move between VRAM and GART and system/swap). I suspect that your target here are the GK20A and GM20B chips which don't have dedicated VRAM, but the ioctl's need to work for everything.
Would it be sufficient to extend NOUVEAU_GEM_NEW or create a NOUVEAU_GEM_NEW_FIXED or something? IOW, why do have to separate the concept of a GEM object and a VM allocation?
Well maybe something like i did for radeon. With radeon you have 2 set of ioctl. One to create/delete bo (GEM stuff) and one to associate a virtual address with a bo. I wanted to let the userspace decide on virtual address of buffer precisely for the same reason CUDA do it ie to allow to map some buffer at same address in GPU address space as in CPU address space. So far we never really took advantage of that on radeon side.
Also on radeon you can map same bo at different virtual address in same process (you will need different file descriptor for each mapping and you can only submit command stream using mapping valid for the file descriptor). Thought this is mostly usefull when sharing same bo accross different process.
I think my radeon virtual address ioclt are nice design but other might disagree. If you want to look at the code :
drivers/gpu/drm/radeon/radeon_vm.c drivers/gpu/drm/radeon/radeon_gem.c
Grep for _va (virtual address per bo) or _vm (virtual address manager per file descriptor) function name and structure name.
On the command stream and bo eviction side everything is as usual on radeon. So a bo can be evicted btw 2 command stream to make room for another one. Either its mapping is invalidated or updated to point to system memory. So most of the logic for everything else remain the same (just need to update the multiple virtual address space).
In addition to this, a way to map/unmap buffers is needed. Ordinarily, one would just use DRM_IOCTL_PRIME_FD_TO_HANDLE to import and map a dmabuf into gem. However, this ioctl will try to grab the virtual address range for this buffer, which will fail in the CUDA case since the virtual address range has been reserved ahead of time. So we perhaps introduce a set of ioctls to map/unmap buffers on top of an already existing virtual address allocation.
My suggestion above is an alternative to this, right? I think dmabufs tend to be used for sharing between devices. I suspect there's more going on here that I don't understand though -- I assume the CUDA use-case is similar to the HSA use-case -- being able to build up data structures that point to one another on the CPU and then process them on the GPU? Can you detail a specific use-case perhaps, including the interactions with the GPU and its address space?
I think you nailed it, it is really about having the same address pointing to the same thing on both the GPU and CPU. But this is also valid and usefull for VRAM. OpenCL 2.0 have various level of transparent address space (probably not the term use in the spec) and the lowest level would need something like what radeon have to work. The most advance level needs more plumbing inside core kernel mm or inside the CPU and GPU hardware.
Jérôme, I believe you were doing the HSA kernel implementation. Perhaps you'd have some feedback on this proposal?
No i did not do the HSA stuff, AMD team leaded by Oded did :)
Cheers, Jérôme
On Tue, Jul 7, 2015 at 8:27 PM, Jerome Glisse j.glisse@gmail.com wrote:
On Tue, Jul 07, 2015 at 11:29:38AM -0400, Ilia Mirkin wrote:
On Mon, Jul 6, 2015 at 8:42 PM, Andrew Chew achew@nvidia.com wrote:
Hello,
I am currently looking into ways to support fixed virtual address allocations and sparse mappings in nouveau, as a step towards supporting CUDA.
CUDA requires that the GPU virtual address for a given buffer match the CPU virtual address. Therefore, when mapping a CUDA buffer, we have to have a way of specifying a particular virtual address to map to (we would ask that the CPU virtual address be used). Currently, as I understand it, the allocator implemented in nvkm/core/mm.c, used to provision virtual addresses, doesn't allow for this (but it's very easy to modify the allocator slightly to allow for this, which I have done locally in my experiments).
In addition, the CUDA use case typically involves allocating a big chunk of address space ahead of time as a way to reserve that chunk for future CUDA use. It then maps individual buffers into that address space as needed. Currently, the virtual address allocation is done during buffer mapping, so in order to support these sparse mappings, it seems to me that the virtual address allocation and buffer mapping need to be decoupled into separate operations.
My current strawman proposal for supporting this is to introduce two new ioctls DRM_IOCTL_NOUVEAU_AS_ALLOC and DRM_IOCTL_NOUVEAU_AS_FREE, that look roughly like this:
#define NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET 0x1 struct drm_nouveau_as_alloc { uint64_t pages; /* in, pages */ uint32_t page_size; /* in, bytes */ uint32_t flags; /* in */ uint64_t offset; /* in/out, byte address */ };
struct drm_nouveau_as_free { uint64_t offset; /* in, byte address */ };
These ioctls just call into the allocator to allocate a range of addresses, resulting in a struct nvkm_vma that tracks that allocation (or releases the struct nvkm_vma back into the virtual address pool in the case of the free ioctl). If NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET is set, offset specifies the requested virtual address. Otherwise, an arbitrary address will be allocated.
Well, this can't just be an address space. You still need bo's, if this is to work with nouveau -- it has to know when to swap things in and out, when they're used, etc. (and/or move between VRAM and GART and system/swap). I suspect that your target here are the GK20A and GM20B chips which don't have dedicated VRAM, but the ioctl's need to work for everything.
Would it be sufficient to extend NOUVEAU_GEM_NEW or create a NOUVEAU_GEM_NEW_FIXED or something? IOW, why do have to separate the concept of a GEM object and a VM allocation?
Well maybe something like i did for radeon. With radeon you have 2 set of ioctl. One to create/delete bo (GEM stuff) and one to associate a virtual address with a bo. I wanted to let the userspace decide on virtual address of buffer precisely for the same reason CUDA do it ie to allow to map some buffer at same address in GPU address space as in CPU address space. So far we never really took advantage of that on radeon side.
Also on radeon you can map same bo at different virtual address in same process (you will need different file descriptor for each mapping and you can only submit command stream using mapping valid for the file descriptor). Thought this is mostly usefull when sharing same bo accross different process.
I think my radeon virtual address ioclt are nice design but other might disagree. If you want to look at the code :
drivers/gpu/drm/radeon/radeon_vm.c drivers/gpu/drm/radeon/radeon_gem.c
Grep for _va (virtual address per bo) or _vm (virtual address manager per file descriptor) function name and structure name.
On the command stream and bo eviction side everything is as usual on radeon. So a bo can be evicted btw 2 command stream to make room for another one. Either its mapping is invalidated or updated to point to system memory. So most of the logic for everything else remain the same (just need to update the multiple virtual address space).
In addition to this, a way to map/unmap buffers is needed. Ordinarily, one would just use DRM_IOCTL_PRIME_FD_TO_HANDLE to import and map a dmabuf into gem. However, this ioctl will try to grab the virtual address range for this buffer, which will fail in the CUDA case since the virtual address range has been reserved ahead of time. So we perhaps introduce a set of ioctls to map/unmap buffers on top of an already existing virtual address allocation.
My suggestion above is an alternative to this, right? I think dmabufs tend to be used for sharing between devices. I suspect there's more going on here that I don't understand though -- I assume the CUDA use-case is similar to the HSA use-case -- being able to build up data structures that point to one another on the CPU and then process them on the GPU? Can you detail a specific use-case perhaps, including the interactions with the GPU and its address space?
I think you nailed it, it is really about having the same address pointing to the same thing on both the GPU and CPU. But this is also valid and usefull for VRAM. OpenCL 2.0 have various level of transparent address space (probably not the term use in the spec) and the lowest level would need something like what radeon have to work. The most advance level needs more plumbing inside core kernel mm or inside the CPU and GPU hardware.
Jérôme, I believe you were doing the HSA kernel implementation. Perhaps you'd have some feedback on this proposal?
No i did not do the HSA stuff, AMD team leaded by Oded did :)
Cheers, Jérôme _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi, So this is very similar to HSA and OpenCL 2.0 requirements. We had an easy life so far for HSA, because amdkfd currently only support APUs, where the system memory is shared between the CPU and GPU cores. In this scenario, we have a dedicated H/W module called IOMMUv2, which uses the CPU page tables to provide access from the GPU core to the system memory.
To support discrete GPUs, amdkfd need to implement one of the following models:
1. Have different virtual address spaces for CPU and GPU buffers, aka OpenCL 1.2. In this model, we prepare the data on the system memory, copy the data to local memory, the GPU works on it, then we copy the results back to system memory. We actually have an implementation of this but it is not upstreamed yet (look below for reference to implementation).
2. Use the GPUVM inside the GPU to access system memory. The limitation here is that the GPUVM uses 40-bit addresses, so the virtual address range must be in the lower 40-bit address range of the CPU. Access speed is limited by PCI-e bandwidth. Another limitation is that the system memory pages must be pinned as the GPU doesn't support page faults.
I think your model is the same as the latter. The latest planning before I left AMD was: 1. Reserve a large chunk of address space in the lower 40-bit address space of the process, when it is created. 2. When a buffer is required by the application, reserve a chunk out of that address space, then create BOs and map them to that address space. I advise to use a fixed size BO (2/4 MB) and if the application require a larger allocation, allocate a list of BOs
The operation of reserving address space and BO creation is one IOCTL, while the mapping of the BO to the address space is a second IOCTL. There are of course unmap and free IOCTLs. The separation is done for a couple of reasons:
1. If the application knows that it wants to use only part of the memory area it allocated, then there is no point in pinning all the BOs. So, the application can map/unmap just part of the allocation.
2. If the application knows that it has finished using the BOs, and it also knows that it will use them later on, it can unmap the BOs (to make them unpinned) but not free them so the memory is still reserved (with its contents intact).
For reference to the first model, look at http://cgit.freedesktop.org/~gabbayo/linux/tree/drivers/gpu/drm/amd/amdkfd/k...
Specifically, at functions:
kfd_ioctl_alloc_memory_of_gpu kfd_ioctl_free_memory_of_gpu kfd_ioctl_map_memory_to_gpu kfd_ioctl_unmap_memory_from_gpu
You can also look at the matching userspace code at:
http://cgit.freedesktop.org/~gabbayo/libhsakmt/tree/src/memory.c?h=libhsakmt...
Hope this helps.
Oded
- Use the GPUVM inside the GPU to access system memory. The
limitation here is that the GPUVM uses 40-bit addresses, so the virtual address range must be in the lower 40-bit address range of the CPU. Access speed is limited by PCI-e bandwidth. Another limitation is that the system memory pages must be pinned as the GPU doesn't support page faults.
The operation of reserving address space and BO creation is one IOCTL, while the mapping of the BO to the address space is a second IOCTL. There are of course unmap and free IOCTLs. The separation is done for a couple of reasons:
- If the application knows that it wants to use only part of the
memory area it allocated, then there is no point in pinning all the BOs. So, the application can map/unmap just part of the allocation.
- If the application knows that it has finished using the BOs, and it
also knows that it will use them later on, it can unmap the BOs (to make them unpinned) but not free them so the memory is still reserved (with its contents intact).
Yes, thanks Oled. I think this is pretty much exactly how I imagine things to work.
I'll post my code soon and see what you guys think. There was some misunderstanding on my part on how bo's work, so I need to rework some stuff.
On Tue, Jul 07, 2015 at 11:29:38AM -0400, Ilia Mirkin wrote:
On Mon, Jul 6, 2015 at 8:42 PM, Andrew Chew achew@nvidia.com wrote:
These ioctls just call into the allocator to allocate a range of addresses, resulting in a struct nvkm_vma that tracks that allocation (or releases the struct nvkm_vma back into the virtual address pool in the case of the free ioctl). If NOUVEAU_AS_ALLOC_FLAGS_FIXED_OFFSET is set, offset specifies the requested virtual address. Otherwise, an arbitrary address will be allocated.
Well, this can't just be an address space. You still need bo's, if this is to work with nouveau -- it has to know when to swap things in and out, when they're used, etc. (and/or move between VRAM and GART and system/swap). I suspect that your target here are the GK20A and GM20B chips which don't have dedicated VRAM, but the ioctl's need to work for everything.
Would it be sufficient to extend NOUVEAU_GEM_NEW or create a NOUVEAU_GEM_NEW_FIXED or something? IOW, why do have to separate the concept of a GEM object and a VM allocation?
You're correct. This is for gk20a and gm20b.
The thing these proposed ioctls are supposed to accomplish is to reserve, ahead of time, a portion of the address space. So at this time, there really aren't any buffer objects yet, and there's nothing to be mapped to the GMMU. That part would come later.
In addition to this, a way to map/unmap buffers is needed. Ordinarily, one would just use DRM_IOCTL_PRIME_FD_TO_HANDLE to import and map a dmabuf into gem. However, this ioctl will try to grab the virtual address range for this buffer, which will fail in the CUDA case since the virtual address range has been reserved ahead of time. So we perhaps introduce a set of ioctls to map/unmap buffers on top of an already existing virtual address allocation.
My suggestion above is an alternative to this, right? I think dmabufs tend to be used for sharing between devices. I suspect there's more going on here that I don't understand though -- I assume the CUDA use-case is similar to the HSA use-case -- being able to build up data structures that point to one another on the CPU and then process them on the GPU? Can you detail a specific use-case perhaps, including the interactions with the GPU and its address space?
The whole dmabufs thing is kind of a side issue. I'll take a look at NOUVEAU_GEM_NEW, but that could be an alternative to this, maybe, if extended (or we make a new NOUVEAU_GEM_NEW_FIXED, as you suggested). Crucially, the NOUVEAU_GEM_NEW_FIXED operation shouldn't result in trying to get a virtual address region and then failing because a previous operation (see above) has reserved it already.
The use case is exactly as you describe. There are data structures built up that contain CPU pointers, and those pointers need to make sense to the GPU as well.
dri-devel@lists.freedesktop.org
-
Andrew Chew -
Ilia Mirkin -
Jerome Glisse -
Oded Gabbay