Hi Steve,
thank you for the fast reply.
Am 20.05.20 um 09:42 schrieb Steve Pronovost:
Echoing what others said, you're not making a DRM driver. The driver should live outside of the DRM code.
Agreed, please see my earlier reply. We'll be moving the driver to drivers/hyperv node or something similar. Apology for the confusion here.
I have one question about the driver API: on Windows, DirectX versions are loosly tied to Windows releases. So I guess you can change the kernel interface among DirectX versions? If so, how would this work on Linux in the long term? If there ever is a DirectX 13 or 14 with incompatible kernel interfaces, how would you plan to update the Linux driver?
You should think of the communication over the VM Bus for the vGPU projection as a strongly versioned interface. We will be keeping compatibility with older version of that interface as it evolves over time so we can continue to run older guest (we already do). This protocol isn't actually tied to the DX API. It is a generic abstraction for the GPU that can be used for any APIs (for example the NVIDIA CUDA driver that we announced is going over the same protocol to access the GPU).
New version of user mode DX can either take advantage or sometime require new services from this kernel abstraction. This mean that pulling a new version of user mode DX can mean having to also pull a new version of this vGPU kernel driver. For WSL, these essentially ships together. The kernel driver ships as part of our WSL2 Linux Kernel integration. User mode DX bits ships with Windows.
Just a friendly advise: maintaining a proprietary component within a Linux environment is tough. You will need a good plan for long-term interface stability and compatibility with the other components.
Best regards Thomas
-----Original Message----- From: Thomas Zimmermann tzimmermann@suse.de Sent: Wednesday, May 20, 2020 12:11 AM To: Sasha Levin sashal@kernel.org; alexander.deucher@amd.com; chris@chris-wilson.co.uk; ville.syrjala@linux.intel.com; Hawking.Zhang@amd.com; tvrtko.ursulin@intel.com Cc: linux-kernel@vger.kernel.org; linux-hyperv@vger.kernel.org; KY Srinivasan kys@microsoft.com; Haiyang Zhang haiyangz@microsoft.com; Stephen Hemminger sthemmin@microsoft.com; wei.liu@kernel.org; Steve Pronovost spronovo@microsoft.com; Iouri Tarassov iourit@microsoft.com; dri-devel@lists.freedesktop.org; linux-fbdev@vger.kernel.org; gregkh@linuxfoundation.org Subject: [EXTERNAL] Re: [RFC PATCH 0/4] DirectX on Linux
Hi
Am 19.05.20 um 18:32 schrieb Sasha Levin:
There is a blog post that goes into more detail about the bigger picture, and walks through all the required pieces to make this work. It is available here: https://devblogs.microsoft.com/directx/directx-heart-linux . The rest of this cover letter will focus on the Linux Kernel bits.
That's quite a surprise. Thanks for your efforts to contribute.
Overview
This is the first draft of the Microsoft Virtual GPU (vGPU) driver. The driver exposes a paravirtualized GPU to user mode applications running in a virtual machine on a Windows host. This enables hardware acceleration in environment such as WSL (Windows Subsystem for Linux) where the Linux virtual machine is able to share the GPU with the Windows host.
The projection is accomplished by exposing the WDDM (Windows Display Driver Model) interface as a set of IOCTL. This allows APIs and user mode driver written against the WDDM GPU abstraction on Windows to be ported to run within a Linux environment. This enables the port of the D3D12 and DirectML APIs as well as their associated user mode driver to Linux. This also enables third party APIs, such as the popular NVIDIA Cuda compute API, to be hardware accelerated within a WSL environment.
Only the rendering/compute aspect of the GPU are projected to the virtual machine, no display functionality is exposed. Further, at this time there are no presentation integration. So although the D3D12 API can be use to render graphics offscreen, there is no path (yet) for pixel to flow from the Linux environment back onto the Windows host desktop. This GPU stack is effectively side-by-side with the native Linux graphics stack.
The driver creates the /dev/dxg device, which can be opened by user mode application and handles their ioctls. The IOCTL interface to the driver is defined in dxgkmthk.h (Dxgkrnl Graphics Port Driver ioctl definitions). The interface matches the D3DKMT interface on Windows. Ioctls are implemented in ioctl.c.
Echoing what others said, you're not making a DRM driver. The driver should live outside of the DRM code.
I have one question about the driver API: on Windows, DirectX versions are loosly tied to Windows releases. So I guess you can change the kernel interface among DirectX versions?
If so, how would this work on Linux in the long term? If there ever is a DirectX 13 or 14 with incompatible kernel interfaces, how would you plan to update the Linux driver?
Best regards Thomas
When a VM starts, hyper-v on the host adds virtual GPU devices to the VM via the hyper-v driver. The host offers several VM bus channels to the VM: the global channel and one channel per virtual GPU, assigned to the VM.
The driver registers with the hyper-v driver (hv_driver) for the arrival of VM bus channels. dxg_probe_device recognizes the vGPU channels and creates the corresponding objects (dxgadapter for vGPUs and dxgglobal for the global channel).
The driver uses the hyper-V VM bus interface to communicate with the host. dxgvmbus.c implements the communication interface.
The global channel has 8GB of IO space assigned by the host. This space is managed by the host and used to give the guest direct CPU access to some allocations. Video memory is allocated on the host except in the case of existing_sysmem allocations. The Windows host allocates memory for the GPU on behalf of the guest. The Linux guest can access that memory by mapping GPU virtual address to allocations and then referencing those GPU virtual address from within GPU command buffers submitted to the GPU. For allocations which require CPU access, the allocation is mapped by the host into a location in the 8GB of IO space reserved in the guest for that purpose. The Windows host uses the nested CPU page table to ensure that this guest IO space always map to the correct location for the allocation as it may migrate between dedicated GPU memory (e.g. VRAM, firmware reserved DDR) and shared system memory (regular DDR) over its lifetime. The Linux guest maps a user mode CPU virtual address to an allocation IO space range for direct access by user mode APIs and drivers.
Implementation of LX_DXLOCK2 ioctl
We would appreciate your feedback on the implementation of the LX_DXLOCK2 ioctl.
This ioctl is used to get a CPU address to an allocation, which is resident in video/system memory on the host. The way it works:
The driver sends the Lock message to the host
The host allocates space in the VM IO space and maps it to the
allocation memory
The host returns the address in IO space for the mapped allocation
The driver (in dxg_map_iospace) allocates a user mode virtual
address range using vm_mmap and maps it to the IO space using io_remap_ofn_range)
- The VA is returned to the application
Internal objects
The following objects are created by the driver (defined in dxgkrnl.h):
dxgadapter - represents a virtual GPU
dxgprocess - tracks per process state (handle table of created objects, list of objects, etc.)
dxgdevice - a container for other objects (contexts, paging queues, allocations, GPU synchronization objects)
dxgcontext - represents thread of GPU execution for packet scheduling.
dxghwqueue - represents thread of GPU execution of hardware
scheduling
dxgallocation - represents a GPU accessible allocation
dxgsyncobject - represents a GPU synchronization object
dxgresource - collection of dxgalloction objects
dxgsharedresource, dxgsharedsyncobj - helper objects to share objects between different dxgdevice objects, which can belong to different
processes
Object handles
All GPU objects, created by the driver, are accessible by a handle (d3dkmt_handle). Each process has its own handle table, which is implemented in hmgr.c. For each API visible object, created by the driver, there is an object, created on the host. For example, the is a dxgprocess object on the host for each dxgprocess object in the VM, etc. The object handles have the same value in the host and the VM, which is done to avoid translation from the guest handles to the host handles.
Signaling CPU events by the host
The WDDM interface provides a way to signal CPU event objects when execution of a context reached certain point. The way it is implemented:
application sends an event_fd via ioctl to the driver
eventfd_ctx_get is used to get a pointer to the file object (eventfd_ctx)
the pointer to sent the host via a VM bus message
when GPU execution reaches a certain point, the host sends a message to the VM with the event pointer
signal_guest_event() handles the messages and eventually eventfd_signal() is called.
Sasha Levin (4): gpu: dxgkrnl: core code gpu: dxgkrnl: hook up dxgkrnl Drivers: hv: vmbus: hook up dxgkrnl gpu: dxgkrnl: create a MAINTAINERS entry
MAINTAINERS | 7 + drivers/gpu/Makefile | 2 +- drivers/gpu/dxgkrnl/Kconfig | 10 + drivers/gpu/dxgkrnl/Makefile | 12 + drivers/gpu/dxgkrnl/d3dkmthk.h | 1635 +++++++++ drivers/gpu/dxgkrnl/dxgadapter.c | 1399 ++++++++ drivers/gpu/dxgkrnl/dxgkrnl.h | 913 ++++++ drivers/gpu/dxgkrnl/dxgmodule.c | 692 ++++ drivers/gpu/dxgkrnl/dxgprocess.c | 355 ++ drivers/gpu/dxgkrnl/dxgvmbus.c | 2955 +++++++++++++++++ drivers/gpu/dxgkrnl/dxgvmbus.h | 859 +++++ drivers/gpu/dxgkrnl/hmgr.c | 593 ++++ drivers/gpu/dxgkrnl/hmgr.h | 107 + drivers/gpu/dxgkrnl/ioctl.c | 5269 ++++++++++++++++++++++++++++++ drivers/gpu/dxgkrnl/misc.c | 280 ++ drivers/gpu/dxgkrnl/misc.h | 288 ++ drivers/video/Kconfig | 2 + include/linux/hyperv.h | 16 + 18 files changed, 15393 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/dxgkrnl/Kconfig create mode 100644 drivers/gpu/dxgkrnl/Makefile create mode 100644 drivers/gpu/dxgkrnl/d3dkmthk.h create mode 100644 drivers/gpu/dxgkrnl/dxgadapter.c create mode 100644 drivers/gpu/dxgkrnl/dxgkrnl.h create mode 100644 drivers/gpu/dxgkrnl/dxgmodule.c create mode 100644 drivers/gpu/dxgkrnl/dxgprocess.c create mode 100644 drivers/gpu/dxgkrnl/dxgvmbus.c create mode 100644 drivers/gpu/dxgkrnl/dxgvmbus.h create mode 100644 drivers/gpu/dxgkrnl/hmgr.c create mode 100644 drivers/gpu/dxgkrnl/hmgr.h create mode 100644 drivers/gpu/dxgkrnl/ioctl.c create mode 100644 drivers/gpu/dxgkrnl/misc.c create mode 100644 drivers/gpu/dxgkrnl/misc.h
-- Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer