Hello,
This patchset introduces memory shrinker for the VirtIO-GPU DRM driver. During OOM, the shrinker will release BOs that are marked as "not needed" by userspace using the new madvise IOCTL. The userspace in this case is the Mesa VirGL driver, it will mark the cached BOs as "not needed", allowing kernel driver to release memory of the cached shmem BOs on lowmem situations, preventing OOM kills.
This patchset includes couple fixes for problems I found while was working on the shrinker, it also includes prerequisite DMA API usage improvement needed by the shrinker.
The Mesa and IGT patches will be kept on hold until this kernel series will be approved and applied.
This patchset was tested using Qemu and crosvm, including both cases of IOMMU off/on.
Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise IGT: https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/tree/virtio-madvise
Dmitry Osipenko (5): drm/virtio: Correct drm_gem_shmem_get_sg_table() error handling drm/virtio: Check whether transferred 2D BO is shmem drm/virtio: Unlock GEM reservations in error code path drm/virtio: Improve DMA API usage for shmem BOs drm/virtio: Add memory shrinker
drivers/gpu/drm/virtio/Makefile | 3 +- drivers/gpu/drm/virtio/virtgpu_drv.c | 22 +++- drivers/gpu/drm/virtio/virtgpu_drv.h | 31 ++++- drivers/gpu/drm/virtio/virtgpu_gem.c | 84 ++++++++++++ drivers/gpu/drm/virtio/virtgpu_gem_shrinker.c | 124 ++++++++++++++++++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 ++++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 17 ++- drivers/gpu/drm/virtio/virtgpu_object.c | 63 +++------ drivers/gpu/drm/virtio/virtgpu_plane.c | 17 ++- drivers/gpu/drm/virtio/virtgpu_vq.c | 30 +++-- include/uapi/drm/virtgpu_drm.h | 14 ++ 11 files changed, 373 insertions(+), 69 deletions(-) create mode 100644 drivers/gpu/drm/virtio/virtgpu_gem_shrinker.c