This series modifies the NV5x+ nouveau display backends to advertise appropriate format modifiers on their display planes in atomic mode setting blobs.
Corresponding modifications to Mesa/userspace are available on the Mesa-dev mailing list as the series:
nouveau: Improved format modifier support
I've tested this on Tesla, Kepler, Pascal, and Turing-class hardware using various formats and all the exposed format modifiers, plus some negative testing with invalid ones.
NOTE: this series depends on the "[PATCH v3] drm: Generalized NV Block Linear DRM format mod" patch submitted to dri-devel.
v2: Used Tesla family instead of NV50 chipset compare to avoid treating oddly numbered NV4x-class chipsets as NV50+ GPUs. Other instances of compares with chipset number in the series were audited, deemed safe, and left as-is for consistency with existing code.
v3: -Rebased on nouveau linux-5.6 @ 137c4ba7163ad9d5696b9fde78b1c0898a9c115b -Noted corresponding Mesa patches are production-worthy now -Better validate bo tile_mode when checking framebuffer size.
James Jones (3): drm/nouveau: Add format mod prop to base/ovly/nvdisp drm/nouveau: Check framebuffer size against bo drm/nouveau: Support NVIDIA format modifiers
drivers/gpu/drm/nouveau/dispnv50/base507c.c | 7 +- drivers/gpu/drm/nouveau/dispnv50/disp.c | 59 ++++++++ drivers/gpu/drm/nouveau/dispnv50/disp.h | 4 + drivers/gpu/drm/nouveau/dispnv50/wndw.c | 35 ++++- drivers/gpu/drm/nouveau/dispnv50/wndwc57e.c | 17 +++ drivers/gpu/drm/nouveau/nouveau_display.c | 158 ++++++++++++++++++++ drivers/gpu/drm/nouveau/nouveau_display.h | 4 + 7 files changed, 276 insertions(+), 8 deletions(-)