This series modifies the NV5x+ nouveau display backends to advertise appropriate format modifiers on their display planes in atomic mode setting blobs.
Corresponding modifications to Mesa/userspace are available on the Mesa-dev gitlab merge request 3724:
https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3724
I've tested this on Tesla, Kepler, Pascal, and Turing-class hardware using various formats and all the exposed format modifiers, plus some negative testing with invalid ones.
NOTE: this series depends on the "[PATCH v3] drm: Generalized NV Block Linear DRM format mod" patch submitted to dri-devel.
v2: Used Tesla family instead of NV50 chipset compare to avoid treating oddly numbered NV4x-class chipsets as NV50+ GPUs. Other instances of compares with chipset number in the series were audited, deemed safe, and left as-is for consistency with existing code.
v3: -Rebased on nouveau linux-5.6 @ 137c4ba7163ad9d5696b9fde78b1c0898a9c115b -Noted corresponding Mesa patches are production-worthy now -Better validate bo tile_mode when checking framebuffer size.
v4: Do not cache kind, tile_mode in nouveau_framebuffer
v5: Resolved against nouveau_framebuffer cleanup
James Jones (3): drm/nouveau: Add format mod prop to base/ovly/nvdisp drm/nouveau: Check framebuffer size against bo drm/nouveau: Support NVIDIA format modifiers
drivers/gpu/drm/nouveau/dispnv50/base507c.c | 7 +- drivers/gpu/drm/nouveau/dispnv50/disp.c | 59 +++++++ drivers/gpu/drm/nouveau/dispnv50/disp.h | 4 + drivers/gpu/drm/nouveau/dispnv50/wndw.c | 47 ++++- drivers/gpu/drm/nouveau/dispnv50/wndwc57e.c | 17 ++ drivers/gpu/drm/nouveau/nouveau_display.c | 183 ++++++++++++++++++++ drivers/gpu/drm/nouveau/nouveau_display.h | 6 + 7 files changed, 312 insertions(+), 11 deletions(-)