Hi,
There were certainly some interesting changes discussed at the allocator workshop during XDC this year, and I'd like to just summarise my thoughts on it and make sure everybody is on the same page.
For those who don't know who I am or my stake in this, I'm the maintainer of the DRM and graphics code for the wlroots Wayland compositor library. I'm ascent12 on Github and Freenode.
My understanding of the issue Nvidia was trying to solve was the in-place transition between different format modifiers. E.g. if a client is to be scanned out, the buffer would need to be transitioned to a non-compressed format that the display controller can work with, but if the client is to be composited, a compressed format would be used, saving on memory bandwidth. Hardware may have more efficient ways to transition between different formats, so it would be good if we can use these and not rely on having to perform a blit if we don't need to. The problem is more general than this, but that was just the example given.
The original solution proposed in James' talk was to add functions to EGL/OpenGL/Vulkan and have the display server perform transitions where required.
Discussions during the workshop at the start tended to having libliftoff handle all of this, but would require libliftoff to have its own rendering context, which I think is bloating the purpose of the library. Also discussed was to have libliftoff ask the compositor to perform the transition if it thinks it was possible.
Another suggestion I made was to make use of Simon's dmabuf hints patch to the wp_linux_dmabuf protocol [1] and leave it up to the client's GPU driver to handle any transitions. This wasn't adequately represented in the lightning talk summarising the workshop, so I'll go over it here now, making sure everyone understands what it is and why I think it is the way we should go forward.
Right now, a Wayland compositor will advertise all of the format+modifier pairs that it supports, but currently does not provide any context for clients as to which one they should actually choose. It's basically up to chance if a client is able to be scanned out and is likely to lead to several suboptimal situations.
The dmabuf hints patch adds a way to suggest a better format to use, based on the current context. This is dynamic, and can be sent multiple times over the lifetime of a surface. The patch also adds a way for the compositor to tell the client which GPU its using, which is useful for clients to know in multi GPU situations.
These hints are in various "tranches", which are just groups of format+modifier pairs of the same preference. The tranches are ordered from most optimal to least optimal. The most optimal tranche would imply direct scanout, while a less optimal tranche would imply compositing, but is not actually defined like that in the protocol.
If a client becomes fullscreen, we would send the format+modifier pairs for the primary plane as the most optimal tranche. If a client is eligible to be scanned out on an overlay plane, we would send the format+modifier pairs for that plane. If a client is partially occluded or otherwise not possible to be scanned out, we'd just have the normal format+modifier pairs that we can use as a texture. Note that the compositor won't send format+modifier pairs which we cannot texture from, even if the plane advertises it's supported. We always need to be able to fall back to compositing.
The hard part of figuring out which clients are "eligible" for being scanned out on an overlay plane could be handled by libliftoff (or something similar) and given back to the compositor to forward to clients. For libliftoff to make a properly informed decision, I think the atomic KMS API needs to be changed. We can only TEST_ONLY for valid buffers, testing the immediate configuration, but doesn't allow us to test for a configuration we WANT to go to. We need some sort of fake framebuffer not backed by any real memory, but will allow us to TEST_ONLY it. Without this, we may tell the client format+modifier pairs that we think will work for scanout, but don't due to whatever hardware limitations or transient issues like memory bandwidth, and we could actually make things worse by having the client transition formats.
As an aside, I would really like these fake framebuffers for my modesetting set up to be a lot cleaner too.
I'm sure this has been discussed before, and I'm not really sure what the implications are from a driver perspective. I'd have to leave it up to people more familiar with KMS and driver internals to comment on this. Even if the solution isn't 100%, something that works most of time would be hugely helpful (especially with RGB formats). Perhaps this is not possible and it would need to live purely in driver-specific code inside of libraries like libliftoff, but it would be nice to not come to that. It seems useful enough for a generic KMS userspace.
As to how dmabuf hints would look client-side, I think this could be managed by the GPU driver pretty easily.
For EGL, if the driver is capable of transitions in-place, they can simply do that as required. If the GPU cannot transition to a new format directly, they can deallocate then reallocate buffers in the new format as they are consumed. This would only lead to a couple of inefficient frames when a new hint is sent, but will reach an optimal situation in the steady state.
Vulkan already has VK_SUBOPTIMAL_KHR for telling the application that it should reallocate its swapchain.
If either the EGL driver or Vulkan application doesn't take these hints to account, things will still continue to work, just not in the most optimal way, basically as it works now.
I believe that the dmabuf hints patch should meet the transition issue that Nvidia was trying to fix. It keeps a lot of the complexity inside of the drivers, and keeps the rendering complexity outside of libliftoff, and doesn't require extensions to EGL/OpenGL/Vulkan as far as I know.
In the future, if any sort of constraints interface is worked out, wp_linux_dmabuf could be extended again to accommodate it. There may already be a need for some extra flags which would be the equivalent of GBM_BO_USE_SCANOUT, but I'm not going to try and design that interface here.
Thanks for your time. Any feedback is welcome.
Scott