Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
Virtual DRM splits the overlay planes of a display controller into multiple virtual devices to allow each plane to be accessed by each process.
This makes it possible to overlay images output from multiple processes on a display. For example, one process displays the camera image without compositor while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease. We want to separate the resources from the master in units of planes, so we proposed virtual DRM. I think the advantage of vDRM is that you can use general DRM APIs in userland.
I don't think that's not the way to go, at least not by touching each single display driver, and not hardcoding the planes in DT.
Thank you for comment. I will reconsider about DT.
What's the actual use case you're doing that for ? Why not using some userland compositor ?
I think when latency is important (e.g., AR, VR, for displaying camera images in IVI systems), there may be use cases where the compositor cannot be used. Normally, when the image is passed through the compositor, it is displayed after 2 VSYNC at most, because the compositor combines the image with VSYNC synchronization. On the other hand, if we use vDRM, the image will be displayed at the next VSYNC, so it will be displayed after 1 VSYNC at most.
Also, since the compositor is a single point of failure, we may not want to make it dependent on it.
Best regards Tomohito Esaki
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
Virtual DRM splits the overlay planes of a display controller into multiple virtual devices to allow each plane to be accessed by each process.
This makes it possible to overlay images output from multiple processes on a display. For example, one process displays the camera image without compositor while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
We want to separate the resources from the master in units of planes, so we proposed virtual DRM. I think the advantage of vDRM is that you can use general DRM APIs in userland.
You do that with DRM leases too.
I don't think that's not the way to go, at least not by touching each single display driver, and not hardcoding the planes in DT.
Thank you for comment. I will reconsider about DT.
What's the actual use case you're doing that for ? Why not using some userland compositor ?
I think when latency is important (e.g., AR, VR, for displaying camera images in IVI systems), there may be use cases where the compositor cannot be used. Normally, when the image is passed through the compositor, it is displayed after 2 VSYNC at most, because the compositor combines the image with VSYNC synchronization. On the other hand, if we use vDRM, the image will be displayed at the next VSYNC, so it will be displayed after 1 VSYNC at most.
As I said in my other email, this is false in the general sense.
Also, since the compositor is a single point of failure, we may not want to make it dependent on it.
This... I'm not quite sure I buy it. If any of all the programs using virtual KMS crashes, you still lose some crucial components from your display. Maybe that program, while crashing, uploads such a bad state to its very own KMS plane, that it causes other KMS planes to malfunction. Then you need to detect this situation and still restart everything, not just the crashed program.
I would think a userspace compositor approach is actually more reliable. You write the compositor to be extremely robust. Exactly because the compositor is in control of the complete display device and not just little pieces of it, it can see what is happening and it can mitigate problems. If you have more unreliable components needing access to display, make those clients to the compositor, so they can crash and malfunction on their own without potentially killing the whole display device. If you are as concerned about latency as XR people are, then use DRM leases.
Also, what if your virtual KMS driver has a bug? Restarting the kernel is much harder that restarting a userspace compositor that hands out DRM leases.
The userspace compositor could even be such that it does nothing more than handing out DRM leases. However, DRM leases have the problem that there is no single entity responsible for keeping the display device working, but that responsibility is split between several processes and none of them sees the whole picture.
Btw. VKMS is an existing DRM driver, so your name choice is conflicting.
Thanks, pq
On Tue, Jun 22, 2021 at 10:12 AM Pekka Paalanen ppaalanen@gmail.com wrote:
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
Virtual DRM splits the overlay planes of a display controller into multiple virtual devices to allow each plane to be accessed by each process.
This makes it possible to overlay images output from multiple processes on a display. For example, one process displays the camera image without compositor while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
We want to separate the resources from the master in units of planes, so we proposed virtual DRM. I think the advantage of vDRM is that you can use general DRM APIs in userland.
You do that with DRM leases too.
I don't think that's not the way to go, at least not by touching each single display driver, and not hardcoding the planes in DT.
Thank you for comment. I will reconsider about DT.
What's the actual use case you're doing that for ? Why not using some userland compositor ?
I think when latency is important (e.g., AR, VR, for displaying camera images in IVI systems), there may be use cases where the compositor cannot be used. Normally, when the image is passed through the compositor, it is displayed after 2 VSYNC at most, because the compositor combines the image with VSYNC synchronization. On the other hand, if we use vDRM, the image will be displayed at the next VSYNC, so it will be displayed after 1 VSYNC at most.
As I said in my other email, this is false in the general sense.
Also, since the compositor is a single point of failure, we may not want to make it dependent on it.
This... I'm not quite sure I buy it. If any of all the programs using virtual KMS crashes, you still lose some crucial components from your display. Maybe that program, while crashing, uploads such a bad state to its very own KMS plane, that it causes other KMS planes to malfunction. Then you need to detect this situation and still restart everything, not just the crashed program.
This, a hundred times. At least in general it's impossible to guarantee resource isolation between different parts of a kms device - everything is shared at least in some driver in funny ways.
The only thing we try to guarantee is that if you keep flipping the same plane with same pixel format, stride, offset, absolutely everything except the memory block unchanged, then that's guaranteed to work. Everything else is off the table.
This is why the drm-lease design ended up with revoke support, because if something goes wrong a superior instance (the compositor, the kernel can't decide that for userspace) needs to decide whom to shoot and revoke their access.
I would think a userspace compositor approach is actually more reliable. You write the compositor to be extremely robust. Exactly because the compositor is in control of the complete display device and not just little pieces of it, it can see what is happening and it can mitigate problems. If you have more unreliable components needing access to display, make those clients to the compositor, so they can crash and malfunction on their own without potentially killing the whole display device. If you are as concerned about latency as XR people are, then use DRM leases.
Also, what if your virtual KMS driver has a bug? Restarting the kernel is much harder that restarting a userspace compositor that hands out DRM leases.
The userspace compositor could even be such that it does nothing more than handing out DRM leases. However, DRM leases have the problem that there is no single entity responsible for keeping the display device working, but that responsibility is split between several processes and none of them sees the whole picture.
Yeah I think a compositor for this use-case, written in Rust and heavily audited/proofed is probably a lot more reliable than cobbling ill-defined kernel driver code on top of barely-defined hw semantics in resource-sharing cases.
Btw. VKMS is an existing DRM driver, so your name choice is conflicting.
Yeah that too :-) -Daniel
Hi, Thank you all for your comments.
On 2021/06/22 17:12, Pekka Paalanen wrote:
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
Virtual DRM splits the overlay planes of a display controller into multiple virtual devices to allow each plane to be accessed by each process.
This makes it possible to overlay images output from multiple processes on a display. For example, one process displays the camera image without compositor while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
In this use case, I understand that this is not possible with DRM lease, am I wrong? I understand that it’s not possible to lease a plane and update planes on the same output independently from different processes in current DRM lease.
If this is correct, what do you think of adding support for plane leases to the DRM lease to handle this case?
Thanks, Tomohito Esaki
On Wed, 23 Jun 2021 15:56:05 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Thank you all for your comments.
On 2021/06/22 17:12, Pekka Paalanen wrote:
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
Virtual DRM splits the overlay planes of a display controller into multiple virtual devices to allow each plane to be accessed by each process.
This makes it possible to overlay images output from multiple processes on a display. For example, one process displays the camera image without compositor while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
In this use case, I understand that this is not possible with DRM lease, am I wrong? I understand that it’s not possible to lease a plane and update planes on the same output independently from different processes in current DRM lease.
If this is correct, what do you think of adding support for plane leases to the DRM lease to handle this case?
Hi,
I would love to see support added for leasing individual planes, especially to replace the virtual DRM proposal which seems to be eradicating everything that atomic modesetting and nuclear pageflip have built over the many years.
However, please note that "on the same output independently" is physically impossible. Semantically, the planes define what a CRTC scans out, and the CRTC defines the scanout timings. Therefore it is not possible to update individual planes independently, they will all always share the timings of the CRTC.
That combined with KMS not allowing multiple updates to be queued at the same time for the same CRTC (atomic commits and legacy pageflips returning EBUSY) makes the plane updates very much inter-dependent.
If you want to avoid EBUSY and have planes update on the vblank you intended, you really need a userspace compositor to pull everything together *before* submitting anything to the kernel.
Thanks, pq
On 2021/06/23 17:39, Pekka Paalanen wrote:
On Wed, 23 Jun 2021 15:56:05 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Thank you all for your comments.
On 2021/06/22 17:12, Pekka Paalanen wrote:
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
Virtual DRM splits the overlay planes of a display controller into multiple virtual devices to allow each plane to be accessed by each process.
This makes it possible to overlay images output from multiple processes on a display. For example, one process displays the camera image without compositor while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
In this use case, I understand that this is not possible with DRM lease, am I wrong? I understand that it’s not possible to lease a plane and update planes on the same output independently from different processes in current DRM lease.
If this is correct, what do you think of adding support for plane leases to the DRM lease to handle this case?
Hi,
I would love to see support added for leasing individual planes, especially to replace the virtual DRM proposal which seems to be eradicating everything that atomic modesetting and nuclear pageflip have built over the many years.
However, please note that "on the same output independently" is physically impossible. Semantically, the planes define what a CRTC scans out, and the CRTC defines the scanout timings. Therefore it is not possible to update individual planes independently, they will all always share the timings of the CRTC.
That combined with KMS not allowing multiple updates to be queued at the same time for the same CRTC (atomic commits and legacy pageflips returning EBUSY) makes the plane updates very much inter-dependent.
If you want to avoid EBUSY and have planes update on the vblank you intended, you really need a userspace compositor to pull everything together *before* submitting anything to the kernel.
Hi,
Thank you for your comments and advice. I will consider leasing a plane.
Thanks, Esaki
On Wed, 23 Jun 2021 18:22:47 +0900 Esaki Tomohito etom@igel.co.jp wrote:
On 2021/06/23 17:39, Pekka Paalanen wrote:
On Wed, 23 Jun 2021 15:56:05 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Thank you all for your comments.
On 2021/06/22 17:12, Pekka Paalanen wrote:
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
On 21.06.21 08:27, Tomohito Esaki wrote:
Hi,
> Virtual DRM splits the overlay planes of a display controller into multiple > virtual devices to allow each plane to be accessed by each process. > > This makes it possible to overlay images output from multiple processes on a > display. For example, one process displays the camera image without compositor > while another process overlays the UI.
Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
In this use case, I understand that this is not possible with DRM lease, am I wrong? I understand that it’s not possible to lease a plane and update planes on the same output independently from different processes in current DRM lease.
If this is correct, what do you think of adding support for plane leases to the DRM lease to handle this case?
Hi,
I would love to see support added for leasing individual planes, especially to replace the virtual DRM proposal which seems to be eradicating everything that atomic modesetting and nuclear pageflip have built over the many years.
However, please note that "on the same output independently" is physically impossible. Semantically, the planes define what a CRTC scans out, and the CRTC defines the scanout timings. Therefore it is not possible to update individual planes independently, they will all always share the timings of the CRTC.
That combined with KMS not allowing multiple updates to be queued at the same time for the same CRTC (atomic commits and legacy pageflips returning EBUSY) makes the plane updates very much inter-dependent.
If you want to avoid EBUSY and have planes update on the vblank you intended, you really need a userspace compositor to pull everything together *before* submitting anything to the kernel.
Hi,
Thank you for your comments and advice. I will consider leasing a plane.
Hi,
I wish you considered a userspace compositor first, once more, with passion.
It does not need to be Weston, and it does not need to use Wayland. Just a userspace daemon that owns the whole display device and somehow talks to whatever else wants stuff on screen.
I have not seen any evidence that leasing individual planes would do you any good. I can easily see it doing you harm. I'm only saying that it would be better than the virtual DRM proposal if you absolutely have to go there. Please, consider not going there at all.
"On the same output independently" is not possible for the very simple reason that the pixel data needs to be streamed serially to a monitor.
Thanks, pq
On 2021/06/23 20:41, Pekka Paalanen wrote:
On Wed, 23 Jun 2021 18:22:47 +0900 Esaki Tomohito etom@igel.co.jp wrote:
On 2021/06/23 17:39, Pekka Paalanen wrote:
On Wed, 23 Jun 2021 15:56:05 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Thank you all for your comments.
On 2021/06/22 17:12, Pekka Paalanen wrote:
On Tue, 22 Jun 2021 13:03:39 +0900 Esaki Tomohito etom@igel.co.jp wrote:
Hi, Enrico Weigelt Thank you for reply.
On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote: > On 21.06.21 08:27, Tomohito Esaki wrote: > > Hi, > >> Virtual DRM splits the overlay planes of a display controller into multiple >> virtual devices to allow each plane to be accessed by each process. >> >> This makes it possible to overlay images output from multiple processes on a >> display. For example, one process displays the camera image without compositor >> while another process overlays the UI. > > Are you attempting to create an simple in-kernel compositor ?
I think the basic idea is the same as DRMlease.
Hi,
indeed. Why not use DRM leases instead?
In this use case, I understand that this is not possible with DRM lease, am I wrong? I understand that it’s not possible to lease a plane and update planes on the same output independently from different processes in current DRM lease.
If this is correct, what do you think of adding support for plane leases to the DRM lease to handle this case?
Hi,
I would love to see support added for leasing individual planes, especially to replace the virtual DRM proposal which seems to be eradicating everything that atomic modesetting and nuclear pageflip have built over the many years.
However, please note that "on the same output independently" is physically impossible. Semantically, the planes define what a CRTC scans out, and the CRTC defines the scanout timings. Therefore it is not possible to update individual planes independently, they will all always share the timings of the CRTC.
That combined with KMS not allowing multiple updates to be queued at the same time for the same CRTC (atomic commits and legacy pageflips returning EBUSY) makes the plane updates very much inter-dependent.
If you want to avoid EBUSY and have planes update on the vblank you intended, you really need a userspace compositor to pull everything together *before* submitting anything to the kernel.
Hi,
Thank you for your comments and advice. I will consider leasing a plane.
Hi,
I wish you considered a userspace compositor first, once more, with passion.
It does not need to be Weston, and it does not need to use Wayland. Just a userspace daemon that owns the whole display device and somehow talks to whatever else wants stuff on screen.
I have not seen any evidence that leasing individual planes would do you any good. I can easily see it doing you harm. I'm only saying that it would be better than the virtual DRM proposal if you absolutely have to go there. Please, consider not going there at all.
"On the same output independently" is not possible for the very simple reason that the pixel data needs to be streamed serially to a monitor.
Hi,
Thank you for your advice. Once again, I'll consider a userspace compositor first.
Best regards Esaki
dri-devel@lists.freedesktop.org
-
Daniel Vetter -
Esaki Tomohito -
Pekka Paalanen