Hi All,
I'm not sure if $subject was a conscious design decision, or an oversight, but that does not really matter.
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
1) Letting the VM-viewer window-system draw the cursor as it normally would draw it.
2) Giving absolute coordinates of the mouse to the VM by e.g. emulating an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
2) Means that any coordinates the window-system inside the VM passes to the VM's gfx-card for where to draw the cursor are basically totally ignored to avoid lag / flicker (and to not have to grab the cursor / confine it to the VM-viewer window and to not have to warp the pointer).
This means that the offset added to the coordinates by e.g. mutter to emulate the hotspot are ignored. For Seamless mouse mode to keep working properly the window-system inside the VM need to pass the VM's gfx-card the correct hotspot when setting the cursor. Which currently is not possible when restricting oneself to the atomic APIs.
Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094 Where this is currently being tracked from the mutter side. Mutter internally has both atomic and legacy paths. The plan for now is to push the hotspot-emulation by shifting coordinates thing into the atomic path, fixing seamless mouse mode when running in legacy mode, combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from using atomic mode.
This is of course a workaround, eventually we would like to see the atomic API extended to allow passing the cursor hot spot.
I'm not really familiar enough with the atomic API to come up with an API design for this, but if there are suggestions on how this should look like from the uAPI side then I can take a shot at implementing this (and hooking it up in mutter's atomic code paths to test it).
Regards,
Hans
p.s.
Before people start discussing how the VM / VM-viewer is broken here and the VM needs to be fixed. Seamless mouse mode exists for at least a decade and has worked fine during this entire decade. It also works fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
Also this problem reproduces with 2 completely independent VM code-bases, it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would not be surprised if other hypervisors are also affected.
And on the API consumer side this problem has been triggered by both mutter and Weston.
Hi,
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
This would also allow the compositor running inside the VM to correctly have control over the cursor position, which is necessary for pointer constraints.
Thanks,
Simon
Hi,
On 3/18/20 3:38 PM, Simon Ser wrote:
Hi,
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
Also note that a subsurface is a Wayland specific solution, where as the VM-viewer may be running on X11, Windows, Mac OS, etc.
This would also allow the compositor running inside the VM to correctly have control over the cursor position, which is necessary for pointer constraints.
Vms basically have 2 mouse modes:
1) Seamless, this works well for all apps which don't do weird things with the cursor. This is what 99% of users want
2) Grab/confine the mouse on the first click inside the VM-viewer window and constantly warp it to the center so that it can move "endlessly" combined with drawing the VM's mouse cursor as a software sprite.
Combined with a special key combo to release the cursor and allow it to leave the VM window in case the user wants to interact with anything else on their desktop. AKA the "this user experience sucks" mode which sometimes is necessary for guests which don't support absolute input coordinates, or for special use cases.
Mode 2. can be used in case of apps inside the guest which want need to constrain the pointer to stay inside a part-of the vm-viewer window, note that the most prominent example of such apps are VM-viewer's themselves and the whole purpose of seamless mode is to not need this less then ideal user experience mode.
Anyways as I mentioned in the p.s. to my original mail already, this is exactly NOT the kind of feedback I'm looking for. Seamless mode exists, it has done so for at least a decade, probably a lot longer.
It works everywhere, across multiple platforms and hypervisors, except with the KMS atomic API. The need to set hotspot coordinates is not something which is up to discussion from my pov. What is up for discussion is how to extend the KMS atomic API to allow this.
Regards,
Hans
Hi,
On 3/18/20 3:38 PM, Simon Ser wrote:
Hi,
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
The video output has latency anyway. Using the host cursor will make the primary plane and cursor plane desynchronized.
Also note that a subsurface is a Wayland specific solution, where as the VM-viewer may be running on X11, Windows, Mac OS, etc.
I'm sure other window systems have similar solutions. You could always blend the cursor yourself.
This would also allow the compositor running inside the VM to correctly have control over the cursor position, which is necessary for pointer constraints.
Vms basically have 2 mouse modes:
- Seamless, this works well for all apps which don't do weird things
with the cursor. This is what 99% of users want
- Grab/confine the mouse on the first click inside the VM-viewer window
and constantly warp it to the center so that it can move "endlessly" combined with drawing the VM's mouse cursor as a software sprite.
Combined with a special key combo to release the cursor and allow it to leave the VM window in case the user wants to interact with anything else on their desktop. AKA the "this user experience sucks" mode which sometimes is necessary for guests which don't support absolute input coordinates, or for special use cases.
Mode 2. can be used in case of apps inside the guest which want need to constrain the pointer to stay inside a part-of the vm-viewer window, note that the most prominent example of such apps are VM-viewer's themselves and the whole purpose of seamless mode is to not need this less then ideal user experience mode.
If you don't care about synchronization and breaking pointer constraints in the guest, yes a new KMS plane property will be required. This sure sounds like abusing the KMS interface though.
Anyways as I mentioned in the p.s. to my original mail already, this is exactly NOT the kind of feedback I'm looking for. Seamless mode exists, it has done so for at least a decade, probably a lot longer.
It works everywhere, across multiple platforms and hypervisors, except with the KMS atomic API. The need to set hotspot coordinates is not something which is up to discussion from my pov. What is up for discussion is how to extend the KMS atomic API to allow this.
That's not how it works.
I'm sorry to say that, but I don't think asking a project to support a feature just because you want that feature is a good mind set. You'll need to convince people maintaining the project that adding the feature is a good idea whether you like it or not.
A new feature is always up for discussion. Atomic KMS is missing the feature, and this can't be seen as a regression, because it never had the feature.
On 2020-03-18 4:22 p.m., Simon Ser wrote:
On 3/18/20 3:38 PM, Simon Ser wrote:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
The video output has latency anyway.
Sounds like you've never tried the two different modes yourself? :) IME it makes a big difference even with a local VM. Even very little latency can make the cursor feel awkward, like it's being held back by a rubber band or something.
Hi,
On 3/19/20 12:35 PM, Michel Dänzer wrote:
On 2020-03-18 4:22 p.m., Simon Ser wrote:
On 3/18/20 3:38 PM, Simon Ser wrote:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
The video output has latency anyway.
Sounds like you've never tried the two different modes yourself? :) IME it makes a big difference even with a local VM. Even very little latency can make the cursor feel awkward, like it's being held back by a rubber band or something.
Right not to mention that the latency may be variable, so the cursor moves in a jittery fashion instead of having it move smoothly matching the smooth way a user normally moves the mouse.
This totally wrecks hand-eye coordination and is just plain awefull.
Regards,
Hans
On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/19/20 12:35 PM, Michel Dänzer wrote:
On 2020-03-18 4:22 p.m., Simon Ser wrote:
On 3/18/20 3:38 PM, Simon Ser wrote:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
The video output has latency anyway.
Sounds like you've never tried the two different modes yourself? :) IME it makes a big difference even with a local VM. Even very little latency can make the cursor feel awkward, like it's being held back by a rubber band or something.
Right not to mention that the latency may be variable, so the cursor moves in a jittery fashion instead of having it move smoothly matching the smooth way a user normally moves the mouse.
This totally wrecks hand-eye coordination and is just plain awefull.
I have experienced it, and while it is painful, I prefer that pain over the pain of accidentally clicking something that was not transmitted to the remote display yet.
Therefore I think the best user experience is to use both types of cursor at the same time: the remote desktop or VM viewer paints the local cursor as an aid, like a phantom, and the cursor from inside the VM is also visible with the latency it naturally has. That means I could actually see that the screen has caught up with my motions before I click something.
Thanks, pq
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2020-03-19 1:54 p.m., Pekka Paalanen wrote:
On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede hdegoede@redhat.com wrote:
On 3/19/20 12:35 PM, Michel Dänzer wrote:
On 2020-03-18 4:22 p.m., Simon Ser wrote:
On 3/18/20 3:38 PM, Simon Ser wrote:
> 1) Letting the VM-viewer window-system draw the cursor > as it normally would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
The video output has latency anyway.
Sounds like you've never tried the two different modes yourself? :) IME it makes a big difference even with a local VM. Even very little latency can make the cursor feel awkward, like it's being held back by a rubber band or something.
Right not to mention that the latency may be variable, so the cursor moves in a jittery fashion instead of having it move smoothly matching the smooth way a user normally moves the mouse.
This totally wrecks hand-eye coordination and is just plain awefull.
I have experienced it, and while it is painful, I prefer that pain over the pain of accidentally clicking something that was not transmitted to the remote display yet.
Unless you mean clicking something while the cursor is moving, not sure how seamless vs not affects this, since the delay of something appearing on the remote display should be the same in both cases?
Therefore I think the best user experience is to use both types of cursor at the same time: the remote desktop or VM viewer paints the local cursor as an aid, like a phantom, and the cursor from inside the VM is also visible with the latency it naturally has. That means I could actually see that the screen has caught up with my motions before I click something.
I'd expect that to result in "duplicate cursor" bug reports — I'd certainly consider it a bug with my user hat on.
- -- Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and X developer
On Thu, 19 Mar 2020 14:51:41 +0100 Michel Dänzer michel@daenzer.net wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2020-03-19 1:54 p.m., Pekka Paalanen wrote:
On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede hdegoede@redhat.com wrote:
On 3/19/20 12:35 PM, Michel Dänzer wrote:
On 2020-03-18 4:22 p.m., Simon Ser wrote:
On 3/18/20 3:38 PM, Simon Ser wrote: > >> 1) Letting the VM-viewer window-system draw the cursor >> as it normally would draw it. > > Why is this important? Can't the VM viewer hide the > cursor and use a sub-surface to manually draw the cursor > plane configured by the guest?
Because then moving the cursor as seen by the user requires a round trip through the VM and that adds latency, esp. when the VM viewer is viewing a VM which is running somewhere else over the network.
The video output has latency anyway.
Sounds like you've never tried the two different modes yourself? :) IME it makes a big difference even with a local VM. Even very little latency can make the cursor feel awkward, like it's being held back by a rubber band or something.
Right not to mention that the latency may be variable, so the cursor moves in a jittery fashion instead of having it move smoothly matching the smooth way a user normally moves the mouse.
This totally wrecks hand-eye coordination and is just plain awefull.
I have experienced it, and while it is painful, I prefer that pain over the pain of accidentally clicking something that was not transmitted to the remote display yet.
Unless you mean clicking something while the cursor is moving, not sure how seamless vs not affects this, since the delay of something appearing on the remote display should be the same in both cases?
How do you know if the cursor is still moving or not, if you don't see the real cursor but only the fake cursor?
I move the mouse, then it takes 0.1 - 10 seconds for the real cursor to reach where I moved it. Only after that I see what the motion caused. Then I can do the next thing.
The mouse motion / cursor response gives me continuous feedback, so that I don't run too far ahead of the remote end.
Therefore I think the best user experience is to use both types of cursor at the same time: the remote desktop or VM viewer paints the local cursor as an aid, like a phantom, and the cursor from inside the VM is also visible with the latency it naturally has. That means I could actually see that the screen has caught up with my motions before I click something.
I'd expect that to result in "duplicate cursor" bug reports — I'd certainly consider it a bug with my user hat on.
The point is to make the phantom cursor look like a phantom cursor, so it cannot be mistaken as the real cursor. It doesn't even need to use the same cursor image as the real cursor, it could be just a translucent spot.
Thanks, pq
Hi,
On 3/19/20 3:48 PM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 14:51:41 +0100 Michel Dänzer michel@daenzer.net wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2020-03-19 1:54 p.m., Pekka Paalanen wrote:
On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede hdegoede@redhat.com wrote:
On 3/19/20 12:35 PM, Michel Dänzer wrote:
On 2020-03-18 4:22 p.m., Simon Ser wrote:
> > On 3/18/20 3:38 PM, Simon Ser wrote: >> >>> 1) Letting the VM-viewer window-system draw the cursor >>> as it normally would draw it. >> >> Why is this important? Can't the VM viewer hide the >> cursor and use a sub-surface to manually draw the cursor >> plane configured by the guest? > > Because then moving the cursor as seen by the user requires > a round trip through the VM and that adds latency, esp. > when the VM viewer is viewing a VM which is running > somewhere else over the network.
The video output has latency anyway.
Sounds like you've never tried the two different modes yourself? :) IME it makes a big difference even with a local VM. Even very little latency can make the cursor feel awkward, like it's being held back by a rubber band or something.
Right not to mention that the latency may be variable, so the cursor moves in a jittery fashion instead of having it move smoothly matching the smooth way a user normally moves the mouse.
This totally wrecks hand-eye coordination and is just plain awefull.
I have experienced it, and while it is painful, I prefer that pain over the pain of accidentally clicking something that was not transmitted to the remote display yet.
Unless you mean clicking something while the cursor is moving, not sure how seamless vs not affects this, since the delay of something appearing on the remote display should be the same in both cases?
How do you know if the cursor is still moving or not, if you don't see the real cursor but only the fake cursor?
I move the mouse, then it takes 0.1 - 10 seconds for the real cursor to reach where I moved it. Only after that I see what the motion caused. Then I can do the next thing.
The mouse motion / cursor response gives me continuous feedback, so that I don't run too far ahead of the remote end.
Therefore I think the best user experience is to use both types of cursor at the same time: the remote desktop or VM viewer paints the local cursor as an aid, like a phantom, and the cursor from inside the VM is also visible with the latency it naturally has. That means I could actually see that the screen has caught up with my motions before I click something.
I'd expect that to result in "duplicate cursor" bug reports — I'd certainly consider it a bug with my user hat on.
The point is to make the phantom cursor look like a phantom cursor, so it cannot be mistaken as the real cursor. It doesn't even need to use the same cursor image as the real cursor, it could be just a translucent spot.
I think you are trying to fix a non existing problem here, accept for say a game of whack a mode, the target which the user is trying to hit is typically static and the VM will receive and process the coordinates where the user stops the motion, before it will receive and process the click. So unless say an icon on a toolbar for some reason is a moving target there is no problem here.
Regards,
Hans
On Wed, 18 Mar 2020 14:38:48 +0000 Simon Ser contact@emersion.fr wrote:
Hi,
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest?
This would also allow the compositor running inside the VM to correctly have control over the cursor position, which is necessary for pointer constraints.
Aren't pointer motion events delivered in absolute and not relative coordinates in the "seamless mode"?
Do we do pointer constraints also with absolute motion?
Thanks, pq
On Thursday, March 19, 2020 10:57 AM, Pekka Paalanen ppaalanen@gmail.com wrote:
On Wed, 18 Mar 2020 14:38:48 +0000 Simon Ser contact@emersion.fr wrote:
Hi,
- Letting the VM-viewer window-system draw the cursor as it normally would draw it.
Why is this important? Can't the VM viewer hide the cursor and use a sub-surface to manually draw the cursor plane configured by the guest? This would also allow the compositor running inside the VM to correctly have control over the cursor position, which is necessary for pointer constraints.
Aren't pointer motion events delivered in absolute and not relative coordinates in the "seamless mode"?
Do we do pointer constraints also with absolute motion?
Some apps do pointer constraints with absolute motion, e.g. games and VNC/VM viewers. It's usually used to prevent the user from leaving the constrained surface by mistake.
Hi,
On 3/18/20 3:28 PM, Hans de Goede wrote:
Hi All,
I'm not sure if $subject was a conscious design decision, or an oversight, but that does not really matter.
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
- Means that any coordinates the window-system inside the VM passes to
This should be: "1) Means that ..."
the VM's gfx-card for where to draw the cursor are basically totally ignored to avoid lag / flicker (and to not have to grab the cursor / confine it to the VM-viewer window and to not have to warp the pointer).
This means that the offset added to the coordinates by e.g. mutter to emulate the hotspot are ignored. For Seamless mouse mode to keep working properly the window-system inside the VM need to pass the VM's gfx-card the correct hotspot when setting the cursor. Which currently is not possible when restricting oneself to the atomic APIs.
Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094 Where this is currently being tracked from the mutter side. Mutter internally has both atomic and legacy paths. The plan for now is to push the hotspot-emulation by shifting coordinates thing into the atomic path, fixing seamless mouse mode when running in legacy mode, combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from using atomic mode.
This is of course a workaround, eventually we would like to see the atomic API extended to allow passing the cursor hot spot.
I'm not really familiar enough with the atomic API to come up with an API design for this, but if there are suggestions on how this should look like from the uAPI side then I can take a shot at implementing this (and hooking it up in mutter's atomic code paths to test it).
Correction, I misunderstood the mutter devs that mutter already has support for atomic, this support is in the works, but not yet there yet. The internal mutter API has been reworked to closer resemble the atomic APIs and that introduced the hotspot emulation and broke seamless mouse mode in VMs.
Mutter does eventually want to switch to using the atomic APIs and then this will become a real problem. This does mean that testing any UAPI extension we come up with for this will be harder then I thought.
Regards,
Hans
p.s.
Before people start discussing how the VM / VM-viewer is broken here and the VM needs to be fixed. Seamless mouse mode exists for at least a decade and has worked fine during this entire decade. It also works fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
Also this problem reproduces with 2 completely independent VM code-bases, it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would not be surprised if other hypervisors are also affected.
And on the API consumer side this problem has been triggered by both mutter and Weston.
On Wed, Mar 18, 2020 at 3:29 PM Hans de Goede hdegoede@redhat.com wrote:
Hi All,
I'm not sure if $subject was a conscious design decision, or an oversight, but that does not really matter.
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Fully aware, we simply never had userspace for this. So - type weston patch - type kernel patch (iirc this might be some work of moving the hotspot coordinates from struct drm_plane to drm_plane_state) - wire up atomic property (it'd just do "CURSOR_HOT_X/Y" and be done with it, only attached to cursor planes that support it ofc) - I think all drivers supporting vm cursor hotspot are now atomic, so would be good to garbage-collect the old legacy infrastructure completely. legacy cursor ioctl will remap to the atomic call transparently like we do now already for everything else - type igt - get it all reviewed and merged.
0 design reasons for omitting this, simply Not Yet Done (tm).
Cheers, Daniel
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
- Means that any coordinates the window-system inside the VM passes to
the VM's gfx-card for where to draw the cursor are basically totally ignored to avoid lag / flicker (and to not have to grab the cursor / confine it to the VM-viewer window and to not have to warp the pointer).
This means that the offset added to the coordinates by e.g. mutter to emulate the hotspot are ignored. For Seamless mouse mode to keep working properly the window-system inside the VM need to pass the VM's gfx-card the correct hotspot when setting the cursor. Which currently is not possible when restricting oneself to the atomic APIs.
Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094 Where this is currently being tracked from the mutter side. Mutter internally has both atomic and legacy paths. The plan for now is to push the hotspot-emulation by shifting coordinates thing into the atomic path, fixing seamless mouse mode when running in legacy mode, combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from using atomic mode.
This is of course a workaround, eventually we would like to see the atomic API extended to allow passing the cursor hot spot.
I'm not really familiar enough with the atomic API to come up with an API design for this, but if there are suggestions on how this should look like from the uAPI side then I can take a shot at implementing this (and hooking it up in mutter's atomic code paths to test it).
Regards,
Hans
p.s.
Before people start discussing how the VM / VM-viewer is broken here and the VM needs to be fixed. Seamless mouse mode exists for at least a decade and has worked fine during this entire decade. It also works fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
Also this problem reproduces with 2 completely independent VM code-bases, it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would not be surprised if other hypervisors are also affected.
And on the API consumer side this problem has been triggered by both mutter and Weston.
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi,
On 3/18/20 4:09 PM, Daniel Vetter wrote:
On Wed, Mar 18, 2020 at 3:29 PM Hans de Goede hdegoede@redhat.com wrote:
Hi All,
I'm not sure if $subject was a conscious design decision, or an oversight, but that does not really matter.
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Fully aware, we simply never had userspace for this. So
- type weston patch
- type kernel patch (iirc this might be some work of moving the
hotspot coordinates from struct drm_plane to drm_plane_state)
- wire up atomic property (it'd just do "CURSOR_HOT_X/Y" and be done
with it, only attached to cursor planes that support it ofc)
- I think all drivers supporting vm cursor hotspot are now atomic, so
would be good to garbage-collect the old legacy infrastructure completely. legacy cursor ioctl will remap to the atomic call transparently like we do now already for everything else
- type igt
- get it all reviewed and merged.
0 design reasons for omitting this, simply Not Yet Done (tm).
Ok, sounds good, thank you. I've put this on my TODO list, but below a bunch of other higher prio items, so it may be a while before I get around to this.
Regards,
Hans
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi,
is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
If so, sounds like hilarity would ensue with Weston.
Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol.
It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client.
No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Thanks, pq
- Means that any coordinates the window-system inside the VM passes to
the VM's gfx-card for where to draw the cursor are basically totally ignored to avoid lag / flicker (and to not have to grab the cursor / confine it to the VM-viewer window and to not have to warp the pointer).
This means that the offset added to the coordinates by e.g. mutter to emulate the hotspot are ignored. For Seamless mouse mode to keep working properly the window-system inside the VM need to pass the VM's gfx-card the correct hotspot when setting the cursor. Which currently is not possible when restricting oneself to the atomic APIs.
Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094 Where this is currently being tracked from the mutter side. Mutter internally has both atomic and legacy paths. The plan for now is to push the hotspot-emulation by shifting coordinates thing into the atomic path, fixing seamless mouse mode when running in legacy mode, combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from using atomic mode.
This is of course a workaround, eventually we would like to see the atomic API extended to allow passing the cursor hot spot.
I'm not really familiar enough with the atomic API to come up with an API design for this, but if there are suggestions on how this should look like from the uAPI side then I can take a shot at implementing this (and hooking it up in mutter's atomic code paths to test it).
Regards,
Hans
p.s.
Before people start discussing how the VM / VM-viewer is broken here and the VM needs to be fixed. Seamless mouse mode exists for at least a decade and has worked fine during this entire decade. It also works fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
Also this problem reproduces with 2 completely independent VM code-bases, it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would not be surprised if other hypervisors are also affected.
And on the API consumer side this problem has been triggered by both mutter and Weston.
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi,
On 3/19/20 11:00 AM, Pekka Paalanen wrote:
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi,
is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
Yes and no, it is not the VM-viewer which is hiding what the display-server inside the VM has put on the KMS cursor plane, the VM-viewer negotiates seamless mouse mode with the hypervisor and then the hypervisor just ignores the cursor-plane except for sending "sprite" changes to the VM-viewer.
If so, sounds like hilarity would ensue with Weston.
Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol.
It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client.
No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Yes it sounds like this will break with Weston, note that it already is broken in Weston, if you run e.g. Fedora 32 beta + its Weston package inside a VirtualBox VM then start gnome-terminal (so that you get a caret cursor instead of the default one) and try to select text. This will result in the wrong text being displayed because Weston does not relay cursor hotspot info to the GPU, also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
Where the symptoms of this are described in more detail (they are identical for Weston and mutter).
Fixing this will require the discussed KMS atomic API changes and also changes on the Weston and mutter side to pass through the hotspot info.
Mutter used to do this, but mutter's internal API-s have been reworked to closer match the atomic KMS API and as part of this passing the hotspot coords was dropped, this is being fixed for the legacy KMS API code-paths now (which atm are the only code paths, atomic support has no landed yet): https://gitlab.gnome.org/GNOME/mutter/merge_requests/1136
Regards,
Hans
- Means that any coordinates the window-system inside the VM passes to
the VM's gfx-card for where to draw the cursor are basically totally ignored to avoid lag / flicker (and to not have to grab the cursor / confine it to the VM-viewer window and to not have to warp the pointer).
This means that the offset added to the coordinates by e.g. mutter to emulate the hotspot are ignored. For Seamless mouse mode to keep working properly the window-system inside the VM need to pass the VM's gfx-card the correct hotspot when setting the cursor. Which currently is not possible when restricting oneself to the atomic APIs.
Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094 Where this is currently being tracked from the mutter side. Mutter internally has both atomic and legacy paths. The plan for now is to push the hotspot-emulation by shifting coordinates thing into the atomic path, fixing seamless mouse mode when running in legacy mode, combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from using atomic mode.
This is of course a workaround, eventually we would like to see the atomic API extended to allow passing the cursor hot spot.
I'm not really familiar enough with the atomic API to come up with an API design for this, but if there are suggestions on how this should look like from the uAPI side then I can take a shot at implementing this (and hooking it up in mutter's atomic code paths to test it).
Regards,
Hans
p.s.
Before people start discussing how the VM / VM-viewer is broken here and the VM needs to be fixed. Seamless mouse mode exists for at least a decade and has worked fine during this entire decade. It also works fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
Also this problem reproduces with 2 completely independent VM code-bases, it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would not be surprised if other hypervisors are also affected.
And on the API consumer side this problem has been triggered by both mutter and Weston.
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
On Thu, 19 Mar 2020 12:49:27 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/19/20 11:00 AM, Pekka Paalanen wrote:
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi,
is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
Yes and no, it is not the VM-viewer which is hiding what the display-server inside the VM has put on the KMS cursor plane, the VM-viewer negotiates seamless mouse mode with the hypervisor and then the hypervisor just ignores the cursor-plane except for sending "sprite" changes to the VM-viewer.
Hi,
I don't think I understand what you're saying, but I assume that I was right in that the VM cursor plane content will not be shown always exactly in the very position the compositor inside the VM puts it. Maybe the example further below explain the issue I envision.
If so, sounds like hilarity would ensue with Weston.
Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol.
It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client.
No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Yes it sounds like this will break with Weston, note that it already is broken in Weston, if you run e.g. Fedora 32 beta + its Weston package inside a VirtualBox VM then start gnome-terminal (so that you get a caret cursor instead of the default one) and try to select text. This will result in the wrong text being displayed because Weston does not relay cursor hotspot info to the GPU, also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
Where the symptoms of this are described in more detail (they are identical for Weston and mutter).
Right, that's the problem with the hotspot.
Fixing this will require the discussed KMS atomic API changes and also changes on the Weston and mutter side to pass through the hotspot info.
The problem I am referring to is that to the user looking at the VM-viewer, suddenly an arbitrary application window (e.g. weston-simple-shm) starts to act as if it was the cursor, when there is no real cursor shown. You have a random window unexpectedly moving around, as if you had started dragging it around with your mouse.
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
It sounds like your VM-viewer makes the assumption that the pointer input device it delivers to the VM is the one that will control the KMS cursor plane position inside the VM. Is that right?
What if the desktop inside the VM is controlled by a remote, e.g. VNC? Then the input events to the VM are completely unrelated to the expected motion of the cursor. Do you just tell the users to stop using the seamless mode in that case?
What if display servers stop using the cursor plane completely, because people may hit a case where a VM-viewer makes the wrong assumption about which input device is associated to which cursor plane inside the VM?
Thanks, pq
Mutter used to do this, but mutter's internal API-s have been reworked to closer match the atomic KMS API and as part of this passing the hotspot coords was dropped, this is being fixed for the legacy KMS API code-paths now (which atm are the only code paths, atomic support has no landed yet): https://gitlab.gnome.org/GNOME/mutter/merge_requests/1136
Regards,
Hans
On Thu, Mar 19, 2020 at 1:58 PM Pekka Paalanen ppaalanen@gmail.com wrote:
On Thu, 19 Mar 2020 12:49:27 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/19/20 11:00 AM, Pekka Paalanen wrote:
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi,
is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
Yes and no, it is not the VM-viewer which is hiding what the display-server inside the VM has put on the KMS cursor plane, the VM-viewer negotiates seamless mouse mode with the hypervisor and then the hypervisor just ignores the cursor-plane except for sending "sprite" changes to the VM-viewer.
Hi,
I don't think I understand what you're saying, but I assume that I was right in that the VM cursor plane content will not be shown always exactly in the very position the compositor inside the VM puts it. Maybe the example further below explain the issue I envision.
If so, sounds like hilarity would ensue with Weston.
Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol.
It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client.
No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Yes it sounds like this will break with Weston, note that it already is broken in Weston, if you run e.g. Fedora 32 beta + its Weston package inside a VirtualBox VM then start gnome-terminal (so that you get a caret cursor instead of the default one) and try to select text. This will result in the wrong text being displayed because Weston does not relay cursor hotspot info to the GPU, also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
Where the symptoms of this are described in more detail (they are identical for Weston and mutter).
Right, that's the problem with the hotspot.
Fixing this will require the discussed KMS atomic API changes and also changes on the Weston and mutter side to pass through the hotspot info.
The problem I am referring to is that to the user looking at the VM-viewer, suddenly an arbitrary application window (e.g. weston-simple-shm) starts to act as if it was the cursor, when there is no real cursor shown. You have a random window unexpectedly moving around, as if you had started dragging it around with your mouse.
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
It sounds like your VM-viewer makes the assumption that the pointer input device it delivers to the VM is the one that will control the KMS cursor plane position inside the VM. Is that right?
What if the desktop inside the VM is controlled by a remote, e.g. VNC? Then the input events to the VM are completely unrelated to the expected motion of the cursor. Do you just tell the users to stop using the seamless mode in that case?
What if display servers stop using the cursor plane completely, because people may hit a case where a VM-viewer makes the wrong assumption about which input device is associated to which cursor plane inside the VM?
Hm that sounds like we should have an additional flag with atomic that enables/disables the hotspot functionality. Then compositors can decide what they want to do with that thing. -Daniel
On Thursday, March 19, 2020 2:16 PM, Daniel Vetter daniel@ffwll.ch wrote:
On Thu, Mar 19, 2020 at 1:58 PM Pekka Paalanen ppaalanen@gmail.com wrote:
On Thu, 19 Mar 2020 12:49:27 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi, On 3/19/20 11:00 AM, Pekka Paalanen wrote:
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot. But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card. Seamless-mode consists of 2 parts:
Letting the VM-viewer window-system draw the cursor as it normally would draw it.
Giving absolute coordinates of the mouse to the VM by e.g. emulating an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi, is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
Yes and no, it is not the VM-viewer which is hiding what the display-server inside the VM has put on the KMS cursor plane, the VM-viewer negotiates seamless mouse mode with the hypervisor and then the hypervisor just ignores the cursor-plane except for sending "sprite" changes to the VM-viewer.
Hi, I don't think I understand what you're saying, but I assume that I was right in that the VM cursor plane content will not be shown always exactly in the very position the compositor inside the VM puts it. Maybe the example further below explain the issue I envision.
If so, sounds like hilarity would ensue with Weston. Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol. It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client. No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Yes it sounds like this will break with Weston, note that it already is broken in Weston, if you run e.g. Fedora 32 beta + its Weston package inside a VirtualBox VM then start gnome-terminal (so that you get a caret cursor instead of the default one) and try to select text. This will result in the wrong text being displayed because Weston does not relay cursor hotspot info to the GPU, also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094 Where the symptoms of this are described in more detail (they are identical for Weston and mutter).
Right, that's the problem with the hotspot.
Fixing this will require the discussed KMS atomic API changes and also changes on the Weston and mutter side to pass through the hotspot info.
The problem I am referring to is that to the user looking at the VM-viewer, suddenly an arbitrary application window (e.g. weston-simple-shm) starts to act as if it was the cursor, when there is no real cursor shown. You have a random window unexpectedly moving around, as if you had started dragging it around with your mouse. The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane. It sounds like your VM-viewer makes the assumption that the pointer input device it delivers to the VM is the one that will control the KMS cursor plane position inside the VM. Is that right? What if the desktop inside the VM is controlled by a remote, e.g. VNC? Then the input events to the VM are completely unrelated to the expected motion of the cursor. Do you just tell the users to stop using the seamless mode in that case? What if display servers stop using the cursor plane completely, because people may hit a case where a VM-viewer makes the wrong assumption about which input device is associated to which cursor plane inside the VM?
Hm that sounds like we should have an additional flag with atomic that enables/disables the hotspot functionality. Then compositors can decide what they want to do with that thing.
A per-plane prop to enable/disable the hotspot would help.
This still doesn't fix issues with multiple input devices, e.g. VNC.
Hi,
On 3/19/20 1:58 PM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 12:49:27 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/19/20 11:00 AM, Pekka Paalanen wrote:
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi,
is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
Yes and no, it is not the VM-viewer which is hiding what the display-server inside the VM has put on the KMS cursor plane, the VM-viewer negotiates seamless mouse mode with the hypervisor and then the hypervisor just ignores the cursor-plane except for sending "sprite" changes to the VM-viewer.
Hi,
I don't think I understand what you're saying, but I assume that I was right in that the VM cursor plane content will not be shown always exactly in the very position the compositor inside the VM puts it.
Right, when seamless mouse mode is enabled the VM's cursor plane will not be shown *at all*, instead the VM-viewer's window-system's cursor plane is shown.
Sprite changes (normal cursor, caret, window-resize cursor, etc.) are forwarded from the VM-s cursor-plane to the VM-viewer which then requests that as cursor to the window-system under which the VM-viewer is running.
Maybe the example further below explain the issue I envision.
If so, sounds like hilarity would ensue with Weston.
Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol.
It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client.
No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Yes it sounds like this will break with Weston, note that it already is broken in Weston, if you run e.g. Fedora 32 beta + its Weston package inside a VirtualBox VM then start gnome-terminal (so that you get a caret cursor instead of the default one) and try to select text. This will result in the wrong text being displayed because Weston does not relay cursor hotspot info to the GPU, also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
Where the symptoms of this are described in more detail (they are identical for Weston and mutter).
Right, that's the problem with the hotspot.
Ack.
Fixing this will require the discussed KMS atomic API changes and also changes on the Weston and mutter side to pass through the hotspot info.
The problem I am referring to is that to the user looking at the VM-viewer, suddenly an arbitrary application window (e.g. weston-simple-shm) starts to act as if it was the cursor, when there is no real cursor shown. You have a random window unexpectedly moving around, as if you had started dragging it around with your mouse.
Correct.
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
Correct.
It sounds like your VM-viewer makes the assumption that the pointer input device it delivers to the VM is the one that will control the KMS cursor plane position inside the VM. Is that right?
Correct.
What if the desktop inside the VM is controlled by a remote, e.g. VNC? Then the input events to the VM are completely unrelated to the expected motion of the cursor. Do you just tell the users to stop using the seamless mode in that case?
A VNC viewer (which is not using seamless mode itself) has the same issue of a mismatch between the cursor position of the window-system it is a client of and the cursor position of the window system inside the VM.
A VNC viewer typically works around this by changing the window-system cursor to transparent and drawing its own cursor, the transparent sprite or disabling of the cursor-plane will get forwarded to the window-system under which the VM-viewer runs so this will work fine.
A real problem though is the absolute input mode, when e.g. the VNC viewer is not using something like seamless mouse mode it will want to do the usual (nasty) tricks of confining the pointer to the window and warping it to the center after each pointer move so that mouse can be moved "endlessly' inside the window. This requires relative input, so emulating a mouse instead of say a drawing tablet, which breaks seamless mode.
So although the way the cursor is drawn typically does not require disabling seamless mode, the input problems do require disabling seamless mode.
In my experience (I have worked on VDI (spice) a couple of years), running a VM-viewer inside a VM is something which not a lot of people do. I guess the fact that this works quite poorly might be one of the reasons people do not do this.
But all VMs I'm familiar with allow disabling seamless mode for compatibility with corner cases like this. This does lead to a seriously degraded user-experience though. For just using an office suit or browser inside a VM seamless mode really is a lot more pleasant for the user. I still remember when seamless mode first got introduced, in the beginning it used to not always work and fixing the fallback to the confine + warp tricks then always was a serious itch which I tried to scratch quickly. Breaking seamless mode also is a good way to quickly accumulate a lot of bug reports from end users in my experience.
What if display servers stop using the cursor plane completely, because people may hit a case where a VM-viewer makes the wrong assumption about which input device is associated to which cursor plane inside the VM?
The confine + warp trick is typically the default mode and only if the guest indicates through e.g. a guest-agent process that seamless mode is supported then seamless mode is enabled.
IOW the VM is careful to not enable it when it might not work.
OTOH most distros now ship with the necessary agents installed by default, so e.g. a F32 beta install will automatically use seamless mode under QEMU/KVM (spice display protocol) or VirtualBox. If a user uses say F32 + a vnc-viewer inside it which needs to do confine + warp tricks then the user will have to disable seamless mode manually. In my prior experience working on VDI this is usually something which users are understanding of and not a problem.
Regards,
Hans
On Thu, 19 Mar 2020 15:30:03 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/19/20 1:58 PM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 12:49:27 +0100 Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/19/20 11:00 AM, Pekka Paalanen wrote:
On Wed, 18 Mar 2020 15:28:02 +0100 Hans de Goede hdegoede@redhat.com wrote:
ATM the Atomic KMS API lacks the ability to set cursor hot-spot coordinates. Mutter (and Weston) have tried to emulate this by shifting the coordinates for where to draw the cursor by the hotspot-coordinates and always using 0x0 for the hotspot.
But this breaks the so called "seamless mouse mode" for virtual-machines and there really is no way to fix this but to allow passing the proper hotspot coordinates to the virtual gfx-card.
Seamless-mode consists of 2 parts:
- Letting the VM-viewer window-system draw the cursor as it normally
would draw it.
- Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the coordinates where the VM-viewer window-system is drawing the cursor to an absolute position using the top left corner of the view as 0x0 and the bottom right corner as max-abs-x,max-abs-y.
Hi,
is the VM-viewer then hiding or autonomously moving what the display server inside VM has put on the KMS cursor plane?
Yes and no, it is not the VM-viewer which is hiding what the display-server inside the VM has put on the KMS cursor plane, the VM-viewer negotiates seamless mouse mode with the hypervisor and then the hypervisor just ignores the cursor-plane except for sending "sprite" changes to the VM-viewer.
Hi,
I don't think I understand what you're saying, but I assume that I was right in that the VM cursor plane content will not be shown always exactly in the very position the compositor inside the VM puts it.
Right, when seamless mouse mode is enabled the VM's cursor plane will not be shown *at all*, instead the VM-viewer's window-system's cursor plane is shown.
Sprite changes (normal cursor, caret, window-resize cursor, etc.) are forwarded from the VM-s cursor-plane to the VM-viewer which then requests that as cursor to the window-system under which the VM-viewer is running.
Maybe the example further below explain the issue I envision.
If so, sounds like hilarity would ensue with Weston.
Weston does not actually know what a cursor is. Weston will happily put any arbitrary non-cursor content onto the KMS cursor plane if it happens to fit. It's just that usually there is a small surface top-most that ends up on the cursor plane, and that surface accidentally happens to be a cursor by Wayland protocol.
It's not difficult to get e.g. weston-simple-shm window to be shown on the KMS cursor plane: just hide the real cursor from the client.
No, it's not an oversight. It is called "making maximal use of the available KMS resources" by using KMS planes as much as possible to avoid compositing by rendering with Pixman or OpenGL.
Yes it sounds like this will break with Weston, note that it already is broken in Weston, if you run e.g. Fedora 32 beta + its Weston package inside a VirtualBox VM then start gnome-terminal (so that you get a caret cursor instead of the default one) and try to select text. This will result in the wrong text being displayed because Weston does not relay cursor hotspot info to the GPU, also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
Where the symptoms of this are described in more detail (they are identical for Weston and mutter).
Right, that's the problem with the hotspot.
Ack.
Fixing this will require the discussed KMS atomic API changes and also changes on the Weston and mutter side to pass through the hotspot info.
The problem I am referring to is that to the user looking at the VM-viewer, suddenly an arbitrary application window (e.g. weston-simple-shm) starts to act as if it was the cursor, when there is no real cursor shown. You have a random window unexpectedly moving around, as if you had started dragging it around with your mouse.
Correct.
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
Correct.
Is that something that should be done?
If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
It sounds like your VM-viewer makes the assumption that the pointer input device it delivers to the VM is the one that will control the KMS cursor plane position inside the VM. Is that right?
Correct.
What if the desktop inside the VM is controlled by a remote, e.g. VNC? Then the input events to the VM are completely unrelated to the expected motion of the cursor. Do you just tell the users to stop using the seamless mode in that case?
A VNC viewer (which is not using seamless mode itself) has the same issue of a mismatch between the cursor position of the window-system it is a client of and the cursor position of the window system inside the VM.
A VNC viewer typically works around this by changing the window-system cursor to transparent and drawing its own cursor, the transparent sprite or disabling of the cursor-plane will get forwarded to the window-system under which the VM-viewer runs so this will work fine.
A real problem though is the absolute input mode, when e.g. the VNC viewer is not using something like seamless mouse mode it will want to do the usual (nasty) tricks of confining the pointer to the window and warping it to the center after each pointer move so that mouse can be moved "endlessly' inside the window. This requires relative input, so emulating a mouse instead of say a drawing tablet, which breaks seamless mode.
So although the way the cursor is drawn typically does not require disabling seamless mode, the input problems do require disabling seamless mode.
In my experience (I have worked on VDI (spice) a couple of years), running a VM-viewer inside a VM is something which not a lot of people do. I guess the fact that this works quite poorly might be one of the reasons people do not do this.
Sorry. I meant that you run a desktop in a VM. Then you control that desktop from some remote via VNC, and you still look at a VM's VM-viewer.
You can replace VNC with any remote input thing. E.g. USB devices plumbed directly through to the VM while still looking at the VM-viewer for output.
Niche? Yes, quite likely.
But all VMs I'm familiar with allow disabling seamless mode for compatibility with corner cases like this. This does lead to a seriously degraded user-experience though. For just using an office suit or browser inside a VM seamless mode really is a lot more pleasant for the user. I still remember when seamless mode first got introduced, in the beginning it used to not always work and fixing the fallback to the confine + warp tricks then always was a serious itch which I tried to scratch quickly. Breaking seamless mode also is a good way to quickly accumulate a lot of bug reports from end users in my experience.
Btw. the warp trick is purely a legacy X11 thing, we have relative motion interfaces nowadays at least in Wayland and I think also with XI2?
But if you need to grab the pointer to allow endless relative motion, then you grab, that hasn't changed from end user perspective.
I think one of the major reasons why Wayland pointer relative motion and confinement extensions were designed was VM- etc. viewers, and of course games.
What if display servers stop using the cursor plane completely, because people may hit a case where a VM-viewer makes the wrong assumption about which input device is associated to which cursor plane inside the VM?
The confine + warp trick is typically the default mode and only if the guest indicates through e.g. a guest-agent process that seamless mode is supported then seamless mode is enabled.
IOW the VM is careful to not enable it when it might not work.
How would the guest-agent know? Does it check that there is literally only one pointer input device and that comes from the VM-viewer? Or does it limit seamless to white-listed display servers perhaps?
OTOH most distros now ship with the necessary agents installed by default, so e.g. a F32 beta install will automatically use seamless mode under QEMU/KVM (spice display protocol) or VirtualBox. If a user uses say F32 + a vnc-viewer inside it which needs to do confine + warp tricks then the user will have to disable seamless mode manually. In my prior experience working on VDI this is usually something which users are understanding of and not a problem.
Yeah, making the association between the input and the cursor plane is unsolved.
Thanks, pq
Hi,
On 3/19/20 4:16 PM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 15:30:03 +0100 Hans de Goede hdegoede@redhat.com wrote:
<snip>
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
Correct.
Is that something that should be done?
If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
<snip>
I think one of the major reasons why Wayland pointer relative motion and confinement extensions were designed was VM- etc. viewers, and of course games.
ack.
What if display servers stop using the cursor plane completely, because people may hit a case where a VM-viewer makes the wrong assumption about which input device is associated to which cursor plane inside the VM?
The confine + warp trick is typically the default mode and only if the guest indicates through e.g. a guest-agent process that seamless mode is supported then seamless mode is enabled.
IOW the VM is careful to not enable it when it might not work.
How would the guest-agent know? Does it check that there is literally only one pointer input device and that comes from the VM-viewer? Or does it limit seamless to white-listed display servers perhaps?
This varies between different solutions. E.g. the spice vdagent has a desktop component which is started with a /etc/xdg/autostart/foo.desktop file, assuming that it will work with any desktops honoring /etc/xdg/autostart.
VirtualBox OTOH assumes it can use seamless mode as soon as the vboxvideo driver loads inside the guest.
Regards,
Hans
On Thursday, March 19, 2020 7:18 PM, Hans de Goede hdegoede@redhat.com wrote:
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
Correct.
Is that something that should be done? If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
That would need to at least be hidden behind a DRM capability, otherwise it would break existing user-space ignoring the hotspot props (e.g. current Weston).
Hi,
On 3/19/20 9:14 PM, Simon Ser wrote:
On Thursday, March 19, 2020 7:18 PM, Hans de Goede hdegoede@redhat.com wrote:
The only way to fix that is to stop Weston from putting non-cursor content on the cursor plane.
Correct.
Is that something that should be done? If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
That would need to at least be hidden behind a DRM capability, otherwise it would break existing user-space ignoring the hotspot props (e.g. current Weston).
Current Weston is already broken, fixing that is what this whole thread is about.
The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
1) Short of disabling seamless mode there is nothing the virtual gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
2) Most existing clients which obviously do not set this to-be-introduced capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients.
Regards,
Hans
Is that something that should be done? If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
That would need to at least be hidden behind a DRM capability, otherwise it would break existing user-space ignoring the hotspot props (e.g. current Weston).
Current Weston is already broken, fixing that is what this whole thread is about.
The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
- Short of disabling seamless mode there is nothing the virtual
gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
- Most existing clients which obviously do not set this to-be-introduced
capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients.
Compositors which don't support the hotspot prop are and will continue to break seamless mode. Setting the prop will never be mandatory, because all user-space today doesn't do it.
Since we need to update all user-space to add support for this prop, it would be a good idea to recognize user-space that doesn't support it. Doing something sensible with user-space that doesn't support the prop is up to the virtual driver.
If we have a capability, the virtual driver could for instance not advertise a cursor plane at all if user-space won't correctly set the hotspot anyway.
Hi,
On 3/19/20 10:07 PM, Simon Ser wrote:
Is that something that should be done? If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
That would need to at least be hidden behind a DRM capability, otherwise it would break existing user-space ignoring the hotspot props (e.g. current Weston).
Current Weston is already broken, fixing that is what this whole thread is about.
The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
- Short of disabling seamless mode there is nothing the virtual
gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
- Most existing clients which obviously do not set this to-be-introduced
capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients.
Compositors which don't support the hotspot prop are and will continue to break seamless mode. Setting the prop will never be mandatory, because all user-space today doesn't do it.
Since we need to update all user-space to add support for this prop, it would be a good idea to recognize user-space that doesn't support it. Doing something sensible with user-space that doesn't support the prop is up to the virtual driver.
If we have a capability, the virtual driver could for instance not advertise a cursor plane at all if user-space won't correctly set the hotspot anyway.
What compositors do we have around today that wouldn't want to support setting the hotspot correctly? Do we like to keep them that way, and do we want to encourage any new compositors to also not do this properly?
/Thomas
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
On Thu, 19 Mar 2020 23:57:09 +0100 Thomas Hellström (VMware) thomas_os@shipmail.org wrote:
Hi,
On 3/19/20 10:07 PM, Simon Ser wrote:
Is that something that should be done? If the hotspot property also had a "disabled" value, then Weston could set the hotspot to disabled when it is using the cursor plane for non-cursor content and not lose the feature. And of course set hotspot correctly when it in fact is a cursor (but for what input?).
I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
That would need to at least be hidden behind a DRM capability, otherwise it would break existing user-space ignoring the hotspot props (e.g. current Weston).
Current Weston is already broken, fixing that is what this whole thread is about.
The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
- Short of disabling seamless mode there is nothing the virtual
gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
- Most existing clients which obviously do not set this to-be-introduced
capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients.
Compositors which don't support the hotspot prop are and will continue to break seamless mode. Setting the prop will never be mandatory, because all user-space today doesn't do it.
Since we need to update all user-space to add support for this prop, it would be a good idea to recognize user-space that doesn't support it. Doing something sensible with user-space that doesn't support the prop is up to the virtual driver.
If we have a capability, the virtual driver could for instance not advertise a cursor plane at all if user-space won't correctly set the hotspot anyway.
What compositors do we have around today that wouldn't want to support setting the hotspot correctly? Do we like to keep them that way, and do we want to encourage any new compositors to also not do this properly?
All compositors that sometimes want to use cursor planes for non-cursor content. If a cursor plane does not actually have any performance benefits, then somehow userspace would need to know to not use it in that case.
I think the capability is a well-designed solution: a driver that knows the cursor plane will not be always exactly like userspace programs it should not advertise the cursor plane at all if userspace does not set the new cap, and the cap is a promise that userspace will set the hotspot correctly.
This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with correct hotspot info because that is a legacy KMS API and we are talking about atomic here. The legacy ioctl can continue to ignore the new cap.
It does mean that if legacy KMS userspace uses the cursor plane for non-cursor content, it keeps being broken for drivers that mangle cursor planes. I can certainly live with that.
It seems people are also forgetting the problem of associating the cursor plane with an input device, so that whatever is looking to mangle the cursor plane behind the KMS app's back would know how to do it right.
My first thought for that is a new cursor plane property with the value of major, minor of the kernel input device that userspace is using to control the cursor plane. This property should be set by userspace only when there is exactly one kernel input device it uses for controlling the cursor plane. Setting this property to none/disabled would be a clear indication that "seamless mode" would be unwanted. The DRM driver or whatever it talks to could then check if the cursor plane is indeed controlled by the input it so far has only assumed and automatically choose correctly between seamless mode or not.
Anyway, these are just ideas. Use them if you wish, I won't NAK anything if you don't.
Now you know what Weston currently does. If Weston needs to change, I guess that is fine with me.
Thanks, pq
On 3/20/20 10:13 AM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 23:57:09 +0100 Thomas Hellström (VMware) thomas_os@shipmail.org wrote:
Hi,
On 3/19/20 10:07 PM, Simon Ser wrote:
> Is that something that should be done? > If the hotspot property also had a "disabled" value, then Weston could > set the hotspot to disabled when it is using the cursor plane for > non-cursor content and not lose the feature. And of course set hotspot > correctly when it in fact is a cursor (but for what input?). I believe cursor planes in the affected virtual gfx-cards do not really have a mode where they can actually be used as a generic overlay plane, certainly not in a useful manner (if anything works it will all be software emulation), implementing a hotspot disabled mode would be tricky and this would needs to be duplicated in all virtual-gfx cards kms drivers.
If I understood Daniel's proposal for how to deal with this properly, then only cursor planes which actually need them would get the new hotspot x/y properties. If we do that then Weston could use the presence of the hotspot x/y properties to detect if it is dealing with a proper hw plane which can also be used as a generic plane; or a virtual-gfx cards cursor-plane, and then just not bother with trying to use the plane as a generic hw plane.
Would that work?
That would need to at least be hidden behind a DRM capability, otherwise it would break existing user-space ignoring the hotspot props (e.g. current Weston).
Current Weston is already broken, fixing that is what this whole thread is about.
The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
- Short of disabling seamless mode there is nothing the virtual
gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
- Most existing clients which obviously do not set this to-be-introduced
capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients.
Compositors which don't support the hotspot prop are and will continue to break seamless mode. Setting the prop will never be mandatory, because all user-space today doesn't do it.
Since we need to update all user-space to add support for this prop, it would be a good idea to recognize user-space that doesn't support it. Doing something sensible with user-space that doesn't support the prop is up to the virtual driver.
If we have a capability, the virtual driver could for instance not advertise a cursor plane at all if user-space won't correctly set the hotspot anyway.
What compositors do we have around today that wouldn't want to support setting the hotspot correctly? Do we like to keep them that way, and do we want to encourage any new compositors to also not do this properly?
All compositors that sometimes want to use cursor planes for non-cursor content. If a cursor plane does not actually have any performance benefits, then somehow userspace would need to know to not use it in that case.
Yes, but is that Weston only? Do we know about others?
I think the capability is a well-designed solution: a driver that knows the cursor plane will not be always exactly like userspace programs it should not advertise the cursor plane at all if userspace does not set the new cap, and the cap is a promise that userspace will set the hotspot correctly.
This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with correct hotspot info because that is a legacy KMS API and we are talking about atomic here. The legacy ioctl can continue to ignore the new cap.
It does mean that if legacy KMS userspace uses the cursor plane for non-cursor content, it keeps being broken for drivers that mangle cursor planes. I can certainly live with that.
What I'm afraid of is that people writing compositors will just leave that cap off, ignoring the huge benefit for virtual environments and future kms-based remoting solutions.
Couldn't compositors that see hotspot properties on a plane just ignore that plane for other stuff than cursors?
It seems people are also forgetting the problem of associating the cursor plane with an input device, so that whatever is looking to mangle the cursor plane behind the KMS app's back would know how to do it right.
My first thought for that is a new cursor plane property with the value of major, minor of the kernel input device that userspace is using to control the cursor plane. This property should be set by userspace only when there is exactly one kernel input device it uses for controlling the cursor plane. Setting this property to none/disabled would be a clear indication that "seamless mode" would be unwanted. The DRM driver or whatever it talks to could then check if the cursor plane is indeed controlled by the input it so far has only assumed and automatically choose correctly between seamless mode or not.
Are you sure this i really needed? VMware's SVGA device checks whether the guest cursor position and host cursor position are reasonably aligned, and if not, composites the guest cursor over the display plane. So if you, for example, attach a passthrough USB mouse to the VM while running in seamless mode, things "just work". Similarly if you were to attach a kms-based vnc solution that behaved in the same way and that created a new input device, things would also look fine, except for temporary cursor jumps when you switch input device.
Anyway, these are just ideas. Use them if you wish, I won't NAK anything if you don't.
Now you know what Weston currently does. If Weston needs to change, I guess that is fine with me.
Thanks, pq
Thanks,
Thomas
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
On Friday, March 20, 2020 11:59 AM, Thomas Hellström thomas_os@shipmail.org wrote:
On 3/20/20 10:13 AM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 23:57:09 +0100 Thomas Hellström (VMware) thomas_os@shipmail.org wrote:
Hi, On 3/19/20 10:07 PM, Simon Ser wrote:
> > Is that something that should be done? > > If the hotspot property also had a "disabled" value, then Weston could > > set the hotspot to disabled when it is using the cursor plane for > > non-cursor content and not lose the feature. And of course set hotspot > > correctly when it in fact is a cursor (but for what input?). > > I believe cursor planes in the affected virtual gfx-cards do not > > really have a mode where they can actually be used as a generic overlay > > plane, certainly not in a useful manner (if anything works it will all > > be software emulation), implementing a hotspot disabled mode would be > > tricky and this would needs to be duplicated in all virtual-gfx cards > > kms drivers. > > If I understood Daniel's proposal for how to deal with this properly, > then only cursor planes which actually need them would get the new > hotspot x/y properties. If we do that then Weston could use the > presence of the hotspot x/y properties to detect if it is dealing > with a proper hw plane which can also be used as a generic > plane; or a virtual-gfx cards cursor-plane, and then just not > bother with trying to use the plane as a generic hw plane. > Would that work? > That would need to at least be hidden behind a DRM capability, otherwise > it would break existing user-space ignoring the hotspot props (e.g. > current Weston). > Current Weston is already broken, fixing that is what this whole > thread is about.
The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
Short of disabling seamless mode there is nothing the virtual gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
Most existing clients which obviously do not set this to-be-introduced capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients. Compositors which don't support the hotspot prop are and will continue to break seamless mode. Setting the prop will never be mandatory, because all user-space today doesn't do it.
Since we need to update all user-space to add support for this prop, it would be a good idea to recognize user-space that doesn't support it. Doing something sensible with user-space that doesn't support the prop is up to the virtual driver. If we have a capability, the virtual driver could for instance not advertise a cursor plane at all if user-space won't correctly set the hotspot anyway. What compositors do we have around today that wouldn't want to support setting the hotspot correctly? Do we like to keep them that way, and do we want to encourage any new compositors to also not do this properly? All compositors that sometimes want to use cursor planes for non-cursor content. If a cursor plane does not actually have any performance benefits, then somehow userspace would need to know to not use it in that case.
Yes, but is that Weston only? Do we know about others?
wlroots plans to do the same as Weston.
I think the capability is a well-designed solution: a driver that knows the cursor plane will not be always exactly like userspace programs it should not advertise the cursor plane at all if userspace does not set the new cap, and the cap is a promise that userspace will set the hotspot correctly. This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with correct hotspot info because that is a legacy KMS API and we are talking about atomic here. The legacy ioctl can continue to ignore the new cap. It does mean that if legacy KMS userspace uses the cursor plane for non-cursor content, it keeps being broken for drivers that mangle cursor planes. I can certainly live with that.
What I'm afraid of is that people writing compositors will just leave that cap off, ignoring the huge benefit for virtual environments and future kms-based remoting solutions.
Couldn't compositors that see hotspot properties on a plane just ignore that plane for other stuff than cursors?
KMS clients will need patching anyway to populate the new prop. Enabling a cap is a one-liner.
Is the plan to intentionally break KMS clients to force them to add support for the property? This increases the number of things KMS clients need to do in order to correctly work. I think the KMS clients' job is already hard enough as-is.
There will always exist user-space which doesn't set the prop. I'm not just thinking about old compositors here: KMS clients are not limited to just compositors. Think about Kodi for instance or DRM lease clients.
It seems people are also forgetting the problem of associating the cursor plane with an input device, so that whatever is looking to mangle the cursor plane behind the KMS app's back would know how to do it right. My first thought for that is a new cursor plane property with the value of major, minor of the kernel input device that userspace is using to control the cursor plane. This property should be set by userspace only when there is exactly one kernel input device it uses for controlling the cursor plane. Setting this property to none/disabled would be a clear indication that "seamless mode" would be unwanted. The DRM driver or whatever it talks to could then check if the cursor plane is indeed controlled by the input it so far has only assumed and automatically choose correctly between seamless mode or not.
Are you sure this i really needed? VMware's SVGA device checks whether the guest cursor position and host cursor position are reasonably aligned, and if not, composites the guest cursor over the display plane. So if you, for example, attach a passthrough USB mouse to the VM while running in seamless mode, things "just work". Similarly if you were to attach a kms-based vnc solution that behaved in the same way and that created a new input device, things would also look fine, except for temporary cursor jumps when you switch input device.
Seems more like a workaround than a real solution.
On 3/20/20 12:27 PM, Simon Ser wrote:
On Friday, March 20, 2020 11:59 AM, Thomas Hellström thomas_os@shipmail.org wrote:
On 3/20/20 10:13 AM, Pekka Paalanen wrote:
On Thu, 19 Mar 2020 23:57:09 +0100 Thomas Hellström (VMware) thomas_os@shipmail.org wrote:
Hi, On 3/19/20 10:07 PM, Simon Ser wrote:
>>> Is that something that should be done? >>> If the hotspot property also had a "disabled" value, then Weston could >>> set the hotspot to disabled when it is using the cursor plane for >>> non-cursor content and not lose the feature. And of course set hotspot >>> correctly when it in fact is a cursor (but for what input?). >>> I believe cursor planes in the affected virtual gfx-cards do not >>> really have a mode where they can actually be used as a generic overlay >>> plane, certainly not in a useful manner (if anything works it will all >>> be software emulation), implementing a hotspot disabled mode would be >>> tricky and this would needs to be duplicated in all virtual-gfx cards >>> kms drivers. >> If I understood Daniel's proposal for how to deal with this properly, >> then only cursor planes which actually need them would get the new >> hotspot x/y properties. If we do that then Weston could use the >> presence of the hotspot x/y properties to detect if it is dealing >> with a proper hw plane which can also be used as a generic >> plane; or a virtual-gfx cards cursor-plane, and then just not >> bother with trying to use the plane as a generic hw plane. >> Would that work? >> That would need to at least be hidden behind a DRM capability, otherwise >> it would break existing user-space ignoring the hotspot props (e.g. >> current Weston). >> Current Weston is already broken, fixing that is what this whole >> thread is about. The virtual gfx-cards drivers simply must now the hotspot for things to work; and a capability is not going to help here for 2 reasons:
Short of disabling seamless mode there is nothing the virtual gfx-cards drivers can do when clients do not pass the hotspot info; and in some cases they cannot even do this as it is under control of a userspace agent process with its own channel to the hypervisor.
Most existing clients which obviously do not set this to-be-introduced capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2 ioctl. Disabling seamless mode when this to-be-introduced capability is not set would cause a huge regression for all these existing clients. Compositors which don't support the hotspot prop are and will continue to break seamless mode. Setting the prop will never be mandatory, because all user-space today doesn't do it.
Since we need to update all user-space to add support for this prop, it would be a good idea to recognize user-space that doesn't support it. Doing something sensible with user-space that doesn't support the prop is up to the virtual driver. If we have a capability, the virtual driver could for instance not advertise a cursor plane at all if user-space won't correctly set the hotspot anyway. What compositors do we have around today that wouldn't want to support setting the hotspot correctly? Do we like to keep them that way, and do we want to encourage any new compositors to also not do this properly? All compositors that sometimes want to use cursor planes for non-cursor content. If a cursor plane does not actually have any performance benefits, then somehow userspace would need to know to not use it in that case.
Yes, but is that Weston only? Do we know about others?
wlroots plans to do the same as Weston.
I think the capability is a well-designed solution: a driver that knows the cursor plane will not be always exactly like userspace programs it should not advertise the cursor plane at all if userspace does not set the new cap, and the cap is a promise that userspace will set the hotspot correctly. This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with correct hotspot info because that is a legacy KMS API and we are talking about atomic here. The legacy ioctl can continue to ignore the new cap. It does mean that if legacy KMS userspace uses the cursor plane for non-cursor content, it keeps being broken for drivers that mangle cursor planes. I can certainly live with that.
What I'm afraid of is that people writing compositors will just leave that cap off, ignoring the huge benefit for virtual environments and future kms-based remoting solutions.
Couldn't compositors that see hotspot properties on a plane just ignore that plane for other stuff than cursors?
KMS clients will need patching anyway to populate the new prop. Enabling a cap is a one-liner.
Is the plan to intentionally break KMS clients to force them to add support for the property?
We can't do that. As Hans pointed out, some KMS clients are already broken, and hardware planes that doesn't fiddle with the cursor (on which Weston works now) don't have to expose the hotspot properties.
This increases the number of things KMS clients need to do in order to correctly work. I think the KMS clients' job is already hard enough as-is.
That's exactly the kind of thinking that I'm afraid will leave virtual- and remoting solutions unsupported.
There will always exist user-space which doesn't set the prop. I'm not just thinking about old compositors here: KMS clients are not limited to just compositors. Think about Kodi for instance or DRM lease clients.
It seems people are also forgetting the problem of associating the cursor plane with an input device, so that whatever is looking to mangle the cursor plane behind the KMS app's back would know how to do it right. My first thought for that is a new cursor plane property with the value of major, minor of the kernel input device that userspace is using to control the cursor plane. This property should be set by userspace only when there is exactly one kernel input device it uses for controlling the cursor plane. Setting this property to none/disabled would be a clear indication that "seamless mode" would be unwanted. The DRM driver or whatever it talks to could then check if the cursor plane is indeed controlled by the input it so far has only assumed and automatically choose correctly between seamless mode or not.
Are you sure this i really needed? VMware's SVGA device checks whether the guest cursor position and host cursor position are reasonably aligned, and if not, composites the guest cursor over the display plane. So if you, for example, attach a passthrough USB mouse to the VM while running in seamless mode, things "just work". Similarly if you were to attach a kms-based vnc solution that behaved in the same way and that created a new input device, things would also look fine, except for temporary cursor jumps when you switch input device.
Seems more like a workaround than a real solution.
That may be the case, but still it works and if you have multiple clients it always allows the active client run in seamless mode. Besides, if people think supporting KMS hotspots is already hard enough, the chance of having them supplying the correct input device is small.
/Thomas
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
On 3/20/20 12:47 PM, Thomas Hellström (VMware) wrote:
On 3/20/20 12:27 PM, Simon Ser wrote:
On Friday, March 20, 2020 11:59 AM, Thomas Hellström thomas_os@shipmail.org wrote:
On 3/20/20 10:13 AM, Pekka Paalanen wrote:
It seems people are also forgetting the problem of associating the cursor plane with an input device, so that whatever is looking to mangle the cursor plane behind the KMS app's back would know how to do it right. My first thought for that is a new cursor plane property with the value of major, minor of the kernel input device that userspace is using to control the cursor plane. This property should be set by userspace only when there is exactly one kernel input device it uses for controlling the cursor plane. Setting this property to none/disabled would be a clear indication that "seamless mode" would be unwanted. The DRM driver or whatever it talks to could then check if the cursor plane is indeed controlled by the input it so far has only assumed and automatically choose correctly between seamless mode or not.
Are you sure this i really needed? VMware's SVGA device checks whether the guest cursor position and host cursor position are reasonably aligned, and if not, composites the guest cursor over the display plane. So if you, for example, attach a passthrough USB mouse to the VM while running in seamless mode, things "just work". Similarly if you were to attach a kms-based vnc solution that behaved in the same way and that created a new input device, things would also look fine, except for temporary cursor jumps when you switch input device.
Seems more like a workaround than a real solution.
That may be the case, but still it works and if you have multiple clients it always allows the active client run in seamless mode. Besides, if people think supporting KMS hotspots is already hard enough, the chance of having them supplying the correct input device is small.
/Thomas
To be clear about this, I'm not *against* a property associating a device with a cursor plane as long as it's just a hint. However in the VMware case, it's unlikely that we will try to implement any kernel driver support for it since we have a working solution and it also may not be possible. (Some remoting solutions only work when seamless mode is enabled).
/Thomas
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
dri-devel@lists.freedesktop.org
-
Daniel Vetter -
Hans de Goede -
Michel Dänzer -
Pekka Paalanen -
Simon Ser -
Thomas Hellström (VMware)