New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
---
Changes from v2: - Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
Changes from v1: - Typos and wording reported by Daniel and Alex --- Documentation/gpu/drm-kms.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst index 87e5023e3f55..bbe254dca635 100644 --- a/Documentation/gpu/drm-kms.rst +++ b/Documentation/gpu/drm-kms.rst @@ -463,6 +463,33 @@ KMS Properties This section of the documentation is primarily aimed at user-space developers. For the driver APIs, see the other sections.
+Requirements +------------ + +KMS drivers might need to add extra properties to support new features. +Each new property introduced in a driver need to meet a few +requirements, in addition to the one mentioned above.: + +- Before the introduction of any vendor-specific properties, they must + be first checked against the generic ones to avoid any conflict or + redundancy. + +- Vendor-specific properties must be prefixed by the vendor's name, + following the syntax "$vendor:$property". + +- Generic properties must be standardized, with some documentation to + describe how the property can be used. + +- Generic properties must provide a generic helper in the core code to + register that property on the object it attaches to. + +- Generic properties content must be decoded by the core and provided in + the object's associated state structure. That includes anything + drivers might want to precompute, like :c:type:`struct drm_clip_rect + <drm_clip_rect>` for planes. + +- An IGT test should be submitted. + Property Types and Blob Property Support ----------------------------------------
Hi Maxime,
On Thu, Jun 10, 2021 at 07:47:31PM +0200, Maxime Ripard wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
Changes from v1:
- Typos and wording reported by Daniel and Alex
Documentation/gpu/drm-kms.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst index 87e5023e3f55..bbe254dca635 100644 --- a/Documentation/gpu/drm-kms.rst +++ b/Documentation/gpu/drm-kms.rst @@ -463,6 +463,33 @@ KMS Properties This section of the documentation is primarily aimed at user-space developers. For the driver APIs, see the other sections.
+Requirements +------------
+KMS drivers might need to add extra properties to support new features. +Each new property introduced in a driver need to meet a few +requirements, in addition to the one mentioned above.:
+- Before the introduction of any vendor-specific properties, they must
- be first checked against the generic ones to avoid any conflict or
- redundancy.
+- Vendor-specific properties must be prefixed by the vendor's name,
- following the syntax "$vendor:$property".
+- Generic properties must be standardized, with some documentation to
- describe how the property can be used.
+- Generic properties must provide a generic helper in the core code to
- register that property on the object it attaches to.
+- Generic properties content must be decoded by the core and provided in
- the object's associated state structure. That includes anything
- drivers might want to precompute, like :c:type:`struct drm_clip_rect
- <drm_clip_rect>` for planes.
+- An IGT test should be submitted.
Property Types and Blob Property Support
Looks nice, thanks for the work!
Reviewed-by: Liviu Dudau liviu.dudau@arm.com
Best regards, Liviu
-- 2.31.1
On Thu, Jun 10, 2021 at 07:47:31PM +0200, Maxime Ripard wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
Changes from v1:
- Typos and wording reported by Daniel and Alex
Documentation/gpu/drm-kms.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst index 87e5023e3f55..bbe254dca635 100644 --- a/Documentation/gpu/drm-kms.rst +++ b/Documentation/gpu/drm-kms.rst @@ -463,6 +463,33 @@ KMS Properties This section of the documentation is primarily aimed at user-space developers. For the driver APIs, see the other sections.
+Requirements +------------
+KMS drivers might need to add extra properties to support new features. +Each new property introduced in a driver need to meet a few +requirements, in addition to the one mentioned above.:
+- Before the introduction of any vendor-specific properties, they must
- be first checked against the generic ones to avoid any conflict or
- redundancy.
+- Vendor-specific properties must be prefixed by the vendor's name,
- following the syntax "$vendor:$property".
+- Generic properties must be standardized, with some documentation to
- describe how the property can be used.
+- Generic properties must provide a generic helper in the core code to
- register that property on the object it attaches to.
+- Generic properties content must be decoded by the core and provided in
- the object's associated state structure. That includes anything
- drivers might want to precompute, like :c:type:`struct drm_clip_rect
- <drm_clip_rect>` for planes.
+- An IGT test should be submitted.
Property Types and Blob Property Support
Acked-by: Rodrigo Vivi rodrigo.vivi@intel.com
-- 2.31.1
On Thu, Jun 10, 2021 at 7:47 PM Maxime Ripard maxime@cerno.tech wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
I'm pretty sure my r-b was without this ... Why exactly do we need this? KMS is meant to be fairly generic (bugs throw a wrench around here sometimes, and semantics can be tricky). If we open up the door to yolo vendor properties in upstream, then that goal is pretty much written off. And we've been there with vendor properties, it's a giantic mess.
Minimally drop my r-b, I'm definitely not in support of this idea.
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
ltdr; I'd go back to v2. And then cc compositor folks on this to get their ack. -Daniel
Changes from v1:
- Typos and wording reported by Daniel and Alex
Documentation/gpu/drm-kms.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst index 87e5023e3f55..bbe254dca635 100644 --- a/Documentation/gpu/drm-kms.rst +++ b/Documentation/gpu/drm-kms.rst @@ -463,6 +463,33 @@ KMS Properties This section of the documentation is primarily aimed at user-space developers. For the driver APIs, see the other sections.
+Requirements +------------
+KMS drivers might need to add extra properties to support new features. +Each new property introduced in a driver need to meet a few +requirements, in addition to the one mentioned above.:
+- Before the introduction of any vendor-specific properties, they must
- be first checked against the generic ones to avoid any conflict or
- redundancy.
+- Vendor-specific properties must be prefixed by the vendor's name,
- following the syntax "$vendor:$property".
+- Generic properties must be standardized, with some documentation to
- describe how the property can be used.
+- Generic properties must provide a generic helper in the core code to
- register that property on the object it attaches to.
+- Generic properties content must be decoded by the core and provided in
- the object's associated state structure. That includes anything
- drivers might want to precompute, like :c:type:`struct drm_clip_rect
- <drm_clip_rect>` for planes.
+- An IGT test should be submitted.
Property Types and Blob Property Support
-- 2.31.1
Hi,
On Thu, Jun 10, 2021 at 11:00:05PM +0200, Daniel Vetter wrote:
On Thu, Jun 10, 2021 at 7:47 PM Maxime Ripard maxime@cerno.tech wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
I'm pretty sure my r-b was without this ...
Yeah, sorry. I wanted to tell you on IRC that you wanted to have a second look, but I shouldn't have kept it and caught you by surprise indeed.
Why exactly do we need this? KMS is meant to be fairly generic (bugs throw a wrench around here sometimes, and semantics can be tricky). If we open up the door to yolo vendor properties in upstream, then that goal is pretty much written off. And we've been there with vendor properties, it's a giantic mess.
Minimally drop my r-b, I'm definitely not in support of this idea.
So the argument Lidiu and Laurent made was that in some cases, getting a generic property right with only a couple of users is hard. So they advocated for the right to keep non-generic properties. I can get the argument, and no-one else said that was wrong, so it felt like the consensus was there.
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
Right, but it's mostly about in-kernel rules though? And you're the one who mentionned CC'ing the driver maintainers in the first iteration?
ltdr; I'd go back to v2. And then cc compositor folks on this to get their ack.
So, Pekka, Simon, is there anyone else I should Cc?
Thanks! Maxime
On 11/06/2021 08:54, Maxime Ripard wrote:
Hi,
On Thu, Jun 10, 2021 at 11:00:05PM +0200, Daniel Vetter wrote:
On Thu, Jun 10, 2021 at 7:47 PM Maxime Ripard maxime@cerno.tech wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
I'm pretty sure my r-b was without this ...
Yeah, sorry. I wanted to tell you on IRC that you wanted to have a second look, but I shouldn't have kept it and caught you by surprise indeed.
Why exactly do we need this? KMS is meant to be fairly generic (bugs throw a wrench around here sometimes, and semantics can be tricky). If we open up the door to yolo vendor properties in upstream, then that goal is pretty much written off. And we've been there with vendor properties, it's a giantic mess.
Minimally drop my r-b, I'm definitely not in support of this idea.
So the argument Lidiu and Laurent made was that in some cases, getting a generic property right with only a couple of users is hard. So they advocated for the right to keep non-generic properties. I can get the argument, and no-one else said that was wrong, so it felt like the consensus was there.
I also think that (maybe mainly on embedded side) we may have 1) esoteric HW features which perhaps can't even be made generic, and 2) features which may or may not be generic, but for which support cannot be added to any common opensource userspace projects like X or Weston, as the only use cases for the features are specialized low level apps (often customer's closed-source apps).
While I agree with Daniel's "gigantic mess" problem, it would also be quite nice to have a way to support all the HW features upstream instead of carrying them in vendor trees.
Tomi
On Fri, Jun 11, 2021 at 09:53:19AM +0300, Tomi Valkeinen wrote:
On 11/06/2021 08:54, Maxime Ripard wrote:
Hi,
On Thu, Jun 10, 2021 at 11:00:05PM +0200, Daniel Vetter wrote:
On Thu, Jun 10, 2021 at 7:47 PM Maxime Ripard maxime@cerno.tech wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
I'm pretty sure my r-b was without this ...
Yeah, sorry. I wanted to tell you on IRC that you wanted to have a second look, but I shouldn't have kept it and caught you by surprise indeed.
Why exactly do we need this? KMS is meant to be fairly generic (bugs throw a wrench around here sometimes, and semantics can be tricky). If we open up the door to yolo vendor properties in upstream, then that goal is pretty much written off. And we've been there with vendor properties, it's a giantic mess.
Minimally drop my r-b, I'm definitely not in support of this idea.
So the argument Lidiu and Laurent made was that in some cases, getting a generic property right with only a couple of users is hard. So they advocated for the right to keep non-generic properties. I can get the argument, and no-one else said that was wrong, so it felt like the consensus was there.
I also think that (maybe mainly on embedded side) we may have 1) esoteric HW features which perhaps can't even be made generic, and 2) features which may or may not be generic, but for which support cannot be added to any common opensource userspace projects like X or Weston, as the only use cases for the features are specialized low level apps (often customer's closed-source apps).
While I agree with Daniel's "gigantic mess" problem, it would also be quite nice to have a way to support all the HW features upstream instead of carrying them in vendor trees.
So this means to be able to accomodate this "vendor properties are totally fine, go wild" exception we also need to throw "open source userspace user, fully reviewed and tested" into the drink?
At least my experience from what I've seen with funky vendor properties isn't so much that we can't figure out a reasonable way to expose them. The problem is that the userspace tends to be a (often closed source) vendor fork of a random compositor somewhere. So if that requirement is somehow a problem, we need to talk about _that_.
Not promising we'll totally merge some vendor properties without spelling out what exactly this means. All that ensures is that people submit patches and then get annoyed because they still can't be merged because the userspace situation is all the same. -Daniel
On Fri, 11 Jun 2021 07:54:07 +0200 Maxime Ripard maxime@cerno.tech wrote:
Hi,
On Thu, Jun 10, 2021 at 11:00:05PM +0200, Daniel Vetter wrote:
On Thu, Jun 10, 2021 at 7:47 PM Maxime Ripard maxime@cerno.tech wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Sorry for intentionally losing the whole CC list, but I'm not comfortable CC'ing such a huge list (of lists) with just this.
ltdr; I'd go back to v2. And then cc compositor folks on this to get their ack.
So, Pekka, Simon, is there anyone else I should Cc?
You can find some Wayland compositor contacts in the following. These are not all compositor projects listed there though. They also signed up for wayland-protocols rather than DRM UAPI discussions, but maybe taking a few won't mind: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/MEMBERS...
Thanks, pq
On Fri, Jun 11, 2021 at 07:54:07AM +0200, Maxime Ripard wrote:
On Thu, Jun 10, 2021 at 11:00:05PM +0200, Daniel Vetter wrote:
On Thu, Jun 10, 2021 at 7:47 PM Maxime Ripard maxime@cerno.tech wrote:
New KMS properties come with a bunch of requirements to avoid each driver from running their own, inconsistent, set of properties, eventually leading to issues like property conflicts, inconsistencies between drivers and semantics, etc.
Let's document what we expect.
Cc: Alexandre Belloni alexandre.belloni@bootlin.com Cc: Alexandre Torgue alexandre.torgue@foss.st.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Alison Wang alison.wang@nxp.com Cc: Alyssa Rosenzweig alyssa.rosenzweig@collabora.com Cc: Andrew Jeffery andrew@aj.id.au Cc: Andrzej Hajda a.hajda@samsung.com Cc: Anitha Chrisanthus anitha.chrisanthus@intel.com Cc: Benjamin Gaignard benjamin.gaignard@linaro.org Cc: Ben Skeggs bskeggs@redhat.com Cc: Boris Brezillon bbrezillon@kernel.org Cc: Brian Starkey brian.starkey@arm.com Cc: Chen Feng puck.chen@hisilicon.com Cc: Chen-Yu Tsai wens@csie.org Cc: Christian Gmeiner christian.gmeiner@gmail.com Cc: "Christian König" christian.koenig@amd.com Cc: Chun-Kuang Hu chunkuang.hu@kernel.org Cc: Edmund Dea edmund.j.dea@intel.com Cc: Eric Anholt eric@anholt.net Cc: Fabio Estevam festevam@gmail.com Cc: Gerd Hoffmann kraxel@redhat.com Cc: Haneen Mohammed hamohammed.sa@gmail.com Cc: Hans de Goede hdegoede@redhat.com Cc: "Heiko Stübner" heiko@sntech.de Cc: Huang Rui ray.huang@amd.com Cc: Hyun Kwon hyun.kwon@xilinx.com Cc: Inki Dae inki.dae@samsung.com Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Jernej Skrabec jernej.skrabec@siol.net Cc: Jerome Brunet jbrunet@baylibre.com Cc: Joel Stanley joel@jms.id.au Cc: John Stultz john.stultz@linaro.org Cc: Jonas Karlman jonas@kwiboo.se Cc: Jonathan Hunter jonathanh@nvidia.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Joonyoung Shim jy0922.shim@samsung.com Cc: Jyri Sarha jyri.sarha@iki.fi Cc: Kevin Hilman khilman@baylibre.com Cc: Kieran Bingham kieran.bingham+renesas@ideasonboard.com Cc: Krzysztof Kozlowski krzysztof.kozlowski@canonical.com Cc: Kyungmin Park kyungmin.park@samsung.com Cc: Laurent Pinchart Laurent.pinchart@ideasonboard.com Cc: Linus Walleij linus.walleij@linaro.org Cc: Liviu Dudau liviu.dudau@arm.com Cc: Lucas Stach l.stach@pengutronix.de Cc: Ludovic Desroches ludovic.desroches@microchip.com Cc: Marek Vasut marex@denx.de Cc: Martin Blumenstingl martin.blumenstingl@googlemail.com Cc: Matthias Brugger matthias.bgg@gmail.com Cc: Maxime Coquelin mcoquelin.stm32@gmail.com Cc: Maxime Ripard mripard@kernel.org Cc: Melissa Wen melissa.srw@gmail.com Cc: Neil Armstrong narmstrong@baylibre.com Cc: Nicolas Ferre nicolas.ferre@microchip.com Cc: "Noralf Trønnes" noralf@tronnes.org Cc: NXP Linux Team linux-imx@nxp.com Cc: Oleksandr Andrushchenko oleksandr_andrushchenko@epam.com Cc: Patrik Jakobsson patrik.r.jakobsson@gmail.com Cc: Paul Cercueil paul@crapouillou.net Cc: Pengutronix Kernel Team kernel@pengutronix.de Cc: Philippe Cornu philippe.cornu@foss.st.com Cc: Philipp Zabel p.zabel@pengutronix.de Cc: Qiang Yu yuq825@gmail.com Cc: Rob Clark robdclark@gmail.com Cc: Robert Foss robert.foss@linaro.org Cc: Rob Herring robh@kernel.org Cc: Rodrigo Siqueira rodrigosiqueiramelo@gmail.com Cc: Rodrigo Vivi rodrigo.vivi@intel.com Cc: Roland Scheidegger sroland@vmware.com Cc: Russell King linux@armlinux.org.uk Cc: Sam Ravnborg sam@ravnborg.org Cc: Sandy Huang hjc@rock-chips.com Cc: Sascha Hauer s.hauer@pengutronix.de Cc: Sean Paul sean@poorly.run Cc: Seung-Woo Kim sw0312.kim@samsung.com Cc: Shawn Guo shawnguo@kernel.org Cc: Stefan Agner stefan@agner.ch Cc: Steven Price steven.price@arm.com Cc: Sumit Semwal sumit.semwal@linaro.org Cc: Thierry Reding thierry.reding@gmail.com Cc: Tian Tao tiantao6@hisilicon.com Cc: Tomeu Vizoso tomeu.vizoso@collabora.com Cc: Tomi Valkeinen tomba@kernel.org Cc: VMware Graphics linux-graphics-maintainer@vmware.com Cc: Xinliang Liu xinliang.liu@linaro.org Cc: Xinwei Kong kong.kongxinwei@hisilicon.com Cc: Yannick Fertre yannick.fertre@foss.st.com Cc: Zack Rusin zackr@vmware.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Signed-off-by: Maxime Ripard maxime@cerno.tech
Changes from v2:
- Take into account the feedback from Laurent and Lidiu to no longer force generic properties, but prefix vendor-specific properties with the vendor name
I'm pretty sure my r-b was without this ...
Yeah, sorry. I wanted to tell you on IRC that you wanted to have a second look, but I shouldn't have kept it and caught you by surprise indeed.
Why exactly do we need this? KMS is meant to be fairly generic (bugs throw a wrench around here sometimes, and semantics can be tricky). If we open up the door to yolo vendor properties in upstream, then that goal is pretty much written off. And we've been there with vendor properties, it's a giantic mess.
Minimally drop my r-b, I'm definitely not in support of this idea.
So the argument Lidiu and Laurent made was that in some cases, getting a generic property right with only a couple of users is hard. So they advocated for the right to keep non-generic properties. I can get the argument, and no-one else said that was wrong, so it felt like the consensus was there.
My argument was two-fold. On one hand, there's the issue of standardizing properties when we have a single example. I don't have a good solution for that chicken-and-egg issue, when we tell vendor they should standardize properties, and when they ask how, we tell them we don't know as it hasn't been done before. The option for allowing a staging playground for vendor properties here isn't something I like, but we'll need something similar one way or another. Perhaps at the cost of not guaranteeing userspace ABI for those properties, and converting them later to something standard (it won't be received well by vendor of course, and will make a push for mainline more difficult to sell, so it's not a great solution).
On the other hand, there are esoteric vendor-specific features for which standardization really doesn't make sense. For those, I think vendor-specific properties are fine, as long as they're properly documented, with a design documentation merged at the same time as the property. A vendor prefix is really just a namespace clash handling tool here.
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
Right, but it's mostly about in-kernel rules though? And you're the one who mentionned CC'ing the driver maintainers in the first iteration?
ltdr; I'd go back to v2. And then cc compositor folks on this to get their ack.
So, Pekka, Simon, is there anyone else I should Cc?
On Thursday, June 10th, 2021 at 23:00, Daniel Vetter daniel.vetter@ffwll.ch wrote:
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
In wlroots we have a policy of only allowing standard KMS properties to be used. Any vendor-specific property is going to be less well-defined, less widely useful, potentially have more design issues, potentially overlap in functionality with other vendor-specific properties, likely have some hardware-specific assumptions, etc.
What matters here is discussing with other driver & user-space folks to make sure the new property's design is sound. Designing uAPI is hard.
If kernel folks are struggling with a user-space implementation, they should discuss with user-space folks to see which project would be interested. There's a chance a compositor will be interested in the new property and will just do the user-space part for you, if not we can suggest candidate projects.
tl;dr strong agree with Daniel here.
On Fri, Jun 11, 2021 at 08:14:59AM +0000, Simon Ser wrote:
On Thursday, June 10th, 2021 at 23:00, Daniel Vetter daniel.vetter@ffwll.ch wrote:
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
In wlroots we have a policy of only allowing standard KMS properties to be used. Any vendor-specific property is going to be less well-defined, less widely useful, potentially have more design issues, potentially overlap in functionality with other vendor-specific properties, likely have some hardware-specific assumptions, etc.
What matters here is discussing with other driver & user-space folks to make sure the new property's design is sound. Designing uAPI is hard.
If kernel folks are struggling with a user-space implementation, they should discuss with user-space folks to see which project would be interested. There's a chance a compositor will be interested in the new property and will just do the user-space part for you, if not we can suggest candidate projects.
tl;dr strong agree with Daniel here.
I think the assumption you and Daniel are making is that the first implementation of a new KMS property can be made standard from day one and that it will work for any late comer driver as is, without having to make changes to its behaviour in a significant way. In my experience that is not the case.
I think we have moved from the times when we were trying to implement in the Linux world features that were available in the hardware but needed a kernel and userspace API. The set of properties that exist in KMS cover a lot of needed functionality and I don't expect to see new properties for stuff that is already supported by hardware.
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard". Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
Best regards, Liviu
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard". Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
Just my 2c - if the mainline kernel isn't willing to commit to a feature for upstream userspace to use, why does that feature belong in the kernel at all? I don't see much value in exposing hardware for the sake of exposing it when, practically, Linux userspace /can't/ use it as-is.
Might these vendor properties be used on downstream Android userspaces? That's not generally an upstream goal to support.
On Fri, Jun 11, 2021 at 08:56:04AM -0400, Alyssa Rosenzweig wrote:
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard". Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
Just my 2c - if the mainline kernel isn't willing to commit to a feature for upstream userspace to use, why does that feature belong in the kernel at all? I don't see much value in exposing hardware for the sake of exposing it when, practically, Linux userspace /can't/ use it as-is.
Might these vendor properties be used on downstream Android userspaces? That's not generally an upstream goal to support.
I think the assumption is that we are willing to commit to supporting a feature for userspace, just that (I personally) lack the confidence that I will be getting the feature right on the first attempt and using only one vendor hardware. And that supporting potential mistakes I might've made in the first version is harder if the feature was deemed "standard".
I'm talking from my experience with the writeback connector. We almost committed the feature twice before more people chipped in and asked us for changes, but that was lucky.
Best regards, Liviu
On Fri, Jun 11, 2021 at 02:34:18PM +0100, Liviu Dudau wrote:
On Fri, Jun 11, 2021 at 08:56:04AM -0400, Alyssa Rosenzweig wrote:
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard". Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
Just my 2c - if the mainline kernel isn't willing to commit to a feature for upstream userspace to use, why does that feature belong in the kernel at all? I don't see much value in exposing hardware for the sake of exposing it when, practically, Linux userspace /can't/ use it as-is.
Might these vendor properties be used on downstream Android userspaces? That's not generally an upstream goal to support.
I think the assumption is that we are willing to commit to supporting a feature for userspace, just that (I personally) lack the confidence that I will be getting the feature right on the first attempt and using only one vendor hardware. And that supporting potential mistakes I might've made in the first version is harder if the feature was deemed "standard".
There's also the issue that the first developer to try to upstream a standard property for a new feature often gets ignored as nobody else has experience with that feature, and thus lack personal interest. That's not a pure technical issue, there's a management problem there too. The right solution may ne to figure out a good process to standardize new property without making it so difficult that everybody will give up and only use downstream kernels. I've heard too many times already from vendors that upstream isn't something they can target as the bar to entry is too high, and when I convince them to submit patches to extend APIs, those patches get ignored (I'm not talking about DRM/KMS only here).
I'm talking from my experience with the writeback connector. We almost committed the feature twice before more people chipped in and asked us for changes, but that was lucky.
On Fri, 11 Jun 2021 13:03:09 +0100 Liviu Dudau liviu.dudau@arm.com wrote:
On Fri, Jun 11, 2021 at 08:14:59AM +0000, Simon Ser wrote:
On Thursday, June 10th, 2021 at 23:00, Daniel Vetter daniel.vetter@ffwll.ch wrote:
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
In wlroots we have a policy of only allowing standard KMS properties to be used. Any vendor-specific property is going to be less well-defined, less widely useful, potentially have more design issues, potentially overlap in functionality with other vendor-specific properties, likely have some hardware-specific assumptions, etc.
What matters here is discussing with other driver & user-space folks to make sure the new property's design is sound. Designing uAPI is hard.
If kernel folks are struggling with a user-space implementation, they should discuss with user-space folks to see which project would be interested. There's a chance a compositor will be interested in the new property and will just do the user-space part for you, if not we can suggest candidate projects.
tl;dr strong agree with Daniel here.
I think the assumption you and Daniel are making is that the first implementation of a new KMS property can be made standard from day one and that it will work for any late comer driver as is, without having to make changes to its behaviour in a significant way. In my experience that is not the case.
I think we have moved from the times when we were trying to implement in the Linux world features that were available in the hardware but needed a kernel and userspace API. The set of properties that exist in KMS cover a lot of needed functionality and I don't expect to see new properties for stuff that is already supported by hardware.
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard".
I agree that is a problem with trying to make generic anything. But it does not mean you should not even try. Maybe trying really hard saves a couple revisions.
What I think should be planned for is revisions. How to add new properties that do the same thing but better, while documenting that a userspace KMS client can use only one revision at a time. You never remove old revisions, unless maybe with a DRM client cap they could disappear from that file description if that is necessary for seeing the new revision.
While designing this, one also needs to take into account that KMS clients need to be able to save and restore properties *they do not understand*. So exposing two revisions of the same feature simultaneously would break save/restore is that's an error.
Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or
But you can't move, you can only add? You can't delete the old property in kernel if it was ever released with a kernel and anyone used it. In the same sentence you also imply that there is a user of it, so removing it will break that user. Then you'll have to track the userspace lifetime to figure out which decade you can try removing it.
should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
You have to carry the backward compatibility in all cases, right?
Userspace OTOH can drop support for older less supported KMS properties while taking advantage of a new revision. Userspace is not required to support old kernels forever.
Here's a wild counter-proposal off a tangent:
How about we make "implemented in and testable with VKMS" the rule, instead of "is generic" for new properties?
VKMS is what compositors (will) use in CI. I would feel hugely less bad about using a property that only one hardware driver ever implements, if also VKMS implements it in a way that compositor CI can observe it working.
I don't expect this proposal to be accepted, but it's food for thought. The major problem for compositor projects is testing as you usually don't have the hardware, IMO. CI tends to not have any hardware.
Thanks, pq
On Mon, Jun 14, 2021 at 05:49:12PM +0300, Pekka Paalanen wrote:
On Fri, 11 Jun 2021 13:03:09 +0100 Liviu Dudau liviu.dudau@arm.com wrote:
On Fri, Jun 11, 2021 at 08:14:59AM +0000, Simon Ser wrote:
On Thursday, June 10th, 2021 at 23:00, Daniel Vetter daniel.vetter@ffwll.ch wrote:
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
In wlroots we have a policy of only allowing standard KMS properties to be used. Any vendor-specific property is going to be less well-defined, less widely useful, potentially have more design issues, potentially overlap in functionality with other vendor-specific properties, likely have some hardware-specific assumptions, etc.
What matters here is discussing with other driver & user-space folks to make sure the new property's design is sound. Designing uAPI is hard.
If kernel folks are struggling with a user-space implementation, they should discuss with user-space folks to see which project would be interested. There's a chance a compositor will be interested in the new property and will just do the user-space part for you, if not we can suggest candidate projects.
tl;dr strong agree with Daniel here.
I think the assumption you and Daniel are making is that the first implementation of a new KMS property can be made standard from day one and that it will work for any late comer driver as is, without having to make changes to its behaviour in a significant way. In my experience that is not the case.
I think we have moved from the times when we were trying to implement in the Linux world features that were available in the hardware but needed a kernel and userspace API. The set of properties that exist in KMS cover a lot of needed functionality and I don't expect to see new properties for stuff that is already supported by hardware.
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard".
I agree that is a problem with trying to make generic anything. But it does not mean you should not even try. Maybe trying really hard saves a couple revisions.
Agree.
What I think should be planned for is revisions. How to add new properties that do the same thing but better, while documenting that a userspace KMS client can use only one revision at a time. You never remove old revisions, unless maybe with a DRM client cap they could disappear from that file description if that is necessary for seeing the new revision.
While designing this, one also needs to take into account that KMS clients need to be able to save and restore properties *they do not understand*. So exposing two revisions of the same feature simultaneously would break save/restore is that's an error.
I quite like the idea of having versions for properties.
Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or
But you can't move, you can only add? You can't delete the old property in kernel if it was ever released with a kernel and anyone used it. In the same sentence you also imply that there is a user of it, so removing it will break that user. Then you'll have to track the userspace lifetime to figure out which decade you can try removing it.
Not that I am supporting the workflow, but I was trying to address the comments that vendors are going to push their own userspace implementation for their vendor properties. If that is the case, when they switch to the standard ones they can drop the support in userspace for their changes. With the implied assumption that you will have fewer vendor implementations hence easier to make changes, KMS properties can be deleted if you know there is no user of them (e.g. the vendor has upgraded all their software to the standard property).
should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
You have to carry the backward compatibility in all cases, right?
Userspace OTOH can drop support for older less supported KMS properties while taking advantage of a new revision. Userspace is not required to support old kernels forever.
Here's a wild counter-proposal off a tangent:
How about we make "implemented in and testable with VKMS" the rule, instead of "is generic" for new properties?
VKMS is what compositors (will) use in CI. I would feel hugely less bad about using a property that only one hardware driver ever implements, if also VKMS implements it in a way that compositor CI can observe it working.
I don't expect this proposal to be accepted, but it's food for thought. The major problem for compositor projects is testing as you usually don't have the hardware, IMO. CI tends to not have any hardware.
While I don't dislike the proposal (I think it is quite sensible), I am worried that for some behaviours VKMS will implement them in a quirky way. To pick (again) the example of writeback, real hardware will have a way to tell if the buffer has been sent successfully to memory and it might take more than one refresh period, while VKMS (if I remember correctly) fakes it and signals the fence at the next vblank. If you code your compositor based on VKMS you might get unexpected artifacts on real hardware.
Best regards, Liviu
Thanks, pq
On Mon, Jun 14, 2021 at 04:24:13PM +0100, Liviu Dudau wrote:
On Mon, Jun 14, 2021 at 05:49:12PM +0300, Pekka Paalanen wrote:
On Fri, 11 Jun 2021 13:03:09 +0100 Liviu Dudau liviu.dudau@arm.com wrote:
On Fri, Jun 11, 2021 at 08:14:59AM +0000, Simon Ser wrote:
On Thursday, June 10th, 2021 at 23:00, Daniel Vetter daniel.vetter@ffwll.ch wrote:
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
In wlroots we have a policy of only allowing standard KMS properties to be used. Any vendor-specific property is going to be less well-defined, less widely useful, potentially have more design issues, potentially overlap in functionality with other vendor-specific properties, likely have some hardware-specific assumptions, etc.
What matters here is discussing with other driver & user-space folks to make sure the new property's design is sound. Designing uAPI is hard.
If kernel folks are struggling with a user-space implementation, they should discuss with user-space folks to see which project would be interested. There's a chance a compositor will be interested in the new property and will just do the user-space part for you, if not we can suggest candidate projects.
tl;dr strong agree with Daniel here.
I think the assumption you and Daniel are making is that the first implementation of a new KMS property can be made standard from day one and that it will work for any late comer driver as is, without having to make changes to its behaviour in a significant way. In my experience that is not the case.
I think we have moved from the times when we were trying to implement in the Linux world features that were available in the hardware but needed a kernel and userspace API. The set of properties that exist in KMS cover a lot of needed functionality and I don't expect to see new properties for stuff that is already supported by hardware.
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard".
I agree that is a problem with trying to make generic anything. But it does not mean you should not even try. Maybe trying really hard saves a couple revisions.
Agree.
What I think should be planned for is revisions. How to add new properties that do the same thing but better, while documenting that a userspace KMS client can use only one revision at a time. You never remove old revisions, unless maybe with a DRM client cap they could disappear from that file description if that is necessary for seeing the new revision.
While designing this, one also needs to take into account that KMS clients need to be able to save and restore properties *they do not understand*. So exposing two revisions of the same feature simultaneously would break save/restore is that's an error.
I quite like the idea of having versions for properties.
It's an interesting idea. We'll need to consider bundles of properties in that case I think, as in a v1 a feature could be implemented by properties A, B and C, while in v2 C could be replaced by D and E (beside A and B themselves also being changed in v2).
Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or
But you can't move, you can only add? You can't delete the old property in kernel if it was ever released with a kernel and anyone used it. In the same sentence you also imply that there is a user of it, so removing it will break that user. Then you'll have to track the userspace lifetime to figure out which decade you can try removing it.
Not that I am supporting the workflow, but I was trying to address the comments that vendors are going to push their own userspace implementation for their vendor properties. If that is the case, when they switch to the standard ones they can drop the support in userspace for their changes. With the implied assumption that you will have fewer vendor implementations hence easier to make changes, KMS properties can be deleted if you know there is no user of them (e.g. the vendor has upgraded all their software to the standard property).
should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
You have to carry the backward compatibility in all cases, right?
Userspace OTOH can drop support for older less supported KMS properties while taking advantage of a new revision. Userspace is not required to support old kernels forever.
Here's a wild counter-proposal off a tangent:
How about we make "implemented in and testable with VKMS" the rule, instead of "is generic" for new properties?
VKMS is what compositors (will) use in CI. I would feel hugely less bad about using a property that only one hardware driver ever implements, if also VKMS implements it in a way that compositor CI can observe it working.
I don't expect this proposal to be accepted, but it's food for thought. The major problem for compositor projects is testing as you usually don't have the hardware, IMO. CI tends to not have any hardware.
While I don't dislike the proposal (I think it is quite sensible), I am worried that for some behaviours VKMS will implement them in a quirky way. To pick (again) the example of writeback, real hardware will have a way to tell if the buffer has been sent successfully to memory and it might take more than one refresh period, while VKMS (if I remember correctly) fakes it and signals the fence at the next vblank. If you code your compositor based on VKMS you might get unexpected artifacts on real hardware.
I'm also worried about properties that related to hardware processing of frames, for instance related to colour spaces, or even scaling. VKMS has a software composer implemented in the kernel, it may get fairly complicated if we need to implement all kind of processing.
On Mon, 14 Jun 2021 19:33:47 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
On Mon, Jun 14, 2021 at 04:24:13PM +0100, Liviu Dudau wrote:
On Mon, Jun 14, 2021 at 05:49:12PM +0300, Pekka Paalanen wrote:
On Fri, 11 Jun 2021 13:03:09 +0100 Liviu Dudau liviu.dudau@arm.com wrote:
On Fri, Jun 11, 2021 at 08:14:59AM +0000, Simon Ser wrote:
On Thursday, June 10th, 2021 at 23:00, Daniel Vetter daniel.vetter@ffwll.ch wrote:
If there's a strong consensus that we really need this then I'm not going to nack this, but this really needs a pile of acks from compositor folks that they're willing to live with the resulting fallout this will likely bring. Your cc list seems to have an absence of compositor folks, but instead every driver maintainer. That's backwards. We make uapi for userspace, not for kernel driver maintainers!
In wlroots we have a policy of only allowing standard KMS properties to be used. Any vendor-specific property is going to be less well-defined, less widely useful, potentially have more design issues, potentially overlap in functionality with other vendor-specific properties, likely have some hardware-specific assumptions, etc.
What matters here is discussing with other driver & user-space folks to make sure the new property's design is sound. Designing uAPI is hard.
If kernel folks are struggling with a user-space implementation, they should discuss with user-space folks to see which project would be interested. There's a chance a compositor will be interested in the new property and will just do the user-space part for you, if not we can suggest candidate projects.
tl;dr strong agree with Daniel here.
I think the assumption you and Daniel are making is that the first implementation of a new KMS property can be made standard from day one and that it will work for any late comer driver as is, without having to make changes to its behaviour in a significant way. In my experience that is not the case.
I think we have moved from the times when we were trying to implement in the Linux world features that were available in the hardware but needed a kernel and userspace API. The set of properties that exist in KMS cover a lot of needed functionality and I don't expect to see new properties for stuff that is already supported by hardware.
What I'm expected to see in the future is new functionality that gets implemented by one hardware vendor and the kernel developers trying to enable that for userspace. It could be that the new property is generic, but there is no way of testing that on more than one implementation yet, so I'd say we are generous calling it "standard property". When the second or third hardware vendor comes along and starts supporting that property with their own set of extra requirements, then we can call it "standard".
I agree that is a problem with trying to make generic anything. But it does not mean you should not even try. Maybe trying really hard saves a couple revisions.
Agree.
What I think should be planned for is revisions. How to add new properties that do the same thing but better, while documenting that a userspace KMS client can use only one revision at a time. You never remove old revisions, unless maybe with a DRM client cap they could disappear from that file description if that is necessary for seeing the new revision.
While designing this, one also needs to take into account that KMS clients need to be able to save and restore properties *they do not understand*. So exposing two revisions of the same feature simultaneously would break save/restore is that's an error.
I quite like the idea of having versions for properties.
It's an interesting idea. We'll need to consider bundles of properties in that case I think, as in a v1 a feature could be implemented by properties A, B and C, while in v2 C could be replaced by D and E (beside A and B themselves also being changed in v2).
Then comes the effort cost: would it be easier to start with a vendor property that only the vendor needs to support (and can submit patches into the compositors to do so) and when the standard property gets added moves to that, or
But you can't move, you can only add? You can't delete the old property in kernel if it was ever released with a kernel and anyone used it. In the same sentence you also imply that there is a user of it, so removing it will break that user. Then you'll have to track the userspace lifetime to figure out which decade you can try removing it.
Not that I am supporting the workflow, but I was trying to address the comments that vendors are going to push their own userspace implementation for their vendor properties. If that is the case, when they switch to the standard ones they can drop the support in userspace for their changes. With the implied assumption that you will have fewer vendor implementations hence easier to make changes, KMS properties can be deleted if you know there is no user of them (e.g. the vendor has upgraded all their software to the standard property).
should we start with a generic property that gets implemented by the compositors (maybe, but then only one vendor supports it) and then later when we actually standardise the property we will have to carry backwards compatibility code in the kernel to handle the old behaviour for old userspace? My proposal to Maxime was for the former option to be reflected in the documentation, but I would like to hear your thoughts.
You have to carry the backward compatibility in all cases, right?
Userspace OTOH can drop support for older less supported KMS properties while taking advantage of a new revision. Userspace is not required to support old kernels forever.
Here's a wild counter-proposal off a tangent:
How about we make "implemented in and testable with VKMS" the rule, instead of "is generic" for new properties?
VKMS is what compositors (will) use in CI. I would feel hugely less bad about using a property that only one hardware driver ever implements, if also VKMS implements it in a way that compositor CI can observe it working.
I don't expect this proposal to be accepted, but it's food for thought. The major problem for compositor projects is testing as you usually don't have the hardware, IMO. CI tends to not have any hardware.
While I don't dislike the proposal (I think it is quite sensible), I am worried that for some behaviours VKMS will implement them in a quirky way. To pick (again) the example of writeback, real hardware will have a way to tell if the buffer has been sent successfully to memory and it might take more than one refresh period, while VKMS (if I remember correctly) fakes it and signals the fence at the next vblank. If you code your compositor based on VKMS you might get unexpected artifacts on real hardware.
I'm also worried about properties that related to hardware processing of frames, for instance related to colour spaces, or even scaling. VKMS has a software composer implemented in the kernel, it may get fairly complicated if we need to implement all kind of processing.
Hi,
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
Requiring a software implementation for testing has another huge benefit (or disadvantage, depending on who you ask): "hardware does secret magic" would not fly anymore. The software implementation would need to deliver the same results, or at least one would need to define the tolerances for e.g. color conversions. Otherwise userspace compositor test suites might pass on the software implementation and fail on hardware implementations. For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
Thanks, pq
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen ppaalanen@gmail.com wrote:
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
[1]: https://github.com/emersion/libliftoff/blob/master/test/libdrm_mock.c
For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote:
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen ppaalanen@gmail.com wrote:
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
On Tue, 15 Jun 2021 12:45:57 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote:
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen ppaalanen@gmail.com wrote:
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
That is where the documented tolerances come into play.
Userspace projects need screenshot-based testing, and we need to know how much tolerance we should allow or expect.
Good reminder about CRCs. CRCs have zero tolerance, so they are not useful for testing properties that have any leeway, are they?
Thanks, pq
On Tuesday, June 15th, 2021 at 12:16, Pekka Paalanen ppaalanen@gmail.com wrote:
Good reminder about CRCs. CRCs have zero tolerance, so they are not useful for testing properties that have any leeway, are they?
IIRC, IGT's alpha blending test currently computes the CRC for all possible roundings, then checks that the hw returns one of the acceptable CRCs.
With more complex color management properties, this approach might not be possible and write-back support in hw drivers would really help.
On Tue, Jun 15, 2021 at 2:09 PM Simon Ser contact@emersion.fr wrote:
On Tuesday, June 15th, 2021 at 12:16, Pekka Paalanen ppaalanen@gmail.com wrote:
Good reminder about CRCs. CRCs have zero tolerance, so they are not useful for testing properties that have any leeway, are they?
IIRC, IGT's alpha blending test currently computes the CRC for all possible roundings, then checks that the hw returns one of the acceptable CRCs.
With more complex color management properties, this approach might not be possible and write-back support in hw drivers would really help.
Yeah CRC based tests have severe limits, and even if you only try to test the extreme stuff there's enough busted hw out there that they will fail. E.g. when scaling I'm sure there's hw that bleeds in pixels from outside the bounding box and can't be fixed, or there's some intel hw where the alpha blending gets the mapping between [0, 0xff] <-> [0.0, 1.0] and you get a nice faint ghost instead of full transparency or opaqueness.
I think for those broken hw is just broken, nothing we can do.
Also writeback isn't supported by enough display hw, and not everyone has access to chamelium or similar.
I think best we can do is that relevant igt have an interactive mode (built-in for crc tests, so you can pause and look at the screen) and then perhaps compare with the vkms reference implementation manually. That's not great, and vkms is also not anywhere near close enough for this either (and like currently you can't even look at it, we'd first need a v4l output mode). -Daniel
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
On Tue, 15 Jun 2021 12:45:57 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote:
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen wrote:
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
Userspace projects need screenshot-based testing, and we need to know how much tolerance we should allow or expect.
Good reminder about CRCs. CRCs have zero tolerance, so they are not useful for testing properties that have any leeway, are they?
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
On Tue, 15 Jun 2021 12:45:57 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote:
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen wrote:
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
What should we infer from that? That the hardware is broken and exposing those KMS properties is a false promise?
If a driver on certain hardware cannot correctly implement a KMS property over the full domain of the input space, should that driver then simply not expose the KMS property at all?
But I would assume that the vendor still wants to expose the features in upstream kernels, yet they cannot use the standard KMS properties for that. Should the driver then expose vendor-specific properties with the disclaimer that the result is not always what one would expect, so that userspace written and tested explicitly for that hardware can still work?
That is, a sufficient justification for a vendor-specific KMS property would be that a standard property already exists, but the hardware is too buggy to make it work. IOW, give up trying to make sense.
I would like to move towards a direction where *hardware* design and testing is eventually guided by Linux KMS property definitions and their tests. If we could have a rule that if a driver cannot correctly implement a property then it must not expose the property, maybe in the long term that might start having an effect?
My underlying assumption is that generic userspace will not use vendor-specific properties.
Or, since we have atomic commits with TEST_ONLY, should it be driver's responsibility to carefully inspect the full state and reject the commit if the hardware is incapable of implementing it correctly? Vendor-specific userspace would know to avoid failing configurations to begin with. I suppose that might put an endless whack-a-mole game on drivers though.
Thanks, pq
Hi Pekka,
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
On Tue, 15 Jun 2021 12:45:57 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote:
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen wrote:
indeed it will, but what else could one do to test userspace KMS clients in generic CI where all you can have is virtual hardware? Maybe in the long run VKMS needs to loop back to a userspace daemon that implements all the complex processing and returns the writeback result via VKMS again? That daemon would then need a single upstream, like the kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual stuff in userspace? Didn't someone already have something like that? It would need to be lifted to be a required part of kernel UAPI submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
For compositor developers like me knowing the exact formulas would be a huge benefit as it would allow me to use KMS to off-load precision-sensitive operations (e.g. professional color management). Otherwise, compositors probably need a switch: "high quality color management? Then do not use KMS features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
What should we infer from that? That the hardware is broken and exposing those KMS properties is a false promise?
No, just that the scaler doesn't document the internal hardware implementation (number of taps in the filters, coefficients, rounding, ...). That's the rule, not the exception, and it doesn't prevent correct operation, images get scaled in a reproducible way (the same input produces the same output).
If a driver on certain hardware cannot correctly implement a KMS property over the full domain of the input space, should that driver then simply not expose the KMS property at all?
The properties involved here would the the SRC and CRTC rectangles for the planes. They don't document pixel-perfect scaling :-)
But I would assume that the vendor still wants to expose the features in upstream kernels, yet they cannot use the standard KMS properties for that. Should the driver then expose vendor-specific properties with the disclaimer that the result is not always what one would expect, so that userspace written and tested explicitly for that hardware can still work?
That is, a sufficient justification for a vendor-specific KMS property would be that a standard property already exists, but the hardware is too buggy to make it work. IOW, give up trying to make sense.
It's not just about buggy hardware, it's also about implementation specificities, such as rounding, filters, order of operations in the color management pipeline (it's relatively easy when you only have two LUTs and a CCM matrix, but if you through 3D LUTs and other tonemapping features in the mix, not all hardware will implement the same pipeline), or various types of image compression (this device implements a "near-lossless" compression scheme that reduces the memory bandwidth by 50% for instance).
I would like to move towards a direction where *hardware* design and testing is eventually guided by Linux KMS property definitions and their tests. If we could have a rule that if a driver cannot correctly implement a property then it must not expose the property, maybe in the long term that might start having an effect?
I didn't even know a 3D LUT existed before getting the datasheet for this device. It's a long shot before the community will acquire enough knowledge to guide hardware design, especially if you consider the feature of tomorrow's devices that nobody knows about yet. A worthwhile goal, but not something we can fully achieve, and not in a short time frame.
My underlying assumption is that generic userspace will not use vendor-specific properties.
I expect some amount of device-specific code in userspace, yes.
There are usually large variations in how the hardware exposes access to a given feature, which leads to code having to convert the standard API parameters to hardware parameters. To a large extend this can be done in drivers, but for some more complex features, it may put too much burden on the kernel. There's a reason mesa is a userspace stack :-)
Or, since we have atomic commits with TEST_ONLY, should it be driver's responsibility to carefully inspect the full state and reject the commit if the hardware is incapable of implementing it correctly? Vendor-specific userspace would know to avoid failing configurations to begin with. I suppose that might put an endless whack-a-mole game on drivers though.
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
Hi Pekka,
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
On Tue, 15 Jun 2021 12:45:57 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote:
On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen wrote:
> indeed it will, but what else could one do to test userspace KMS > clients in generic CI where all you can have is virtual hardware? Maybe > in the long run VKMS needs to loop back to a userspace daemon that > implements all the complex processing and returns the writeback result > via VKMS again? That daemon would then need a single upstream, like the > kernel, where it is maintained and correctness verified.
The complex processing must be implemented even without write-back, because user-space can ask for CRCs of the CRTC.
> Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual > stuff in userspace? Didn't someone already have something like that? > It would need to be lifted to be a required part of kernel UAPI > submissions, I suppose like IGT is nowadays.
FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full software implementation with write-back connectors, but allows to expose virtual planes and check atomic commits in CI.
> For compositor developers like me knowing the exact formulas would be a huge > benefit as it would allow me to use KMS to off-load precision-sensitive > operations (e.g. professional color management). Otherwise, compositors > probably need a switch: "high quality color management? Then do not use KMS > features."
I think for alpha blending there are already rounding issues depending on the hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
What should we infer from that? That the hardware is broken and exposing those KMS properties is a false promise?
No, just that the scaler doesn't document the internal hardware implementation (number of taps in the filters, coefficients, rounding, ...). That's the rule, not the exception, and it doesn't prevent correct operation, images get scaled in a reproducible way (the same input produces the same output).
If a driver on certain hardware cannot correctly implement a KMS property over the full domain of the input space, should that driver then simply not expose the KMS property at all?
The properties involved here would the the SRC and CRTC rectangles for the planes. They don't document pixel-perfect scaling :-)
But I would assume that the vendor still wants to expose the features in upstream kernels, yet they cannot use the standard KMS properties for that. Should the driver then expose vendor-specific properties with the disclaimer that the result is not always what one would expect, so that userspace written and tested explicitly for that hardware can still work?
That is, a sufficient justification for a vendor-specific KMS property would be that a standard property already exists, but the hardware is too buggy to make it work. IOW, give up trying to make sense.
It's not just about buggy hardware, it's also about implementation specificities, such as rounding, filters, order of operations in the color management pipeline (it's relatively easy when you only have two LUTs and a CCM matrix, but if you through 3D LUTs and other tonemapping features in the mix, not all hardware will implement the same pipeline), or various types of image compression (this device implements a "near-lossless" compression scheme that reduces the memory bandwidth by 50% for instance).
Rounding shouldn't result in needing wide tolerances.
Filters are more difficult, but at least we can factor them out when testing other things. Filters could be tested in isolation with some image difference metrics rather than per-pixel independent comparisons.
The order of operations in the color management pipeline is very important. We can't work with "whatever". All the variability in hardware is exactly why I have been calling out for defined abstract color pipeline in the DRM UAPI docs, so that drivers would know which properties to map to their elements, so that userspace can have any possibility of using them correctly. If the hardware has a block that doesn't fit in the abstract pipeline, you get to add things to the abstract pipeline, or invent a whole another abstract pipeline and document that.
Lossy compression needs its own KMS properties to ensure it can be disabled when necessary. Near-lossless is not lossless, if a difference can be measured. The driver or hardware cannot guess if the end user is ok with "near-lossless" or not, so you have to be conservative and assume not ok, offering an opt-in for lossy.
...
My underlying assumption is that generic userspace will not use vendor-specific properties.
I expect some amount of device-specific code in userspace, yes.
If we had a reliable way to test device-specific code without the hardware and automatically in CI, then maybe.
There are usually large variations in how the hardware exposes access to a given feature, which leads to code having to convert the standard API parameters to hardware parameters. To a large extend this can be done in drivers, but for some more complex features, it may put too much burden on the kernel. There's a reason mesa is a userspace stack :-)
If we get a Khronos standardised 2D composition API... oh wait.
Nothing wrong with userspace libraries, but it does mean that driver developers need to contribute to them, like they do to Mesa. Is there any of that going on for KMS?
In the mean time, DRM UAPI basically must define a 2D composition API, or the new KMS properties will not see use outside of vendor trees.
Or, since we have atomic commits with TEST_ONLY, should it be driver's responsibility to carefully inspect the full state and reject the commit if the hardware is incapable of implementing it correctly? Vendor-specific userspace would know to avoid failing configurations to begin with. I suppose that might put an endless whack-a-mole game on drivers though.
Thanks, pq
Hi Pekka,
On Thu, Jun 17, 2021 at 02:33:11PM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
On Tue, 15 Jun 2021 12:45:57 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 07:15:18AM +0000, Simon Ser wrote: > On Tuesday, June 15th, 2021 at 09:03, Pekka Paalanen wrote: > > > indeed it will, but what else could one do to test userspace KMS > > clients in generic CI where all you can have is virtual hardware? Maybe > > in the long run VKMS needs to loop back to a userspace daemon that > > implements all the complex processing and returns the writeback result > > via VKMS again? That daemon would then need a single upstream, like the > > kernel, where it is maintained and correctness verified. > > The complex processing must be implemented even without write-back, because > user-space can ask for CRCs of the CRTC. > > > Or an LD_PRELOAD that hijacks all KMS ioctls and implements virtual > > stuff in userspace? Didn't someone already have something like that? > > It would need to be lifted to be a required part of kernel UAPI > > submissions, I suppose like IGT is nowadays. > > FWIW, I have a mock libdrm [1] for libliftoff. This is nowhere near a full > software implementation with write-back connectors, but allows to expose > virtual planes and check atomic commits in CI. > > [1]: https://github.com/emersion/libliftoff/blob/master/test/libdrm_mock.c > > > For compositor developers like me knowing the exact formulas would be a huge > > benefit as it would allow me to use KMS to off-load precision-sensitive > > operations (e.g. professional color management). Otherwise, compositors > > probably need a switch: "high quality color management? Then do not use KMS > > features." > > I think for alpha blending there are already rounding issues depending on the > hardware. I wouldn't keep my hopes up for any guarantee that all hw uses the > exact same formulae for color management stuff.
Good, because otherwise you would be very quickly disappointed :-)
For scaling we would also need to replicate the exact same filter taps, which are often not documented.
That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
What should we infer from that? That the hardware is broken and exposing those KMS properties is a false promise?
No, just that the scaler doesn't document the internal hardware implementation (number of taps in the filters, coefficients, rounding, ...). That's the rule, not the exception, and it doesn't prevent correct operation, images get scaled in a reproducible way (the same input produces the same output).
If a driver on certain hardware cannot correctly implement a KMS property over the full domain of the input space, should that driver then simply not expose the KMS property at all?
The properties involved here would the the SRC and CRTC rectangles for the planes. They don't document pixel-perfect scaling :-)
But I would assume that the vendor still wants to expose the features in upstream kernels, yet they cannot use the standard KMS properties for that. Should the driver then expose vendor-specific properties with the disclaimer that the result is not always what one would expect, so that userspace written and tested explicitly for that hardware can still work?
That is, a sufficient justification for a vendor-specific KMS property would be that a standard property already exists, but the hardware is too buggy to make it work. IOW, give up trying to make sense.
It's not just about buggy hardware, it's also about implementation specificities, such as rounding, filters, order of operations in the color management pipeline (it's relatively easy when you only have two LUTs and a CCM matrix, but if you through 3D LUTs and other tonemapping features in the mix, not all hardware will implement the same pipeline), or various types of image compression (this device implements a "near-lossless" compression scheme that reduces the memory bandwidth by 50% for instance).
Rounding shouldn't result in needing wide tolerances.
Filters are more difficult, but at least we can factor them out when testing other things. Filters could be tested in isolation with some image difference metrics rather than per-pixel independent comparisons.
The metrics I was using had both a tolerance on the pixel value and on the number of pixels accepted outside of the value tolerance. I'm sure we can improve it (perhaps taking locality into account), but that's heuristics, and keeping heuristics working across a wide variety of use cases is hard.
The filter I mentioned, by the way, is the scaler filter. Out of curiosity, do any of the devices you work on document with pixel-perfect precision how the hardware scaler is implemented ?
The order of operations in the color management pipeline is very important. We can't work with "whatever". All the variability in hardware is exactly why I have been calling out for defined abstract color pipeline in the DRM UAPI docs, so that drivers would know which properties to map to their elements, so that userspace can have any possibility of using them correctly. If the hardware has a block that doesn't fit in the abstract pipeline, you get to add things to the abstract pipeline, or invent a whole another abstract pipeline and document that.
One very typical difference between devices is the order of the processing blocks. By modelling the KMS pipeline as degamma -> ccm -> gamma, we can accommodate hardware that have any combination of [1-2] * 1D LUTs + 1 * CCM. Now, throw one 3D LUT into the mix, at different points in the pipeline depending on the device, and it will start getting complicated, even if the use case is quite simple and common. This is getting a bit out of topic, but how would you solve this one in particular ?
Lossy compression needs its own KMS properties to ensure it can be disabled when necessary. Near-lossless is not lossless, if a difference can be measured. The driver or hardware cannot guess if the end user is ok with "near-lossless" or not, so you have to be conservative and assume not ok, offering an opt-in for lossy.
Sure, but what would be the barrier to entry for such a property that would enable the compression (it could actually be a pixel format modifier) ? Would it only need to be documented ? Would we need a software implementation in VKMS and/or in IGT ? The compression algorithm is proprietary and not documented, so the latter can't be done.
...
My underlying assumption is that generic userspace will not use vendor-specific properties.
I expect some amount of device-specific code in userspace, yes.
If we had a reliable way to test device-specific code without the hardware and automatically in CI, then maybe.
There are usually large variations in how the hardware exposes access to a given feature, which leads to code having to convert the standard API parameters to hardware parameters. To a large extend this can be done in drivers, but for some more complex features, it may put too much burden on the kernel. There's a reason mesa is a userspace stack :-)
If we get a Khronos standardised 2D composition API... oh wait.
Nothing wrong with userspace libraries, but it does mean that driver developers need to contribute to them, like they do to Mesa. Is there any of that going on for KMS?
Not that I'm aware of, but I think that's a direction we can consider seriously.
In the mean time, DRM UAPI basically must define a 2D composition API, or the new KMS properties will not see use outside of vendor trees.
Or, since we have atomic commits with TEST_ONLY, should it be driver's responsibility to carefully inspect the full state and reject the commit if the hardware is incapable of implementing it correctly? Vendor-specific userspace would know to avoid failing configurations to begin with. I suppose that might put an endless whack-a-mole game on drivers though.
On Thu, 17 Jun 2021 16:37:14 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
Hi Pekka,
On Thu, Jun 17, 2021 at 02:33:11PM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
...
That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
What should we infer from that? That the hardware is broken and exposing those KMS properties is a false promise?
No, just that the scaler doesn't document the internal hardware implementation (number of taps in the filters, coefficients, rounding, ...). That's the rule, not the exception, and it doesn't prevent correct operation, images get scaled in a reproducible way (the same input produces the same output).
If a driver on certain hardware cannot correctly implement a KMS property over the full domain of the input space, should that driver then simply not expose the KMS property at all?
The properties involved here would the the SRC and CRTC rectangles for the planes. They don't document pixel-perfect scaling :-)
But I would assume that the vendor still wants to expose the features in upstream kernels, yet they cannot use the standard KMS properties for that. Should the driver then expose vendor-specific properties with the disclaimer that the result is not always what one would expect, so that userspace written and tested explicitly for that hardware can still work?
That is, a sufficient justification for a vendor-specific KMS property would be that a standard property already exists, but the hardware is too buggy to make it work. IOW, give up trying to make sense.
It's not just about buggy hardware, it's also about implementation specificities, such as rounding, filters, order of operations in the color management pipeline (it's relatively easy when you only have two LUTs and a CCM matrix, but if you through 3D LUTs and other tonemapping features in the mix, not all hardware will implement the same pipeline), or various types of image compression (this device implements a "near-lossless" compression scheme that reduces the memory bandwidth by 50% for instance).
Rounding shouldn't result in needing wide tolerances.
Filters are more difficult, but at least we can factor them out when testing other things. Filters could be tested in isolation with some image difference metrics rather than per-pixel independent comparisons.
The metrics I was using had both a tolerance on the pixel value and on the number of pixels accepted outside of the value tolerance. I'm sure we can improve it (perhaps taking locality into account), but that's heuristics, and keeping heuristics working across a wide variety of use cases is hard.
Hi Laurent,
I was thinking of using a more, um, scientific error measures, e.g. sum or squared errors (SSE) or average SSE over the whole re-scaled/filtered image result, ignoring pixels outside of that.
What one normally uses when matching images in computer vision, for example. Could even add a threshold such that simple rounding-level errors would not even register in SSE. I'm sure there is plenty of literature on that, but it may be behind a paywall like IEEE.
SSE may or may not need to computed from light-linear pixel values, too. If one wanted to go even further, I'm sure there are computational models about human color and brightness difference sensitivity that could be used to weigh the errors.
The filter I mentioned, by the way, is the scaler filter. Out of curiosity, do any of the devices you work on document with pixel-perfect precision how the hardware scaler is implemented ?
I don't work on drivers, so wouldn't even look for hardware docs. I go by what KMS UAPI documents because that is the API I work with and nothing else. And yes, I ignore all the scaling filter issues for now.
Because I don't have a way to get feedback (writeback connectors maybe not existing and not hooked up in Weston quite yet), testing scaling/filtering precision has not been on-topic yet. Right now I'm interested in color correctness rather than geometrical filtering. For traditional color management, the expected pixel values are quite precise.
The order of operations in the color management pipeline is very important. We can't work with "whatever". All the variability in hardware is exactly why I have been calling out for defined abstract color pipeline in the DRM UAPI docs, so that drivers would know which properties to map to their elements, so that userspace can have any possibility of using them correctly. If the hardware has a block that doesn't fit in the abstract pipeline, you get to add things to the abstract pipeline, or invent a whole another abstract pipeline and document that.
One very typical difference between devices is the order of the processing blocks. By modelling the KMS pipeline as degamma -> ccm -> gamma, we can accommodate hardware that have any combination of [1-2] * 1D LUTs + 1 * CCM. Now, throw one 3D LUT into the mix, at
But you cannot represent pipelines like 1D LUT -> 1D LUT -> CCM because the abstract pipeline just doesn't have the elements for that. OTOH, maybe that ordering does not even make sense to have in hardware? So maybe not all combinations are actually needed.
different points in the pipeline depending on the device, and it will start getting complicated, even if the use case is quite simple and common. This is getting a bit out of topic, but how would you solve this one in particular ?
By defining all the points in the abstract color pipeline where a 3D LUT could exist. Then each point would probably need its own KMS property.
We already have the KMS pipeline exactly as degamma -> ctm -> gamma and drivers need to respect that order.
If the combinatorial explosion gets out of hand, maybe we need a KMS property to switch to whole another abstract pipeline which defines a different ordering on the same and/or different KMS properties.
From what I've learnt recently, if you have a 3D LUT, you want a 1D LUT on each side of it for memory vs. precision optimization. And after the degamma -> ctm -> gamma pipeline you may want one more ctm for RGB-to-YCbCr conversion. So I have hope that the abstract pipeline with all actually implemented hardware features might not go totally out of hand.
Lossy compression needs its own KMS properties to ensure it can be disabled when necessary. Near-lossless is not lossless, if a difference can be measured. The driver or hardware cannot guess if the end user is ok with "near-lossless" or not, so you have to be conservative and assume not ok, offering an opt-in for lossy.
Sure, but what would be the barrier to entry for such a property that would enable the compression (it could actually be a pixel format modifier) ? Would it only need to be documented ? Would we need a software implementation in VKMS and/or in IGT ? The compression algorithm is proprietary and not documented, so the latter can't be done.
Good questions. Shows that the idea of strictly requiring a VKMS implementation won't fly, which is what I expected.
Saying it could be a pixel format modifier is a really good point. A modifier cannot be the only thing to control it. Userspace does not decode modifiers, so it cannot filter it out when it wants lossless pixels. There must be something else to control it.
As a userspace dev, I would be ok with documenting a KMS property as "improves blah blah, but also does significant violence to your pixels", so I would know that this is something I need to consider.
You could argue that all KMS properties do violence to pixels, but that's not a useful definition. It would just mean that in some use cases I would never off-load anything to KMS. Depending on the use case that might still be true even if the errors were limited to reasonable rounding errors. I need an idea of how much error does KMS processing do, and ultimately I expect compositors to also need a last resort button for "do not trust KMS processing at all" which then makes the display server always use the exact same simplest possible KMS configuration and let the end user deal with the KMS errors via color-profiling his monitor.
It's quite a different thing to have color processing elements in an unexpected order in the pipeline than it is to have a scaling filter doing slightly unknown operations.
...
My underlying assumption is that generic userspace will not use vendor-specific properties.
I expect some amount of device-specific code in userspace, yes.
If we had a reliable way to test device-specific code without the hardware and automatically in CI, then maybe.
There are usually large variations in how the hardware exposes access to a given feature, which leads to code having to convert the standard API parameters to hardware parameters. To a large extend this can be done in drivers, but for some more complex features, it may put too much burden on the kernel. There's a reason mesa is a userspace stack :-)
If we get a Khronos standardised 2D composition API... oh wait.
Nothing wrong with userspace libraries, but it does mean that driver developers need to contribute to them, like they do to Mesa. Is there any of that going on for KMS?
Not that I'm aware of, but I think that's a direction we can consider seriously.
That would be awesome if the API is generic.
In the mean time, DRM UAPI basically must define a 2D composition API, or the new KMS properties will not see use outside of vendor trees.
Thanks, pq
Hi Pekka,
On Fri, Jun 18, 2021 at 11:55:38AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 16:37:14 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 02:33:11PM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote:
On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
...
> That is where the documented tolerances come into play.
This is something I've experimented with a while ago, when developing automated tests for the rcar-du driver. When playing with different input images we had to constantly increases tolerances, up to a point where the tests started to miss real problems :-(
What should we infer from that? That the hardware is broken and exposing those KMS properties is a false promise?
No, just that the scaler doesn't document the internal hardware implementation (number of taps in the filters, coefficients, rounding, ...). That's the rule, not the exception, and it doesn't prevent correct operation, images get scaled in a reproducible way (the same input produces the same output).
If a driver on certain hardware cannot correctly implement a KMS property over the full domain of the input space, should that driver then simply not expose the KMS property at all?
The properties involved here would the the SRC and CRTC rectangles for the planes. They don't document pixel-perfect scaling :-)
But I would assume that the vendor still wants to expose the features in upstream kernels, yet they cannot use the standard KMS properties for that. Should the driver then expose vendor-specific properties with the disclaimer that the result is not always what one would expect, so that userspace written and tested explicitly for that hardware can still work?
That is, a sufficient justification for a vendor-specific KMS property would be that a standard property already exists, but the hardware is too buggy to make it work. IOW, give up trying to make sense.
It's not just about buggy hardware, it's also about implementation specificities, such as rounding, filters, order of operations in the color management pipeline (it's relatively easy when you only have two LUTs and a CCM matrix, but if you through 3D LUTs and other tonemapping features in the mix, not all hardware will implement the same pipeline), or various types of image compression (this device implements a "near-lossless" compression scheme that reduces the memory bandwidth by 50% for instance).
Rounding shouldn't result in needing wide tolerances.
Filters are more difficult, but at least we can factor them out when testing other things. Filters could be tested in isolation with some image difference metrics rather than per-pixel independent comparisons.
The metrics I was using had both a tolerance on the pixel value and on the number of pixels accepted outside of the value tolerance. I'm sure we can improve it (perhaps taking locality into account), but that's heuristics, and keeping heuristics working across a wide variety of use cases is hard.
Hi Laurent,
I was thinking of using a more, um, scientific error measures, e.g. sum or squared errors (SSE) or average SSE over the whole re-scaled/filtered image result, ignoring pixels outside of that.
There are quite a few matching algorithms implement in imagemagick, we have tried them all, and they all suffered for the issue I initially described; we had to push the thresholds high enough to avoid false positives, and we then saw false negatives.
What one normally uses when matching images in computer vision, for example. Could even add a threshold such that simple rounding-level errors would not even register in SSE. I'm sure there is plenty of literature on that, but it may be behind a paywall like IEEE.
SSE may or may not need to computed from light-linear pixel values, too. If one wanted to go even further, I'm sure there are computational models about human color and brightness difference sensitivity that could be used to weigh the errors.
This isn't about human perception, but about the ability to check if the driver correctly configures the device :-) For driver testing, I don't care if the colours look right or not if userspace misconfigures the colour pipeline. What I need is to catch regressions when my driver messes up the hardware configuration. This particular device processes the images in stripes, and it's very easy for a stripe to be one pixel off (or actually a fraction of a pixel when the scaler is involved).
Testing if a device correctly implements colour processing is of course also useful, but is a distinct goal. Both need to be considered, when stating requirements such as fully documented properties with a VKMS software implementation and pixel-perfect tests in IGT.
The filter I mentioned, by the way, is the scaler filter. Out of curiosity, do any of the devices you work on document with pixel-perfect precision how the hardware scaler is implemented ?
I don't work on drivers, so wouldn't even look for hardware docs. I go by what KMS UAPI documents because that is the API I work with and nothing else. And yes, I ignore all the scaling filter issues for now.
Because I don't have a way to get feedback (writeback connectors maybe not existing and not hooked up in Weston quite yet), testing scaling/filtering precision has not been on-topic yet. Right now I'm interested in color correctness rather than geometrical filtering. For traditional color management, the expected pixel values are quite precise.
The order of operations in the color management pipeline is very important. We can't work with "whatever". All the variability in hardware is exactly why I have been calling out for defined abstract color pipeline in the DRM UAPI docs, so that drivers would know which properties to map to their elements, so that userspace can have any possibility of using them correctly. If the hardware has a block that doesn't fit in the abstract pipeline, you get to add things to the abstract pipeline, or invent a whole another abstract pipeline and document that.
One very typical difference between devices is the order of the processing blocks. By modelling the KMS pipeline as degamma -> ccm -> gamma, we can accommodate hardware that have any combination of [1-2] * 1D LUTs + 1 * CCM. Now, throw one 3D LUT into the mix, at
But you cannot represent pipelines like 1D LUT -> 1D LUT -> CCM because the abstract pipeline just doesn't have the elements for that. OTOH, maybe that ordering does not even make sense to have in hardware? So maybe not all combinations are actually needed.
If userspace wanted such a pipeline (I'm not sure why it would), then it could just combine the two LUTs in one.
different points in the pipeline depending on the device, and it will start getting complicated, even if the use case is quite simple and common. This is getting a bit out of topic, but how would you solve this one in particular ?
By defining all the points in the abstract color pipeline where a 3D LUT could exist. Then each point would probably need its own KMS property.
And when we'll add the next object ? This can't scale I'm afraid. You'll have a spec that tells you that things can be in any order, and may well be able to expose the order to userspace, but you won't be able to implement a generic userspace that makes use of it.
BTW, if we want to expose a finer-grained topology of processing blocks, I'd recommend looking at the Media Controller API, it was designed just for that purpose.
We already have the KMS pipeline exactly as degamma -> ctm -> gamma and drivers need to respect that order.
If the combinatorial explosion gets out of hand, maybe we need a KMS property to switch to whole another abstract pipeline which defines a different ordering on the same and/or different KMS properties.
From what I've learnt recently, if you have a 3D LUT, you want a 1D LUT on each side of it for memory vs. precision optimization. And after the degamma -> ctm -> gamma pipeline you may want one more ctm for RGB-to-YCbCr conversion. So I have hope that the abstract pipeline with all actually implemented hardware features might not go totally out of hand.
On the output of the CRTC, I have, in this order
- RGB to YUV conversion (can only use presets for BT.601 and BT.709, with limited or full range) - 3D LUT - 1D LUT - YUV to RGB conversion (presets only)
The RGB to YUV and YUV to RGB conversions can be bypassed. That's it.
There's also a histogram engine, to allow implementation of dynamic gamma correction. This would require userspace to read the histogram data for every frame, and update the LUTs accordingly.
Lossy compression needs its own KMS properties to ensure it can be disabled when necessary. Near-lossless is not lossless, if a difference can be measured. The driver or hardware cannot guess if the end user is ok with "near-lossless" or not, so you have to be conservative and assume not ok, offering an opt-in for lossy.
Sure, but what would be the barrier to entry for such a property that would enable the compression (it could actually be a pixel format modifier) ? Would it only need to be documented ? Would we need a software implementation in VKMS and/or in IGT ? The compression algorithm is proprietary and not documented, so the latter can't be done.
Good questions. Shows that the idea of strictly requiring a VKMS implementation won't fly, which is what I expected.
Saying it could be a pixel format modifier is a really good point. A modifier cannot be the only thing to control it. Userspace does not decode modifiers, so it cannot filter it out when it wants lossless pixels. There must be something else to control it.
The hardware implementation is such that the decompression is implemented ahead of the display DMA engine, which means that it depends on how the framebuffer memory is mapped. The display controller itself isn't aware of the compression. The driver needs to map framebuffers to the device through a specific memory window to decompress them.
As a userspace dev, I would be ok with documenting a KMS property as "improves blah blah, but also does significant violence to your pixels", so I would know that this is something I need to consider.
You could argue that all KMS properties do violence to pixels, but that's not a useful definition. It would just mean that in some use cases I would never off-load anything to KMS. Depending on the use case that might still be true even if the errors were limited to reasonable rounding errors. I need an idea of how much error does KMS processing do, and ultimately I expect compositors to also need a last resort button for "do not trust KMS processing at all" which then makes the display server always use the exact same simplest possible KMS configuration and let the end user deal with the KMS errors via color-profiling his monitor.
I really think we'll end up needing device-specific userspace components. Given the example above, with histogram calculation and dynamic gamma correction, there's no way we'll be able to have a single userspace implementation that would support these kind of features for all devices. It doesn't have to be solved now, but we probably want to start thinking about an API to plug device-specific components in compositors. This is the kind of problem that the libcamera project is tackling on the camera side, I believe a similar approach will be needed for displays too.
It's quite a different thing to have color processing elements in an unexpected order in the pipeline than it is to have a scaling filter doing slightly unknown operations.
That I agree with, those are two different issues. We all have our cross to bear :-)
...
My underlying assumption is that generic userspace will not use vendor-specific properties.
I expect some amount of device-specific code in userspace, yes.
If we had a reliable way to test device-specific code without the hardware and automatically in CI, then maybe.
There are usually large variations in how the hardware exposes access to a given feature, which leads to code having to convert the standard API parameters to hardware parameters. To a large extend this can be done in drivers, but for some more complex features, it may put too much burden on the kernel. There's a reason mesa is a userspace stack :-)
If we get a Khronos standardised 2D composition API... oh wait.
Nothing wrong with userspace libraries, but it does mean that driver developers need to contribute to them, like they do to Mesa. Is there any of that going on for KMS?
Not that I'm aware of, but I think that's a direction we can consider seriously.
That would be awesome if the API is generic.
In the mean time, DRM UAPI basically must define a 2D composition API, or the new KMS properties will not see use outside of vendor trees.
On Fri, 18 Jun 2021 12:58:49 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
Hi Pekka,
On Fri, Jun 18, 2021 at 11:55:38AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 16:37:14 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 02:33:11PM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote: > On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
...
This isn't about human perception, but about the ability to check if the driver correctly configures the device :-) For driver testing, I don't care if the colours look right or not if userspace misconfigures the colour pipeline. What I need is to catch regressions when my driver messes up the hardware configuration. This particular device processes the images in stripes, and it's very easy for a stripe to be one pixel off (or actually a fraction of a pixel when the scaler is involved).
Testing if a device correctly implements colour processing is of course also useful, but is a distinct goal. Both need to be considered, when stating requirements such as fully documented properties with a VKMS software implementation and pixel-perfect tests in IGT.
Right. Your driver testing requirements are different. You know how your hardware works, so you can build your tests to suit.
What this thread is about is KMS UAPI, and verifying that a KMS implementation matches the KMS UAPI specification. The UAPI specification is the only thing I can look at when writing userspace. As a userspace and compositor developer, I am specifically concerned about how humans will perceive the final image.
If you use your hardware-specific tests to ensure that your driver programs the hardware correctly, and IGT tests ensure that the KMS UAPI is implemented to the KMS UAPI spec, then I would hope that everything is fine. By reading the KMS UAPI spec I can decide to use or not use specific KMS features.
...in an ideal world.
One very typical difference between devices is the order of the processing blocks. By modelling the KMS pipeline as degamma -> ccm -> gamma, we can accommodate hardware that have any combination of [1-2] * 1D LUTs + 1 * CCM. Now, throw one 3D LUT into the mix, at
But you cannot represent pipelines like 1D LUT -> 1D LUT -> CCM because the abstract pipeline just doesn't have the elements for that. OTOH, maybe that ordering does not even make sense to have in hardware? So maybe not all combinations are actually needed.
If userspace wanted such a pipeline (I'm not sure why it would), then it could just combine the two LUTs in one.
Maybe? You can also combine the 1D LUTs into the 3D LUT in the middle, too, but the result is not generally the same as using them separately when the 3D LUT size is limited.
different points in the pipeline depending on the device, and it will start getting complicated, even if the use case is quite simple and common. This is getting a bit out of topic, but how would you solve this one in particular ?
By defining all the points in the abstract color pipeline where a 3D LUT could exist. Then each point would probably need its own KMS property.
And when we'll add the next object ? This can't scale I'm afraid. You'll have a spec that tells you that things can be in any order, and may well be able to expose the order to userspace, but you won't be able to implement a generic userspace that makes use of it.
Are you saying that generic userspace cannot happen? On top of KMS UAPI, that is.
Why would a generic userspace library API be a more feasible effort?
BTW, if we want to expose a finer-grained topology of processing blocks, I'd recommend looking at the Media Controller API, it was designed just for that purpose.
Sure. However, right now we have KMS properties instead. Or maybe they should just go all unused?
Maybe the rule should be to not add any more KMS properties and tell people to design something completely new based on Media Controller API instead?
I've never looked at Media Controller API, so I have no idea how it would suit display servers.
I very much think that I cannot use processing blocks exposed by KMS if I cannot have guarantees that they work the same across all Linux drivers.
We already have the KMS pipeline exactly as degamma -> ctm -> gamma and drivers need to respect that order.
If the combinatorial explosion gets out of hand, maybe we need a KMS property to switch to whole another abstract pipeline which defines a different ordering on the same and/or different KMS properties.
From what I've learnt recently, if you have a 3D LUT, you want a 1D LUT on each side of it for memory vs. precision optimization. And after the degamma -> ctm -> gamma pipeline you may want one more ctm for RGB-to-YCbCr conversion. So I have hope that the abstract pipeline with all actually implemented hardware features might not go totally out of hand.
On the output of the CRTC, I have, in this order
- RGB to YUV conversion (can only use presets for BT.601 and BT.709, with limited or full range)
- 3D LUT
- 1D LUT
- YUV to RGB conversion (presets only)
The RGB to YUV and YUV to RGB conversions can be bypassed. That's it.
There's also a histogram engine, to allow implementation of dynamic gamma correction. This would require userspace to read the histogram data for every frame, and update the LUTs accordingly.
Interesting, but almost none of that would be used by the color management pipeline model I'm familiar with. I wonder what kind of use cases doing LUTs in YUV have been designed for, and *after* blending, even.
...
I really think we'll end up needing device-specific userspace components. Given the example above, with histogram calculation and dynamic gamma correction, there's no way we'll be able to have a single userspace implementation that would support these kind of features for all devices. It doesn't have to be solved now, but we probably want to start thinking about an API to plug device-specific components in compositors. This is the kind of problem that the libcamera project is tackling on the camera side, I believe a similar approach will be needed for displays too.
To be able to plug device-specific components in compositors, we need a driver-agnostic API through which compositors can use those device specific components. Still the same problem, now just in userspace than at the UAPI level.
Thanks, pq
Hi Pekka,
On Fri, Jun 18, 2021 at 02:32:00PM +0300, Pekka Paalanen wrote:
On Fri, 18 Jun 2021 12:58:49 +0300 Laurent Pinchart wrote:
On Fri, Jun 18, 2021 at 11:55:38AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 16:37:14 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 02:33:11PM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote: > On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote: > > On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
...
This isn't about human perception, but about the ability to check if the driver correctly configures the device :-) For driver testing, I don't care if the colours look right or not if userspace misconfigures the colour pipeline. What I need is to catch regressions when my driver messes up the hardware configuration. This particular device processes the images in stripes, and it's very easy for a stripe to be one pixel off (or actually a fraction of a pixel when the scaler is involved).
Testing if a device correctly implements colour processing is of course also useful, but is a distinct goal. Both need to be considered, when stating requirements such as fully documented properties with a VKMS software implementation and pixel-perfect tests in IGT.
Right. Your driver testing requirements are different. You know how your hardware works, so you can build your tests to suit.
What this thread is about is KMS UAPI, and verifying that a KMS implementation matches the KMS UAPI specification. The UAPI specification is the only thing I can look at when writing userspace. As a userspace and compositor developer, I am specifically concerned about how humans will perceive the final image.
Yes, the thread has gone in a few different directions.
If you use your hardware-specific tests to ensure that your driver programs the hardware correctly, and IGT tests ensure that the KMS UAPI is implemented to the KMS UAPI spec, then I would hope that everything is fine. By reading the KMS UAPI spec I can decide to use or not use specific KMS features.
...in an ideal world.
I wish I knew where that world was ;-)
I do agree with you though, you need predictability in userspace, otherwise it's quite useless (or not very useful). This brings back many memories (so unfortunately not too distant) with cameras. No latter than yesterday, I heard from a person with lots of involvement in GPU API design that GPUs seem easy compared to cameras. Hopefully this means we'll be able to solve this on the display side :-)
One very typical difference between devices is the order of the processing blocks. By modelling the KMS pipeline as degamma -> ccm -> gamma, we can accommodate hardware that have any combination of [1-2] * 1D LUTs + 1 * CCM. Now, throw one 3D LUT into the mix, at
But you cannot represent pipelines like 1D LUT -> 1D LUT -> CCM because the abstract pipeline just doesn't have the elements for that. OTOH, maybe that ordering does not even make sense to have in hardware? So maybe not all combinations are actually needed.
If userspace wanted such a pipeline (I'm not sure why it would), then it could just combine the two LUTs in one.
Maybe? You can also combine the 1D LUTs into the 3D LUT in the middle, too, but the result is not generally the same as using them separately when the 3D LUT size is limited.
Yes, my 3D LUT has 17 points, I've heard about LUTs up to 33 points (with bilinear interpolation), that's quite little if used as a 1D LUT. There's also the fact that interpolation will result in cross-talk between the three colour channels.
different points in the pipeline depending on the device, and it will start getting complicated, even if the use case is quite simple and common. This is getting a bit out of topic, but how would you solve this one in particular ?
By defining all the points in the abstract color pipeline where a 3D LUT could exist. Then each point would probably need its own KMS property.
And when we'll add the next object ? This can't scale I'm afraid. You'll have a spec that tells you that things can be in any order, and may well be able to expose the order to userspace, but you won't be able to implement a generic userspace that makes use of it.
Are you saying that generic userspace cannot happen? On top of KMS UAPI, that is.
Why would a generic userspace library API be a more feasible effort?
I'll make a parallel with the camera world again, because I think display hardware is taking a similar path, just later.
We used to have capture devices that offered a fairly high-level interface at the hardware or firmware level, with a set of features that were more or less standard. Devices could be exposed with an abstract model, as done in KMS today, with a set of standardized controls (V4L2 uses the word control when KMS uses property). We had way less automated testing, and didn't enough attention to consistency of the userspace API (we're talking about code from the beginning of the century, if not earlier, the world was different), so different drivers were not consistent in how the API was implemented (for instance the API didn't specify gain units, so drivers would use different units), but more or less, you could write a standard V4L2 application that would work with most devices, with a few quirks here and there.
Then the world changed. Cameras become more complex, with the raw sensor and the ISP exposed to Linux in the OMAP3 that Nokia used in the N900 and N9 phones. For a while we tried to stick with V4L2, and implement ISP control within the kernel, with the help of a userspace daemon behind the scenes to which the kernel driver called back. The daemon was required, as control of the camera required complicated computation that really benefited from floating point math. It was horrible, and the daemon was closed-source, but from an application point of view, it was still high-level V4L2 as before. This hit a big wall though, to expose a high-level API on top of a low-level hardware interface, we had to bridge the gap in the kernel. This meant hardcoding use cases on the kernel side, which clearly wouldn't scale, as not everybody had the same use cases (an industrial vision application and a smartphone camera application really don't see the world the same way). We were facing a dead end.
We then decided to expose a lower-level API from kernel to userspace, giving userspace the ability to control everything. That's how the Media Controller API was born (its purpose is to expose the internal topology of the device as a graph of connected entities, that's about it), as well as the V4L2 subdev userspace API (in KMS term, that would mean more or less exposing DRM bridges to userspace, with the ability to control them individually). This simplified the kernel implementation (no more hardcoded use cases, no more daemon behind the scenes), but at the same time resulted in an API that wasn't a good fit for applications to consume directly. One step closer to the hardware, one step further from the user. You can think of this as the API exposed by GPU kernel drivers, it's not enough, a userspace component to translate to a higher-level API is needed.
This was followed by 10 years of limbo as Nokia pulled the plug from Linux (and then smartphone) development. Fast forward, at the end of 2018, we created libcamera as the mesa of the camera world to solve this issue. It offers a standard camera stack with a high-level API, where device-specific code can be added (and kept to the minimum, as a centralized framework means we can share code that is common between devices).
I see lots of parallel with display pipelines getting more complex. It wouldn't be fair to expose a very low-level API and ask every single compositor to know about all the devices out there, but at the same time, we're reaching a level of hardware complexity that I believe makes it a dead end to try to abstract and standardize everything at the kernel level. This doesn't mean we should allow all vendors to throw in any kind of crappy properties or other API extensions, we can still enforce some ground rules, such as proper documentation, reasonable effort for standardization, and reference open-source userspace implementation in a project we would consider to be the equalivant of mesa for KMS.
BTW, if we want to expose a finer-grained topology of processing blocks, I'd recommend looking at the Media Controller API, it was designed just for that purpose.
Sure. However, right now we have KMS properties instead. Or maybe they should just go all unused?
Maybe the rule should be to not add any more KMS properties and tell people to design something completely new based on Media Controller API instead?
No, I think that would be too harsh, my point was that if we want to expose how the building blocks are layed out, the MC API is meant to expose a graph of connected entities, so it's a good match. Controlling the device should still go through KMS properties I believe.
I've never looked at Media Controller API, so I have no idea how it would suit display servers.
I very much think that I cannot use processing blocks exposed by KMS if I cannot have guarantees that they work the same across all Linux drivers.
We already have the KMS pipeline exactly as degamma -> ctm -> gamma and drivers need to respect that order.
If the combinatorial explosion gets out of hand, maybe we need a KMS property to switch to whole another abstract pipeline which defines a different ordering on the same and/or different KMS properties.
From what I've learnt recently, if you have a 3D LUT, you want a 1D LUT on each side of it for memory vs. precision optimization. And after the degamma -> ctm -> gamma pipeline you may want one more ctm for RGB-to-YCbCr conversion. So I have hope that the abstract pipeline with all actually implemented hardware features might not go totally out of hand.
On the output of the CRTC, I have, in this order
- RGB to YUV conversion (can only use presets for BT.601 and BT.709, with limited or full range)
- 3D LUT
- 1D LUT
- YUV to RGB conversion (presets only)
The RGB to YUV and YUV to RGB conversions can be bypassed. That's it.
There's also a histogram engine, to allow implementation of dynamic gamma correction. This would require userspace to read the histogram data for every frame, and update the LUTs accordingly.
Interesting, but almost none of that would be used by the color management pipeline model I'm familiar with. I wonder what kind of use cases doing LUTs in YUV have been designed for, and *after* blending, even.
I'm afraid I can't tell as I don't know. This reaches the limit of my display colour management knowledge.
...
I really think we'll end up needing device-specific userspace components. Given the example above, with histogram calculation and dynamic gamma correction, there's no way we'll be able to have a single userspace implementation that would support these kind of features for all devices. It doesn't have to be solved now, but we probably want to start thinking about an API to plug device-specific components in compositors. This is the kind of problem that the libcamera project is tackling on the camera side, I believe a similar approach will be needed for displays too.
To be able to plug device-specific components in compositors, we need a driver-agnostic API through which compositors can use those device specific components. Still the same problem, now just in userspace than at the UAPI level.
Agreed, it pushes the problem to a different place :-) I however think it may be easier to solve it in userspace, as it's a better place to hardcode use cases. A general-purpose compositor may not want to make use of all hardware features, while a specific-purpose compositor may want to exercise the hardware features differently. This however opens the door to introducing badly-designed and un(der-)documented properties, which is not something I want, so we would still need to set ground rules, such as documentation and reference implementation in a given userspace compositor/stack.
On that note, I think we need to do a bit more hand-holding. I'm sitting on a patch series that is several years old to add 3D LUT support to my driver, and while I believe I'm quite experienced with KMS development, adding support for this in a reference compositor in userspace is a really big task that I don't know how to tackle. I also know very little about colour management, and have nobody in the team who has that knowledge and could help me. I'm thus facing a bit of a dead-end, not because I don't want to spend the time required for this, but because the bar is so high that I just can't do it. I would expect many developers to be facing the same challenges. If the community doesn't make an effort in fostering collaboration between developers (most kernel developers won't even know who to ask for help if they have to step out of the kernel to userspace) and providing good documentation, we'll just scare developers away. It took me weeks to figure out how to run Weston on this device, as it has no GPU that I can use, and falling back to a software compositor required finding out that lots of documentation is outdated and referenced command line options that were not accurate anymore (or I could mention painfully finding out that an issue I was blocked with for days was solved by plugging in a USB mouse to the board, as Weston didn't want to start without an input device). Another developer from my team tried before me and gave up, thinking it was impossible. That's the reality that many kernel developers are facing, the same way I suppose than me saying saying "you just need to recompile the kernel with this option enabled" is not very helpful for lots of userspace developers.
On Fri, 18 Jun 2021 15:19:15 +0300 Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
Hi Pekka,
On Fri, Jun 18, 2021 at 02:32:00PM +0300, Pekka Paalanen wrote:
On Fri, 18 Jun 2021 12:58:49 +0300 Laurent Pinchart wrote:
On Fri, Jun 18, 2021 at 11:55:38AM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 16:37:14 +0300 Laurent Pinchart wrote:
On Thu, Jun 17, 2021 at 02:33:11PM +0300, Pekka Paalanen wrote:
On Thu, 17 Jun 2021 13:29:48 +0300 Laurent Pinchart wrote: > On Thu, Jun 17, 2021 at 10:27:01AM +0300, Pekka Paalanen wrote: > > On Thu, 17 Jun 2021 00:05:24 +0300 Laurent Pinchart wrote: > > > On Tue, Jun 15, 2021 at 01:16:56PM +0300, Pekka Paalanen wrote:
...
One very typical difference between devices is the order of the processing blocks. By modelling the KMS pipeline as degamma -> ccm -> gamma, we can accommodate hardware that have any combination of [1-2] * 1D LUTs + 1 * CCM. Now, throw one 3D LUT into the mix, at
But you cannot represent pipelines like 1D LUT -> 1D LUT -> CCM because the abstract pipeline just doesn't have the elements for that. OTOH, maybe that ordering does not even make sense to have in hardware? So maybe not all combinations are actually needed.
If userspace wanted such a pipeline (I'm not sure why it would), then it could just combine the two LUTs in one.
Maybe? You can also combine the 1D LUTs into the 3D LUT in the middle, too, but the result is not generally the same as using them separately when the 3D LUT size is limited.
Yes, my 3D LUT has 17 points, I've heard about LUTs up to 33 points (with bilinear interpolation), that's quite little if used as a 1D LUT. There's also the fact that interpolation will result in cross-talk between the three colour channels.
Hi Laurent,
that is not what I meant, but that's true too.
What I meant was this: https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/582#note_9512...
pre_curve and post_curve are the 1D LUTs on each side of a size-limited 3D LUT. pre_curve controls the 3D LUT tap positioning to concentrate on the areas where the approximated function needs more taps.
Why would a generic userspace library API be a more feasible effort?
I'll make a parallel with the camera world again, because I think display hardware is taking a similar path, just later.
We used to have capture devices that offered a fairly high-level interface at the hardware or firmware level, with a set of features that were more or less standard. Devices could be exposed with an abstract model, as done in KMS today, with a set of standardized controls (V4L2 uses the word control when KMS uses property). We had way less automated testing, and didn't enough attention to consistency of the userspace API (we're talking about code from the beginning of the century, if not earlier, the world was different), so different drivers were not consistent in how the API was implemented (for instance the API didn't specify gain units, so drivers would use different units), but more or less, you could write a standard V4L2 application that would work with most devices, with a few quirks here and there.
Then the world changed. Cameras become more complex, with the raw sensor and the ISP exposed to Linux in the OMAP3 that Nokia used in the N900 and N9 phones. For a while we tried to stick with V4L2, and implement ISP control within the kernel, with the help of a userspace daemon behind the scenes to which the kernel driver called back. The daemon was required, as control of the camera required complicated computation that really benefited from floating point math. It was horrible, and the daemon was closed-source, but from an application point of view, it was still high-level V4L2 as before. This hit a big wall though, to expose a high-level API on top of a low-level hardware interface, we had to bridge the gap in the kernel. This meant hardcoding use cases on the kernel side, which clearly wouldn't scale, as not everybody had the same use cases (an industrial vision application and a smartphone camera application really don't see the world the same way). We were facing a dead end.
We then decided to expose a lower-level API from kernel to userspace, giving userspace the ability to control everything. That's how the Media Controller API was born (its purpose is to expose the internal topology of the device as a graph of connected entities, that's about it), as well as the V4L2 subdev userspace API (in KMS term, that would mean more or less exposing DRM bridges to userspace, with the ability to control them individually). This simplified the kernel implementation (no more hardcoded use cases, no more daemon behind the scenes), but at the same time resulted in an API that wasn't a good fit for applications to consume directly. One step closer to the hardware, one step further from the user. You can think of this as the API exposed by GPU kernel drivers, it's not enough, a userspace component to translate to a higher-level API is needed.
This was followed by 10 years of limbo as Nokia pulled the plug from Linux (and then smartphone) development. Fast forward, at the end of 2018, we created libcamera as the mesa of the camera world to solve this issue. It offers a standard camera stack with a high-level API, where device-specific code can be added (and kept to the minimum, as a centralized framework means we can share code that is common between devices).
I see lots of parallel with display pipelines getting more complex. It wouldn't be fair to expose a very low-level API and ask every single compositor to know about all the devices out there, but at the same time, we're reaching a level of hardware complexity that I believe makes it a dead end to try to abstract and standardize everything at the kernel level. This doesn't mean we should allow all vendors to throw in any kind of crappy properties or other API extensions, we can still enforce some ground rules, such as proper documentation, reasonable effort for standardization, and reference open-source userspace implementation in a project we would consider to be the equalivant of mesa for KMS.
Thank you very much for explaining this! It makes perfect sense to me. It also sounds like a ¤&!%¤# amount of work, but really does sound like the best plan in the long term.
GPUs have OpenGL and Vulkan, but I suppose with cameras and display, there is nothing standard to aim for, for either hardware designers nor software designers? Chicken-and-egg.
BTW, if we want to expose a finer-grained topology of processing blocks, I'd recommend looking at the Media Controller API, it was designed just for that purpose.
Sure. However, right now we have KMS properties instead. Or maybe they should just go all unused?
Maybe the rule should be to not add any more KMS properties and tell people to design something completely new based on Media Controller API instead?
No, I think that would be too harsh, my point was that if we want to expose how the building blocks are layed out, the MC API is meant to expose a graph of connected entities, so it's a good match. Controlling the device should still go through KMS properties I believe.
Ok.
...
To be able to plug device-specific components in compositors, we need a driver-agnostic API through which compositors can use those device specific components. Still the same problem, now just in userspace than at the UAPI level.
Agreed, it pushes the problem to a different place :-) I however think it may be easier to solve it in userspace, as it's a better place to hardcode use cases. A general-purpose compositor may not want to make use of all hardware features, while a specific-purpose compositor may want to exercise the hardware features differently. This however opens the door to introducing badly-designed and un(der-)documented properties, which is not something I want, so we would still need to set ground rules, such as documentation and reference implementation in a given userspace compositor/stack.
Right, sounds fine, but I think it also shifts the KMS UAPI paradigm a little, away from generic and towards driver-specific. If the DRM maintainers agree to that, I'm fine with it. It's just different from what I have understood so far, which is why I'm so keen on getting UAPI documented precisely.
On that note, I think we need to do a bit more hand-holding. I'm sitting on a patch series that is several years old to add 3D LUT support to my driver, and while I believe I'm quite experienced with KMS development, adding support for this in a reference compositor in userspace is a really big task that I don't know how to tackle. I also know very little about colour management, and have nobody in the team who has that knowledge and could help me. I'm thus facing a bit of a dead-end, not because I don't want to spend the time required for this, but because the bar is so high that I just can't do it. I would expect many developers to be facing the same challenges. If the community doesn't make an effort in fostering collaboration between developers (most kernel developers won't even know who to ask for help if they have to step out of the kernel to userspace) and providing good documentation, we'll just scare developers away.
That's why I'm screaming here, to make contact. :-D
And Simor Ser is another, and I have seen him even offer to implement userspace bits for new KMS properties in development.
But a 3D LUT in display is a really tiny piece in the landscape of color management. Color management on Wayland has been talked about, oh I don't remember, for a decade? Weston code has been experimented on for the past few years at least. But only last week, the very first piece of actual color management exclusive code was merged in Weston upstream.
Getting a display server project into a phase where it would actually start experimenting on KMS features can take a decade - not just work but finding the funding. Weston is not quite there yet either, not even with all the currently open merge requests.
Then, if your 3D LUT is after blending, I'm not sure I would have use for it in Weston! If it was on a KMS plane before blending, then yes, I would have use for it. All this depends on how the blending color space is chosen, and the idea we have right now does not have use for a 3D LUT after blending - except maybe for fullscreen single-buffer scenegraph, e.g. video and games.
It took me weeks to figure out how to run Weston on this device, as it has no GPU that I can use, and falling back to a software compositor required finding out that lots of documentation is outdated and referenced command line options that were not accurate anymore (or I could mention painfully finding out that an issue I was blocked with for days was solved by plugging in a USB mouse to the board, as Weston didn't want to start without an input device). Another developer from my team tried before me and gave up, thinking it was impossible. That's the reality that many kernel developers are facing, the same way I suppose than me saying saying "you just need to recompile the kernel with this option enabled" is not very helpful for lots of userspace developers.
Why have I not heard of your problems with Weston?!
We have Gitlab issues, mailing list and IRC. You're warmly welcome to ask when you hit a wall. :-)
I even read #dri-devel very often, so if you mentioned Weston there, it would have had a good chance to reach me.
Thanks, pq
dri-devel@lists.freedesktop.org
-
Alyssa Rosenzweig -
Daniel Vetter -
Daniel Vetter -
Laurent Pinchart -
Liviu Dudau -
Maxime Ripard -
Pekka Paalanen -
Rodrigo Vivi -
Simon Ser -
Tomi Valkeinen