Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts, -Brian
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything. -Daniel
On Mon, Aug 21, 2017 at 06:01:24PM +0200, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This problem already exists (ignoring modifiers) in the case of any v4l2 <-> DRM buffer sharing (direct video scanout, for instance).
I was hoping it would be possible to draw enough equivalency between the different definitions to manage a useful subset through a 1:1 lookup table. If that's not possible then this gets a whole lot more tricky. Are you already aware of incompatibilities which would prevent it?
-Brian
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
-Daniel
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
Regards,
Hans
Hi Hans,
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
Thanks, -Brian
Regards,
Hans
On 08/24/17 13:14, Brian Starkey wrote:
Hi Hans,
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Regards,
Hans
On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
On 08/24/17 13:14, Brian Starkey wrote:
Hi Hans,
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
Cheers, -Brian
Regards,
Hans
On 24/08/17 14:26, Brian Starkey wrote:
On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
On 08/24/17 13:14, Brian Starkey wrote:
Hi Hans,
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
So this patch limits the number of formats to 64 (being the size of the bit mask). I was hoping these modifiers applied to all formats, but unfortunately that isn't the case apparently.
How it would work with my proposal is that the integer menu control would reflect the list of supported modifiers for the currently selected format. If you change format, then the available modifier list changes as well. The advantage is that there is no '64 formats' limitation, something that I feel very uncomfortable about since some devices support a *lot* of formats. The disadvantage is that it is harder to get a quick overview of all combinations for formats and modifiers.
This has more to do with limitations in the V4L2 API than with supporting modifiers in general. We need something better to give userspace a quick overview of all combinations of pixelformats, framesizes, frameintervals and now modifiers. However, that's our problem :-)
Regards,
Hans
Cheers, -Brian
Regards,
Hans
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
On Fri, Aug 25, 2017 at 10:14:03AM +0200, Hans Verkuil wrote:
On 24/08/17 14:26, Brian Starkey wrote:
On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
On 08/24/17 13:14, Brian Starkey wrote:
Hi Hans,
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey brian.starkey@arm.com wrote: > Hi all, > > I couldn't find this topic talked about elsewhere, but apologies if > it's a duplicate - I'll be glad to be steered in the direction of a > thread. > > We'd like to support DRM format modifiers in v4l2 in order to share > the description of different (mostly proprietary) buffer formats > between e.g. a v4l2 device and a DRM device. > > DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and > are a vendor-namespaced 64-bit value used to describe various > vendor-specific buffer layouts. They are combined with a (DRM) FourCC > code to give a complete description of the data contained in a buffer. > > The same modifier definition is used in the Khronos EGL extension > EGL_EXT_image_dma_buf_import_modifiers, and is supported in the > Wayland linux-dmabuf protocol. > > > This buffer information could of course be described in the > vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the > information already defined in drm_fourcc.h. Additionally, there > would be quite a format explosion where a device supports a dozen or > more formats, all of which can use one or more different > layouts/compression schemes. > > So, I'm wondering if anyone has views on how/whether this could be > incorporated? > > I spoke briefly about this to Laurent at LPC last year, and he > suggested v4l2_control as one approach. > > I also wondered if could be added in v4l2_pix_format_mplane - looks > like there's 8 bytes left before it exceeds the 200 bytes, or could go > in the reserved portion of v4l2_plane_pix_format. > > Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
So this patch limits the number of formats to 64 (being the size of the bit mask).
It's not limited to 64 formats. Right now no DRM drivers support more than 64 formats, but when they do, the "offset" field in struct drm_format_modifier can be set to 64 and then the bit mask represents formats 65 through 128 (see the comment on that struct):
* If the number formats grew to 128, and formats 98-102 are * supported with the modifier: * * 0x0000003c00000000 0000000000000000 * ^ * |__offset = 64, formats = 0x3c00000000
I was hoping these modifiers applied to all formats, but unfortunately that isn't the case apparently.
Yeah, if only it were so simple :-)
How it would work with my proposal is that the integer menu control would reflect the list of supported modifiers for the currently selected format. If you change format, then the available modifier list changes as well.
Ah yes, I need to get used to thinking about stateful APIs - that works.
The advantage is that there is no '64 formats' limitation, something that I feel very uncomfortable about since some devices support a *lot* of formats. The disadvantage is that it is harder to get a quick overview of all combinations for formats and modifiers.
This has more to do with limitations in the V4L2 API than with supporting modifiers in general. We need something better to give userspace a quick overview of all combinations of pixelformats, framesizes, frameintervals and now modifiers. However, that's our problem :-)
Regards,
Hans
Thanks for the inputs,
-Brian
Cheers, -Brian
Regards,
Hans
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi Brian,
On Tuesday, 29 August 2017 12:19:43 EEST Brian Starkey wrote:
On Fri, Aug 25, 2017 at 10:14:03AM +0200, Hans Verkuil wrote:
On 24/08/17 14:26, Brian Starkey wrote:
On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
On 08/24/17 13:14, Brian Starkey wrote:
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote: > On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey wrote: >> Hi all, >> >> I couldn't find this topic talked about elsewhere, but apologies if >> it's a duplicate - I'll be glad to be steered in the direction of a >> thread. >> >> We'd like to support DRM format modifiers in v4l2 in order to share >> the description of different (mostly proprietary) buffer formats >> between e.g. a v4l2 device and a DRM device. >> >> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h >> and are a vendor-namespaced 64-bit value used to describe various >> vendor-specific buffer layouts. They are combined with a (DRM) >> FourCC code to give a complete description of the data contained in >> a buffer. >> >> The same modifier definition is used in the Khronos EGL extension >> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the >> Wayland linux-dmabuf protocol. >> >> >> This buffer information could of course be described in the >> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the >> information already defined in drm_fourcc.h. Additionally, there >> would be quite a format explosion where a device supports a dozen or >> more formats, all of which can use one or more different >> layouts/compression schemes. >> >> So, I'm wondering if anyone has views on how/whether this could be >> incorporated? >> >> I spoke briefly about this to Laurent at LPC last year, and he >> suggested v4l2_control as one approach. >> >> I also wondered if could be added in v4l2_pix_format_mplane - looks >> like there's 8 bytes left before it exceeds the 200 bytes, or could >> go in the reserved portion of v4l2_plane_pix_format. >> >> Thanks for any thoughts, > > One problem is that the modifers sometimes reference the DRM fourcc > codes. v4l has a different (and incompatible set) of fourcc codes, > whereas all the protocols and specs (you can add DRI3.1 for Xorg to > that list btw) use both drm fourcc and drm modifiers. > > This might or might not make this proposal unworkable, but it's > something I'd at least review carefully. > > Otherwise I think it'd be great if we could have one namespace for > all modifiers, that's pretty much why we have them. Please also note > that for drm_fourcc.h we don't require an in-kernel user for a new > modifier since a bunch of them might need to be allocated just for > userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example > for this would be compressed surfaces with fast-clearing, which is > planned for i915 (but current hw can't scan it out). And we really > want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
So this patch limits the number of formats to 64 (being the size of the bit mask).
It's not limited to 64 formats. Right now no DRM drivers support more than 64 formats, but when they do, the "offset" field in struct drm_format_modifier can be set to 64 and then the bit mask represents formats 65 through 128 (see the comment on that struct):
- If the number formats grew to 128, and formats 98-102 are
- supported with the modifier:
- 0x0000003c00000000 0000000000000000
^|__offset = 64, formats = 0x3c00000000I was hoping these modifiers applied to all formats, but unfortunately that isn't the case apparently.
Yeah, if only it were so simple :-)
How it would work with my proposal is that the integer menu control would reflect the list of supported modifiers for the currently selected format. If you change format, then the available modifier list changes as well.
Ah yes, I need to get used to thinking about stateful APIs - that works.
No, you don't, we need to make enumeration stateless :-)
The advantage is that there is no '64 formats' limitation, something that I feel very uncomfortable about since some devices support a *lot* of formats. The disadvantage is that it is harder to get a quick overview of all combinations for formats and modifiers.
This has more to do with limitations in the V4L2 API than with supporting modifiers in general. We need something better to give userspace a quick overview of all combinations of pixelformats, framesizes, frameintervals and now modifiers. However, that's our problem :-)
Le jeudi 24 août 2017 à 13:26 +0100, Brian Starkey a écrit :
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
The problem is that the list of modifiers depends on the format selected. Having to call S_FMT to obtain this list is quite inefficient.
Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think an other one made it the same way recently, something from Mediatek if I remember. Though, unlike the Intel one, the same modifier does not have various result depending on the hardware revision.
regards, Nicolas
On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne nicolas@ndufresne.ca wrote:
Le jeudi 24 août 2017 à 13:26 +0100, Brian Starkey a écrit :
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
The problem is that the list of modifiers depends on the format selected. Having to call S_FMT to obtain this list is quite inefficient.
Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think an other one made it the same way recently, something from Mediatek if I remember. Though, unlike the Intel one, the same modifier does not have various result depending on the hardware revision.
Note on the intel modifers: On most recent platforms (iirc gen9) the modifier is well defined and always describes the same byte layout. We simply didn't want to rewrite our entire software stack for all the old gunk platforms, hence the language. I guess we could/should describe the layout in detail, but atm we're the only ones using it.
On your topic of v4l2 encoding the drm fourcc+modifier combo into a special v4l fourcc: That's exactly the mismatch I was thinking of. There's other examples of v4l2 fourcc being more specific than their drm counters (e.g. specific way the different planes are laid out). -Daniel
On Mon, Aug 28, 2017 at 10:49:07PM +0200, Daniel Vetter wrote:
On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne nicolas@ndufresne.ca wrote:
Le jeudi 24 ao??t 2017 ?? 13:26 +0100, Brian Starkey a ??crit :
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
The problem is that the list of modifiers depends on the format selected. Having to call S_FMT to obtain this list is quite inefficient.
Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think an other one made it the same way recently, something from Mediatek if I remember. Though, unlike the Intel one, the same modifier does not have various result depending on the hardware revision.
Note on the intel modifers: On most recent platforms (iirc gen9) the modifier is well defined and always describes the same byte layout. We simply didn't want to rewrite our entire software stack for all the old gunk platforms, hence the language. I guess we could/should describe the layout in detail, but atm we're the only ones using it.
On your topic of v4l2 encoding the drm fourcc+modifier combo into a special v4l fourcc: That's exactly the mismatch I was thinking of. There's other examples of v4l2 fourcc being more specific than their drm counters (e.g. specific way the different planes are laid out).
I'm not entirely clear on the v4l2 fourccs being more specific than DRM ones - do you mean e.g. NV12 vs NV12M? Specifically in the case of multi-planar formats I think it's a non-issue because modifiers are allowed to alter the number of planes and the meanings of them. Also V4L2 NV12M is a superset of NV12 - so NV12M would always be able to describe a DRM NV12 buffer.
I don't see the "special v4l2 format already exists" case as a problem either. It would be up to any drivers that already have special formats to decide if they want to also support it via a more generic modifiers API or not.
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Cheers, -Brian
-Daniel
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
On Mon, Aug 28, 2017 at 10:49:07PM +0200, Daniel Vetter wrote:
On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne nicolas@ndufresne.ca wrote:
Le jeudi 24 ao??t 2017 ?? 13:26 +0100, Brian Starkey a ??crit :
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
The problem is that the list of modifiers depends on the format selected. Having to call S_FMT to obtain this list is quite inefficient.
Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think an other one made it the same way recently, something from Mediatek if I remember. Though, unlike the Intel one, the same modifier does not have various result depending on the hardware revision.
Note on the intel modifers: On most recent platforms (iirc gen9) the modifier is well defined and always describes the same byte layout. We simply didn't want to rewrite our entire software stack for all the old gunk platforms, hence the language. I guess we could/should describe the layout in detail, but atm we're the only ones using it.
On your topic of v4l2 encoding the drm fourcc+modifier combo into a special v4l fourcc: That's exactly the mismatch I was thinking of. There's other examples of v4l2 fourcc being more specific than their drm counters (e.g. specific way the different planes are laid out).
I'm not entirely clear on the v4l2 fourccs being more specific than DRM ones - do you mean e.g. NV12 vs NV12M? Specifically in the case of multi-planar formats I think it's a non-issue because modifiers are allowed to alter the number of planes and the meanings of them. Also V4L2 NV12M is a superset of NV12 - so NV12M would always be able to describe a DRM NV12 buffer.
I don't see the "special v4l2 format already exists" case as a problem either. It would be up to any drivers that already have special formats to decide if they want to also support it via a more generic modifiers API or not.
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-) -Daniel
On 30/08/17 09:50, Daniel Vetter wrote:
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
On Mon, Aug 28, 2017 at 10:49:07PM +0200, Daniel Vetter wrote:
On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne nicolas@ndufresne.ca wrote:
Le jeudi 24 ao??t 2017 ?? 13:26 +0100, Brian Starkey a ??crit :
What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it.
But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware.
Yes, makes sense.
> > However, in DRM the API lets you get the supported formats for each > modifier as-well-as the modifier list itself. I'm not sure how exactly > to provide that in a control.
We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers.
So enumerating these modifiers would work out-of-the-box.
Right. So I guess the supported set of formats could be somehow enumerated in the menu item string. In DRM the pairs are (modifier + bitmask) where bits represent formats in the supported formats list (commit db1689aa61bd in drm-next). Printing a hex representation of the bitmask would be functional but I concede not very pretty.
The problem is that the list of modifiers depends on the format selected. Having to call S_FMT to obtain this list is quite inefficient.
Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think an other one made it the same way recently, something from Mediatek if I remember. Though, unlike the Intel one, the same modifier does not have various result depending on the hardware revision.
Note on the intel modifers: On most recent platforms (iirc gen9) the modifier is well defined and always describes the same byte layout. We simply didn't want to rewrite our entire software stack for all the old gunk platforms, hence the language. I guess we could/should describe the layout in detail, but atm we're the only ones using it.
On your topic of v4l2 encoding the drm fourcc+modifier combo into a special v4l fourcc: That's exactly the mismatch I was thinking of. There's other examples of v4l2 fourcc being more specific than their drm counters (e.g. specific way the different planes are laid out).
I'm not entirely clear on the v4l2 fourccs being more specific than DRM ones - do you mean e.g. NV12 vs NV12M? Specifically in the case of multi-planar formats I think it's a non-issue because modifiers are allowed to alter the number of planes and the meanings of them. Also V4L2 NV12M is a superset of NV12 - so NV12M would always be able to describe a DRM NV12 buffer.
I don't see the "special v4l2 format already exists" case as a problem either. It would be up to any drivers that already have special formats to decide if they want to also support it via a more generic modifiers API or not.
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-)
Probably not something we'll do: there are I believe only two drivers that are affected (exynos & mediatek), so they can do that in their driver.
Question: how many modifiers will typically apply to a format? I ask because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link between a fourcc and modifiers:
https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
The __u32 reserved[4] array can be used to provide a bitmask to modifier indices (for the integer menu control). It's similar to what drm does, except instead of modifiers mapping to fourccs it is the other way around.
This would avoid having to change the modifiers control whenever a new format is set and it makes it easy to enumerate all combinations.
But this only works if the total number of modifiers used by a single driver is expected to remain small (let's say no more than 64).
Regards,
Hans
On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
On 30/08/17 09:50, Daniel Vetter wrote:
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-)
Probably not something we'll do: there are I believe only two drivers that are affected (exynos & mediatek), so they can do that in their driver.
Question: how many modifiers will typically apply to a format? I ask because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link between a fourcc and modifiers:
https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
The __u32 reserved[4] array can be used to provide a bitmask to modifier indices (for the integer menu control). It's similar to what drm does, except instead of modifiers mapping to fourccs it is the other way around.
This would avoid having to change the modifiers control whenever a new format is set and it makes it easy to enumerate all combinations.
But this only works if the total number of modifiers used by a single driver is expected to remain small (let's say no more than 64).
In our current (yet to be submitted) description, we've got around a dozen modifiers for any one format to describe our compression variants. We have a lot of on/off toggles which leads to combinatorial expansion, so it can grow pretty quickly (though I am trying to limit the valid combinations as much as possible).
How about if the mask fills up then VIDIOC_ENUM_FMT can return another fmtdsc with the same FourCC and different modifier bitmask, where the second one's modifier bitmask is for the next "N" modifiers?
-Brian
Regards,
Hans
On 30/08/17 11:36, Brian Starkey wrote:
On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
On 30/08/17 09:50, Daniel Vetter wrote:
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-)
Probably not something we'll do: there are I believe only two drivers that are affected (exynos & mediatek), so they can do that in their driver.
Question: how many modifiers will typically apply to a format? I ask because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link between a fourcc and modifiers:
https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
The __u32 reserved[4] array can be used to provide a bitmask to modifier indices (for the integer menu control). It's similar to what drm does, except instead of modifiers mapping to fourccs it is the other way around.
This would avoid having to change the modifiers control whenever a new format is set and it makes it easy to enumerate all combinations.
But this only works if the total number of modifiers used by a single driver is expected to remain small (let's say no more than 64).
In our current (yet to be submitted) description, we've got around a dozen modifiers for any one format to describe our compression variants. We have a lot of on/off toggles which leads to combinatorial expansion, so it can grow pretty quickly (though I am trying to limit the valid combinations as much as possible).
How about if the mask fills up then VIDIOC_ENUM_FMT can return another fmtdsc with the same FourCC and different modifier bitmask, where the second one's modifier bitmask is for the next "N" modifiers?
I was thinking along similar lines, but it could cause some problems with the ABI since applications currently assume that no fourcc will appear twice when enumerating formats. Admittedly, we never explicitly said in the spec that that can't happen, but it is kind of expected.
There are ways around that, but if possible I'd like to avoid that.
In theory there are up to 128 bits available but I can't help thinking that if you create more than, say, 64 modifiers for a HW platform you have a big mess anyway.
If I am wrong, then I need to know because then I can prepare for it (or whoever is going to actually implement this...)
If the number of modifiers is expected to be limited then making 64 bits available would be good enough, at least for now.
BTW, is a modifier always optional? I.e. for all fourccs, is the unmodified format always available? Or are there fourccs that require the use of a modifier?
Regards,
Hans
On Wed, Aug 30, 2017 at 11:53:58AM +0200, Hans Verkuil wrote:
On 30/08/17 11:36, Brian Starkey wrote:
On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
On 30/08/17 09:50, Daniel Vetter wrote:
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-)
Probably not something we'll do: there are I believe only two drivers that are affected (exynos & mediatek), so they can do that in their driver.
Question: how many modifiers will typically apply to a format? I ask because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link between a fourcc and modifiers:
https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
The __u32 reserved[4] array can be used to provide a bitmask to modifier indices (for the integer menu control). It's similar to what drm does, except instead of modifiers mapping to fourccs it is the other way around.
This would avoid having to change the modifiers control whenever a new format is set and it makes it easy to enumerate all combinations.
But this only works if the total number of modifiers used by a single driver is expected to remain small (let's say no more than 64).
In our current (yet to be submitted) description, we've got around a dozen modifiers for any one format to describe our compression variants. We have a lot of on/off toggles which leads to combinatorial expansion, so it can grow pretty quickly (though I am trying to limit the valid combinations as much as possible).
How about if the mask fills up then VIDIOC_ENUM_FMT can return another fmtdsc with the same FourCC and different modifier bitmask, where the second one's modifier bitmask is for the next "N" modifiers?
I was thinking along similar lines, but it could cause some problems with the ABI since applications currently assume that no fourcc will appear twice when enumerating formats. Admittedly, we never explicitly said in the spec that that can't happen, but it is kind of expected.
There are ways around that, but if possible I'd like to avoid that.
In theory there are up to 128 bits available but I can't help thinking that if you create more than, say, 64 modifiers for a HW platform you have a big mess anyway.
If I am wrong, then I need to know because then I can prepare for it (or whoever is going to actually implement this...)
You're probably right, but I can't speak for everyone. From the current state of drm_fourcc.h it looks like 64 would be plenty (there aren't anywhere near 64 modifiers even defined right now). Adding in the Arm compression formats will expand it a lot, but still not to 64 (yet).
If the number of modifiers is expected to be limited then making 64 bits available would be good enough, at least for now.
BTW, is a modifier always optional? I.e. for all fourccs, is the unmodified format always available? Or are there fourccs that require the use of a modifier?
We do actually have one or two formats which are only supported with a modifier (on our HW).
-Brian
Regards,
Hans
Hi Brian,
On Wednesday, 30 August 2017 13:32:01 EEST Brian Starkey wrote:
On Wed, Aug 30, 2017 at 11:53:58AM +0200, Hans Verkuil wrote:
On 30/08/17 11:36, Brian Starkey wrote:
On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
On 30/08/17 09:50, Daniel Vetter wrote:
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
The fact is, adding special formats for each combination is unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-)
Probably not something we'll do: there are I believe only two drivers that are affected (exynos & mediatek), so they can do that in their driver.
Question: how many modifiers will typically apply to a format? I ask because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link between a fourcc and modifiers:
https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
The __u32 reserved[4] array can be used to provide a bitmask to modifier indices (for the integer menu control). It's similar to what drm does, except instead of modifiers mapping to fourccs it is the other way around.
This would avoid having to change the modifiers control whenever a new format is set and it makes it easy to enumerate all combinations.
But this only works if the total number of modifiers used by a single driver is expected to remain small (let's say no more than 64).
In our current (yet to be submitted) description, we've got around a dozen modifiers for any one format to describe our compression variants. We have a lot of on/off toggles which leads to combinatorial expansion, so it can grow pretty quickly (though I am trying to limit the valid combinations as much as possible).
How about if the mask fills up then VIDIOC_ENUM_FMT can return another fmtdsc with the same FourCC and different modifier bitmask, where the second one's modifier bitmask is for the next "N" modifiers?
I was thinking along similar lines, but it could cause some problems with the ABI since applications currently assume that no fourcc will appear twice when enumerating formats. Admittedly, we never explicitly said in the spec that that can't happen, but it is kind of expected.
There are ways around that, but if possible I'd like to avoid that.
In theory there are up to 128 bits available but I can't help thinking that if you create more than, say, 64 modifiers for a HW platform you have a big mess anyway.
If I am wrong, then I need to know because then I can prepare for it (or whoever is going to actually implement this...)
You're probably right, but I can't speak for everyone. From the current state of drm_fourcc.h it looks like 64 would be plenty (there aren't anywhere near 64 modifiers even defined right now). Adding in the Arm compression formats will expand it a lot, but still not to 64 (yet).
Do all those modifiers make sense on the V4L2 side ? I expect that some modifiers will mostly be used for buffers shared between the GPU and the display engine, while others will be used by codecs. The sets will likely overlap, but might not be identical.
If the number of modifiers is expected to be limited then making 64 bits available would be good enough, at least for now.
BTW, is a modifier always optional? I.e. for all fourccs, is the unmodified format always available? Or are there fourccs that require the use of a modifier?
We do actually have one or two formats which are only supported with a modifier (on our HW).
Hi Laurent,
On Thu, Aug 31, 2017 at 05:51:33PM +0300, Laurent Pinchart wrote:
Hi Brian,
On Wednesday, 30 August 2017 13:32:01 EEST Brian Starkey wrote:
On Wed, Aug 30, 2017 at 11:53:58AM +0200, Hans Verkuil wrote:
On 30/08/17 11:36, Brian Starkey wrote:
On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
On 30/08/17 09:50, Daniel Vetter wrote:
On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote: > The fact is, adding special formats for each combination is > unmanageable - we're talking dozens in the case of our hardware.
Hm right, we can just remap the special combos to the drm-fourcc + modifier style. Bonus point if v4l does that in the core so not everyone has to reinvent that wheel :-)
Probably not something we'll do: there are I believe only two drivers that are affected (exynos & mediatek), so they can do that in their driver.
Question: how many modifiers will typically apply to a format? I ask because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link between a fourcc and modifiers:
https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
The __u32 reserved[4] array can be used to provide a bitmask to modifier indices (for the integer menu control). It's similar to what drm does, except instead of modifiers mapping to fourccs it is the other way around.
This would avoid having to change the modifiers control whenever a new format is set and it makes it easy to enumerate all combinations.
But this only works if the total number of modifiers used by a single driver is expected to remain small (let's say no more than 64).
In our current (yet to be submitted) description, we've got around a dozen modifiers for any one format to describe our compression variants. We have a lot of on/off toggles which leads to combinatorial expansion, so it can grow pretty quickly (though I am trying to limit the valid combinations as much as possible).
How about if the mask fills up then VIDIOC_ENUM_FMT can return another fmtdsc with the same FourCC and different modifier bitmask, where the second one's modifier bitmask is for the next "N" modifiers?
I was thinking along similar lines, but it could cause some problems with the ABI since applications currently assume that no fourcc will appear twice when enumerating formats. Admittedly, we never explicitly said in the spec that that can't happen, but it is kind of expected.
There are ways around that, but if possible I'd like to avoid that.
In theory there are up to 128 bits available but I can't help thinking that if you create more than, say, 64 modifiers for a HW platform you have a big mess anyway.
If I am wrong, then I need to know because then I can prepare for it (or whoever is going to actually implement this...)
You're probably right, but I can't speak for everyone. From the current state of drm_fourcc.h it looks like 64 would be plenty (there aren't anywhere near 64 modifiers even defined right now). Adding in the Arm compression formats will expand it a lot, but still not to 64 (yet).
Do all those modifiers make sense on the V4L2 side ? I expect that some modifiers will mostly be used for buffers shared between the GPU and the display engine, while others will be used by codecs. The sets will likely overlap, but might not be identical.
All of the ones from drm_fourcc.h - I expect not. In the case of Arm's framebuffer compression though, it's used in all our media IPs; Display, GPU and Video, and ideally on all data exchanged between them. For the most part the modifiers describing it could apply to all three.
There's also the question of what you're calling a codec - mem2mem rotation blocks/scalers/etc. exposed via V4L2 would likely want the same set as any display device which consumes their output (which I think puts them firmly in the shared between the XXX and display engine camp).
Cheers, -Brian
If the number of modifiers is expected to be limited then making 64 bits available would be good enough, at least for now.
BTW, is a modifier always optional? I.e. for all fourccs, is the unmodified format always available? Or are there fourccs that require the use of a modifier?
We do actually have one or two formats which are only supported with a modifier (on our HW).
-- Regards,
Laurent Pinchart
Hi Brian,
On Thursday, 24 August 2017 14:14:31 EEST Brian Starkey wrote:
On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
On 08/21/2017 06:01 PM, Daniel Vetter wrote:
On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey wrote:
Hi all,
I couldn't find this topic talked about elsewhere, but apologies if it's a duplicate - I'll be glad to be steered in the direction of a thread.
We'd like to support DRM format modifiers in v4l2 in order to share the description of different (mostly proprietary) buffer formats between e.g. a v4l2 device and a DRM device.
DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and are a vendor-namespaced 64-bit value used to describe various vendor-specific buffer layouts. They are combined with a (DRM) FourCC code to give a complete description of the data contained in a buffer.
The same modifier definition is used in the Khronos EGL extension EGL_EXT_image_dma_buf_import_modifiers, and is supported in the Wayland linux-dmabuf protocol.
This buffer information could of course be described in the vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the information already defined in drm_fourcc.h. Additionally, there would be quite a format explosion where a device supports a dozen or more formats, all of which can use one or more different layouts/compression schemes.
So, I'm wondering if anyone has views on how/whether this could be incorporated?
I spoke briefly about this to Laurent at LPC last year, and he suggested v4l2_control as one approach.
I also wondered if could be added in v4l2_pix_format_mplane - looks like there's 8 bytes left before it exceeds the 200 bytes, or could go in the reserved portion of v4l2_plane_pix_format.
We're considering reworking the format ioctls at some point. We don't necessarily need to wait until then to implement support for modifiers, but we will have an opportunity to integrate them with formats at that point.
Thanks for any thoughts,
One problem is that the modifers sometimes reference the DRM fourcc codes. v4l has a different (and incompatible set) of fourcc codes, whereas all the protocols and specs (you can add DRI3.1 for Xorg to that list btw) use both drm fourcc and drm modifiers.
This might or might not make this proposal unworkable, but it's something I'd at least review carefully.
Otherwise I think it'd be great if we could have one namespace for all modifiers, that's pretty much why we have them. Please also note that for drm_fourcc.h we don't require an in-kernel user for a new modifier since a bunch of them might need to be allocated just for userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example for this would be compressed surfaces with fast-clearing, which is planned for i915 (but current hw can't scan it out). And we really want to have one namespace for everything.
Who sets these modifiers? Kernel or userspace? Or can it be set by both? I assume any userspace code that sets/reads this is code specific for that hardware?
I think normally the modifier would be set by userspace. However it might not necessarily be device-specific code. In DRM the intention is for userspace to query the set of modifiers which are supported, and then use them without necessarily knowing exactly what they mean (insofar as that is possible).
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
Userspace could certainly set modifiers blindly, but the point of modifiers is to generate side effects benefitial to the use case at hand (for instance by optimizing the memory access pattern). To use them meaningfully userspace would need to have at least an idea of the side effects they generate.
I think Laurent's suggestion of using a 64 bit V4L2 control for this makes the most sense.
Especially if you can assume that whoever sets this knows the hardware.
I think this only makes sense if you pass buffers from one HW device to another.
Because you cannot expect generic video capture code to be able to interpret all the zillion different combinations of modifiers.
I don't quite follow this last bit. The control could report the set of supported modifiers.
However, in DRM the API lets you get the supported formats for each modifier as-well-as the modifier list itself. I'm not sure how exactly to provide that in a control.
Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
Userspace could certainly set modifiers blindly, but the point of modifiers is to generate side effects benefitial to the use case at hand (for instance by optimizing the memory access pattern). To use them meaningfully userspace would need to have at least an idea of the side effects they generate.
Generic userspace will basically pick some random combination. To allow generically picking the optimal configuration we could indeed rely on the application knowledge, but we could also enhance the spec so that the order in the enumeration becomes meaningful.
regards, Nicolas
Hi Nicolas,
On Thursday, 31 August 2017 19:12:58 EEST Nicolas Dufresne wrote:
Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
Userspace could certainly set modifiers blindly, but the point of modifiers is to generate side effects benefitial to the use case at hand (for instance by optimizing the memory access pattern). To use them meaningfully userspace would need to have at least an idea of the side effects they generate.
Generic userspace will basically pick some random combination.
In that case userspace could set no modifier at all by default (except in the case where unmodified formats are not supported by the hardware, but I don't expect that to be the most common case).
To allow generically picking the optimal configuration we could indeed rely on the application knowledge, but we could also enhance the spec so that the order in the enumeration becomes meaningful.
I'm not sure how far we should go. I could imagine a system where the API would report capabilities for modifiers (e.g. this modifier lowers the bandwidth, this one enhances the quality, ...), but going in that direction, where do we stop ? In practice I expect userspace to know some information about the hardware, so I'd rather avoid over-engineering the API.
On Fri, Sep 1, 2017 at 3:13 AM, Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
Hi Nicolas,
On Thursday, 31 August 2017 19:12:58 EEST Nicolas Dufresne wrote:
Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
Userspace could certainly set modifiers blindly, but the point of modifiers is to generate side effects benefitial to the use case at hand (for instance by optimizing the memory access pattern). To use them meaningfully userspace would need to have at least an idea of the side effects they generate.
Generic userspace will basically pick some random combination.
In that case userspace could set no modifier at all by default (except in the case where unmodified formats are not supported by the hardware, but I don't expect that to be the most common case).
To allow generically picking the optimal configuration we could indeed rely on the application knowledge, but we could also enhance the spec so that the order in the enumeration becomes meaningful.
I'm not sure how far we should go. I could imagine a system where the API would report capabilities for modifiers (e.g. this modifier lowers the bandwidth, this one enhances the quality, ...), but going in that direction, where do we stop ? In practice I expect userspace to know some information about the hardware, so I'd rather avoid over-engineering the API.
I think in the (hopefully not too) long term, something like https://github.com/cubanismo/allocator/ is the way forward. That doesn't quite solve how v4l2 kernel part sorts out w/ corresponding userspace .so what is preferable, but at least that is compartmentalized to v4l2.. on the gl/vk side of things there will ofc be a hardware specific userspace part that knows what it prefers. For v4l2, it probably makes sense to sort out what the userspace level API is and work backwards from there, rather than risk trying to design a kernel uapi that might turn out to be the wrong thing.
BR, -R
On Fri, Sep 1, 2017 at 2:43 PM, Rob Clark robdclark@gmail.com wrote:
On Fri, Sep 1, 2017 at 3:13 AM, Laurent Pinchart laurent.pinchart@ideasonboard.com wrote:
Hi Nicolas,
On Thursday, 31 August 2017 19:12:58 EEST Nicolas Dufresne wrote:
Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
e.g. if I have two devices which support MODIFIER_FOO, I could attempt to share a buffer between them which uses MODIFIER_FOO without necessarily knowing exactly what it is/does.
Userspace could certainly set modifiers blindly, but the point of modifiers is to generate side effects benefitial to the use case at hand (for instance by optimizing the memory access pattern). To use them meaningfully userspace would need to have at least an idea of the side effects they generate.
Generic userspace will basically pick some random combination.
In that case userspace could set no modifier at all by default (except in the case where unmodified formats are not supported by the hardware, but I don't expect that to be the most common case).
To allow generically picking the optimal configuration we could indeed rely on the application knowledge, but we could also enhance the spec so that the order in the enumeration becomes meaningful.
I'm not sure how far we should go. I could imagine a system where the API would report capabilities for modifiers (e.g. this modifier lowers the bandwidth, this one enhances the quality, ...), but going in that direction, where do we stop ? In practice I expect userspace to know some information about the hardware, so I'd rather avoid over-engineering the API.
I think in the (hopefully not too) long term, something like https://github.com/cubanismo/allocator/ is the way forward. That doesn't quite solve how v4l2 kernel part sorts out w/ corresponding userspace .so what is preferable, but at least that is compartmentalized to v4l2.. on the gl/vk side of things there will ofc be a hardware specific userspace part that knows what it prefers. For v4l2, it probably makes sense to sort out what the userspace level API is and work backwards from there, rather than risk trying to design a kernel uapi that might turn out to be the wrong thing.
I thought for kms the plan is to make the ordering meaningful, because it doesn't necessarily match the gl/vk one. E.g. on intel gl would prefer Y compressed, Y, X, untiled. Whereas display would be Y compressed, X (much easier to scan out, in many cases allows more planes to be used), Y (is necessary for 90° rotation), untiled. So if drm_hwc really wants to use all the planes, it could prioritize the display over rendering and request X instead of Y tiled.
I think the same would go for v4l. -Daniel
dri-devel@lists.freedesktop.org
-
Brian Starkey -
Daniel Vetter -
Hans Verkuil -
Laurent Pinchart -
Nicolas Dufresne -
Rob Clark