On Fri, Apr 16, 2021 at 12:25 AM Ian Romanick idr@freedesktop.org wrote:

On 4/15/21 8:59 AM, Matthew Auld wrote:

...

Add a note about the two-step process.

Suggested-by: Daniel Vetter daniel@ffwll.ch Signed-off-by: Matthew Auld matthew.auld@intel.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Jordan Justen jordan.l.justen@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Kenneth Graunke kenneth@whitecape.org Cc: Jason Ekstrand jason@jlekstrand.net Cc: Dave Airlie airlied@gmail.com Cc: dri-devel@lists.freedesktop.org Cc: mesa-dev@lists.freedesktop.org

---

include/uapi/drm/i915_drm.h | 57 ++++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 11 deletions(-)

diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h index d9c954a5a456..ef36f1a0adde 100644 --- a/include/uapi/drm/i915_drm.h +++ b/include/uapi/drm/i915_drm.h @@ -2210,14 +2210,23 @@ struct drm_i915_perf_oa_config { __u64 flex_regs_ptr; };

+/**

• • struct drm_i915_query_item - An individual query for the kernel to process.
• • The behaviour is determined by the @query_id. Note that exactly what

Since we just had a big discussion about this on mesa-dev w.r.t. Mesa code and documentation... does the kernel have a policy about which flavor (pun intended) of English should be used?

I'm not finding it documented in https://dri.freedesktop.org/docs/drm/doc-guide/sphinx.html but I thought we've discussed it. Adding linux-doc and Jon Corbet. -Daniel

...

• • @data_ptr is also depends on the specific @query_id.
• */

struct drm_i915_query_item {

• /** @query_id: The id for this query */
  __u64 query_id;
  

#define DRM_I915_QUERY_TOPOLOGY_INFO 1 #define DRM_I915_QUERY_ENGINE_INFO 2 #define DRM_I915_QUERY_PERF_CONFIG 3 /* Must be kept compact -- no holes and well documented */

• /*
  

• /**
  

•  * @length:
  

•  *
   * When set to zero by userspace, this is filled with the size of the
   * data to be written at the data_ptr pointer. The kernel sets this
   * value to a negative value to signal an error on a particular query
  

@@ -2225,21 +2234,26 @@ struct drm_i915_query_item { */ __s32 length;

• /*
  

• /**
  

•  * @flags:
  

•  *
   * When query_id == DRM_I915_QUERY_TOPOLOGY_INFO, must be 0.
   *
   * When query_id == DRM_I915_QUERY_PERF_CONFIG, must be one of the
  

•  * following :
  

•  *         - DRM_I915_QUERY_PERF_CONFIG_LIST
  

•  *         - DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID
  

•  *         - DRM_I915_QUERY_PERF_CONFIG_FOR_UUID
  

•  * following:
  

•  *
  

•  *      - DRM_I915_QUERY_PERF_CONFIG_LIST
  

•  *      - DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID
  

•  *      - DRM_I915_QUERY_PERF_CONFIG_FOR_UUID
   */
  __u32 flags;
  

#define DRM_I915_QUERY_PERF_CONFIG_LIST 1 #define DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID 2 #define DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_ID 3

• /*
  

• /**
  

•  * @data_ptr:
  

•  *
   * Data will be written at the location pointed by data_ptr when the
   * value of length matches the length of the data to be written by the
   * kernel.
  

@@ -2247,16 +2261,37 @@ struct drm_i915_query_item { __u64 data_ptr; };

+/**

• • struct drm_i915_query - Supply an array of drm_i915_query_item for the kernel
• • to fill out.
• • Note that this is generally a two step process for each drm_i915_query_item
• • in the array:
• • 1.) Call the DRM_IOCTL_I915_QUERY, giving it our array of
• • drm_i915_query_item, with drm_i915_query_item.size set to zero. The
• • kernel will then fill in the size, in bytes, which tells userspace how
• • memory it needs to allocate for the blob(say for an array of
• • properties).
• • 2.) Next we call DRM_IOCTL_I915_QUERY again, this time with the
• • drm_i915_query_item.data_ptr equal to our newly allocated blob. Note
• • that the i915_query_item.size should still be the same as what the
• • kernel previously set. At this point the kernel can fill in the blob.
• */

struct drm_i915_query {

• /** @num_items: The number of elements in the @items_ptr array */
  __u32 num_items;
  
  

• /*
  

•  * Unused for now. Must be cleared to zero.
  

• /**
  

•  * @flags: Unused for now. Must be cleared to zero.
   */
  __u32 flags;
  
  

• /*
  

•  * This points to an array of num_items drm_i915_query_item structures.
  

• /**
  

•  * @items_ptr: This points to an array of num_items drm_i915_query_item
  

•  * structures.
   */
  __u64 items_ptr;
  

};

---

mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev

On 15/04/2021 16:59, Matthew Auld wrote:

Add a note about the two-step process.

Suggested-by: Daniel Vetter daniel@ffwll.ch Signed-off-by: Matthew Auld matthew.auld@intel.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Jordan Justen jordan.l.justen@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Kenneth Graunke kenneth@whitecape.org Cc: Jason Ekstrand jason@jlekstrand.net Cc: Dave Airlie airlied@gmail.com Cc: dri-devel@lists.freedesktop.org Cc: mesa-dev@lists.freedesktop.org

---

include/uapi/drm/i915_drm.h | 57 ++++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 11 deletions(-)

diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h index d9c954a5a456..ef36f1a0adde 100644 --- a/include/uapi/drm/i915_drm.h +++ b/include/uapi/drm/i915_drm.h @@ -2210,14 +2210,23 @@ struct drm_i915_perf_oa_config { __u64 flex_regs_ptr; };

+/**

• • struct drm_i915_query_item - An individual query for the kernel to process.
• • The behaviour is determined by the @query_id. Note that exactly what
• • @data_ptr is also depends on the specific @query_id.
• */ struct drm_i915_query_item {
• /** @query_id: The id for this query */ __u64 query_id; #define DRM_I915_QUERY_TOPOLOGY_INFO 1 #define DRM_I915_QUERY_ENGINE_INFO 2 #define DRM_I915_QUERY_PERF_CONFIG 3 /* Must be kept compact -- no holes and well documented */

• /*

• /**

• * @length:
  

• *
  

  • When set to zero by userspace, this is filled with the size of the
  • data to be written at the data_ptr pointer. The kernel sets this
  • value to a negative value to signal an error on a particular query

@@ -2225,21 +2234,26 @@ struct drm_i915_query_item { */ __s32 length;

• /*

• /**

• * @flags:
  

• *
  

  • When query_id == DRM_I915_QUERY_TOPOLOGY_INFO, must be 0.
  • When query_id == DRM_I915_QUERY_PERF_CONFIG, must be one of the

• * following :
  

• *         - DRM_I915_QUERY_PERF_CONFIG_LIST
  

• *         - DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID
  

• *         - DRM_I915_QUERY_PERF_CONFIG_FOR_UUID
  

• * following:
  

• *
  

• *	- DRM_I915_QUERY_PERF_CONFIG_LIST
  

• *      - DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID
  

• *      - DRM_I915_QUERY_PERF_CONFIG_FOR_UUID
  

  */ __u32 flags; #define DRM_I915_QUERY_PERF_CONFIG_LIST 1 #define DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID 2 #define DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_ID 3

• /*

• /**

• * @data_ptr:
  

• *
  

  • Data will be written at the location pointed by data_ptr when the
  • value of length matches the length of the data to be written by the
  • kernel.

@@ -2247,16 +2261,37 @@ struct drm_i915_query_item { __u64 data_ptr; };

+/**

• • struct drm_i915_query - Supply an array of drm_i915_query_item for the kernel
• • to fill out.
• • Note that this is generally a two step process for each drm_i915_query_item
• • in the array:
• • 1.) Call the DRM_IOCTL_I915_QUERY, giving it our array of
• • drm_i915_query_item, with drm_i915_query_item.size set to zero. The
• • kernel will then fill in the size, in bytes, which tells userspace how
• • memory it needs to allocate for the blob(say for an array of
• • properties).
• • 2.) Next we call DRM_IOCTL_I915_QUERY again, this time with the
• • drm_i915_query_item.data_ptr equal to our newly allocated blob. Note
• • that the i915_query_item.size should still be the same as what the
• • kernel previously set. At this point the kernel can fill in the blob.

I'd also document the one step alternative where userspace passes in a buffer equal or larger than the required size. For many small queries (most?) this actually works just as well and with one ioctl only.

Regards,

Tvrtko

• */ struct drm_i915_query {
• /** @num_items: The number of elements in the @items_ptr array */ __u32 num_items;

• /*

• * Unused for now. Must be cleared to zero.
  

• /**

• * @flags: Unused for now. Must be cleared to zero.
  

  */ __u32 flags;

• /*

• * This points to an array of num_items drm_i915_query_item structures.
  

• /**

• * @items_ptr: This points to an array of num_items drm_i915_query_item
  

• * structures.
  

  */ __u64 items_ptr; };

On Thu, Apr 15, 2021 at 11:04 AM Matthew Auld matthew.auld@intel.com wrote:

Add an entry for the new uAPI needed for DG1.

v2(Daniel):

• include the overall upstreaming plan
• add a note for mmap, there are differences here for TTM vs i915
• bunch of other suggestions from Daniel

v3: (Daniel)

• add a note for set/get caching stuff
• add some more docs for existing query and extensions stuff
• add an actual code example for regions query
• bunch of other stuff

(Jason)

• uAPI change(!): - try a simpler design with the placements extension - rather than have a generic setparam which can cover multiple use cases, have each extension be responsible for one thing only

Signed-off-by: Matthew Auld matthew.auld@intel.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Jordan Justen jordan.l.justen@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Kenneth Graunke kenneth@whitecape.org Cc: Jason Ekstrand jason@jlekstrand.net Cc: Dave Airlie airlied@gmail.com Cc: dri-devel@lists.freedesktop.org Cc: mesa-dev@lists.freedesktop.org

---

Documentation/gpu/rfc/i915_gem_lmem.h | 255 ++++++++++++++++++++++++ Documentation/gpu/rfc/i915_gem_lmem.rst | 139 +++++++++++++ Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 398 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.h create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.rst

diff --git a/Documentation/gpu/rfc/i915_gem_lmem.h b/Documentation/gpu/rfc/i915_gem_lmem.h new file mode 100644 index 000000000000..2a82a452e9f2 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.h @@ -0,0 +1,255 @@ +/*

• • Note that drm_i915_query_item and drm_i915_query are existing bits of uAPI.
• • For the regions query we are just adding a new query id, so no actual new
• • ioctl or anything, but including it here for reference.
• */

+struct drm_i915_query_item { +#define DRM_I915_QUERY_MEMORY_REGIONS 0xdeadbeaf

•   ....
  

•    __u64 query_id;
  

•    /*
  

•     * When set to zero by userspace, this is filled with the size of the
  

•     * data to be written at the data_ptr pointer. The kernel sets this
  

•     * value to a negative value to signal an error on a particular query
  

•     * item.
  

•     */
  

•    __s32 length;
  

•    __u32 flags;
  

•    /*
  

•     * Data will be written at the location pointed by data_ptr when the
  

•     * value of length matches the length of the data to be written by the
  

•     * kernel.
  

•     */
  

•    __u64 data_ptr;
  

+};

+struct drm_i915_query {

•    __u32 num_items;
  

•    /*
  

•     * Unused for now. Must be cleared to zero.
  

•     */
  

•    __u32 flags;
  

•    /*
  

•     * This points to an array of num_items drm_i915_query_item structures.
  

•     */
  

•    __u64 items_ptr;
  

+};

+#define DRM_IOCTL_I915_QUERY DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_QUERY, struct drm_i915_query)

+/**

• • enum drm_i915_gem_memory_class
• */

+enum drm_i915_gem_memory_class {

•   /** @I915_MEMORY_CLASS_SYSTEM: system memory */
  

•   I915_MEMORY_CLASS_SYSTEM = 0,
  

•   /** @I915_MEMORY_CLASS_DEVICE: device local-memory */
  

•   I915_MEMORY_CLASS_DEVICE,
  

+};

+/**

• • struct drm_i915_gem_memory_class_instance
• */

+struct drm_i915_gem_memory_class_instance {

•   /** @memory_class: see enum drm_i915_gem_memory_class */
  

•   __u16 memory_class;
  

•   /** @memory_instance: which instance */
  

•   __u16 memory_instance;
  

+};

+/**

• • struct drm_i915_memory_region_info
• • Describes one region as known to the driver.
• • Note that we reserve quite a lot of stuff here for potential future work. As
• • an example we might want expose the capabilities(see caps) for a given
• • region, which could include things like if the region is CPU
• • mappable/accessible etc.

I get caps but I'm seriously at a loss as to what the rest of this would be used for. Why are caps and flags both there and separate? Flags are typically something you set, not query. Also, what's with rsvd1 at the end? This smells of substantial over-building to me.

I thought to myself, "maybe I'm missing a future use-case" so I looked at the internal tree and none of this is being used there either. This indicates to me that either I'm missing something and there's code somewhere I don't know about or, with three years of building on internal branches, we still haven't proven that any of this is needed. If it's the latter, which I strongly suspect, maybe we should drop the unnecessary bits and only add them back in if and when we have proof that they're useful.

To be clear, I don't mind the query API as such and the class/instance stuff seems fine and I really like being able to get the sizes directly. What concerns me is all this extra future-proofing that we have zero proof is actually useful. In my experience, when you build out like this without so much as a use-case, you always end up building the wrong thing.

• */

+struct drm_i915_memory_region_info {

•   /** @region: class:instance pair encoding */
  

•   struct drm_i915_gem_memory_class_instance region;
  

•   /** @rsvd0: MBZ */
  

•   __u32 rsvd0;
  

•   /** @caps: MBZ */
  

•   __u64 caps;
  

•   /** @flags: MBZ */
  

•   __u64 flags;
  

•   /** @probed_size: Memory probed by the driver (-1 = unknown) */
  

•   __u64 probed_size;
  

•   /** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
  

•   __u64 unallocated_size;
  

•   /** @rsvd1: MBZ */
  

•   __u64 rsvd1[8];
  

+};

+/**

• • struct drm_i915_query_memory_regions
• • Region info query enumerates all regions known to the driver by filling in
• • an array of struct drm_i915_memory_region_info structures.
• • Example for getting the list of supported regions:
• • .. code-block:: C

• • struct drm_i915_query_memory_regions *info;
    

• • struct drm_i915_query_item item = {
    

• •         .query_id = DRM_I915_QUERY_MEMORY_REGIONS;
    

• • };
    

• • struct drm_i915_query query = {
    

• •         .num_items = 1,
    

• •         .items_ptr = (uintptr_t)&item,
    

• • };
    

• • int err, i;
    

• • // First query the size of the blob we need, this needs to be large
    

• • // enough to hold our array of regions. The kernel will fill out the
    

• • // item.length for us, which is the number of bytes we need.
    

• • err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
    

• • if (err) ...
    

• • info = calloc(1, item.length);
    

• • // Now that we allocated the required number of bytes, we call the ioctl
    

• • // again, this time with the data_ptr pointing to our newly allocated
    

• • // blob, which the kernel can then populate with the all the region info.
    

• • item.data_ptr = (uintptr_t)&info,
    

• • err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
    

• • if (err) ...
    

• • // We can now access each region in the array
    

• • for (i = 0; i < info->num_regions; i++) {
    

• •         struct drm_i915_memory_region_info mr = info->regions[i];
    

• •         u16 class = mr.region.class;
    

• •         u16 instance = mr.region.instance;
    

• •         ....
    

• • }
    

• • free(info);
    

• */

+struct drm_i915_query_memory_regions {

•   /** @num_regions: Number of supported regions */
  

•   __u32 num_regions;
  

•   /** @rsvd: MBZ */
  

•   __u32 rsvd[3];
  

Why pad to 16B instead of 8B?

•   /** @regions: Info about each supported region */
  

•   struct drm_i915_memory_region_info regions[];
  

+};

+#define DRM_I915_GEM_CREATE_EXT 0xdeadbeaf +#define DRM_IOCTL_I915_GEM_CREATE_EXT DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_CREATE_EXT, struct drm_i915_gem_create_ext)

+/**

• • struct drm_i915_gem_create_ext
• • Existing gem_create behaviour, with added extension support.
• • Note that in the future we want to have our buffer flags here, at least for
• • the stuff that is immutable. Previously we would have two ioctls, one to
• • create the object with gem_create, and another to apply various parameters,
• • however this creates some ambiguity for the params which are considered
• • immutable. Also in general we're phasing out the various SET/GET ioctls.
• */

+struct drm_i915_gem_create_ext {

•   /**
  

•    * @size: Requested size for the object.
  

•    *
  

•    * The (page-aligned) allocated size for the object will be returned.
  

•    *
  

•    * Note that for some devices we have might have further minimum
  

•    * page-size restrictions(larger than 4K), like for device local-memory.
  

•    * However in general the final size here should always reflect any
  

•    * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
  

•    * extension to place the object in device local-memory.
  

•    */
  

•   __u64 size;
  

•   /**
  

•    * @handle: Returned handle for the object.
  

•    *
  

•    * Object handles are nonzero.
  

•    */
  

•   __u32 handle;
  

•   /** @flags: MBZ */
  

•   __u32 flags;
  

•   /**
  

•    * @extensions: The chain of extensions to apply to this object.
  

•    *
  

•    * This will be useful in the future when we need to support several
  

•    * different extensions, and we need to apply more than one when
  

•    * creating the object. See struct i915_user_extension.
  

•    *
  

•    * If we don't supply any extensions then we get the same old gem_create
  

•    * behaviour.
  

•    *
  

•    * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
  

•    * drm_i915_gem_create_ext_memory_regions
  

•    */
  

+#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0

•   __u64 extensions;
  

+};

+/**

• • struct drm_i915_gem_create_ext_memory_regions
• • I915_GEM_CREATE_EXT_MEMORY_REGIONS extension:
• • Set the object with the desired set of placements/regions in priority
• • order(each entry must be unique and supported by the device), as an array of
• • drm_i915_gem_memory_class_instance, or an equivalent layout of class:instance
• • pair encodings. See DRM_I915_QUERY_MEMORY_REGIONS for how to query the
• • supported regions for a device.
• • As an example, on discrete devices, if we wish to set the placement as
• • device local-memory we can do something like:
• • .. code-block:: C

• • struct drm_i915_gem_memory_class_instance region_lmem = {
    

• •          .memory_class = I915_MEMORY_CLASS_DEVICE,
    

• •          .memory_instance = 0,
    

• •  };
    

• •  struct drm_i915_gem_create_ext_memory_regions regions = {
    

• •          .base = { .name = I915_GEM_CREATE_EXT_MEMORY_REGIONS },
    

• •          .regions = (uintptr_t)&region_lmem,
    

• •          .num_regions = 1,
    

• •  };
    

• •  struct drm_i915_gem_create_ext create_ext = {
    

• •          .size = 16 * PAGE_SIZE,
    

• •          .extensions = (uintptr_t)&regions,
    

• •  };
    

• •  int err = ioctl(fd, DRM_IOCTL_I915_GEM_CREATE_EXT, &create_ext);
    

• •  if (err) ...
    

• • At which point we get the object handle in create_ext.handle, if all went
• • well.
• */

+struct drm_i915_gem_create_ext_memory_regions {

•   /** @base: Extension link. See struct i915_user_extension. */
  

•   struct i915_user_extension base;
  

•   /** @pad: MBZ */
  

•   __u32 pad;
  

•   /** @num_regions: Number of elements in the placements array. */
  

•   __u32 num_regions;
  

•   /**
  

•    * @regions: The placements array.
  

•    *
  

•    * Should be an array of drm_i915_gem_memory_class_instance.
  

•    */
  

•   __u64 regions;
  

+}; diff --git a/Documentation/gpu/rfc/i915_gem_lmem.rst b/Documentation/gpu/rfc/i915_gem_lmem.rst new file mode 100644 index 000000000000..52f1db15ae94 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.rst @@ -0,0 +1,139 @@ +========================= +I915 DG1/LMEM RFC Section +=========================

+Upstream plan +============= +For upstream the overall plan for landing all the DG1 stuff and turning it for +real, with all the uAPI bits is:

+* Merge basic HW enabling of DG1(still without pciid) +* Merge the uAPI bits behind special CONFIG_BROKEN(or so) flag

•    * At this point we can still make changes, but importantly this lets us
  

•      start running IGTs which can utilize local-memory in CI
  

+* Convert over to TTM, make sure it all keeps working +* Add pciid for DG1 and turn on uAPI for real

+New object placement and region query uAPI +========================================== +Starting from DG1 we need to give userspace the ability to allocate buffers from +device local-memory. Currently the driver supports gem_create, which can place +buffers in system memory via shmem, and the usual assortment of other +interfaces, like dumb buffers and userptr.

+To support this new capability, while also providing a uAPI which will work +beyond just DG1, we propose to offer three new bits of uAPI:

+Query uAPI +---------- +Existing query interface +^^^^^^^^^^^^^^^^^^^^^^^^ +.. kernel-doc:: include/uapi/drm/i915_drm.h

•    :functions: drm_i915_query_item drm_i915_query
  

+DRM_I915_QUERY_MEMORY_REGIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +New query ID which allows userspace to discover the list of supported memory +regions(like system-memory and local-memory) for a given device. We identify +each region with a class and instance pair, which should be unique. The class +here would be DEVICE or SYSTEM, and the instance would be zero, on platforms +like DG1.

+Side note: The class/instance design is borrowed from our existing engine uAPI, +where we describe every physical engine in terms of its class, and the +particular instance, since we can have more than one per class.

+In the future we also want to expose more information which can further +describe the capabilities of a region.

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_memory_class drm_i915_gem_memory_class_instance drm_i915_memory_region_info drm_i915_query_memory_regions
  

+GEM_CREATE_EXT +-------------- +New ioctl which is basically just gem_create but now allows userspace to +provide a chain of possible extensions. Note that if we don't provide any +extensions then we get the exact same behaviour as gem_create.

+Side note: We also need to support PXP[1] in the near future, which is also +applicable to integrated platforms, and adds its own gem_create_ext extension, +which basically lets userspace mark a buffer as "protected".

A bit off-topic, but do we really need a whole extension for that? Or can we just throw a bit in flags? I'm a big fan of landing create_ext anyway; I like extensibility. I'm just questioning whether or not that one needs its own struct.

--Jason

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_create_ext
  

+It's raining extensions +^^^^^^^^^^^^^^^^^^^^^^^ +As noted above, extensions can be supplied as a chain in gem_create_ext using the +existing i915_user_extension. This will be useful in the future when we need to +support several different extensions, and we need to apply more than one when +creating the object.

+.. kernel-doc:: include/uapi/drm/i915_drm.h

•    :functions: i915_user_extension
  

+I915_GEM_CREATE_EXT_MEMORY_REGIONS +---------------------------------- +Implemented as an extension for gem_create_ext, we would now allow userspace to +optionally provide an immutable list of preferred placements at creation time, +in priority order, for a given buffer object. For the placements we expect +them each to use the class/instance encoding, as per the output of the regions +query. Having the list in priority order will be useful in the future when +placing an object, say during eviction.

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_create_ext_memory_regions
  

+One fair criticism here is that this seems a little over-engineered[2]. If we +just consider DG1 then yes, a simple gem_create.flags or something is totally +all that's needed to tell the kernel to allocate the buffer in local-memory or +whatever. However looking to the future we need uAPI which can also support +upcoming Xe HP multi-tile architecture in a sane way, where there can be +multiple local-memory instances for a given device, and so using both class and +instance in our uAPI to describe regions is desirable, although specifically +for DG1 it's uninteresting, since we only have a single local-memory instance.

+Existing uAPI issues +==================== +Some potential issues we still need to resolve.

+I915 MMAP +--------- +In i915 there are multiple ways to MMAP GEM object, including mapping the same +object using different mapping types(WC vs WB), i.e multiple active mmaps per +object. TTM expects one MMAP at most for the lifetime of the object. If it +turns out that we have to backpedal here, there might be some potential +userspace fallout.

+I915 SET/GET_CACHING +-------------------- +In i915 we have set/get_caching ioctl. TTM doesn't let us to change this, but +DG1 doesn't support non-snooped pcie transactions, so we can just always +allocate as WB for smem-only buffers. If/when our hw gains support for +non-snooped pcie transactions then we must fix this mode at allocation time as +a new GEM extension.

+This is related to the mmap problem, because in general (meaning, when we're +not running on intel cpus) the cpu mmap must not, ever, be inconsistent with +allocation mode.

+Possible idea is to let the kernel picks the mmap mode for userspace from the +following table:

+smem-only: WB. Userspace does not need to call clflush.

+smem+lmem: We allocate uncached memory, and give userspace a WC mapping +for when the buffer is in smem, and WC when it's in lmem. GPU does snooped +access, which is a bit inefficient.

+lmem only: always WC

+This means on discrete you only get a single mmap mode, all others must be +rejected. That's probably going to be a new default mode or something like +that.

+Links +===== +[1] https://patchwork.freedesktop.org/series/86798/

+[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5599#note_553791 diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst index a8621f7dab8b..05670442ca1b 100644 --- a/Documentation/gpu/rfc/index.rst +++ b/Documentation/gpu/rfc/index.rst @@ -15,3 +15,7 @@ host such documentation:

• Once the code has landed move all the documentation to the right places in the main core, helper or driver sections.

+.. toctree::

• i915_gem_lmem.rst

-- 2.26.3

On 4/16/2021 10:02 AM, Daniel Vetter wrote:

On Fri, Apr 16, 2021 at 6:38 PM Jason Ekstrand jason@jlekstrand.net wrote:

...

On Thu, Apr 15, 2021 at 11:04 AM Matthew Auld matthew.auld@intel.com wrote:

...

Add an entry for the new uAPI needed for DG1.

v2(Daniel):

• include the overall upstreaming plan
• add a note for mmap, there are differences here for TTM vs i915
• bunch of other suggestions from Daniel

v3: (Daniel)

• add a note for set/get caching stuff
• add some more docs for existing query and extensions stuff
• add an actual code example for regions query
• bunch of other stuff

(Jason)

• uAPI change(!): - try a simpler design with the placements extension - rather than have a generic setparam which can cover multiple use cases, have each extension be responsible for one thing only

Signed-off-by: Matthew Auld matthew.auld@intel.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Jordan Justen jordan.l.justen@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Kenneth Graunke kenneth@whitecape.org Cc: Jason Ekstrand jason@jlekstrand.net Cc: Dave Airlie airlied@gmail.com Cc: dri-devel@lists.freedesktop.org Cc: mesa-dev@lists.freedesktop.org

---

Documentation/gpu/rfc/i915_gem_lmem.h | 255 ++++++++++++++++++++++++ Documentation/gpu/rfc/i915_gem_lmem.rst | 139 +++++++++++++ Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 398 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.h create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.rst

diff --git a/Documentation/gpu/rfc/i915_gem_lmem.h b/Documentation/gpu/rfc/i915_gem_lmem.h new file mode 100644 index 000000000000..2a82a452e9f2 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.h @@ -0,0 +1,255 @@ +/*

• • Note that drm_i915_query_item and drm_i915_query are existing bits of uAPI.
• • For the regions query we are just adding a new query id, so no actual new
• • ioctl or anything, but including it here for reference.
• */

+struct drm_i915_query_item { +#define DRM_I915_QUERY_MEMORY_REGIONS 0xdeadbeaf

•   ....
  

•    __u64 query_id;
  

•    /*
  

•     * When set to zero by userspace, this is filled with the size of the
  

•     * data to be written at the data_ptr pointer. The kernel sets this
  

•     * value to a negative value to signal an error on a particular query
  

•     * item.
  

•     */
  

•    __s32 length;
  

•    __u32 flags;
  

•    /*
  

•     * Data will be written at the location pointed by data_ptr when the
  

•     * value of length matches the length of the data to be written by the
  

•     * kernel.
  

•     */
  

•    __u64 data_ptr;
  

+};

+struct drm_i915_query {

•    __u32 num_items;
  

•    /*
  

•     * Unused for now. Must be cleared to zero.
  

•     */
  

•    __u32 flags;
  

•    /*
  

•     * This points to an array of num_items drm_i915_query_item structures.
  

•     */
  

•    __u64 items_ptr;
  

+};

+#define DRM_IOCTL_I915_QUERY DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_QUERY, struct drm_i915_query)

+/**

• • enum drm_i915_gem_memory_class
• */

+enum drm_i915_gem_memory_class {

•   /** @I915_MEMORY_CLASS_SYSTEM: system memory */
  

•   I915_MEMORY_CLASS_SYSTEM = 0,
  

•   /** @I915_MEMORY_CLASS_DEVICE: device local-memory */
  

•   I915_MEMORY_CLASS_DEVICE,
  

+};

+/**

• • struct drm_i915_gem_memory_class_instance
• */

+struct drm_i915_gem_memory_class_instance {

•   /** @memory_class: see enum drm_i915_gem_memory_class */
  

•   __u16 memory_class;
  

•   /** @memory_instance: which instance */
  

•   __u16 memory_instance;
  

+};

+/**

• • struct drm_i915_memory_region_info
• • Describes one region as known to the driver.
• • Note that we reserve quite a lot of stuff here for potential future work. As
• • an example we might want expose the capabilities(see caps) for a given
• • region, which could include things like if the region is CPU
• • mappable/accessible etc.

I get caps but I'm seriously at a loss as to what the rest of this would be used for. Why are caps and flags both there and separate? Flags are typically something you set, not query. Also, what's with rsvd1 at the end? This smells of substantial over-building to me.

I thought to myself, "maybe I'm missing a future use-case" so I looked at the internal tree and none of this is being used there either. This indicates to me that either I'm missing something and there's code somewhere I don't know about or, with three years of building on internal branches, we still haven't proven that any of this is needed. If it's the latter, which I strongly suspect, maybe we should drop the unnecessary bits and only add them back in if and when we have proof that they're useful.

To be clear, I don't mind the query API as such and the class/instance stuff seems fine and I really like being able to get the sizes directly. What concerns me is all this extra future-proofing that we have zero proof is actually useful. In my experience, when you build out like this without so much as a use-case, you always end up building the wrong thing.

...

• */

+struct drm_i915_memory_region_info {

•   /** @region: class:instance pair encoding */
  

•   struct drm_i915_gem_memory_class_instance region;
  

•   /** @rsvd0: MBZ */
  

•   __u32 rsvd0;
  

•   /** @caps: MBZ */
  

•   __u64 caps;
  

•   /** @flags: MBZ */
  

•   __u64 flags;
  

•   /** @probed_size: Memory probed by the driver (-1 = unknown) */
  

•   __u64 probed_size;
  

•   /** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
  

•   __u64 unallocated_size;
  

•   /** @rsvd1: MBZ */
  

•   __u64 rsvd1[8];
  

+};

+/**

• • struct drm_i915_query_memory_regions
• • Region info query enumerates all regions known to the driver by filling in
• • an array of struct drm_i915_memory_region_info structures.
• • Example for getting the list of supported regions:
• • .. code-block:: C

• • struct drm_i915_query_memory_regions *info;
    

• • struct drm_i915_query_item item = {
    

• •         .query_id = DRM_I915_QUERY_MEMORY_REGIONS;
    

• • };
    

• • struct drm_i915_query query = {
    

• •         .num_items = 1,
    

• •         .items_ptr = (uintptr_t)&item,
    

• • };
    

• • int err, i;
    

• • // First query the size of the blob we need, this needs to be large
    

• • // enough to hold our array of regions. The kernel will fill out the
    

• • // item.length for us, which is the number of bytes we need.
    

• • err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
    

• • if (err) ...
    

• • info = calloc(1, item.length);
    

• • // Now that we allocated the required number of bytes, we call the ioctl
    

• • // again, this time with the data_ptr pointing to our newly allocated
    

• • // blob, which the kernel can then populate with the all the region info.
    

• • item.data_ptr = (uintptr_t)&info,
    

• • err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
    

• • if (err) ...
    

• • // We can now access each region in the array
    

• • for (i = 0; i < info->num_regions; i++) {
    

• •         struct drm_i915_memory_region_info mr = info->regions[i];
    

• •         u16 class = mr.region.class;
    

• •         u16 instance = mr.region.instance;
    

• •         ....
    

• • }
    

• • free(info);
    

• */

+struct drm_i915_query_memory_regions {

•   /** @num_regions: Number of supported regions */
  

•   __u32 num_regions;
  

•   /** @rsvd: MBZ */
  

•   __u32 rsvd[3];
  

Why pad to 16B instead of 8B?

...

•   /** @regions: Info about each supported region */
  

•   struct drm_i915_memory_region_info regions[];
  

+};

+#define DRM_I915_GEM_CREATE_EXT 0xdeadbeaf +#define DRM_IOCTL_I915_GEM_CREATE_EXT DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_CREATE_EXT, struct drm_i915_gem_create_ext)

+/**

• • struct drm_i915_gem_create_ext
• • Existing gem_create behaviour, with added extension support.
• • Note that in the future we want to have our buffer flags here, at least for
• • the stuff that is immutable. Previously we would have two ioctls, one to
• • create the object with gem_create, and another to apply various parameters,
• • however this creates some ambiguity for the params which are considered
• • immutable. Also in general we're phasing out the various SET/GET ioctls.
• */

+struct drm_i915_gem_create_ext {

•   /**
  

•    * @size: Requested size for the object.
  

•    *
  

•    * The (page-aligned) allocated size for the object will be returned.
  

•    *
  

•    * Note that for some devices we have might have further minimum
  

•    * page-size restrictions(larger than 4K), like for device local-memory.
  

•    * However in general the final size here should always reflect any
  

•    * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
  

•    * extension to place the object in device local-memory.
  

•    */
  

•   __u64 size;
  

•   /**
  

•    * @handle: Returned handle for the object.
  

•    *
  

•    * Object handles are nonzero.
  

•    */
  

•   __u32 handle;
  

•   /** @flags: MBZ */
  

•   __u32 flags;
  

•   /**
  

•    * @extensions: The chain of extensions to apply to this object.
  

•    *
  

•    * This will be useful in the future when we need to support several
  

•    * different extensions, and we need to apply more than one when
  

•    * creating the object. See struct i915_user_extension.
  

•    *
  

•    * If we don't supply any extensions then we get the same old gem_create
  

•    * behaviour.
  

•    *
  

•    * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
  

•    * drm_i915_gem_create_ext_memory_regions
  

•    */
  

+#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0

•   __u64 extensions;
  

+};

+/**

• • struct drm_i915_gem_create_ext_memory_regions
• • I915_GEM_CREATE_EXT_MEMORY_REGIONS extension:
• • Set the object with the desired set of placements/regions in priority
• • order(each entry must be unique and supported by the device), as an array of
• • drm_i915_gem_memory_class_instance, or an equivalent layout of class:instance
• • pair encodings. See DRM_I915_QUERY_MEMORY_REGIONS for how to query the
• • supported regions for a device.
• • As an example, on discrete devices, if we wish to set the placement as
• • device local-memory we can do something like:
• • .. code-block:: C

• • struct drm_i915_gem_memory_class_instance region_lmem = {
    

• •          .memory_class = I915_MEMORY_CLASS_DEVICE,
    

• •          .memory_instance = 0,
    

• •  };
    

• •  struct drm_i915_gem_create_ext_memory_regions regions = {
    

• •          .base = { .name = I915_GEM_CREATE_EXT_MEMORY_REGIONS },
    

• •          .regions = (uintptr_t)&region_lmem,
    

• •          .num_regions = 1,
    

• •  };
    

• •  struct drm_i915_gem_create_ext create_ext = {
    

• •          .size = 16 * PAGE_SIZE,
    

• •          .extensions = (uintptr_t)&regions,
    

• •  };
    

• •  int err = ioctl(fd, DRM_IOCTL_I915_GEM_CREATE_EXT, &create_ext);
    

• •  if (err) ...
    

• • At which point we get the object handle in create_ext.handle, if all went
• • well.
• */

+struct drm_i915_gem_create_ext_memory_regions {

•   /** @base: Extension link. See struct i915_user_extension. */
  

•   struct i915_user_extension base;
  

•   /** @pad: MBZ */
  

•   __u32 pad;
  

•   /** @num_regions: Number of elements in the placements array. */
  

•   __u32 num_regions;
  

•   /**
  

•    * @regions: The placements array.
  

•    *
  

•    * Should be an array of drm_i915_gem_memory_class_instance.
  

•    */
  

•   __u64 regions;
  

+}; diff --git a/Documentation/gpu/rfc/i915_gem_lmem.rst b/Documentation/gpu/rfc/i915_gem_lmem.rst new file mode 100644 index 000000000000..52f1db15ae94 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.rst @@ -0,0 +1,139 @@ +========================= +I915 DG1/LMEM RFC Section +=========================

+Upstream plan +============= +For upstream the overall plan for landing all the DG1 stuff and turning it for +real, with all the uAPI bits is:

+* Merge basic HW enabling of DG1(still without pciid) +* Merge the uAPI bits behind special CONFIG_BROKEN(or so) flag

•    * At this point we can still make changes, but importantly this lets us
  

•      start running IGTs which can utilize local-memory in CI
  

+* Convert over to TTM, make sure it all keeps working +* Add pciid for DG1 and turn on uAPI for real

+New object placement and region query uAPI +========================================== +Starting from DG1 we need to give userspace the ability to allocate buffers from +device local-memory. Currently the driver supports gem_create, which can place +buffers in system memory via shmem, and the usual assortment of other +interfaces, like dumb buffers and userptr.

+To support this new capability, while also providing a uAPI which will work +beyond just DG1, we propose to offer three new bits of uAPI:

+Query uAPI +---------- +Existing query interface +^^^^^^^^^^^^^^^^^^^^^^^^ +.. kernel-doc:: include/uapi/drm/i915_drm.h

•    :functions: drm_i915_query_item drm_i915_query
  

+DRM_I915_QUERY_MEMORY_REGIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +New query ID which allows userspace to discover the list of supported memory +regions(like system-memory and local-memory) for a given device. We identify +each region with a class and instance pair, which should be unique. The class +here would be DEVICE or SYSTEM, and the instance would be zero, on platforms +like DG1.

+Side note: The class/instance design is borrowed from our existing engine uAPI, +where we describe every physical engine in terms of its class, and the +particular instance, since we can have more than one per class.

+In the future we also want to expose more information which can further +describe the capabilities of a region.

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_memory_class drm_i915_gem_memory_class_instance drm_i915_memory_region_info drm_i915_query_memory_regions
  

+GEM_CREATE_EXT +-------------- +New ioctl which is basically just gem_create but now allows userspace to +provide a chain of possible extensions. Note that if we don't provide any +extensions then we get the exact same behaviour as gem_create.

+Side note: We also need to support PXP[1] in the near future, which is also +applicable to integrated platforms, and adds its own gem_create_ext extension, +which basically lets userspace mark a buffer as "protected".

A bit off-topic, but do we really need a whole extension for that? Or can we just throw a bit in flags? I'm a big fan of landing create_ext anyway; I like extensibility. I'm just questioning whether or not that one needs its own struct.

Yeah for PXP just a flag is all we need. If we cant/don't want to stuff that into the main structure then we could do an empty extension: It's presence would indicate that we want PXP enabled.

There is actually a potential future use for the PXP extension as there are different PXP object use types; we currently plan on supporting only one, but I'd prefer to keep the extension option open for potential future expansion.

Definitely we shouldn't build the entire setparam machinery for gem bo copied over from ctx_create_ext while Jason is working on around 100 or so igt+kernel patches to undo the unecessary uapi complexity this caused over there in ctx land.

Also adding Daniele and Lionel to keep them in the loop here on what's happening to gem_create_ext. Daniele I think for merging we need a) resubmit the pxp patches to dri-devel per new process that we have since 2-3 weeks now and then b) land the uapi bits only after this patch here from Matt has landed, to make sure we don't end up with the wrong version of gem_create_ext. Is that ok?

Sure. I might send a new rev for review before this lands, but I'll note the dependency in the cover letter and rebase and re-send afterwards.

Daniele

-Daniel

...

--Jason

...

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_create_ext
  

+It's raining extensions +^^^^^^^^^^^^^^^^^^^^^^^ +As noted above, extensions can be supplied as a chain in gem_create_ext using the +existing i915_user_extension. This will be useful in the future when we need to +support several different extensions, and we need to apply more than one when +creating the object.

+.. kernel-doc:: include/uapi/drm/i915_drm.h

•    :functions: i915_user_extension
  

+I915_GEM_CREATE_EXT_MEMORY_REGIONS +---------------------------------- +Implemented as an extension for gem_create_ext, we would now allow userspace to +optionally provide an immutable list of preferred placements at creation time, +in priority order, for a given buffer object. For the placements we expect +them each to use the class/instance encoding, as per the output of the regions +query. Having the list in priority order will be useful in the future when +placing an object, say during eviction.

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_create_ext_memory_regions
  

+One fair criticism here is that this seems a little over-engineered[2]. If we +just consider DG1 then yes, a simple gem_create.flags or something is totally +all that's needed to tell the kernel to allocate the buffer in local-memory or +whatever. However looking to the future we need uAPI which can also support +upcoming Xe HP multi-tile architecture in a sane way, where there can be +multiple local-memory instances for a given device, and so using both class and +instance in our uAPI to describe regions is desirable, although specifically +for DG1 it's uninteresting, since we only have a single local-memory instance.

+Existing uAPI issues +==================== +Some potential issues we still need to resolve.

+I915 MMAP +--------- +In i915 there are multiple ways to MMAP GEM object, including mapping the same +object using different mapping types(WC vs WB), i.e multiple active mmaps per +object. TTM expects one MMAP at most for the lifetime of the object. If it +turns out that we have to backpedal here, there might be some potential +userspace fallout.

+I915 SET/GET_CACHING +-------------------- +In i915 we have set/get_caching ioctl. TTM doesn't let us to change this, but +DG1 doesn't support non-snooped pcie transactions, so we can just always +allocate as WB for smem-only buffers. If/when our hw gains support for +non-snooped pcie transactions then we must fix this mode at allocation time as +a new GEM extension.

+This is related to the mmap problem, because in general (meaning, when we're +not running on intel cpus) the cpu mmap must not, ever, be inconsistent with +allocation mode.

+Possible idea is to let the kernel picks the mmap mode for userspace from the +following table:

+smem-only: WB. Userspace does not need to call clflush.

+smem+lmem: We allocate uncached memory, and give userspace a WC mapping +for when the buffer is in smem, and WC when it's in lmem. GPU does snooped +access, which is a bit inefficient.

+lmem only: always WC

+This means on discrete you only get a single mmap mode, all others must be +rejected. That's probably going to be a new default mode or something like +that.

+Links +===== +[1] https://patchwork.freedesktop.org/series/86798/

+[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5599#note_553791 diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst index a8621f7dab8b..05670442ca1b 100644 --- a/Documentation/gpu/rfc/index.rst +++ b/Documentation/gpu/rfc/index.rst @@ -15,3 +15,7 @@ host such documentation:

• Once the code has landed move all the documentation to the right places in the main core, helper or driver sections.

+.. toctree::

• i915_gem_lmem.rst

-- 2.26.3

---

dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel

On Mon, Apr 19, 2021 at 7:02 AM Matthew Auld matthew.auld@intel.com wrote:

On 16/04/2021 17:38, Jason Ekstrand wrote:

...

On Thu, Apr 15, 2021 at 11:04 AM Matthew Auld matthew.auld@intel.com wrote:

...

Add an entry for the new uAPI needed for DG1.

v2(Daniel):

• include the overall upstreaming plan
• add a note for mmap, there are differences here for TTM vs i915
• bunch of other suggestions from Daniel

v3: (Daniel)

• add a note for set/get caching stuff
• add some more docs for existing query and extensions stuff
• add an actual code example for regions query
• bunch of other stuff

(Jason)

• uAPI change(!): - try a simpler design with the placements extension - rather than have a generic setparam which can cover multiple use cases, have each extension be responsible for one thing only

Signed-off-by: Matthew Auld matthew.auld@intel.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Jordan Justen jordan.l.justen@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Kenneth Graunke kenneth@whitecape.org Cc: Jason Ekstrand jason@jlekstrand.net Cc: Dave Airlie airlied@gmail.com Cc: dri-devel@lists.freedesktop.org Cc: mesa-dev@lists.freedesktop.org

---

Documentation/gpu/rfc/i915_gem_lmem.h | 255 ++++++++++++++++++++++++ Documentation/gpu/rfc/i915_gem_lmem.rst | 139 +++++++++++++ Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 398 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.h create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.rst

diff --git a/Documentation/gpu/rfc/i915_gem_lmem.h b/Documentation/gpu/rfc/i915_gem_lmem.h new file mode 100644 index 000000000000..2a82a452e9f2 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.h @@ -0,0 +1,255 @@ +/*

• • Note that drm_i915_query_item and drm_i915_query are existing bits of uAPI.
• • For the regions query we are just adding a new query id, so no actual new
• • ioctl or anything, but including it here for reference.
• */

+struct drm_i915_query_item { +#define DRM_I915_QUERY_MEMORY_REGIONS 0xdeadbeaf

•   ....
  

•    __u64 query_id;
  

•    /*
  

•     * When set to zero by userspace, this is filled with the size of the
  

•     * data to be written at the data_ptr pointer. The kernel sets this
  

•     * value to a negative value to signal an error on a particular query
  

•     * item.
  

•     */
  

•    __s32 length;
  

•    __u32 flags;
  

•    /*
  

•     * Data will be written at the location pointed by data_ptr when the
  

•     * value of length matches the length of the data to be written by the
  

•     * kernel.
  

•     */
  

•    __u64 data_ptr;
  

+};

+struct drm_i915_query {

•    __u32 num_items;
  

•    /*
  

•     * Unused for now. Must be cleared to zero.
  

•     */
  

•    __u32 flags;
  

•    /*
  

•     * This points to an array of num_items drm_i915_query_item structures.
  

•     */
  

•    __u64 items_ptr;
  

+};

+#define DRM_IOCTL_I915_QUERY DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_QUERY, struct drm_i915_query)

+/**

• • enum drm_i915_gem_memory_class
• */

+enum drm_i915_gem_memory_class {

•   /** @I915_MEMORY_CLASS_SYSTEM: system memory */
  

•   I915_MEMORY_CLASS_SYSTEM = 0,
  

•   /** @I915_MEMORY_CLASS_DEVICE: device local-memory */
  

•   I915_MEMORY_CLASS_DEVICE,
  

+};

+/**

• • struct drm_i915_gem_memory_class_instance
• */

+struct drm_i915_gem_memory_class_instance {

•   /** @memory_class: see enum drm_i915_gem_memory_class */
  

•   __u16 memory_class;
  

•   /** @memory_instance: which instance */
  

•   __u16 memory_instance;
  

+};

+/**

• • struct drm_i915_memory_region_info
• • Describes one region as known to the driver.
• • Note that we reserve quite a lot of stuff here for potential future work. As
• • an example we might want expose the capabilities(see caps) for a given
• • region, which could include things like if the region is CPU
• • mappable/accessible etc.

I get caps but I'm seriously at a loss as to what the rest of this would be used for. Why are caps and flags both there and separate? Flags are typically something you set, not query. Also, what's with rsvd1 at the end? This smells of substantial over-building to me.

I thought to myself, "maybe I'm missing a future use-case" so I looked at the internal tree and none of this is being used there either. This indicates to me that either I'm missing something and there's code somewhere I don't know about or, with three years of building on internal branches, we still haven't proven that any of this is needed. If it's the latter, which I strongly suspect, maybe we should drop the unnecessary bits and only add them back in if and when we have proof that they're useful.

Do you mean just drop caps/flags here, but keep/inflate rsvd0/rsvd1, which is less opinionated about future unknowns? If so, makes sense to me.

I meant drop flags and rsvd1. We need rsvd0 for padding and I can see some value to caps. We may want to advertise, for instance, what mapping coherency types are available per-heap. But I don't see any use for any of the other fields.

...

To be clear, I don't mind the query API as such and the class/instance stuff seems fine and I really like being able to get the sizes directly. What concerns me is all this extra future-proofing that we have zero proof is actually useful. In my experience, when you build out like this without so much as a use-case, you always end up building the wrong thing.

...

• */

+struct drm_i915_memory_region_info {

•   /** @region: class:instance pair encoding */
  

•   struct drm_i915_gem_memory_class_instance region;
  

•   /** @rsvd0: MBZ */
  

•   __u32 rsvd0;
  

•   /** @caps: MBZ */
  

•   __u64 caps;
  

•   /** @flags: MBZ */
  

•   __u64 flags;
  

•   /** @probed_size: Memory probed by the driver (-1 = unknown) */
  

•   __u64 probed_size;
  

•   /** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
  

•   __u64 unallocated_size;
  

•   /** @rsvd1: MBZ */
  

•   __u64 rsvd1[8];
  

+};

+/**

• • struct drm_i915_query_memory_regions
• • Region info query enumerates all regions known to the driver by filling in
• • an array of struct drm_i915_memory_region_info structures.
• • Example for getting the list of supported regions:
• • .. code-block:: C

• • struct drm_i915_query_memory_regions *info;
    

• • struct drm_i915_query_item item = {
    

• •         .query_id = DRM_I915_QUERY_MEMORY_REGIONS;
    

• • };
    

• • struct drm_i915_query query = {
    

• •         .num_items = 1,
    

• •         .items_ptr = (uintptr_t)&item,
    

• • };
    

• • int err, i;
    

• • // First query the size of the blob we need, this needs to be large
    

• • // enough to hold our array of regions. The kernel will fill out the
    

• • // item.length for us, which is the number of bytes we need.
    

• • err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
    

• • if (err) ...
    

• • info = calloc(1, item.length);
    

• • // Now that we allocated the required number of bytes, we call the ioctl
    

• • // again, this time with the data_ptr pointing to our newly allocated
    

• • // blob, which the kernel can then populate with the all the region info.
    

• • item.data_ptr = (uintptr_t)&info,
    

• • err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
    

• • if (err) ...
    

• • // We can now access each region in the array
    

• • for (i = 0; i < info->num_regions; i++) {
    

• •         struct drm_i915_memory_region_info mr = info->regions[i];
    

• •         u16 class = mr.region.class;
    

• •         u16 instance = mr.region.instance;
    

• •         ....
    

• • }
    

• • free(info);
    

• */

+struct drm_i915_query_memory_regions {

•   /** @num_regions: Number of supported regions */
  

•   __u32 num_regions;
  

•   /** @rsvd: MBZ */
  

•   __u32 rsvd[3];
  

Why pad to 16B instead of 8B?

It's copy-pasta from engine_info. I can shrink it if you want? I don't have a strong opinion.

Yeah, I'd shrink to ust a __u32. We could probably drop it entirely but aligning to 8B seems like a good idea before an array.

--Jason

...

...

•   /** @regions: Info about each supported region */
  

•   struct drm_i915_memory_region_info regions[];
  

+};

+#define DRM_I915_GEM_CREATE_EXT 0xdeadbeaf +#define DRM_IOCTL_I915_GEM_CREATE_EXT DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_CREATE_EXT, struct drm_i915_gem_create_ext)

+/**

• • struct drm_i915_gem_create_ext
• • Existing gem_create behaviour, with added extension support.
• • Note that in the future we want to have our buffer flags here, at least for
• • the stuff that is immutable. Previously we would have two ioctls, one to
• • create the object with gem_create, and another to apply various parameters,
• • however this creates some ambiguity for the params which are considered
• • immutable. Also in general we're phasing out the various SET/GET ioctls.
• */

+struct drm_i915_gem_create_ext {

•   /**
  

•    * @size: Requested size for the object.
  

•    *
  

•    * The (page-aligned) allocated size for the object will be returned.
  

•    *
  

•    * Note that for some devices we have might have further minimum
  

•    * page-size restrictions(larger than 4K), like for device local-memory.
  

•    * However in general the final size here should always reflect any
  

•    * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
  

•    * extension to place the object in device local-memory.
  

•    */
  

•   __u64 size;
  

•   /**
  

•    * @handle: Returned handle for the object.
  

•    *
  

•    * Object handles are nonzero.
  

•    */
  

•   __u32 handle;
  

•   /** @flags: MBZ */
  

•   __u32 flags;
  

•   /**
  

•    * @extensions: The chain of extensions to apply to this object.
  

•    *
  

•    * This will be useful in the future when we need to support several
  

•    * different extensions, and we need to apply more than one when
  

•    * creating the object. See struct i915_user_extension.
  

•    *
  

•    * If we don't supply any extensions then we get the same old gem_create
  

•    * behaviour.
  

•    *
  

•    * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
  

•    * drm_i915_gem_create_ext_memory_regions
  

•    */
  

+#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0

•   __u64 extensions;
  

+};

+/**

• • struct drm_i915_gem_create_ext_memory_regions
• • I915_GEM_CREATE_EXT_MEMORY_REGIONS extension:
• • Set the object with the desired set of placements/regions in priority
• • order(each entry must be unique and supported by the device), as an array of
• • drm_i915_gem_memory_class_instance, or an equivalent layout of class:instance
• • pair encodings. See DRM_I915_QUERY_MEMORY_REGIONS for how to query the
• • supported regions for a device.
• • As an example, on discrete devices, if we wish to set the placement as
• • device local-memory we can do something like:
• • .. code-block:: C

• • struct drm_i915_gem_memory_class_instance region_lmem = {
    

• •          .memory_class = I915_MEMORY_CLASS_DEVICE,
    

• •          .memory_instance = 0,
    

• •  };
    

• •  struct drm_i915_gem_create_ext_memory_regions regions = {
    

• •          .base = { .name = I915_GEM_CREATE_EXT_MEMORY_REGIONS },
    

• •          .regions = (uintptr_t)&region_lmem,
    

• •          .num_regions = 1,
    

• •  };
    

• •  struct drm_i915_gem_create_ext create_ext = {
    

• •          .size = 16 * PAGE_SIZE,
    

• •          .extensions = (uintptr_t)&regions,
    

• •  };
    

• •  int err = ioctl(fd, DRM_IOCTL_I915_GEM_CREATE_EXT, &create_ext);
    

• •  if (err) ...
    

• • At which point we get the object handle in create_ext.handle, if all went
• • well.
• */

+struct drm_i915_gem_create_ext_memory_regions {

•   /** @base: Extension link. See struct i915_user_extension. */
  

•   struct i915_user_extension base;
  

•   /** @pad: MBZ */
  

•   __u32 pad;
  

•   /** @num_regions: Number of elements in the placements array. */
  

•   __u32 num_regions;
  

•   /**
  

•    * @regions: The placements array.
  

•    *
  

•    * Should be an array of drm_i915_gem_memory_class_instance.
  

•    */
  

•   __u64 regions;
  

+}; diff --git a/Documentation/gpu/rfc/i915_gem_lmem.rst b/Documentation/gpu/rfc/i915_gem_lmem.rst new file mode 100644 index 000000000000..52f1db15ae94 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.rst @@ -0,0 +1,139 @@ +========================= +I915 DG1/LMEM RFC Section +=========================

+Upstream plan +============= +For upstream the overall plan for landing all the DG1 stuff and turning it for +real, with all the uAPI bits is:

+* Merge basic HW enabling of DG1(still without pciid) +* Merge the uAPI bits behind special CONFIG_BROKEN(or so) flag

•    * At this point we can still make changes, but importantly this lets us
  

•      start running IGTs which can utilize local-memory in CI
  

+* Convert over to TTM, make sure it all keeps working +* Add pciid for DG1 and turn on uAPI for real

+New object placement and region query uAPI +========================================== +Starting from DG1 we need to give userspace the ability to allocate buffers from +device local-memory. Currently the driver supports gem_create, which can place +buffers in system memory via shmem, and the usual assortment of other +interfaces, like dumb buffers and userptr.

+To support this new capability, while also providing a uAPI which will work +beyond just DG1, we propose to offer three new bits of uAPI:

+Query uAPI +---------- +Existing query interface +^^^^^^^^^^^^^^^^^^^^^^^^ +.. kernel-doc:: include/uapi/drm/i915_drm.h

•    :functions: drm_i915_query_item drm_i915_query
  

+DRM_I915_QUERY_MEMORY_REGIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +New query ID which allows userspace to discover the list of supported memory +regions(like system-memory and local-memory) for a given device. We identify +each region with a class and instance pair, which should be unique. The class +here would be DEVICE or SYSTEM, and the instance would be zero, on platforms +like DG1.

+Side note: The class/instance design is borrowed from our existing engine uAPI, +where we describe every physical engine in terms of its class, and the +particular instance, since we can have more than one per class.

+In the future we also want to expose more information which can further +describe the capabilities of a region.

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_memory_class drm_i915_gem_memory_class_instance drm_i915_memory_region_info drm_i915_query_memory_regions
  

+GEM_CREATE_EXT +-------------- +New ioctl which is basically just gem_create but now allows userspace to +provide a chain of possible extensions. Note that if we don't provide any +extensions then we get the exact same behaviour as gem_create.

+Side note: We also need to support PXP[1] in the near future, which is also +applicable to integrated platforms, and adds its own gem_create_ext extension, +which basically lets userspace mark a buffer as "protected".

A bit off-topic, but do we really need a whole extension for that? Or can we just throw a bit in flags? I'm a big fan of landing create_ext anyway; I like extensibility. I'm just questioning whether or not that one needs its own struct.

--Jason

...

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_create_ext
  

+It's raining extensions +^^^^^^^^^^^^^^^^^^^^^^^ +As noted above, extensions can be supplied as a chain in gem_create_ext using the +existing i915_user_extension. This will be useful in the future when we need to +support several different extensions, and we need to apply more than one when +creating the object.

+.. kernel-doc:: include/uapi/drm/i915_drm.h

•    :functions: i915_user_extension
  

+I915_GEM_CREATE_EXT_MEMORY_REGIONS +---------------------------------- +Implemented as an extension for gem_create_ext, we would now allow userspace to +optionally provide an immutable list of preferred placements at creation time, +in priority order, for a given buffer object. For the placements we expect +them each to use the class/instance encoding, as per the output of the regions +query. Having the list in priority order will be useful in the future when +placing an object, say during eviction.

+.. kernel-doc:: Documentation/gpu/rfc/i915_gem_lmem.h

•    :functions: drm_i915_gem_create_ext_memory_regions
  

+One fair criticism here is that this seems a little over-engineered[2]. If we +just consider DG1 then yes, a simple gem_create.flags or something is totally +all that's needed to tell the kernel to allocate the buffer in local-memory or +whatever. However looking to the future we need uAPI which can also support +upcoming Xe HP multi-tile architecture in a sane way, where there can be +multiple local-memory instances for a given device, and so using both class and +instance in our uAPI to describe regions is desirable, although specifically +for DG1 it's uninteresting, since we only have a single local-memory instance.

+Existing uAPI issues +==================== +Some potential issues we still need to resolve.

+I915 MMAP +--------- +In i915 there are multiple ways to MMAP GEM object, including mapping the same +object using different mapping types(WC vs WB), i.e multiple active mmaps per +object. TTM expects one MMAP at most for the lifetime of the object. If it +turns out that we have to backpedal here, there might be some potential +userspace fallout.

+I915 SET/GET_CACHING +-------------------- +In i915 we have set/get_caching ioctl. TTM doesn't let us to change this, but +DG1 doesn't support non-snooped pcie transactions, so we can just always +allocate as WB for smem-only buffers. If/when our hw gains support for +non-snooped pcie transactions then we must fix this mode at allocation time as +a new GEM extension.

+This is related to the mmap problem, because in general (meaning, when we're +not running on intel cpus) the cpu mmap must not, ever, be inconsistent with +allocation mode.

+Possible idea is to let the kernel picks the mmap mode for userspace from the +following table:

+smem-only: WB. Userspace does not need to call clflush.

+smem+lmem: We allocate uncached memory, and give userspace a WC mapping +for when the buffer is in smem, and WC when it's in lmem. GPU does snooped +access, which is a bit inefficient.

+lmem only: always WC

+This means on discrete you only get a single mmap mode, all others must be +rejected. That's probably going to be a new default mode or something like +that.

+Links +===== +[1] https://patchwork.freedesktop.org/series/86798/

+[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5599#note_553791 diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst index a8621f7dab8b..05670442ca1b 100644 --- a/Documentation/gpu/rfc/index.rst +++ b/Documentation/gpu/rfc/index.rst @@ -15,3 +15,7 @@ host such documentation:

• Once the code has landed move all the documentation to the right places in the main core, helper or driver sections.

+.. toctree::

• i915_gem_lmem.rst

-- 2.26.3

On Tue, Apr 20, 2021 at 11:34 AM Tvrtko Ursulin tvrtko.ursulin@linux.intel.com wrote:

On 19/04/2021 16:19, Jason Ekstrand wrote:

...

On Mon, Apr 19, 2021 at 7:02 AM Matthew Auld matthew.auld@intel.com wrote:

...

On 16/04/2021 17:38, Jason Ekstrand wrote:

...

On Thu, Apr 15, 2021 at 11:04 AM Matthew Auld matthew.auld@intel.com wrote:

...

Add an entry for the new uAPI needed for DG1.

v2(Daniel): - include the overall upstreaming plan - add a note for mmap, there are differences here for TTM vs i915 - bunch of other suggestions from Daniel v3: (Daniel) - add a note for set/get caching stuff - add some more docs for existing query and extensions stuff - add an actual code example for regions query - bunch of other stuff (Jason) - uAPI change(!): - try a simpler design with the placements extension - rather than have a generic setparam which can cover multiple use cases, have each extension be responsible for one thing only

Signed-off-by: Matthew Auld matthew.auld@intel.com Cc: Joonas Lahtinen joonas.lahtinen@linux.intel.com Cc: Jordan Justen jordan.l.justen@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Kenneth Graunke kenneth@whitecape.org Cc: Jason Ekstrand jason@jlekstrand.net Cc: Dave Airlie airlied@gmail.com Cc: dri-devel@lists.freedesktop.org Cc: mesa-dev@lists.freedesktop.org

---

Documentation/gpu/rfc/i915_gem_lmem.h | 255 ++++++++++++++++++++++++ Documentation/gpu/rfc/i915_gem_lmem.rst | 139 +++++++++++++ Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 398 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.h create mode 100644 Documentation/gpu/rfc/i915_gem_lmem.rst

diff --git a/Documentation/gpu/rfc/i915_gem_lmem.h b/Documentation/gpu/rfc/i915_gem_lmem.h new file mode 100644 index 000000000000..2a82a452e9f2 --- /dev/null +++ b/Documentation/gpu/rfc/i915_gem_lmem.h @@ -0,0 +1,255 @@ +/*

• • Note that drm_i915_query_item and drm_i915_query are existing bits of uAPI.
• • For the regions query we are just adding a new query id, so no actual new
• • ioctl or anything, but including it here for reference.
• */

+struct drm_i915_query_item { +#define DRM_I915_QUERY_MEMORY_REGIONS 0xdeadbeaf

•   ....
  

•    __u64 query_id;
  

•    /*
  

•     * When set to zero by userspace, this is filled with the size of the
  

•     * data to be written at the data_ptr pointer. The kernel sets this
  

•     * value to a negative value to signal an error on a particular query
  

•     * item.
  

•     */
  

•    __s32 length;
  

•    __u32 flags;
  

•    /*
  

•     * Data will be written at the location pointed by data_ptr when the
  

•     * value of length matches the length of the data to be written by the
  

•     * kernel.
  

•     */
  

•    __u64 data_ptr;
  

+};

+struct drm_i915_query {

•    __u32 num_items;
  

•    /*
  

•     * Unused for now. Must be cleared to zero.
  

•     */
  

•    __u32 flags;
  

•    /*
  

•     * This points to an array of num_items drm_i915_query_item structures.
  

•     */
  

•    __u64 items_ptr;
  

+};

+#define DRM_IOCTL_I915_QUERY DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_QUERY, struct drm_i915_query)

+/**

• • enum drm_i915_gem_memory_class
• */

+enum drm_i915_gem_memory_class {

•   /** @I915_MEMORY_CLASS_SYSTEM: system memory */
  

•   I915_MEMORY_CLASS_SYSTEM = 0,
  

•   /** @I915_MEMORY_CLASS_DEVICE: device local-memory */
  

•   I915_MEMORY_CLASS_DEVICE,
  

+};

+/**

• • struct drm_i915_gem_memory_class_instance
• */

+struct drm_i915_gem_memory_class_instance {

•   /** @memory_class: see enum drm_i915_gem_memory_class */
  

•   __u16 memory_class;
  

•   /** @memory_instance: which instance */
  

•   __u16 memory_instance;
  

+};

+/**

• • struct drm_i915_memory_region_info
• • Describes one region as known to the driver.
• • Note that we reserve quite a lot of stuff here for potential future work. As
• • an example we might want expose the capabilities(see caps) for a given
• • region, which could include things like if the region is CPU
• • mappable/accessible etc.

I get caps but I'm seriously at a loss as to what the rest of this would be used for. Why are caps and flags both there and separate? Flags are typically something you set, not query. Also, what's with rsvd1 at the end? This smells of substantial over-building to me.

I thought to myself, "maybe I'm missing a future use-case" so I looked at the internal tree and none of this is being used there either. This indicates to me that either I'm missing something and there's code somewhere I don't know about or, with three years of building on internal branches, we still haven't proven that any of this is needed. If it's the latter, which I strongly suspect, maybe we should drop the unnecessary bits and only add them back in if and when we have proof that they're useful.

Do you mean just drop caps/flags here, but keep/inflate rsvd0/rsvd1, which is less opinionated about future unknowns? If so, makes sense to me.

I meant drop flags and rsvd1. We need rsvd0 for padding and I can see some value to caps. We may want to advertise, for instance, what mapping coherency types are available per-heap. But I don't see any use for any of the other fields.

I'd suggest making sure at least enough rsvd fields remain so that flags could be added later if needed. Experience from engine info shows that both were required in order to extend the query via re-purposing the rsvds and adding flag bits to indicate when a certain rsvd contains a new piece of information.

Looking at DII, all I see is we started using caps. I already said I'm fine with caps. I can already imagine some useful ones like specifying what kinds of mappings we can do.

If we're concerned about more complicated stuff, I argue that we have no ability to predict what that will look like and so just throwing in a bunch of __u32 rsvd[N] is blind guessing. I'm seeing a lot of that in the recently added APIs such as the flags and rsvd[4] in i915_user_extension. What's that there for? Why can't you put that information in the extension struct which derives from it? Maybe it's so that we can extend it. But we added that struct as part of an extension mechanism!?!

If we want to make things extensible, Vulkan actually provides some prior art for this in the form of allowing queries to be extended just like input structs. We could add a __u64 extensions field to memory_region_info and, if we ever need to query more info, the client can provide a chain of additional per-region queries. Yeah, there are problems with it and it gets a bit clunky but it does work pretty well.

I probably cannot go into too much detail here, but anyway the point is just to make sure too much is not stripped out so that instead of simply adding fields/flags we have to add a new query in the future. IMO some rsvd fields are not really harmful and if they can make things easier in the future why not.

Maybe it's my tired and generally grumpy state of mind but I'm not particularly favorable towards "why not?" as a justification for immutable kernel APIs. We've already found a few places where Zoidberg API design has caused us problems. We need an answer to "why?" Future extensibility is a potentially valid answer but we need to do a better job of thinking through it than we have in the past.

--Jason