On 29/06/2021 22:35, Matthew Brost wrote:

Add entry for i915 GuC submission / DRM scheduler integration plan. Follow up patch with details of new parallel submission uAPI to come.

v2: (Daniel Vetter)

• Expand explaination of why bonding isn't supported for GuC submission
• CC some of the DRM scheduler maintainers
• Add priority inheritance / boosting use case
• Add reasoning for removing in order assumptions

(Daniel Stone)

• Add links to priority spec

v4: (Tvrtko)

• Add TODOs section

(Daniel Vetter)

• Pull in 1 line from following patch

v5: (Checkpatch)

• Fix typos

Cc: Christian König christian.koenig@amd.com Cc: Luben Tuikov luben.tuikov@amd.com Cc: Alex Deucher alexander.deucher@amd.com Cc: Steven Price steven.price@arm.com Cc: Jon Bloomfield jon.bloomfield@intel.com Cc: Dave Airlie airlied@gmail.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Jason Ekstrand jason@jlekstrand.net Cc: dri-devel@lists.freedesktop.org Signed-off-by: Matthew Brost matthew.brost@intel.com Reviewed-by: Daniel Vetter daniel.vetter@ffwll.ch Acked-by: Dave Airlie airlied@redhat.com

---

Documentation/gpu/rfc/i915_scheduler.rst | 91 ++++++++++++++++++++++++ Documentation/gpu/rfc/index.rst | 4 ++ 2 files changed, 95 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst

diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst new file mode 100644 index 000000000000..7acd386a6b49 --- /dev/null +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -0,0 +1,91 @@ +========================================= +I915 GuC Submission/DRM Scheduler Section +=========================================

+Upstream plan +============= +For upstream the overall plan for landing GuC submission and integrating the +i915 with the DRM scheduler is:

+* Merge basic GuC submission

• • Basic submission support for all gen11+ platforms
• • Not enabled by default on any current platforms but can be enabled via

•  modparam enable_guc
  

Just a quick reminder that a lot of users have this parameter set in their grub command line because of the incorrect assumption that quick sync requires the HuC which requires the GuC command submission.

Just bear that in mind and make sure there is a minimum of quality in the GuC backend before allowing users to use the GuC, even behind this kernel parameter.

Maybe a warning in the kernel logs saying that this feature is experimental, and use "enable_guc=1" for the fullest quick sync support will improve the user satisfaction, and save you some time handling unwanted bugs.

• • Lots of rework will need to be done to integrate with DRM scheduler so

•  no need to nit pick everything in the code, it just should be
  

•  functional, no major coding style / layering errors, and not regress
  

•  execlists
  

• • Update IGTs / selftests as needed to work with GuC submission
• • Enable CI on supported platforms for a baseline

What's the plan to keep the comparison between GuC and execlists? The CI machines cannot just test the GuC as it would expose users to regressions in the default mode, and machines are hard to come by in the CI lab.

• • Rework / get CI heathly for GuC submission in place as needed

+* Merge new parallel submission uAPI

• • Bonding uAPI completely incompatible with GuC submission, plus it has

•  severe design issues in general, which is why we want to retire it no
  

•  matter what
  

• • New uAPI adds I915_CONTEXT_ENGINES_EXT_PARALLEL context setup step

•  which configures a slot with N contexts
  

• • After I915_CONTEXT_ENGINES_EXT_PARALLEL a user can submit N batches to

•  a slot in a single execbuf IOCTL and the batches run on the GPU in
  

•  paralllel
  

FYI, this is a typo

• • Initially only for GuC submission but execlists can be supported if

•  needed
  

+* Convert the i915 to use the DRM scheduler

• • GuC submission backend fully integrated with DRM scheduler

• * All request queues removed from backend (e.g. all backpressure
  

•   handled in DRM scheduler)
  

• * Resets / cancels hook in DRM scheduler
  

• * Watchdog hooks into DRM scheduler
  

• * Lots of complexity of the GuC backend can be pulled out once
  

•   integrated with DRM scheduler (e.g. state machine gets
  

•   simplier, locking gets simplier, etc...)
  

As much as I would like more consistency in the English language, simplier is not a word in the dictionary.

• • Execlists backend will minimum required to hook in the DRM scheduler

Can't parse this sentence. How about: "Minimum integration of the execlists backend in the DRM scheduler"?

Other than these typographical nitpicks, I think the plan is sane provided that you can find enough CI machines and add the warning.

• * Legacy interface
  

• * Features like timeslicing / preemption / virtual engines would
  

•   be difficult to integrate with the DRM scheduler and these
  

•   features are not required for GuC submission as the GuC does
  

•   these things for us
  

• * ROI low on fully integrating into DRM scheduler
  

• * Fully integrating would add lots of complexity to DRM
  

•   scheduler
  

• • Port i915 priority inheritance / boosting feature in DRM scheduler

• * Used for i915 page flip, may be useful to other DRM drivers as
  

•   well
  

• * Will be an optional feature in the DRM scheduler
  

• • Remove in-order completion assumptions from DRM scheduler

• * Even when using the DRM scheduler the backends will handle
  

•   preemption, timeslicing, etc... so it is possible for jobs to
  

•   finish out of order
  

• • Pull out i915 priority levels and use DRM priority levels
• • Optimize DRM scheduler as needed

+TODOs for GuC submission upstream +=================================

+* Need an update to GuC firmware / i915 to enable error state capture +* Open source tool to decode GuC logs +* Public GuC spec

+New uAPI for basic GuC submission +================================= +No major changes are required to the uAPI for basic GuC submission. The only +change is a new scheduler attribute: I915_SCHEDULER_CAP_STATIC_PRIORITY_MAP. +This attribute indicates the 2k i915 user priority levels are statically mapped +into 3 levels as follows:

+* -1k to -1 Low priority +* 0 Medium priority +* 1 to 1k High priority

+This is needed because the GuC only has 4 priority bands. The highest priority +band is reserved with the kernel. This aligns with the DRM scheduler priority +levels too.

+Spec references: +---------------- +* https://www.khronos.org/registry/EGL/extensions/IMG/EGL_IMG_context_priority... +* https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/chap5.html... +* https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-pri...

+New parallel submission uAPI +============================ +Details to come in a following patch. diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst index 05670442ca1b..91e93a705230 100644 --- a/Documentation/gpu/rfc/index.rst +++ b/Documentation/gpu/rfc/index.rst @@ -19,3 +19,7 @@ host such documentation: .. toctree::

  i915_gem_lmem.rst

+.. toctree::

• i915_scheduler.rst

On 29/06/2021 22:35, Matthew Brost wrote:

Add entry for i915 new parallel submission uAPI plan.

v2: (Daniel Vetter):

• Expand logical order explaination
• Add dummy header
• Only allow N BBs in execbuf IOCTL
• Configure parallel submission per slot not per gem context

v3: (Marcin Ślusarz):

• Lot's of typos / bad english fixed

(Tvrtko Ursulin):

• Consistent pseudo code, clean up wording in descriptions

v4: (Daniel Vetter)

• Drop flags
• Add kernel doc
• Reword a few things / fix typos

(Tvrtko)

• Reword a few things / fix typos

v5: (Checkpatch)

• Fix typos

(Docs)

• Fix warning

Cc: Tvrtko Ursulin tvrtko.ursulin@intel.com Cc: Tony Ye tony.ye@intel.com CC: Carl Zhang carl.zhang@intel.com Cc: Daniel Vetter daniel.vetter@intel.com Cc: Jason Ekstrand jason@jlekstrand.net Signed-off-by: Matthew Brost matthew.brost@intel.com Acked-by: Daniel Vetter daniel.vetter@ffwll.ch Acked-by: Tony Ye tony.ye@intel.com

---

Documentation/gpu/rfc/i915_parallel_execbuf.h | 122 ++++++++++++++++++ Documentation/gpu/rfc/i915_scheduler.rst | 59 ++++++++- 2 files changed, 180 insertions(+), 1 deletion(-) create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h

diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h new file mode 100644 index 000000000000..8cbe2c4e0172 --- /dev/null +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h @@ -0,0 +1,122 @@ +/* SPDX-License-Identifier: MIT */ +/*

• • Copyright © 2021 Intel Corporation
• */

+#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */

+/**

• • struct drm_i915_context_engines_parallel_submit - Configure engine for
• • parallel submission.
• • Setup a slot in the context engine map to allow multiple BBs to be submitted
• • in a single execbuf IOCTL. Those BBs will then be scheduled to run on the GPU
• • in parallel. Multiple hardware contexts are created internally in the i915
• • run these BBs.

The previous sentence makes little sense.

Once a slot is configured for N BBs only N BBs can be

• • submitted in each execbuf IOCTL and this is implicit behavior e.g. The user
• • doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL knows how
• • many BBs there are based on the slot's configuration. The N BBs are the last
• • N buffer objects or first N if I915_EXEC_BATCH_FIRST is set.
• • The default placement behavior is to create implicit bonds between each
• • context if each context maps to more than 1 physical engine (e.g. context is
• • a virtual engine). Also we only allow contexts of same engine class and these
• • contexts must be in logically contiguous order. Examples of the placement
• • behavior described below. Lastly, the default is to not allow BBs to
• • preempted mid BB rather insert coordinated preemption on all hardware
• • contexts between each set of BBs. Flags may be added in the future to change
• • both of these default behaviors.
• • Returns -EINVAL if hardware context placement configuration is invalid or if
• • the placement configuration isn't supported on the platform / submission
• • interface.
• • Returns -ENODEV if extension isn't supported on the platform / submission
• • interface.
• • .. code-block:: none
• • Example 1 pseudo code:
• • CS[X] = generic engine of same class, logical instance X
• • INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
• • set_engines(INVALID)
• • set_parallel(engine_index=0, width=2, num_siblings=1,

• •      engines=CS[0],CS[1])
    

• • Results in the following valid placement:
• • CS[0], CS[1]
• • Example 2 pseudo code:
• • CS[X] = generic engine of same class, logical instance X
• • INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
• • set_engines(INVALID)
• • set_parallel(engine_index=0, width=2, num_siblings=2,

• •      engines=CS[0],CS[2],CS[1],CS[3])
    

• • Results in the following valid placements:
• • CS[0], CS[1]
• • CS[2], CS[3]
• • This can also be thought of as 2 virtual engines described by 2-D array
• • in the engines the field with bonds placed between each index of the
• • virtual engines. e.g. CS[0] is bonded to CS[1], CS[2] is bonded to
• • CS[3].
• • VE[0] = CS[0], CS[2]
• • VE[1] = CS[1], CS[3]
• • Example 3 pseudo code:
• • CS[X] = generic engine of same class, logical instance X
• • INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
• • set_engines(INVALID)
• • set_parallel(engine_index=0, width=2, num_siblings=2,

• •      engines=CS[0],CS[1],CS[1],CS[3])
    

• • Results in the following valid and invalid placements:
• • CS[0], CS[1]
• • CS[1], CS[3] - Not logical contiguous, return -EINVAL
• */

+struct drm_i915_context_engines_parallel_submit {

• /**

• * @base: base user extension.
  

• */
  

• struct i915_user_extension base;
• /**

• * @engine_index: slot for parallel engine
  

• */
  

• __u16 engine_index;
• /**

• * @width: number of contexts per parallel engine
  

• */
  

• __u16 width;
• /**

• * @num_siblings: number of siblings per context
  

• */
  

• __u16 num_siblings;
• /**

• * @mbz16: reserved for future use; must be zero
  

• */
  

• __u16 mbz16;
• /**

• * @flags: all undefined flags must be zero, currently not defined flags
  

• */
  

• __u64 flags;
• /**

• * @mbz64: reserved for future use; must be zero
  

• */
  

• __u64 mbz64[3];
• /**

• * @engines: 2-d array of engine instances to configure parallel engine
  

• *
  

• * length = width (i) * num_siblings (j)
  

• * index = j + i * num_siblings
  

• */
  

• struct i915_engine_class_instance engines[0];

+} __packed;

diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst index 7acd386a6b49..cbda75065dad 100644 --- a/Documentation/gpu/rfc/i915_scheduler.rst +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -88,4 +88,61 @@ Spec references:

New parallel submission uAPI

-Details to come in a following patch. +The existing bonding uAPI is completely broken with GuC submission because +whether a submission is a single context submit or parallel submit isn't known +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple +contexts in parallel with the GuC the context must be explicitly registered with +N contexts and all N contexts must be submitted in a single command to the GuC. +The GuC interfaces do not support dynamically changing between N contexts as the +bonding uAPI does. Hence the need for a new parallel submission interface. Also +the legacy bonding uAPI is quite confusing and not intuitive at all. Furthermore +I915_SUBMIT_FENCE is by design a future fence, so not really something we should +continue to support.

A proprietary firmware interface should not be considered a good reason to retire a Linux uAPI. It is messages like this that make me super uneasy about the entire ordeal with the GuC: we would have no say on its interface, and Linux would be an afterthought of the firmware developers.

Anyway, I'm sure there are plenty of user-facing reasons to the original interface sucks, so please focus on that instead.

+The new parallel submission uAPI consists of 3 parts:

+* Export engines logical mapping +* A 'set_parallel' extension to configure contexts for parallel

• submission

+* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL

+Export engines logical mapping +------------------------------ +Certain use cases require BBs to be placed on engine instances in logical order +(e.g. split-frame on gen11+). The logical mapping of engine instances can change +based on fusing.

This sentence can be quite confusing: How about "The logical mapping of engine instances can change from one GPU to another, as some engines can be fused-out at the factory for various reasons".

The above makes it clear that this would be fixed on a per-GPU basis, and what type of fusing we are talking about, making it easier to google for.

Rather than making UMDs be aware of fusing, simply expose the +logical mapping with the existing query engine info IOCTL. Also the GuC +submission interface currently only supports submitting multiple contexts to +engines in logical order which is a new requirement compared to execlists. +Lastly, all current platforms have at most 2 engine instances and the logical +order is the same as uAPI order. This will change on platforms with more than 2 +engine instances.

+A single bit will be added to drm_i915_engine_info.flags indicating that the +logical instance has been returned and a new field, +drm_i915_engine_info.logical_instance, returns the logical instance.

+A 'set_parallel' extension to configure contexts for parallel submission +------------------------------------------------------------------------ +The 'set_parallel' extension configures a slot for parallel submission of N BBs. +It is a setup step that must be called before using any of the contexts. See +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for +similar existing examples. Once a slot is configured for parallel submission the +execbuf2 IOCTL can be called submitting N BBs in a single IOCTL. Initially only +supports GuC submission. Execlists supports can be added later if needed.

+Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and +drm_i915_context_engines_parallel_submit to the uAPI to implement this +extension.

+.. kernel-doc:: Documentation/gpu/rfc/i915_parallel_execbuf.h

•    :functions: drm_i915_context_engines_parallel_submit
  

+Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL +------------------------------------------------------------------- +Contexts that have been configured with the 'set_parallel' extension can only +submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N objects +in the drm_i915_gem_exec_object2 list or the first N if I915_EXEC_BATCH_FIRST is +set. The number of BBs is implicit based on the slot submitted and how it has +been configured by 'set_parallel' or other extensions. No uAPI changes are +required to the execbuf2 IOCTL.

Good job! The interface is pretty well described, and setting up ahead of time how many parallel contexts will be run should make it easier to reason with than the current interface.

That being said, a uAPI is not designed just with a firmware and driver in mind. Would you mind adding a list of use cases where this interface would be needed, and a small analysis about the impact of such change?

I mean, the userspace was designed with the old interface in mind, and forcing them to set up the number of parallel execution at context creation may lead them to submit empty command buffers when they would need a lower parallelism. Any recommendations there?

Cheers, Martin