On 7 May 2015 15:37, "Chris Wilson" chris@chris-wilson.co.uk wrote:

On Thu, May 07, 2015 at 03:15:50PM +0100, Robert Bragg wrote:

...

+static int init_oa_buffer(struct perf_event *event) +{

• struct drm_i915_private *dev_priv =
  

•         container_of(event->pmu, typeof(*dev_priv), oa_pmu.pmu);
  

• struct drm_i915_gem_object *bo;
  

• int ret;
  

• BUG_ON(!IS_HASWELL(dev_priv->dev));
  

• BUG_ON(!mutex_is_locked(&dev_priv->dev->struct_mutex));
  

• BUG_ON(dev_priv->oa_pmu.oa_buffer.obj);
  

• spin_lock_init(&dev_priv->oa_pmu.oa_buffer.flush_lock);
  

• /* NB: We over allocate the OA buffer due to the way raw sample
  

data

...

•  * gets copied from the gpu mapped circular buffer into the perf
  

•  * circular buffer so that only one copy is required.
  

•  *
  

•  * For each perf sample (raw->size + 4) needs to be 8 byte
  

aligned,

...

•  * where the 4 corresponds to the 32bit raw->size member that's
  

•  * added to the sample header that userspace sees.
  

•  *
  

•  * Due to the + 4 for the size member: when we copy a report to
  

the

...

•  * userspace facing perf buffer we always copy an additional 4
  

bytes

...

•  * from the subsequent report to make up for the miss alignment,
  

but

...

•  * when a report is at the end of the gpu mapped buffer we need to
  

•  * read 4 bytes past the end of the buffer.
  

•  */
  

• bo = i915_gem_alloc_object(dev_priv->dev, OA_BUFFER_SIZE +
  

PAGE_SIZE);

...

• if (bo == NULL) {
  

•         DRM_ERROR("Failed to allocate OA buffer\n");
  

•         ret = -ENOMEM;
  

•         goto err;
  

• }
  

• dev_priv->oa_pmu.oa_buffer.obj = bo;
  

• ret = i915_gem_object_set_cache_level(bo, I915_CACHE_LLC);
  

• if (ret)
  

•         goto err_unref;
  

• /* PreHSW required 512K alignment, HSW requires 16M */
  

• ret = i915_gem_obj_ggtt_pin(bo, SZ_16M, 0);
  

• if (ret)
  

•         goto err_unref;
  

• dev_priv->oa_pmu.oa_buffer.gtt_offset =
  

i915_gem_obj_ggtt_offset(bo);

...

• dev_priv->oa_pmu.oa_buffer.addr = vmap_oa_buffer(bo);
  

You can look forward to both i915_gem_object_create_internal() and i915_gem_object_pin_vmap()

Okey, will do, thanks.

...

• /* Pre-DevBDW: OABUFFER must be set with counters off,
  

•  * before OASTATUS1, but after OASTATUS2 */
  

• I915_WRITE(GEN7_OASTATUS2, dev_priv->oa_pmu.oa_buffer.gtt_offset |
  

•            GEN7_OASTATUS2_GGTT); /* head */
  

• I915_WRITE(GEN7_OABUFFER, dev_priv->oa_pmu.oa_buffer.gtt_offset);
  

• I915_WRITE(GEN7_OASTATUS1, dev_priv->oa_pmu.oa_buffer.gtt_offset |
  

•            GEN7_OASTATUS1_OABUFFER_SIZE_16M); /* tail */
  

• DRM_DEBUG_DRIVER("OA Buffer initialized, gtt offset = 0x%x, vaddr
  

= %p",

...

•                  dev_priv->oa_pmu.oa_buffer.gtt_offset,
  

•                  dev_priv->oa_pmu.oa_buffer.addr);
  

• return 0;
  

+err_unref:

• drm_gem_object_unreference_unlocked(&bo->base);
  

But what I really what to say was: mutex deadlock^^^

Yikes, I've pushed an updated patch addressing this and can reply with a new patch here in a bit.

Thanks, - Robert

-Chris

-- Chris Wilson, Intel Open Source Technology Centre

On 7 May 2015 15:58, "Chris Wilson" chris@chris-wilson.co.uk wrote:

On Thu, May 07, 2015 at 03:15:50PM +0100, Robert Bragg wrote:

...

• /* We bypass the default perf core perf_paranoid_cpu() ||
  

•  * CAP_SYS_ADMIN check by using the PERF_PMU_CAP_IS_DEVICE
  

•  * flag and instead authenticate based on whether the current
  

•  * pid owns the specified context, or require CAP_SYS_ADMIN
  

•  * when collecting cross-context metrics.
  

•  */
  

• dev_priv->oa_pmu.specific_ctx = NULL;
  

• if (oa_attr.single_context) {
  

•         u32 ctx_id = oa_attr.ctx_id;
  

•         unsigned int drm_fd = oa_attr.drm_fd;
  

•         struct fd fd = fdget(drm_fd);
  

•         if (fd.file) {
  

Specify a ctx and not providing the right fd should be its own error, either EBADF or EINVAL.

Right, I went for both in the end; EBADF if fdget fails and EINVAL if the fd is ok but we fail to lookup a context with it.

...

•                 dev_priv->oa_pmu.specific_ctx =
  

•                         lookup_context(dev_priv, fd.file, ctx_id);
  

•         }
  

Missing fdput

Ah yes; fixed.

...

• }
  

• if (!dev_priv->oa_pmu.specific_ctx && !capable(CAP_SYS_ADMIN))
  

•         return -EACCES;
  

• mutex_lock(&dev_priv->dev->struct_mutex);
  

i915_mutex_interruptible, probably best to couple into the GPU error handling here as well especially as init_oa_buffer() will go onto touch GPU internals.

Ok, using i915_mutex_interruptible makes sense, I've also moved the locking into init_oa_buffer.

About the GPU error handling, do you have any thoughts on what could be most helpful here? I'm thinking a.t.m of extending i915_capture_reg_state() in i915_gpu_error.c to capture the OACONTROL + OASTATUS state and perhaps all the UCGCTL unit clock gating state too.

...

• ret = init_oa_buffer(event);
  

• mutex_unlock(&dev_priv->dev->struct_mutex);
  

• if (ret)
  

•         return ret;
  

• BUG_ON(dev_priv->oa_pmu.exclusive_event);
  

• dev_priv->oa_pmu.exclusive_event = event;
  

• event->destroy = i915_oa_event_destroy;
  

• /* PRM - observability performance counters:
  

•  *
  

•  *   OACONTROL, performance counter enable, note:
  

•  *
  

•  *   "When this bit is set, in order to have coherent counts,
  

•  *   RC6 power state and trunk clock gating must be disabled.
  

•  *   This can be achieved by programming MMIO registers as
  

•  *   0xA094=0 and 0xA090[31]=1"
  

•  *
  

•  *   In our case we are expected that taking pm + FORCEWAKE
  

•  *   references will effectively disable RC6 and trunk clock
  

•  *   gating.
  

•  */
  

• intel_runtime_pm_get(dev_priv);
  

• intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
  

That is a nuisance. Aside: Why isn't OA inside the powerctx? Is a subset valid with forcewake? It does perturb the system greatly to disable rc6, so I wonder if it could be made optional?

Yes, it's a shame.

I probably only really know enough about the OA unit design to be dangerous and won't try and comment in detail here, but I think there's more to it than not saving state in a power context. As I understand it, there were a number of design changes made to enable OA+RC6 support for BDW+, including having the OA unit automatically write out reports to the OA buffer when entering RC6.

I think just FORCEWAKE_RENDER would work here, but only say that because it looks like HSW only has the render forcewake domain from what I could tell.

I think I need to update the comment above these lines as I don't think these will affect crclk gating; these just handle disabling RC6.

The WIP patch I sent out basically represents me trying to get to the bottom of the clock gating constraints we have.

At this point I think I need to disable CS unit gating via UCGCTL1, as well as DOP gating for the render trunk clock via MISCCPCTL but I'm not entirely confident about that just yet. At least empirically I see these fixing some issues in rudimentary testing.

...

• return 0;
  

+}

+static void update_oacontrol(struct drm_i915_private *dev_priv) +{

• BUG_ON(!spin_is_locked(&dev_priv->oa_pmu.lock));
  

• if (dev_priv->oa_pmu.event_active) {
  

•         unsigned long ctx_id = 0;
  

•         bool pinning_ok = false;
  

•         if (dev_priv->oa_pmu.specific_ctx) {
  

•                 struct intel_context *ctx =
  

•                         dev_priv->oa_pmu.specific_ctx;
  

•                 struct drm_i915_gem_object *obj =
  

•                         ctx->legacy_hw_ctx.rcs_state;
  

If only there was ctx->legacy_hw_ctx.rcs_vma...

ok, not sure if this is a prod to add that, a heads up that this is coming or seething because some prior attempt to add this was nack'd.

...

•                 if (i915_gem_obj_is_pinned(obj)) {
  

•                         ctx_id = i915_gem_obj_ggtt_offset(obj);
  

•                         pinning_ok = true;
  

•                 }
  

•         }
  

•         if ((ctx_id == 0 || pinning_ok)) {
  

•                 bool periodic = dev_priv->oa_pmu.periodic;
  

•                 u32 period_exponent = dev_priv->oa_pmu.period_exponent;
  

•                 u32 report_format = dev_priv->oa_pmu.oa_buffer.format;
  

•                 I915_WRITE(GEN7_OACONTROL,
  

•                            (ctx_id & GEN7_OACONTROL_CTX_MASK) |
  

•                            (period_exponent <<
  

•                             GEN7_OACONTROL_TIMER_PERIOD_SHIFT) |
  

•                            (periodic ?
  

•                             GEN7_OACONTROL_TIMER_ENABLE : 0) |
  

•                            (report_format <<
  

•                             GEN7_OACONTROL_FORMAT_SHIFT) |
  

•                            (ctx_id ?
  

•                             GEN7_OACONTROL_PER_CTX_ENABLE : 0) |
  

•                            GEN7_OACONTROL_ENABLE);
  

I notice you don't use any write barriers...

ok, so I still haven't put write barriers within update_oacontrol() itself, but I've now added mmiowb()s just before unlocking which is done outside of the update_oacontrol(). I think a barrier just within update_oacontrol() could be ok a.t.m while the pinning hooks currently just use update_oacontol(), but in case we might introduce more overlapping mmio configuration within these critical sections, waiting until the unlock might be preferable. On the other hand, a.t.m the pinning callbacks now have redundant wb()s while there is no specific context filtering - not sure if that should be a concern.

Looking at this, I also didn't feel happy with the way I reset oa_pmu->specific_context when destroying an event, considering that ->specific_context being set is what determines whether the pinning callbacks may call update_oacontrol() asynchronously with respect to the pmu methods. Although we know oacontrol will be disabled by the time we come to destroy an event, it didn't seem great that that we could be continuing to run update_oacontrol() up to the point where we are resetting all the clock gating, power management and NOA enable state. I'll attach a separate patch to see if you agree this is worth changing.

Thanks for the comments.

- Robert

-Chris

Chris Wilson, Intel Open Source Technology Centre

On Fri, May 8, 2015 at 5:21 PM, Peter Zijlstra peterz@infradead.org wrote:

So I've not yet went through the entire series; but I'm wondering if its at all possible to re-use some of this work:

lkml.kernel.org/r/1428453299-19121-1-git-send-email-sukadev@linux.vnet.ibm.com

That's for a Power8 HV call that can basically return an array of values; which on a superficial level sounds a bit like what this GPU hardware does.

Thanks for this pointer.

I think the main similarity here is the ability to capture multiple counters consistent for the same point in time, but in our case we don't have an explicitly controlled transaction mechanism like this.

Although we can collect a large set of counters in a latched fashion - so they are self consistent - the selection of counters included in our OA unit reports is more rigid.

Most of our counters aren't independently aggregated, they are derived from signals selected as part of the OA unit configuration and the values are only maintained by the OA unit itself, so a re-configuration to select different signals will discard the counter values of currently selected signals.

afik re-configuring our signal selection is also relatively slow too (I was told this last week at least, but I haven't tested it myself) and so it's really geared towards applications or tools choosing a configuration to maintain for a relatively long time while profiling a workload.

I think the other big difference here is that we don't have a way to explicitly trigger a report to be written from the cpu. (Although we can read the OA counters via mmio, it's only intended for debug purposes as this subverts the hw latching of counters) This means it would be difficult to try and treat this like a transaction including a fixed set of event->read()s without a way for pmu->commit_txn() to trigger a report.

Let me read more of this..

Thanks.

- Robert

On Fri, May 8, 2015 at 5:24 PM, Peter Zijlstra peterz@infradead.org wrote:

On Thu, May 07, 2015 at 03:15:43PM +0100, Robert Bragg wrote:

...

I've changed the uapi for configuring the i915_oa specific attributes when calling perf_event_open(2) whereby instead of cramming lots of bitfields into the perf_event_attr config members, I'm now daisy-chaining a drm_i915_oa_event_attr_t structure off of a single config member that's extensible and validated in the same way as the perf_event_attr struct. I've found this much nicer to work with while being neatly extensible too.

This worries me a bit.. is there more background for this?

Would it maybe be helpful to see the before and after? I had kept this uapi change in a separate patch for a while locally but in the end decided to squash it before sending out my updated series.

Although I did find it a bit awkward with the bitfields, I was mainly concerned about the extensibility of packing logically separate attributes into the config members and had heard similar concerns from a few others who had been experimenting with my patches too.

A few simple attributes I can think of a.t.m that we might want to add in the future are: - control of the OABUFFER size - a way to ask the kernel to collect reports at the beginning and end of batch buffers, in addition to periodic reports - alternative ways to uniquely identify a context to support tools profiling a single context not necessarily owned by the current process

It could also be interesting to expose some counter configuration through these attributes too. E.g. on Broadwell+ we have 14 'Flexible EU' counters included in the OA unit reports, each with a 16bit configuration.

In a more extreme case it might also be useful to allow userspace to specify a complete counter config, which (depending on the configuration) could be over 100 32bit values to select the counter signals + configure the corresponding combining logic.

Since this pmu is in a device driver it also seemed reasonably appropriate to de-couple it slightly from the core perf_event_attr structure by allowing driver extensible attributes.

I wonder if it might be less worrisome if the i915_oa_copy_attr() code were instead a re-usable utility perhaps maintained in events/core.c, so if other pmu drivers were to follow suite there would be less risk of a mistake being made here?

Regards, - Robert

On Tue, May 19, 2015 at 3:53 PM, Peter Zijlstra peterz@infradead.org wrote:

On Fri, May 15, 2015 at 02:07:29AM +0100, Robert Bragg wrote:

...

On Fri, May 8, 2015 at 5:24 PM, Peter Zijlstra peterz@infradead.org wrote:

...

On Thu, May 07, 2015 at 03:15:43PM +0100, Robert Bragg wrote:

...

I've changed the uapi for configuring the i915_oa specific attributes when calling perf_event_open(2) whereby instead of cramming lots of bitfields into the perf_event_attr config members, I'm now daisy-chaining a drm_i915_oa_event_attr_t structure off of a single config member that's extensible and validated in the same way as the perf_event_attr struct. I've found this much nicer to work with while being neatly extensible too.

This worries me a bit.. is there more background for this?

Would it maybe be helpful to see the before and after? I had kept this uapi change in a separate patch for a while locally but in the end decided to squash it before sending out my updated series.

Although I did find it a bit awkward with the bitfields, I was mainly concerned about the extensibility of packing logically separate attributes into the config members and had heard similar concerns from a few others who had been experimenting with my patches too.

A few simple attributes I can think of a.t.m that we might want to add in the future are:

• control of the OABUFFER size
• a way to ask the kernel to collect reports at the beginning and end

of batch buffers, in addition to periodic reports

• alternative ways to uniquely identify a context to support tools

profiling a single context not necessarily owned by the current process

It could also be interesting to expose some counter configuration through these attributes too. E.g. on Broadwell+ we have 14 'Flexible EU' counters included in the OA unit reports, each with a 16bit configuration.

In a more extreme case it might also be useful to allow userspace to specify a complete counter config, which (depending on the configuration) could be over 100 32bit values to select the counter signals + configure the corresponding combining logic.

Since this pmu is in a device driver it also seemed reasonably appropriate to de-couple it slightly from the core perf_event_attr structure by allowing driver extensible attributes.

I wonder if it might be less worrisome if the i915_oa_copy_attr() code were instead a re-usable utility perhaps maintained in events/core.c, so if other pmu drivers were to follow suite there would be less risk of a mistake being made here?

So I had a peek at:

https://01.org/sites/default/files/documentation/observability_performance_c...

In an attempt to inform myself of how the hardware worked. But the document is rather sparse (and does not include broadwell).

Thanks. Sorry I can't easily fix this myself, but there is work ongoing to update this documentation. In the interim I can try and cover some of the key details here...

So from what I can gather there's two ways to observe the counters, through MMIO or trough the ring-buffer, which in turn seems triggered by a MI_REPORT_PERF_COUNT command.

There are three ways; mmio, MI_REPORT_PERF_COUNT via the ring-buffer and periodic sampling where the HW writes into a circular 'oabuffer'.

I think it's best to discount mmio as a debug feature since none of the counters are useful in isolation and mmio subverts the latching mechanism we get with the other two methods. We typically at least want to relate a counter to the number of clock cycles elapsed or the gpu time, or thread spawn counts.

This pmu driver is primarily for exposing periodic metrics, while it's up to applications to choose to emit MI_REPORT_PERF_COUNT commands as part of their command streams so I think we can mostly ignore MI_REPORT_PERF_COUNT here too.

[ Now the document talks about shortcomings of this scheme, where the MI_REPORT_PERF_COUNT command can only be placed every other command, but a single command can contain so much computation that this is not fine grained enough -- leading to the suggestion of a timer/cycle based reporting, but that is not actually mentioned afaict ]

It's in there, though unfortunately the documentation isn't very clear currently. The 'Performance Event Counting' section seems to be the appropriate place to introduce the periodic sampling feature, but just reading it myself it really only talks about the limitations of reporting like you say. I'll see if I can prod to get this improved.

If you see page 18 "Performance Statistics Registers":

OACONTROL has a 'Timer Period' field and 'Timer Enable' OABUFFER points to a circular buffer for periodic reports OASTATUS1/2 contain the head/tail pointers

Now the MI_REPORT_PERF_COUNT can select a vector (Counter Select) of which events it will write out.

This covers the regular 'A' counters. Is this correct?

There isn't a counter select in MI_REPORT_PERF_COUNT commands. I guess you either saw 'GTT select' which is for dealing with the different address spaces that the destination buffer can reside in, or in the overview you saw the mention of the 'Counter Select' field following the description of MI_REPORT_PERF_COUNT but just missed that it was referring to the OACONTROL register.

The counter configuration is shared by all three modes of reading the counters. (Though beware of confusing terms here, imho the above motioned 'Counter Select' would be better described as a 'Report Format' which is only relevant to MI_REPORT_PERF_COUNT and periodic sampling, not mmio reading)

The A counters have fixed semantics on Haswell so there's nothing to configure with these. On Broadwell some of these do become configurable as "Flexible EU Counters" which is something for us to keep in mind for later.

The semantics of the B counters are determined through the configuration of a MUX to select signals of interest and the configuration of some fixed-function ('boolean') logic that can affect how those signals are combined before feeding into one of the B counters. This configuration process is relatively slow, involving in the order of 100 register writes.

After configuring the counter semantics, then for periodic and MI_REPORT_PERF_COUNT usage (which work in terms of reports), we have to choose the layout of those reports.

OACONTROL has a report format field ('Counter Select') that gives us a limited way to trade off how many A or B counters are included, and the memory bandwidth consumed writing out reports. So it doesn't affect the configuration of counters per-se, it's just a way to ignore some of the currently configured counters by selecting smaller reports. Just looking at the OACONTROL documentation though I see something amiss, as this field isn't only pre-Haswell. The possible formats/layouts for Haswell are documented on page 10.

Then there are the 'B' counters, which appear to be programmable through the CEC MMIO registers.

The fixed-function logic affecting B counters is configured via those CEC registers as well as some per-counter 'report trigger' and 'start trigger' registers not currently described. Overall though programming the B counters is a combination of the fixed-function logic and the MUX configuration that determines the signals that feed into this logic, before counting.

These B events can also be part of the MI_REPORT_PERF_COUNT vector.

Right?

Right, part of MI_REPORT_PERF_COUNT and periodic reports.

So for me the 'natural' way to represent this in perf would be through event groups. Create a perf event for every single event -- yes this is 53 events.

So when I was first looking at this work I had considered the possibility of separate events, and these are some of the things that in the end made me forward the hardware's raw report data via a single event instead...

There are 100s of possible B counters depending on the MUX configuration + fixed-function logic in addition to the A counters. A choice would need to be made about whether to expose events for the configurable counters that aren't inherently associated with any semantics, or instead defining events for counters with specific semantics (with 100s of possible counters to define). The later would seem more complex for userspace and the kernel if they both now have to understand the constraints on what counters can be used together. I guess with either approach we would also need to have some form of dedicated group leader event accepting attributes for configuring the state that affects the group as a whole, such as the counter configuration (3D vs GPGPU vs media etc). I'm not sure where we would handle the context-id + drm file descriptor attributes for initiating single context profiling but guess we'd need to authenticate each individual event open. It's not clear if we'd configure the report layout via the group leader, or try to automatically choose the most compact format based on the group members. I'm not sure how pmus currently handle the opening of enabled events on an enabled group but I think there would need to be limitations in our case that new members can't result in a reconfigure of the counters if that might loose the current counter values known to userspace.

From a user's pov, there's no real freedom to mix and match which

counters are configured together, and there's only some limited ability to ignore some of the currently selected counters by not including them in reports.

Something to understand here is that we have to work with sets of pre-defined MUX + fixed-function logic configurations that have been validated to give useful metrics for specific use cases, such as benchmarking 3D rendering, GPGPU or media workloads.

As it is currently the kernel doesn't need to know anything about the semantics of individual counters being selected, so it's currently convenient that we can aim to maintain all the counter meta data we have in userspace according to the changing needs of tools or drivers (e.g. names, descriptions, units, max values, normalization equations), de-coupled from the kernel, instead of splitting it between the kernel and userspace.

A benefit of being able to change the report size is to reduce memory bandwidth usage that can skew measurements. It's possible to request the gpu to write out periodic snapshots at a very high frequency (we can program a period as low as 160 nanoseconds) and higher frequencies can start to expose some interesting details about how the gpu is utilized - though with notable observer effects too. How careful we are to not waste bandwidth is expected to determine what sampling resolutions we can achieve before significantly impacting what we are measuring.

Splitting the counters up looked like it could increase the bandwidth we use quite a bit. The main difference comes from requiring 64bit values instead of the 32bit values in our raw reports. This can be offset partly since there are quite a few 'reserved'/redundant A counters that don't need forwarding. As an example in the most extreme case, instead of an 8 byte perf_event_header + 4byte raw_size + 256 byte reports + 4 byte padding every 160ns ~= 1.5GB/s, we might have 33 A counters (ignoring redundant ones) + 16 configurable counters = 400 byte struct read_format (using PERF_FORMAT_GROUP) + 8 byte perf_event_header every 160ns ~= 2.4GB/s. On the other hand though we could choose to forward only 2 or 3 counters of interest at these high frequencies which isn't possible currently.

Use the MMIO reads for the regular read() interface, and use a hrtimer placing MI_REPORT_PERF_COUNT commands, with a counter select mask covering the all events in the current group, for sampling.

Unfortunately due to the mmio limitations and the need to relate counters I can't imagine many use cases for directly accessing the counters individually via the read() interface.

MI_REPORT_PERF_COUNT commands are really only intended for collecting reports in sync with a command stream. We are experimenting currently with an extension of my PMU driver that emits MI_REPORT_PERF_COUNT commands automatically around the batches of commands submitted by userspace so we can do a better job of filtering metrics across many gpu contexts, but for now the expectation is that the kernel shouldn't be emitting MI_REPORT_PERF_COUNT commands. We emit MI_REPORT_PERF_COUNT commands within Mesa for example to implement the GL_INTEL_performance_query extension, at the start and end of a query around a sequence of commands that the application is interested in measuring.

Then obtain the vector values using PERF_SAMPLE_READ and PERF_FORMAT_READ -- and use the read txn support to consume the ring-buffer vectors instead of the MMIO registers.

It sounds potentially doable to consume periodic OABUFFER reports via the read_txn support.

You can use the perf_event_attr::config to select the counter (A0-A44, B0-B7) and use perf_event_attr::config1 (low and high dword) for the corresponding CEC registers.

Hopefully covered above, but since the fixed-function state is so dependent on the MUX configuration I think it currently makes sense to treat the MUX plus logic state (including the CEC state) a tightly coupled unit.

The Flexible EU Counters for Broadwell+ could be more amenable to this kind of independent configuration, as I don't believe they are dependant on the MUX configuration.

One idea that's come up a lot though is having the possibility of being able to configure an event with a full MUX + fixed-function state description.

This does not require random per driver ABI extentions for perf_event_attr, nor your custom output format.

Am I missing something obvious here?

Definitely nothing 'obvious' since the current documentation is notably incomplete a.t.m, but I don't think we were on the same page about how the hardware works and our use cases.

Hopefully some of my above comments help clarify some details.

I think I'll hold off from looking at changing anything specific until you've had a chance to read my comments above in case you have more thoughts and feedback.

Thanks for looking at this, - Robert

On Thu, May 21, 2015 at 12:17:48AM +0100, Robert Bragg wrote:

...

So for me the 'natural' way to represent this in perf would be through event groups. Create a perf event for every single event -- yes this is 53 events.

So when I was first looking at this work I had considered the possibility of separate events, and these are some of the things that in the end made me forward the hardware's raw report data via a single event instead...

There are 100s of possible B counters depending on the MUX configuration + fixed-function logic in addition to the A counters.

There's only 8 B counters, what there is 100s of is configurations, and that's no different from any other PMU. There's 100s of events to select on the CPU side as well - yet only 4 (general purpose) counters (8 if you disable HT on recent machines) per CPU.

Of course, you could create more than 8 events at any one time, and then perf will round-robin the configuration for you.

A choice would need to be made about whether to expose events for the configurable counters that aren't inherently associated with any semantics, or instead defining events for counters with specific semantics (with 100s of possible counters to define). The later would seem more complex for userspace and the kernel if they both now have to understand the constraints on what counters can be used together.

Again, no different from existing PMUs. The most 'interesting' part of any PMU driver is event scheduling.

The explicit design of perf was to put event scheduling _in_ the kernel and not allow userspace direct access to the hardware (as previous PMU models did).

I guess with either approach we would also need to have some form of dedicated group leader event accepting attributes for configuring the state that affects the group as a whole, such as the counter configuration (3D vs GPGPU vs media etc).

That depends a bit on how flexible you want to switch between these modes; with the cost being in the 100s of MMIO register writes, it might just not make sense to make this too dynamic.

An option might be to select the mode through the PMU driver's sysfs files. Another option might be to expose the B counters 3 times and have each set be mutually exclusive. That is, as soon as you've created a 3D event, you can no longer create GPGPU/Media events.

I'm not sure where we would handle the context-id + drm file descriptor attributes for initiating single context profiling but guess we'd need to authenticate each individual event open.

Event open is a slow path, so that should not be a problem, right?

It's not clear if we'd configure the report layout via the group leader, or try to automatically choose the most compact format based on the group members. I'm not sure how pmus currently handle the opening of enabled events on an enabled group but

With the PERF_FORMAT_GROUP layout changing in-flight. I would recommend not doing that -- decoding the output will be 'interesting' but not impossible.

I think there would need to be limitations in our case that new members can't result in a reconfigure of the counters if that might loose the current counter values known to userspace.

I'm not entirely sure what you mean here.

From a user's pov, there's no real freedom to mix and match which counters are configured together, and there's only some limited ability to ignore some of the currently selected counters by not including them in reports.

It is 'impossible' to create a group that is not programmable. That is, the pmu::event_init() call _should_ verify that the addition of the event to the group (as given by event->group_leader) is valid.

Something to understand here is that we have to work with sets of pre-defined MUX + fixed-function logic configurations that have been validated to give useful metrics for specific use cases, such as benchmarking 3D rendering, GPGPU or media workloads.

This is fine; as stated above. Since these are limited pieces of 'firmware' which are expensive to load, you don't have to do a fully dynamic solution here.

As it is currently the kernel doesn't need to know anything about the semantics of individual counters being selected, so it's currently convenient that we can aim to maintain all the counter meta data we have in userspace according to the changing needs of tools or drivers (e.g. names, descriptions, units, max values, normalization equations), de-coupled from the kernel, instead of splitting it between the kernel and userspace.

And that fully violates the design premise of perf. The kernel should be in control of resources, not userspace.

If we'd have put userspace in charge, we could now not profile the same task by two (or more) different observers. But with kernel side counter management that is no problem at all.

A benefit of being able to change the report size is to reduce memory bandwidth usage that can skew measurements. It's possible to request the gpu to write out periodic snapshots at a very high frequency (we can program a period as low as 160 nanoseconds) and higher frequencies can start to expose some interesting details about how the gpu is utilized - though with notable observer effects too. How careful we are to not waste bandwidth is expected to determine what sampling resolutions we can achieve before significantly impacting what we are measuring.

Splitting the counters up looked like it could increase the bandwidth we use quite a bit. The main difference comes from requiring 64bit values instead of the 32bit values in our raw reports. This can be offset partly since there are quite a few 'reserved'/redundant A counters that don't need forwarding. As an example in the most extreme case, instead of an 8 byte perf_event_header + 4byte raw_size + 256 byte reports + 4 byte padding every 160ns ~= 1.5GB/s, we might have 33 A counters (ignoring redundant ones) + 16 configurable counters = 400

In that PDF there's only 8 configurable 'B' counters.

byte struct read_format (using PERF_FORMAT_GROUP) + 8 byte perf_event_header every 160ns ~= 2.4GB/s. On the other hand though we could choose to forward only 2 or 3 counters of interest at these high frequencies which isn't possible currently.

Right; although you'd have to spend some cpu cycles on the format shift. Which might make it moot again. That said; I would really prefer we start out with trying to make the generic format stuff work before trying to come up with special case hacks.

...

Use the MMIO reads for the regular read() interface, and use a hrtimer placing MI_REPORT_PERF_COUNT commands, with a counter select mask covering the all events in the current group, for sampling.

Unfortunately due to the mmio limitations and the need to relate counters I can't imagine many use cases for directly accessing the counters individually via the read() interface.

Fair enough.

MI_REPORT_PERF_COUNT commands are really only intended for collecting reports in sync with a command stream. We are experimenting currently with an extension of my PMU driver that emits MI_REPORT_PERF_COUNT commands automatically around the batches of commands submitted by userspace so we can do a better job of filtering metrics across many gpu contexts, but for now the expectation is that the kernel shouldn't be emitting MI_REPORT_PERF_COUNT commands. We emit MI_REPORT_PERF_COUNT commands within Mesa for example to implement the GL_INTEL_performance_query extension, at the start and end of a query around a sequence of commands that the application is interested in measuring.

I will try and read up on the GL_INTEL_performance_query thing.

...

You can use the perf_event_attr::config to select the counter (A0-A44, B0-B7) and use perf_event_attr::config1 (low and high dword) for the corresponding CEC registers.

Hopefully covered above, but since the fixed-function state is so dependent on the MUX configuration I think it currently makes sense to treat the MUX plus logic state (including the CEC state) a tightly coupled unit.

Oh wait, the A counters are also affected by the MUX programming! That wasn't clear to me before this.

Same difference though; you could still do either the sysfs aided flips or the mutually exclusive counter types to deal with this; all still assuming reprogramming all that is 'expensive'.

The Flexible EU Counters for Broadwell+ could be more amenable to this kind of independent configuration, as I don't believe they are dependant on the MUX configuration.

That sounds good :-)

One idea that's come up a lot though is having the possibility of being able to configure an event with a full MUX + fixed-function state description.

Right; seeing how there's only a very limited number of these MUX programs validated (3D, GPGPU, Media, any others?) this should be doable. And as mentioned before, you could disable the other types once you create one in order to limit the reprogramming thing.

...

This does not require random per driver ABI extentions for perf_event_attr, nor your custom output format.

Am I missing something obvious here?

Definitely nothing 'obvious' since the current documentation is notably incomplete a.t.m, but I don't think we were on the same page about how the hardware works and our use cases.

Hopefully some of my above comments help clarify some details.

Yes, thanks!

On Wed, May 27, 2015 at 4:39 PM, <> wrote:

On Thu, May 21, 2015 at 12:17:48AM +0100, Robert Bragg wrote:

...

...

So for me the 'natural' way to represent this in perf would be through event groups. Create a perf event for every single event -- yes this is 53 events.

So when I was first looking at this work I had considered the possibility of separate events, and these are some of the things that in the end made me forward the hardware's raw report data via a single event instead...

There are 100s of possible B counters depending on the MUX configuration + fixed-function logic in addition to the A counters.

There's only 8 B counters, what there is 100s of is configurations, and that's no different from any other PMU. There's 100s of events to select on the CPU side as well - yet only 4 (general purpose) counters (8 if you disable HT on recent machines) per CPU.

To clarify one thing here; In the PRM you'll see a reference to 'Reserved' slots in the report layouts and these effectively correspond to a further 8 configurable counters. These further counters get referred to as 'C' counters for 'custom' and similar to the B counters they are defined based on the MUX configuration except they skip the boolean logic pipeline.

Thanks for the comparison with the CPU.

My main thought here is to beware of considering these configurable counters as 'general purpose' if that might also imply orthogonality. All our configurable counters build on top of a common/shared MUX configuration so I'd tend to think of them more like a 'general purpose counter set'.

Of course, you could create more than 8 events at any one time, and then perf will round-robin the configuration for you.

This hardware really isn't designed to allow round-robin reconfiguration. If we change the metric set then we will loose the aggregate value of our B counters. Besides writing the large MUX config + boolean logic state we would also need to save the current aggregated counter values only maintained by the OA unit so they could be restored later as well as any internal state that's simply not accessible to us. If we reconfigure to try and round-robin between sets of counters each reconfiguration will trash state that we have no way of restoring. The counters have to no longer be in use if there's no going back once we reconfigure.

...

A choice would need to be made about whether to expose events for the configurable counters that aren't inherently associated with any semantics, or instead defining events for counters with specific semantics (with 100s of possible counters to define). The later would seem more complex for userspace and the kernel if they both now have to understand the constraints on what counters can be used together.

Again, no different from existing PMUs. The most 'interesting' part of any PMU driver is event scheduling.

The explicit design of perf was to put event scheduling _in_ the kernel and not allow userspace direct access to the hardware (as previous PMU models did).

I'm thinking your 'event scheduling' here refers to the kernel being able to handle multiple concurrent users of the same pmu, but then we may be talking cross purposes...

I was describing how (if we exposed events for counters with well defined semantics - as opposed to just 16 events for the B/C counters) the driver would need to do some kind of matching based on the events added to a group to deduce which MUX configuration should be used and saying that in this case userspace would need to be very aware of which events belong to a specific MUX configuration.

I don't think we have the hardware flexibility to support scheduling in terms of multiple concurrent users, unless they happen to want exactly the same configuration.

I also don't think we have a use case for accessing gpu metrics where concurrent access is really important. It's possible to think of cases where it might be nice, except they are unlikely to involve the same configuration so given the current OA unit design I haven't had plans to try and support concurrent access and haven't had any requests for it so far.

...

I guess with either approach we would also need to have some form of dedicated group leader event accepting attributes for configuring the state that affects the group as a whole, such as the counter configuration (3D vs GPGPU vs media etc).

That depends a bit on how flexible you want to switch between these modes; with the cost being in the 100s of MMIO register writes, it might just not make sense to make this too dynamic.

An option might be to select the mode through the PMU driver's sysfs files. Another option might be to expose the B counters 3 times and have each set be mutually exclusive. That is, as soon as you've created a 3D event, you can no longer create GPGPU/Media events.

Hmm, interesting idea to have global/pmu state be controlled via sysfs...

I suppose the security model doesn't seem as clear, with sysfs access gated by file permissions while we want to gate some control based on whether you have access to a gpu context. The risk of multiple users looking to open a group of events and first setting a global mode via sysfs seems like it could also silently misconfigure your events with the wrong mode. Maybe it could become immutable once one event is opened and you could double check afterwards.

Besides the mode/MUX config choice, other state that is shared by the group includes: - single context vs system wide profiling - the timer exponent

Using sysfs I think we'd have to be careful to not expose configuration options that relate to the security policy if there's a risk of a non-privileged process affecting it.

...

I'm not sure where we would handle the context-id + drm file descriptor attributes for initiating single context profiling but guess we'd need to authenticate each individual event open.

Event open is a slow path, so that should not be a problem, right?

Yeah I don't think I'd be concerned about duplicating the checks per-event from a performance pov.

I'm more concerned about differentiating security checks that relate to the final configuration as a whole vs individual events. In this case we use a drm file descriptor + context handle to allow profiling a single gpu context that the current process has access to, but the choice of profiling a single context or across all contexts affects the final hardware configuration too which relates to the group. Even if a process has the privileges to additionally open an event for system wide metrics, this choice is part of the OA unit configuration and incompatible with single-context filtering. For reference, on Haswell the reports don't include a context identifier which makes it difficult to try and emulate a single-context filtering event with a system wide hardware configuration.

I can see that we'd be able to cross-check event compatibility for each addition to the group but I also can't help but see this as an example of how tightly coupled the configuration of these counters are.

...

It's not clear if we'd configure the report layout via the group leader, or try to automatically choose the most compact format based on the group members. I'm not sure how pmus currently handle the opening of enabled events on an enabled group but

With the PERF_FORMAT_GROUP layout changing in-flight. I would recommend not doing that -- decoding the output will be 'interesting' but not impossible.

Right, I wouldn't want to have to handle such corner cases so I think I'd want to be able to report an error back to userspace attempting to extend a group that's ever been activated.

...

I think there would need to be limitations in our case that new members can't result in a reconfigure of the counters if that might loose the current counter values known to userspace.

I'm not entirely sure what you mean here.

This is referring to the issue of us not being able to explicitly save and restore the state of the OA unit to be able to allow reconfiguring the counters while they are in use.

The counter values for B/C counters are only maintained by the OA unit and in the case of B counters which supports e.g. referencing delayed values, there is state held in the unit that we have no way to explicitly access and save.

...

From a user's pov, there's no real freedom to mix and match which counters are configured together, and there's only some limited ability to ignore some of the currently selected counters by not including them in reports.

It is 'impossible' to create a group that is not programmable. That is, the pmu::event_init() call _should_ verify that the addition of the event to the group (as given by event->group_leader) is valid.

...

Something to understand here is that we have to work with sets of pre-defined MUX + fixed-function logic configurations that have been validated to give useful metrics for specific use cases, such as benchmarking 3D rendering, GPGPU or media workloads.

This is fine; as stated above. Since these are limited pieces of 'firmware' which are expensive to load, you don't have to do a fully dynamic solution here.

...

As it is currently the kernel doesn't need to know anything about the semantics of individual counters being selected, so it's currently convenient that we can aim to maintain all the counter meta data we have in userspace according to the changing needs of tools or drivers (e.g. names, descriptions, units, max values, normalization equations), de-coupled from the kernel, instead of splitting it between the kernel and userspace.

And that fully violates the design premise of perf. The kernel should be in control of resources, not userspace.

This isn't referring to programmable resources though; it's only talking about where the higher-level meta data about these counters is maintained, including names and descriptions detailing the specific counter semantics as well as a description of the equations/expressions that should be used to normalize these counters to be useful to users.

This seems very comparable to other PMUs that expose _RAW, hardware specific data via samples that rely on a userspace library like libpfm4 to understand.

I see different tools have slightly different needs and may want to normalize some counters differently. This data is still evolving over time, and overall feel it's going to be more practical/maintainable for us to try and keep this semantic data consolidated to userspace.

For reference; what I'm really referring to here is an established XML description of OA counters maintained and validated by engineers closely involved with the hardware. I think it makes sense for us to factor into this trade-off the benefit we get from being able to easily leverage this data but considering the relative lack of stability of this data makes me prefer to limit ourselves to just putting the MUX and boolean logic configurations in the kernel.

If we'd have put userspace in charge, we could now not profile the same task by two (or more) different observers. But with kernel side counter management that is no problem at all.

Sorry, I think my comment must have come across wrong because I wasn't talking about a choice that affects who's responsible for configuring the counters or supporting multiple observers or not.

I was talking about a trade-off for where we maintain the higher level information about the counters that can be exposed, especially the code for normalizing individual counters.

...

A benefit of being able to change the report size is to reduce memory bandwidth usage that can skew measurements. It's possible to request the gpu to write out periodic snapshots at a very high frequency (we can program a period as low as 160 nanoseconds) and higher frequencies can start to expose some interesting details about how the gpu is utilized - though with notable observer effects too. How careful we are to not waste bandwidth is expected to determine what sampling resolutions we can achieve before significantly impacting what we are measuring.

Splitting the counters up looked like it could increase the bandwidth we use quite a bit. The main difference comes from requiring 64bit values instead of the 32bit values in our raw reports. This can be offset partly since there are quite a few 'reserved'/redundant A counters that don't need forwarding. As an example in the most extreme case, instead of an 8 byte perf_event_header + 4byte raw_size + 256 byte reports + 4 byte padding every 160ns ~= 1.5GB/s, we might have 33 A counters (ignoring redundant ones) + 16 configurable counters = 400

In that PDF there's only 8 configurable 'B' counters.

Sorry that it's another example of being incomplete. The current document only says enough about B counters to show how to access the gpu clock that is required to normalise many of the A counters. There are also the 8 C counters I mentioned earlier derived from the MUX configuration but without the boolean logic.

...

byte struct read_format (using PERF_FORMAT_GROUP) + 8 byte perf_event_header every 160ns ~= 2.4GB/s. On the other hand though we could choose to forward only 2 or 3 counters of interest at these high frequencies which isn't possible currently.

Right; although you'd have to spend some cpu cycles on the format shift. Which might make it moot again. That said; I would really prefer we start out with trying to make the generic format stuff work before trying to come up with special case hacks.

...

...

Use the MMIO reads for the regular read() interface, and use a hrtimer placing MI_REPORT_PERF_COUNT commands, with a counter select mask covering the all events in the current group, for sampling.

Unfortunately due to the mmio limitations and the need to relate counters I can't imagine many use cases for directly accessing the counters individually via the read() interface.

Fair enough.

...

MI_REPORT_PERF_COUNT commands are really only intended for collecting reports in sync with a command stream. We are experimenting currently with an extension of my PMU driver that emits MI_REPORT_PERF_COUNT commands automatically around the batches of commands submitted by userspace so we can do a better job of filtering metrics across many gpu contexts, but for now the expectation is that the kernel shouldn't be emitting MI_REPORT_PERF_COUNT commands. We emit MI_REPORT_PERF_COUNT commands within Mesa for example to implement the GL_INTEL_performance_query extension, at the start and end of a query around a sequence of commands that the application is interested in measuring.

I will try and read up on the GL_INTEL_performance_query thing.

You can see my code for this here, in case that's helpful: https://github.com/rib/mesa/tree/wip/rib/oa-hsw-4.0.0

You might also be interested in my more recent 'codegen' branch too (work in progress though): https://github.com/rib/mesa/tree/wip/rib/oa-hsw-codegen

Here you can see an example of the meta data I mentioned earlier, e.g. describing the equations for normalizing the counters for one metrics set:

https://github.com/rib/mesa/blob/wip/rib/oa-hsw-codegen/src/mesa/drivers/dri...

This set corresponds to the "3D" metric set enabled in the i915_oa pmu driver.

More example data can also be seen in my gputop tool, with 'render basic', 'compute basic', 'compute extended', 'memory reads', 'memory writes' and 'sampler balance' metric sets:

https://github.com/rib/gputop/blob/641d7644df64a871f36f3c283cfa18a2f1530813/...

...

...

You can use the perf_event_attr::config to select the counter (A0-A44, B0-B7) and use perf_event_attr::config1 (low and high dword) for the corresponding CEC registers.

Hopefully covered above, but since the fixed-function state is so dependent on the MUX configuration I think it currently makes sense to treat the MUX plus logic state (including the CEC state) a tightly coupled unit.

Oh wait, the A counters are also affected by the MUX programming! That wasn't clear to me before this.

Er actually you were right before, the raw A counters aren't directly affected by the MUX configuration. In this paragraph 'fixed-function state' was referring to the boolean logic state for B counters in case that was unclear and the CEC ('custom event counter') registers are a part of that boolean logic state.

That said though, the raw A counters aren't really /useful/ without the MUX programming because most of them are so closely related to the gpu clock (which we access via a B or C counter) that we have to program the MUX before we can normalize the A counters to be meaningful to users.

Same difference though; you could still do either the sysfs aided flips or the mutually exclusive counter types to deal with this; all still assuming reprogramming all that is 'expensive'.

...

The Flexible EU Counters for Broadwell+ could be more amenable to this kind of independent configuration, as I don't believe they are dependant on the MUX configuration.

That sounds good :-)

...

One idea that's come up a lot though is having the possibility of being able to configure an event with a full MUX + fixed-function state description.

Right; seeing how there's only a very limited number of these MUX programs validated (3D, GPGPU, Media, any others?) this should be doable. And as mentioned before, you could disable the other types once you create one in order to limit the reprogramming thing.

Here I meant being able to support userspace supplying a full MUX configuration and boolean logic configuration (i.e. large arrays of data), for userspace tools being able to experiment with new configurations during the early stages of building and validating these configurations - so mostly interesting to the engineers responsible for defining and testing these configurations in the first place since it means they can share experimental configurations with tools with the caveat that you'd likely need root privileges to use the interface. It's appealing to have a low barrier for enabling people to test new configurations without the need for users to build a custom kernel.

I'm not worried about supporting this in the short term, but it's still a usecase that could be helpful to support eventually, that I want to keep in mind at least.

About the number of different configurations, I've been using 3D, gpgpu and media as summary examples, but there are quite a few configurations really...

On Haswell these are names of some of the sets I'm aware of... Render Metrics Basic Gen7.5 Compute Metrics Basic Gen7.5 Compute Metrics Extended Gen7.5 Render Metrics Slice Balance Gen7.5 Memory Reads Distribution Gen7.5 Memory Writes Distribution Gen7.5 Metric set SamplerBalance Memory Reads on Write Port Distribution Gen7.5 Stencil PMA Hold Metrics Gen7.5 Media Memory Reads Distribution Gen7.5 Media Memory Writes Distribution Gen7.5 Media VME Pipe Gen7.5

For Broadwell we have more; for example... Render Metrics Basic Gen8 Compute Metrics Basic Gen8 Render Metrics for 3D Pipeline Profile Memory Reads Distribution Gen8 Memory Writes Distribution Gen8 Compute Metrics Extended Gen8 Compute Metrics L3 Cache Gen8 Data Port Reads Coalescing Gen8 Data Port Writes Coalescing Gen8 Metric set HDCAndSF Metric set L3_1 Metric set L3_2 Metric set L3_3 Metric set L3_4 Metric set RasterizerAndPixelBackend Metric set Sampler_1 Metric set Sampler_2 Metric set TDL_1 Metric set TDL_2 Stencil PMA Hold Metrics Gen8 Media Memory Reads Distribution Gen8 Media Memory Writes Distribution Gen8 Media VME Pipe Gen8 HDC URB Coalescing Metrics Gen8

I'm not sure that it will end up making sense to publish all of these, depending on what kind of testing these have been through to validate that they give helpful data, and on the other hand these are evolving and there may be more over time, but hopefully it at least gives a ballpark idea of the number of configurations that could be interesting to enable between Haswell and Broadwell.

...

...

This does not require random per driver ABI extentions for perf_event_attr, nor your custom output format.

Am I missing something obvious here?

Definitely nothing 'obvious' since the current documentation is notably incomplete a.t.m, but I don't think we were on the same page about how the hardware works and our use cases.

Hopefully some of my above comments help clarify some details.

Yes, thanks!

Ok, I think there are some clarifications on the nature of our OA hardware that we're drilling into here that might also help explain more of the trade-offs that were considered regarding forwarding raw OA hardware reports.

I wonder if it could be good for us to take a little bit of a step back here just to re-consider the implications of enabling a pmu that really doesn't relate to the cpu or tasks running on cpus - since I think this is the first example of that, and it's also had a bearing on how we currently forward samples via perf.

I think the basic implications to consider are that for our use cases we wouldn't expect to be requesting cpu centric info in samples, such as _IP, _CALLCHAIN, _CPU, _BRANCH_STACK and_REGS/STACK_USER and that we wouldn't open events for specific pids/tasks.

I don't think we'd expect to use existing userspace tools with this pmu, such as perf, and instead:

- We're accessing perf from within OpenGL for implementing a performance query extension that applications/games/tools can then use to get metrics for a single gpu context owned by OpenGL. - We're creating new tools that are geared for viewing gpu metrics, including gputop and grafips

I have a feeling that Ingo and yourself may currently be thinking of the OA unit like an uncore pmu with the idea that tools like perf should Just Work™ with OA counters, while I don't think that's going to be the case.

Even so, I still feel that we benefit from being able to leverage the perf infrastructure to support our use cases and even though perf is currently quite cpu centric I've been happy that it's turned out to be a good fit for exposing device metrics too and ignoring/bypassing some of cpu specific bits for our needs has been straight forward.

I don't bring this up as a tangent to the question of having one raw-report event vs multiple single counter events, but rather to highlight that our gpu focused use cases might be a bit different to what you're used to and I hope our tooling and OpenGL work can inform this discussion too; representing the practical ends that we're most eager to support.

Regards, - Robert