Thanks, I was finally able to reproduce this. Seems like it's only reproducible
if the app runs at >= ~10K FPS.

It happens because the client submits so many PresentPixmap requests within a
single vertical blank period that the corresponding DRM vblank events fill up
the kernel DRM event queue. Then when the compositor sends a PresentPixmap
request, the page flip ioctl gets -ENOMEM from drm_event_reserve_init and
propagates it. (That the modesetting driver can't deal with this is its bug)

Options for addressing this:

The kernel might allow queuing page flip events even when the queue is full
with vblank events.

The xserver present code should be able to re-use a single DRM vblank event for
multiple client requests with the same target MSC.