I suspect what happens is that some client occasionally asks the X server to
probe  the connected displays, similar to xrandr. This powers up the dGPU, in
order to probe its display connectors. That takes some time, during which the X
server freezes.

Assuming you don't need the dGPU display outputs, adding the below to
/etc/X11/xorg.conf may serve as a workaround. You can still use the dGPU for
applications by setting the environment variable DRI_PRIME=1.

Section "ServerFlags"
        Option  "AutoAddGPU" "off"
EndSection

Section "Device"
        Identifier "Device0"
        Option  "AccelMethod" "glamor"
        Option  "DRI" "3"
EndSection