(In reply to jimijames.bove from comment #4)
> Before I switched to AMD, I was passing an NVidia GPU (GTX 660) into my
> virtual machine, and I could unbind and rebind it between nouveau and
> vfio-pci as much as I wanted. No trouble at all. Even while X was running,
> once DRI3 support came.

You can still do that with DRI3, you just have to prevent Xorg from using the
secondary GPU, e.g. via

Section "ServerFlags"
       Option  "AutoAddGPU" "off"
EndSection

in /etc/X11/xorg.conf.