I put the issue in DRM/radeon instead of mesa/radeonsi because I thought it would be related to power management. I tried echo high > /sys/class/drm/card1/device/power_dpm_force_performance_level and performance > /sys/class/drm/card1/device/power_dpm_state but it makes no difference, still just as slow.