https://bugs.freedesktop.org/show_bug.cgi?id=104825
--- Comment #1 from Harry Wentland harry.wentland@amd.com --- This patch https://patchwork.freedesktop.org/patch/198719/ should fix it, but there could be some other issues as well.
amd-staging-drm-next has fixes for a whole bunch of driver unload issues, including what you're seeing. It's hosted at https://cgit.freedesktop.org/~agd5f/linux/log/?h=amd-staging-drm-next
Can you try the patch and/or amd-staging-drm-next?