https://bugs.freedesktop.org/show_bug.cgi?id=110143
--- Comment #2 from andrew.m.mcmahon@gmail.com --- Just out of interest RBDoom3-BFG works perfectly. Consider compiling the latest master rather than running Doom 3 through Wine/Proton. Your RX580 shouldn't have any issue - my R9 285 can handle it. https://github.com/RobertBeckebans/RBDOOM-3-BFG
There's also dhewm 3 for the non-bfg/vanilla edition of doom 3: https://github.com/dhewm/dhewm3