The situation has massively improved since I opened this issue at least with
very recent kernels. 

I'm hesitant to close it, at least not before doing some testing to see if it
can still be reproduced.