TLDR: The issue is with LLVM >3.9.

I started bisecting between the two versions I knew had differences (13.0.5 and
17.0.1). I started with 17.0.1 and was using LLVM 4.0, but then to compile
13.0.5 I had to use to LLVM 3.9. This gave me an idea to test the latest
version with LLVM 3.9 and surprise, no more glitches!

So I tested 17.0.1 and a recent git version (060a6434) with LLVM 3.9 (the one
from Debian) and the glitches are gone. So for now I have a workaround that is
ok for me.