When I use modesetting driver with Option "PageFlip" "false", the stuttering is gone (however, as expected tearing is not fully prevented anymore). So there might be an actual connection to pageflipping?