Nick reported on IRC that this doesn't happen with weston, so it seems to be gnome-shell/sway compositor issues.