https://bugzilla.kernel.org/show_bug.cgi?id=206987
--- Comment #38 from Petteri Aimonen (jpa@kernelbug.mail.kapsi.fi) --- @krakopo I must say I don't have any idea what could be happening on your machine. It could be explained if the kernel thread was being pre-empted, but pre-emption is disabled by kernel_fpu_begin().
It may help to ask in bug 207979 also, it has some of the long time x86 maintainers on CC.