git bisected this to d57c0edfe00d3274b50f91ce3076ed0e82d28782. Reverting this solves the problem on all kernels including latest 4.9.0.