You are probably seeing this bug (bug is in the pci hotplug system): https://bugzilla.kernel.org/show_bug.cgi?id=61891 See if the latest patch there helps.