Re: nouveau regression with 5.7 caused by "PCI/PM: Assume ports without DLL Link Active train links in 100 ms"
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
git bisect log (had to do a second bisect, that's why the first bad and good commits appear a bit random):
git bisect start # bad: [a92b984a110863b42a3abf32e3f049b02b19e350] clk: samsung: exynos5433: Add IGNORE_UNUSED flag to sclk_i2s1 git bisect bad a92b984a110863b42a3abf32e3f049b02b19e350 # good: [4da858c086433cd012c0bb16b5921f6fafe3f803] Merge branch 'linux-5.7' of git://github.com/skeggsb/linux into drm-fixes git bisect good 4da858c086433cd012c0bb16b5921f6fafe3f803 # good: [d5dfe4f1b44ed532653c2335267ad9599c8a698e] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma git bisect good d5dfe4f1b44ed532653c2335267ad9599c8a698e # good: [b24e451cfb8c33ef5b8b4a80e232706b089914fb] ipv6: fix IPV6_ADDRFORM operation logic git bisect good b24e451cfb8c33ef5b8b4a80e232706b089914fb # good: [d843ffbce812742986293f974d55ba404e91872f] nvmet: fix memory leak when removing namespaces and controllers concurrently git bisect good d843ffbce812742986293f974d55ba404e91872f # good: [be66f10a60e3ec0b589898f78a428bcb34095730] staging: wfx: fix output of rx_stats on big endian hosts git bisect good be66f10a60e3ec0b589898f78a428bcb34095730 # good: [a4482984c41f5cc1d217aa189fe51bbbc0500f98] s390/qdio: consistently restore the IRQ handler git bisect good a4482984c41f5cc1d217aa189fe51bbbc0500f98 # good: [bec32a54a4de62b46466f4da1beb9ddd42db81b8] f2fs: fix potential use-after-free issue git bisect good bec32a54a4de62b46466f4da1beb9ddd42db81b8 # bad: [044aaaa8b1b15adb397ce423a6d97920a46b3893] habanalabs: increase timeout during reset git bisect bad 044aaaa8b1b15adb397ce423a6d97920a46b3893 # good: [6fe8ed270763a6a2e350bf37eee0f3857482ed48] arm64: dts: qcom: db820c: Fix invalid pm8994 supplies git bisect good 6fe8ed270763a6a2e350bf37eee0f3857482ed48 # good: [363e8bfc96b4e9d9e0a885408cecaf23df468523] tty: n_gsm: Fix waking up upper tty layer when room available git bisect good 363e8bfc96b4e9d9e0a885408cecaf23df468523 # bad: [afaff825e3a436f9d1e3986530133b1c91b54cd1] PCI/PM: Assume ports without DLL Link Active train links in 100 ms git bisect bad afaff825e3a436f9d1e3986530133b1c91b54cd1 # good: [be0ed15d88c65de0e28ff37a3b242e65a782fd98] HID: Add quirks for Trust Panora Graphic Tablet git bisect good be0ed15d88c65de0e28ff37a3b242e65a782fd98 # first bad commit: [afaff825e3a436f9d1e3986530133b1c91b54cd1] PCI/PM: Assume ports without DLL Link Active train links in 100 ms
[+cc Sasha -- stable kernel regression] [+cc Patrick, Kai-Heng, LKML]
On Fri, Jul 17, 2020 at 12:10:39AM +0200, Karol Herbst wrote:
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
Yeah, that stinks. We had another similar report from Patrick:
https://lore.kernel.org/r/CAErSpo5sTeK_my1dEhWp7aHD0xOp87+oHYWkTjbL7ALgDbXo-...
Apparently the problem is ec411e02b7a2 ("PCI/PM: Assume ports without DLL Link Active train links in 100 ms"), which Patrick found was backported to v5.4.49 as 828b192c57e8, and you found was backported to v5.7.6 as afaff825e3a4.
Oddly, Patrick reported that v5.7.7 worked correctly, even though it still contains afaff825e3a4.
I guess in the absence of any other clues we'll have to revert it. I hate to do that because that means we'll have slow resume of Thunderbolt-connected devices again, but that's better than having GPUs completely broken.
Could you and Patrick open bugzilla.kernel.org reports, attach dmesg logs and "sudo lspci -vv" output, and add the URLs to Kai-Heng's original report at https://bugzilla.kernel.org/show_bug.cgi?id=206837 and to this thread?
There must be a way to fix the slow resume problem without breaking the GPUs.
Bjorn
On Fri, Jul 17, 2020 at 1:54 AM Bjorn Helgaas helgaas@kernel.org wrote:
[+cc Sasha -- stable kernel regression] [+cc Patrick, Kai-Heng, LKML]
On Fri, Jul 17, 2020 at 12:10:39AM +0200, Karol Herbst wrote:
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
Yeah, that stinks. We had another similar report from Patrick:
https://lore.kernel.org/r/CAErSpo5sTeK_my1dEhWp7aHD0xOp87+oHYWkTjbL7ALgDbXo-...
Apparently the problem is ec411e02b7a2 ("PCI/PM: Assume ports without DLL Link Active train links in 100 ms"), which Patrick found was backported to v5.4.49 as 828b192c57e8, and you found was backported to v5.7.6 as afaff825e3a4.
Oddly, Patrick reported that v5.7.7 worked correctly, even though it still contains afaff825e3a4.
I guess in the absence of any other clues we'll have to revert it. I hate to do that because that means we'll have slow resume of Thunderbolt-connected devices again, but that's better than having GPUs completely broken.
Could you and Patrick open bugzilla.kernel.org reports, attach dmesg logs and "sudo lspci -vv" output, and add the URLs to Kai-Heng's original report at https://bugzilla.kernel.org/show_bug.cgi?id=206837 and to this thread?
There must be a way to fix the slow resume problem without breaking the GPUs.
I wouldn't be surprised if this is related to the Intel bridge we check against for Nouveau.. I still have to check on another laptop with the same bridge our workaround was required as well but wouldn't be surprised if it shows the same problem. Will get you the information from both systems tomorrow then.
Bjorn
Filed at https://bugzilla.kernel.org/show_bug.cgi?id=208597
oddly enough I wasn't able to reproduce it on my XPS 9560, will ping once something breaks.
On Fri, Jul 17, 2020 at 2:43 AM Karol Herbst kherbst@redhat.com wrote:
On Fri, Jul 17, 2020 at 1:54 AM Bjorn Helgaas helgaas@kernel.org wrote:
[+cc Sasha -- stable kernel regression] [+cc Patrick, Kai-Heng, LKML]
On Fri, Jul 17, 2020 at 12:10:39AM +0200, Karol Herbst wrote:
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
Yeah, that stinks. We had another similar report from Patrick:
https://lore.kernel.org/r/CAErSpo5sTeK_my1dEhWp7aHD0xOp87+oHYWkTjbL7ALgDbXo-...
Apparently the problem is ec411e02b7a2 ("PCI/PM: Assume ports without DLL Link Active train links in 100 ms"), which Patrick found was backported to v5.4.49 as 828b192c57e8, and you found was backported to v5.7.6 as afaff825e3a4.
Oddly, Patrick reported that v5.7.7 worked correctly, even though it still contains afaff825e3a4.
I guess in the absence of any other clues we'll have to revert it. I hate to do that because that means we'll have slow resume of Thunderbolt-connected devices again, but that's better than having GPUs completely broken.
Could you and Patrick open bugzilla.kernel.org reports, attach dmesg logs and "sudo lspci -vv" output, and add the URLs to Kai-Heng's original report at https://bugzilla.kernel.org/show_bug.cgi?id=206837 and to this thread?
There must be a way to fix the slow resume problem without breaking the GPUs.
I wouldn't be surprised if this is related to the Intel bridge we check against for Nouveau.. I still have to check on another laptop with the same bridge our workaround was required as well but wouldn't be surprised if it shows the same problem. Will get you the information from both systems tomorrow then.
Bjorn
Hi,
[Sorry for the delay, I was on vacation]
On Fri, Jul 17, 2020 at 01:32:10PM +0200, Karol Herbst wrote:
Thanks for reporting.
I'll check your logs and try to figure if there is something we can do to make both nouveau and TBT working at the same time.
oddly enough I wasn't able to reproduce it on my XPS 9560, will ping once something breaks.
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
On Tue, 2020-07-21 at 15:22 +0300, Mika Westerberg wrote:
Hi,
[Sorry for the delay, I was on vacation]
On Fri, Jul 17, 2020 at 01:32:10PM +0200, Karol Herbst wrote:
Thanks for reporting.
I'll check your logs and try to figure if there is something we can do to make both nouveau and TBT working at the same time.
oddly enough I wasn't able to reproduce it on my XPS 9560, will ping once something breaks.
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
On Tue, 2020-07-21 at 18:27 +0300, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Will give it a shot today and let you know the result
On Tue, 2020-07-21 at 12:00 -0400, Lyude Paul wrote:
On Tue, 2020-07-21 at 18:27 +0300, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Will give it a shot today and let you know the result
Ahh-actually, I thought the laptop I had locally could reproduce this bug but that doesn't appear to be the case whoops. Karol Herbst still has access to a machine that can test this though, so they'll likely get to trying the patch today or tommorrow
On Tue, Jul 21, 2020 at 02:24:19PM -0400, Lyude Paul wrote:
On Tue, 2020-07-21 at 12:00 -0400, Lyude Paul wrote:
On Tue, 2020-07-21 at 18:27 +0300, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Will give it a shot today and let you know the result
Ahh-actually, I thought the laptop I had locally could reproduce this bug but that doesn't appear to be the case whoops. Karol Herbst still has access to a machine that can test this though, so they'll likely get to trying the patch today or tommorrow
OK sounds good :)
On 7/21/20 10:27 AM, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Hi Mika,
I can confirm that this patch applied to 5.4.52 fixes the issue with hybrid graphics on the Thinkpad X1 Extreme gen2.
Thanks,
Pat
On Tue, Jul 21, 2020 at 01:37:12PM -0500, Patrick Volkerding wrote:
On 7/21/20 10:27 AM, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Hi Mika,
I can confirm that this patch applied to 5.4.52 fixes the issue with hybrid graphics on the Thinkpad X1 Extreme gen2.
Great, thanks for testing!
On Wed, Jul 22, 2020 at 11:25 AM Mika Westerberg mika.westerberg@linux.intel.com wrote:
On Tue, Jul 21, 2020 at 01:37:12PM -0500, Patrick Volkerding wrote:
On 7/21/20 10:27 AM, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Hi Mika,
I can confirm that this patch applied to 5.4.52 fixes the issue with hybrid graphics on the Thinkpad X1 Extreme gen2.
Great, thanks for testing!
yeah, works on the P1G2 as well.
On Thu, Jul 23, 2020 at 10:30:58PM +0200, Karol Herbst wrote:
On Wed, Jul 22, 2020 at 11:25 AM Mika Westerberg mika.westerberg@linux.intel.com wrote:
On Tue, Jul 21, 2020 at 01:37:12PM -0500, Patrick Volkerding wrote:
On 7/21/20 10:27 AM, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Hi Mika,
I can confirm that this patch applied to 5.4.52 fixes the issue with hybrid graphics on the Thinkpad X1 Extreme gen2.
Great, thanks for testing!
yeah, works on the P1G2 as well.
Thanks for testing!
Since we have the revert queued for this release cycle, I think I will send an updated version of "PCI/PM: Assume ports without DLL Link Active train links in 100 ms" after v5.9-rc1 is released that has this workaround in place.
(I'm continuing my vacation so will be offline next week).
On Fri, Jul 24, 2020 at 12:57:51PM +0300, Mika Westerberg wrote:
On Thu, Jul 23, 2020 at 10:30:58PM +0200, Karol Herbst wrote:
On Wed, Jul 22, 2020 at 11:25 AM Mika Westerberg mika.westerberg@linux.intel.com wrote:
On Tue, Jul 21, 2020 at 01:37:12PM -0500, Patrick Volkerding wrote:
On 7/21/20 10:27 AM, Mika Westerberg wrote:
On Tue, Jul 21, 2020 at 11:01:55AM -0400, Lyude Paul wrote:
Sure thing. Also, feel free to let me know if you'd like access to one of the systems we saw breaking with this patch - I'm fairly sure I've got one of them locally at my apartment and don't mind setting up AMT/KVM/SSH
Probably no need for remote access (thanks for the offer, though). I attached a test patch to the bug report:
https://bugzilla.kernel.org/show_bug.cgi?id=208597
that tries to work it around (based on the ->pm_cap == 0). I wonder if anyone would have time to try it out.
Hi Mika,
I can confirm that this patch applied to 5.4.52 fixes the issue with hybrid graphics on the Thinkpad X1 Extreme gen2.
Great, thanks for testing!
yeah, works on the P1G2 as well.
Thanks for testing!
Since we have the revert queued for this release cycle, I think I will send an updated version of "PCI/PM: Assume ports without DLL Link Active train links in 100 ms" after v5.9-rc1 is released that has this workaround in place.
(I'm continuing my vacation so will be offline next week).
Sounds fine, sorry for interrupting your vacation!
On Fri, Jul 17, 2020 at 02:43:52AM +0200, Karol Herbst wrote:
On Fri, Jul 17, 2020 at 1:54 AM Bjorn Helgaas helgaas@kernel.org wrote:
[+cc Sasha -- stable kernel regression] [+cc Patrick, Kai-Heng, LKML]
On Fri, Jul 17, 2020 at 12:10:39AM +0200, Karol Herbst wrote:
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
Yeah, that stinks. We had another similar report from Patrick:
https://lore.kernel.org/r/CAErSpo5sTeK_my1dEhWp7aHD0xOp87+oHYWkTjbL7ALgDbXo-...
Apparently the problem is ec411e02b7a2 ("PCI/PM: Assume ports without DLL Link Active train links in 100 ms"), which Patrick found was backported to v5.4.49 as 828b192c57e8, and you found was backported to v5.7.6 as afaff825e3a4.
Oddly, Patrick reported that v5.7.7 worked correctly, even though it still contains afaff825e3a4.
I guess in the absence of any other clues we'll have to revert it. I hate to do that because that means we'll have slow resume of Thunderbolt-connected devices again, but that's better than having GPUs completely broken.
Could you and Patrick open bugzilla.kernel.org reports, attach dmesg logs and "sudo lspci -vv" output, and add the URLs to Kai-Heng's original report at https://bugzilla.kernel.org/show_bug.cgi?id=206837 and to this thread?
There must be a way to fix the slow resume problem without breaking the GPUs.
I wouldn't be surprised if this is related to the Intel bridge we check against for Nouveau.. I still have to check on another laptop with the same bridge our workaround was required as well but wouldn't be surprised if it shows the same problem. Will get you the information from both systems tomorrow then.
I take it that ec411e02b7a2 will be reverted upstream?
On Fri, Jul 17, 2020 at 10:43:18AM -0400, Sasha Levin wrote:
On Fri, Jul 17, 2020 at 02:43:52AM +0200, Karol Herbst wrote:
On Fri, Jul 17, 2020 at 1:54 AM Bjorn Helgaas helgaas@kernel.org wrote:
On Fri, Jul 17, 2020 at 12:10:39AM +0200, Karol Herbst wrote:
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
Yeah, that stinks. We had another similar report from Patrick:
https://lore.kernel.org/r/CAErSpo5sTeK_my1dEhWp7aHD0xOp87+oHYWkTjbL7ALgDbXo-...
Apparently the problem is ec411e02b7a2 ("PCI/PM: Assume ports without DLL Link Active train links in 100 ms"), which Patrick found was backported to v5.4.49 as 828b192c57e8, and you found was backported to v5.7.6 as afaff825e3a4.
Oddly, Patrick reported that v5.7.7 worked correctly, even though it still contains afaff825e3a4.
I guess in the absence of any other clues we'll have to revert it. I hate to do that because that means we'll have slow resume of Thunderbolt-connected devices again, but that's better than having GPUs completely broken.
Could you and Patrick open bugzilla.kernel.org reports, attach dmesg logs and "sudo lspci -vv" output, and add the URLs to Kai-Heng's original report at https://bugzilla.kernel.org/show_bug.cgi?id=206837 and to this thread?
There must be a way to fix the slow resume problem without breaking the GPUs.
I wouldn't be surprised if this is related to the Intel bridge we check against for Nouveau.. I still have to check on another laptop with the same bridge our workaround was required as well but wouldn't be surprised if it shows the same problem. Will get you the information from both systems tomorrow then.
I take it that ec411e02b7a2 will be reverted upstream?
Yes, unless we have a better fix soon. I applied the revert to my for-linus branch, so it will appear in -next soon. I think it's a little late to get it in -rc5, so I'll probably ask Linus to pull it next week for -rc6.
Bjorn
On Thu, 2020-07-16 at 18:54 -0500, Bjorn Helgaas wrote:
[+cc Sasha -- stable kernel regression] [+cc Patrick, Kai-Heng, LKML]
On Fri, Jul 17, 2020 at 12:10:39AM +0200, Karol Herbst wrote:
On Tue, Jul 7, 2020 at 9:30 PM Karol Herbst kherbst@redhat.com wrote:
Hi everybody,
with the mentioned commit Nouveau isn't able to load firmware onto the GPU on one of my systems here. Even though the issue doesn't always happen I am quite confident this is the commit breaking it.
I am still digging into the issue and trying to figure out what exactly breaks, but it shows up in different ways. Either we are not able to boot the engines on the GPU or the GPU becomes unresponsive. Btw, this is also a system where our runtime power management issue shows up, so maybe there is indeed something funky with the bridge controller.
Just pinging you in case you have an idea on how this could break Nouveau
most of the times it shows up like this: nouveau 0000:01:00.0: acr: AHESASC binary failed
Sometimes it works at boot and fails at runtime resuming with random faults. So I will be investigating a bit more, but yeah... I am super sure the commit triggered this issue, no idea if it actually causes it.
so yeah.. I reverted that locally and never ran into issues again. Still valid on latest 5.7. So can we get this reverted or properly fixed? This breaks runtime pm for us on at least some hardware.
Yeah, that stinks. We had another similar report from Patrick:
https://lore.kernel.org/r/CAErSpo5sTeK_my1dEhWp7aHD0xOp87+oHYWkTjbL7ALgDbXo-...
Apparently the problem is ec411e02b7a2 ("PCI/PM: Assume ports without DLL Link Active train links in 100 ms"), which Patrick found was backported to v5.4.49 as 828b192c57e8, and you found was backported to v5.7.6 as afaff825e3a4.
Oddly, Patrick reported that v5.7.7 worked correctly, even though it still contains afaff825e3a4.
I guess in the absence of any other clues we'll have to revert it. I hate to do that because that means we'll have slow resume of Thunderbolt-connected devices again, but that's better than having GPUs completely broken.
Could you and Patrick open bugzilla.kernel.org reports, attach dmesg logs and "sudo lspci -vv" output, and add the URLs to Kai-Heng's original report at https://bugzilla.kernel.org/show_bug.cgi?id=206837 and to this thread?
There must be a way to fix the slow resume problem without breaking the GPUs.
Isn't it possible to tell whether a PCI device is connected through thunderbolt or not? We could probably get away with just defaulting to 100ms for thunderbolt devices without DLL Link Active specified, and then default to the old delay value for non-thunderbolt devices.
Bjorn
On Fri, Jul 17, 2020 at 03:04:10PM -0400, Lyude Paul wrote:
Isn't it possible to tell whether a PCI device is connected through thunderbolt or not? We could probably get away with just defaulting to 100ms for thunderbolt devices without DLL Link Active specified, and then default to the old delay value for non-thunderbolt devices.
pci_is_thunderbolt_attached()
On Fri, Jul 17, 2020 at 09:52:09PM +0200, Lukas Wunner wrote:
On Fri, Jul 17, 2020 at 03:04:10PM -0400, Lyude Paul wrote:
Isn't it possible to tell whether a PCI device is connected through thunderbolt or not? We could probably get away with just defaulting to 100ms for thunderbolt devices without DLL Link Active specified, and then default to the old delay value for non-thunderbolt devices.
pci_is_thunderbolt_attached()
That only works with some devices. I think we should try to keep the fact that some PCIe links may be tunneled over TBT/USB4 transparent to the PCI core and try to treat them as "standard" PCIe links if possible at all :)
dri-devel@lists.freedesktop.org
-
Bjorn Helgaas -
Karol Herbst -
Lukas Wunner -
Lyude Paul -
Mika Westerberg -
Patrick Volkerding -
Sasha Levin