Re: [PATCH v3] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges
I think there is something I totally forgot about:
When there was never a driver bound to the GPU, and if runtime power management gets enabled on that device, runtime suspend/resume works as expected (I am not 100% sure on if that always works, but I will recheck that). In the past I know that at some point I "bisected" the nouveau driver to figure out what actually breaks it and found out that some script executed with the help of an on-chip engine (signed script, signed firmware, both vbios provided) makes it break. Debugging the script lead me to the PCIe link speed changes done inside the script breaking it.
But as "reverting" the speed change didn't make it work reliably again, I think I need to get back on that and check if it's something else. I will try to convert the script into C or python code to make it more accessible to debug and hopefully I'll find something I overlooked the last time.
On Mon, Oct 21, 2019 at 6:40 PM Karol Herbst kherbst@redhat.com wrote:
On Mon, Oct 21, 2019 at 5:46 PM Mika Westerberg mika.westerberg@intel.com wrote:
On Mon, Oct 21, 2019 at 04:49:09PM +0200, Karol Herbst wrote:
On Mon, Oct 21, 2019 at 4:09 PM Mika Westerberg mika.westerberg@intel.com wrote:
On Mon, Oct 21, 2019 at 03:54:09PM +0200, Karol Herbst wrote:
I really would like to provide you more information about such workaround but I'm not aware of any ;-) I have not seen any issues like this when D3cold is properly implemented in the platform. That's why I'm bit skeptical that this has anything to do with specific Intel PCIe ports. More likely it is some power sequence in the _ON/_OFF() methods that is run differently on Windows.
yeah.. maybe. I really don't know what's the actual root cause. I just know that with this workaround it works perfectly fine on my and some other systems it was tested on. Do you know who would be best to approach to get proper documentation about those methods and what are the actual prerequisites of those methods?
Those should be documented in the ACPI spec. Chapter 7 should explain power resources and the device power methods in detail.
either I looked up the wrong spec or the documentation isn't really saying much there.
Well it explains those methods, _PSx, _PRx and _ON()/_OFF(). In case of PCIe device you also want to check PCIe spec. PCIe 5.0 section 5.8 "PCI Function Power State Transitions" has a picture about the supported power state transitions and there we can find that function must be in D3hot before it can be transitioned into D3cold so if the _OFF() for example blindly assumes that the device is in D0 when it is called, it is a bug in the BIOS.
BTW, where can I find acpidump of such system?
I am sure it's uploaded somewhere already. But it's not an issue of just one system. It's essentially hitting every single laptop with a skylake or kaby lake CPU + Nvidia GPU. I didn't see any system where it's actually working right now (and we are pestering nvidia about this issue for over a year already with no solution)
I've attached an acpidump from my system.
On Tue, Oct 22, 2019 at 11:16:14AM +0200, Karol Herbst wrote:
I think there is something I totally forgot about:
When there was never a driver bound to the GPU, and if runtime power management gets enabled on that device, runtime suspend/resume works as expected (I am not 100% sure on if that always works, but I will recheck that).
AFAIK, if there is no driver bound to the PCI device it is left to D0 regardless of the runtime PM state which could explain why it works in that case (it is never put into D3hot).
I looked at the acpidump you sent and there is one thing that may explain the differences between Windows and Linux. Not sure if you were aware of this already, though. The power resource PGOF() method has this:
If (((OSYS <= 0x07D9) || ((OSYS == 0x07DF) && (_REV == 0x05)))) { ... }
If I read it right, the later condition tries to detect Linux which fails nowadays but if you have acpi_rev_override in the command line (or the machine is listed in acpi_rev_dmi_table) this check passes and does some magic which is not clear to me. There is similar in PGON() side which is used to turn the device back on.
You can check what actually happens when _ON()/_OFF() is called by passing something like below to the kernel command line:
acpi.trace_debug_layer=0x80 acpi.trace_debug_level=0x10 acpi.trace_method_name=_SB.PCI0.PEG0.PG00._ON acpi.trace_state=method
(See also Documentation/firmware-guide/acpi/method-tracing.rst).
Trace goes to system dmesg.
On Tue, Oct 22, 2019 at 2:45 PM Mika Westerberg mika.westerberg@intel.com wrote:
On Tue, Oct 22, 2019 at 11:16:14AM +0200, Karol Herbst wrote:
I think there is something I totally forgot about:
When there was never a driver bound to the GPU, and if runtime power management gets enabled on that device, runtime suspend/resume works as expected (I am not 100% sure on if that always works, but I will recheck that).
AFAIK, if there is no driver bound to the PCI device it is left to D0 regardless of the runtime PM state which could explain why it works in that case (it is never put into D3hot).
I looked at the acpidump you sent and there is one thing that may explain the differences between Windows and Linux. Not sure if you were aware of this already, though. The power resource PGOF() method has this:
If (((OSYS <= 0x07D9) || ((OSYS == 0x07DF) && (_REV == 0x05)))) { ... }
I think this is the fallback to some older method of runtime suspending the device, and I think it will end up touching different registers on the bridge controller which do not show the broken behaviour.
You'll find references to following variables which all cause a link to be powered down: Q0L2 (newest), P0L2, P0LD (oldest, I think).
Maybe I remember incorrectly and have to read the code again... okay, the fallback path uses P0LD indeed. That's actually the only register of those being documented by Intel afaik.
If I read it right, the later condition tries to detect Linux which fails nowadays but if you have acpi_rev_override in the command line (or the machine is listed in acpi_rev_dmi_table) this check passes and does some magic which is not clear to me. There is similar in PGON() side which is used to turn the device back on.
You can check what actually happens when _ON()/_OFF() is called by passing something like below to the kernel command line:
acpi.trace_debug_layer=0x80 acpi.trace_debug_level=0x10 acpi.trace_method_name=_SB.PCI0.PEG0.PG00._ON acpi.trace_state=method
(See also Documentation/firmware-guide/acpi/method-tracing.rst).
Trace goes to system dmesg.
This sounds to be very helpful, I'll give it a try.
On Tue, Oct 22, 2019 at 02:51:53PM +0200, Karol Herbst wrote:
On Tue, Oct 22, 2019 at 2:45 PM Mika Westerberg mika.westerberg@intel.com wrote:
On Tue, Oct 22, 2019 at 11:16:14AM +0200, Karol Herbst wrote:
I think there is something I totally forgot about:
When there was never a driver bound to the GPU, and if runtime power management gets enabled on that device, runtime suspend/resume works as expected (I am not 100% sure on if that always works, but I will recheck that).
AFAIK, if there is no driver bound to the PCI device it is left to D0 regardless of the runtime PM state which could explain why it works in that case (it is never put into D3hot).
I looked at the acpidump you sent and there is one thing that may explain the differences between Windows and Linux. Not sure if you were aware of this already, though. The power resource PGOF() method has this:
If (((OSYS <= 0x07D9) || ((OSYS == 0x07DF) && (_REV == 0x05)))) { ... }
I think this is the fallback to some older method of runtime suspending the device, and I think it will end up touching different registers on the bridge controller which do not show the broken behaviour.
I think it actually tries to identify older Windows and then Linux (the _REV == 0x05 check comes from that). So at least some point Dell people have experiment this on Linux.
dri-devel@lists.freedesktop.org
-
Karol Herbst -
Mika Westerberg