https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #1 from coolo1mc@gmail.com --- Getting the exact same issue with my vega 56, system hangs when I log in to lightdm, fans spin up and just get louder and louder, shutting down doesn't work. Reverting to 4.15 didn't seem to fix the issue either, even though it was working fine before upgrading to 4.16

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #3 from Stefan stfn+freedesktop@hijnn.net --- Vega8 / Ryzen 2400G btw.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #5 from coolo1mc@gmail.com --- Is there any additional info we need to get? Anything we can test? My system is currently unusable until this is fixed and it has been 3 months since being reported and haven't heard anything but more reports

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #7 from coolo1mc@gmail.com --- I discovered that the cause of this for me was pywal, when I ran it my gpu hung, but if I didn't run it, it was otherwise fine. Another cause is cemu through wine with mesa_mild

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #9 from dxxf@volny.cz --- So one more update: My boot issue went away after updating: - kernel-firmware to 20180525 (as there were some amdgpu firmware updates in 20180518). - libLLVM6 from 6.0.0rc1 to 6.0.0 (and I strongly suspect this was the cause as I had VM pagefault issues before with libLLVM5 - but only in some OpenGL applications, not at boot).

https://bugs.freedesktop.org/show_bug.cgi?id=105251

Barry G barry@grussling.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |barry@grussling.com

--- Comment #11 from Barry G barry@grussling.com --- Created attachment 140645 --> https://bugs.freedesktop.org/attachment.cgi?id=140645&action=edit Complete DMESG from boot to lockup

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #13 from Barry G barry@grussling.com --- I upgraded my linux-firmware to 20180606.d114732-1 and it had no affect on the issue. Still locks running the script with the same dmesg.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #15 from dergottdergrunten@gmail.com --- I think my issue is related. I get black screen boots roughly every 2/3 times I boot up my computer. This last time, it booted up, kernel panic'd and I could still see the output so I took some pictures.

https://imgur.com/gallery/T69zIjX

Info:

$ uname -a Linux itx-dev.local 4.17.12-arch1-1-ARCH #1 SMP PREEMPT Fri Aug 3 07:16:41 UTC 2018 x86_64 GNU/Linux

$ cat /proc/cpuinfo processor : 5 vendor_id : AuthenticAMD cpu family : 23 model : 17 model name : AMD Ryzen 5 2400G with Radeon Vega Graphics stepping : 0

$ pacman -Qs amdgpu local/xf86-video-amdgpu 18.0.1-2 (xorg-drivers) X.org amdgpu video driver

$ pacman -Qs mesa local/glu 9.0.0-5 Mesa OpenGL Utility library local/lib32-libva-mesa-driver 18.1.5-1 VA-API implementation for gallium (32-bit) local/lib32-mesa 18.1.5-1 An open-source implementation of the OpenGL specification (32-bit) local/lib32-vulkan-radeon 18.1.5-1 Radeon's Vulkan mesa driver (32-bit) local/libva-mesa-driver 18.1.5-1 VA-API implementation for gallium local/mesa 18.1.5-1 An open-source implementation of the OpenGL specification local/mesa-vdpau 18.1.5-1 Mesa VDPAU drivers local/vulkan-radeon 18.1.5-1 Radeon's Vulkan mesa driver

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #17 from CheatCodesOfLife mesa@suchmail.net --- (In reply to Andrey Grodzovsky from comment #16)

Hi everyone, I've tried with latest kernel and latest VEGA10 firmware and wasn't able to reproduce this problem.

From the logs it seems all of you are running 4.17.x kernel or earlier - try latest 4.18 and latest firmware form here -

https://cgit.freedesktop.org/~agd5f/linux/log/?h=amd-staging-drm-next https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/

Hi,

I can reproduce this every time, on kernel 4.18 with mesa 18.3 and a Vega64.

Simply try to open Mario Kart 8 in Cemu with wine, and the system will crash with the exact same dmesg.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #19 from CheatCodesOfLife mesa@suchmail.net --- (In reply to Andrey Grodzovsky from comment #18)

(In reply to CheatCodesOfLife from comment #17)

...

(In reply to Andrey Grodzovsky from comment #16)

...

Hi everyone, I've tried with latest kernel and latest VEGA10 firmware and wasn't able to reproduce this problem.

From the logs it seems all of you are running 4.17.x kernel or earlier - try latest 4.18 and latest firmware form here -

https://cgit.freedesktop.org/~agd5f/linux/log/?h=amd-staging-drm-next https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/

Hi,

I can reproduce this every time, on kernel 4.18 with mesa 18.3 and a Vega64.

Simply try to open Mario Kart 8 in Cemu with wine, and the system will crash with the exact same dmesg.

I had mesa 18.2 so I updated to 18.3 - still nothing. Could you provide glxinfo dump ? What LLVM are you using ? I have 7.

I have had this problem with mesa 18.2 and LLVM7.

Currently on mesa 18.3 and LLVM8.

I also had this result with a Vega56, and I know people online who have the same problem. Nobody can open Mario Kart 8 in Cemu with wine if they have a Vega card.

I've attached my glxinfo > glxinfo.txt

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #21 from Andrey Grodzovsky andrey.grodzovsky@amd.com --- (In reply to CheatCodesOfLife from comment #17)

(In reply to Andrey Grodzovsky from comment #16)

...

Hi everyone, I've tried with latest kernel and latest VEGA10 firmware and wasn't able to reproduce this problem.

From the logs it seems all of you are running 4.17.x kernel or earlier - try latest 4.18 and latest firmware form here -

https://cgit.freedesktop.org/~agd5f/linux/log/?h=amd-staging-drm-next https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/

Hi,

I can reproduce this every time, on kernel 4.18 with mesa 18.3 and a Vega64.

Simply try to open Mario Kart 8 in Cemu with wine, and the system will crash with the exact same dmesg.

I had mesa 18.2 so I updated to 18.3 - still nothing. Could you provide glxinfo dump ? What LLVM are you using ? I have 7.(In reply to CheatCodesOfLife from comment #20)

Created attachment 141210 [details] glxinfo dump as requested

Thanks for the info, is there any other way you reproduce it without the wine platform ?

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #23 from Andrey Grodzovsky andrey.grodzovsky@amd.com --- (In reply to CheatCodesOfLife from comment #22)

You're welcome.

Not the exact same problem, no. I can get a hard-lock by trying to use amdvlk to play rpcs3, but it doesn't produce the same error and it's not as consistent (takes up to 15 minutes to crash)

Not sure if it's worth noting but I went back and tried every Cemu version back to 1.5 and a lot of wine versions going back to 2.8. It happens every time as soon as the game loads.

Let's try to get some debug info for the VMC page fault then -

Clone and build our open source register analyzer from here - https://cgit.freedesktop.org/amd/umr/ Install trace-cmd utility Load driver with cmd line parameter amdgpu.vm_fault_stop=2 from grub P.S Best to use latest kernel from here - https://cgit.freedesktop.org/~agd5f/linux/log/?h=amd-staging-drm-next

After desktop is loaded type

sudo trace-cmd start -e dma_fence -e gpu_scheduler -e amdgpu -v -e "amdgpu:amdgpu_mm_rreg" -e "amdgpu:amdgpu_mm_wreg" -e "amdgpu:amdgpu_iv" to enable kernel event tracing log

If possible to launch the game from shell then prepend the command with GALLIUM_DDEBUG=always to dump all the MESA commands into files in ~/ddebug_dumps/

Start the game. When the problem happens do the following -

as root cd /sys/kernel/debug/tracing && cat trace > event_dump

as normal user or root sudo umr -lb > umr_dump sudo umr -O verbose,use_colour -R gfx[.] >> umr_dump sudo umr -O halt_waves,use_colour -wa >> umr_dump dmesg > dmesg_dump

Upload a tar/zip of all those files + all the files from ~/ddebug_dumps/

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #25 from CheatCodesOfLife mesa@suchmail.net --- Created attachment 141269 --> https://bugs.freedesktop.org/attachment.cgi?id=141269&action=edit debug files

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #27 from Andrey Grodzovsky andrey.grodzovsky@amd.com ---

(In reply to CheatCodesOfLife from comment #25)

Created attachment 141269 [details] debug files

Since your kernel build doesn't have the latest AMD code I don't have ALL the trace logs so I can't be curtain but it does looks like the address reported by GPU fault is bad address, it's above any VA range seen in logs. I would need you to run Cemu.exe with GALLIUM_DDEBUG=always environment variable and upload logs from from ~/ddebug_dumps/

From googling it looks like WINE will pass down any ENVs picked from shell to

the apps it runs so should be easy - just run GALLIUM_DDEBUG=always 'WINE launch commands' from shell. Also provide all the other logs like last time.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #29 from Andrey Grodzovsky andrey.grodzovsky@amd.com --- Sorry , this link https://cgit.freedesktop.org/mesa/mesa/commit/?id=c5c6e0187fd5d535c304ca3fd6...

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #31 from CheatCodesOfLife mesa@suchmail.net --- (In reply to Andrey Grodzovsky from comment #29)

Sorry , this link https://cgit.freedesktop.org/mesa/mesa/commit/ ?id=c5c6e0187fd5d535c304ca3fd62de0f5e636c0c2

Yeah, I am using mesa. I've setup the amd-drm-next kernel kernel. This is the command I used to launch Cemu:

GALLIUM_HUD="fps" GALLIUM_DDEBUG=always wine64 Cemu.exe

(I switched on the GALLIUM_HUD as well so that I could verify that wine was receiving the ENVs, which it is.)

uname -a Linux nihonium2 4.18.0-rc1-5024f8dfe478 #1 SMP PREEMPT Sat Aug 25 05:10:49 AEST 2018 x86_64 GNU/Linux

The logs are attached (this time it took 3 tries to actually launch cemu due to an unrelated issue so the archive is 14mb)

https://bugs.freedesktop.org/show_bug.cgi?id=105251

CheatCodesOfLife mesa@suchmail.net changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |mesa@suchmail.net

--- Comment #33 from CheatCodesOfLife mesa@suchmail.net --- Created attachment 141277 --> https://bugs.freedesktop.org/attachment.cgi?id=141277&action=edit amd3.tar.gz dmesg, trace, ddebug logs

Sorry about that. I don't have that dmesg any more but I did the whole process again and attached it. This time I have confirmed all the files are in the archive.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #35 from Andrey Grodzovsky andrey.grodzovsky@amd.com --- (In reply to Andrey Grodzovsky from comment #34)

(In reply to CheatCodesOfLife from comment #33)

...

Created attachment 141277 [details] amd3.tar.gz dmesg, trace, ddebug logs

Sorry about that. I don't have that dmesg any more but I did the whole process again and attached it. This time I have confirmed all the files are in the archive.

But where is the trace file ? :) Any way I will try to check with what I have.

Any way, doesn't matter, could you please redo the capture and this time instead of GALLIUM_DDEBUG=always do GALLIUM_DDEBUG=1000 ? This way we can get one big dump file when VM_FAULT happens with all the info.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #37 from Marek Olšák maraeo@gmail.com --- Created attachment 141323 --> https://bugs.freedesktop.org/attachment.cgi?id=141323&action=edit patch - fix ddebug BO list reporting

Hi,

Can you please get a new ddebug report with the attached patch? Thanks.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #39 from CheatCodesOfLife mesa@suchmail.net --- Created attachment 141342 --> https://bugs.freedesktop.org/attachment.cgi?id=141342&action=edit logs after building the patched mesa

Hi,

Thanks for the logging patch.

I have applied patched that into the latest master branch from the mesa github page, built it and ran the game again with the new version.

The logs are attached.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #41 from CheatCodesOfLife mesa@suchmail.net --- (In reply to Andrey Grodzovsky from comment #40)

Marek Olšák, I still don't see the expected debug output. I looked for 'Buffer list' CheatCodesOfLife, can you verify please you are running the patched version of MESA ? We tested yesterday the new prints and they do show on VM_FAULTs.

This is most likely my fault as I'm new to most of this sort of thing. This is what I did, maybe you'll see where I went wrong:

- Patch This is the patched version of src/gallium/drivers/radeonsi/si_gfx_cs.c http://termbin.com/ypet

- Build I installed this build of mesa to a different prefix, rather than overriding my system install (I use this computer for work, everything).

System install: glxinfo |grep Mesa\ 18 OpenGL core profile version string: 4.5 (Core Profile) Mesa 18.3.0-devel (git-e345247092) OpenGL version string: 4.4 (Compatibility Profile) Mesa 18.3.0-devel (git-e345247092) OpenGL ES profile version string: OpenGL ES 3.2 Mesa 18.3.0-devel (git-e345247092)

New build:

OpenGL core profile version string: 4.5 (Core Profile) Mesa 18.3.0-devel (git-a72dbc461b) OpenGL version string: 4.4 (Compatibility Profile) Mesa 18.3.0-devel (git-a72dbc461b) OpenGL ES profile version string: OpenGL ES 3.2 Mesa 18.3.0-devel (git-a72dbc461b)

- Running: I then ran Cemu like this: LD_LIBRARY_PATH=/home/paul/mesa_log/lib/ GALLIUM_HUD="fps" GALLIUM_DDEBUG=1000 wine64 Cemu.exe

I know wine lets you do this because this is how we used to use a fork of mesa called 'mesa_mild' to get the required compatibility profile prior to mesa 18.2 which provided core compatibility 4.4

If installing to a prefix like that isn't adequate for this testing, let me know and I'll re-install the OS on an external drive, do a system-wide install of this patched mesa and try again.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #43 from CheatCodesOfLife mesa@suchmail.net --- (In reply to Marek Olšák from comment #42)

The file is incomplete, but I don't know why. Can you try it again? Maybe it'll be complete next time. It's better to use the REISUB key sequence to reboot the machine. (put it in google)

Hi Marek,

Yep, I'll do this tonight (including the REISUB to reboot).

In which file should I grep for 'Buffer list' to ensure it's worked before posting here?

And is fine that I've sandbox'd the install to /home/paul/mesa_log rather than a system-install?

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #45 from CheatCodesOfLife mesa@suchmail.net --- I've just tried it again a couple of times, and this time I'm sitting there tailing (-f) the ddebug file and nothing is being added to it after GFX_

tail -f ~/ddebug_dumps/Cemu.13f.exe_1990_00000000 HQD_IB_BUSY = 0 CP_CPF_STALLED_STAT1 <- RING_FETCHING_DATA = 1 INDR1_FETCHING_DATA = 1 INDR2_FETCHING_DATA = 0 STATE_FETCHING_DATA = 0 TCIU_WAITING_ON_FREE = 0 TCIU_WAITING_ON_TAGS = 0 UTCL2IU_WAITING_ON_FREE = 0 UTCL2IU_WAITING_ON_TAGS = 0 GFX_

It's been 10 minutes, that's the end of it. I don't think it's a reboot / flush logs to the filesystem issue since I'm still SSH'd in and following the log file. No "Buffer list" in the file either :(

I also tried building the latest master branch and applying the patch again, same thing.

On the monitor in the terminal where I ran wine, it says "Hang detection timeout is 1000ms." Not sure if that's relevant.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #49 from Andrey Grodzovsky andrey.grodzovsky@amd.com --- (In reply to CheatCodesOfLife from comment #48)

Created attachment 141425 [details] logs and trace

Hi,

I have applied the patch, ran through the process and attached the logs. The file doesn't appear to be truncated anymore.

Looks like still not Buffer list in the log...

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #51 from Michel Dänzer michel@daenzer.net --- One thing you could try is setting the synchronous attribute on the ddebug dump file before the hang:

chattr +S ~/ddebug_dump/*

Of course, you'll have to wait for the file to be created before doing this.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #53 from CheatCodesOfLife mesa@suchmail.net --- Created attachment 141524 --> https://bugs.freedesktop.org/attachment.cgi?id=141524&action=edit vega_crasher Logs + trace with patched mesa

Hi Michel,

Even with the chattr +S command, the buffer list is not present :(

I also ran the vega_crasher from zzyxpaw and am able to reproduce that on my system. I have attached the output and it includes the Buffer Lists.

For some reason, when Cemu + Mario Kart 8 crashes, the file gets truncated, but when the vega_crasher tool crashes, the files are not truncated. This leads me to believe I'm not doing anything wrong? lol.

Other than that, the symptoms are the same. Mouse moves for a little while then it stops.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #55 from zzyxpaw@gmail.com --- Any updates to this? I can still reproduce with the latest amd-drm-next kernel.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #57 from Antonio Chirizzi antonio.chirizzi@gmail.com --- Hello there, I am seeing the same problems on my Ryzen 2700U which is freezing as well, with the latest Ubuntu kernel: 4.19.0-041900-generic.

I am luckily able to ssh back in and try to shut it down, but it won't completely.

This is what I see on the kern.log, and in my case the whole graphics is frozen. I am running linuxmint cinnamon 19, with the latest Ubuntu kernel.

Oct 28 11:30:58 antonioRyzen kernel: [22639.758782] gmc_v9_0_process_interrupt: 10 callbacks suppressed Oct 28 11:30:58 antonioRyzen kernel: [22639.758789] amdgpu 0000:02:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:7 pasid:32769, for p rocess cinnamon pid 1459 thread amdgpu_cs:0 pid 1463 Oct 28 11:30:58 antonioRyzen kernel: [22639.758789] ) Oct 28 11:30:58 antonioRyzen kernel: [22639.758797] amdgpu 0000:02:00.0: at address 0x00000001010e1000 from 27 Oct 28 11:30:58 antonioRyzen kernel: [22639.758801] amdgpu 0000:02:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00701031 Oct 28 11:30:58 antonioRyzen kernel: [22639.758818] amdgpu 0000:02:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:7 pasid:32769, for process cinnamon pid 1459 thread amdgpu_cs:0 pid 1463 Oct 28 11:30:58 antonioRyzen kernel: [22639.758818] ) Oct 28 11:30:58 antonioRyzen kernel: [22639.758822] amdgpu 0000:02:00.0: at address 0x00000001010e0000 from 27 Oct 28 11:30:58 antonioRyzen kernel: [22639.758825] amdgpu 0000:02:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000 Oct 28 11:30:58 antonioRyzen kernel: [22639.758834] amdgpu 0000:02:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:7 pasid:32769, for process cinnamon pid 1459 thread amdgpu_cs:0 pid 1463 Oct 28 11:30:58 antonioRyzen kernel: [22639.758834] ) Oct 28 11:30:58 antonioRyzen kernel: [22639.758839] amdgpu 0000:02:00.0: at address 0x00000001010e0000 from 27 Oct 28 11:30:58 antonioRyzen kernel: [22639.758841] amdgpu 0000:02:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000 Oct 28 11:30:58 antonioRyzen kernel: [22639.758850] amdgpu 0000:02:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:7 pasid:32769, for process cinnamon pid 1459 thread amdgpu_cs:0 pid 1463 Oct 28 11:30:58 antonioRyzen kernel: [22639.758850] ) Oct 28 11:30:58 antonioRyzen kernel: [22639.758853] amdgpu 0000:02:00.0: at address 0x00000001010e0000 from 27 Oct 28 11:30:58 antonioRyzen kernel: [22639.758855] amdgpu 0000:02:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000 Oct 28 11:30:58 antonioRyzen kernel: [22639.758863] amdgpu 0000:02:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:7 pasid:32769, for process cinnamon pid 1459 thread amdgpu_cs:0 pid 1463 Oct 28 11:30:58 antonioRyzen kernel: [22639.758863] ) Oct 28 11:30:58 antonioRyzen kernel: [22639.758867] amdgpu 0000:02:00.0: at address 0x00000001010e0000 from 27 Oct 28 11:30:58 antonioRyzen kernel: [22639.758869] amdgpu 0000:02:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000 Oct 28 11:30:58 antonioRyzen kernel: [22639.758877] amdgpu 0000:02:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:7 pasid:32769, for process cinnamon pid 1459 thread amdgpu_cs:0 pid 1463 ... Oct 28 11:30:58 antonioRyzen kernel: [22639.758931] ) Oct 28 11:30:58 antonioRyzen kernel: [22639.758935] amdgpu 0000:02:00.0: at address 0x00000001010e1000 from 27 Oct 28 11:30:58 antonioRyzen kernel: [22639.758937] amdgpu 0000:02:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000 Oct 28 11:31:08 antonioRyzen kernel: [22649.811916] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=1445040, emitt ed seq=1445042 Oct 28 11:31:08 antonioRyzen kernel: [22649.811925] [drm] GPU recovery disabled. Oct 28 11:33:58 antonioRyzen kernel: [22820.391906] INFO: task kworker/u32:1:19683 blocked for more than 120 seconds. Oct 28 11:33:58 antonioRyzen kernel: [22820.391914] Not tainted 4.19.0-041900-generic #201810221809 Oct 28 11:33:58 antonioRyzen kernel: [22820.391917] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 28 11:33:58 antonioRyzen kernel: [22820.391920] kworker/u32:1 D 0 19683 2 0x80000000 Oct 28 11:33:58 antonioRyzen kernel: [22820.391943] Workqueue: events_unbound commit_work [drm_kms_helper] Oct 28 11:33:58 antonioRyzen kernel: [22820.391945] Call Trace: Oct 28 11:33:58 antonioRyzen kernel: [22820.391956] __schedule+0x29e/0x840 Oct 28 11:33:58 antonioRyzen kernel: [22820.391959] schedule+0x2c/0x80 Oct 28 11:33:58 antonioRyzen kernel: [22820.391962] schedule_timeout+0x258/0x360 Oct 28 11:33:58 antonioRyzen kernel: [22820.392050] ? optc1_get_crtc_scanoutpos+0x69/0xa0 [amdgpu] Oct 28 11:33:58 antonioRyzen kernel: [22820.392062] dma_fence_default_wait+0x20a/0x280 Oct 28 11:33:58 antonioRyzen kernel: [22820.392065] ? dma_fence_release+0xa0/0xa0 Oct 28 11:33:58 antonioRyzen kernel: [22820.392068] dma_fence_wait_timeout+0xe7/0x110 Oct 28 11:33:58 antonioRyzen kernel: [22820.392071] reservation_object_wait_timeout_rcu+0x201/0x340 Oct 28 11:33:58 antonioRyzen kernel: [22820.392140] ? amdgpu_get_vblank_counter_kms+0x111/0x160 [amdgpu] Oct 28 11:33:58 antonioRyzen kernel: [22820.392222] amdgpu_dm_do_flip+0x12c/0x370 [amdgpu] Oct 28 11:33:58 antonioRyzen kernel: [22820.392305] amdgpu_dm_atomic_commit_tail+0x7ac/0xea0 [amdgpu]

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #59 from Benjamin Hodgetts ben@xnode.org --- It's not just Cemu, it looks like it happens in Yuzu too. If you Google for "VMC page fault" then you'll find people running into that error in various other programs too.

Personally, this is what I got when running MK8 in Cemu: ============== [Sat Nov 17 22:29:43 2018] amdgpu 0000:03:00.0: [gfxhub] VMC page fault (src_id:0 ring:155 vmid:3 pasid:32769, for process Cemu.exe pid 963 thread Cemu.exe:cs0 pid 1035) at address 0x000080014189a000 from 27 VM_L2_PROTECTION_FAULT_STATUS:0x00301137 (repeated 7 times over the space of a few minutes) [Sat Nov 17 22:32:44 2018] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma1 timeout, signaled seq=1020, emitted seq=1023 ==============

And then this when trying to run Super Mario Odyssey in Yuzu: ============== [Sat Nov 17 22:47:26 2018] amdgpu 0000:03:00.0: [gfxhub] VMC page fault (src_id:0 ring:156 vmid:3 pasid:32769, for process yuzu pid 960 thread yuzu:cs0 pid 972 at address 0x000080bd27743000 from 27 VM_L2_PROTECTION_FAULT_STATUS:0x00301138 [Sat Nov 17 22:47:36 2018] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=24703, emitted seq=24704 ==============

I'll look into getting the info dump that was requested earlier in the thread to see if that helps, but the seemingly abandoned state of this bug is rather concerning.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #61 from Benjamin Hodgetts ben@xnode.org --- Created attachment 142797 --> https://bugs.freedesktop.org/attachment.cgi?id=142797&action=edit New Error Since 4.19.X

Ok, so some time between my last report and now (started happening since 4.19.something, I don't know which version specifically) this problem has changed in how it manifests itself. Previously you'd get the "[gfxhub] VMC page fault" messages. Now it manifests itself in considerably more serious looking errors (with none of the "VMC page faults" in sight). Log file attached.

Once something triggers this, the card will become basically unresponsive and anything that tries to use it will start throwing more of the errors seen in the attached log.

It's not random though. For example I can run Unigine valley/superposition or Elder Scrolls Online (via Wine+DXVK) for as long as I like, stress-testing or benchmarking and it'll be fine. But as soon a I try one of the problem programs, it'll basically "break" the graphics card until I hard reset.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #63 from e88z4 felix.adrianto@gmail.com --- Hi

I came across this bug report that might be related to my bug report #109022

https://bugs.freedesktop.org/show_bug.cgi?id=109022

I got the VMC Page Fault error as well while playing Yuzu with RX580. The bug can be reproduced easily at the same spot each time. GPU crashed but can be accessed through SSH.

If you think my bug is very similar with this bug, maybe I can help debugging.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #65 from zzyxpaw@gmail.com --- I've just tested the vega_crasher on the latest kernel from the linux-amd-staging-drm-next-git package (archlinux) and it didn't crash.

% uname -a Linux erebor 5.0.0-rc1-amd-staging-drm-next-git-b8cd95e15410+ #1 SMP PREEMPT Sat Feb 16 02:30:22 PST 2019 x86_64 GNU/Linux

That said, I'm still experiencing random crashes. I'll try and and get a debug dump next time it happens, but it looks a lot like what is described on this thread: https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-sour...

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #67 from udo udovdh@xs4all.nl --- I get multiple of these:

392.377183] amdgpu 0000:09:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:5 pasid:32772, for process firefox pid 4467 thread firefox:cs0 pid 4565 ) [ 392.377194] amdgpu 0000:09:00.0: at address 0x00000001013d4000 from 27 [ 392.377200] amdgpu 0000:09:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00501031

(...)

[ 402.621544] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=28019, emitted seq=28022 [ 402.621551] [drm] GPU recovery disabled.

Fedora 30 on Gigabyte X470 AORUS ULTRA GAMING w/ AMD Ryzen 5 2400G with Radeon Vega Graphics running git mesa and git xf86-video-amdgpu.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #69 from Hleb Valoshka 375gnu@gmail.com --- I have this fault with 2400G and mesa 18.3 & 19.1.1 with Linux 4.19 (other versions haven't been tested).

It seems that Vega is unable to handle tiny VBO correctly. I have an old application that uses a lot of immediate mode GL functions to create small billboards using GL_QUADS like the following one:

glTexCoord2f(0, 0); glVertex(v0 * Size); glTexCoord2f(1, 0); glVertex(v1 * Size); glTexCoord2f(1, 1); glVertex(v2 * Size); glTexCoord2f(0, 1); glVertex(v3 * Size);

Initially I have replaced this code with static GLfloat Vtx[] = { -1, -1, 0, 0, 0, 1, -1, 0, 1, 0, 1, 1, 0, 1, 1, -1, 1, 0, 0, 1 };

glBufferData(GL_ARRAY_BUFFER, sizeof(Vtx), Vtx, GL_STATIC_DRAW); glEnableClientState(GL_VERTEX_ARRAY); glEnableClientState(GL_TEXTURE_COORD_ARRAY); glVertexPointer(3, GL_FLOAT, 5*sizeof(GLfloat), 0); glTexCoordPointer(2, GL_FLOAT, 5*sizeof(GLfloat), 3*sizeof(GLfloat));

+ I use VAO if it's available.

As a variant I used independent arrays for position and texture coordinates. But with the same fault.

So as a result I added required data to another related VBO which contains 8192 vertices. Now I don't have this fault.

I know that OpenGL doesn't like herds of small VBOs, but the hardware failure is not an expected result if we use them.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #71 from deltasquared ds2.bugs.freedesktop@gmail.com --- I would like to pitch into this as it seems this particular problem has been plaguing me for some months now. Currently running kernel 5.2.1-arch1-1-ARCH and I will still occasionally get errors like this when running minetest (they seem to be subtly different from the others in this thread upon reading):

[ 5699.136659] amdgpu 0000:0b:00.0: [gfxhub] no-retry page fault (src_id:0 ring:155 vmid:5 pasid:32770, for process minetest pid 7127 thread minetest:cs0 pid 7133) [ 5699.136662] amdgpu 0000:0b:00.0: in page starting at address 0x000080014034d000 from 27 [ 5699.136664] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00501136 [ 5704.343299] [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out. [ 5709.259775] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=443165, emitted seq=443167 [ 5709.259860] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process minetest pid 7127 thread minetest:cs0 pid 7133 [ 5709.259862] [drm] GPU recovery disabled. *repeat last four lines endlessly...*

Relevant hardware is a ryzen 2200G (vega 8 GPU). The issue has survived swapping almost every component in my system so I think it is safe to rule out hardware brokenness in my case at least. Mercifully it seems the rest of the system survives this hence being able to capture the dmesg output, but with the gpu hard locked obviously the only recourse is to then reboot (after gathering some output for a while).

I haven't yet been able to obtain an API trace from minetest when it becomes difficult. Furthermore it doesn't do so reliably - I can often play for hours, but then the crash will strike and then the issue can sometimes persist across a few reboots if I press minetest to try and load a world again fast enough. Heck idk, is it a case of the precise 3D cloud pattern in the menu background at the time? Sounds like it would be useful for me to have apitrace running in the background whenever I run it on the off chance I can catch it in the act.

zzyxpaw's "vega crasher" in message #52 has reliably been able to cause GPU lock-up. Same sort of story: black window will pop up, nothing happens, and either lock-up occurs after a moment, or (interestingly) attempting to move the window in X11 will cause the lock-up immediately.

If there is any more data (such as attempting to get an apitrace) that would be useful I am willing to attempt to gather it, as this issue is the only blemish on an otherwise perfectly stable system.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #73 from Juan A. Suarez jasuarez@igalia.com --- (In reply to Pierre-Eric Pelloux-Prayer from comment #72)

This MR https://gitlab.freedesktop.org/mesa/mesa/merge_requests/1265 should improve the situation. It has been merged last week.

An incorrect program (like "vega_crasher") should hit an assert (if they're enabled in Mesa) or produce an incorrect rendering but shouldn't hang the GPU anymore.

It could be good if people could report here if this improved with this MR.

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #75 from deltasquared ds2.bugs.freedesktop@gmail.com --- After compiling mesa-git on commit 0661c357c60 from the AUR pkgbuild, I can now confirm my system seems to have become impervious to the above "vega_crasher" program.

Output from said program after resizing and moving vega_crasher's window a bit, in case it was important:

L CALLBACK: type = 0x8251, severity = 0x826b, message = LLVM diagnostic (remark): <unknown>:0:0: 9 instructions in function GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 16 VGPRS: 24 Code Size: 52 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = LLVM diagnostic (remark): <unknown>:0:0: 12 instructions in function GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 16 VGPRS: 8 Code Size: 92 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 16 VGPRS: 24 Code Size: 44 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 16 VGPRS: 8 Code Size: 80 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = LLVM diagnostic (remark): <unknown>:0:0: 2 instructions in function GL CALLBACK: type = 0x8251, severity = 0x826b, message = LLVM diagnostic (remark): <unknown>:0:0: 3 instructions in function GL CALLBACK: type = 0x8251, severity = 0x826b, message = LLVM diagnostic (remark): <unknown>:0:0: 4 instructions in function GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 16 VGPRS: 24 Code Size: 44 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 16 VGPRS: 8 Code Size: 136 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = LLVM diagnostic (remark): <unknown>:0:0: 16 instructions in function GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 24 VGPRS: 24 Code Size: 92 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0 GL CALLBACK: type = 0x8251, severity = 0x826b, message = Shader Stats: SGPRS: 24 VGPRS: 24 Code Size: 88 LDS: 0 Scratch: 0 Max Waves: 10 Spilled SGPRs: 0 Spilled VGPRs: 0 PrivMem VGPRs: 0

Minetest will take longer to test as the pkgbuild doesn't enable asserts, and also because of adformentioned $dayjob. I guess in that case I'd only know if I saw garbled output; it was never very consistent when it occured but would always be during the loading bar screen (but when it did happen some very colourful blocky corruption would result).

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #77 from deltasquared ds2.bugs.freedesktop@gmail.com --- Created attachment 144845 --> https://bugs.freedesktop.org/attachment.cgi?id=144845&action=edit vega_crasher after patch, black central output, on ryzen 2200G with vega 8 graphics

Screenshot time (1/2). It seems sometimes vega_crasher will render black - I haven't thoroughly looked over the code so I'm not sure if this is the adformentioned incorrect result (where an assert would have been hit).

https://bugs.freedesktop.org/show_bug.cgi?id=105251

--- Comment #79 from deltasquared ds2.bugs.freedesktop@gmail.com --- OK, have managed to get an unrelated crash from starting minetest now with the mentioned patch so at this point I think that case is unrelated. (Certainly seems to be more subtle, this MT crash has never been as reliable to trigger as some of the other things on this thread).

Will endeavour to file a bug separately. Any suggestions on information to kickstart such a related bug would be appreciated, else I will reach out to various channels on freenode first to get that ball rolling.