https://bugzilla.kernel.org/show_bug.cgi?id=78221
Bug ID: 78221 Summary: 3.16 RC1: AMD R9 270 GPU locks up on some heavy 2D activity - GPU VM fault occurs. (possibly DMA copying issue strikes back?) Product: Drivers Version: 2.5 Kernel Version: 3.16-rc1 Hardware: x86-64 OS: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: Video(DRI - non Intel) Assignee: drivers_video-dri@kernel-bugs.osdl.org Reporter: t3st3r@mail.ru Regression: No
Configuration: AMD R9 270 with 3.16-rc1 kernel. Ubuntu 14.04 + oibaf PPA used as recent opensource graphics stack (MESA 10.3 git). XFCE used as DE, compositing is off.
To reproduce: Intermittent bug. R9 270 GPU can sometimes lock up on some heavy 2D-based loads. Known way to toggle bug is to install Battle for Wesnoth game from ubuntu repos and allow game to display some map with many units/heavy animations when running in windowed mode (BfW is SDL-based, 2D-only game, it does not makes GL calls, etc). In random intervals, usually in range of 30 minutes, GPU could lose stability and could lock up.
Special considerations: 1) GPU recovery often succeeds in this case. Yet its bad to have 20 seconds of black screen and sometimes it can fail to recover after multiple GPU crashes. 2) Kernels prior 3.15 had similar issue. Kernel 3.15 (release version but not -RCs) has been rock solid and never crashed GPU to the best of my knowledge, no matter what I attempted. Now 3.16-rc1 crashes again. 3) Taking 2) into account and such verioning I suspect it could have something to do with commit b5be1a839a33634393394e4782edaa37a4bc1a1e or somewhere around. Possibly its what reintroduced lockups in 3.16 again. Maybe underlying reasons of deadlocks were not fixed for R9 270? 4) GPU seems to be stable under 3D loads I've attempted - it does not crashes even after hours of quite demanding 3D loads like games, etc. Only some specific 2D loads can cause such issues.
Crash details are looking like this: ===CUT=== Jun 18 03:10:01 localhost kernel: [26125.102351] radeon 0000:01:00.0: ring 0 stalled for more than 10263msec Jun 18 03:10:01 localhost kernel: [26125.102362] radeon 0000:01:00.0: GPU lockup (waiting for 0x0000000000445274 last fence id 0x0000000000445273 on ring 0) Jun 18 03:10:01 localhost kernel: [26125.102370] radeon 0000:01:00.0: failed to get a new IB (-35) Jun 18 03:10:01 localhost kernel: [26125.671219] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0018 address=0x00000000803dae00 flags=0x0000] Jun 18 03:10:01 localhost kernel: [26125.671232] AMD-Vi: Event logged [ Jun 18 03:10:01 localhost kernel: [26125.671232] radeon 0000:01:00.0: Saved 23200 dwords of commands on ring 0. Jun 18 03:10:01 localhost kernel: [26125.671240] IO_PAGE_FAULT device=01:00.0 domain=0x0018 address=0x00000000803dae30 flags=0x0020] Jun 18 03:10:01 localhost kernel: [26125.671243] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0018 address=0x0000000080000100 flags=0x0020] Jun 18 03:10:01 localhost kernel: [26125.671248] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0018 address=0x00000000803dad00 flags=0x0000] Jun 18 03:10:01 localhost kernel: [26125.671361] radeon 0000:01:00.0: GPU softreset: 0x0000006C Jun 18 03:10:01 localhost kernel: [26125.671365] radeon 0000:01:00.0: GRBM_STATUS = 0xA0003028 Jun 18 03:10:01 localhost kernel: [26125.671367] radeon 0000:01:00.0: GRBM_STATUS_SE0 = 0x00000006 Jun 18 03:10:01 localhost kernel: [26125.671370] radeon 0000:01:00.0: GRBM_STATUS_SE1 = 0x00000006 Jun 18 03:10:01 localhost kernel: [26125.671372] radeon 0000:01:00.0: SRBM_STATUS = 0x200000C0 Jun 18 03:10:01 localhost kernel: [26125.671483] radeon 0000:01:00.0: SRBM_STATUS2 = 0x00000000 Jun 18 03:10:01 localhost kernel: [26125.671485] radeon 0000:01:00.0: R_008674_CP_STALLED_STAT1 = 0x00000000 Jun 18 03:10:01 localhost kernel: [26125.671487] radeon 0000:01:00.0: R_008678_CP_STALLED_STAT2 = 0x00010000 Jun 18 03:10:01 localhost kernel: [26125.671489] radeon 0000:01:00.0: R_00867C_CP_BUSY_STAT = 0x00000002 Jun 18 03:10:01 localhost kernel: [26125.671492] radeon 0000:01:00.0: R_008680_CP_STAT = 0x80010243 Jun 18 03:10:01 localhost kernel: [26125.671494] radeon 0000:01:00.0: R_00D034_DMA_STATUS_REG = 0x44483146 Jun 18 03:10:01 localhost kernel: [26125.671496] radeon 0000:01:00.0: R_00D834_DMA_STATUS_REG = 0x44C84246 Jun 18 03:10:01 localhost kernel: [26125.671499] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:01 localhost kernel: [26125.671501] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:02 localhost kernel: [26126.218193] radeon 0000:01:00.0: GRBM_SOFT_RESET=0x0000DDFF Jun 18 03:10:02 localhost kernel: [26126.218247] radeon 0000:01:00.0: SRBM_SOFT_RESET=0x00100140 Jun 18 03:10:02 localhost kernel: [26126.219404] radeon 0000:01:00.0: GRBM_STATUS = 0x00003028 Jun 18 03:10:02 localhost kernel: [26126.219407] radeon 0000:01:00.0: GRBM_STATUS_SE0 = 0x00000006 Jun 18 03:10:02 localhost kernel: [26126.219409] radeon 0000:01:00.0: GRBM_STATUS_SE1 = 0x00000006 Jun 18 03:10:02 localhost kernel: [26126.219411] radeon 0000:01:00.0: SRBM_STATUS = 0x200000C0 Jun 18 03:10:02 localhost kernel: [26126.219522] radeon 0000:01:00.0: SRBM_STATUS2 = 0x00000000 Jun 18 03:10:02 localhost kernel: [26126.219524] radeon 0000:01:00.0: R_008674_CP_STALLED_STAT1 = 0x00000000 Jun 18 03:10:02 localhost kernel: [26126.219526] radeon 0000:01:00.0: R_008678_CP_STALLED_STAT2 = 0x00000000 Jun 18 03:10:02 localhost kernel: [26126.219528] radeon 0000:01:00.0: R_00867C_CP_BUSY_STAT = 0x00000000 Jun 18 03:10:02 localhost kernel: [26126.219530] radeon 0000:01:00.0: R_008680_CP_STAT = 0x00000000 Jun 18 03:10:02 localhost kernel: [26126.219533] radeon 0000:01:00.0: R_00D034_DMA_STATUS_REG = 0x44C83D57 Jun 18 03:10:02 localhost kernel: [26126.219535] radeon 0000:01:00.0: R_00D834_DMA_STATUS_REG = 0x44C83D57 Jun 18 03:10:02 localhost kernel: [26126.219780] radeon 0000:01:00.0: GPU reset succeeded, trying to resume Jun 18 03:10:02 localhost kernel: [26126.246602] [drm] probing gen 2 caps for device 1002:5a16 = 31cd02/0 Jun 18 03:10:02 localhost kernel: [26126.246606] [drm] PCIE gen 2 link speeds already enabled Jun 18 03:10:02 localhost kernel: [26126.250269] [drm] PCIE GART of 1024M enabled (table at 0x0000000000276000). Jun 18 03:10:02 localhost kernel: [26126.250405] radeon 0000:01:00.0: WB enabled Jun 18 03:10:02 localhost kernel: [26126.250408] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00 and cpu addr 0xffff88041651fc00 Jun 18 03:10:02 localhost kernel: [26126.250411] radeon 0000:01:00.0: fence driver on ring 1 use gpu addr 0x0000000080000c04 and cpu addr 0xffff88041651fc04 Jun 18 03:10:02 localhost kernel: [26126.250413] radeon 0000:01:00.0: fence driver on ring 2 use gpu addr 0x0000000080000c08 and cpu addr 0xffff88041651fc08 Jun 18 03:10:02 localhost kernel: [26126.250415] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c and cpu addr 0xffff88041651fc0c Jun 18 03:10:02 localhost kernel: [26126.250417] radeon 0000:01:00.0: fence driver on ring 4 use gpu addr 0x0000000080000c10 and cpu addr 0xffff88041651fc10 Jun 18 03:10:02 localhost kernel: [26126.251393] radeon 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000000075a18 and cpu addr 0xffffc90011db5a18 Jun 18 03:10:02 localhost kernel: [26126.436426] [drm] ring test on 0 succeeded in 3 usecs Jun 18 03:10:02 localhost kernel: [26126.436432] [drm] ring test on 1 succeeded in 1 usecs Jun 18 03:10:02 localhost kernel: [26126.436437] [drm] ring test on 2 succeeded in 1 usecs Jun 18 03:10:02 localhost kernel: [26126.436500] [drm] ring test on 3 succeeded in 2 usecs Jun 18 03:10:02 localhost kernel: [26126.436510] [drm] ring test on 4 succeeded in 1 usecs Jun 18 03:10:02 localhost kernel: [26126.613588] [drm] ring test on 5 succeeded in 2 usecs Jun 18 03:10:02 localhost kernel: [26126.613595] [drm] UVD initialized successfully. Jun 18 03:10:04 localhost kernel: [26128.294193] SysRq : Emergency Sync Jun 18 03:10:04 localhost kernel: [26128.309427] Emergency Sync complete Jun 18 03:10:12 localhost kernel: [26136.610250] radeon 0000:01:00.0: ring 0 stalled for more than 10001msec Jun 18 03:10:12 localhost kernel: [26136.610260] radeon 0000:01:00.0: GPU lockup (waiting for 0x0000000000445396 last fence id 0x0000000000445273 on ring 0) Jun 18 03:10:12 localhost kernel: [26136.610266] [drm:r600_ib_test] *ERROR* radeon: fence wait failed (-35). Jun 18 03:10:12 localhost kernel: [26136.610273] [drm:radeon_ib_ring_tests] *ERROR* radeon: failed testing IB on GFX ring (-35). Jun 18 03:10:12 localhost kernel: [26136.610278] radeon 0000:01:00.0: ib ring test failed (-35). Jun 18 03:10:13 localhost kernel: [26137.083011] SysRq : Emergency Sync Jun 18 03:10:13 localhost kernel: [26137.098644] Emergency Sync complete Jun 18 03:10:13 localhost kernel: [26137.164779] radeon 0000:01:00.0: GPU softreset: 0x00000048 Jun 18 03:10:13 localhost kernel: [26137.164782] radeon 0000:01:00.0: GRBM_STATUS = 0xA0003028 Jun 18 03:10:13 localhost kernel: [26137.164785] radeon 0000:01:00.0: GRBM_STATUS_SE0 = 0x00000006 Jun 18 03:10:13 localhost kernel: [26137.164787] radeon 0000:01:00.0: GRBM_STATUS_SE1 = 0x00000006 Jun 18 03:10:13 localhost kernel: [26137.164789] radeon 0000:01:00.0: SRBM_STATUS = 0x200000C0 Jun 18 03:10:13 localhost kernel: [26137.164911] radeon 0000:01:00.0: SRBM_STATUS2 = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.164913] radeon 0000:01:00.0: R_008674_CP_STALLED_STAT1 = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.164918] radeon 0000:01:00.0: R_008678_CP_STALLED_STAT2 = 0x00010000 Jun 18 03:10:13 localhost kernel: [26137.164925] radeon 0000:01:00.0: R_00867C_CP_BUSY_STAT = 0x00000002 Jun 18 03:10:13 localhost kernel: [26137.164930] radeon 0000:01:00.0: R_008680_CP_STAT = 0x80010243 Jun 18 03:10:13 localhost kernel: [26137.164933] radeon 0000:01:00.0: R_00D034_DMA_STATUS_REG = 0x44C83D57 Jun 18 03:10:13 localhost kernel: [26137.164935] radeon 0000:01:00.0: R_00D834_DMA_STATUS_REG = 0x44C83D57 Jun 18 03:10:13 localhost kernel: [26137.164937] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.164940] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.709342] radeon 0000:01:00.0: GRBM_SOFT_RESET=0x0000DDFF Jun 18 03:10:13 localhost kernel: [26137.709397] radeon 0000:01:00.0: SRBM_SOFT_RESET=0x00000100 Jun 18 03:10:13 localhost kernel: [26137.710554] radeon 0000:01:00.0: GRBM_STATUS = 0x00003028 Jun 18 03:10:13 localhost kernel: [26137.710556] radeon 0000:01:00.0: GRBM_STATUS_SE0 = 0x00000006 Jun 18 03:10:13 localhost kernel: [26137.710558] radeon 0000:01:00.0: GRBM_STATUS_SE1 = 0x00000006 Jun 18 03:10:13 localhost kernel: [26137.710560] radeon 0000:01:00.0: SRBM_STATUS = 0x200000C0 Jun 18 03:10:13 localhost kernel: [26137.710681] radeon 0000:01:00.0: SRBM_STATUS2 = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.710683] radeon 0000:01:00.0: R_008674_CP_STALLED_STAT1 = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.710685] radeon 0000:01:00.0: R_008678_CP_STALLED_STAT2 = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.710687] radeon 0000:01:00.0: R_00867C_CP_BUSY_STAT = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.710689] radeon 0000:01:00.0: R_008680_CP_STAT = 0x00000000 Jun 18 03:10:13 localhost kernel: [26137.710695] radeon 0000:01:00.0: R_00D034_DMA_STATUS_REG = 0x44C83D57 Jun 18 03:10:13 localhost kernel: [26137.710706] radeon 0000:01:00.0: R_00D834_DMA_STATUS_REG = 0x44C83D57 Jun 18 03:10:13 localhost kernel: [26137.710960] radeon 0000:01:00.0: GPU reset succeeded, trying to resume Jun 18 03:10:13 localhost kernel: [26137.724290] [drm] probing gen 2 caps for device 1002:5a16 = 31cd02/0 Jun 18 03:10:13 localhost kernel: [26137.724294] [drm] PCIE gen 2 link speeds already enabled Jun 18 03:10:13 localhost kernel: [26137.728012] [drm] PCIE GART of 1024M enabled (table at 0x0000000000276000). Jun 18 03:10:13 localhost kernel: [26137.728147] radeon 0000:01:00.0: WB enabled Jun 18 03:10:13 localhost kernel: [26137.728150] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00 and cpu addr 0xffff88041651fc00 Jun 18 03:10:13 localhost kernel: [26137.728152] radeon 0000:01:00.0: fence driver on ring 1 use gpu addr 0x0000000080000c04 and cpu addr 0xffff88041651fc04 Jun 18 03:10:13 localhost kernel: [26137.728154] radeon 0000:01:00.0: fence driver on ring 2 use gpu addr 0x0000000080000c08 and cpu addr 0xffff88041651fc08 Jun 18 03:10:13 localhost kernel: [26137.728156] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c and cpu addr 0xffff88041651fc0c Jun 18 03:10:13 localhost kernel: [26137.728158] radeon 0000:01:00.0: fence driver on ring 4 use gpu addr 0x0000000080000c10 and cpu addr 0xffff88041651fc10 Jun 18 03:10:13 localhost kernel: [26137.729140] radeon 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000000075a18 and cpu addr 0xffffc90011db5a18 Jun 18 03:10:14 localhost kernel: [26137.914127] [drm] ring test on 0 succeeded in 3 usecs Jun 18 03:10:14 localhost kernel: [26137.914133] [drm] ring test on 1 succeeded in 1 usecs Jun 18 03:10:14 localhost kernel: [26137.914138] [drm] ring test on 2 succeeded in 1 usecs Jun 18 03:10:14 localhost kernel: [26137.914202] [drm] ring test on 3 succeeded in 2 usecs Jun 18 03:10:14 localhost kernel: [26137.914211] [drm] ring test on 4 succeeded in 1 usecs Jun 18 03:10:14 localhost kernel: [26138.091289] [drm] ring test on 5 succeeded in 2 usecs Jun 18 03:10:14 localhost kernel: [26138.091296] [drm] UVD initialized successfully. Jun 18 03:10:14 localhost kernel: [26138.091385] [drm] ib test on ring 0 succeeded in 0 usecs Jun 18 03:10:14 localhost kernel: [26138.091475] [drm] ib test on ring 1 succeeded in 0 usecs Jun 18 03:10:14 localhost kernel: [26138.091601] [drm] ib test on ring 2 succeeded in 0 usecs Jun 18 03:10:14 localhost kernel: [26138.091644] [drm] ib test on ring 3 succeeded in 0 usecs Jun 18 03:10:14 localhost kernel: [26138.091679] [drm] ib test on ring 4 succeeded in 0 usecs Jun 18 03:10:16 localhost kernel: [26140.121922] SysRq : Keyboard mode set to system default Jun 18 03:10:18 localhost kernel: [26141.841306] SysRq : Emergency Sync Jun 18 03:10:18 localhost kernel: [26141.866011] Emergency Sync complete Jun 18 03:10:18 localhost kernel: [26142.361117] SysRq : Emergency Sync Jun 18 03:10:18 localhost kernel: [26142.383541] Emergency Sync complete Jun 18 03:10:24 localhost kernel: [26148.238943] radeon 0000:01:00.0: ring 5 stalled for more than 10000msec Jun 18 03:10:24 localhost kernel: [26148.238952] radeon 0000:01:00.0: GPU lockup (waiting for 0x0000000000000004 last fence id 0x0000000000000002 on ring 5) Jun 18 03:10:24 localhost kernel: [26148.238958] [drm:uvd_v1_0_ib_test] *ERROR* radeon: fence wait failed (-35). Jun 18 03:10:24 localhost kernel: [26148.238966] [drm:radeon_ib_ring_tests] *ERROR* radeon: failed testing IB on ring 5 (-35). Jun 18 03:10:24 localhost kernel: [26148.238995] [drm:radeon_pm_resume_dpm] *ERROR* radeon: dpm resume failed Jun 18 03:10:24 localhost kernel: [26148.262523] radeon 0000:01:00.0: GPU fault detected: 146 0x05428804 Jun 18 03:10:24 localhost kernel: [26148.262534] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x0000A1AA Jun 18 03:10:24 localhost kernel: [26148.262539] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x02088004 Jun 18 03:10:24 localhost kernel: [26148.262544] VM fault (0x04, vmid 1) at page 41386, read from TC (136) Jun 18 03:10:24 localhost kernel: [26148.262552] radeon 0000:01:00.0: GPU fault detected: 146 0x06824804 Jun 18 03:10:24 localhost kernel: [26148.262556] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262559] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262563] VM fault (0x00, vmid 0) at page 0, read from unknown (0) Jun 18 03:10:24 localhost kernel: [26148.262570] radeon 0000:01:00.0: GPU fault detected: 146 0x05428404 Jun 18 03:10:24 localhost kernel: [26148.262573] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262577] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262580] VM fault (0x00, vmid 0) at page 0, read from unknown (0) Jun 18 03:10:24 localhost kernel: [26148.262586] radeon 0000:01:00.0: GPU fault detected: 146 0x06c24404 Jun 18 03:10:24 localhost kernel: [26148.262590] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262593] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262597] VM fault (0x00, vmid 0) at page 0, read from unknown (0) Jun 18 03:10:24 localhost kernel: [26148.262603] radeon 0000:01:00.0: GPU fault detected: 146 0x06228804 Jun 18 03:10:24 localhost kernel: [26148.262606] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262610] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262613] VM fault (0x00, vmid 0) at page 0, read from unknown (0) Jun 18 03:10:24 localhost kernel: [26148.262619] radeon 0000:01:00.0: GPU fault detected: 146 0x07024804 Jun 18 03:10:24 localhost kernel: [26148.262623] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262626] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262629] VM fault (0x00, vmid 0) at page 0, read from unknown (0) Jun 18 03:10:24 localhost kernel: [26148.262635] radeon 0000:01:00.0: GPU fault detected: 146 0x0802c804 Jun 18 03:10:24 localhost kernel: [26148.262639] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262642] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262646] VM fault (0x00, vmid 0) at page 0, read from unknown (0) Jun 18 03:10:24 localhost kernel: [26148.262652] radeon 0000:01:00.0: GPU fault detected: 146 0x0882c404 Jun 18 03:10:24 localhost kernel: [26148.262655] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 Jun 18 03:10:24 localhost kernel: [26148.262659] radeon 0000:01:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x00000000 ===CUT=== (message about VM fault repeats several thousands times, some messages omitted as they make full log about 3.5Mb).