On Mon, Jan 10, 2011 at 8:24 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
AMD chipset.
They don't seem to have have any errata's for this GFX business. But they do have a passthrough mode - hopefull that is not what you have passed in?
Since you mentioned that you were able to passthrough other PCI devices that means the IOMMU is working. This narrows it down.
Yes I could pass-through a network card to VM. I also tried passing through a old Sound Card to Windows 7 machine. It did not work as expected, Windows 7 does not have the old drivers.
Heh.
The big difference between other PCI devices and the graphics card is that the graphic card has its own VMM and "radeon_gart_bind" ends up programming the bus address (so virt_to_phys of the pages, and you could put in a printk there to see what it is) in the graphic VMM. This means that when the graphic card is told to fetch the writeback data it ends up using the address that was programmed in there to look. Here the IOMMU should step in and see "oh, this DMA address is for this guest, let me patch it up and actually look where this would fall in the guest space. Oh OK, let me use this value real value which correspond the guests real value."
It will take some time for me to understand every thing you have written :)
Ok. Ping me if you have some questions. Reading this might help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM
But the IOMMU has a couple of knobs. One of those is the passthrough code and the GFX mapping code. The first should be easy to spot (should tell you on bootup whether you got that enabled or not), while the other looks to be a bunch of workarounds when passing in GFX cards b/c they .. well, not sure what, but they look to not work correctly with the IOMMU.
By GFX Mapping, Do you mean the code that maps VGA frame-buffers to Guest OS?
No. It is the IOMMU kernel code that checks to see if this is a GFX card and if so does something fancy. But the workarounds for this appear to be exclusive to Intel chipset so you shouldn't hit any.
One more silly question on the frame-buffer mapping.
I was reading a paper on xen vga pass-though. It says "When applying PCI pass-through, certain memory areas of the physical machine are mapped to the VM. When the guest OS writes to one of those memory addresses, Xen will make sure it is actually written at the appropriate address. This implicates that no other VM is able to make use of that device. The frame-buffer is mapped to the machine's main memory at address 0xA0000 to 0xC0000. This memory range must be mapped to the VM's memory in order for the OS to address the video adapter."
<nods> > > My question is.... > In my case, the host machine is using Nvidia graphics card, so the > host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by > host. Now if the ATI card maps it's framebuffer to this same address > wouldn't it cause any problem.
Ah, but it does not. The first card that is called by the BIOS (so your motherboard BIOS) sets itself up to use that memory. The other card has to wait. Keep in mind that the A0000 to C0000 is the old style VGA framebuffer (80x25) or if you program the card properly it can do VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic video card passed in which sets this device up (and touching that area in the guest ends up in VNC window or SDL window).
Anyhow back to your host.. If you grep in your kernel log you should see something like: [drm] fb mappable at 0xE0142000
This is the framebuffer address.
If the the machine has only 1 graphics card those statement make sense, but I could not understand what will happen when there are two graphics cards, each connected to different monitor and assigned to different VM, where would each machines frame-buffer reside in the host memory.
Check the kernel log. Usually it is also the first BAR in the pci device.
Does Xen allow passing-though multiple graphics cards, each to different VM?
Yes (with some patches posted by the AMD and Intel folks).. (look for threads that have some GFX or Radeon or something like that in its title).
If you see 'identity map for device' where the device is your GFX then that looks to be the bug. It shouldn't set the identity mapping on it.
I guess the identity map is used only on the Intel chipset. Here are msgs on host machine when addition debugging parameter (amd_iommu_dump) was added.
prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 00000002 AMD 00000001) [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW 00000001 AMD 00000001) [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.012918] Performance Events: AMD PMU driver. [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 [ 0.300011] System has AMD C1E enabled [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 devid_to: 00:14.4 [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor (6 cpu cores) (version 2.20.00) [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 [ 12.350018] EDAC amd64: This node reports that Memory ECC is currently disabled, set F3x44[22] (0000:00:18.3). [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
What motherboard is this?
ASUS M4A89TD Pro/USB3, it is mentioned on the link http://wiki.xensource.com/xenwiki/VTdHowTo
prasad@prasad-kvm:~$ lspci | grep -i RV 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (rev 80) 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (Secondary) (rev 80)
So, how come you aren't passing in 02:00.1?
I tried passing through 02.00.1 as well. When I pass it to a Linux machine I do not see the device bind to a driver, I don't see the drm logs in the dmsg. So I thought I am not supposed to pass the 02:00:1.
Should I be passing 02.00.1 instead of 02.00.0?
Look for 'DMAR GFX' on Google. Might shed some more light.
No DMAR for AMD. I think I should have a look at DMAR GFX.
I am really thankful to you for your support.
Hopefully I haven't confused you any further :-)
Thanks and Regards, Prasad