On Fre, 2011-01-07 at 02:42 +0000, Prasad Joshi wrote:
On Mon, Dec 27, 2010 at 3:30 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
On Tue, Dec 21, 2010 at 04:51:04AM +0000, Prasad Joshi wrote:
Hello All,
This is probably my first mail on this mailing list. I am working on a university project on Nvidia or any GPU Pass-through in KVM. I have a
You picking the hard ones first, eh? I would suggest you do Intel or ATI as they have actually posted patches for this.
Thanks a lot for your reply.
Yes indeed, I started with the ATI Radeon RV370 card. Passing the Nvidia card is the ultimate aim of the project.
question related to framebuffer.
I was reading few documents on XEN VGA pass-through, the document says the first step for pass-through is mapping the framebuffers in the VM at specific address, specifically speaking 0xA0000 till 0xC0000.
I would suggest you look at the code - there have been some patches posted by AMD engineers on xen-devel for this.
Yes I am looking at those patches as well, including some research papers on the same topic.
Here is what I have tried so far. I have 2 graphics cards connected to my machine
- Nvidia: This is being used by host machine.
- ATI: Want to pass-through this card to the VM.
I tried to pass-through the ATI card to VM. First, there was a problem with ROM BIOS. Then I passed a correct ROM BIOS to QEMU-KVM. Now when I start the VM with ATI card in pass-through mode.
I see some garbage output on the monitor attached to the ATI card. I also saw following messages in the system log.
[ 2.162294] [drm] Initialized drm 1.1.0 20060810 [ 2.459594] [drm] radeon defaulting to kernel modesetting. [ 2.459596] [drm] radeon kernel modesetting enabled. [ 2.766698] radeon 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, high) -> IRQ 10 [ 2.766734] radeon 0000:00:04.0: setting latency timer to 64 [ 2.783512] [drm] initializing kernel modesetting (RV380 0x1002:0x5B64). [ 2.792407] [drm] register mmio base: 0x40000000 [ 2.792408] [drm] register mmio size: 65536 [ 2.797177] [drm] Generation 2 PCI interface, using max accessible memory [ 2.797275] radeon 0000:00:04.0: VRAM: 128M 0x00000000F8000000 - 0x00000000FFFFFFFF (128M used) [ 2.797284] radeon 0000:00:04.0: GTT: 512M 0x00000000D8000000 - 0x00000000F7FFFFFF [ 2.798370] radeon 0000:00:04.0: irq 40 for MSI/MSI-X [ 2.870703] radeon 0000:00:04.0: radeon: using MSI. [ 3.151162] [drm] radeon: irq initialized. [ 3.151539] [drm] Detected VRAM RAM=128M, BAR=128M [ 3.151541] [drm] RAM width 128bits DDR [ 3.151610] [drm] radeon: 128M of VRAM memory ready [ 3.151611] [drm] radeon: 512M of GTT memory ready. [ 3.151627] [drm] GART: num cpu pages 131072, num gpu pages 131072 [ 3.152701] [drm] radeon: 1 quad pipes, 1 Z pipes initialized. [ 3.544943] [drm] PCIE GART of 512M enabled (table at 0xF8040000). [ 3.548479] radeon 0000:00:04.0: WB enabled [ 3.549020] [drm] Loading R300 Microcode [ 3.554278] [drm] radeon: ring at 0x00000000D8001000 [ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration [ 3.793709] [drm] radeon: cp finalized [ 3.834875] radeon 0000:00:04.0: ffff8800320b4c00 unpin not necessary [ 3.841759] [drm] Radeon Display Connectors [ 3.841766] [drm] Connector 0: [ 3.841771] [drm] VGA [ 3.841777] [drm] DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60 [ 3.841781] [drm] Encoders: [ 3.841785] [drm] CRT1: INTERNAL_DAC1 [ 3.841788] [drm] Connector 1: [ 3.841791] [drm] DVI-I [ 3.841794] [drm] HPD1 [ 3.841799] [drm] DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64 [ 3.841803] [drm] Encoders: [ 3.841806] [drm] CRT2: INTERNAL_DAC2 [ 3.841809] [drm] DFP1: INTERNAL_TMDS1 [ 6.158753] [drm] fb mappable at 0xF8040000 [ 6.158756] [drm] vram apper at 0xF8000000 [ 6.158757] [drm] size 5242880 [ 6.158758] [drm] fb depth is 24 [ 6.158759] [drm] pitch is 5120 [ 13.914196] fb0: radeondrmfb frame buffer device [ 13.914198] drm: registered panic notifier [ 13.914683] [drm] Initialized radeon 2.7.0 20080528 for 0000:00:04.0 on minor 0
The errors
[ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
Seem to be disabling GPU functionality. For now, I am interested in Graphics functionality of the ATI card and not GPGPU functionality.
I will appreciate if you can help me with some information on solving this problem. Let me know if I should be asking this question on another forum. Please keep me in CC.
Looks like dri-devel material, added to CC.
As a first guess, maybe the system memory pages accessed by the GPU GART aren't the same ones intended for this by the VM.
The errors
[ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
Seem to be disabling GPU functionality. For now, I am interested in Graphics functionality of the ATI card and not GPGPU functionality.
I will appreciate if you can help me with some information on solving this problem. Let me know if I should be asking this question on another forum. Please keep me in CC.
Looks like dri-devel material, added to CC.
As a first guess, maybe the system memory pages accessed by the GPU GART aren't the same ones intended for this by the VM.
I actually have patches for that (sorry for the late reply).
Try: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/ttm.pci-api.v3
But I am not sure if they would make any difference here. You said you are using KVM-QEMU - you must also be using an IOMMU? Does the IOMMU give you any errors?
2011/1/7 Konrad Rzeszutek Wilk konrad.wilk@oracle.com:
The errors
[ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
Seem to be disabling GPU functionality. For now, I am interested in Graphics functionality of the ATI card and not GPGPU functionality.
I will appreciate if you can help me with some information on solving this problem. Let me know if I should be asking this question on another forum. Please keep me in CC.
Looks like dri-devel material, added to CC.
As a first guess, maybe the system memory pages accessed by the GPU GART aren't the same ones intended for this by the VM.
I actually have patches for that (sorry for the late reply).
Try: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/ttm.pci-api.v3
But I am not sure if they would make any difference here. You said you are using KVM-QEMU - you must also be using an IOMMU? Does the IOMMU give you any errors?
Yes I want to pass-through the ATI Radeon card to VM in KVM. I have enabled the IOMMU support and it works perfectly alright. I tried passing through a network card it worked.
Should I try the patch you suggested?
On Fri, Jan 07, 2011 at 05:47:59PM +0000, Prasad Joshi wrote:
2011/1/7 Konrad Rzeszutek Wilk konrad.wilk@oracle.com:
The errors
[ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
Seem to be disabling GPU functionality. For now, I am interested in Graphics functionality of the ATI card and not GPGPU functionality.
I will appreciate if you can help me with some information on solving this problem. Let me know if I should be asking this question on another forum. Please keep me in CC.
Looks like dri-devel material, added to CC.
As a first guess, maybe the system memory pages accessed by the GPU GART aren't the same ones intended for this by the VM.
I actually have patches for that (sorry for the late reply).
Try: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/ttm.pci-api.v3
But I am not sure if they would make any difference here. You said you are using KVM-QEMU - you must also be using an IOMMU? Does the IOMMU give you any errors?
Yes I want to pass-through the ATI Radeon card to VM in KVM. I have enabled the IOMMU support and it works perfectly alright. I tried passing through a network card it worked.
Should I try the patch you suggested?
Go for it. I am still at lost why you would see those errors unless.. well, what is the DMA code that gets turned on when you run your guest? What do you see for 'PCI-DMA' ? Is it bounce buffer or swiotlb? How much memory do you pass to the guest? less than 4GB?
On Fri, Jan 7, 2011 at 7:24 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
On Fri, Jan 07, 2011 at 05:47:59PM +0000, Prasad Joshi wrote:
2011/1/7 Konrad Rzeszutek Wilk konrad.wilk@oracle.com:
The errors
[ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
Seem to be disabling GPU functionality. For now, I am interested in Graphics functionality of the ATI card and not GPGPU functionality.
I will appreciate if you can help me with some information on solving this problem. Let me know if I should be asking this question on another forum. Please keep me in CC.
Looks like dri-devel material, added to CC.
As a first guess, maybe the system memory pages accessed by the GPU GART aren't the same ones intended for this by the VM.
I actually have patches for that (sorry for the late reply).
Try: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/ttm.pci-api.v3
I could access the git repository, but was not able to get those patches form the repository. Frankly speaking, I don't know how to get those patches.
But I am not sure if they would make any difference here. You said you are using KVM-QEMU - you must also be using an IOMMU? Does the IOMMU give you any errors?
Yes I want to pass-through the ATI Radeon card to VM in KVM. I have enabled the IOMMU support and it works perfectly alright. I tried passing through a network card it worked.
Should I try the patch you suggested?
Go for it. I am still at lost why you would see those errors unless.. well, what is the DMA code that gets turned on when you run your guest? What do you see for 'PCI-DMA' ? Is it bounce buffer or swiotlb? How much memory do you pass to the guest? less than 4GB?
I am trying these things on Uni server, I will let you know the answers by tomorrow.
Thanks and Regards, Prasad
On Fri, Jan 7, 2011 at 7:24 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
On Fri, Jan 07, 2011 at 05:47:59PM +0000, Prasad Joshi wrote:
2011/1/7 Konrad Rzeszutek Wilk konrad.wilk@oracle.com:
The errors
[ 3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) [ 3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 3.782542] radeon 0000:00:04.0: failled initializing CP (-22). [ 3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
Seem to be disabling GPU functionality. For now, I am interested in Graphics functionality of the ATI card and not GPGPU functionality.
I will appreciate if you can help me with some information on solving this problem. Let me know if I should be asking this question on another forum. Please keep me in CC.
Looks like dri-devel material, added to CC.
As a first guess, maybe the system memory pages accessed by the GPU GART aren't the same ones intended for this by the VM.
I actually have patches for that (sorry for the late reply).
Try: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/ttm.pci-api.v3
But I am not sure if they would make any difference here. You said you are using KVM-QEMU - you must also be using an IOMMU? Does the IOMMU give you any errors?
Yes I want to pass-through the ATI Radeon card to VM in KVM. I have enabled the IOMMU support and it works perfectly alright. I tried passing through a network card it worked.
Should I try the patch you suggested?
Go for it. I am still at lost why you would see those errors unless..
Tried your patches on Guest machine, it did not help. I am attaching the messages from guest machine.
well, what is the DMA code that gets turned on when you run your guest?
Looking into the qemu-kvm code to find more information about DMA.
What do you see for 'PCI-DMA' ?
Is it bounce buffer or swiotlb?
It is the default configuration. I did not pass swiotlb parameter on kernel boot line.
The host kernel is booted with iommu=1.
How much memory do you pass to the guest? less than 4GB?
root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=02:00.0
I am passing 1GB memory to the guest machine.
I have 2 graphics cards (Nvidia and ATI) attached to my system. The Nvidia graphics card is used by host machine and I am trying to assign the ATI graphics card to the Guest machine.
When I unbind the ATI graphics card from the host machine I see these messages on the host machine Jan 10 11:59:15 prasad-kvm kernel: [ 3396.912767] pci-stub: invalid id string "" Jan 10 11:59:59 prasad-kvm kernel: [ 3441.180601] pci-stub 0000:02:00.1: claimed by stub Jan 10 12:00:08 prasad-kvm kernel: [ 3449.901870] [drm] radeon: finishing device. Jan 10 12:00:08 prasad-kvm kernel: [ 3449.902161] [drm] radeon: cp finalized Jan 10 12:00:08 prasad-kvm kernel: [ 3449.903435] radeon 0000:02:00.0: ffff88022ae3d200 unpin not necessary Jan 10 12:00:08 prasad-kvm kernel: [ 3449.903490] [drm] radeon: ttm finalized Jan 10 12:00:08 prasad-kvm kernel: [ 3449.903551] vga_switcheroo: disabled
Thanks a lot for helping me out.
Thanks and Regards, Prasad
Tried your patches on Guest machine, it did not help. I am attaching the messages from guest machine.
Ok, then the issue is not with the guest but the IOMMU.
well, what is the DMA code that gets turned on when you run your guest?
Looking into the qemu-kvm code to find more information about DMA.
What do you see for 'PCI-DMA' ?
Is it bounce buffer or swiotlb?
It is the default configuration. I did not pass swiotlb parameter on kernel boot line.
That is the nop one. So using virt_to_phys to get DMA address.
The host kernel is booted with iommu=1.
Is this a AMD or Intel chipset?
How much memory do you pass to the guest? less than 4GB?
root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=02:00.0
I am passing 1GB memory to the guest machine.
Since you mentioned that you were able to passthrough other PCI devices that means the IOMMU is working. This narrows it down.
The big difference between other PCI devices and the graphics card is that the graphic card has its own VMM and "radeon_gart_bind" ends up programming the bus address (so virt_to_phys of the pages, and you could put in a printk there to see what it is) in the graphic VMM. This means that when the graphic card is told to fetch the writeback data it ends up using the address that was programmed in there to look. Here the IOMMU should step in and see "oh, this DMA address is for this guest, let me patch it up and actually look where this would fall in the guest space. Oh OK, let me use this value real value which correspond the guests real value."
But the IOMMU has a couple of knobs. One of those is the passthrough code and the GFX mapping code. The first should be easy to spot (should tell you on bootup whether you got that enabled or not), while the other looks to be a bunch of workarounds when passing in GFX cards b/c they .. well, not sure what, but they look to not work correctly with the IOMMU.
If you see 'identity map for device' where the device is your GFX then that looks to be the bug. It shouldn't set the identity mapping on it.
Look for 'DMAR GFX' on Google. Might shed some more light.
Hello Konard,
Thanks a lot for your efforts and time.
On Mon, Jan 10, 2011 at 6:17 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Tried your patches on Guest machine, it did not help. I am attaching the messages from guest machine.
Ok, then the issue is not with the guest but the IOMMU.
well, what is the DMA code that gets turned on when you run your guest?
Looking into the qemu-kvm code to find more information about DMA.
What do you see for 'PCI-DMA' ?
Is it bounce buffer or swiotlb?
It is the default configuration. I did not pass swiotlb parameter on kernel boot line.
That is the nop one. So using virt_to_phys to get DMA address.
The host kernel is booted with iommu=1.
Is this a AMD or Intel chipset?
AMD chipset.
How much memory do you pass to the guest? less than 4GB?
root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=02:00.0
I am passing 1GB memory to the guest machine.
Since you mentioned that you were able to passthrough other PCI devices that means the IOMMU is working. This narrows it down.
Yes I could pass-through a network card to VM. I also tried passing through a old Sound Card to Windows 7 machine. It did not work as expected, Windows 7 does not have the old drivers.
The big difference between other PCI devices and the graphics card is that the graphic card has its own VMM and "radeon_gart_bind" ends up programming the bus address (so virt_to_phys of the pages, and you could put in a printk there to see what it is) in the graphic VMM. This means that when the graphic card is told to fetch the writeback data it ends up using the address that was programmed in there to look. Here the IOMMU should step in and see "oh, this DMA address is for this guest, let me patch it up and actually look where this would fall in the guest space. Oh OK, let me use this value real value which correspond the guests real value."
It will take some time for me to understand every thing you have written :)
But the IOMMU has a couple of knobs. One of those is the passthrough code and the GFX mapping code. The first should be easy to spot (should tell you on bootup whether you got that enabled or not), while the other looks to be a bunch of workarounds when passing in GFX cards b/c they .. well, not sure what, but they look to not work correctly with the IOMMU.
By GFX Mapping, Do you mean the code that maps VGA frame-buffers to Guest OS?
One more silly question on the frame-buffer mapping.
I was reading a paper on xen vga pass-though. It says "When applying PCI pass-through, certain memory areas of the physical machine are mapped to the VM. When the guest OS writes to one of those memory addresses, Xen will make sure it is actually written at the appropriate address. This implicates that no other VM is able to make use of that device. The frame-buffer is mapped to the machine's main memory at address 0xA0000 to 0xC0000. This memory range must be mapped to the VM's memory in order for the OS to address the video adapter."
My question is.... In my case, the host machine is using Nvidia graphics card, so the host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by host. Now if the ATI card maps it's framebuffer to this same address wouldn't it cause any problem.
If the the machine has only 1 graphics card those statement make sense, but I could not understand what will happen when there are two graphics cards, each connected to different monitor and assigned to different VM, where would each machines frame-buffer reside in the host memory.
Does Xen allow passing-though multiple graphics cards, each to different VM?
I am really sorry if those questions does not make any sense.
If you see 'identity map for device' where the device is your GFX then that looks to be the bug. It shouldn't set the identity mapping on it.
I guess the identity map is used only on the Intel chipset. Here are msgs on host machine when addition debugging parameter (amd_iommu_dump) was added.
prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 00000002 AMD 00000001) [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW 00000001 AMD 00000001) [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.012918] Performance Events: AMD PMU driver. [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 [ 0.300011] System has AMD C1E enabled [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 devid_to: 00:14.4 [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor (6 cpu cores) (version 2.20.00) [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 [ 12.350018] EDAC amd64: This node reports that Memory ECC is currently disabled, set F3x44[22] (0000:00:18.3). [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
prasad@prasad-kvm:~$ lspci | grep -i RV 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (rev 80) 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (Secondary) (rev 80)
Look for 'DMAR GFX' on Google. Might shed some more light.
No DMAR for AMD. I think I should have a look at DMAR GFX.
I am really thankful to you for your support.
Thanks and Regards, Prasad
AMD chipset.
They don't seem to have have any errata's for this GFX business. But they do have a passthrough mode - hopefull that is not what you have passed in?
Since you mentioned that you were able to passthrough other PCI devices that means the IOMMU is working. This narrows it down.
Yes I could pass-through a network card to VM. I also tried passing through a old Sound Card to Windows 7 machine. It did not work as expected, Windows 7 does not have the old drivers.
Heh.
The big difference between other PCI devices and the graphics card is that the graphic card has its own VMM and "radeon_gart_bind" ends up programming the bus address (so virt_to_phys of the pages, and you could put in a printk there to see what it is) in the graphic VMM. This means that when the graphic card is told to fetch the writeback data it ends up using the address that was programmed in there to look. Here the IOMMU should step in and see "oh, this DMA address is for this guest, let me patch it up and actually look where this would fall in the guest space. Oh OK, let me use this value real value which correspond the guests real value."
It will take some time for me to understand every thing you have written :)
Ok. Ping me if you have some questions. Reading this might help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM
But the IOMMU has a couple of knobs. One of those is the passthrough code and the GFX mapping code. The first should be easy to spot (should tell you on bootup whether you got that enabled or not), while the other looks to be a bunch of workarounds when passing in GFX cards b/c they .. well, not sure what, but they look to not work correctly with the IOMMU.
By GFX Mapping, Do you mean the code that maps VGA frame-buffers to Guest OS?
No. It is the IOMMU kernel code that checks to see if this is a GFX card and if so does something fancy. But the workarounds for this appear to be exclusive to Intel chipset so you shouldn't hit any.
One more silly question on the frame-buffer mapping.
I was reading a paper on xen vga pass-though. It says "When applying PCI pass-through, certain memory areas of the physical machine are mapped to the VM. When the guest OS writes to one of those memory addresses, Xen will make sure it is actually written at the appropriate address. This implicates that no other VM is able to make use of that device. The frame-buffer is mapped to the machine's main memory at address 0xA0000 to 0xC0000. This memory range must be mapped to the VM's memory in order for the OS to address the video adapter."
<nods>
My question is.... In my case, the host machine is using Nvidia graphics card, so the host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by host. Now if the ATI card maps it's framebuffer to this same address wouldn't it cause any problem.
Ah, but it does not. The first card that is called by the BIOS (so your motherboard BIOS) sets itself up to use that memory. The other card has to wait. Keep in mind that the A0000 to C0000 is the old style VGA framebuffer (80x25) or if you program the card properly it can do VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic video card passed in which sets this device up (and touching that area in the guest ends up in VNC window or SDL window).
Anyhow back to your host.. If you grep in your kernel log you should see something like: [drm] fb mappable at 0xE0142000
This is the framebuffer address.
If the the machine has only 1 graphics card those statement make sense, but I could not understand what will happen when there are two graphics cards, each connected to different monitor and assigned to different VM, where would each machines frame-buffer reside in the host memory.
Check the kernel log. Usually it is also the first BAR in the pci device.
Does Xen allow passing-though multiple graphics cards, each to different VM?
Yes (with some patches posted by the AMD and Intel folks).. (look for threads that have some GFX or Radeon or something like that in its title).
If you see 'identity map for device' where the device is your GFX then that looks to be the bug. It shouldn't set the identity mapping on it.
I guess the identity map is used only on the Intel chipset. Here are msgs on host machine when addition debugging parameter (amd_iommu_dump) was added.
prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 00000002 AMD 00000001) [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW 00000001 AMD 00000001) [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.012918] Performance Events: AMD PMU driver. [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 [ 0.300011] System has AMD C1E enabled [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 devid_to: 00:14.4 [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor (6 cpu cores) (version 2.20.00) [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 [ 12.350018] EDAC amd64: This node reports that Memory ECC is currently disabled, set F3x44[22] (0000:00:18.3). [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
What motherboard is this?
prasad@prasad-kvm:~$ lspci | grep -i RV 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (rev 80) 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (Secondary) (rev 80)
So, how come you aren't passing in 02:00.1?
Look for 'DMAR GFX' on Google. Might shed some more light.
No DMAR for AMD. I think I should have a look at DMAR GFX.
I am really thankful to you for your support.
Hopefully I haven't confused you any further :-)
Thanks and Regards, Prasad
On Mon, Jan 10, 2011 at 8:24 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
AMD chipset.
They don't seem to have have any errata's for this GFX business. But they do have a passthrough mode - hopefull that is not what you have passed in?
Since you mentioned that you were able to passthrough other PCI devices that means the IOMMU is working. This narrows it down.
Yes I could pass-through a network card to VM. I also tried passing through a old Sound Card to Windows 7 machine. It did not work as expected, Windows 7 does not have the old drivers.
Heh.
The big difference between other PCI devices and the graphics card is that the graphic card has its own VMM and "radeon_gart_bind" ends up programming the bus address (so virt_to_phys of the pages, and you could put in a printk there to see what it is) in the graphic VMM. This means that when the graphic card is told to fetch the writeback data it ends up using the address that was programmed in there to look. Here the IOMMU should step in and see "oh, this DMA address is for this guest, let me patch it up and actually look where this would fall in the guest space. Oh OK, let me use this value real value which correspond the guests real value."
It will take some time for me to understand every thing you have written :)
Ok. Ping me if you have some questions. Reading this might help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM
But the IOMMU has a couple of knobs. One of those is the passthrough code and the GFX mapping code. The first should be easy to spot (should tell you on bootup whether you got that enabled or not), while the other looks to be a bunch of workarounds when passing in GFX cards b/c they .. well, not sure what, but they look to not work correctly with the IOMMU.
By GFX Mapping, Do you mean the code that maps VGA frame-buffers to Guest OS?
No. It is the IOMMU kernel code that checks to see if this is a GFX card and if so does something fancy. But the workarounds for this appear to be exclusive to Intel chipset so you shouldn't hit any.
One more silly question on the frame-buffer mapping.
I was reading a paper on xen vga pass-though. It says "When applying PCI pass-through, certain memory areas of the physical machine are mapped to the VM. When the guest OS writes to one of those memory addresses, Xen will make sure it is actually written at the appropriate address. This implicates that no other VM is able to make use of that device. The frame-buffer is mapped to the machine's main memory at address 0xA0000 to 0xC0000. This memory range must be mapped to the VM's memory in order for the OS to address the video adapter."
<nods> > > My question is.... > In my case, the host machine is using Nvidia graphics card, so the > host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by > host. Now if the ATI card maps it's framebuffer to this same address > wouldn't it cause any problem.
Ah, but it does not. The first card that is called by the BIOS (so your motherboard BIOS) sets itself up to use that memory. The other card has to wait. Keep in mind that the A0000 to C0000 is the old style VGA framebuffer (80x25) or if you program the card properly it can do VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic video card passed in which sets this device up (and touching that area in the guest ends up in VNC window or SDL window).
Anyhow back to your host.. If you grep in your kernel log you should see something like: [drm] fb mappable at 0xE0142000
This is the framebuffer address.
If the the machine has only 1 graphics card those statement make sense, but I could not understand what will happen when there are two graphics cards, each connected to different monitor and assigned to different VM, where would each machines frame-buffer reside in the host memory.
Check the kernel log. Usually it is also the first BAR in the pci device.
Does Xen allow passing-though multiple graphics cards, each to different VM?
Yes (with some patches posted by the AMD and Intel folks).. (look for threads that have some GFX or Radeon or something like that in its title).
If you see 'identity map for device' where the device is your GFX then that looks to be the bug. It shouldn't set the identity mapping on it.
I guess the identity map is used only on the Intel chipset. Here are msgs on host machine when addition debugging parameter (amd_iommu_dump) was added.
prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 00000002 AMD 00000001) [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW 00000001 AMD 00000001) [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.012918] Performance Events: AMD PMU driver. [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 [ 0.300011] System has AMD C1E enabled [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 devid_to: 00:14.4 [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor (6 cpu cores) (version 2.20.00) [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 [ 12.350018] EDAC amd64: This node reports that Memory ECC is currently disabled, set F3x44[22] (0000:00:18.3). [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
What motherboard is this?
ASUS M4A89TD Pro/USB3, it is mentioned on the link http://wiki.xensource.com/xenwiki/VTdHowTo
prasad@prasad-kvm:~$ lspci | grep -i RV 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (rev 80) 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (Secondary) (rev 80)
So, how come you aren't passing in 02:00.1?
I tried passing through 02.00.1 as well. When I pass it to a Linux machine I do not see the device bind to a driver, I don't see the drm logs in the dmsg. So I thought I am not supposed to pass the 02:00:1.
Should I be passing 02.00.1 instead of 02.00.0?
Look for 'DMAR GFX' on Google. Might shed some more light.
No DMAR for AMD. I think I should have a look at DMAR GFX.
I am really thankful to you for your support.
Hopefully I haven't confused you any further :-)
Thanks and Regards, Prasad
On Tue, Jan 11, 2011 at 11:58 AM, Prasad Joshi prasadjoshi124@gmail.com wrote:
On Mon, Jan 10, 2011 at 8:24 PM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
AMD chipset.
They don't seem to have have any errata's for this GFX business. But they do have a passthrough mode - hopefull that is not what you have passed in?
Since you mentioned that you were able to passthrough other PCI devices that means the IOMMU is working. This narrows it down.
Yes I could pass-through a network card to VM. I also tried passing through a old Sound Card to Windows 7 machine. It did not work as expected, Windows 7 does not have the old drivers.
Heh.
The big difference between other PCI devices and the graphics card is that the graphic card has its own VMM and "radeon_gart_bind" ends up programming the bus address (so virt_to_phys of the pages, and you could put in a printk there to see what it is) in the graphic VMM. This means that when the graphic card is told to fetch the writeback data it ends up using the address that was programmed in there to look. Here the IOMMU should step in and see "oh, this DMA address is for this guest, let me patch it up and actually look where this would fall in the guest space. Oh OK, let me use this value real value which correspond the guests real value."
It will take some time for me to understand every thing you have written :)
Ok. Ping me if you have some questions. Reading this might help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM
But the IOMMU has a couple of knobs. One of those is the passthrough code and the GFX mapping code. The first should be easy to spot (should tell you on bootup whether you got that enabled or not), while the other looks to be a bunch of workarounds when passing in GFX cards b/c they .. well, not sure what, but they look to not work correctly with the IOMMU.
By GFX Mapping, Do you mean the code that maps VGA frame-buffers to Guest OS?
No. It is the IOMMU kernel code that checks to see if this is a GFX card and if so does something fancy. But the workarounds for this appear to be exclusive to Intel chipset so you shouldn't hit any.
One more silly question on the frame-buffer mapping.
I was reading a paper on xen vga pass-though. It says "When applying PCI pass-through, certain memory areas of the physical machine are mapped to the VM. When the guest OS writes to one of those memory addresses, Xen will make sure it is actually written at the appropriate address. This implicates that no other VM is able to make use of that device. The frame-buffer is mapped to the machine's main memory at address 0xA0000 to 0xC0000. This memory range must be mapped to the VM's memory in order for the OS to address the video adapter."
<nods> > > My question is.... > In my case, the host machine is using Nvidia graphics card, so the > host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by > host. Now if the ATI card maps it's framebuffer to this same address > wouldn't it cause any problem.
Ah, but it does not. The first card that is called by the BIOS (so your motherboard BIOS) sets itself up to use that memory. The other card has to wait. Keep in mind that the A0000 to C0000 is the old style VGA framebuffer (80x25) or if you program the card properly it can do VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic video card passed in which sets this device up (and touching that area in the guest ends up in VNC window or SDL window).
Anyhow back to your host.. If you grep in your kernel log you should see something like: [drm] fb mappable at 0xE0142000
This is the framebuffer address.
If the the machine has only 1 graphics card those statement make sense, but I could not understand what will happen when there are two graphics cards, each connected to different monitor and assigned to different VM, where would each machines frame-buffer reside in the host memory.
Check the kernel log. Usually it is also the first BAR in the pci device.
Does Xen allow passing-though multiple graphics cards, each to different VM?
Yes (with some patches posted by the AMD and Intel folks).. (look for threads that have some GFX or Radeon or something like that in its title).
If you see 'identity map for device' where the device is your GFX then that looks to be the bug. It shouldn't set the identity mapping on it.
I guess the identity map is used only on the Intel chipset. Here are msgs on host machine when addition debugging parameter (amd_iommu_dump) was added.
prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 00000002 AMD 00000001) [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW 00000001 AMD 00000001) [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ root=/dev/sda1 ro iommu=1 amd_iommu_dump [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.012918] Performance Events: AMD PMU driver. [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 [ 0.300011] System has AMD C1E enabled [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 devid_to: 00:14.4 [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor (6 cpu cores) (version 2.20.00) [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 [ 12.350018] EDAC amd64: This node reports that Memory ECC is currently disabled, set F3x44[22] (0000:00:18.3). [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
What motherboard is this?
ASUS M4A89TD Pro/USB3, it is mentioned on the link http://wiki.xensource.com/xenwiki/VTdHowTo
prasad@prasad-kvm:~$ lspci | grep -i RV 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (rev 80) 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL V3100 (PCIE)] (Secondary) (rev 80)
So, how come you aren't passing in 02:00.1?
I tried passing through 02.00.1 as well. When I pass it to a Linux machine I do not see the device bind to a driver, I don't see the drm logs in the dmsg. So I thought I am not supposed to pass the 02:00:1.
Should I be passing 02.00.1 instead of 02.00.0?
02:00.0 is the actual device. 02:00.1 is just a placeholder for supporting dualhead certain other OSes.
Alex
Look for 'DMAR GFX' on Google. Might shed some more light.
No DMAR for AMD. I think I should have a look at DMAR GFX.
I am really thankful to you for your support.
Hopefully I haven't confused you any further :-)
Thanks and Regards, Prasad
dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel
What motherboard is this?
ASUS M4A89TD Pro/USB3, it is mentioned on the link http://wiki.xensource.com/xenwiki/VTdHowTo
I am not sure how the implementation of the IOMMU in the Xen hypervisor is different from the Linux implementation. Usually it is lock-step, but it might be different.
Also, Xen's QEMU has different mechanism for handling PCI passthrough so ...
Lots of variables to figure out why it does not work for you. It might make sense to start first with a working case and from there start switching over to KVM.
dri-devel@lists.freedesktop.org
-
Alex Deucher -
Konrad Rzeszutek Wilk -
Michel Dänzer -
Prasad Joshi