<p dir="ltr">Hi Andrew,</p>
<p dir="ltr">Just wanted to quickly say that I really appreciate your prompt reply and hope you'll be happy to assist further if possible. I've just gotten slightly sidetracked by some other issues but will come back to this in the next week and provide more background info and results of workaround attempts.</p>
<p dir="ltr">Cheers,<br>
Blair</p>
<div class="gmail_extra"><br><div class="gmail_quote">On 28 Sep 2016 2:13 AM, "Andrew J Younge" <<a href="mailto:ajyounge@indiana.edu">ajyounge@indiana.edu</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Blair,<br>
<br>
I'm very interested to hear more about your project using virtualzed<br>
GPUs, and hopefully JP and/or myself can be of help here.<br>
<br>
So in the past we've struggled with the usage of PCI bridges as a<br>
connector between multiple GPUs. This was first seen with Xen and<br>
S2070 servers (which has 4 older GPUs across Nvidia PCI bridges) and<br>
found that the ACS was prohibiting the successful passthrough of the<br>
GPU. While we just decided to use discrete independent adapters moving<br>
forward, we've never gone back and tried this with KVM. With that, I<br>
can expect the same issues as the ACS cannot guarantee proper<br>
isolation of the device. Looking at the K80 GPUs, I'm seeing that<br>
there are 3 PLX bridges for each GPU pair (see my output below for a<br>
native system w/out KVM), and I'd estimate likely these would be on<br>
the same iommu group. This could be the problem.<br>
<br>
I have heard that such a patch exists in KVM for you to override the<br>
IOMMU groups and ACS protections, however I don't have any experience<br>
with it directly [1]. In our experiments, we used an updated SeaBIOS,<br>
whereas the link provided below details a UEFI BIOS. This may have<br>
different implications that I don't have experience with.<br>
Furthermore, I assume this patch will likely just be ignoring all of<br>
ACS, which is going to be an obvious and potentially severe security<br>
risk. In a purely academic environment such a security risk may not<br>
matter, but it should be noted nonetheless.<br>
<br>
So, lets take a few steps back to confirm things. Are you able to<br>
actually pass both K80 GPUs through to a running KVM instance, and<br>
have the Nvidia drivers loaded? Any dmesg output errors here may go a<br>
long way. Are you also passing through the PCI bridge device (lspci<br>
should show one)? If you're actually making it that far, it may next<br>
be worth simply running a regular CUDA application set first before<br>
trying any GPUDirect methods. For our GPUDirect usage, we were<br>
specifically leveraging the RDMA support with an InfiniBand adapter<br>
rather than CUDA P2P, so your mileage may vary there as well.<br>
<br>
Hopefully this is helpful in finding your problem. With this, I'd be<br>
interested to hear if the ACS override mechanism, or any other option<br>
works for enabling passthrough with K80 GPUs (we have a few dozen<br>
non-virtualized for another project). If you have any other<br>
non-bridged GPU cards (like a K20 or C2075) lying around, it may be<br>
worth giving that a try to try to rule-out other potential issues<br>
first.<br>
<br>
[1] <a href="https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Bypassing_the_IOMMU_groups_.28ACS_override_patch.29" rel="noreferrer" target="_blank">https://wiki.archlinux.org/<wbr>index.php/PCI_passthrough_via_<wbr>OVMF#Bypassing_the_IOMMU_<wbr>groups_.28ACS_override_patch.<wbr>29</a><br>
<br>
[root@r-001 ~]# lspci | grep -i -e PLX -e nvidia<br>
02:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
03:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
03:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
04:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
05:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
06:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
07:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
07:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
08:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
09:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
82:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
83:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
83:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
84:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
85:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
86:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
87:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
87:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI<br>
Express Gen 3 (8.0 GT/s) Switch (rev ca)<br>
88:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
89:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)<br>
[root@r-001 ~]# nvidia-smi topo --matrix<br>
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 mlx4_0 CPU Affinity<br>
GPU0 X PIX PHB PHB SOC SOC SOC SOC SOC 0-11,24-35<br>
GPU1 PIX X PHB PHB SOC SOC SOC SOC SOC 0-11,24-35<br>
GPU2 PHB PHB X PIX SOC SOC SOC SOC SOC 0-11,24-35<br>
GPU3 PHB PHB PIX X SOC SOC SOC SOC SOC 0-11,24-35<br>
GPU4 SOC SOC SOC SOC X PIX PHB PHB PHB 12-23,36-47<br>
GPU5 SOC SOC SOC SOC PIX X PHB PHB PHB 12-23,36-47<br>
GPU6 SOC SOC SOC SOC PHB PHB X PIX PHB 12-23,36-47<br>
GPU7 SOC SOC SOC SOC PHB PHB PIX X PHB 12-23,36-47<br>
mlx4_0 SOC SOC SOC SOC PHB PHB PHB PHB X<br>
<br>
Legend:<br>
<br>
X = Self<br>
SOC = Path traverses a socket-level link (e.g. QPI)<br>
PHB = Path traverses a PCIe host bridge<br>
PXB = Path traverses multiple PCIe internal switches<br>
PIX = Path traverses a PCIe internal switch<br>
<br>
<br>
Cheers,<br>
Andrew<br>
<br>
<br>
Andrew J. Younge<br>
School of Informatics & Computing<br>
Indiana University / Bloomington, IN USA<br>
<a href="mailto:ajyounge@indiana.edu">ajyounge@indiana.edu</a> / <a href="http://ajyounge.com" rel="noreferrer" target="_blank">http://ajyounge.com</a><br>
<br>
<br>
On Tue, Sep 27, 2016 at 4:37 AM, Blair Bethwaite<br>
<<a href="mailto:blair.bethwaite@gmail.com">blair.bethwaite@gmail.com</a>> wrote:<br>
> Hi Andrew, hi John -<br>
><br>
> I've just started trying to get CUDA P2P working in our virtualized<br>
> HPC environment. I figure this must be something you solved already in<br>
> order to produce the aforementioned paper, but having read it a couple<br>
> of times I don't think it provides enough detail about the guest<br>
> config, hoping you can shed some light...<br>
><br>
> The issue I'm grappling with is that despite using a qemu-kvm machine<br>
> type (q35) with an emulated PCIe bus and seeing that indeed the P2P<br>
> capable GPUs (NVIDIA K80s) are attached to that bus, and nvidia-smi<br>
> sees them as sharing a PHB, the simpleP2P CUDA sample fails when<br>
> checking their ability to communicate with each other. Is there some<br>
> magic config I might be missing, did you need to make any PCI-ACS<br>
> changes?<br>
><br>
> Best regards,<br>
> Blair<br>
><br>
><br>
> On 16 March 2016 at 07:57, Blair Bethwaite <<a href="mailto:blair.bethwaite@gmail.com">blair.bethwaite@gmail.com</a>> wrote:<br>
>><br>
>> Hi Andrew,<br>
>><br>
>> On 16 March 2016 at 05:28, Andrew J Younge <<a href="mailto:ajyounge@indiana.edu">ajyounge@indiana.edu</a>> wrote:<br>
>> > point to a recent publication of ours at VEE15 titled "Supporting High<br>
>> > Performance Molecular Dynamics in Virtualized Clusters using IOMMU,<br>
>> > SR-IOV, and GPUDirect." In the paper we show that using Nvidia GPUs<br>
>> ...<br>
>> > <a href="http://dl.acm.org/citation.cfm?id=2731194" rel="noreferrer" target="_blank">http://dl.acm.org/citation.<wbr>cfm?id=2731194</a><br>
>><br>
>> Oooh interesting - GPUDirect too. That's something I've been wanting<br>
>> to try out in our environment. Will take a look a your paper...<br>
>><br>
>> --<br>
>> Cheers,<br>
>> ~Blairo<br>
><br>
><br>
> --<br>
> Cheers,<br>
> ~Blairo<br>
</blockquote></div></div>