On Thu, Apr 4, 2019 at 8:55 PM Manuel Sopena Ballesteros <manuel.sb@garvan.org.au> wrote:

Dear Openstack community,

I recently setup pci-passthrough against nvme drives to directly attach the disks to the vm as a block device.

I created 2 vms on the same physical host and I can see the disks (nvme0n1) on both of them:

[centos@test-nvme-small ~]$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

vda 253:0 0 10G 0 disk

└─vda1 253:1 0 10G 0 part /

nvme0n1 259:0 0 1.8T 0 disk

[centos@test-nvme-small-2 ~]$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

vda 253:0 0 10G 0 disk

└─vda1 253:1 0 10G 0 part /

nvme0n1 259:0 0 1.8T 0 disk

So I wanted to check which nvme device was attached to which vm. I thought one option would be to dump the vm xml file from virsh which can be found below (I am only putting the information between the disks section but happy to attach the hole file if needed):

Device/disk configuration for instance-00000094

<disk type='file' device='disk'>

<driver name='qemu' type='qcow2' cache='none'/>

<source file='/var/lib/nova/instances/fd95cc45-1501-4693-8643-944be2ff4625/disk'/>

<backingStore type='file' index='1'>

<format type='raw'/>

<source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/>

<backingStore/>

</backingStore>

<target dev='vda' bus='virtio'/>

<alias name='virtio-disk0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

</disk>

Device/disk configuration for instance-00000093

<disk type='file' device='disk'>

<driver name='qemu' type='qcow2' cache='none'/>

<source file='/var/lib/nova/instances/7e6a055c-1b4e-458c-89cd-cb8c1d10e939/disk'/>

<backingStore type='file' index='1'>

<format type='raw'/>

<source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/>

<backingStore/>

</backingStore>

<target dev='vda' bus='virtio'/>

<alias name='virtio-disk0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

</disk>

Based on this output and If I am not mistaken I am under the impression that the host pci device attached to both vms is 0000:00:04.0

Questions:

Is there an Openstack way to check which pci devices are attached to which vms instead of having to dump the vm xml file?

I don't think so. I'm not the best at answering this.

Am I right assuming that the same disk is attached to both vms? If yes, how can that happen?

That's the root device, that's not the PCI passthrough device

Thank you very much

NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com

On Fri, 2019-04-05 at 00:52 +0000, Manuel Sopena Ballesteros wrote:

Dear Openstack community,

I recently setup pci-passthrough against nvme drives to directly attach the disks to the vm as a block device.

I created 2 vms on the same physical host and I can see the disks (nvme0n1) on both of them:

[centos@test-nvme-small ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk └─vda1 253:1 0 10G 0 part / nvme0n1 259:0 0 1.8T 0 disk

[centos@test-nvme-small-2 ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk └─vda1 253:1 0 10G 0 part / nvme0n1 259:0 0 1.8T 0 disk

So I wanted to check which nvme device was attached to which vm. I thought one option would be to dump the vm xml file from virsh which can be found below (I am only putting the information between the disks section but happy to attach the hole file if needed):

Device/disk configuration for instance-00000094

<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/fd95cc45-1501-4693-8643-944be2ff4625/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>

Device/disk configuration for instance-00000093

<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/7e6a055c-1b4e-458c-89cd-cb8c1d10e939/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>

can you provide the full xmls these diss are not the nvme disk this is the info for the root disk wihc is a qcow 2 file on you host. when you are passing the pci devices form the host they will show up in the libvirt xml as a hostdev element. it will look something like this <devices> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> </source> </hostdev> </devices> in this case teh address in the source element will be the host pci address of the nvme device you may also have other fields like a second guest address element which will be outside the source element

Based on this output and If I am not mistaken I am under the impression that the host pci device attached to both vms is 0000:00:04.0

no this is the pci address where the virtio disk (your root disk) will be presented to the guest.

Questions:

Is there an Openstack way to check which pci devices are attached to which vms instead of having to dump the vm xml file?

technically its sored in the nova db in the pci_devices table but i would much perfer operators to look at the xmls first before touching the db.

Am I right assuming that the same disk is attached to both vms? no luclly that is not whats happening. if it was all maner of things would be broken :) If yes, how can that happen?

Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.