Openstack attaching same pci device to 2 different vms?
Dear Openstack community, I recently setup pci-passthrough against nvme drives to directly attach the disks to the vm as a block device. I created 2 vms on the same physical host and I can see the disks (nvme0n1) on both of them: [centos@test-nvme-small ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk └─vda1 253:1 0 10G 0 part / nvme0n1 259:0 0 1.8T 0 disk [centos@test-nvme-small-2 ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk └─vda1 253:1 0 10G 0 part / nvme0n1 259:0 0 1.8T 0 disk So I wanted to check which nvme device was attached to which vm. I thought one option would be to dump the vm xml file from virsh which can be found below (I am only putting the information between the disks section but happy to attach the hole file if needed): Device/disk configuration for instance-00000094 <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/fd95cc45-1501-4693-8643-944be2ff4625/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> Device/disk configuration for instance-00000093 <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/7e6a055c-1b4e-458c-89cd-cb8c1d10e939/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> Based on this output and If I am not mistaken I am under the impression that the host pci device attached to both vms is 0000:00:04.0 Questions: Is there an Openstack way to check which pci devices are attached to which vms instead of having to dump the vm xml file? Am I right assuming that the same disk is attached to both vms? If yes, how can that happen? Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
On Thu, Apr 4, 2019 at 8:55 PM Manuel Sopena Ballesteros <manuel.sb@garvan.org.au> wrote:
Dear Openstack community,
I recently setup pci-passthrough against nvme drives to directly attach the disks to the vm as a block device.
I created 2 vms on the same physical host and I can see the disks (nvme0n1) on both of them:
[centos@test-nvme-small ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 10G 0 disk
└─vda1 253:1 0 10G 0 part /
nvme0n1 259:0 0 1.8T 0 disk
[centos@test-nvme-small-2 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 10G 0 disk
└─vda1 253:1 0 10G 0 part /
nvme0n1 259:0 0 1.8T 0 disk
So I wanted to check which nvme device was attached to which vm. I thought one option would be to dump the vm xml file from virsh which can be found below (I am only putting the information between the disks section but happy to attach the hole file if needed):
Device/disk configuration for instance-00000094
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/fd95cc45-1501-4693-8643-944be2ff4625/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
Device/disk configuration for instance-00000093
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/7e6a055c-1b4e-458c-89cd-cb8c1d10e939/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
Based on this output and If I am not mistaken I am under the impression that the host pci device attached to both vms is 0000:00:04.0
Questions:
Is there an Openstack way to check which pci devices are attached to which vms instead of having to dump the vm xml file?
I don't think so. I'm not the best at answering this.
Am I right assuming that the same disk is attached to both vms? If yes, how can that happen?
That's the root device, that's not the PCI passthrough device
Thank you very much
NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com
Ok, Yes sorry, my bad, I got confused because I was looking in the wrong section. I believe the right place to look for the pass-through pci devices is under hostdev section <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> Thank you for your help Mohammed Manuel -----Original Message----- From: Mohammed Naser [mailto:mnaser@vexxhost.com] Sent: Friday, April 5, 2019 12:13 PM To: Manuel Sopena Ballesteros Cc: openstack@lists.openstack.org Subject: Re: Openstack attaching same pci device to 2 different vms? On Thu, Apr 4, 2019 at 8:55 PM Manuel Sopena Ballesteros <manuel.sb@garvan.org.au> wrote:
Dear Openstack community,
I recently setup pci-passthrough against nvme drives to directly attach the disks to the vm as a block device.
I created 2 vms on the same physical host and I can see the disks (nvme0n1) on both of them:
[centos@test-nvme-small ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 10G 0 disk
└─vda1 253:1 0 10G 0 part /
nvme0n1 259:0 0 1.8T 0 disk
[centos@test-nvme-small-2 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 10G 0 disk
└─vda1 253:1 0 10G 0 part /
nvme0n1 259:0 0 1.8T 0 disk
So I wanted to check which nvme device was attached to which vm. I thought one option would be to dump the vm xml file from virsh which can be found below (I am only putting the information between the disks section but happy to attach the hole file if needed):
Device/disk configuration for instance-00000094
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/fd95cc45-1501-4693-8643-944be2ff4625/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
Device/disk configuration for instance-00000093
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/7e6a055c-1b4e-458c-89cd-cb8c1d10e939/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
Based on this output and If I am not mistaken I am under the impression that the host pci device attached to both vms is 0000:00:04.0
Questions:
Is there an Openstack way to check which pci devices are attached to which vms instead of having to dump the vm xml file?
I don't think so. I'm not the best at answering this.
Am I right assuming that the same disk is attached to both vms? If yes, how can that happen?
That's the root device, that's not the PCI passthrough device
Thank you very much
NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
On Fri, 2019-04-05 at 00:52 +0000, Manuel Sopena Ballesteros wrote:
Dear Openstack community,
I recently setup pci-passthrough against nvme drives to directly attach the disks to the vm as a block device.
I created 2 vms on the same physical host and I can see the disks (nvme0n1) on both of them:
[centos@test-nvme-small ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk └─vda1 253:1 0 10G 0 part / nvme0n1 259:0 0 1.8T 0 disk
[centos@test-nvme-small-2 ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk └─vda1 253:1 0 10G 0 part / nvme0n1 259:0 0 1.8T 0 disk
So I wanted to check which nvme device was attached to which vm. I thought one option would be to dump the vm xml file from virsh which can be found below (I am only putting the information between the disks section but happy to attach the hole file if needed):
Device/disk configuration for instance-00000094
<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/fd95cc45-1501-4693-8643-944be2ff4625/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>
Device/disk configuration for instance-00000093
<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/7e6a055c-1b4e-458c-89cd-cb8c1d10e939/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/4cc6eebe175e35178cb81853818a1eb103cea937'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>
can you provide the full xmls these diss are not the nvme disk this is the info for the root disk wihc is a qcow 2 file on you host. when you are passing the pci devices form the host they will show up in the libvirt xml as a hostdev element. it will look something like this <devices> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> </source> </hostdev> </devices> in this case teh address in the source element will be the host pci address of the nvme device you may also have other fields like a second guest address element which will be outside the source element
Based on this output and If I am not mistaken I am under the impression that the host pci device attached to both vms is 0000:00:04.0
no this is the pci address where the virtio disk (your root disk) will be presented to the guest.
Questions:
Is there an Openstack way to check which pci devices are attached to which vms instead of having to dump the vm xml file?
technically its sored in the nova db in the pci_devices table but i would much perfer operators to look at the xmls first before touching the db.
Am I right assuming that the same disk is attached to both vms? no luclly that is not whats happening. if it was all maner of things would be broken :) If yes, how can that happen?
Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
participants (3)
-
Manuel Sopena Ballesteros
-
Mohammed Naser
-
Sean Mooney