<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Dear Openstack community,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am trying to attach NVMe drives to my vms and would like to check the configuration I have because for some reason it is not working.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is my physical host details:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@zeus-59 ~]# lspci -nn -D | grep -i ssd<o:p></o:p></p>
<p class="MsoNormal">0000:03:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:04:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:06:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:07:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:08:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:09:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:84:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:85:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:86:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal">0000:87:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My idea is to attach either 0000:03:00.0 or 0000:04:00.0 to a vm<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is how I identify the block devices for each pci device:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@zeus-59 ~]# ls -l /sys/bus/pci/devices/0000\:03\:00.0/nvme<o:p></o:p></p>
<p class="MsoNormal">total 0<o:p></o:p></p>
<p class="MsoNormal">drwxr-xr-x. 4 root root 0 Apr 3 14:02 nvme0<o:p></o:p></p>
<p class="MsoNormal">[root@zeus-59 ~]# ls -l /sys/bus/pci/devices/0000\:04\:00.0/nvme<o:p></o:p></p>
<p class="MsoNormal">total 0<o:p></o:p></p>
<p class="MsoNormal">drwxr-xr-x. 4 root root 0 Apr 3 14:02 nvme1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I installed the OS on the physical machine leaving those 2 drives intact:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@zeus-59 ~]# lsblk<o:p></o:p></p>
<p class="MsoNormal">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT<o:p></o:p></p>
<p class="MsoNormal">sda 8:0 0 59.6G 0 disk<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">†</span>€sda1 8:1 0 1G 0 part /boot<o:p></o:p></p>
<p class="MsoNormal">„€sda2 8:2 0 4G 0 part [SWAP]<o:p></o:p></p>
<p class="MsoNormal">sr0 11:0 1 906M 0 rom<o:p></o:p></p>
<p class="MsoNormal">nvme0n1 259:15 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">nvme1n1 259:0 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">nvme2n1 259:11 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme2n1p1 259:12 0 1.8T 0 part /var/lib/docker/btrfs<o:p></o:p></p>
<p class="MsoNormal">nvme3n1 259:6 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme3n1p1 259:7 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal">nvme4n1 259:13 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme4n1p1 259:14 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal">nvme5n1 259:5 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme5n1p1 259:8 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal">nvme6n1 259:17 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme6n1p1 259:18 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal">nvme7n1 259:9 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme7n1p1 259:10 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal">nvme8n1 259:3 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme8n1p1 259:4 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal">nvme9n1 259:1 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme9n1p1 259:2 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The next thing is to configure Openstack<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">nova api config<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[pci]<o:p></o:p></p>
<p class="MsoNormal">alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">nova scheduler config<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[filter_scheduler]<o:p></o:p></p>
<p class="MsoNormal">enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter<o:p></o:p></p>
<p class="MsoNormal">available_filters = nova.scheduler.filters.all_filters<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">nova compute config<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[filter_scheduler]<o:p></o:p></p>
<p class="MsoNormal">enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter<o:p></o:p></p>
<p class="MsoNormal">available_filters = nova.scheduler.filters.all_filters<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[pci]<o:p></o:p></p>
<p class="MsoNormal">passthrough_whitelist = [ {"address":"0000:03:00.0"}, {"address":"0000:04:00.0"} ]<o:p></o:p></p>
<p class="MsoNormal">alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Then I create my flavor<o:p></o:p></p>
<p class="MsoNormal">openstack flavor create nvme.small --ram 64000 --disk 10 --vcpus 7 --property "pci_passthrough:alias"="nvme:1"<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI","sans-serif";color:#172B4D;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI","sans-serif";color:#172B4D;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal">I created the vm towards the host with the free nvme drives<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@openstack-deployment ~]# openstack server create --flavor nvme.small --image centos7.5-image --nic net-id=hpc --security-group admin --key-name mykey --availability-zone nova:zeus-59.localdomain test_nvme_small<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Vm creation fails and I can see this error in the nova compute logs<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2019-04-03 16:50:35.512 7 ERROR nova.virt.libvirt.guest [req-8478a99f-bad0-43d5-b405-55c45e8d8cae 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Error launching a defined domain with XML: <domain type='kvm'><o:p></o:p></p>
<p class="MsoNormal"> <name>instance-00000090</name><o:p></o:p></p>
<p class="MsoNormal">…<o:p></o:p></p>
<p class="MsoNormal"> <devices><o:p></o:p></p>
<p class="MsoNormal"> <emulator>/usr/libexec/qemu-kvm</emulator><o:p></o:p></p>
<p class="MsoNormal"> <disk type='file' device='disk'><o:p></o:p></p>
<p class="MsoNormal"> <driver name='qemu' type='qcow2' cache='none'/><o:p></o:p></p>
<p class="MsoNormal"> <source file='/var/lib/nova/instances/a261fcd4-eca3-4982-8b2a-1df33087ab40/disk'/><o:p></o:p></p>
<p class="MsoNormal"> <target dev='vda' bus='virtio'/><o:p></o:p></p>
<p class="MsoNormal"> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/><o:p></o:p></p>
<p class="MsoNormal"> </disk><o:p></o:p></p>
<p class="MsoNormal">…<o:p></o:p></p>
<p class="MsoNormal">2019-04-03 16:50:35.513 7 ERROR nova.virt.libvirt.driver [req-8478a99f-bad0-43d5-b405-55c45e8d8cae 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: a261fcd4-eca3-4982-8b2a-1df33087ab40] Failed
to start libvirt guest: libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any idea or advice would be very helpful<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you very much<o:p></o:p></p>
</div>
<div style="font-size:8pt; font-family: Arial">NOTICE</div>
<div style="font-size:8pt; font-family: Arial">Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are
not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution
of viruses or similar in electronic communications. This notice should not be removed.
</div>
</body>
</html>