<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am trying to configure Openstack Ocata with support for SR-IOV on Intel X710 NICs, but am struggling to launch any VMs.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I've followed the steps available at: https://docs.openstack.org/ocata/networking-guide/config-sriov.html, but am encountering an issue where my available hosts are filtered.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#####<o:p></o:p></p>
<p class="MsoNormal">Problem<o:p></o:p></p>
<p class="MsoNormal">#####<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Launching a VM, configured using a port-id that uses "--binding:vnic_type direct" fails to launch.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#####<o:p></o:p></p>
<p class="MsoNormal">Details<o:p></o:p></p>
<p class="MsoNormal">#####<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Following the guide available at <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> https://docs.openstack.org/ocata/networking-guide/config-sriov.html
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have tried to enable the use of Virtual Function (VF) ports using SR-IOV on an on Intel X710 NIC. When I come to spin-up a VM using the steps documented, e.g.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> openstack server create --flavor m1.large --image cirros --nic port-id=$port_id test-sriov<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal">the instantiation fails. In /var/log/nova-conductor.log, I can see: -<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 2018-01-05 18:49:04.470 36491 WARNING nova.scheduler.utils [req-788817bb-4e78-4a1b-8bf5-240a20472cd8 0218d276a04e40ab806d4075671ef01b dd2323e3ab4f45f29b6e7840fb7c0d2c - - -] Failed to compute_task_build_instances:
No valid host was found. There are not enough hosts available.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> Traceback (most recent call last):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 218, in inner<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> return func(*args, **kwargs)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 98, in select_destinations<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> dests = self.driver.select_destinations(ctxt, spec_obj)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> raise exception.NoValidHost(reason=reason)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> NoValidHost: No valid host was found. There are not enough hosts available.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <span style="font-family:Consolas">2018-01-05 18:49:04.471 36491 WARNING nova.scheduler.utils [req-788817bb-4e78-4a1b-8bf5-240a20472cd8 0218d276a04e40ab806d4075671ef01b dd2323e3ab4f45f29b6e7840fb7c0d2c - - -] [instance: f18454bb-ff7e-4171-8ebd-982197e353f7]
Setting instance to ERROR state.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Note: NoValidHost: No valid host was found. There are not enough hosts available.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In /var/log/nova-scheduler.log I also see the following: -<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <span style="font-family:Consolas">2018-01-05 18:49:04.371 37065 INFO nova.filters [...] Filter PciPassthroughFilter returned 0 hosts<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 2018-01-05 18:49:04.372 37065 INFO nova.filters [...] Filtering removed all hosts for the request with instance ID 'f18454bb-ff7e-4171-8ebd-982197e353f7'. Filter results: ['PciPassthroughFilter:
(start: 2, end: 0)']<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Initially, I felt this might be because the VM flavor used didn't have a "pci_passthrough:alias" value set. I've made sure to set my flavor like so: -<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> +----------------------------+--------------------------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | Field | Value |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> +----------------------------+--------------------------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | OS-FLV-DISABLED:disabled | False |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | OS-FLV-EXT-DATA:ephemeral | 0 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | access_project_ids | af8fadc445164950a6d7ad4f91e1e06b, dd2323e3ab4f45f29b6e7840fb7c0d2c |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | disk | 5 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | id | fc76fed6-0583-4c6b-908b-3c49d6cd6652 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | name | m2.mini |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | os-flavor-access:is_public | False |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | properties | pci_passthrough:alias='a1:1' |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | ram | 512 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | rxtx_factor | 1.0 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | swap | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> | vcpus | 1 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> +----------------------------+--------------------------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">where "alias" is set in /etc/nova/nova.conf on the controller as: -
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> [pci]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> alias = { "vendor_id":"8086", "product_id":"154c", "device_type":"type-VF","name":"a1" }<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I've checked the PCI details on my VF's, and they are as follows: -<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 81:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 81:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 81:02.0 Ethernet controller [0200]: Intel Corporation XL710/X710 Virtual Function [8086:154c] (rev 01)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 81:02.1 Ethernet controller [0200]: Intel Corporation XL710/X710 Virtual Function [8086:154c] (rev 01)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 81:02.2 Ethernet controller [0200]: Intel Corporation XL710/X710 Virtual Function [8086:154c] (rev 01)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> [and so on... 30x VFs in total]<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As far as other configuration files look, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/etc/nova/nova.conf on the Compute is configured as so: -<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> [pci]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> passthrough_whitelist ={ "devname": "enp129s0f0", "physical_network": "physnet2"}<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">the /etc/neutron/plugins/ml2/sriov_agent.ini on the Compute is configured to map my physical network to the NIC as follows: -<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas"> [sriov_nic]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> physical_device_mappings =physnet2:enp129s0f0<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">and firewall disabled with: -<span style="font-family:Consolas"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> [securitygroup]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> firewall_driver = neutron.agent.firewall.NoopFirewallDriver<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/etc/neutron/plugins/ml2/ml2_conf.ini on the Controller has mechanism_drivers configures as: -
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <span style="font-family:Consolas">mechanism_drivers = linuxbridge,sriovnicswitch<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And lastly, /etc/nova/nova.conf on the Controller as the following filters set: -
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <span style="font-family:Consolas">[DEFAULT]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> scheduler_default_filters =PciPassthroughFilter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> scheduler_available_filters = nova.scheduler.filters.all_filters<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I've been investigating this for some time now, and am able to confirm that without pursuing the SR-IOV route I am able to create VLAN Provider networks on my NIC interface quite happily, and VMs can connect to them with no issue. Are there
any obvious debugging routes I should investigate to try and resolve this?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for replies!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ben<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">Ben Thomas<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">Senior Research Associate<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">UK 5G Testbeds and Trials - University of Bristol<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">Level 0 - Merchant Venturers Building<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">Woodland Road, Clifton, Bristol, BS8 1UB, United Kingdom<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">Email: ben.thomas@bristol.ac.uk
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><a href="http://www.bristol.ac.uk/smart"><span style="color:blue">Smart Internet Lab</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>