<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:678585375;
mso-list-type:hybrid;
mso-list-template-ids:-33403498 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">See my comments inline
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Sam Stoelinga [mailto:sammiestoel@gmail.com]
<br>
<b>Sent:</b> Sunday, July 19, 2015 10:37 AM<br>
<b>To:</b> Moshe Levi<br>
<b>Cc:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack] [Neutron][SRIOV][docs] Enabling SRIOV on OpenStack Juno step-by-step guide<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">I think it was not fair to say it's not up to date. It seems it's up to date, but current downsides of existing OpenStack wikis on SRIOV are missing info, many different Wikis and hard to consume the info:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">1. <a href="https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking">https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. <a href="https://wiki.openstack.org/wiki/Nova-neutron-sriov">https://wiki.openstack.org/wiki/Nova-neutron-sriov</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3. <a href="https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support">https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">4. <a href="https://wiki.openstack.org/wiki/Pci_passthrough">https://wiki.openstack.org/wiki/Pci_passthrough</a><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] – I think the only wiki users should be using is </span><a href="https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking">https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking</a>
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[2],[3] are the blueprints and [4] is PCI-Pass-through without SR-IOV and it refer to [1] which is good.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal">Things that I noticed that were missing / could be better:<o:p></o:p></p>
</div>
<div>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span dir="LTR"></span>Adding PCIDeviceFilter to nova-scheduler<o:p></o:p></p>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] I think you mean the PciPassthroughFilter and you are correct indeed it is missing. I will update the wiki.</span><o:p></o:p></pre>
</div>
<div>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span dir="LTR"></span>How to enable VFs on compute nodes<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] – This vendor specific but we can add links to vendor wiki page on how to configure their NIC to support SR-IOV</span><span style="color:#1F497D"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">3. This wiki: <a href="https://wiki.openstack.org/wiki/Nova-neutron-sriov">https://wiki.openstack.org/wiki/Nova-neutron-sriov</a> contains incorrect information.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Current wrong: "neutron port-create <net-uuid-from-step-1> --name sriov_port
<b>--vnic-type direct</b> "<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">should be "neutron port-create <net-uuid-from-step-1> --name sriov_port<b> --binding:vnic_type direc</b>t"<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] – this is blueprint but I will check if I can update it.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoListParagraph" style="margin-left:41.25pt;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span>Make it more clear that agent_required = False is totally fine and may be better. From what I read you have to disable the firewall functionality if you enable sriov-agent? Not sure if that understanding
is correct.<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] – I agree this is totally need clarification. The agent_required=False is used when you have Intel NIC that doesn’t support admin up/down change. SR-IOV
NIC that support admin up/down change should be configured with agent_required =True.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I will update the wiki explaining that flag. (by the way we hope to change it in liberty and deprecate the agent_required flag)
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I would prefer this information to have release bound documentation in for example the Networking Guide: <a href="http://docs.openstack.org/networking-guide/">http://docs.openstack.org/networking-guide/</a> or the Cloud Administrator Guide: <a href="http://docs.openstack.org/admin-guide-cloud/content/">http://docs.openstack.org/admin-guide-cloud/content/</a><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] – Ok, I guess this is required change in
<a href="https://review.openstack.org/#/admin/projects/openstack/openstack-manuals">
<span style="color:#1F497D;text-decoration:none">openstack/openstack-manuals</span></a> repository. Can you help and adding documentations there? Just put me as review. If not I will try do it myself or find someone in Mellanox.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">I believe that using the pci sys interface is vendor compatible. Would be great if you could confirm. Could you try testing $ echo '7' > /sys/class/net/eth3/device/sriov_numvfs on a mellanox card? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This way we don't have to write vendor specific docs on enabling VFs :) <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">When using modprobe ixgbe max_vfs=7 it tells you that using max_vfs is deprecated and that the pci sys interface should be used. That's how I found out about this.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[ML] – unfortunately it is not generic in Melllanox you need to configure number of VFs and number of probes (also Single Port or Duel Port ) it is more complicate
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">L</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">see
<a href="https://community.mellanox.com/docs/DOC-1484">https://community.mellanox.com/docs/DOC-1484</a>
<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Sun, Jul 19, 2015 at 2:44 PM, Moshe Levi <<a href="mailto:moshele@mellanox.com" target="_blank">moshele@mellanox.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Sam,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Can you explain why you think that the <a href="https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking" target="_blank">https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking</a>
is out of date?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Moreover you blog explain how to configure SR-IOV on Intel NIC, but keep in mind Neutron SRIOV is generic
and can support any other vendors such as Mellanox.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Maybe will should add links to how to configure SR-IOV NIC for several Vendors. We can start with
Mellanox and Intel NIC. What do you think?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Sam Stoelinga [mailto:<a href="mailto:sammiestoel@gmail.com" target="_blank">sammiestoel@gmail.com</a>]
<br>
<b>Sent:</b> Saturday, July 18, 2015 5:55 PM<br>
<b>To:</b> <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
<b>Subject:</b> [Openstack] [Neutron][SRIOV][docs] Enabling SRIOV on OpenStack Juno step-by-step guide</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi networking gurus,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">While it may be easy for many of you to enable Neutron SRIOV on OpenStack it wasn't a smooth ride for me. I documented exactly which steps were required to enable SRIOV on OpenStack
on my blog: <a href="http://samos-it.com/posts/sriov-openstack-juno-fuel-6-1.html" target="_blank">http://samos-it.com/posts/sriov-openstack-juno-fuel-6-1.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It seems there is no official documentation yet other than 2 out of date wiki pages. I would like take the content of my blog post to official OpenStack docs if you guys/girls think
it's useful for the broader audience.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Sam Stoelinga<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>