<div dir="ltr"><div><div><div><div><div><div><br></div>Based on my testing you can control NUMA node placement if you are also using SRIOV and each PF is grouped in its own<br></div>physical network.<br><br>How did I test ?<br><br>First set your flavor to use a single numa node.<br><br>hw_numa_nodes=1<br><br></div>Then when creating your neutron port(s), call a neutron network created against the physical network which resides in the desired NUMA node.<br><br></div><div>For example say you want to drive resource alignment within NUMA 1<br></div><br>Because of 'hw_numa_nodes=1' and because your neutron ports are created against physcial network in NUMA 1, nova will try to grab memory/cpu from the same. <br></div>If not enough resources available in NUMA1 on all applicable hosts then instantiation fails.<br><br></div>I tested numerous times and tried to account for multiple variables, in each test I was able to select NUMA alignment using the above method.<br><br><br><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 8, 2016 at 2:43 PM, Kaustubh Kelkar <span dir="ltr"><<a href="mailto:kaustubh.kelkar@casa-systems.com" target="_blank">kaustubh.kelkar@casa-systems.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Although it contradicts the idea of a cloud, I believe the CPU mapping between the guest and the host is a valid case for NFV applications. The best that one
can do is to ensure vCPU and virtual memory are mapped to single NUMA node within the host and to make sure the CPUs don’t float within the NUMA.<br>
<br>
A while back, I was able to do this on a Kilo based lab for performance benchmarking:<br>
<a href="https://ask.openstack.org/en/question/87711/numa-awareness-during-instance-placement/" target="_blank">https://ask.openstack.org/en/question/87711/numa-awareness-during-instance-placement/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">While the answer may not be up to date with respect to newer versions of OpenStack, in addition to numa_* extra specs, you could look at cpu_policy and cpu_thread_policy
as well.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">-Kaustubh<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Arne Wiebalck [mailto:<a href="mailto:arne.wiebalck@cern.ch" target="_blank">arne.wiebalck@cern.ch</a>]
<br>
<b>Sent:</b> Friday, July 8, 2016 3:11 PM<br>
<b>To:</b> Brent Troge <<a href="mailto:brenttroge2016@gmail.com" target="_blank">brenttroge2016@gmail.com</a>><br>
<b>Cc:</b> <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
<b>Subject:</b> Re: [Openstack] vCPU -> pCPU MAPPING<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">We have use cases in our cloud which require vCPU-to-NUMA_node pinning<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">to maximise the CPU performance available in the guests. From what we’ve<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">seen, there was no further improvement when the vCPUs were mapped<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">one-to-one to pCPUs (we did not study this in detail, though, as with the<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">NUMA node pinning the performance was sufficiently close to the physical<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">one).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">To implement this, we specify the numa_nodes extra_spec for the corresponding<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">flavor and rely on nova’s placement policy.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">HTH,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> Arne<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">—<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Arne Wiebalck<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">CERN IT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 08 Jul 2016, at 19:22, Steve Gordon <<a href="mailto:sgordon@redhat.com" target="_blank">sgordon@redhat.com</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">----- Original Message -----<br style="text-align:start;word-spacing:0px">
<br>
</span><u></u><u></u></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">From: "Brent Troge" <</span><a href="mailto:brenttroge2016@gmail.com" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">brenttroge2016@gmail.com</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">><br>
To:<span> </span></span><a href="mailto:openstack@lists.openstack.org" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">openstack@lists.openstack.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Sent: Friday, July 8, 2016 9:59:58 AM<br>
Subject: [Openstack] vCPU -> pCPU MAPPING<br>
<br>
context - high performance private cloud with cpu pinning<br>
<br>
Is it possible to map vCPUs to specific pCPUs ?<br>
Currently I see you can only direct which vCPUs are mapped to a specific<br>
NUMA node<br>
<br>
hw:numa_cpus.0=1,2,3,4<u></u><u></u></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Just in addition to Jay's comment, the above does not do what I suspect you think it does. The above tells Nova to expose vCPUs 1, 2, 3, and 4 in *guest* NUMA node 0 when building the guest NUMA topology in the Libvirt XML. Nova will endeavor to map these vCPUs
to pCPUs on the same NUMA node on the host as *each other* but that will not necessarily be NUMA node *0* on the host depending on resource availability.<br>
<br>
Thanks,<br>
<br>
Steve<br>
<br style="text-align:start;word-spacing:0px">
<br>
</span><u></u><u></u></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">However, to get even more granular, is it possible to create a flavor which<br>
maps vCPU to specific pCPU within a numa node ?<br>
<br>
Something like:<br>
hw:numa_cpus.<NUMA-NODE>-<pCPU>=<vCPU><br>
<br>
hw:numa_cpus.0-1=1<br>
hw:numa_cpus.0-2=2<br>
<br>
<br>
Thanks!<br>
<br>
_______________________________________________<br>
Mailing list: </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Post to : </span><a href="mailto:openstack@lists.openstack.org" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">openstack@lists.openstack.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Unsubscribe : </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><u></u><u></u></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
_______________________________________________<br>
Mailing list:<span> </span></span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Post to :<span> </span></span><a href="mailto:openstack@lists.openstack.org" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">openstack@lists.openstack.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Unsubscribe :<span> </span></span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
</div>
</blockquote></div><br></div>