<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Although it contradicts the idea of a cloud, I believe the CPU mapping between the guest and the host is a valid case for NFV applications. The best that one
can do is to ensure vCPU and virtual memory are mapped to single NUMA node within the host and to make sure the CPUs don’t float within the NUMA.<br>
<br>
A while back, I was able to do this on a Kilo based lab for performance benchmarking:<br>
<a href="https://ask.openstack.org/en/question/87711/numa-awareness-during-instance-placement/">https://ask.openstack.org/en/question/87711/numa-awareness-during-instance-placement/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">While the answer may not be up to date with respect to newer versions of OpenStack, in addition to numa_* extra specs, you could look at cpu_policy and cpu_thread_policy
as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Kaustubh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Arne Wiebalck [mailto:arne.wiebalck@cern.ch]
<br>
<b>Sent:</b> Friday, July 8, 2016 3:11 PM<br>
<b>To:</b> Brent Troge <brenttroge2016@gmail.com><br>
<b>Cc:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack] vCPU -> pCPU MAPPING<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">We have use cases in our cloud which require vCPU-to-NUMA_node pinning<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to maximise the CPU performance available in the guests. From what we’ve<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">seen, there was no further improvement when the vCPUs were mapped<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">one-to-one to pCPUs (we did not study this in detail, though, as with the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">NUMA node pinning the performance was sufficiently close to the physical<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">one).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">To implement this, we specify the numa_nodes extra_spec for the corresponding<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">flavor and rely on nova’s placement policy.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">HTH,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Arne<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">—<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Arne Wiebalck<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">CERN IT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 08 Jul 2016, at 19:22, Steve Gordon <<a href="mailto:sgordon@redhat.com">sgordon@redhat.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">----- Original Message -----<br style="orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">From: "Brent Troge" <</span><a href="mailto:brenttroge2016@gmail.com"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">brenttroge2016@gmail.com</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">><br>
To:<span class="apple-converted-space"> </span></span><a href="mailto:openstack@lists.openstack.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">openstack@lists.openstack.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Sent: Friday, July 8, 2016 9:59:58 AM<br>
Subject: [Openstack] vCPU -> pCPU MAPPING<br>
<br>
context - high performance private cloud with cpu pinning<br>
<br>
Is it possible to map vCPUs to specific pCPUs ?<br>
Currently I see you can only direct which vCPUs are mapped to a specific<br>
NUMA node<br>
<br>
hw:numa_cpus.0=1,2,3,4<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Just in addition to Jay's comment, the above does not do what I suspect you think it does. The above tells Nova to expose vCPUs 1, 2, 3, and 4 in *guest* NUMA node 0 when building the guest NUMA topology in the Libvirt XML. Nova will endeavor to map these vCPUs
to pCPUs on the same NUMA node on the host as *each other* but that will not necessarily be NUMA node *0* on the host depending on resource availability.<br>
<br>
Thanks,<br>
<br>
Steve<br>
<br style="orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">However, to get even more granular, is it possible to create a flavor which<br>
maps vCPU to specific pCPU within a numa node ?<br>
<br>
Something like:<br>
hw:numa_cpus.<NUMA-NODE>-<pCPU>=<vCPU><br>
<br>
hw:numa_cpus.0-1=1<br>
hw:numa_cpus.0-2=2<br>
<br>
<br>
Thanks!<br>
<br>
_______________________________________________<br>
Mailing list: </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Post to : </span><a href="mailto:openstack@lists.openstack.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">openstack@lists.openstack.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Unsubscribe : </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
_______________________________________________<br>
Mailing list:<span class="apple-converted-space"> </span></span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Post to :<span class="apple-converted-space"> </span></span><a href="mailto:openstack@lists.openstack.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">openstack@lists.openstack.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Unsubscribe :<span class="apple-converted-space"> </span></span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>