<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Hi All,</p>
<p></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">After revisiting the spec [1] again and again, I got to know few points please check and let me know about my understanding:</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><b>Understanding:<span class="Apple-converted-space">
</span>If the ``vcpu_pin_set`` is set on compute node A in the Stein release then we can say that this<span class="Apple-converted-space">
</span>node is used to host the dedicated instance on it and if user upgrades from Stein to Train and if operator doesn’t define ``[compute] cpu_dedicated_set`` set then simply fallback to ``vcpu_pin_set`` and report it as PCPU inventory.<span class="Apple-converted-space"> </span></b></p>
<br>
<p></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Considering multiple combinations of various configuration options, I think we will need to implement below business rules so that the issue highlighted in the previous email about
the scheduler pre-filter can be solved.</p>
<br>
<p></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Rule 1:</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">If operator sets ``[compute] cpu_shared_set`` in Train.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">1.If pinned instances are found then we can simply say that this compute node is used as dedicated in the previous release so raise an error that says to set ``[compute] cpu_dedicated_set``
config option otherwise report it as VCPU inventory.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Rule 2:</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">If operator sets ``[compute] cpu_dedicated_set``<span class="Apple-converted-space">
</span>in Train.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">1. Report inventory as PCPU</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">2. If instances are found, check for host numa topology pinned_cpus, if pinned_cpus is not empty, that means this compute node is used as dedicated in the previous release and if empty,
then raise an error that this compute node is used as shared compute node in previous release.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Rule 3:</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">If operator sets None of the options (``[compute] cpu_dedicated_set``, ``[compute] cpu_shared_set``, ``vcpu_pin_set``)<span class="Apple-converted-space">
</span>in Train.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">1. If instances are found, check for host numa topology pinned_cpus, if<span class="Apple-converted-space">
</span>pinned_cpus is not empty, then raise an error that this compute node is used as dedicated compute node in previous release so set ``[compute] cpu_dedicated_set``, otherwise report inventory as VCPU.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">2. If no instances, report inventory as VCPU.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Rule 4:<span class="Apple-converted-space"> </span></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">If operator sets ``vcpu_pin_set`` config option in Train.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">1. If instances are found, check for host numa topology pinned_cpus, if pinned_cpus is empty, that means this compute node is used for non-pinned instances in the previous release,
so raise an error otherwise report it as PCPU inventory.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">2. If no instances, report inventory as PCPU.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Rule 5:</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">If operator sets ``vcpu_pin_set`` and ``[compute] cpu_dedicated_set`` or ``[compute] cpu_shared_set`` config options in Train</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">1. Simply raise an error</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Above business rules 3 and 4 are very important in order to solve the scheduler pre-filter issue highlighted in my previous email.</p>
<br>
<p></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">As of today, in either case, `vcpu_pin_set``<span class="Apple-converted-space">
</span>is set or not set on the compute node, it can used for both pinned or non-pinned instances depending on whether this host belongs to an aggregate with “pinned” metadata. But as per business rule #3 , if<span class="Apple-converted-space">
</span>``vcpu_pin_set`` is not set,<span class="Apple-converted-space"> </span>we are considering it to be used for non-pinned instances only.<span class="Apple-converted-space">
</span>Do you think this could cause an issue in providing backward compatibility?</p>
<br>
<p></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Please provide your suggestions on the above business rules.</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">[1]: <a href="https://review.opendev.org/#/c/555081/28/specs/train/approved/cpu-resources.rst@409" class="OWAAutoLink" id="LPlnk681407" previewremoved="true">https://review.opendev.org/#/c/555081/28/specs/train/approved/cpu-resources.rst@409</a> </p>
<p></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);"></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(228, 175, 10);"><br>
</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(228, 175, 10);"><br>
</p>
<p></p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">Thanks and Regards,</p>
<p style="margin: 0px; font: 12px "Helvetica Neue"; color: rgb(69, 69, 69);">-Bhagyashri Shewale-</p>
<p></p>
<p></p>
<p></p>
<p></p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Shewale, Bhagyashri<br>
<b>Sent:</b> Wednesday, June 12, 2019 6:10:04 PM<br>
<b>To:</b> openstack-discuss@lists.openstack.org; openstack@fried.cc; smooney@redhat.com; sfinucan@redhat.com; jaypipes@gmail.com<br>
<b>Subject:</b> [nova] Spec: Standardize CPU resource tracking</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="">
<p style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,Helvetica,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:12pt; margin-top:0px; margin-bottom:0px">
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Hi All,</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Currently I am working on implementation of cpu pinning upgrade part as mentioned in the spec [1].</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">While implementing the scheduler pre-filter as mentioned in [1], I have encountered one big issue:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Proposed change in spec: In scheduler pre-filter we are going to alias request_spec.flavor.extra_spec and request_spec.image.properties form ``hw:cpu_policy`` to ``resources=(V|P)CPU:${flavor.vcpus}``
of existing instances.<span class="x_Apple-converted-space"> </span></p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">So when user will create a new instance<span class="x_Apple-converted-space">
</span>or execute instance actions like shelve, unshelve, resize, evacuate and migration<span class="x_Apple-converted-space">
</span>post upgrade it will go through scheduler pre-filter which will set alias for `hw:cpu_policy` in request_spec flavor ``extra specs`` and image metadata properties. In below particular case, it won’t work:-</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">For example:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">I have two compute nodes say A and B:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">On Stein:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Compute node A configurations:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">vcpu_pin_set=0-3 (used as dedicated CPU, This host is added in aggregate which has “pinned” metadata)</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Compute node B Configuration:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">vcpu_pin_set=0-3 (used as dedicated CPU, This host is added in aggregate which has “pinned” metadata)</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">On Train, two possible scenarios:</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Compute node A configurations: (Consider the new cpu pinning implementation is merged into Train)</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">vcpu_pin_set=0-3<span class="x_Apple-converted-space">
</span>(Keep same settings as in Stein)</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Compute node B Configuration: (Consider the new cpu pinning implementation is merged into Train)</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">cpu_dedicated_set=0-3 (change to the new config option)</p>
<ol style="list-style-type:decimal">
<li style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px"><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px">Consider that one instance say `test ` is created using flavor having old extra specs (hw:cpu_policy=dedicated,
"aggregate_instance_extra_specs:pinned": "true") in Stein release and now upgraded Nova to Train with the above configuration.</span></li><li style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px"><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px"><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px">Now when user will perform</span><span class="x_Apple-converted-space" style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px"> </span><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px">instance
action say shelve/unshelve scheduler pre-filter will change the request_spec flavor extra spec from ``hw:cpu_policy`` to </span><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px; white-space:pre"></span><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px; white-space:pre"></span><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px">``resources=PCPU:$<no.
of cpus>`` which ultimately will return only compute node B from placement service. Here, we expect it should have retuned both </span><span style="color:rgb(69,69,69); font-family:"Helvetica Neue"; font-size:12px">Compute A and Compute B.</span></span></li><li style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">If user creates a new instance using old extra specs (hw:cpu_policy=dedicated, "aggregate_instance_extra_specs:pinned": "true") on Train release<span class="x_Apple-converted-space"> </span>with
the above configuration then it will return only compute node B from placement service where as it should have returned both compute Node A and B.</li></ol>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px"><span style="white-space:pre"></span></p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Problem: As Compute node A is still configured to be used to boot instances with dedicated CPUs same behavior as Stein, it will not be returned by placement service due to the changes in
the scheduler pre-filter logic.</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Propose changes:<span class="x_Apple-converted-space"> </span></p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Earlier in the spec [2]: The online data migration was proposed to change flavor extra specs and image metadata properties of request_spec and instance object. Based on the instance host,
we can get the NumaTopology of the host which will contain the new configuration options set on the compute host. Based on the NumaTopology of host, we can change instance and request_spec flavor extra specs.</p>
<ol style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,Helvetica,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:12pt; list-style-type:decimal">
<li style="margin:0px; font:12px "Helvetica Neue"; color:rgb(69,69,69)">Remove cpu_policy from extra specs</li><li style="margin:0px; font:12px "Helvetica Neue"; color:rgb(69,69,69)">Add “resources:PCPU=<count>” in extra specs</li></ol>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">We can also change the flavor extra specs and image metadata properties of instance and request_spec object using the reshape functionality.</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px">Please give us your feedback on the proposed solution so that we can update specs accordingly.</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
[1]: <a href="https://review.opendev.org/#/c/555081/28/specs/train/approved/cpu-resources.rst@451" class="x_OWAAutoLink" id="LPlnk830191">https://review.opendev.org/#/c/555081/28/specs/train/approved/cpu-resources.rst@451</a> <span style="text-decoration-line:underline"><span class="x_Apple-converted-space"></span></span></p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
[2]: <a href="https://review.opendev.org/#/c/555081/23..28/specs/train/approved/cpu-resources.rst" class="x_OWAAutoLink" id="LPlnk469968">https://review.opendev.org/#/c/555081/23..28/specs/train/approved/cpu-resources.rst</a> </p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
<br>
</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
Thanks and Regards,</p>
<p style="color:rgb(69,69,69); font:12px "Helvetica Neue"; margin:0px; min-height:14px">
-Bhagyashri Shewale-</p>
<p style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,Helvetica,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:12pt">
</p>
<p style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,Helvetica,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:12pt">
</p>
</div>
</div>
Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying
promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.
</body>
</html>