<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Yes, this is almost exactly what I did. No, I am not running mysql in a HA deployment and I have ran nova-manage api_db sync several times throughout the process below.
<div class=""><br class="">
</div>
<div class=""> I think I found a work around but im not sure how feasible this is.
<div class=""><br class="">
</div>
<div class="">I first, changed the reallocation ratio to 1:1. In the nova.conf on the controller. Nova would not accept this for some reason and seemed like it needed to be changed on the compute node. So I deleted the hypervisor, resource provider, and compute
service. Changed the ratios on the compute node itself, and then re-added it back in. Now the capacity changed to 64 which is the number of cores on the systems. When starting a vm, it still gets the same number for “used” in the placement-api.log: See below: </div>
<div class=""><br class="">
</div>
<div class="">New ratios</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">~# openstack resource provider inventory list 554f2a3b-924e-440c-9847-596064ea0f3f<br style="box-sizing: inherit;" class="">+----------------+------------------+----------+----------+----------+-----------+--------+<br style="box-sizing: inherit;" class="">| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total |<br style="box-sizing: inherit;" class="">+----------------+------------------+----------+----------+----------+-----------+--------+<br style="box-sizing: inherit;" class="">| VCPU | 1.0 | 1 | 64 | 0 | 1 | 64 |<br style="box-sizing: inherit;" class="">| MEMORY_MB | 1.0 | 1 | 515655 | 512 | 1 | 515655 |<br style="box-sizing: inherit;" class="">| DISK_GB | 1.0 | 1 | 7096 | 0 | 1 | 7096 |<br style="box-sizing: inherit;" class="">+----------------+------------------+----------+----------+----------+-----------+--------+</pre>
<div class=""><br class="">
</div>
</div>
<div class="">Error from placement.log</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">2021-06-30 13:49:24.877 4381 WARNING placement.objects.allocation [req-7dc8930f-1eac-401a-ade7-af36e64c2ba8 a770bde56c9d49e68facb792cf69088c 6da06417e0004cbb87c1e64fe1978de5 - default default] Over capacity for VCPU on resource provider c4199e84-8259-4d0e-9361-9b0d9e6e66b7. Needed: 4, Used: 8206, Capacity: 64.0</pre>
<div class=""><br class="">
</div>
<div class="">With that in mind, I did the same procedure again but set the ratio to 1024</div>
<div class=""><br class="">
</div>
<div class="">New ratios</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">~# openstack resource provider inventory list 519c1e10-3546-4e3b-a017-3e831376cde8<br style="box-sizing: inherit;" class="">+----------------+------------------+----------+----------+----------+-----------+--------+<br style="box-sizing: inherit;" class="">| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total |<br style="box-sizing: inherit;" class="">+----------------+------------------+----------+----------+----------+-----------+--------+<br style="box-sizing: inherit;" class="">| VCPU | 1024.0 | 1 | 64 | 0 | 1 | 64 |<br style="box-sizing: inherit;" class="">| MEMORY_MB | 1.0 | 1 | 515655 | 512 | 1 | 515655 |<br style="box-sizing: inherit;" class="">| DISK_GB | 1.0 | 1 | 7096 | 0 | 1 | 7096 |<br style="box-sizing: inherit;" class="">+----------------+------------------+----------+----------+----------+-----------+--------+</pre>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
</div>
<div class="">Now I can spin up vms without issues. </div>
<div class=""><br class="">
</div>
<div class="">I have 1 test AZ with 2 hosts inside. I have set these hosts to the ratio above. I was able to spin up approx 45 4x core VMs without issues and no signs of it hitting an upper limit on the host. </div>
<div class=""><br class="">
</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">120 | VCPU=64 | 519c1e10-3546-4e3b-a017-3e831376cde8 | VCPU=88/65536<br style="box-sizing: inherit;" class="">23 | VCPU=64 | 8f97a3ba-98a0-475e-a3cf-41425569b2cb | VCPU=96/65536</pre>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
</div>
<div class="">I have 2 problems with this fix. </div>
<div class=""><br class="">
</div>
<div class="">1) the overcommit is now super high and I have no way, besides quotas, to guarantee the system won’t be over provisioned. </div>
<div class="">2) I still don’t know how that “used” resources value is being calculated. When this issue first started, the “used” resources were a different number. Over the past two days, the used resources for a 4 core virtual machine have remained at 8206
but I have no way to guarantee this. </div>
<div class=""><br class="">
</div>
<div class="">My initial tests when this started was to compare the resource values when building different size vms. Here is that list: </div>
<div class=""><br class="">
</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">1 core - 4107<br style="box-sizing: inherit;" class="">2 core - 4108<br style="box-sizing: inherit;" class="">4 core- 4110<br style="box-sizing: inherit;" class="">8 core - 4114<br style="box-sizing: inherit;" class="">16 core - 4122<br style="box-sizing: inherit;" class="">32 core - 8234</pre>
<div class=""><br class="">
</div>
</div>
<div class="">The number on the right is the number the “used” value used to be. Yesterday and today, it has changed to 8206 for a 4 core vm, I have not tested the rest. </div>
<div class=""><br class="">
</div>
<div class="">Before I commit to combing through the placement api source code to figure out how the “used” value in the placement log is being calculated, im hoping someone knows where and how that value is being calculated. It does not seem to be a fixed
value in the database and it doesn’t seem to be effected by the allocation ratios. </div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thank you in advance!!</div>
<div class="">
<div dir="auto" style="text-align: start; text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>-Jeff Mazzone<br class="">
Senior Linux Systems Administrator<br class="">
Center for Translational Data Science<br class="">
University of Chicago.</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 30, 2021, at 2:40 PM, Laurent Dumont <<a href="mailto:laurentfdumont@gmail.com" class="">laurentfdumont@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">In
some cases, the DEBUG messages are a bit verbose but can really walk you through the allocation/scheduling process. You could increase it for nova and restart the api + scheduler on the controllers. I wonder if a desync of the DB could be in cause? Are you
running an HA deployment for the mysql backend?</span></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>