<div dir="ltr">In some cases, the DEBUG messages are a bit verbose but can really walk you through the allocation/scheduling process. You could increase it for nova and restart the api + scheduler on the controllers. I wonder if a desync of the DB could be in cause? Are you running an HA deployment for the mysql backend?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 30, 2021 at 1:44 PM Jeffrey Mazzone <<a href="mailto:jmazzone@uchicago.edu">jmazzone@uchicago.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
<div><font color="#be37f4">Any other logs with <span style="font-family:monospace"> </span><span style="font-family:monospace">Unable to create allocation for 'VCPU' on resource provider?</span></font></div>
<div><font color="#be37f4"><span style="font-family:monospace"><br>
</span></font></div>
<div>No, the 3 logs listed are the only logs where it is showing this message and VCPU is the only thing it fails for. No memory or disk allocation failures, always VCPU.<font face="monospace"> </font></div>
<font color="#be37f4">
<div><font color="#be37f4"><br>
</font></div>
<div><font color="#be37f4"><br>
</font></div>
At this point if you list the resource provider usage on 3f9d0deb-936c-474a-bdee-d3df049f073d again then do you still see 4 VCPU used, or 8206 used?</font>
<div><font color="#be37f4"><span><br>
</span></font></div>
<div><span>The usage shows everything correctly: </span></div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">~# openstack resource provider usage show 3f9d0deb-936c-474a-bdee-d3df049f073d<br style="box-sizing:inherit">+----------------+-------+<br style="box-sizing:inherit">| resource_class | usage |<br style="box-sizing:inherit">+----------------+-------+<br style="box-sizing:inherit">| VCPU | 4 |<br style="box-sizing:inherit">| MEMORY_MB | 8192 |<br style="box-sizing:inherit">| DISK_GB | 10 |<br style="box-sizing:inherit">+----------------+-------+</pre>
<div><br>
</div>
<div>Allocations shows the same: </div>
<div><br>
</div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">~# openstack resource provider show 3f9d0deb-936c-474a-bdee-d3df049f073d --allocations<br style="box-sizing:inherit">+-------------+--------------------------------------------------------------------------------------------------------+<br style="box-sizing:inherit">| Field | Value |<br style="box-sizing:inherit">+-------------+--------------------------------------------------------------------------------------------------------+<br style="box-sizing:inherit">| uuid | 3f9d0deb-936c-474a-bdee-d3df049f073d |<br style="box-sizing:inherit">| name | kh09-50 |<br style="box-sizing:inherit">| generation | 244 |<br style="box-sizing:inherit">| allocations | {'4a6fe4c2-ece4-45c2-b7a2-fdfd41308988': {'resources': {'VCPU': 4, 'MEMORY_MB': 8192, 'DISK_GB': 10}}} |<br style="box-sizing:inherit">+-------------+--------------------------------------------------------------------------------------------------------+</pre>
<div><br>
</div>
</div>
<div>Allocation candidate list shows all 228 servers in the cluster available: </div>
<div><br>
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">~# openstack allocation candidate list --resource VCPU=4 -c "resource provider" -f value | wc -l<br style="box-sizing:inherit">228</pre>
<div><br>
</div>
<div> Starting a new vm on that host shows the following in the logs: </div>
<div><br>
</div>
<div>Placement-api.log</div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">2021-06-30 12:27:21.335 4382 WARNING placement.objects.allocation [req-f4d74abc-7b18-407a-85e7-f1c268bd5e53 a770bde56c9d49e68facb792cf69088c 6da06417e0004cbb87c1e64fe1978de5 - default default] Over capacity for VCPU on resource provider 0e0d8ec8-bb31-4da5-a813-bd73560ff7d6. Needed: 4, Used: 8206, Capacity: 1024.0</pre>
<div><br>
</div>
</div>
<div>nova-scheduler.log</div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">2021-06-30 12:27:21.429 6895 WARNING nova.scheduler.client.report [req-3106f4da-1df9-4370-b56b-8ba6b62980dc aacc7911abf349b783eed20ad176c034 23920ecfbf294e71ad558aa49cb17de8 - default default] Failed to save allocation for a9296e22-4b50-45b7-a442-1fce0a844bcd. Got HTTP 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider '3f9d0deb-936c-474a-bdee-d3df049f073d'. The requested amount would exceed the capacity. ", "code": "placement.undefined_code", "request_id": "req-e9f12a3a-3136-4501-8bd6-4add31f0eb82"}]}</pre>
<div><br>
</div>
</div>
<div>I really can’t figure out where this, what’s seems to be last minute, calculation of used resources comes from. </div>
<div><br>
</div>
<div><font color="#be37f4">Given you also have an Ussuri deployment, you could call the nova-audit command to see whether you would have orphaned allocations :</font></div>
<div><dl><dt><code><font color="#be37f4"><span>nova-manage</span> <span>placement</span> <span>audit</span> <span>[--verbose]</span> <span>[--delete]</span> <span>[--resource_provider</span> <span><uuid>]</span></font></code></dt></dl></div>
<div><br>
</div>
<div>When running this command, it says the UUID does not exist. </div>
<div><br>
</div>
<div><br>
</div>
<div>Thank you! I truly appreciate everyones help. </div>
<div><br>
</div>
<div>-Jeff M</div>
<div>
<div><br>
</div>
<br>
</div>
</div>
</div>
</blockquote></div>