<div dir="ltr">I'm curious to see if I can reproduce the issue in my test-env. I never tried puppet-openstack so might as well see how it goes!<div><br></div><div>The ServerFault issue mentions the puppet-openstack integration being used to deploy Ussuri? Specifically, the puppet modules being at the 17.4 version?</div><div><br></div><div>But looking at <a href="https://docs.openstack.org/puppet-openstack-guide/latest/install/releases.html">https://docs.openstack.org/puppet-openstack-guide/latest/install/releases.html</a> - the modules for Ussuri should be at 16.x? Could it be some kind of weird setup of the deployment modules for Ussuri/placement that didn't go as planned? </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 30, 2021 at 9:13 PM Jeffrey Mazzone <<a href="mailto:jmazzone@uchicago.edu">jmazzone@uchicago.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
<blockquote type="cite">
<div>On Jun 30, 2021, at 5:06 PM, melanie witt <<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a>> wrote:</div>
<br>
</blockquote>
<font color="#4d22b4">I suggest you run the 'openstack resource provider show <RP UUID> --allocations' command as Balazs mentioned earlier to show all of the allocations (used resources) on the compute node. I also suggest you run the 'nova-manage
placement audit' tool [1] as Sylvain mentioned earlier to show whether there are any orphaned allocations, i.e. allocations that are for instances that no longer exist. The consumer UUID is the instance UUID.</font>
<div><font color="#4d22b4"><span><br>
</span></font></div>
<div>I did both of those suggestions. "openstack resource provider show <RP UUID> —allocations" shows what is expected. No additional orphaned vms and the resources used is correct. Here is an example of a different set of hosts and zones. This host
had 2x 16 core vms on it before the cluster went into this state. You can see them both below. The nova-manage audit commands do not show any orphans either. </div>
<div><br>
</div>
<div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;word-break:normal;border-radius:4px"><font color="#1d1c1d" face="Monaco, Menlo, Consolas, Courier New, monospace"><span style="white-space:pre-wrap">~# openstack resource provider show 41ecee2a-ec24-48e5-8b9d-24065d67238a --allocations
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| uuid | 41ecee2a-ec24-48e5-8b9d-24065d67238a |
| name | kh09-56 |
| generation | 55 |
| root_provider_uuid | 41ecee2a-ec24-48e5-8b9d-24065d67238a |
| parent_provider_uuid | None |
| allocations | {'d6b9d19c-1ba9-44c2-97ab-90098509b872': {'resources': {'DISK_GB': 50, 'MEMORY_MB': 16384, 'VCPU': 16}, 'consumer_generation': 1}, 'e0a8401a-0bb6-4612-a496-6a794ebe6cd0': {'resources': {'DISK_GB': 50, 'MEMORY_MB': 16384, 'VCPU': 16}, 'consumer_generation': 1}} |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+</span></font></pre>
</div>
<div><br>
</div>
<div>Usage on the resource provider: </div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">~# openstack resource provider usage show 41ecee2a-ec24-48e5-8b9d-24065d67238a<br style="box-sizing:inherit">+----------------+-------+<br style="box-sizing:inherit">| resource_class | usage |<br style="box-sizing:inherit">+----------------+-------+<br style="box-sizing:inherit">| VCPU | 32 |<br style="box-sizing:inherit">| MEMORY_MB | 32768 |<br style="box-sizing:inherit">| DISK_GB | 100 |<br style="box-sizing:inherit">+----------------+-------+</pre>
<div><br>
</div>
</div>
<div>All of that looks correct. Requesting it to check allocations for a 4 VCPU vm also shows it as a candidate: </div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">~# openstack allocation candidate list --resource VCPU=4 | grep 41ecee2a-ec24-48e5-8b9d-24065d67238a<br style="box-sizing:inherit">| 41 | VCPU=4 | 41ecee2a-ec24-48e5-8b9d-24065d67238a | VCPU=32/1024,MEMORY_MB=32768/772714,DISK_GB=100/7096</pre>
<div><br>
</div>
</div>
<div>In the placement database, under the used column, also shows the correct values for the information provided above with those 2 vms on it: </div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">+---------------------+------------+-------+----------------------+--------------------------------------+-------------------+-------+<br style="box-sizing:inherit">| created_at | updated_at | id | resource_provider_id | consumer_id | resource_class_id | used |<br style="box-sizing:inherit">+---------------------+------------+-------+----------------------+--------------------------------------+-------------------+-------+<br style="box-sizing:inherit">| 2021-06-02 18:45:05 | NULL | 4060 | 125 | e0a8401a-0bb6-4612-a496-6a794ebe6cd0 | 2 | 50 |<br style="box-sizing:inherit">| 2021-06-02 18:45:05 | NULL | 4061 | 125 | e0a8401a-0bb6-4612-a496-6a794ebe6cd0 | 1 | 16384 |<br style="box-sizing:inherit">| 2021-06-02 18:45:05 | NULL | 4062 | 125 | e0a8401a-0bb6-4612-a496-6a794ebe6cd0 | 0 | 16 |<br style="box-sizing:inherit">| 2021-06-04 18:39:13 | NULL | 7654 | 125 | d6b9d19c-1ba9-44c2-97ab-90098509b872 | 2 | 50 |<br style="box-sizing:inherit">| 2021-06-04 18:39:13 | NULL | 7655 | 125 | d6b9d19c-1ba9-44c2-97ab-90098509b872 | 1 | 16384 |<br style="box-sizing:inherit">| 2021-06-04 18:39:13 | NULL | 7656 | 125 | d6b9d19c-1ba9-44c2-97ab-90098509b872 | 0 | 16 |</pre>
<div><br>
</div>
</div>
<div><br>
</div>
<div>Trying to build a vm though.. I get the placement error with the improperly calculated “Used” values. </div>
<div><br>
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">2021-06-30 19:51:39.732 43832 WARNING placement.objects.allocation [req-de225c66-8297-4b34-9380-26cf9385d658 a770bde56c9d49e68facb792cf69088c 6da06417e0004cbb87c1e64fe1978de5 - default default] Over capacity for VCPU on resource provider b749130c-a368-4332-8a1f-8411851b4b2a. Needed: 4, Used: 18509, Capacity: 1024.0</pre>
<div><br>
</div>
<div>Outside of changing the allocation ratio, im completely lost. Im confident it has to do with that improper calculation of the used value but how is it being calculated if it isn’t being added up from fixed values in the database as has been suggested? </div>
<div><br>
</div>
<div>Thanks in advance!</div>
<div>-Jeff M</div>
<div>
<div><br>
<br>
</div>
<div><br>
<blockquote type="cite">
<div><br>
</div>
<div><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">The
tl;dr on how the value is calculated is there's a table called 'allocations' in the placement database that holds all the values for resource providers and resource classes and it has a 'used' column. If you add up all of the 'used' values for a resource class
(VCPU) and resource provider (compute node) then that will be the total used of that resource on that resource provider. You can see this data by 'openstack resource provider show <RP UUID> --allocations' as well.</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">The
allocation ratio will not affect the value of 'used' but it will affect the working value of 'total' to be considered higher than it actually is in order to oversubscribe. If a compute node has 64 cores and cpu_allocation ratio is 16 then 64 * 16 = 1024 cores
will be allowed for placement on that compute node.</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">You
likely have "orphaned" allocations for the compute node/resource provider that are not mapped to instances any more and you can use 'nova-manage placement audit' to find those and optionally delete them. Doing that will cleanup your resource provider. First,
I would run it without specifying --delete just to see what it shows without modifying anything.</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote></div>