Open Stack

The other symptom is that the scheduler will send single VMs to a full hypervisor and overload it even though we have cpu_allocation_ratio and ram_allocation_ratio set to 1:

root at us01odc-dev1-ctrl1:~# os hypervisor list --long
+----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+
| ID | Hypervisor Hostname                      | Hypervisor Type | Host IP       | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |
+----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+
|  1 | us01odc-dev1-hv003.internal.synopsys.com | QEMU            | 10.195.116.16 | up    |         42 |    16 |         161792 |    128888 |
|  3 | us01odc-dev1-hv002.internal.synopsys.com | QEMU            | 10.195.116.15 | up    |         43 |    16 |         165888 |    128888 |
|  4 | us01odc-dev1-hv001.internal.synopsys.com | QEMU            | 10.195.116.14 | up    |         38 |    16 |         161792 |    128888 |
+----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+

In the logs I see the scheduler returning 1 host:

/var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.930 895454 DEBUG nova.filters [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filter NUMATopologyFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104

It weighs the host and reports negative RAM:

/var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.930 895454 DEBUG nova.scheduler.filter_scheduler [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Weighed [WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 683008MB io_ops: 1 instances: 5, weight: 0.0]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454

Then it selects that host:

/var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.931 895454 DEBUG nova.scheduler.utils [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Attempting to claim resources in the placement API for instance 834dd112-26f8-424c-a92b-23423baa185a claim_resources /usr/lib/python2.7/dist-packages/nova/scheduler/utils.py:935
/var/log/nova/nova-scheduler.log:2019-11-19 16:38:21.567 895454 DEBUG nova.scheduler.filter_scheduler [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: 834dd112-26f8-424c-a92b-23423baa185a] Selected host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 683008MB io_ops: 1 instances: 5 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346

The VM builds successfully and goes to ACTIVE.

What should I be looking for here? Obviously I broke the scheduler, but my nova config is the same as the working cluster.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20191120/284b27ed/attachment.html>