<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">The other symptom is that the scheduler will send single VMs to a full hypervisor and overload it even though we have cpu_allocation_ratio and ram_allocation_ratio set to 1:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">root@us01odc-dev1-ctrl1:~# os hypervisor list --long<o:p></o:p></p>
<p class="MsoNormal">+----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+<o:p></o:p></p>
<p class="MsoNormal">| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |<o:p></o:p></p>
<p class="MsoNormal">+----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+<o:p></o:p></p>
<p class="MsoNormal">| 1 | us01odc-dev1-hv003.internal.synopsys.com | QEMU | 10.195.116.16 | up | 42 | 16 | 161792 | 128888 |<o:p></o:p></p>
<p class="MsoNormal">| 3 | us01odc-dev1-hv002.internal.synopsys.com | QEMU | 10.195.116.15 | up | 43 | 16 | 165888 | 128888 |<o:p></o:p></p>
<p class="MsoNormal">| 4 | us01odc-dev1-hv001.internal.synopsys.com | QEMU | 10.195.116.14 | up | 38 | 16 | 161792 | 128888 |<o:p></o:p></p>
<p class="MsoNormal">+----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the logs I see the scheduler returning 1 host:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.930 895454 DEBUG nova.filters [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filter
NUMATopologyFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It weighs the host and reports negative RAM:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.930 895454 DEBUG nova.scheduler.filter_scheduler [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81]
Weighed [WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 683008MB io_ops: 1 instances: 5, weight: 0.0]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Then it selects that host:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.931 895454 DEBUG nova.scheduler.utils [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81]
Attempting to claim resources in the placement API for instance 834dd112-26f8-424c-a92b-23423baa185a claim_resources /usr/lib/python2.7/dist-packages/nova/scheduler/utils.py:935<o:p></o:p></p>
<p class="MsoNormal">/var/log/nova/nova-scheduler.log:2019-11-19 16:38:21.567 895454 DEBUG nova.scheduler.filter_scheduler [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81]
[instance: 834dd112-26f8-424c-a92b-23423baa185a] Selected host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 683008MB io_ops: 1 instances: 5 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The VM builds successfully and goes to ACTIVE.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What should I be looking for here? Obviously I broke the scheduler, but my nova config is the same as the working cluster.<o:p></o:p></p>
</div>
</body>
</html>