<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">I’m following this document to setup CPU pinning on Rocky:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">https://www.redhat.com/en/blog/driving-fast-lane-cpu-pinning-and-numa-topology-awareness-openstack-compute<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I followed all of the steps except for modifying non-pinned flavors and I have one aggregate containing a single NUMA-capable host:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">root@us01odc-dev1-ctrl1:/var/log/nova# os aggregate list<o:p></o:p></p>
<p class="MsoNormal">+----+-------+-------------------+<o:p></o:p></p>
<p class="MsoNormal">| ID | Name | Availability Zone |<o:p></o:p></p>
<p class="MsoNormal">+----+-------+-------------------+<o:p></o:p></p>
<p class="MsoNormal">| 4 | perf3 | None |<o:p></o:p></p>
<p class="MsoNormal">+----+-------+-------------------+<o:p></o:p></p>
<p class="MsoNormal">root@us01odc-dev1-ctrl1:/var/log/nova# os aggregate show 4<o:p></o:p></p>
<p class="MsoNormal">+-------------------+----------------------------+<o:p></o:p></p>
<p class="MsoNormal">| Field | Value |<o:p></o:p></p>
<p class="MsoNormal">+-------------------+----------------------------+<o:p></o:p></p>
<p class="MsoNormal">| availability_zone | None |<o:p></o:p></p>
<p class="MsoNormal">| created_at | 2019-10-30T23:05:41.000000 |<o:p></o:p></p>
<p class="MsoNormal">| deleted | False |<o:p></o:p></p>
<p class="MsoNormal">| deleted_at | None |<o:p></o:p></p>
<p class="MsoNormal">| hosts | [u'us01odc-dev1-hv003'] |<o:p></o:p></p>
<p class="MsoNormal">| id | 4 |<o:p></o:p></p>
<p class="MsoNormal">| name | perf3 |<o:p></o:p></p>
<p class="MsoNormal">| properties | pinned='true' |<o:p></o:p></p>
<p class="MsoNormal">| updated_at | None |<o:p></o:p></p>
<p class="MsoNormal">+-------------------+----------------------------+<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a flavor with the NUMA properties:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">root@us01odc-dev1-ctrl1:/var/log/nova# os flavor show s1.perf3<o:p></o:p></p>
<p class="MsoNormal">+----------------------------+-------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">| Field | Value |<o:p></o:p></p>
<p class="MsoNormal">+----------------------------+-------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">| OS-FLV-DISABLED:disabled | False |<o:p></o:p></p>
<p class="MsoNormal">| OS-FLV-EXT-DATA:ephemeral | 0 |<o:p></o:p></p>
<p class="MsoNormal">| access_project_ids | None |<o:p></o:p></p>
<p class="MsoNormal">| disk | 35 |<o:p></o:p></p>
<p class="MsoNormal">| id | be3d21c4-7e91-42a2-b832-47f42fdd3907 |<o:p></o:p></p>
<p class="MsoNormal">| name | s1.perf3 |<o:p></o:p></p>
<p class="MsoNormal">| os-flavor-access:is_public | True |<o:p></o:p></p>
<p class="MsoNormal">| properties | aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated' |<o:p></o:p></p>
<p class="MsoNormal">| ram | 30720 |<o:p></o:p></p>
<p class="MsoNormal">| rxtx_factor | 1.0 |<o:p></o:p></p>
<p class="MsoNormal">| swap | 7168 |<o:p></o:p></p>
<p class="MsoNormal">| vcpus | 4 |<o:p></o:p></p>
<p class="MsoNormal">+----------------------------+-------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I create a VM with that flavor: <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">openstack server create --flavor s1.perf3 --image NOT-QSC-CentOS6.10-19P1-v4 --network it-network alberttest4<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">but it goes to error status, and I see this in the logs:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2019-10-30 16:17:55.590 3248800 INFO nova.virt.hardware [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Computed NUMA topology CPU pinning:
usable pCPUs: [[4], [5], [6], [7]], vCPUs mapping: [(0, 4), (1, 5), (2, 6), (3, 7)]<o:p></o:p></p>
<p class="MsoNormal">2019-10-30 16:17:55.595 3248800 INFO nova.virt.hardware [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Computed NUMA topology CPU pinning:
usable pCPUs: [[0], [1], [2], [3], [4], [5], [6], [7]], vCPUs mapping: [(0, 0), (1, 1), (2, 2), (3, 3)]<o:p></o:p></p>
<p class="MsoNormal">2019-10-30 16:17:55.595 3248800 INFO nova.filters [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filter AggregateInstanceExtraSpecsFilter
returned 0 hosts<o:p></o:p></p>
<p class="MsoNormal">2019-10-30 16:17:55.596 3248800 INFO nova.filters [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filtering removed all hosts for the
request with instance ID '73b1e584-0ce4-478c-a706-c5892609dc3f'. Filter results: ['RetryFilter: (start: 3, end: 3)', 'AvailabilityZoneFilter: (start: 3, end: 3)', 'CoreFilter: (start: 3, end: 2)', 'RamFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2,
end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'ServerGroupAntiAffinityFilter: (start: 2, end: 2)', 'ServerGroupAffinityFilter: (start: 2, end: 2)', 'DifferentHostFilter: (start: 2, end: 2)', 'SameHostFilter:
(start: 2, end: 2)', 'NUMATopologyFilter: (start: 2, end: 2)', 'AggregateInstanceExtraSpecsFilter: (start: 2, end: 0)']<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It looks like my hypervisor is passing the hw:cpu_policy='dedicated' requirement but it is failing on “pinned=true”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The interesting part of the problem is that if I add a second apparently identical hypervisor to the aggregate it starts working. I create s1.perf3 VMs and they land on us01odc-dev1-hv002 and the XML shows that they are correctly pinned.
When us01odc-dev1-hv002 is full then they start failing again.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What should I be looking for here? What could cause one apparently identical hypervisor to fail AggregateInstanceExtraSpecsFilter while another one passes?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the nova-compute log of the failing hypervisor I see this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2019-10-31 10:43:01.147 1103 INFO nova.compute.resource_tracker [req-dda65a9c-9d0a-4888-b4cb-0bf4423dd2f3 - - - - -] Instance 4856d505-c220-4873-b881-836b5b75f7bb has allocations against this compute host but is not found in the database.<o:p></o:p></p>
<p class="MsoNormal">2019-10-31 10:43:01.148 1103 INFO nova.compute.resource_tracker [req-dda65a9c-9d0a-4888-b4cb-0bf4423dd2f3 - - - - -] Final resource view: name=us01odc-dev1-hv003.internal.synopsys.com phys_ram=128888MB used_ram=38912MB phys_disk=1208GB
used_disk=297GB total_vcpus=8 used_vcpus=6 pci_stats=[]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Openstack can’t find a VM with UUID 4856d505-c220-4873-b881-836b5b75f7bb. There are no VMs on hv003 but I can create a non-pinned VM there and it works. Do I have a “phantom” VM that is consuming resources on hv003? How can I fix that?<o:p></o:p></p>
</div>
</body>
</html>