<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here:
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-from-centos-7-to-centos-8">https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-from-centos-7-to-centos-8</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Everything seems to work fine until I try to deploy the first replacement controller into the cluster. I upgrade RMQ, ES and Kibana, then follow the “remove existing controller” process to remove control0, create a Centos 8 VM, bootstrap
the cluster, pull containers to the new control0, and everything is still working. Then I type the last command “kolla-ansible -i multinode deploy --limit control”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The RMQ install works and I see all 3 nodes up in the RMQ admin, but it takes a long time to complete “TASK [service-ks-register : masakari | Creating users] “ and then hangs on “TASK [service-ks-register : masakari | Creating roles]”.
At this time the new control0 becomes unreachable and drops out of the RMQ cluster. I can still ping it but console hangs along with new and existing ssh sessions. It appears that the CPU may be maxed out and not allowing interrupts. Eventually I see error
“fatal: [control0]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">RMQ seems fine on the 2 old controllers; they just don’t see the new control0 active:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">(rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status<o:p></o:p></p>
<p class="MsoNormal">Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ...<o:p></o:p></p>
<p class="MsoNormal">[{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0',<o:p></o:p></p>
<p class="MsoNormal"> 'rabbit@chrnc-void-testupgrade-control-0-replace',<o:p></o:p></p>
<p class="MsoNormal"> 'rabbit@chrnc-void-testupgrade-control-1',<o:p></o:p></p>
<p class="MsoNormal"> 'rabbit@chrnc-void-testupgrade-control-2']}]},<o:p></o:p></p>
<p class="MsoNormal">{running_nodes,['rabbit@chrnc-void-testupgrade-control-1',<o:p></o:p></p>
<p class="MsoNormal"> 'rabbit@chrnc-void-testupgrade-control-2']},<o:p></o:p></p>
<p class="MsoNormal">{cluster_name,<<"rabbit@chrnc-void-testupgrade-control-0.dev.chtrse.com">>},<o:p></o:p></p>
<p class="MsoNormal">{partitions,[]},<o:p></o:p></p>
<p class="MsoNormal">{alarms,[{'rabbit@chrnc-void-testupgrade-control-1',[]},<o:p></o:p></p>
<p class="MsoNormal"> {'rabbit@chrnc-void-testupgrade-control-2',[]}]}]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After this the HAProxy IP is pingable but openstack commands are failing:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">(openstack) [root@chrnc-void-testupgrade-build openstack]# osl<o:p></o:p></p>
<p class="MsoNormal">Failed to discover available identity versions when contacting http://172.16.0.100:35357/v3. Attempting to parse version from URL.<o:p></o:p></p>
<p class="MsoNormal">Gateway Timeout (HTTP 504)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After about an hour my open ssh session on the new control0 responded and confirmed that the CPU is maxed out:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@chrnc-void-testupgrade-control-0-replace /]# uptime<o:p></o:p></p>
<p class="MsoNormal">17:41:55 up 1:55, 0 users, load average: 157.87, 299.75, 388.69<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I built new heat stacks and tried it a few times, and it consistently fails on masakari. Do I need to change something in my masakari config before upgrading Train from Centos 7 to Centos 8?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
The contents of this e-mail message and <br>any attachments are intended solely for the <br>addressee(s) and may contain confidential <br>and/or legally privileged information. If you<br>are not the intended recipient of this message<br>or if this message has been addressed to you <br>in error, please immediately alert the sender<br>by reply e-mail and then delete this message <br>and any attachments. If you are not the <br>intended recipient, you are notified that <br>any use, dissemination, distribution, copying,<br>or storage of this message or any attachment <br>is strictly prohibited.</body>
</html>