<div dir="ltr"><span id="gmail-:36.co" class="gmail-tL8wMe gmail-EMoHub" style="text-align:left" dir="ltr">controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status<br>
Password: <br>
Cluster status of node rabbit@localhost ...<br>
[{nodes,[{disc,[rabbit@localhost]}]},<br>
{running_nodes,[rabbit@localhost]},<br>
{cluster_name,<<"rabbit@controller-0">>},<br>
{partitions,[]},<br>
{alarms,[{rabbit@localhost,[]}]}]</span></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com">C-Albert.Braden@charter.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_3324821667827655833WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>>
<br>
<b>Sent:</b> Thursday, June 17, 2021 9:58 AM<br>
<b>To:</b> Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>><br>
<b>Cc:</b> openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background:rgb(35,92,112) none repeat scroll 0% 0%">
<strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">Hi Albert,</span><span style="color:black"><u></u><u></u></span></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">Sorry for the inconvenience.</span><span style="color:black"><u></u><u></u></span></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.</span><span style="color:black"><u></u><u></u></span></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">But I still have the same issue. Please find scheduler and conductor logs.</span><span style="color:black"><u></u><u></u></span></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%"># Scheduler Logs</span><span style="color:black"><u></u><u></u></span></pre>
</div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"><a href="http://paste.openstack.org/show/806729/" target="_blank">http://paste.openstack.org/show/806729/</a></span><u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">## Conductor logs</span><span style="color:black"><u></u><u></u></span></pre>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"><a href="http://paste.openstack.org/show/806727/" target="_blank">http://paste.openstack.org/show/806727/</a></span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">On Thu, Jun 17, 2021 at 6:55 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>> wrote:</span><u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%">Both links show the conductor log. It tells us that no valid host was found. The scheduler
log should show why no valid host was found.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">
open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>>
<br>
<b>Sent:</b> Thursday, June 17, 2021 2:02 AM<br>
<b>To:</b> Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>><br>
<b>Cc:</b> openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs</span><u></u><u></u></p>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background-size:auto;background-origin:padding-box;background-clip:border-box">
<b><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white;background:white none repeat scroll 0% 0%">CAUTION:</span></b><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white;background:white none repeat scroll 0% 0%"> The e-mail below is from an external source. Please
exercise caution before opening attachments, clicking links, or following guidance.
</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">Found nova-conductor and nova-scheduler logs [1],[2].</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">But I have no idea about what caused this issue.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">[1]
<a href="http://paste.openstack.org/show/806720/" target="_blank">http://paste.openstack.org/show/806720/</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">[2]
<a href="http://paste.openstack.org/show/806721/" target="_blank">http://paste.openstack.org/show/806721/</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(136,136,136);background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">On Wed, Jun 16, 2021 at 8:08 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>> wrote:</span><u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%">Someone will need to dig through the starlingx documentation and figure out where
the log files are located. Do you want to do that?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%"><a href="https://docs.starlingx.io/" target="_blank">https://docs.starlingx.io/</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">
open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, June 16, 2021 9:12 AM<br>
<b>To:</b> Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>><br>
<b>Cc:</b> openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs</span><u></u><u></u></p>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background-size:auto;background-origin:padding-box;background-clip:border-box">
<b><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white;background:white none repeat scroll 0% 0%">CAUTION:</span></b><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white;background:white none repeat scroll 0% 0%"> The e-mail below is from an external source. Please
exercise caution before opening attachments, clicking links, or following guidance.
</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">I can see sm-scheduler.log not sure if its the correct log file you mentioned, but I can't see conductor log.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">This environment have been deployed using starlingx.</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">On Wed, Jun 16, 2021 at 6:16 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>> wrote:</span><u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%">Can you check the scheduler and conductor logs on the controllers? There should be
entries describing why the instance failed to schedule. You may need to set “debug=true” in nova.conf to get more details.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125);background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">
open infra <</span><span style="background:white none repeat scroll 0% 0%"><a href="mailto:openinfradn@gmail.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">openinfradn@gmail.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">>
<br>
<b>Sent:</b> Wednesday, June 16, 2021 1:36 AM<br>
<b>To:</b> openstack-discuss <</span><span style="background:white none repeat scroll 0% 0%"><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">openstack-discuss@lists.openstack.org</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif;background:white none repeat scroll 0% 0%">><br>
<b>Subject:</b> [EXTERNAL] Error creating VMs</span><u></u><u></u></p>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background-size:auto;background-origin:padding-box;background-clip:border-box">
<b><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white;background:white none repeat scroll 0% 0%">CAUTION:</span></b><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white;background:white none repeat scroll 0% 0%"> The e-mail below is from an external source. Please
exercise caution before opening attachments, clicking links, or following guidance.
</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">Hi</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">After setting up OpenStack environment using STX R5 (two controllers, two storage nodes and one worker), I have deployed a VM. VM ended up with ERROR
status.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">I highly appreciate if someone can guide to dig further (what logs to check ) or to fix this issue.</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"><a href="http://paste.openstack.org/show/806626/" target="_blank">http://paste.openstack.org/show/806626/</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">Regards</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">Danishka</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%"> </span><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">The contents of this e-mail message and
<br>
any attachments are intended solely for the <br>
addressee(s) and may contain confidential <br>
and/or legally privileged information. If you<br>
are not the intended recipient of this message<br>
or if this message has been addressed to you <br>
in error, please immediately alert the sender<br>
by reply e-mail and then delete this message <br>
and any attachments. If you are not the <br>
intended recipient, you are notified that <br>
any use, dissemination, distribution, copying,<br>
or storage of this message or any attachment <br>
is strictly prohibited.</span><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">The contents of this e-mail message and
<br>
any attachments are intended solely for the <br>
addressee(s) and may contain confidential <br>
and/or legally privileged information. If you<br>
are not the intended recipient of this message<br>
or if this message has been addressed to you <br>
in error, please immediately alert the sender<br>
by reply e-mail and then delete this message <br>
and any attachments. If you are not the <br>
intended recipient, you are notified that <br>
any use, dissemination, distribution, copying,<br>
or storage of this message or any attachment <br>
is strictly prohibited.</span><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal"><span style="background:white none repeat scroll 0% 0%">The contents of this e-mail message and
<br>
any attachments are intended solely for the <br>
addressee(s) and may contain confidential <br>
and/or legally privileged information. If you<br>
are not the intended recipient of this message<br>
or if this message has been addressed to you <br>
in error, please immediately alert the sender<br>
by reply e-mail and then delete this message <br>
and any attachments. If you are not the <br>
intended recipient, you are notified that <br>
any use, dissemination, distribution, copying,<br>
or storage of this message or any attachment <br>
is strictly prohibited.</span><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
The contents of this e-mail message and <br>any attachments are intended solely for the <br>addressee(s) and may contain confidential <br>and/or legally privileged information. If you<br>are not the intended recipient of this message<br>or if this message has been addressed to you <br>in error, please immediately alert the sender<br>by reply e-mail and then delete this message <br>and any attachments. If you are not the <br>intended recipient, you are notified that <br>any use, dissemination, distribution, copying,<br>or storage of this message or any attachment <br>is strictly prohibited.</div>
</blockquote></div>