<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.gmail-tl8wme
{mso-style-name:gmail-tl8wme;}
span.gmail-m3855489329660706895gmail-tl8wme
{mso-style-name:gmail-m_3855489329660706895gmail-tl8wme;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I see “no valid host” at 10:48:39 in the conductor log:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">2021-06-17T10:48:39.65107847Z stdout F nova.exception.NoValidHost: No valid host was found.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In the scheduler log at 10:37:43 we see the scheduler starting and the RMQ error, followed by the scheduling failure at 10:48:39. It looks like the scheduler
can’t connect to RMQ.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">2021-06-17T10:37:43.038362995Z stdout F 2021-06-17 10:37:43.038 1 INFO nova.service [-] Starting scheduler node (version 21.2.1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">2021-06-17T10:37:43.08082925Z stdout F 2021-06-17 10:37:43.079 1 ERROR oslo.messaging._drivers.impl_rabbit [req-775b2b25-b9ef-4642-a6fd-7574eab7cc37 - - - - -] Connection
failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">2021-06-17T10:48:39.197318418Z stdout F 2021-06-17 10:48:39.196 1 INFO nova.scheduler.manager [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89
e8813293073545f99658adbec2f80c1d - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> open infra <openinfradn@gmail.com>
<br>
<b>Sent:</b> Thursday, June 17, 2021 12:38 PM<br>
<b>To:</b> Braden, Albert <C-Albert.Braden@charter.com><br>
<b>Cc:</b> openstack-discuss <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #5A5A5A 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#235C70">
<strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><o:p></o:p></p>
</div>
<div>
<div id="gmail-:3o.ma">
<p class="MsoNormal"><span class="gmail-tl8wme">Another instance created at 2021-06-17T16:10:06Z, and it also failed.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="gmail-tl8wme">I can't see rabbitmq errors around that timestamp.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div id="gmail-:4d.ma">
<p class="MsoNormal"><span class="gmail-tl8wme">scheduler log </span><a href="https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA" target="_blank">http://paste.openstack.org/show/806737/</a><o:p></o:p></p>
</div>
<div id="gmail-:4e.ma">
<p class="MsoNormal"><span class="gmail-tl8wme">conducter log </span><a href="https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ" target="_blank">http://paste.openstack.org/show/806738/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div id="gmail-:4f.ma">
<p class="MsoNormal"><span class="gmail-tl8wme">At 2021-06-17 15:52:38.810, system was able to connect to AMQP </span><o:p></o:p></p>
</div>
<div id="gmail-:4g.ma">
<p class="MsoNormal"><a href="https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g" target="_blank">http://paste.openstack.org/show/806739/</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="gmail-:4c.ma">
<p class="MsoNormal"><span class="gmail-tl8wme">controller-0:/var/log/pods$ sudo rabbitmqctl list_queues</span><br>
<span class="gmail-tl8wme">Password: </span><br>
<span class="gmail-tl8wme">Listing queues ...</span><br>
<span class="gmail-tl8wme">sysinv.ceph_manager.192.168.204.1 0</span><br>
<span class="gmail-tl8wme">sysinv.ceph_manager 0</span><br>
<span class="gmail-tl8wme">barbican.workers 0</span><br>
<span class="gmail-tl8wme">sysinv.fpga_agent_manager.controller-0 0</span><br>
<span class="gmail-tl8wme">barbican.workers.barbican.queue 0</span><br>
<span class="gmail-tl8wme">sysinv.conductor_manager 0</span><br>
<span class="gmail-tl8wme">sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0</span><br>
<span class="gmail-tl8wme">sysinv.agent_manager 0</span><br>
<span class="gmail-tl8wme">sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0</span><br>
<a href="http://notifications.info">notifications.info</a><span class="gmail-tl8wme"> 0</span><br>
<span class="gmail-tl8wme">sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0</span><br>
<span class="gmail-tl8wme">sysinv.fpga_agent_manager 0</span><br>
<span class="gmail-tl8wme">sysinv-keystone-listener-workers 0</span><br>
<span class="gmail-tl8wme">sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0</span><br>
<span class="gmail-tl8wme">sysinv.agent_manager.controller-0 0</span><br>
<span class="gmail-tl8wme">barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0</span><br>
<span class="gmail-tl8wme">sysinv.conductor_manager.controller-0 0</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com">C-Albert.Braden@charter.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface?
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">This might be a clue: </span><o:p></o:p></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></pre>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Courier New";color:black">ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89
e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Courier New";color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking
issue that prevents them from connecting.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> open infra <</span><a href="mailto:openinfradn@gmail.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">openinfradn@gmail.com</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">>
<br>
<b>Sent:</b> Thursday, June 17, 2021 10:06 AM<br>
<b>To:</b> Braden, Albert <</span><a href="mailto:C-Albert.Braden@charter.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">C-Albert.Braden@charter.com</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> openstack-discuss <</span><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">openstack-discuss@lists.openstack.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:solid #5A5A5A 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#235C70;background-position-x:0%;background-position-y:0%;background-attachment:scroll">
<strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmail-m3855489329660706895gmail-tl8wme">controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status</span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme">Password: </span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme">Cluster status of node rabbit@localhost ...</span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme">[{nodes,[{disc,[rabbit@localhost]}]},</span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme"> {running_nodes,[rabbit@localhost]},</span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme"> {cluster_name,<<"rabbit@controller-0">>},</span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme"> {partitions,[]},</span><br>
<span class="gmail-m3855489329660706895gmail-tl8wme"> {alarms,[{rabbit@localhost,[]}]}]</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> open infra <</span><a href="mailto:openinfradn@gmail.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">openinfradn@gmail.com</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">>
<br>
<b>Sent:</b> Thursday, June 17, 2021 9:58 AM<br>
<b>To:</b> Braden, Albert <</span><a href="mailto:C-Albert.Braden@charter.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">C-Albert.Braden@charter.com</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> openstack-discuss <</span><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">openstack-discuss@lists.openstack.org</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:solid #5A5A5A 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#235C70;background-position-x:0%;background-position-y:0%;background-attachment:scroll">
<strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<pre><span style="color:black;background:white">Hi Albert,</span><o:p></o:p></pre>
<pre><span style="color:black;background:white">Sorry for the inconvenience.</span><o:p></o:p></pre>
<pre><span style="color:black;background:white">Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.</span><o:p></o:p></pre>
<pre><span style="color:black;background:white">But I still have the same issue. Please find scheduler and conductor logs.</span><o:p></o:p></pre>
<pre><span style="color:black;background:white"># Scheduler Logs</span><o:p></o:p></pre>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
</div>
The contents of this e-mail message and <br>any attachments are intended solely for the <br>addressee(s) and may contain confidential <br>and/or legally privileged information. If you<br>are not the intended recipient of this message<br>or if this message has been addressed to you <br>in error, please immediately alert the sender<br>by reply e-mail and then delete this message <br>and any attachments. If you are not the <br>intended recipient, you are notified that <br>any use, dissemination, distribution, copying,<br>or storage of this message or any attachment <br>is strictly prohibited.</body>
</html>