<div dir="ltr"><div>VM provisioning is resolved.</div><div>But still having an issue with openstack networking.</div><div><br></div><div><div>"Unable to create the network. No tenant network is available for allocation."</div><div><br></div></div><div><a href="http://paste.openstack.org/show/807014/">http://paste.openstack.org/show/807014/</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 17, 2021 at 11:14 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com">C-Albert.Braden@charter.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_1305777423259021146WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">I see “no valid host” at 10:48:39 in the conductor log:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:black">2021-06-17T10:48:39.65107847Z stdout F nova.exception.NoValidHost: No valid host was found.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">In the scheduler log at 10:37:43 we see the scheduler starting and the RMQ error, followed by the scheduling failure at 10:48:39. It looks like the scheduler
can’t connect to RMQ.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:black">2021-06-17T10:37:43.038362995Z stdout F 2021-06-17 10:37:43.038 1 INFO nova.service [-] Starting scheduler node (version 21.2.1)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:black">2021-06-17T10:37:43.08082925Z stdout F 2021-06-17 10:37:43.079 1 ERROR oslo.messaging._drivers.impl_rabbit [req-775b2b25-b9ef-4642-a6fd-7574eab7cc37 - - - - -] Connection
failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:black">2021-06-17T10:48:39.197318418Z stdout F 2021-06-17 10:48:39.196 1 INFO nova.scheduler.manager [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89
e8813293073545f99658adbec2f80c1d - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>>
<br>
<b>Sent:</b> Thursday, June 17, 2021 12:38 PM<br>
<b>To:</b> Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>><br>
<b>Cc:</b> openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background:rgb(35,92,112) none repeat scroll 0% 0%">
<strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><u></u><u></u></p>
</div>
<div>
<div id="gmail-m_1305777423259021146gmail-:3o.ma">
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-tl8wme">Another instance created at 2021-06-17T16:10:06Z, and it also failed.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-tl8wme">I can't see rabbitmq errors around that timestamp.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div id="gmail-m_1305777423259021146gmail-:4d.ma">
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-tl8wme">scheduler log </span><a href="https://www.google.com/url?q=http://paste.openstack.org/show/806737/&sa=D&source=hangouts&ust=1624034044021000&usg=AFQjCNHSJ6dWR_hMfE6e6YUw14demT_mZA" target="_blank">http://paste.openstack.org/show/806737/</a><u></u><u></u></p>
</div>
<div id="gmail-m_1305777423259021146gmail-:4e.ma">
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-tl8wme">conducter log </span><a href="https://www.google.com/url?q=http://paste.openstack.org/show/806738/&sa=D&source=hangouts&ust=1624034079136000&usg=AFQjCNHuMa5Eerlpx0-wlOM-fBaDXpHptQ" target="_blank">http://paste.openstack.org/show/806738/</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div id="gmail-m_1305777423259021146gmail-:4f.ma">
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-tl8wme">At 2021-06-17 15:52:38.810, system was able to connect to AMQP </span><u></u><u></u></p>
</div>
<div id="gmail-m_1305777423259021146gmail-:4g.ma">
<p class="MsoNormal"><a href="https://www.google.com/url?q=http://paste.openstack.org/show/806739/&sa=D&source=hangouts&ust=1624034275886000&usg=AFQjCNFree0y1N8q-PQnd6Unz5AQZG5q2g" target="_blank">http://paste.openstack.org/show/806739/</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div id="gmail-m_1305777423259021146gmail-:4c.ma">
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-tl8wme">controller-0:/var/log/pods$ sudo rabbitmqctl list_queues</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">Password: </span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">Listing queues ...</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.ceph_manager.192.168.204.1 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.ceph_manager 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">barbican.workers 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.fpga_agent_manager.controller-0 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">barbican.workers.barbican.queue 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.conductor_manager 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.agent_manager_fanout_fed76e414eb04da084ab35a1c27e1bf1 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.agent_manager 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.ceph_manager_fanout_d585fb522f46431da60741573a7f8575 0</span><br>
<a href="http://notifications.info" target="_blank">notifications.info</a><span class="gmail-m_1305777423259021146gmail-tl8wme"> 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.conductor_manager_fanout_cba8926fa47f4780a9e17f3d9b889500 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.fpga_agent_manager 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv-keystone-listener-workers 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.fpga_agent_manager_fanout_f00a01bfd3f54a64860f8d7454e9a78e 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.agent_manager.controller-0 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">barbican.workers_fanout_2c97c319faa943e88eaed4c101e530c7 0</span><br>
<span class="gmail-m_1305777423259021146gmail-tl8wme">sysinv.conductor_manager.controller-0 0</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 17, 2021 at 7:51 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">It looks like RMQ is up but services can’t connect to it. What do you see in the RMQ web interface?
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<pre><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">This might be a clue: </span><u></u><u></u></pre>
<pre><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></pre>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:black">ERROR oslo.messaging._drivers.impl_rabbit [req-b30bb3d6-0ab1-4dbb-aed9-1d9cece3eac7 d9f7048c1cd947cfa8ecef128a6cee89
e8813293073545f99658adbec2f80c1d - default default] Connection failed: failed to resolve broker hostname (retrying in 0 seconds): OSError: failed to resolve broker hostname</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Check for a typo in your config that points services to an incorrect RMQ hostname, or a networking
issue that prevents them from connecting.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> open infra <</span><a href="mailto:openinfradn@gmail.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">openinfradn@gmail.com</span></a><span style="font-size:11pt;font-family:"Calibri",sans-serif">>
<br>
<b>Sent:</b> Thursday, June 17, 2021 10:06 AM<br>
<b>To:</b> Braden, Albert <</span><a href="mailto:C-Albert.Braden@charter.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">C-Albert.Braden@charter.com</span></a><span style="font-size:11pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> openstack-discuss <</span><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">openstack-discuss@lists.openstack.org</span></a><span style="font-size:11pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background:rgb(35,92,112) none repeat scroll 0% 0%">
<strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme">controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status</span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme">Password: </span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme">Cluster status of node rabbit@localhost ...</span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme">[{nodes,[{disc,[rabbit@localhost]}]},</span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme"> {running_nodes,[rabbit@localhost]},</span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme"> {cluster_name,<<"rabbit@controller-0">>},</span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme"> {partitions,[]},</span><br>
<span class="gmail-m_1305777423259021146gmail-m3855489329660706895gmail-tl8wme"> {alarms,[{rabbit@localhost,[]}]}]</span><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 17, 2021 at 7:32 PM Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com" target="_blank">C-Albert.Braden@charter.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">It looks like your RMQ is broken. What do you get from “rabbitmqctl cluster_status”?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> open infra <</span><a href="mailto:openinfradn@gmail.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">openinfradn@gmail.com</span></a><span style="font-size:11pt;font-family:"Calibri",sans-serif">>
<br>
<b>Sent:</b> Thursday, June 17, 2021 9:58 AM<br>
<b>To:</b> Braden, Albert <</span><a href="mailto:C-Albert.Braden@charter.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">C-Albert.Braden@charter.com</span></a><span style="font-size:11pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> openstack-discuss <</span><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">openstack-discuss@lists.openstack.org</span></a><span style="font-size:11pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: [EXTERNAL] Error creating VMs</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border:1pt solid rgb(90,90,90);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background:rgb(35,92,112) none repeat scroll 0% 0%">
<strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10pt;font-family:"Calibri",sans-serif;color:white"> The e-mail below is from an external source. Please exercise caution before
opening attachments, clicking links, or following guidance. </span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">Hi Albert,</span><u></u><u></u></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">Sorry for the inconvenience.</span><u></u><u></u></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">Please note that I have recreated both the data network at starlingx (physical network of openstack) and the network of openstack.</span><u></u><u></u></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%">But I still have the same issue. Please find scheduler and conductor logs.</span><u></u><u></u></pre>
<pre><span style="color:black;background:white none repeat scroll 0% 0%"># Scheduler Logs</span><u></u><u></u></pre>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</blockquote>
</div>
</div>
The contents of this e-mail message and <br>any attachments are intended solely for the <br>addressee(s) and may contain confidential <br>and/or legally privileged information. If you<br>are not the intended recipient of this message<br>or if this message has been addressed to you <br>in error, please immediately alert the sender<br>by reply e-mail and then delete this message <br>and any attachments. If you are not the <br>intended recipient, you are notified that <br>any use, dissemination, distribution, copying,<br>or storage of this message or any attachment <br>is strictly prohibited.</div>
</blockquote></div>