<div dir="ltr">Maybe the following could provide a bit more data :<div><ul><li>Launch a test instance in the tenant project experiencing the issue.</li><li>tcpdump directly on the instance TAP interface - confirm if you are seeing DHCP DISCOVER/REQUEST/OFFER</li><li>Would also allow you to see the Cloudinit traffic.</li></ul></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 4, 2019 at 7:06 PM Grant Morley <<a href="mailto:grant@civo.com">grant@civo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Cory,</p>
<p>Thanks for the response. I'll take a look at the metadata service
from the instance and from OpenStack itself tomorrow now. It's
midnight here in the UK and I need to get some rest. Thanks for
the tip, hopefully I'll find something useful to go on from there.</p>
<p>Grant,<br>
</p>
<div>On 04/12/2019 23:49, Cory Hawkless
wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
they failing to contact the metadata service and hanging
during the boot process while they try and receive metadata?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">From
the VM can you hit
<a href="http://169.254.169.254" target="_blank">http://169.254.169.254</a>
– That’s the default IP of the metadata server, it should
respond with a basic page showing some date based
subdirectories<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If
it doesn’t respond you can start following the metadata
service path instead of DHCP<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Given
that the machines come up with an IP eventually leads me to
think the DHCP service is actually working ok.<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_1274259296692322191__MailEndCompose"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></a></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext" lang="EN-US"> Grant Morley [<a href="mailto:grant@civo.com" target="_blank">mailto:grant@civo.com</a>]
<br>
<b>Sent:</b> Thursday, 5 December 2019 10:10 AM<br>
<b>To:</b> Eric K. Miller
<a href="mailto:emiller@genesishosting.com" target="_blank"><emiller@genesishosting.com></a>;
<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
<b>Cc:</b> Ian Banks <a href="mailto:ian@civo.com" target="_blank"><ian@civo.com></a><br>
<b>Subject:</b> Re: DHCP timeout when creating instances
for specific tenants<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p>Hi Eric,<u></u><u></u></p>
<p>Thanks for getting back to me. I am fairly sure it is a DHCP
error. The instances are getting an IP when they eventually
boot, it is just taking a long time for them to bring up
networking. The strange thing is, it only seems to be new
tenants. All existing tenants are absolutely fine. <u></u><u></u></p>
<p>I can check DNS as well just to be on the safe side, however
I wasn't seeing any errors in the Nova or Neutron logs when
the instance(s) were being created.<u></u><u></u></p>
<p>Regards,<u></u><u></u></p>
<div>
<p class="MsoNormal">On 04/12/2019 22:47, Eric K. Miller
wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi
Grant,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
you sure this is a DHCP timeout and not a DNS resolution
issue? I ask because we have seen a strange DNS issue
occur that can cause something similar.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
the VMs being assigned an IP after they finally boot?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Eric
K. Miller</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Genesis
Hosting Solutions, LLC</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Try
our Genesis Public Cloud - powered by OpenStack!eut</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="https://genesishosting.com/" target="_blank">https://genesishosting.com/</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0cm 0cm 0cm 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif">
Grant Morley [<a href="mailto:grant@civo.com" target="_blank">mailto:grant@civo.com</a>]
<br>
<b>Sent:</b> Wednesday, December 04, 2019 11:00 AM<br>
<b>To:</b> <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
<b>Cc:</b> Ian Banks<br>
<b>Subject:</b> DHCP timeout when creating instances
for specific tenants</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p>Hi all,<u></u><u></u></p>
<p>I wonder if anyone can help shed any light on an odd
issue we are seeing with only a couple of specific
tenants. Basically if they launch an instance they are
taking about 5 minutes to launch rather than our usual 30
second or so launch.<u></u><u></u></p>
<p>We are seeing the following on the instance logs:<u></u><u></u></p>
<p><a href="https://pastebin.com/hDstsd8G" target="_blank">https://pastebin.com/hDstsd8G</a><u></u><u></u></p>
<p>Weirdly it only seems to be happening for 1 or 2 new
tenants. I have tested this on our personal account and a
few other customers have tested and their instances launch
really quickly as expected.
<u></u><u></u></p>
<p>Is there anything specific during the tenant creation
that can cause this issue? Or are there any logs in nova /
neutron I should be looking out for that might shed some
light?
<u></u><u></u></p>
<p>I haven't seen anything that is obvious. Any help would
be much appreciated as we are a little stumped at the
moment.<u></u><u></u></p>
<p>Many thanks,<u></u><u></u></p>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Verdana,sans-serif;color:rgb(168,216,255)"><img id="gmail-m_1274259296692322191_x0000_i1025" src="https://www.civo.com/images/email-logo.jpg" border="0"></span><u></u><u></u></p>
<div style="margin-left:67.5pt;margin-bottom:3.75pt">
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:Verdana,sans-serif;color:rgb(35,157,255)">Grant
Morley</span><span style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(109,190,255)"></span><u></u><u></u></p>
</div>
<div style="margin-left:67.5pt;margin-bottom:7.5pt">
<p class="MsoNormal"><span style="font-size:11.5pt;font-family:Verdana,sans-serif;color:rgb(136,136,136)">Cloud
Lead, Civo Ltd</span><u></u><u></u></p>
</div>
<div style="margin-left:67.5pt">
<p class="MsoNormal"><span style="font-size:10pt;font-family:Helvetica,sans-serif"><a href="https://www.civo.com/" target="_blank"><span style="font-family:Verdana,sans-serif;text-decoration:none">www.civo.com</span></a></span><span style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(136,136,136)">
| </span><span style="font-size:10pt;font-family:Helvetica,sans-serif"><a href="https://www.civo.com/signup" target="_blank"><span style="font-family:Verdana,sans-serif;text-decoration:none">Signup
for an account!</span></a></span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div>