<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi all,</p>
<p>It looks like the issue was actually with the Ubuntu image for
both 16.04 and 18.04. We changed the dhcp timeout in
"/etc/dhcp/dhclient.conf" from 300 seconds down to 2 seconds and
the instances then worked absolutely fine. Not sure why it was
only happening for some tenants and not others but that has
resolved it.</p>
<p>I am still going to look into the metadata service as the fix
doesn't feel right to me still.</p>
<p>Thanks again for all your help.</p>
<p>Grant<br>
</p>
<div class="moz-cite-prefix">On 05/12/2019 00:09, Laurent Dumont
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAOAKi8wHLMASHoq-7-wM1eZ105j=MpQsP3hs-n7t1zYU=r4pSw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Maybe the following could provide a bit more data :
<div>
<ul>
<li>Launch a test instance in the tenant project
experiencing the issue.</li>
<li>tcpdump directly on the instance TAP interface - confirm
if you are seeing DHCP DISCOVER/REQUEST/OFFER</li>
<li>Would also allow you to see the Cloudinit traffic.</li>
</ul>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Dec 4, 2019 at 7:06 PM
Grant Morley <<a href="mailto:grant@civo.com"
moz-do-not-send="true">grant@civo.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Cory,</p>
<p>Thanks for the response. I'll take a look at the metadata
service from the instance and from OpenStack itself
tomorrow now. It's midnight here in the UK and I need to
get some rest. Thanks for the tip, hopefully I'll find
something useful to go on from there.</p>
<p>Grant,<br>
</p>
<div>On 04/12/2019 23:49, Cory Hawkless wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
they failing to contact the metadata service and
hanging during the boot process while they try and
receive metadata?</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">From
the VM can you hit <a href="http://169.254.169.254"
target="_blank" moz-do-not-send="true">http://169.254.169.254</a>
– That’s the default IP of the metadata server, it
should respond with a basic page showing some date
based subdirectories</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If
it doesn’t respond you can start following the
metadata service path instead of DHCP</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Given
that the machines come up with an IP eventually
leads me to think the DHCP service is actually
working ok.</span></p>
<p class="MsoNormal"><a
name="m_1274259296692322191__MailEndCompose"
moz-do-not-send="true"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></a></p>
<div>
<div
style="border-right:none;border-bottom:none;border-left:none;border-top:1pt
solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext"
lang="EN-US">From:</span></b><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext"
lang="EN-US"> Grant Morley [<a
href="mailto:grant@civo.com" target="_blank"
moz-do-not-send="true">mailto:grant@civo.com</a>]
<br>
<b>Sent:</b> Thursday, 5 December 2019 10:10 AM<br>
<b>To:</b> Eric K. Miller <a
href="mailto:emiller@genesishosting.com"
target="_blank" moz-do-not-send="true"><emiller@genesishosting.com></a>;
<a
href="mailto:openstack-operators@lists.openstack.org"
target="_blank" moz-do-not-send="true">openstack-operators@lists.openstack.org</a><br>
<b>Cc:</b> Ian Banks <a
href="mailto:ian@civo.com" target="_blank"
moz-do-not-send="true"><ian@civo.com></a><br>
<b>Subject:</b> Re: DHCP timeout when creating
instances for specific tenants</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p>Hi Eric,</p>
<p>Thanks for getting back to me. I am fairly sure it is
a DHCP error. The instances are getting an IP when
they eventually boot, it is just taking a long time
for them to bring up networking. The strange thing is,
it only seems to be new tenants. All existing tenants
are absolutely fine. </p>
<p>I can check DNS as well just to be on the safe side,
however I wasn't seeing any errors in the Nova or
Neutron logs when the instance(s) were being created.</p>
<p>Regards,</p>
<div>
<p class="MsoNormal">On 04/12/2019 22:47, Eric K.
Miller wrote:</p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi
Grant,</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
you sure this is a DHCP timeout and not a DNS
resolution issue? I ask because we have seen a
strange DNS issue occur that can cause something
similar.</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
the VMs being assigned an IP after they finally
boot?</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Eric
K. Miller</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Genesis
Hosting Solutions, LLC</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Try
our Genesis Public Cloud - powered by
OpenStack!eut</span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a
href="https://genesishosting.com/"
target="_blank" moz-do-not-send="true">https://genesishosting.com/</a></span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<div
style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt
solid blue;padding:0cm 0cm 0cm 4pt">
<div>
<div
style="border-right:none;border-bottom:none;border-left:none;border-top:1pt
solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span
style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span
style="font-size:10pt;font-family:Tahoma,sans-serif"> Grant Morley [<a
href="mailto:grant@civo.com"
target="_blank" moz-do-not-send="true">mailto:grant@civo.com</a>]
<br>
<b>Sent:</b> Wednesday, December 04, 2019
11:00 AM<br>
<b>To:</b> <a
href="mailto:openstack-operators@lists.openstack.org"
target="_blank" moz-do-not-send="true">openstack-operators@lists.openstack.org</a><br>
<b>Cc:</b> Ian Banks<br>
<b>Subject:</b> DHCP timeout when creating
instances for specific tenants</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p>Hi all,</p>
<p>I wonder if anyone can help shed any light on an
odd issue we are seeing with only a couple of
specific tenants. Basically if they launch an
instance they are taking about 5 minutes to launch
rather than our usual 30 second or so launch.</p>
<p>We are seeing the following on the instance logs:</p>
<p><a href="https://pastebin.com/hDstsd8G"
target="_blank" moz-do-not-send="true">https://pastebin.com/hDstsd8G</a></p>
<p>Weirdly it only seems to be happening for 1 or 2
new tenants. I have tested this on our personal
account and a few other customers have tested and
their instances launch really quickly as expected.
</p>
<p>Is there anything specific during the tenant
creation that can cause this issue? Or are there
any logs in nova / neutron I should be looking out
for that might shed some light? </p>
<p>I haven't seen anything that is obvious. Any help
would be much appreciated as we are a little
stumped at the moment.</p>
<p>Many thanks,</p>
<div>
<p class="MsoNormal">-- </p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Verdana,sans-serif;color:rgb(168,216,255)"><img
id="gmail-m_1274259296692322191_x0000_i1025"
src="https://www.civo.com/images/email-logo.jpg"
moz-do-not-send="true" border="0"></span></p>
<div
style="margin-left:67.5pt;margin-bottom:3.75pt">
<p class="MsoNormal"><span
style="font-size:13.5pt;font-family:Verdana,sans-serif;color:rgb(35,157,255)">Grant
Morley</span><span
style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(109,190,255)"></span></p>
</div>
<div
style="margin-left:67.5pt;margin-bottom:7.5pt">
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:Verdana,sans-serif;color:rgb(136,136,136)">Cloud
Lead, Civo Ltd</span></p>
</div>
<div style="margin-left:67.5pt">
<p class="MsoNormal"><span
style="font-size:10pt;font-family:Helvetica,sans-serif"><a
href="https://www.civo.com/"
target="_blank" moz-do-not-send="true"><span
style="font-family:Verdana,sans-serif;text-decoration:none">www.civo.com</span></a></span><span
style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(136,136,136)">
| </span><span
style="font-size:10pt;font-family:Helvetica,sans-serif"><a
href="https://www.civo.com/signup"
target="_blank" moz-do-not-send="true"><span
style="font-family:Verdana,sans-serif;text-decoration:none">Signup for
an account!</span></a></span></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<div class="moz-signature">-- <br>
<div><br>
</div>
<div>
<div style="font-family: helvetica, Arial; font-size: 13px;"><font
face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"><span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"><img
src="https://www.civo.com/images/email-logo.jpg"
style="width: 70px; color: #a8d8ff; font-family: 'Lucida
Grande', Verdana, Arial, sans-serif; font-size: 11px;
line-height: 15px; float: left; margin-top: 0px;
margin-right: 10px"></span></font>
<div style="margin-left: 90px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"><span
style="font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; font-size: 18px; color:
rgb(35, 157, 255);">Grant Morley</span> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"><span
style="font-size: 15px;"></span></font></div>
<div style="margin-left: 90px; line-height: 15px; color:
#888888; margin-bottom: 10px;"><font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span
style="font-size: 15px;">Cloud Lead, Civo Ltd</span></font></div>
<div style="font-family: Hind, helvetica, arial; margin-left:
90px;"><a href="https://www.civo.com/" style="color: rgb(35,
157, 255); font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; line-height: 15px;
text-decoration: none;">www.civo.com</a><font
color="#888888"> | </font><a
href="https://www.civo.com/signup" style="color: rgb(35,
157, 255); font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; line-height: 15px;
margin-right: 6px; text-decoration: none;">Signup for an
account!</a></div>
</div>
</div>
</div>
</body>
</html>