<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>HI Eric,</p>
<p>We are indeed already setting the DNS servers explicitly in the
subnet so I don't think that is the issue ( from what I can tell )
I was wondering if it could be an issue with Neutron not
responding in time for the DHCP request from the instance, however
I haven't yet found any evidence of this.<br>
</p>
<p>The only other thought I had was that it could be an issue with
RabbitMQ somehow and potentially increasing the "rpc timeout" on
neutron to see if that helps as I have seen some errors stating
that RabbitMQ didn't respond to a message request in time. However
I think it could be a red herring as I would assume if RabbitMQ
was to blame, existing tenants and instances would also be
suffering.</p>
<p>Grant<br>
</p>
<div class="moz-cite-prefix">On 04/12/2019 23:46, Eric K. Miller
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:046E9C0290DD9149B106B72FC9156BEA04771513@gmsxchsvr01.thecreation.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi
Grant,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I
didn't see any DNS errors either. The solution was to
explicitly configure the dns servers in the subnet. Are you
doing this already? Or are you relying on the dnsmasq
processes created for the router to respond to DNS queries
(and forward them respectively)?<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><br>
Eric<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in
0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Grant Morley [<a class="moz-txt-link-freetext" href="mailto:grant@civo.com">mailto:grant@civo.com</a>] <br>
<b>Sent:</b> Wednesday, December 04, 2019 5:40 PM<br>
<b>To:</b> Eric K. Miller;
<a class="moz-txt-link-abbreviated" href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
<b>Cc:</b> Ian Banks<br>
<b>Subject:</b> Re: DHCP timeout when creating
instances for specific tenants<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hi Eric,<o:p></o:p></p>
<p>Thanks for getting back to me. I am fairly sure it is a
DHCP error. The instances are getting an IP when they
eventually boot, it is just taking a long time for them to
bring up networking. The strange thing is, it only seems to
be new tenants. All existing tenants are absolutely fine. <o:p></o:p></p>
<p>I can check DNS as well just to be on the safe side,
however I wasn't seeing any errors in the Nova or Neutron
logs when the instance(s) were being created.<o:p></o:p></p>
<p>Regards,<o:p></o:p></p>
<div>
<p class="MsoNormal">On 04/12/2019 22:47, Eric K. Miller
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi
Grant,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Are
you sure this is a DHCP timeout and not a DNS resolution
issue? I ask because we have seen a strange DNS issue
occur that can cause something similar.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Are
the VMs being assigned an IP after they finally boot?</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Eric
K. Miller</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Genesis
Hosting Solutions, LLC</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Try
our Genesis Public Cloud - powered by OpenStack!eut</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a
href="https://genesishosting.com/"
moz-do-not-send="true">https://genesishosting.com/</a></span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue
1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
Grant Morley [<a href="mailto:grant@civo.com"
moz-do-not-send="true">mailto:grant@civo.com</a>]
<br>
<b>Sent:</b> Wednesday, December 04, 2019 11:00 AM<br>
<b>To:</b> <a
href="mailto:openstack-operators@lists.openstack.org"
moz-do-not-send="true">openstack-operators@lists.openstack.org</a><br>
<b>Cc:</b> Ian Banks<br>
<b>Subject:</b> DHCP timeout when creating
instances for specific tenants</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p>Hi all,<o:p></o:p></p>
<p>I wonder if anyone can help shed any light on an odd
issue we are seeing with only a couple of specific
tenants. Basically if they launch an instance they are
taking about 5 minutes to launch rather than our usual
30 second or so launch.<o:p></o:p></p>
<p>We are seeing the following on the instance logs:<o:p></o:p></p>
<p><a href="https://pastebin.com/hDstsd8G"
moz-do-not-send="true">https://pastebin.com/hDstsd8G</a><o:p></o:p></p>
<p>Weirdly it only seems to be happening for 1 or 2 new
tenants. I have tested this on our personal account and
a few other customers have tested and their instances
launch really quickly as expected. <o:p></o:p></p>
<p>Is there anything specific during the tenant creation
that can cause this issue? Or are there any logs in nova
/ neutron I should be looking out for that might shed
some light? <o:p></o:p></p>
<p>I haven't seen anything that is obvious. Any help would
be much appreciated as we are a little stumped at the
moment.<o:p></o:p></p>
<p>Many thanks,<o:p></o:p></p>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:"Verdana","sans-serif";color:#A8D8FF;border:solid
windowtext 1.0pt;padding:0in"><img
id="_x0000_i1025"
src="cid:part5.A19B6599.A5773C21@civo.com"
alt="Image removed by sender." class=""
width="100" height="100" border="0"></span><o:p></o:p></p>
<div style="margin-left:67.5pt;margin-bottom:3.75pt">
<p class="MsoNormal"
style="mso-line-height-alt:11.25pt"><span
style="font-size:13.5pt;font-family:"Verdana","sans-serif";color:#239DFF">Grant
Morley</span><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:#6DBEFF">
</span><o:p></o:p></p>
</div>
<div style="margin-left:67.5pt;margin-bottom:7.5pt">
<p class="MsoNormal"
style="mso-line-height-alt:11.25pt"><span
style="font-size:11.5pt;font-family:"Verdana","sans-serif";color:#888888">Cloud
Lead, Civo Ltd</span><o:p></o:p></p>
</div>
<div style="margin-left:67.5pt">
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a
href="https://www.civo.com/"
moz-do-not-send="true"><span
style="font-family:"Verdana","sans-serif";text-decoration:none">www.civo.com</span></a></span><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:#888888">
| </span><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a
href="https://www.civo.com/signup"
moz-do-not-send="true"><span
style="font-family:"Verdana","sans-serif";text-decoration:none">Signup
for an account!</span></a></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>