<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>HI Eric,</p>
    <p>We are indeed already setting the DNS servers explicitly in the
      subnet so I don't think that is the issue ( from what I can tell )
      I was wondering if it could be an issue with Neutron not
      responding in time for the DHCP request from the instance, however
      I haven't yet found any evidence of this.<br>
    </p>
    <p>The only other thought I had was that it could be an issue with
      RabbitMQ somehow and potentially increasing the "rpc timeout" on
      neutron to see if that helps as I have seen some errors stating
      that RabbitMQ didn't respond to a message request in time. However
      I think it could be a red herring as I would assume if RabbitMQ
      was to blame, existing tenants and instances would also be
      suffering.</p>
    <p>Grant<br>
    </p>
    <div class="moz-cite-prefix">On 04/12/2019 23:46, Eric K. Miller
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:046E9C0290DD9149B106B72FC9156BEA04771513@gmsxchsvr01.thecreation.com">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <meta name="Generator" content="Microsoft Word 14 (filtered
        medium)">
      <!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
      <style><!--
/* Font Definitions */
@font-face
        {font-family:Helvetica;
        panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
        {font-family:Helvetica;
        panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
        {font-family:Verdana;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";
        color:black;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";
        color:black;}
span.EmailStyle18
        {mso-style-type:personal;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
span.EmailStyle19
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
      <div class="WordSection1">
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi
            Grant,<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I
            didn't see any DNS errors either.  The solution was to
            explicitly configure the dns servers in the subnet.  Are you
            doing this already?  Or are you relying on the dnsmasq
            processes created for the router to respond to DNS queries
            (and forward them respectively)?<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><br>
            Eric<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <div style="border:none;border-left:solid blue 1.5pt;padding:0in
          0in 0in 4.0pt">
          <div>
            <div style="border:none;border-top:solid #B5C4DF
              1.0pt;padding:3.0pt 0in 0in 0in">
              <p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
                  Grant Morley [<a class="moz-txt-link-freetext" href="mailto:grant@civo.com">mailto:grant@civo.com</a>] <br>
                  <b>Sent:</b> Wednesday, December 04, 2019 5:40 PM<br>
                  <b>To:</b> Eric K. Miller;
                  <a class="moz-txt-link-abbreviated" href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
                  <b>Cc:</b> Ian Banks<br>
                  <b>Subject:</b> Re: DHCP timeout when creating
                  instances for specific tenants<o:p></o:p></span></p>
            </div>
          </div>
          <p class="MsoNormal"><o:p> </o:p></p>
          <p>Hi Eric,<o:p></o:p></p>
          <p>Thanks for getting back to me. I am fairly sure it is a
            DHCP error. The instances are getting an IP when they
            eventually boot, it is just taking a long time for them to
            bring up networking. The strange thing is, it only seems to
            be new tenants. All existing tenants are absolutely fine. <o:p></o:p></p>
          <p>I can check DNS as well just to be on the safe side,
            however I wasn't seeing any errors in the Nova or Neutron
            logs when the instance(s) were being created.<o:p></o:p></p>
          <p>Regards,<o:p></o:p></p>
          <div>
            <p class="MsoNormal">On 04/12/2019 22:47, Eric K. Miller
              wrote:<o:p></o:p></p>
          </div>
          <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi
                Grant,</span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Are
                you sure this is a DHCP timeout and not a DNS resolution
                issue?  I ask because we have seen a strange DNS issue
                occur that can cause something similar.</span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Are
                the VMs being assigned an IP after they finally boot?</span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Eric
                K. Miller</span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Genesis
                Hosting Solutions, LLC</span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Try
                our Genesis Public Cloud - powered by OpenStack!eut</span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a
                  href="https://genesishosting.com/"
                  moz-do-not-send="true">https://genesishosting.com/</a></span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
            <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
            <div style="border:none;border-left:solid blue
              1.5pt;padding:0in 0in 0in 4.0pt">
              <div>
                <div style="border:none;border-top:solid #B5C4DF
                  1.0pt;padding:3.0pt 0in 0in 0in">
                  <p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
                      Grant Morley [<a href="mailto:grant@civo.com"
                        moz-do-not-send="true">mailto:grant@civo.com</a>]
                      <br>
                      <b>Sent:</b> Wednesday, December 04, 2019 11:00 AM<br>
                      <b>To:</b> <a
                        href="mailto:openstack-operators@lists.openstack.org"
                        moz-do-not-send="true">openstack-operators@lists.openstack.org</a><br>
                      <b>Cc:</b> Ian Banks<br>
                      <b>Subject:</b> DHCP timeout when creating
                      instances for specific tenants</span><o:p></o:p></p>
                </div>
              </div>
              <p class="MsoNormal"> <o:p></o:p></p>
              <p>Hi all,<o:p></o:p></p>
              <p>I wonder if anyone can help shed any light on an odd
                issue we are seeing with only a couple of specific
                tenants. Basically if they launch an instance they are
                taking about 5 minutes to launch rather than our usual
                30 second or so launch.<o:p></o:p></p>
              <p>We are seeing the following on the instance logs:<o:p></o:p></p>
              <p><a href="https://pastebin.com/hDstsd8G"
                  moz-do-not-send="true">https://pastebin.com/hDstsd8G</a><o:p></o:p></p>
              <p>Weirdly it only seems to be happening for 1 or 2 new
                tenants. I have tested this on our personal account and
                a few other customers have tested and their instances
                launch really quickly as expected. <o:p></o:p></p>
              <p>Is there anything specific during the tenant creation
                that can cause this issue? Or are there any logs in nova
                / neutron I should be looking out for that might shed
                some light? <o:p></o:p></p>
              <p>I haven't seen anything that is obvious. Any help would
                be much appreciated as we are a little stumped at the
                moment.<o:p></o:p></p>
              <p>Many thanks,<o:p></o:p></p>
              <div>
                <p class="MsoNormal">-- <o:p></o:p></p>
                <div>
                  <p class="MsoNormal"> <o:p></o:p></p>
                </div>
                <div>
                  <div>
                    <p class="MsoNormal"><span
style="font-size:8.5pt;font-family:"Verdana","sans-serif";color:#A8D8FF;border:solid
                        windowtext 1.0pt;padding:0in"><img
                          id="_x0000_i1025"
                          src="cid:part5.A19B6599.A5773C21@civo.com"
                          alt="Image removed by sender." class=""
                          width="100" height="100" border="0"></span><o:p></o:p></p>
                    <div style="margin-left:67.5pt;margin-bottom:3.75pt">
                      <p class="MsoNormal"
                        style="mso-line-height-alt:11.25pt"><span
style="font-size:13.5pt;font-family:"Verdana","sans-serif";color:#239DFF">Grant
                          Morley</span><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:#6DBEFF">
                        </span><o:p></o:p></p>
                    </div>
                    <div style="margin-left:67.5pt;margin-bottom:7.5pt">
                      <p class="MsoNormal"
                        style="mso-line-height-alt:11.25pt"><span
style="font-size:11.5pt;font-family:"Verdana","sans-serif";color:#888888">Cloud
                          Lead, Civo Ltd</span><o:p></o:p></p>
                    </div>
                    <div style="margin-left:67.5pt">
                      <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a
                            href="https://www.civo.com/"
                            moz-do-not-send="true"><span
style="font-family:"Verdana","sans-serif";text-decoration:none">www.civo.com</span></a></span><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:#888888">
                          | </span><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a
                            href="https://www.civo.com/signup"
                            moz-do-not-send="true"><span
style="font-family:"Verdana","sans-serif";text-decoration:none">Signup
                              for an account!</span></a></span><o:p></o:p></p>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
  </body>
</html>