<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi all,</p>
    <p>It looks like the issue was actually with the Ubuntu image for
      both 16.04 and 18.04. We changed the dhcp timeout in
      "/etc/dhcp/dhclient.conf" from 300 seconds down to 2 seconds and
      the instances then worked absolutely fine. Not sure why it was
      only happening for some tenants and not others but that has
      resolved it.</p>
    <p>I am still going to look into the metadata service as the fix
      doesn't feel right to me still.</p>
    <p>Thanks again for all your help.</p>
    <p>Grant<br>
    </p>
    <div class="moz-cite-prefix">On 05/12/2019 00:09, Laurent Dumont
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAOAKi8wHLMASHoq-7-wM1eZ105j=MpQsP3hs-n7t1zYU=r4pSw@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">Maybe the following could provide a bit more data :
        <div>
          <ul>
            <li>Launch a test instance in the tenant project
              experiencing the issue.</li>
            <li>tcpdump directly on the instance TAP interface - confirm
              if you are seeing DHCP DISCOVER/REQUEST/OFFER</li>
            <li>Would also allow you to see the Cloudinit traffic.</li>
          </ul>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Wed, Dec 4, 2019 at 7:06 PM
          Grant Morley <<a href="mailto:grant@civo.com"
            moz-do-not-send="true">grant@civo.com</a>> wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div bgcolor="#FFFFFF">
            <p>Hi Cory,</p>
            <p>Thanks for the response. I'll take a look at the metadata
              service from the instance and from OpenStack itself
              tomorrow now. It's midnight here in the UK and I need to
              get some rest. Thanks for the tip, hopefully I'll find
              something useful to go on from there.</p>
            <p>Grant,<br>
            </p>
            <div>On 04/12/2019 23:49, Cory Hawkless wrote:<br>
            </div>
            <blockquote type="cite">
              <div>
                <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
                    they failing to contact the metadata service and
                    hanging during the boot process while they try and
                    receive metadata?</span></p>
                <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">From
                    the VM can you hit <a href="http://169.254.169.254"
                      target="_blank" moz-do-not-send="true">http://169.254.169.254</a>
                    – That’s the default IP of the metadata server, it
                    should respond with a basic page showing some date
                    based subdirectories</span></p>
                <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If
                    it doesn’t respond you can start following the
                    metadata service path instead of DHCP</span></p>
                <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Given
                    that the machines come up with an IP eventually
                    leads me to think the DHCP service is actually
                    working ok.</span></p>
                <p class="MsoNormal"><a
                    name="m_1274259296692322191__MailEndCompose"
                    moz-do-not-send="true"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></a></p>
                <div>
                  <div
style="border-right:none;border-bottom:none;border-left:none;border-top:1pt
                    solid rgb(225,225,225);padding:3pt 0cm 0cm">
                    <p class="MsoNormal"><b><span
                          style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext"
                          lang="EN-US">From:</span></b><span
                        style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext"
                        lang="EN-US"> Grant Morley [<a
                          href="mailto:grant@civo.com" target="_blank"
                          moz-do-not-send="true">mailto:grant@civo.com</a>]
                        <br>
                        <b>Sent:</b> Thursday, 5 December 2019 10:10 AM<br>
                        <b>To:</b> Eric K. Miller <a
                          href="mailto:emiller@genesishosting.com"
                          target="_blank" moz-do-not-send="true"><emiller@genesishosting.com></a>;
                        <a
                          href="mailto:openstack-operators@lists.openstack.org"
                          target="_blank" moz-do-not-send="true">openstack-operators@lists.openstack.org</a><br>
                        <b>Cc:</b> Ian Banks <a
                          href="mailto:ian@civo.com" target="_blank"
                          moz-do-not-send="true"><ian@civo.com></a><br>
                        <b>Subject:</b> Re: DHCP timeout when creating
                        instances for specific tenants</span></p>
                  </div>
                </div>
                <p class="MsoNormal"> </p>
                <p>Hi Eric,</p>
                <p>Thanks for getting back to me. I am fairly sure it is
                  a DHCP error. The instances are getting an IP when
                  they eventually boot, it is just taking a long time
                  for them to bring up networking. The strange thing is,
                  it only seems to be new tenants. All existing tenants
                  are absolutely fine. </p>
                <p>I can check DNS as well just to be on the safe side,
                  however I wasn't seeing any errors in the Nova or
                  Neutron logs when the instance(s) were being created.</p>
                <p>Regards,</p>
                <div>
                  <p class="MsoNormal">On 04/12/2019 22:47, Eric K.
                    Miller wrote:</p>
                </div>
                <blockquote style="margin-top:5pt;margin-bottom:5pt">
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi
                      Grant,</span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
                      you sure this is a DHCP timeout and not a DNS
                      resolution issue?  I ask because we have seen a
                      strange DNS issue occur that can cause something
                      similar.</span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Are
                      the VMs being assigned an IP after they finally
                      boot?</span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Eric
                      K. Miller</span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Genesis
                      Hosting Solutions, LLC</span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Try
                      our Genesis Public Cloud - powered by
                      OpenStack!eut</span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a
                        href="https://genesishosting.com/"
                        target="_blank" moz-do-not-send="true">https://genesishosting.com/</a></span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
                  <p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
                  <div
style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt
                    solid blue;padding:0cm 0cm 0cm 4pt">
                    <div>
                      <div
style="border-right:none;border-bottom:none;border-left:none;border-top:1pt
                        solid rgb(181,196,223);padding:3pt 0cm 0cm">
                        <p class="MsoNormal"><b><span
                              style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span
style="font-size:10pt;font-family:Tahoma,sans-serif"> Grant Morley [<a
                              href="mailto:grant@civo.com"
                              target="_blank" moz-do-not-send="true">mailto:grant@civo.com</a>]
                            <br>
                            <b>Sent:</b> Wednesday, December 04, 2019
                            11:00 AM<br>
                            <b>To:</b> <a
                              href="mailto:openstack-operators@lists.openstack.org"
                              target="_blank" moz-do-not-send="true">openstack-operators@lists.openstack.org</a><br>
                            <b>Cc:</b> Ian Banks<br>
                            <b>Subject:</b> DHCP timeout when creating
                            instances for specific tenants</span></p>
                      </div>
                    </div>
                    <p class="MsoNormal"> </p>
                    <p>Hi all,</p>
                    <p>I wonder if anyone can help shed any light on an
                      odd issue we are seeing with only a couple of
                      specific tenants. Basically if they launch an
                      instance they are taking about 5 minutes to launch
                      rather than our usual 30 second or so launch.</p>
                    <p>We are seeing the following on the instance logs:</p>
                    <p><a href="https://pastebin.com/hDstsd8G"
                        target="_blank" moz-do-not-send="true">https://pastebin.com/hDstsd8G</a></p>
                    <p>Weirdly it only seems to be happening for 1 or 2
                      new tenants. I have tested this on our personal
                      account and a few other customers have tested and
                      their instances launch really quickly as expected.
                    </p>
                    <p>Is there anything specific during the tenant
                      creation that can cause this issue? Or are there
                      any logs in nova / neutron I should be looking out
                      for that might shed some light? </p>
                    <p>I haven't seen anything that is obvious. Any help
                      would be much appreciated as we are a little
                      stumped at the moment.</p>
                    <p>Many thanks,</p>
                    <div>
                      <p class="MsoNormal">-- </p>
                      <div>
                        <p class="MsoNormal"> </p>
                      </div>
                      <div>
                        <div>
                          <p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Verdana,sans-serif;color:rgb(168,216,255)"><img
id="gmail-m_1274259296692322191_x0000_i1025"
                                src="https://www.civo.com/images/email-logo.jpg"
                                moz-do-not-send="true" border="0"></span></p>
                          <div
                            style="margin-left:67.5pt;margin-bottom:3.75pt">
                            <p class="MsoNormal"><span
style="font-size:13.5pt;font-family:Verdana,sans-serif;color:rgb(35,157,255)">Grant
                                Morley</span><span
style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(109,190,255)"></span></p>
                          </div>
                          <div
                            style="margin-left:67.5pt;margin-bottom:7.5pt">
                            <p class="MsoNormal"><span
style="font-size:11.5pt;font-family:Verdana,sans-serif;color:rgb(136,136,136)">Cloud
                                Lead, Civo Ltd</span></p>
                          </div>
                          <div style="margin-left:67.5pt">
                            <p class="MsoNormal"><span
                                style="font-size:10pt;font-family:Helvetica,sans-serif"><a
                                  href="https://www.civo.com/"
                                  target="_blank" moz-do-not-send="true"><span
style="font-family:Verdana,sans-serif;text-decoration:none">www.civo.com</span></a></span><span
style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(136,136,136)">
                                | </span><span
                                style="font-size:10pt;font-family:Helvetica,sans-serif"><a
                                  href="https://www.civo.com/signup"
                                  target="_blank" moz-do-not-send="true"><span
style="font-family:Verdana,sans-serif;text-decoration:none">Signup for
                                    an account!</span></a></span></p>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </blockquote>
              </div>
            </blockquote>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <div class="moz-signature">-- <br>
      <div><br>
      </div>
      <div>
        <div style="font-family: helvetica, Arial; font-size: 13px;"><font
            face="Lucida Grande, Verdana, Arial, sans-serif"
            color="#A8D8FF"><span style="font-size: 11px; line-height:
              15px; float: left; margin-right: 10px;"><img
                src="https://www.civo.com/images/email-logo.jpg"
                style="width: 70px; color: #a8d8ff; font-family: 'Lucida
                Grande', Verdana, Arial, sans-serif; font-size: 11px;
                line-height: 15px; float: left; margin-top: 0px;
                margin-right: 10px"></span></font>
          <div style="margin-left: 90px; line-height: 15px; color:
            rgb(109, 190, 255); margin-bottom: 5px;"><span
              style="font-family: Hind, "Lucida Grande",
              Verdana, Arial, sans-serif; font-size: 18px; color:
              rgb(35, 157, 255);">Grant Morley</span> <font face="Hind,
              Lucida Grande, Verdana, Arial, sans-serif"><span
                style="font-size: 15px;"></span></font></div>
          <div style="margin-left: 90px; line-height: 15px; color:
            #888888; margin-bottom: 10px;"><font face="Hind, Lucida
              Grande, Verdana, Arial, sans-serif"><span
                style="font-size: 15px;">Cloud Lead, Civo Ltd</span></font></div>
          <div style="font-family: Hind, helvetica, arial; margin-left:
            90px;"><a href="https://www.civo.com/" style="color: rgb(35,
              157, 255); font-family: Hind, "Lucida Grande",
              Verdana, Arial, sans-serif; line-height: 15px;
              text-decoration: none;">www.civo.com</a><font
              color="#888888"> | </font><a
              href="https://www.civo.com/signup" style="color: rgb(35,
              157, 255); font-family: Hind, "Lucida Grande",
              Verdana, Arial, sans-serif; line-height: 15px;
              margin-right: 6px; text-decoration: none;">Signup for an
              account!</a></div>
        </div>
      </div>
    </div>
  </body>
</html>