<div dir="ltr">Hi Grant,<div><br></div><div>Sorry to hear that. Can you do me a favor and file a bug against neutron with your env so we can try to reproduce it?</div><div>Include everything like if you have l2pop enabled, which agent you are using, and which L3 settings you are using (L3HA/DVR/etc).</div><div><br></div><div>Thanks,</div><div>Kevin Benton</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 8, 2016 at 1:50 AM, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <p>Hi Kevin,</p>
    <p>Sorry for the late reply, We have tried doing that and we were
      still seeing the same issues.  I don't think the bug was quite the
      same as what we were seeing. <br>
    </p>
    <p>Unfortunately we have had to roll back to Mitaka as we had a
      tight deadline and not being able to create networks / have HA was
      fairly critical. Interestingly, now we are back on Mitaka,
      everything is working fine.</p>
    <p>I will try and get a testing environment set up to see if I get
      the same results as we were seeing when we upgraded to Newton from
      Mitaka. I am not sure if it is something to do with our specific
      set up, but we have followed the OSA guidelines and as everything
      was working on Liberty and Mitaka I assume we have it all set up
      correctly.</p>
    <p>I will keep you posted to our findings, as we may be onto another
      bug.</p>
    <p>Regards,<br>
    </p><div><div class="h5">
    <br>
    <div class="m_-703245763040761213moz-cite-prefix">On 06/12/16 14:07, Kevin Benton wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">There was a bug that the fixes just recently merged
        for where removing a router on the L3 agent was done in the
        wrong order and it resulted in issues cleaning up the interfaces
        with Linux Bridge + L3HA. <a href="https://bugs.launchpad.net/neutron/+bug/1629159" target="_blank">https://bugs.launchpad.<wbr>net/neutron/+bug/1629159</a>
        <div><br>
        </div>
        <div>It could be the case that there is an orphaned veth pair in
          a deleted namespace from the same router when it was removed
          from the L3 agent.</div>
        <div><br>
        </div>
        <div>For each L3 agent, can you shutdown the L3 agent, run the
          netns cleanup script, ensure all keepalived processes are
          dead, and then start the agent again?</div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Tue, Dec 6, 2016 at 4:59 AM, Grant
          Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <p>They both appear to be "ACTIVE" which is what I would
                expect:</p>
              <p>root@management-1-utility-cont<wbr>ainer-f1222d05:~#
                neutron port-show 8cd027f1-9f8c-4077-9c8a-92abc6<wbr>2fadd4<br>
                +-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
                | Field                 |
                Value                         <wbr>                              <wbr>                        
                |<br>
                +-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
                | admin_state_up        |
                True                          <wbr>                              <wbr>                        
                |<br>
                | allowed_address_pairs
                |                             <wbr>                              <wbr>                          
                |<br>
                | binding:host_id       |
                network-1-neutron-agents-conta<wbr>iner-11d47568                 <wbr>                        
                |<br>
                | binding:profile       |
                {}                            <wbr>                              <wbr>                        
                |<br>
                | binding:vif_details   | {"port_filter":
                true}                         <wbr>                              <wbr>        
                |<br>
                | binding:vif_type      |
                bridge                        <wbr>                              <wbr>                        
                |<br>
                | binding:vnic_type     |
                normal                        <wbr>                              <wbr>                        
                |<br>
                | created_at            |
                2016-12-05T10:58:01Z          <wbr>                              <wbr>                        
                |<br>
                | description          
                |                             <wbr>                              <wbr>                          
                |<br>
                | device_id             |
                a8a10308-d62f-420f-99cf-f3727e<wbr>f2b784                        <wbr>                        
                |<br>
                | device_owner          |
                network:router_ha_interface   <wbr>                              <wbr>                        
                |<br>
                | extra_dhcp_opts      
                |                             <wbr>                              <wbr>                          
                |<br>
                | fixed_ips             | {"subnet_id":
                "6495d542-4b78-40df-84af-31500<wbr>aaa0bf8",
                "ip_address": "169.254.192.5"} |<br>
                | id                    |
                8cd027f1-9f8c-4077-9c8a-92abc6<wbr>2fadd4                        <wbr>                        
                |<br>
                | mac_address           |
                fa:16:3e:58:a1:a4             <wbr>                              <wbr>                        
                |<br>
                | name                  | HA port tenant
                e0ffdeb1e910469d9e625b95f2fa6c<wbr>54                            <wbr>         
                |<br>
                | network_id            |
                2b04fc3a-5c0d-4f55-996f-8888d8<wbr>bd1e1d                        <wbr>                        
                |<br>
                | port_security_enabled |
                False                         <wbr>                              <wbr>                        
                |<br>
                | project_id           
                |                             <wbr>                              <wbr>                          
                |<br>
                | revision_number       |
                23                            <wbr>                              <wbr>                        
                |<br>
                | security_groups      
                |                             <wbr>                              <wbr>                          
                |<br>
                | status                |
                ACTIVE                        <wbr>                              <wbr>                        
                |<br>
                | tenant_id            
                |                             <wbr>                              <wbr>                          
                |<br>
                | updated_at            |
                2016-12-06T10:18:00Z          <wbr>                              <wbr>                        
                |<br>
                +-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
                root@management-1-utility-cont<wbr>ainer-f1222d05:~#
                neutron port-show bda1f324-3178-46e5-8638-0f454b<wbr>a09cab<br>
                +-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
                | Field                 |
                Value                         <wbr>                              <wbr>                        
                |<br>
                +-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
                | admin_state_up        |
                True                          <wbr>                              <wbr>                        
                |<br>
                | allowed_address_pairs
                |                             <wbr>                              <wbr>                          
                |<br>
                | binding:host_id       |
                network-2-neutron-agents-conta<wbr>iner-40906bfc                 <wbr>                        
                |<br>
                | binding:profile       |
                {}                            <wbr>                              <wbr>                        
                |<br>
                | binding:vif_details   | {"port_filter":
                true}                         <wbr>                              <wbr>        
                |<br>
                | binding:vif_type      |
                bridge                        <wbr>                              <wbr>                        
                |<br>
                | binding:vnic_type     |
                normal                        <wbr>                              <wbr>                        
                |<br>
                | created_at            |
                2016-12-05T10:58:01Z          <wbr>                              <wbr>                        
                |<br>
                | description          
                |                             <wbr>                              <wbr>                          
                |<br>
                | device_id             |
                a8a10308-d62f-420f-99cf-f3727e<wbr>f2b784                        <wbr>                        
                |<br>
                | device_owner          |
                network:router_ha_interface   <wbr>                              <wbr>                        
                |<br>
                | extra_dhcp_opts      
                |                             <wbr>                              <wbr>                          
                |<br>
                | fixed_ips             | {"subnet_id":
                "6495d542-4b78-40df-84af-31500<wbr>aaa0bf8",
                "ip_address": "169.254.192.1"} |<br>
                | id                    |
                bda1f324-3178-46e5-8638-0f454b<wbr>a09cab                        <wbr>                        
                |<br>
                | mac_address           |
                fa:16:3e:c3:8a:14             <wbr>                              <wbr>                        
                |<br>
                | name                  | HA port tenant
                e0ffdeb1e910469d9e625b95f2fa6c<wbr>54                            <wbr>         
                |<br>
                | network_id            |
                2b04fc3a-5c0d-4f55-996f-8888d8<wbr>bd1e1d                        <wbr>                        
                |<br>
                | port_security_enabled |
                False                         <wbr>                              <wbr>                        
                |<br>
                | project_id           
                |                             <wbr>                              <wbr>                          
                |<br>
                | revision_number       |
                15                            <wbr>                              <wbr>                        
                |<br>
                | security_groups      
                |                             <wbr>                              <wbr>                          
                |<br>
                | status                |
                ACTIVE                        <wbr>                              <wbr>                        
                |<br>
                | tenant_id            
                |                             <wbr>                              <wbr>                          
                |<br>
                | updated_at            |
                2016-12-05T14:35:16Z          <wbr>                              <wbr>                        
                |<br>
                +-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
                <br>
              </p>
              <div>
                <div class="m_-703245763040761213h5">
                  <p><br>
                  </p>
                  <br>
                  <div class="m_-703245763040761213m_-5715076046573262841moz-cite-prefix">On
                    06/12/16 12:53, Kevin Benton wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">Can you do a 'neutron port-show' for
                      both of those HA ports to check their status
                      field?</div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Tue, Dec 6, 2016 at
                        2:29 AM, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF">
                            <p>Hi Kevin & Neil,</p>
                            <p>Many thanks for the reply. I have
                              attached a screen shot showing that we
                              cannot ping between the L3 HA nodes on the
                              router name spaces. This was previously
                              working fine with Mitaka, and has only
                              stopped working since the upgrade to
                              Newton.</p>
                            <p>From the packet captures and TCP dumps,
                              the traffic doesn't seem to be even
                              leaving the namespace.</p>
                            <p>On the attachment, the left hand side
                              shows the state of keepalived showing both
                              HA agents as master and the ring hand side
                              is the ping attempt.</p>
                            <p>Regards,<br>
                            </p>
                            <div>
                              <div class="m_-703245763040761213m_-5715076046573262841h5">
                                <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119moz-cite-prefix">On
                                  06/12/16 10:14, Kevin Benton wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr">Yes, that is a
                                    misleading warning. What is
                                    happening is that it's trying to
                                    load the interface driver as an
                                    alias first, which results in a
                                    stevedore warning that you see and
                                    then it falls back to loading it by
                                    the class path, which is what you
                                    have configured. We will need to see
                                    if there is a way we can suppress
                                    that warning somehow when we make
                                    the call to load by an alias and it
                                    fails.
                                    <div><br>
                                    </div>
                                    <div>If you switch your interface to
                                      just 'linuxbridge', that should
                                      get rid of the warning.</div>
                                    <div><br>
                                      <div><br>
                                      </div>
                                      <div>For both L3 HA nodes becoming
                                        master, we need a little more
                                        info to figure out the root
                                        cause. Can you try switching
                                        into the router namespace on one
                                        of the L3 HA nodes and see if
                                        you can ping the other router
                                        instance across the L3 HA
                                        network for that router?</div>
                                    </div>
                                  </div>
                                  <div class="gmail_extra"><br>
                                    <div class="gmail_quote">On Mon, Dec
                                      5, 2016 at 7:54 AM, Neil Jerram <span dir="ltr"><<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:neil@tigera.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:neil@tigera.io" target="_blank">neil@tigera.io</a>></span>
                                      wrote:<br>
                                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                        <div dir="ltr">
                                          <div>
                                            <div>
                                              <div>I have also recently
                                                been seeing 'Could not
                                                load
                                                <whatever>InterfaceDriver'
                                                warnings from the DHCP
                                                agent, and haven't yet
                                                understood that -
                                                although I'm pretty sure
                                                that my interface driver
                                                is being loaded really -
                                                or else none of my
                                                networking function
                                                would work at all.<br>
                                                <br>
                                              </div>
                                              So it's possible that that
                                              part of your report is
                                              benign, and just a
                                              misleading warning.  That
                                              said, I am still worried
                                              about it too, and would
                                              like to understand it
                                              properly.<br>
                                              <br>
                                            </div>
                                            I'm not aware of seeing the
                                            other symptoms you
                                            mentioned.<br>
                                            <br>
                                          </div>
                                               Neil<br>
                                          <br>
                                        </div>
                                        <br>
                                        <div class="gmail_quote">
                                          <div>
                                            <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119h5">
                                              <div dir="ltr">On Mon, Dec
                                                5, 2016 at 3:14 PM Grant
                                                Morley <<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>>
                                                wrote:<br>
                                              </div>
                                            </div>
                                          </div>
                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                            <div>
                                              <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119h5">
                                                <div text="#000000" bgcolor="#FFFFFF" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Hi
                                                    All,</p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
                                                    have just upgraded
                                                    from Mitaka to
                                                    Newton. We are
                                                    running OSA and we
                                                    seem to have come
                                                    across some weird
                                                    networking issues
                                                    since the upgrade.
                                                    Basically network
                                                    access to instances
                                                    is very intermittent
                                                    and seems to
                                                    randomly stop
                                                    working. <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                  </p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
                                                    are running neutron
                                                    in HA and it appears
                                                    that both of the
                                                    neutron nodes are
                                                    now trying to be
                                                    master and are both
                                                    trying to bring up
                                                    the gateway IP
                                                    addresses which
                                                    would be causing
                                                    conflicts. <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                  </p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
                                                    are also seeing a
                                                    lot of the following
                                                    in the
                                                    "neutron-dhcp-agent"
                                                    log files:</p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">2016-12-05
                                                    14:42:24.837 2020
                                                    WARNING
                                                    stevedore.named
                                                    [req-1955d0a1-1453-4c65-a93a-5<wbr>4e8ea39b230
1ac995c0729142289f7237222f3358<wbr>06 3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20
                                                    - - -] Could not
                                                    load
                                                    neutron.agent.linux.interface.<wbr>BridgeInterfaceDriver<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                    2016-12-05
                                                    14:42:42.803 2020
                                                    INFO
                                                    neutron.agent.dhcp.agent
[req-fad7d2bb-9d3c-4192-868a-0<wbr>164b382aecf
                                                    1ac995c0729142289f7237222f3358<wbr>06
3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20 - - -] Trigger reload_allocations
                                                    for port
                                                    admin_state_up=True,
allowed_address_pairs=[], binding:host_id=, binding:profile=,
                                                    binding:vif_details=,
binding:vif_type=unbound, binding:vnic_type=normal,
                                                    created_at=2016-12-05T14:42:42<wbr>Z,
                                                    description=,
                                                    device_id=8752effa-2ff2-4ce1-b<wbr>e70-e9f2243612cb,
device_owner=network:floatingi<wbr>p, extra_dhcp_opts=[],
                                                    fixed_ips=[{u'subnet_id':
u'4ca7db2d-544a-4a97-b5a4-3cbf<wbr>2467a4b7', u'ip_address':
                                                    u'XXX.XXX.XXX.XXX'}],
id=b3cf223d-8e76-484a-a649-d8a<wbr>7dd435124,
                                                    mac_address=fa:16:3e:ff:0d:50,
                                                    name=,
                                                    network_id=af5db886-0178-4f8d-<wbr>9189-f55f773b37fa,
port_security_enabled=False, project_id=, revision_number=4,
                                                    security_groups=[],
                                                    status=N/A,
                                                    tenant_id=,
                                                    updated_at=2016-12-05T14:42:42<wbr>Z</p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
                                                    am a bit concerned
                                                    about neutron not
                                                    being able to load
                                                    the Bridge interface
                                                    driver.</p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Has
                                                    anyone else come
                                                    across this at all
                                                    or have any
                                                    pointers? This was
                                                    working fine in
                                                    Mitaka it just seems
                                                    since the upgrade to
                                                    Newton, we have
                                                    these issues.</p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
                                                    am able to provide
                                                    more logs if they
                                                    are needed.</p>
                                                  <p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Regards,<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                  </p>
                                                  <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-signaturem_4273622011722304119m_-4652561670223368530gmail_msg">--
                                                    <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                    <div style="height:13px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                    </div>
                                                    <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                      <font class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Lucida
                                                        Grande, Verdana,
                                                        Arial,
                                                        sans-serif" color="#A8D8FF">
                                                        <span style="font-size:11px;line-height:15px;float:left;margin-right:10px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                          <img class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        </span> </font>
                                                      <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        <span class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Grant
                                                          Morley</span>
                                                        <span style="color:rgb(35,157,255);font-size:15px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        </span> <font class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind,
                                                          Lucida Grande,
                                                          Verdana,
                                                          Arial,
                                                          sans-serif"> <span style="font-size:15px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                          </span> </font>
                                                      </div>
                                                      <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        <font class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind,
                                                          Lucida Grande,
                                                          Verdana,
                                                          Arial,
                                                          sans-serif"> <span style="font-size:15px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Cloud
                                                          Lead</span> </font>
                                                      </div>
                                                      <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        <span style="color:#616161" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Absolute</span>
                                                        <span style="color:#98a5b0" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">DevOps
                                                          Ltd</span> <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        Units H, J &
                                                        K, Gateway 1000,
                                                        Whittle Way,
                                                        Stevenage,
                                                        Herts, SG1 2FP </div>
                                                      <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                                        <a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-txt-link-abbreviatedm_4273622011722304119m_-4652561670223368530gmail_msg" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>
                                                        <a class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">0845
                                                          874 0580</a> </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
______________________________<wbr>_________________<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                            OpenStack-operators mailing
                                            list<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                            <a href="mailto:OpenStack-operators@lists.openstack.org" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                            <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                          </blockquote>
                                        </div>
                                        <br>
                                        ______________________________<wbr>_________________<br>
                                        OpenStack-operators mailing list<br>
                                        <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br>
                                        <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br>
                                        <br>
                                      </blockquote>
                                    </div>
                                    <br>
                                  </div>
                                </blockquote>
                                <br>
                                <div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119moz-signature">--
                                  <br>
                                  <div style="height:13px"> </div>
                                  <div> <font face="Lucida Grande,
                                      Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px">
                                        <img src="cid:part14.70F6E578.E836212B@absolutedevops.io">
                                      </span> </font>
                                    <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
                                      <span>Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px">
                                      </span> <font face="Hind, Lucida
                                        Grande, Verdana, Arial,
                                        sans-serif"> <span style="font-size:15px"> </span>
                                      </font> </div>
                                    <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
                                      <font face="Hind, Lucida Grande,
                                        Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud
                                          Lead</span> </font> </div>
                                    <div> <span style="color:#616161">Absolute</span>
                                      <span style="color:#98a5b0">DevOps
                                        Ltd</span> <br>
                                      Units H, J & K, Gateway 1000,
                                      Whittle Way, Stevenage, Herts, SG1
                                      2FP </div>
                                    <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
                                      <a class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a>
                                      <a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>
                                      <a>0845 874
                                        0580</a> </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </blockquote>
                  <br>
                  <div class="m_-703245763040761213m_-5715076046573262841moz-signature">-- <br>
                    <div style="height:13px"> </div>
                    <div> <font face="Lucida Grande, Verdana, Arial,
                        sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px">
                          <img src="cid:part18.538AA7AD.CAD7AA0A@absolutedevops.io">
                        </span> </font>
                      <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
                        <span>Grant
                          Morley</span> <span style="color:rgb(35,157,255);font-size:15px">
                        </span> <font face="Hind, Lucida Grande,
                          Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
                      <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
                        <font face="Hind, Lucida Grande, Verdana, Arial,
                          sans-serif"> <span style="font-size:15px">Cloud
                            Lead</span> </font> </div>
                      <div>
                        <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps Ltd</span> <br>
                        Units H, J & K, Gateway 1000, Whittle Way,
                        Stevenage, Herts, SG1 2FP </div>
                      <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
                        <a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a>0845
                          874 0580</a> </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <div class="m_-703245763040761213moz-signature">-- <br>
      
      
      <div style="height:13px"> </div>
      <div> <font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img src="cid:part22.C188757D.FAA19A47@absolutedevops.io" style="color:#a8d8ff;font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px"> </span>
        </font>
        <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
        <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind,
            Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
        <div style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps
            Ltd</span> <br>
          Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
          Herts, SG1 2FP
        </div>
        <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a href="mailto:grant@absolutedevops.i" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px">0845 874 0580</a> </div>
      </div>
    </div>
  </div></div></div>

</blockquote></div><br></div>