<div dir="ltr">There was a bug that the fixes just recently merged for where removing a router on the L3 agent was done in the wrong order and it resulted in issues cleaning up the interfaces with Linux Bridge + L3HA. <a href="https://bugs.launchpad.net/neutron/+bug/1629159">https://bugs.launchpad.net/neutron/+bug/1629159</a><div><br></div><div>It could be the case that there is an orphaned veth pair in a deleted namespace from the same router when it was removed from the L3 agent.</div><div><br></div><div>For each L3 agent, can you shutdown the L3 agent, run the netns cleanup script, ensure all keepalived processes are dead, and then start the agent again?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 6, 2016 at 4:59 AM, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <p>They both appear to be "ACTIVE" which is what I would expect:</p>
    <p>root@management-1-utility-<wbr>container-f1222d05:~# neutron port-show
      8cd027f1-9f8c-4077-9c8a-<wbr>92abc62fadd4<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
      | Field                 |
Value                         <wbr>                              <wbr>                        
      |<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
      | admin_state_up        |
True                          <wbr>                              <wbr>                        
      |<br>
      | allowed_address_pairs
|                             <wbr>                              <wbr>                          
      |<br>
      | binding:host_id       |
network-1-neutron-agents-<wbr>container-11d47568            <wbr>                             
      |<br>
      | binding:profile       |
{}                            <wbr>                              <wbr>                        
      |<br>
      | binding:vif_details   | {"port_filter":
      true}                         <wbr>                              <wbr>        
      |<br>
      | binding:vif_type      |
bridge                        <wbr>                              <wbr>                        
      |<br>
      | binding:vnic_type     |
normal                        <wbr>                              <wbr>                        
      |<br>
      | created_at            |
2016-12-05T10:58:01Z          <wbr>                              <wbr>                        
      |<br>
      | description          
|                             <wbr>                              <wbr>                          
      |<br>
      | device_id             |
a8a10308-d62f-420f-99cf-<wbr>f3727ef2b784                  <wbr>                              
      |<br>
      | device_owner          |
network:router_ha_interface   <wbr>                              <wbr>                        
      |<br>
      | extra_dhcp_opts      
|                             <wbr>                              <wbr>                          
      |<br>
      | fixed_ips             | {"subnet_id":
      "6495d542-4b78-40df-84af-<wbr>31500aaa0bf8", "ip_address":
      "169.254.192.5"} |<br>
      | id                    |
8cd027f1-9f8c-4077-9c8a-<wbr>92abc62fadd4                  <wbr>                              
      |<br>
      | mac_address           |
fa:16:3e:58:a1:a4             <wbr>                              <wbr>                        
      |<br>
      | name                  | HA port tenant
      e0ffdeb1e910469d9e625b95f2fa6c<wbr>54                            <wbr>         
      |<br>
      | network_id            |
2b04fc3a-5c0d-4f55-996f-<wbr>8888d8bd1e1d                  <wbr>                              
      |<br>
      | port_security_enabled |
False                         <wbr>                              <wbr>                        
      |<br>
      | project_id           
|                             <wbr>                              <wbr>                          
      |<br>
      | revision_number       |
23                            <wbr>                              <wbr>                        
      |<br>
      | security_groups      
|                             <wbr>                              <wbr>                          
      |<br>
      | status                |
ACTIVE                        <wbr>                              <wbr>                        
      |<br>
      | tenant_id            
|                             <wbr>                              <wbr>                          
      |<br>
      | updated_at            |
2016-12-06T10:18:00Z          <wbr>                              <wbr>                        
      |<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
      root@management-1-utility-<wbr>container-f1222d05:~# neutron port-show
      bda1f324-3178-46e5-8638-<wbr>0f454ba09cab<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
      | Field                 |
Value                         <wbr>                              <wbr>                        
      |<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
      | admin_state_up        |
True                          <wbr>                              <wbr>                        
      |<br>
      | allowed_address_pairs
|                             <wbr>                              <wbr>                          
      |<br>
      | binding:host_id       |
network-2-neutron-agents-<wbr>container-40906bfc            <wbr>                             
      |<br>
      | binding:profile       |
{}                            <wbr>                              <wbr>                        
      |<br>
      | binding:vif_details   | {"port_filter":
      true}                         <wbr>                              <wbr>        
      |<br>
      | binding:vif_type      |
bridge                        <wbr>                              <wbr>                        
      |<br>
      | binding:vnic_type     |
normal                        <wbr>                              <wbr>                        
      |<br>
      | created_at            |
2016-12-05T10:58:01Z          <wbr>                              <wbr>                        
      |<br>
      | description          
|                             <wbr>                              <wbr>                          
      |<br>
      | device_id             |
a8a10308-d62f-420f-99cf-<wbr>f3727ef2b784                  <wbr>                              
      |<br>
      | device_owner          |
network:router_ha_interface   <wbr>                              <wbr>                        
      |<br>
      | extra_dhcp_opts      
|                             <wbr>                              <wbr>                          
      |<br>
      | fixed_ips             | {"subnet_id":
      "6495d542-4b78-40df-84af-<wbr>31500aaa0bf8", "ip_address":
      "169.254.192.1"} |<br>
      | id                    |
bda1f324-3178-46e5-8638-<wbr>0f454ba09cab                  <wbr>                              
      |<br>
      | mac_address           |
fa:16:3e:c3:8a:14             <wbr>                              <wbr>                        
      |<br>
      | name                  | HA port tenant
      e0ffdeb1e910469d9e625b95f2fa6c<wbr>54                            <wbr>         
      |<br>
      | network_id            |
2b04fc3a-5c0d-4f55-996f-<wbr>8888d8bd1e1d                  <wbr>                              
      |<br>
      | port_security_enabled |
False                         <wbr>                              <wbr>                        
      |<br>
      | project_id           
|                             <wbr>                              <wbr>                          
      |<br>
      | revision_number       |
15                            <wbr>                              <wbr>                        
      |<br>
      | security_groups      
|                             <wbr>                              <wbr>                          
      |<br>
      | status                |
ACTIVE                        <wbr>                              <wbr>                        
      |<br>
      | tenant_id            
|                             <wbr>                              <wbr>                          
      |<br>
      | updated_at            |
2016-12-05T14:35:16Z          <wbr>                              <wbr>                        
      |<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
      <br>
    </p><div><div class="h5">
    <p><br>
    </p>
    <br>
    <div class="m_-5715076046573262841moz-cite-prefix">On 06/12/16 12:53, Kevin Benton wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Can you do a 'neutron port-show' for both of those
        HA ports to check their status field?</div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Tue, Dec 6, 2016 at 2:29 AM, Grant
          Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <p>Hi Kevin & Neil,</p>
              <p>Many thanks for the reply. I have attached a screen
                shot showing that we cannot ping between the L3 HA nodes
                on the router name spaces. This was previously working
                fine with Mitaka, and has only stopped working since the
                upgrade to Newton.</p>
              <p>From the packet captures and TCP dumps, the traffic
                doesn't seem to be even leaving the namespace.</p>
              <p>On the attachment, the left hand side shows the state
                of keepalived showing both HA agents as master and the
                ring hand side is the ping attempt.</p>
              <p>Regards,<br>
              </p>
              <div>
                <div class="m_-5715076046573262841h5">
                  <div class="m_-5715076046573262841m_4273622011722304119moz-cite-prefix">On
                    06/12/16 10:14, Kevin Benton wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">Yes, that is a misleading warning.
                      What is happening is that it's trying to load the
                      interface driver as an alias first, which results
                      in a stevedore warning that you see and then it
                      falls back to loading it by the class path, which
                      is what you have configured. We will need to see
                      if there is a way we can suppress that warning
                      somehow when we make the call to load by an alias
                      and it fails.
                      <div><br>
                      </div>
                      <div>If you switch your interface to just
                        'linuxbridge', that should get rid of the
                        warning.</div>
                      <div><br>
                        <div><br>
                        </div>
                        <div>For both L3 HA nodes becoming master, we
                          need a little more info to figure out the root
                          cause. Can you try switching into the router
                          namespace on one of the L3 HA nodes and see if
                          you can ping the other router instance across
                          the L3 HA network for that router?</div>
                      </div>
                    </div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Mon, Dec 5, 2016 at
                        7:54 AM, Neil Jerram <span dir="ltr"><<a href="mailto:neil@tigera.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:neil@tigera.io" target="_blank">neil@tigera.io</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div dir="ltr">
                            <div>
                              <div>
                                <div>I have also recently been seeing
                                  'Could not load
                                  <whatever>InterfaceDriver'
                                  warnings from the DHCP agent, and
                                  haven't yet understood that - although
                                  I'm pretty sure that my interface
                                  driver is being loaded really - or
                                  else none of my networking function
                                  would work at all.<br>
                                  <br>
                                </div>
                                So it's possible that that part of your
                                report is benign, and just a misleading
                                warning.  That said, I am still worried
                                about it too, and would like to
                                understand it properly.<br>
                                <br>
                              </div>
                              I'm not aware of seeing the other symptoms
                              you mentioned.<br>
                              <br>
                            </div>
                                 Neil<br>
                            <br>
                          </div>
                          <br>
                          <div class="gmail_quote">
                            <div>
                              <div class="m_-5715076046573262841m_4273622011722304119h5">
                                <div dir="ltr">On Mon, Dec 5, 2016 at
                                  3:14 PM Grant Morley <<a href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>>
                                  wrote:<br>
                                </div>
                              </div>
                            </div>
                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                              <div>
                                <div class="m_-5715076046573262841m_4273622011722304119h5">
                                  <div text="#000000" bgcolor="#FFFFFF" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Hi
                                      All,</p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
                                      have just upgraded from Mitaka to
                                      Newton. We are running OSA and we
                                      seem to have come across some
                                      weird networking issues since the
                                      upgrade. Basically network access
                                      to instances is very intermittent
                                      and seems to randomly stop
                                      working. <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                    </p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
                                      are running neutron in HA and it
                                      appears that both of the neutron
                                      nodes are now trying to be master
                                      and are both trying to bring up
                                      the gateway IP addresses which
                                      would be causing conflicts. <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                    </p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
                                      are also seeing a lot of the
                                      following in the
                                      "neutron-dhcp-agent" log files:</p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">2016-12-05
                                      14:42:24.837 2020 WARNING
                                      stevedore.named
                                      [req-1955d0a1-1453-4c65-a93a-5<wbr>4e8ea39b230
                                      1ac995c0729142289f7237222f3358<wbr>06
                                      3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20
                                      - - -] Could not load
                                      neutron.agent.linux.interface.<wbr>BridgeInterfaceDriver<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                      2016-12-05 14:42:42.803 2020 INFO
                                      neutron.agent.dhcp.agent
                                      [req-fad7d2bb-9d3c-4192-868a-0<wbr>164b382aecf
                                      1ac995c0729142289f7237222f3358<wbr>06
                                      3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20
                                      - - -] Trigger reload_allocations
                                      for port admin_state_up=True,
                                      allowed_address_pairs=[],
                                      binding:host_id=,
                                      binding:profile=,
                                      binding:vif_details=,
                                      binding:vif_type=unbound,
                                      binding:vnic_type=normal,
                                      created_at=2016-12-05T14:42:42<wbr>Z,
                                      description=,
                                      device_id=8752effa-2ff2-4ce1-b<wbr>e70-e9f2243612cb,
                                      device_owner=network:floatingi<wbr>p,
                                      extra_dhcp_opts=[],
                                      fixed_ips=[{u'subnet_id':
                                      u'4ca7db2d-544a-4a97-b5a4-3cbf<wbr>2467a4b7',
                                      u'ip_address':
                                      u'XXX.XXX.XXX.XXX'}],
                                      id=b3cf223d-8e76-484a-a649-d8a<wbr>7dd435124,
                                      mac_address=fa:16:3e:ff:0d:50,
                                      name=,
                                      network_id=af5db886-0178-4f8d-<wbr>9189-f55f773b37fa,
                                      port_security_enabled=False,
                                      project_id=, revision_number=4,
                                      security_groups=[], status=N/A,
                                      tenant_id=,
                                      updated_at=2016-12-05T14:42:42<wbr>Z</p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
                                      am a bit concerned about neutron
                                      not being able to load the Bridge
                                      interface driver.</p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Has
                                      anyone else come across this at
                                      all or have any pointers? This was
                                      working fine in Mitaka it just
                                      seems since the upgrade to Newton,
                                      we have these issues.</p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
                                      am able to provide more logs if
                                      they are needed.</p>
                                    <p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Regards,<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                    </p>
                                    <div class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-signature
m_4273622011722304119m_-4652561670223368530gmail_msg">-- <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                      <div style="height:13px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                      </div>
                                      <div class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                        <font class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Lucida Grande, Verdana,
                                          Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <img class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                          </span> </font>
                                        <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <span class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Grant
                                            Morley</span> <span style="color:rgb(35,157,255);font-size:15px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> </span> <font class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind,
                                            Lucida Grande, Verdana,
                                            Arial, sans-serif"> <span style="font-size:15px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                            </span> </font> </div>
                                        <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <font class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind, Lucida Grande,
                                            Verdana, Arial, sans-serif">
                                            <span style="font-size:15px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Cloud Lead</span>
                                          </font> </div>
                                        <div class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                          <span style="color:#616161" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Absolute</span>
                                          <span style="color:#98a5b0" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">DevOps
                                            Ltd</span> <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                                          Units H, J & K, Gateway
                                          1000, Whittle Way, Stevenage,
                                          Herts, SG1 2FP </div>
                                        <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <a class="m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a>
                                          <a class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-txt-link-abbreviated
m_4273622011722304119m_-4652561670223368530gmail_msg" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>
                                          <a class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">0845
                                            874 0580</a> </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                              ______________________________<wbr>_________________<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                              OpenStack-operators mailing list<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                              <a href="mailto:OpenStack-operators@lists.openstack.org" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                              <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
                            </blockquote>
                          </div>
                          <br>
                          ______________________________<wbr>_________________<br>
                          OpenStack-operators mailing list<br>
                          <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br>
                          <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br>
                          <br>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </blockquote>
                  <br>
                  <div class="m_-5715076046573262841m_4273622011722304119moz-signature">-- <br>
                    <div style="height:13px"> </div>
                    <div> <font face="Lucida Grande, Verdana, Arial,
                        sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px">
                          <img src="cid:part12.1A418A5E.0DC4605D@absolutedevops.io">
                        </span> </font>
                      <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
                        <span>Grant
                          Morley</span> <span style="color:rgb(35,157,255);font-size:15px">
                        </span> <font face="Hind, Lucida Grande,
                          Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
                      <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
                        <font face="Hind, Lucida Grande, Verdana, Arial,
                          sans-serif"> <span style="font-size:15px">Cloud
                            Lead</span> </font> </div>
                      <div>
                        <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps Ltd</span> <br>
                        Units H, J & K, Gateway 1000, Whittle Way,
                        Stevenage, Herts, SG1 2FP </div>
                      <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
                        <a class="m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a class="m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a>0845
                          874 0580</a> </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <div class="m_-5715076046573262841moz-signature">-- <br>
      
      
      <div style="height:13px"> </div>
      <div> <font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img src="cid:part16.9EF23BD0.C65D693F@absolutedevops.io" style="color:#a8d8ff;font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px"> </span>
        </font>
        <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
        <div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind,
            Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
        <div style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps
            Ltd</span> <br>
          Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
          Herts, SG1 2FP
        </div>
        <div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a href="mailto:grant@absolutedevops.i" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px">0845 874 0580</a> </div>
      </div>
    </div>
  </div></div></div>

</blockquote></div><br></div>