<div dir="ltr">Hi Grant,<div><br></div><div>Sorry to hear that. Can you do me a favor and file a bug against neutron with your env so we can try to reproduce it?</div><div>Include everything like if you have l2pop enabled, which agent you are using, and which L3 settings you are using (L3HA/DVR/etc).</div><div><br></div><div>Thanks,</div><div>Kevin Benton</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 8, 2016 at 1:50 AM, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Kevin,</p>
<p>Sorry for the late reply, We have tried doing that and we were
still seeing the same issues. I don't think the bug was quite the
same as what we were seeing. <br>
</p>
<p>Unfortunately we have had to roll back to Mitaka as we had a
tight deadline and not being able to create networks / have HA was
fairly critical. Interestingly, now we are back on Mitaka,
everything is working fine.</p>
<p>I will try and get a testing environment set up to see if I get
the same results as we were seeing when we upgraded to Newton from
Mitaka. I am not sure if it is something to do with our specific
set up, but we have followed the OSA guidelines and as everything
was working on Liberty and Mitaka I assume we have it all set up
correctly.</p>
<p>I will keep you posted to our findings, as we may be onto another
bug.</p>
<p>Regards,<br>
</p><div><div class="h5">
<br>
<div class="m_-703245763040761213moz-cite-prefix">On 06/12/16 14:07, Kevin Benton wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">There was a bug that the fixes just recently merged
for where removing a router on the L3 agent was done in the
wrong order and it resulted in issues cleaning up the interfaces
with Linux Bridge + L3HA. <a href="https://bugs.launchpad.net/neutron/+bug/1629159" target="_blank">https://bugs.launchpad.<wbr>net/neutron/+bug/1629159</a>
<div><br>
</div>
<div>It could be the case that there is an orphaned veth pair in
a deleted namespace from the same router when it was removed
from the L3 agent.</div>
<div><br>
</div>
<div>For each L3 agent, can you shutdown the L3 agent, run the
netns cleanup script, ensure all keepalived processes are
dead, and then start the agent again?</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Dec 6, 2016 at 4:59 AM, Grant
Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>They both appear to be "ACTIVE" which is what I would
expect:</p>
<p>root@management-1-utility-cont<wbr>ainer-f1222d05:~#
neutron port-show 8cd027f1-9f8c-4077-9c8a-92abc6<wbr>2fadd4<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| Field |
Value <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| admin_state_up |
True <wbr> <wbr>
|<br>
| allowed_address_pairs
| <wbr> <wbr>
|<br>
| binding:host_id |
network-1-neutron-agents-conta<wbr>iner-11d47568 <wbr>
|<br>
| binding:profile |
{} <wbr> <wbr>
|<br>
| binding:vif_details | {"port_filter":
true} <wbr> <wbr>
|<br>
| binding:vif_type |
bridge <wbr> <wbr>
|<br>
| binding:vnic_type |
normal <wbr> <wbr>
|<br>
| created_at |
2016-12-05T10:58:01Z <wbr> <wbr>
|<br>
| description
| <wbr> <wbr>
|<br>
| device_id |
a8a10308-d62f-420f-99cf-f3727e<wbr>f2b784 <wbr>
|<br>
| device_owner |
network:router_ha_interface <wbr> <wbr>
|<br>
| extra_dhcp_opts
| <wbr> <wbr>
|<br>
| fixed_ips | {"subnet_id":
"6495d542-4b78-40df-84af-31500<wbr>aaa0bf8",
"ip_address": "169.254.192.5"} |<br>
| id |
8cd027f1-9f8c-4077-9c8a-92abc6<wbr>2fadd4 <wbr>
|<br>
| mac_address |
fa:16:3e:58:a1:a4 <wbr> <wbr>
|<br>
| name | HA port tenant
e0ffdeb1e910469d9e625b95f2fa6c<wbr>54 <wbr>
|<br>
| network_id |
2b04fc3a-5c0d-4f55-996f-8888d8<wbr>bd1e1d <wbr>
|<br>
| port_security_enabled |
False <wbr> <wbr>
|<br>
| project_id
| <wbr> <wbr>
|<br>
| revision_number |
23 <wbr> <wbr>
|<br>
| security_groups
| <wbr> <wbr>
|<br>
| status |
ACTIVE <wbr> <wbr>
|<br>
| tenant_id
| <wbr> <wbr>
|<br>
| updated_at |
2016-12-06T10:18:00Z <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
root@management-1-utility-cont<wbr>ainer-f1222d05:~#
neutron port-show bda1f324-3178-46e5-8638-0f454b<wbr>a09cab<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| Field |
Value <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| admin_state_up |
True <wbr> <wbr>
|<br>
| allowed_address_pairs
| <wbr> <wbr>
|<br>
| binding:host_id |
network-2-neutron-agents-conta<wbr>iner-40906bfc <wbr>
|<br>
| binding:profile |
{} <wbr> <wbr>
|<br>
| binding:vif_details | {"port_filter":
true} <wbr> <wbr>
|<br>
| binding:vif_type |
bridge <wbr> <wbr>
|<br>
| binding:vnic_type |
normal <wbr> <wbr>
|<br>
| created_at |
2016-12-05T10:58:01Z <wbr> <wbr>
|<br>
| description
| <wbr> <wbr>
|<br>
| device_id |
a8a10308-d62f-420f-99cf-f3727e<wbr>f2b784 <wbr>
|<br>
| device_owner |
network:router_ha_interface <wbr> <wbr>
|<br>
| extra_dhcp_opts
| <wbr> <wbr>
|<br>
| fixed_ips | {"subnet_id":
"6495d542-4b78-40df-84af-31500<wbr>aaa0bf8",
"ip_address": "169.254.192.1"} |<br>
| id |
bda1f324-3178-46e5-8638-0f454b<wbr>a09cab <wbr>
|<br>
| mac_address |
fa:16:3e:c3:8a:14 <wbr> <wbr>
|<br>
| name | HA port tenant
e0ffdeb1e910469d9e625b95f2fa6c<wbr>54 <wbr>
|<br>
| network_id |
2b04fc3a-5c0d-4f55-996f-8888d8<wbr>bd1e1d <wbr>
|<br>
| port_security_enabled |
False <wbr> <wbr>
|<br>
| project_id
| <wbr> <wbr>
|<br>
| revision_number |
15 <wbr> <wbr>
|<br>
| security_groups
| <wbr> <wbr>
|<br>
| status |
ACTIVE <wbr> <wbr>
|<br>
| tenant_id
| <wbr> <wbr>
|<br>
| updated_at |
2016-12-05T14:35:16Z <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
<br>
</p>
<div>
<div class="m_-703245763040761213h5">
<p><br>
</p>
<br>
<div class="m_-703245763040761213m_-5715076046573262841moz-cite-prefix">On
06/12/16 12:53, Kevin Benton wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Can you do a 'neutron port-show' for
both of those HA ports to check their status
field?</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Dec 6, 2016 at
2:29 AM, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Kevin & Neil,</p>
<p>Many thanks for the reply. I have
attached a screen shot showing that we
cannot ping between the L3 HA nodes on the
router name spaces. This was previously
working fine with Mitaka, and has only
stopped working since the upgrade to
Newton.</p>
<p>From the packet captures and TCP dumps,
the traffic doesn't seem to be even
leaving the namespace.</p>
<p>On the attachment, the left hand side
shows the state of keepalived showing both
HA agents as master and the ring hand side
is the ping attempt.</p>
<p>Regards,<br>
</p>
<div>
<div class="m_-703245763040761213m_-5715076046573262841h5">
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119moz-cite-prefix">On
06/12/16 10:14, Kevin Benton wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Yes, that is a
misleading warning. What is
happening is that it's trying to
load the interface driver as an
alias first, which results in a
stevedore warning that you see and
then it falls back to loading it by
the class path, which is what you
have configured. We will need to see
if there is a way we can suppress
that warning somehow when we make
the call to load by an alias and it
fails.
<div><br>
</div>
<div>If you switch your interface to
just 'linuxbridge', that should
get rid of the warning.</div>
<div><br>
<div><br>
</div>
<div>For both L3 HA nodes becoming
master, we need a little more
info to figure out the root
cause. Can you try switching
into the router namespace on one
of the L3 HA nodes and see if
you can ping the other router
instance across the L3 HA
network for that router?</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Dec
5, 2016 at 7:54 AM, Neil Jerram <span dir="ltr"><<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:neil@tigera.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:neil@tigera.io" target="_blank">neil@tigera.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>I have also recently
been seeing 'Could not
load
<whatever>InterfaceDriver'
warnings from the DHCP
agent, and haven't yet
understood that -
although I'm pretty sure
that my interface driver
is being loaded really -
or else none of my
networking function
would work at all.<br>
<br>
</div>
So it's possible that that
part of your report is
benign, and just a
misleading warning. That
said, I am still worried
about it too, and would
like to understand it
properly.<br>
<br>
</div>
I'm not aware of seeing the
other symptoms you
mentioned.<br>
<br>
</div>
Neil<br>
<br>
</div>
<br>
<div class="gmail_quote">
<div>
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119h5">
<div dir="ltr">On Mon, Dec
5, 2016 at 3:14 PM Grant
Morley <<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>>
wrote:<br>
</div>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119h5">
<div text="#000000" bgcolor="#FFFFFF" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Hi
All,</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
have just upgraded
from Mitaka to
Newton. We are
running OSA and we
seem to have come
across some weird
networking issues
since the upgrade.
Basically network
access to instances
is very intermittent
and seems to
randomly stop
working. <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
are running neutron
in HA and it appears
that both of the
neutron nodes are
now trying to be
master and are both
trying to bring up
the gateway IP
addresses which
would be causing
conflicts. <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
are also seeing a
lot of the following
in the
"neutron-dhcp-agent"
log files:</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">2016-12-05
14:42:24.837 2020
WARNING
stevedore.named
[req-1955d0a1-1453-4c65-a93a-5<wbr>4e8ea39b230
1ac995c0729142289f7237222f3358<wbr>06 3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20
- - -] Could not
load
neutron.agent.linux.interface.<wbr>BridgeInterfaceDriver<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
2016-12-05
14:42:42.803 2020
INFO
neutron.agent.dhcp.agent
[req-fad7d2bb-9d3c-4192-868a-0<wbr>164b382aecf
1ac995c0729142289f7237222f3358<wbr>06
3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20 - - -] Trigger reload_allocations
for port
admin_state_up=True,
allowed_address_pairs=[], binding:host_id=, binding:profile=,
binding:vif_details=,
binding:vif_type=unbound, binding:vnic_type=normal,
created_at=2016-12-05T14:42:42<wbr>Z,
description=,
device_id=8752effa-2ff2-4ce1-b<wbr>e70-e9f2243612cb,
device_owner=network:floatingi<wbr>p, extra_dhcp_opts=[],
fixed_ips=[{u'subnet_id':
u'4ca7db2d-544a-4a97-b5a4-3cbf<wbr>2467a4b7', u'ip_address':
u'XXX.XXX.XXX.XXX'}],
id=b3cf223d-8e76-484a-a649-d8a<wbr>7dd435124,
mac_address=fa:16:3e:ff:0d:50,
name=,
network_id=af5db886-0178-4f8d-<wbr>9189-f55f773b37fa,
port_security_enabled=False, project_id=, revision_number=4,
security_groups=[],
status=N/A,
tenant_id=,
updated_at=2016-12-05T14:42:42<wbr>Z</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
am a bit concerned
about neutron not
being able to load
the Bridge interface
driver.</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Has
anyone else come
across this at all
or have any
pointers? This was
working fine in
Mitaka it just seems
since the upgrade to
Newton, we have
these issues.</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
am able to provide
more logs if they
are needed.</p>
<p class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Regards,<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</p>
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-signaturem_4273622011722304119m_-4652561670223368530gmail_msg">--
<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<div style="height:13px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</div>
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<font class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Lucida
Grande, Verdana,
Arial,
sans-serif" color="#A8D8FF">
<span style="font-size:11px;line-height:15px;float:left;margin-right:10px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<img class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</span> </font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<span class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Grant
Morley</span>
<span style="color:rgb(35,157,255);font-size:15px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</span> <font class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind,
Lucida Grande,
Verdana,
Arial,
sans-serif"> <span style="font-size:15px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</span> </font>
</div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<font class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind,
Lucida Grande,
Verdana,
Arial,
sans-serif"> <span style="font-size:15px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Cloud
Lead</span> </font>
</div>
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<span style="color:#616161" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Absolute</span>
<span style="color:#98a5b0" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">DevOps
Ltd</span> <br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
Units H, J &
K, Gateway 1000,
Whittle Way,
Stevenage,
Herts, SG1 2FP </div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-txt-link-abbreviatedm_4273622011722304119m_-4652561670223368530gmail_msg" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>
<a class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">0845
874 0580</a> </div>
</div>
</div>
</div>
</div>
</div>
______________________________<wbr>_________________<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
OpenStack-operators mailing
list<br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<a href="mailto:OpenStack-operators@lists.openstack.org" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</blockquote>
</div>
<br>
______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<div class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119moz-signature">--
<br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande,
Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px">
<img src="cid:part14.70F6E578.E836212B@absolutedevops.io">
</span> </font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
<span>Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px">
</span> <font face="Hind, Lucida
Grande, Verdana, Arial,
sans-serif"> <span style="font-size:15px"> </span>
</font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
<font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud
Lead</span> </font> </div>
<div> <span style="color:#616161">Absolute</span>
<span style="color:#98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000,
Whittle Way, Stevenage, Herts, SG1
2FP </div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
<a class="m_-703245763040761213m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a>
<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>
<a>0845 874
0580</a> </div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<div class="m_-703245763040761213m_-5715076046573262841moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial,
sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px">
<img src="cid:part18.538AA7AD.CAD7AA0A@absolutedevops.io">
</span> </font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
<span>Grant
Morley</span> <span style="color:rgb(35,157,255);font-size:15px">
</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
<font face="Hind, Lucida Grande, Verdana, Arial,
sans-serif"> <span style="font-size:15px">Cloud
Lead</span> </font> </div>
<div>
<span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way,
Stevenage, Herts, SG1 2FP </div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
<a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a class="m_-703245763040761213m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a>0845
874 0580</a> </div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<div class="m_-703245763040761213moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img src="cid:part22.C188757D.FAA19A47@absolutedevops.io" style="color:#a8d8ff;font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px"> </span>
</font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
<div style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a href="mailto:grant@absolutedevops.i" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-703245763040761213moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px">0845 874 0580</a> </div>
</div>
</div>
</div></div></div>
</blockquote></div><br></div>