<div dir="ltr">There was a bug that the fixes just recently merged for where removing a router on the L3 agent was done in the wrong order and it resulted in issues cleaning up the interfaces with Linux Bridge + L3HA. <a href="https://bugs.launchpad.net/neutron/+bug/1629159">https://bugs.launchpad.net/neutron/+bug/1629159</a><div><br></div><div>It could be the case that there is an orphaned veth pair in a deleted namespace from the same router when it was removed from the L3 agent.</div><div><br></div><div>For each L3 agent, can you shutdown the L3 agent, run the netns cleanup script, ensure all keepalived processes are dead, and then start the agent again?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 6, 2016 at 4:59 AM, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>They both appear to be "ACTIVE" which is what I would expect:</p>
<p>root@management-1-utility-<wbr>container-f1222d05:~# neutron port-show
8cd027f1-9f8c-4077-9c8a-<wbr>92abc62fadd4<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| Field |
Value <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| admin_state_up |
True <wbr> <wbr>
|<br>
| allowed_address_pairs
| <wbr> <wbr>
|<br>
| binding:host_id |
network-1-neutron-agents-<wbr>container-11d47568 <wbr>
|<br>
| binding:profile |
{} <wbr> <wbr>
|<br>
| binding:vif_details | {"port_filter":
true} <wbr> <wbr>
|<br>
| binding:vif_type |
bridge <wbr> <wbr>
|<br>
| binding:vnic_type |
normal <wbr> <wbr>
|<br>
| created_at |
2016-12-05T10:58:01Z <wbr> <wbr>
|<br>
| description
| <wbr> <wbr>
|<br>
| device_id |
a8a10308-d62f-420f-99cf-<wbr>f3727ef2b784 <wbr>
|<br>
| device_owner |
network:router_ha_interface <wbr> <wbr>
|<br>
| extra_dhcp_opts
| <wbr> <wbr>
|<br>
| fixed_ips | {"subnet_id":
"6495d542-4b78-40df-84af-<wbr>31500aaa0bf8", "ip_address":
"169.254.192.5"} |<br>
| id |
8cd027f1-9f8c-4077-9c8a-<wbr>92abc62fadd4 <wbr>
|<br>
| mac_address |
fa:16:3e:58:a1:a4 <wbr> <wbr>
|<br>
| name | HA port tenant
e0ffdeb1e910469d9e625b95f2fa6c<wbr>54 <wbr>
|<br>
| network_id |
2b04fc3a-5c0d-4f55-996f-<wbr>8888d8bd1e1d <wbr>
|<br>
| port_security_enabled |
False <wbr> <wbr>
|<br>
| project_id
| <wbr> <wbr>
|<br>
| revision_number |
23 <wbr> <wbr>
|<br>
| security_groups
| <wbr> <wbr>
|<br>
| status |
ACTIVE <wbr> <wbr>
|<br>
| tenant_id
| <wbr> <wbr>
|<br>
| updated_at |
2016-12-06T10:18:00Z <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
root@management-1-utility-<wbr>container-f1222d05:~# neutron port-show
bda1f324-3178-46e5-8638-<wbr>0f454ba09cab<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| Field |
Value <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
| admin_state_up |
True <wbr> <wbr>
|<br>
| allowed_address_pairs
| <wbr> <wbr>
|<br>
| binding:host_id |
network-2-neutron-agents-<wbr>container-40906bfc <wbr>
|<br>
| binding:profile |
{} <wbr> <wbr>
|<br>
| binding:vif_details | {"port_filter":
true} <wbr> <wbr>
|<br>
| binding:vif_type |
bridge <wbr> <wbr>
|<br>
| binding:vnic_type |
normal <wbr> <wbr>
|<br>
| created_at |
2016-12-05T10:58:01Z <wbr> <wbr>
|<br>
| description
| <wbr> <wbr>
|<br>
| device_id |
a8a10308-d62f-420f-99cf-<wbr>f3727ef2b784 <wbr>
|<br>
| device_owner |
network:router_ha_interface <wbr> <wbr>
|<br>
| extra_dhcp_opts
| <wbr> <wbr>
|<br>
| fixed_ips | {"subnet_id":
"6495d542-4b78-40df-84af-<wbr>31500aaa0bf8", "ip_address":
"169.254.192.1"} |<br>
| id |
bda1f324-3178-46e5-8638-<wbr>0f454ba09cab <wbr>
|<br>
| mac_address |
fa:16:3e:c3:8a:14 <wbr> <wbr>
|<br>
| name | HA port tenant
e0ffdeb1e910469d9e625b95f2fa6c<wbr>54 <wbr>
|<br>
| network_id |
2b04fc3a-5c0d-4f55-996f-<wbr>8888d8bd1e1d <wbr>
|<br>
| port_security_enabled |
False <wbr> <wbr>
|<br>
| project_id
| <wbr> <wbr>
|<br>
| revision_number |
15 <wbr> <wbr>
|<br>
| security_groups
| <wbr> <wbr>
|<br>
| status |
ACTIVE <wbr> <wbr>
|<br>
| tenant_id
| <wbr> <wbr>
|<br>
| updated_at |
2016-12-05T14:35:16Z <wbr> <wbr>
|<br>
+-----------------------+-----<wbr>------------------------------<wbr>------------------------------<wbr>---------------------+<br>
<br>
</p><div><div class="h5">
<p><br>
</p>
<br>
<div class="m_-5715076046573262841moz-cite-prefix">On 06/12/16 12:53, Kevin Benton wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Can you do a 'neutron port-show' for both of those
HA ports to check their status field?</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Dec 6, 2016 at 2:29 AM, Grant
Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Kevin & Neil,</p>
<p>Many thanks for the reply. I have attached a screen
shot showing that we cannot ping between the L3 HA nodes
on the router name spaces. This was previously working
fine with Mitaka, and has only stopped working since the
upgrade to Newton.</p>
<p>From the packet captures and TCP dumps, the traffic
doesn't seem to be even leaving the namespace.</p>
<p>On the attachment, the left hand side shows the state
of keepalived showing both HA agents as master and the
ring hand side is the ping attempt.</p>
<p>Regards,<br>
</p>
<div>
<div class="m_-5715076046573262841h5">
<div class="m_-5715076046573262841m_4273622011722304119moz-cite-prefix">On
06/12/16 10:14, Kevin Benton wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Yes, that is a misleading warning.
What is happening is that it's trying to load the
interface driver as an alias first, which results
in a stevedore warning that you see and then it
falls back to loading it by the class path, which
is what you have configured. We will need to see
if there is a way we can suppress that warning
somehow when we make the call to load by an alias
and it fails.
<div><br>
</div>
<div>If you switch your interface to just
'linuxbridge', that should get rid of the
warning.</div>
<div><br>
<div><br>
</div>
<div>For both L3 HA nodes becoming master, we
need a little more info to figure out the root
cause. Can you try switching into the router
namespace on one of the L3 HA nodes and see if
you can ping the other router instance across
the L3 HA network for that router?</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Dec 5, 2016 at
7:54 AM, Neil Jerram <span dir="ltr"><<a href="mailto:neil@tigera.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:neil@tigera.io" target="_blank">neil@tigera.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>I have also recently been seeing
'Could not load
<whatever>InterfaceDriver'
warnings from the DHCP agent, and
haven't yet understood that - although
I'm pretty sure that my interface
driver is being loaded really - or
else none of my networking function
would work at all.<br>
<br>
</div>
So it's possible that that part of your
report is benign, and just a misleading
warning. That said, I am still worried
about it too, and would like to
understand it properly.<br>
<br>
</div>
I'm not aware of seeing the other symptoms
you mentioned.<br>
<br>
</div>
Neil<br>
<br>
</div>
<br>
<div class="gmail_quote">
<div>
<div class="m_-5715076046573262841m_4273622011722304119h5">
<div dir="ltr">On Mon, Dec 5, 2016 at
3:14 PM Grant Morley <<a href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>>
wrote:<br>
</div>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="m_-5715076046573262841m_4273622011722304119h5">
<div text="#000000" bgcolor="#FFFFFF" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Hi
All,</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
have just upgraded from Mitaka to
Newton. We are running OSA and we
seem to have come across some
weird networking issues since the
upgrade. Basically network access
to instances is very intermittent
and seems to randomly stop
working. <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
are running neutron in HA and it
appears that both of the neutron
nodes are now trying to be master
and are both trying to bring up
the gateway IP addresses which
would be causing conflicts. <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">We
are also seeing a lot of the
following in the
"neutron-dhcp-agent" log files:</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">2016-12-05
14:42:24.837 2020 WARNING
stevedore.named
[req-1955d0a1-1453-4c65-a93a-5<wbr>4e8ea39b230
1ac995c0729142289f7237222f3358<wbr>06
3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20
- - -] Could not load
neutron.agent.linux.interface.<wbr>BridgeInterfaceDriver<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
2016-12-05 14:42:42.803 2020 INFO
neutron.agent.dhcp.agent
[req-fad7d2bb-9d3c-4192-868a-0<wbr>164b382aecf
1ac995c0729142289f7237222f3358<wbr>06
3cc95dbe91c84e3e8ebbb9893ee54d<wbr>20
- - -] Trigger reload_allocations
for port admin_state_up=True,
allowed_address_pairs=[],
binding:host_id=,
binding:profile=,
binding:vif_details=,
binding:vif_type=unbound,
binding:vnic_type=normal,
created_at=2016-12-05T14:42:42<wbr>Z,
description=,
device_id=8752effa-2ff2-4ce1-b<wbr>e70-e9f2243612cb,
device_owner=network:floatingi<wbr>p,
extra_dhcp_opts=[],
fixed_ips=[{u'subnet_id':
u'4ca7db2d-544a-4a97-b5a4-3cbf<wbr>2467a4b7',
u'ip_address':
u'XXX.XXX.XXX.XXX'}],
id=b3cf223d-8e76-484a-a649-d8a<wbr>7dd435124,
mac_address=fa:16:3e:ff:0d:50,
name=,
network_id=af5db886-0178-4f8d-<wbr>9189-f55f773b37fa,
port_security_enabled=False,
project_id=, revision_number=4,
security_groups=[], status=N/A,
tenant_id=,
updated_at=2016-12-05T14:42:42<wbr>Z</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
am a bit concerned about neutron
not being able to load the Bridge
interface driver.</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Has
anyone else come across this at
all or have any pointers? This was
working fine in Mitaka it just
seems since the upgrade to Newton,
we have these issues.</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">I
am able to provide more logs if
they are needed.</p>
<p class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Regards,<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</p>
<div class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-signature
m_4273622011722304119m_-4652561670223368530gmail_msg">-- <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<div style="height:13px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</div>
<div class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<font class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Lucida Grande, Verdana,
Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <img class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</span> </font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <span class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Grant
Morley</span> <span style="color:rgb(35,157,255);font-size:15px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> </span> <font class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind,
Lucida Grande, Verdana,
Arial, sans-serif"> <span style="font-size:15px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <font class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" face="Hind, Lucida Grande,
Verdana, Arial, sans-serif">
<span style="font-size:15px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Cloud Lead</span>
</font> </div>
<div class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<span style="color:#616161" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">Absolute</span>
<span style="color:#98a5b0" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">DevOps
Ltd</span> <br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
Units H, J & K, Gateway
1000, Whittle Way, Stevenage,
Herts, SG1 2FP </div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg"> <a class="m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a>
<a class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530m_-994859292978489524moz-txt-link-abbreviated
m_4273622011722304119m_-4652561670223368530gmail_msg" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>
<a class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">0845
874 0580</a> </div>
</div>
</div>
</div>
</div>
</div>
______________________________<wbr>_________________<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
OpenStack-operators mailing list<br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<a href="mailto:OpenStack-operators@lists.openstack.org" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br class="m_-5715076046573262841m_4273622011722304119m_-4652561670223368530gmail_msg">
</blockquote>
</div>
<br>
______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<div class="m_-5715076046573262841m_4273622011722304119moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial,
sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px">
<img src="cid:part12.1A418A5E.0DC4605D@absolutedevops.io">
</span> </font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
<span>Grant
Morley</span> <span style="color:rgb(35,157,255);font-size:15px">
</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
<font face="Hind, Lucida Grande, Verdana, Arial,
sans-serif"> <span style="font-size:15px">Cloud
Lead</span> </font> </div>
<div>
<span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way,
Stevenage, Herts, SG1 2FP </div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
<a class="m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a class="m_-5715076046573262841m_4273622011722304119moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a>0845
874 0580</a> </div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<div class="m_-5715076046573262841moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img src="cid:part16.9EF23BD0.C65D693F@absolutedevops.io" style="color:#a8d8ff;font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px"> </span>
</font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
<div style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="http://www.absolutedevops.io" target="_blank">www.absolutedevops.io</a> <a href="mailto:grant@absolutedevops.i" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank"></a><a class="m_-5715076046573262841moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a> <a style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px">0845 874 0580</a> </div>
</div>
</div>
</div></div></div>
</blockquote></div><br></div>