Open Stack

Hi All,

We have  noticed in our neutron-l3-agent logs that there are a number of 
routers that neutron seems to think exist ( but they no longer exist in 
the data base )  and it is constantly trying to delete them. However 
because they don't appear to exist,  so neutron seems to get stuck in a 
loop of trying to delete  routers that are no longer there.

We originally noticed that neutron was trying to delete the 
"qrouter-xxxx-xxx-xxxx" and that didn't exist. So we added the router 
manually with "ip netns add qrouter-xxx-xxx-xxx-xxx" which then deletes 
the router however we then get the following error messages:

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent [-] Error while 
deleting router da7b633a-233b-46d1-ba3d-315b3bee6a61
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent Traceback (most 
recent call last):
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/agent.py", 
line 359, in _safe_router_removed
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self._router_removed(router_id)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/agent.py", 
line 377, in _router_removed
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent ri.delete(self)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", 
line 380, in delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent super(HaRouter, 
self).delete(agent)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 361, in delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.process_delete(agent)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/common/utils.py", 
line 385, in call
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent self.logger(e)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.force_reraise()
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
six.reraise(self.type_, self.value, self.tb)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/common/utils.py", 
line 382, in call
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent return 
func(*args, **kwargs)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 972, in process_delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self._process_external_on_delete(agent)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 794, in _process_external_on_delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self._process_external_gateway(ex_gw_port, agent.pd)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 693, in _process_external_gateway
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.external_gateway_removed(self.ex_gw_port, interface_name)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", 
line 371, in external_gateway_removed
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent interface_name)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 668, in external_gateway_removed
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
ip_addr['prefixlen']))
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 286, in remove_external_gateway_ip
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
device.delete_addr_and_conntrack_state(ip_cidr)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 290, in delete_addr_and_conntrack_state
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.addr.delete(cidr)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 580, in delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 'dev', self.name))
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 361, in _as_root
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
use_root_namespace=use_root_namespace)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 94, in _as_root
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
log_fail_as_error=self.log_fail_as_error)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 103, in _execute
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
log_fail_as_error=log_fail_as_error)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/utils.py", 
line 140, in execute
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent     raise 
RuntimeError(msg)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent RuntimeError: 
Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-c4080b5c-c9"

Has anyone come across this before?

We don't seem to have an entry for them anywhere from what we can see.

Regards,


-- 
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io <http://www.absolutedevops.io/> 
grant at absolutedevops.io <mailto:grant at absolutedevops.i> 0845 874 0580
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170608/f3edaaa4/attachment.html>