Open Stack

Ignore that now all,

Managed to fix it by restarting the l3-agent. Looks like it must have 
been cached in memory.

Thanks,


On 08/06/17 18:07, Grant Morley wrote:
>
> Hi All,
>
> We have  noticed in our neutron-l3-agent logs that there are a number 
> of routers that neutron seems to think exist ( but they no longer 
> exist in the data base )  and it is constantly trying to delete them. 
> However because they don't appear to exist,  so neutron seems to get 
> stuck in a loop of trying to delete routers that are no longer there.
>
> We originally noticed that neutron was trying to delete the 
> "qrouter-xxxx-xxx-xxxx" and that didn't exist. So we added the router 
> manually with "ip netns add qrouter-xxx-xxx-xxx-xxx" which then 
> deletes the router however we then get the following error messages:
>
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent [-] Error 
> while deleting router da7b633a-233b-46d1-ba3d-315b3bee6a61
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent Traceback 
> (most recent call last):
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/agent.py", 
> line 359, in _safe_router_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self._router_removed(router_id)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/agent.py", 
> line 377, in _router_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent ri.delete(self)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", 
> line 380, in delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> super(HaRouter, self).delete(agent)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
> line 361, in delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self.process_delete(agent)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/common/utils.py", 
> line 385, in call
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent self.logger(e)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/oslo_utils/excutils.py", 
> line 220, in __exit__
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self.force_reraise()
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/oslo_utils/excutils.py", 
> line 196, in force_reraise
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> six.reraise(self.type_, self.value, self.tb)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/common/utils.py", 
> line 382, in call
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent return 
> func(*args, **kwargs)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
> line 972, in process_delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self._process_external_on_delete(agent)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
> line 794, in _process_external_on_delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self._process_external_gateway(ex_gw_port, agent.pd)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
> line 693, in _process_external_gateway
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self.external_gateway_removed(self.ex_gw_port, interface_name)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", 
> line 371, in external_gateway_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent interface_name)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
> line 668, in external_gateway_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> ip_addr['prefixlen']))
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
> line 286, in remove_external_gateway_ip
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> device.delete_addr_and_conntrack_state(ip_cidr)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
> line 290, in delete_addr_and_conntrack_state
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> self.addr.delete(cidr)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
> line 580, in delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 'dev', 
> self.name))
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
> line 361, in _as_root
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> use_root_namespace=use_root_namespace)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
> line 94, in _as_root
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> log_fail_as_error=self.log_fail_as_error)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
> line 103, in _execute
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
> log_fail_as_error=log_fail_as_error)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/utils.py", 
> line 140, in execute
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent raise 
> RuntimeError(msg)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent RuntimeError: 
> Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device 
> "qg-c4080b5c-c9"
>
> Has anyone come across this before?
>
> We don't seem to have an entry for them anywhere from what we can see.
>
> Regards,
>
>
> -- 
> Grant Morley
> Cloud Lead
> Absolute DevOps Ltd
> Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
> www.absolutedevops.io <http://www.absolutedevops.io/> 
> grant at absolutedevops.io <mailto:grant at absolutedevops.i> 0845 874 0580
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io <http://www.absolutedevops.io/> 
grant at absolutedevops.io <mailto:grant at absolutedevops.i> 0845 874 0580
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170608/a4c1aced/attachment.html>