<div dir="ltr">I patched this back in liberty.<div>What version of Openstack are you using ?</div><div><br></div><div>Saverio</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-06-08 19:36 GMT+02:00 Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Ignore that now all,</p>
<p>Managed to fix it by restarting the l3-agent. Looks like it must
have been cached in memory.</p>
<p>Thanks,<br>
</p><div><div class="h5">
<br>
<div class="m_-7157055308795890042moz-cite-prefix">On 08/06/17 18:07, Grant Morley wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<p>Hi All,</p>
<p>We have noticed in our neutron-l3-agent logs that there are a
number of routers that neutron seems to think exist ( but they
no longer exist in the data base ) and it is constantly trying
to delete them. However because they don't appear to exist, so
neutron seems to get stuck in a loop of trying to delete
routers that are no longer there.</p>
<p>We originally noticed that neutron was trying to delete the
"qrouter-xxxx-xxx-xxxx" and that didn't exist. So we added the
router manually with "ip netns add qrouter-xxx-xxx-xxx-xxx"
which then deletes the router however we then get the following
error messages:</p>
<p>2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent [-]
Error while deleting router da7b633a-233b-46d1-ba3d-<wbr>315b3bee6a61<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
Traceback (most recent call last):<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>agent.py",
line 359, in _safe_router_removed<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self._router_removed(router_<wbr>id)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>agent.py",
line 377, in _router_removed<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
ri.delete(self)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/ha_<wbr>router.py",
line 380, in delete<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
super(HaRouter, self).delete(agent)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py",
line 361, in delete<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self.process_delete(agent)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/common/utils.<wbr>py",
line 385, in call<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self.logger(e)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/oslo_utils/excutils.<wbr>py",
line 220, in __exit__<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self.force_reraise()<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/oslo_utils/excutils.<wbr>py",
line 196, in force_reraise<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
six.reraise(self.type_, self.value, self.tb)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/common/utils.<wbr>py",
line 382, in call<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
return func(*args, **kwargs)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py",
line 972, in process_delete<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self._process_external_on_<wbr>delete(agent)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py",
line 794, in _process_external_on_delete<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self._process_external_<wbr>gateway(ex_gw_port, agent.pd)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py",
line 693, in _process_external_gateway<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self.external_gateway_removed(<wbr>self.ex_gw_port, interface_name)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/ha_<wbr>router.py",
line 371, in external_gateway_removed<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
interface_name)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py",
line 668, in external_gateway_removed<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
ip_addr['prefixlen']))<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py",
line 286, in remove_external_gateway_ip<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
device.delete_addr_and_<wbr>conntrack_state(ip_cidr)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>ip_lib.py",
line 290, in delete_addr_and_conntrack_<wbr>state<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
self.addr.delete(cidr)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>ip_lib.py",
line 580, in delete<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
'dev', <a href="http://self.name" target="_blank">self.name</a>))<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>ip_lib.py",
line 361, in _as_root<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
use_root_namespace=use_root_<wbr>namespace)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>ip_lib.py",
line 94, in _as_root<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
log_fail_as_error=self.log_<wbr>fail_as_error)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>ip_lib.py",
line 103, in _execute<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
log_fail_as_error=log_fail_as_<wbr>error)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent File
"/openstack/venvs/neutron-13.<wbr>3.8/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py",
line 140, in execute<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
raise RuntimeError(msg)<br>
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot
find device "qg-c4080b5c-c9"</p>
<p>Has anyone come across this before?</p>
<p>We don't seem to have an entry for them anywhere from what we
can see.</p>
<p>Regards,<br>
</p>
<p><br>
</p>
<div class="m_-7157055308795890042moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font color="#A8D8FF" face="Lucida Grande, Verdana,
Arial, sans-serif"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img style="color:#a8d8ff;font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px"> </span> </font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span>
<font face="Hind, Lucida Grande, Verdana, Arial, sans-serif">
<span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
<div style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP </div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">www.absolutedevops.io</a>
<a href="mailto:grant@absolutedevops.i" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">grant@absolutedevops.io</a>
<a>0845 874 0580</a>
</div>
</div>
</div>
<br>
<fieldset class="m_-7157055308795890042mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
OpenStack-operators mailing list
<a class="m_-7157055308795890042moz-txt-link-abbreviated" href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.<wbr>openstack.org</a>
<a class="m_-7157055308795890042moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a>
</pre>
</blockquote><span class="">
<br>
<div class="m_-7157055308795890042moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font color="#A8D8FF" face="Lucida Grande, Verdana, Arial,
sans-serif"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img style="color:#a8d8ff;font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px"> </span>
</font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
<div style="font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:#616161">Absolute</span> <span style="color:#98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">www.absolutedevops.io</a> <a href="mailto:grant@absolutedevops.i" style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">grant@absolutedevops.io</a> <a style="color:#239dff;font-family:Hind,'Lucida Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px">0845 874 0580</a> </div>
</div>
</div>
</span></div>
<br>______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br></blockquote></div><br></div>