<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Actually, I just noticed that a patch which has been pending review since Sep is not merged, and that actually fixes the root cause.<br>
<div><br>
https://review.openstack.org/#/c/119269/<br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px"><br>
Having said that, I think the code can definitely use some touching. If router_ids is not supposed to be None, it should be asserted (enforcing a contract with the plugin APIs!) instead of letting python crib about it. Also, the code in plugins/ml2/plugin.py
should not even be call l3plugin.notify_routers_updated if the router_ids set is empty. I think assert also belongs there.<br>
<br>
-Sunil<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF992712"><font color="#000000" face="Tahoma" size="2"><b>From:</b> Salvatore Orlando [sorlando@nicira.com]<br>
<b>Sent:</b> Sunday, January 11, 2015 7:25 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">I think Kevin is right - the actual root cause lies in plugins for which disassociate_floating_ips returns None rather than an (empty) iterable.
<div><br>
</div>
<div>A quick check revealed that at least:</div>
<div>neutron.plugins.embrane.base_plugin.EmbranePlugin</div>
<div>neutron.plugions.cisco.service_plugins.cisco_router_plugin.CiscoRouterPlugin</div>
<div><br>
</div>
<div>can return None in some circumstances.</div>
<div>I reckon the plugins should be fixed.</div>
<div>While Sunil's patch fixes the issue it does not make a lot of sense since it call notify_routers_updated with None for routers_id.</div>
<div>I agree with his intention of making the ml2 plugin more robust, but in my opinion when router_ids is None or empty the notification method should probably not be called at all. </div>
<div><br>
</div>
<div>Salvatore</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 11 January 2015 at 12:02, Kevin Benton <span dir="ltr">
<<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">That only seems to bury the root cause because router_ids shouldn't be none in this case with the reference L3 plugin.
<div><br>
</div>
<div>The ML2 plugin calls disassociate_floatingips with do_notify set to False so it should always be at least an empty set coming back.</div>
<div><br>
</div>
<div>Which L3 plugin were you using? Perhaps the implementation of disassociate_floatingips was incorrect. </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="h5">On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar <span dir="ltr"><<a href="mailto:sunil@embrane.com" target="_blank">sunil@embrane.com</a>></span> wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div>
<div class="h5">
<div>
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">This trivial patch fixes the tracebacks:<br>
<br>
$ cat disassociate_floating_ips.patch <br>
--- neutron/db/l3_db.py.orig 2015-01-10 22:20:30.101506298 -0800<br>
+++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800<br>
@@ -1257,4 +1257,4 @@<br>
<br>
def notify_routers_updated(self, context, router_ids):<br>
super(L3_NAT_db_mixin, self).notify_routers_updated(<span><br>
- context, list(router_ids), 'disassociate_floatingips', {})<br>
</span>+ context, list(router_ids) if router_ids else None, 'disassociate_floatingips', {})<br>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px"><br>
-Sunil<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="font-family:Times New Roman; color:#000000; font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma"><b>From:</b> Sunil Kumar [<a href="mailto:sunil@embrane.com" target="_blank">sunil@embrane.com</a>]<br>
<b>Sent:</b> Saturday, January 10, 2015 7:07 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?<br>
</font><br>
</div>
<div>
<div>
<div></div>
<div>
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">Not sure if its something seen by others. I hit this when I run tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master:<br>
<div><br>
2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995<br>
2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore "db-access" internal_lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206<br>
2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229<br>
2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043<br>
2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238<br>
2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback (most recent call last):<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource result = method(request=request, **args)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 198, in delete_floatingip<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource self).delete_floatingip(context, id)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1237, in delete_floatingip<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource router_id = self._delete_floatingip(context, id)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 902, in _delete_floatingip<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource l3_port_check=False)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1050, in delete_port<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource l3plugin.notify_routers_updated(context, router_ids)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1260, in notify_routers_updated<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource context, list(router_ids), 'disassociate_floatingips', {})<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource TypeError: 'NoneType' object is not iterable<br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px"><br>
Looks like the code is assuming that router_ids can never be None, which clearly is the case here. Is that a bug?<br>
<br>
Looking elsewhere in the l3_db.py, L3RpcNotifierMixin.notify_routers_updated() does make a check for router_ids (which means that that function does expect it to be empty some times), but the list() is killing it before it reaches that.<br>
<br>
This backtrace repeats itself many many times in the neutron logs.<br>
<br>
Thanks for your help.<br>
-Sunil<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
__________________________________________________________________________<span class=""><br>
OpenStack Development Mailing List (not for usage questions)<br>
</span>Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<span class="HOEnZb"><font color="#888888"><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>
<div>Kevin Benton</div>
</div>
</font></span></div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>