<div dir="ltr">I think Kevin is right - the actual root cause lies in plugins for which disassociate_floating_ips returns None rather than an (empty) iterable.<div><br></div><div>A quick check revealed that at least:</div><div>neutron.plugins.embrane.base_plugin.EmbranePlugin</div><div>neutron.plugions.cisco.service_plugins.cisco_router_plugin.CiscoRouterPlugin</div><div><br></div><div>can return None in some circumstances.</div><div>I reckon the plugins should be fixed.</div><div>While Sunil's patch fixes the issue it does not make a lot of sense since it call notify_routers_updated with None for routers_id.</div><div>I agree with his intention of making the ml2 plugin more robust, but in my opinion when router_ids is None or empty the notification method should probably not be called at all. </div><div><br></div><div>Salvatore</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 11 January 2015 at 12:02, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That only seems to bury the root cause because router_ids shouldn't be none in this case with the reference L3 plugin.<div><br></div><div>The ML2 plugin calls disassociate_floatingips with do_notify set to False so it should always be at least an empty set coming back.</div><div><br></div><div>Which L3 plugin were you using? Perhaps the implementation of disassociate_floatingips was incorrect. </div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar <span dir="ltr"><<a href="mailto:sunil@embrane.com" target="_blank">sunil@embrane.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">This trivial patch fixes the tracebacks:<br>
<br>
$ cat disassociate_floating_ips.patch <br>
--- neutron/db/l3_db.py.orig 2015-01-10 22:20:30.101506298 -0800<br>
+++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800<br>
@@ -1257,4 +1257,4 @@<br>
<br>
def notify_routers_updated(self, context, router_ids):<br>
super(L3_NAT_db_mixin, self).notify_routers_updated(<span><br>
- context, list(router_ids), 'disassociate_floatingips', {})<br></span>
+ context, list(router_ids) if router_ids else None, 'disassociate_floatingips', {})<br>
<div><br>
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px"><br>
-Sunil<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma"><b>From:</b> Sunil Kumar [<a href="mailto:sunil@embrane.com" target="_blank">sunil@embrane.com</a>]<br>
<b>Sent:</b> Saturday, January 10, 2015 7:07 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?<br>
</font><br>
</div><div><div>
<div></div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Not sure if its something seen by others. I hit this when I run tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master:<br>
<div><br>
2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995<br>
2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore "db-access" internal_lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206<br>
2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229<br>
2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043<br>
2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238<br>
2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback (most recent call last):<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource result = method(request=request, **args)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 198, in delete_floatingip<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource self).delete_floatingip(context, id)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1237, in delete_floatingip<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource router_id = self._delete_floatingip(context, id)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 902, in _delete_floatingip<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource l3_port_check=False)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1050, in delete_port<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource l3plugin.notify_routers_updated(context, router_ids)<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1260, in notify_routers_updated<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource context, list(router_ids), 'disassociate_floatingips', {})<br>
2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource TypeError: 'NoneType' object is not iterable<br>
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px"><br>
Looks like the code is assuming that router_ids can never be None, which clearly is the case here. Is that a bug?<br>
<br>
Looking elsewhere in the l3_db.py, L3RpcNotifierMixin.notify_routers_updated() does make a check for router_ids (which means that that function does expect it to be empty some times), but the list() is killing it before it reaches that.<br>
<br>
This backtrace repeats itself many many times in the neutron logs.<br>
<br>
Thanks for your help.<br>
-Sunil<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br></div></div>__________________________________________________________________________<span class=""><br>
OpenStack Development Mailing List (not for usage questions)<br></span>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div><div>Kevin Benton</div></div>
</font></span></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>