Open Stack

I think Kevin is right - the actual root cause lies in plugins for which
disassociate_floating_ips returns None rather than an (empty) iterable.

A quick check revealed that at least:
neutron.plugins.embrane.base_plugin.EmbranePlugin
neutron.plugions.cisco.service_plugins.cisco_router_plugin.CiscoRouterPlugin

can return None in some circumstances.
I reckon the plugins should be fixed.
While Sunil's patch fixes the issue it does not make a lot of sense since
it call notify_routers_updated with None for routers_id.
I agree with his intention of making the ml2 plugin more robust, but in my
opinion when router_ids is None or empty the notification method should
probably not be called at all.

Salvatore

On 11 January 2015 at 12:02, Kevin Benton <blak111 at gmail.com> wrote:

> That only seems to bury the root cause because router_ids shouldn't be
> none in this case with the reference L3 plugin.
>
> The ML2 plugin calls disassociate_floatingips with do_notify set to False
> so it should always be at least an empty set coming back.
>
> Which L3 plugin were you using? Perhaps the implementation of
> disassociate_floatingips was incorrect.
>
> On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar <sunil at embrane.com> wrote:
>
>>  This trivial patch fixes the tracebacks:
>>
>> $ cat disassociate_floating_ips.patch
>> --- neutron/db/l3_db.py.orig    2015-01-10 22:20:30.101506298 -0800
>> +++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800
>> @@ -1257,4 +1257,4 @@
>>
>>      def notify_routers_updated(self, context, router_ids):
>>          super(L3_NAT_db_mixin, self).notify_routers_updated(
>> -            context, list(router_ids), 'disassociate_floatingips', {})
>> +            context, list(router_ids) if router_ids else None,
>> 'disassociate_floatingips', {})
>>
>>
>> -Sunil
>>     ------------------------------
>> *From:* Sunil Kumar [sunil at embrane.com]
>> *Sent:* Saturday, January 10, 2015 7:07 PM
>> *To:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* [openstack-dev] [Neutron][L3][Devstack] Bug during delete
>> floating IPs?
>>
>>   Not sure if its something seen by others. I hit this when I run
>> tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master:
>>
>> 2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin
>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port
>> e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port
>> /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995
>> 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils
>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore
>> "db-access" internal_lock
>> /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206
>> 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils
>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access"
>> lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229
>> 2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin
>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for
>> e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip
>> delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043
>> 2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils
>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access"
>> lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238
>> 2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource
>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback
>> (most recent call last):
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource     result =
>> method(request=request, **args)
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource
>> obj_deleter(request.context, id, **kwargs)
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 198, in
>> delete_floatingip
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource
>> self).delete_floatingip(context, id)
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1237, in
>> delete_floatingip
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource     router_id
>> = self._delete_floatingip(context, id)
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/db/l3_db.py", line 902, in
>> _delete_floatingip
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource
>> l3_port_check=False)
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1050, in
>> delete_port
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource
>> l3plugin.notify_routers_updated(context, router_ids)
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource   File
>> "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1260, in
>> notify_routers_updated
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource     context,
>> list(router_ids), 'disassociate_floatingips', {})
>> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource TypeError:
>> 'NoneType' object is not iterable
>>
>> Looks like the code is assuming that router_ids can never be None, which
>> clearly is the case here. Is that a bug?
>>
>> Looking elsewhere in the l3_db.py,
>> L3RpcNotifierMixin.notify_routers_updated() does make a check for
>> router_ids (which means that that function does expect it to be empty some
>> times), but the list() is killing it before it reaches that.
>>
>> This backtrace repeats itself many many times in the neutron logs.
>>
>> Thanks for your help.
>> -Sunil
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150111/65de34a8/attachment.html>