[openstack-dev] Fwd: Simultaneous launch of multiple instances from horizon leads to multiple LANs assigned to new tenant

Nikolay Starodubtsev starodubcevna at gmail.com
Mon Apr 8 06:25:35 UTC 2013


Hi, all!
I started work on this bug and post a comment on launchpad, but no one
anwer to me, so here it is:
I've done a little search. Is this bug still actual? When I try to
reproduce it I have another bug:
https://bugs.launchpad.net/nova/+bug/1161812
 I have a look at nova/compute/

manager.py and see that when we try to create instance we called:
network_info = self._allocate_network(context, instance, requested_networks)

Here is _allocate_network code:

def _allocate_network(self, context, instance, requested_networks):
         """Allocate networks for an instance and return the network info"""
         self._instance_update(context, instance['uuid'],
                               vm_state=vm_states.BUILDING,
                               task_state=task_states.NETWORKING,
                               expected_task_state=None)
        is_vpn = instance['image_ref'] == str(FLAGS.vpn_image_id)
         try:
             # allocate and get network info
             network_info = self.network_api.allocate_for_instance(
                                 context, instance, vpn=is_vpn,
                                 requested_networks=requested_networks)
         except Exception:
             LOG.exception(_('Instance failed network setup'),
                           instance=instance)
             raise

         LOG.debug(_('Instance network_info: |%s|'), network_info,
                   instance=instance)

         return network_info

So it try to give network to instance, not to project/tenant. Is this bug
still actual?

From: Devananda van der Veen <devananda.vdv at gmail.com>
Date: Fri, Feb 8, 2013 at 10:12 PM
Subject: Re: [openstack-dev] Simultaneous launch of multiple instances from
horizon leads to multiple LANs assigned to new tenant
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>


This sounds like the "select; if result: update; else: insert" race
condition and subsequent deadlock issue. We talked at length about it in
the db meeting yesterday. I've tagged this bug as "db" related.


On Thu, Feb 7, 2013 at 7:43 AM, rezroo <reza at dslextreme.com> wrote:

>  Ok - thanks.  I reported a bug:
>
> https://bugs.launchpad.net/nova/+bug/1118457
>
> Rez
>
>
> On 02/06/2013 10:58 PM, Vishvananda Ishaya wrote:
>
> Going from memory of the code I believe this is possible. Iirc the lock is
> a select for update so assuming innodb it will prevent two workers from
> assigning the same network to multiple tenants but won't prevent two
> networks being assigned to the same tenant. This is probably an untested
> consequence of clan plus multi host. A decent workaround would be to assign
> a network to a tenant immediately after creation instead of lazily.
> On Feb 5, 2013 12:56 PM, "rezroo" <reza at dslextreme.com> wrote:
>
>>  I've run into this issue and I'd like to get some opinions before
>> logging a defect. I'm using folsom, deployed using VlanManager.  It seems
>> to me that the assignment of a network to a tenant is lazy evaluated until
>> the first instance for the project is launched.  This can be confirmed by
>> creating a new tenant and then doing 'nova-manage network list'. If one now
>> launches multiple tenants by setting the number of instances to 4 or 5 then
>> there is a race condition and it is possible that two or more vlan/networks
>> are assigned to the tenant. I'm running into this issue.
>>
>> I posted to Openstack-operators already and I got one reply back claiming
>> that this is a configuration issue, without any insight as to what is
>> misconfigured. Do you know what kind of misconfiguration leads to this? Or
>> what concurrency/locking mechanism prevents this from happening?
>> Presumably a select is done in the networks table to see if a vlan is
>> already assigned to a tenant.  If you have sufficiently fast environment it
>> is possible that two simultaneous queries are done, both with no results,
>> and two networks are assigned and committed to the database.  This is what
>> happened for me.  What misconfiguration should I look for, or what
>> concurrency/locking mechanism is there to prevent this?
>>
>> Or I can log a defect.
>>
>> Thanks,
>> Rez
>>
>> -------- Original Message --------  Subject: Re: [Openstack-operators]
>> Simultaneous launch of multiple instances from horizon leads to multiple
>> LANs assigned to new tenant  Date: Tue, 05 Feb 2013 05:41:37 -0800  From:
>> rezroo <reza at dslextreme.com> <reza at dslextreme.com>  To: Vivek Singh
>> Raghuwanshi <vivekraghuwanshi at gmail.com> <vivekraghuwanshi at gmail.com>
>>
>> Still using nova-network due to its better integration with Horizon.
>> Do you know what kind of misconfiguration leads to this? Or what
>> concurrency/locking mechanism prevents this from happening?  So presumably
>> a select is done in the networks table to see if a vlan is already assigned
>> to a tenant.  If you have sufficiently fast environment it is possible that
>> two simultaneous queries are done, both with no results, and two networks
>> are assigned and committed to the database.  This is what happened for me.
>> What misconfiguration should I look for, or what concurrency/locking
>> mechanism is there to prevent this?
>> Thanks,
>> Rez
>>
>> On 02/04/2013 11:03 PM, Vivek Singh Raghuwanshi wrote:
>>
>> This only happens due to misconfiguration, but are you still with
>> nova-network.
>> its advisable to switch your  cloud with Quantum
>>
>> On Tue, Feb 5, 2013 at 5:05 AM, rezroo <reza at dslextreme.com> wrote:
>>
>>> Hi - I'm wondering if others have seen this defect and if it has already
>>> been reported or not.
>>> I'm using folsom, deployed using VlanManager.
>>> When a new tenant is created no vlan/network is assigned to it until the
>>> first instance for the project is launched.  This can be confirmed by
>>> creating a new tenant and then doing 'nova-manage network list'.
>>> If now one launches multiple tenants by setting the number of instances
>>> to 4 or 5 then there is a race condition and it is possible that two or
>>> more vlan/networks are assigned to the tenant.
>>> Should I log this as a defect?
>>> Thanks,
>>> Rez
>>>
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>>
>>
>> --
>> ViVek Raghuwanshi
>> Mobile -+91-09595950504
>> Skype - vivek_raghuwanshi
>> IRC vivekraghuwanshi
>> http://vivekraghuwanshi.wordpress.com/
>> http://in.linkedin.com/in/vivekraghuwanshi
>>
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing listOpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 

Kind regards, Yuriy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130408/b25026e1/attachment.html>


More information about the OpenStack-dev mailing list