Open Stack

Let me take a look and circle back to you in a bit. This is a very
sensitive part of the code, so we need to
Handle properly any change.

Thanks,

Edgar

On 11/20/13 5:46 AM, "Isaku Yamahata" <isaku.yamahata at gmail.com> wrote:

>On Tue, Nov 19, 2013 at 08:59:38AM -0800,
>Edgar Magana <emagana at plumgrid.com> wrote:
>
>> Do you have in mind any implementation, any BP?
>> We could actually work on this together, all plugins will get the
>>benefits
>> of a better implementation.
>
>Yes, let's work together. Here is my blueprint (it's somewhat old.
>So needs to be updated.)
>https://blueprints.launchpad.net/neutron/+spec/fix-races-of-db-based-plugi
>n
>https://docs.google.com/file/d/0B4LNMvjOzyDuU2xNd0piS3JBMHM/edit
>
>Although I've thought of status change(adding more status) and locking
>protocol so far, TaskFlow seems something to look at before starting and
>another possible approach is decoupling backend process from api call
>as Salvatore suggested like NVP plugin.
>Even with taskflow or decoupling approach, some kind of enhancing status
>change/locking protocol will be necessary for performance of creating
>many ports at once.
>
>thanks,
>
>> 
>> Thanks,
>> 
>> Edgar
>> 
>> On 11/19/13 3:57 AM, "Isaku Yamahata" <isaku.yamahata at gmail.com> wrote:
>> 
>> >On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>> >Robert Kukura <rkukura at redhat.com> wrote:
>> >
>> >> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>> >> > Developers,
>> >> > 
>> >> > This topic has been discussed before but I do not remember if we
>>have
>> >>a
>> >> > good solution or not.
>> >> 
>> >> The ML2 plugin addresses this by calling each MechanismDriver twice.
>>The
>> >> create_network_precommit() method is called as part of the DB
>> >> transaction, and the create_network_postcommit() method is called
>>after
>> >> the transaction has been committed. Interactions with devices or
>> >> controllers are done in the postcommit methods. If the postcommit
>>method
>> >> raises an exception, the plugin deletes that partially-created
>>resource
>> >> and returns the exception to the client. You might consider a similar
>> >> approach in your plugin.
>> >
>> >Splitting works into two phase, pre/post, is good approach.
>> >But there still remains race window.
>> >Once the transaction is committed, the result is visible to outside.
>> >So the concurrent request to same resource will be racy.
>> >There is a window after pre_xxx_yyy before post_xxx_yyy() where
>> >other requests can be handled.
>> >
>> >The state machine needs to be enhanced, I think. (plugins need
>> >modification)
>> >For example, adding more states like pending_{create, delete, update}.
>> >Also we would like to consider serializing between operation of ports
>> >and subnets. or between operation of subnets and network depending on
>> >performance requirement.
>> >(Or carefully audit complex status change. i.e.
>> >changing port during subnet/network update/deletion.)
>> >
>> >I think it would be useful to establish reference locking policy
>> >for ML2 plugin for SDN controllers.
>> >Thoughts or comments? If this is considered useful and acceptable,
>> >I'm willing to help.
>> >
>> >thanks,
>> >Isaku Yamahata
>> >
>> >> -Bob
>> >> 
>> >> > Basically, if concurrent API calls are sent to Neutron, all of them
>> >>are
>> >> > sent to the plug-in level where two actions have to be made:
>> >> > 
>> >> > 1. DB transaction ? No just for data persistence but also to
>>collect
>> >>the
>> >> > information needed for the next action
>> >> > 2. Plug-in back-end implementation ? In our case is a call to the
>> >>python
>> >> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>> >> > 
>> >> > For instance:
>> >> > 
>> >> > def create_port(self, context, port):
>> >> >         with context.session.begin(subtransactions=True):
>> >> >             # Plugin DB - Port Create and Return port
>> >> >             port_db = super(NeutronPluginPLUMgridV2,
>> >> > self).create_port(context,
>> >> >            
>> >> port)
>> >> >             device_id = port_db["device_id"]
>> >> >             if port_db["device_owner"] == "network:router_gateway":
>> >> >                 router_db = self._get_router(context, device_id)
>> >> >             else:
>> >> >                 router_db = None
>> >> >             try:
>> >> >                 LOG.debug(_("PLUMgrid Library: create_port()
>>called"))
>> >> > # Back-end implementation
>> >> >                 self._plumlib.create_port(port_db, router_db)
>> >> >             except Exception:
>> >> >             ?
>> >> > 
>> >> > The way we have implemented at the plugin-level in Havana (even in
>> >> > Grizzly) is that both action are wrapped in the same "transaction"
>> >>which
>> >> > automatically rolls back any operation done to its original state
>> >> > protecting mostly the DB of having any inconsistency state or left
>> >>over
>> >> > data if the back-end part fails.=.
>> >> > The problem that we are experiencing is when concurrent calls to
>>the
>> >> > same API are sent, the number of operation at the plug-in back-end
>>are
>> >> > long enough to make the next concurrent API call to get stuck at
>>the
>> >>DB
>> >> > transaction level, which creates a hung state for the Neutron
>>Server
>> >>to
>> >> > the point that all concurrent API calls will fail.
>> >> > 
>> >> > This can be fixed if we include some "locking" system such as
>>calling:
>> >> > 
>> >> > from neutron.common import utile
>> >> > ?
>> >> > 
>> >> > @utils.synchronized('any-name', external=True)
>> >> > def create_port(self, context, port):
>> >> > ?
>> >> > 
>> >> > Obviously, this will create a serialization of all concurrent calls
>> >> > which will ends up in having a really bad performance. Does anyone
>> >>has a
>> >> > better solution?
>> >> > 
>> >> > Thanks,
>> >> > 
>> >> > Edgar
>> >> > 
>> >> > 
>> >> > _______________________________________________
>> >> > OpenStack-dev mailing list
>> >> > OpenStack-dev at lists.openstack.org
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > 
>> >> 
>> >> 
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >-- 
>> >Isaku Yamahata <isaku.yamahata at gmail.com>
>> 
>> 
>
>-- 
>Isaku Yamahata <isaku.yamahata at gmail.com>