Open Stack

Hi guys,

New to the discussion, so please correct if I'm off base.

if I understand correctly, the controllers handle the state/race-condition in a controller specify way.  So what we are talking about is a new state-flow for inside of the openstack db to describe the state of the network as known to openstack.   Is this correct?
Thanks,
john
-----Original Message-----
From: Isaku Yamahata [mailto:isaku.yamahata at gmail.com] 
Sent: Wednesday, November 20, 2013 5:46 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: Robert Kukura; Isaku Yamahata
Subject: Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

On Tue, Nov 19, 2013 at 08:59:38AM -0800, Edgar Magana <emagana at plumgrid.com> wrote:

> Do you have in mind any implementation, any BP?
> We could actually work on this together, all plugins will get the 
> benefits of a better implementation.

Yes, let's work together. Here is my blueprint (it's somewhat old.
So needs to be updated.)
https://blueprints.launchpad.net/neutron/+spec/fix-races-of-db-based-plugin
https://docs.google.com/file/d/0B4LNMvjOzyDuU2xNd0piS3JBMHM/edit

Although I've thought of status change(adding more status) and locking protocol so far, TaskFlow seems something to look at before starting and another possible approach is decoupling backend process from api call as Salvatore suggested like NVP plugin.
Even with taskflow or decoupling approach, some kind of enhancing status change/locking protocol will be necessary for performance of creating many ports at once.

thanks,

> 
> Thanks,
> 
> Edgar
> 
> On 11/19/13 3:57 AM, "Isaku Yamahata" <isaku.yamahata at gmail.com> wrote:
> 
> >On Mon, Nov 18, 2013 at 03:55:49PM -0500, Robert Kukura 
> ><rkukura at redhat.com> wrote:
> >
> >> On 11/18/2013 03:25 PM, Edgar Magana wrote:
> >> > Developers,
> >> > 
> >> > This topic has been discussed before but I do not remember if we 
> >> > have
> >>a
> >> > good solution or not.
> >> 
> >> The ML2 plugin addresses this by calling each MechanismDriver 
> >> twice. The
> >> create_network_precommit() method is called as part of the DB 
> >> transaction, and the create_network_postcommit() method is called 
> >> after the transaction has been committed. Interactions with devices 
> >> or controllers are done in the postcommit methods. If the 
> >> postcommit method raises an exception, the plugin deletes that 
> >> partially-created resource and returns the exception to the client. 
> >> You might consider a similar approach in your plugin.
> >
> >Splitting works into two phase, pre/post, is good approach.
> >But there still remains race window.
> >Once the transaction is committed, the result is visible to outside.
> >So the concurrent request to same resource will be racy.
> >There is a window after pre_xxx_yyy before post_xxx_yyy() where other 
> >requests can be handled.
> >
> >The state machine needs to be enhanced, I think. (plugins need
> >modification)
> >For example, adding more states like pending_{create, delete, update}.
> >Also we would like to consider serializing between operation of ports 
> >and subnets. or between operation of subnets and network depending on 
> >performance requirement.
> >(Or carefully audit complex status change. i.e.
> >changing port during subnet/network update/deletion.)
> >
> >I think it would be useful to establish reference locking policy for 
> >ML2 plugin for SDN controllers.
> >Thoughts or comments? If this is considered useful and acceptable, 
> >I'm willing to help.
> >
> >thanks,
> >Isaku Yamahata
> >
> >> -Bob
> >> 
> >> > Basically, if concurrent API calls are sent to Neutron, all of 
> >> > them
> >>are
> >> > sent to the plug-in level where two actions have to be made:
> >> > 
> >> > 1. DB transaction ? No just for data persistence but also to 
> >> > collect
> >>the
> >> > information needed for the next action 2. Plug-in back-end 
> >> > implementation ? In our case is a call to the
> >>python
> >> > library than consequentially calls PLUMgrid REST GW (soon SAL)
> >> > 
> >> > For instance:
> >> > 
> >> > def create_port(self, context, port):
> >> >         with context.session.begin(subtransactions=True):
> >> >             # Plugin DB - Port Create and Return port
> >> >             port_db = super(NeutronPluginPLUMgridV2, 
> >> > self).create_port(context,
> >> >               
> >> port)
> >> >             device_id = port_db["device_id"]
> >> >             if port_db["device_owner"] == "network:router_gateway":
> >> >                 router_db = self._get_router(context, device_id)
> >> >             else:
> >> >                 router_db = None
> >> >             try:
> >> >                 LOG.debug(_("PLUMgrid Library: create_port() 
> >> > called")) # Back-end implementation
> >> >                 self._plumlib.create_port(port_db, router_db)
> >> >             except Exception:
> >> >             Š
> >> > 
> >> > The way we have implemented at the plugin-level in Havana (even 
> >> > in
> >> > Grizzly) is that both action are wrapped in the same "transaction"
> >>which
> >> > automatically rolls back any operation done to its original state 
> >> > protecting mostly the DB of having any inconsistency state or 
> >> > left
> >>over
> >> > data if the back-end part fails.=.
> >> > The problem that we are experiencing is when concurrent calls to 
> >> > the same API are sent, the number of operation at the plug-in 
> >> > back-end are long enough to make the next concurrent API call to 
> >> > get stuck at the
> >>DB
> >> > transaction level, which creates a hung state for the Neutron 
> >> > Server
> >>to
> >> > the point that all concurrent API calls will fail.
> >> > 
> >> > This can be fixed if we include some "locking" system such as calling:
> >> > 
> >> > from neutron.common import utile
> >> > Š
> >> > 
> >> > @utils.synchronized('any-name', external=True) def 
> >> > create_port(self, context, port):
> >> > Š
> >> > 
> >> > Obviously, this will create a serialization of all concurrent 
> >> > calls which will ends up in having a really bad performance. Does 
> >> > anyone
> >>has a
> >> > better solution?
> >> > 
> >> > Thanks,
> >> > 
> >> > Edgar
> >> > 
> >> > 
> >> > _______________________________________________
> >> > OpenStack-dev mailing list
> >> > OpenStack-dev at lists.openstack.org 
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > 
> >> 
> >> 
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >--
> >Isaku Yamahata <isaku.yamahata at gmail.com>
> 
> 

--
Isaku Yamahata <isaku.yamahata at gmail.com>

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev