<div dir="ltr"><div>Hi, Isaku and Edgar,</div><div><br></div>As part of the effort to implement L3 router service type framework, I have reworked L3 plugin to introduce a 2-step process, precommit and postcommit, similar to ML2. If you plan to work on L3 code, we can collaborate.<div>
<br></div><div><a href="https://blueprints.launchpad.net/neutron/+spec/l3-router-service-type-framework">https://blueprints.launchpad.net/neutron/+spec/l3-router-service-type-framework</a></div><div><br></div><div>Also, for advanced services such as FW and LBaas, there already is a state transition logic in the plugin. For example, a firewall instance can have CREATE, UPDATE and DELETE_PENDING states. </div>
<div><br></div><div>Thanks,</div><div>Gary</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 20, 2013 at 8:55 AM, Edgar Magana <span dir="ltr"><<a href="mailto:emagana@plumgrid.com" target="_blank">emagana@plumgrid.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Let me take a look and circle back to you in a bit. This is a very<br>
sensitive part of the code, so we need to<br>
Handle properly any change.<br>
<br>
Thanks,<br>
<br>
Edgar<br>
<div class="HOEnZb"><div class="h5"><br>
On 11/20/13 5:46 AM, "Isaku Yamahata" <<a href="mailto:isaku.yamahata@gmail.com">isaku.yamahata@gmail.com</a>> wrote:<br>
<br>
>On Tue, Nov 19, 2013 at 08:59:38AM -0800,<br>
>Edgar Magana <<a href="mailto:emagana@plumgrid.com">emagana@plumgrid.com</a>> wrote:<br>
><br>
>> Do you have in mind any implementation, any BP?<br>
>> We could actually work on this together, all plugins will get the<br>
>>benefits<br>
>> of a better implementation.<br>
><br>
>Yes, let's work together. Here is my blueprint (it's somewhat old.<br>
>So needs to be updated.)<br>
><a href="https://blueprints.launchpad.net/neutron/+spec/fix-races-of-db-based-plugi" target="_blank">https://blueprints.launchpad.net/neutron/+spec/fix-races-of-db-based-plugi</a><br>
>n<br>
><a href="https://docs.google.com/file/d/0B4LNMvjOzyDuU2xNd0piS3JBMHM/edit" target="_blank">https://docs.google.com/file/d/0B4LNMvjOzyDuU2xNd0piS3JBMHM/edit</a><br>
><br>
>Although I've thought of status change(adding more status) and locking<br>
>protocol so far, TaskFlow seems something to look at before starting and<br>
>another possible approach is decoupling backend process from api call<br>
>as Salvatore suggested like NVP plugin.<br>
>Even with taskflow or decoupling approach, some kind of enhancing status<br>
>change/locking protocol will be necessary for performance of creating<br>
>many ports at once.<br>
><br>
>thanks,<br>
><br>
>><br>
>> Thanks,<br>
>><br>
>> Edgar<br>
>><br>
>> On 11/19/13 3:57 AM, "Isaku Yamahata" <<a href="mailto:isaku.yamahata@gmail.com">isaku.yamahata@gmail.com</a>> wrote:<br>
>><br>
>> >On Mon, Nov 18, 2013 at 03:55:49PM -0500,<br>
>> >Robert Kukura <<a href="mailto:rkukura@redhat.com">rkukura@redhat.com</a>> wrote:<br>
>> ><br>
>> >> On 11/18/2013 03:25 PM, Edgar Magana wrote:<br>
>> >> > Developers,<br>
>> >> ><br>
>> >> > This topic has been discussed before but I do not remember if we<br>
>>have<br>
>> >>a<br>
>> >> > good solution or not.<br>
>> >><br>
>> >> The ML2 plugin addresses this by calling each MechanismDriver twice.<br>
>>The<br>
>> >> create_network_precommit() method is called as part of the DB<br>
>> >> transaction, and the create_network_postcommit() method is called<br>
>>after<br>
>> >> the transaction has been committed. Interactions with devices or<br>
>> >> controllers are done in the postcommit methods. If the postcommit<br>
>>method<br>
>> >> raises an exception, the plugin deletes that partially-created<br>
>>resource<br>
>> >> and returns the exception to the client. You might consider a similar<br>
>> >> approach in your plugin.<br>
>> ><br>
>> >Splitting works into two phase, pre/post, is good approach.<br>
>> >But there still remains race window.<br>
>> >Once the transaction is committed, the result is visible to outside.<br>
>> >So the concurrent request to same resource will be racy.<br>
>> >There is a window after pre_xxx_yyy before post_xxx_yyy() where<br>
>> >other requests can be handled.<br>
>> ><br>
>> >The state machine needs to be enhanced, I think. (plugins need<br>
>> >modification)<br>
>> >For example, adding more states like pending_{create, delete, update}.<br>
>> >Also we would like to consider serializing between operation of ports<br>
>> >and subnets. or between operation of subnets and network depending on<br>
>> >performance requirement.<br>
>> >(Or carefully audit complex status change. i.e.<br>
>> >changing port during subnet/network update/deletion.)<br>
>> ><br>
>> >I think it would be useful to establish reference locking policy<br>
>> >for ML2 plugin for SDN controllers.<br>
>> >Thoughts or comments? If this is considered useful and acceptable,<br>
>> >I'm willing to help.<br>
>> ><br>
>> >thanks,<br>
>> >Isaku Yamahata<br>
>> ><br>
>> >> -Bob<br>
>> >><br>
>> >> > Basically, if concurrent API calls are sent to Neutron, all of them<br>
>> >>are<br>
>> >> > sent to the plug-in level where two actions have to be made:<br>
>> >> ><br>
>> >> > 1. DB transaction ? No just for data persistence but also to<br>
>>collect<br>
>> >>the<br>
>> >> > information needed for the next action<br>
>> >> > 2. Plug-in back-end implementation ? In our case is a call to the<br>
>> >>python<br>
>> >> > library than consequentially calls PLUMgrid REST GW (soon SAL)<br>
>> >> ><br>
>> >> > For instance:<br>
>> >> ><br>
>> >> > def create_port(self, context, port):<br>
>> >> > with context.session.begin(subtransactions=True):<br>
>> >> > # Plugin DB - Port Create and Return port<br>
>> >> > port_db = super(NeutronPluginPLUMgridV2,<br>
>> >> > self).create_port(context,<br>
>> >> ><br>
>> >> port)<br>
>> >> > device_id = port_db["device_id"]<br>
>> >> > if port_db["device_owner"] == "network:router_gateway":<br>
>> >> > router_db = self._get_router(context, device_id)<br>
>> >> > else:<br>
>> >> > router_db = None<br>
>> >> > try:<br>
>> >> > LOG.debug(_("PLUMgrid Library: create_port()<br>
>>called"))<br>
>> >> > # Back-end implementation<br>
>> >> > self._plumlib.create_port(port_db, router_db)<br>
>> >> > except Exception:<br>
>> >> > ?<br>
>> >> ><br>
>> >> > The way we have implemented at the plugin-level in Havana (even in<br>
>> >> > Grizzly) is that both action are wrapped in the same "transaction"<br>
>> >>which<br>
>> >> > automatically rolls back any operation done to its original state<br>
>> >> > protecting mostly the DB of having any inconsistency state or left<br>
>> >>over<br>
>> >> > data if the back-end part fails.=.<br>
>> >> > The problem that we are experiencing is when concurrent calls to<br>
>>the<br>
>> >> > same API are sent, the number of operation at the plug-in back-end<br>
>>are<br>
>> >> > long enough to make the next concurrent API call to get stuck at<br>
>>the<br>
>> >>DB<br>
>> >> > transaction level, which creates a hung state for the Neutron<br>
>>Server<br>
>> >>to<br>
>> >> > the point that all concurrent API calls will fail.<br>
>> >> ><br>
>> >> > This can be fixed if we include some "locking" system such as<br>
>>calling:<br>
>> >> ><br>
>> >> > from neutron.common import utile<br>
>> >> > ?<br>
>> >> ><br>
>> >> > @utils.synchronized('any-name', external=True)<br>
>> >> > def create_port(self, context, port):<br>
>> >> > ?<br>
>> >> ><br>
>> >> > Obviously, this will create a serialization of all concurrent calls<br>
>> >> > which will ends up in having a really bad performance. Does anyone<br>
>> >>has a<br>
>> >> > better solution?<br>
>> >> ><br>
>> >> > Thanks,<br>
>> >> ><br>
>> >> > Edgar<br>
>> >> ><br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > OpenStack-dev mailing list<br>
>> >> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> >> ><br>
>> >><br>
>> >><br>
>> >> _______________________________________________<br>
>> >> OpenStack-dev mailing list<br>
>> >> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>> >--<br>
>> >Isaku Yamahata <<a href="mailto:isaku.yamahata@gmail.com">isaku.yamahata@gmail.com</a>><br>
>><br>
>><br>
><br>
>--<br>
>Isaku Yamahata <<a href="mailto:isaku.yamahata@gmail.com">isaku.yamahata@gmail.com</a>><br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>