<div dir="ltr">Hi Edgar, <div><br></div><div>We had a similar issue and worked around by something like the following (which I believe similar to what Aaron said): </div><div><br></div><div><pre style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap;word-wrap:normal;color:rgb(51,51,51);font-size:12px;line-height:16px;background-color:rgb(245,245,245)">
@@ -45,6 +45,8 @@ SNAT_RULE_PROPERTY = {OS_TENANT_ROUTER_RULE_KEY: SNAT_RULE}
class MidonetResourceNotFound(q_exc.NotFound):
message = _('MidoNet %(resource_type)s %(id)s could not be found')
+from eventlet.semaphore <span style="color:rgb(0,0,145)">import</span> Semaphore
+PORT_ALLOC_SEM = Semaphore()
class MidonetPluginV2(db_base_plugin_v2.QuantumDbPluginV2,
l3_db.L3_NAT_db_mixin):
@@ -428,21 +430,31 @@ class MidonetPluginV2(db_base_plugin_v2.QuantumDbPluginV2,
# set midonet port id to quantum port id and create a DB record.
port_data['id'] = bridge_port.get_id()
- session = context.session
- with session.begin(subtransactions=True):
- qport = <span style="color:rgb(0,0,145)">super</span>(MidonetPluginV2, self).create_port(context, port)
- <span style="color:rgb(0,0,145)">if</span> is_compute_interface:
- # get ip and mac from DB record.
- fixed_ip = qport['fixed_ips'][0]['ip_address']
- mac = qport['mac_address']
-
+ qport = None
+ with PORT_ALLOC_SEM:
+ session = context.session
+ with session.begin(subtransactions=True):
+ qport = <span style="color:rgb(0,0,145)">super</span>(MidonetPluginV2, self).create_port(context, port)
+ <span style="color:rgb(0,0,145)">if</span> is_compute_interface:
+ # get ip and mac from DB record.
+ id = qport['id']
+ fixed_ip = qport['fixed_ips'][0]['ip_address']
+ mac = qport['mac_address']
+
+ <span style="color:rgb(0,0,145)">if</span> qport and is_compute_interface:
+ <span style="color:rgb(0,0,145)">try</span>:
# create dhcp host entry under the bridge.
dhcp_subnets = bridge.get_dhcp_subnets()
<span style="color:rgb(0,0,145)">if</span> len(dhcp_subnets) > 0:
dhcp_subnets[0].add_dhcp_host().ip_addr(fixed_ip)\
.mac_addr(mac)\
.create()
- <span style="color:rgb(0,0,145)">return</span> qport
+ <span style="color:rgb(0,0,145)">return</span> qport
+ except Exception:
+ self.delete_port(context, id)
+ <span style="color:rgb(0,0,145)">return</span> None
+ <span style="color:rgb(0,0,145)">else</span>:
+ <span style="color:rgb(0,0,145)">return</span> qport
def update_port(self, context, id, port):
"""
</pre></div><div><br></div><div>We are also looking to fix this for upstream icehouce. </div><div><br></div><div>Also, I have just submitted a (regression) test for this in tempest: </div><div><a href="https://review.openstack.org/#/c/57355">https://review.openstack.org/#/c/57355</a><br>
</div><div><br></div><div>Hope the test makes sense.</div><div><br></div><div>Thanks,</div><div>Tomoe</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Nov 19, 2013 at 5:25 AM, Edgar Magana <span dir="ltr"><<a href="mailto:emagana@plumgrid.com" target="_blank">emagana@plumgrid.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word"><div>Developers,</div><div><br></div><div>
This topic has been discussed before but I do not remember if we have a good solution or not.</div><div>Basically, if concurrent API calls are sent to Neutron, all of them are sent to the plug-in level where two actions have to be made:</div>
<div><div><br></div><div>1. DB transaction – No just for data persistence but also to collect the information needed for the next action</div><div>2. Plug-in back-end implementation – In our case is a call to the python library than consequentially calls PLUMgrid REST GW (soon SAL)</div>
<div><br></div><div>For instance:</div><div><br></div><div><div>def create_port(self, context, port):</div><div> with context.session.begin(subtransactions=True):</div><div> # Plugin DB - Port Create and Return port</div>
<div> port_db = super(NeutronPluginPLUMgridV2, self).create_port(context,</div><div> port)</div><div> device_id = port_db["device_id"]</div>
<div> if port_db["device_owner"] == "network:router_gateway":</div><div> router_db = self._get_router(context, device_id)</div><div> else:</div><div> router_db = None</div>
<div> try:</div><div> LOG.debug(_("PLUMgrid Library: create_port() called"))</div><div><span style="white-space:pre-wrap"> </span># Back-end implementation</div><div> self._plumlib.create_port(port_db, router_db)</div>
<div> except Exception:</div><div> …</div></div><div><br></div><div>The way we have implemented at the plugin-level in Havana (even in Grizzly) is that both action are wrapped in the same "transaction" which automatically rolls back any operation done to its original state protecting mostly the DB of having any inconsistency state or left over data if the back-end part fails.=.</div>
<div>The problem that we are experiencing is when concurrent calls to the same API are sent, the number of operation at the plug-in back-end are long enough to make the next concurrent API call to get stuck at the DB transaction level, which creates a hung state for the Neutron Server to the point that all concurrent API calls will fail.</div>
</div><div><br></div><div>This can be fixed if we include some "locking" system such as calling:</div><div><br></div><div>from neutron.common import utile</div><div>…</div><div><br></div><div>@utils.synchronized('any-name', external=True)</div>
<div>def create_port(self, context, port):</div><div>…</div><div><br></div><div>Obviously, this will create a serialization of all concurrent calls which will ends up in having a really bad performance. Does anyone has a better solution?</div>
<div><br></div><div>Thanks,</div><div><br></div><div>Edgar</div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>