<div dir="ltr">Unfortunately we didn't switch to the new sql driver until liberty so that probably wouldn't be a safe switch in Kilo.<div><br></div><div>Adding a retry will help, but unfortunately that will still block your call for 60 seconds with that driver until the timeout exception is triggered.</div><div>We worked around this in ML2 by identifying the calls that could yield while holding a DB lock and then acquiring a semaphore before doing each one.</div><div>You can see an example here: <a href="https://github.com/openstack/neutron/blob/363eeb06104662ee38aeed04af043899379f6ab8/neutron/plugins/ml2/plugin.py#L1074">https://github.com/openstack/neutron/blob/363eeb06104662ee38aeed04af043899379f6ab8/neutron/plugins/ml2/plugin.py#L1074</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 10, 2016 at 11:27 PM, Divya <span dir="ltr"><<a href="mailto:divya.hc@gmail.com" target="_blank">divya.hc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Mike for the response. I am part of Nuage openstack team. We are looking in to the issue.<div>An extra delete_port call in NuagePlugin's add_router_interface triggers db lockout when insert into routerport ( this is in core neutron ).</div><div>Are you suggesting NuagePlugin should retry in this case or should core neutron, add-router_interface should retry??</div><div>Will give it a try.</div><div><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, May 10, 2016 at 4:54 PM, Mike Bayer <span dir="ltr"><<a href="mailto:mbayer@redhat.com" target="_blank">mbayer@redhat.com</a>></span> wrote:<br></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br>
<br>
On 05/10/2016 04:57 PM, Divya wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
I am trying to run this rally test on stable/kilo<br>
<a href="https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/neutron/create_and_delete_routers.json" rel="noreferrer" target="_blank">https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/neutron/create_and_delete_routers.json</a><br>
<br>
with concurrency 50 and iterations 2000.<br>
<br>
This test basically cretaes routers and subnets<br>
and then calls<br>
router-interface-add<br>
router-interface-delete<br>
<br>
<br>
And i am running this against 3rd party Nuage plugin.<br>
<br>
In the NuagePlugin:<br>
<br>
add_router_interface is something like this:<br>
----------------------------<br>
super().add_router_interface<br>
try:<br>
some calls to external rest server<br>
super().delete_port<br>
except:<br>
<br>
remove_router_interface:<br>
-----------------------------------<br>
super().remove_router_interface<br>
some calls to external rest server<br>
super().create_port()<br>
some calls to external rest server<br>
<br>
<br>
If i comment delete_port in the add_router_interface, i am not hitting<br>
the db lockout issue.<br>
delete_port or any other operations are not within any transaction.<br>
So not sure, why this is leading to db lock timeouts in insert to routerport<br>
<br>
error trace<br>
<a href="http://paste.openstack.org/show/496626/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/496626/</a><br>
<br>
<br>
<br>
Really appreciate any help on this.<br>
</blockquote>
<br>
<br></div></div>
I'm not on the Neutron team, but in general, Openstack applications should be employing retry logic internally which anticipates database deadlocks like these and retries the operation. I'd report this stack trace (especially if it is reproducible) as a bug to this plugin's launchpad project.<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
Divya<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div></div><br></div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>