<div dir="ltr">><span style="font-size:12.8px">What is left to be done in opt.2 is to handle situation when object is trying to update the state and the revision is not the latest.</span><div><br></div><div><br></div><div>Once the revision counter is added, this will generate an error (StaleData I think) that will be caught by the retry decorator so the process can be started over by fetching the latest state.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 13, 2016 at 1:22 AM, Korzeniewski, Artur <span dir="ltr"><<a href="mailto:artur.korzeniewski@intel.com" target="_blank">artur.korzeniewski@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Ilya,<br>
Thanks for investigating the concurrency issue. It is indeed a problem from multithreaded neutron-server point of view.<br>
<br>
Regarding the opt.2, there is ongoing work to add the revision number[1] to main neutron resources (port, network, subnet...) This is connected to spec [2] and Launchpad bug [3]: push all object changes as AMQP notifications. I think [1] is implementing part of your idea about version number. If version number will be added to standard attributes table, it will automagically appear as a new field in object.<br>
<br>
What is left to be done in opt.2 is to handle situation when object is trying to update the state and the revision is not the latest. Then, the object should before update() fetch the current state and try to merge the requested changes to the current state of the object, and then apply it in the DB. Also it would send the newest object state via AMQP notification, obsoleting the previous configuration.<br>
<br>
We can also check how nova is handling the concurrency issue in their OVO usage model.<br>
<br>
Regards,<br>
Artur<br>
IRC: korzen<br>
<br>
[1] <a href="https://review.openstack.org/#/c/303966/11" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/303966/11</a><br>
[2] <a href="https://review.openstack.org/#/c/225995" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/225995</a><br>
[3] <a href="https://bugs.launchpad.net/neutron/+bug/1516195" rel="noreferrer" target="_blank">https://bugs.launchpad.net/neutron/+bug/1516195</a><br>
<div class="HOEnZb"><div class="h5"><br>
-----Original Message-----<br>
From: Ilya Chukhnakov [mailto:<a href="mailto:ichukhnakov@mirantis.com">ichukhnakov@mirantis.com</a>]<br>
Sent: Thursday, May 12, 2016 8:26 PM<br>
To: OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
Subject: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues<br>
<br>
Hi everyone.<br>
<br>
I’ve recently found that straightforward use of NeutronDbObject is prone to concurrency-related problems.<br>
<br>
I’ve submitted a patch set [3] with some tests to show that without special treatment using NeutronDbObject could lead to unexpected results.<br>
<br>
Further patch sets will provide acquire_object/acquire_objects contextmanager methods to the NeutronDbObject class. These methods are to be used in place of get_object/get_objects whenever the user intends to make changes to the object.<br>
These methods would start an autonested_transaction.<br>
<br>
There are (at least) two potential options for the implementation:<br>
<br>
1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).<br>
<br>
pros:<br>
- the object is guaranteed to not be changed while within the context<br>
<br>
cons:<br>
- prone to deadlocks ([1] and potentially when locking multiple objects)<br>
<br>
2. Lock-free CAS based on object version counter. Can use SqlAlchemy version<br>
counter [2] or add our own. If conflicting changes are detected upon exiting<br>
the context (i.e. version counter held differs from the one in the DB), will<br>
raise OSLO RetryRequest exception.<br>
<br>
pros:<br>
- does not require locking<br>
<br>
cons:<br>
- require an additional field in the models<br>
<br>
While opt.2 only prevents the conflicting changes, but does not guarantee that the object does not change while within the context, opt.1 may seem preferential. But even with opt.1 the user should not expect that the changes made to the object while within the context will get to the database as the autonested_transaction could fail on flush/commit.<br>
<br>
So I’d like to hear others’ opinion on the problem and which of the two implementation options would be preferred? Or maybe someone has a better idea.<br>
<br>
[1] <a href="https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad</a><br>
[2] <a href="http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html" rel="noreferrer" target="_blank">http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html</a><br>
<br>
[3] <a href="https://review.openstack.org/#/c/315705/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/315705/</a><br>
<br>
--<br>
Thanks,<br>
Ilya<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>