[openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues
michal.dulko at intel.com
Mon May 16 15:18:44 UTC 2016
On 05/12/2016 08:26 PM, Ilya Chukhnakov wrote:
> Hi everyone.
> I’ve recently found that straightforward use of NeutronDbObject is prone to
> concurrency-related problems.
> I’ve submitted a patch set  with some tests to show that without special
> treatment using NeutronDbObject could lead to unexpected results.
> Further patch sets will provide acquire_object/acquire_objects contextmanager
> methods to the NeutronDbObject class. These methods are to be used in place of
> get_object/get_objects whenever the user intends to make changes to the object.
> These methods would start an autonested_transaction.
> There are (at least) two potential options for the implementation:
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
> - the object is guaranteed to not be changed while within the context
> - prone to deadlocks ( and potentially when locking multiple objects)
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
> counter  or add our own. If conflicting changes are detected upon exiting
> the context (i.e. version counter held differs from the one in the DB), will
> raise OSLO RetryRequest exception.
> - does not require locking
> - require an additional field in the models
> While opt.2 only prevents the conflicting changes, but does not guarantee that
> the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the changes
> made to the object while within the context will get to the database as the
> autonested_transaction could fail on flush/commit.
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better idea.
>  https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
>  http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
>  https://review.openstack.org/#/c/315705/
In Cinder we're handling similar problems related to race conditions
between status check and status change with conditional updates.
Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND
status='status_allowing_transition_to_foo';". You can check out o.vo
layer of that stuff at .
You could provide fields you care not to be modified in the
expected_values and retry DB operations on failure.
In general the problem you're mentioning is related more to the
concurrent DB updates and o.vo never aimed for magically solving that
problem. I believe you've had same problem with raw SQLA objects.
More information about the OpenStack-dev