Open Stack

On 05/12/2016 08:26 PM, Ilya Chukhnakov wrote:
> Hi everyone.
>
> I’ve recently found that straightforward use of NeutronDbObject is prone to
> concurrency-related problems.
>
> I’ve submitted a patch set [3] with some tests to show that without special
> treatment using NeutronDbObject could lead to unexpected results.
>
> Further patch sets will provide acquire_object/acquire_objects contextmanager
> methods to the NeutronDbObject class. These methods are to be used in place of
> get_object/get_objects whenever the user intends to make changes to the object.
> These methods would start an autonested_transaction.
>
> There are (at least) two potential options for the implementation:
>
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
>
>    pros:
>      - the object is guaranteed to not be changed while within the context
>
>    cons:
>      - prone to deadlocks ([1] and potentially when locking multiple objects)
>
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
>    counter [2] or add our own. If conflicting changes are detected upon exiting
>    the context (i.e. version counter held differs from the one in the DB), will
>    raise OSLO RetryRequest exception.
>
>    pros:
>      - does not require locking
>
>    cons:
>      - require an additional field in the models
>
> While opt.2 only prevents the conflicting changes, but does not guarantee that
> the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the changes
> made to the object while within the context will get to the database as the
> autonested_transaction could fail on flush/commit.
>
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better idea.
>
> [1] https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
>
> [3] https://review.openstack.org/#/c/315705/

In Cinder we're handling similar problems related to race conditions
between status check and status change with conditional updates.
Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND
status='status_allowing_transition_to_foo';". You can check out o.vo
layer of that stuff at [1].

You could provide fields you care not to be modified in the
expected_values and retry DB operations on failure.

In general the problem you're mentioning is related more to the
concurrent DB updates and o.vo never aimed for magically solving that
problem. I believe you've had same problem with raw SQLA objects.

[1]
https://github.com/openstack/cinder/blob/master/cinder/objects/base.py#L173-L289