Open Stack

Hi Ilya,
Thanks for investigating the concurrency issue. It is indeed a problem from multithreaded neutron-server point of view.

Regarding the opt.2, there is ongoing work to add the revision number[1] to main neutron resources (port, network, subnet...) This is connected to spec [2] and Launchpad bug [3]: push all object changes as AMQP notifications. I think [1] is implementing part of your idea about version number. If version number will be added to standard attributes table, it will automagically appear as a new field in object.

What is left to be done in opt.2 is to handle situation when object is trying to update the state and the revision is not the latest. Then, the object should before update() fetch the current state and try to merge the requested changes to the current state of the object, and then apply it in the DB. Also it would send the newest object state via AMQP notification, obsoleting the previous configuration.

We can also check how nova is handling the concurrency issue in their OVO usage model.

Regards,
Artur
IRC: korzen

[1] https://review.openstack.org/#/c/303966/11
[2] https://review.openstack.org/#/c/225995
[3] https://bugs.launchpad.net/neutron/+bug/1516195

-----Original Message-----
From: Ilya Chukhnakov [mailto:ichukhnakov at mirantis.com] 
Sent: Thursday, May 12, 2016 8:26 PM
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

Hi everyone.

I’ve recently found that straightforward use of NeutronDbObject is prone to concurrency-related problems.

I’ve submitted a patch set [3] with some tests to show that without special treatment using NeutronDbObject could lead to unexpected results.

Further patch sets will provide acquire_object/acquire_objects contextmanager methods to the NeutronDbObject class. These methods are to be used in place of get_object/get_objects whenever the user intends to make changes to the object.
These methods would start an autonested_transaction.

There are (at least) two potential options for the implementation:

1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).

   pros:
     - the object is guaranteed to not be changed while within the context

   cons:
     - prone to deadlocks ([1] and potentially when locking multiple objects)

2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
   counter [2] or add our own. If conflicting changes are detected upon exiting
   the context (i.e. version counter held differs from the one in the DB), will
   raise OSLO RetryRequest exception.

   pros:
     - does not require locking

   cons:
     - require an additional field in the models

While opt.2 only prevents the conflicting changes, but does not guarantee that the object does not change while within the context, opt.1 may seem preferential. But even with opt.1 the user should not expect that the changes made to the object while within the context will get to the database as the autonested_transaction could fail on flush/commit.

So I’d like to hear others’ opinion on the problem and which of the two implementation options would be preferred? Or maybe someone has a better idea.

[1] https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
[2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html

[3] https://review.openstack.org/#/c/315705/

--
Thanks,
Ilya
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev