Open Stack

This works as long as you have 1 DB and don't fail over to a secondary
slave DB.

Now u can say we all must use percona (or similar) for this, but then
that¹s a change in deployment as well (and imho a bigger one). This is
where the concept of a quorum in zookeeper comes into play, the
transaction log that zookeeper maintains will be consistent among all
members in that quorum. This is a typical zookeeper deployment strategy
(select how many nodes u want for that quorum being an important question).

It also doesn't handle the case where u can automatically recover from the
current resource owner (nova-compute for example) dying.

Your atomic "check-if-owner-is-empty-and-store-instance-as-owner" is now
user initiated instead of being automatic (zookeeper provides these kind
of notifications via its watch concept). So that makes it hard for say an
automated system (heat?) to react to these failures in any other way than
repeated polling (or repeated retries or periodic tasks) which means that
heat will not be able to react to failure in a 'live' manner. So this to
me is the liveness question that zookeeper is designed to help out with,
of course u can simulate this in a DB and repeated polling (as long as u
don't try to do anything complicated with mysql, like replicas/slaves with
transaction logs that may not be caught up and that u might have to fail
over to if problems happen, since u are on your own if this happens).

-Josh

On 10/30/13 1:02 PM, "Chris Friesen" <chris.friesen at windriver.com> wrote:

>On 10/30/2013 01:34 PM, Joshua Harlow wrote:
>> To me u just made state consistency be a lock by another name.
>>
>> A lock protects a region of code from being mutually accessed
>
>Personally I view a lock as protecting a set of data from being mutually
>accessed.
>
>> The question to me becomes what
>> happens to that state consistency when its running in a distributed
>> system, which all of openstack is running in. At that point u need a way
>> to ensure multiple servers (going through various states) are not
>> manipulating the same resources at the same time (delete volume from
>> cinder, while attaching it in nova). Those 2 separate services do not
>> likely share the same state transitions (and will likely not as they
>> become tightly coupled at that point). So then u need some type of
>> coordination system to ensure the ordering of these 2 resource actions
>> is done in a consistent manner.
>
>This sort of thing seems solvable by a "reserve-before-use" kind of
>model, without needing any mutex locking as such.
>
>When attaching, do an atomic
>"check-if-owner-is-empty-and-store-instance-as-owner" transaction to
>store the instance as the owner of the volume very early. Then reload
>from the database to make sure the instance is the current owner, and
>now you're guaranteed that nobody can delete it under your feet.
>
>When deleting, if the current owner is set and the owner instance exists
>then bail out with an error.
>
>This is essentially akin to using atomic-test-and-set instead of a mutex.
>
>Chris
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev