Open Stack

On 02/19/2016 06:05 PM, Joshua Harlow wrote:
> Hi all,
>
> After reading over the following interesting article about redis and redlock
> (IMHO it's good overview of distributed locking in general):
>
> http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html#protecting-a-resource-with-a-lock
> (I personally recommend people read the whole article as well, as it's rather
> interesting, as well as the response from the redis author at
> http://antirez.com/news/101).
>
> It got me wondering if with all the locking and such that is getting used in
> openstack (distributed or not) that as we move to more distributed locking
> mechanisms (for scale reasons, HA, active-active...) that we might need to have
> a way to fence modifications of a storage entry (say belonging to a resource, ie
> a volume, a network...) with a token (or sequence-id) so that the problems
> mentioned in that blog do not affect openstack (apparently issues like it have
> affected hbase) and the more we think about it now (vs. later) the better we
> will be.
>
> Anyone have any thoughts on this?
>
> Perhaps tooz can along with its lock API also provide a token for each lock that
> can be used to interact with a storage layer (and that token can checked by the
> storage layer to avoid storage layer corruption).

It may also be beneficial to take a page from POSIX "robust mutexes" and 
introduce a way for the new lock holder to be notified that the previous lock 
holder's lease expired rather than the lock being unlocked cleanly.

This alerts the new lock holder that the data protected by the lock may be 
self-inconsistent since the old lock holder may have died while in the middle of 
modifying things.

Chris