[openstack-dev] [nova] Distributed locking

Joshua Harlow harlowja at yahoo-inc.com
Wed Jun 25 17:35:23 UTC 2014


Could u expand on this and how it would work.

I'm pretty skeptical of new ad-hoc locking implementations so just want to
ensure it's flushed out in detail.

What would the two local locks be, where would they be, what would the
'conducting' being doing to coordinate?

-----Original Message-----
From: John Garbutt <john at johngarbutt.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Wednesday, June 25, 2014 at 1:08 AM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [nova] Distributed locking

>So just to keep the ML up with some of the discussion we had in IRC
>the other day...
>
>Most resources in Nova are owned by a particular nova-compute. So the
>locks on the resources are effectively held by the nova-compute that
>owns the resource.
>
>We already effectively have a cross nova-compute "lock" holding in the
>capacity reservations during migrate/resize.
>
>But to cut a long story short, if the image cache is actually just a
>copy from one of the nova-compute nodes that already have that image
>into the local (shared) folder for another nova-compute, then we can
>get away without a global lock, and just have two local locks on
>either end and some "conducting" to co-ordinate things.
>
>Its not perfect, but its an option.
>
>Thanks,
>John
>
>
>On 17 June 2014 18:18, Clint Byrum <clint at fewbar.com> wrote:
>> Excerpts from Matthew Booth's message of 2014-06-17 01:36:11 -0700:
>>> On 17/06/14 00:28, Joshua Harlow wrote:
>>> > So this is a reader/write lock then?
>>> >
>>> > I have seen https://github.com/python-zk/kazoo/pull/141 come up in
>>>the
>>> > kazoo (zookeeper python library) but there was a lack of a
>>>maintainer for
>>> > that 'recipe', perhaps if we really find this needed we can help get
>>>that
>>> > pull request 'sponsored' so that it can be used for this purpose?
>>> >
>>> >
>>> > As far as resiliency, the thing I was thinking about was how correct
>>>do u
>>> > want this lock to be?
>>> >
>>> > If u say go with memcached and a locking mechanism using it this
>>>will not
>>> > be correct but it might work good enough under normal usage. So
>>>that¹s why
>>> > I was wondering about what level of correctness do you want and what
>>>do
>>> > you want to happen if a server that is maintaining the lock record
>>>dies.
>>> > In memcaches case this will literally be 1 server, even if sharding
>>>is
>>> > being used, since a key hashes to one server. So if that one server
>>>goes
>>> > down (or a network split happens) then it is possible for two
>>>entities to
>>> > believe they own the same lock (and if the network split recovers
>>>this
>>> > gets even weirder); so that¹s what I was wondering about when
>>>mentioning
>>> > resiliency and how much incorrectness you are willing to tolerate.
>>>
>>> From my POV, the most important things are:
>>>
>>> * 2 nodes must never believe they hold the same lock
>>> * A node must eventually get the lock
>>>
>>
>> If these are musts, then memcache is a no-go for locking. memcached is
>> likely to delete anything it is storing in its RAM, at any time. Also
>> if you have several memcache servers, a momentary network blip could
>> lead to acquiring the lock erroneously.
>>
>> The only thing it is useful for is coalescing, where a broken lock just
>> means wasted resources, erroneous errors, etc. If consistency is needed,
>> then you need a consistent backend.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list