Open Stack

You're right Joshua.

Tooz HEAD points to 0f4e1198fdcbd6a29d77c67d105d201ed0fbd9e0.

With regards to etcd and zookeeper's versions, they are:
zookeeper-3.4.5+28-1.cdh4.7.1.p0.13.el6.x86_64,
etcd-2.2.5-2.el7.0.1.x86_64.

John.

On Thu, Jul 21, 2016 at 8:14 PM, Joshua Harlow <harlowja at fastmail.com> wrote:
> Hi John,
>
> Thanks for gathering this info,
>
> Do you have the versions of the backend that were used here (particularly
> relevant for etcd which has a new release pretty frequently).
>
> It'd be useful to capture that info also :)
>
> John Schwarz wrote:
>>
>> Hi everyone,
>>
>> Following [1], a few of us sat down during the last day of the Austin
>> Summit and discussed the possibility of adding formal support for
>> Tooz, specifically for the locking mechanism it provides. The
>> conclusion we reached was that benchmarks should be done to show if
>> and how Tooz affects the normal operation of Neutron (i.e. if locking
>> a resource using Zookeeper takes 3 seconds, it's not worthwhile at
>> all).
>>
>> We've finally finished the benchmarks and they are available at [2].
>> They test a specific case: when creating an HA router a lock-free
>> algorithm is used to assign a vrid to a router (this is later used for
>> keepalived), and the benchmark specifically checks the effects of
>> locking that function with either Zookeeper or Etcd, using the no-Tooz
>> case as a baseline. The locking was checked in 2 different ways - one
>> which presents no contention (acquire() always succeeds immediately)
>> and one which presents contentions (acquire() may block until a
>> similar process for the invoking tenant is complete).
>>
>> The benchmarks show that while using Tooz does raise the cost of an
>> operation, the effects are not as bad as we initially feared. In the
>> simple, single simultaneous request, using Zookeeper raised the
>> average time it took to create a router by 1.5% (from 11.811 to 11.988
>> seconds). On the more-realistic case of 6 simultaneous requests,
>> Zookeeper raised the cost by 3.74% (from 16.533 to 17.152 seconds).
>>
>> It is important to note that the setup itself was overloaded - it was
>> built on a single baremetal hosting 5 VMs (4 of which were
>> controllers) and thus we were unable to go further - for example, 10
>> concurrent requests overloaded the server and caused some race
>> conditions to appear in the L3 scheduler (bugs will be opened soon),
>> so for this reason we haven't tested heavier samples and limited
>> ourselves to 6 simultaneous requests.
>>
>> Also important to note that some kind of race condition was noticed in
>> tooz's etcd driver. We've discussed this with the tooz devs and
>> provided a patch that is supposed to fix them [3].
>> Lastly, races in the L3 HA Scheduler were found and we are yet to dig
>> into them and find out their cause - bugs will be opened for these as
>> well.
>>
>> I've opened the summary [2] for comments so you're welcome to open a
>> discussion about the results both in the ML and on the doc itself.
>>
>> (CC to all those who attended the Austin Summit meeting and other
>> interested parties)
>> Happy locking,
>>
>> [1]:
>> http://lists.openstack.org/pipermail/openstack-dev/2016-April/093199.html
>> [2]:
>> https://docs.google.com/document/d/1jdI8gkQKBE0G9koR0nLiW02d5rwyWv_-gAp7yavt4w8
>> [3]: https://review.openstack.org/#/c/342096/
>>
>> --
>> John Schwarz,
>> Senior Software Engineer,
>> Red Hat.
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
John Schwarz,
Senior Software Engineer,
Red Hat.