Open Stack

>I believe Clint already linked to
>https://aphyr.com/posts/309-knossos-redis-and-linearizability or
>similar - but 'known for general ease of use and reliability' is uhm,
>a bold claim. Its worth comparing that (and the other redis writeups)
>to this one: https://aphyr.com/posts/291-call-me-maybe-zookeeper. "Use
>zookeeper, its mature".

Those write ups are from 2013 and with general improvements in Redis over the last two years I'd find it hard to believe that they're still relevant, however its worth testing to confirm if Redis is deemed as a viable option.

>The openjdk is present on the same linux distributions, and has been
>used in both open source and proprietary programs for decades. *what*
>license implications are you speaking of?

The license issues would be related to deployers using Oracle java which may or may not be needed by certain deployers for scale and performance requirements. While I do not have specific performance numbers at my fingertips to illustrate general performance issues using zookeeper at scale with OpenJDK, I have, in the past, compared OpenJDK to Oracle Java and found that Oracle java was quite a bit more stable and packed far more performance capabilities.   I did find [ http://blog.cloud-benchmarks.org/2015/07/17/cassandra-write-performance-on-gce-and-aws.html ] which claims a 32% performance improvement with casandra using Oracle Java 8 over OpenJDK on top of the fact that it was less prone to crashes but that may not be entirely relevant to this case. Also there's no denying that Oracle has a questionable history dealing with Opensource projects regarding Java in the past and because performance / stability concerns may require the use of Oracle Java which will undoubtedly come with questionable license requirements.

>I believe you can do this with zookeeper - both single process, or
>three processes on one machine to emulate a cluster - very easily.
>Quoting http://qnalist.com/questions/29943/java-heap-size-for-zookeeper
>- "It's more dependent on your workload than anything. If you're
>storing on order of hundreds of small znodes then 1gb is going to [be]
>more then fine." Obviously we should test this and confirm it, but
>developer efficiency is a key part of any decision, and AFAIK there is
>absolutely nothing in the way as far as zookeeper goes. 

This would be worth while testing to ensure that a developer really can use typical work machines to test without major performance impacts or changes to their workflow.

>Just like rabbitmq and openvswitch, its a mature thing, written in a language
>other than Python, which needs its own care and feeding (and that
>feeding is something like 90% zk specific, not 'java headaches').

Agreed that with different solutions care is needed to facilitate optimal language environments however I'm not sure OpenVSwitch/RabbitMQ may be the best comparison to the point of maturity. As I'm sure you're well aware, both of these pieces of software have been "mature" for some time however up until real recently stable OvS has been known to reach havoc in large scale production environments and RabbitMQ clustering is still really something to be desired. The point being is just because something is "mature" doesn't mean that its the most stable and or the right solution.

> The default should be suitable for use in the majority of clouds

IDK if we can quantify that Zookeeper fits the "majority" of clouds as I believe there'll be scale issues while using OpenJDK forcing the use of Oracla Java and the potential for inadvertent license issues. That said, I have no real means to say Redis will be any better, however prior operational experience tells me that managing zookeeper at real scale is a PITA.

I too wouldn't mind seeing a solution using consul put forth as the default. It's really an interesting approach, provides multiple backends (http/dns/etc), should scale to multiple DCs without much  if any hacking, written in go (using 1.5.1+ in master and 1.4.1+ in stable) which will come with some impressive performance capabilities, is under active development, licensed "Mozilla Public License, version 2.0", should be fairly minimal in terms of resource requirements on development machines, and hits many of the other shinneyness factors developers / deployers may be interested in. 


All said I'm very interested in a DLM solution and if there's anything I can do to help make it go please let me know.

--

Kevin Carter
IRC: cloudnull


________________________________________
From: Robert Collins <robertc at robertcollins.net>
Sent: Tuesday, November 10, 2015 12:55 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all] Outcome of distributed lock manager discussion @ the summit

On 10 November 2015 at 19:24, Kevin Carter <kevin.carter at rackspace.com> wrote:
> Hello all,
>
> The rational behind using a solution like zookeeper makes sense however in reviewing the thread I found myself asking if there was a better way to address the problem without the addition of a Java based solution as the default. While it has been covered that the current implementation would be a reference and that "other" driver support in Tooz would allow for any backend a deployer may want, the work being proposed within devstack [0] would become the default development case thus making it the de-facto standard and I think we could do better in terms of supporting developers and delivering capability.
>
> My thoughts on using Redis+Redislock instead of Java+Zookeeper as the default option:
> * Tooz already support redislock
> * Redis has an established cluster system known for general ease of use and reliability on distributed systems.

I believe Clint already linked to
https://aphyr.com/posts/309-knossos-redis-and-linearizability or
similar - but 'known for general ease of use and reliability' is uhm,
a bold claim. Its worth comparing that (and the other redis writeups)
to this one: https://aphyr.com/posts/291-call-me-maybe-zookeeper. "Use
zookeeper, its mature".

> * Several OpenStack projects already support Redis as a backend option or have extended capabilities using a Redis.
> * Redis can be implemented in RHEL, SUSE, and DEB based systems with ease.
> * Redis is Opensource software licensed under the "three clause BSD license" and would not have any of the same questionable license implications as found when dealing with anything Java.

The openjdk is present on the same linux distributions, and has been
used in both open source and proprietary programs for decades. *what*
license implications are you speaking of?

> * The inclusion of Redis would work on a single node allowing developers to continue work using VMs running on Laptops with 4GB or ram but would also scale to support the multi-controller use case with ease. This would also give developers the ability to work on a systems that will actually resemble production.

I believe you can do this with zookeeper - both single process, or
three processes on one machine to emulate a cluster - very easily.
Quoting http://qnalist.com/questions/29943/java-heap-size-for-zookeeper
- "It's more dependent on your workload than anything. If you're
storing on order of hundreds of small znodes then 1gb is going to [be]
more then fine." Obviously we should test this and confirm it, but
developer efficiency is a key part of any decision, and AFAIK there is
absolutely nothing in the way as far as zookeeper goes. Just like
rabbitmq and openvswitch, its a mature thing, written in a language
other than Python, which needs its own care and feeding (and that
feeding is something like 90% zk specific, not 'java headaches').

> * Redislock will bring with it no additional developer facing language dependencies (Redis is written in ANSI C and works ... without external dependencies [1]) while also providing a plethora of language bindings [2].
>
>
> I apologize for questioning the proposed solution so late into the development of this thread and for not making the summit conversations to talk more with everyone whom worked on the proposal. While the ship may have sailed on this point for now I figured I'd ask why we might go down the path of Zookeeper+Java when a solution with likely little to no development effort already exists, can support just about any production/development environment, has lots of bindings, and (IMHO) would integrate with the larger community easier; many OpenStack developers and deployers already know Redis. With the inclusion of ZK+Java in DevStack and the act of making it the default it essentially creates new hard dependencies one of which is Java and I'd like to avoid that if at all possible; basically I think we can do better.


I think its fine to raise the question, but lets perhaps set some priorities.

1) The default should be mature
2) The default should be suitable for developer use on a modest laptop
3) The default should be suitable for use in the majority of clouds

I believe zk meets all those priorities, and redis does not. Its
possible that etcd and or consul do: though they are much newer and so
perhaps fail on the maturity scale. I'm certain redis does not - at
least, not unless the previously reported defects have been fixed in
the last 2 years.

-Rob



--
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev