Open Stack

On Mon, Oct 31, 2022, at 4:29 AM, Christian Rohmann wrote:
>

snip

>    d) Then there only is Zookeeper left, which also ticks all the 
> (feature-)boxes 
> (https://docs.openstack.org/tooz/latest/user/compatibility.html) and is 
> quite a proven tool for coordination also outside of the OpenStack 
> ecosystem.
> On the downside it's not really that well known and common (anymore) 
> outside the "data processing" context (see 
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy).
> Being a Java application it requires a JVM and its dependencies and is 
> quite memory heavy to store just a few megabytes of config data. Looking 
> at more and more people putting their OS control plane into something 
> like Kubernetes it also seems even less suitable to be "moved around" a 
> lot. Another issue might be the lack of a recent and non-EoL version 
> packaged in Ubuntu - see 
> https://bugs.launchpad.net/ubuntu/+source/zookeeper/+bug/1854331. Maybe 
> (!) this could be an indication of how commonly it is used outside of 
> e.g. Support from TLS was only added in 3.5.5 
> (https://zookeeper.apache.org/doc/r3.5.5/zookeeperAdmin.html#Quorum+TLS)
>

Zuul relies on Zookeeper for its coordination and shared state (without tooz). This is nice because it means we can look at the OpenDev Zuul ZK cluster stats for more info. We currently run a three node cluster. Each node is a 4vcpu 4GB memory VM. The JVM itself seems to consume just under a gig of memory per node. Total system memory stats can be seen here [0]. According to `docker image list` the zookeeper container images we are running are 265MB large. If you scroll to the bottom of this grafana dashboard [1] you'll see operating stats for the cluster.

All that to show that zookeeper isn't free, but it also isn't terribly expensive to run either. Particularly when it tends to fill an important role of preventing software from trampling over itself.

As far as installing it goes, we've been happily using the official docker images [2]. They have worked well for us and have been kept up to date (including TLS support). If you don't want to use those images the tarballs upstream publishes [3] include init scripts that can be used to manage zookeeper as a proper service. You just download, verify, extract, and execute the script (assuming you have java installed) and the service runs.

I'm not going to try and convince anyone that they should use Zookeeper or not. I just want to put concrete details on some of these concerns.

[0] http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=70034&rra_id=all
[1] https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=now-7d&to=now
[2] https://hub.docker.com/_/zookeeper
[3] https://zookeeper.apache.org/releases.html