Open Stack

Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
> On 03/14/2017 02:50 PM, Julien Danjou wrote:
> > On Tue, Mar 14 2017, Jay Pipes wrote:
> >
> >> Not tooz, because I'm not interested in a DLM nor leader election library
> >> (that's what the underlying etcd3 cluster handles for me), only a fast service
> >> liveness/healthcheck system, but it shows usage of etcd3 and Google Protocol
> >> Buffers implementing a simple API for liveness checking and host maintenance
> >> reporting.
> >
> > Cool cool. So that's the same feature that we implemented in tooz 3
> > years ago. It's called "group membership". You create a group, make
> > nodes join it, and you know who's dead/alive and get notified when their
> > status change.
> 
> The point of os-lively is not to provide a thin API over ZooKeeper's 
> group membership interface. The point of os-lively is to remove the need 
> to have a database (RDBMS) record of a service in Nova.
> 

That's also the point of tooz's group membership API:

https://docs.openstack.org/developer/tooz/compatibility.html#grouping

> tooz simply abstracts a group membership API across a number of drivers. 
> I don't need that. I need a way to maintain a service record (with 
> maintenance period information, region, and an evolvable data record 
> format) and query those service records in an RDBMS-like manner but 
> without the RDBMS being involved.
> 
> >> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
> >> service liveness checking, which should dramatically reduce the amount of both
> >> DB traffic as well as conductor/MQ service update traffic.
> >
> > Interesting. Joshua and Vilob tried to push usage of tooz group
> > membership a couple of years ago, but it got nowhere. Well, no, they got
> > 2 specs written IIRC:
> >
> >   https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
> >
> > But then it died for whatever reasons on Nova side.
> 
> It died because it didn't actually solve a problem.
> 
> The problem is that even if we incorporate tooz, we would still need to 
> have a service table in the RDBMS and continue to query it over and over 
> again in the scheduler and API nodes.
> 

Most likely it was designed with hesitance to have a tooz requirement
to be a source of truth. But it's certainly not a problem for most tooz
backends to be a source of truth. Certainly not for etcd or ZK, which
are both designed to be that.

> I want all service information in the same place, and I don't want to 
> use an RDBMS for that information. etcd3 provides an ideal place to 
> store service record information. Google Protocol Buffers is an ideal 
> data format for evolvable versioned objects. os-lively presents an API 
> that solves the problem I want to solve in Nova. tooz didn't.
> 

Was there something inherent in tooz's design that prevented you from
adding it to tooz's group API? Said API already includes liveness (watch
the group that corresponds to the service you want).

The only thing missing is being able to get groups and group members
by secondary indexes. etcd3's built in indexes by field are pretty nice
for that, but ZK can likely also do it too by maintaining the index in
the driver.

I understand abstractions can seem pretty cumbersome when you're moving
fast. It's not something I want to see stand in your way. But it would
be nice to see where there's deficiency in tooz so we can be there for
the next project that needs it and maybe eventually factor out direct
etcd3 usage so users who have maybe chosen ZK as their tooz backend can
also benefit from your work.