Open Stack

On 03/14/2017 05:01 PM, Clint Byrum wrote:
> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
>>> On Tue, Mar 14 2017, Jay Pipes wrote:
>>>
>>>> Not tooz, because I'm not interested in a DLM nor leader election library
>>>> (that's what the underlying etcd3 cluster handles for me), only a fast service
>>>> liveness/healthcheck system, but it shows usage of etcd3 and Google Protocol
>>>> Buffers implementing a simple API for liveness checking and host maintenance
>>>> reporting.
>>>
>>> Cool cool. So that's the same feature that we implemented in tooz 3
>>> years ago. It's called "group membership". You create a group, make
>>> nodes join it, and you know who's dead/alive and get notified when their
>>> status change.
>>
>> The point of os-lively is not to provide a thin API over ZooKeeper's
>> group membership interface. The point of os-lively is to remove the need
>> to have a database (RDBMS) record of a service in Nova.
>
> That's also the point of tooz's group membership API:
>
> https://docs.openstack.org/developer/tooz/compatibility.html#grouping

Did you take a look at the code I wrote in os-lively? What part of the 
tooz group membership API do you think I would have used?

Again, this was a weekend project that I was moving fast on. I looked at 
tooz and didn't see how I could use it for my purposes, which was to 
store a versioned object in a consistent key/value store with support 
for transactional semantics when storing index and data records at the 
same time [1]

https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511

etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional 
semantics in a consistent key/value store that I needed.

tooz is cool, but it's not what I was looking for. It's solving a 
different problem than I was trying to solve.

This isn't a case of NIH, despite what Julien is trying to intimate in 
his emails.

>> tooz simply abstracts a group membership API across a number of drivers.
>> I don't need that. I need a way to maintain a service record (with
>> maintenance period information, region, and an evolvable data record
>> format) and query those service records in an RDBMS-like manner but
>> without the RDBMS being involved.
>>
>>>> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
>>>> service liveness checking, which should dramatically reduce the amount of both
>>>> DB traffic as well as conductor/MQ service update traffic.
>>>
>>> Interesting. Joshua and Vilob tried to push usage of tooz group
>>> membership a couple of years ago, but it got nowhere. Well, no, they got
>>> 2 specs written IIRC:
>>>
>>>   https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>>>
>>> But then it died for whatever reasons on Nova side.
>>
>> It died because it didn't actually solve a problem.
>>
>> The problem is that even if we incorporate tooz, we would still need to
>> have a service table in the RDBMS and continue to query it over and over
>> again in the scheduler and API nodes.
>
> Most likely it was designed with hesitance to have a tooz requirement
> to be a source of truth. But it's certainly not a problem for most tooz
> backends to be a source of truth. Certainly not for etcd or ZK, which
> are both designed to be that.
>
>> I want all service information in the same place, and I don't want to
>> use an RDBMS for that information. etcd3 provides an ideal place to
>> store service record information. Google Protocol Buffers is an ideal
>> data format for evolvable versioned objects. os-lively presents an API
>> that solves the problem I want to solve in Nova. tooz didn't.
>
> Was there something inherent in tooz's design that prevented you from
> adding it to tooz's group API? Said API already includes liveness (watch
> the group that corresponds to the service you want).

See above about transactional semantics.

I'm actually happy to add an etcd3 group membership driver to tooz, 
though. After the experience gained this weekend using etcd3, I'd like 
to do that.

Still doesn't mean that tooz would be the appropriate choice for what I 
was trying to do with os-lively, though.

> The only thing missing is being able to get groups and group members
> by secondary indexes. etcd3's built in indexes by field are pretty nice

Not sure what you're talking about. etcd3 doesn't have any indexing by 
field. I built the os-lively library primarily as a well-defined set of 
index overlays (by uuid, by host, by service type, and by region) over 
etcd3's key/value store.

> for that, but ZK can likely also do it too by maintaining the index in
> the driver.

Maybe, I'm not sure, I didn't spend much time this weekend looking at 
ZooKeeper.

> I understand abstractions can seem pretty cumbersome when you're moving
> fast. It's not something I want to see stand in your way. But it would
> be nice to see where there's deficiency in tooz so we can be there for
> the next project that needs it and maybe eventually factor out direct
> etcd3 usage so users who have maybe chosen ZK as their tooz backend can
> also benefit from your work.

It's not a deficiency in tooz. It's a different problem domain. Look at 
the os-lively API and show me how you think I could have used tooz to 
implement that API.

Best,
-jay