Open Stack

On 03/15/2017 03:13 AM, Jay Pipes wrote:
> On 03/14/2017 05:01 PM, Clint Byrum wrote:
>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
>>>> On Tue, Mar 14 2017, Jay Pipes wrote:
>>>>
>>>>> Not tooz, because I'm not interested in a DLM nor leader election
>>>>> library
>>>>> (that's what the underlying etcd3 cluster handles for me), only a
>>>>> fast service
>>>>> liveness/healthcheck system, but it shows usage of etcd3 and Google
>>>>> Protocol
>>>>> Buffers implementing a simple API for liveness checking and host
>>>>> maintenance
>>>>> reporting.
>>>>
>>>> Cool cool. So that's the same feature that we implemented in tooz 3
>>>> years ago. It's called "group membership". You create a group, make
>>>> nodes join it, and you know who's dead/alive and get notified when
>>>> their
>>>> status change.
>>>
>>> The point of os-lively is not to provide a thin API over ZooKeeper's
>>> group membership interface. The point of os-lively is to remove the need
>>> to have a database (RDBMS) record of a service in Nova.
>>
>> That's also the point of tooz's group membership API:
>>
>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping
> 
> Did you take a look at the code I wrote in os-lively? What part of the
> tooz group membership API do you think I would have used?
> 
> Again, this was a weekend project that I was moving fast on. I looked at
> tooz and didn't see how I could use it for my purposes, which was to
> store a versioned object in a consistent key/value store with support
> for transactional semantics when storing index and data records at the
> same time [1]
> 
> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
> 
> 
> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
> semantics in a consistent key/value store that I needed.
> 
> tooz is cool, but it's not what I was looking for. It's solving a
> different problem than I was trying to solve.
> 
> This isn't a case of NIH, despite what Julien is trying to intimate in
> his emails.
> 
>>> tooz simply abstracts a group membership API across a number of drivers.
>>> I don't need that. I need a way to maintain a service record (with
>>> maintenance period information, region, and an evolvable data record
>>> format) and query those service records in an RDBMS-like manner but
>>> without the RDBMS being involved.
>>>
>>>>> servicegroup API with os-lively and eliminate Nova's use of an
>>>>> RDBMS for
>>>>> service liveness checking, which should dramatically reduce the
>>>>> amount of both
>>>>> DB traffic as well as conductor/MQ service update traffic.
>>>>
>>>> Interesting. Joshua and Vilob tried to push usage of tooz group
>>>> membership a couple of years ago, but it got nowhere. Well, no, they
>>>> got
>>>> 2 specs written IIRC:
>>>>
>>>>  
>>>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>>>>
>>>>
>>>> But then it died for whatever reasons on Nova side.
>>>
>>> It died because it didn't actually solve a problem.
>>>
>>> The problem is that even if we incorporate tooz, we would still need to
>>> have a service table in the RDBMS and continue to query it over and over
>>> again in the scheduler and API nodes.
>>
>> Most likely it was designed with hesitance to have a tooz requirement
>> to be a source of truth. But it's certainly not a problem for most tooz
>> backends to be a source of truth. Certainly not for etcd or ZK, which
>> are both designed to be that.
>>
>>> I want all service information in the same place, and I don't want to
>>> use an RDBMS for that information. etcd3 provides an ideal place to
>>> store service record information. Google Protocol Buffers is an ideal
>>> data format for evolvable versioned objects. os-lively presents an API
>>> that solves the problem I want to solve in Nova. tooz didn't.
>>
>> Was there something inherent in tooz's design that prevented you from
>> adding it to tooz's group API? Said API already includes liveness (watch
>> the group that corresponds to the service you want).
> 
> See above about transactional semantics.
> 
> I'm actually happy to add an etcd3 group membership driver to tooz,
> though. After the experience gained this weekend using etcd3, I'd like
> to do that.
> 
> Still doesn't mean that tooz would be the appropriate choice for what I
> was trying to do with os-lively, though.
> 
>> The only thing missing is being able to get groups and group members
>> by secondary indexes. etcd3's built in indexes by field are pretty nice
> 
> Not sure what you're talking about. etcd3 doesn't have any indexing by
> field. I built the os-lively library primarily as a well-defined set of
> index overlays (by uuid, by host, by service type, and by region) over
> etcd3's key/value store.
> 
>> for that, but ZK can likely also do it too by maintaining the index in
>> the driver.
> 
> Maybe, I'm not sure, I didn't spend much time this weekend looking at
> ZooKeeper.

a) awesome. when the rest of this dips momentarily into words that might
sound negative, please hear it all wrapped in an "awesome" and know that
my personal desire is to see the thing you're working on be successful
without undue burden...

b) In Tokyo, we had the big discussion about DLMs (where at least my
intent going in to the room was to get us to pick one and only one).
There were three camps in the room who were all vocal:

1) YES! Let's just pick one, I don't care which one
2) I hate Java I don't want to run Zookeeper, so we can't pick that
3) I hate go/don't trust coreos I don't want to run etcd so we can't
pick that

Because of 2 and 3 the group represented by 1 lost and we ended up with:
"crap, we have to use an abstraction library"

I'd argue that unless something has changed significantly, having Nova
grow a direct depend on etcd when the DLM discussion brought us to "the
operators in the room have expressed a need for a pluggable choice
between at least zk and etcd" should be pretty much a non-starter.

Now, being that I was personally in group 1, I'd be THRILLED if we
could, as a community, decide to pick one and skip having an abstraction
library. I still don't care which one - and you know I love gRPC/protobuf.

But I do think that given the anti-etcd sentiment that was expressed was
equally as vehement as the anti-zk sentiment, that we need to circle
back and make a legit call on this topic.

If we can pick one, I think having special-purpose libraries like
os-lively for specific purposes would be neat.

If we still can't pick one, then I think adding the liveness check you
implemented for os-lively as a new feature in tooz and also implementing
the same thing in the zk driver would be necessary. (of course, that'll
probably depend on getting etcd3 support added to tooz and making sure
there is a good functional test for etcd3.

c) awesome

>> I understand abstractions can seem pretty cumbersome when you're moving
>> fast. It's not something I want to see stand in your way. But it would
>> be nice to see where there's deficiency in tooz so we can be there for
>> the next project that needs it and maybe eventually factor out direct
>> etcd3 usage so users who have maybe chosen ZK as their tooz backend can
>> also benefit from your work.
> 
> It's not a deficiency in tooz. It's a different problem domain. Look at
> the os-lively API and show me how you think I could have used tooz to
> implement that API.

I think I already said this above - but what I was reading/hearing is
"why not add add the feature you need to tooz" ... not "tooz does that
already" - however, as you said, you were doing quick weekend POC work,
so it's possible that adding this to tooz is a next step.