Open Stack

Monty, Team,

Sorry for the top post:

Support for etcd/tooz in devstack (with file driver as default) -
https://review.openstack.org/#/c/445432/

As of right now both zookeeper driver and etcd driver is working fine:
https://review.openstack.org/#/c/445630/
https://review.openstack.org/#/c/445629/

The problem we have from before is that we do not have any CI jobs
that used zookeeper.

I am leaning towards just throwing the etcd as default and if folks
are interested in zookeeper then they can add specific CI jobs with
DLM_BACKEND variable set.

Thanks,
Dims

On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor <mordred at inaugust.com> wrote:
> On 03/15/2017 03:13 AM, Jay Pipes wrote:
>> On 03/14/2017 05:01 PM, Clint Byrum wrote:
>>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>>>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
>>>>> On Tue, Mar 14 2017, Jay Pipes wrote:
>>>>>
>>>>>> Not tooz, because I'm not interested in a DLM nor leader election
>>>>>> library
>>>>>> (that's what the underlying etcd3 cluster handles for me), only a
>>>>>> fast service
>>>>>> liveness/healthcheck system, but it shows usage of etcd3 and Google
>>>>>> Protocol
>>>>>> Buffers implementing a simple API for liveness checking and host
>>>>>> maintenance
>>>>>> reporting.
>>>>>
>>>>> Cool cool. So that's the same feature that we implemented in tooz 3
>>>>> years ago. It's called "group membership". You create a group, make
>>>>> nodes join it, and you know who's dead/alive and get notified when
>>>>> their
>>>>> status change.
>>>>
>>>> The point of os-lively is not to provide a thin API over ZooKeeper's
>>>> group membership interface. The point of os-lively is to remove the need
>>>> to have a database (RDBMS) record of a service in Nova.
>>>
>>> That's also the point of tooz's group membership API:
>>>
>>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping
>>
>> Did you take a look at the code I wrote in os-lively? What part of the
>> tooz group membership API do you think I would have used?
>>
>> Again, this was a weekend project that I was moving fast on. I looked at
>> tooz and didn't see how I could use it for my purposes, which was to
>> store a versioned object in a consistent key/value store with support
>> for transactional semantics when storing index and data records at the
>> same time [1]
>>
>> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
>>
>>
>> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
>> semantics in a consistent key/value store that I needed.
>>
>> tooz is cool, but it's not what I was looking for. It's solving a
>> different problem than I was trying to solve.
>>
>> This isn't a case of NIH, despite what Julien is trying to intimate in
>> his emails.
>>
>>>> tooz simply abstracts a group membership API across a number of drivers.
>>>> I don't need that. I need a way to maintain a service record (with
>>>> maintenance period information, region, and an evolvable data record
>>>> format) and query those service records in an RDBMS-like manner but
>>>> without the RDBMS being involved.
>>>>
>>>>>> servicegroup API with os-lively and eliminate Nova's use of an
>>>>>> RDBMS for
>>>>>> service liveness checking, which should dramatically reduce the
>>>>>> amount of both
>>>>>> DB traffic as well as conductor/MQ service update traffic.
>>>>>
>>>>> Interesting. Joshua and Vilob tried to push usage of tooz group
>>>>> membership a couple of years ago, but it got nowhere. Well, no, they
>>>>> got
>>>>> 2 specs written IIRC:
>>>>>
>>>>>
>>>>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>>>>>
>>>>>
>>>>> But then it died for whatever reasons on Nova side.
>>>>
>>>> It died because it didn't actually solve a problem.
>>>>
>>>> The problem is that even if we incorporate tooz, we would still need to
>>>> have a service table in the RDBMS and continue to query it over and over
>>>> again in the scheduler and API nodes.
>>>
>>> Most likely it was designed with hesitance to have a tooz requirement
>>> to be a source of truth. But it's certainly not a problem for most tooz
>>> backends to be a source of truth. Certainly not for etcd or ZK, which
>>> are both designed to be that.
>>>
>>>> I want all service information in the same place, and I don't want to
>>>> use an RDBMS for that information. etcd3 provides an ideal place to
>>>> store service record information. Google Protocol Buffers is an ideal
>>>> data format for evolvable versioned objects. os-lively presents an API
>>>> that solves the problem I want to solve in Nova. tooz didn't.
>>>
>>> Was there something inherent in tooz's design that prevented you from
>>> adding it to tooz's group API? Said API already includes liveness (watch
>>> the group that corresponds to the service you want).
>>
>> See above about transactional semantics.
>>
>> I'm actually happy to add an etcd3 group membership driver to tooz,
>> though. After the experience gained this weekend using etcd3, I'd like
>> to do that.
>>
>> Still doesn't mean that tooz would be the appropriate choice for what I
>> was trying to do with os-lively, though.
>>
>>> The only thing missing is being able to get groups and group members
>>> by secondary indexes. etcd3's built in indexes by field are pretty nice
>>
>> Not sure what you're talking about. etcd3 doesn't have any indexing by
>> field. I built the os-lively library primarily as a well-defined set of
>> index overlays (by uuid, by host, by service type, and by region) over
>> etcd3's key/value store.
>>
>>> for that, but ZK can likely also do it too by maintaining the index in
>>> the driver.
>>
>> Maybe, I'm not sure, I didn't spend much time this weekend looking at
>> ZooKeeper.
>
> a) awesome. when the rest of this dips momentarily into words that might
> sound negative, please hear it all wrapped in an "awesome" and know that
> my personal desire is to see the thing you're working on be successful
> without undue burden...
>
> b) In Tokyo, we had the big discussion about DLMs (where at least my
> intent going in to the room was to get us to pick one and only one).
> There were three camps in the room who were all vocal:
>
> 1) YES! Let's just pick one, I don't care which one
> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
> pick that
>
> Because of 2 and 3 the group represented by 1 lost and we ended up with:
> "crap, we have to use an abstraction library"
>
> I'd argue that unless something has changed significantly, having Nova
> grow a direct depend on etcd when the DLM discussion brought us to "the
> operators in the room have expressed a need for a pluggable choice
> between at least zk and etcd" should be pretty much a non-starter.
>
> Now, being that I was personally in group 1, I'd be THRILLED if we
> could, as a community, decide to pick one and skip having an abstraction
> library. I still don't care which one - and you know I love gRPC/protobuf.
>
> But I do think that given the anti-etcd sentiment that was expressed was
> equally as vehement as the anti-zk sentiment, that we need to circle
> back and make a legit call on this topic.
>
> If we can pick one, I think having special-purpose libraries like
> os-lively for specific purposes would be neat.
>
> If we still can't pick one, then I think adding the liveness check you
> implemented for os-lively as a new feature in tooz and also implementing
> the same thing in the zk driver would be necessary. (of course, that'll
> probably depend on getting etcd3 support added to tooz and making sure
> there is a good functional test for etcd3.
>
> c) awesome
>
>>> I understand abstractions can seem pretty cumbersome when you're moving
>>> fast. It's not something I want to see stand in your way. But it would
>>> be nice to see where there's deficiency in tooz so we can be there for
>>> the next project that needs it and maybe eventually factor out direct
>>> etcd3 usage so users who have maybe chosen ZK as their tooz backend can
>>> also benefit from your work.
>>
>> It's not a deficiency in tooz. It's a different problem domain. Look at
>> the os-lively API and show me how you think I could have used tooz to
>> implement that API.
>
> I think I already said this above - but what I was reading/hearing is
> "why not add add the feature you need to tooz" ... not "tooz does that
> already" - however, as you said, you were doing quick weekend POC work,
> so it's possible that adding this to tooz is a next step.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims