Open Stack

On 03/15/2017 02:26 PM, Davanum Srinivas wrote:
> On Wed, Mar 15, 2017 at 9:19 AM, Monty Taylor <mordred at inaugust.com> wrote:
>> On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
>>> Monty, Team,
>>>
>>> Sorry for the top post:
>>>
>>> Support for etcd/tooz in devstack (with file driver as default) -
>>> https://review.openstack.org/#/c/445432/
>>>
>>> As of right now both zookeeper driver and etcd driver is working fine:
>>> https://review.openstack.org/#/c/445630/
>>> https://review.openstack.org/#/c/445629/
>>>
>>> The problem we have from before is that we do not have any CI jobs
>>> that used zookeeper.
>>>
>>> I am leaning towards just throwing the etcd as default and if folks
>>> are interested in zookeeper then they can add specific CI jobs with
>>> DLM_BACKEND variable set.
>>
>> That doesn't bother me - zk as the default choice was because at the
>> time zk worked and etcd did not.
>>
>> That said - etcd3 is a newer/better thing - so maybe instead of driving
>> etcd home as a default before we add etcd3 support, we just change tooz
>> to support etcd3, add the devstack jobs to use that, and start from a
>> position that doesn't involve dealing with any legacy?
> 
> Yep, jd__ and i confirmed that things work with 3.x

WOOT!

> Thanks,
> Dims
> 
>>
>>> On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor <mordred at inaugust.com> wrote:
>>>> On 03/15/2017 03:13 AM, Jay Pipes wrote:
>>>>> On 03/14/2017 05:01 PM, Clint Byrum wrote:
>>>>>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>>>>>>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
>>>>>>>> On Tue, Mar 14 2017, Jay Pipes wrote:
>>>>>>>>
>>>>>>>>> Not tooz, because I'm not interested in a DLM nor leader election
>>>>>>>>> library
>>>>>>>>> (that's what the underlying etcd3 cluster handles for me), only a
>>>>>>>>> fast service
>>>>>>>>> liveness/healthcheck system, but it shows usage of etcd3 and Google
>>>>>>>>> Protocol
>>>>>>>>> Buffers implementing a simple API for liveness checking and host
>>>>>>>>> maintenance
>>>>>>>>> reporting.
>>>>>>>>
>>>>>>>> Cool cool. So that's the same feature that we implemented in tooz 3
>>>>>>>> years ago. It's called "group membership". You create a group, make
>>>>>>>> nodes join it, and you know who's dead/alive and get notified when
>>>>>>>> their
>>>>>>>> status change.
>>>>>>>
>>>>>>> The point of os-lively is not to provide a thin API over ZooKeeper's
>>>>>>> group membership interface. The point of os-lively is to remove the need
>>>>>>> to have a database (RDBMS) record of a service in Nova.
>>>>>>
>>>>>> That's also the point of tooz's group membership API:
>>>>>>
>>>>>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping
>>>>>
>>>>> Did you take a look at the code I wrote in os-lively? What part of the
>>>>> tooz group membership API do you think I would have used?
>>>>>
>>>>> Again, this was a weekend project that I was moving fast on. I looked at
>>>>> tooz and didn't see how I could use it for my purposes, which was to
>>>>> store a versioned object in a consistent key/value store with support
>>>>> for transactional semantics when storing index and data records at the
>>>>> same time [1]
>>>>>
>>>>> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
>>>>>
>>>>>
>>>>> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
>>>>> semantics in a consistent key/value store that I needed.
>>>>>
>>>>> tooz is cool, but it's not what I was looking for. It's solving a
>>>>> different problem than I was trying to solve.
>>>>>
>>>>> This isn't a case of NIH, despite what Julien is trying to intimate in
>>>>> his emails.
>>>>>
>>>>>>> tooz simply abstracts a group membership API across a number of drivers.
>>>>>>> I don't need that. I need a way to maintain a service record (with
>>>>>>> maintenance period information, region, and an evolvable data record
>>>>>>> format) and query those service records in an RDBMS-like manner but
>>>>>>> without the RDBMS being involved.
>>>>>>>
>>>>>>>>> servicegroup API with os-lively and eliminate Nova's use of an
>>>>>>>>> RDBMS for
>>>>>>>>> service liveness checking, which should dramatically reduce the
>>>>>>>>> amount of both
>>>>>>>>> DB traffic as well as conductor/MQ service update traffic.
>>>>>>>>
>>>>>>>> Interesting. Joshua and Vilob tried to push usage of tooz group
>>>>>>>> membership a couple of years ago, but it got nowhere. Well, no, they
>>>>>>>> got
>>>>>>>> 2 specs written IIRC:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>>>>>>>>
>>>>>>>>
>>>>>>>> But then it died for whatever reasons on Nova side.
>>>>>>>
>>>>>>> It died because it didn't actually solve a problem.
>>>>>>>
>>>>>>> The problem is that even if we incorporate tooz, we would still need to
>>>>>>> have a service table in the RDBMS and continue to query it over and over
>>>>>>> again in the scheduler and API nodes.
>>>>>>
>>>>>> Most likely it was designed with hesitance to have a tooz requirement
>>>>>> to be a source of truth. But it's certainly not a problem for most tooz
>>>>>> backends to be a source of truth. Certainly not for etcd or ZK, which
>>>>>> are both designed to be that.
>>>>>>
>>>>>>> I want all service information in the same place, and I don't want to
>>>>>>> use an RDBMS for that information. etcd3 provides an ideal place to
>>>>>>> store service record information. Google Protocol Buffers is an ideal
>>>>>>> data format for evolvable versioned objects. os-lively presents an API
>>>>>>> that solves the problem I want to solve in Nova. tooz didn't.
>>>>>>
>>>>>> Was there something inherent in tooz's design that prevented you from
>>>>>> adding it to tooz's group API? Said API already includes liveness (watch
>>>>>> the group that corresponds to the service you want).
>>>>>
>>>>> See above about transactional semantics.
>>>>>
>>>>> I'm actually happy to add an etcd3 group membership driver to tooz,
>>>>> though. After the experience gained this weekend using etcd3, I'd like
>>>>> to do that.
>>>>>
>>>>> Still doesn't mean that tooz would be the appropriate choice for what I
>>>>> was trying to do with os-lively, though.
>>>>>
>>>>>> The only thing missing is being able to get groups and group members
>>>>>> by secondary indexes. etcd3's built in indexes by field are pretty nice
>>>>>
>>>>> Not sure what you're talking about. etcd3 doesn't have any indexing by
>>>>> field. I built the os-lively library primarily as a well-defined set of
>>>>> index overlays (by uuid, by host, by service type, and by region) over
>>>>> etcd3's key/value store.
>>>>>
>>>>>> for that, but ZK can likely also do it too by maintaining the index in
>>>>>> the driver.
>>>>>
>>>>> Maybe, I'm not sure, I didn't spend much time this weekend looking at
>>>>> ZooKeeper.
>>>>
>>>> a) awesome. when the rest of this dips momentarily into words that might
>>>> sound negative, please hear it all wrapped in an "awesome" and know that
>>>> my personal desire is to see the thing you're working on be successful
>>>> without undue burden...
>>>>
>>>> b) In Tokyo, we had the big discussion about DLMs (where at least my
>>>> intent going in to the room was to get us to pick one and only one).
>>>> There were three camps in the room who were all vocal:
>>>>
>>>> 1) YES! Let's just pick one, I don't care which one
>>>> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
>>>> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
>>>> pick that
>>>>
>>>> Because of 2 and 3 the group represented by 1 lost and we ended up with:
>>>> "crap, we have to use an abstraction library"
>>>>
>>>> I'd argue that unless something has changed significantly, having Nova
>>>> grow a direct depend on etcd when the DLM discussion brought us to "the
>>>> operators in the room have expressed a need for a pluggable choice
>>>> between at least zk and etcd" should be pretty much a non-starter.
>>>>
>>>> Now, being that I was personally in group 1, I'd be THRILLED if we
>>>> could, as a community, decide to pick one and skip having an abstraction
>>>> library. I still don't care which one - and you know I love gRPC/protobuf.
>>>>
>>>> But I do think that given the anti-etcd sentiment that was expressed was
>>>> equally as vehement as the anti-zk sentiment, that we need to circle
>>>> back and make a legit call on this topic.
>>>>
>>>> If we can pick one, I think having special-purpose libraries like
>>>> os-lively for specific purposes would be neat.
>>>>
>>>> If we still can't pick one, then I think adding the liveness check you
>>>> implemented for os-lively as a new feature in tooz and also implementing
>>>> the same thing in the zk driver would be necessary. (of course, that'll
>>>> probably depend on getting etcd3 support added to tooz and making sure
>>>> there is a good functional test for etcd3.
>>>>
>>>> c) awesome
>>>>
>>>>>> I understand abstractions can seem pretty cumbersome when you're moving
>>>>>> fast. It's not something I want to see stand in your way. But it would
>>>>>> be nice to see where there's deficiency in tooz so we can be there for
>>>>>> the next project that needs it and maybe eventually factor out direct
>>>>>> etcd3 usage so users who have maybe chosen ZK as their tooz backend can
>>>>>> also benefit from your work.
>>>>>
>>>>> It's not a deficiency in tooz. It's a different problem domain. Look at
>>>>> the os-lively API and show me how you think I could have used tooz to
>>>>> implement that API.
>>>>
>>>> I think I already said this above - but what I was reading/hearing is
>>>> "why not add add the feature you need to tooz" ... not "tooz does that
>>>> already" - however, as you said, you were doing quick weekend POC work,
>>>> so it's possible that adding this to tooz is a next step.
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170315/1ab83a3e/attachment.pgp>