[openstack-dev] [oslo] zeromq work for kilo

Kapil Thangavelu kapil.thangavelu at canonical.com
Thu Sep 18 20:26:13 UTC 2014


Hi Davanum

The two primary fixes to oslo.messaging outstanding to get things running
are

https://review.openstack.org/#/c/84938/
https://review.openstack.org/#/c/120745/

cheers,

Kapil


On Thu, Sep 18, 2014 at 11:32 AM, Davanum Srinivas <davanum at gmail.com>
wrote:

> Kapil,
>
> I see just 2 relevant reviews for zmq in the oslo.messaging queue:
>
> https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z
>
> Are there others i am missing? ("fix critical items", "tests" from your
> email)
>
> thanks,
> dims
>
> On Thu, Sep 18, 2014 at 10:16 AM, Kapil Thangavelu
> <kapil.thangavelu at canonical.com> wrote:
> >
> >
> > On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco <flavio at redhat.com>
> wrote:
> >>
> >> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> >> > This thread [1] has turned more “future focused", so I’m moving the
> >> > conversation to the -dev list where we usually have those sorts of
> >> > discussions.
> >> >
> >> > [1]
> >> >
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >> >
> >>
> >>
> >> I saw this mentioned in the `openstack` thread but I'd like us to
> >> reconsider it.
> >>
> >> Since we've gone through the "hey please don't deprecate it, I'll help"
> >> process a couple of times with the zmq driver, I'm thinking we should
> >> pull it out of the code base anyway.
> >
> >
> > I think the primary issue has been two fold, one is the lack of
> > reviews/merges on extant patches that fix critical items that have been
> > outstanding for months. I think that is compounded by the other issue
> which
> > was the lack of tests. We've sprinted last week on adding in both unit
> > tests, the extant patches and functionally verifying them by automating
> > cloud deployments with zmq for the messaging driver. We're also
> committing
> > as a company to supporting it on an ongoing basis. If we get the gates
> going
> > for kilo, i don't see any reason for the churn below, if the gates don't
> get
> > going we can yank to external in kilo anyway.
> >
> > cheers,
> >
> > Kapil
> >
> >>
> >> Here's a rough plan of what I think we should do until the zmq is
> >> updated and has a proper gate working:
> >>
> >> 1. Pull it out the code base into its own repo (stackforge?)
> >> 2. Setup the basic `oslo.messaging` gates for it
> >> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> >> 4. Make lots of noise and make sure people using zmq knows that they
> >> have to install a separate package now. (I know this means creating a
> >> new package for many distros).
> >>
> >> If there are folks interested in maintaining this driver, they can still
> >> do it with the driver outside the code base. If it ever gets enough
> >> attention and maturity, it could be re-proposed for inclusion into the
> >> code base.
> >>
> >> I know there are folks interested in a broker-less solution and we now
> >> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> >> very good solution and also shares most of the semantics we rely on
> >> protocol-wise.
> >>
> >> I didn't go through the whole thread so, I'm sorry if I'm repeating
> >> something that has already been said/proposed/discussed.
> >>
> >> Flavio
> >>
> >> > On Sep 17, 2014, at 7:54 AM, James Page <james.page at ubuntu.com>
> wrote:
> >> >
> >> >> Signed PGP part
> >> >> Hi Li
> >> >>
> >> >> On 17/09/14 11:58, Li Ma wrote:
> >> >>>> The scale potential is very appealing and is something I want to
> >> >>>> test - - hopefully in the next month or so.
> >> >>>>
> >> >>>> Canonical are interested in helping to maintain this driver and
> >> >>>> hopefully we help any critical issues prior to Juno release.
> >> >>>>
> >> >>>>
> >> >>> That sounds good. I just went through all the bugs reported in the
> >> >>> community.
> >> >>>
> >> >>> The only critical bug which makes ZeroMQ malfunction is
> >> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >> >>> corresponding review is under way:
> >> >>> https://review.openstack.org/#/c/84938/
> >> >>
> >> >> Agreed
> >> >>
> >> >>> Others are tagged to 'zmq' in
> >> >>> https://bugs.launchpad.net/oslo.messaging
> >> >>
> >> >> Looking through Doug's suggested list of information and collating
> >> >> what I know of from our work last week:
> >> >>
> >> >> 1) documentation for how to configure and use zeromq with
> >> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> >> in the messaging library repository)
> >> >>
> >> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> >> that work into some general documentation on how best to configure
> >> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> >> at all.
> >> >>
> >> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> >> cycle to make this a little easier to configure and digest - for
> >> >> example, right now no systemd unit/upstart configuration/sysv init
> >> >> script is provided to manage the zmq-receiver.
> >> >>
> >> >> I'd also like to document the current design of the ZMQ driver - Doug
> >> >> - where is the best place todo this? I thought in the source tree
> >> >> somewhere.
> >> >
> >> > The documentation in the oslo.messaging repository [2] would be a good
> >> > place to start for that. If we decide deployers/operators need the
> >> > information we can either refer to it from the guides managed by the
> >> > documentation team or we can move/copy the information. How about if
> you
> >> > start a new drivers subdirectory there, and add information about
> zmq. We
> >> > can have other driver authors provide similar detail about their
> drivers in
> >> > the same directory.
> >> >
> >> > [2]
> >> >
> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
> >> >
> >> >>
> >> >> 2) a list of the critical bugs that need to be fixed + any existing
> >> >> patches associated with those bugs, so they can be reviewed early in
> >> >> kilo
> >> >>
> >> >> This blocks operation of nova+neutron environements:
> >> >>
> >> >> https://bugs.launchpad.net/oslo.messaging/+bug/1301723
> >> >>      Summary: Message was sent to wrong node with zmq as rpc_backend
> >> >>      Patch: https://review.openstack.org/84938
> >> >>
> >> >> Also notifcations are effectively unimplemented which prevents use
> >> >> with Ceilometer so I'd also add:
> >> >>
> >> >> https://bugs.launchpad.net/oslo.messaging/+bug/1368154
> >> >>      Summary: https://bugs.launchpad.net/oslo.messaging/+bug/
> >> >>      Patch: https://review.openstack.org/120745
> >> >
> >> > That’s a good list, and shorter than I expected. I have added these
> bugs
> >> > to the next-kilo milestone.
> >> >
> >> >>
> >> >> 3) an analysis of what it would take to be able to run functional
> >> >> tests for zeromq on our CI infrastructure, not necessarily the full
> >> >> tempest run or devstack-gate job, probably functional tests we place
> >> >> in the tree with the driver (we will be doing this for all of the
> >> >> drivers) + besides writing new functional tests, we need to bring the
> >> >> unit tests for zeromq into the oslo.messaging repository
> >> >>
> >> >> Kapil Thangavelu started work on both functional tests for the ZMQ
> >> >> driver last week; the output from the sprint is here:
> >> >>
> >> >>    https://github.com/ostack-musketeers/oslo.messaging
> >> >>
> >> >> it covers the ZMQ driver (including messaging through the
> zmq-receiver
> >> >> proxy) and the associated MatchMakers (local, ring, redis) at a
> >> >> varying levels of coverage, but I feel it moves things in the right
> >> >> direction - Kapil's going to raise a review for this in the next
> >> >> couple of days.
> >> >>
> >> >> Doug - has any structure been agreed within the oslo.messaging tree
> >> >> for unit/functional test splits? Right now we have them all in one
> >> >> place.
> >> >
> >> > I think we will want them split up, but we don’t have an agreed
> existing
> >> > structure for that. I would like to see a test framework of some sort
> that
> >> > defines the tests in a way that can be used to run the same
> functional for
> >> > all of the drivers as separate jobs (with appropriate hooks for
> ensuring the
> >> > needed services are running, etc.). Setting that up warrants its own
> spec,
> >> > because there are going to be quite a few details to work out. We
> will also
> >> > need to participate in the larger conversation about how to set up
> those
> >> > functional test jobs to be consistent with the other projects.
> >> >
> >> >>
> >> >> Edward Hope-Morley also worked on getting devstack working with ZMQ:
> >> >>
> >> >>    https://github.com/ostack-musketeers/devstack
> >> >>
> >> >> that's still WIP but again we'll get any changes submitted for review
> >> >> ASAP.
> >> >
> >> > That’s good to have, but I don’t necessarily consider it a requirement
> >> > for in-project functional tests.
> >> >
> >> >>
> >> >> 4) and some improvements that we would like to make longer term
> >> >>
> >> >> a) Connection re-use on outbound messaging avoiding the current tcp
> >> >> setup overhead for every sent message.  This may also bring further
> >> >> performance benefits due to underlying messaging batching in ZMQ.
> >> >
> >> > This sounds like it would be a good thing to do, but making what we
> have
> >> > work correctly and testing it feels more important for now.
> >> >
> >> >>
> >> >> b) Moving from tcp PUSH/PULL sockets between servers to DEALER/DEALER
> >> >> (or something similar) to allow for heartbeating and more immediate
> >> >> failure detection
> >> >
> >> > I would need to understand how much of a rewrite that represents
> before
> >> > commenting further.
> >> >
> >> >>
> >> >> c) Crypto support
> >> >
> >> > There are some other discussions about adding crypto to messaging,
> and I
> >> > hope we can do that without having to touch each driver, if possible.
> >> >
> >> >>
> >> >> Cheers
> >> >>
> >> >> James
> >> >>
> >> >> --
> >> >> James Page
> >> >> Ubuntu and Debian Developer
> >> >> james.page at ubuntu.com
> >> >> jamespage at debian.org
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > OpenStack-dev mailing list
> >> > OpenStack-dev at lists.openstack.org
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >>
> >> --
> >> @flaper87
> >> Flavio Percoco
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140918/92d57ec9/attachment.html>


More information about the OpenStack-dev mailing list