<div dir="ltr">Hi Davanum<div><br></div><div>The two primary fixes to oslo.messaging outstanding to get things running are</div><div><br></div><div><a href="https://review.openstack.org/#/c/84938/" target="_blank">https://review.openstack.org/#/c/84938/</a><br></div><div><a href="https://review.openstack.org/#/c/120745/" target="_blank">https://review.openstack.org/#/c/120745/</a></div><div><br></div><div>cheers,</div><div><br></div><div>Kapil</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 18, 2014 at 11:32 AM, Davanum Srinivas <span dir="ltr"><<a href="mailto:davanum@gmail.com" target="_blank">davanum@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Kapil,<br>
<br>
I see just 2 relevant reviews for zmq in the oslo.messaging queue:<br>
<a href="https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z" target="_blank">https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z</a><br>
<br>
Are there others i am missing? ("fix critical items", "tests" from your email)<br>
<br>
thanks,<br>
dims<br>
<div><div class="h5"><br>
On Thu, Sep 18, 2014 at 10:16 AM, Kapil Thangavelu<br>
<<a href="mailto:kapil.thangavelu@canonical.com">kapil.thangavelu@canonical.com</a>> wrote:<br>
><br>
><br>
> On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco <<a href="mailto:flavio@redhat.com">flavio@redhat.com</a>> wrote:<br>
>><br>
>> On 09/17/2014 04:34 PM, Doug Hellmann wrote:<br>
>> > This thread [1] has turned more “future focused", so I’m moving the<br>
>> > conversation to the -dev list where we usually have those sorts of<br>
>> > discussions.<br>
>> ><br>
>> > [1]<br>
>> > <a href="http://lists.openstack.org/pipermail/openstack/2014-September/009253.html" target="_blank">http://lists.openstack.org/pipermail/openstack/2014-September/009253.html</a><br>
>> ><br>
>><br>
>><br>
>> I saw this mentioned in the `openstack` thread but I'd like us to<br>
>> reconsider it.<br>
>><br>
>> Since we've gone through the "hey please don't deprecate it, I'll help"<br>
>> process a couple of times with the zmq driver, I'm thinking we should<br>
>> pull it out of the code base anyway.<br>
><br>
><br>
> I think the primary issue has been two fold, one is the lack of<br>
> reviews/merges on extant patches that fix critical items that have been<br>
> outstanding for months. I think that is compounded by the other issue which<br>
> was the lack of tests. We've sprinted last week on adding in both unit<br>
> tests, the extant patches and functionally verifying them by automating<br>
> cloud deployments with zmq for the messaging driver. We're also committing<br>
> as a company to supporting it on an ongoing basis. If we get the gates going<br>
> for kilo, i don't see any reason for the churn below, if the gates don't get<br>
> going we can yank to external in kilo anyway.<br>
><br>
> cheers,<br>
><br>
> Kapil<br>
><br>
>><br>
>> Here's a rough plan of what I think we should do until the zmq is<br>
>> updated and has a proper gate working:<br>
>><br>
>> 1. Pull it out the code base into its own repo (stackforge?)<br>
>> 2. Setup the basic `oslo.messaging` gates for it<br>
>> 3. Release the driver on pypi as: `oslo.messaging.zmq`<br>
>> 4. Make lots of noise and make sure people using zmq knows that they<br>
>> have to install a separate package now. (I know this means creating a<br>
>> new package for many distros).<br>
>><br>
>> If there are folks interested in maintaining this driver, they can still<br>
>> do it with the driver outside the code base. If it ever gets enough<br>
>> attention and maturity, it could be re-proposed for inclusion into the<br>
>> code base.<br>
>><br>
>> I know there are folks interested in a broker-less solution and we now<br>
>> have the not-tested-in-production-yet amqp1.0 driver which I think is a<br>
>> very good solution and also shares most of the semantics we rely on<br>
>> protocol-wise.<br>
>><br>
>> I didn't go through the whole thread so, I'm sorry if I'm repeating<br>
>> something that has already been said/proposed/discussed.<br>
>><br>
>> Flavio<br>
>><br>
>> > On Sep 17, 2014, at 7:54 AM, James Page <<a href="mailto:james.page@ubuntu.com">james.page@ubuntu.com</a>> wrote:<br>
>> ><br>
>> >> Signed PGP part<br>
>> >> Hi Li<br>
>> >><br>
>> >> On 17/09/14 11:58, Li Ma wrote:<br>
>> >>>> The scale potential is very appealing and is something I want to<br>
>> >>>> test - - hopefully in the next month or so.<br>
>> >>>><br>
>> >>>> Canonical are interested in helping to maintain this driver and<br>
>> >>>> hopefully we help any critical issues prior to Juno release.<br>
>> >>>><br>
>> >>>><br>
>> >>> That sounds good. I just went through all the bugs reported in the<br>
>> >>> community.<br>
>> >>><br>
>> >>> The only critical bug which makes ZeroMQ malfunction is<br>
>> >>> <a href="https://bugs.launchpad.net/oslo.messaging/+bug/1301723" target="_blank">https://bugs.launchpad.net/oslo.messaging/+bug/1301723</a> and the<br>
>> >>> corresponding review is under way:<br>
>> >>> <a href="https://review.openstack.org/#/c/84938/" target="_blank">https://review.openstack.org/#/c/84938/</a><br>
>> >><br>
>> >> Agreed<br>
>> >><br>
>> >>> Others are tagged to 'zmq' in<br>
>> >>> <a href="https://bugs.launchpad.net/oslo.messaging" target="_blank">https://bugs.launchpad.net/oslo.messaging</a><br>
>> >><br>
>> >> Looking through Doug's suggested list of information and collating<br>
>> >> what I know of from our work last week:<br>
>> >><br>
>> >> 1) documentation for how to configure and use zeromq with<br>
>> >> oslo.messaging (note, not the version in oslo-incubator, the version<br>
>> >> in the messaging library repository)<br>
>> >><br>
>> >> As part of our sprint, I worked on automating deployment of OpenStack<br>
>> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig<br>
>> >> that work into some general documentation on how best to configure<br>
>> >> ZeroMQ with OpenStack - the current documentation is a bit raw and<br>
>> >> does not talk about how to configure the oslo-messaging-zmq-receiver<br>
>> >> at all.<br>
>> >><br>
>> >> I also plan some packaging updates for Debian/Ubuntu in our next dev<br>
>> >> cycle to make this a little easier to configure and digest - for<br>
>> >> example, right now no systemd unit/upstart configuration/sysv init<br>
>> >> script is provided to manage the zmq-receiver.<br>
>> >><br>
>> >> I'd also like to document the current design of the ZMQ driver - Doug<br>
>> >> - where is the best place todo this? I thought in the source tree<br>
>> >> somewhere.<br>
>> ><br>
>> > The documentation in the oslo.messaging repository [2] would be a good<br>
>> > place to start for that. If we decide deployers/operators need the<br>
>> > information we can either refer to it from the guides managed by the<br>
>> > documentation team or we can move/copy the information. How about if you<br>
>> > start a new drivers subdirectory there, and add information about zmq. We<br>
>> > can have other driver authors provide similar detail about their drivers in<br>
>> > the same directory.<br>
>> ><br>
>> > [2]<br>
>> > <a href="http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source" target="_blank">http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source</a><br>
>> ><br>
>> >><br>
>> >> 2) a list of the critical bugs that need to be fixed + any existing<br>
>> >> patches associated with those bugs, so they can be reviewed early in<br>
>> >> kilo<br>
>> >><br>
>> >> This blocks operation of nova+neutron environements:<br>
>> >><br>
>> >> <a href="https://bugs.launchpad.net/oslo.messaging/+bug/1301723" target="_blank">https://bugs.launchpad.net/oslo.messaging/+bug/1301723</a><br>
>> >> Summary: Message was sent to wrong node with zmq as rpc_backend<br>
>> >> Patch: <a href="https://review.openstack.org/84938" target="_blank">https://review.openstack.org/84938</a><br>
>> >><br>
>> >> Also notifcations are effectively unimplemented which prevents use<br>
>> >> with Ceilometer so I'd also add:<br>
>> >><br>
>> >> <a href="https://bugs.launchpad.net/oslo.messaging/+bug/1368154" target="_blank">https://bugs.launchpad.net/oslo.messaging/+bug/1368154</a><br>
>> >> Summary: <a href="https://bugs.launchpad.net/oslo.messaging/+bug/" target="_blank">https://bugs.launchpad.net/oslo.messaging/+bug/</a><br>
>> >> Patch: <a href="https://review.openstack.org/120745" target="_blank">https://review.openstack.org/120745</a><br>
>> ><br>
>> > That’s a good list, and shorter than I expected. I have added these bugs<br>
>> > to the next-kilo milestone.<br>
>> ><br>
>> >><br>
>> >> 3) an analysis of what it would take to be able to run functional<br>
>> >> tests for zeromq on our CI infrastructure, not necessarily the full<br>
>> >> tempest run or devstack-gate job, probably functional tests we place<br>
>> >> in the tree with the driver (we will be doing this for all of the<br>
>> >> drivers) + besides writing new functional tests, we need to bring the<br>
>> >> unit tests for zeromq into the oslo.messaging repository<br>
>> >><br>
>> >> Kapil Thangavelu started work on both functional tests for the ZMQ<br>
>> >> driver last week; the output from the sprint is here:<br>
>> >><br>
>> >> <a href="https://github.com/ostack-musketeers/oslo.messaging" target="_blank">https://github.com/ostack-musketeers/oslo.messaging</a><br>
>> >><br>
>> >> it covers the ZMQ driver (including messaging through the zmq-receiver<br>
>> >> proxy) and the associated MatchMakers (local, ring, redis) at a<br>
>> >> varying levels of coverage, but I feel it moves things in the right<br>
>> >> direction - Kapil's going to raise a review for this in the next<br>
>> >> couple of days.<br>
>> >><br>
>> >> Doug - has any structure been agreed within the oslo.messaging tree<br>
>> >> for unit/functional test splits? Right now we have them all in one<br>
>> >> place.<br>
>> ><br>
>> > I think we will want them split up, but we don’t have an agreed existing<br>
>> > structure for that. I would like to see a test framework of some sort that<br>
>> > defines the tests in a way that can be used to run the same functional for<br>
>> > all of the drivers as separate jobs (with appropriate hooks for ensuring the<br>
>> > needed services are running, etc.). Setting that up warrants its own spec,<br>
>> > because there are going to be quite a few details to work out. We will also<br>
>> > need to participate in the larger conversation about how to set up those<br>
>> > functional test jobs to be consistent with the other projects.<br>
>> ><br>
>> >><br>
>> >> Edward Hope-Morley also worked on getting devstack working with ZMQ:<br>
>> >><br>
>> >> <a href="https://github.com/ostack-musketeers/devstack" target="_blank">https://github.com/ostack-musketeers/devstack</a><br>
>> >><br>
>> >> that's still WIP but again we'll get any changes submitted for review<br>
>> >> ASAP.<br>
>> ><br>
>> > That’s good to have, but I don’t necessarily consider it a requirement<br>
>> > for in-project functional tests.<br>
>> ><br>
>> >><br>
>> >> 4) and some improvements that we would like to make longer term<br>
>> >><br>
>> >> a) Connection re-use on outbound messaging avoiding the current tcp<br>
>> >> setup overhead for every sent message. This may also bring further<br>
>> >> performance benefits due to underlying messaging batching in ZMQ.<br>
>> ><br>
>> > This sounds like it would be a good thing to do, but making what we have<br>
>> > work correctly and testing it feels more important for now.<br>
>> ><br>
>> >><br>
>> >> b) Moving from tcp PUSH/PULL sockets between servers to DEALER/DEALER<br>
>> >> (or something similar) to allow for heartbeating and more immediate<br>
>> >> failure detection<br>
>> ><br>
>> > I would need to understand how much of a rewrite that represents before<br>
>> > commenting further.<br>
>> ><br>
>> >><br>
>> >> c) Crypto support<br>
>> ><br>
>> > There are some other discussions about adding crypto to messaging, and I<br>
>> > hope we can do that without having to touch each driver, if possible.<br>
>> ><br>
>> >><br>
>> >> Cheers<br>
>> >><br>
>> >> James<br>
>> >><br>
>> >> --<br>
>> >> James Page<br>
>> >> Ubuntu and Debian Developer<br>
>> >> <a href="mailto:james.page@ubuntu.com">james.page@ubuntu.com</a><br>
>> >> <a href="mailto:jamespage@debian.org">jamespage@debian.org</a><br>
>> >><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > OpenStack-dev mailing list<br>
>> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>><br>
>><br>
>> --<br>
>> @flaper87<br>
>> Flavio Percoco<br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
<br>
--<br>
</div></div>Davanum Srinivas :: <a href="https://twitter.com/dims" target="_blank">https://twitter.com/dims</a><br>
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>