Open Stack

Hope this thread isn't dead.

Mike - thanks for highlighting some really key issues at scale.

On a related note, can someone from the Ceilometer comment about the store and forward requirement? Currently scaling RabbitMQ is non-trivial. Though cells help make the problem smaller, as Paul Mathews points out in the video below, cells don't make the problems go away. Looking at the experience in the community, Qpid isn't an option either.

Cheers,
Subbu


On Dec 9, 2013, at 4:36 PM, Mike Wilson <geekinutah at gmail.com> wrote:

> This is the first time I've heard of the dispatch router, I'm really excited now that I've looked at it a bit. Thx Gordon and Russell for bringing this up. I'm very familiar with the scaling issues associated with any kind of brokered messaging solution. We grew an Openstack installation to about 7,000 nodes and started having significant scaling issues with the qpid broker. We've talked about our problems at a couple summits in a fair amount of detail[1][2]. I won't bother repeating the information in this thread.
> 
> I really like the idea of separating the logic of routing away from the the message emitter. Russell mentioned the 0mq matchmaker, we essentially ditched the qpid broker for direct communication via 0mq and it's matchmaker. It still has a lot of problems which dispatch seems to address. For example, in ceilometer we have store-and-forward behavior as a requirement. This kind of communication requires a broker but 0mq doesn't really officially support one, which means we would probably end up with some broker as part of OpenStack. Matchmaker is also a fairly basic implementation of what is essentially a directory. For any sort of serious production use case you end up sprinkling JSON files all over the place or maintaining a Redis backend. I feel like the matchmaker needs a bunch more work to make modifying the directory simpler for operations. I would rather put that work into a separate project like dispatch than have to maintain essentially a one off in Openstack's codebase.
> 
> I wonder how this fits into messaging from a driver perspective in Openstack or even how this fits into oslo.messaging? Right now we have topics for binaries(compute, network, consoleauth, etc), hostname.service_topic for nodes, fanout queue per node (not sure if kombu also has this) and different exchanges per project. If we can abstract the routing from the emission of the message all we really care about is emitter, endpoint, messaging pattern (fanout, store and forward, etc). Also not sure if there's a dispatch analogue in the rabbit world, if not we need to have some mapping of concepts etc between impls.
> 
> So many questions, but in general I'm really excited about this and eager to contribute. For sure I will start playing with this in Bluehost's environments that haven't been completely 0mqized. I also have some lingering concerns about qpid in general. Beyond scaling issues I've run into some other terrible bugs that motivated our move away from it. Again, these are mentioned in our presentations at summits and I'd be happy to talk more about them in a separate discussion. I've also been able to talk to some other qpid+openstack users who have seen the same bugs. Another large installation that comes to mind is Qihoo 360 in China. They run a few thousand nodes with qpid for messaging and are familiar with the snags we run into.
> 
> Gordon,
> 
> I would really appreciate if you could watch those two talks and comment. The bugs are probably separate from the dispatch router discussion, but it does dampen my enthusiasm a bit not knowing how to fix issues beyond scale :-(. 
> 
> -Mike Wilson
> 
> [1] http://www.openstack.org/summit/portland-2013/session-videos/presentation/using-openstack-in-a-traditional-hosting-environment
> [2] http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/going-brokerless-the-transition-from-qpid-to-0mq
> 
> 
> 
> 
> On Mon, Dec 9, 2013 at 4:29 PM, Mark McLoughlin <markmc at redhat.com> wrote:
> On Mon, 2013-12-09 at 16:05 +0100, Flavio Percoco wrote:
> > Greetings,
> >
> > As $subject mentions, I'd like to start discussing the support for
> > AMQP 1.0[0] in oslo.messaging. We already have rabbit and qpid drivers
> > for earlier (and different!) versions of AMQP, the proposal would be
> > to add an additional driver for a _protocol_ not a particular broker.
> > (Both RabbitMQ and Qpid support AMQP 1.0 now).
> >
> > By targeting a clear mapping on to a protocol, rather than a specific
> > implementation, we would simplify the task in the future for anyone
> > wishing to move to any other system that spoke AMQP 1.0. That would no
> > longer require a new driver, merely different configuration and
> > deployment. That would then allow openstack to more easily take
> > advantage of any emerging innovations in this space.
> 
> Sounds sane to me.
> 
> To put it another way, assuming all AMQP 1.0 client libraries are equal,
> all the operator cares about is that we have a driver that connect into
> whatever AMQP 1.0 messaging topology they want to use.
> 
> Of course, not all client libraries will be equal, so if we don't offer
> the choice of library/driver to the operator, then the onus is on us to
> pick the best client library for this driver.
> 
> (Enjoying the rest of this thread too, thanks to Gordon for his
> insights)
> 
> Mark.
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev