Open Stack

On 12/09/2013 04:10 PM, Russell Bryant wrote:
> From looking it appears that RabbitMQ's support is via an experimental
> plugin.  I don't know any more about it.  Has anyone looked at it in
> detail?

I believe initial support was added in 3.1.0: 
http://www.rabbitmq.com/release-notes/README-3.1.0.txt

I have certainly successfully tested basic interaction with RabbitMQ 
over 1.0.

>      https://www.rabbitmq.com/specification.html
>
> As I understand it, "Qpid" supports it, in that it's a completely new
> implementation as a library (Proton) under the Qpid project umbrella.
>   There's also a message router called Dispatch.  This is not to be
> confused with the existing Qpid libraries, or the existing Qpid broker
> (qpidd).

Yes, proton is a library that encapsulates the AMQP 1.0 encoding and 
protocol rules. It is used in the existing native broker (i.e. qpidd) to 
offer 1.0 support (as well as the qpid::messaging c++ client library).

In addition there is as you mention the 'Dispatch Router'. This is an 
alternative architecture for an intermediary to address some of the 
issues that can arise with qpidd or other brokers (distributed in nature 
to scale better, end-to-end reliability rather than store and forward etc).

So the Qpid project offers both new components as well as 1.0 support 
and smooth transition for existing components. (Disclosure, I'm a 
developer in the Qpid community also).

(There are of course other implementations also e.g., ActiveMQ, 
ApolloMQ, HornetQ, Microsoft ServiceBus, SwiftMQ.)

>      http://qpid.apache.org/proton/
>      http://qpid.apache.org/components/dispatch-router/
>

[...]

> All of this sounds fine to me.  Surely a single driver for multiple
> systems is an improvement.  What's not really mentioned though is why
> we should care about AMQP 1.0 beyond that.  Why is it architecturally
> superior?  It has been discussed on this list some before, but I
> figure it's worth re-visiting if some code is going to show up soon.

Personally I think there is benefit to having a standardised, open 
wire-protocol as the basis for communication in systems like OpenStack, 
rather than having the driver tied to a particular implementation 
throughout (and having more of the key details of the interaction as 
details of the implementation of the driver). The bytes over the wire 
are another level of interface and having that tightly specified can be 
valuable.

Having one driver that still offers choice with regard to intermediaries 
used (I avoid the term broker in case it is implies particular 
approaches), is I think an advantage. Hypothetically for example it 
would have been an advantage had the same driver been usable against 
both RabbitMQ and Qpid previously. The (bumpy!) evolution  of AMQP meant 
that wasn't quite possible since they both spoke different versions 
ofthe early protocol. AMQP 1.0 might in the future avoid needing new 
drivers in such cases however, making it easier to adopt alternative or 
emerging solutions.

AMQP is not the only messaging protocol of course, However its general 
purpose nature and the fact that both RabbitMQ and Qpid really came 
about through AMQP makes it a reasonable choice.

> In the case of Nova (and others that followed Nova's messaging
> patterns), I firmly believe that for scaling reasons, we need to move
> toward it becoming the norm to use peer-to-peer messaging for most
> things.  For example, the API and conductor services should be talking
> directly to compute nodes instead of through a broker.

Is scale the only reason for preferring direct communication? I don't 
think an intermediary based solution _necessarily_ scales less 
effectively (providing it is distributed in nature, which for example is 
one of the central aims of the dispatch router in Qpid).

That's not to argue that peer-to-peer shouldn't be used, just trying to 
understand all the factors.

One other pattern that can benefit from intermediated message flow is in 
load balancing. If the processing entities are effectively 'pulling' 
messages, this can more naturally balance the load according to capacity 
than when the producer of the workload is trying to determine the best 
balance.

>  The exception
> to that is cases where we use a publish-subscribe model, and a broker
> serves that really well.  Notifications and notification consumers
> (such as Ceilometer) are the prime example.

The 'fanout' RPC cast would perhaps be another?

> In terms of existing messaging drivers, you could accomplish this with
> a combination of both RabbitMQ or Qpid for brokered messaging and
> ZeroMQ for the direct messaging cases.  It would require only a small
> amount of additional code to allow you to select a separate driver for
> each case.
>
> Based on my understanding, AMQP 1.0 could be used for both of these
> patterns.  It seems ideal long term to be able to use the same
> protocol for everything we need.

That is certainly true. AMQP 1.0 is fully symmetric so it can be used 
directly peer-to-peer as well as between intermediaries. In fact, apart 
from the establishment of the connection in the first place, a process 
need not see any difference in the interaction either way.

> We could use only ZeroMQ, as well.  It doesn't have the
> publish-subscribe stuff we need built in necessarily.  Surely that has
> been done multiple times by others already, though.  We could build it
> too, if we had to.

Indeed. However the benefit of choosing a protocol is that you can use 
solutions developed outside OpenStack or any other single project.

> Can you (or someone) elaborate further on what will make this solution
> superior to our existing options?

Superior is a very bold claim to make :-) I do personally think that an 
AMQP 1.0 based solution would be worthwhile for the reasons above. Given 
a hypothetical choice between say the current qpid driver and one that 
could talk to different back-ends, over a standard protocol for which 
e.g. semantic monitoring tools could be developed and which would make 
reasoning about partial upgrades or migrations easier, I know which I 
would lean to. Obviously that is not the choice here, since one already 
exists and the other is as yet hypothetical. However, as I say I think 
this could be a worthwhile journey and that would justify at least 
taking some initial steps.

--Gordon.