Open Stack

On 12/09/2013 12:56 PM, Gordon Sim wrote:
>> In the case of Nova (and others that followed Nova's messaging
>> patterns), I firmly believe that for scaling reasons, we need to move
>> toward it becoming the norm to use peer-to-peer messaging for most
>> things.  For example, the API and conductor services should be talking
>> directly to compute nodes instead of through a broker.
> 
> Is scale the only reason for preferring direct communication? I don't
> think an intermediary based solution _necessarily_ scales less
> effectively (providing it is distributed in nature, which for example is
> one of the central aims of the dispatch router in Qpid).
> 
> That's not to argue that peer-to-peer shouldn't be used, just trying to
> understand all the factors.

Scale is the primary one.  If the intermediary based solution is easily
distributed to handle our scaling needs, that would probably be fine,
too.  That just hasn't been our experience so far with both RabbitMQ and
Qpid.

> One other pattern that can benefit from intermediated message flow is in
> load balancing. If the processing entities are effectively 'pulling'
> messages, this can more naturally balance the load according to capacity
> than when the producer of the workload is trying to determine the best
> balance.

Yes, that's another factor.  Today, we rely on the message broker's
behavior to equally distribute messages to a set of consumers.

One example is how Nova components talk to the nova-scheduler service.
All instances of the nova-scheduler service are reading off a single
'scheduler' queue, so messages hit them round-robin.

In the case of the zeromq driver, this logic is embedded in the client.
 It has to know about all consumers and handles choosing where each
message goes itself.  See references to the 'matchmaker' code for this.

Honestly, using a distributed more lightweight router like Dispatch
sounds *much* nicer.

>>  The exception
>> to that is cases where we use a publish-subscribe model, and a broker
>> serves that really well.  Notifications and notification consumers
>> (such as Ceilometer) are the prime example.
> 
> The 'fanout' RPC cast would perhaps be another?

Good point.

In Nova we have been working to get rid of the usage of this pattern.
In the latest code the only place it's used AFAIK is in some code we
expect to mark deprecated (nova-network).

>> In terms of existing messaging drivers, you could accomplish this with
>> a combination of both RabbitMQ or Qpid for brokered messaging and
>> ZeroMQ for the direct messaging cases.  It would require only a small
>> amount of additional code to allow you to select a separate driver for
>> each case.
>>
>> Based on my understanding, AMQP 1.0 could be used for both of these
>> patterns.  It seems ideal long term to be able to use the same
>> protocol for everything we need.
> 
> That is certainly true. AMQP 1.0 is fully symmetric so it can be used
> directly peer-to-peer as well as between intermediaries. In fact, apart
> from the establishment of the connection in the first place, a process
> need not see any difference in the interaction either way.
> 
>> We could use only ZeroMQ, as well.  It doesn't have the
>> publish-subscribe stuff we need built in necessarily.  Surely that has
>> been done multiple times by others already, though.  We could build it
>> too, if we had to.
> 
> Indeed. However the benefit of choosing a protocol is that you can use
> solutions developed outside OpenStack or any other single project.
> 
>> Can you (or someone) elaborate further on what will make this solution
>> superior to our existing options?
> 
> Superior is a very bold claim to make :-) I do personally think that an
> AMQP 1.0 based solution would be worthwhile for the reasons above. Given
> a hypothetical choice between say the current qpid driver and one that
> could talk to different back-ends, over a standard protocol for which
> e.g. semantic monitoring tools could be developed and which would make
> reasoning about partial upgrades or migrations easier, I know which I
> would lean to. Obviously that is not the choice here, since one already
> exists and the other is as yet hypothetical. However, as I say I think
> this could be a worthwhile journey and that would justify at least
> taking some initial steps.

Thanks for sharing some additional insight.

I was already quite optimistic, but you've helped solidify that.  I'm
very interested in diving deeper into how Dispatch would fit into the
various ways OpenStack is using messaging today.  I'd like to get a
better handle on how the use of Dispatch as an intermediary would scale
out for a deployment that consists of 10s of thousands of compute nodes,
for example.

Is it roughly just that you can have a network of N Dispatch routers
that route messages from point A to point B, and for notifications we
would use a traditional message broker (qpidd or rabbitmq) ?

-- 
Russell Bryant