Open Stack

On 09/12/13 17:37 -0500, Russell Bryant wrote:
>On 12/09/2013 05:16 PM, Gordon Sim wrote:
>> On 12/09/2013 07:15 PM, Russell Bryant wrote:

[...]

>>>> One other pattern that can benefit from intermediated message flow is in
>>>> load balancing. If the processing entities are effectively 'pulling'
>>>> messages, this can more naturally balance the load according to capacity
>>>> than when the producer of the workload is trying to determine the best
>>>> balance.
>>>
>>> Yes, that's another factor.  Today, we rely on the message broker's
>>> behavior to equally distribute messages to a set of consumers.
>>
>> Sometimes you even _want_ message distribution to be 'unequal', if the
>> load varies by message or the capacity by consumer. E.g. If one consumer
>> is particularly slow (or is given a particularly arduous task), it may
>> not be optimal for it to receive the same portion of subsequent messages
>> as other less heavily loaded or more powerful consumers.
>
>Indeed.  We haven't tried to do that anywhere, but it would be an
>improvement for some cases.

Agreed, this is something that worths experimenting.

[...]

>>> I'm very interested in diving deeper into how Dispatch would fit into
>>> the various ways OpenStack is using messaging today.  I'd like to get
>>> a better handle on how the use of Dispatch as an intermediary would
>>> scale out for a deployment that consists of 10s of thousands of
>>> compute nodes, for example.
>>>
>>> Is it roughly just that you can have a network of N Dispatch routers
>>> that route messages from point A to point B, and for notifications we
>>> would use a traditional message broker (qpidd or rabbitmq) ?
>>
>> For scaling the basic idea is that not all connections are made to the
>> same process and therefore not all messages need to travel through a
>> single intermediary process.
>>
>> So for N different routers, each have a portion of the total number of
>> publishers and consumers connected to them. Though client can
>> communicate even if they are not connected to the same router, each
>> router only needs to handle the messages sent by the publishers directly
>> attached, or sent to the consumer directly attached. It never needs to
>> see messages between publishers and consumer that are not directly
>> attached.
>>
>> To address your example, the 10s of thousands of compute nodes would be
>> spread across N routers. Assuming these were all interconnected, a
>> message from the scheduler would only travel through at most two of
>> these N routers (the one the scheduler was connected to and the one the
>> receiving compute node was connected to). No process needs to be able to
>> handle 10s of thousands of connections itself (as contrasted with full
>> direct, non-intermediated communication, where the scheduler would need
>> to manage connections to each of the compute nodes).
>>
>> This basic pattern is the same as networks of brokers, but Dispatch
>> router has been designed from the start to simply focus on that problem
>> (and not deal with all other broker related features, such as
>> transactions, durability, specialised queueing etc).
>
>Soudns awesome.  :-)
>
>> The other difference is that Dispatch Router does not accept
>> responsibility for messages, i.e. it does not offer any
>> store-and-forward behaviour. Any acknowledgement is end-to-end. This
>> avoids it having to replicate messages. On failure they can if needed by
>> replayed by the original sender.
>
>I think the lack of store-and-forward is OK.
>
>Right now, all of the Nova code is written to assume that the messaging
>is unreliable and that any message could get lost.  It may result in an
>operation failing, but it should fail gracefully.  Doing end-to-end
>acknowledgement may actually be an improvement.

This is interesting and a very important point. I wonder what the
reliability expectations of other services w.r.t OpenStack messaging
are.

I agree on the fact that p2p acknowledgement could be an improvement
but I'm also wondering how this (if ever) will affect projects - in
terms of requiring changes. One of the goals of this new driver is to
not require any changes on the existing projects.

Also, a bit different but related topic, are there cases where tasks
are re-scheduled in nova? If so, what does nova do in this case? Are
those task sent back to `nova-scheduler` for re-scheduling?

Cheers,
FF

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131210/8323d123/attachment.pgp>