Open Stack

On 12/20/2013 05:27 PM, Herndon, John Luke wrote:
>
> On Dec 20, 2013, at 8:48 AM, Julien Danjou <julien at danjou.info>
> wrote:
>> Anyway, my main concern here is that I am not very enthusiast
>> about using the executor to do that. I wonder if there is not a way
>> to ask the broker to get as many as message as it has up to a
>> limit?
>>
>> You would have 100 messages waiting in the notifications.info
>> queue, and you would be able to tell to oslo.messaging that you
>> want to read up to 10 messages at a time. If the underlying
>> protocol (e.g. AMQP) can support that too, it would be more
>> efficient too.
>
> Yeah, I like this idea. As far as I can tell, AMQP doesn’t support
> grabbing more than a single message at a time, but we could
> definitely have the broker store up the batch before sending it
> along.

AMQP (in all it's versions) allows for a subscription with a 
configurable amount of 'prefetch', which means the broker can send lots 
of messages without waiting for the client to request them one at a time.

That's not quite the same as the batching I think you are looking for, 
but it does allow the broker to do its own batching. My guess is the 
rabbit driver is already using basic.consume rather than basic.get 
anyway(?), so the broker is free to batch as it sees fit.  (I haven't 
actually dug into the kombu code to verify that however, perhaps someone 
else here can confirm?)

However you still need the client to have some way of batching up the 
messages and then processing them together.

> Other protocols may support bulk consumption. My one concern
> with this approach is error handling. Currently the executors treat
> each notification individually. So let’s say the broker hands 100
> messages at a time. When client is done processing the messages, the
> broker needs to know if message 25 had an error or not. We would
> somehow need to communicate back to the broker which messages failed.
> I think this may take some refactoring of executors/dispatchers. What
> do you think?

I've have some related questions, that I haven't yet satisfactorily 
answered yet. The extra context here may be useful in doing so.

(1) What are the expectations around message delivery guarantees for 
insertion into a store? I.e. if there is a failure, is it ok to get 
duplicate entries for notifications? (I'm assuming losing notifications 
is not acceptable).

(2) What would you want the broker to do with the failed messages? What 
sort of things might fail? Is it related to the message content itself? 
Or is it failures suspected to be of a temporal nature?

(3) How important is ordering ? If a failure causes some notifications 
to be inserted out of order is that a problem at all?