Open Stack

On Dec 20, 2013, at 10:47 AM, Julien Danjou <julien at danjou.info> wrote:

> On Fri, Dec 20 2013, Herndon, John Luke wrote:
> 
>> Yeah, I like this idea. As far as I can tell, AMQP doesn’t support grabbing
>> more than a single message at a time, but we could definitely have the
>> broker store up the batch before sending it along. Other protocols may
>> support bulk consumption. My one concern with this approach is error
>> handling. Currently the executors treat each notification individually. So
>> let’s say the broker hands 100 messages at a time. When client is done
>> processing the messages, the broker needs to know if message 25 had an error
>> or not. We would somehow need to communicate back to the broker which
>> messages failed. I think this may take some refactoring of
>> executors/dispatchers. What do you think?
> 
> Yeah, it definitely needs to change the messaging API a bit to handle
> such a case. But in the end that will be a good thing to support such a
> case, it being natively supported by the broker or not.
> 
> For brokers where it's not possible, it may be simple enough to have a
> "get_one_notification_nb()" method that would either return a
> notification or None if there's none to read, and would that
> consequently have to be _non-blocking_.
> 
> So if the transport is smart we write:
> 
>  # Return up to max_number_of_notifications_to_read
>  notifications =
>      transport.get_notificatations(conf.max_number_of_notifications_to_read)
>  storage.record(notifications)
> 
> Otherwise we do:
> 
>  for i in range(conf.max_number_of_notifications_to_read):
>      notification = transport.get_one_notification_nb():
>      if notification:
>          notifications.append(notification)
>      else:
>          break
>   storage.record(notifications)
> 
> So it's just about having the right primitive in oslo.messaging, we can
> then build on top of that wherever that is.
> 

I think this will work. I was considering putting in a timeout so the broker would not send off all of the messages immediately, and implement using blocking calls. If the consumer consumes faster than the publishers are publishing, this just becomes single-notification batches. So it may be beneficial to wait for more messages to arrive before sending off the batch. If the batch is full before the timeout is reached, then the batch would be sent off.

> -- 
> Julien Danjou
> /* Free Software hacker * independent consultant
>   http://julien.danjou.info */

-----------------
John Herndon
HP Cloud
john.herndon at hp.com



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4958 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/7e939c94/attachment.bin>