Open Stack

On Fri, Dec 20 2013, Herndon, John Luke wrote:

> Yeah, I like this idea. As far as I can tell, AMQP doesn’t support grabbing
> more than a single message at a time, but we could definitely have the
> broker store up the batch before sending it along. Other protocols may
> support bulk consumption. My one concern with this approach is error
> handling. Currently the executors treat each notification individually. So
> let’s say the broker hands 100 messages at a time. When client is done
> processing the messages, the broker needs to know if message 25 had an error
> or not. We would somehow need to communicate back to the broker which
> messages failed. I think this may take some refactoring of
> executors/dispatchers. What do you think?

Yeah, it definitely needs to change the messaging API a bit to handle
such a case. But in the end that will be a good thing to support such a
case, it being natively supported by the broker or not.

For brokers where it's not possible, it may be simple enough to have a
"get_one_notification_nb()" method that would either return a
notification or None if there's none to read, and would that
consequently have to be _non-blocking_.

So if the transport is smart we write:

  # Return up to max_number_of_notifications_to_read
  notifications =
      transport.get_notificatations(conf.max_number_of_notifications_to_read)
  storage.record(notifications)

Otherwise we do:

  for i in range(conf.max_number_of_notifications_to_read):
      notification = transport.get_one_notification_nb():
      if notification:
          notifications.append(notification)
      else:
          break
   storage.record(notifications)

So it's just about having the right primitive in oslo.messaging, we can
then build on top of that wherever that is.

-- 
Julien Danjou
/* Free Software hacker * independent consultant
   http://julien.danjou.info */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/02407564/attachment.pgp>