[openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

Herndon, John Luke john.herndon at hp.com
Fri Dec 20 14:29:13 UTC 2013


Hi Nadya,

Yep, that’s right, the notifications stick around on the server until they
are acknowledged so there is extra overhead involved. I only have experience
with rabbitmq, so I can’t speak for other transports, but we have used this
strategy internally for other purposes, and have reached > 10k
messages/second on a single consumer using batch message consumption (i.e.,
consume N messages, process them, then ack all N at once). We’ve found that
being able to acknowledge the entire batch of messages at a time leads to a
huge performance increase. This is another motivating factor for moving
towards batches. But to your point, making this configurable is the right
way to go just in case other transports don’t react as well.

Thanks,
-john


From:  Nadya Privalova <nprivalova at mirantis.com>
Reply-To:  "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date:  Fri, 20 Dec 2013 15:25:55 +0400
To:  "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject:  Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in
Batches

Hi John,

As for me your ideas look very interesting. As I understood notification
messages will be kept in MQ for some time (during batch-basket is being
filled), right? I'm concerned about the additional load that will be on MQ
(Rabbit). 

Thanks,
Nadya


On Fri, Dec 20, 2013 at 3:31 AM, Herndon, John Luke <john.herndon at hp.com>
wrote:
> Hi Folks,
> 
> The Rackspace-HP team has been putting a lot of effort into performance
> testing event collection in the ceilometer storage drivers[0]. Based on
> some results of this testing, we would like to support batch consumption
> of notifications, as it will greatly improve insertion performance. Batch
> consumption in this case means waiting for a certain number of
> notifications to arrive before sending to the storage
> driver.
> 
> I¹d like to get feedback from the community about this feature, and how we
> are planning to implement it. Here is what I’m currently thinking:
> 
> 1) This seems to fit well into oslo.messaging - batching may be a feature
> that other projects will find useful. After reviewing the changes that
> sileht has been working on in oslo.messaging, I think the right way to
> start off is to create a new executor that builds up a batch of
> notifications, and sends the batch to the dispatcher. We’d also add a
> timeout, so if a certain amount of time passes and the batch isn’t filled
> up, the notifications will be dispatched anyway. I’ve started a
> blueprint for this change and am filling in the details as I go along [1].
> 
> 2) In ceilometer, initialize the notification listener with the batch
> executor instead of the eventlet executor (this should probably be
> configurable)[2]. We can then send the entire batch of notifications to
> the storage driver to be processed as events, while maintaining the
> current method for converting notifications into samples.
> 
> 3) Error handling becomes more difficult. The executor needs to know if
> any of the notifications should be requeued. I think the right way to
> solve this is to return a list of notifications to requeue from the
> handler. Any better ideas?
> 
> Is this the right approach to take? I¹m not an oslo.messaging expert, so
> if there is a proper way to implement this change, I¹m all ears!
> 
> Thanks, happy holidays!
> -john
> 
> 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing
> 1:
> https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages
> 2: https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

_______________________________________________ OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/63e770dd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5443 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/63e770dd/attachment.bin>


More information about the OpenStack-dev mailing list