Open Stack

Hi Folks,

The Rackspace-HP team has been putting a lot of effort into performance
testing event collection in the ceilometer storage drivers[0]. Based on
some results of this testing, we would like to support batch consumption
of notifications, as it will greatly improve insertion performance. Batch
consumption in this case means waiting for a certain number of
notifications to arrive before sending to the storage
driver. 

I¹d like to get feedback from the community about this feature, and how we
are planning to implement it. Here is what I’m currently thinking:

1) This seems to fit well into oslo.messaging - batching may be a feature
that other projects will find useful. After reviewing the changes that
sileht has been working on in oslo.messaging, I think the right way to
start off is to create a new executor that builds up a batch of
notifications, and sends the batch to the dispatcher. We’d also add a
timeout, so if a certain amount of time passes and the batch isn’t filled
up, the notifications will be dispatched anyway. I’ve started a
blueprint for this change and am filling in the details as I go along [1].

2) In ceilometer, initialize the notification listener with the batch
executor instead of the eventlet executor (this should probably be
configurable)[2]. We can then send the entire batch of notifications to
the storage driver to be processed as events, while maintaining the
current method for converting notifications into samples.

3) Error handling becomes more difficult. The executor needs to know if
any of the notifications should be requeued. I think the right way to
solve this is to return a list of notifications to requeue from the
handler. Any better ideas?

Is this the right approach to take? I¹m not an oslo.messaging expert, so
if there is a proper way to implement this change, I¹m all ears!

Thanks, happy holidays!
-john

0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing
1: 
https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages
2: https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5443 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131219/89241e87/attachment.bin>