[openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

Doug Hellmann doug.hellmann at dreamhost.com
Fri Dec 20 20:17:21 UTC 2013


On Fri, Dec 20, 2013 at 12:15 PM, Herndon, John Luke <john.herndon at hp.com>wrote:

>
> On Dec 20, 2013, at 8:10 AM, Doug Hellmann <doug.hellmann at dreamhost.com>
> wrote:
>
>
>
>
> On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke <john.herndon at hp.com>wrote:
>
>> Hi Folks,
>>
>> The Rackspace-HP team has been putting a lot of effort into performance
>> testing event collection in the ceilometer storage drivers[0]. Based on
>> some results of this testing, we would like to support batch consumption
>> of notifications, as it will greatly improve insertion performance. Batch
>> consumption in this case means waiting for a certain number of
>> notifications to arrive before sending to the storage
>> driver.
>>
>> I¹d like to get feedback from the community about this feature, and how we
>> are planning to implement it. Here is what I’m currently thinking:
>>
>> 1) This seems to fit well into oslo.messaging - batching may be a feature
>> that other projects will find useful. After reviewing the changes that
>> sileht has been working on in oslo.messaging, I think the right way to
>> start off is to create a new executor that builds up a batch of
>> notifications, and sends the batch to the dispatcher. We’d also add a
>> timeout, so if a certain amount of time passes and the batch isn’t filled
>> up, the notifications will be dispatched anyway. I’ve started a
>> blueprint for this change and am filling in the details as I go along [1].
>>
>
> IIRC, the executor is meant to differentiate between threading, eventlet,
> other async implementations, or other methods for dealing with the I/O. It
> might be better to implement the batching at the dispatcher level instead.
> That way no matter what I/O processing is in place, the batching will occur.
>
>
> I thought about doing it in the dispatcher. One problem I see is handling
> message acks. It looks like the current executors are built around single
> messages andre-queueing single messages if problems occur. If we build up a
> batch in the dispatcher, either the executor has to wait for the whole
> batch to be committed (which wouldn’t work in the case of the blocking
> executor, or would leave a lot of green threads hanging around in the case
> of the eventlet executor), or the executor has to be modified to allow
> acking to be handled out of band. So, I was thinking it would be cleaner to
> write a new executor that is responsible for acking/requeueing the entire
> batch. Maybe I’m missing something?
>

No, you're right. Were you going to use eventlet again for the new
executor?



>
>
>> 2) In ceilometer, initialize the notification listener with the batch
>> executor instead of the eventlet executor (this should probably be
>> configurable)[2]. We can then send the entire batch of notifications to
>> the storage driver to be processed as events, while maintaining the
>> current method for converting notifications into samples.
>>
>> 3) Error handling becomes more difficult. The executor needs to know if
>> any of the notifications should be requeued. I think the right way to
>> solve this is to return a list of notifications to requeue from the
>> handler. Any better ideas?
>>
>
> Which "handler" do you mean?
>
>
> Ah, sorry - handler is whichever method is registered to receive the batch
> from the dispatcher. In ceilometer’s case, this would be
> process_notifications I think.
>
>
> Doug
>
>
>
>>
>> Is this the right approach to take? I¹m not an oslo.messaging expert, so
>> if there is a proper way to implement this change, I¹m all ears!
>>
>> Thanks, happy holidays!
>> -john
>>
>> 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing
>> 1:
>>
>> https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages
>> 2:
>> https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> -----------------
> John Herndon
> HP Cloud
> john.herndon at hp.com
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/d8e69302/attachment.html>


More information about the OpenStack-dev mailing list