Open Stack

On Dec 20, 2013, at 8:10 AM, Doug Hellmann <doug.hellmann at dreamhost.com> wrote:

> 
> 
> 
> On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke <john.herndon at hp.com> wrote:
> Hi Folks,
> 
> The Rackspace-HP team has been putting a lot of effort into performance
> testing event collection in the ceilometer storage drivers[0]. Based on
> some results of this testing, we would like to support batch consumption
> of notifications, as it will greatly improve insertion performance. Batch
> consumption in this case means waiting for a certain number of
> notifications to arrive before sending to the storage
> driver.
> 
> I¹d like to get feedback from the community about this feature, and how we
> are planning to implement it. Here is what I’m currently thinking:
> 
> 1) This seems to fit well into oslo.messaging - batching may be a feature
> that other projects will find useful. After reviewing the changes that
> sileht has been working on in oslo.messaging, I think the right way to
> start off is to create a new executor that builds up a batch of
> notifications, and sends the batch to the dispatcher. We’d also add a
> timeout, so if a certain amount of time passes and the batch isn’t filled
> up, the notifications will be dispatched anyway. I’ve started a
> blueprint for this change and am filling in the details as I go along [1].
> 
> IIRC, the executor is meant to differentiate between threading, eventlet, other async implementations, or other methods for dealing with the I/O. It might be better to implement the batching at the dispatcher level instead. That way no matter what I/O processing is in place, the batching will occur.
> 

I thought about doing it in the dispatcher. One problem I see is handling message acks. It looks like the current executors are built around single messages andre-queueing single messages if problems occur. If we build up a batch in the dispatcher, either the executor has to wait for the whole batch to be committed (which wouldn’t work in the case of the blocking executor, or would leave a lot of green threads hanging around in the case of the eventlet executor), or the executor has to be modified to allow acking to be handled out of band. So, I was thinking it would be cleaner to write a new executor that is responsible for acking/requeueing the entire batch. Maybe I’m missing something?

> 
> 2) In ceilometer, initialize the notification listener with the batch
> executor instead of the eventlet executor (this should probably be
> configurable)[2]. We can then send the entire batch of notifications to
> the storage driver to be processed as events, while maintaining the
> current method for converting notifications into samples.
> 
> 3) Error handling becomes more difficult. The executor needs to know if
> any of the notifications should be requeued. I think the right way to
> solve this is to return a list of notifications to requeue from the
> handler. Any better ideas?
> 
> Which "handler" do you mean?

Ah, sorry - handler is whichever method is registered to receive the batch from the dispatcher. In ceilometer’s case, this would be process_notifications I think.

> Doug
> 
>  
> 
> Is this the right approach to take? I¹m not an oslo.messaging expert, so
> if there is a proper way to implement this change, I¹m all ears!
> 
> Thanks, happy holidays!
> -john
> 
> 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing
> 1:
> https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages
> 2: https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-----------------
John Herndon
HP Cloud
john.herndon at hp.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/c0672462/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4958 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131220/c0672462/attachment.bin>