<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Dec 20, 2013, at 8:10 AM, Doug Hellmann <<a href="mailto:doug.hellmann@dreamhost.com">doug.hellmann@dreamhost.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"><div dir="ltr"><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke <span dir="ltr"><<a href="mailto:john.herndon@hp.com" target="_blank">john.herndon@hp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; position: static; z-index: auto;">Hi Folks,<br>
<br>
The Rackspace-HP team has been putting a lot of effort into performance<br>
testing event collection in the ceilometer storage drivers[0]. Based on<br>
some results of this testing, we would like to support batch consumption<br>
of notifications, as it will greatly improve insertion performance. Batch<br>
consumption in this case means waiting for a certain number of<br>
notifications to arrive before sending to the storage<br>
driver.<br>
<br>
I¹d like to get feedback from the community about this feature, and how we<br>
are planning to implement it. Here is what I’m currently thinking:<br>
<br>
1) This seems to fit well into oslo.messaging - batching may be a feature<br>
that other projects will find useful. After reviewing the changes that<br>
sileht has been working on in oslo.messaging, I think the right way to<br>
start off is to create a new executor that builds up a batch of<br>
notifications, and sends the batch to the dispatcher. We’d also add a<br>
timeout, so if a certain amount of time passes and the batch isn’t filled<br>
up, the notifications will be dispatched anyway. I’ve started a<br>
blueprint for this change and am filling in the details as I go along [1].<br></blockquote><div><br></div><div><div class="gmail_default" style="font-size:small">IIRC, the executor is meant to differentiate between threading, eventlet, other async implementations, or other methods for dealing with the I/O. It might be better to implement the batching at the dispatcher level instead. That way no matter what I/O processing is in place, the batching will occur.</div>
</div><div><br></div></div></div></div></blockquote><div><br></div><div>I thought about doing it in the dispatcher. One problem I see is handling message acks. It looks like the current executors are built around single messages andre-queueing single messages if problems occur. If we build up a batch in the dispatcher, either the executor has to wait for the whole batch to be committed (which wouldn’t work in the case of the blocking executor, or would leave a lot of green threads hanging around in the case of the eventlet executor), or the executor has to be modified to allow acking to be handled out of band. So, I was thinking it would be cleaner to write a new executor that is responsible for acking/requeueing the entire batch. Maybe I’m missing something?</div><br><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2) In ceilometer, initialize the notification listener with the batch<br>
executor instead of the eventlet executor (this should probably be<br>
configurable)[2]. We can then send the entire batch of notifications to<br>
the storage driver to be processed as events, while maintaining the<br>
current method for converting notifications into samples.<br>
<br>
3) Error handling becomes more difficult. The executor needs to know if<br>
any of the notifications should be requeued. I think the right way to<br>
solve this is to return a list of notifications to requeue from the<br>
handler. Any better ideas?<br></blockquote><div><br></div><div><div class="gmail_default" style="font-size:small">Which "handler" do you mean?</div></div></div></div></div></blockquote><div><br></div>Ah, sorry - handler is whichever method is registered to receive the batch from the dispatcher. In ceilometer’s case, this would be process_notifications I think.<br><br><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>
<div class="gmail_default" style="font-size:small">Doug</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Is this the right approach to take? I¹m not an oslo.messaging expert, so<br>
if there is a proper way to implement this change, I¹m all ears!<br>
<br>
Thanks, happy holidays!<br>
-john<br>
<br>
0: <a href="https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing" target="_blank">https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing</a><br>
1:<br>
<a href="https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages" target="_blank">https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages</a><br>
2: <a href="https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification" target="_blank">https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification</a><br>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></blockquote></div><br><div apple-content-edited="true">
<div>-----------------</div><div>John Herndon</div><div>HP Cloud</div><div><a href="mailto:john.herndon@hp.com">john.herndon@hp.com</a></div><div><br></div><br class="Apple-interchange-newline">
</div>
<br></body></html>