<div dir="ltr">Sandy,<div><br></div><div>Seems like we should think about how we can combine our approaches.</div><div>Rally makes load using python clients (e.g. ceilometer python client) using different amount of users/tenatns/active_users/... So it address #2 point. </div>
<div><br></div><div>About profiling part.</div><div>Actually we attempted to make profiling system based on tomograph + zipkin. But after we finished work around it we got complex and unstable solution. So we take a look at ceilometer and seems like it is the perfect solution for storing profiling data. So we are almost done with this part. Only thing that we need now is virtualization system, that could be ported from zipkin. </div>
<div><br></div><div><br></div><div>So, it will be nice if you will be able to join our efforts. And help with testing ceilometer & build OpenStack profiling system. </div><div><br></div><div><br></div><div>Best regards,</div>
<div>Boris Pavlovic</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 29, 2013 at 9:05 PM, Sandy Walsh <span dir="ltr"><<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
<br>
On 11/29/2013 11:32 AM, Nadya Privalova wrote:<br>
> Hello Sandy,<br>
><br>
> I'm very interested in performance results for Ceilometer. Now we have<br>
> successfully installed Ceilometer in the HA-lab with 200 computes and 3<br>
> controllers. Now it works pretty good with MySQL. Our next steps are:<br>
><br>
> 1. Configure alarms<br>
> 2. Try to use Rally for OpenStack performance with MySQL and MongoDB<br>
> (<a href="https://wiki.openstack.org/wiki/Rally" target="_blank">https://wiki.openstack.org/wiki/Rally</a>)<br>
><br>
> We are open to any suggestions.<br>
<br>
</div>Awesome, as a group we really need to start a similar effort as the<br>
storage driver tests for ceilometer in general.<br>
<br>
I assume you're just pulling Samples via the agent? We're really just<br>
focused on event storage and retrieval.<br>
<br>
There seems to be three levels of load testing required:<br>
1. testing through the collectors (either sample or event collection)<br>
2. testing load on the CM api<br>
3. testing the storage drivers.<br>
<br>
Sounds like you're addressing #1, we're addressing #3 and Tempest<br>
integration tests will be handling #2.<br>
<br>
I should also add that we've instrumented the db and ceilometer hosts<br>
using Diamond to statsd/graphite for tracking load on the hosts while<br>
the tests are underway. This will help with determining how many<br>
collectors we need, where the bottlenecks are coming from, etc.<br>
<br>
It might be nice to standardize on that so we can compare results?<br>
<br>
-S<br>
<div class="im"><br>
><br>
> Thanks,<br>
> Nadya<br>
><br>
><br>
><br>
> On Wed, Nov 27, 2013 at 9:42 PM, Sandy Walsh <<a href="mailto:sandy.walsh@rackspace.com">sandy.walsh@rackspace.com</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:sandy.walsh@rackspace.com">sandy.walsh@rackspace.com</a>>> wrote:<br>
><br>
> Hey!<br>
><br>
> We've ballparked that we need to store a million events per day. To<br>
> that end, we're flip-flopping between sql and no-sql solutions,<br>
> hybrid solutions that include elastic search and other schemes.<br>
> Seems every road we go down has some limitations. So, we've started<br>
> working on test suite for load testing the ceilometer storage<br>
> drivers. The intent is to have a common place to record our findings<br>
> and compare with the efforts of others.<br>
><br>
> There's an etherpad where we're tracking our results [1] and a test<br>
> suite that we're building out [2]. The test suite works against a<br>
> fork of ceilometer where we can keep our experimental storage driver<br>
> tweaks [3].<br>
><br>
> The test suite hits the storage drivers directly, bypassing the api,<br>
> but still uses the ceilometer models. We've added support for<br>
> dumping the results to statsd/graphite for charting of performance<br>
> results in real-time.<br>
><br>
> If you're interested in large scale deployments of ceilometer, we<br>
> would welcome any assistance.<br>
><br>
> Thanks!<br>
> -Sandy<br>
><br>
> [1] <a href="https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing" target="_blank">https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing</a><br>
> [2] <a href="https://github.com/rackerlabs/ceilometer-load-tests" target="_blank">https://github.com/rackerlabs/ceilometer-load-tests</a><br>
> [3] <a href="https://github.com/rackerlabs/instrumented-ceilometer" target="_blank">https://github.com/rackerlabs/instrumented-ceilometer</a><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
</div></div>> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<div class="HOEnZb"><div class="h5">><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>