<div dir="ltr">Matt,<div><br></div><div>thank you so much for covering [1], [2] and [3] points - I'll ping folks who've written these lines directly and will try to find out the answers.</div><div><br></div><div>Cheers,</div><div>Dina</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 30, 2015 at 1:42 AM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br>
On 10/29/2015 10:55 AM, Matt Riedemann wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
On 10/29/2015 9:30 AM, Dina Belova wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey folks!<br>
<br>
On Tuesday we had great summit session about performance team kick-off<br>
and yesterday it was a great LDT session as well and I’m really glad to<br>
see how much does the OpenStack performance topic is important for all<br>
of us. 40 minutes session surely was not enough to analyse everyone’s<br>
feedback and bottlenecks people usually see, so I’ll try to finalise<br>
what have been discussed and the next steps in this email.<br>
<br>
Performance team kick-off session<br>
(<a href="https://etherpad.openstack.org/p/mitaka-cross-project-performance-team-kick-off" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/mitaka-cross-project-performance-team-kick-off</a>)<br>
<br>
can be shortly described with the following points:<br>
<br>
* IBM, Intel, HP, Mirantis, Rackspace, Red Hat, Yahoo! and others were<br>
taking part in the session<br>
* Various tools are used right now for OpenStack benchmarking and<br>
profiling right now:<br>
o Rally (IBM, HP, Mirantis, Yahoo!)<br>
o Shaker (Mirantis, merging its functionality to Rally right now)<br>
o Gatling (Rackspace)<br>
o Zipkin (Yahoo!)<br>
o JMeter (Yandex)<br>
o and others…<br>
* Various issues have been seen during the OpenStack cloud operating<br>
(full list can be found here -<br>
<a href="https://etherpad.openstack.org/p/openstack-performance-issues" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/openstack-performance-issues</a>). Most<br>
mentioned issues were the following:<br>
o performance of DB-related layers (DB itself and oslo.db) - it is<br>
about 7 abstraction DB layers in Nova; performance of Nova<br>
conductor was mentioned several times<br>
o performance of MQ-related layers (MQ itself and oslo.messaging)<br>
* Different companies are using different standards for performance<br>
benchmarking (both control plane and data plane testing)<br>
* The most wished output from the team due to the comments will be:<br>
o agree on the “performance testing standard”, including answers<br>
on the following questions:<br>
+ what tools need to be used for OpenStack performance<br>
benchmarking?<br>
+ what benchmarking meters need to be covered? what we would<br>
like to compare?<br>
+ what scenarios need to be covered?<br>
+ how can we compare performance of different cloud<br>
deployments?<br>
+ what performance deployment patterns can be used for various<br>
workloads?<br>
o share test plans and perform benchmarking tests<br>
o create methodologies and documentation about best OpenStack<br>
deployment and performance testing practices<br>
<br>
<br>
We’re going to cover all these topics further. First of all IRC channel<br>
for the discussions was created: *#openstack-performance*. We’re going<br>
to have weekly meeting related to current progress on that channel,<br>
doodle with the voting can be found here:<br>
<a href="http://doodle.com/poll/wv6qt8eqtc3mdkuz#table" rel="noreferrer" target="_blank">http://doodle.com/poll/wv6qt8eqtc3mdkuz#table</a><br>
(I was brave enough not to include timeslots that were overlapping<br>
with some of mine really hard-to-move activities :))<br>
<br>
Let’s have next week as a voting time, and have first IRC meeting in our<br>
channel the week after next. We can start our further discussions with<br>
“performance” and “performance testing” terms definition and<br>
benchmarking tools analysis.<br>
<br>
Cheers,<br>
Dina<br>
<br>
<br>
__________________________________________________________________________<br>
<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
<br>
Thanks for writing this up, it's great to see people getting together<br>
and sharing info on performance issues and trying to pinpoint the big ones.<br>
<br>
I poked through the performance issues etherpad and was wondering how<br>
many people with DB issues, particularly for nova-conductor, are using a<br>
level of oslo.db that's new enough to be using pymysql rather than<br>
mysql-python because from what I remember there were eventlet issues<br>
without pymysql. That was added to oslo.db 1.12.0 [1].<br>
<br>
The nova-conductor workers / CPU usage is also a known issue in the<br>
large ops gate job [2] but I'm not aware of anyone spending the time<br>
drilling into what exactly is causing a lot of that overhead and if any<br>
of it is abnormal.<br>
<br>
Finally, wrt DB, I'd also be interested to know if Rackspace, or anyone<br>
else, is still running with the direct-to-sql stuff that comstud wrote<br>
for nova [3] and if that still shows significant performance<br>
improvements over using sqlalchemy ORM. Not to open that can of worms in<br>
the -dev list here again, but it'd be an interesting data point.<br>
<br>
[1] <a href="https://review.openstack.org/#/c/184392/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/184392/</a><br>
[2] <a href="https://review.openstack.org/#/c/228636/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/228636/</a><br>
[3] <a href="https://blueprints.launchpad.net/nova/+spec/db-mysqldb-impl" rel="noreferrer" target="_blank">https://blueprints.launchpad.net/nova/+spec/db-mysqldb-impl</a><br>
<br>
</blockquote>
<br></div></div>
Oops, forgot to copy the ops list on the last reply.<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"><p style="font-size:small;margin:0px;font-family:Helvetica">Best regards,</p><p style="font-size:small;margin:0px;font-family:Helvetica">Dina Belova</p><p style="font-size:small;margin:0px;font-family:Helvetica">Software Engineer</p><p style="font-size:small;margin:0px;font-family:Helvetica">Mirantis Inc.</p></div></div></div>
</div>