<div dir="ltr">Mark,<div><br></div><div>yes, sorry for not mentioning it here. it's 3PM - 4PM UTC time zone.</div><div><br></div><div>Cheers,</div><div>Dina</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 10, 2015 at 2:51 AM, Mark Wagner <span dir="ltr"><<a href="mailto:mwagner@redhat.com" target="_blank">mwagner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
For clarification, this is 3-4 PM (15:00 - 16:00) UTC, correct ?.<br>
<br>
-mark<br>
<span class=""><br>
----- Original Message -----<br>
> From: "Dina Belova" <<a href="mailto:dbelova@mirantis.com">dbelova@mirantis.com</a>><br>
> To: "Matt Riedemann" <<a href="mailto:mriedem@linux.vnet.ibm.com">mriedem@linux.vnet.ibm.com</a>><br>
> Cc: "OpenStack Development Mailing List" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>>, <a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
</span><span class="">> Sent: Monday, November 9, 2015 4:30:55 AM<br>
> Subject: Re: [Openstack-operators] [openstack-dev] Performance Team summit session results<br>
><br>
</span>> Folks,<br>
><br>
> due to the doodle <<a href="http://doodle.com/poll/wv6qt8eqtc3mdkuz#table" rel="noreferrer" target="_blank">http://doodle.com/poll/wv6qt8eqtc3mdkuz#table</a>> 3:00 -<br>
<span class="">> 4:00 UTC Tuesdays (starting from tomorrow) is ok for all voted people.<br>
> Although for the US folks with PST time zone it'll be very early due to the<br>
> time zone change happened for US on November, 1st. Still hope seeing you<br>
</span>> there on *#openstack-performance* channel :)<br>
<span class="">><br>
> I've created primary wiki pages for the team<br>
</span>> <<a href="https://wiki.openstack.org/wiki/Performance_Team" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/Performance_Team</a>> and its meetings<br>
> <<a href="https://wiki.openstack.org/wiki/Meetings/Performance" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/Meetings/Performance</a>> - please feel free<br>
<div class="HOEnZb"><div class="h5">> to add more items to the agenda.<br>
><br>
> See you tomorrow :)<br>
><br>
> Cheers,<br>
> Dina<br>
><br>
><br>
> On Mon, Nov 9, 2015 at 5:38 PM, Dina Belova <<a href="mailto:dbelova@mirantis.com">dbelova@mirantis.com</a>> wrote:<br>
><br>
> > Matt,<br>
> ><br>
> > thank you so much for covering [1], [2] and [3] points - I'll ping folks<br>
> > who've written these lines directly and will try to find out the answers.<br>
> ><br>
> > Cheers,<br>
> > Dina<br>
> ><br>
> > On Fri, Oct 30, 2015 at 1:42 AM, Matt Riedemann <<br>
> > <a href="mailto:mriedem@linux.vnet.ibm.com">mriedem@linux.vnet.ibm.com</a>> wrote:<br>
> ><br>
> >><br>
> >><br>
> >> On 10/29/2015 10:55 AM, Matt Riedemann wrote:<br>
> >><br>
> >>><br>
> >>><br>
> >>> On 10/29/2015 9:30 AM, Dina Belova wrote:<br>
> >>><br>
> >>>> Hey folks!<br>
> >>>><br>
> >>>> On Tuesday we had great summit session about performance team kick-off<br>
> >>>> and yesterday it was a great LDT session as well and I’m really glad to<br>
> >>>> see how much does the OpenStack performance topic is important for all<br>
> >>>> of us. 40 minutes session surely was not enough to analyse everyone’s<br>
> >>>> feedback and bottlenecks people usually see, so I’ll try to finalise<br>
> >>>> what have been discussed and the next steps in this email.<br>
> >>>><br>
> >>>> Performance team kick-off session<br>
> >>>> (<br>
> >>>> <a href="https://etherpad.openstack.org/p/mitaka-cross-project-performance-team-kick-off" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/mitaka-cross-project-performance-team-kick-off</a><br>
> >>>> )<br>
> >>>><br>
> >>>> can be shortly described with the following points:<br>
> >>>><br>
> >>>> * IBM, Intel, HP, Mirantis, Rackspace, Red Hat, Yahoo! and others were<br>
> >>>> taking part in the session<br>
> >>>> * Various tools are used right now for OpenStack benchmarking and<br>
> >>>> profiling right now:<br>
> >>>> o Rally (IBM, HP, Mirantis, Yahoo!)<br>
> >>>> o Shaker (Mirantis, merging its functionality to Rally right now)<br>
> >>>> o Gatling (Rackspace)<br>
> >>>> o Zipkin (Yahoo!)<br>
> >>>> o JMeter (Yandex)<br>
> >>>> o and others…<br>
> >>>> * Various issues have been seen during the OpenStack cloud operating<br>
> >>>> (full list can be found here -<br>
> >>>> <a href="https://etherpad.openstack.org/p/openstack-performance-issues" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/openstack-performance-issues</a>).<br>
> >>>> Most<br>
> >>>> mentioned issues were the following:<br>
> >>>> o performance of DB-related layers (DB itself and oslo.db) - it is<br>
> >>>> about 7 abstraction DB layers in Nova; performance of Nova<br>
> >>>> conductor was mentioned several times<br>
> >>>> o performance of MQ-related layers (MQ itself and oslo.messaging)<br>
> >>>> * Different companies are using different standards for performance<br>
> >>>> benchmarking (both control plane and data plane testing)<br>
> >>>> * The most wished output from the team due to the comments will be:<br>
> >>>> o agree on the “performance testing standard”, including answers<br>
> >>>> on the following questions:<br>
> >>>> + what tools need to be used for OpenStack performance<br>
> >>>> benchmarking?<br>
> >>>> + what benchmarking meters need to be covered? what we would<br>
> >>>> like to compare?<br>
> >>>> + what scenarios need to be covered?<br>
> >>>> + how can we compare performance of different cloud<br>
> >>>> deployments?<br>
> >>>> + what performance deployment patterns can be used for various<br>
> >>>> workloads?<br>
> >>>> o share test plans and perform benchmarking tests<br>
> >>>> o create methodologies and documentation about best OpenStack<br>
> >>>> deployment and performance testing practices<br>
> >>>><br>
> >>>><br>
> >>>> We’re going to cover all these topics further. First of all IRC channel<br>
> >>>> for the discussions was created: *#openstack-performance*. We’re going<br>
> >>>> to have weekly meeting related to current progress on that channel,<br>
> >>>> doodle with the voting can be found here:<br>
> >>>> <a href="http://doodle.com/poll/wv6qt8eqtc3mdkuz#table" rel="noreferrer" target="_blank">http://doodle.com/poll/wv6qt8eqtc3mdkuz#table</a><br>
> >>>> (I was brave enough not to include timeslots that were overlapping<br>
> >>>> with some of mine really hard-to-move activities :))<br>
> >>>><br>
> >>>> Let’s have next week as a voting time, and have first IRC meeting in our<br>
> >>>> channel the week after next. We can start our further discussions with<br>
> >>>> “performance” and “performance testing” terms definition and<br>
> >>>> benchmarking tools analysis.<br>
> >>>><br>
> >>>> Cheers,<br>
> >>>> Dina<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> __________________________________________________________________________<br>
> >>>><br>
> >>>> OpenStack Development Mailing List (not for usage questions)<br>
> >>>> Unsubscribe:<br>
> >>>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> >>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>>><br>
> >>>><br>
> >>> Thanks for writing this up, it's great to see people getting together<br>
> >>> and sharing info on performance issues and trying to pinpoint the big<br>
> >>> ones.<br>
> >>><br>
> >>> I poked through the performance issues etherpad and was wondering how<br>
> >>> many people with DB issues, particularly for nova-conductor, are using a<br>
> >>> level of oslo.db that's new enough to be using pymysql rather than<br>
> >>> mysql-python because from what I remember there were eventlet issues<br>
> >>> without pymysql. That was added to oslo.db 1.12.0 [1].<br>
> >>><br>
> >>> The nova-conductor workers / CPU usage is also a known issue in the<br>
> >>> large ops gate job [2] but I'm not aware of anyone spending the time<br>
> >>> drilling into what exactly is causing a lot of that overhead and if any<br>
> >>> of it is abnormal.<br>
> >>><br>
> >>> Finally, wrt DB, I'd also be interested to know if Rackspace, or anyone<br>
> >>> else, is still running with the direct-to-sql stuff that comstud wrote<br>
> >>> for nova [3] and if that still shows significant performance<br>
> >>> improvements over using sqlalchemy ORM. Not to open that can of worms in<br>
> >>> the -dev list here again, but it'd be an interesting data point.<br>
> >>><br>
> >>> [1] <a href="https://review.openstack.org/#/c/184392/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/184392/</a><br>
> >>> [2] <a href="https://review.openstack.org/#/c/228636/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/228636/</a><br>
> >>> [3] <a href="https://blueprints.launchpad.net/nova/+spec/db-mysqldb-impl" rel="noreferrer" target="_blank">https://blueprints.launchpad.net/nova/+spec/db-mysqldb-impl</a><br>
> >>><br>
> >>><br>
> >> Oops, forgot to copy the ops list on the last reply.<br>
> >><br>
> >> --<br>
> >><br>
> >> Thanks,<br>
> >><br>
> >> Matt Riedemann<br>
> >><br>
> >><br>
> >> _______________________________________________<br>
> >> OpenStack-operators mailing list<br>
> >> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
> >><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Best regards,<br>
> ><br>
> > Dina Belova<br>
> ><br>
> > Software Engineer<br>
> ><br>
> > Mirantis Inc.<br>
> ><br>
><br>
><br>
><br>
> --<br>
><br>
> Best regards,<br>
><br>
> Dina Belova<br>
><br>
> Software Engineer<br>
><br>
> Mirantis Inc.<br>
><br>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"><p style="font-size:small;margin:0px;font-family:Helvetica">Best regards,</p><p style="font-size:small;margin:0px;font-family:Helvetica">Dina Belova</p><p style="font-size:small;margin:0px;font-family:Helvetica">Software Engineer</p><p style="font-size:small;margin:0px;font-family:Helvetica">Mirantis Inc.</p></div></div></div>
</div>