Open Stack

On Jul 13, 2014, at 9:29 AM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:

> Signed PGP part
> On 12/07/14 03:17, Mike Bayer wrote:
> >
> > On 7/11/14, 7:26 PM, Carl Baldwin wrote:
> >>
> >>
> >> On Jul 11, 2014 5:32 PM, "Vishvananda Ishaya"
> >> <vishvananda at gmail.com
> > <mailto:vishvananda at gmail.com>> wrote:
> >>>
> >>> I have tried using pymysql in place of mysqldb and in real
> >>> world
> > concurrency
> >>> tests against cinder and nova it performs slower. I was
> >>> inspired by
> > the mention
> >>> of mysql-connector so I just tried that option instead.
> > Mysql-connector seems
> >>> to be slightly slower as well, which leads me to believe that
> >>> the
> > blocking inside of
> >>
> >> Do you have some numbers?  "Seems to be slightly slower" doesn't
> > really stand up as an argument against the numbers that have been
> > posted in this thread.

Numbers are highly dependent on a number of other factors, but I was
seeing 100 concurrent list commands against cinder going from an average
of 400 ms to an average of around 600 ms with both msql-connector and pymsql.

It is also worth mentioning that my test of 100 concurrent creates from the
same project in cinder leads to average response times over 3 seconds. Note that
creates return before the request is sent to the node for processing, so
this is just the api creating the db record and sticking a message on the queue.
A huge part of the slowdown is in quota reservation processing which does a row
lock on the project id.

Before we are sure that an eventlet friendly backend “gets rid of all deadlocks”, I will
mention that trying this test against connector leads to some requests timing out
at our load balancer (5 minute timeout), so we may actually be introducing deadlocks
where the retry_on_deadlock operator is used.

Consider the above anecdotal for the moment, since I can’t verify for sure that
switching the sql driver didn’t introduce some other race or unrelated problem.

Let me just caution that we can’t recommend replacing our mysql backend without
real performance and load testing.

Vish

> >>
> >>> sqlalchemy is not the main bottleneck across projects.
> >>>
> >>> Vish
> >>>
> >>> P.S. The performanace in all cases was abysmal, so performance
> >>> work
> > definitely
> >>> needs to be done, but just the guess that replacing our mysql
> > library is going to
> >>> solve all of our performance problems appears to be incorrect
> >>> at
> > first blush.
> >>
> >> The motivation is still mostly deadlock relief but more
> >> performance
> > work should be done.  I agree with you there.  I'm still hopeful
> > for some improvement from this.
> >
> >
> > To identify performance that's alleviated by async you have to
> > establish up front that IO blocking is the issue, which would
> > entail having code that's blazing fast until you start running it
> > against concurrent connections, at which point you can identify via
> > profiling that IO operations are being serialized.   This is a very
> > specific issue.
> >
> > In contrast, to identify why some arbitrary openstack app is slow,
> > my bet is that async is often not the big issue.   Every day I look
> > at openstack code and talk to people working on things,  I see
> > many performance issues that have nothing to do with concurrency,
> > and as I detailed in my wiki page at
> > https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy there is a
> > long road to cleaning up all the excessive queries, hundreds of
> > unnecessary rows and columns being pulled over the network,
> > unindexed lookups, subquery joins, hammering of Python-intensive
> > operations (often due to the nature of OS apps as lots and lots of
> > tiny API calls) that can be cached.   There's a clear path to tons
> > better performance documented there and most of it is not about
> > async  - which means that successful async isn't going to solve all
> > those issues.
> >
> 
> Of course there is a long road to decent performance, and switching a
> library won't magically fix all out issues. But if it will fix
> deadlocks, and give 30% to 150% performance boost for different
> operations, and since the switch is almost smooth, this is something
> worth doing.
> 
> >
> >
> >
> > _______________________________________________ OpenStack-dev
> > mailing list OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140714/3a36e247/attachment.pgp>