Open Stack

Clark,

You make a good point.  It's there some resistance to this or is it just a
matter of asking?

Carl
On Jul 11, 2014 12:23 PM, "Clark Boylan" <clark.boylan at gmail.com> wrote:

> Before we get too far ahead of ourselves mysql-connector is not hosted
> on pypi. Instead it is an external package link. We recently managed
> to remove all packages that are hosted as external package links from
> openstack and will not add new ones in. Before we can use
> mysql-connector in the gate oracle will need to publish
> mysql-connector on pypi properly.
>
> That said there is at least one other pure python alternative,
> PyMySQL. PyMySQL supports py3k and pypy. We should look at using
> PyMySQL instead if we want to start with a reasonable path to getting
> this in the gate.
>
> Clark
>
> On Fri, Jul 11, 2014 at 10:07 AM, Miguel Angel Ajo Pelayo
> <mangelajo at redhat.com> wrote:
> > +1 here too,
> >
> > Amazed with the performance gains, x2.4 seems a lot,
> > and we'd get rid of deadlocks.
> >
> >
> >
> > ----- Original Message -----
> >> +1
> >>
> >> I'm pretty excited about the possibilities here.  I've had this
> >> mysqldb/eventlet contention in the back of my mind for some time now.
> >> I'm glad to see some work being done in this area.
> >>
> >> Carl
> >>
> >> On Fri, Jul 11, 2014 at 7:04 AM, Ihar Hrachyshka <ihrachys at redhat.com>
> wrote:
> >> > -----BEGIN PGP SIGNED MESSAGE-----
> >> > Hash: SHA512
> >> >
> >> > On 09/07/14 13:17, Ihar Hrachyshka wrote:
> >> >> Hi all,
> >> >>
> >> >> Multiple projects are suffering from db lock timeouts due to
> >> >> deadlocks deep in mysqldb library that we use to interact with
> >> >> mysql servers. In essence, the problem is due to missing eventlet
> >> >> support in mysqldb module, meaning when a db lock is encountered,
> >> >> the library does not yield to the next green thread, allowing other
> >> >> threads to eventually unlock the grabbed lock, and instead it just
> >> >> blocks the main thread, that eventually raises timeout exception
> >> >> (OperationalError).
> >> >>
> >> >> The failed operation is not retried, leaving failing request not
> >> >> served. In Nova, there is a special retry mechanism for deadlocks,
> >> >> though I think it's more a hack than a proper fix.
> >> >>
> >> >> Neutron is one of the projects that suffer from those timeout
> >> >> errors a lot. Partly it's due to lack of discipline in how we do
> >> >> nested calls in l3_db and ml2_plugin code, but that's not something
> >> >> to change in foreseeable future, so we need to find another
> >> >> solution that is applicable for Juno. Ideally, the solution should
> >> >> be applicable for Icehouse too to allow distributors to resolve
> >> >> existing deadlocks without waiting for Juno.
> >> >>
> >> >> We've had several discussions and attempts to introduce a solution
> >> >> to the problem. Thanks to oslo.db guys, we now have more or less
> >> >> clear view on the cause of the failures and how to easily fix them.
> >> >> The solution is to switch mysqldb to something eventlet aware. The
> >> >> best candidate is probably MySQL Connector module that is an
> >> >> official MySQL client for Python and that shows some (preliminary)
> >> >> good results in terms of performance.
> >> >
> >> > I've made additional testing, creating 2000 networks in parallel (10
> >> > thread workers) for both drivers and comparing results.
> >> >
> >> > With mysqldb: 215.81 sec
> >> > With mysql-connector: 88.66
> >> >
> >> > ~2.4 times performance boost, ok? ;)
> >> >
> >> > I think we should switch to that library *even* if we forget about all
> >> > the nasty deadlocks we experience now.
> >> >
> >> >>
> >> >> I've posted a Neutron spec for the switch to the new client in Juno
> >> >> at [1]. Ideally, switch is just a matter of several fixes to
> >> >> oslo.db that would enable full support for the new driver already
> >> >> supported by SQLAlchemy, plus 'connection' string modified in
> >> >> service configuration files, plus documentation updates to refer to
> >> >> the new official way to configure services for MySQL. The database
> >> >> code won't, ideally, require any major changes, though some
> >> >> adaptation for the new client library may be needed. That said,
> >> >> Neutron does not seem to require any changes, though it was
> >> >> revealed that there are some alembic migration rules in Keystone or
> >> >> Glance that need (trivial) modifications.
> >> >>
> >> >> You can see how trivial the switch can be achieved for a service
> >> >> based on example for Neutron [2].
> >> >>
> >> >> While this is a Neutron specific proposal, there is an obvious wish
> >> >> to switch to the new library globally throughout all the projects,
> >> >> to reduce devops burden, among other things. My vision is that,
> >> >> ideally, we switch all projects to the new library in Juno, though
> >> >> we still may leave several projects for K in case any issues arise,
> >> >> similar to the way projects switched to oslo.messaging during two
> >> >> cycles instead of one. Though looking at how easy Neutron can be
> >> >> switched to the new library, I wouldn't expect any issues that
> >> >> would postpone the switch till K.
> >> >>
> >> >> It was mentioned in comments to the spec proposal that there were
> >> >> some discussions at the latest summit around possible switch in
> >> >> context of Nova that revealed some concerns, though they do not
> >> >> seem to be documented anywhere. So if you know anything about it,
> >> >> please comment.
> >> >>
> >> >> So, we'd like to hear from other projects what's your take on that
> >> >> move, whether you see any issues or have concerns about it.
> >> >>
> >> >> Thanks for your comments, /Ihar
> >> >>
> >> >> [1]: https://review.openstack.org/#/c/104905/ [2]:
> >> >> https://review.openstack.org/#/c/105209/
> >> >>
> >> >> _______________________________________________ OpenStack-dev
> >> >> mailing list OpenStack-dev at lists.openstack.org
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> > -----BEGIN PGP SIGNATURE-----
> >> > Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> >> > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >> >
> >> > iQEcBAEBCgAGBQJTv9LHAAoJEC5aWaUY1u57d2cIAIAthLuM6qxN9fVjPwoICEae
> >> > oSOLvaDNPpZ+xBBqKI+2l5aFiBXSkHzgCfWGHEZB4e+5odAzt8r3Dg5eG/hwckGt
> >> > iZLPGLxcmvD5K0cRoSSPWkPC4KkOwKw0yQHl/JQarDcHQlLgO64jx3bzlB1LDxRu
> >> > R/Bvqo1SBo8g/cupWyxJXNViu9z7zAlvcHLRg4j/AfNTsTDZRrSgbMF2/gLTMvN2
> >> > FPtkjBvZq++zOva5G5/TySr1b3QRBFCG0uetVbcVF//90XOw+O++rUiDW1v7vkA9
> >> > OS2sCIXmx1i8kt9yuvs0h11MS8qfX9rSXREJXyPq6NDmePdQdKFsozMdTmqaDfU=
> >> > =JfiC
> >> > -----END PGP SIGNATURE-----
> >> >
> >> > _______________________________________________
> >> > OpenStack-dev mailing list
> >> > OpenStack-dev at lists.openstack.org
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140711/0e4d4501/attachment.html>