Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 09/07/14 15:00, Roman Podoliaka wrote:
> Hi Ihar,
> 
> AFAIU, the switch is a matter of pip install + specifying the
> correct db URI in the config files. I'm not sure why you are filing
> a spec in Neutron project. IMHO, this has nothing to do with
> projects, but rather a purely deployment question. E.g. don't we
> have PostgreSQL+psycopg2 or MySQL+pymysql deployments of OpenStack
> right now?

The issue was raised in Neutron because it suffers a lot from those
deadlocks, and because initially I saw the switch as local to this
specific project, that would lead other projects by example and not as
an enforced rule. I would be glad to put the spec in some other,
better place. Do you know any?

I don't know whether we have other MySQL deployments not using
MySQLdb; if so, they are not configured as per official documentation.

See:
- -
http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-database-controller.html
- -
http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-database-node.html
- -
http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron-ml2-controller-node.html

If we want people to use a new module, we need to update the docs not
to refer to MySQLdb. Or at least inform them that deadlocks may occur
when using the library [and I think this is enough to just remove any
references to the library from the documentation.]

> 
> I think what you really want is to change the defaults we test in
> the gate, which is a different problem.

It's lots of different things to do:

- - some work to be tracked in oslo.db (and old db code from incubator?);
- - update neutron code if needed (though preliminary testing didn't
reveal any specific changes for this specific project though, while
others may need trivial updates);
- - switch defaults in devstack;
- - update documentation to refer to the new library.

> 
> Thanks, Roman
> 
> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka
> <ihrachys at redhat.com> wrote: Hi all,
> 
> Multiple projects are suffering from db lock timeouts due to
> deadlocks deep in mysqldb library that we use to interact with
> mysql servers. In essence, the problem is due to missing eventlet
> support in mysqldb module, meaning when a db lock is encountered,
> the library does not yield to the next green thread, allowing other
> threads to eventually unlock the grabbed lock, and instead it just
> blocks the main thread, that eventually raises timeout exception
> (OperationalError).
> 
> The failed operation is not retried, leaving failing request not 
> served. In Nova, there is a special retry mechanism for deadlocks, 
> though I think it's more a hack than a proper fix.
> 
> Neutron is one of the projects that suffer from those timeout
> errors a lot. Partly it's due to lack of discipline in how we do
> nested calls in l3_db and ml2_plugin code, but that's not something
> to change in foreseeable future, so we need to find another
> solution that is applicable for Juno. Ideally, the solution should
> be applicable for Icehouse too to allow distributors to resolve
> existing deadlocks without waiting for Juno.
> 
> We've had several discussions and attempts to introduce a solution
> to the problem. Thanks to oslo.db guys, we now have more or less
> clear view on the cause of the failures and how to easily fix them.
> The solution is to switch mysqldb to something eventlet aware. The
> best candidate is probably MySQL Connector module that is an
> official MySQL client for Python and that shows some (preliminary)
> good results in terms of performance.
> 
> I've posted a Neutron spec for the switch to the new client in Juno
> at [1]. Ideally, switch is just a matter of several fixes to
> oslo.db that would enable full support for the new driver already
> supported by SQLAlchemy, plus 'connection' string modified in
> service configuration files, plus documentation updates to refer to
> the new official way to configure services for MySQL. The database
> code won't, ideally, require any major changes, though some
> adaptation for the new client library may be needed. That said,
> Neutron does not seem to require any changes, though it was
> revealed that there are some alembic migration rules in Keystone or
> Glance that need (trivial) modifications.
> 
> You can see how trivial the switch can be achieved for a service
> based on example for Neutron [2].
> 
> While this is a Neutron specific proposal, there is an obvious wish
> to switch to the new library globally throughout all the projects,
> to reduce devops burden, among other things. My vision is that,
> ideally, we switch all projects to the new library in Juno, though
> we still may leave several projects for K in case any issues arise,
> similar to the way projects switched to oslo.messaging during two
> cycles instead of one. Though looking at how easy Neutron can be
> switched to the new library, I wouldn't expect any issues that
> would postpone the switch till K.
> 
> It was mentioned in comments to the spec proposal that there were
> some discussions at the latest summit around possible switch in
> context of Nova that revealed some concerns, though they do not
> seem to be documented anywhere. So if you know anything about it,
> please comment.
> 
> So, we'd like to hear from other projects what's your take on that 
> move, whether you see any issues or have concerns about it.
> 
> Thanks for your comments, /Ihar
> 
> [1]: https://review.openstack.org/#/c/104905/ [2]:
> https://review.openstack.org/#/c/105209/
>> 
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTvUEAAAoJEC5aWaUY1u57cHUIAOq+ij/rNDJv1PWmYTlaU8EB
PJNdwEos9Kl2Zj37VYfJllg3oSUG8ueLXBuBXJldjgWTZZS0psNz7tmcUjTox0f/
wvpQfiFpJAXMSqRg9AYqg9dYPRAoM6PN6sSgFuNSLi1mKmRabyLafG2G5pXxWd5H
C+Ku+DNpSi6qjqFnaVr7BC9Ya8wS0uhi9LGKzee/m+HjKpmJoEPO6dC1+LoDvXY6
5XFTKXk36e5WMvka+j6ZP7eS4BCfml3Gu6NjdoskLd1bEqAcIeABBO+CSRYwF1/G
m2vpXX4yUG2GF5QsA4cAwvXYNo91UYtpAMhiMVzPFGQyV8rCeRYuirdd9d5Fu2E=
=YvJj
-----END PGP SIGNATURE-----