Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Some updates on the matter:

- - oslo-spec was approved with narrowed scope which is now 'enabled
mysqlconnector as an alternative in gate' instead of 'switch the
default db driver to mysqlconnector'. We'll revisit the switch part
the next cycle once we have the new driver running in gate and real
benchmarking is heavy-lifted.

- - there are several patches that are needed to make devstack and
tempest passing deployment and testing. Those are collected under the
hood of: https://review.openstack.org/#/c/114207/ Not much of them.

- - we'll need a new oslo.db release to bump versions (this is needed to
set raise_on_warnings=False for the new driver, which was incorrectly
set to True in sqlalchemy till very recently). This is expected to be
released this month (as per Roman Podoliaka).

- - once the corresponding patch for sqlalchemy-migrate is merged, we'll
also need a new version released for this.

- - on PyPI side, no news for now. The last time I've heard from Geert
(the maintainer of MySQL Connector for Python), he was working on
this. I suspect there are some legal considerations running inside
Oracle. I'll update once I know more about that.

- - once all the relevant patches land in affected projects and
devstack, I'm going to introduce a separate gate job to run against
mysqlconnector.

Cheers,
/Ihar

On 22/07/14 15:03, Ihar Hrachyshka wrote:
> FYI: I've moved the spec to oslo space since the switch is not
> really limited to neutron, and most of coding is to be done in
> oslo.db (though not much anyway).
> 
> New spec: https://review.openstack.org/#/c/108355/
> 
> On 09/07/14 13:17, Ihar Hrachyshka wrote:
>> Hi all,
> 
>> Multiple projects are suffering from db lock timeouts due to 
>> deadlocks deep in mysqldb library that we use to interact with 
>> mysql servers. In essence, the problem is due to missing
>> eventlet support in mysqldb module, meaning when a db lock is
>> encountered, the library does not yield to the next green thread,
>> allowing other threads to eventually unlock the grabbed lock, and
>> instead it just blocks the main thread, that eventually raises
>> timeout exception (OperationalError).
> 
>> The failed operation is not retried, leaving failing request not
>>  served. In Nova, there is a special retry mechanism for
>> deadlocks, though I think it's more a hack than a proper fix.
> 
>> Neutron is one of the projects that suffer from those timeout 
>> errors a lot. Partly it's due to lack of discipline in how we do 
>> nested calls in l3_db and ml2_plugin code, but that's not
>> something to change in foreseeable future, so we need to find
>> another solution that is applicable for Juno. Ideally, the
>> solution should be applicable for Icehouse too to allow
>> distributors to resolve existing deadlocks without waiting for
>> Juno.
> 
>> We've had several discussions and attempts to introduce a
>> solution to the problem. Thanks to oslo.db guys, we now have more
>> or less clear view on the cause of the failures and how to easily
>> fix them. The solution is to switch mysqldb to something eventlet
>> aware. The best candidate is probably MySQL Connector module that
>> is an official MySQL client for Python and that shows some
>> (preliminary) good results in terms of performance.
> 
>> I've posted a Neutron spec for the switch to the new client in
>> Juno at [1]. Ideally, switch is just a matter of several fixes
>> to oslo.db that would enable full support for the new driver
>> already supported by SQLAlchemy, plus 'connection' string
>> modified in service configuration files, plus documentation
>> updates to refer to the new official way to configure services
>> for MySQL. The database code won't, ideally, require any major
>> changes, though some adaptation for the new client library may be
>> needed. That said, Neutron does not seem to require any changes,
>> though it was revealed that there are some alembic migration
>> rules in Keystone or Glance that need (trivial) modifications.
> 
>> You can see how trivial the switch can be achieved for a service 
>> based on example for Neutron [2].
> 
>> While this is a Neutron specific proposal, there is an obvious
>> wish to switch to the new library globally throughout all the
>> projects, to reduce devops burden, among other things. My vision
>> is that, ideally, we switch all projects to the new library in
>> Juno, though we still may leave several projects for K in case
>> any issues arise, similar to the way projects switched to
>> oslo.messaging during two cycles instead of one. Though looking
>> at how easy Neutron can be switched to the new library, I
>> wouldn't expect any issues that would postpone the switch till
>> K.
> 
>> It was mentioned in comments to the spec proposal that there
>> were some discussions at the latest summit around possible switch
>> in context of Nova that revealed some concerns, though they do
>> not seem to be documented anywhere. So if you know anything about
>> it, please comment.
> 
>> So, we'd like to hear from other projects what's your take on
>> that move, whether you see any issues or have concerns about it.
> 
>> Thanks for your comments, /Ihar
> 
>> [1]: https://review.openstack.org/#/c/104905/ [2]: 
>> https://review.openstack.org/#/c/105209/
> 
>> _______________________________________________ OpenStack-dev 
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>> 
> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJT7gsfAAoJEC5aWaUY1u57hsMIANWwJ+Ai4jh2SlT8dSy3oUD1
yyYBnEJKdxp69Oro/qVGULbCizZ82lwHw5OTPGjFIuxpgzFbAneBTMwp2MqZvDwE
gbEQRVJ8wIc+isqWy7hW8Vd/bRuzcm0tVDH8mHow5ujKLeT1T4OSYBK5Wx80OMta
WFpIUVnSWUBxT6jnYNMcpzK8N6JbM9TJie4fycKsDFGsCAtoIzi2HdOUafhFN9MY
K08iItlnOKJpAJsNqk5VUtxIS6qsI5TO1gedMuRQcWbDL9czTyNvxUIOZn404hNx
c89ZSrFnJEQ7iwmd4dWv6rDumv8JjQAJTg6jDri6oLzE10AS1vrFdf8ac/v/xOU=
=vGYx
-----END PGP SIGNATURE-----