Open Stack

On Aug 15, 2014, at 9:29 AM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:

> Signed PGP part
> Some updates on the matter:
> 
> - oslo-spec was approved with narrowed scope which is now 'enabled
> mysqlconnector as an alternative in gate' instead of 'switch the
> default db driver to mysqlconnector'. We'll revisit the switch part
> the next cycle once we have the new driver running in gate and real
> benchmarking is heavy-lifted.
> 
> - there are several patches that are needed to make devstack and
> tempest passing deployment and testing. Those are collected under the
> hood of: https://review.openstack.org/#/c/114207/ Not much of them.
> 
> - we'll need a new oslo.db release to bump versions (this is needed to
> set raise_on_warnings=False for the new driver, which was incorrectly
> set to True in sqlalchemy till very recently). This is expected to be
> released this month (as per Roman Podoliaka).

This release is currently blocked on landing some changes in projects using the library so they don’t break when the new version starts using different exception classes. We’re tracking that work in https://etherpad.openstack.org/p/sqla_exceptions_caught

It looks like we’re down to 2 patches, one for cinder (https://review.openstack.org/#/c/111760/) and one for glance (https://review.openstack.org/#/c/109655). Roman, can you verify that those are the only two projects that need changes for the exception issue?

> 
> - once the corresponding patch for sqlalchemy-migrate is merged, we'll
> also need a new version released for this.
> 
> - on PyPI side, no news for now. The last time I've heard from Geert
> (the maintainer of MySQL Connector for Python), he was working on
> this. I suspect there are some legal considerations running inside
> Oracle. I'll update once I know more about that.

If we don’t have the new package on PyPI, how do we plan to include it in the gate? Are there options to allow an exception, or to make the mirroring software download it anyway?

Doug

> 
> - once all the relevant patches land in affected projects and
> devstack, I'm going to introduce a separate gate job to run against
> mysqlconnector.
> 
> Cheers,
> /Ihar
> 
> On 22/07/14 15:03, Ihar Hrachyshka wrote:
> > FYI: I've moved the spec to oslo space since the switch is not
> > really limited to neutron, and most of coding is to be done in
> > oslo.db (though not much anyway).
> >
> > New spec: https://review.openstack.org/#/c/108355/
> >
> > On 09/07/14 13:17, Ihar Hrachyshka wrote:
> >> Hi all,
> >
> >> Multiple projects are suffering from db lock timeouts due to
> >> deadlocks deep in mysqldb library that we use to interact with
> >> mysql servers. In essence, the problem is due to missing
> >> eventlet support in mysqldb module, meaning when a db lock is
> >> encountered, the library does not yield to the next green thread,
> >> allowing other threads to eventually unlock the grabbed lock, and
> >> instead it just blocks the main thread, that eventually raises
> >> timeout exception (OperationalError).
> >
> >> The failed operation is not retried, leaving failing request not
> >>  served. In Nova, there is a special retry mechanism for
> >> deadlocks, though I think it's more a hack than a proper fix.
> >
> >> Neutron is one of the projects that suffer from those timeout
> >> errors a lot. Partly it's due to lack of discipline in how we do
> >> nested calls in l3_db and ml2_plugin code, but that's not
> >> something to change in foreseeable future, so we need to find
> >> another solution that is applicable for Juno. Ideally, the
> >> solution should be applicable for Icehouse too to allow
> >> distributors to resolve existing deadlocks without waiting for
> >> Juno.
> >
> >> We've had several discussions and attempts to introduce a
> >> solution to the problem. Thanks to oslo.db guys, we now have more
> >> or less clear view on the cause of the failures and how to easily
> >> fix them. The solution is to switch mysqldb to something eventlet
> >> aware. The best candidate is probably MySQL Connector module that
> >> is an official MySQL client for Python and that shows some
> >> (preliminary) good results in terms of performance.
> >
> >> I've posted a Neutron spec for the switch to the new client in
> >> Juno at [1]. Ideally, switch is just a matter of several fixes
> >> to oslo.db that would enable full support for the new driver
> >> already supported by SQLAlchemy, plus 'connection' string
> >> modified in service configuration files, plus documentation
> >> updates to refer to the new official way to configure services
> >> for MySQL. The database code won't, ideally, require any major
> >> changes, though some adaptation for the new client library may be
> >> needed. That said, Neutron does not seem to require any changes,
> >> though it was revealed that there are some alembic migration
> >> rules in Keystone or Glance that need (trivial) modifications.
> >
> >> You can see how trivial the switch can be achieved for a service
> >> based on example for Neutron [2].
> >
> >> While this is a Neutron specific proposal, there is an obvious
> >> wish to switch to the new library globally throughout all the
> >> projects, to reduce devops burden, among other things. My vision
> >> is that, ideally, we switch all projects to the new library in
> >> Juno, though we still may leave several projects for K in case
> >> any issues arise, similar to the way projects switched to
> >> oslo.messaging during two cycles instead of one. Though looking
> >> at how easy Neutron can be switched to the new library, I
> >> wouldn't expect any issues that would postpone the switch till
> >> K.
> >
> >> It was mentioned in comments to the spec proposal that there
> >> were some discussions at the latest summit around possible switch
> >> in context of Nova that revealed some concerns, though they do
> >> not seem to be documented anywhere. So if you know anything about
> >> it, please comment.
> >
> >> So, we'd like to hear from other projects what's your take on
> >> that move, whether you see any issues or have concerns about it.
> >
> >> Thanks for your comments, /Ihar
> >
> >> [1]: https://review.openstack.org/#/c/104905/ [2]:
> >> https://review.openstack.org/#/c/105209/
> >
> >> _______________________________________________ OpenStack-dev
> >> mailing list OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >>
> >
> > _______________________________________________ OpenStack-dev
> > mailing list OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev