Open Stack

Hi Ihar,

AFAIU, the switch is a matter of pip install + specifying the correct
db URI in the config files. I'm not sure why you are filing a spec in
Neutron project. IMHO, this has nothing to do with projects, but
rather a purely deployment question. E.g. don't we have
PostgreSQL+psycopg2 or MySQL+pymysql deployments of OpenStack right
now?

I think what you really want is to change the defaults we test in the
gate, which is a different problem.

Thanks,
Roman

On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Hi all,
>
> Multiple projects are suffering from db lock timeouts due to deadlocks
> deep in mysqldb library that we use to interact with mysql servers. In
> essence, the problem is due to missing eventlet support in mysqldb
> module, meaning when a db lock is encountered, the library does not
> yield to the next green thread, allowing other threads to eventually
> unlock the grabbed lock, and instead it just blocks the main thread,
> that eventually raises timeout exception (OperationalError).
>
> The failed operation is not retried, leaving failing request not
> served. In Nova, there is a special retry mechanism for deadlocks,
> though I think it's more a hack than a proper fix.
>
> Neutron is one of the projects that suffer from those timeout errors a
> lot. Partly it's due to lack of discipline in how we do nested calls
> in l3_db and ml2_plugin code, but that's not something to change in
> foreseeable future, so we need to find another solution that is
> applicable for Juno. Ideally, the solution should be applicable for
> Icehouse too to allow distributors to resolve existing deadlocks
> without waiting for Juno.
>
> We've had several discussions and attempts to introduce a solution to
> the problem. Thanks to oslo.db guys, we now have more or less clear
> view on the cause of the failures and how to easily fix them. The
> solution is to switch mysqldb to something eventlet aware. The best
> candidate is probably MySQL Connector module that is an official MySQL
> client for Python and that shows some (preliminary) good results in
> terms of performance.
>
> I've posted a Neutron spec for the switch to the new client in Juno at
> [1]. Ideally, switch is just a matter of several fixes to oslo.db that
> would enable full support for the new driver already supported by
> SQLAlchemy, plus 'connection' string modified in service configuration
> files, plus documentation updates to refer to the new official way to
> configure services for MySQL. The database code won't, ideally,
> require any major changes, though some adaptation for the new client
> library may be needed. That said, Neutron does not seem to require any
> changes, though it was revealed that there are some alembic migration
> rules in Keystone or Glance that need (trivial) modifications.
>
> You can see how trivial the switch can be achieved for a service based
> on example for Neutron [2].
>
> While this is a Neutron specific proposal, there is an obvious wish to
> switch to the new library globally throughout all the projects, to
> reduce devops burden, among other things. My vision is that, ideally,
> we switch all projects to the new library in Juno, though we still may
> leave several projects for K in case any issues arise, similar to the
> way projects switched to oslo.messaging during two cycles instead of
> one. Though looking at how easy Neutron can be switched to the new
> library, I wouldn't expect any issues that would postpone the switch
> till K.
>
> It was mentioned in comments to the spec proposal that there were some
> discussions at the latest summit around possible switch in context of
> Nova that revealed some concerns, though they do not seem to be
> documented anywhere. So if you know anything about it, please comment.
>
> So, we'd like to hear from other projects what's your take on that
> move, whether you see any issues or have concerns about it.
>
> Thanks for your comments,
> /Ihar
>
> [1]: https://review.openstack.org/#/c/104905/
> [2]: https://review.openstack.org/#/c/105209/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBCgAGBQJTvSS+AAoJEC5aWaUY1u57uk0IAMNIW4e59fU8uiF7eg8KwIgU
> 5vjzDP4GX454Oxm0h5q3Olc0nXIeB6zSBGDoomgLk9+4AS250ihGRA/V10iDEJTF
> yubcvknep/ZfF+lKkmgBA3WNXJgTXffXeN2bimC6t5zJA+8Cmn3lUPu0djt0GWs7
> AktufkPbVj7ZauN6w9OpW4AnoZX1fARvynCilTuHYu+lb8nQ/Hatqu5dgqdeDyRp
> jodLoN1ow3VYR7Cq5jocqhw719aiKLJdlUgWVHNL5A5oTR1Uxu0AdleeUzXVUvFm
> +EQO0Xe+slMSBgzBsgPJAiX0Vkc6kfJdFHR571QUWCXaXF1nUEIkgra/7j+0Uqs=
> =bgds
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev