Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 09/07/14 15:40, Sean Dague wrote:
> On 07/09/2014 09:00 AM, Roman Podoliaka wrote:
>> Hi Ihar,
>> 
>> AFAIU, the switch is a matter of pip install + specifying the
>> correct db URI in the config files. I'm not sure why you are
>> filing a spec in Neutron project. IMHO, this has nothing to do
>> with projects, but rather a purely deployment question. E.g.
>> don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of
>> OpenStack right now?
>> 
>> I think what you really want is to change the defaults we test in
>> the gate, which is a different problem.
> 
> Because this is really a *new* driver. As you can see by the
> attempted run, it doesn't work with alembic given the definitions
> that neutron has. So it's not like this is currently compatible
> with OpenStack code.

Well, to fix that, you just need to specify raise_on_warnings=False
for connection (it's default for mysqldb but not mysql-connector).
I've done it in devstack patch for now, but probably it belongs to
oslo.db.

> 
>> 
>> Thanks, Roman
>> 
>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka
>> <ihrachys at redhat.com> wrote: Hi all,
>> 
>> Multiple projects are suffering from db lock timeouts due to
>> deadlocks deep in mysqldb library that we use to interact with
>> mysql servers. In essence, the problem is due to missing eventlet
>> support in mysqldb module, meaning when a db lock is encountered,
>> the library does not yield to the next green thread, allowing
>> other threads to eventually unlock the grabbed lock, and instead
>> it just blocks the main thread, that eventually raises timeout
>> exception (OperationalError).
>> 
>> The failed operation is not retried, leaving failing request not 
>> served. In Nova, there is a special retry mechanism for
>> deadlocks, though I think it's more a hack than a proper fix.
>> 
>> Neutron is one of the projects that suffer from those timeout
>> errors a lot. Partly it's due to lack of discipline in how we do
>> nested calls in l3_db and ml2_plugin code, but that's not
>> something to change in foreseeable future, so we need to find
>> another solution that is applicable for Juno. Ideally, the
>> solution should be applicable for Icehouse too to allow
>> distributors to resolve existing deadlocks without waiting for
>> Juno.
>> 
>> We've had several discussions and attempts to introduce a
>> solution to the problem. Thanks to oslo.db guys, we now have more
>> or less clear view on the cause of the failures and how to easily
>> fix them. The solution is to switch mysqldb to something eventlet
>> aware. The best candidate is probably MySQL Connector module that
>> is an official MySQL client for Python and that shows some
>> (preliminary) good results in terms of performance.
>> 
>> I've posted a Neutron spec for the switch to the new client in
>> Juno at [1]. Ideally, switch is just a matter of several fixes to
>> oslo.db that would enable full support for the new driver already
>> supported by SQLAlchemy, plus 'connection' string modified in
>> service configuration files, plus documentation updates to refer
>> to the new official way to configure services for MySQL. The
>> database code won't, ideally, require any major changes, though
>> some adaptation for the new client library may be needed. That
>> said, Neutron does not seem to require any changes, though it was
>> revealed that there are some alembic migration rules in Keystone
>> or Glance that need (trivial) modifications.
>> 
>> You can see how trivial the switch can be achieved for a service
>> based on example for Neutron [2].
>> 
>> While this is a Neutron specific proposal, there is an obvious
>> wish to switch to the new library globally throughout all the
>> projects, to reduce devops burden, among other things. My vision
>> is that, ideally, we switch all projects to the new library in
>> Juno, though we still may leave several projects for K in case
>> any issues arise, similar to the way projects switched to
>> oslo.messaging during two cycles instead of one. Though looking
>> at how easy Neutron can be switched to the new library, I
>> wouldn't expect any issues that would postpone the switch till
>> K.
>> 
>> It was mentioned in comments to the spec proposal that there were
>> some discussions at the latest summit around possible switch in
>> context of Nova that revealed some concerns, though they do not
>> seem to be documented anywhere. So if you know anything about it,
>> please comment.
>> 
>> So, we'd like to hear from other projects what's your take on
>> that move, whether you see any issues or have concerns about it.
>> 
>> Thanks for your comments, /Ihar
>> 
>> [1]: https://review.openstack.org/#/c/104905/ [2]:
>> https://review.openstack.org/#/c/105209/
>>> 
>>> _______________________________________________ OpenStack-dev
>>> mailing list OpenStack-dev at lists.openstack.org 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>> 
_______________________________________________
>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>> 
> 
> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTvVUaAAoJEC5aWaUY1u57likH/33btpNhAm8OjkQUj0ilBHFb
HKxJDBVoWa+ioXrgYMcrwCZp8PAsE7c6jYRZW1Aaa4Ge7KeTXoAhYojF4T8p1HhQ
Q3nAbxSdG1Hlq6m+6NzIGoAPxsSCgoo4hNWtyGfoBLlbpufr8QZMvXhKLObUvrZF
wto3oAGXAnYu6XKjP4Y7zFMPz+t4HwIrOYNlRBlkmH+/7dzhtnVbDiJ6AsQiJ5lF
+6yC5iaQEFl13TJ+5dQgTIqP5jwk9uFr/nhakj+4hbZWpAjXf0fqY+jNaUL3dRDh
hFuy4f0sRlCgSiq7x6Rt9NO5UYnwSd7yHLjZBDyzCdOQrdweDSOH6XB26K3yJr0=
=kSGB
-----END PGP SIGNATURE-----