Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 09/07/14 16:59, Roman Podoliaka wrote:
> Hi all,
> 
> Not sure what issues you are talking about, but I just replaced 
> "mysql" with "mysql+mysqlconnector" in my db connection string  in 
> neutron.conf and "neutron-db-manage upgrade head" worked like a
> charm for an empty schema.

Have you enabled metering plugin as devstack in gate does? That's what
fails. If it's not enabled, everything succeeds, as you've mentioned.
If it's enabled, sqlconnector driver raise error on warning due to
'CREATE TABLE IF NOT EXISTS' statement is used. The statement raises
warning because some backends do not support it. Though I don't think
there's a way to rewrite the migration rule without using the
statement: using alembic.op.create_table does not generate IF NOT
EXISTS, and we need it for offline migration where we can't check
table presence in runtime.

> 
> Ihar, could please elaborate on what changes to oslo.db are
> needed? (as an oslo.db developer I'm very interested in this part
> :) )

You can find some patches at Angus dashboard [1]. As for the change I
refer to, we need to set raise_on_warnings for mysqlconnector dialect
in create_engine().

[1]:
https://review.openstack.org/#/q/owner:%22Angus+Lees+%253Cgus%2540inodes.org%253E%22+status:open,n,z

> 
> Thanks, Roman
> 
> On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka
> <ihrachys at redhat.com> wrote: On 09/07/14 15:40, Sean Dague wrote:
>>>> On 07/09/2014 09:00 AM, Roman Podoliaka wrote:
>>>>> Hi Ihar,
>>>>> 
>>>>> AFAIU, the switch is a matter of pip install + specifying
>>>>> the correct db URI in the config files. I'm not sure why
>>>>> you are filing a spec in Neutron project. IMHO, this has
>>>>> nothing to do with projects, but rather a purely deployment
>>>>> question. E.g. don't we have PostgreSQL+psycopg2 or
>>>>> MySQL+pymysql deployments of OpenStack right now?
>>>>> 
>>>>> I think what you really want is to change the defaults we
>>>>> test in the gate, which is a different problem.
>>>> 
>>>> Because this is really a *new* driver. As you can see by the 
>>>> attempted run, it doesn't work with alembic given the
>>>> definitions that neutron has. So it's not like this is
>>>> currently compatible with OpenStack code.
> 
> Well, to fix that, you just need to specify
> raise_on_warnings=False for connection (it's default for mysqldb
> but not mysql-connector). I've done it in devstack patch for now,
> but probably it belongs to oslo.db.
> 
>>>> 
>>>>> 
>>>>> Thanks, Roman
>>>>> 
>>>>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka 
>>>>> <ihrachys at redhat.com> wrote: Hi all,
>>>>> 
>>>>> Multiple projects are suffering from db lock timeouts due
>>>>> to deadlocks deep in mysqldb library that we use to
>>>>> interact with mysql servers. In essence, the problem is due
>>>>> to missing eventlet support in mysqldb module, meaning when
>>>>> a db lock is encountered, the library does not yield to the
>>>>> next green thread, allowing other threads to eventually
>>>>> unlock the grabbed lock, and instead it just blocks the
>>>>> main thread, that eventually raises timeout exception
>>>>> (OperationalError).
>>>>> 
>>>>> The failed operation is not retried, leaving failing
>>>>> request not served. In Nova, there is a special retry
>>>>> mechanism for deadlocks, though I think it's more a hack
>>>>> than a proper fix.
>>>>> 
>>>>> Neutron is one of the projects that suffer from those
>>>>> timeout errors a lot. Partly it's due to lack of discipline
>>>>> in how we do nested calls in l3_db and ml2_plugin code, but
>>>>> that's not something to change in foreseeable future, so we
>>>>> need to find another solution that is applicable for Juno.
>>>>> Ideally, the solution should be applicable for Icehouse too
>>>>> to allow distributors to resolve existing deadlocks without
>>>>> waiting for Juno.
>>>>> 
>>>>> We've had several discussions and attempts to introduce a 
>>>>> solution to the problem. Thanks to oslo.db guys, we now
>>>>> have more or less clear view on the cause of the failures
>>>>> and how to easily fix them. The solution is to switch
>>>>> mysqldb to something eventlet aware. The best candidate is
>>>>> probably MySQL Connector module that is an official MySQL
>>>>> client for Python and that shows some (preliminary) good
>>>>> results in terms of performance.
>>>>> 
>>>>> I've posted a Neutron spec for the switch to the new client
>>>>> in Juno at [1]. Ideally, switch is just a matter of several
>>>>> fixes to oslo.db that would enable full support for the new
>>>>> driver already supported by SQLAlchemy, plus 'connection'
>>>>> string modified in service configuration files, plus
>>>>> documentation updates to refer to the new official way to
>>>>> configure services for MySQL. The database code won't,
>>>>> ideally, require any major changes, though some adaptation
>>>>> for the new client library may be needed. That said,
>>>>> Neutron does not seem to require any changes, though it
>>>>> was revealed that there are some alembic migration rules in
>>>>> Keystone or Glance that need (trivial) modifications.
>>>>> 
>>>>> You can see how trivial the switch can be achieved for a
>>>>> service based on example for Neutron [2].
>>>>> 
>>>>> While this is a Neutron specific proposal, there is an
>>>>> obvious wish to switch to the new library globally
>>>>> throughout all the projects, to reduce devops burden, among
>>>>> other things. My vision is that, ideally, we switch all
>>>>> projects to the new library in Juno, though we still may
>>>>> leave several projects for K in case any issues arise,
>>>>> similar to the way projects switched to oslo.messaging
>>>>> during two cycles instead of one. Though looking at how
>>>>> easy Neutron can be switched to the new library, I wouldn't
>>>>> expect any issues that would postpone the switch till K.
>>>>> 
>>>>> It was mentioned in comments to the spec proposal that
>>>>> there were some discussions at the latest summit around
>>>>> possible switch in context of Nova that revealed some
>>>>> concerns, though they do not seem to be documented
>>>>> anywhere. So if you know anything about it, please
>>>>> comment.
>>>>> 
>>>>> So, we'd like to hear from other projects what's your take
>>>>> on that move, whether you see any issues or have concerns
>>>>> about it.
>>>>> 
>>>>> Thanks for your comments, /Ihar
>>>>> 
>>>>> [1]: https://review.openstack.org/#/c/104905/ [2]: 
>>>>> https://review.openstack.org/#/c/105209/
>>>>>> 
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>>
>
>>>>>> 
_______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org 
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>> 
_______________________________________________ OpenStack-dev
>>>> mailing list OpenStack-dev at lists.openstack.org 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>
>>
>>>> 
_______________________________________________
>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTvV17AAoJEC5aWaUY1u57A04IAIm6M1puMLE4XHrTtQI9nVNH
RPDAhZ/v38bZ61kIOYIFR2wEYddQQ1xtP6FVKf+aUZ2PPM5S5AHzCy9HGn024aGA
IaPGReFaxyL5z+eV60AQabrx5r1K0FMjED3WE7MNR8dkypl7lGWGBBtI9/L6f0ag
0UoIUTheQj0caJXNwlsKAlKA5nVmjAUbzjzC0sJCYDyYxv/A1svWPIdIMkYPspO/
B6sg38m6SCx7QvANRnNXSLL3aw/zZ26iPoewbQcpSZtiLI9GvzuHhlglem5Ey3Is
+FauDqtPv6HM20aEV2GRS68enDP/eZ6GjI1pKtAL5fVtuy4naH0QfFsGgS+ayPw=
=3C7C
-----END PGP SIGNATURE-----