Open Stack

Hi all,

Not sure what issues you are talking about, but I just replaced
"mysql" with "mysql+mysqlconnector" in my db connection string  in
neutron.conf and "neutron-db-manage upgrade head" worked like a charm
for an empty schema.

Ihar, could please elaborate on what changes to oslo.db are needed?
(as an oslo.db developer I'm very interested in this part :) )

Thanks,
Roman

On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 09/07/14 15:40, Sean Dague wrote:
>> On 07/09/2014 09:00 AM, Roman Podoliaka wrote:
>>> Hi Ihar,
>>>
>>> AFAIU, the switch is a matter of pip install + specifying the
>>> correct db URI in the config files. I'm not sure why you are
>>> filing a spec in Neutron project. IMHO, this has nothing to do
>>> with projects, but rather a purely deployment question. E.g.
>>> don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of
>>> OpenStack right now?
>>>
>>> I think what you really want is to change the defaults we test in
>>> the gate, which is a different problem.
>>
>> Because this is really a *new* driver. As you can see by the
>> attempted run, it doesn't work with alembic given the definitions
>> that neutron has. So it's not like this is currently compatible
>> with OpenStack code.
>
> Well, to fix that, you just need to specify raise_on_warnings=False
> for connection (it's default for mysqldb but not mysql-connector).
> I've done it in devstack patch for now, but probably it belongs to
> oslo.db.
>
>>
>>>
>>> Thanks, Roman
>>>
>>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka
>>> <ihrachys at redhat.com> wrote: Hi all,
>>>
>>> Multiple projects are suffering from db lock timeouts due to
>>> deadlocks deep in mysqldb library that we use to interact with
>>> mysql servers. In essence, the problem is due to missing eventlet
>>> support in mysqldb module, meaning when a db lock is encountered,
>>> the library does not yield to the next green thread, allowing
>>> other threads to eventually unlock the grabbed lock, and instead
>>> it just blocks the main thread, that eventually raises timeout
>>> exception (OperationalError).
>>>
>>> The failed operation is not retried, leaving failing request not
>>> served. In Nova, there is a special retry mechanism for
>>> deadlocks, though I think it's more a hack than a proper fix.
>>>
>>> Neutron is one of the projects that suffer from those timeout
>>> errors a lot. Partly it's due to lack of discipline in how we do
>>> nested calls in l3_db and ml2_plugin code, but that's not
>>> something to change in foreseeable future, so we need to find
>>> another solution that is applicable for Juno. Ideally, the
>>> solution should be applicable for Icehouse too to allow
>>> distributors to resolve existing deadlocks without waiting for
>>> Juno.
>>>
>>> We've had several discussions and attempts to introduce a
>>> solution to the problem. Thanks to oslo.db guys, we now have more
>>> or less clear view on the cause of the failures and how to easily
>>> fix them. The solution is to switch mysqldb to something eventlet
>>> aware. The best candidate is probably MySQL Connector module that
>>> is an official MySQL client for Python and that shows some
>>> (preliminary) good results in terms of performance.
>>>
>>> I've posted a Neutron spec for the switch to the new client in
>>> Juno at [1]. Ideally, switch is just a matter of several fixes to
>>> oslo.db that would enable full support for the new driver already
>>> supported by SQLAlchemy, plus 'connection' string modified in
>>> service configuration files, plus documentation updates to refer
>>> to the new official way to configure services for MySQL. The
>>> database code won't, ideally, require any major changes, though
>>> some adaptation for the new client library may be needed. That
>>> said, Neutron does not seem to require any changes, though it was
>>> revealed that there are some alembic migration rules in Keystone
>>> or Glance that need (trivial) modifications.
>>>
>>> You can see how trivial the switch can be achieved for a service
>>> based on example for Neutron [2].
>>>
>>> While this is a Neutron specific proposal, there is an obvious
>>> wish to switch to the new library globally throughout all the
>>> projects, to reduce devops burden, among other things. My vision
>>> is that, ideally, we switch all projects to the new library in
>>> Juno, though we still may leave several projects for K in case
>>> any issues arise, similar to the way projects switched to
>>> oslo.messaging during two cycles instead of one. Though looking
>>> at how easy Neutron can be switched to the new library, I
>>> wouldn't expect any issues that would postpone the switch till
>>> K.
>>>
>>> It was mentioned in comments to the spec proposal that there were
>>> some discussions at the latest summit around possible switch in
>>> context of Nova that revealed some concerns, though they do not
>>> seem to be documented anywhere. So if you know anything about it,
>>> please comment.
>>>
>>> So, we'd like to hear from other projects what's your take on
>>> that move, whether you see any issues or have concerns about it.
>>>
>>> Thanks for your comments, /Ihar
>>>
>>> [1]: https://review.openstack.org/#/c/104905/ [2]:
>>> https://review.openstack.org/#/c/105209/
>>>>
>>>> _______________________________________________ OpenStack-dev
>>>> mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>>
> _______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>>
>>
>>
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBCgAGBQJTvVUaAAoJEC5aWaUY1u57likH/33btpNhAm8OjkQUj0ilBHFb
> HKxJDBVoWa+ioXrgYMcrwCZp8PAsE7c6jYRZW1Aaa4Ge7KeTXoAhYojF4T8p1HhQ
> Q3nAbxSdG1Hlq6m+6NzIGoAPxsSCgoo4hNWtyGfoBLlbpufr8QZMvXhKLObUvrZF
> wto3oAGXAnYu6XKjP4Y7zFMPz+t4HwIrOYNlRBlkmH+/7dzhtnVbDiJ6AsQiJ5lF
> +6yC5iaQEFl13TJ+5dQgTIqP5jwk9uFr/nhakj+4hbZWpAjXf0fqY+jNaUL3dRDh
> hFuy4f0sRlCgSiq7x6Rt9NO5UYnwSd7yHLjZBDyzCdOQrdweDSOH6XB26K3yJr0=
> =kSGB
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev