<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
On 7/9/14, 10:59 AM, Roman Podoliaka wrote:<br>
<span style="white-space: pre;">> Hi all,<br>
><br>
> Not sure what issues you are talking about, but I just
replaced<br>
> "mysql" with "mysql+mysqlconnector" in my db connection
string in<br>
> neutron.conf and "neutron-db-manage upgrade head" worked like
a charm<br>
> for an empty schema.<br>
><br>
> Ihar, could please elaborate on what changes to oslo.db are
needed?<br>
> (as an oslo.db developer I'm very interested in this part :)
)<br>
><br>
> Thanks,<br>
> Roman<br>
><br>
> On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka
<a class="moz-txt-link-rfc2396E" href="mailto:ihrachys@redhat.com"><ihrachys@redhat.com></a> wrote:</span><br>
<blockquote type="cite">On 09/07/14 15:40, Sean Dague wrote:<br>
>>> On 07/09/2014 09:00 AM, Roman Podoliaka wrote:<br>
>>>> Hi Ihar,<br>
>>>><br>
>>>> AFAIU, the switch is a matter of pip install +
specifying the<br>
>>>> correct db URI in the config files. I'm not sure
why you are<br>
>>>> filing a spec in Neutron project. IMHO, this has
nothing to do<br>
>>>> with projects, but rather a purely deployment
question. E.g.<br>
>>>> don't we have PostgreSQL+psycopg2 or
MySQL+pymysql deployments of<br>
>>>> OpenStack right now?<br>
>>>><br>
>>>> I think what you really want is to change the
defaults we test in<br>
>>>> the gate, which is a different problem.<br>
>>><br>
>>> Because this is really a *new* driver. As you can see
by the<br>
>>> attempted run, it doesn't work with alembic given the
definitions<br>
>>> that neutron has. So it's not like this is currently
compatible<br>
>>> with OpenStack code.<br>
<br>
Well, to fix that, you just need to specify
raise_on_warnings=False<br>
for connection (it's default for mysqldb but not mysql-connector).<br>
I've done it in devstack patch for now, but probably it belongs to<br>
</blockquote>
<br>
this is also semi-my fault as mysqlconnector apparently defaults
this to False now, but for some reason the SQLAlchemy mysqlconnector
dialect is flipping it to True (this dialect was contributed by
MySQL-connector's folks, so not sure why the inconsistency, perhaps
they changed their minds)<br>
<br>
<br>
<br>
<blockquote type="cite">oslo.db.<br>
<br>
>>><br>
>>>><br>
>>>> Thanks, Roman<br>
>>>><br>
>>>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka<br>
>>>> <a class="moz-txt-link-rfc2396E" href="mailto:ihrachys@redhat.com"><ihrachys@redhat.com></a> wrote: Hi all,<br>
>>>><br>
>>>> Multiple projects are suffering from db lock
timeouts due to<br>
>>>> deadlocks deep in mysqldb library that we use to
interact with<br>
>>>> mysql servers. In essence, the problem is due to
missing eventlet<br>
>>>> support in mysqldb module, meaning when a db lock
is encountered,<br>
>>>> the library does not yield to the next green
thread, allowing<br>
>>>> other threads to eventually unlock the grabbed
lock, and instead<br>
>>>> it just blocks the main thread, that eventually
raises timeout<br>
>>>> exception (OperationalError).<br>
>>>><br>
>>>> The failed operation is not retried, leaving
failing request not<br>
>>>> served. In Nova, there is a special retry
mechanism for<br>
>>>> deadlocks, though I think it's more a hack than a
proper fix.<br>
>>>><br>
>>>> Neutron is one of the projects that suffer from
those timeout<br>
>>>> errors a lot. Partly it's due to lack of
discipline in how we do<br>
>>>> nested calls in l3_db and ml2_plugin code, but
that's not<br>
>>>> something to change in foreseeable future, so we
need to find<br>
>>>> another solution that is applicable for Juno.
Ideally, the<br>
>>>> solution should be applicable for Icehouse too to
allow<br>
>>>> distributors to resolve existing deadlocks
without waiting for<br>
>>>> Juno.<br>
>>>><br>
>>>> We've had several discussions and attempts to
introduce a<br>
>>>> solution to the problem. Thanks to oslo.db guys,
we now have more<br>
>>>> or less clear view on the cause of the failures
and how to easily<br>
>>>> fix them. The solution is to switch mysqldb to
something eventlet<br>
>>>> aware. The best candidate is probably MySQL
Connector module that<br>
>>>> is an official MySQL client for Python and that
shows some<br>
>>>> (preliminary) good results in terms of
performance.<br>
>>>><br>
>>>> I've posted a Neutron spec for the switch to the
new client in<br>
>>>> Juno at [1]. Ideally, switch is just a matter of
several fixes to<br>
>>>> oslo.db that would enable full support for the
new driver already<br>
>>>> supported by SQLAlchemy, plus 'connection' string
modified in<br>
>>>> service configuration files, plus documentation
updates to refer<br>
>>>> to the new official way to configure services for
MySQL. The<br>
>>>> database code won't, ideally, require any major
changes, though<br>
>>>> some adaptation for the new client library may be
needed. That<br>
>>>> said, Neutron does not seem to require any
changes, though it was<br>
>>>> revealed that there are some alembic migration
rules in Keystone<br>
>>>> or Glance that need (trivial) modifications.<br>
>>>><br>
>>>> You can see how trivial the switch can be
achieved for a service<br>
>>>> based on example for Neutron [2].<br>
>>>><br>
>>>> While this is a Neutron specific proposal, there
is an obvious<br>
>>>> wish to switch to the new library globally
throughout all the<br>
>>>> projects, to reduce devops burden, among other
things. My vision<br>
>>>> is that, ideally, we switch all projects to the
new library in<br>
>>>> Juno, though we still may leave several projects
for K in case<br>
>>>> any issues arise, similar to the way projects
switched to<br>
>>>> oslo.messaging during two cycles instead of one.
Though looking<br>
>>>> at how easy Neutron can be switched to the new
library, I<br>
>>>> wouldn't expect any issues that would postpone
the switch till<br>
>>>> K.<br>
>>>><br>
>>>> It was mentioned in comments to the spec proposal
that there were<br>
>>>> some discussions at the latest summit around
possible switch in<br>
>>>> context of Nova that revealed some concerns,
though they do not<br>
>>>> seem to be documented anywhere. So if you know
anything about it,<br>
>>>> please comment.<br>
>>>><br>
>>>> So, we'd like to hear from other projects what's
your take on<br>
>>>> that move, whether you see any issues or have
concerns about it.<br>
>>>><br>
>>>> Thanks for your comments, /Ihar<br>
>>>><br>
>>>> [1]: <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/104905/">https://review.openstack.org/#/c/104905/</a>
[2]:<br>
>>>> <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/105209/">https://review.openstack.org/#/c/105209/</a><br>
>>>>><br>
>>>>>
_______________________________________________ OpenStack-dev<br>
>>>>> mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>><br>
>>>><br>
>>>>><br>
_______________________________________________<br>
>>>> OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>><br>
>>><br>
>>>><br>
>>><br>
>>><br>
>>> _______________________________________________
OpenStack-dev<br>
>>> mailing list <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
</blockquote>
<span style="white-space: pre;">>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br>
<br>
<br>
</body>
</html>